diff --git a/.env.example b/.env.example index 1213459f..b9ba79a1 100644 --- a/.env.example +++ b/.env.example @@ -6,18 +6,33 @@ # ⚠️ 所有标注「必填」的项若没配置，`docker compose up` 会直接失败退出，避免把默认/示例值带到生产环境。 # ==================== 数据库（Docker 模式必填） ==================== +# +# ⚠️ Docker 栈已切换到 PostgreSQL 16（此前为 MySQL）。老部署升级前请先读 +# docker-compose.yml 顶部的迁移说明：旧 mysql_data 卷不会被读取，需要先 +# mysqldump 再用 pgloader 等工具导入，或钉在切换前的 tag 上继续用 MySQL。 DB_HOST=localhost -DB_PORT=3306 +DB_PORT=5432 DB_NAME=mateclaw + +# 应用连接账号（最小权限角色，由 docker/postgres/init/10-app-role.sh 首次 +# 初始化时自动创建，仅拥有 mateclaw schema，不是超级用户）。 DB_USERNAME=mateclaw # ⚠️ 必填且请改成强密码（至少 16 位，含大小写+数字+符号）。 # docker-compose.yml 会通过 ${DB_PASSWORD:?} 强制要求此项。 DB_PASSWORD=change-me-strong-user-password -# ⚠️ MySQL root 账号密码，仅用于容器内初始化。请改成与 DB_PASSWORD 不同的强密码。 -DB_ROOT_PASSWORD=change-me-strong-root-password +# ⚠️ PostgreSQL 引导超级账号，仅用于容器内初始化和运维。 +# 请改成与 DB_PASSWORD 不同的强密码。 +DB_ADMIN_USERNAME=mateclaw_admin +DB_ADMIN_PASSWORD=change-me-strong-admin-password + +# ==================== 搜索（可选） ==================== + +# WebSearch 工具的云端搜索 API（可选，二选一或都不配；不配可用 SearXNG sidecar）。 +SERPER_API_KEY= +TAVILY_API_KEY= # ==================== 安全（强烈建议覆盖） ==================== @@ -123,6 +138,27 @@ MATE_WIKI_WATCHER_INTERVAL_MS=300000 # 并记得在 docker-compose.yml 的 volumes 里把对应宿主机目录挂进容器。 MATECLAW_SKILL_WORKSPACE_ROOT= +# ── Skill ZIP 上传大小上限（MB，可选）──────────────────────────── +# 技能包上传/市场安装的应用层上限，默认单文件 1MB、整包 50MB。 +# 解包时整包缓存在内存里，整包上限调多大，单次安装峰值内存就可能吃多大。 +# 同时注意 Spring 层 spring.servlet.multipart 的上限（默认 100MB/200MB）。 +MATECLAW_SKILL_UPLOAD_MAX_ENTRY_SIZE_MB= +MATECLAW_SKILL_UPLOAD_MAX_TOTAL_SIZE_MB= + +# ── Python pip 镜像源（可选）────────────────────────────────────── +# skill 里 Python 脚本缺包时 pip install 走的源。默认用 PyPI 官方源。 +# pip 原生读 PIP_INDEX_URL / PIP_TRUSTED_HOST 环境变量，容器内自动继承。 +# HTTP 源会自动从 URL 推导 PIP_TRUSTED_HOST；自签 HTTPS 需手动填。 +# 互联网加速: https://pypi.tuna.tsinghua.edu.cn/simple +# 局域网私有源: http://192.168.1.100:8080/simple（trusted-host 自动推导） +#PIP_INDEX_URL=https://pypi.tuna.tsinghua.edu.cn/simple +#PIP_TRUSTED_HOST= +# ── 桌面版补充（非 Docker，宿主机直接跑 Java）───────────────────── +# 桌面版不继承上面的 Docker 变量。用 Spring 配置注入 Python 子进程； +# 也可直接设系统环境变量 PIP_INDEX_URL / PIP_TRUSTED_HOST（覆盖更全）。 +#MATECLAW_PIP_INDEX_URL= +#MATECLAW_PIP_TRUSTED_HOST= + # ── Maven 镜像（国内加速）───────────────────────────────────────── # 在中国大陆构建时取消注释，将 Aliyun 仓库优先级提前，大幅提速 mvn 拉包。 # 空值（默认）使用 US Maven Central → Google CDN → Aliyun 的顺序。 diff --git a/.gitignore b/.gitignore index 5e5ab1e8..e946f6b7 100644 --- a/.gitignore +++ b/.gitignore @@ -119,3 +119,8 @@ outputs/ # Ignore stray npm/yarn lockfiles so they are not committed by mistake. package-lock.json yarn.lock + +# Python bytecode caches generated when skill scripts (e.g. skills/*/scripts/*.py) +# are executed. Never commit or sync these. +__pycache__/ +*.pyc diff --git a/README.md b/README.md index 524afd5c..09e12138 100644 --- a/README.md +++ b/README.md @@ -103,6 +103,9 @@ You hire coworkers, not chat boxes. Each one has a **Role**, a **Goal**, a **Bac ### Multimodal creation Text-to-speech · Speech-to-text · Image · Music · Video · 3D. First-class, not add-ons. **Sidecar routing** (1.3.0+) means a text-only main model + an image attachment no longer dead-ends — a configured vision model describes the image, and the main model answers. **Image edit** lands too: refer to an earlier conversation attachment by `msg::` and ask the model to recolor or restyle it. Four **document-generation tools** (`DocxRenderTool` / `XlsxRenderTool` / `PptxRenderTool` / `PdfRenderTool`) render Markdown straight to Office files inside the JVM — no subprocess, no Office install. +### Content Studio (1.8.0+) +A flagship *scene*, not a tool — a seeded "Content Studio" employee turns one sentence into a publishable post: pick-topic → research → draft → illustrate → **de-AI** → lay out → deliver. **WeChat Official Account (公众号)** articles land in your draft box as inline-style HTML with body images uploaded into WeChat; **Xiaohongshu (小红书)** notes package as ≥3 vertical 3:4 cards with an online preview. De-AI-ification runs against a **measurable AI-trace score**; every delivery is compliance-scanned and logged to a **content calendar** that dedups by topic fingerprint. + ### Enterprise-ready RBAC + JWT. **Personal Access Tokens** for headless scripts and CI. **HMAC-SHA-256 outbound webhook signing**. **Distributed Cron lock** so multi-instance deployments don't double-fire. Full audit trail. Flyway-managed schema that auto-heals on upgrade. One JAR to ship. MySQL in production, H2 for dev — nothing to change in your code. @@ -217,6 +220,19 @@ Full docs at **[claw.mate.vip/docs](https://claw.mate.vip/docs)** — setup, arc ## Roadmap +**v1.8.0 (shipped 2026-07-12)** — the employee turns *outward and does a whole job*: **Content Studio**, the first flagship scene built end-to-end on MateClaw's own primitives: + +- **Content Studio — one sentence to a publishable post** — a seeded "Content Studio" employee runs pick-topic → research → draft → illustrate → de-AI → layout → deliver. **WeChat Official Account (公众号)** image-text articles (inline-style HTML → draft box) and **Xiaohongshu (小红书)** image-first notes (≥3 vertical 3:4 cards + online preview) ship first-class +- **De-AI-ification you can measure** — a heuristic AI-trace score (no LLM, deterministic) drives a detect → rewrite → re-check loop, capped at 3 rounds +- **A publish chain hardened for real operation** — body images uploaded into WeChat (no broken external links), AES-GCM-encrypted secrets, reused service + persisted token, retry + Chinese error hints, a guaranteed fallback cover; draft-box-first, publish approval-gated +- **A content calendar that dedups and remembers** — every delivery is compliance-scanned and auto-recorded, a topic fingerprint stops repeat picks, and a read-only Content Calendar page shows drafted/packaged/published/failed +- **The browser agent sees by reference** — an accessibility-tree ref snapshot + interact-by-ref (click the element, not a pixel), real-browser privacy guardrails, and a controlled CDP escape hatch +- **Sharper attention, tighter loops** — attention anchoring & environment awareness (MCP tool provenance + pinned skill constraints + event notifications), a tool-call loop guard, and a post-mutation verify reminder + +Plus: a fast-load pass (initial load down ~78%), a chat context-occupancy panel, cross-KB wikilinks, MCP progress notifications, a Volcano Engine provider, and the public Docker stack on PostgreSQL 16. + +Full story in the [v1.8.0 release notes](https://claw.mate.vip/docs/en/releases/1.8.0). + **v1.7.0 (shipped 2026-07-04)** — a *productionization pass*: once it's in real collaboration, close every loop you can't see, gather, reach, fit, or connect: - **All three approval paths close the loop** — workflow `await_approval` actually pushes to channels and resolves → resumes, the WebChat (API-key) channel can approve/deny and replay, and Feishu/WeCom card clicks resolve workflow approvals directly diff --git a/README_zh.md b/README_zh.md index a23506d5..69aefcaa 100644 --- a/README_zh.md +++ b/README_zh.md @@ -103,6 +103,9 @@ MateClaw 的 **LLM Wiki** 把它消化成结构化页面，页面之间自己长 ### 多模态创作 语音合成 · 语音识别 · 图片 · 音乐 · 视频 · 3D。一等公民，不是附加插件。**多模态旁路**（1.3.0+）让纯文本主模型遇到图片附件时自动调用配置好的视觉模型转描述，主对话保持便宜。**图像编辑**也到位：用 `msg::` 引用会话里更早的某张图，让模型改色、改风格。**4 个文档生成工具**（`DocxRenderTool` / `XlsxRenderTool` / `PptxRenderTool` / `PdfRenderTool`）在 JVM 内把 Markdown 直接渲染成 Office 文件——不 fork 子进程、不依赖 npm、不需要装 Office。 +### 内容工作室（1.8.0+） +一个招牌*场景*,不是工具——预置的「内容工作室」员工把一句话变成可发布成品:选题 → 搜集 → 成文 → 配图 → **去 AI 化** → 排版 → 交付。**微信公众号(公众号)** 文章以内联样式 HTML 躺进你的草稿箱,正文图自动上传进微信;**小红书** 笔记打包成 ≥3 张竖版 3:4 卡片并在线预览。去 AI 化对着一个**可度量的 AI 痕迹评分**跑;每次交付都被合规扫描并记进一个按选题指纹去重的**内容日历**。 + ### 企业就绪 RBAC + JWT。**Personal Access Token** 给无人值守脚本和 CI 用。**Webhook 出站 HMAC-SHA-256 签名**。**Cron 分布式锁**多实例不双发。完整审计事件流。Flyway 管理数据库 schema，升级时自愈。一个 JAR 交付。生产用 MySQL，开发用 H2，代码零改动。 @@ -217,6 +220,19 @@ mateclaw/ ## 路线图 +**v1.8.0（2026-07-12 发布）** — 员工*转向对外、干完一整件活*:**内容工作室**——第一个完全用 MateClaw 自身原子能力端到端搭起来的招牌场景: + +- **内容工作室——一句话到可发布成品** — 预置「内容工作室」员工跑通 选题 → 搜集 → 成文 → 配图 → 去 AI 化 → 排版 → 交付。**微信公众号(公众号)** 图文文章(内联样式 HTML → 草稿箱)与 **小红书** 以图为主图文笔记(≥3 张竖版 3:4 卡片 + 在线预览)首批一等公民 +- **可度量的去 AI 化** — 启发式 AI 痕迹评分(无 LLM、确定性)驱动 检测 → 改写 → 复检 闭环,硬上限 3 轮 +- **为长期投产而加固的发布链** — 正文图上传进微信(不再外链发布即裂)、AES-GCM 加密密钥、服务复用 + token 持久化、重试 + 中文错误提示、兜底封面;草稿箱优先,发表走审批 +- **会去重、会记账的内容日历** — 每次交付都合规扫描 + 自动落台账、选题指纹防重复选题、只读内容日历页展示草稿/已打包/已发布/失败 +- **浏览器 Agent 按引用去看** — 无障碍树 ref 快照 + 按 ref 交互(点元素而非像素)、真实浏览器隐私护栏、受控 CDP 逃生舱 +- **注意力更聚焦、循环更收得住** — 注意力锚定与环境感知(MCP 工具溯源 + skill 约束固定 + 事件通知)、工具调用循环护栏、改动后校验提醒 + +外加:一次快加载优化(初始加载 ↓约 78%)、聊天上下文占用面板、跨知识库 wikilink、MCP 进度通知、火山方舟供应商,以及公开 Docker 栈切到 PostgreSQL 16。 + +完整故事见 [v1.8.0 release notes](https://claw.mate.vip/docs/zh/releases/1.8.0)。 + **v1.7.0（2026-07-04 发布）** — 一次*生产化加固*：把它放进真正的协作里之后，那些看不见、收不拢、够不着、装不下、连不通的地方全补上： - **审批三条链路彻底闭环** — 工作流 `await_approval` 真的推到渠道并 resolve→恢复执行、WebChat（API-Key）渠道能批准/拒绝并重放、飞书/企微点卡片直接 resolve 工作流审批 diff --git a/docker-compose.yml b/docker-compose.yml index 66fbab91..6a8a0411 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,35 +1,57 @@ -version: '3.8' - +# ============================================================================ +# ⚠️ DATABASE ENGINE: PostgreSQL (was MySQL before) +# +# This stack now runs on PostgreSQL 16. Switching the DB engine is a BREAKING +# change for existing deployments: `docker compose up -d` on a host that +# previously ran the MySQL stack starts a FRESH, EMPTY PostgreSQL volume +# (postgres_data). The old `mysql_data` volume is NOT read and the app +# re-seeds default data — existing data is not lost, but it is also NOT +# visible to the new stack. +# +# Treat this as FRESH-INSTALL-ONLY. To carry data across from a MySQL +# deployment, dump the OLD stack BEFORE pulling this change, e.g.: +# docker compose exec mysql mysqldump -u"$DB_USERNAME" -p"$DB_PASSWORD" "$DB_NAME" > mateclaw-mysql.sql +# then load into PostgreSQL with a cross-engine tool such as pgloader — +# there is no automatic MySQL → PostgreSQL migration here. Alternatively, +# keep running MySQL by pinning your checkout to a pre-switch tag and setting +# SPRING_PROFILES_ACTIVE=mysql (the mysql Spring profile remains supported). +# ============================================================================ services: - # MySQL 数据库 + # PostgreSQL 数据库 # # ⚠️ 密码通过环境变量传入，必须从 .env 文件提供。首次部署前： # 1. cp .env.example .env - # 2. 编辑 .env 把 DB_ROOT_PASSWORD / DB_PASSWORD 改成强密码 + # 2. 编辑 .env 把 DB_ADMIN_PASSWORD / DB_PASSWORD 改成强密码 # 未设置会直接在 `docker compose up` 时报错，避免把默认密码带到生产环境。 - mysql: - image: mysql:8.0 - container_name: mateclaw-mysql + postgres: + image: postgres:16 + container_name: mateclaw-postgres restart: unless-stopped environment: - MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD:?DB_ROOT_PASSWORD is required in .env} - MYSQL_DATABASE: ${DB_NAME:-mateclaw} - MYSQL_USER: ${DB_USERNAME:-mateclaw} - MYSQL_PASSWORD: ${DB_PASSWORD:?DB_PASSWORD is required in .env} + # Bootstrap/superuser — owns the cluster, used only for init + admin tasks. + POSTGRES_DB: ${DB_NAME:-mateclaw} + POSTGRES_USER: ${DB_ADMIN_USERNAME:-mateclaw_admin} + POSTGRES_PASSWORD: ${DB_ADMIN_PASSWORD:?DB_ADMIN_PASSWORD is required in .env} + # Least-privilege application role created by the init script below; this + # is the account the server connects with (NOT a superuser). + APP_DB_USERNAME: ${DB_USERNAME:-mateclaw} + APP_DB_PASSWORD: ${DB_PASSWORD:?DB_PASSWORD is required in .env} TZ: Asia/Shanghai - ports: - - "3306:3306" + # No host port on purpose — the DB is only reachable from the compose + # network. For ad-hoc inspection use `docker compose exec postgres psql`, + # or temporarily add: ports: ["127.0.0.1:5432:5432"] volumes: - - mysql_data:/var/lib/mysql - # Schema and seed data are managed by Flyway on application startup. - # Do NOT mount legacy schema.sql / data.sql here — Flyway creates all - # tables from V1 baseline and applies incremental migrations automatically. - command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci + - postgres_data:/var/lib/postgresql/data + # Runs once on first init (empty data dir): creates the restricted app + # role and the mateclaw schema it owns. See docker/postgres/init/. + - ./docker/postgres/init:/docker-entrypoint-initdb.d:ro healthcheck: - test: ["CMD", "mysqladmin", "ping", "-h", "localhost"] + test: ["CMD-SHELL", "pg_isready -U ${DB_ADMIN_USERNAME:-mateclaw_admin} -d ${DB_NAME:-mateclaw}"] interval: 10s timeout: 5s retries: 5 + networks: + - mateclaw-net # SearXNG 搜索引擎（keyless 搜索 provider） # @@ -37,6 +59,8 @@ services: # works out of the box (upstream image ships JSON disabled + Limiter enabled, # both of which silently break mateclaw's SearXNGSearchProvider). # No host bind-mount — edit docker/searxng/settings.yml and rebuild. + # Internal-only: the app reaches it via the compose network. To debug from + # the host, temporarily add: ports: ["127.0.0.1:8088:8080"] searxng: build: context: ./docker/searxng @@ -47,14 +71,14 @@ services: - SEARXNG_SECRET=${SEARXNG_SECRET:-mateclaw-dev-searxng-secret-change-me} - UWSGI_WORKERS=2 - UWSGI_THREADS=4 - ports: - - "8088:8080" healthcheck: # Healthz needs json format, so this also doubles as an integration check. test: ["CMD", "wget", "--spider", "-q", "http://localhost:8080/healthz"] interval: 30s timeout: 5s retries: 3 + networks: + - mateclaw-net # MateClaw 后端服务 mateclaw-server: @@ -66,14 +90,14 @@ services: container_name: mateclaw-server restart: unless-stopped depends_on: - mysql: + postgres: condition: service_healthy searxng: condition: service_healthy environment: - SPRING_PROFILES_ACTIVE: mysql - DB_HOST: mysql - DB_PORT: 3306 + SPRING_PROFILES_ACTIVE: postgres + DB_HOST: postgres + DB_PORT: 5432 DB_NAME: ${DB_NAME:-mateclaw} DB_USERNAME: ${DB_USERNAME:-mateclaw} DB_PASSWORD: ${DB_PASSWORD:?DB_PASSWORD is required in .env} @@ -82,6 +106,7 @@ services: # Settings → Models → Add Provider # Keys are stored in mate_model_provider and hot-reloaded. SERPER_API_KEY: ${SERPER_API_KEY:-} + TAVILY_API_KEY: ${TAVILY_API_KEY:-} JWT_SECRET: ${JWT_SECRET:-} MATECLAW_CORS_ALLOWED_ORIGINS: ${MATECLAW_CORS_ALLOWED_ORIGINS:-} # SearXNG: tell the app where to reach the sidecar container @@ -133,6 +158,18 @@ services: # 已安装的 skill、运行时积累的 LESSONS.md 以及 skill 运行产物，容器重启不丢。 # 内置 skill 仍由 JAR classpath 每次启动现场释放，空卷不会丢内置文件。 MATECLAW_SKILL_WORKSPACE_ROOT: ${MATECLAW_SKILL_WORKSPACE_ROOT:-/app/data/skills} + # Skill ZIP 上传/安装大小上限（MB）。解包过程整包缓存在内存里， + # max-total 调多大，单次安装的峰值内存就可能吃多大。 + MATECLAW_SKILL_UPLOAD_MAX_ENTRY_SIZE_MB: ${MATECLAW_SKILL_UPLOAD_MAX_ENTRY_SIZE_MB:-1} + MATECLAW_SKILL_UPLOAD_MAX_TOTAL_SIZE_MB: ${MATECLAW_SKILL_UPLOAD_MAX_TOTAL_SIZE_MB:-50} + # pip 镜像源配置（可选）。skill 里的 Python 脚本缺包时 pip install 会走这个源。 + # 留空则用 PyPI 默认源（pypi.org）。pip 原生读 PIP_INDEX_URL / PIP_TRUSTED_HOST + # 环境变量，容器内所有进程（JVM、Python 子进程、bash）自动继承，无需额外配置。 + # HTTP 源会自动从 URL 推导 PIP_TRUSTED_HOST；自签 HTTPS 需手动填。 + # - 互联网加速：PIP_INDEX_URL=https://pypi.tuna.tsinghua.edu.cn/simple + # - 局域网私有源：PIP_INDEX_URL=http://192.168.1.100:8080/simple（trusted-host 自动推导） + PIP_INDEX_URL: ${PIP_INDEX_URL:-} + PIP_TRUSTED_HOST: ${PIP_TRUSTED_HOST:-} # Chromium needs a real /dev/shm. Docker defaults to 64MB which causes # SIGBUS / "Target page closed" errors under load. 2GB is the usual # recommendation for Playwright / headless chrome. @@ -141,11 +178,17 @@ services: - "18080:18088" # host:container — app listens on 18088 inside the container - "1455:1455" volumes: - # server_data covers /app/data — H2 DB, wiki-uploads, AND the skill - # workspace (MATECLAW_SKILL_WORKSPACE_ROOT=/app/data/skills above), so a - # single volume persists everything. No separate skills volume needed. + # server_data covers /app/data — wiki-uploads AND the skill workspace + # (MATECLAW_SKILL_WORKSPACE_ROOT=/app/data/skills above), so a single + # volume persists everything. No separate skills volume needed. - server_data:/app/data + networks: + - mateclaw-net volumes: - mysql_data: + postgres_data: server_data: + +networks: + mateclaw-net: + driver: bridge diff --git a/docker/postgres/init/10-app-role.sh b/docker/postgres/init/10-app-role.sh new file mode 100755 index 00000000..388bd9c3 --- /dev/null +++ b/docker/postgres/init/10-app-role.sh @@ -0,0 +1,43 @@ +#!/bin/sh +# ============================================================================ +# Create a least-privilege application role for the MateClaw server. +# +# Runs once, during first container init (empty data dir), as the bootstrap +# superuser (POSTGRES_USER) against POSTGRES_DB. The app role: +# - can log in and CONNECT to the database, +# - owns the `mateclaw` schema (so Flyway can create/alter tables in it), +# - is NOT a superuser and cannot touch other databases/roles. +# +# The server connects as APP_DB_USERNAME / APP_DB_PASSWORD. +# ============================================================================ +set -e + +# Pass credentials as psql variables (-v) rather than interpolating them into +# the SQL text. The quoted heredoc (<<'EOSQL') keeps the body literal, and psql +# does the quoting: :'var' -> safe string literal, :"var" -> safe identifier. +# CREATE ROLE is generated via format(%I, %L) + \gexec so a password containing +# a quote (or an exotic role name) can't break or inject into the statement. +psql -v ON_ERROR_STOP=1 \ + --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" \ + -v app_user="$APP_DB_USERNAME" \ + -v app_pw="$APP_DB_PASSWORD" \ + -v db="$POSTGRES_DB" <<'EOSQL' + SELECT format('CREATE ROLE %I LOGIN PASSWORD %L', :'app_user', :'app_pw') + WHERE NOT EXISTS (SELECT FROM pg_roles WHERE rolname = :'app_user') + \gexec + + -- CONNECT to use the database; CREATE so the role can create schemas in it. + -- CREATE is required because Flyway's init-sql runs CREATE SCHEMA IF NOT + -- EXISTS, and PostgreSQL checks the database-level CREATE privilege *before* + -- the IF NOT EXISTS short-circuit — so even a pre-existing schema is denied + -- without it. Still scoped to this one database; not a cluster superuser. + GRANT CONNECT, CREATE ON DATABASE :"db" TO :"app_user"; + + -- The app owns its schema so Flyway DDL works, without cluster superuser rights. + CREATE SCHEMA IF NOT EXISTS mateclaw AUTHORIZATION :"app_user"; + + -- Default to the app schema on every connection from this role. + ALTER ROLE :"app_user" SET search_path TO mateclaw, public; +EOSQL + +echo "[init] application role '${APP_DB_USERNAME}' and schema 'mateclaw' ready" diff --git a/mateclaw-desktop/package.json b/mateclaw-desktop/package.json index 77eb1c22..f7fbdef7 100644 --- a/mateclaw-desktop/package.json +++ b/mateclaw-desktop/package.json @@ -1,6 +1,6 @@ { "name": "mateclaw-desktop", - "version": "1.7.0", + "version": "1.8.0", "description": "MateClaw Desktop - AI Assistant powered by Spring AI Alibaba", "author": "MateClaw Team", "license": "Apache-2.0", diff --git a/mateclaw-desktop/scripts/README.md b/mateclaw-desktop/scripts/README.md new file mode 100644 index 00000000..b6ed4528 --- /dev/null +++ b/mateclaw-desktop/scripts/README.md @@ -0,0 +1,170 @@ +# MateClaw Desktop - Build & Publish Scripts + +## Scripts + +| Script | Description | +|--------|-------------| +| `build.sh` | Build backend JAR (frontend + Spring Boot) | +| `download-jre.sh` | Download Adoptium JRE 21 for target platform | +| `build-all-platforms.sh` | One-click build for all platforms (macOS + Windows) | +| `publish-github.sh` | Publish release artifacts to GitHub Releases | + +## Quick Start + +### Full Build + Publish (recommended) + +```bash +cd mateclaw-desktop + +# Step 1: Build for all platforms (no publish) +bash scripts/build-all-platforms.sh --all + +# Step 2: Publish to GitHub (via proxy) +bash scripts/publish-github.sh --proxy +``` + +### Step-by-Step Build + +```bash +cd mateclaw-desktop + +# 1. Build backend JAR (frontend + Spring Boot) +bash scripts/build.sh + +# 2. Download JRE for target platform +bash scripts/download-jre.sh mac-arm64 # Apple Silicon +bash scripts/download-jre.sh mac-x64 # Intel Mac +bash scripts/download-jre.sh win-x64 # Windows x64 +bash scripts/download-jre.sh win-arm64 # Windows ARM + +# 3. Build frontend and package +npm run build +npx electron-builder --mac # macOS +npx electron-builder --win --x64 # Windows x64 +npx electron-builder --win --arm64 # Windows ARM +``` + +## build.sh + +Build the backend JAR, includes three steps: + +1. Build Vue 3 frontend to `mateclaw-server/src/main/resources/static` +2. Package Spring Boot fat JAR via Maven +3. Copy JAR to `mateclaw-desktop/resources/app.jar` + +```bash +bash scripts/build.sh +``` + +**Prerequisites:** Node.js (pnpm or npm), Maven (or mvnw) + +## download-jre.sh + +Download Adoptium JRE 21 for the target platform. + +```bash +# Auto-detect current platform +bash scripts/download-jre.sh + +# Specify platform +bash scripts/download-jre.sh mac-arm64 +bash scripts/download-jre.sh mac-x64 +bash scripts/download-jre.sh win-x64 +bash scripts/download-jre.sh win-arm64 +``` + +JRE will be saved to `resources/jre/{os}-{arch}/`. + +## build-all-platforms.sh + +Orchestrates the full build pipeline: JAR build -> JRE download -> frontend compile -> electron-builder package. + +```bash +bash scripts/build-all-platforms.sh --all # macOS + Windows +bash scripts/build-all-platforms.sh --mac-only # macOS only +bash scripts/build-all-platforms.sh --win-only # Windows only +``` + +Build artifacts are output to `release/` directory. + +## publish-github.sh + +Publish build artifacts from `release/` to GitHub Releases via `gh` CLI. + +### Basic Usage + +```bash +bash scripts/publish-github.sh # Direct upload +bash scripts/publish-github.sh --draft # Create draft release +bash scripts/publish-github.sh --tag=v1.1.0 # Custom tag +``` + +### With Proxy (for slow or restricted networks) + +```bash +bash scripts/publish-github.sh --proxy # Default proxy: 127.0.0.1:7890 +bash scripts/publish-github.sh --proxy=192.168.1.1:8080 # Custom proxy +bash scripts/publish-github.sh --proxy --draft # Draft + proxy +``` + +### Retry Failed Upload + +```bash +bash scripts/publish-github.sh --proxy --retry # Delete old release, re-upload +bash scripts/publish-github.sh --proxy --retry --draft # Retry as draft +``` + +### All Options + +| Option | Description | +|--------|-------------| +| `--proxy` | Use proxy `127.0.0.1:7890` for GitHub upload | +| `--proxy=host:port` | Use custom proxy address | +| `--draft` | Create as draft release | +| `--tag=vX.Y.Z` | Custom tag (default: `v{version}` from package.json) | +| `--retry` | Delete existing release for this tag before re-uploading | + +**Prerequisites:** [gh CLI](https://cli.github.com/) installed and authenticated (`gh auth login`) + +## npm Scripts + +These scripts are also available as npm commands: + +```bash +npm run setup:jar # build.sh +npm run setup:jre # download-jre.sh +npm run setup # build.sh + download-jre.sh +npm run package:mac # Build frontend + electron-builder --mac +npm run package:win # Build frontend + electron-builder --win +npm run package:all # build-all-platforms.sh --all +npm run publish:github # publish-github.sh +npm run publish:github:draft # publish-github.sh --draft +``` + +## Typical Workflow + +``` +build.sh download-jre.sh + | | + v v +resources/app.jar resources/jre/{os}-{arch}/ + | | + +--------- electron-builder ---+ + | + v + release/ + ├── MateClaw_1.0.0_arm64.dmg + ├── MateClaw_1.0.0_x64.dmg + ├── MateClaw_1.0.0_arm64.zip + ├── MateClaw_1.0.0_x64.zip + ├── MateClaw_1.0.0_arm64_Setup.exe + ├── MateClaw_1.0.0_x64_Setup.exe + ├── latest-mac.yml + └── latest.yml + | + v + publish-github.sh + | + v + GitHub Releases +``` diff --git a/mateclaw-desktop/scripts/__tests__/trim-playwright-driver.test.cjs b/mateclaw-desktop/scripts/__tests__/trim-playwright-driver.test.cjs new file mode 100644 index 00000000..2167f258 --- /dev/null +++ b/mateclaw-desktop/scripts/__tests__/trim-playwright-driver.test.cjs @@ -0,0 +1,151 @@ +const assert = require('node:assert/strict') +const fs = require('node:fs') +const os = require('node:os') +const path = require('node:path') +const test = require('node:test') + +const { + createZipBuffer, + readZipEntries, + resolveDriverDirectory, + trimDriverBundleInAppJar, + ZIP_STORED, + default: afterPack, +} = require('../trim-playwright-driver.cjs') + +test('resolveDriverDirectory maps electron platform and arch to Playwright driver folder', () => { + assert.equal(resolveDriverDirectory('darwin', 'x64'), 'driver/mac') + assert.equal(resolveDriverDirectory('darwin', 'arm64'), 'driver/mac-arm64') + assert.equal(resolveDriverDirectory('linux', 'x64'), 'driver/linux') + assert.equal(resolveDriverDirectory('linux', 'arm64'), 'driver/linux-arm64') + assert.equal(resolveDriverDirectory('win32', 'x64'), 'driver/win32_x64') + assert.equal(resolveDriverDirectory('win32', 'arm64'), 'driver/win32_x64') + assert.deepEqual(resolveDriverDirectory('darwin', 'universal'), ['driver/mac', 'driver/mac-arm64']) +}) + +test('trimDriverBundleInAppJar keeps only the target driver and preserves nested jar as STORED', async () => { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'mateclaw-driver-trim-')) + const appJarPath = path.join(tmp, 'app.jar') + + const driverBundle = createZipBuffer([ + entry('META-INF/MANIFEST.MF', 'Manifest-Version: 1.0\n'), + entry('driver/mac/node', 'mac'), + entry('driver/mac-arm64/node', 'mac-arm64'), + entry('driver/linux/node', 'linux'), + entry('driver/linux-arm64/node', 'linux-arm64'), + entry('driver/win32_x64/node.exe', 'win32'), + entry('com/microsoft/playwright/Driver.class', 'class'), + ]) + + const appJar = createZipBuffer([ + entry('BOOT-INF/classpath.idx', '- "BOOT-INF/lib/driver-bundle-1.52.0.jar"\n'), + entry('BOOT-INF/lib/driver-bundle-1.52.0.jar', driverBundle, ZIP_STORED), + entry('BOOT-INF/lib/other.jar', 'other', ZIP_STORED), + ]) + + fs.writeFileSync(appJarPath, appJar) + + const result = await trimDriverBundleInAppJar(appJarPath, 'driver/mac-arm64') + assert.equal(result.removedDriverEntries, 4) + + const outerEntries = readZipEntries(fs.readFileSync(appJarPath)) + const nestedEntry = outerEntries.find((item) => item.name === 'BOOT-INF/lib/driver-bundle-1.52.0.jar') + assert.equal(nestedEntry.method, ZIP_STORED) + + const innerEntries = readZipEntries(nestedEntry.data) + const names = innerEntries.map((item) => item.name).sort() + + assert.deepEqual(names, [ + 'META-INF/MANIFEST.MF', + 'com/microsoft/playwright/Driver.class', + 'driver/mac-arm64/node', + ]) +}) + +test('trimDriverBundleInAppJar throws when driver-bundle is missing', async () => { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'mateclaw-driver-trim-')) + const appJarPath = path.join(tmp, 'app.jar') + + fs.writeFileSync(appJarPath, createZipBuffer([ + entry('BOOT-INF/lib/other.jar', 'other', ZIP_STORED), + ])) + + await assert.rejects( + () => trimDriverBundleInAppJar(appJarPath, 'driver/mac-arm64'), + /Playwright driver-bundle jar not found/ + ) +}) + +test('trimDriverBundleInAppJar throws when no target driver entries are kept', async () => { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'mateclaw-driver-trim-')) + const appJarPath = path.join(tmp, 'app.jar') + fs.writeFileSync(appJarPath, createAppJarWithDriverBundle()) + + await assert.rejects( + () => trimDriverBundleInAppJar(appJarPath, 'driver/mac-arm64-v2'), + /No entries kept under driver\/mac-arm64-v2/ + ) +}) + +test('trimDriverBundleInAppJar can keep both mac drivers for universal builds', async () => { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'mateclaw-driver-trim-')) + const appJarPath = path.join(tmp, 'app.jar') + fs.writeFileSync(appJarPath, createAppJarWithDriverBundle()) + + const result = await trimDriverBundleInAppJar(appJarPath, ['driver/mac', 'driver/mac-arm64']) + assert.equal(result.keptDriverEntries, 2) + + const outerEntries = readZipEntries(fs.readFileSync(appJarPath)) + const nestedEntry = outerEntries.find((item) => item.name === 'BOOT-INF/lib/driver-bundle-1.52.0.jar') + const driverNames = readZipEntries(nestedEntry.data) + .filter((item) => item.name.startsWith('driver/')) + .map((item) => item.name) + .sort() + + assert.deepEqual(driverNames, [ + 'driver/mac-arm64/node', + 'driver/mac/node', + ]) +}) + +test('afterPack throws when app.jar is missing', async () => { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'mateclaw-driver-trim-')) + + await assert.rejects( + () => afterPack({ + appOutDir: tmp, + arch: 'arm64', + electronPlatformName: 'darwin', + packager: { + appInfo: { productFilename: 'MateClaw' }, + }, + }), + /app\.jar not found/ + ) +}) + +function entry(name, data, method) { + return { + name, + data: Buffer.isBuffer(data) ? data : Buffer.from(data), + method, + } +} + +function createAppJarWithDriverBundle() { + const driverBundle = createZipBuffer([ + entry('META-INF/MANIFEST.MF', 'Manifest-Version: 1.0\n'), + entry('driver/mac/node', 'mac'), + entry('driver/mac-arm64/node', 'mac-arm64'), + entry('driver/linux/node', 'linux'), + entry('driver/linux-arm64/node', 'linux-arm64'), + entry('driver/win32_x64/node.exe', 'win32'), + entry('com/microsoft/playwright/Driver.class', 'class'), + ]) + + return createZipBuffer([ + entry('BOOT-INF/classpath.idx', '- "BOOT-INF/lib/driver-bundle-1.52.0.jar"\n'), + entry('BOOT-INF/lib/driver-bundle-1.52.0.jar', driverBundle, ZIP_STORED), + entry('BOOT-INF/lib/other.jar', 'other', ZIP_STORED), + ]) +} diff --git a/mateclaw-desktop/scripts/branding.cjs b/mateclaw-desktop/scripts/branding.cjs new file mode 100644 index 00000000..35aa5a39 --- /dev/null +++ b/mateclaw-desktop/scripts/branding.cjs @@ -0,0 +1,153 @@ +/** + * scripts/branding.cjs — Vite plugin for build-time white-label branding. + * + * Reads brand settings from branding.config.json (or BRAND_* env overrides) + * and replaces hardcoded "MateClaw" strings in all built files — source code + * stays untouched. + * + * Supported env overrides: + * BRAND_NAME, BRAND_TAGLINE, BRAND_TEAM, BRAND_COPYRIGHT, + * BRAND_APP_ID, BRAND_GITHUB_URL, BRAND_LOGO_FILE + */ +'use strict' + +const fs = require('fs') +const path = require('path') + +function loadBrandConfig(rootDir) { + const configPath = path.join(rootDir, 'branding.config.json') + let config = {} + if (fs.existsSync(configPath)) { + config = JSON.parse(fs.readFileSync(configPath, 'utf-8')) + } + + // Env vars override the config file. + const env = process.env + return { + name: env.BRAND_NAME || config.name || 'MateClaw', + tagline: env.BRAND_TAGLINE || config.tagline || 'AI Personal Assistant', + team: env.BRAND_TEAM || config.team || 'MateClaw Team', + copyright: env.BRAND_COPYRIGHT || config.copyright || 'Copyright © 2026 MateClaw Team', + appId: env.BRAND_APP_ID || config.appId || 'vip.mate.mateclaw', + githubUrl: env.BRAND_GITHUB_URL || config.githubUrl || 'https://github.com/matevip/mateclaw', + logoFile: env.BRAND_LOGO_FILE || config.logoFile || 'mateclaw_logo_s.png', + } +} + +/** + * Build the string-replacement table. + * + * Order matters: longer/more-specific patterns are replaced first to avoid + * partial matches (e.g. "MateClaw Team" before "MateClaw"). + */ +function buildReplacements(brand) { + const replacements = [] + + // 1. Copyright line (most specific) + replacements.push([ + 'Copyright © 2026 MateClaw Team', + brand.copyright, + ]) + + // 2. Team name + replacements.push(['MateClaw Team', brand.team]) + + // 3. GitHub URLs + replacements.push([ + 'https://github.com/matevip/mateclaw/issues', + brand.githubUrl + '/issues', + ]) + replacements.push([ + 'https://github.com/matevip/mateclaw', + brand.githubUrl, + ]) + + // 4. Logo file path + replacements.push([ + 'mateclaw_logo_s.png', + brand.logoFile, + ]) + + // 5. Tagline + replacements.push([ + 'AI Personal Assistant', + brand.tagline, + ]) + + // 6. Split-span brand name in App.vue template: + // MateClaw + // Replace the inner text so styling classes are preserved but the text + // changes. We split the brand name: first half gets "mate" class, second + // half gets "claw" class. If it's a single word, it all goes in "mate". + var half = Math.ceil(brand.name.length / 2) + var firstPart = brand.name.slice(0, half) + var secondPart = brand.name.slice(half) + replacements.push([ + '>MateClaw<', + '>' + firstPart + '' + secondPart + '<', + ]) + + // 7. Brand name (catch-all, must come last) + // Only replace the exact word "MateClaw", not "mateclaw" (lowercase, + // which is used in H2 database paths and Spring Boot properties that + // are coupled with the server and must NOT change). + replacements.push(['MateClaw', brand.name]) + + return replacements +} + +function applyReplacements(code, replacements) { + var result = code + for (var i = 0; i < replacements.length; i++) { + var from = replacements[i][0] + var to = replacements[i][1] + // Use split/join for reliable literal string replacement (no regex + // escaping issues). + result = result.split(from).join(to) + } + return result +} + +/** + * Vite plugin entry point. + * + * Usage in vite.config.ts: + * import { brandingPlugin } from './scripts/branding.cjs' + * plugins: [brandingPlugin()] + */ +function brandingPlugin(options) { + options = options || {} + var rootDir = options.rootDir || process.cwd() + var brand = loadBrandConfig(rootDir) + var replacements = buildReplacements(brand) + + var isDefault = + brand.name === 'MateClaw' && + brand.tagline === 'AI Personal Assistant' && + brand.team === 'MateClaw Team' + + if (!isDefault) { + console.log('[branding] White-label build: "' + brand.name + '" (tagline: "' + brand.tagline + '")') + } + + return { + name: 'mateclaw-branding', + enforce: 'pre', + + // Transform JS/TS/Vue source before compilation + transform: function (code, id) { + if (id.indexOf('node_modules') !== -1) return null + // Only process source files that might contain brand strings. + if (!/\.(ts|js|vue|html|css|cjs|mjs)$/.test(id)) return null + var result = applyReplacements(code, replacements) + return result !== code ? { code: result, map: null } : null + }, + + // Transform index.html + transformIndexHtml: function (html) { + return applyReplacements(html, replacements) + }, + } +} + +module.exports = { brandingPlugin: brandingPlugin, loadBrandConfig: loadBrandConfig, buildReplacements: buildReplacements } diff --git a/mateclaw-desktop/scripts/build-all-platforms.sh b/mateclaw-desktop/scripts/build-all-platforms.sh new file mode 100755 index 00000000..14f388eb --- /dev/null +++ b/mateclaw-desktop/scripts/build-all-platforms.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# +# scripts/build-all-platforms.sh — Build MateClaw desktop packages for all +# platforms (macOS + Windows) in the specified build mode. +# +# Usage: +# scripts/build-all-platforms.sh --all # local mode (default), both platforms +# scripts/build-all-platforms.sh --local # local mode, both platforms +# scripts/build-all-platforms.sh --remote # remote mode, both platforms +# +set -euo pipefail + +MODE="--all" +case "${1:-}" in + --all) MODE="local" ;; + --local) MODE="local" ;; + --remote) MODE="remote" ;; + *) echo "Usage: $0 [--all|--local|--remote]"; exit 1 ;; +esac + +echo "==> Building all platforms, BUILD_MODE=$MODE" + +if [ "$MODE" = "remote" ]; then + echo "==> macOS (remote/lite)" + BUILD_MODE=remote npx electron-builder --mac + echo "==> Windows (remote/lite)" + BUILD_MODE=remote npx electron-builder --win +else + echo "==> macOS (local/full)" + BUILD_MODE=local npx electron-builder --mac + echo "==> Windows (local/full)" + BUILD_MODE=local npx electron-builder --win +fi + +echo "==> All builds complete." diff --git a/mateclaw-desktop/scripts/build.sh b/mateclaw-desktop/scripts/build.sh new file mode 100755 index 00000000..35d6ee73 --- /dev/null +++ b/mateclaw-desktop/scripts/build.sh @@ -0,0 +1,30 @@ +#!/usr/bin/env bash +# +# scripts/build.sh — Build the MateClaw Spring Boot backend JAR and place it +# at resources/app.jar so electron-builder can bundle it into the desktop app. +# +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +SERVER_DIR="$(cd "$PROJECT_ROOT/../mateclaw-server" && pwd)" +RESOURCES_DIR="$PROJECT_ROOT/resources" + +echo "==> Building mateclaw-server JAR from $SERVER_DIR" + +# Build the Spring Boot fat JAR (skip tests for packaging speed) +cd "$SERVER_DIR" +mvn clean package -DskipTests -Dmaven.test.skip=true -q + +# Locate the built JAR +JAR_FILE=$(ls "$SERVER_DIR"/target/mateclaw-server-*.jar 2>/dev/null | head -1) +if [ -z "$JAR_FILE" ]; then + echo "ERROR: Could not find built JAR in $SERVER_DIR/target/" + exit 1 +fi + +echo "==> Copying $JAR_FILE → $RESOURCES_DIR/app.jar" +mkdir -p "$RESOURCES_DIR" +cp "$JAR_FILE" "$RESOURCES_DIR/app.jar" + +echo "==> Done. JAR size: $(du -h "$RESOURCES_DIR/app.jar" | cut -f1)" diff --git a/mateclaw-desktop/scripts/download-jre.sh b/mateclaw-desktop/scripts/download-jre.sh new file mode 100755 index 00000000..5fca8087 --- /dev/null +++ b/mateclaw-desktop/scripts/download-jre.sh @@ -0,0 +1,86 @@ +#!/usr/bin/env bash +# +# scripts/download-jre.sh — Download Eclipse Temurin JRE 21 for the current +# macOS architecture (or both) and extract into resources/jre//. +# +# Usage: +# scripts/download-jre.sh # auto-detect current arch +# scripts/download-jre.sh arm64 # arm64 only +# scripts/download-jre.sh x64 # x64 only +# scripts/download-jre.sh all # both arches +# +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +JRE_DIR="$PROJECT_ROOT/resources/jre" + +# Temurin 21 (LTS) JRE downloads via Adoptium API +ADOPTIUM_BASE="https://api.adoptium.net/v3/binary/latest/21/ga/mac" + +download_and_extract() { + local arch="$1" + local folder="$2" + local url="$ADOPTIUM_BASE/$arch/jre/hotspot/normal/eclipse?project=jdk" + local tmpfile="$JRE_DIR/jre-mac-$arch.tar.gz" + + echo "==> Downloading Temurin 21 JRE for macOS $arch" + mkdir -p "$JRE_DIR" + curl -L --fail -o "$tmpfile" "$url" + + echo "==> Extracting to $JRE_DIR/$folder" + rm -rf "$JRE_DIR/$folder" + mkdir -p "$JRE_DIR/$folder" + + # Temurin macOS tar.gz extracts to: jdk-21.x.x+jre/Contents/Home/... + # We want $folder/Contents/Home/... so move the inner Contents up. + local extract_tmp="$JRE_DIR/.tmp-$arch" + rm -rf "$extract_tmp" + mkdir -p "$extract_tmp" + tar -xzf "$tmpfile" -C "$extract_tmp" + + # Find the extracted top-level directory and move its Contents + local extracted_dir + extracted_dir=$(find "$extract_tmp" -maxdepth 1 -type d -name "jdk-*" | head -1) + if [ -z "$extracted_dir" ]; then + # Fallback: some tarballs extract Contents directly + if [ -d "$extract_tmp/Contents" ]; then + mv "$extract_tmp/Contents" "$JRE_DIR/$folder/Contents" + else + echo "ERROR: Could not find extracted JDK directory" + exit 1 + fi + else + mv "$extracted_dir/Contents" "$JRE_DIR/$folder/Contents" + fi + + rm -rf "$extract_tmp" "$tmpfile" + + # Verify java binary exists + local java_bin="$JRE_DIR/$folder/Contents/Home/bin/java" + if [ -f "$java_bin" ]; then + echo "==> OK: $java_bin" + else + echo "ERROR: java binary not found at $java_bin" + exit 1 + fi +} + +TARGET="${1:-auto}" +if [ "$TARGET" = "auto" ]; then + case "$(uname -m)" in + arm64) TARGET="arm64" ;; + x86_64) TARGET="x64" ;; + *) echo "Unsupported arch: $(uname -m)"; exit 1 ;; + esac +fi + +case "$TARGET" in + arm64) download_and_extract "aarch64" "mac-arm64" ;; + x64) download_and_extract "x64" "mac-x64" ;; + all) download_and_extract "aarch64" "mac-arm64" + download_and_extract "x64" "mac-x64" ;; + *) echo "Usage: $0 [arm64|x64|all]"; exit 1 ;; +esac + +echo "==> JRE setup complete." diff --git a/mateclaw-desktop/scripts/publish-github.sh b/mateclaw-desktop/scripts/publish-github.sh new file mode 100755 index 00000000..777db90a --- /dev/null +++ b/mateclaw-desktop/scripts/publish-github.sh @@ -0,0 +1,324 @@ +#!/usr/bin/env bash +# +# Publish MateClaw Desktop release artifacts to GitHub Releases +# +# Prerequisites: +# - gh CLI installed and authenticated (gh auth login) +# - Build artifacts exist in release/ directory (run build-all-platforms.sh first) +# +# Usage: +# bash scripts/publish-github.sh # Create release (direct) +# bash scripts/publish-github.sh --proxy # Create release via proxy (127.0.0.1:7890) +# bash scripts/publish-github.sh --proxy=host:port # Create release via custom proxy +# bash scripts/publish-github.sh --draft # Create draft release +# bash scripts/publish-github.sh --tag=v1.1.0 # Custom tag (default: v{version} from package.json) +# bash scripts/publish-github.sh --retry # Delete existing release and re-upload +# bash scripts/publish-github.sh --proxy --retry --draft # Combine options +# + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +DESKTOP_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" +RELEASE_DIR="$DESKTOP_DIR/release" + +# ─── Parse arguments ─────────────────────────────────────────────────────── + +DRAFT="" +CUSTOM_TAG="" +USE_PROXY="" +PROXY_ADDR="127.0.0.1:7890" +RETRY=false +NOTES_FILE_OVERRIDE="" +NOTES_LANG="en" + +for arg in "$@"; do + case "$arg" in + --draft) DRAFT="--draft" ;; + --tag=*) CUSTOM_TAG="${arg#--tag=}" ;; + --proxy) USE_PROXY=true ;; + --proxy=*) USE_PROXY=true; PROXY_ADDR="${arg#--proxy=}" ;; + --retry) RETRY=true ;; + --notes-file=*) NOTES_FILE_OVERRIDE="${arg#--notes-file=}" ;; + --notes-lang=*) NOTES_LANG="${arg#--notes-lang=}" ;; + -h|--help) + echo "Usage: $0 [--draft] [--tag=vX.Y.Z] [--proxy[=host:port]] [--retry] [--notes-file=path] [--notes-lang=en|zh]" + echo "" + echo "Options:" + echo " --draft Create as draft release" + echo " --tag=vX.Y.Z Custom tag (default: v{version} from package.json)" + echo " --proxy Use proxy 127.0.0.1:7890 for uploading to GitHub" + echo " --proxy=host:port Use custom proxy address" + echo " --retry Delete existing release for this tag before uploading" + echo " --notes-file=path Use this markdown file as release notes (highest priority)" + echo " --notes-lang=en|zh Preferred language for auto-detected notes (default: en)" + echo "" + echo "Release notes resolution order:" + echo " 1. --notes-file=path (explicit override)" + echo " 2. ../docs/{notes-lang}/releases/X.Y.Z.md (curated, preferred lang)" + echo " 3. ../docs/{other-lang}/releases/X.Y.Z.md (curated, fallback lang)" + echo " 4. Tag annotation (if multi-line) (legacy)" + echo " 5. Tagged commit message body (legacy)" + echo " 6. --generate-notes (GitHub auto)" + echo "" + echo "Examples:" + echo " # Step 1: Build locally (no publish)" + echo " bash scripts/build-all-platforms.sh --all" + echo "" + echo " # Step 2: Publish to GitHub" + echo " bash scripts/publish-github.sh --proxy # via proxy, en notes" + echo " bash scripts/publish-github.sh --proxy --notes-lang=zh # zh notes" + echo " bash scripts/publish-github.sh --proxy --draft # draft (preview)" + echo " bash scripts/publish-github.sh --proxy --retry # delete + re-upload" + echo " bash scripts/publish-github.sh --notes-file=NOTES.md # custom notes file" + echo "" + echo "Run build-all-platforms.sh first to generate artifacts." + exit 0 + ;; + *) + echo "Unknown option: $arg" + echo "Usage: $0 [--draft] [--tag=vX.Y.Z] [--proxy[=host:port]] [--retry] [--notes-file=path] [--notes-lang=en|zh]" + exit 1 + ;; + esac +done + +if [ "$NOTES_LANG" != "en" ] && [ "$NOTES_LANG" != "zh" ]; then + echo "❌ --notes-lang must be 'en' or 'zh' (got: $NOTES_LANG)" + exit 1 +fi + +# ─── Setup proxy ─────────────────────────────────────────────────────────── + +if [ "$USE_PROXY" = true ]; then + export https_proxy="http://${PROXY_ADDR}" + export http_proxy="http://${PROXY_ADDR}" + export HTTPS_PROXY="http://${PROXY_ADDR}" + export HTTP_PROXY="http://${PROXY_ADDR}" + echo "🌐 Proxy enabled: ${PROXY_ADDR}" + echo "" +fi + +# ─── Check prerequisites ────────────────────────────────────────────────── + +if ! command -v gh &>/dev/null; then + echo "❌ gh CLI not found. Install: https://cli.github.com/" + exit 1 +fi + +if ! gh auth status &>/dev/null; then + echo "❌ gh not authenticated. Run: gh auth login" + exit 1 +fi + +# ─── Read version from package.json ─────────────────────────────────────── + +VERSION=$(node -p "require('$DESKTOP_DIR/package.json').version") +TAG="${CUSTOM_TAG:-v${VERSION}}" + +echo "╔════════════════════════════════════════════════════════╗" +echo "║ MateClaw Desktop - Publish to GitHub ║" +echo "╚════════════════════════════════════════════════════════╝" +echo "" +echo " Version: $VERSION" +echo " Tag: $TAG" +echo " Mode: ${DRAFT:-release}" +echo " Proxy: ${USE_PROXY:+${PROXY_ADDR}}${USE_PROXY:-off}" +echo " Retry: $RETRY" +echo "" + +# ─── Retry: delete existing release ─────────────────────────────────────── + +if [ "$RETRY" = true ]; then + echo "🔄 Checking for existing release $TAG ..." + if gh release view "$TAG" &>/dev/null; then + echo " Found existing release, deleting..." + gh release delete "$TAG" --yes --cleanup-tag 2>/dev/null || gh release delete "$TAG" --yes 2>/dev/null + echo " ✅ Old release deleted." + else + echo " No existing release found, proceeding." + fi + echo "" +fi + +# ─── Collect release artifacts ───────────────────────────────────────────── + +if [ ! -d "$RELEASE_DIR" ]; then + echo "❌ Release directory not found: $RELEASE_DIR" + echo " Run build-all-platforms.sh first." + exit 1 +fi + +ARTIFACTS=() + +# macOS artifacts +for pattern in "MateClaw*.dmg" "MateClaw*.zip"; do + while IFS= read -r -d '' f; do + ARTIFACTS+=("$f") + done < <(find "$RELEASE_DIR" -maxdepth 1 -name "$pattern" -print0 2>/dev/null) +done + +# Windows artifacts (skip the architecture-merged installer — only ship per-arch builds) +while IFS= read -r -d '' f; do + ARTIFACTS+=("$f") +done < <(find "$RELEASE_DIR" -maxdepth 1 -name "MateClaw*Setup*.exe" \ + ! -name "MateClaw_${VERSION}_Setup.exe" -print0 2>/dev/null) + +# Linux artifacts +while IFS= read -r -d '' f; do + ARTIFACTS+=("$f") +done < <(find "$RELEASE_DIR" -maxdepth 1 -name "MateClaw*.AppImage" -print0 2>/dev/null) + +# Auto-update metadata files +for pattern in "latest-mac.yml" "latest.yml" "latest-linux.yml"; do + if [ -f "$RELEASE_DIR/$pattern" ]; then + ARTIFACTS+=("$RELEASE_DIR/$pattern") + fi +done + +# blockmap files (for differential updates) — also skip merged installer's blockmap +while IFS= read -r -d '' f; do + ARTIFACTS+=("$f") +done < <(find "$RELEASE_DIR" -maxdepth 1 -name "*.blockmap" \ + ! -name "MateClaw_${VERSION}_Setup.exe.blockmap" -print0 2>/dev/null) + +if [ ${#ARTIFACTS[@]} -eq 0 ]; then + echo "❌ No release artifacts found in $RELEASE_DIR" + echo " Run build-all-platforms.sh first." + exit 1 +fi + +echo " Artifacts to upload:" +TOTAL_SIZE=0 +for f in "${ARTIFACTS[@]}"; do + SIZE_BYTES=$(stat -f%z "$f" 2>/dev/null || stat -c%s "$f" 2>/dev/null || echo 0) + SIZE_HUMAN=$(du -h "$f" | cut -f1) + TOTAL_SIZE=$((TOTAL_SIZE + SIZE_BYTES)) + echo " • $(basename "$f") ($SIZE_HUMAN)" +done +TOTAL_HUMAN=$(echo "$TOTAL_SIZE" | awk '{ + if ($1 >= 1073741824) printf "%.1f GB", $1/1073741824; + else if ($1 >= 1048576) printf "%.0f MB", $1/1048576; + else printf "%.0f KB", $1/1024; +}') +echo "" +echo " Total: $TOTAL_HUMAN" +echo "" + +# ─── Resolve release notes (priority chain) ─────────────────────────────── +# 1. --notes-file=path (explicit override) +# 2. ../docs/{notes-lang}/releases/X.Y.Z.md (curated, preferred lang) +# 3. ../docs/{other-lang}/releases/X.Y.Z.md (curated, fallback lang) +# 4. Tag annotation (if multi-line) (legacy fallback) +# 5. Tagged commit message body (legacy fallback) +# 6. --generate-notes (GitHub auto) + +NOTES_FILE=$(mktemp) +trap "rm -f '$NOTES_FILE'" EXIT + +REPO_ROOT="$(cd "$DESKTOP_DIR/.." && pwd)" +NOTES_LANG_OTHER=$([ "$NOTES_LANG" = "en" ] && echo "zh" || echo "en") +NOTES_PRIMARY="$REPO_ROOT/docs/${NOTES_LANG}/releases/${VERSION}.md" +NOTES_FALLBACK="$REPO_ROOT/docs/${NOTES_LANG_OTHER}/releases/${VERSION}.md" + +if [ -n "$NOTES_FILE_OVERRIDE" ]; then + if [ ! -f "$NOTES_FILE_OVERRIDE" ]; then + echo "❌ --notes-file path not found: $NOTES_FILE_OVERRIDE" + exit 1 + fi + cat "$NOTES_FILE_OVERRIDE" > "$NOTES_FILE" + echo "📝 Using --notes-file: $NOTES_FILE_OVERRIDE" +elif [ -f "$NOTES_PRIMARY" ]; then + cat "$NOTES_PRIMARY" > "$NOTES_FILE" + echo "📝 Using docs/${NOTES_LANG}/releases/${VERSION}.md as release notes." +elif [ -f "$NOTES_FALLBACK" ]; then + cat "$NOTES_FALLBACK" > "$NOTES_FILE" + echo "📝 Using docs/${NOTES_LANG_OTHER}/releases/${VERSION}.md as release notes (preferred '$NOTES_LANG' not found)." +else + # Legacy fallback: tag annotation, then commit body + TAG_MSG=$(git tag -l --format='%(contents)' "$TAG" 2>/dev/null | sed '/^$/d') + if [ -n "$TAG_MSG" ] && [ "$(echo "$TAG_MSG" | wc -l)" -gt 2 ]; then + echo "$TAG_MSG" > "$NOTES_FILE" + echo "📝 Using tag annotation as release notes." + else + COMMIT_MSG=$(git log -1 --format='%B' "$TAG" 2>/dev/null | tail -n +2 | sed '/^$/d') + if [ -n "$COMMIT_MSG" ]; then + echo "$COMMIT_MSG" > "$NOTES_FILE" + echo "📝 Using commit message as release notes." + else + echo "⚠️ No release notes found (no docs/X.Y.Z.md, tag annotation, or commit body), using --generate-notes." + fi + fi +fi +echo "" + +# ─── Rewrite latest.yml so it does not reference the merged installer ───── +# +# electron-builder generates a multi-arch merged Windows installer +# (MateClaw_X.Y.Z_Setup.exe, ~600 MB) alongside the per-arch installers +# (MateClaw_X.Y.Z_x64_Setup.exe, MateClaw_X.Y.Z_arm64_Setup.exe). The +# merged build is too large to be worth uploading, but the default +# latest.yml top-level `path:` and `files[0]` both point at it. If we +# upload latest.yml as-is while skipping the merged exe, electron-updater +# on Windows hits 404 on every check and the whole update path breaks. +# +# Fix: rewrite latest.yml in place so `path` / `sha512` / `files[]` only +# reference assets that are actually being uploaded. Use x64 as the +# default top-level target (arm64 clients still match via files[]). + +LATEST_YML="$RELEASE_DIR/latest.yml" +MERGED_EXE_NAME="MateClaw_${VERSION}_Setup.exe" +X64_EXE_NAME="MateClaw_${VERSION}_x64_Setup.exe" +ARM64_EXE_NAME="MateClaw_${VERSION}_arm64_Setup.exe" + +if [ -f "$LATEST_YML" ]; then + echo "🧹 Rewriting latest.yml to drop the merged installer ..." + (cd "$DESKTOP_DIR" && node -e " + const fs = require('fs'); + const yaml = require('js-yaml'); + const file = '$LATEST_YML'; + const merged = '$MERGED_EXE_NAME'; + const x64 = '$X64_EXE_NAME'; + const arm64 = '$ARM64_EXE_NAME'; + const doc = yaml.load(fs.readFileSync(file, 'utf8')); + const before = (doc.files || []).length; + doc.files = (doc.files || []).filter(f => f.url !== merged); + const x64Entry = doc.files.find(f => f.url === x64); + const arm64Entry = doc.files.find(f => f.url === arm64); + if (!x64Entry || !arm64Entry) { + console.error('FATAL: per-arch installer entries missing from latest.yml'); + console.error(' files:', doc.files.map(f => f.url)); + process.exit(1); + } + doc.path = x64Entry.url; + doc.sha512 = x64Entry.sha512; + fs.writeFileSync(file, yaml.dump(doc, { lineWidth: -1 })); + console.log(' files[] entries: ' + before + ' → ' + doc.files.length); + console.log(' top-level path: ' + doc.path); + ") + echo "" +fi + +# ─── Create GitHub Release ───────────────────────────────────────────────── + +echo "🚀 Creating GitHub Release $TAG ..." +echo "" + +if [ -s "$NOTES_FILE" ]; then + gh release create "$TAG" \ + --title "MateClaw $VERSION" \ + --notes-file "$NOTES_FILE" \ + $DRAFT \ + "${ARTIFACTS[@]}" +else + gh release create "$TAG" \ + --title "MateClaw $VERSION" \ + --generate-notes \ + $DRAFT \ + "${ARTIFACTS[@]}" +fi + +echo "" +echo "✅ Release published successfully!" +echo "" +gh release view "$TAG" --json url -q '.url' diff --git a/mateclaw-desktop/scripts/trim-playwright-driver.cjs b/mateclaw-desktop/scripts/trim-playwright-driver.cjs new file mode 100644 index 00000000..9f4c2bd3 --- /dev/null +++ b/mateclaw-desktop/scripts/trim-playwright-driver.cjs @@ -0,0 +1,360 @@ +const fs = require('node:fs') +const path = require('node:path') +const zlib = require('node:zlib') + +const ZIP_STORED = 0 +const ZIP_DEFLATED = 8 +const DRIVER_BUNDLE_PATTERN = /^BOOT-INF\/lib\/driver-bundle-[^/]+\.jar$/ + +const ARCH_X64 = 1 +const ARCH_ARM64 = 3 + +async function afterPack(context) { + const platform = context.electronPlatformName || context.packager?.platform?.name + const arch = normalizeArch(context.arch) + const keepDriverDirectory = resolveDriverDirectory(platform, arch) + const appJarPath = findAppJar(context) + + if (!appJarPath) { + throw new Error(`[trim-playwright-driver] app.jar not found in ${context.appOutDir}`) + } + + const result = await trimDriverBundleInAppJar(appJarPath, keepDriverDirectory) + const keptLabel = asArray(keepDriverDirectory).join(', ') + console.log( + `[trim-playwright-driver] ${path.relative(process.cwd(), appJarPath)}: ` + + `kept ${keptLabel}, removed ${result.removedDriverEntries} driver entries, ` + + `${formatBytes(result.beforeBytes)} -> ${formatBytes(result.afterBytes)}` + ) +} + +function findAppJar(context) { + const appOutDir = context.appOutDir + const productFilename = context.packager?.appInfo?.productFilename || context.packager?.appInfo?.productName || 'MateClaw' + const platform = context.electronPlatformName || context.packager?.platform?.name + + const candidates = [] + if (platform === 'darwin') { + candidates.push(path.join(appOutDir, `${productFilename}.app`, 'Contents', 'Resources', 'app.jar')) + } + candidates.push(path.join(appOutDir, 'resources', 'app.jar')) + + for (const candidate of candidates) { + if (fs.existsSync(candidate)) return candidate + } + + const found = findFirstFile(appOutDir, 'app.jar', 4) + return found +} + +function findFirstFile(root, fileName, maxDepth, depth = 0) { + if (!root || depth > maxDepth || !fs.existsSync(root)) return null + for (const dirent of fs.readdirSync(root, { withFileTypes: true })) { + const fullPath = path.join(root, dirent.name) + if (dirent.isFile() && dirent.name === fileName) return fullPath + if (dirent.isDirectory()) { + const found = findFirstFile(fullPath, fileName, maxDepth, depth + 1) + if (found) return found + } + } + return null +} + +async function trimDriverBundleInAppJar(appJarPath, keepDriverDirectory) { + const appJarBuffer = fs.readFileSync(appJarPath) + const outerEntries = readZipEntries(appJarBuffer) + const driverBundleEntry = outerEntries.find((entry) => DRIVER_BUNDLE_PATTERN.test(entry.name)) + const keepDriverDirectories = asArray(keepDriverDirectory) + + if (!driverBundleEntry) { + throw new Error(`Playwright driver-bundle jar not found in ${appJarPath}`) + } + + const innerEntries = readZipEntries(driverBundleEntry.data) + let removedDriverEntries = 0 + let keptDriverEntries = 0 + + const trimmedInnerEntries = innerEntries.filter((entry) => { + if (!entry.name.startsWith('driver/')) return true + if (keepDriverDirectories.some((directory) => entry.name.startsWith(`${directory}/`))) { + keptDriverEntries += 1 + return true + } + removedDriverEntries += 1 + return false + }) + + if (keptDriverEntries === 0) { + throw new Error( + `No entries kept under ${keepDriverDirectories.join(', ')}; Playwright driver layout may have changed` + ) + } + + const trimmedDriverBundle = createZipBuffer(trimmedInnerEntries.map(cloneEntryForWrite)) + const rewrittenOuterEntries = outerEntries.map((entry) => { + if (entry.name !== driverBundleEntry.name) return cloneEntryForWrite(entry) + return { + ...cloneEntryForWrite(entry), + data: trimmedDriverBundle, + method: ZIP_STORED, + } + }) + + const rewrittenAppJar = createZipBuffer(rewrittenOuterEntries) + fs.writeFileSync(appJarPath, rewrittenAppJar) + + return { + beforeBytes: appJarBuffer.length, + afterBytes: rewrittenAppJar.length, + driverBundleName: driverBundleEntry.name, + keptDriverEntries, + removedDriverEntries, + } +} + +function cloneEntryForWrite(entry) { + return { + name: entry.name, + data: Buffer.from(entry.data), + method: entry.method, + date: entry.date, + comment: entry.comment, + externalAttributes: entry.externalAttributes, + } +} + +function resolveDriverDirectory(platform, arch) { + const normalizedArch = normalizeArch(arch) + + if (platform === 'darwin') { + if (normalizedArch === 'arm64') return 'driver/mac-arm64' + if (normalizedArch === 'x64') return 'driver/mac' + if (normalizedArch === 'universal') return ['driver/mac', 'driver/mac-arm64'] + } + + if (platform === 'linux') { + if (normalizedArch === 'arm64') return 'driver/linux-arm64' + if (normalizedArch === 'x64') return 'driver/linux' + } + + if (platform === 'win32') { + return 'driver/win32_x64' + } + + throw new Error(`Unsupported platform/arch for Playwright driver trim: ${platform}/${arch}`) +} + +function asArray(value) { + return Array.isArray(value) ? value : [value] +} + +function normalizeArch(arch) { + if (arch === ARCH_X64 || arch === 'x64') return 'x64' + if (arch === ARCH_ARM64 || arch === 'arm64') return 'arm64' + if (arch === 'universal' || arch === 4) return 'universal' + return String(arch) +} + +function readZipEntries(buffer) { + const eocdOffset = findEndOfCentralDirectory(buffer) + const centralDirectorySize = buffer.readUInt32LE(eocdOffset + 12) + const centralDirectoryOffset = buffer.readUInt32LE(eocdOffset + 16) + const entries = [] + + let offset = centralDirectoryOffset + const centralDirectoryEnd = centralDirectoryOffset + centralDirectorySize + + while (offset < centralDirectoryEnd) { + const signature = buffer.readUInt32LE(offset) + if (signature !== 0x02014b50) { + throw new Error(`Invalid central directory signature at offset ${offset}`) + } + + const flags = buffer.readUInt16LE(offset + 8) + const method = buffer.readUInt16LE(offset + 10) + const dosTime = buffer.readUInt16LE(offset + 12) + const dosDate = buffer.readUInt16LE(offset + 14) + const crc = buffer.readUInt32LE(offset + 16) + const compressedSize = buffer.readUInt32LE(offset + 20) + const uncompressedSize = buffer.readUInt32LE(offset + 24) + const fileNameLength = buffer.readUInt16LE(offset + 28) + const extraLength = buffer.readUInt16LE(offset + 30) + const commentLength = buffer.readUInt16LE(offset + 32) + const externalAttributes = buffer.readUInt32LE(offset + 38) + const localHeaderOffset = buffer.readUInt32LE(offset + 42) + const name = buffer.toString('utf8', offset + 46, offset + 46 + fileNameLength) + const comment = buffer.subarray(offset + 46 + fileNameLength + extraLength, offset + 46 + fileNameLength + extraLength + commentLength) + + const localSignature = buffer.readUInt32LE(localHeaderOffset) + if (localSignature !== 0x04034b50) { + throw new Error(`Invalid local file header signature for ${name}`) + } + + const localNameLength = buffer.readUInt16LE(localHeaderOffset + 26) + const localExtraLength = buffer.readUInt16LE(localHeaderOffset + 28) + const dataOffset = localHeaderOffset + 30 + localNameLength + localExtraLength + const compressedData = buffer.subarray(dataOffset, dataOffset + compressedSize) + + let data + if (method === ZIP_STORED) { + data = Buffer.from(compressedData) + } else if (method === ZIP_DEFLATED) { + data = zlib.inflateRawSync(compressedData) + } else { + throw new Error(`Unsupported ZIP method ${method} for ${name}`) + } + + if (data.length !== uncompressedSize) { + throw new Error(`Unexpected uncompressed size for ${name}: ${data.length} !== ${uncompressedSize}`) + } + + entries.push({ + name, + data, + method, + flags, + crc, + date: dosToDate(dosDate, dosTime), + comment: Buffer.from(comment), + externalAttributes, + }) + + offset += 46 + fileNameLength + extraLength + commentLength + } + + return entries +} + +function createZipBuffer(entries) { + const localParts = [] + const centralParts = [] + let offset = 0 + + for (const entry of entries) { + const nameBuffer = Buffer.from(entry.name) + const data = Buffer.isBuffer(entry.data) ? entry.data : Buffer.from(entry.data || '') + const method = entry.method ?? ZIP_DEFLATED + const compressedData = method === ZIP_STORED ? data : zlib.deflateRawSync(data) + const crc = crc32(data) + const { dosDate, dosTime } = dateToDos(entry.date) + + const localHeader = Buffer.alloc(30 + nameBuffer.length) + localHeader.writeUInt32LE(0x04034b50, 0) + localHeader.writeUInt16LE(20, 4) + localHeader.writeUInt16LE(0x0800, 6) + localHeader.writeUInt16LE(method, 8) + localHeader.writeUInt16LE(dosTime, 10) + localHeader.writeUInt16LE(dosDate, 12) + localHeader.writeUInt32LE(crc, 14) + localHeader.writeUInt32LE(compressedData.length, 18) + localHeader.writeUInt32LE(data.length, 22) + localHeader.writeUInt16LE(nameBuffer.length, 26) + localHeader.writeUInt16LE(0, 28) + nameBuffer.copy(localHeader, 30) + + localParts.push(localHeader, compressedData) + + const comment = Buffer.isBuffer(entry.comment) ? entry.comment : Buffer.alloc(0) + const centralHeader = Buffer.alloc(46 + nameBuffer.length + comment.length) + centralHeader.writeUInt32LE(0x02014b50, 0) + centralHeader.writeUInt16LE(20, 4) + centralHeader.writeUInt16LE(20, 6) + centralHeader.writeUInt16LE(0x0800, 8) + centralHeader.writeUInt16LE(method, 10) + centralHeader.writeUInt16LE(dosTime, 12) + centralHeader.writeUInt16LE(dosDate, 14) + centralHeader.writeUInt32LE(crc, 16) + centralHeader.writeUInt32LE(compressedData.length, 20) + centralHeader.writeUInt32LE(data.length, 24) + centralHeader.writeUInt16LE(nameBuffer.length, 28) + centralHeader.writeUInt16LE(0, 30) + centralHeader.writeUInt16LE(comment.length, 32) + centralHeader.writeUInt16LE(0, 34) + centralHeader.writeUInt16LE(0, 36) + centralHeader.writeUInt32LE(entry.externalAttributes || 0, 38) + centralHeader.writeUInt32LE(offset, 42) + nameBuffer.copy(centralHeader, 46) + comment.copy(centralHeader, 46 + nameBuffer.length) + centralParts.push(centralHeader) + + offset += localHeader.length + compressedData.length + } + + const centralDirectoryOffset = offset + const centralDirectory = Buffer.concat(centralParts) + const centralDirectorySize = centralDirectory.length + + const eocd = Buffer.alloc(22) + eocd.writeUInt32LE(0x06054b50, 0) + eocd.writeUInt16LE(0, 4) + eocd.writeUInt16LE(0, 6) + eocd.writeUInt16LE(entries.length, 8) + eocd.writeUInt16LE(entries.length, 10) + eocd.writeUInt32LE(centralDirectorySize, 12) + eocd.writeUInt32LE(centralDirectoryOffset, 16) + eocd.writeUInt16LE(0, 20) + + return Buffer.concat([...localParts, centralDirectory, eocd]) +} + +function findEndOfCentralDirectory(buffer) { + const minOffset = Math.max(0, buffer.length - 0xffff - 22) + for (let offset = buffer.length - 22; offset >= minOffset; offset -= 1) { + if (buffer.readUInt32LE(offset) === 0x06054b50) return offset + } + throw new Error('End of central directory not found') +} + +function dateToDos(date) { + const value = date instanceof Date ? date : new Date(1980, 0, 1, 0, 0, 0) + const year = Math.max(1980, value.getFullYear()) + return { + dosDate: ((year - 1980) << 9) | ((value.getMonth() + 1) << 5) | value.getDate(), + dosTime: (value.getHours() << 11) | (value.getMinutes() << 5) | Math.floor(value.getSeconds() / 2), + } +} + +function dosToDate(dosDate, dosTime) { + const day = dosDate & 0x1f + const month = (dosDate >> 5) & 0x0f + const year = ((dosDate >> 9) & 0x7f) + 1980 + const second = (dosTime & 0x1f) * 2 + const minute = (dosTime >> 5) & 0x3f + const hour = (dosTime >> 11) & 0x1f + return new Date(year, Math.max(0, month - 1), day || 1, hour, minute, second) +} + +function crc32(buffer) { + let crc = 0xffffffff + for (let i = 0; i < buffer.length; i += 1) { + crc = CRC_TABLE[(crc ^ buffer[i]) & 0xff] ^ (crc >>> 8) + } + return (crc ^ 0xffffffff) >>> 0 +} + +function makeCrcTable() { + const table = new Uint32Array(256) + for (let i = 0; i < 256; i += 1) { + let value = i + for (let bit = 0; bit < 8; bit += 1) { + value = value & 1 ? 0xedb88320 ^ (value >>> 1) : value >>> 1 + } + table[i] = value >>> 0 + } + return table +} + +function formatBytes(bytes) { + return `${(bytes / 1024 / 1024).toFixed(1)} MB` +} + +const CRC_TABLE = makeCrcTable() + +module.exports = afterPack +module.exports.default = afterPack +module.exports.createZipBuffer = createZipBuffer +module.exports.readZipEntries = readZipEntries +module.exports.resolveDriverDirectory = resolveDriverDirectory +module.exports.trimDriverBundleInAppJar = trimDriverBundleInAppJar +module.exports.ZIP_STORED = ZIP_STORED +module.exports.ZIP_DEFLATED = ZIP_DEFLATED diff --git a/mateclaw-server/Dockerfile b/mateclaw-server/Dockerfile index 2898c2ae..4e9cd4b5 100644 --- a/mateclaw-server/Dockerfile +++ b/mateclaw-server/Dockerfile @@ -47,6 +47,7 @@ COPY pom.xml ./pom.xml COPY mateclaw-plugin-api/pom.xml mateclaw-plugin-api/pom.xml COPY mateclaw-server/pom.xml mateclaw-server/pom.xml COPY mateclaw-plugin-sample/pom.xml mateclaw-plugin-sample/pom.xml +COPY mateclaw-plugin-search-sample/pom.xml mateclaw-plugin-search-sample/pom.xml # Pre-fetch backend dependencies through the reactor so the parent POM, # dependencyManagement, and internal module versions all resolve consistently. @@ -99,8 +100,19 @@ RUN apt-get update \ tesseract-ocr \ tesseract-ocr-chi-sim \ tzdata \ + python3-pip \ + python-is-python3 \ + python3-dev \ + build-essential \ && rm -rf /var/lib/apt/lists/* +# Remove PEP 668's EXTERNALLY-MANAGED marker so pip can install packages +# system-wide without --break-system-packages. This is a container — there is +# no host Python environment to protect. Skill scripts and LLM-generated code +# need to `pip install` on the fly; PEP 668 would block every install with +# "error: externally-managed-environment". +RUN rm -f /usr/lib/python3*/EXTERNALLY-MANAGED + # Tell Playwright Java where Microsoft's image stored the browsers. # BrowserLauncher's BUNDLED strategy will then succeed without extra config. ENV PLAYWRIGHT_BROWSERS_PATH=/ms-playwright \ @@ -117,6 +129,7 @@ ENV PLAYWRIGHT_BROWSERS_PATH=/ms-playwright \ ENV SPRING_PROFILES_ACTIVE=mysql COPY --from=builder /build/mateclaw-server/target/*.jar app.jar + EXPOSE 18088 EXPOSE 1455 ENTRYPOINT ["java", "-jar", "app.jar"] diff --git a/mateclaw-server/pom.xml b/mateclaw-server/pom.xml index e04bb74f..f25d542c 100644 --- a/mateclaw-server/pom.xml +++ b/mateclaw-server/pom.xml @@ -263,6 +263,19 @@ jsoup + + + + com.github.binarywang + weixin-java-mp + 4.6.0 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-2010.xsd b/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-2010.xsd new file mode 100644 index 00000000..f65f7777 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-2010.xsd @@ -0,0 +1,560 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-2012.xsd b/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-2012.xsd new file mode 100644 index 00000000..6b00755a --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-2012.xsd @@ -0,0 +1,67 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-2018.xsd b/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-2018.xsd new file mode 100644 index 00000000..f321d333 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-2018.xsd @@ -0,0 +1,14 @@ + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-cex-2018.xsd b/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-cex-2018.xsd new file mode 100644 index 00000000..364c6a9b --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-cex-2018.xsd @@ -0,0 +1,20 @@ + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-cid-2016.xsd b/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-cid-2016.xsd new file mode 100644 index 00000000..fed9d15b --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-cid-2016.xsd @@ -0,0 +1,13 @@ + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-sdtdatahash-2020.xsd b/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-sdtdatahash-2020.xsd new file mode 100644 index 00000000..680cf154 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-sdtdatahash-2020.xsd @@ -0,0 +1,4 @@ + + + + diff --git a/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-symex-2015.xsd b/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-symex-2015.xsd new file mode 100644 index 00000000..89ada908 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/docx/scripts/office/schemas/microsoft/wml-symex-2015.xsd @@ -0,0 +1,8 @@ + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/docx/scripts/office/soffice.py b/mateclaw-server/src/main/resources/skills/docx/scripts/office/soffice.py new file mode 100644 index 00000000..d64118bb --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/docx/scripts/office/soffice.py @@ -0,0 +1,220 @@ +""" +Helper for running LibreOffice (soffice) in environments where AF_UNIX +sockets may be blocked (e.g., sandboxed VMs). Detects the restriction +at runtime and applies an LD_PRELOAD shim if needed. + +Usage: + from office.soffice import run_soffice, get_soffice_cmd, get_soffice_env + + # Option 1 – run soffice directly + result = run_soffice(["--headless", "--convert-to", "pdf", "input.docx"]) + + # Option 2 – get env dict for your own subprocess calls + env = get_soffice_env() + subprocess.run([get_soffice_cmd(), ...], env=env) +""" + +import os +import platform +import shutil +import socket +import subprocess +import tempfile +from pathlib import Path + + +def get_soffice_cmd() -> str: + """Return the soffice command name for the current platform.""" + # Prefer PATH first on all platforms. + path_cmd = shutil.which("soffice") + if path_cmd: + return path_cmd + + if platform.system() == "Windows": + # Windows can expose soffice as .com or .exe; try both on PATH first. + for candidate_name in ("soffice.com", "soffice.exe"): + path_candidate = shutil.which(candidate_name) + if path_candidate: + return path_candidate + + # On Windows, try to find soffice.exe via common install paths + for prog_dir in ( + os.environ.get("PROGRAMFILES", r"C:\Program Files"), + os.environ.get("PROGRAMFILES(X86)", r"C:\Program Files (x86)"), + ): + if not prog_dir: + continue + program_dir = Path(prog_dir) / "LibreOffice" / "program" + for exe_name in ("soffice.com", "soffice.exe"): + candidate = program_dir / exe_name + if candidate.exists(): + return str(candidate) + return "soffice" # fallback, hope it's on PATH + return "soffice" + + +def get_soffice_env() -> dict: + env = os.environ.copy() + # SAL_USE_VCLPLUGIN=svp is Linux-only (headless rendering) + if platform.system() == "Linux": + env["SAL_USE_VCLPLUGIN"] = "svp" + + if _needs_shim(): + shim = _ensure_shim() + env["LD_PRELOAD"] = str(shim) + + return env + + +def run_soffice(args: list[str], **kwargs) -> subprocess.CompletedProcess: + env = get_soffice_env() + return subprocess.run([get_soffice_cmd()] + args, env=env, **kwargs) + + + +_SHIM_SO = Path(tempfile.gettempdir()) / "lo_socket_shim.so" + + +def _needs_shim() -> bool: + # AF_UNIX / LD_PRELOAD shim is Linux-only; skip on Windows and macOS + if not hasattr(socket, "AF_UNIX"): + return False + try: + s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + s.close() + return False + except OSError: + return True + + +def _ensure_shim() -> Path: + if _SHIM_SO.exists(): + return _SHIM_SO + + src = Path(tempfile.gettempdir()) / "lo_socket_shim.c" + src.write_text(_SHIM_SOURCE) + subprocess.run( + ["gcc", "-shared", "-fPIC", "-o", str(_SHIM_SO), str(src), "-ldl"], + check=True, + capture_output=True, + ) + src.unlink() + return _SHIM_SO + + + +_SHIM_SOURCE = r""" +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include + +static int (*real_socket)(int, int, int); +static int (*real_socketpair)(int, int, int, int[2]); +static int (*real_listen)(int, int); +static int (*real_accept)(int, struct sockaddr *, socklen_t *); +static int (*real_close)(int); +static int (*real_read)(int, void *, size_t); + +/* Per-FD bookkeeping (FDs >= 1024 are passed through unshimmed). */ +static int is_shimmed[1024]; +static int peer_of[1024]; +static int wake_r[1024]; /* accept() blocks reading this */ +static int wake_w[1024]; /* close() writes to this */ +static int listener_fd = -1; /* FD that received listen() */ + +__attribute__((constructor)) +static void init(void) { + real_socket = dlsym(RTLD_NEXT, "socket"); + real_socketpair = dlsym(RTLD_NEXT, "socketpair"); + real_listen = dlsym(RTLD_NEXT, "listen"); + real_accept = dlsym(RTLD_NEXT, "accept"); + real_close = dlsym(RTLD_NEXT, "close"); + real_read = dlsym(RTLD_NEXT, "read"); + for (int i = 0; i < 1024; i++) { + peer_of[i] = -1; + wake_r[i] = -1; + wake_w[i] = -1; + } +} + +/* ---- socket ---------------------------------------------------------- */ +int socket(int domain, int type, int protocol) { + if (domain == AF_UNIX) { + int fd = real_socket(domain, type, protocol); + if (fd >= 0) return fd; + /* socket(AF_UNIX) blocked – fall back to socketpair(). */ + int sv[2]; + if (real_socketpair(domain, type, protocol, sv) == 0) { + if (sv[0] >= 0 && sv[0] < 1024) { + is_shimmed[sv[0]] = 1; + peer_of[sv[0]] = sv[1]; + int wp[2]; + if (pipe(wp) == 0) { + wake_r[sv[0]] = wp[0]; + wake_w[sv[0]] = wp[1]; + } + } + return sv[0]; + } + errno = EPERM; + return -1; + } + return real_socket(domain, type, protocol); +} + +/* ---- listen ---------------------------------------------------------- */ +int listen(int sockfd, int backlog) { + if (sockfd >= 0 && sockfd < 1024 && is_shimmed[sockfd]) { + listener_fd = sockfd; + return 0; + } + return real_listen(sockfd, backlog); +} + +/* ---- accept ---------------------------------------------------------- */ +int accept(int sockfd, struct sockaddr *addr, socklen_t *addrlen) { + if (sockfd >= 0 && sockfd < 1024 && is_shimmed[sockfd]) { + /* Block until close() writes to the wake pipe. */ + if (wake_r[sockfd] >= 0) { + char buf; + real_read(wake_r[sockfd], &buf, 1); + } + errno = ECONNABORTED; + return -1; + } + return real_accept(sockfd, addr, addrlen); +} + +/* ---- close ----------------------------------------------------------- */ +int close(int fd) { + if (fd >= 0 && fd < 1024 && is_shimmed[fd]) { + int was_listener = (fd == listener_fd); + is_shimmed[fd] = 0; + + if (wake_w[fd] >= 0) { /* unblock accept() */ + char c = 0; + write(wake_w[fd], &c, 1); + real_close(wake_w[fd]); + wake_w[fd] = -1; + } + if (wake_r[fd] >= 0) { real_close(wake_r[fd]); wake_r[fd] = -1; } + if (peer_of[fd] >= 0) { real_close(peer_of[fd]); peer_of[fd] = -1; } + + if (was_listener) + _exit(0); /* conversion done – exit */ + } + return real_close(fd); +} +""" + + + +if __name__ == "__main__": + import sys + result = run_soffice(sys.argv[1:]) + sys.exit(result.returncode) diff --git a/mateclaw-server/src/main/resources/skills/docx/scripts/office/unpack.py b/mateclaw-server/src/main/resources/skills/docx/scripts/office/unpack.py new file mode 100755 index 00000000..00152533 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/docx/scripts/office/unpack.py @@ -0,0 +1,132 @@ +"""Unpack Office files (DOCX, PPTX, XLSX) for editing. + +Extracts the ZIP archive, pretty-prints XML files, and optionally: +- Merges adjacent runs with identical formatting (DOCX only) +- Simplifies adjacent tracked changes from same author (DOCX only) + +Usage: + python unpack.py [options] + +Examples: + python unpack.py document.docx unpacked/ + python unpack.py presentation.pptx unpacked/ + python unpack.py document.docx unpacked/ --merge-runs false +""" + +import argparse +import sys +import zipfile +from pathlib import Path + +import defusedxml.minidom + +from helpers.merge_runs import merge_runs as do_merge_runs +from helpers.simplify_redlines import simplify_redlines as do_simplify_redlines + +SMART_QUOTE_REPLACEMENTS = { + "\u201c": "“", + "\u201d": "”", + "\u2018": "‘", + "\u2019": "’", +} + + +def unpack( + input_file: str, + output_directory: str, + merge_runs: bool = True, + simplify_redlines: bool = True, +) -> tuple[None, str]: + input_path = Path(input_file) + output_path = Path(output_directory) + suffix = input_path.suffix.lower() + + if not input_path.exists(): + return None, f"Error: {input_file} does not exist" + + if suffix not in {".docx", ".pptx", ".xlsx"}: + return None, f"Error: {input_file} must be a .docx, .pptx, or .xlsx file" + + try: + output_path.mkdir(parents=True, exist_ok=True) + + with zipfile.ZipFile(input_path, "r") as zf: + zf.extractall(output_path) + + xml_files = list(output_path.rglob("*.xml")) + list(output_path.rglob("*.rels")) + for xml_file in xml_files: + _pretty_print_xml(xml_file) + + message = f"Unpacked {input_file} ({len(xml_files)} XML files)" + + if suffix == ".docx": + if simplify_redlines: + simplify_count, _ = do_simplify_redlines(str(output_path)) + message += f", simplified {simplify_count} tracked changes" + + if merge_runs: + merge_count, _ = do_merge_runs(str(output_path)) + message += f", merged {merge_count} runs" + + for xml_file in xml_files: + _escape_smart_quotes(xml_file) + + return None, message + + except zipfile.BadZipFile: + return None, f"Error: {input_file} is not a valid Office file" + except Exception as e: + return None, f"Error unpacking: {e}" + + +def _pretty_print_xml(xml_file: Path) -> None: + try: + content = xml_file.read_text(encoding="utf-8") + dom = defusedxml.minidom.parseString(content) + xml_file.write_bytes(dom.toprettyxml(indent=" ", encoding="utf-8")) + except Exception: + pass + + +def _escape_smart_quotes(xml_file: Path) -> None: + try: + content = xml_file.read_text(encoding="utf-8") + for char, entity in SMART_QUOTE_REPLACEMENTS.items(): + content = content.replace(char, entity) + xml_file.write_text(content, encoding="utf-8") + except Exception: + pass + + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description="Unpack an Office file (DOCX, PPTX, XLSX) for editing" + ) + parser.add_argument("input_file", help="Office file to unpack") + parser.add_argument("output_directory", help="Output directory") + parser.add_argument( + "--merge-runs", + type=lambda x: x.lower() == "true", + default=True, + metavar="true|false", + help="Merge adjacent runs with identical formatting (DOCX only, default: true)", + ) + parser.add_argument( + "--simplify-redlines", + type=lambda x: x.lower() == "true", + default=True, + metavar="true|false", + help="Merge adjacent tracked changes from same author (DOCX only, default: true)", + ) + args = parser.parse_args() + + _, message = unpack( + args.input_file, + args.output_directory, + merge_runs=args.merge_runs, + simplify_redlines=args.simplify_redlines, + ) + print(message) + + if "Error" in message: + sys.exit(1) diff --git a/mateclaw-server/src/main/resources/skills/docx/scripts/office/validate.py b/mateclaw-server/src/main/resources/skills/docx/scripts/office/validate.py new file mode 100755 index 00000000..03b01f6e --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/docx/scripts/office/validate.py @@ -0,0 +1,111 @@ +""" +Command line tool to validate Office document XML files against XSD schemas and tracked changes. + +Usage: + python validate.py [--original ] [--auto-repair] [--author NAME] + +The first argument can be either: +- An unpacked directory containing the Office document XML files +- A packed Office file (.docx/.pptx/.xlsx) which will be unpacked to a temp directory + +Auto-repair fixes: +- paraId/durableId values that exceed OOXML limits +- Missing xml:space="preserve" on w:t elements with whitespace +""" + +import argparse +import sys +import tempfile +import zipfile +from pathlib import Path + +from validators import DOCXSchemaValidator, PPTXSchemaValidator, RedliningValidator + + +def main(): + parser = argparse.ArgumentParser(description="Validate Office document XML files") + parser.add_argument( + "path", + help="Path to unpacked directory or packed Office file (.docx/.pptx/.xlsx)", + ) + parser.add_argument( + "--original", + required=False, + default=None, + help="Path to original file (.docx/.pptx/.xlsx). If omitted, all XSD errors are reported and redlining validation is skipped.", + ) + parser.add_argument( + "-v", + "--verbose", + action="store_true", + help="Enable verbose output", + ) + parser.add_argument( + "--auto-repair", + action="store_true", + help="Automatically repair common issues (hex IDs, whitespace preservation)", + ) + parser.add_argument( + "--author", + default="Claude", + help="Author name for redlining validation (default: Claude)", + ) + args = parser.parse_args() + + path = Path(args.path) + assert path.exists(), f"Error: {path} does not exist" + + original_file = None + if args.original: + original_file = Path(args.original) + assert original_file.is_file(), f"Error: {original_file} is not a file" + assert original_file.suffix.lower() in [".docx", ".pptx", ".xlsx"], ( + f"Error: {original_file} must be a .docx, .pptx, or .xlsx file" + ) + + file_extension = (original_file or path).suffix.lower() + assert file_extension in [".docx", ".pptx", ".xlsx"], ( + f"Error: Cannot determine file type from {path}. Use --original or provide a .docx/.pptx/.xlsx file." + ) + + if path.is_file() and path.suffix.lower() in [".docx", ".pptx", ".xlsx"]: + temp_dir = tempfile.mkdtemp() + with zipfile.ZipFile(path, "r") as zf: + zf.extractall(temp_dir) + unpacked_dir = Path(temp_dir) + else: + assert path.is_dir(), f"Error: {path} is not a directory or Office file" + unpacked_dir = path + + match file_extension: + case ".docx": + validators = [ + DOCXSchemaValidator(unpacked_dir, original_file, verbose=args.verbose), + ] + if original_file: + validators.append( + RedliningValidator(unpacked_dir, original_file, verbose=args.verbose, author=args.author) + ) + case ".pptx": + validators = [ + PPTXSchemaValidator(unpacked_dir, original_file, verbose=args.verbose), + ] + case _: + print(f"Error: Validation not supported for file type {file_extension}") + sys.exit(1) + + if args.auto_repair: + total_repairs = sum(v.repair() for v in validators) + if total_repairs: + print(f"Auto-repaired {total_repairs} issue(s)") + + success = all(v.validate() for v in validators) + + if success: + print("All validations PASSED!") + + sys.exit(0 if success else 1) + + +if __name__ == "__main__": + main() diff --git a/mateclaw-server/src/main/resources/skills/docx/scripts/office/validators/__init__.py b/mateclaw-server/src/main/resources/skills/docx/scripts/office/validators/__init__.py new file mode 100644 index 00000000..db092ece --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/docx/scripts/office/validators/__init__.py @@ -0,0 +1,15 @@ +""" +Validation modules for Word document processing. +""" + +from .base import BaseSchemaValidator +from .docx import DOCXSchemaValidator +from .pptx import PPTXSchemaValidator +from .redlining import RedliningValidator + +__all__ = [ + "BaseSchemaValidator", + "DOCXSchemaValidator", + "PPTXSchemaValidator", + "RedliningValidator", +] diff --git a/mateclaw-server/src/main/resources/skills/docx/scripts/office/validators/base.py b/mateclaw-server/src/main/resources/skills/docx/scripts/office/validators/base.py new file mode 100644 index 00000000..db4a06a2 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/docx/scripts/office/validators/base.py @@ -0,0 +1,847 @@ +""" +Base validator with common validation logic for document files. +""" + +import re +from pathlib import Path + +import defusedxml.minidom +import lxml.etree + + +class BaseSchemaValidator: + + IGNORED_VALIDATION_ERRORS = [ + "hyphenationZone", + "purl.org/dc/terms", + ] + + UNIQUE_ID_REQUIREMENTS = { + "comment": ("id", "file"), + "commentrangestart": ("id", "file"), + "commentrangeend": ("id", "file"), + "bookmarkstart": ("id", "file"), + "bookmarkend": ("id", "file"), + "sldid": ("id", "file"), + "sldmasterid": ("id", "global"), + "sldlayoutid": ("id", "global"), + "cm": ("authorid", "file"), + "sheet": ("sheetid", "file"), + "definedname": ("id", "file"), + "cxnsp": ("id", "file"), + "sp": ("id", "file"), + "pic": ("id", "file"), + "grpsp": ("id", "file"), + } + + EXCLUDED_ID_CONTAINERS = { + "sectionlst", + } + + ELEMENT_RELATIONSHIP_TYPES = {} + + SCHEMA_MAPPINGS = { + "word": "ISO-IEC29500-4_2016/wml.xsd", + "ppt": "ISO-IEC29500-4_2016/pml.xsd", + "xl": "ISO-IEC29500-4_2016/sml.xsd", + "[Content_Types].xml": "ecma/fouth-edition/opc-contentTypes.xsd", + "app.xml": "ISO-IEC29500-4_2016/shared-documentPropertiesExtended.xsd", + "core.xml": "ecma/fouth-edition/opc-coreProperties.xsd", + "custom.xml": "ISO-IEC29500-4_2016/shared-documentPropertiesCustom.xsd", + ".rels": "ecma/fouth-edition/opc-relationships.xsd", + "people.xml": "microsoft/wml-2012.xsd", + "commentsIds.xml": "microsoft/wml-cid-2016.xsd", + "commentsExtensible.xml": "microsoft/wml-cex-2018.xsd", + "commentsExtended.xml": "microsoft/wml-2012.xsd", + "chart": "ISO-IEC29500-4_2016/dml-chart.xsd", + "theme": "ISO-IEC29500-4_2016/dml-main.xsd", + "drawing": "ISO-IEC29500-4_2016/dml-main.xsd", + } + + MC_NAMESPACE = "http://schemas.openxmlformats.org/markup-compatibility/2006" + XML_NAMESPACE = "http://www.w3.org/XML/1998/namespace" + + PACKAGE_RELATIONSHIPS_NAMESPACE = ( + "http://schemas.openxmlformats.org/package/2006/relationships" + ) + OFFICE_RELATIONSHIPS_NAMESPACE = ( + "http://schemas.openxmlformats.org/officeDocument/2006/relationships" + ) + CONTENT_TYPES_NAMESPACE = ( + "http://schemas.openxmlformats.org/package/2006/content-types" + ) + + MAIN_CONTENT_FOLDERS = {"word", "ppt", "xl"} + + OOXML_NAMESPACES = { + "http://schemas.openxmlformats.org/officeDocument/2006/math", + "http://schemas.openxmlformats.org/officeDocument/2006/relationships", + "http://schemas.openxmlformats.org/schemaLibrary/2006/main", + "http://schemas.openxmlformats.org/drawingml/2006/main", + "http://schemas.openxmlformats.org/drawingml/2006/chart", + "http://schemas.openxmlformats.org/drawingml/2006/chartDrawing", + "http://schemas.openxmlformats.org/drawingml/2006/diagram", + "http://schemas.openxmlformats.org/drawingml/2006/picture", + "http://schemas.openxmlformats.org/drawingml/2006/spreadsheetDrawing", + "http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing", + "http://schemas.openxmlformats.org/wordprocessingml/2006/main", + "http://schemas.openxmlformats.org/presentationml/2006/main", + "http://schemas.openxmlformats.org/spreadsheetml/2006/main", + "http://schemas.openxmlformats.org/officeDocument/2006/sharedTypes", + "http://www.w3.org/XML/1998/namespace", + } + + def __init__(self, unpacked_dir, original_file=None, verbose=False): + self.unpacked_dir = Path(unpacked_dir).resolve() + self.original_file = Path(original_file) if original_file else None + self.verbose = verbose + + self.schemas_dir = Path(__file__).parent.parent / "schemas" + + patterns = ["*.xml", "*.rels"] + self.xml_files = [ + f for pattern in patterns for f in self.unpacked_dir.rglob(pattern) + ] + + if not self.xml_files: + print(f"Warning: No XML files found in {self.unpacked_dir}") + + def validate(self): + raise NotImplementedError("Subclasses must implement the validate method") + + def repair(self) -> int: + return self.repair_whitespace_preservation() + + def repair_whitespace_preservation(self) -> int: + repairs = 0 + + for xml_file in self.xml_files: + try: + content = xml_file.read_text(encoding="utf-8") + dom = defusedxml.minidom.parseString(content) + modified = False + + for elem in dom.getElementsByTagName("*"): + if elem.tagName.endswith(":t") and elem.firstChild: + text = elem.firstChild.nodeValue + if text and (text.startswith((' ', '\t')) or text.endswith((' ', '\t'))): + if elem.getAttribute("xml:space") != "preserve": + elem.setAttribute("xml:space", "preserve") + text_preview = repr(text[:30]) + "..." if len(text) > 30 else repr(text) + print(f" Repaired: {xml_file.name}: Added xml:space='preserve' to {elem.tagName}: {text_preview}") + repairs += 1 + modified = True + + if modified: + xml_file.write_bytes(dom.toxml(encoding="UTF-8")) + + except Exception: + pass + + return repairs + + def validate_xml(self): + errors = [] + + for xml_file in self.xml_files: + try: + lxml.etree.parse(str(xml_file)) + except lxml.etree.XMLSyntaxError as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {e.lineno}: {e.msg}" + ) + except Exception as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Unexpected error: {str(e)}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} XML violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All XML files are well-formed") + return True + + def validate_namespaces(self): + errors = [] + + for xml_file in self.xml_files: + try: + root = lxml.etree.parse(str(xml_file)).getroot() + declared = set(root.nsmap.keys()) - {None} + + for attr_val in [ + v for k, v in root.attrib.items() if k.endswith("Ignorable") + ]: + undeclared = set(attr_val.split()) - declared + errors.extend( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Namespace '{ns}' in Ignorable but not declared" + for ns in undeclared + ) + except lxml.etree.XMLSyntaxError: + continue + + if errors: + print(f"FAILED - {len(errors)} namespace issues:") + for error in errors: + print(error) + return False + if self.verbose: + print("PASSED - All namespace prefixes properly declared") + return True + + def validate_unique_ids(self): + errors = [] + global_ids = {} + + for xml_file in self.xml_files: + try: + root = lxml.etree.parse(str(xml_file)).getroot() + file_ids = {} + + mc_elements = root.xpath( + ".//mc:AlternateContent", namespaces={"mc": self.MC_NAMESPACE} + ) + for elem in mc_elements: + elem.getparent().remove(elem) + + for elem in root.iter(): + tag = ( + elem.tag.split("}")[-1].lower() + if "}" in elem.tag + else elem.tag.lower() + ) + + if tag in self.UNIQUE_ID_REQUIREMENTS: + in_excluded_container = any( + ancestor.tag.split("}")[-1].lower() in self.EXCLUDED_ID_CONTAINERS + for ancestor in elem.iterancestors() + ) + if in_excluded_container: + continue + + attr_name, scope = self.UNIQUE_ID_REQUIREMENTS[tag] + + id_value = None + for attr, value in elem.attrib.items(): + attr_local = ( + attr.split("}")[-1].lower() + if "}" in attr + else attr.lower() + ) + if attr_local == attr_name: + id_value = value + break + + if id_value is not None: + if scope == "global": + if id_value in global_ids: + prev_file, prev_line, prev_tag = global_ids[ + id_value + ] + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {elem.sourceline}: Global ID '{id_value}' in <{tag}> " + f"already used in {prev_file} at line {prev_line} in <{prev_tag}>" + ) + else: + global_ids[id_value] = ( + xml_file.relative_to(self.unpacked_dir), + elem.sourceline, + tag, + ) + elif scope == "file": + key = (tag, attr_name) + if key not in file_ids: + file_ids[key] = {} + + if id_value in file_ids[key]: + prev_line = file_ids[key][id_value] + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {elem.sourceline}: Duplicate {attr_name}='{id_value}' in <{tag}> " + f"(first occurrence at line {prev_line})" + ) + else: + file_ids[key][id_value] = elem.sourceline + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} ID uniqueness violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All required IDs are unique") + return True + + def validate_file_references(self): + errors = [] + + rels_files = list(self.unpacked_dir.rglob("*.rels")) + + if not rels_files: + if self.verbose: + print("PASSED - No .rels files found") + return True + + all_files = [] + for file_path in self.unpacked_dir.rglob("*"): + if ( + file_path.is_file() + and file_path.name != "[Content_Types].xml" + and not file_path.name.endswith(".rels") + ): + all_files.append(file_path.resolve()) + + all_referenced_files = set() + + if self.verbose: + print( + f"Found {len(rels_files)} .rels files and {len(all_files)} target files" + ) + + for rels_file in rels_files: + try: + rels_root = lxml.etree.parse(str(rels_file)).getroot() + + rels_dir = rels_file.parent + + referenced_files = set() + broken_refs = [] + + for rel in rels_root.findall( + ".//ns:Relationship", + namespaces={"ns": self.PACKAGE_RELATIONSHIPS_NAMESPACE}, + ): + target = rel.get("Target") + if target and not target.startswith( + ("http", "mailto:") + ): + if target.startswith("/"): + target_path = self.unpacked_dir / target.lstrip("/") + elif rels_file.name == ".rels": + target_path = self.unpacked_dir / target + else: + base_dir = rels_dir.parent + target_path = base_dir / target + + try: + target_path = target_path.resolve() + if target_path.exists() and target_path.is_file(): + referenced_files.add(target_path) + all_referenced_files.add(target_path) + else: + broken_refs.append((target, rel.sourceline)) + except (OSError, ValueError): + broken_refs.append((target, rel.sourceline)) + + if broken_refs: + rel_path = rels_file.relative_to(self.unpacked_dir) + for broken_ref, line_num in broken_refs: + errors.append( + f" {rel_path}: Line {line_num}: Broken reference to {broken_ref}" + ) + + except Exception as e: + rel_path = rels_file.relative_to(self.unpacked_dir) + errors.append(f" Error parsing {rel_path}: {e}") + + unreferenced_files = set(all_files) - all_referenced_files + + if unreferenced_files: + for unref_file in sorted(unreferenced_files): + unref_rel_path = unref_file.relative_to(self.unpacked_dir) + errors.append(f" Unreferenced file: {unref_rel_path}") + + if errors: + print(f"FAILED - Found {len(errors)} relationship validation errors:") + for error in errors: + print(error) + print( + "CRITICAL: These errors will cause the document to appear corrupt. " + + "Broken references MUST be fixed, " + + "and unreferenced files MUST be referenced or removed." + ) + return False + else: + if self.verbose: + print( + "PASSED - All references are valid and all files are properly referenced" + ) + return True + + def validate_all_relationship_ids(self): + import lxml.etree + + errors = [] + + for xml_file in self.xml_files: + if xml_file.suffix == ".rels": + continue + + rels_dir = xml_file.parent / "_rels" + rels_file = rels_dir / f"{xml_file.name}.rels" + + if not rels_file.exists(): + continue + + try: + rels_root = lxml.etree.parse(str(rels_file)).getroot() + rid_to_type = {} + + for rel in rels_root.findall( + f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship" + ): + rid = rel.get("Id") + rel_type = rel.get("Type", "") + if rid: + if rid in rid_to_type: + rels_rel_path = rels_file.relative_to(self.unpacked_dir) + errors.append( + f" {rels_rel_path}: Line {rel.sourceline}: " + f"Duplicate relationship ID '{rid}' (IDs must be unique)" + ) + type_name = ( + rel_type.split("/")[-1] if "/" in rel_type else rel_type + ) + rid_to_type[rid] = type_name + + xml_root = lxml.etree.parse(str(xml_file)).getroot() + + r_ns = self.OFFICE_RELATIONSHIPS_NAMESPACE + rid_attrs_to_check = ["id", "embed", "link"] + for elem in xml_root.iter(): + for attr_name in rid_attrs_to_check: + rid_attr = elem.get(f"{{{r_ns}}}{attr_name}") + if not rid_attr: + continue + xml_rel_path = xml_file.relative_to(self.unpacked_dir) + elem_name = ( + elem.tag.split("}")[-1] if "}" in elem.tag else elem.tag + ) + + if rid_attr not in rid_to_type: + errors.append( + f" {xml_rel_path}: Line {elem.sourceline}: " + f"<{elem_name}> r:{attr_name} references non-existent relationship '{rid_attr}' " + f"(valid IDs: {', '.join(sorted(rid_to_type.keys())[:5])}{'...' if len(rid_to_type) > 5 else ''})" + ) + elif attr_name == "id" and self.ELEMENT_RELATIONSHIP_TYPES: + expected_type = self._get_expected_relationship_type( + elem_name + ) + if expected_type: + actual_type = rid_to_type[rid_attr] + if expected_type not in actual_type.lower(): + errors.append( + f" {xml_rel_path}: Line {elem.sourceline}: " + f"<{elem_name}> references '{rid_attr}' which points to '{actual_type}' " + f"but should point to a '{expected_type}' relationship" + ) + + except Exception as e: + xml_rel_path = xml_file.relative_to(self.unpacked_dir) + errors.append(f" Error processing {xml_rel_path}: {e}") + + if errors: + print(f"FAILED - Found {len(errors)} relationship ID reference errors:") + for error in errors: + print(error) + print("\nThese ID mismatches will cause the document to appear corrupt!") + return False + else: + if self.verbose: + print("PASSED - All relationship ID references are valid") + return True + + def _get_expected_relationship_type(self, element_name): + elem_lower = element_name.lower() + + if elem_lower in self.ELEMENT_RELATIONSHIP_TYPES: + return self.ELEMENT_RELATIONSHIP_TYPES[elem_lower] + + if elem_lower.endswith("id") and len(elem_lower) > 2: + prefix = elem_lower[:-2] + if prefix.endswith("master"): + return prefix.lower() + elif prefix.endswith("layout"): + return prefix.lower() + else: + if prefix == "sld": + return "slide" + return prefix.lower() + + if elem_lower.endswith("reference") and len(elem_lower) > 9: + prefix = elem_lower[:-9] + return prefix.lower() + + return None + + def validate_content_types(self): + errors = [] + + content_types_file = self.unpacked_dir / "[Content_Types].xml" + if not content_types_file.exists(): + print("FAILED - [Content_Types].xml file not found") + return False + + try: + root = lxml.etree.parse(str(content_types_file)).getroot() + declared_parts = set() + declared_extensions = set() + + for override in root.findall( + f".//{{{self.CONTENT_TYPES_NAMESPACE}}}Override" + ): + part_name = override.get("PartName") + if part_name is not None: + declared_parts.add(part_name.lstrip("/")) + + for default in root.findall( + f".//{{{self.CONTENT_TYPES_NAMESPACE}}}Default" + ): + extension = default.get("Extension") + if extension is not None: + declared_extensions.add(extension.lower()) + + declarable_roots = { + "sld", + "sldLayout", + "sldMaster", + "presentation", + "document", + "workbook", + "worksheet", + "theme", + } + + media_extensions = { + "png": "image/png", + "jpg": "image/jpeg", + "jpeg": "image/jpeg", + "gif": "image/gif", + "bmp": "image/bmp", + "tiff": "image/tiff", + "wmf": "image/x-wmf", + "emf": "image/x-emf", + } + + all_files = list(self.unpacked_dir.rglob("*")) + all_files = [f for f in all_files if f.is_file()] + + for xml_file in self.xml_files: + path_str = str(xml_file.relative_to(self.unpacked_dir)).replace( + "\\", "/" + ) + + if any( + skip in path_str + for skip in [".rels", "[Content_Types]", "docProps/", "_rels/"] + ): + continue + + try: + root_tag = lxml.etree.parse(str(xml_file)).getroot().tag + root_name = root_tag.split("}")[-1] if "}" in root_tag else root_tag + + if root_name in declarable_roots and path_str not in declared_parts: + errors.append( + f" {path_str}: File with <{root_name}> root not declared in [Content_Types].xml" + ) + + except Exception: + continue + + for file_path in all_files: + if file_path.suffix.lower() in {".xml", ".rels"}: + continue + if file_path.name == "[Content_Types].xml": + continue + if "_rels" in file_path.parts or "docProps" in file_path.parts: + continue + + extension = file_path.suffix.lstrip(".").lower() + if extension and extension not in declared_extensions: + if extension in media_extensions: + relative_path = file_path.relative_to(self.unpacked_dir) + errors.append( + f' {relative_path}: File with extension \'{extension}\' not declared in [Content_Types].xml - should add: ' + ) + + except Exception as e: + errors.append(f" Error parsing [Content_Types].xml: {e}") + + if errors: + print(f"FAILED - Found {len(errors)} content type declaration errors:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print( + "PASSED - All content files are properly declared in [Content_Types].xml" + ) + return True + + def validate_file_against_xsd(self, xml_file, verbose=False): + xml_file = Path(xml_file).resolve() + unpacked_dir = self.unpacked_dir.resolve() + + is_valid, current_errors = self._validate_single_file_xsd( + xml_file, unpacked_dir + ) + + if is_valid is None: + return None, set() + elif is_valid: + return True, set() + + original_errors = self._get_original_file_errors(xml_file) + + assert current_errors is not None + new_errors = current_errors - original_errors + + new_errors = { + e for e in new_errors + if not any(pattern in e for pattern in self.IGNORED_VALIDATION_ERRORS) + } + + if new_errors: + if verbose: + relative_path = xml_file.relative_to(unpacked_dir) + print(f"FAILED - {relative_path}: {len(new_errors)} new error(s)") + for error in list(new_errors)[:3]: + truncated = error[:250] + "..." if len(error) > 250 else error + print(f" - {truncated}") + return False, new_errors + else: + if verbose: + print( + f"PASSED - No new errors (original had {len(current_errors)} errors)" + ) + return True, set() + + def validate_against_xsd(self): + new_errors = [] + original_error_count = 0 + valid_count = 0 + skipped_count = 0 + + for xml_file in self.xml_files: + relative_path = str(xml_file.relative_to(self.unpacked_dir)) + is_valid, new_file_errors = self.validate_file_against_xsd( + xml_file, verbose=False + ) + + if is_valid is None: + skipped_count += 1 + continue + elif is_valid and not new_file_errors: + valid_count += 1 + continue + elif is_valid: + original_error_count += 1 + valid_count += 1 + continue + + new_errors.append(f" {relative_path}: {len(new_file_errors)} new error(s)") + for error in list(new_file_errors)[:3]: + new_errors.append( + f" - {error[:250]}..." if len(error) > 250 else f" - {error}" + ) + + if self.verbose: + print(f"Validated {len(self.xml_files)} files:") + print(f" - Valid: {valid_count}") + print(f" - Skipped (no schema): {skipped_count}") + if original_error_count: + print(f" - With original errors (ignored): {original_error_count}") + print( + f" - With NEW errors: {len(new_errors) > 0 and len([e for e in new_errors if not e.startswith(' ')]) or 0}" + ) + + if new_errors: + print("\nFAILED - Found NEW validation errors:") + for error in new_errors: + print(error) + return False + else: + if self.verbose: + print("\nPASSED - No new XSD validation errors introduced") + return True + + def _get_schema_path(self, xml_file): + if xml_file.name in self.SCHEMA_MAPPINGS: + return self.schemas_dir / self.SCHEMA_MAPPINGS[xml_file.name] + + if xml_file.suffix == ".rels": + return self.schemas_dir / self.SCHEMA_MAPPINGS[".rels"] + + if "charts/" in str(xml_file) and xml_file.name.startswith("chart"): + return self.schemas_dir / self.SCHEMA_MAPPINGS["chart"] + + if "theme/" in str(xml_file) and xml_file.name.startswith("theme"): + return self.schemas_dir / self.SCHEMA_MAPPINGS["theme"] + + if xml_file.parent.name in self.MAIN_CONTENT_FOLDERS: + return self.schemas_dir / self.SCHEMA_MAPPINGS[xml_file.parent.name] + + return None + + def _clean_ignorable_namespaces(self, xml_doc): + xml_string = lxml.etree.tostring(xml_doc, encoding="unicode") + xml_copy = lxml.etree.fromstring(xml_string) + + for elem in xml_copy.iter(): + attrs_to_remove = [] + + for attr in elem.attrib: + if "{" in attr: + ns = attr.split("}")[0][1:] + if ns not in self.OOXML_NAMESPACES: + attrs_to_remove.append(attr) + + for attr in attrs_to_remove: + del elem.attrib[attr] + + self._remove_ignorable_elements(xml_copy) + + return lxml.etree.ElementTree(xml_copy) + + def _remove_ignorable_elements(self, root): + elements_to_remove = [] + + for elem in list(root): + if not hasattr(elem, "tag") or callable(elem.tag): + continue + + tag_str = str(elem.tag) + if tag_str.startswith("{"): + ns = tag_str.split("}")[0][1:] + if ns not in self.OOXML_NAMESPACES: + elements_to_remove.append(elem) + continue + + self._remove_ignorable_elements(elem) + + for elem in elements_to_remove: + root.remove(elem) + + def _preprocess_for_mc_ignorable(self, xml_doc): + root = xml_doc.getroot() + + if f"{{{self.MC_NAMESPACE}}}Ignorable" in root.attrib: + del root.attrib[f"{{{self.MC_NAMESPACE}}}Ignorable"] + + return xml_doc + + def _validate_single_file_xsd(self, xml_file, base_path): + schema_path = self._get_schema_path(xml_file) + if not schema_path: + return None, None + + try: + with open(schema_path, "rb") as xsd_file: + parser = lxml.etree.XMLParser() + xsd_doc = lxml.etree.parse( + xsd_file, parser=parser, base_url=str(schema_path) + ) + schema = lxml.etree.XMLSchema(xsd_doc) + + with open(xml_file, "r") as f: + xml_doc = lxml.etree.parse(f) + + xml_doc, _ = self._remove_template_tags_from_text_nodes(xml_doc) + xml_doc = self._preprocess_for_mc_ignorable(xml_doc) + + relative_path = xml_file.relative_to(base_path) + if ( + relative_path.parts + and relative_path.parts[0] in self.MAIN_CONTENT_FOLDERS + ): + xml_doc = self._clean_ignorable_namespaces(xml_doc) + + if schema.validate(xml_doc): + return True, set() + else: + errors = set() + for error in schema.error_log: + errors.add(error.message) + return False, errors + + except Exception as e: + return False, {str(e)} + + def _get_original_file_errors(self, xml_file): + if self.original_file is None: + return set() + + import tempfile + import zipfile + + xml_file = Path(xml_file).resolve() + unpacked_dir = self.unpacked_dir.resolve() + relative_path = xml_file.relative_to(unpacked_dir) + + with tempfile.TemporaryDirectory() as temp_dir: + temp_path = Path(temp_dir) + + with zipfile.ZipFile(self.original_file, "r") as zip_ref: + zip_ref.extractall(temp_path) + + original_xml_file = temp_path / relative_path + + if not original_xml_file.exists(): + return set() + + is_valid, errors = self._validate_single_file_xsd( + original_xml_file, temp_path + ) + return errors if errors else set() + + def _remove_template_tags_from_text_nodes(self, xml_doc): + warnings = [] + template_pattern = re.compile(r"\{\{[^}]*\}\}") + + xml_string = lxml.etree.tostring(xml_doc, encoding="unicode") + xml_copy = lxml.etree.fromstring(xml_string) + + def process_text_content(text, content_type): + if not text: + return text + matches = list(template_pattern.finditer(text)) + if matches: + for match in matches: + warnings.append( + f"Found template tag in {content_type}: {match.group()}" + ) + return template_pattern.sub("", text) + return text + + for elem in xml_copy.iter(): + if not hasattr(elem, "tag") or callable(elem.tag): + continue + tag_str = str(elem.tag) + if tag_str.endswith("}t") or tag_str == "t": + continue + + elem.text = process_text_content(elem.text, "text content") + elem.tail = process_text_content(elem.tail, "tail content") + + return lxml.etree.ElementTree(xml_copy), warnings + + +if __name__ == "__main__": + raise RuntimeError("This module should not be run directly.") diff --git a/mateclaw-server/src/main/resources/skills/docx/scripts/office/validators/docx.py b/mateclaw-server/src/main/resources/skills/docx/scripts/office/validators/docx.py new file mode 100644 index 00000000..0a0b0bfd --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/docx/scripts/office/validators/docx.py @@ -0,0 +1,447 @@ +""" +Validator for Word document XML files against XSD schemas. +""" + +import os +import random +import re +import tempfile +import zipfile + +import defusedxml.minidom +import lxml.etree + +from .base import BaseSchemaValidator + + +class DOCXSchemaValidator(BaseSchemaValidator): + + WORD_2006_NAMESPACE = "http://schemas.openxmlformats.org/wordprocessingml/2006/main" + W14_NAMESPACE = "http://schemas.microsoft.com/office/word/2010/wordml" + W16CID_NAMESPACE = "http://schemas.microsoft.com/office/word/2016/wordml/cid" + + ELEMENT_RELATIONSHIP_TYPES = {} + + def validate(self): + if not self.validate_xml(): + return False + + all_valid = True + if not self.validate_namespaces(): + all_valid = False + + if not self.validate_unique_ids(): + all_valid = False + + if not self.validate_file_references(): + all_valid = False + + if not self.validate_content_types(): + all_valid = False + + if not self.validate_against_xsd(): + all_valid = False + + if not self.validate_whitespace_preservation(): + all_valid = False + + if not self.validate_deletions(): + all_valid = False + + if not self.validate_insertions(): + all_valid = False + + if not self.validate_all_relationship_ids(): + all_valid = False + + if not self.validate_id_constraints(): + all_valid = False + + if not self.validate_comment_markers(): + all_valid = False + + self.compare_paragraph_counts() + + return all_valid + + def validate_whitespace_preservation(self): + errors = [] + + for xml_file in self.xml_files: + if xml_file.name != "document.xml": + continue + + try: + root = lxml.etree.parse(str(xml_file)).getroot() + + for elem in root.iter(f"{{{self.WORD_2006_NAMESPACE}}}t"): + if elem.text: + text = elem.text + if re.search(r"^[ \t\n\r]", text) or re.search( + r"[ \t\n\r]$", text + ): + xml_space_attr = f"{{{self.XML_NAMESPACE}}}space" + if ( + xml_space_attr not in elem.attrib + or elem.attrib[xml_space_attr] != "preserve" + ): + text_preview = ( + repr(text)[:50] + "..." + if len(repr(text)) > 50 + else repr(text) + ) + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {elem.sourceline}: w:t element with whitespace missing xml:space='preserve': {text_preview}" + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} whitespace preservation violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All whitespace is properly preserved") + return True + + def validate_deletions(self): + errors = [] + + for xml_file in self.xml_files: + if xml_file.name != "document.xml": + continue + + try: + root = lxml.etree.parse(str(xml_file)).getroot() + namespaces = {"w": self.WORD_2006_NAMESPACE} + + for t_elem in root.xpath(".//w:del//w:t", namespaces=namespaces): + if t_elem.text: + text_preview = ( + repr(t_elem.text)[:50] + "..." + if len(repr(t_elem.text)) > 50 + else repr(t_elem.text) + ) + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {t_elem.sourceline}: found within : {text_preview}" + ) + + for instr_elem in root.xpath( + ".//w:del//w:instrText", namespaces=namespaces + ): + text_preview = ( + repr(instr_elem.text or "")[:50] + "..." + if len(repr(instr_elem.text or "")) > 50 + else repr(instr_elem.text or "") + ) + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {instr_elem.sourceline}: found within (use ): {text_preview}" + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} deletion validation violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - No w:t elements found within w:del elements") + return True + + def count_paragraphs_in_unpacked(self): + count = 0 + + for xml_file in self.xml_files: + if xml_file.name != "document.xml": + continue + + try: + root = lxml.etree.parse(str(xml_file)).getroot() + paragraphs = root.findall(f".//{{{self.WORD_2006_NAMESPACE}}}p") + count = len(paragraphs) + except Exception as e: + print(f"Error counting paragraphs in unpacked document: {e}") + + return count + + def count_paragraphs_in_original(self): + original = self.original_file + if original is None: + return 0 + + count = 0 + + try: + with tempfile.TemporaryDirectory() as temp_dir: + with zipfile.ZipFile(original, "r") as zip_ref: + zip_ref.extractall(temp_dir) + + doc_xml_path = os.path.join(temp_dir, "word", "document.xml") + root = lxml.etree.parse(doc_xml_path).getroot() + + paragraphs = root.findall(f".//{{{self.WORD_2006_NAMESPACE}}}p") + count = len(paragraphs) + + except Exception as e: + print(f"Error counting paragraphs in original document: {e}") + + return count + + def validate_insertions(self): + errors = [] + + for xml_file in self.xml_files: + if xml_file.name != "document.xml": + continue + + try: + root = lxml.etree.parse(str(xml_file)).getroot() + namespaces = {"w": self.WORD_2006_NAMESPACE} + + invalid_elements = root.xpath( + ".//w:ins//w:delText[not(ancestor::w:del)]", namespaces=namespaces + ) + + for elem in invalid_elements: + text_preview = ( + repr(elem.text or "")[:50] + "..." + if len(repr(elem.text or "")) > 50 + else repr(elem.text or "") + ) + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {elem.sourceline}: within : {text_preview}" + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} insertion validation violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - No w:delText elements within w:ins elements") + return True + + def compare_paragraph_counts(self): + original_count = self.count_paragraphs_in_original() + new_count = self.count_paragraphs_in_unpacked() + + diff = new_count - original_count + diff_str = f"+{diff}" if diff > 0 else str(diff) + print(f"\nParagraphs: {original_count} → {new_count} ({diff_str})") + + def _parse_id_value(self, val: str, base: int = 16) -> int: + return int(val, base) + + def validate_id_constraints(self): + errors = [] + para_id_attr = f"{{{self.W14_NAMESPACE}}}paraId" + durable_id_attr = f"{{{self.W16CID_NAMESPACE}}}durableId" + + for xml_file in self.xml_files: + try: + for elem in lxml.etree.parse(str(xml_file)).iter(): + if val := elem.get(para_id_attr): + if self._parse_id_value(val, base=16) >= 0x80000000: + errors.append( + f" {xml_file.name}:{elem.sourceline}: paraId={val} >= 0x80000000" + ) + + if val := elem.get(durable_id_attr): + if xml_file.name == "numbering.xml": + try: + if self._parse_id_value(val, base=10) >= 0x7FFFFFFF: + errors.append( + f" {xml_file.name}:{elem.sourceline}: " + f"durableId={val} >= 0x7FFFFFFF" + ) + except ValueError: + errors.append( + f" {xml_file.name}:{elem.sourceline}: " + f"durableId={val} must be decimal in numbering.xml" + ) + else: + if self._parse_id_value(val, base=16) >= 0x7FFFFFFF: + errors.append( + f" {xml_file.name}:{elem.sourceline}: " + f"durableId={val} >= 0x7FFFFFFF" + ) + except Exception: + pass + + if errors: + print(f"FAILED - {len(errors)} ID constraint violations:") + for e in errors: + print(e) + elif self.verbose: + print("PASSED - All paraId/durableId values within constraints") + return not errors + + def validate_comment_markers(self): + errors = [] + + document_xml = None + comments_xml = None + for xml_file in self.xml_files: + if xml_file.name == "document.xml" and "word" in str(xml_file): + document_xml = xml_file + elif xml_file.name == "comments.xml": + comments_xml = xml_file + + if not document_xml: + if self.verbose: + print("PASSED - No document.xml found (skipping comment validation)") + return True + + try: + doc_root = lxml.etree.parse(str(document_xml)).getroot() + namespaces = {"w": self.WORD_2006_NAMESPACE} + + range_starts = { + elem.get(f"{{{self.WORD_2006_NAMESPACE}}}id") + for elem in doc_root.xpath( + ".//w:commentRangeStart", namespaces=namespaces + ) + } + range_ends = { + elem.get(f"{{{self.WORD_2006_NAMESPACE}}}id") + for elem in doc_root.xpath( + ".//w:commentRangeEnd", namespaces=namespaces + ) + } + references = { + elem.get(f"{{{self.WORD_2006_NAMESPACE}}}id") + for elem in doc_root.xpath( + ".//w:commentReference", namespaces=namespaces + ) + } + + orphaned_ends = range_ends - range_starts + for comment_id in sorted( + orphaned_ends, key=lambda x: int(x) if x and x.isdigit() else 0 + ): + errors.append( + f' document.xml: commentRangeEnd id="{comment_id}" has no matching commentRangeStart' + ) + + orphaned_starts = range_starts - range_ends + for comment_id in sorted( + orphaned_starts, key=lambda x: int(x) if x and x.isdigit() else 0 + ): + errors.append( + f' document.xml: commentRangeStart id="{comment_id}" has no matching commentRangeEnd' + ) + + comment_ids = set() + if comments_xml and comments_xml.exists(): + comments_root = lxml.etree.parse(str(comments_xml)).getroot() + comment_ids = { + elem.get(f"{{{self.WORD_2006_NAMESPACE}}}id") + for elem in comments_root.xpath( + ".//w:comment", namespaces=namespaces + ) + } + + marker_ids = range_starts | range_ends | references + invalid_refs = marker_ids - comment_ids + for comment_id in sorted( + invalid_refs, key=lambda x: int(x) if x and x.isdigit() else 0 + ): + if comment_id: + errors.append( + f' document.xml: marker id="{comment_id}" references non-existent comment' + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append(f" Error parsing XML: {e}") + + if errors: + print(f"FAILED - {len(errors)} comment marker violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All comment markers properly paired") + return True + + def repair(self) -> int: + repairs = super().repair() + repairs += self.repair_durableId() + return repairs + + def repair_durableId(self) -> int: + repairs = 0 + + for xml_file in self.xml_files: + try: + content = xml_file.read_text(encoding="utf-8") + dom = defusedxml.minidom.parseString(content) + modified = False + + for elem in dom.getElementsByTagName("*"): + if not elem.hasAttribute("w16cid:durableId"): + continue + + durable_id = elem.getAttribute("w16cid:durableId") + needs_repair = False + + if xml_file.name == "numbering.xml": + try: + needs_repair = ( + self._parse_id_value(durable_id, base=10) >= 0x7FFFFFFF + ) + except ValueError: + needs_repair = True + else: + try: + needs_repair = ( + self._parse_id_value(durable_id, base=16) >= 0x7FFFFFFF + ) + except ValueError: + needs_repair = True + + if needs_repair: + value = random.randint(1, 0x7FFFFFFE) + if xml_file.name == "numbering.xml": + new_id = str(value) + else: + new_id = f"{value:08X}" + + elem.setAttribute("w16cid:durableId", new_id) + print( + f" Repaired: {xml_file.name}: durableId {durable_id} → {new_id}" + ) + repairs += 1 + modified = True + + if modified: + xml_file.write_bytes(dom.toxml(encoding="UTF-8")) + + except Exception: + pass + + return repairs + + +if __name__ == "__main__": + raise RuntimeError("This module should not be run directly.") diff --git a/mateclaw-server/src/main/resources/skills/docx/scripts/office/validators/pptx.py b/mateclaw-server/src/main/resources/skills/docx/scripts/office/validators/pptx.py new file mode 100644 index 00000000..09842aa9 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/docx/scripts/office/validators/pptx.py @@ -0,0 +1,275 @@ +""" +Validator for PowerPoint presentation XML files against XSD schemas. +""" + +import re + +from .base import BaseSchemaValidator + + +class PPTXSchemaValidator(BaseSchemaValidator): + + PRESENTATIONML_NAMESPACE = ( + "http://schemas.openxmlformats.org/presentationml/2006/main" + ) + + ELEMENT_RELATIONSHIP_TYPES = { + "sldid": "slide", + "sldmasterid": "slidemaster", + "notesmasterid": "notesmaster", + "sldlayoutid": "slidelayout", + "themeid": "theme", + "tablestyleid": "tablestyles", + } + + def validate(self): + if not self.validate_xml(): + return False + + all_valid = True + if not self.validate_namespaces(): + all_valid = False + + if not self.validate_unique_ids(): + all_valid = False + + if not self.validate_uuid_ids(): + all_valid = False + + if not self.validate_file_references(): + all_valid = False + + if not self.validate_slide_layout_ids(): + all_valid = False + + if not self.validate_content_types(): + all_valid = False + + if not self.validate_against_xsd(): + all_valid = False + + if not self.validate_notes_slide_references(): + all_valid = False + + if not self.validate_all_relationship_ids(): + all_valid = False + + if not self.validate_no_duplicate_slide_layouts(): + all_valid = False + + return all_valid + + def validate_uuid_ids(self): + import lxml.etree + + errors = [] + uuid_pattern = re.compile( + r"^[\{\(]?[0-9A-Fa-f]{8}-?[0-9A-Fa-f]{4}-?[0-9A-Fa-f]{4}-?[0-9A-Fa-f]{4}-?[0-9A-Fa-f]{12}[\}\)]?$" + ) + + for xml_file in self.xml_files: + try: + root = lxml.etree.parse(str(xml_file)).getroot() + + for elem in root.iter(): + for attr, value in elem.attrib.items(): + attr_name = attr.split("}")[-1].lower() + if attr_name == "id" or attr_name.endswith("id"): + if self._looks_like_uuid(value): + if not uuid_pattern.match(value): + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {elem.sourceline}: ID '{value}' appears to be a UUID but contains invalid hex characters" + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} UUID ID validation errors:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All UUID-like IDs contain valid hex values") + return True + + def _looks_like_uuid(self, value): + clean_value = value.strip("{}()").replace("-", "") + return len(clean_value) == 32 and all(c.isalnum() for c in clean_value) + + def validate_slide_layout_ids(self): + import lxml.etree + + errors = [] + + slide_masters = list(self.unpacked_dir.glob("ppt/slideMasters/*.xml")) + + if not slide_masters: + if self.verbose: + print("PASSED - No slide masters found") + return True + + for slide_master in slide_masters: + try: + root = lxml.etree.parse(str(slide_master)).getroot() + + rels_file = slide_master.parent / "_rels" / f"{slide_master.name}.rels" + + if not rels_file.exists(): + errors.append( + f" {slide_master.relative_to(self.unpacked_dir)}: " + f"Missing relationships file: {rels_file.relative_to(self.unpacked_dir)}" + ) + continue + + rels_root = lxml.etree.parse(str(rels_file)).getroot() + + valid_layout_rids = set() + for rel in rels_root.findall( + f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship" + ): + rel_type = rel.get("Type", "") + if "slideLayout" in rel_type: + valid_layout_rids.add(rel.get("Id")) + + for sld_layout_id in root.findall( + f".//{{{self.PRESENTATIONML_NAMESPACE}}}sldLayoutId" + ): + r_id = sld_layout_id.get( + f"{{{self.OFFICE_RELATIONSHIPS_NAMESPACE}}}id" + ) + layout_id = sld_layout_id.get("id") + + if r_id and r_id not in valid_layout_rids: + errors.append( + f" {slide_master.relative_to(self.unpacked_dir)}: " + f"Line {sld_layout_id.sourceline}: sldLayoutId with id='{layout_id}' " + f"references r:id='{r_id}' which is not found in slide layout relationships" + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {slide_master.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} slide layout ID validation errors:") + for error in errors: + print(error) + print( + "Remove invalid references or add missing slide layouts to the relationships file." + ) + return False + else: + if self.verbose: + print("PASSED - All slide layout IDs reference valid slide layouts") + return True + + def validate_no_duplicate_slide_layouts(self): + import lxml.etree + + errors = [] + slide_rels_files = list(self.unpacked_dir.glob("ppt/slides/_rels/*.xml.rels")) + + for rels_file in slide_rels_files: + try: + root = lxml.etree.parse(str(rels_file)).getroot() + + layout_rels = [ + rel + for rel in root.findall( + f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship" + ) + if "slideLayout" in rel.get("Type", "") + ] + + if len(layout_rels) > 1: + errors.append( + f" {rels_file.relative_to(self.unpacked_dir)}: has {len(layout_rels)} slideLayout references" + ) + + except Exception as e: + errors.append( + f" {rels_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print("FAILED - Found slides with duplicate slideLayout references:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All slides have exactly one slideLayout reference") + return True + + def validate_notes_slide_references(self): + import lxml.etree + + errors = [] + notes_slide_references = {} + + slide_rels_files = list(self.unpacked_dir.glob("ppt/slides/_rels/*.xml.rels")) + + if not slide_rels_files: + if self.verbose: + print("PASSED - No slide relationship files found") + return True + + for rels_file in slide_rels_files: + try: + root = lxml.etree.parse(str(rels_file)).getroot() + + for rel in root.findall( + f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship" + ): + rel_type = rel.get("Type", "") + if "notesSlide" in rel_type: + target = rel.get("Target", "") + if target: + normalized_target = target.replace("../", "") + + slide_name = rels_file.stem.replace( + ".xml", "" + ) + + if normalized_target not in notes_slide_references: + notes_slide_references[normalized_target] = [] + notes_slide_references[normalized_target].append( + (slide_name, rels_file) + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {rels_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + for target, references in notes_slide_references.items(): + if len(references) > 1: + slide_names = [ref[0] for ref in references] + errors.append( + f" Notes slide '{target}' is referenced by multiple slides: {', '.join(slide_names)}" + ) + for slide_name, rels_file in references: + errors.append(f" - {rels_file.relative_to(self.unpacked_dir)}") + + if errors: + print( + f"FAILED - Found {len([e for e in errors if not e.startswith(' ')])} notes slide reference validation errors:" + ) + for error in errors: + print(error) + print("Each slide may optionally have its own slide file.") + return False + else: + if self.verbose: + print("PASSED - All notes slide references are unique") + return True + + +if __name__ == "__main__": + raise RuntimeError("This module should not be run directly.") diff --git a/mateclaw-server/src/main/resources/skills/docx/scripts/office/validators/redlining.py b/mateclaw-server/src/main/resources/skills/docx/scripts/office/validators/redlining.py new file mode 100644 index 00000000..71c81b6b --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/docx/scripts/office/validators/redlining.py @@ -0,0 +1,247 @@ +""" +Validator for tracked changes in Word documents. +""" + +import subprocess +import tempfile +import zipfile +from pathlib import Path + + +class RedliningValidator: + + def __init__(self, unpacked_dir, original_docx, verbose=False, author="Claude"): + self.unpacked_dir = Path(unpacked_dir) + self.original_docx = Path(original_docx) + self.verbose = verbose + self.author = author + self.namespaces = { + "w": "http://schemas.openxmlformats.org/wordprocessingml/2006/main" + } + + def repair(self) -> int: + return 0 + + def validate(self): + modified_file = self.unpacked_dir / "word" / "document.xml" + if not modified_file.exists(): + print(f"FAILED - Modified document.xml not found at {modified_file}") + return False + + try: + import xml.etree.ElementTree as ET + + tree = ET.parse(modified_file) + root = tree.getroot() + + del_elements = root.findall(".//w:del", self.namespaces) + ins_elements = root.findall(".//w:ins", self.namespaces) + + author_del_elements = [ + elem + for elem in del_elements + if elem.get(f"{{{self.namespaces['w']}}}author") == self.author + ] + author_ins_elements = [ + elem + for elem in ins_elements + if elem.get(f"{{{self.namespaces['w']}}}author") == self.author + ] + + if not author_del_elements and not author_ins_elements: + if self.verbose: + print(f"PASSED - No tracked changes by {self.author} found.") + return True + + except Exception: + pass + + with tempfile.TemporaryDirectory() as temp_dir: + temp_path = Path(temp_dir) + + try: + with zipfile.ZipFile(self.original_docx, "r") as zip_ref: + zip_ref.extractall(temp_path) + except Exception as e: + print(f"FAILED - Error unpacking original docx: {e}") + return False + + original_file = temp_path / "word" / "document.xml" + if not original_file.exists(): + print( + f"FAILED - Original document.xml not found in {self.original_docx}" + ) + return False + + try: + import xml.etree.ElementTree as ET + + modified_tree = ET.parse(modified_file) + modified_root = modified_tree.getroot() + original_tree = ET.parse(original_file) + original_root = original_tree.getroot() + except ET.ParseError as e: + print(f"FAILED - Error parsing XML files: {e}") + return False + + self._remove_author_tracked_changes(original_root) + self._remove_author_tracked_changes(modified_root) + + modified_text = self._extract_text_content(modified_root) + original_text = self._extract_text_content(original_root) + + if modified_text != original_text: + error_message = self._generate_detailed_diff( + original_text, modified_text + ) + print(error_message) + return False + + if self.verbose: + print(f"PASSED - All changes by {self.author} are properly tracked") + return True + + def _generate_detailed_diff(self, original_text, modified_text): + error_parts = [ + f"FAILED - Document text doesn't match after removing {self.author}'s tracked changes", + "", + "Likely causes:", + " 1. Modified text inside another author's or tags", + " 2. Made edits without proper tracked changes", + " 3. Didn't nest inside when deleting another's insertion", + "", + "For pre-redlined documents, use correct patterns:", + " - To reject another's INSERTION: Nest inside their ", + " - To restore another's DELETION: Add new AFTER their ", + "", + ] + + git_diff = self._get_git_word_diff(original_text, modified_text) + if git_diff: + error_parts.extend(["Differences:", "============", git_diff]) + else: + error_parts.append("Unable to generate word diff (git not available)") + + return "\n".join(error_parts) + + def _get_git_word_diff(self, original_text, modified_text): + try: + with tempfile.TemporaryDirectory() as temp_dir: + temp_path = Path(temp_dir) + + original_file = temp_path / "original.txt" + modified_file = temp_path / "modified.txt" + + original_file.write_text(original_text, encoding="utf-8") + modified_file.write_text(modified_text, encoding="utf-8") + + result = subprocess.run( + [ + "git", + "diff", + "--word-diff=plain", + "--word-diff-regex=.", + "-U0", + "--no-index", + str(original_file), + str(modified_file), + ], + capture_output=True, + text=True, + ) + + if result.stdout.strip(): + lines = result.stdout.split("\n") + content_lines = [] + in_content = False + for line in lines: + if line.startswith("@@"): + in_content = True + continue + if in_content and line.strip(): + content_lines.append(line) + + if content_lines: + return "\n".join(content_lines) + + result = subprocess.run( + [ + "git", + "diff", + "--word-diff=plain", + "-U0", + "--no-index", + str(original_file), + str(modified_file), + ], + capture_output=True, + text=True, + ) + + if result.stdout.strip(): + lines = result.stdout.split("\n") + content_lines = [] + in_content = False + for line in lines: + if line.startswith("@@"): + in_content = True + continue + if in_content and line.strip(): + content_lines.append(line) + return "\n".join(content_lines) + + except (subprocess.CalledProcessError, FileNotFoundError, Exception): + pass + + return None + + def _remove_author_tracked_changes(self, root): + ins_tag = f"{{{self.namespaces['w']}}}ins" + del_tag = f"{{{self.namespaces['w']}}}del" + author_attr = f"{{{self.namespaces['w']}}}author" + + for parent in root.iter(): + to_remove = [] + for child in parent: + if child.tag == ins_tag and child.get(author_attr) == self.author: + to_remove.append(child) + for elem in to_remove: + parent.remove(elem) + + deltext_tag = f"{{{self.namespaces['w']}}}delText" + t_tag = f"{{{self.namespaces['w']}}}t" + + for parent in root.iter(): + to_process = [] + for child in parent: + if child.tag == del_tag and child.get(author_attr) == self.author: + to_process.append((child, list(parent).index(child))) + + for del_elem, del_index in reversed(to_process): + for elem in del_elem.iter(): + if elem.tag == deltext_tag: + elem.tag = t_tag + + for child in reversed(list(del_elem)): + parent.insert(del_index, child) + parent.remove(del_elem) + + def _extract_text_content(self, root): + p_tag = f"{{{self.namespaces['w']}}}p" + t_tag = f"{{{self.namespaces['w']}}}t" + + paragraphs = [] + for p_elem in root.findall(f".//{p_tag}"): + text_parts = [] + for t_elem in p_elem.findall(f".//{t_tag}"): + if t_elem.text: + text_parts.append(t_elem.text) + paragraph_text = "".join(text_parts) + if paragraph_text: + paragraphs.append(paragraph_text) + + return "\n".join(paragraphs) + + +if __name__ == "__main__": + raise RuntimeError("This module should not be run directly.") diff --git a/mateclaw-server/src/main/resources/skills/docx/scripts/templates/comments.xml b/mateclaw-server/src/main/resources/skills/docx/scripts/templates/comments.xml new file mode 100644 index 00000000..cd01a7d7 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/docx/scripts/templates/comments.xml @@ -0,0 +1,3 @@ + + + diff --git a/mateclaw-server/src/main/resources/skills/docx/scripts/templates/commentsExtended.xml b/mateclaw-server/src/main/resources/skills/docx/scripts/templates/commentsExtended.xml new file mode 100644 index 00000000..411003cc --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/docx/scripts/templates/commentsExtended.xml @@ -0,0 +1,3 @@ + + + diff --git a/mateclaw-server/src/main/resources/skills/docx/scripts/templates/commentsExtensible.xml b/mateclaw-server/src/main/resources/skills/docx/scripts/templates/commentsExtensible.xml new file mode 100644 index 00000000..f5572d71 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/docx/scripts/templates/commentsExtensible.xml @@ -0,0 +1,3 @@ + + + diff --git a/mateclaw-server/src/main/resources/skills/docx/scripts/templates/commentsIds.xml b/mateclaw-server/src/main/resources/skills/docx/scripts/templates/commentsIds.xml new file mode 100644 index 00000000..32f1629f --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/docx/scripts/templates/commentsIds.xml @@ -0,0 +1,3 @@ + + + diff --git a/mateclaw-server/src/main/resources/skills/docx/scripts/templates/people.xml b/mateclaw-server/src/main/resources/skills/docx/scripts/templates/people.xml new file mode 100644 index 00000000..3803d2de --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/docx/scripts/templates/people.xml @@ -0,0 +1,3 @@ + + + diff --git a/mateclaw-server/src/main/resources/skills/gzh_article/SKILL.md b/mateclaw-server/src/main/resources/skills/gzh_article/SKILL.md new file mode 100644 index 00000000..7526b14b --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/gzh_article/SKILL.md @@ -0,0 +1,130 @@ +--- +name: gzh_article +description: '公众号图文创作 / 推文 / 官方号文章 (official account article) — 端到端：选题→搜集→成文→配图→去AI化→公众号内联样式排版→交付/草稿箱。honors user persona & style memory.' +version: 1.3.0 +tags: +- 公众号 +- 图文 +- 内容创作 +- writing +- wechat +platforms: + - macos + - linux + - windows +--- + +# 公众号图文创作 + +把一个选题做成可直接粘进公众号编辑器的图文推文。7 个阶段，每步都接到平台真实工具上。 + +> 📌 **动笔前务必先读 `references/gzh_platform_rules.md`** —— 那是微信公众号的真实平台规矩（封面尺寸、标题/摘要、编辑器排版、诱导分享/关注红线、群发频次与时机、原创机制），懂了这些才叫"会做公众号"，而不只是"会写字"。下面 SOP 的成文/配图/自查各步都以它为准。 + +## 开工前：读取共享人设记忆 + +先用 `recall_structured` 取回并全程遵守： + +- `content_persona` — 人设 / 口吻 +- `writing_style_gzh` — 公众号文风 +- `topic_interests` — 选题方向 +- `banned_words` — 禁用 / 敏感词 +- `signature_blocks` — 固定开场 / 结尾 / 引导关注段 + +取不到就用中性默认，不要编造。 + +## 七步 SOP + +### 1. 选题 + +用 `web_search`（`freshness=week`、`language=zh-CN`）围绕 `topic_interests` 找近期热点和角度，`count` 取 5–8。给用户 3–5 个候选选题（每个带一句话切入角度），让其确认或补充后再往下。 + +### 2. 搜集汇总（参考文章） + +用户给出参考公众号链接时： + +- **优先**用可能存在的 `wechat_article_extract` 工具直接抽正文（若该工具可用，用 `load_skill` 或工具列表确认）。 +- 没有该工具，就用 `browser_use`：先 `action=open` 打开链接，再 `action=snapshot` 抓取页面可见正文。 +- 把每篇参考的**核心观点、结构、可借鉴角度**提炼成要点。 + +> **红线**：本技能产出**原创**内容，参考只用于找角度、补事实，并在文末标注引用来源。**严禁洗稿 / 搬运 / 逐段改写**他人文章。 + +### 3. 成文 + +按公众号结构模板成文（详见 `references/gzh_structure.md`）： + +1. **钩子引言** — 用具体场景 / 反常识数据 / 一个问题抓住读者。 +2. **3–5 个小标题分节** — 每节一个论点，配**具体案例或数据**，不要空谈。 +3. **金句** — 每节或结尾埋一句可摘录的话。 +4. **结尾行动号召 + 引导关注** — 用 `signature_blocks` 里的固定收尾。 + +全程遵守 `writing_style_gzh` + `content_persona`。 + +**同时定好标题和摘要**（打开率的命门，见 `references/gzh_platform_rules.md`）： +- **标题**：≤30 字为佳（后台上限 64），带具体信息 / 数字 / 悬念 / 情绪；**与正文一致**，不做标题党、不用"震惊体"、别满屏感叹号（会被限流）。给 2–3 个候选让用户挑。 +- **摘要（digest）**：**手写 ≤120 字**，补一句标题没说完的钩子（不填会被系统乱截正文前 54 字）。交付时作为 `gzh_publish` 的 `digest` 传入。 + +### 4. 配图 + +用 `image_generate`（`action=generate`）： + +- **封面头图**：`aspectRatio=landscape`（公众号头图是 **2.35:1**，约 900×383 / 1080×460）。prompt 写清主题、风格、留白；**封面文字要大而少**（缩略图很小，一行主标题足矣）。 +- **分享/朋友圈封面**：如需单独出，用 `aspectRatio=square`（**1:1**，≥500×500）。 +- **关键小节配图**：按需为 2–3 个重点小节各生成一张，风格与封面统一。 + +`image_generate` 只认 `landscape` / `portrait` / `square` 三种比例，其它比例映射到最近的一个（如 3:4 → portrait）。 + +### 5. 去 AI 化 + +`load_skill deai_humanize`，然后对全文跑它的"打分→改写→复检"循环，`platform=gzh`，目标 `score ≤ 55`。 + +### 6. 违禁词 + 平台规则自查 + +对照 `banned_words` 和 `references/compliance_checklist.md`，命中就**标注并给出替换建议**，不要静默通过。两类都要扫： +- **广告法**：极限词（最/第一/唯一/国家级/100%）、虚假功效、承诺收益、敏感内容、侵权。 +- **微信平台红线**（比广告法更容易封号，见 `references/gzh_platform_rules.md` 第 5 节）：**诱导分享**（集赞/助力/分享解锁）、**诱导关注**（关注才能看全文）、**违规外链/二维码**、标题党。引导互动只用话术、不用利诱。 + +**产品教程/操作类文章配真实截图**：写"如何用 MateClaw 做 XX"这类教程时，用 `capture_screenshot(path)` 截真实后台界面（`path` 为站内相对路径，如 `/chat`、`/channels`、`/agents`、`/skills`），把返回的图片 URL 以 `![步骤说明](URL)` **直接嵌进对应步骤的 Markdown**，替代【截图】占位；再整体交给 `gzh_package`。这样成品里是真实产品截图，不用手动补图。 + +### 7. 打包交付（gzh_package —— 在线预览 + 素材下载） + +**默认用 `gzh_package` 交付**。你只需把成稿正文以 **Markdown** 形式传入，服务端会转成公众号内联样式 HTML（公众号不认 ` + + +
+

{{TITLE}}

+

{{SUBTITLE}}

+
    + {{POINTS}} + +
+ + diff --git a/mateclaw-server/src/main/resources/skills/xhs_note/references/xhs_card_cover.html b/mateclaw-server/src/main/resources/skills/xhs_note/references/xhs_card_cover.html new file mode 100644 index 00000000..7d5648d9 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xhs_note/references/xhs_card_cover.html @@ -0,0 +1,64 @@ + + + + + + + + +
+ {{BADGE}} +

{{TITLE}}

+

{{SUBTITLE}}

+
{{FOOTER}}
+ + diff --git a/mateclaw-server/src/main/resources/skills/xhs_note/references/xhs_card_end.html b/mateclaw-server/src/main/resources/skills/xhs_note/references/xhs_card_end.html new file mode 100644 index 00000000..43c3fceb --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xhs_note/references/xhs_card_end.html @@ -0,0 +1,55 @@ + + + + + + + + +
💛
+

{{TITLE}}

+
{{CTA}}
+

{{TAGS}}

+
{{HANDLE}}
+ + diff --git a/mateclaw-server/src/main/resources/skills/xhs_note/references/xhs_card_quote.html b/mateclaw-server/src/main/resources/skills/xhs_note/references/xhs_card_quote.html new file mode 100644 index 00000000..7ca8e612 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xhs_note/references/xhs_card_quote.html @@ -0,0 +1,81 @@ + + + + + + + + +
+
{{TAG}}
+
“
+

{{QUOTE}}

+
+

{{SOURCE}}

+
{{FOOTER}}
+ + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/helpers/__init__.py b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/helpers/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/helpers/merge_runs.py b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/helpers/merge_runs.py new file mode 100644 index 00000000..ad7c25ee --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/helpers/merge_runs.py @@ -0,0 +1,199 @@ +"""Merge adjacent runs with identical formatting in DOCX. + +Merges adjacent elements that have identical properties. +Works on runs in paragraphs and inside tracked changes (, ). + +Also: +- Removes rsid attributes from runs (revision metadata that doesn't affect rendering) +- Removes proofErr elements (spell/grammar markers that block merging) +""" + +from pathlib import Path + +import defusedxml.minidom + + +def merge_runs(input_dir: str) -> tuple[int, str]: + doc_xml = Path(input_dir) / "word" / "document.xml" + + if not doc_xml.exists(): + return 0, f"Error: {doc_xml} not found" + + try: + dom = defusedxml.minidom.parseString(doc_xml.read_text(encoding="utf-8")) + root = dom.documentElement + + _remove_elements(root, "proofErr") + _strip_run_rsid_attrs(root) + + containers = {run.parentNode for run in _find_elements(root, "r")} + + merge_count = 0 + for container in containers: + merge_count += _merge_runs_in(container) + + doc_xml.write_bytes(dom.toxml(encoding="UTF-8")) + return merge_count, f"Merged {merge_count} runs" + + except Exception as e: + return 0, f"Error: {e}" + + + + +def _find_elements(root, tag: str) -> list: + results = [] + + def traverse(node): + if node.nodeType == node.ELEMENT_NODE: + name = node.localName or node.tagName + if name == tag or name.endswith(f":{tag}"): + results.append(node) + for child in node.childNodes: + traverse(child) + + traverse(root) + return results + + +def _get_child(parent, tag: str): + for child in parent.childNodes: + if child.nodeType == child.ELEMENT_NODE: + name = child.localName or child.tagName + if name == tag or name.endswith(f":{tag}"): + return child + return None + + +def _get_children(parent, tag: str) -> list: + results = [] + for child in parent.childNodes: + if child.nodeType == child.ELEMENT_NODE: + name = child.localName or child.tagName + if name == tag or name.endswith(f":{tag}"): + results.append(child) + return results + + +def _is_adjacent(elem1, elem2) -> bool: + node = elem1.nextSibling + while node: + if node == elem2: + return True + if node.nodeType == node.ELEMENT_NODE: + return False + if node.nodeType == node.TEXT_NODE and node.data.strip(): + return False + node = node.nextSibling + return False + + + + +def _remove_elements(root, tag: str): + for elem in _find_elements(root, tag): + if elem.parentNode: + elem.parentNode.removeChild(elem) + + +def _strip_run_rsid_attrs(root): + for run in _find_elements(root, "r"): + for attr in list(run.attributes.values()): + if "rsid" in attr.name.lower(): + run.removeAttribute(attr.name) + + + + +def _merge_runs_in(container) -> int: + merge_count = 0 + run = _first_child_run(container) + + while run: + while True: + next_elem = _next_element_sibling(run) + if next_elem and _is_run(next_elem) and _can_merge(run, next_elem): + _merge_run_content(run, next_elem) + container.removeChild(next_elem) + merge_count += 1 + else: + break + + _consolidate_text(run) + run = _next_sibling_run(run) + + return merge_count + + +def _first_child_run(container): + for child in container.childNodes: + if child.nodeType == child.ELEMENT_NODE and _is_run(child): + return child + return None + + +def _next_element_sibling(node): + sibling = node.nextSibling + while sibling: + if sibling.nodeType == sibling.ELEMENT_NODE: + return sibling + sibling = sibling.nextSibling + return None + + +def _next_sibling_run(node): + sibling = node.nextSibling + while sibling: + if sibling.nodeType == sibling.ELEMENT_NODE: + if _is_run(sibling): + return sibling + sibling = sibling.nextSibling + return None + + +def _is_run(node) -> bool: + name = node.localName or node.tagName + return name == "r" or name.endswith(":r") + + +def _can_merge(run1, run2) -> bool: + rpr1 = _get_child(run1, "rPr") + rpr2 = _get_child(run2, "rPr") + + if (rpr1 is None) != (rpr2 is None): + return False + if rpr1 is None: + return True + return rpr1.toxml() == rpr2.toxml() + + +def _merge_run_content(target, source): + for child in list(source.childNodes): + if child.nodeType == child.ELEMENT_NODE: + name = child.localName or child.tagName + if name != "rPr" and not name.endswith(":rPr"): + target.appendChild(child) + + +def _consolidate_text(run): + t_elements = _get_children(run, "t") + + for i in range(len(t_elements) - 1, 0, -1): + curr, prev = t_elements[i], t_elements[i - 1] + + if _is_adjacent(prev, curr): + prev_text = prev.firstChild.data if prev.firstChild else "" + curr_text = curr.firstChild.data if curr.firstChild else "" + merged = prev_text + curr_text + + if prev.firstChild: + prev.firstChild.data = merged + else: + prev.appendChild(run.ownerDocument.createTextNode(merged)) + + if merged.startswith(" ") or merged.endswith(" "): + prev.setAttribute("xml:space", "preserve") + elif prev.hasAttribute("xml:space"): + prev.removeAttribute("xml:space") + + run.removeChild(curr) diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/helpers/simplify_redlines.py b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/helpers/simplify_redlines.py new file mode 100644 index 00000000..db963bb9 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/helpers/simplify_redlines.py @@ -0,0 +1,197 @@ +"""Simplify tracked changes by merging adjacent w:ins or w:del elements. + +Merges adjacent elements from the same author into a single element. +Same for elements. This makes heavily-redlined documents easier to +work with by reducing the number of tracked change wrappers. + +Rules: +- Only merges w:ins with w:ins, w:del with w:del (same element type) +- Only merges if same author (ignores timestamp differences) +- Only merges if truly adjacent (only whitespace between them) +""" + +import xml.etree.ElementTree as ET +import zipfile +from pathlib import Path + +import defusedxml.minidom + +WORD_NS = "http://schemas.openxmlformats.org/wordprocessingml/2006/main" + + +def simplify_redlines(input_dir: str) -> tuple[int, str]: + doc_xml = Path(input_dir) / "word" / "document.xml" + + if not doc_xml.exists(): + return 0, f"Error: {doc_xml} not found" + + try: + dom = defusedxml.minidom.parseString(doc_xml.read_text(encoding="utf-8")) + root = dom.documentElement + + merge_count = 0 + + containers = _find_elements(root, "p") + _find_elements(root, "tc") + + for container in containers: + merge_count += _merge_tracked_changes_in(container, "ins") + merge_count += _merge_tracked_changes_in(container, "del") + + doc_xml.write_bytes(dom.toxml(encoding="UTF-8")) + return merge_count, f"Simplified {merge_count} tracked changes" + + except Exception as e: + return 0, f"Error: {e}" + + +def _merge_tracked_changes_in(container, tag: str) -> int: + merge_count = 0 + + tracked = [ + child + for child in container.childNodes + if child.nodeType == child.ELEMENT_NODE and _is_element(child, tag) + ] + + if len(tracked) < 2: + return 0 + + i = 0 + while i < len(tracked) - 1: + curr = tracked[i] + next_elem = tracked[i + 1] + + if _can_merge_tracked(curr, next_elem): + _merge_tracked_content(curr, next_elem) + container.removeChild(next_elem) + tracked.pop(i + 1) + merge_count += 1 + else: + i += 1 + + return merge_count + + +def _is_element(node, tag: str) -> bool: + name = node.localName or node.tagName + return name == tag or name.endswith(f":{tag}") + + +def _get_author(elem) -> str: + author = elem.getAttribute("w:author") + if not author: + for attr in elem.attributes.values(): + if attr.localName == "author" or attr.name.endswith(":author"): + return attr.value + return author + + +def _can_merge_tracked(elem1, elem2) -> bool: + if _get_author(elem1) != _get_author(elem2): + return False + + node = elem1.nextSibling + while node and node != elem2: + if node.nodeType == node.ELEMENT_NODE: + return False + if node.nodeType == node.TEXT_NODE and node.data.strip(): + return False + node = node.nextSibling + + return True + + +def _merge_tracked_content(target, source): + while source.firstChild: + child = source.firstChild + source.removeChild(child) + target.appendChild(child) + + +def _find_elements(root, tag: str) -> list: + results = [] + + def traverse(node): + if node.nodeType == node.ELEMENT_NODE: + name = node.localName or node.tagName + if name == tag or name.endswith(f":{tag}"): + results.append(node) + for child in node.childNodes: + traverse(child) + + traverse(root) + return results + + +def get_tracked_change_authors(doc_xml_path: Path) -> dict[str, int]: + if not doc_xml_path.exists(): + return {} + + try: + tree = ET.parse(doc_xml_path) + root = tree.getroot() + except ET.ParseError: + return {} + + namespaces = {"w": WORD_NS} + author_attr = f"{{{WORD_NS}}}author" + + authors: dict[str, int] = {} + for tag in ["ins", "del"]: + for elem in root.findall(f".//w:{tag}", namespaces): + author = elem.get(author_attr) + if author: + authors[author] = authors.get(author, 0) + 1 + + return authors + + +def _get_authors_from_docx(docx_path: Path) -> dict[str, int]: + try: + with zipfile.ZipFile(docx_path, "r") as zf: + if "word/document.xml" not in zf.namelist(): + return {} + with zf.open("word/document.xml") as f: + tree = ET.parse(f) + root = tree.getroot() + + namespaces = {"w": WORD_NS} + author_attr = f"{{{WORD_NS}}}author" + + authors: dict[str, int] = {} + for tag in ["ins", "del"]: + for elem in root.findall(f".//w:{tag}", namespaces): + author = elem.get(author_attr) + if author: + authors[author] = authors.get(author, 0) + 1 + return authors + except (zipfile.BadZipFile, ET.ParseError): + return {} + + +def infer_author(modified_dir: Path, original_docx: Path, default: str = "Claude") -> str: + modified_xml = modified_dir / "word" / "document.xml" + modified_authors = get_tracked_change_authors(modified_xml) + + if not modified_authors: + return default + + original_authors = _get_authors_from_docx(original_docx) + + new_changes: dict[str, int] = {} + for author, count in modified_authors.items(): + original_count = original_authors.get(author, 0) + diff = count - original_count + if diff > 0: + new_changes[author] = diff + + if not new_changes: + return default + + if len(new_changes) == 1: + return next(iter(new_changes)) + + raise ValueError( + f"Multiple authors added new changes: {new_changes}. " + "Cannot infer which author to validate." + ) diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/pack.py b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/pack.py new file mode 100755 index 00000000..db29ed8b --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/pack.py @@ -0,0 +1,159 @@ +"""Pack a directory into a DOCX, PPTX, or XLSX file. + +Validates with auto-repair, condenses XML formatting, and creates the Office file. + +Usage: + python pack.py [--original ] [--validate true|false] + +Examples: + python pack.py unpacked/ output.docx --original input.docx + python pack.py unpacked/ output.pptx --validate false +""" + +import argparse +import sys +import shutil +import tempfile +import zipfile +from pathlib import Path + +import defusedxml.minidom + +from validators import DOCXSchemaValidator, PPTXSchemaValidator, RedliningValidator + +def pack( + input_directory: str, + output_file: str, + original_file: str | None = None, + validate: bool = True, + infer_author_func=None, +) -> tuple[None, str]: + input_dir = Path(input_directory) + output_path = Path(output_file) + suffix = output_path.suffix.lower() + + if not input_dir.is_dir(): + return None, f"Error: {input_dir} is not a directory" + + if suffix not in {".docx", ".pptx", ".xlsx"}: + return None, f"Error: {output_file} must be a .docx, .pptx, or .xlsx file" + + if validate and original_file: + original_path = Path(original_file) + if original_path.exists(): + success, output = _run_validation( + input_dir, original_path, suffix, infer_author_func + ) + if output: + print(output) + if not success: + return None, f"Error: Validation failed for {input_dir}" + + with tempfile.TemporaryDirectory() as temp_dir: + temp_content_dir = Path(temp_dir) / "content" + shutil.copytree(input_dir, temp_content_dir) + + for pattern in ["*.xml", "*.rels"]: + for xml_file in temp_content_dir.rglob(pattern): + _condense_xml(xml_file) + + output_path.parent.mkdir(parents=True, exist_ok=True) + with zipfile.ZipFile(output_path, "w", zipfile.ZIP_DEFLATED) as zf: + for f in temp_content_dir.rglob("*"): + if f.is_file(): + zf.write(f, f.relative_to(temp_content_dir)) + + return None, f"Successfully packed {input_dir} to {output_file}" + + +def _run_validation( + unpacked_dir: Path, + original_file: Path, + suffix: str, + infer_author_func=None, +) -> tuple[bool, str | None]: + output_lines = [] + validators = [] + + if suffix == ".docx": + author = "Claude" + if infer_author_func: + try: + author = infer_author_func(unpacked_dir, original_file) + except ValueError as e: + print(f"Warning: {e} Using default author 'Claude'.", file=sys.stderr) + + validators = [ + DOCXSchemaValidator(unpacked_dir, original_file), + RedliningValidator(unpacked_dir, original_file, author=author), + ] + elif suffix == ".pptx": + validators = [PPTXSchemaValidator(unpacked_dir, original_file)] + + if not validators: + return True, None + + total_repairs = sum(v.repair() for v in validators) + if total_repairs: + output_lines.append(f"Auto-repaired {total_repairs} issue(s)") + + success = all(v.validate() for v in validators) + + if success: + output_lines.append("All validations PASSED!") + + return success, "\n".join(output_lines) if output_lines else None + + +def _condense_xml(xml_file: Path) -> None: + try: + with open(xml_file, encoding="utf-8") as f: + dom = defusedxml.minidom.parse(f) + + for element in dom.getElementsByTagName("*"): + if element.tagName.endswith(":t"): + continue + + for child in list(element.childNodes): + if ( + child.nodeType == child.TEXT_NODE + and child.nodeValue + and child.nodeValue.strip() == "" + ) or child.nodeType == child.COMMENT_NODE: + element.removeChild(child) + + xml_file.write_bytes(dom.toxml(encoding="UTF-8")) + except Exception as e: + print(f"ERROR: Failed to parse {xml_file.name}: {e}", file=sys.stderr) + raise + + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description="Pack a directory into a DOCX, PPTX, or XLSX file" + ) + parser.add_argument("input_directory", help="Unpacked Office document directory") + parser.add_argument("output_file", help="Output Office file (.docx/.pptx/.xlsx)") + parser.add_argument( + "--original", + help="Original file for validation comparison", + ) + parser.add_argument( + "--validate", + type=lambda x: x.lower() == "true", + default=True, + metavar="true|false", + help="Run validation with auto-repair (default: true)", + ) + args = parser.parse_args() + + _, message = pack( + args.input_directory, + args.output_file, + original_file=args.original, + validate=args.validate, + ) + print(message) + + if "Error" in message: + sys.exit(1) diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chart.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chart.xsd new file mode 100644 index 00000000..6454ef9a --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chart.xsd @@ -0,0 +1,1499 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chartDrawing.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chartDrawing.xsd new file mode 100644 index 00000000..afa4f463 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chartDrawing.xsd @@ -0,0 +1,146 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-diagram.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-diagram.xsd new file mode 100644 index 00000000..64e66b8a --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-diagram.xsd @@ -0,0 +1,1085 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-lockedCanvas.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-lockedCanvas.xsd new file mode 100644 index 00000000..687eea82 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-lockedCanvas.xsd @@ -0,0 +1,11 @@ + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-main.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-main.xsd new file mode 100644 index 00000000..6ac81b06 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-main.xsd @@ -0,0 +1,3081 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-picture.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-picture.xsd new file mode 100644 index 00000000..1dbf0514 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-picture.xsd @@ -0,0 +1,23 @@ + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-spreadsheetDrawing.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-spreadsheetDrawing.xsd new file mode 100644 index 00000000..f1af17db --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-spreadsheetDrawing.xsd @@ -0,0 +1,185 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-wordprocessingDrawing.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-wordprocessingDrawing.xsd new file mode 100644 index 00000000..0a185ab6 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-wordprocessingDrawing.xsd @@ -0,0 +1,287 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/pml.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/pml.xsd new file mode 100644 index 00000000..14ef4888 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/pml.xsd @@ -0,0 +1,1676 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-additionalCharacteristics.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-additionalCharacteristics.xsd new file mode 100644 index 00000000..c20f3bf1 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-additionalCharacteristics.xsd @@ -0,0 +1,28 @@ + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-bibliography.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-bibliography.xsd new file mode 100644 index 00000000..ac602522 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-bibliography.xsd @@ -0,0 +1,144 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-commonSimpleTypes.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-commonSimpleTypes.xsd new file mode 100644 index 00000000..424b8ba8 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-commonSimpleTypes.xsd @@ -0,0 +1,174 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlDataProperties.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlDataProperties.xsd new file mode 100644 index 00000000..2bddce29 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlDataProperties.xsd @@ -0,0 +1,25 @@ + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlSchemaProperties.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlSchemaProperties.xsd new file mode 100644 index 00000000..8a8c18ba --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlSchemaProperties.xsd @@ -0,0 +1,18 @@ + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesCustom.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesCustom.xsd new file mode 100644 index 00000000..5c42706a --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesCustom.xsd @@ -0,0 +1,59 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesExtended.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesExtended.xsd new file mode 100644 index 00000000..853c341c --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesExtended.xsd @@ -0,0 +1,56 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesVariantTypes.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesVariantTypes.xsd new file mode 100644 index 00000000..da835ee8 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesVariantTypes.xsd @@ -0,0 +1,195 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-math.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-math.xsd new file mode 100644 index 00000000..87ad2658 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-math.xsd @@ -0,0 +1,582 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-relationshipReference.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-relationshipReference.xsd new file mode 100644 index 00000000..9e86f1b2 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-relationshipReference.xsd @@ -0,0 +1,25 @@ + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/sml.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/sml.xsd new file mode 100644 index 00000000..d0be42e7 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/sml.xsd @@ -0,0 +1,4439 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-main.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-main.xsd new file mode 100644 index 00000000..8821dd18 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-main.xsd @@ -0,0 +1,570 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-officeDrawing.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-officeDrawing.xsd new file mode 100644 index 00000000..ca2575c7 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-officeDrawing.xsd @@ -0,0 +1,509 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-presentationDrawing.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-presentationDrawing.xsd new file mode 100644 index 00000000..dd079e60 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-presentationDrawing.xsd @@ -0,0 +1,12 @@ + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-spreadsheetDrawing.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-spreadsheetDrawing.xsd new file mode 100644 index 00000000..3dd6cf62 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-spreadsheetDrawing.xsd @@ -0,0 +1,108 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-wordprocessingDrawing.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-wordprocessingDrawing.xsd new file mode 100644 index 00000000..f1041e34 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-wordprocessingDrawing.xsd @@ -0,0 +1,96 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/wml.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/wml.xsd new file mode 100644 index 00000000..9c5b7a63 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/wml.xsd @@ -0,0 +1,3646 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/xml.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/xml.xsd new file mode 100644 index 00000000..0f13678d --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ISO-IEC29500-4_2016/xml.xsd @@ -0,0 +1,116 @@ + + + + + + See http://www.w3.org/XML/1998/namespace.html and + http://www.w3.org/TR/REC-xml for information about this namespace. + + This schema document describes the XML namespace, in a form + suitable for import by other schema documents. + + Note that local names in this namespace are intended to be defined + only by the World Wide Web Consortium or its subgroups. The + following names are currently defined in this namespace and should + not be used with conflicting semantics by any Working Group, + specification, or document instance: + + base (as an attribute name): denotes an attribute whose value + provides a URI to be used as the base for interpreting any + relative URIs in the scope of the element on which it + appears; its value is inherited. This name is reserved + by virtue of its definition in the XML Base specification. + + lang (as an attribute name): denotes an attribute whose value + is a language code for the natural language of the content of + any element; its value is inherited. This name is reserved + by virtue of its definition in the XML specification. + + space (as an attribute name): denotes an attribute whose + value is a keyword indicating what whitespace processing + discipline is intended for the content of the element; its + value is inherited. This name is reserved by virtue of its + definition in the XML specification. + + Father (in any context at all): denotes Jon Bosak, the chair of + the original XML Working Group. This name is reserved by + the following decision of the W3C XML Plenary and + XML Coordination groups: + + In appreciation for his vision, leadership and dedication + the W3C XML Plenary on this 10th day of February, 2000 + reserves for Jon Bosak in perpetuity the XML name + xml:Father + + + + + This schema defines attributes and an attribute group + suitable for use by + schemas wishing to allow xml:base, xml:lang or xml:space attributes + on elements they define. + + To enable this, such a schema must import this schema + for the XML namespace, e.g. as follows: + <schema . . .> + . . . + <import namespace="http://www.w3.org/XML/1998/namespace" + schemaLocation="http://www.w3.org/2001/03/xml.xsd"/> + + Subsequently, qualified reference to any of the attributes + or the group defined below will have the desired effect, e.g. + + <type . . .> + . . . + <attributeGroup ref="xml:specialAttrs"/> + + will define a type which will schema-validate an instance + element with any of those attributes + + + + In keeping with the XML Schema WG's standard versioning + policy, this schema document will persist at + http://www.w3.org/2001/03/xml.xsd. + At the date of issue it can also be found at + http://www.w3.org/2001/xml.xsd. + The schema document at that URI may however change in the future, + in order to remain compatible with the latest version of XML Schema + itself. In other words, if the XML Schema namespace changes, the version + of this document at + http://www.w3.org/2001/xml.xsd will change + accordingly; the version at + http://www.w3.org/2001/03/xml.xsd will not change. + + + + + + In due course, we should install the relevant ISO 2- and 3-letter + codes as the enumerated possible values . . . + + + + + + + + + + + + + + + See http://www.w3.org/TR/xmlbase/ for + information about this attribute. + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ecma/fouth-edition/opc-contentTypes.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ecma/fouth-edition/opc-contentTypes.xsd new file mode 100644 index 00000000..a6de9d27 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ecma/fouth-edition/opc-contentTypes.xsd @@ -0,0 +1,42 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ecma/fouth-edition/opc-coreProperties.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ecma/fouth-edition/opc-coreProperties.xsd new file mode 100644 index 00000000..10e978b6 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ecma/fouth-edition/opc-coreProperties.xsd @@ -0,0 +1,50 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ecma/fouth-edition/opc-digSig.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ecma/fouth-edition/opc-digSig.xsd new file mode 100644 index 00000000..4248bf7a --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ecma/fouth-edition/opc-digSig.xsd @@ -0,0 +1,49 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ecma/fouth-edition/opc-relationships.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ecma/fouth-edition/opc-relationships.xsd new file mode 100644 index 00000000..56497467 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/ecma/fouth-edition/opc-relationships.xsd @@ -0,0 +1,33 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/mce/mc.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/mce/mc.xsd new file mode 100644 index 00000000..ef725457 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/mce/mc.xsd @@ -0,0 +1,75 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-2010.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-2010.xsd new file mode 100644 index 00000000..f65f7777 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-2010.xsd @@ -0,0 +1,560 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-2012.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-2012.xsd new file mode 100644 index 00000000..6b00755a --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-2012.xsd @@ -0,0 +1,67 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-2018.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-2018.xsd new file mode 100644 index 00000000..f321d333 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-2018.xsd @@ -0,0 +1,14 @@ + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-cex-2018.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-cex-2018.xsd new file mode 100644 index 00000000..364c6a9b --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-cex-2018.xsd @@ -0,0 +1,20 @@ + + + + + + + + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-cid-2016.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-cid-2016.xsd new file mode 100644 index 00000000..fed9d15b --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-cid-2016.xsd @@ -0,0 +1,13 @@ + + + + + + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-sdtdatahash-2020.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-sdtdatahash-2020.xsd new file mode 100644 index 00000000..680cf154 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-sdtdatahash-2020.xsd @@ -0,0 +1,4 @@ + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-symex-2015.xsd b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-symex-2015.xsd new file mode 100644 index 00000000..89ada908 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/schemas/microsoft/wml-symex-2015.xsd @@ -0,0 +1,8 @@ + + + + + + + + diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/soffice.py b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/soffice.py new file mode 100644 index 00000000..072354fb --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/soffice.py @@ -0,0 +1,218 @@ +""" +Helper for running LibreOffice (soffice) in environments where AF_UNIX +sockets may be blocked (e.g., sandboxed VMs). Detects the restriction +at runtime and applies an LD_PRELOAD shim if needed. + +Usage: + from office.soffice import run_soffice, get_soffice_cmd, get_soffice_env + + # Option 1 – run soffice directly + result = run_soffice(["--headless", "--convert-to", "pdf", "input.docx"]) + + # Option 2 – get env dict for your own subprocess calls + env = get_soffice_env() + subprocess.run([get_soffice_cmd(), ...], env=env) +""" + +import os +import platform +import shutil +import socket +import subprocess +import tempfile +from pathlib import Path + + +def get_soffice_cmd() -> str: + """Return the soffice command name for the current platform.""" + # Prefer PATH first on all platforms. + path_cmd = shutil.which("soffice") + if path_cmd: + return path_cmd + + if platform.system() == "Windows": + # Windows can expose soffice as .com or .exe; try both on PATH first. + for candidate_name in ("soffice.com", "soffice.exe"): + path_candidate = shutil.which(candidate_name) + if path_candidate: + return path_candidate + + for prog_dir in ( + os.environ.get("PROGRAMFILES", r"C:\Program Files"), + os.environ.get("PROGRAMFILES(X86)", r"C:\Program Files (x86)"), + ): + if not prog_dir: + continue + program_dir = Path(prog_dir) / "LibreOffice" / "program" + for exe_name in ("soffice.com", "soffice.exe"): + candidate = program_dir / exe_name + if candidate.exists(): + return str(candidate) + return "soffice" + return "soffice" + + +def get_soffice_env() -> dict: + env = os.environ.copy() + if platform.system() == "Linux": + env["SAL_USE_VCLPLUGIN"] = "svp" + + if _needs_shim(): + shim = _ensure_shim() + env["LD_PRELOAD"] = str(shim) + + return env + + +def run_soffice(args: list[str], **kwargs) -> subprocess.CompletedProcess: + env = get_soffice_env() + return subprocess.run([get_soffice_cmd()] + args, env=env, **kwargs) + + + +_SHIM_SO = Path(tempfile.gettempdir()) / "lo_socket_shim.so" + + +def _needs_shim() -> bool: + # AF_UNIX / LD_PRELOAD shim is Linux-only; skip on Windows and macOS + if not hasattr(socket, "AF_UNIX"): + return False + try: + s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + s.close() + return False + except OSError: + return True + + +def _ensure_shim() -> Path: + if _SHIM_SO.exists(): + return _SHIM_SO + + src = Path(tempfile.gettempdir()) / "lo_socket_shim.c" + src.write_text(_SHIM_SOURCE) + subprocess.run( + ["gcc", "-shared", "-fPIC", "-o", str(_SHIM_SO), str(src), "-ldl"], + check=True, + capture_output=True, + ) + src.unlink() + return _SHIM_SO + + + +_SHIM_SOURCE = r""" +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include + +static int (*real_socket)(int, int, int); +static int (*real_socketpair)(int, int, int, int[2]); +static int (*real_listen)(int, int); +static int (*real_accept)(int, struct sockaddr *, socklen_t *); +static int (*real_close)(int); +static int (*real_read)(int, void *, size_t); + +/* Per-FD bookkeeping (FDs >= 1024 are passed through unshimmed). */ +static int is_shimmed[1024]; +static int peer_of[1024]; +static int wake_r[1024]; /* accept() blocks reading this */ +static int wake_w[1024]; /* close() writes to this */ +static int listener_fd = -1; /* FD that received listen() */ + +__attribute__((constructor)) +static void init(void) { + real_socket = dlsym(RTLD_NEXT, "socket"); + real_socketpair = dlsym(RTLD_NEXT, "socketpair"); + real_listen = dlsym(RTLD_NEXT, "listen"); + real_accept = dlsym(RTLD_NEXT, "accept"); + real_close = dlsym(RTLD_NEXT, "close"); + real_read = dlsym(RTLD_NEXT, "read"); + for (int i = 0; i < 1024; i++) { + peer_of[i] = -1; + wake_r[i] = -1; + wake_w[i] = -1; + } +} + +/* ---- socket ---------------------------------------------------------- */ +int socket(int domain, int type, int protocol) { + if (domain == AF_UNIX) { + int fd = real_socket(domain, type, protocol); + if (fd >= 0) return fd; + /* socket(AF_UNIX) blocked – fall back to socketpair(). */ + int sv[2]; + if (real_socketpair(domain, type, protocol, sv) == 0) { + if (sv[0] >= 0 && sv[0] < 1024) { + is_shimmed[sv[0]] = 1; + peer_of[sv[0]] = sv[1]; + int wp[2]; + if (pipe(wp) == 0) { + wake_r[sv[0]] = wp[0]; + wake_w[sv[0]] = wp[1]; + } + } + return sv[0]; + } + errno = EPERM; + return -1; + } + return real_socket(domain, type, protocol); +} + +/* ---- listen ---------------------------------------------------------- */ +int listen(int sockfd, int backlog) { + if (sockfd >= 0 && sockfd < 1024 && is_shimmed[sockfd]) { + listener_fd = sockfd; + return 0; + } + return real_listen(sockfd, backlog); +} + +/* ---- accept ---------------------------------------------------------- */ +int accept(int sockfd, struct sockaddr *addr, socklen_t *addrlen) { + if (sockfd >= 0 && sockfd < 1024 && is_shimmed[sockfd]) { + /* Block until close() writes to the wake pipe. */ + if (wake_r[sockfd] >= 0) { + char buf; + real_read(wake_r[sockfd], &buf, 1); + } + errno = ECONNABORTED; + return -1; + } + return real_accept(sockfd, addr, addrlen); +} + +/* ---- close ----------------------------------------------------------- */ +int close(int fd) { + if (fd >= 0 && fd < 1024 && is_shimmed[fd]) { + int was_listener = (fd == listener_fd); + is_shimmed[fd] = 0; + + if (wake_w[fd] >= 0) { /* unblock accept() */ + char c = 0; + write(wake_w[fd], &c, 1); + real_close(wake_w[fd]); + wake_w[fd] = -1; + } + if (wake_r[fd] >= 0) { real_close(wake_r[fd]); wake_r[fd] = -1; } + if (peer_of[fd] >= 0) { real_close(peer_of[fd]); peer_of[fd] = -1; } + + if (was_listener) + _exit(0); /* conversion done – exit */ + } + return real_close(fd); +} +""" + + + +if __name__ == "__main__": + import sys + result = run_soffice(sys.argv[1:]) + sys.exit(result.returncode) diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/unpack.py b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/unpack.py new file mode 100755 index 00000000..00152533 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/unpack.py @@ -0,0 +1,132 @@ +"""Unpack Office files (DOCX, PPTX, XLSX) for editing. + +Extracts the ZIP archive, pretty-prints XML files, and optionally: +- Merges adjacent runs with identical formatting (DOCX only) +- Simplifies adjacent tracked changes from same author (DOCX only) + +Usage: + python unpack.py [options] + +Examples: + python unpack.py document.docx unpacked/ + python unpack.py presentation.pptx unpacked/ + python unpack.py document.docx unpacked/ --merge-runs false +""" + +import argparse +import sys +import zipfile +from pathlib import Path + +import defusedxml.minidom + +from helpers.merge_runs import merge_runs as do_merge_runs +from helpers.simplify_redlines import simplify_redlines as do_simplify_redlines + +SMART_QUOTE_REPLACEMENTS = { + "\u201c": "“", + "\u201d": "”", + "\u2018": "‘", + "\u2019": "’", +} + + +def unpack( + input_file: str, + output_directory: str, + merge_runs: bool = True, + simplify_redlines: bool = True, +) -> tuple[None, str]: + input_path = Path(input_file) + output_path = Path(output_directory) + suffix = input_path.suffix.lower() + + if not input_path.exists(): + return None, f"Error: {input_file} does not exist" + + if suffix not in {".docx", ".pptx", ".xlsx"}: + return None, f"Error: {input_file} must be a .docx, .pptx, or .xlsx file" + + try: + output_path.mkdir(parents=True, exist_ok=True) + + with zipfile.ZipFile(input_path, "r") as zf: + zf.extractall(output_path) + + xml_files = list(output_path.rglob("*.xml")) + list(output_path.rglob("*.rels")) + for xml_file in xml_files: + _pretty_print_xml(xml_file) + + message = f"Unpacked {input_file} ({len(xml_files)} XML files)" + + if suffix == ".docx": + if simplify_redlines: + simplify_count, _ = do_simplify_redlines(str(output_path)) + message += f", simplified {simplify_count} tracked changes" + + if merge_runs: + merge_count, _ = do_merge_runs(str(output_path)) + message += f", merged {merge_count} runs" + + for xml_file in xml_files: + _escape_smart_quotes(xml_file) + + return None, message + + except zipfile.BadZipFile: + return None, f"Error: {input_file} is not a valid Office file" + except Exception as e: + return None, f"Error unpacking: {e}" + + +def _pretty_print_xml(xml_file: Path) -> None: + try: + content = xml_file.read_text(encoding="utf-8") + dom = defusedxml.minidom.parseString(content) + xml_file.write_bytes(dom.toprettyxml(indent=" ", encoding="utf-8")) + except Exception: + pass + + +def _escape_smart_quotes(xml_file: Path) -> None: + try: + content = xml_file.read_text(encoding="utf-8") + for char, entity in SMART_QUOTE_REPLACEMENTS.items(): + content = content.replace(char, entity) + xml_file.write_text(content, encoding="utf-8") + except Exception: + pass + + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description="Unpack an Office file (DOCX, PPTX, XLSX) for editing" + ) + parser.add_argument("input_file", help="Office file to unpack") + parser.add_argument("output_directory", help="Output directory") + parser.add_argument( + "--merge-runs", + type=lambda x: x.lower() == "true", + default=True, + metavar="true|false", + help="Merge adjacent runs with identical formatting (DOCX only, default: true)", + ) + parser.add_argument( + "--simplify-redlines", + type=lambda x: x.lower() == "true", + default=True, + metavar="true|false", + help="Merge adjacent tracked changes from same author (DOCX only, default: true)", + ) + args = parser.parse_args() + + _, message = unpack( + args.input_file, + args.output_directory, + merge_runs=args.merge_runs, + simplify_redlines=args.simplify_redlines, + ) + print(message) + + if "Error" in message: + sys.exit(1) diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/validate.py b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/validate.py new file mode 100755 index 00000000..03b01f6e --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/validate.py @@ -0,0 +1,111 @@ +""" +Command line tool to validate Office document XML files against XSD schemas and tracked changes. + +Usage: + python validate.py [--original ] [--auto-repair] [--author NAME] + +The first argument can be either: +- An unpacked directory containing the Office document XML files +- A packed Office file (.docx/.pptx/.xlsx) which will be unpacked to a temp directory + +Auto-repair fixes: +- paraId/durableId values that exceed OOXML limits +- Missing xml:space="preserve" on w:t elements with whitespace +""" + +import argparse +import sys +import tempfile +import zipfile +from pathlib import Path + +from validators import DOCXSchemaValidator, PPTXSchemaValidator, RedliningValidator + + +def main(): + parser = argparse.ArgumentParser(description="Validate Office document XML files") + parser.add_argument( + "path", + help="Path to unpacked directory or packed Office file (.docx/.pptx/.xlsx)", + ) + parser.add_argument( + "--original", + required=False, + default=None, + help="Path to original file (.docx/.pptx/.xlsx). If omitted, all XSD errors are reported and redlining validation is skipped.", + ) + parser.add_argument( + "-v", + "--verbose", + action="store_true", + help="Enable verbose output", + ) + parser.add_argument( + "--auto-repair", + action="store_true", + help="Automatically repair common issues (hex IDs, whitespace preservation)", + ) + parser.add_argument( + "--author", + default="Claude", + help="Author name for redlining validation (default: Claude)", + ) + args = parser.parse_args() + + path = Path(args.path) + assert path.exists(), f"Error: {path} does not exist" + + original_file = None + if args.original: + original_file = Path(args.original) + assert original_file.is_file(), f"Error: {original_file} is not a file" + assert original_file.suffix.lower() in [".docx", ".pptx", ".xlsx"], ( + f"Error: {original_file} must be a .docx, .pptx, or .xlsx file" + ) + + file_extension = (original_file or path).suffix.lower() + assert file_extension in [".docx", ".pptx", ".xlsx"], ( + f"Error: Cannot determine file type from {path}. Use --original or provide a .docx/.pptx/.xlsx file." + ) + + if path.is_file() and path.suffix.lower() in [".docx", ".pptx", ".xlsx"]: + temp_dir = tempfile.mkdtemp() + with zipfile.ZipFile(path, "r") as zf: + zf.extractall(temp_dir) + unpacked_dir = Path(temp_dir) + else: + assert path.is_dir(), f"Error: {path} is not a directory or Office file" + unpacked_dir = path + + match file_extension: + case ".docx": + validators = [ + DOCXSchemaValidator(unpacked_dir, original_file, verbose=args.verbose), + ] + if original_file: + validators.append( + RedliningValidator(unpacked_dir, original_file, verbose=args.verbose, author=args.author) + ) + case ".pptx": + validators = [ + PPTXSchemaValidator(unpacked_dir, original_file, verbose=args.verbose), + ] + case _: + print(f"Error: Validation not supported for file type {file_extension}") + sys.exit(1) + + if args.auto_repair: + total_repairs = sum(v.repair() for v in validators) + if total_repairs: + print(f"Auto-repaired {total_repairs} issue(s)") + + success = all(v.validate() for v in validators) + + if success: + print("All validations PASSED!") + + sys.exit(0 if success else 1) + + +if __name__ == "__main__": + main() diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/validators/__init__.py b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/validators/__init__.py new file mode 100644 index 00000000..db092ece --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/validators/__init__.py @@ -0,0 +1,15 @@ +""" +Validation modules for Word document processing. +""" + +from .base import BaseSchemaValidator +from .docx import DOCXSchemaValidator +from .pptx import PPTXSchemaValidator +from .redlining import RedliningValidator + +__all__ = [ + "BaseSchemaValidator", + "DOCXSchemaValidator", + "PPTXSchemaValidator", + "RedliningValidator", +] diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/validators/base.py b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/validators/base.py new file mode 100644 index 00000000..db4a06a2 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/validators/base.py @@ -0,0 +1,847 @@ +""" +Base validator with common validation logic for document files. +""" + +import re +from pathlib import Path + +import defusedxml.minidom +import lxml.etree + + +class BaseSchemaValidator: + + IGNORED_VALIDATION_ERRORS = [ + "hyphenationZone", + "purl.org/dc/terms", + ] + + UNIQUE_ID_REQUIREMENTS = { + "comment": ("id", "file"), + "commentrangestart": ("id", "file"), + "commentrangeend": ("id", "file"), + "bookmarkstart": ("id", "file"), + "bookmarkend": ("id", "file"), + "sldid": ("id", "file"), + "sldmasterid": ("id", "global"), + "sldlayoutid": ("id", "global"), + "cm": ("authorid", "file"), + "sheet": ("sheetid", "file"), + "definedname": ("id", "file"), + "cxnsp": ("id", "file"), + "sp": ("id", "file"), + "pic": ("id", "file"), + "grpsp": ("id", "file"), + } + + EXCLUDED_ID_CONTAINERS = { + "sectionlst", + } + + ELEMENT_RELATIONSHIP_TYPES = {} + + SCHEMA_MAPPINGS = { + "word": "ISO-IEC29500-4_2016/wml.xsd", + "ppt": "ISO-IEC29500-4_2016/pml.xsd", + "xl": "ISO-IEC29500-4_2016/sml.xsd", + "[Content_Types].xml": "ecma/fouth-edition/opc-contentTypes.xsd", + "app.xml": "ISO-IEC29500-4_2016/shared-documentPropertiesExtended.xsd", + "core.xml": "ecma/fouth-edition/opc-coreProperties.xsd", + "custom.xml": "ISO-IEC29500-4_2016/shared-documentPropertiesCustom.xsd", + ".rels": "ecma/fouth-edition/opc-relationships.xsd", + "people.xml": "microsoft/wml-2012.xsd", + "commentsIds.xml": "microsoft/wml-cid-2016.xsd", + "commentsExtensible.xml": "microsoft/wml-cex-2018.xsd", + "commentsExtended.xml": "microsoft/wml-2012.xsd", + "chart": "ISO-IEC29500-4_2016/dml-chart.xsd", + "theme": "ISO-IEC29500-4_2016/dml-main.xsd", + "drawing": "ISO-IEC29500-4_2016/dml-main.xsd", + } + + MC_NAMESPACE = "http://schemas.openxmlformats.org/markup-compatibility/2006" + XML_NAMESPACE = "http://www.w3.org/XML/1998/namespace" + + PACKAGE_RELATIONSHIPS_NAMESPACE = ( + "http://schemas.openxmlformats.org/package/2006/relationships" + ) + OFFICE_RELATIONSHIPS_NAMESPACE = ( + "http://schemas.openxmlformats.org/officeDocument/2006/relationships" + ) + CONTENT_TYPES_NAMESPACE = ( + "http://schemas.openxmlformats.org/package/2006/content-types" + ) + + MAIN_CONTENT_FOLDERS = {"word", "ppt", "xl"} + + OOXML_NAMESPACES = { + "http://schemas.openxmlformats.org/officeDocument/2006/math", + "http://schemas.openxmlformats.org/officeDocument/2006/relationships", + "http://schemas.openxmlformats.org/schemaLibrary/2006/main", + "http://schemas.openxmlformats.org/drawingml/2006/main", + "http://schemas.openxmlformats.org/drawingml/2006/chart", + "http://schemas.openxmlformats.org/drawingml/2006/chartDrawing", + "http://schemas.openxmlformats.org/drawingml/2006/diagram", + "http://schemas.openxmlformats.org/drawingml/2006/picture", + "http://schemas.openxmlformats.org/drawingml/2006/spreadsheetDrawing", + "http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing", + "http://schemas.openxmlformats.org/wordprocessingml/2006/main", + "http://schemas.openxmlformats.org/presentationml/2006/main", + "http://schemas.openxmlformats.org/spreadsheetml/2006/main", + "http://schemas.openxmlformats.org/officeDocument/2006/sharedTypes", + "http://www.w3.org/XML/1998/namespace", + } + + def __init__(self, unpacked_dir, original_file=None, verbose=False): + self.unpacked_dir = Path(unpacked_dir).resolve() + self.original_file = Path(original_file) if original_file else None + self.verbose = verbose + + self.schemas_dir = Path(__file__).parent.parent / "schemas" + + patterns = ["*.xml", "*.rels"] + self.xml_files = [ + f for pattern in patterns for f in self.unpacked_dir.rglob(pattern) + ] + + if not self.xml_files: + print(f"Warning: No XML files found in {self.unpacked_dir}") + + def validate(self): + raise NotImplementedError("Subclasses must implement the validate method") + + def repair(self) -> int: + return self.repair_whitespace_preservation() + + def repair_whitespace_preservation(self) -> int: + repairs = 0 + + for xml_file in self.xml_files: + try: + content = xml_file.read_text(encoding="utf-8") + dom = defusedxml.minidom.parseString(content) + modified = False + + for elem in dom.getElementsByTagName("*"): + if elem.tagName.endswith(":t") and elem.firstChild: + text = elem.firstChild.nodeValue + if text and (text.startswith((' ', '\t')) or text.endswith((' ', '\t'))): + if elem.getAttribute("xml:space") != "preserve": + elem.setAttribute("xml:space", "preserve") + text_preview = repr(text[:30]) + "..." if len(text) > 30 else repr(text) + print(f" Repaired: {xml_file.name}: Added xml:space='preserve' to {elem.tagName}: {text_preview}") + repairs += 1 + modified = True + + if modified: + xml_file.write_bytes(dom.toxml(encoding="UTF-8")) + + except Exception: + pass + + return repairs + + def validate_xml(self): + errors = [] + + for xml_file in self.xml_files: + try: + lxml.etree.parse(str(xml_file)) + except lxml.etree.XMLSyntaxError as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {e.lineno}: {e.msg}" + ) + except Exception as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Unexpected error: {str(e)}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} XML violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All XML files are well-formed") + return True + + def validate_namespaces(self): + errors = [] + + for xml_file in self.xml_files: + try: + root = lxml.etree.parse(str(xml_file)).getroot() + declared = set(root.nsmap.keys()) - {None} + + for attr_val in [ + v for k, v in root.attrib.items() if k.endswith("Ignorable") + ]: + undeclared = set(attr_val.split()) - declared + errors.extend( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Namespace '{ns}' in Ignorable but not declared" + for ns in undeclared + ) + except lxml.etree.XMLSyntaxError: + continue + + if errors: + print(f"FAILED - {len(errors)} namespace issues:") + for error in errors: + print(error) + return False + if self.verbose: + print("PASSED - All namespace prefixes properly declared") + return True + + def validate_unique_ids(self): + errors = [] + global_ids = {} + + for xml_file in self.xml_files: + try: + root = lxml.etree.parse(str(xml_file)).getroot() + file_ids = {} + + mc_elements = root.xpath( + ".//mc:AlternateContent", namespaces={"mc": self.MC_NAMESPACE} + ) + for elem in mc_elements: + elem.getparent().remove(elem) + + for elem in root.iter(): + tag = ( + elem.tag.split("}")[-1].lower() + if "}" in elem.tag + else elem.tag.lower() + ) + + if tag in self.UNIQUE_ID_REQUIREMENTS: + in_excluded_container = any( + ancestor.tag.split("}")[-1].lower() in self.EXCLUDED_ID_CONTAINERS + for ancestor in elem.iterancestors() + ) + if in_excluded_container: + continue + + attr_name, scope = self.UNIQUE_ID_REQUIREMENTS[tag] + + id_value = None + for attr, value in elem.attrib.items(): + attr_local = ( + attr.split("}")[-1].lower() + if "}" in attr + else attr.lower() + ) + if attr_local == attr_name: + id_value = value + break + + if id_value is not None: + if scope == "global": + if id_value in global_ids: + prev_file, prev_line, prev_tag = global_ids[ + id_value + ] + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {elem.sourceline}: Global ID '{id_value}' in <{tag}> " + f"already used in {prev_file} at line {prev_line} in <{prev_tag}>" + ) + else: + global_ids[id_value] = ( + xml_file.relative_to(self.unpacked_dir), + elem.sourceline, + tag, + ) + elif scope == "file": + key = (tag, attr_name) + if key not in file_ids: + file_ids[key] = {} + + if id_value in file_ids[key]: + prev_line = file_ids[key][id_value] + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {elem.sourceline}: Duplicate {attr_name}='{id_value}' in <{tag}> " + f"(first occurrence at line {prev_line})" + ) + else: + file_ids[key][id_value] = elem.sourceline + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} ID uniqueness violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All required IDs are unique") + return True + + def validate_file_references(self): + errors = [] + + rels_files = list(self.unpacked_dir.rglob("*.rels")) + + if not rels_files: + if self.verbose: + print("PASSED - No .rels files found") + return True + + all_files = [] + for file_path in self.unpacked_dir.rglob("*"): + if ( + file_path.is_file() + and file_path.name != "[Content_Types].xml" + and not file_path.name.endswith(".rels") + ): + all_files.append(file_path.resolve()) + + all_referenced_files = set() + + if self.verbose: + print( + f"Found {len(rels_files)} .rels files and {len(all_files)} target files" + ) + + for rels_file in rels_files: + try: + rels_root = lxml.etree.parse(str(rels_file)).getroot() + + rels_dir = rels_file.parent + + referenced_files = set() + broken_refs = [] + + for rel in rels_root.findall( + ".//ns:Relationship", + namespaces={"ns": self.PACKAGE_RELATIONSHIPS_NAMESPACE}, + ): + target = rel.get("Target") + if target and not target.startswith( + ("http", "mailto:") + ): + if target.startswith("/"): + target_path = self.unpacked_dir / target.lstrip("/") + elif rels_file.name == ".rels": + target_path = self.unpacked_dir / target + else: + base_dir = rels_dir.parent + target_path = base_dir / target + + try: + target_path = target_path.resolve() + if target_path.exists() and target_path.is_file(): + referenced_files.add(target_path) + all_referenced_files.add(target_path) + else: + broken_refs.append((target, rel.sourceline)) + except (OSError, ValueError): + broken_refs.append((target, rel.sourceline)) + + if broken_refs: + rel_path = rels_file.relative_to(self.unpacked_dir) + for broken_ref, line_num in broken_refs: + errors.append( + f" {rel_path}: Line {line_num}: Broken reference to {broken_ref}" + ) + + except Exception as e: + rel_path = rels_file.relative_to(self.unpacked_dir) + errors.append(f" Error parsing {rel_path}: {e}") + + unreferenced_files = set(all_files) - all_referenced_files + + if unreferenced_files: + for unref_file in sorted(unreferenced_files): + unref_rel_path = unref_file.relative_to(self.unpacked_dir) + errors.append(f" Unreferenced file: {unref_rel_path}") + + if errors: + print(f"FAILED - Found {len(errors)} relationship validation errors:") + for error in errors: + print(error) + print( + "CRITICAL: These errors will cause the document to appear corrupt. " + + "Broken references MUST be fixed, " + + "and unreferenced files MUST be referenced or removed." + ) + return False + else: + if self.verbose: + print( + "PASSED - All references are valid and all files are properly referenced" + ) + return True + + def validate_all_relationship_ids(self): + import lxml.etree + + errors = [] + + for xml_file in self.xml_files: + if xml_file.suffix == ".rels": + continue + + rels_dir = xml_file.parent / "_rels" + rels_file = rels_dir / f"{xml_file.name}.rels" + + if not rels_file.exists(): + continue + + try: + rels_root = lxml.etree.parse(str(rels_file)).getroot() + rid_to_type = {} + + for rel in rels_root.findall( + f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship" + ): + rid = rel.get("Id") + rel_type = rel.get("Type", "") + if rid: + if rid in rid_to_type: + rels_rel_path = rels_file.relative_to(self.unpacked_dir) + errors.append( + f" {rels_rel_path}: Line {rel.sourceline}: " + f"Duplicate relationship ID '{rid}' (IDs must be unique)" + ) + type_name = ( + rel_type.split("/")[-1] if "/" in rel_type else rel_type + ) + rid_to_type[rid] = type_name + + xml_root = lxml.etree.parse(str(xml_file)).getroot() + + r_ns = self.OFFICE_RELATIONSHIPS_NAMESPACE + rid_attrs_to_check = ["id", "embed", "link"] + for elem in xml_root.iter(): + for attr_name in rid_attrs_to_check: + rid_attr = elem.get(f"{{{r_ns}}}{attr_name}") + if not rid_attr: + continue + xml_rel_path = xml_file.relative_to(self.unpacked_dir) + elem_name = ( + elem.tag.split("}")[-1] if "}" in elem.tag else elem.tag + ) + + if rid_attr not in rid_to_type: + errors.append( + f" {xml_rel_path}: Line {elem.sourceline}: " + f"<{elem_name}> r:{attr_name} references non-existent relationship '{rid_attr}' " + f"(valid IDs: {', '.join(sorted(rid_to_type.keys())[:5])}{'...' if len(rid_to_type) > 5 else ''})" + ) + elif attr_name == "id" and self.ELEMENT_RELATIONSHIP_TYPES: + expected_type = self._get_expected_relationship_type( + elem_name + ) + if expected_type: + actual_type = rid_to_type[rid_attr] + if expected_type not in actual_type.lower(): + errors.append( + f" {xml_rel_path}: Line {elem.sourceline}: " + f"<{elem_name}> references '{rid_attr}' which points to '{actual_type}' " + f"but should point to a '{expected_type}' relationship" + ) + + except Exception as e: + xml_rel_path = xml_file.relative_to(self.unpacked_dir) + errors.append(f" Error processing {xml_rel_path}: {e}") + + if errors: + print(f"FAILED - Found {len(errors)} relationship ID reference errors:") + for error in errors: + print(error) + print("\nThese ID mismatches will cause the document to appear corrupt!") + return False + else: + if self.verbose: + print("PASSED - All relationship ID references are valid") + return True + + def _get_expected_relationship_type(self, element_name): + elem_lower = element_name.lower() + + if elem_lower in self.ELEMENT_RELATIONSHIP_TYPES: + return self.ELEMENT_RELATIONSHIP_TYPES[elem_lower] + + if elem_lower.endswith("id") and len(elem_lower) > 2: + prefix = elem_lower[:-2] + if prefix.endswith("master"): + return prefix.lower() + elif prefix.endswith("layout"): + return prefix.lower() + else: + if prefix == "sld": + return "slide" + return prefix.lower() + + if elem_lower.endswith("reference") and len(elem_lower) > 9: + prefix = elem_lower[:-9] + return prefix.lower() + + return None + + def validate_content_types(self): + errors = [] + + content_types_file = self.unpacked_dir / "[Content_Types].xml" + if not content_types_file.exists(): + print("FAILED - [Content_Types].xml file not found") + return False + + try: + root = lxml.etree.parse(str(content_types_file)).getroot() + declared_parts = set() + declared_extensions = set() + + for override in root.findall( + f".//{{{self.CONTENT_TYPES_NAMESPACE}}}Override" + ): + part_name = override.get("PartName") + if part_name is not None: + declared_parts.add(part_name.lstrip("/")) + + for default in root.findall( + f".//{{{self.CONTENT_TYPES_NAMESPACE}}}Default" + ): + extension = default.get("Extension") + if extension is not None: + declared_extensions.add(extension.lower()) + + declarable_roots = { + "sld", + "sldLayout", + "sldMaster", + "presentation", + "document", + "workbook", + "worksheet", + "theme", + } + + media_extensions = { + "png": "image/png", + "jpg": "image/jpeg", + "jpeg": "image/jpeg", + "gif": "image/gif", + "bmp": "image/bmp", + "tiff": "image/tiff", + "wmf": "image/x-wmf", + "emf": "image/x-emf", + } + + all_files = list(self.unpacked_dir.rglob("*")) + all_files = [f for f in all_files if f.is_file()] + + for xml_file in self.xml_files: + path_str = str(xml_file.relative_to(self.unpacked_dir)).replace( + "\\", "/" + ) + + if any( + skip in path_str + for skip in [".rels", "[Content_Types]", "docProps/", "_rels/"] + ): + continue + + try: + root_tag = lxml.etree.parse(str(xml_file)).getroot().tag + root_name = root_tag.split("}")[-1] if "}" in root_tag else root_tag + + if root_name in declarable_roots and path_str not in declared_parts: + errors.append( + f" {path_str}: File with <{root_name}> root not declared in [Content_Types].xml" + ) + + except Exception: + continue + + for file_path in all_files: + if file_path.suffix.lower() in {".xml", ".rels"}: + continue + if file_path.name == "[Content_Types].xml": + continue + if "_rels" in file_path.parts or "docProps" in file_path.parts: + continue + + extension = file_path.suffix.lstrip(".").lower() + if extension and extension not in declared_extensions: + if extension in media_extensions: + relative_path = file_path.relative_to(self.unpacked_dir) + errors.append( + f' {relative_path}: File with extension \'{extension}\' not declared in [Content_Types].xml - should add: ' + ) + + except Exception as e: + errors.append(f" Error parsing [Content_Types].xml: {e}") + + if errors: + print(f"FAILED - Found {len(errors)} content type declaration errors:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print( + "PASSED - All content files are properly declared in [Content_Types].xml" + ) + return True + + def validate_file_against_xsd(self, xml_file, verbose=False): + xml_file = Path(xml_file).resolve() + unpacked_dir = self.unpacked_dir.resolve() + + is_valid, current_errors = self._validate_single_file_xsd( + xml_file, unpacked_dir + ) + + if is_valid is None: + return None, set() + elif is_valid: + return True, set() + + original_errors = self._get_original_file_errors(xml_file) + + assert current_errors is not None + new_errors = current_errors - original_errors + + new_errors = { + e for e in new_errors + if not any(pattern in e for pattern in self.IGNORED_VALIDATION_ERRORS) + } + + if new_errors: + if verbose: + relative_path = xml_file.relative_to(unpacked_dir) + print(f"FAILED - {relative_path}: {len(new_errors)} new error(s)") + for error in list(new_errors)[:3]: + truncated = error[:250] + "..." if len(error) > 250 else error + print(f" - {truncated}") + return False, new_errors + else: + if verbose: + print( + f"PASSED - No new errors (original had {len(current_errors)} errors)" + ) + return True, set() + + def validate_against_xsd(self): + new_errors = [] + original_error_count = 0 + valid_count = 0 + skipped_count = 0 + + for xml_file in self.xml_files: + relative_path = str(xml_file.relative_to(self.unpacked_dir)) + is_valid, new_file_errors = self.validate_file_against_xsd( + xml_file, verbose=False + ) + + if is_valid is None: + skipped_count += 1 + continue + elif is_valid and not new_file_errors: + valid_count += 1 + continue + elif is_valid: + original_error_count += 1 + valid_count += 1 + continue + + new_errors.append(f" {relative_path}: {len(new_file_errors)} new error(s)") + for error in list(new_file_errors)[:3]: + new_errors.append( + f" - {error[:250]}..." if len(error) > 250 else f" - {error}" + ) + + if self.verbose: + print(f"Validated {len(self.xml_files)} files:") + print(f" - Valid: {valid_count}") + print(f" - Skipped (no schema): {skipped_count}") + if original_error_count: + print(f" - With original errors (ignored): {original_error_count}") + print( + f" - With NEW errors: {len(new_errors) > 0 and len([e for e in new_errors if not e.startswith(' ')]) or 0}" + ) + + if new_errors: + print("\nFAILED - Found NEW validation errors:") + for error in new_errors: + print(error) + return False + else: + if self.verbose: + print("\nPASSED - No new XSD validation errors introduced") + return True + + def _get_schema_path(self, xml_file): + if xml_file.name in self.SCHEMA_MAPPINGS: + return self.schemas_dir / self.SCHEMA_MAPPINGS[xml_file.name] + + if xml_file.suffix == ".rels": + return self.schemas_dir / self.SCHEMA_MAPPINGS[".rels"] + + if "charts/" in str(xml_file) and xml_file.name.startswith("chart"): + return self.schemas_dir / self.SCHEMA_MAPPINGS["chart"] + + if "theme/" in str(xml_file) and xml_file.name.startswith("theme"): + return self.schemas_dir / self.SCHEMA_MAPPINGS["theme"] + + if xml_file.parent.name in self.MAIN_CONTENT_FOLDERS: + return self.schemas_dir / self.SCHEMA_MAPPINGS[xml_file.parent.name] + + return None + + def _clean_ignorable_namespaces(self, xml_doc): + xml_string = lxml.etree.tostring(xml_doc, encoding="unicode") + xml_copy = lxml.etree.fromstring(xml_string) + + for elem in xml_copy.iter(): + attrs_to_remove = [] + + for attr in elem.attrib: + if "{" in attr: + ns = attr.split("}")[0][1:] + if ns not in self.OOXML_NAMESPACES: + attrs_to_remove.append(attr) + + for attr in attrs_to_remove: + del elem.attrib[attr] + + self._remove_ignorable_elements(xml_copy) + + return lxml.etree.ElementTree(xml_copy) + + def _remove_ignorable_elements(self, root): + elements_to_remove = [] + + for elem in list(root): + if not hasattr(elem, "tag") or callable(elem.tag): + continue + + tag_str = str(elem.tag) + if tag_str.startswith("{"): + ns = tag_str.split("}")[0][1:] + if ns not in self.OOXML_NAMESPACES: + elements_to_remove.append(elem) + continue + + self._remove_ignorable_elements(elem) + + for elem in elements_to_remove: + root.remove(elem) + + def _preprocess_for_mc_ignorable(self, xml_doc): + root = xml_doc.getroot() + + if f"{{{self.MC_NAMESPACE}}}Ignorable" in root.attrib: + del root.attrib[f"{{{self.MC_NAMESPACE}}}Ignorable"] + + return xml_doc + + def _validate_single_file_xsd(self, xml_file, base_path): + schema_path = self._get_schema_path(xml_file) + if not schema_path: + return None, None + + try: + with open(schema_path, "rb") as xsd_file: + parser = lxml.etree.XMLParser() + xsd_doc = lxml.etree.parse( + xsd_file, parser=parser, base_url=str(schema_path) + ) + schema = lxml.etree.XMLSchema(xsd_doc) + + with open(xml_file, "r") as f: + xml_doc = lxml.etree.parse(f) + + xml_doc, _ = self._remove_template_tags_from_text_nodes(xml_doc) + xml_doc = self._preprocess_for_mc_ignorable(xml_doc) + + relative_path = xml_file.relative_to(base_path) + if ( + relative_path.parts + and relative_path.parts[0] in self.MAIN_CONTENT_FOLDERS + ): + xml_doc = self._clean_ignorable_namespaces(xml_doc) + + if schema.validate(xml_doc): + return True, set() + else: + errors = set() + for error in schema.error_log: + errors.add(error.message) + return False, errors + + except Exception as e: + return False, {str(e)} + + def _get_original_file_errors(self, xml_file): + if self.original_file is None: + return set() + + import tempfile + import zipfile + + xml_file = Path(xml_file).resolve() + unpacked_dir = self.unpacked_dir.resolve() + relative_path = xml_file.relative_to(unpacked_dir) + + with tempfile.TemporaryDirectory() as temp_dir: + temp_path = Path(temp_dir) + + with zipfile.ZipFile(self.original_file, "r") as zip_ref: + zip_ref.extractall(temp_path) + + original_xml_file = temp_path / relative_path + + if not original_xml_file.exists(): + return set() + + is_valid, errors = self._validate_single_file_xsd( + original_xml_file, temp_path + ) + return errors if errors else set() + + def _remove_template_tags_from_text_nodes(self, xml_doc): + warnings = [] + template_pattern = re.compile(r"\{\{[^}]*\}\}") + + xml_string = lxml.etree.tostring(xml_doc, encoding="unicode") + xml_copy = lxml.etree.fromstring(xml_string) + + def process_text_content(text, content_type): + if not text: + return text + matches = list(template_pattern.finditer(text)) + if matches: + for match in matches: + warnings.append( + f"Found template tag in {content_type}: {match.group()}" + ) + return template_pattern.sub("", text) + return text + + for elem in xml_copy.iter(): + if not hasattr(elem, "tag") or callable(elem.tag): + continue + tag_str = str(elem.tag) + if tag_str.endswith("}t") or tag_str == "t": + continue + + elem.text = process_text_content(elem.text, "text content") + elem.tail = process_text_content(elem.tail, "tail content") + + return lxml.etree.ElementTree(xml_copy), warnings + + +if __name__ == "__main__": + raise RuntimeError("This module should not be run directly.") diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/validators/docx.py b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/validators/docx.py new file mode 100644 index 00000000..0a0b0bfd --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/validators/docx.py @@ -0,0 +1,447 @@ +""" +Validator for Word document XML files against XSD schemas. +""" + +import os +import random +import re +import tempfile +import zipfile + +import defusedxml.minidom +import lxml.etree + +from .base import BaseSchemaValidator + + +class DOCXSchemaValidator(BaseSchemaValidator): + + WORD_2006_NAMESPACE = "http://schemas.openxmlformats.org/wordprocessingml/2006/main" + W14_NAMESPACE = "http://schemas.microsoft.com/office/word/2010/wordml" + W16CID_NAMESPACE = "http://schemas.microsoft.com/office/word/2016/wordml/cid" + + ELEMENT_RELATIONSHIP_TYPES = {} + + def validate(self): + if not self.validate_xml(): + return False + + all_valid = True + if not self.validate_namespaces(): + all_valid = False + + if not self.validate_unique_ids(): + all_valid = False + + if not self.validate_file_references(): + all_valid = False + + if not self.validate_content_types(): + all_valid = False + + if not self.validate_against_xsd(): + all_valid = False + + if not self.validate_whitespace_preservation(): + all_valid = False + + if not self.validate_deletions(): + all_valid = False + + if not self.validate_insertions(): + all_valid = False + + if not self.validate_all_relationship_ids(): + all_valid = False + + if not self.validate_id_constraints(): + all_valid = False + + if not self.validate_comment_markers(): + all_valid = False + + self.compare_paragraph_counts() + + return all_valid + + def validate_whitespace_preservation(self): + errors = [] + + for xml_file in self.xml_files: + if xml_file.name != "document.xml": + continue + + try: + root = lxml.etree.parse(str(xml_file)).getroot() + + for elem in root.iter(f"{{{self.WORD_2006_NAMESPACE}}}t"): + if elem.text: + text = elem.text + if re.search(r"^[ \t\n\r]", text) or re.search( + r"[ \t\n\r]$", text + ): + xml_space_attr = f"{{{self.XML_NAMESPACE}}}space" + if ( + xml_space_attr not in elem.attrib + or elem.attrib[xml_space_attr] != "preserve" + ): + text_preview = ( + repr(text)[:50] + "..." + if len(repr(text)) > 50 + else repr(text) + ) + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {elem.sourceline}: w:t element with whitespace missing xml:space='preserve': {text_preview}" + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} whitespace preservation violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All whitespace is properly preserved") + return True + + def validate_deletions(self): + errors = [] + + for xml_file in self.xml_files: + if xml_file.name != "document.xml": + continue + + try: + root = lxml.etree.parse(str(xml_file)).getroot() + namespaces = {"w": self.WORD_2006_NAMESPACE} + + for t_elem in root.xpath(".//w:del//w:t", namespaces=namespaces): + if t_elem.text: + text_preview = ( + repr(t_elem.text)[:50] + "..." + if len(repr(t_elem.text)) > 50 + else repr(t_elem.text) + ) + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {t_elem.sourceline}: found within : {text_preview}" + ) + + for instr_elem in root.xpath( + ".//w:del//w:instrText", namespaces=namespaces + ): + text_preview = ( + repr(instr_elem.text or "")[:50] + "..." + if len(repr(instr_elem.text or "")) > 50 + else repr(instr_elem.text or "") + ) + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {instr_elem.sourceline}: found within (use ): {text_preview}" + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} deletion validation violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - No w:t elements found within w:del elements") + return True + + def count_paragraphs_in_unpacked(self): + count = 0 + + for xml_file in self.xml_files: + if xml_file.name != "document.xml": + continue + + try: + root = lxml.etree.parse(str(xml_file)).getroot() + paragraphs = root.findall(f".//{{{self.WORD_2006_NAMESPACE}}}p") + count = len(paragraphs) + except Exception as e: + print(f"Error counting paragraphs in unpacked document: {e}") + + return count + + def count_paragraphs_in_original(self): + original = self.original_file + if original is None: + return 0 + + count = 0 + + try: + with tempfile.TemporaryDirectory() as temp_dir: + with zipfile.ZipFile(original, "r") as zip_ref: + zip_ref.extractall(temp_dir) + + doc_xml_path = os.path.join(temp_dir, "word", "document.xml") + root = lxml.etree.parse(doc_xml_path).getroot() + + paragraphs = root.findall(f".//{{{self.WORD_2006_NAMESPACE}}}p") + count = len(paragraphs) + + except Exception as e: + print(f"Error counting paragraphs in original document: {e}") + + return count + + def validate_insertions(self): + errors = [] + + for xml_file in self.xml_files: + if xml_file.name != "document.xml": + continue + + try: + root = lxml.etree.parse(str(xml_file)).getroot() + namespaces = {"w": self.WORD_2006_NAMESPACE} + + invalid_elements = root.xpath( + ".//w:ins//w:delText[not(ancestor::w:del)]", namespaces=namespaces + ) + + for elem in invalid_elements: + text_preview = ( + repr(elem.text or "")[:50] + "..." + if len(repr(elem.text or "")) > 50 + else repr(elem.text or "") + ) + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {elem.sourceline}: within : {text_preview}" + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} insertion validation violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - No w:delText elements within w:ins elements") + return True + + def compare_paragraph_counts(self): + original_count = self.count_paragraphs_in_original() + new_count = self.count_paragraphs_in_unpacked() + + diff = new_count - original_count + diff_str = f"+{diff}" if diff > 0 else str(diff) + print(f"\nParagraphs: {original_count} → {new_count} ({diff_str})") + + def _parse_id_value(self, val: str, base: int = 16) -> int: + return int(val, base) + + def validate_id_constraints(self): + errors = [] + para_id_attr = f"{{{self.W14_NAMESPACE}}}paraId" + durable_id_attr = f"{{{self.W16CID_NAMESPACE}}}durableId" + + for xml_file in self.xml_files: + try: + for elem in lxml.etree.parse(str(xml_file)).iter(): + if val := elem.get(para_id_attr): + if self._parse_id_value(val, base=16) >= 0x80000000: + errors.append( + f" {xml_file.name}:{elem.sourceline}: paraId={val} >= 0x80000000" + ) + + if val := elem.get(durable_id_attr): + if xml_file.name == "numbering.xml": + try: + if self._parse_id_value(val, base=10) >= 0x7FFFFFFF: + errors.append( + f" {xml_file.name}:{elem.sourceline}: " + f"durableId={val} >= 0x7FFFFFFF" + ) + except ValueError: + errors.append( + f" {xml_file.name}:{elem.sourceline}: " + f"durableId={val} must be decimal in numbering.xml" + ) + else: + if self._parse_id_value(val, base=16) >= 0x7FFFFFFF: + errors.append( + f" {xml_file.name}:{elem.sourceline}: " + f"durableId={val} >= 0x7FFFFFFF" + ) + except Exception: + pass + + if errors: + print(f"FAILED - {len(errors)} ID constraint violations:") + for e in errors: + print(e) + elif self.verbose: + print("PASSED - All paraId/durableId values within constraints") + return not errors + + def validate_comment_markers(self): + errors = [] + + document_xml = None + comments_xml = None + for xml_file in self.xml_files: + if xml_file.name == "document.xml" and "word" in str(xml_file): + document_xml = xml_file + elif xml_file.name == "comments.xml": + comments_xml = xml_file + + if not document_xml: + if self.verbose: + print("PASSED - No document.xml found (skipping comment validation)") + return True + + try: + doc_root = lxml.etree.parse(str(document_xml)).getroot() + namespaces = {"w": self.WORD_2006_NAMESPACE} + + range_starts = { + elem.get(f"{{{self.WORD_2006_NAMESPACE}}}id") + for elem in doc_root.xpath( + ".//w:commentRangeStart", namespaces=namespaces + ) + } + range_ends = { + elem.get(f"{{{self.WORD_2006_NAMESPACE}}}id") + for elem in doc_root.xpath( + ".//w:commentRangeEnd", namespaces=namespaces + ) + } + references = { + elem.get(f"{{{self.WORD_2006_NAMESPACE}}}id") + for elem in doc_root.xpath( + ".//w:commentReference", namespaces=namespaces + ) + } + + orphaned_ends = range_ends - range_starts + for comment_id in sorted( + orphaned_ends, key=lambda x: int(x) if x and x.isdigit() else 0 + ): + errors.append( + f' document.xml: commentRangeEnd id="{comment_id}" has no matching commentRangeStart' + ) + + orphaned_starts = range_starts - range_ends + for comment_id in sorted( + orphaned_starts, key=lambda x: int(x) if x and x.isdigit() else 0 + ): + errors.append( + f' document.xml: commentRangeStart id="{comment_id}" has no matching commentRangeEnd' + ) + + comment_ids = set() + if comments_xml and comments_xml.exists(): + comments_root = lxml.etree.parse(str(comments_xml)).getroot() + comment_ids = { + elem.get(f"{{{self.WORD_2006_NAMESPACE}}}id") + for elem in comments_root.xpath( + ".//w:comment", namespaces=namespaces + ) + } + + marker_ids = range_starts | range_ends | references + invalid_refs = marker_ids - comment_ids + for comment_id in sorted( + invalid_refs, key=lambda x: int(x) if x and x.isdigit() else 0 + ): + if comment_id: + errors.append( + f' document.xml: marker id="{comment_id}" references non-existent comment' + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append(f" Error parsing XML: {e}") + + if errors: + print(f"FAILED - {len(errors)} comment marker violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All comment markers properly paired") + return True + + def repair(self) -> int: + repairs = super().repair() + repairs += self.repair_durableId() + return repairs + + def repair_durableId(self) -> int: + repairs = 0 + + for xml_file in self.xml_files: + try: + content = xml_file.read_text(encoding="utf-8") + dom = defusedxml.minidom.parseString(content) + modified = False + + for elem in dom.getElementsByTagName("*"): + if not elem.hasAttribute("w16cid:durableId"): + continue + + durable_id = elem.getAttribute("w16cid:durableId") + needs_repair = False + + if xml_file.name == "numbering.xml": + try: + needs_repair = ( + self._parse_id_value(durable_id, base=10) >= 0x7FFFFFFF + ) + except ValueError: + needs_repair = True + else: + try: + needs_repair = ( + self._parse_id_value(durable_id, base=16) >= 0x7FFFFFFF + ) + except ValueError: + needs_repair = True + + if needs_repair: + value = random.randint(1, 0x7FFFFFFE) + if xml_file.name == "numbering.xml": + new_id = str(value) + else: + new_id = f"{value:08X}" + + elem.setAttribute("w16cid:durableId", new_id) + print( + f" Repaired: {xml_file.name}: durableId {durable_id} → {new_id}" + ) + repairs += 1 + modified = True + + if modified: + xml_file.write_bytes(dom.toxml(encoding="UTF-8")) + + except Exception: + pass + + return repairs + + +if __name__ == "__main__": + raise RuntimeError("This module should not be run directly.") diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/validators/pptx.py b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/validators/pptx.py new file mode 100644 index 00000000..09842aa9 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/validators/pptx.py @@ -0,0 +1,275 @@ +""" +Validator for PowerPoint presentation XML files against XSD schemas. +""" + +import re + +from .base import BaseSchemaValidator + + +class PPTXSchemaValidator(BaseSchemaValidator): + + PRESENTATIONML_NAMESPACE = ( + "http://schemas.openxmlformats.org/presentationml/2006/main" + ) + + ELEMENT_RELATIONSHIP_TYPES = { + "sldid": "slide", + "sldmasterid": "slidemaster", + "notesmasterid": "notesmaster", + "sldlayoutid": "slidelayout", + "themeid": "theme", + "tablestyleid": "tablestyles", + } + + def validate(self): + if not self.validate_xml(): + return False + + all_valid = True + if not self.validate_namespaces(): + all_valid = False + + if not self.validate_unique_ids(): + all_valid = False + + if not self.validate_uuid_ids(): + all_valid = False + + if not self.validate_file_references(): + all_valid = False + + if not self.validate_slide_layout_ids(): + all_valid = False + + if not self.validate_content_types(): + all_valid = False + + if not self.validate_against_xsd(): + all_valid = False + + if not self.validate_notes_slide_references(): + all_valid = False + + if not self.validate_all_relationship_ids(): + all_valid = False + + if not self.validate_no_duplicate_slide_layouts(): + all_valid = False + + return all_valid + + def validate_uuid_ids(self): + import lxml.etree + + errors = [] + uuid_pattern = re.compile( + r"^[\{\(]?[0-9A-Fa-f]{8}-?[0-9A-Fa-f]{4}-?[0-9A-Fa-f]{4}-?[0-9A-Fa-f]{4}-?[0-9A-Fa-f]{12}[\}\)]?$" + ) + + for xml_file in self.xml_files: + try: + root = lxml.etree.parse(str(xml_file)).getroot() + + for elem in root.iter(): + for attr, value in elem.attrib.items(): + attr_name = attr.split("}")[-1].lower() + if attr_name == "id" or attr_name.endswith("id"): + if self._looks_like_uuid(value): + if not uuid_pattern.match(value): + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {elem.sourceline}: ID '{value}' appears to be a UUID but contains invalid hex characters" + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} UUID ID validation errors:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All UUID-like IDs contain valid hex values") + return True + + def _looks_like_uuid(self, value): + clean_value = value.strip("{}()").replace("-", "") + return len(clean_value) == 32 and all(c.isalnum() for c in clean_value) + + def validate_slide_layout_ids(self): + import lxml.etree + + errors = [] + + slide_masters = list(self.unpacked_dir.glob("ppt/slideMasters/*.xml")) + + if not slide_masters: + if self.verbose: + print("PASSED - No slide masters found") + return True + + for slide_master in slide_masters: + try: + root = lxml.etree.parse(str(slide_master)).getroot() + + rels_file = slide_master.parent / "_rels" / f"{slide_master.name}.rels" + + if not rels_file.exists(): + errors.append( + f" {slide_master.relative_to(self.unpacked_dir)}: " + f"Missing relationships file: {rels_file.relative_to(self.unpacked_dir)}" + ) + continue + + rels_root = lxml.etree.parse(str(rels_file)).getroot() + + valid_layout_rids = set() + for rel in rels_root.findall( + f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship" + ): + rel_type = rel.get("Type", "") + if "slideLayout" in rel_type: + valid_layout_rids.add(rel.get("Id")) + + for sld_layout_id in root.findall( + f".//{{{self.PRESENTATIONML_NAMESPACE}}}sldLayoutId" + ): + r_id = sld_layout_id.get( + f"{{{self.OFFICE_RELATIONSHIPS_NAMESPACE}}}id" + ) + layout_id = sld_layout_id.get("id") + + if r_id and r_id not in valid_layout_rids: + errors.append( + f" {slide_master.relative_to(self.unpacked_dir)}: " + f"Line {sld_layout_id.sourceline}: sldLayoutId with id='{layout_id}' " + f"references r:id='{r_id}' which is not found in slide layout relationships" + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {slide_master.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} slide layout ID validation errors:") + for error in errors: + print(error) + print( + "Remove invalid references or add missing slide layouts to the relationships file." + ) + return False + else: + if self.verbose: + print("PASSED - All slide layout IDs reference valid slide layouts") + return True + + def validate_no_duplicate_slide_layouts(self): + import lxml.etree + + errors = [] + slide_rels_files = list(self.unpacked_dir.glob("ppt/slides/_rels/*.xml.rels")) + + for rels_file in slide_rels_files: + try: + root = lxml.etree.parse(str(rels_file)).getroot() + + layout_rels = [ + rel + for rel in root.findall( + f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship" + ) + if "slideLayout" in rel.get("Type", "") + ] + + if len(layout_rels) > 1: + errors.append( + f" {rels_file.relative_to(self.unpacked_dir)}: has {len(layout_rels)} slideLayout references" + ) + + except Exception as e: + errors.append( + f" {rels_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print("FAILED - Found slides with duplicate slideLayout references:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All slides have exactly one slideLayout reference") + return True + + def validate_notes_slide_references(self): + import lxml.etree + + errors = [] + notes_slide_references = {} + + slide_rels_files = list(self.unpacked_dir.glob("ppt/slides/_rels/*.xml.rels")) + + if not slide_rels_files: + if self.verbose: + print("PASSED - No slide relationship files found") + return True + + for rels_file in slide_rels_files: + try: + root = lxml.etree.parse(str(rels_file)).getroot() + + for rel in root.findall( + f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship" + ): + rel_type = rel.get("Type", "") + if "notesSlide" in rel_type: + target = rel.get("Target", "") + if target: + normalized_target = target.replace("../", "") + + slide_name = rels_file.stem.replace( + ".xml", "" + ) + + if normalized_target not in notes_slide_references: + notes_slide_references[normalized_target] = [] + notes_slide_references[normalized_target].append( + (slide_name, rels_file) + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {rels_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + for target, references in notes_slide_references.items(): + if len(references) > 1: + slide_names = [ref[0] for ref in references] + errors.append( + f" Notes slide '{target}' is referenced by multiple slides: {', '.join(slide_names)}" + ) + for slide_name, rels_file in references: + errors.append(f" - {rels_file.relative_to(self.unpacked_dir)}") + + if errors: + print( + f"FAILED - Found {len([e for e in errors if not e.startswith(' ')])} notes slide reference validation errors:" + ) + for error in errors: + print(error) + print("Each slide may optionally have its own slide file.") + return False + else: + if self.verbose: + print("PASSED - All notes slide references are unique") + return True + + +if __name__ == "__main__": + raise RuntimeError("This module should not be run directly.") diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/validators/redlining.py b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/validators/redlining.py new file mode 100644 index 00000000..71c81b6b --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/office/validators/redlining.py @@ -0,0 +1,247 @@ +""" +Validator for tracked changes in Word documents. +""" + +import subprocess +import tempfile +import zipfile +from pathlib import Path + + +class RedliningValidator: + + def __init__(self, unpacked_dir, original_docx, verbose=False, author="Claude"): + self.unpacked_dir = Path(unpacked_dir) + self.original_docx = Path(original_docx) + self.verbose = verbose + self.author = author + self.namespaces = { + "w": "http://schemas.openxmlformats.org/wordprocessingml/2006/main" + } + + def repair(self) -> int: + return 0 + + def validate(self): + modified_file = self.unpacked_dir / "word" / "document.xml" + if not modified_file.exists(): + print(f"FAILED - Modified document.xml not found at {modified_file}") + return False + + try: + import xml.etree.ElementTree as ET + + tree = ET.parse(modified_file) + root = tree.getroot() + + del_elements = root.findall(".//w:del", self.namespaces) + ins_elements = root.findall(".//w:ins", self.namespaces) + + author_del_elements = [ + elem + for elem in del_elements + if elem.get(f"{{{self.namespaces['w']}}}author") == self.author + ] + author_ins_elements = [ + elem + for elem in ins_elements + if elem.get(f"{{{self.namespaces['w']}}}author") == self.author + ] + + if not author_del_elements and not author_ins_elements: + if self.verbose: + print(f"PASSED - No tracked changes by {self.author} found.") + return True + + except Exception: + pass + + with tempfile.TemporaryDirectory() as temp_dir: + temp_path = Path(temp_dir) + + try: + with zipfile.ZipFile(self.original_docx, "r") as zip_ref: + zip_ref.extractall(temp_path) + except Exception as e: + print(f"FAILED - Error unpacking original docx: {e}") + return False + + original_file = temp_path / "word" / "document.xml" + if not original_file.exists(): + print( + f"FAILED - Original document.xml not found in {self.original_docx}" + ) + return False + + try: + import xml.etree.ElementTree as ET + + modified_tree = ET.parse(modified_file) + modified_root = modified_tree.getroot() + original_tree = ET.parse(original_file) + original_root = original_tree.getroot() + except ET.ParseError as e: + print(f"FAILED - Error parsing XML files: {e}") + return False + + self._remove_author_tracked_changes(original_root) + self._remove_author_tracked_changes(modified_root) + + modified_text = self._extract_text_content(modified_root) + original_text = self._extract_text_content(original_root) + + if modified_text != original_text: + error_message = self._generate_detailed_diff( + original_text, modified_text + ) + print(error_message) + return False + + if self.verbose: + print(f"PASSED - All changes by {self.author} are properly tracked") + return True + + def _generate_detailed_diff(self, original_text, modified_text): + error_parts = [ + f"FAILED - Document text doesn't match after removing {self.author}'s tracked changes", + "", + "Likely causes:", + " 1. Modified text inside another author's or tags", + " 2. Made edits without proper tracked changes", + " 3. Didn't nest inside when deleting another's insertion", + "", + "For pre-redlined documents, use correct patterns:", + " - To reject another's INSERTION: Nest inside their ", + " - To restore another's DELETION: Add new AFTER their ", + "", + ] + + git_diff = self._get_git_word_diff(original_text, modified_text) + if git_diff: + error_parts.extend(["Differences:", "============", git_diff]) + else: + error_parts.append("Unable to generate word diff (git not available)") + + return "\n".join(error_parts) + + def _get_git_word_diff(self, original_text, modified_text): + try: + with tempfile.TemporaryDirectory() as temp_dir: + temp_path = Path(temp_dir) + + original_file = temp_path / "original.txt" + modified_file = temp_path / "modified.txt" + + original_file.write_text(original_text, encoding="utf-8") + modified_file.write_text(modified_text, encoding="utf-8") + + result = subprocess.run( + [ + "git", + "diff", + "--word-diff=plain", + "--word-diff-regex=.", + "-U0", + "--no-index", + str(original_file), + str(modified_file), + ], + capture_output=True, + text=True, + ) + + if result.stdout.strip(): + lines = result.stdout.split("\n") + content_lines = [] + in_content = False + for line in lines: + if line.startswith("@@"): + in_content = True + continue + if in_content and line.strip(): + content_lines.append(line) + + if content_lines: + return "\n".join(content_lines) + + result = subprocess.run( + [ + "git", + "diff", + "--word-diff=plain", + "-U0", + "--no-index", + str(original_file), + str(modified_file), + ], + capture_output=True, + text=True, + ) + + if result.stdout.strip(): + lines = result.stdout.split("\n") + content_lines = [] + in_content = False + for line in lines: + if line.startswith("@@"): + in_content = True + continue + if in_content and line.strip(): + content_lines.append(line) + return "\n".join(content_lines) + + except (subprocess.CalledProcessError, FileNotFoundError, Exception): + pass + + return None + + def _remove_author_tracked_changes(self, root): + ins_tag = f"{{{self.namespaces['w']}}}ins" + del_tag = f"{{{self.namespaces['w']}}}del" + author_attr = f"{{{self.namespaces['w']}}}author" + + for parent in root.iter(): + to_remove = [] + for child in parent: + if child.tag == ins_tag and child.get(author_attr) == self.author: + to_remove.append(child) + for elem in to_remove: + parent.remove(elem) + + deltext_tag = f"{{{self.namespaces['w']}}}delText" + t_tag = f"{{{self.namespaces['w']}}}t" + + for parent in root.iter(): + to_process = [] + for child in parent: + if child.tag == del_tag and child.get(author_attr) == self.author: + to_process.append((child, list(parent).index(child))) + + for del_elem, del_index in reversed(to_process): + for elem in del_elem.iter(): + if elem.tag == deltext_tag: + elem.tag = t_tag + + for child in reversed(list(del_elem)): + parent.insert(del_index, child) + parent.remove(del_elem) + + def _extract_text_content(self, root): + p_tag = f"{{{self.namespaces['w']}}}p" + t_tag = f"{{{self.namespaces['w']}}}t" + + paragraphs = [] + for p_elem in root.findall(f".//{p_tag}"): + text_parts = [] + for t_elem in p_elem.findall(f".//{t_tag}"): + if t_elem.text: + text_parts.append(t_elem.text) + paragraph_text = "".join(text_parts) + if paragraph_text: + paragraphs.append(paragraph_text) + + return "\n".join(paragraphs) + + +if __name__ == "__main__": + raise RuntimeError("This module should not be run directly.") diff --git a/mateclaw-server/src/main/resources/skills/xlsx/scripts/recalc.py b/mateclaw-server/src/main/resources/skills/xlsx/scripts/recalc.py new file mode 100755 index 00000000..1501abc0 --- /dev/null +++ b/mateclaw-server/src/main/resources/skills/xlsx/scripts/recalc.py @@ -0,0 +1,209 @@ +""" +Excel Formula Recalculation Script +Recalculates all formulas in an Excel file using LibreOffice +""" + +import json +import os +import platform +import subprocess +import sys +from pathlib import Path + +from office.soffice import get_soffice_cmd, get_soffice_env + +from openpyxl import load_workbook + +MACRO_DIR_MACOS = "~/Library/Application Support/LibreOffice/4/user/basic/Standard" +MACRO_DIR_LINUX = "~/.config/libreoffice/4/user/basic/Standard" +MACRO_DIR_WINDOWS = "~/AppData/Roaming/LibreOffice/4/user/basic/Standard" +MACRO_FILENAME = "Module1.xba" + +RECALCULATE_MACRO = """ + + + Sub RecalculateAndSave() + ThisComponent.calculateAll() + ThisComponent.store() + ThisComponent.close(True) + End Sub +""" + + +def has_gtimeout(): + try: + subprocess.run( + ["gtimeout", "--version"], capture_output=True, timeout=1, check=False + ) + return True + except (FileNotFoundError, subprocess.TimeoutExpired): + return False + + +def _get_macro_dir() -> str: + """Return the LibreOffice macro directory for the current platform.""" + system = platform.system() + if system == "Darwin": + return MACRO_DIR_MACOS + elif system == "Windows": + return MACRO_DIR_WINDOWS + else: + return MACRO_DIR_LINUX + + +def setup_libreoffice_macro(): + macro_dir = os.path.expanduser(_get_macro_dir()) + macro_file = os.path.join(macro_dir, MACRO_FILENAME) + + if ( + os.path.exists(macro_file) + and "RecalculateAndSave" in Path(macro_file).read_text() + ): + return True + + if not os.path.exists(macro_dir): + subprocess.run( + [get_soffice_cmd(), "--headless", "--terminate_after_init"], + capture_output=True, + timeout=10, + env=get_soffice_env(), + ) + os.makedirs(macro_dir, exist_ok=True) + + try: + Path(macro_file).write_text(RECALCULATE_MACRO) + return True + except Exception: + return False + + +def recalc(filename, timeout=30): + if not Path(filename).exists(): + return {"error": f"File {filename} does not exist"} + + abs_path = str(Path(filename).absolute()) + + if not setup_libreoffice_macro(): + return {"error": "Failed to setup LibreOffice macro"} + + cmd = [ + get_soffice_cmd(), + "--headless", + "--norestore", + "vnd.sun.star.script:Standard.Module1.RecalculateAndSave?language=Basic&location=application", + abs_path, + ] + + system = platform.system() + if system == "Linux": + cmd = ["timeout", str(timeout)] + cmd + elif system == "Darwin" and has_gtimeout(): + cmd = ["gtimeout", str(timeout)] + cmd + # On Windows, rely on subprocess timeout parameter instead of external tool + + try: + result = subprocess.run( + cmd, + capture_output=True, + text=True, + env=get_soffice_env(), + timeout=timeout if system == "Windows" else None, + ) + except subprocess.TimeoutExpired: + return { + "error": ( + f"LibreOffice recalculation timed out after {timeout} seconds" + ), + } + + if result.returncode != 0 and result.returncode != 124: + error_msg = result.stderr or "Unknown error during recalculation" + if "Module1" in error_msg or "RecalculateAndSave" not in error_msg: + return {"error": "LibreOffice macro not configured properly"} + return {"error": error_msg} + + try: + wb = load_workbook(filename, data_only=True) + + excel_errors = [ + "#VALUE!", + "#DIV/0!", + "#REF!", + "#NAME?", + "#NULL!", + "#NUM!", + "#N/A", + ] + error_details = {err: [] for err in excel_errors} + total_errors = 0 + + for sheet_name in wb.sheetnames: + ws = wb[sheet_name] + for row in ws.iter_rows(): + for cell in row: + if cell.value is not None and isinstance(cell.value, str): + for err in excel_errors: + if err in cell.value: + location = f"{sheet_name}!{cell.coordinate}" + error_details[err].append(location) + total_errors += 1 + break + + wb.close() + + result = { + "status": "success" if total_errors == 0 else "errors_found", + "total_errors": total_errors, + "error_summary": {}, + } + + for err_type, locations in error_details.items(): + if locations: + result["error_summary"][err_type] = { + "count": len(locations), + "locations": locations[:20], + } + + wb_formulas = load_workbook(filename, data_only=False) + formula_count = 0 + for sheet_name in wb_formulas.sheetnames: + ws = wb_formulas[sheet_name] + for row in ws.iter_rows(): + for cell in row: + if ( + cell.value + and isinstance(cell.value, str) + and cell.value.startswith("=") + ): + formula_count += 1 + wb_formulas.close() + + result["total_formulas"] = formula_count + + return result + + except Exception as e: + return {"error": str(e)} + + +def main(): + if len(sys.argv) < 2: + print("Usage: python recalc.py [timeout_seconds]") + print("\nRecalculates all formulas in an Excel file using LibreOffice") + print("\nReturns JSON with error details:") + print(" - status: 'success' or 'errors_found'") + print(" - total_errors: Total number of Excel errors found") + print(" - total_formulas: Number of formulas in the file") + print(" - error_summary: Breakdown by error type with locations") + print(" - #VALUE!, #DIV/0!, #REF!, #NAME?, #NULL!, #NUM!, #N/A") + sys.exit(1) + + filename = sys.argv[1] + timeout = int(sys.argv[2]) if len(sys.argv) > 2 else 30 + + result = recalc(filename, timeout) + print(json.dumps(result, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/mateclaw-server/src/test/java/vip/mate/agent/context/CompactionSurvivalComparisonTest.java b/mateclaw-server/src/test/java/vip/mate/agent/context/CompactionSurvivalComparisonTest.java new file mode 100644 index 00000000..c1dad7e6 --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/agent/context/CompactionSurvivalComparisonTest.java @@ -0,0 +1,261 @@ +package vip.mate.agent.context; + +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.springframework.ai.chat.messages.AssistantMessage; +import org.springframework.ai.chat.messages.Message; +import org.springframework.ai.chat.messages.ToolResponseMessage; +import org.springframework.ai.chat.messages.UserMessage; + +import java.util.ArrayList; +import java.util.List; + +/** + * New-vs-old compaction comparison harness. + * + *

This file is intentionally written to compile against BOTH: + *

    + *
  • {@code /data/mateclaw} — the new (post-six-moves) codebase
    • + *
  • {@code /data/mateclaw/mateclaw-old} — the pre-six-moves codebase
    • + *
+ * + *

It only uses APIs that exist in both: + *

    + *
  • {@code new ConversationWindowManager(null, null, null)}
    • + *
  • {@code mgr.softTrimToolResults(messages)}
    • + *
  • {@code mgr.hardClearToolResults(messages)}
    • + *
  • {@code mgr.prePruneForSummary(messages)}
    • + *
  • {@code TokenEstimator.estimateTokens(...)}
    • + *
+ * + *

It does NOT use {@code isExemptTool} or {@code spillEvictToolResults} + * (those don't exist on the old code). The same source file produces + * DIFFERENT metrics on the two codebases — that difference IS the + * demonstration that Move 4 works. + * + *

Each test prints a {@code [METRIC]} line to stdout in a stable + * key=value format so the new-code run and old-code run can be diffed. + */ +class CompactionSurvivalComparisonTest { + + // ==================== markers ==================== + + private static final String LOAD_SKILL_BODY_TEMPLATE = + "[mate-skill-md]\n# skill-%d\nconstraints:\n- CONSTRAINT_MARKER_%d\n" + + "- Always confirm before writing\n" + + "- Never delete files outside /tmp\n" + + "- Use exactly 4-space indentation\n"; + + private static final String DELEGATE_BODY_TEMPLATE = + "[sub-agent transcript %d]\nuser: list files\nassistant: DELEGATE_MARKER_%d done\n"; + + private static final String READ_FILE_BODY_TEMPLATE = + "file content %d\n" + "x".repeat(1500) + "\nREADFILE_MARKER_%d\n"; + + // ==================== helpers ==================== + + private static ConversationWindowManager newManager() { + return new ConversationWindowManager(null, null, null); + } + + private static ToolResponseMessage trm(String toolName, String body) { + return ToolResponseMessage.builder() + .responses(List.of(new ToolResponseMessage.ToolResponse( + "call-" + toolName + "-" + System.nanoTime(), toolName, body))) + .build(); + } + + /** + * Extract ALL text from a message, including {@link ToolResponseMessage} + * response data (which {@code message.getText()} does NOT return). + */ + private static String extractAllText(Message m) { + if (m == null) return ""; + if (m instanceof ToolResponseMessage trm) { + StringBuilder sb = new StringBuilder(); + for (ToolResponseMessage.ToolResponse r : trm.getResponses()) { + if (r.responseData() != null) sb.append(r.responseData()); + } + return sb.toString(); + } + String t = m.getText(); + return t == null ? "" : t; + } + + /** + * Token counter that ALSO counts {@link ToolResponseMessage} response data + * (the production {@code TokenEstimator.estimateTokens(Message)} only reads + * {@code getText()}, which is null for tool responses). + */ + private static int realTokens(List messages) { + int total = 0; + for (Message m : messages) { + total += TokenEstimator.estimateTokens(extractAllText(m)) + + 4; // PER_MESSAGE_OVERHEAD + } + return total; + } + + private static int countSurvivingMarkers(List messages, String markerPrefix, int total) { + int survived = 0; + for (int i = 0; i < total; i++) { + String marker = markerPrefix + i; + boolean found = false; + for (Message m : messages) { + if (extractAllText(m).contains(marker)) { + found = true; + break; + } + } + if (found) survived++; + } + return survived; + } + + private static void runAllThreePhases(ConversationWindowManager mgr, List messages) { + mgr.softTrimToolResults(messages); + mgr.hardClearToolResults(messages); + mgr.prePruneForSummary(messages); + } + + // ==================== tests ==================== + + @Test + @DisplayName("Scenario A: 50 load_skill + 50 delegate + 50 read_file, single full compaction") + void scenarioA_bulkSurvivalRate() { + ConversationWindowManager mgr = newManager(); + List messages = new ArrayList<>(); + messages.add(new UserMessage("Load the shopping skill and run a sub-agent, then read 50 files.")); + + int N = 50; + for (int i = 0; i < N; i++) { + messages.add(new AssistantMessage("loading skill " + i)); + messages.add(trm("load_skill", String.format(LOAD_SKILL_BODY_TEMPLATE, i, i))); + messages.add(new AssistantMessage("delegating " + i)); + messages.add(trm("delegateToAgent", String.format(DELEGATE_BODY_TEMPLATE, i, i))); + messages.add(new AssistantMessage("reading file " + i)); + messages.add(trm("read_file", String.format(READ_FILE_BODY_TEMPLATE, i, i))); + } + + int tokensBefore = realTokens(messages); + long start = System.nanoTime(); + runAllThreePhases(mgr, messages); + long elapsedMs = (System.nanoTime() - start) / 1_000_000; + int tokensAfter = realTokens(messages); + + int loadSkillSurvived = countSurvivingMarkers(messages, "CONSTRAINT_MARKER_", N); + int delegateSurvived = countSurvivingMarkers(messages, "DELEGATE_MARKER_", N); + int readFileSurvived = countSurvivingMarkers(messages, "READFILE_MARKER_", N); + + System.out.printf( + "[METRIC] scenario=A load_skill_survival=%d/%d delegate_survival=%d/%d " + + "readfile_survival=%d/%d tokens_before=%d tokens_after=%d time_ms=%d%n", + loadSkillSurvived, N, delegateSurvived, N, readFileSurvived, N, + tokensBefore, tokensAfter, elapsedMs); + } + + @Test + @DisplayName("Scenario B: 100-round extreme compression, one load_skill pinned at the head") + void scenarioB_hundredRoundExtremeCompression() { + ConversationWindowManager mgr = newManager(); + List messages = new ArrayList<>(); + messages.add(new UserMessage("Load the shopping skill, then do 100 file-reads.")); + messages.add(new AssistantMessage("loading skill")); + messages.add(trm("load_skill", + "[mate-skill-md]\n# ckjia-shopping\nconstraints:\n" + + "- ROOT_CONSTRAINT_MARKER\n" + + "- Always confirm before writing\n" + + "- Never delete files outside /tmp\n")); + + int rounds = 100; + int tokensConsumedTotal = 0; + long totalTimeMs = 0; + for (int r = 0; r < rounds; r++) { + messages.add(new AssistantMessage("reading file " + r)); + messages.add(trm("read_file", String.format(READ_FILE_BODY_TEMPLATE, r, r))); + + int before = realTokens(messages); + long start = System.nanoTime(); + runAllThreePhases(mgr, messages); + long elapsed = (System.nanoTime() - start) / 1_000_000; + int after = realTokens(messages); + tokensConsumedTotal += (before - after); + totalTimeMs += elapsed; + } + + int tokensFinal = realTokens(messages); + boolean rootSurvived = messages.stream() + .anyMatch(m -> extractAllText(m).contains("ROOT_CONSTRAINT_MARKER")); + + System.out.printf( + "[METRIC] scenario=B rounds=%d root_constraint_survived=%s tokens_final=%d " + + "tokens_consumed_by_compaction=%d total_compaction_time_ms=%d%n", + rounds, rootSurvived, tokensFinal, tokensConsumedTotal, totalTimeMs); + } + + @Test + @DisplayName("Scenario C: per-tool survival rate breakdown across 20 load_skill of varying body sizes") + void scenarioC_perToolSurvivalBreakdown() { + ConversationWindowManager mgr = newManager(); + List messages = new ArrayList<>(); + messages.add(new UserMessage("Load 20 skills of varying body sizes.")); + + int N = 20; + int[] bodySizes = {100, 500, 1000, 2000, 5000}; + for (int i = 0; i < N; i++) { + int size = bodySizes[i % bodySizes.length]; + StringBuilder body = new StringBuilder(); + body.append("[mate-skill-md]\n# skill-").append(i).append("\nconstraints:\n"); + body.append("- CONSTRAINT_MARKER_").append(i).append("\n"); + while (body.length() < size) body.append('x'); + messages.add(trm("load_skill", body.toString())); + } + + int tokensBefore = realTokens(messages); + long start = System.nanoTime(); + runAllThreePhases(mgr, messages); + long elapsedMs = (System.nanoTime() - start) / 1_000_000; + int tokensAfter = realTokens(messages); + + int survived = countSurvivingMarkers(messages, "CONSTRAINT_MARKER_", N); + + System.out.printf( + "[METRIC] scenario=C load_skill_count=%d survived=%d survival_rate=%.2f " + + "tokens_before=%d tokens_after=%d time_ms=%d%n", + N, survived, (survived * 100.0 / N), + tokensBefore, tokensAfter, elapsedMs); + } + + @Test + @DisplayName("Scenario D: token consumption per round (steady-state)") + void scenarioD_tokenConsumptionPerRound() { + ConversationWindowManager mgr = newManager(); + List messages = new ArrayList<>(); + messages.add(new UserMessage("Run a long task with one pinned skill.")); + messages.add(trm("load_skill", + "[mate-skill-md]\n# pinned\nconstraints:\n- PINNED_MARKER\n")); + + int rounds = 30; + StringBuilder perRound = new StringBuilder(); + for (int r = 0; r < rounds; r++) { + messages.add(new AssistantMessage("step " + r)); + messages.add(trm("read_file", String.format(READ_FILE_BODY_TEMPLATE, r, r))); + + int before = realTokens(messages); + long start = System.nanoTime(); + runAllThreePhases(mgr, messages); + long elapsed = (System.nanoTime() - start) / 1_000_000; + int after = realTokens(messages); + + if (r > 0) perRound.append(","); + perRound.append(String.format("%d:%d:%d", r, before - after, elapsed)); + } + + boolean pinnedSurvived = messages.stream() + .anyMatch(m -> extractAllText(m).contains("PINNED_MARKER")); + + System.out.printf( + "[METRIC] scenario=D rounds=%d pinned_survived=%s per_round=round:tokens_consumed:time_ms{%s}%n", + rounds, pinnedSurvived, perRound); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/agent/context/ContextCompressionLedgerSurvivalTest.java b/mateclaw-server/src/test/java/vip/mate/agent/context/ContextCompressionLedgerSurvivalTest.java new file mode 100644 index 00000000..e8cd37a2 --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/agent/context/ContextCompressionLedgerSurvivalTest.java @@ -0,0 +1,312 @@ +package vip.mate.agent.context; + +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.springframework.ai.chat.messages.AssistantMessage; +import org.springframework.ai.chat.messages.Message; +import org.springframework.ai.chat.messages.ToolResponseMessage; +import org.springframework.ai.chat.messages.UserMessage; +import org.springframework.ai.chat.model.ChatModel; +import org.springframework.ai.chat.model.ChatResponse; +import org.springframework.ai.chat.model.Generation; +import org.springframework.ai.chat.prompt.Prompt; +import vip.mate.agent.progress.ProgressLedger; +import vip.mate.agent.progress.ProgressLedgerService; +import vip.mate.agent.progress.ProgressStatus; +import vip.mate.config.ConversationWindowProperties; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +/** + * Black-box test: verifies that ProgressLedger entries (pinned, auto-recorded, + * regular) survive all four stages of ConversationWindowManager compression. + * + *

This is the key accuracy guarantee for the B-class changes: + *

    + *
  • B2 pins skill constraints into the ledger's {@code pinned} map
    • + *
  • B3 stores the ledger in the DB, separate from the message list
    • + *
  • ReasoningNode loads the ledger fresh each turn into + * {@code nonHistoryPrefix}, which is NEVER touched by compaction
    • + *
+ * + *

Contrast with A1 (PRUNE_EXEMPT_TOOLS): A1 tried to protect + * {@code load_skill} tool results inside the {@code messages} list, but the + * four-stage pipeline (Soft Trim / Hard Clear / Pre-Prune / LLM Summary) does + * NOT honor PRUNE_EXEMPT_TOOLS — so the tool result body IS destroyed. + * B2/B3 solves this by extracting constraints into the ledger BEFORE + * compression can touch them. + */ +class ContextCompressionLedgerSurvivalTest { + + private InMemoryProgressLedgerService ledgerService; + private ConversationWindowManager manager; + private ChatModel chatModel; + + @BeforeEach + void setUp() { + ledgerService = new InMemoryProgressLedgerService(); + + ConversationWindowProperties props = new ConversationWindowProperties(); + props.setFirstUserAnchorEnabled(true); + props.setFirstUserAnchorMaxTokens(400); + manager = new ConversationWindowManager(props, null, null); + + chatModel = mock(ChatModel.class); + ChatResponse response = new ChatResponse(List.of( + new Generation(new AssistantMessage("SUMMARY_OF_COMPRESSED_HISTORY")))); + when(chatModel.call(any(Prompt.class))).thenReturn(response); + } + + // ==================== Core survival test ==================== + + @Test + void allThreeLedgerEntryTypesSurviveCompression() { + String convId = "conv-survival"; + + // 1. Pin constraints (simulating B2: ActionNode.load_skill → pinSkillConstraints) + ledgerService.upsertPinned(convId, "pin_research_0", + "🔒 research: Never fabricate citations", "Never fabricate citations"); + ledgerService.upsertPinned(convId, "pin_research_1", + "🔒 research: Always cite primary sources", "Always cite primary sources"); + + // 2. Auto-record tool calls (simulating B5: ActionNode.autoRecordToolCalls) + ledgerService.upsertAutoRecorded(convId, "web_search", "web_search", "found 5 results"); + ledgerService.upsertAutoRecorded(convId, "read_file", "read_file", "read paper.pdf"); + ledgerService.upsertAutoRecorded(convId, "write_file", "write_file", "wrote draft.md"); + + // 3. Regular entries (simulating LLM: progress_update) + ledgerService.upsert(convId, "step_literature_review", "Literature Review", + ProgressStatus.DONE, "surveyed 12 papers"); + ledgerService.upsert(convId, "step_draft_outline", "Draft Outline", + ProgressStatus.IN_PROGRESS, "writing section 3"); + ledgerService.upsert(convId, "step_final_edit", "Final Edit", + ProgressStatus.PENDING, null); + + // 4. Build a long message history that triggers compression + List history = buildLongHistoryWithLoadSkill(60); + + // 5. Snapshot the ledger BEFORE compression + ProgressLedger beforeLedger = ledgerService.load(convId); + String beforeSnapshot = beforeLedger.renderSnapshot(); + assertThat(beforeSnapshot).contains("🔒 固定约束"); + assertThat(beforeSnapshot).contains("🔧 自动记录"); + assertThat(beforeSnapshot).contains("✅ 已完成"); + assertThat(beforeSnapshot).contains("🔄 进行中"); + assertThat(beforeSnapshot).contains("⏳ 待办"); + + // 6. Run PTL compression (the most aggressive: all 4 stages) + List compacted = manager.compactForRetry(history, chatModel, convId, 1L); + + // 7. The compacted message list should be shorter + assertThat(compacted).hasSizeLessThan(history.size()); + + // 8. CRITICAL: The ledger is unaffected — it lives in the DB, not in messages + ProgressLedger afterLedger = ledgerService.load(convId); + assertThat(afterLedger.pinnedEntries()).hasSize(2); + assertThat(afterLedger.asMap()) + .containsKeys("auto_web_search", "auto_read_file", "auto_write_file", + "step_literature_review", "step_draft_outline", "step_final_edit"); + + // 9. The rendered snapshot is identical before and after compression + String afterSnapshot = afterLedger.renderSnapshot(); + assertThat(afterSnapshot).isEqualTo(beforeSnapshot); + } + + // ==================== A1 gap proof: load_skill body destroyed, constraints survive ==================== + + @Test + void loadSkillBodyDestroyedByCompressionButConstraintsSurviveInLedger() { + String convId = "conv-a1-gap"; + + // B2: pin constraints from a loaded skill + ledgerService.upsertPinned(convId, "pin_security_0", + "🔒 security: Never run rm -rf", "Never run rm -rf"); + ledgerService.upsertPinned(convId, "pin_security_1", + "🔒 security: Confirm before shell_exec", "Confirm before shell_exec"); + + // Build a history with a load_skill tool result (>200 chars, will be pruned) + String loadSkillResult = "SKILL.md loaded successfully. This skill provides security auditing " + + "capabilities. " + "Constraint: Never run rm -rf. ".repeat(20) + + "Always confirm before shell_exec. " + "More padding ".repeat(20); + + List history = new ArrayList<>(); + history.add(new UserMessage("Load the security skill")); + history.add(AssistantMessage.builder() + .content("Loading security skill") + .toolCalls(List.of(new AssistantMessage.ToolCall( + "call-1", "function", "load_skill", "{\"name\":\"security\"}"))) + .build()); + history.add(ToolResponseMessage.builder() + .responses(List.of(new ToolResponseMessage.ToolResponse( + "call-1", "load_skill", loadSkillResult))) + .build()); + // Pad to trigger compression + for (int i = 0; i < 50; i++) { + history.add(new UserMessage("Turn " + i + " — ".repeat(50) + " filler content")); + history.add(new AssistantMessage("Response " + i + " — ".repeat(50) + " filler response")); + } + + // Run compression + List compacted = manager.compactForRetry(history, chatModel, convId, 1L); + + // The load_skill tool result body in messages should have been pruned/replaced + // (A1 gap: PRUNE_EXEMPT_TOOLS is not honored by the 4-stage pipeline) + boolean loadSkillBodySurvived = compacted.stream() + .filter(m -> m instanceof ToolResponseMessage) + .map(m -> (ToolResponseMessage) m) + .flatMap(trm -> trm.getResponses().stream()) + .anyMatch(r -> "load_skill".equals(r.name()) + && r.responseData() != null + && r.responseData().contains("Never run rm -rf")); + // The body was destroyed by compression (known A1 gap) + assertThat(loadSkillBodySurvived).isFalse(); + + // BUT: the constraints pinned by B2 are in the ledger, which is unaffected + ProgressLedger ledger = ledgerService.load(convId); + assertThat(ledger.pinnedEntries()).hasSize(2); + assertThat(ledger.renderSnapshot()).contains("Never run rm -rf"); + assertThat(ledger.renderSnapshot()).contains("Confirm before shell_exec"); + } + + // ==================== Auto-recorded entries bounded after compression ==================== + + @Test + void autoRecordedEntriesStayBoundedAfterManyToolCallsAndCompression() { + String convId = "conv-bounded"; + + // Simulate 10 tool calls — auto-record should bound to MAX_AUTO_RECORDED + for (int i = 0; i < 10; i++) { + ledgerService.upsertAutoRecorded(convId, "tool_" + i, "tool_" + i, "result " + i); + } + + // Run compression on a minimal history (compression doesn't affect ledger) + List history = new ArrayList<>(); + history.add(new UserMessage("Do task")); + history.add(new AssistantMessage("Done")); + manager.compactForRetry(history, chatModel, convId, 1L); + + // Ledger still bounded + ProgressLedger ledger = ledgerService.load(convId); + long autoCount = ledger.asMap().keySet().stream() + .filter(k -> k.startsWith(ProgressLedger.AUTO_RECORDED_PREFIX)) + .count(); + assertThat(autoCount).isEqualTo(ProgressLedgerService.MAX_AUTO_RECORDED); + } + + // ==================== Multiple compression cycles ==================== + + @Test + void ledgerSurvivesMultipleCompressionCycles() { + String convId = "conv-multi-cycle"; + + // Setup all three entry types + ledgerService.upsertPinned(convId, "pin_skill_0", "🔒 skill: Rule", "Rule"); + ledgerService.upsertAutoRecorded(convId, "read_file", "read_file", "result"); + ledgerService.upsert(convId, "step_1", "Step 1", ProgressStatus.DONE, "done"); + + String snapshotBefore = ledgerService.load(convId).renderSnapshot(); + + // Run compression 3 times (simulating 3 user turns with PTL) + List history = buildLongHistoryWithLoadSkill(40); + for (int cycle = 0; cycle < 3; cycle++) { + manager.compactForRetry(new ArrayList<>(history), chatModel, convId, 1L); + } + + // Ledger is still intact + String snapshotAfter = ledgerService.load(convId).renderSnapshot(); + assertThat(snapshotAfter).isEqualTo(snapshotBefore); + } + + // ==================== Backward compat: old flat-map JSON ==================== + + @Test + void oldFlatMapLedgerMigratesToWrapperFormatWithEmptyPinned() { + String convId = "conv-legacy"; + // Write old-format JSON (flat map, no wrapper) + String oldJson = "{\"step_1\":{\"key\":\"step_1\",\"label\":\"Step 1\"," + + "\"status\":\"DONE\",\"note\":\"done\",\"updatedAt\":\"2025-01-01T00:00:00Z\"}}"; + ledgerService.store.put(convId, oldJson); + + ProgressLedger ledger = ledgerService.load(convId); + assertThat(ledger.asMap()).containsKey("step_1"); + assertThat(ledger.pinnedEntries()).isEmpty(); // migrated with empty pinned + + // After an upsert, the JSON should be in wrapper format + ledgerService.upsert(convId, "step_2", "Step 2", ProgressStatus.PENDING, null); + String newJson = ledgerService.store.get(convId); + assertThat(newJson).contains("\"entries\""); + assertThat(newJson).contains("\"pinned\""); + } + + // ==================== Helpers ==================== + + /** + * Build a long message history that includes a load_skill call/result, + * enough to trigger the PTL compression path. + */ + private List buildLongHistoryWithLoadSkill(int fillerTurns) { + List messages = new ArrayList<>(); + messages.add(new UserMessage("Load the research skill and do a literature review")); + + // load_skill tool call + result + messages.add(AssistantMessage.builder() + .content("I'll load the research skill first") + .toolCalls(List.of(new AssistantMessage.ToolCall( + "call-ls", "function", "load_skill", "{\"name\":\"research\"}"))) + .build()); + messages.add(ToolResponseMessage.builder() + .responses(List.of(new ToolResponseMessage.ToolResponse( + "call-ls", "load_skill", + "Skill 'research' loaded. Constraints: Never fabricate citations. " + + "Always cite primary sources. " + "Padding ".repeat(30)))) + .build()); + + // Filler turns to build up history + for (int i = 0; i < fillerTurns; i++) { + messages.add(new UserMessage("Question " + i + ": " + "x".repeat(200))); + messages.add(AssistantMessage.builder() + .content("Answer " + i + ": " + "y".repeat(200)) + .toolCalls(i % 5 == 0 ? List.of(new AssistantMessage.ToolCall( + "call-" + i, "function", "web_search", + "{\"q\":\"query-" + i + "\"}")) : List.of()) + .build()); + if (i % 5 == 0) { + messages.add(ToolResponseMessage.builder() + .responses(List.of(new ToolResponseMessage.ToolResponse( + "call-" + i, "web_search", "Search result " + i + ": " + "z".repeat(300)))) + .build()); + } + } + return messages; + } + + // ==================== In-memory ledger service ==================== + + private static final class InMemoryProgressLedgerService extends ProgressLedgerService { + final Map store = new ConcurrentHashMap<>(); + + InMemoryProgressLedgerService() { + super(null, new ObjectMapper().registerModule(new JavaTimeModule())); + } + + @Override + protected String loadLedgerJson(String conversationId) { + return store.get(conversationId); + } + + @Override + protected void saveLedgerJson(String conversationId, String json) { + store.put(conversationId, json); + } + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/agent/context/ConversationWindowManagerExemptAndSpillTest.java b/mateclaw-server/src/test/java/vip/mate/agent/context/ConversationWindowManagerExemptAndSpillTest.java new file mode 100644 index 00000000..22867f22 --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/agent/context/ConversationWindowManagerExemptAndSpillTest.java @@ -0,0 +1,300 @@ +package vip.mate.agent.context; + +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.springframework.ai.chat.messages.AssistantMessage; +import org.springframework.ai.chat.messages.Message; +import org.springframework.ai.chat.messages.ToolResponseMessage; +import org.springframework.ai.chat.messages.UserMessage; +import vip.mate.agent.graph.executor.ToolResultStorage; + +import java.util.ArrayList; +import java.util.List; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.mock; + +/** + * Move 4 — behavioral tests that would FAIL on the pre-Move-4 code. + * + *

Pre-Move-4 bugs being verified: + *

    + *
  1. {@code softTrimToolResults}, {@code hardClearToolResults}, and + * {@code prePruneForSummary} did NOT check {@code PRUNE_EXEMPT_TOOLS}. + * A {@code load_skill} or {@code delegateToAgent} result would be + * trimmed/cleared/pruned, silently dropping the skill's constraints + * or the sub-agent's transcript.
    2. + *
  2. There was no lossless spill-evict path before the LLM summary — + * every over-budget conversation paid the LLM-summary token cost + * even when spilling to disk would have sufficed.
    3. + *
+ * + *

Each test below asserts the NEW behavior. To confirm they would fail + * on the old code, revert the Move 4 changes in ConversationWindowManager + * and re-run — every test in this class should fail. + */ +class ConversationWindowManagerExemptAndSpillTest { + + // Reuse the same constants the production code uses. + private static final String LOAD_SKILL_BODY = + "[mate-skill-md]\n# ckjia-shopping\nconstraints:\n- Always confirm before writing\n"; + private static final String DELEGATE_BODY = + "[sub-agent transcript]\nuser: list files\nassistant: ..."; + private static final String READ_FILE_BODY = "x".repeat(2000); + + private static ConversationWindowManager newManager() { + // Constructor: (ConversationWindowProperties, MemoryManager, ConversationService) + // — all can be null for the phases we test (softTrim/hardClear/prePrune + // don't touch memory or conversation service). + return new ConversationWindowManager(null, null, null); + } + + private static ToolResponseMessage trm(String toolName, String body) { + return ToolResponseMessage.builder() + .responses(List.of(new ToolResponseMessage.ToolResponse( + "call-" + toolName, toolName, body))) + .build(); + } + + // ==================== Move 4.1: isExemptTool ==================== + + @Test + @DisplayName("isExemptTool(load_skill) → true (NEW: method did not exist pre-Move-4)") + void loadSkillIsExempt() { + ToolResponseMessage.ToolResponse r = new ToolResponseMessage.ToolResponse( + "id", "load_skill", "body"); + assertTrue(ConversationWindowManager.isExemptTool(r), + "load_skill must be exempt — pre-Move-4 this method did not exist"); + } + + @Test + @DisplayName("isExemptTool(delegateToAgent) → true") + void delegateIsExempt() { + ToolResponseMessage.ToolResponse r = new ToolResponseMessage.ToolResponse( + "id", "delegateToAgent", "body"); + assertTrue(ConversationWindowManager.isExemptTool(r)); + } + + @Test + @DisplayName("isExemptTool(delegateParallel) → true") + void delegateParallelIsExempt() { + ToolResponseMessage.ToolResponse r = new ToolResponseMessage.ToolResponse( + "id", "delegateParallel", "body"); + assertTrue(ConversationWindowManager.isExemptTool(r)); + } + + @Test + @DisplayName("isExemptTool(read_file) → false (non-exempt tool)") + void readFileIsNotExempt() { + ToolResponseMessage.ToolResponse r = new ToolResponseMessage.ToolResponse( + "id", "read_file", "body"); + assertFalse(ConversationWindowManager.isExemptTool(r)); + } + + @Test + @DisplayName("isExemptTool(null) → false (defensive)") + void nullIsNotExempt() { + assertFalse(ConversationWindowManager.isExemptTool(null)); + } + + // ==================== Move 4.2: softTrimToolResults preserves exempt ==================== + + @Test + @DisplayName("softTrimToolResults preserves load_skill body verbatim (would FAIL pre-Move-4)") + void softTrimPreservesLoadSkillBody() { + // Pre-Move-4: softTrimToolResults only checked isSpillMarker — + // load_skill body would be truncated to ~400 chars, destroying + // the skill constraints. Move 4 adds isExemptTool check. + ConversationWindowManager mgr = newManager(); + List messages = new ArrayList<>(List.of( + trm("load_skill", LOAD_SKILL_BODY), + trm("read_file", READ_FILE_BODY))); + + int trimmed = mgr.softTrimToolResults(messages); + + // read_file WAS trimmed (non-exempt) + assertTrue(trimmed >= 1, "non-exempt read_file should be trimmed"); + // load_skill body is UNCHANGED + ToolResponseMessage loadSkillTrm = (ToolResponseMessage) messages.get(0); + assertEquals(LOAD_SKILL_BODY, loadSkillTrm.getResponses().get(0).responseData(), + "load_skill body must survive softTrim verbatim — " + + "pre-Move-4 this would have been truncated"); + assertTrue(loadSkillTrm.getResponses().get(0).responseData().contains( + "Always confirm before writing"), + "constraint text must survive — pre-Move-4 it was lost"); + } + + @Test + @DisplayName("softTrimToolResults preserves delegateToAgent body verbatim") + void softTrimPreservesDelegateBody() { + ConversationWindowManager mgr = newManager(); + List messages = new ArrayList<>(List.of( + trm("delegateToAgent", DELEGATE_BODY))); + + mgr.softTrimToolResults(messages); + + ToolResponseMessage trm = (ToolResponseMessage) messages.get(0); + assertEquals(DELEGATE_BODY, trm.getResponses().get(0).responseData(), + "delegateToAgent body must survive softTrim verbatim"); + } + + // ==================== Move 4.3: hardClearToolResults preserves exempt ==================== + + @Test + @DisplayName("hardClearToolResults preserves load_skill body verbatim (would FAIL pre-Move-4)") + void hardClearPreservesLoadSkillBody() { + // Pre-Move-4: hardClearToolResults only checked isSpillMarker — + // load_skill body would be replaced with "[旧工具输出已清理]". + // This is the most destructive bypass: Phase 2 wipes the entire + // skill constraints, then Phase 3 LLM summary can't reconstruct + // them because they're already gone. + ConversationWindowManager mgr = newManager(); + List messages = new ArrayList<>(List.of( + trm("load_skill", LOAD_SKILL_BODY), + trm("read_file", READ_FILE_BODY))); + + int cleared = mgr.hardClearToolResults(messages); + + // read_file WAS cleared (non-exempt) + assertTrue(cleared >= 1, "non-exempt read_file should be cleared"); + // load_skill body is UNCHANGED + ToolResponseMessage loadSkillTrm = (ToolResponseMessage) messages.get(0); + assertEquals(LOAD_SKILL_BODY, loadSkillTrm.getResponses().get(0).responseData(), + "load_skill body must survive hardClear verbatim — " + + "pre-Move-4 this would have been replaced with a placeholder"); + assertTrue(loadSkillTrm.getResponses().get(0).responseData().contains( + "Always confirm before writing"), + "constraint text must survive hardClear"); + } + + @Test + @DisplayName("hardClearToolResults preserves delegateToAgent body verbatim") + void hardClearPreservesDelegateBody() { + ConversationWindowManager mgr = newManager(); + List messages = new ArrayList<>(List.of( + trm("delegateToAgent", DELEGATE_BODY))); + + mgr.hardClearToolResults(messages); + + ToolResponseMessage trm = (ToolResponseMessage) messages.get(0); + assertEquals(DELEGATE_BODY, trm.getResponses().get(0).responseData(), + "delegateToAgent body must survive hardClear verbatim"); + } + + // ==================== Move 4.4: prePruneForSummary preserves exempt ==================== + + @Test + @DisplayName("prePruneForSummary preserves load_skill body verbatim (would FAIL pre-Move-4)") + void prePrunePreservesLoadSkillBody() { + // Pre-Move-4: prePruneForSummary replaced ANY non-spill body >200 chars + // with "[旧工具输出已清理以节省上下文空间]". load_skill's SKILL.md + // snapshot is typically >200 chars, so it was ALWAYS pruned here. + ConversationWindowManager mgr = newManager(); + List messages = new ArrayList<>(List.of( + trm("load_skill", LOAD_SKILL_BODY), + trm("read_file", READ_FILE_BODY))); + + int pruned = mgr.prePruneForSummary(messages); + + // read_file WAS pruned (non-exempt, >200 chars) + assertTrue(pruned >= 1, "non-exempt read_file should be pruned"); + // load_skill body is UNCHANGED + ToolResponseMessage loadSkillTrm = (ToolResponseMessage) messages.get(0); + assertEquals(LOAD_SKILL_BODY, loadSkillTrm.getResponses().get(0).responseData(), + "load_skill body must survive prePrune verbatim — " + + "pre-Move-4 this would have been replaced with a placeholder"); + } + + @Test + @DisplayName("prePruneForSummary skips ToolResponseMessage entirely when all responses are exempt") + void prePruneSkipsAllExemptMessage() { + ConversationWindowManager mgr = newManager(); + List messages = new ArrayList<>(List.of( + trm("load_skill", LOAD_SKILL_BODY))); + + int pruned = mgr.prePruneForSummary(messages); + + assertEquals(0, pruned, + "a ToolResponseMessage with only exempt responses must not be pruned at all"); + } + + // ==================== Move 4.5: spillEvictToolResults (new method) ==================== + + @Test + @DisplayName("spillEvictToolResults returns 0 when toolResultStorage is null (defensive)") + void spillEvictNoStorage() { + ConversationWindowManager mgr = newManager(); + List messages = new ArrayList<>(List.of( + trm("read_file", READ_FILE_BODY))); + int spilled = mgr.spillEvictToolResults(messages, "conv-1", "/tmp/ws"); + assertEquals(0, spilled, "null toolResultStorage must no-op"); + } + + @Test + @DisplayName("spillEvictToolResults returns 0 when conversationId is null") + void spillEvictNoConversationId() { + // Even with storage wired, null conversationId must no-op to avoid + // writing spill files to a meaningless path. + ConversationWindowManager mgr = newManagerWithStorage(mock(ToolResultStorage.class)); + List messages = new ArrayList<>(List.of( + trm("read_file", READ_FILE_BODY))); + int spilled = mgr.spillEvictToolResults(messages, null, "/tmp/ws"); + assertEquals(0, spilled); + } + + @Test + @DisplayName("spillEvictToolResults skips exempt tools (load_skill not spilled)") + void spillEvictSkipsExempt() { + // Exempt tools must never be spilled — their content is not + // safely recoverable (load_skill returns a snapshot that may + // have been edited since). + ToolResultStorage storage = mock(ToolResultStorage.class); + ConversationWindowManager mgr = newManagerWithStorage(storage); + List messages = new ArrayList<>(List.of( + trm("load_skill", LOAD_SKILL_BODY))); + + int spilled = mgr.spillEvictToolResults(messages, "conv-1", "/tmp/ws"); + + assertEquals(0, spilled, "load_skill must not be spilled"); + } + + // ==================== Move 4.6: Phase 2.7 integration ==================== + + @Test + @DisplayName("Move 4 invariant: exempt tools survive ALL three phases unmodified") + void exemptToolsSurviveAllPhases() { + // This is the integration test: run softTrim → hardClear → prePrune + // in sequence (same order as compactMessages) and verify the + // load_skill body is still intact at the end. + ConversationWindowManager mgr = newManager(); + List messages = new ArrayList<>(List.of( + new UserMessage("load the shopping skill"), + trm("load_skill", LOAD_SKILL_BODY), + new AssistantMessage("Now let me read a file"), + trm("read_file", READ_FILE_BODY))); + + mgr.softTrimToolResults(messages); + mgr.hardClearToolResults(messages); + mgr.prePruneForSummary(messages); + + ToolResponseMessage loadSkillTrm = (ToolResponseMessage) messages.get(1); + assertEquals(LOAD_SKILL_BODY, loadSkillTrm.getResponses().get(0).responseData(), + "load_skill body must survive ALL three phases — " + + "this is the core Move 4 fix for 'compression causes attention failure'"); + + // read_file WAS modified (cleared to placeholder) + ToolResponseMessage readFileTrm = (ToolResponseMessage) messages.get(3); + assertFalse(readFileTrm.getResponses().get(0).responseData().equals(READ_FILE_BODY), + "non-exempt read_file should have been modified by at least one phase"); + } + + // ==================== helpers ==================== + + private static ConversationWindowManager newManagerWithStorage(ToolResultStorage storage) { + ConversationWindowManager mgr = new ConversationWindowManager(null, null, null); + mgr.setToolResultStorage(storage); + return mgr; + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/agent/graph/guard/ToolLoopGuardTest.java b/mateclaw-server/src/test/java/vip/mate/agent/graph/guard/ToolLoopGuardTest.java new file mode 100644 index 00000000..66c498a5 --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/agent/graph/guard/ToolLoopGuardTest.java @@ -0,0 +1,186 @@ +package vip.mate.agent.graph.guard; + +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.springframework.ai.chat.messages.AssistantMessage; +import org.springframework.ai.chat.messages.ToolResponseMessage; + +import java.util.List; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.*; + +/** + * Pure-logic tests for the tool-call loop guard: signature canonicalization, + * the failure heuristic, and the three detectors' threshold boundaries. + */ +class ToolLoopGuardTest { + + private static AssistantMessage.ToolCall call(String id, String name, String args) { + return new AssistantMessage.ToolCall(id, "function", name, args); + } + + private static ToolResponseMessage.ToolResponse result(String id, String name, String data) { + return new ToolResponseMessage.ToolResponse(id, name, data); + } + + /** Run N consecutive rounds of the same single call/result pair, chaining stats. */ + private static ToolLoopGuard.Evaluation runRounds(int rounds, String name, String args, String data) { + Map stats = Map.of(); + ToolLoopGuard.Evaluation eval = null; + for (int i = 0; i < rounds; i++) { + eval = ToolLoopGuard.evaluate(stats, + List.of(call("c1", name, args)), + List.of(result("c1", name, data))); + stats = eval.stats(); + } + return eval; + } + + // ==================== failure heuristic ==================== + + @Test + @DisplayName("失败判定：执行器异常前缀 / 安全拦截 / 工具自身错误前缀 / JSON error 字段") + void failureHeuristic() { + assertTrue(ToolLoopGuard.isFailure("Tool execution failed: boom")); + assertTrue(ToolLoopGuard.isFailure("[安全拦截] rm -rf 被拒绝。请使用更安全的替代方案。")); + assertTrue(ToolLoopGuard.isFailure("Error: file not found")); + assertTrue(ToolLoopGuard.isFailure("错误：路径不存在")); + assertTrue(ToolLoopGuard.isFailure("{\"error\":\"path outside workspace\"}")); + assertTrue(ToolLoopGuard.isFailure("{\"success\": false, \"message\":\"denied\"}")); + + assertFalse(ToolLoopGuard.isFailure("{\"filePath\":\"a.txt\",\"bytesWritten\":42}")); + assertFalse(ToolLoopGuard.isFailure("{\"error\": null, \"rows\": 3}")); + assertFalse(ToolLoopGuard.isFailure("{\"error\": \"\", \"rows\": 3}")); + assertFalse(ToolLoopGuard.isFailure("plain successful output")); + assertFalse(ToolLoopGuard.isFailure(null)); + assertFalse(ToolLoopGuard.isFailure(" ")); + } + + // ==================== signature canonicalization ==================== + + @Test + @DisplayName("参数规范化：键序与空白差异命中同一签名") + void canonicalization_keyOrderAndWhitespace() { + String a = ToolLoopGuard.canonicalizeArguments("{\"b\":1,\"a\":2}"); + String b = ToolLoopGuard.canonicalizeArguments("{ \"a\" : 2, \"b\" : 1 }"); + assertEquals(a, b); + + // Non-JSON falls back to the trimmed raw string. + assertEquals("not-json", ToolLoopGuard.canonicalizeArguments(" not-json ")); + assertEquals("", ToolLoopGuard.canonicalizeArguments(null)); + } + + @Test + @DisplayName("同参失败：不同键序也累计到同一计数器") + void exactFailure_keyOrderInsensitive() { + Map stats = Map.of(); + ToolLoopGuard.Evaluation e1 = ToolLoopGuard.evaluate(stats, + List.of(call("c1", "read_file", "{\"b\":1,\"a\":2}")), + List.of(result("c1", "read_file", "Error: nope"))); + ToolLoopGuard.Evaluation e2 = ToolLoopGuard.evaluate(e1.stats(), + List.of(call("c1", "read_file", "{\"a\":2,\"b\":1}")), + List.of(result("c1", "read_file", "Error: nope"))); + // 2nd identical-arg failure crosses the warn threshold. + assertEquals(1, e2.warnings().size()); + assertTrue(e2.warnings().get(0).contains("相同参数")); + } + + // ==================== detector 1: exact failure ==================== + + @Test + @DisplayName("同参失败：1 次不警告，2 次警告，5 次熔断") + void exactFailure_thresholds() { + assertTrue(runRounds(1, "web_search", "{\"q\":\"x\"}", "Error: rate limited").warnings().isEmpty()); + + ToolLoopGuard.Evaluation warn = runRounds(2, "web_search", "{\"q\":\"x\"}", "Error: rate limited"); + assertEquals(1, warn.warnings().size()); + assertFalse(warn.shouldHalt()); + + ToolLoopGuard.Evaluation halt = runRounds(5, "web_search", "{\"q\":\"x\"}", "Error: rate limited"); + assertTrue(halt.shouldHalt()); + assertTrue(halt.haltReason().contains("web_search")); + } + + @Test + @DisplayName("同参失败：中途成功清零计数") + void exactFailure_successResets() { + Map stats = runRounds(4, "web_search", "{\"q\":\"x\"}", "Error: rate limited").stats(); + // One success on the same signature clears the streak. + ToolLoopGuard.Evaluation ok = ToolLoopGuard.evaluate(stats, + List.of(call("c1", "web_search", "{\"q\":\"x\"}")), + List.of(result("c1", "web_search", "10 results found"))); + assertFalse(ok.shouldHalt()); + // Next failure starts from 1 again — no warning. + ToolLoopGuard.Evaluation after = ToolLoopGuard.evaluate(ok.stats(), + List.of(call("c1", "web_search", "{\"q\":\"x\"}")), + List.of(result("c1", "web_search", "Error: rate limited"))); + assertTrue(after.warnings().isEmpty()); + } + + // ==================== detector 2: per-tool failure ==================== + + @Test + @DisplayName("同工具换参失败：3 次警告，8 次熔断") + void sameToolFailure_thresholds() { + Map stats = Map.of(); + ToolLoopGuard.Evaluation eval = null; + for (int i = 0; i < 8; i++) { + eval = ToolLoopGuard.evaluate(stats, + List.of(call("c1", "read_file", "{\"path\":\"/guess/" + i + "\"}")), + List.of(result("c1", "read_file", "Error: no such file"))); + stats = eval.stats(); + if (i == 1) { + assertTrue(eval.warnings().isEmpty(), "2 failures with different args: below warn threshold"); + } + if (i == 2) { + assertEquals(1, eval.warnings().size(), "3rd failure warns"); + assertTrue(eval.warnings().get(0).contains("已失败 3 次")); + } + } + assertTrue(eval.shouldHalt(), "8th failure halts"); + } + + // ==================== detector 3: idempotent no-progress ==================== + + @Test + @DisplayName("只读工具无进展：第 2 次相同结果警告，第 5 次熔断，结果变化清零") + void noProgress_thresholds() { + assertTrue(runRounds(1, "read_file", "{\"path\":\"a\"}", "same content").warnings().isEmpty()); + + ToolLoopGuard.Evaluation warn = runRounds(2, "read_file", "{\"path\":\"a\"}", "same content"); + assertEquals(1, warn.warnings().size()); + assertTrue(warn.warnings().get(0).contains("完全相同的结果")); + + ToolLoopGuard.Evaluation halt = runRounds(5, "read_file", "{\"path\":\"a\"}", "same content"); + assertTrue(halt.shouldHalt()); + + // A changed result resets the streak. + Map stats = runRounds(4, "read_file", "{\"path\":\"a\"}", "same content").stats(); + ToolLoopGuard.Evaluation changed = ToolLoopGuard.evaluate(stats, + List.of(call("c1", "read_file", "{\"path\":\"a\"}")), + List.of(result("c1", "read_file", "different content"))); + assertFalse(changed.shouldHalt()); + assertTrue(changed.warnings().isEmpty()); + } + + @Test + @DisplayName("变更类工具不参与无进展检测") + void noProgress_mutatingToolsExempt() { + // Same successful write repeated 6 times — legitimate, never flagged. + ToolLoopGuard.Evaluation eval = runRounds(6, "write_file", + "{\"filePath\":\"a.txt\",\"content\":\"x\"}", + "{\"filePath\":\"a.txt\",\"bytesWritten\":1}"); + assertTrue(eval.warnings().isEmpty()); + assertFalse(eval.shouldHalt()); + } + + @Test + @DisplayName("空批次与空历史安全返回") + void emptyInputsAreSafe() { + ToolLoopGuard.Evaluation eval = ToolLoopGuard.evaluate(null, List.of(), List.of()); + assertTrue(eval.warnings().isEmpty()); + assertFalse(eval.shouldHalt()); + assertTrue(eval.stats().isEmpty()); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/agent/graph/node/EnvironmentNotificationRenderingTest.java b/mateclaw-server/src/test/java/vip/mate/agent/graph/node/EnvironmentNotificationRenderingTest.java new file mode 100644 index 00000000..0dca7c4d --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/agent/graph/node/EnvironmentNotificationRenderingTest.java @@ -0,0 +1,255 @@ +package vip.mate.agent.graph.node; + +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.springframework.ai.chat.messages.Message; +import org.springframework.ai.chat.messages.SystemMessage; +import vip.mate.agent.runtime.EnvironmentEventRouter; +import vip.mate.agent.runtime.EnvironmentNotification; +import vip.mate.agent.runtime.RunningConversationRegistry; +import vip.mate.skill.event.SkillUpdatedEvent; +import vip.mate.tool.mcp.event.McpConnectionLostEvent; +import vip.mate.tool.mcp.event.McpServerChangedEvent; +import vip.mate.tool.mcp.event.McpServerRemovedEvent; + +import java.time.Instant; +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * Black-box test for the C-class environment-awareness pipeline: + * MCP / skill event fires → {@link EnvironmentEventRouter} translates it → + * {@link RunningConversationRegistry} queues it → next reasoning turn drains + * the queue → {@link ReasoningNode#renderEnvironmentNotifications} renders + * the LLM-visible block. + * + *

Verifies the LLM actually receives actionable text on the next + * turn — not just that an event was queued. This is the user-facing + * contract: the agent must be told, in plain language, which tool prefix + * broke, which skill changed, and what to do about it. + * + *

The rendering helper is exercised via the real production static + * method on {@link ReasoningNode} (package-private for testability), so a + * format regression in the helper fails this test rather than a duplicate. + */ +class EnvironmentNotificationRenderingTest { + + private RunningConversationRegistry registry; + private EnvironmentEventRouter router; + + @BeforeEach + void setUp() { + registry = new RunningConversationRegistry(); + router = new EnvironmentEventRouter(registry); + } + + // ==================== Render helper contract ==================== + + @Test + void renderNullReturnsNull() { + assertThat(ReasoningNode.renderEnvironmentNotifications(null)).isNull(); + } + + @Test + void renderEmptyReturnsNull() { + // Empty drain should NOT inject "(no notifications)" noise — the + // caller checks for null and skips the SystemMessage entirely. + assertThat(ReasoningNode.renderEnvironmentNotifications(List.of())).isNull(); + } + + @Test + void renderSingleNotificationHasHeaderAndDirective() { + EnvironmentNotification n = new EnvironmentNotification( + "mcp-lost", "⚠️ server 7 down", Instant.now()); + + String block = ReasoningNode.renderEnvironmentNotifications(List.of(n)); + + assertThat(block).isNotNull(); + assertThat(block).contains("📢 环境变更通知"); + assertThat(block).contains("⚠️ server 7 down"); + // Authority + directive tail must be present — the LLM is told this + // is Java-injected truth, not a heuristic, and must adapt. + assertThat(block).contains("Java 运行时检测"); + assertThat(block).contains("立即据此调整计划"); + } + + @Test + void renderMultipleNotificationsAllVisible() { + List notes = List.of( + new EnvironmentNotification("mcp-lost", "⚠️ server 7 down", Instant.now()), + new EnvironmentNotification("skill-updated", "🔄 web-scraper updated", Instant.now()), + new EnvironmentNotification("mcp-removed", "❌ server 3 removed", Instant.now())); + + String block = ReasoningNode.renderEnvironmentNotifications(notes); + + assertThat(block).isNotNull(); + assertThat(block).contains("⚠️ server 7 down"); + assertThat(block).contains("🔄 web-scraper updated"); + assertThat(block).contains("❌ server 3 removed"); + // Each notification renders as its own bullet + assertThat(block.split("\n")).anyMatch(line -> line.trim().startsWith("- ⚠️")); + } + + // ==================== End-to-end: event → drain → render ==================== + + @Test + void mcpConnectionLostEventEndToEnd_producesActionableLLMText() { + registry.register("conv-1", 42L); + router.onMcpConnectionLost(new McpConnectionLostEvent(7L, "stdio-process-exited")); + + List notes = registry.drain("conv-1"); + String block = ReasoningNode.renderEnvironmentNotifications(notes); + + assertThat(block).isNotNull(); + // Critical actionable info: serverId, tool prefix, explicit "don't retry" + assertThat(block).contains("7"); + assertThat(block).contains("mcp_7_"); + assertThat(block).contains("不要反复重试"); + } + + @Test + void mcpServerRemovedEventEndToEnd_producesActionableLLMText() { + registry.register("conv-1", 42L); + router.onMcpServerRemoved(new McpServerRemovedEvent(3L, "fetch-server")); + + String block = ReasoningNode.renderEnvironmentNotifications(registry.drain("conv-1")); + + assertThat(block).isNotNull(); + assertThat(block).contains("fetch-server"); + assertThat(block).contains("mcp_3_"); + assertThat(block).contains("永久失效"); + } + + @Test + void mcpServerChangedEventEndToEnd_producesActionableLLMText() { + registry.register("conv-1", 42L); + router.onMcpServerChanged(new McpServerChangedEvent("mcp-rescan-complete")); + + String block = ReasoningNode.renderEnvironmentNotifications(registry.drain("conv-1")); + + assertThat(block).isNotNull(); + assertThat(block).contains("MCP 工具列表已变更"); + assertThat(block).contains("mcp-rescan-complete"); + } + + @Test + void skillUpdatedEventEndToEnd_producesActionableLLMText() { + registry.register("conv-1", 42L); + router.onSkillUpdated(new SkillUpdatedEvent(10L, "web-scraper", "update")); + + String block = ReasoningNode.renderEnvironmentNotifications(registry.drain("conv-1")); + + assertThat(block).isNotNull(); + assertThat(block).contains("web-scraper"); + assertThat(block).contains("已更新"); + // The LLM is told to re-load_skill to refresh constraints + assertThat(block).contains("load_skill"); + } + + @Test + void multipleEventsFiredMidTurnAllReachNextReasoningTurn() { + // Simulate a chaotic mid-turn environment: 3 changes fire while the + // agent is mid-tool-call. The next reasoning turn must see ALL of + // them, in order, in a single SystemMessage. + registry.register("conv-1", 42L); + router.onMcpServerChanged(new McpServerChangedEvent("change-A")); + router.onMcpConnectionLost(new McpConnectionLostEvent(5L, "lost-B")); + router.onSkillUpdated(new SkillUpdatedEvent(1L, "skill-C", "update")); + + List notes = registry.drain("conv-1"); + String block = ReasoningNode.renderEnvironmentNotifications(notes); + + assertThat(notes).hasSize(3); + assertThat(block).isNotNull(); + // Order preserved: change-A first, lost-B second, skill-C third + int idxA = block.indexOf("change-A"); + int idxB = block.indexOf("mcp_5_"); + int idxC = block.indexOf("skill-C"); + assertThat(idxA).isGreaterThan(-1); + assertThat(idxB).isGreaterThan(idxA); + assertThat(idxC).isGreaterThan(idxB); + } + + // ==================== At-most-once delivery ==================== + + @Test + void drainIsEmptyOnSecondCallSoNotificationIsInjectedAtMostOnce() { + registry.register("conv-1", 42L); + router.onMcpConnectionLost(new McpConnectionLostEvent(7L, "first")); + + List first = registry.drain("conv-1"); + List second = registry.drain("conv-1"); + + assertThat(first).hasSize(1); + assertThat(second).isEmpty(); + // The second turn's render must be skipped (returns null) — the + // notification must NOT echo into a third turn. + assertThat(ReasoningNode.renderEnvironmentNotifications(second)).isNull(); + } + + @Test + void newEventsFiredAfterFirstDrainAreVisibleOnSecondTurn() { + // Sequential delivery: event 1 fires → drained on turn 1 → event 2 + // fires → drained on turn 2. Each turn sees only what's new since + // the previous drain. + registry.register("conv-1", 42L); + router.onMcpConnectionLost(new McpConnectionLostEvent(1L, "first")); + + String turn1 = ReasoningNode.renderEnvironmentNotifications(registry.drain("conv-1")); + assertThat(turn1).contains("mcp_1_"); + + router.onMcpConnectionLost(new McpConnectionLostEvent(2L, "second")); + String turn2 = ReasoningNode.renderEnvironmentNotifications(registry.drain("conv-1")); + assertThat(turn2).contains("mcp_2_"); + assertThat(turn2).doesNotContain("mcp_1_"); // first event not re-delivered + } + + // ==================== Survives compression (nonHistoryPrefix invariant) ==================== + + @Test + void notificationBlockIsExactlyOneSystemMessage_soSurvivesCompressionByDesign() { + // The C4 injection site is nonHistoryPrefix, which is built fresh + // every reasoning turn and NEVER touched by ConversationWindowManager + // compaction (verified by ContextCompressionLedgerSurvivalTest for + // the ledger snapshot — same invariant applies here). This test + // pins the SHAPE of the injected payload so a future refactor that + // accidentally puts the notification into the history window (which + // IS subject to compaction) would fail. + registry.register("conv-1", 42L); + router.onMcpConnectionLost(new McpConnectionLostEvent(7L, "test")); + + List notes = registry.drain("conv-1"); + String block = ReasoningNode.renderEnvironmentNotifications(notes); + + Message injection = new SystemMessage(block); + assertThat(injection).isInstanceOf(SystemMessage.class); + // SystemMessage is treated as non-history by the window manager — + // it's never trimmed, never compacted, never summarised away. + assertThat(injection.getText()).isEqualTo(block); + } + + // ==================== Inactive conversation safety ==================== + + @Test + void eventFiredWhenNoConversationActiveIsDroppedSilently() { + // No active conversation → router must not throw, no notification + // lingers to be delivered to a future conversation that happens to + // reuse the same conversationId. + router.onMcpConnectionLost(new McpConnectionLostEvent(7L, "no-listener")); + + registry.register("conv-1", 42L); // register AFTER the event + assertThat(registry.drain("conv-1")).isEmpty(); + } + + @Test + void eventFiredAfterUnregisterIsDroppedSilently() { + registry.register("conv-1", 42L); + registry.unregister("conv-1"); + router.onMcpConnectionLost(new McpConnectionLostEvent(7L, "post-unregister")); + + // Re-registering must NOT receive the event that fired while inactive + registry.register("conv-1", 42L); + assertThat(registry.drain("conv-1")).isEmpty(); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/agent/graph/node/ObservationNodeLoopGuardTest.java b/mateclaw-server/src/test/java/vip/mate/agent/graph/node/ObservationNodeLoopGuardTest.java new file mode 100644 index 00000000..f23e32a8 --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/agent/graph/node/ObservationNodeLoopGuardTest.java @@ -0,0 +1,200 @@ +package vip.mate.agent.graph.node; + +import com.alibaba.cloud.ai.graph.OverAllState; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.springframework.ai.chat.messages.AssistantMessage; +import org.springframework.ai.chat.messages.ToolResponseMessage; +import vip.mate.agent.graph.observation.ObservationProcessor; +import vip.mate.config.GraphObservationProperties; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.*; +import static vip.mate.agent.graph.state.MateClawStateKeys.*; + +/** + * ObservationNode wiring for the tool-call loop guard and the one-shot + * post-mutation verification reminder: warnings land in the observation text, + * a halt lands in the ERROR slot (routing to graceful wrap-up), and the + * reminder fires exactly once per run. + */ +class ObservationNodeLoopGuardTest { + + private ObservationNode node() { + return new ObservationNode(new ObservationProcessor(new GraphObservationProperties())); + } + + private static AssistantMessage.ToolCall call(String id, String name, String args) { + return new AssistantMessage.ToolCall(id, "function", name, args); + } + + private static ToolResponseMessage.ToolResponse result(String id, String name, String data) { + return new ToolResponseMessage.ToolResponse(id, name, data); + } + + private OverAllState state(Map loopStats, Boolean reminderInjected, + List calls, + List results) { + Map m = new HashMap<>(); + m.put(CURRENT_ITERATION, 1); + m.put(MAX_ITERATIONS, 25); + m.put(OBSERVATION_HISTORY, new ArrayList()); + m.put(TOOL_CALLS, calls); + m.put(TOOL_RESULTS, results); + m.put(TOOL_CALL_COUNT, 0); + if (loopStats != null) { + m.put(TOOL_LOOP_STATS, loopStats); + } + if (reminderInjected != null) { + m.put(MUTATION_REMINDER_INJECTED, reminderInjected); + } + return new OverAllState(m); + } + + @SuppressWarnings("unchecked") + private static String lastObservation(Map out) { + List history = (List) out.get(OBSERVATION_HISTORY); + return history.get(history.size() - 1); + } + + @Test + @DisplayName("同参二次失败：警告注入观察文本，计数器写回状态") + void warnInjectedIntoObservation() throws Exception { + List calls = List.of(call("c1", "web_search", "{\"q\":\"x\"}")); + List results = + List.of(result("c1", "web_search", "Error: rate limited")); + + Map round1 = node().apply(state(null, null, calls, results)); + assertFalse(lastObservation(round1).contains("循环警告"), "1st failure: no warning yet"); + Map stats = (Map) round1.get(TOOL_LOOP_STATS); + assertNotNull(stats, "counters must be written back to state"); + + Map round2 = node().apply(state(stats, null, calls, results)); + assertTrue(lastObservation(round2).contains("循环警告"), "2nd identical failure warns"); + assertNull(round2.get(ERROR), "warning must not set the error slot"); + } + + @Test + @DisplayName("同参五次失败：置 ERROR 走优雅收尾路由") + void haltSetsError() throws Exception { + List calls = List.of(call("c1", "web_search", "{\"q\":\"x\"}")); + List results = + List.of(result("c1", "web_search", "Error: rate limited")); + + Map stats = null; + Map out = null; + for (int i = 0; i < 5; i++) { + out = node().apply(state(stats, null, calls, results)); + stats = (Map) out.get(TOOL_LOOP_STATS); + } + assertNotNull(out.get(ERROR), "5th identical failure must halt via the ERROR slot"); + assertTrue(((String) out.get(ERROR)).contains("web_search")); + } + + @Test + @DisplayName("成功写文件：验证提醒注入一次，后续轮不重复") + void verificationReminderFiresOnce() throws Exception { + List calls = + List.of(call("c1", "write_file", "{\"filePath\":\"a.txt\",\"content\":\"x\"}")); + List results = + List.of(result("c1", "write_file", "{\"filePath\":\"a.txt\",\"bytesWritten\":1}")); + + Map round1 = node().apply(state(null, null, calls, results)); + assertTrue(lastObservation(round1).contains("验证提醒"), "first successful mutation reminds"); + assertEquals(Boolean.TRUE, round1.get(MUTATION_REMINDER_INJECTED)); + + Map round2 = node().apply(state( + (Map) round1.get(TOOL_LOOP_STATS), true, calls, results)); + assertFalse(lastObservation(round2).contains("验证提醒"), "reminder is one-shot per run"); + } + + @Test + @DisplayName("写文件失败不触发验证提醒") + void failedMutationDoesNotRemind() throws Exception { + List calls = + List.of(call("c1", "write_file", "{\"filePath\":\"a.txt\",\"content\":\"x\"}")); + List results = + List.of(result("c1", "write_file", "Tool execution failed: disk full")); + + Map out = node().apply(state(null, null, calls, results)); + assertFalse(lastObservation(out).contains("验证提醒")); + assertNull(out.get(MUTATION_REMINDER_INJECTED)); + } + + @Test + @DisplayName("只读工具正常成功：无警告、无提醒、无 ERROR") + void healthyRoundIsUntouched() throws Exception { + List calls = List.of(call("c1", "read_file", "{\"path\":\"a\"}")); + List results = + List.of(result("c1", "read_file", "file content")); + + Map out = node().apply(state(null, null, calls, results)); + String obs = lastObservation(out); + assertFalse(obs.contains("循环")); + assertFalse(obs.contains("验证提醒")); + assertNull(out.get(ERROR)); + } + + // ==================== warning events (UI visibility) ==================== + + @SuppressWarnings("unchecked") + private static List warningEvents(Map out) { + var events = (List) out.get(PENDING_EVENTS); + if (events == null) return List.of(); + return events.stream() + .filter(e -> vip.mate.agent.GraphEventPublisher.EVENT_WARNING.equals(e.type())) + .toList(); + } + + @Test + @DisplayName("循环警告轮：PENDING_EVENTS 携带 warning 事件（source=loop_guard）") + void warningRoundEmitsWarningEvent() throws Exception { + List calls = List.of(call("c1", "web_search", "{\"q\":\"x\"}")); + List results = + List.of(result("c1", "web_search", "Error: rate limited")); + + Map round1 = node().apply(state(null, null, calls, results)); + assertTrue(warningEvents(round1).isEmpty(), "1st failure: no warning event"); + + Map round2 = node().apply(state( + (Map) round1.get(TOOL_LOOP_STATS), null, calls, results)); + var events = warningEvents(round2); + assertEquals(1, events.size(), "2nd identical failure emits one warning event"); + assertEquals("loop_guard", events.get(0).data().get("source")); + assertTrue(String.valueOf(events.get(0).data().get("message")).contains("循环警告")); + } + + @Test + @DisplayName("熔断轮：额外携带循环熔断 warning 事件") + void haltRoundEmitsHaltWarningEvent() throws Exception { + List calls = List.of(call("c1", "web_search", "{\"q\":\"x\"}")); + List results = + List.of(result("c1", "web_search", "Error: rate limited")); + + Map stats = null; + Map out = null; + for (int i = 0; i < 5; i++) { + out = node().apply(state(stats, null, calls, results)); + stats = (Map) out.get(TOOL_LOOP_STATS); + } + var events = warningEvents(out); + assertFalse(events.isEmpty()); + assertTrue(events.stream().anyMatch(e -> + String.valueOf(e.data().get("message")).contains("循环熔断"))); + } + + @Test + @DisplayName("健康轮：PENDING_EVENTS 无 warning 事件") + void healthyRoundEmitsNoWarningEvent() throws Exception { + List calls = List.of(call("c1", "read_file", "{\"path\":\"a\"}")); + List results = + List.of(result("c1", "read_file", "file content")); + + Map out = node().apply(state(null, null, calls, results)); + assertTrue(warningEvents(out).isEmpty()); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/agent/graph/node/ReasoningNodeLoadedSkillsHintTest.java b/mateclaw-server/src/test/java/vip/mate/agent/graph/node/ReasoningNodeLoadedSkillsHintTest.java new file mode 100644 index 00000000..42e4b021 --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/agent/graph/node/ReasoningNodeLoadedSkillsHintTest.java @@ -0,0 +1,153 @@ +package vip.mate.agent.graph.node; + +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +import java.util.LinkedHashSet; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Move 1 — coverage for {@link ReasoningNode#renderLoadedSkillsHint}. + * + *

Move 1 splits the previously-monolithic skill catalog rendering into: + *

    + *
  1. A static catalog segment rendered with {@code Set.of()} for + * loadedThisRun, so it stays prompt-cache-friendly across turns.
    2. + *
  2. A per-turn volatile suffix that tells the model which skills it + * already pulled in via {@code load_skill} this run.
    3. + *
+ * + *

This test exercises the volatile-suffix helper directly. The helper + * is package-private static so tests can assert its exact format without + * duplicating it. The contract being verified: + *

    + *
  • {@code null} or empty input → {@code null} (caller skips injection).
    • + *
  • Non-empty input → a single-line SystemMessage body.
    • + *
  • Each skill name is wrapped in backticks.
    • + *
  • The text explicitly says "do not re-load" so the model knows not + * to invoke {@code load_skill} again.
    • + *
  • Order is preserved (LinkedHashSet semantics) so the most-recently + * loaded skill is named first — useful when the model needs to + * disambiguate two skills with overlapping tool names.
    • + *
+ */ +class ReasoningNodeLoadedSkillsHintTest { + + @Test + @DisplayName("null loadedThisRun → null (caller skips injection)") + void nullSetReturnsNull() { + assertNull(ReasoningNode.renderLoadedSkillsHint(null), + "null input must return null so the caller can skip injection"); + } + + @Test + @DisplayName("empty loadedThisRun → null (caller skips injection)") + void emptySetReturnsNull() { + assertNull(ReasoningNode.renderLoadedSkillsHint(Set.of()), + "empty input must return null so the caller can skip injection"); + } + + @Test + @DisplayName("single skill → hint contains backtick-wrapped name + 'do not re-load'") + void singleSkillRendersHint() { + String hint = ReasoningNode.renderLoadedSkillsHint(Set.of("ckjia-shopping")); + + assertNotNull(hint, "non-empty input must produce a hint"); + assertTrue(hint.contains("`ckjia-shopping`"), + "skill name must be wrapped in backticks; hint was: " + hint); + assertTrue(hint.contains("do not re-load"), + "hint must explicitly say 'do not re-load'; hint was: " + hint); + assertTrue(hint.contains("loaded this run"), + "hint must mention 'loaded this run' for context; hint was: " + hint); + } + + @Test + @DisplayName("multiple skills → comma-separated backtick-wrapped names") + void multipleSkillsAreCommaSeparated() { + // LinkedHashSet so iteration order is deterministic in the assertion + Set loaded = new LinkedHashSet<>(); + loaded.add("ckjia-shopping"); + loaded.add("browser-cdp"); + loaded.add("pdf-builtin"); + + String hint = ReasoningNode.renderLoadedSkillsHint(loaded); + + assertNotNull(hint); + assertTrue(hint.contains("`ckjia-shopping`")); + assertTrue(hint.contains("`browser-cdp`")); + assertTrue(hint.contains("`pdf-builtin`")); + // All three on one line, comma-separated + assertTrue(hint.contains("`ckjia-shopping`, `browser-cdp`, `pdf-builtin`"), + "multiple skills must be comma-separated on one line; hint was: " + hint); + } + + @Test + @DisplayName("hint is a single line (no embedded newlines)") + void hintIsSingleLine() { + String hint = ReasoningNode.renderLoadedSkillsHint(Set.of("skill-a", "skill-b")); + + assertNotNull(hint); + assertFalse(hint.contains("\n"), + "hint must be a single line so it doesn't break SystemMessage formatting; hint was: " + hint); + assertTrue(hint.endsWith("."), + "hint must end with a period; hint was: " + hint); + } + + @Test + @DisplayName("hint order matches the input iteration order (LinkedHashSet)") + void hintPreservesIterationOrder() { + Set loaded = new LinkedHashSet<>(); + loaded.add("third-loaded"); + loaded.add("first-loaded"); + loaded.add("second-loaded"); + + String hint = ReasoningNode.renderLoadedSkillsHint(loaded); + + assertNotNull(hint); + int firstIdx = hint.indexOf("`first-loaded`"); + int secondIdx = hint.indexOf("`second-loaded`"); + int thirdIdx = hint.indexOf("`third-loaded`"); + assertTrue(thirdIdx < firstIdx && firstIdx < secondIdx, + "iteration order must be preserved (third loaded first); hint was: " + hint); + } + + @Test + @DisplayName("Move 1 invariant: hint text is volatile-suffix material, NOT a static-catalog segment") + void hintTextIsVolatileSuffixMaterial() { + // The hint must NOT contain language that implies it's part of the + // static catalog — that would confuse the model about which skills + // are statically visible vs. loaded this run. + String hint = ReasoningNode.renderLoadedSkillsHint(Set.of("any-skill")); + + assertNotNull(hint); + assertFalse(hint.contains("| Skill |"), + "hint must not look like a catalog table row; hint was: " + hint); + assertFalse(hint.contains("### "), + "hint must not look like a catalog section header; hint was: " + hint); + } + + @Test + @DisplayName("Move 1 vs Move 2 boundary: hint does not duplicate constraints content") + void hintDoesNotDuplicateConstraints() { + // The hint's job is to tell the model "you already loaded this + // skill, don't load it again" — it must NOT also embed the skill's + // constraints. Those live in the Constraints column of the static + // catalog (Move 2) and the ProgressLedger (B-class). Putting them + // here too would (a) duplicate token spend, (b) break the + // prompt-cache stability of the static prefix, and (c) violate + // the "position is semantics" principle. + String hint = ReasoningNode.renderLoadedSkillsHint(Set.of("ckjia-shopping")); + + assertNotNull(hint); + // Spot-check common constraint phrases that should NOT appear here + assertFalse(hint.contains("Constraints"), + "hint must not duplicate the Constraints column; hint was: " + hint); + assertFalse(hint.contains("allowed tools"), + "hint must not duplicate the allowed-tools block; hint was: " + hint); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/agent/progress/ProgressLedgerPrefixGuardTest.java b/mateclaw-server/src/test/java/vip/mate/agent/progress/ProgressLedgerPrefixGuardTest.java new file mode 100644 index 00000000..631c1940 --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/agent/progress/ProgressLedgerPrefixGuardTest.java @@ -0,0 +1,379 @@ +package vip.mate.agent.progress; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/** + * White-box tests for the prefix guard in {@link ProgressLedgerService#upsert} + * and the display-name / key-uniqueness behavior of + * {@link ProgressLedgerService#upsertAutoRecorded}. + */ +class ProgressLedgerPrefixGuardTest { + + private InMemoryService service; + + @BeforeEach + void setUp() { + service = new InMemoryService(); + } + + // ==================== Prefix Guard ==================== + + @Test + void upsertRejectsAutoPrefix() { + assertThatThrownBy(() -> + service.upsert("conv-1", "auto_read_file", "label", ProgressStatus.DONE, "note")) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("reserved"); + } + + @Test + void upsertRejectsPinPrefix() { + assertThatThrownBy(() -> + service.upsert("conv-1", "pin_my-skill_0", "label", ProgressStatus.DONE, "note")) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("reserved"); + } + + @Test + void upsertAcceptsRegularKey() { + service.upsert("conv-1", "step_research", "Research", ProgressStatus.IN_PROGRESS, "working"); + ProgressLedger ledger = service.load("conv-1"); + assertThat(ledger.asMap()).containsKey("step_research"); + } + + @Test + void upsertAcceptsStepPrefix() { + // "step_" is fine — only "auto_" and "pin_" are reserved + service.upsert("conv-1", "step_1", "Step 1", ProgressStatus.PENDING, null); + ProgressLedger ledger = service.load("conv-1"); + assertThat(ledger.asMap()).containsKey("step_1"); + } + + // ==================== Auto-Recorded: Key Uniqueness ==================== + + @Test + void autoRecordedUsesFullToolNameAsKey() { + service.upsertAutoRecorded("conv-1", "mcp_4_search_a1b2c3", "search", "found 3 results"); + ProgressLedger ledger = service.load("conv-1"); + // Key should be auto_mcp_4_search_a1b2c3 (full name), NOT auto_search + assertThat(ledger.asMap()).containsKey("auto_mcp_4_search_a1b2c3"); + ProgressEntry entry = ledger.asMap().get("auto_mcp_4_search_a1b2c3"); + assertThat(entry.getLabel()).isEqualTo("search"); // display name is slug only + } + + @Test + void autoRecordedDifferentServersNoCollision() { + // Two MCP servers both exposing "search" — must NOT collide + service.upsertAutoRecorded("conv-1", "mcp_4_search_a1b2c3", "search", "server A result"); + service.upsertAutoRecorded("conv-1", "mcp_7_search_x9y8z7", "search", "server B result"); + + ProgressLedger ledger = service.load("conv-1"); + assertThat(ledger.asMap()) + .containsKey("auto_mcp_4_search_a1b2c3") + .containsKey("auto_mcp_7_search_x9y8z7"); + assertThat(ledger.asMap()).hasSize(2); // two distinct entries + } + + @Test + void autoRecordedDoesNotOverwriteLlmEntry() { + // LLM writes a regular entry first + service.upsert("conv-1", "read_file", "Read File", ProgressStatus.IN_PROGRESS, "LLM tracking"); + // Java tries to auto-record the same tool — should be a no-op because + // the key "auto_read_file" is different from "read_file" + service.upsertAutoRecorded("conv-1", "read_file", "read_file", "file content"); + + ProgressLedger ledger = service.load("conv-1"); + // Both entries coexist — LLM entry under "read_file", auto under "auto_read_file" + assertThat(ledger.asMap()).hasSize(2); + assertThat(ledger.asMap().get("read_file").getNote()).isEqualTo("LLM tracking"); + assertThat(ledger.asMap().get("auto_read_file").getStatus()).isEqualTo(ProgressStatus.DONE); + } + + // ==================== Auto-Recorded: Bounding ==================== + + @Test + void autoRecordedBoundedToMaxFive() { + for (int i = 0; i < 10; i++) { + service.upsertAutoRecorded("conv-1", "tool_" + i, "tool_" + i, "result " + i); + } + ProgressLedger ledger = service.load("conv-1"); + long autoCount = ledger.asMap().keySet().stream() + .filter(k -> k.startsWith(ProgressLedger.AUTO_RECORDED_PREFIX)) + .count(); + assertThat(autoCount).isEqualTo(ProgressLedgerService.MAX_AUTO_RECORDED); + } + + @Test + void autoRecordedEvictsOldestFirst() { + service.upsertAutoRecorded("conv-1", "tool_a", "tool_a", "first"); + service.upsertAutoRecorded("conv-1", "tool_b", "tool_b", "second"); + service.upsertAutoRecorded("conv-1", "tool_c", "tool_c", "third"); + service.upsertAutoRecorded("conv-1", "tool_d", "tool_d", "fourth"); + service.upsertAutoRecorded("conv-1", "tool_e", "tool_e", "fifth"); + // Now at max — adding a 6th should evict tool_a (oldest) + service.upsertAutoRecorded("conv-1", "tool_f", "tool_f", "sixth"); + + ProgressLedger ledger = service.load("conv-1"); + assertThat(ledger.asMap()).doesNotContainKey("auto_tool_a"); + assertThat(ledger.asMap()).containsKey("auto_tool_f"); + } + + // ==================== Pinned Entries ==================== + + @Test + void upsertPinnedWritesToPinnedMap() { + service.upsertPinned("conv-1", "pin_skill_0", "🔒 skill: Rule A", "Rule A detail"); + ProgressLedger ledger = service.load("conv-1"); + assertThat(ledger.pinnedEntries()).containsKey("pin_skill_0"); + assertThat(ledger.pinnedEntries().get("pin_skill_0").getLabel()).isEqualTo("🔒 skill: Rule A"); + } + + @Test + void upsertDoesNotTouchPinnedMap() { + service.upsertPinned("conv-1", "pin_skill_0", "🔒 skill: Rule A", "Rule A"); + // LLM upsert should only touch entries, not pinned + service.upsert("conv-1", "step_1", "Step 1", ProgressStatus.DONE, "done"); + + ProgressLedger ledger = service.load("conv-1"); + assertThat(ledger.pinnedEntries()).hasSize(1); + assertThat(ledger.asMap()).hasSize(1); + assertThat(ledger.asMap()).containsKey("step_1"); + } + + @Test + void clearPinnedByPrefixRemovesMatchingEntries() { + service.upsertPinned("conv-1", "pin_skillA_0", "A: Rule 0", "detail"); + service.upsertPinned("conv-1", "pin_skillA_1", "A: Rule 1", "detail"); + service.upsertPinned("conv-1", "pin_skillB_0", "B: Rule 0", "detail"); + + service.clearPinnedByPrefix("conv-1", "pin_skillA_"); + + ProgressLedger ledger = service.load("conv-1"); + assertThat(ledger.pinnedEntries()).hasSize(1); + assertThat(ledger.pinnedEntries()).containsKey("pin_skillB_0"); + } + + // ==================== Rendering ==================== + + @Test + void renderSnapshotShowsAllThreeSections() { + // Pinned (B2) + service.upsertPinned("conv-1", "pin_skill_0", "🔒 skill: No delete", "No delete outside /ws"); + // Auto-recorded (B5) + service.upsertAutoRecorded("conv-1", "read_file", "read_file", "read config.yaml"); + // Regular (LLM) + service.upsert("conv-1", "step_research", "Research", ProgressStatus.IN_PROGRESS, "investigating"); + + ProgressLedger ledger = service.load("conv-1"); + String snapshot = ledger.renderSnapshot(); + + assertThat(snapshot).contains("🔒 固定约束"); + assertThat(snapshot).contains("🔧 自动记录"); + assertThat(snapshot).contains("🔄 进行中"); + assertThat(snapshot).contains("No delete outside /ws"); + assertThat(snapshot).contains("read_file"); + assertThat(snapshot).contains("Research"); + } + + @Test + void renderSnapshotNullWhenEmpty() { + ProgressLedger ledger = service.load("conv-1"); + assertThat(ledger.renderSnapshot()).isNull(); + } + + @Test + void staleReminderExcludesAutoRecordedFromRegularCheck() { + // Only auto-recorded entries — should be treated as "no regular entries" + service.upsertAutoRecorded("conv-1", "read_file", "read_file", "result"); + + ProgressLedger ledger = service.load("conv-1"); + // With only auto-recorded entries, the ledger is NOT "empty" (size > 0) + // but has no regular entries — stale reminder should nudge + String reminder = ledger.renderStaleReminder(10, java.time.Instant.now()); + // At iteration 10 (> EMPTY_LEDGER_NUDGE_ITERATIONS=5), should nudge + assertThat(reminder).contains("进度账本是空的"); + } + + // ==================== Concurrency ==================== + + @Test + void concurrentUpsertAndAutoRecordAreSafe() throws InterruptedException { + int threads = 8; + int perThread = 20; + ExecutorService pool = Executors.newFixedThreadPool(threads); + CountDownLatch latch = new CountDownLatch(threads); + AtomicInteger errors = new AtomicInteger(); + + for (int t = 0; t < threads; t++) { + final int tid = t; + pool.submit(() -> { + try { + for (int i = 0; i < perThread; i++) { + String key = "step_t" + tid + "_i" + i; + service.upsert("conv-1", key, key, ProgressStatus.PENDING, null); + service.upsertAutoRecorded("conv-1", "tool_" + tid + "_" + i, + "tool_" + i, "result"); + } + } catch (Exception e) { + errors.incrementAndGet(); + } finally { + latch.countDown(); + } + }); + } + latch.await(30, TimeUnit.SECONDS); + pool.shutdown(); + + assertThat(errors.get()).isZero(); + // Auto-recorded entries are bounded to MAX_AUTO_RECORDED regardless of concurrency + ProgressLedger ledger = service.load("conv-1"); + long autoCount = ledger.asMap().keySet().stream() + .filter(k -> k.startsWith(ProgressLedger.AUTO_RECORDED_PREFIX)) + .count(); + assertThat(autoCount).isLessThanOrEqualTo(ProgressLedgerService.MAX_AUTO_RECORDED); + } + + // ==================== Batch auto-record ==================== + + @Test + void batchInsertProducesSameResultAsSequential() { + // Insert 3 entries via batch on conv-A, 3 entries via sequential on conv-B + List batch = List.of( + new ProgressLedgerService.AutoRecordEntry("web_search", "web_search", "found 5 results"), + new ProgressLedgerService.AutoRecordEntry("read_file", "read_file", "read paper.pdf"), + new ProgressLedgerService.AutoRecordEntry("write_file", "write_file", "wrote draft.md")); + + service.upsertAutoRecordedBatch("conv-A", batch); + + service.upsertAutoRecorded("conv-B", "web_search", "web_search", "found 5 results"); + service.upsertAutoRecorded("conv-B", "read_file", "read_file", "read paper.pdf"); + service.upsertAutoRecorded("conv-B", "write_file", "write_file", "wrote draft.md"); + + // Both conversations should have identical auto-recorded entries + ProgressLedger ledgerA = service.load("conv-A"); + ProgressLedger ledgerB = service.load("conv-B"); + assertThat(ledgerA.asMap().keySet()).isEqualTo(ledgerB.asMap().keySet()); + for (String key : ledgerA.asMap().keySet()) { + ProgressEntry a = ledgerA.asMap().get(key); + ProgressEntry b = ledgerB.asMap().get(key); + assertThat(a.getLabel()).isEqualTo(b.getLabel()); + assertThat(a.getStatus()).isEqualTo(b.getStatus()); + assertThat(a.getNote()).isEqualTo(b.getNote()); + } + } + + @Test + void batchInsertBoundedToMaxFiveEvenWithLargeBatch() { + // Insert 10 entries in a single batch — should be bounded to MAX_AUTO_RECORDED + List big = new java.util.ArrayList<>(); + for (int i = 0; i < 10; i++) { + big.add(new ProgressLedgerService.AutoRecordEntry("tool_" + i, "tool_" + i, "result " + i)); + } + service.upsertAutoRecordedBatch("conv-1", big); + + ProgressLedger ledger = service.load("conv-1"); + long autoCount = ledger.asMap().keySet().stream() + .filter(k -> k.startsWith(ProgressLedger.AUTO_RECORDED_PREFIX)) + .count(); + assertThat(autoCount).isEqualTo(ProgressLedgerService.MAX_AUTO_RECORDED); + // The newest 5 (tool_5 through tool_9) should survive; oldest evicted + assertThat(ledger.asMap()).containsKey("auto_tool_9"); + assertThat(ledger.asMap()).containsKey("auto_tool_5"); + assertThat(ledger.asMap()).doesNotContainKey("auto_tool_4"); + } + + @Test + void batchInsertSkipsNullAndBlankToolNames() { + List mixed = List.of( + new ProgressLedgerService.AutoRecordEntry(null, "null-tool", "result"), + new ProgressLedgerService.AutoRecordEntry("", "blank-tool", "result"), + new ProgressLedgerService.AutoRecordEntry(" ", "whitespace-tool", "result"), + new ProgressLedgerService.AutoRecordEntry("valid_tool", "valid", "valid result")); + + service.upsertAutoRecordedBatch("conv-1", mixed); + + ProgressLedger ledger = service.load("conv-1"); + assertThat(ledger.asMap()).containsKey("auto_valid_tool"); + assertThat(ledger.asMap()).doesNotContainKey("auto_null"); + assertThat(ledger.asMap()).doesNotContainKey("auto_"); + } + + @Test + void batchInsertDoesNotOverwriteLlmAuthoredEntries() { + // LLM writes an entry with the same key the batch would use + service.upsertPinned("conv-1", "pin_skip", "pinned", "pinned-note"); + // Simulate LLM writing via direct map manipulation — write to entries + // with the auto_ prefix (this is what the LLM would do if the prefix + // guard weren't there; the guard prevents it, but we test the batch's + // skip-if-exists behavior by pre-seeding via the internal store) + // Actually, since upsert() rejects auto_ prefix, we can't pre-seed + // via the public API. Instead, test that batch doesn't overwrite + // an entry it just inserted in the same batch (dedup within batch). + List dupBatch = List.of( + new ProgressLedgerService.AutoRecordEntry("dup_tool", "dup", "first result"), + new ProgressLedgerService.AutoRecordEntry("dup_tool", "dup", "second result")); + + service.upsertAutoRecordedBatch("conv-1", dupBatch); + + ProgressLedger ledger = service.load("conv-1"); + ProgressEntry entry = ledger.asMap().get("auto_dup_tool"); + assertThat(entry).isNotNull(); + // First insert wins; second is skipped (same behavior as sequential) + assertThat(entry.getNote()).isEqualTo("first result"); + } + + @Test + void emptyBatchIsNoOp() { + service.upsertAutoRecordedBatch("conv-1", List.of()); + assertThat(service.load("conv-1").isEmpty()).isTrue(); + } + + @Test + void nullAndBlankConversationIdIgnoredInBatch() { + List batch = List.of( + new ProgressLedgerService.AutoRecordEntry("tool", "tool", "result")); + service.upsertAutoRecordedBatch(null, batch); + service.upsertAutoRecordedBatch("", batch); + service.upsertAutoRecordedBatch(" ", batch); + // No exception, no state change + } + + // ==================== Test infrastructure ==================== + + /** + * In-memory subclass that overrides DB I/O — same pattern as + * {@link ProgressLedgerServiceConcurrencyTest}. + */ + private static final class InMemoryService extends ProgressLedgerService { + private final Map store = new ConcurrentHashMap<>(); + + InMemoryService() { + super(null, new ObjectMapper().registerModule(new JavaTimeModule())); + } + + @Override + protected String loadLedgerJson(String conversationId) { + return store.get(conversationId); + } + + @Override + protected void saveLedgerJson(String conversationId, String json) { + store.put(conversationId, json); + } + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/agent/progress/ProgressLedgerStaleReminderTest.java b/mateclaw-server/src/test/java/vip/mate/agent/progress/ProgressLedgerStaleReminderTest.java index b997979a..991ea89d 100644 --- a/mateclaw-server/src/test/java/vip/mate/agent/progress/ProgressLedgerStaleReminderTest.java +++ b/mateclaw-server/src/test/java/vip/mate/agent/progress/ProgressLedgerStaleReminderTest.java @@ -23,27 +23,27 @@ class ProgressLedgerStaleReminderTest { private static final Instant NOW = Instant.parse("2026-05-24T19:30:00Z"); @Test - @DisplayName("Iteration < 10 → no reminder regardless of ledger state.") + @DisplayName("Iteration < 3 → no reminder regardless of ledger state.") void warmupPeriodNoReminder() { assertNull(ProgressLedger.empty().renderStaleReminder(0, NOW)); - assertNull(ProgressLedger.empty().renderStaleReminder(5, NOW)); - assertNull(ProgressLedger.empty().renderStaleReminder(9, NOW)); + assertNull(ProgressLedger.empty().renderStaleReminder(1, NOW)); + assertNull(ProgressLedger.empty().renderStaleReminder(2, NOW)); } @Test - @DisplayName("Empty ledger between iter 10 and 14 → still no reminder.") + @DisplayName("Empty ledger between iter 3 and 4 → still no reminder.") void emptyLedgerBelowNudgeThreshold() { - assertNull(ProgressLedger.empty().renderStaleReminder(10, NOW)); - assertNull(ProgressLedger.empty().renderStaleReminder(14, NOW)); + assertNull(ProgressLedger.empty().renderStaleReminder(3, NOW)); + assertNull(ProgressLedger.empty().renderStaleReminder(4, NOW)); } @Test - @DisplayName("Empty ledger at iter ≥ 15 → emit empty-ledger reminder.") + @DisplayName("Empty ledger at iter ≥ 5 → emit empty-ledger reminder.") void emptyLedgerTriggersReminder() { - String out = ProgressLedger.empty().renderStaleReminder(15, NOW); + String out = ProgressLedger.empty().renderStaleReminder(5, NOW); assertNotNull(out); assertTrue(out.contains("进度账本是空的"), out); - assertTrue(out.contains("15 轮"), out); + assertTrue(out.contains("5 轮"), out); assertTrue(out.contains("progress_update"), out); } @@ -57,7 +57,7 @@ class ProgressLedgerStaleReminderTest { } @Test - @DisplayName("Non-empty ledger with last update ≥ 90s ago → emit stale reminder.") + @DisplayName("Non-empty ledger with last update ≥ 45s ago → emit stale reminder.") void staleUpdateTriggersReminder() { Map entries = new LinkedHashMap<>(); entries.put("a", new ProgressEntry("a", "A", ProgressStatus.DONE, null, @@ -78,7 +78,7 @@ class ProgressLedgerStaleReminderTest { Map entries = new LinkedHashMap<>(); entries.put("a", new ProgressEntry("a", "A", ProgressStatus.DONE, null, NOW.minusSeconds(600))); - assertNull(new ProgressLedger(entries).renderStaleReminder(5, NOW)); + assertNull(new ProgressLedger(entries).renderStaleReminder(2, NOW)); } @Test diff --git a/mateclaw-server/src/test/java/vip/mate/agent/runtime/RunningConversationRegistryTest.java b/mateclaw-server/src/test/java/vip/mate/agent/runtime/RunningConversationRegistryTest.java new file mode 100644 index 00000000..5efec5fe --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/agent/runtime/RunningConversationRegistryTest.java @@ -0,0 +1,309 @@ +package vip.mate.agent.runtime; + +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import vip.mate.skill.event.SkillUpdatedEvent; +import vip.mate.tool.mcp.event.McpConnectionLostEvent; +import vip.mate.tool.mcp.event.McpServerChangedEvent; +import vip.mate.tool.mcp.event.McpServerRemovedEvent; + +import java.time.Instant; +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * White-box tests for {@link RunningConversationRegistry} and + * {@link EnvironmentEventRouter}. These verify the C-class event-routing + * pipeline: register → event fires → notification queued → drain on next turn. + */ +class RunningConversationRegistryTest { + + private RunningConversationRegistry registry; + private EnvironmentEventRouter router; + + @BeforeEach + void setUp() { + registry = new RunningConversationRegistry(); + router = new EnvironmentEventRouter(registry); + } + + // ==================== Registry lifecycle ==================== + + @Test + void registerMakesConversationActive() { + assertThat(registry.isActive("conv-1")).isFalse(); + registry.register("conv-1", 42L); + assertThat(registry.isActive("conv-1")).isTrue(); + assertThat(registry.activeConversations()).contains("conv-1"); + } + + @Test + void unregisterMakesConversationInactive() { + registry.register("conv-1", 42L); + registry.unregister("conv-1"); + assertThat(registry.isActive("conv-1")).isFalse(); + assertThat(registry.activeConversations()).doesNotContain("conv-1"); + } + + @Test + void registerIsIdempotent() { + registry.register("conv-1", 42L); + registry.register("conv-1", 42L); // no-op, just refreshes lastActiveAt + assertThat(registry.activeConversations()).hasSize(1); + } + + @Test + void unregisterUnknownConversationIsSafe() { + registry.unregister("never-registered"); + // no exception thrown + } + + @Test + void nullAndBlankConversationIdIgnored() { + registry.register(null, 42L); + registry.register("", 42L); + registry.register(" ", 42L); + assertThat(registry.activeConversations()).isEmpty(); + } + + // ==================== Queue + drain ==================== + + @Test + void drainReturnsEmptyForInactiveConversation() { + List notes = registry.drain("never-active"); + assertThat(notes).isEmpty(); + } + + @Test + void drainEmptiesTheQueue() { + registry.register("conv-1", 42L); + registry.enqueue("conv-1", new EnvironmentNotification("test", "msg-1", Instant.now())); + registry.enqueue("conv-1", new EnvironmentNotification("test", "msg-2", Instant.now())); + + List first = registry.drain("conv-1"); + assertThat(first).hasSize(2); + + List second = registry.drain("conv-1"); + assertThat(second).isEmpty(); // queue was drained + } + + @Test + void enqueueToInactiveConversationDropsMessage() { + // Event fires when no conversation is active — message is lost (by design) + registry.enqueue("never-active", new EnvironmentNotification("test", "msg", Instant.now())); + assertThat(registry.drain("never-active")).isEmpty(); + } + + @Test + void queueBoundedToTenEvictsOldest() { + registry.register("conv-1", 42L); + for (int i = 0; i < 15; i++) { + registry.enqueue("conv-1", new EnvironmentNotification("test", "msg-" + i, Instant.now())); + } + List notes = registry.drain("conv-1"); + assertThat(notes).hasSize(RunningConversationRegistry.MAX_NOTIFICATIONS_PER_CONVERSATION); + // Oldest messages (msg-0 through msg-4) should have been evicted + assertThat(notes.get(0).message()).isEqualTo("msg-5"); + assertThat(notes.get(9).message()).isEqualTo("msg-14"); + } + + // ==================== Broadcast ==================== + + @Test + void broadcastReachesAllActiveConversations() { + registry.register("conv-A", 1L); + registry.register("conv-B", 2L); + registry.register("conv-C", 3L); + + registry.broadcast(new EnvironmentNotification("mcp-lost", "server down", Instant.now())); + + assertThat(registry.drain("conv-A")).hasSize(1); + assertThat(registry.drain("conv-B")).hasSize(1); + assertThat(registry.drain("conv-C")).hasSize(1); + } + + @Test + void broadcastToNoActiveConversationsIsNoOp() { + registry.broadcast(new EnvironmentNotification("test", "msg", Instant.now())); + // no exception, no side effect + } + + // ==================== Event Router ==================== + + @Test + void mcpServerChangedEventQueuesNotification() { + registry.register("conv-1", 42L); + router.onMcpServerChanged(new McpServerChangedEvent("mcp-tools-changed:99")); + + List notes = registry.drain("conv-1"); + assertThat(notes).hasSize(1); + assertThat(notes.get(0).type()).isEqualTo("mcp-changed"); + assertThat(notes.get(0).message()).contains("MCP 工具列表已变更"); + assertThat(notes.get(0).message()).contains("mcp-tools-changed:99"); + } + + @Test + void mcpConnectionLostEventQueuesNotificationWithServerId() { + registry.register("conv-1", 42L); + router.onMcpConnectionLost(new McpConnectionLostEvent(7L, "stdio-process-exited")); + + List notes = registry.drain("conv-1"); + assertThat(notes).hasSize(1); + assertThat(notes.get(0).type()).isEqualTo("mcp-lost"); + assertThat(notes.get(0).message()).contains("7"); + assertThat(notes.get(0).message()).contains("mcp_7_"); + } + + @Test + void mcpServerRemovedEventQueuesNotification() { + registry.register("conv-1", 42L); + router.onMcpServerRemoved(new McpServerRemovedEvent(3L, "my-server")); + + List notes = registry.drain("conv-1"); + assertThat(notes).hasSize(1); + assertThat(notes.get(0).type()).isEqualTo("mcp-removed"); + assertThat(notes.get(0).message()).contains("my-server"); + assertThat(notes.get(0).message()).contains("mcp_3_"); + } + + @Test + void skillUpdatedEventQueuesNotification() { + registry.register("conv-1", 42L); + router.onSkillUpdated(new SkillUpdatedEvent(10L, "web-scraper", "update")); + + List notes = registry.drain("conv-1"); + assertThat(notes).hasSize(1); + assertThat(notes.get(0).type()).isEqualTo("skill-updated"); + assertThat(notes.get(0).message()).contains("web-scraper"); + assertThat(notes.get(0).message()).contains("已更新"); + } + + @Test + void skillUpdatedEnableEventUsesCorrectVerb() { + registry.register("conv-1", 42L); + router.onSkillUpdated(new SkillUpdatedEvent(11L, "data-processor", "enable")); + + List notes = registry.drain("conv-1"); + assertThat(notes.get(0).message()).contains("已启用"); + } + + @Test + void eventWithNoActiveConversationsIsSilentlyDropped() { + // No conversation registered — router should not throw + router.onMcpServerChanged(new McpServerChangedEvent("test")); + router.onMcpConnectionLost(new McpConnectionLostEvent(1L, "test")); + router.onMcpServerRemoved(new McpServerRemovedEvent(1L, "test")); + // No exception thrown + } + + @Test + void multipleEventsQueueInOrder() { + registry.register("conv-1", 42L); + router.onMcpServerChanged(new McpServerChangedEvent("change-1")); + router.onMcpConnectionLost(new McpConnectionLostEvent(5L, "lost-1")); + router.onSkillUpdated(new SkillUpdatedEvent(1L, "skill", "update")); + + List notes = registry.drain("conv-1"); + assertThat(notes).hasSize(3); + assertThat(notes.get(0).type()).isEqualTo("mcp-changed"); + assertThat(notes.get(1).type()).isEqualTo("mcp-lost"); + assertThat(notes.get(2).type()).isEqualTo("skill-updated"); + } + + // ==================== Stale-handle cleanup ==================== + + @Test + void cleanupStaleRemovesOldHandles() throws Exception { + registry.register("stale-conv", 1L); + // Backdate lastActiveAt to 1 hour ago via reflection + backdateLastActive("stale-conv", java.time.Instant.now().minusSeconds(3600)); + + registry.register("fresh-conv", 2L); // fresh — just registered + + int removed = registry.cleanupStale(java.time.Duration.ofMinutes(30)); + + assertThat(removed).isEqualTo(1); + assertThat(registry.isActive("stale-conv")).isFalse(); + assertThat(registry.isActive("fresh-conv")).isTrue(); + } + + @Test + void cleanupStaleKeepsActiveConversations() { + registry.register("conv-1", 1L); + registry.register("conv-2", 2L); + registry.register("conv-3", 3L); + + int removed = registry.cleanupStale(java.time.Duration.ofMinutes(30)); + + assertThat(removed).isZero(); + assertThat(registry.activeConversations()).hasSize(3); + } + + @Test + void cleanupStaleWithZeroOrNegativeDurationIsNoOp() { + registry.register("conv-1", 1L); + assertThat(registry.cleanupStale(java.time.Duration.ZERO)).isZero(); + assertThat(registry.cleanupStale(java.time.Duration.ofMillis(-1))).isZero(); + assertThat(registry.cleanupStale(null)).isZero(); + assertThat(registry.isActive("conv-1")).isTrue(); + } + + @Test + void cleanupStaleDoesNotRemoveRefreshedHandle() throws Exception { + registry.register("conv-1", 1L); + // Backdate, then re-register (which refreshes lastActiveAt) + backdateLastActive("conv-1", java.time.Instant.now().minusSeconds(3600)); + registry.register("conv-1", 1L); // refresh + + int removed = registry.cleanupStale(java.time.Duration.ofMinutes(30)); + + assertThat(removed).isZero(); + assertThat(registry.isActive("conv-1")).isTrue(); + } + + @Test + void cleanupStaleRemovesMultipleStaleHandles() throws Exception { + registry.register("stale-1", 1L); + registry.register("stale-2", 2L); + registry.register("stale-3", 3L); + registry.register("fresh-1", 4L); + + for (String conv : new String[]{"stale-1", "stale-2", "stale-3"}) { + backdateLastActive(conv, java.time.Instant.now().minusSeconds(3600)); + } + + int removed = registry.cleanupStale(java.time.Duration.ofMinutes(30)); + + assertThat(removed).isEqualTo(3); + assertThat(registry.isActive("stale-1")).isFalse(); + assertThat(registry.isActive("stale-2")).isFalse(); + assertThat(registry.isActive("stale-3")).isFalse(); + assertThat(registry.isActive("fresh-1")).isTrue(); + } + + @Test + void scheduledCleanupIsSafeToCallWithNoActiveConversations() { + registry.scheduledCleanup(); + // No exception, no side effect + } + + /** + * Helper: use reflection to backdate a conversation handle's + * {@code lastActiveAt} field, simulating a stale registration. + */ + private void backdateLastActive(String conversationId, java.time.Instant past) throws Exception { + java.lang.reflect.Field activeField = RunningConversationRegistry.class + .getDeclaredField("active"); + activeField.setAccessible(true); + @SuppressWarnings("unchecked") + java.util.concurrent.ConcurrentMap active = + (java.util.concurrent.ConcurrentMap) activeField.get(registry); + Object handle = active.get(conversationId); + assertThat(handle).as("handle must exist for conv " + conversationId).isNotNull(); + java.lang.reflect.Field lastActiveField = handle.getClass() + .getDeclaredField("lastActiveAt"); + lastActiveField.setAccessible(true); + lastActiveField.set(handle, past); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/channel/webchat/WebChatAttachmentE2ETest.java b/mateclaw-server/src/test/java/vip/mate/channel/webchat/WebChatAttachmentE2ETest.java index a41140de..1d6136c2 100644 --- a/mateclaw-server/src/test/java/vip/mate/channel/webchat/WebChatAttachmentE2ETest.java +++ b/mateclaw-server/src/test/java/vip/mate/channel/webchat/WebChatAttachmentE2ETest.java @@ -13,6 +13,7 @@ import reactor.core.publisher.Flux; import vip.mate.MateClawApplication; import vip.mate.agent.AgentService; import vip.mate.agent.model.AgentEntity; +import vip.mate.workspace.core.service.ChatUploadLocationResolver; import java.io.ByteArrayOutputStream; import java.io.IOException; @@ -282,9 +283,11 @@ class WebChatAttachmentE2ETest { assertThat(parts).contains("\"fileName\":\"note.txt\""); assertThat(parts).contains("\"contentType\":\"text/plain\""); assertThat(parts).contains("\"path\":\""); - // Path points into the conversation's upload dir on disk. + // Path points into the conversation's upload dir on disk. The dir uses + // the sanitized conversation id (cid carries ':' which is path-illegal + // on Windows), so assert against the sanitized segment. String path = extractStringField(parts, "path"); - assertThat(path).contains(cid); + assertThat(path).contains(ChatUploadLocationResolver.sanitizeSegment(cid)); assertThat(Files.isRegularFile(Path.of(path))).isTrue(); // The bytes on disk match what we uploaded. assertThat(Files.readString(Path.of(path))).isEqualTo(fileBody); diff --git a/mateclaw-server/src/test/java/vip/mate/channel/webchat/WebChatFileServiceTest.java b/mateclaw-server/src/test/java/vip/mate/channel/webchat/WebChatFileServiceTest.java index 03223a72..9e8c17b3 100644 --- a/mateclaw-server/src/test/java/vip/mate/channel/webchat/WebChatFileServiceTest.java +++ b/mateclaw-server/src/test/java/vip/mate/channel/webchat/WebChatFileServiceTest.java @@ -4,6 +4,7 @@ import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; import org.springframework.mock.web.MockMultipartFile; +import vip.mate.workspace.core.service.ChatUploadLocationResolver; import vip.mate.workspace.core.service.ChatUploadLocationResolverTestSupport; import java.io.IOException; @@ -39,7 +40,9 @@ class WebChatFileServiceTest { @AfterEach void cleanup() throws IOException { - Path dir = Paths.get("data", "chat-uploads", CONV); + // Attachments land under the sanitized segment (CONV carries ':'), so + // clean that dir — not the raw-id one. + Path dir = Paths.get("data", "chat-uploads", ChatUploadLocationResolver.sanitizeSegment(CONV)); if (Files.exists(dir)) { try (Stream walk = Files.walk(dir)) { walk.sorted(Comparator.reverseOrder()).forEach(p -> { diff --git a/mateclaw-server/src/test/java/vip/mate/content/service/ContentItemServiceTest.java b/mateclaw-server/src/test/java/vip/mate/content/service/ContentItemServiceTest.java new file mode 100644 index 00000000..8fec3895 --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/content/service/ContentItemServiceTest.java @@ -0,0 +1,64 @@ +package vip.mate.content.service; + +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import vip.mate.content.model.ContentItemEntity; +import vip.mate.content.repository.ContentItemMapper; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.*; + +/** + * Pin {@link ContentItemService#record} idempotency: re-packaging the same topic + * on the same platform within the dedup window updates the existing ledger row + * instead of inserting a duplicate; a genuinely new topic inserts a fresh row. + */ +class ContentItemServiceTest { + + private ContentItemMapper mapper; + private ContentItemService service; + + @BeforeEach + void setUp() { + mapper = mock(ContentItemMapper.class); + service = new ContentItemService(mapper); + } + + @Test + @DisplayName("re-package of a recent same topic updates the existing row, no duplicate insert") + void rePackageUpdatesInsteadOfInserting() { + ContentItemEntity existing = new ContentItemEntity(); + existing.setId(555L); + existing.setStatus("packaged"); + when(mapper.selectOne(any())).thenReturn(existing); + + Long id = service.record(1L, "gzh", "科技数码选题", "新标题", "packaged", "http://p/2", null); + + assertEquals(555L, id, "should return the existing row's id"); + verify(mapper, never()).insert(any(ContentItemEntity.class)); + verify(mapper, times(1)).updateById(existing); + assertEquals("新标题", existing.getTitle(), "title refreshed on the existing row"); + assertEquals("http://p/2", existing.getPreviewUrl()); + } + + @Test + @DisplayName("a new topic (no recent match) inserts a fresh row") + void newTopicInserts() { + when(mapper.selectOne(any())).thenReturn(null); + + service.record(1L, "xhs", "全新选题", "标题", "packaged", "http://p/1", null); + + verify(mapper, times(1)).insert(any(ContentItemEntity.class)); + verify(mapper, never()).updateById(any(ContentItemEntity.class)); + } + + @Test + @DisplayName("fingerprint falls back to title when topic is null") + void fingerprintFallsBackToTitle() { + assertEquals(ContentItemService.fingerprint("我的标题"), + ContentItemService.fingerprint("我的标题")); + assertNotEquals(ContentItemService.fingerprint("甲"), ContentItemService.fingerprint("乙")); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/skill/manifest/SkillManifestConstraintsParsingTest.java b/mateclaw-server/src/test/java/vip/mate/skill/manifest/SkillManifestConstraintsParsingTest.java new file mode 100644 index 00000000..f84e660f --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/skill/manifest/SkillManifestConstraintsParsingTest.java @@ -0,0 +1,110 @@ +package vip.mate.skill.manifest; + +import org.junit.jupiter.api.Test; +import vip.mate.skill.runtime.SkillFrontmatterParser; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * White-box test: verifies that {@link SkillManifestParser} now populates the + * {@code constraints} field from YAML frontmatter. Before the fix, the field + * was declared on {@link SkillManifest} but the parser never called + * {@code .constraints(...)} on the builder, so B2 (pinSkillConstraints) and + * agent-4 (catalog anchor) were dead code. + */ +class SkillManifestConstraintsParsingTest { + + private final SkillManifestParser parser = new SkillManifestParser(new SkillFrontmatterParser()); + + @Test + void constraintsBlockPopulatesField() { + String skillMd = """ + --- + name: my-skill + description: A skill with constraints + constraints: + - Never delete files outside /workspace + - Always confirm before running shell commands + - Use read_file before write_file + --- + # My Skill + Body content here. + """; + + SkillManifest manifest = parser.parse(skillMd); + + assertThat(manifest).isNotNull(); + assertThat(manifest.getConstraints()) + .hasSize(3) + .containsExactly( + "Never delete files outside /workspace", + "Always confirm before running shell commands", + "Use read_file before write_file"); + } + + @Test + void noConstraintsYieldsEmptyList() { + String skillMd = """ + --- + name: simple-skill + description: A skill without constraints + --- + # Simple Skill + """; + + SkillManifest manifest = parser.parse(skillMd); + + assertThat(manifest).isNotNull(); + assertThat(manifest.getConstraints()).isEmpty(); + } + + @Test + void singleStringConstraintWrapsIntoOneElementList() { + String skillMd = """ + --- + name: single-constraint-skill + constraints: "Always be polite" + --- + # Single Constraint Skill + """; + + SkillManifest manifest = parser.parse(skillMd); + + assertThat(manifest).isNotNull(); + assertThat(manifest.getConstraints()).hasSize(1); + assertThat(manifest.getConstraints().get(0)).isEqualTo("Always be polite"); + } + + @Test + void constraintsNotLeakedIntoExtras() { + String skillMd = """ + --- + name: extras-test + constraints: + - Rule A + --- + # Extras Test + """; + + SkillManifest manifest = parser.parse(skillMd); + + // constraints should be a typed field, NOT in extras + assertThat(manifest.getExtras()).doesNotContainKey("constraints"); + } + + @Test + void constraintsSurviveRoundTripThroughBuilder() { + SkillManifest original = SkillManifest.builder() + .name("round-trip") + .constraints(java.util.List.of("Rule 1", "Rule 2")) + .build(); + + // Rebuild from the same values — verifies the field is wired correctly + SkillManifest rebuilt = SkillManifest.builder() + .name(original.getName()) + .constraints(original.getConstraints()) + .build(); + + assertThat(rebuilt.getConstraints()).isEqualTo(original.getConstraints()); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/skill/runtime/SkillRuntimeServiceConstraintsAndToolsTest.java b/mateclaw-server/src/test/java/vip/mate/skill/runtime/SkillRuntimeServiceConstraintsAndToolsTest.java new file mode 100644 index 00000000..a2819860 --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/skill/runtime/SkillRuntimeServiceConstraintsAndToolsTest.java @@ -0,0 +1,359 @@ +package vip.mate.skill.runtime; + +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import vip.mate.skill.acp.AcpSkillBridge; +import vip.mate.skill.lessons.SkillLessonsService; +import vip.mate.skill.manifest.SkillManifest; +import vip.mate.skill.mcp.McpSkillBridge; +import vip.mate.skill.model.SkillEntity; +import vip.mate.skill.runtime.model.ResolvedSkill; +import vip.mate.skill.service.SkillService; +import vip.mate.skill.usage.SkillUsageService; + +import java.util.List; +import java.util.Set; +import java.util.LinkedHashSet; + +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +/** + * Move 2 & Move 3 — coverage for the new catalog segments: + *
    + *
  1. Move 2: a 4th {@code Constraints} column in the catalog table, + * populated only for bound skills whose manifest declares + * {@code constraints[]}.
    2. + *
  2. Move 3: a {@code ### Bound skill allowed tools} block after the + * table, listing each bound skill's effective tool allowlist.
    3. + *
+ * + *

Existing tests in {@link SkillRuntimeServicePromptBudgetTest} use a + * {@code resolved()} helper that builds a {@link ResolvedSkill} without a + * manifest, so neither the Constraints column nor the allowed-tools block + * is exercised. This class fills that gap. + */ +class SkillRuntimeServiceConstraintsAndToolsTest { + + @Test + @DisplayName("Move 2: bound skill with manifest constraints renders a Constraints cell") + void boundSkillWithConstraintsRendersConstraintsColumn() { + SkillService skillService = mock(SkillService.class); + SkillPackageResolver resolver = mock(SkillPackageResolver.class); + SkillLessonsService lessonsService = mock(SkillLessonsService.class); + McpSkillBridge mcpBridge = mock(McpSkillBridge.class); + AcpSkillBridge acpBridge = mock(AcpSkillBridge.class); + SkillUsageService usageService = mock(SkillUsageService.class); + + SkillEntity entity = entity(99L, "ckjia-shopping", "builtin"); + when(skillService.listEnabledSkills()).thenReturn(List.of(entity)); + SkillManifest manifest = SkillManifest.builder() + .constraints(List.of("Always confirm before writing", "Use markdown links")) + .build(); + ResolvedSkill bound = resolvedWithManifest(entity, manifest); + when(resolver.resolve(entity)).thenReturn(bound); + when(mcpBridge.listMcpDerivedResolvedSkills()).thenReturn(List.of()); + when(acpBridge.listAcpDerivedResolvedSkills()).thenReturn(List.of()); + when(usageService.recentLoadedSkillNames(null, 8)).thenReturn(Set.of()); + when(usageService.frequentlyLoadedSkillNames(8)).thenReturn(Set.of()); + + SkillRuntimeService runtime = new SkillRuntimeService( + skillService, resolver, lessonsService, mcpBridge, acpBridge, usageService); + + String prompt = runtime.buildSkillPromptEnhancement(Set.of(99L), null, 8192); + + // The table header has the Constraints column + assertTrue(prompt.contains("| Skill | Status | Description | Constraints |"), + "catalog table must declare the Constraints column; prompt was: " + prompt); + // Both constraints survive in the cell + assertTrue(prompt.contains("Always confirm before writing"), + "first constraint must render in the Constraints cell; prompt was: " + prompt); + assertTrue(prompt.contains("Use markdown links"), + "second constraint must render in the Constraints cell; prompt was: " + prompt); + } + + @Test + @DisplayName("Move 2: non-bound (recommended) skill omits constraints even when manifest has them") + void nonBoundSkillOmitsConstraints() { + SkillService skillService = mock(SkillService.class); + SkillPackageResolver resolver = mock(SkillPackageResolver.class); + SkillLessonsService lessonsService = mock(SkillLessonsService.class); + McpSkillBridge mcpBridge = mock(McpSkillBridge.class); + AcpSkillBridge acpBridge = mock(AcpSkillBridge.class); + SkillUsageService usageService = mock(SkillUsageService.class); + + SkillEntity entity = entity(7L, "pdf-builtin", "builtin"); + when(skillService.listEnabledSkills()).thenReturn(List.of(entity)); + // Manifest has constraints, but the agent does NOT bind this skill + SkillManifest manifest = SkillManifest.builder() + .constraints(List.of("Never delete source files")) + .build(); + ResolvedSkill skill = resolvedWithManifest(entity, manifest); + when(resolver.resolve(entity)).thenReturn(skill); + when(mcpBridge.listMcpDerivedResolvedSkills()).thenReturn(List.of()); + when(acpBridge.listAcpDerivedResolvedSkills()).thenReturn(List.of()); + when(usageService.recentLoadedSkillNames(null, 8)).thenReturn(Set.of()); + when(usageService.frequentlyLoadedSkillNames(8)).thenReturn(Set.of()); + + SkillRuntimeService runtime = new SkillRuntimeService( + skillService, resolver, lessonsService, mcpBridge, acpBridge, usageService); + + // boundSkillIds = null → "recommended" branch, no skill is bound + String prompt = runtime.buildSkillPromptEnhancement(null, null, 8192); + + assertTrue(prompt.contains("| Skill | Status | Description | Constraints |"), + "header still declares Constraints column; prompt was: " + prompt); + assertFalse(prompt.contains("Never delete source files"), + "non-bound skill constraints must NOT render; prompt was: " + prompt); + } + + @Test + @DisplayName("Move 2: long constraints are truncated to the per-cell budget") + void longConstraintsAreTruncated() { + SkillService skillService = mock(SkillService.class); + SkillPackageResolver resolver = mock(SkillPackageResolver.class); + SkillLessonsService lessonsService = mock(SkillLessonsService.class); + McpSkillBridge mcpBridge = mock(McpSkillBridge.class); + AcpSkillBridge acpBridge = mock(AcpSkillBridge.class); + SkillUsageService usageService = mock(SkillUsageService.class); + + SkillEntity entity = entity(42L, "long-constraints-skill", "builtin"); + when(skillService.listEnabledSkills()).thenReturn(List.of(entity)); + // Build a constraint well past CONSTRAINTS_SUMMARY_LIMIT (80 chars) + String longConstraint = "A".repeat(120); + SkillManifest manifest = SkillManifest.builder() + .constraints(List.of(longConstraint)) + .build(); + ResolvedSkill bound = resolvedWithManifest(entity, manifest); + when(resolver.resolve(entity)).thenReturn(bound); + when(mcpBridge.listMcpDerivedResolvedSkills()).thenReturn(List.of()); + when(acpBridge.listAcpDerivedResolvedSkills()).thenReturn(List.of()); + when(usageService.recentLoadedSkillNames(null, 8)).thenReturn(Set.of()); + when(usageService.frequentlyLoadedSkillNames(8)).thenReturn(Set.of()); + + SkillRuntimeService runtime = new SkillRuntimeService( + skillService, resolver, lessonsService, mcpBridge, acpBridge, usageService); + + String prompt = runtime.buildSkillPromptEnhancement(Set.of(42L), null, 8192); + + // The truncation marker appears + assertTrue(prompt.contains("..."), + "truncated constraints must end with '...'; prompt was: " + prompt); + // The full 120-char string is NOT present + assertFalse(prompt.contains("A".repeat(120)), + "long constraint must be truncated, not rendered whole; prompt was: " + prompt); + // The truncated prefix IS present (80 chars) + assertTrue(prompt.contains("A".repeat(80)), + "truncated prefix (80 'A's) must be in the cell; prompt was: " + prompt); + } + + @Test + @DisplayName("Move 2: pipe characters in constraints are escaped to protect the table layout") + void pipeInConstraintsIsEscaped() { + SkillService skillService = mock(SkillService.class); + SkillPackageResolver resolver = mock(SkillPackageResolver.class); + SkillLessonsService lessonsService = mock(SkillLessonsService.class); + McpSkillBridge mcpBridge = mock(McpSkillBridge.class); + AcpSkillBridge acpBridge = mock(AcpSkillBridge.class); + SkillUsageService usageService = mock(SkillUsageService.class); + + SkillEntity entity = entity(11L, "pipe-skill", "builtin"); + when(skillService.listEnabledSkills()).thenReturn(List.of(entity)); + SkillManifest manifest = SkillManifest.builder() + .constraints(List.of("Use option A | option B")) + .build(); + ResolvedSkill bound = resolvedWithManifest(entity, manifest); + when(resolver.resolve(entity)).thenReturn(bound); + when(mcpBridge.listMcpDerivedResolvedSkills()).thenReturn(List.of()); + when(acpBridge.listAcpDerivedResolvedSkills()).thenReturn(List.of()); + when(usageService.recentLoadedSkillNames(null, 8)).thenReturn(Set.of()); + when(usageService.frequentlyLoadedSkillNames(8)).thenReturn(Set.of()); + + SkillRuntimeService runtime = new SkillRuntimeService( + skillService, resolver, lessonsService, mcpBridge, acpBridge, usageService); + + String prompt = runtime.buildSkillPromptEnhancement(Set.of(11L), null, 8192); + + assertTrue(prompt.contains("Use option A \\| option B"), + "pipe must be escaped as \\|; prompt was: " + prompt); + } + + @Test + @DisplayName("Move 3: bound skill with allowedTools renders the 'Bound skill allowed tools' block") + void boundSkillWithAllowedToolsRendersBlock() { + SkillService skillService = mock(SkillService.class); + SkillPackageResolver resolver = mock(SkillPackageResolver.class); + SkillLessonsService lessonsService = mock(SkillLessonsService.class); + McpSkillBridge mcpBridge = mock(McpSkillBridge.class); + AcpSkillBridge acpBridge = mock(AcpSkillBridge.class); + SkillUsageService usageService = mock(SkillUsageService.class); + + SkillEntity entity = entity(99L, "ckjia-shopping", "builtin"); + when(skillService.listEnabledSkills()).thenReturn(List.of(entity)); + SkillManifest manifest = SkillManifest.builder() + .allowedTools(List.of("read_file", "execute_code")) + .build(); + ResolvedSkill bound = resolvedWithManifest(entity, manifest); + when(resolver.resolve(entity)).thenReturn(bound); + when(mcpBridge.listMcpDerivedResolvedSkills()).thenReturn(List.of()); + when(acpBridge.listAcpDerivedResolvedSkills()).thenReturn(List.of()); + when(usageService.recentLoadedSkillNames(null, 8)).thenReturn(Set.of()); + when(usageService.frequentlyLoadedSkillNames(8)).thenReturn(Set.of()); + + SkillRuntimeService runtime = new SkillRuntimeService( + skillService, resolver, lessonsService, mcpBridge, acpBridge, usageService); + + String prompt = runtime.buildSkillPromptEnhancement(Set.of(99L), null, 8192); + + assertTrue(prompt.contains("### Bound skill allowed tools"), + "allowed-tools block header must render; prompt was: " + prompt); + assertTrue(prompt.contains("`ckjia-shopping`: `read_file`, `execute_code`"), + "allowed-tools line must list skill + tools; prompt was: " + prompt); + } + + @Test + @DisplayName("Move 3: bound skill with no allowedTools omits the block entirely") + void boundSkillWithoutAllowedToolsOmitsBlock() { + SkillService skillService = mock(SkillService.class); + SkillPackageResolver resolver = mock(SkillPackageResolver.class); + SkillLessonsService lessonsService = mock(SkillLessonsService.class); + McpSkillBridge mcpBridge = mock(McpSkillBridge.class); + AcpSkillBridge acpBridge = mock(AcpSkillBridge.class); + SkillUsageService usageService = mock(SkillUsageService.class); + + SkillEntity entity = entity(99L, "doc-only-skill", "builtin"); + when(skillService.listEnabledSkills()).thenReturn(List.of(entity)); + // Manifest exists (so Constraints column could render) but has no + // allowedTools — the skill is documentation-only. + SkillManifest manifest = SkillManifest.builder() + .constraints(List.of("Read-only")) + .build(); + ResolvedSkill bound = resolvedWithManifest(entity, manifest); + when(resolver.resolve(entity)).thenReturn(bound); + when(mcpBridge.listMcpDerivedResolvedSkills()).thenReturn(List.of()); + when(acpBridge.listAcpDerivedResolvedSkills()).thenReturn(List.of()); + when(usageService.recentLoadedSkillNames(null, 8)).thenReturn(Set.of()); + when(usageService.frequentlyLoadedSkillNames(8)).thenReturn(Set.of()); + + SkillRuntimeService runtime = new SkillRuntimeService( + skillService, resolver, lessonsService, mcpBridge, acpBridge, usageService); + + String prompt = runtime.buildSkillPromptEnhancement(Set.of(99L), null, 8192); + + assertFalse(prompt.contains("### Bound skill allowed tools"), + "no allowedTools → block must be omitted; prompt was: " + prompt); + // Constraints still render + assertTrue(prompt.contains("Read-only"), + "constraints still render even without allowedTools; prompt was: " + prompt); + } + + @Test + @DisplayName("Move 3: non-bound skill with allowedTools does NOT render the block") + void nonBoundSkillWithAllowedToolsOmitsBlock() { + SkillService skillService = mock(SkillService.class); + SkillPackageResolver resolver = mock(SkillPackageResolver.class); + SkillLessonsService lessonsService = mock(SkillLessonsService.class); + McpSkillBridge mcpBridge = mock(McpSkillBridge.class); + AcpSkillBridge acpBridge = mock(AcpSkillBridge.class); + SkillUsageService usageService = mock(SkillUsageService.class); + + SkillEntity entity = entity(7L, "pdf-builtin", "builtin"); + when(skillService.listEnabledSkills()).thenReturn(List.of(entity)); + SkillManifest manifest = SkillManifest.builder() + .allowedTools(List.of("read_file")) + .build(); + ResolvedSkill skill = resolvedWithManifest(entity, manifest); + when(resolver.resolve(entity)).thenReturn(skill); + when(mcpBridge.listMcpDerivedResolvedSkills()).thenReturn(List.of()); + when(acpBridge.listAcpDerivedResolvedSkills()).thenReturn(List.of()); + when(usageService.recentLoadedSkillNames(null, 8)).thenReturn(Set.of()); + when(usageService.frequentlyLoadedSkillNames(8)).thenReturn(Set.of()); + + SkillRuntimeService runtime = new SkillRuntimeService( + skillService, resolver, lessonsService, mcpBridge, acpBridge, usageService); + + // boundSkillIds = null → no skill is bound + String prompt = runtime.buildSkillPromptEnhancement(null, null, 8192); + + assertFalse(prompt.contains("### Bound skill allowed tools"), + "non-bound skill must not render the block; prompt was: " + prompt); + } + + @Test + @DisplayName("Move 1+2+3: static catalog (render(Set.of())) omits both bound-only segments") + void staticCatalogRenderedWithEmptyBoundSetOmitsBoundSegments() { + // This mirrors how ReasoningNode now calls render(Set.of()) — + // the static catalog must NOT contain bound-only segments + // (Constraints cells, allowed-tools block), because no skill is + // bound in the static-render pass. + SkillService skillService = mock(SkillService.class); + SkillPackageResolver resolver = mock(SkillPackageResolver.class); + SkillLessonsService lessonsService = mock(SkillLessonsService.class); + McpSkillBridge mcpBridge = mock(McpSkillBridge.class); + AcpSkillBridge acpBridge = mock(AcpSkillBridge.class); + SkillUsageService usageService = mock(SkillUsageService.class); + + SkillEntity entity = entity(99L, "skill-with-everything", "builtin"); + when(skillService.listEnabledSkills()).thenReturn(List.of(entity)); + SkillManifest manifest = SkillManifest.builder() + .constraints(List.of("Always confirm")) + .allowedTools(List.of("read_file")) + .build(); + ResolvedSkill skill = resolvedWithManifest(entity, manifest); + when(resolver.resolve(entity)).thenReturn(skill); + when(mcpBridge.listMcpDerivedResolvedSkills()).thenReturn(List.of()); + when(acpBridge.listAcpDerivedResolvedSkills()).thenReturn(List.of()); + when(usageService.recentLoadedSkillNames(null, 8)).thenReturn(Set.of()); + when(usageService.frequentlyLoadedSkillNames(8)).thenReturn(Set.of()); + + SkillRuntimeService runtime = new SkillRuntimeService( + skillService, resolver, lessonsService, mcpBridge, acpBridge, usageService); + + // boundSkillIds = null simulates the static-render pass (no skill + // is "bound" from the cache-stability perspective) + String prompt = runtime.buildSkillPromptEnhancement(null, null, 8192); + + // Header still present (it's part of the table layout) + assertTrue(prompt.contains("| Skill | Status | Description | Constraints |"), + "header still declares the column; prompt was: " + prompt); + // But the bound-only content is absent + assertFalse(prompt.contains("Always confirm"), + "bound-only constraints must not render in static pass; prompt was: " + prompt); + assertFalse(prompt.contains("### Bound skill allowed tools"), + "bound-only allowed-tools block must not render in static pass; prompt was: " + prompt); + } + + // ============================ helpers ============================ + + private static SkillEntity entity(Long id, String name, String type) { + SkillEntity entity = new SkillEntity(); + entity.setId(id); + entity.setName(name); + entity.setDescription("Description for " + name); + entity.setSkillType(type); + entity.setEnabled(true); + entity.setSecurityScanStatus("PASSED"); + return entity; + } + + private static ResolvedSkill resolvedWithManifest(SkillEntity entity, SkillManifest manifest) { + // passesActiveGate requires hasAnyActiveFeature() when a manifest is + // present — without an activeFeatures entry the skill is filtered out + // by refreshActiveSkills() and the catalog comes back empty. + Set activeFeatures = new LinkedHashSet<>(); + activeFeatures.add("default"); + return ResolvedSkill.builder() + .id(entity.getId()) + .name(entity.getName()) + .description(entity.getDescription()) + .enabled(Boolean.TRUE.equals(entity.getEnabled())) + .runtimeAvailable(true) + .dependencyReady(true) + .securityBlocked(false) + .manifest(manifest) + .activeFeatures(activeFeatures) + .build(); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/skill/runtime/SkillScriptExecutionServicePipMirrorTest.java b/mateclaw-server/src/test/java/vip/mate/skill/runtime/SkillScriptExecutionServicePipMirrorTest.java new file mode 100644 index 00000000..4edce54e --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/skill/runtime/SkillScriptExecutionServicePipMirrorTest.java @@ -0,0 +1,165 @@ +package vip.mate.skill.runtime; + +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.springframework.test.util.ReflectionTestUtils; + +import java.nio.file.Path; +import java.util.Locale; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assumptions.assumeTrue; + +/** + * Tests for the pip mirror env-var injection in + * {@link SkillScriptExecutionService}. + * + *

Two scenarios: + *

    + *
  • Desktop fallback — {@code PIP_INDEX_URL} absent from the + * process env → Spring config ({@code mateclaw.pip.index-url}) is + * injected so Python subprocesses can still find the mirror.
    • + *
  • Docker / env-var precedence — {@code PIP_INDEX_URL} already + * present in the subprocess env (set via docker-compose or system env) + * → Spring config does NOT override it.
    • + *
+ * + *

Bash-gated so the suite stays green on hosts without bash (e.g. Windows CI). + */ +class SkillScriptExecutionServicePipMirrorTest { + + private static final boolean IS_WINDOWS = + System.getProperty("os.name", "").toLowerCase(Locale.ROOT).contains("win"); + + private static final String BASH_SCRIPT = + "echo PIP_INDEX_URL=$PIP_INDEX_URL\n" + + "echo PIP_TRUSTED_HOST=$PIP_TRUSTED_HOST\n"; + + private SkillScriptExecutionService newService(String indexUrl, String trustedHost) { + SkillScriptExecutionService svc = new SkillScriptExecutionService(); + ReflectionTestUtils.setField(svc, "pipIndexUrl", indexUrl); + ReflectionTestUtils.setField(svc, "pipTrustedHost", trustedHost); + return svc; + } + + @Test + @DisplayName("desktop fallback: Spring config injected when PIP_INDEX_URL absent from env") + void springConfigInjectedWhenEnvAbsent(@TempDir Path dir) { + assumeTrue(!IS_WINDOWS && hasInterpreter("bash")); + // Skip if the host already has PIP_INDEX_URL set — the inherited env + // var would make containsKey() true, hiding the Spring fallback path. + assumeTrue(System.getenv("PIP_INDEX_URL") == null, + "PIP_INDEX_URL already set in host environment"); + + var svc = newService("http://192.168.1.100:8080/simple", "192.168.1.100"); + + var result = svc.executeCode("bash", BASH_SCRIPT, dir, null, Map.of(), null); + + assertThat(result.getExitCode()).isZero(); + assertThat(result.getStdout()).contains("PIP_INDEX_URL=http://192.168.1.100:8080/simple"); + assertThat(result.getStdout()).contains("PIP_TRUSTED_HOST=192.168.1.100"); + } + + @Test + @DisplayName("Docker case: env var takes precedence over Spring config") + void envVarTakesPrecedenceOverSpringConfig(@TempDir Path dir) { + assumeTrue(!IS_WINDOWS && hasInterpreter("bash")); + + var svc = newService("http://spring-fallback:8080/simple", "spring-fallback"); + + // Simulate Docker: PIP_INDEX_URL already in the subprocess env + var result = svc.executeCode("bash", BASH_SCRIPT, dir, null, + Map.of("PIP_INDEX_URL", "http://docker-env:9090/simple", + "PIP_TRUSTED_HOST", "docker-env"), + null); + + assertThat(result.getExitCode()).isZero(); + assertThat(result.getStdout()).contains("PIP_INDEX_URL=http://docker-env:9090/simple"); + assertThat(result.getStdout()).contains("PIP_TRUSTED_HOST=docker-env"); + assertThat(result.getStdout()).doesNotContain("spring-fallback"); + } + + @Test + @DisplayName("no config: nothing injected, pip uses defaults") + void noPipConfigNoInjection(@TempDir Path dir) { + assumeTrue(!IS_WINDOWS && hasInterpreter("bash")); + assumeTrue(System.getenv("PIP_INDEX_URL") == null, + "PIP_INDEX_URL already set in host environment"); + + var svc = newService("", ""); + + var result = svc.executeCode("bash", BASH_SCRIPT, dir, null, Map.of(), null); + + assertThat(result.getExitCode()).isZero(); + assertThat(result.getStdout()).doesNotContain("http://"); + } + + @Test + @DisplayName("HTTPS index-url, no trusted-host → not auto-derived") + void onlyIndexUrlSetHttps(@TempDir Path dir) { + assumeTrue(!IS_WINDOWS && hasInterpreter("bash")); + assumeTrue(System.getenv("PIP_INDEX_URL") == null, + "PIP_INDEX_URL already set in host environment"); + + var svc = newService("https://pypi.tuna.tsinghua.edu.cn/simple", ""); + + var result = svc.executeCode("bash", BASH_SCRIPT, dir, null, Map.of(), null); + + assertThat(result.getExitCode()).isZero(); + assertThat(result.getStdout()) + .contains("PIP_INDEX_URL=https://pypi.tuna.tsinghua.edu.cn/simple"); + // HTTPS + valid cert → no trusted-host needed, should NOT be auto-derived + assertThat(result.getStdout()).contains("PIP_TRUSTED_HOST=\n"); + } + + @Test + @DisplayName("HTTP index-url, no trusted-host → auto-derived from URL") + void httpAutoDeriveTrustedHost(@TempDir Path dir) { + assumeTrue(!IS_WINDOWS && hasInterpreter("bash")); + assumeTrue(System.getenv("PIP_INDEX_URL") == null, + "PIP_INDEX_URL already set in host environment"); + + var svc = newService("http://192.168.1.100:8080/simple", ""); + + var result = svc.executeCode("bash", BASH_SCRIPT, dir, null, Map.of(), null); + + assertThat(result.getExitCode()).isZero(); + assertThat(result.getStdout()) + .contains("PIP_INDEX_URL=http://192.168.1.100:8080/simple"); + // HTTP source → trusted-host auto-derived from URL + assertThat(result.getStdout()).contains("PIP_TRUSTED_HOST=192.168.1.100"); + } + + @Test + @DisplayName("HTTP index-url with explicit trusted-host → not overwritten") + void httpExplicitTrustedHostNotOverwritten(@TempDir Path dir) { + assumeTrue(!IS_WINDOWS && hasInterpreter("bash")); + + var svc = newService("http://192.168.1.100:8080/simple", "my-mirror.local"); + + // Simulate Docker: both env vars already set + var result = svc.executeCode("bash", BASH_SCRIPT, dir, null, + Map.of("PIP_INDEX_URL", "http://10.0.0.5:9090/simple", + "PIP_TRUSTED_HOST", "10.0.0.5"), + null); + + assertThat(result.getExitCode()).isZero(); + assertThat(result.getStdout()).contains("PIP_INDEX_URL=http://10.0.0.5:9090/simple"); + assertThat(result.getStdout()).contains("PIP_TRUSTED_HOST=10.0.0.5"); + // Neither Spring config nor auto-derive should override + assertThat(result.getStdout()).doesNotContain("192.168.1.100"); + assertThat(result.getStdout()).doesNotContain("my-mirror.local"); + } + + private static boolean hasInterpreter(String name) { + try { + Process p = new ProcessBuilder(name, "--version") + .redirectErrorStream(true).start(); + return p.waitFor(10, java.util.concurrent.TimeUnit.SECONDS) && p.exitValue() == 0; + } catch (Exception e) { + return false; + } + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/system/service/SettingCryptoTest.java b/mateclaw-server/src/test/java/vip/mate/system/service/SettingCryptoTest.java new file mode 100644 index 00000000..81639556 --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/system/service/SettingCryptoTest.java @@ -0,0 +1,58 @@ +package vip.mate.system.service; + +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.*; + +/** + * Pin {@link SettingCrypto}: AES-GCM round-trips, ciphertext is prefixed and + * randomized per call, and legacy plaintext (no prefix) passes through so + * secrets stay readable during migration. + */ +class SettingCryptoTest { + + private final SettingCrypto crypto = new SettingCrypto("unit-test-key"); + + @Test + @DisplayName("encrypt → decrypt round-trips, ciphertext is prefixed and differs from plaintext") + void roundTrip() { + String secret = "wx-app-secret-1234567890"; + String enc = crypto.encrypt(secret); + assertTrue(enc.startsWith("enc:v1:"), "ciphertext must carry the version prefix"); + assertNotEquals(secret, enc); + assertEquals(secret, crypto.decrypt(enc)); + } + + @Test + @DisplayName("legacy plaintext (no prefix) is returned unchanged") + void legacyPlaintextPassthrough() { + assertEquals("old-plain-secret", crypto.decrypt("old-plain-secret")); + } + + @Test + @DisplayName("each encryption uses a fresh IV → different ciphertext, same plaintext") + void randomizedIv() { + String a = crypto.encrypt("same-value"); + String b = crypto.encrypt("same-value"); + assertNotEquals(a, b, "distinct IVs must yield distinct ciphertext"); + assertEquals("same-value", crypto.decrypt(a)); + assertEquals("same-value", crypto.decrypt(b)); + } + + @Test + @DisplayName("blank/null pass through untouched") + void blankPassthrough() { + assertEquals("", crypto.encrypt("")); + assertNull(crypto.encrypt(null)); + assertNull(crypto.decrypt(null)); + } + + @Test + @DisplayName("wrong key cannot read another key's ciphertext") + void wrongKeyFailsClosed() { + String enc = crypto.encrypt("top-secret"); + String recovered = new SettingCrypto("a-different-key").decrypt(enc); + assertEquals("", recovered, "a wrong key must not return the real secret"); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/system/service/SystemSettingBoolApiTest.java b/mateclaw-server/src/test/java/vip/mate/system/service/SystemSettingBoolApiTest.java index a4c6f55c..0f3a78f7 100644 --- a/mateclaw-server/src/test/java/vip/mate/system/service/SystemSettingBoolApiTest.java +++ b/mateclaw-server/src/test/java/vip/mate/system/service/SystemSettingBoolApiTest.java @@ -46,7 +46,8 @@ class SystemSettingBoolApiTest { @BeforeEach void setUp() { - service = new SystemSettingService(mapper, new SearchProviderRegistry(List.of()), mock(PluginManager.class)); + service = new SystemSettingService(mapper, new SearchProviderRegistry(List.of()), + new SettingCrypto("test-key"), mock(PluginManager.class)); } private SystemSettingEntity row(String value) { diff --git a/mateclaw-server/src/test/java/vip/mate/system/service/SystemSettingServiceCatalogTest.java b/mateclaw-server/src/test/java/vip/mate/system/service/SystemSettingServiceCatalogTest.java index 999b2d3f..18b004a0 100644 --- a/mateclaw-server/src/test/java/vip/mate/system/service/SystemSettingServiceCatalogTest.java +++ b/mateclaw-server/src/test/java/vip/mate/system/service/SystemSettingServiceCatalogTest.java @@ -68,7 +68,7 @@ class SystemSettingServiceCatalogTest { @DisplayName("marks builtin providers as builtin=true with no pluginName") void builtinEntry() { SearchProviderRegistry registry = new SearchProviderRegistry(List.of(stub("serper", 300, true, false))); - service = new SystemSettingService(mapper, registry, pluginManager); + service = new SystemSettingService(mapper, registry, new SettingCrypto("test-key"), pluginManager); SearchProviderCatalogResponse catalog = service.getSearchProviderCatalog(); @@ -86,7 +86,7 @@ class SystemSettingServiceCatalogTest { SearchProviderRegistry registry = new SearchProviderRegistry(List.of()); registry.registerPluginProvider(stub("my-search", 500, true, true)); when(pluginManager.getPluginNameForSearchProvider("my-search")).thenReturn("my-plugin"); - service = new SystemSettingService(mapper, registry, pluginManager); + service = new SystemSettingService(mapper, registry, new SettingCrypto("test-key"), pluginManager); SearchProviderCatalogResponse catalog = service.getSearchProviderCatalog(); @@ -105,7 +105,7 @@ class SystemSettingServiceCatalogTest { stub("duckduckgo", 100, false, true))); registry.registerPluginProvider(stub("my-search", 200, true, true)); when(pluginManager.getPluginNameForSearchProvider("my-search")).thenReturn("my-plugin"); - service = new SystemSettingService(mapper, registry, pluginManager); + service = new SystemSettingService(mapper, registry, new SettingCrypto("test-key"), pluginManager); SearchProviderCatalogResponse catalog = service.getSearchProviderCatalog(); @@ -128,7 +128,7 @@ class SystemSettingServiceCatalogTest { @DisplayName("surfaces the resolved provider id and source alongside the catalog") void resolvedSurfaced() { SearchProviderRegistry registry = new SearchProviderRegistry(List.of(stub("duckduckgo", 100, false, true))); - service = new SystemSettingService(mapper, registry, pluginManager); + service = new SystemSettingService(mapper, registry, new SettingCrypto("test-key"), pluginManager); SearchProviderCatalogResponse catalog = service.getSearchProviderCatalog(); @@ -140,7 +140,7 @@ class SystemSettingServiceCatalogTest { @DisplayName("resolvedId/resolvedSource are null when no provider is available at all") void resolvedNullWhenNothingAvailable() { SearchProviderRegistry registry = new SearchProviderRegistry(List.of(stub("serper", 300, true, false))); - service = new SystemSettingService(mapper, registry, pluginManager); + service = new SystemSettingService(mapper, registry, new SettingCrypto("test-key"), pluginManager); SearchProviderCatalogResponse catalog = service.getSearchProviderCatalog(); diff --git a/mateclaw-server/src/test/java/vip/mate/tool/browser/BrowserPrivacyGuardTest.java b/mateclaw-server/src/test/java/vip/mate/tool/browser/BrowserPrivacyGuardTest.java new file mode 100644 index 00000000..65a31c04 --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/tool/browser/BrowserPrivacyGuardTest.java @@ -0,0 +1,86 @@ +package vip.mate.tool.browser; + +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +import java.util.List; + +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Unit tests for {@link BrowserPrivacyGuard} classification and blocking. + * The audit mapper is never touched by {@code isSensitive}/{@code blockReason}, + * so a null mapper is fine here. + */ +class BrowserPrivacyGuardTest { + + private BrowserPrivacyGuard guardWith(List sensitive, List trusted) { + BrowserProperties props = new BrowserProperties(); + props.getPrivacy().setSensitiveHosts(sensitive); + props.getPrivacy().setTrustedHosts(trusted); + return new BrowserPrivacyGuard(props, null); + } + + @Test + @DisplayName("Heuristic flags banking / webmail / login / admin pages") + void heuristicFlagsSensitivePages() { + BrowserPrivacyGuard g = guardWith(List.of(), List.of()); + assertTrue(g.isSensitive("https://www.chase.com/banking")); + assertTrue(g.isSensitive("https://mail.google.com/mail/u/0")); + assertTrue(g.isSensitive("https://github.com/login")); + assertTrue(g.isSensitive("https://example.com/account/settings")); + assertTrue(g.isSensitive("https://admin.example.com/")); + } + + @Test + @DisplayName("Ordinary content pages are not flagged") + void ordinaryPagesNotFlagged() { + BrowserPrivacyGuard g = guardWith(List.of(), List.of()); + assertFalse(g.isSensitive("https://example.com/products/42")); + assertFalse(g.isSensitive("https://news.example.com/article/hello-world")); + assertFalse(g.isSensitive("")); + assertFalse(g.isSensitive(null)); + } + + @Test + @DisplayName("trustedHosts overrides both the heuristic and sensitiveHosts") + void trustedOverridesEverything() { + BrowserPrivacyGuard g = guardWith(List.of("internal.corp"), List.of("mail.google.com")); + assertFalse(g.isSensitive("https://mail.google.com/mail")); + // subdomain of a trusted host is also trusted + BrowserPrivacyGuard g2 = guardWith(List.of(), List.of("example.com")); + assertFalse(g2.isSensitive("https://admin.example.com/login")); + } + + @Test + @DisplayName("sensitiveHosts adds hosts the heuristic would miss") + void sensitiveHostsExtendHeuristic() { + BrowserPrivacyGuard g = guardWith(List.of("intranet.corp"), List.of()); + assertTrue(g.isSensitive("https://intranet.corp/dashboard")); + assertTrue(g.isSensitive("https://hr.intranet.corp/")); + } + + @Test + @DisplayName("blockReason only fires for a user-managed browser on a sensitive page") + void blockReasonScope() { + BrowserPrivacyGuard g = guardWith(List.of(), List.of()); + // sensitive page but self-spawned browser → allowed + assertNull(g.blockReason(false, "https://github.com/login", "screenshot")); + // user-managed browser but ordinary page → allowed + assertNull(g.blockReason(true, "https://example.com/products", "eval")); + // user-managed browser on a sensitive page → blocked + assertNotNull(g.blockReason(true, "https://github.com/login", "eval")); + } + + @Test + @DisplayName("Disabling the guard allows everything") + void disabledGuardAllows() { + BrowserProperties props = new BrowserProperties(); + props.getPrivacy().setEnabled(false); + BrowserPrivacyGuard g = new BrowserPrivacyGuard(props, null); + assertNull(g.blockReason(true, "https://www.chase.com/banking", "screenshot")); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/tool/browser/PageSnapshotProbe.java b/mateclaw-server/src/test/java/vip/mate/tool/browser/PageSnapshotProbe.java new file mode 100644 index 00000000..4fed6a64 --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/tool/browser/PageSnapshotProbe.java @@ -0,0 +1,116 @@ +package vip.mate.tool.browser; + +import cn.hutool.json.JSONObject; +import com.microsoft.playwright.Browser; +import com.microsoft.playwright.ElementHandle; +import com.microsoft.playwright.Page; +import com.microsoft.playwright.Playwright; +import vip.mate.common.net.SsrfProperties; + +import java.util.List; + +/** + * Manual end-to-end probe for {@link PageSnapshotScript}. Not a JUnit test — run via + * {@code mvn -q test-compile exec:java -Dexec.mainClass=vip.mate.tool.browser.PageSnapshotProbe + * -Dexec.classpathScope=test} (use {@code test-compile}, not {@code compile}, so this + * test-scoped class is built). + * + *

Launches a real browser and drives the EXACT path the tool uses — + * {@code page.querySelector("body").evaluate(PageSnapshotScript.SNAPSHOT_JS, opts)} — + * against a known page. It exists to lock the most subtle failure mode found in + * review: Playwright's {@code ElementHandle.evaluate} passes the element as the + * FIRST POSITIONAL ARGUMENT (never as {@code this}). If the snapshot function is + * ever reverted to read {@code this}, the walk starts from the wrong root and the + * tree comes back empty (or throws) — this probe then fails loudly instead of the + * regression shipping silently, the way the original {@code this}-based version did. + * + *

Asserts, on a fixed HTML page: + *

    + *
  • references are assigned (non-empty) and materialised as {@code data-mate-ref};
    • + *
  • interactive elements appear with the right role + accessible name + * (incl. wrapping-label resolution and the {@code display:none} filter);
    • + *
  • {@code [data-mate-ref='e1']} resolves back to the expected element.
    • + *
+ * Exits non-zero on any failed assertion. + */ +public final class PageSnapshotProbe { + + private static final String HTML = """ + +

Probe Form

+ + Learn more + + +
+ + """; + + private static int failures = 0; + + public static void main(String[] args) { + System.out.println("=== PageSnapshotScript probe ==="); + BrowserProperties props = new BrowserProperties(); + BrowserLauncher launcher = new BrowserLauncher(props, new SsrfProperties()); + + int exit = 0; + try (Playwright pw = Playwright.create()) { + BrowserLauncher.Result r = launcher.launch(pw, false); + if (!r.isSuccess()) { + System.err.println("FAIL: could not launch a browser: " + r.getFailureSummary()); + System.exit(1); + } + try (Browser browser = r.getBrowser()) { + Page page = r.getPage(); + page.setContent(HTML); + + // EXACT tool path: element handle + Hutool JSONObject opts. + ElementHandle root = page.querySelector("body"); + JSONObject opts = new JSONObject(); + opts.set("maxLen", 20_000); + opts.set("includeNonInteractive", true); + String json = (String) root.evaluate(PageSnapshotScript.SNAPSHOT_JS, opts); + PageSnapshotScript.Result snap = PageSnapshotScript.Result.fromJson(json); + + System.out.println("tree:\n" + snap.tree()); + System.out.println("refs: " + snap.refs()); + + List refs = snap.refs(); + String tree = snap.tree(); + + // The regression guard: the this-vs-first-arg bug makes this empty. + check(!refs.isEmpty(), "references assigned (element passed as first arg, not `this`)"); + check(refs.size() >= 4, "at least 4 interactive refs (input, link, select, button), got " + refs.size()); + + check(tree.contains("textbox \"Customer name:\""), "input resolves wrapping-label name"); + check(tree.contains("link \"Learn more\""), "anchor rendered as link with text"); + check(tree.contains("combobox"), "select rendered as combobox"); + check(tree.contains("button \"Submit\""), "button rendered with text"); + check(tree.contains("heading \"Probe Form\" [level=1]"), "h1 rendered as heading level 1"); + check(!tree.contains("Hidden Button"), "display:none subtree is filtered out"); + + boolean e1Resolves = page.querySelector("[data-mate-ref='e1']") != null; + check(e1Resolves, "[data-mate-ref='e1'] resolves back to a live element"); + + if (failures == 0) { + System.out.println("\nALL CHECKS PASSED (" + refs.size() + " refs) via " + r.getStrategy()); + } else { + System.err.println("\n" + failures + " CHECK(S) FAILED"); + exit = 1; + } + } + } catch (Exception e) { + System.err.println("FAIL: probe threw: " + e); + e.printStackTrace(); + exit = 1; + } + System.exit(exit); + } + + private static void check(boolean ok, String what) { + System.out.println((ok ? " [PASS] " : " [FAIL] ") + what); + if (!ok) { + failures++; + } + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/tool/builtin/ComplianceScannerTest.java b/mateclaw-server/src/test/java/vip/mate/tool/builtin/ComplianceScannerTest.java new file mode 100644 index 00000000..08fe3711 --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/tool/builtin/ComplianceScannerTest.java @@ -0,0 +1,68 @@ +package vip.mate.tool.builtin; + +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +import java.util.List; + +import static org.junit.jupiter.api.Assertions.*; + +/** + * Pin {@link ComplianceScanner}: high-risk categories (极限词 / 诱导 / 承诺收益) are + * flagged as high-risk so the publish path can hard-block them, medical-efficacy + * is a non-high-risk hit, and clean copy scans clean. + */ +class ComplianceScannerTest { + + @Test + @DisplayName("广告法 极限词 → high-risk hit") + void adLawSuperlative() { + ComplianceScanner.Result r = ComplianceScanner.scan("我们是全国第一、效果最好的品牌"); + assertFalse(r.clean()); + assertTrue(r.hasHighRisk()); + assertTrue(ComplianceScanner.report(r).contains("广告法极限词")); + } + + @Test + @DisplayName("WeChat 诱导 words → high-risk hit") + void weChatInduce() { + ComplianceScanner.Result r = ComplianceScanner.scan("集赞 20 个送礼品，分享到朋友圈解锁全文"); + assertTrue(r.hasHighRisk()); + assertTrue(ComplianceScanner.report(r).contains("微信诱导")); + } + + @Test + @DisplayName("promised returns → high-risk hit") + void promisedReturns() { + assertTrue(ComplianceScanner.scan("保本理财，稳赚不赔").hasHighRisk()); + } + + @Test + @DisplayName("medical efficacy → hit but NOT high-risk") + void medicalEfficacyNotHighRisk() { + ComplianceScanner.Result r = ComplianceScanner.scan("这款茶能排毒养颜"); + assertFalse(r.clean()); + assertFalse(r.hasHighRisk(), "医疗功效 is a warning, not a hard block"); + } + + @Test + @DisplayName("extra banned words merge in as a non-high-risk 自定义禁用词 category") + void extraBannedWords() { + ComplianceScanner.Result r = ComplianceScanner.scan( + "这段文字提到了竞品X和内部代号Y", List.of("竞品X", "内部代号Y", "没出现的词")); + assertFalse(r.clean()); + assertFalse(r.hasHighRisk(), "custom banned words are a warning, not a hard block"); + String rep = ComplianceScanner.report(r); + assertTrue(rep.contains("自定义禁用词")); + assertTrue(rep.contains("竞品X") && rep.contains("内部代号Y")); + assertFalse(rep.contains("没出现的词"), "only actually-present terms are reported"); + } + + @Test + @DisplayName("clean copy scans clean") + void cleanCopy() { + ComplianceScanner.Result r = ComplianceScanner.scan("这是我上周做的三道家常菜，步骤和用量都写清楚了。"); + assertTrue(r.clean()); + assertTrue(ComplianceScanner.report(r).contains("未命中")); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/tool/builtin/ContentItemToolTest.java b/mateclaw-server/src/test/java/vip/mate/tool/builtin/ContentItemToolTest.java new file mode 100644 index 00000000..44d557b1 --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/tool/builtin/ContentItemToolTest.java @@ -0,0 +1,94 @@ +package vip.mate.tool.builtin; + +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import vip.mate.content.model.ContentItemEntity; +import vip.mate.content.service.ContentItemService; + +import java.time.LocalDateTime; +import java.util.List; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.ArgumentMatchers.*; +import static org.mockito.Mockito.*; + +/** + * Pin {@link ContentItemTool}: the fingerprint is stable across cosmetic + * differences, and check_recent / record / mark_published delegate correctly. + */ +class ContentItemToolTest { + + private ContentItemService service; + private ContentItemTool tool; + + @BeforeEach + void setUp() { + service = mock(ContentItemService.class); + tool = new ContentItemTool(service); + } + + @Test + @DisplayName("fingerprint ignores case / whitespace / punctuation but distinguishes real topics") + void fingerprintStable() { + String a = ContentItemService.fingerprint("周末咖啡探店"); + String b = ContentItemService.fingerprint(" 周末 咖啡，探店！ "); + assertEquals(a, b, "cosmetic differences must collapse to the same fingerprint"); + assertNotEquals(a, ContentItemService.fingerprint("露营装备清单"), "different topics differ"); + } + + @Test + @DisplayName("check_recent: empty history → not a repeat") + void checkRecentEmpty() { + when(service.findRecent(eq("gzh"), eq("周末咖啡探店"), anyInt())).thenReturn(List.of()); + String out = tool.content_item("check_recent", "gzh", "周末咖啡探店", + null, null, null, null, 14, null, null); + assertTrue(out.contains("未重复"), out); + } + + @Test + @DisplayName("check_recent: recent same-topic row → flagged as repeat with its title") + void checkRecentRepeat() { + ContentItemEntity prior = new ContentItemEntity(); + prior.setTitle("上周那篇咖啡探店"); + prior.setStatus("published"); + prior.setCreateTime(LocalDateTime.now().minusDays(3)); + when(service.findRecent(eq("gzh"), eq("周末咖啡探店"), anyInt())).thenReturn(List.of(prior)); + + String out = tool.content_item("check_recent", "gzh", "周末咖啡探店", + null, null, null, null, 14, null, null); + assertTrue(out.contains("疑似重复"), out); + assertTrue(out.contains("上周那篇咖啡探店"), "should show the prior title"); + } + + @Test + @DisplayName("record: delegates to the service and reports the item id") + void recordDelegates() { + when(service.record(any(), eq("xhs"), eq("露营装备清单"), any(), any(), any(), any())) + .thenReturn(999L); + String out = tool.content_item("record", "xhs", "露营装备清单", + "新手露营必带的8样东西", "packaged", "http://x/preview", null, null, null, null); + verify(service, times(1)).record(any(), eq("xhs"), eq("露营装备清单"), + eq("新手露营必带的8样东西"), eq("packaged"), eq("http://x/preview"), isNull()); + assertTrue(out.contains("999"), out); + } + + @Test + @DisplayName("mark_published: reports success / not-found from the service") + void markPublished() { + when(service.markPublished(123L, "media_abc")).thenReturn(true); + assertTrue(tool.content_item("mark_published", null, null, null, null, + null, "media_abc", null, 123L, null).contains("已标记为已发布")); + + when(service.markPublished(404L, null)).thenReturn(false); + assertTrue(tool.content_item("mark_published", null, null, null, null, + null, null, null, 404L, null).startsWith("Error:")); + } + + @Test + @DisplayName("unknown action is rejected") + void unknownAction() { + assertTrue(tool.content_item("frobnicate", null, null, null, null, null, null, null, null, null) + .startsWith("Error:")); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/tool/builtin/GzhPackageCoverHealingTest.java b/mateclaw-server/src/test/java/vip/mate/tool/builtin/GzhPackageCoverHealingTest.java new file mode 100644 index 00000000..10e11418 --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/tool/builtin/GzhPackageCoverHealingTest.java @@ -0,0 +1,132 @@ +package vip.mate.tool.builtin; + +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import vip.mate.content.service.ContentItemService; +import vip.mate.tool.document.GeneratedFileCache; + +import java.nio.file.Path; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.ArgumentMatchers.*; +import static org.mockito.Mockito.*; + +/** + * Reproduce and pin the fix for the broken-cover bug: when the model references + * the cover by its logical filename ({@code cover_xyz.png}) instead of the + * issued id, the URL id-pattern can't parse it, so the old tool embedded the raw + * (non-serving) reference and the preview showed a broken image. The packager + * must now (a) self-heal a name-based reference to the real generated image, and + * (b) when a cover genuinely can't be resolved, drop it and warn rather than ship + * a broken {@code }. + */ +class GzhPackageCoverHealingTest { + + private GeneratedFileCache cache; + private ContentItemService contentSvc; + private GzhPackageTool tool; + + @BeforeEach + void setUp(@TempDir Path tempDir) { + cache = new GeneratedFileCache(tempDir); + contentSvc = mock(ContentItemService.class); + tool = new GzhPackageTool(cache, contentSvc); + } + + private static final String BODY = "## 小节一\n\n正文一段。\n\n## 小节二\n\n又一段。"; + + @Test + @DisplayName("name-based cover reference self-heals to the real generated image") + void nameBasedReferenceHeals() { + String id = cache.put("PNGBYTES".getBytes(), "cover_cat_7pits.png", "image/png"); + // The model echoed the cover by filename — the id pattern stops at the '_'. + String out = tool.gzh_package( + "养猫第一年烧掉3万块", + BODY, + "/api/v1/files/generated/cover_cat_7pits.png", + "内容工作室", + null, + null); + + assertTrue(out.contains("/api/v1/files/generated/" + id), + "cover should be healed to the real generated id; got:\n" + out); + assertFalse(out.contains("generated/cover_cat_7pits.png"), + "the broken name-based reference must not be embedded"); + assertTrue(out.contains("") + void unresolvableCoverUsesPlaceholder() { + String out = tool.gzh_package( + "标题", + BODY, + "/api/v1/files/generated/deadbeef-0000-0000-0000-000000000000", + "内容工作室", + null, + null); + + assertTrue(out.contains("⚠️"), "an unresolved cover must be flagged; got:\n" + out); + assertTrue(out.contains("占位封面"), "a placeholder cover must be substituted"); + assertTrue(out.contains(""; + WxMpService wx = wxReturning("http://mmbiz.qpic.cn/mmbiz_png/abc/0"); + + GzhPublishTool.ImageInlineResult r = tool.inlineContentImages(wx, html); + + assertEquals(1, r.uploaded()); + assertTrue(r.failed().isEmpty()); + assertTrue(r.html().contains("http://mmbiz.qpic.cn/mmbiz_png/abc/0"), "src must be rewritten"); + assertFalse(r.html().contains("/api/v1/files/generated/"), "the external ref must be gone"); + } + + @Test + @DisplayName("an image already on mp.weixin.qq.com is left untouched and not re-uploaded") + void leavesWeChatImageAlone() throws Exception { + String html = ""; + WxMpService wx = wxReturning("http://mmbiz.qpic.cn/should-not-be-used"); + + GzhPublishTool.ImageInlineResult r = tool.inlineContentImages(wx, html); + + assertEquals(0, r.uploaded()); + assertTrue(r.failed().isEmpty()); + assertTrue(r.html().contains("mp.weixin.qq.com/existing.png")); + verify(wx, never()).getMaterialService(); + } + + @Test + @DisplayName("an unresolvable body image is reported as failed but does not block the rest") + void unresolvableImageReportedNotBlocking() throws Exception { + String good = cache.put("PNGDATA".getBytes(), "ok.png", "image/png"); + String html = "" + + ""; + WxMpService wx = wxReturning("http://mmbiz.qpic.cn/mmbiz_png/ok/0"); + + GzhPublishTool.ImageInlineResult r = tool.inlineContentImages(wx, html); + + assertEquals(1, r.uploaded(), "the good image still uploads"); + assertEquals(1, r.failed().size(), "the missing image is reported"); + assertTrue(r.html().contains("http://mmbiz.qpic.cn/mmbiz_png/ok/0")); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/tool/builtin/XhsPackageTest.java b/mateclaw-server/src/test/java/vip/mate/tool/builtin/XhsPackageTest.java new file mode 100644 index 00000000..f6e49857 --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/tool/builtin/XhsPackageTest.java @@ -0,0 +1,87 @@ +package vip.mate.tool.builtin; + +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import vip.mate.content.service.ContentItemService; +import vip.mate.tool.document.GeneratedFileCache; + +import static org.mockito.Mockito.mock; + +import java.nio.charset.StandardCharsets; +import java.nio.file.Path; +import java.util.regex.Matcher; + +import static org.junit.jupiter.api.Assertions.*; + +/** + * Pin {@link XhsPackageTool}: 小红书 is image-first, so packaging must (a) refuse + * a note with fewer than 3 resolvable images, (b) render an image-first preview + * (the images come before the copy), and (c) self-heal an image referenced by + * filename instead of its issued id. + */ +class XhsPackageTest { + + private GeneratedFileCache cache; + private XhsPackageTool tool; + + @BeforeEach + void setUp(@TempDir Path tempDir) { + cache = new GeneratedFileCache(tempDir); + tool = new XhsPackageTool(cache, mock(ContentItemService.class)); + } + + private String putImg(String name) { + String id = cache.put("PNGDATA".getBytes(), name, "image/png"); + return "/api/v1/files/generated/" + id; + } + + /** Read back the online-preview HTML that the tool stored, given its result text. */ + private String previewHtml(String out) { + Matcher m = GeneratedFileCache.GENERATED_URL_PATTERN.matcher(out); + assertTrue(m.find(), "result should contain a preview URL"); + return new String(cache.get(m.group(1)).orElseThrow().bytes(), StandardCharsets.UTF_8); + } + + @Test + @DisplayName("fewer than 3 images → refused, no preview minted") + void refusesUnderThreeImages() { + String imgs = putImg("cover.png") + "," + putImg("c1.png"); + String out = tool.xhs_package("夏日穿搭", "正文", "穿搭,夏天", imgs, null, null); + assertTrue(out.contains("至少需要 3 张"), "should demand >=3 images; got:\n" + out); + assertFalse(out.contains("在线预览"), "must not produce a preview when refused"); + } + + @Test + @DisplayName("3 images → packaged; preview is image-first (images before the copy)") + void packagesThreeImagesImageFirst() { + String imgs = putImg("cover.png") + "," + putImg("c1.png") + "," + putImg("c2.png"); + String out = tool.xhs_package("3天2夜厦门citywalk", "第一天去了鼓浪屿\n人不多", "厦门,citywalk,旅行", imgs, null, null); + + assertTrue(out.contains("在线预览"), "should return a preview link"); + assertTrue(out.contains("素材下载"), "should return a material zip"); + assertTrue(out.contains("3 张图"), "should report the image count"); + + String html = previewHtml(out); + int imgCount = html.split("=3; got:\n" + out); + assertTrue(out.contains("3 张图"), "healed image should be counted"); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/tool/disclosure/ToolDisclosureServiceTest.java b/mateclaw-server/src/test/java/vip/mate/tool/disclosure/ToolDisclosureServiceTest.java index bd343fb6..94fb91cb 100644 --- a/mateclaw-server/src/test/java/vip/mate/tool/disclosure/ToolDisclosureServiceTest.java +++ b/mateclaw-server/src/test/java/vip/mate/tool/disclosure/ToolDisclosureServiceTest.java @@ -148,11 +148,14 @@ class ToolDisclosureServiceTest { } @Test - @DisplayName("MCP tool whose server has no tier set defaults to core (visible)") - void mcpDefaultsCoreWhenServerTierUnset() { + @DisplayName("Move 5: MCP tool whose server has no tier set defaults to EXTENSION (on-demand)") + void mcpDefaultsExtensionWhenServerTierUnset() { + // Move 5: MCP tools default to EXTENSION so they don't flood the + // CORE tool list. Pre-Move-4 this returned CORE. var svc = service(List.of(), List.of(server(7L, "github", null)), List.of(mcpDto("mcp_github_create_issue", 7L))); - assertEquals(DisclosureTier.CORE, svc.resolveTierByName("mcp_github_create_issue")); + assertEquals(DisclosureTier.EXTENSION, svc.resolveTierByName("mcp_github_create_issue"), + "Move 5: MCP tools with no explicit tier must default to EXTENSION"); } @Test diff --git a/mateclaw-server/src/test/java/vip/mate/tool/document/GeneratedFileCacheFindByFilenameTest.java b/mateclaw-server/src/test/java/vip/mate/tool/document/GeneratedFileCacheFindByFilenameTest.java new file mode 100644 index 00000000..f0cc4136 --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/tool/document/GeneratedFileCacheFindByFilenameTest.java @@ -0,0 +1,71 @@ +package vip.mate.tool.document; + +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.nio.file.Path; +import java.util.Optional; + +import static org.junit.jupiter.api.Assertions.*; + +/** + * Pin {@link GeneratedFileCache#findIdByFilename}: recover a file's issued id + * from its logical filename, so a reference that points at the name instead of + * the id (which {@code GENERATED_URL_PATTERN} cannot parse) can still be + * resolved. Covers the in-memory hit, the disk fallback, the mime-prefix + * filter, case-insensitivity, and the misses. + */ +class GeneratedFileCacheFindByFilenameTest { + + private Path dir; + private GeneratedFileCache cache; + + @BeforeEach + void setUp(@TempDir Path tempDir) { + dir = tempDir; + cache = new GeneratedFileCache(tempDir); + } + + @Test + @DisplayName("in-memory: filename + image mime → the issued id") + void memoryHit() { + String id = cache.put("PNG".getBytes(), "cover_cat_7pits.png", "image/png"); + assertEquals(Optional.of(id), cache.findIdByFilename("cover_cat_7pits.png", "image/")); + } + + @Test + @DisplayName("filename match is case-insensitive") + void caseInsensitive() { + String id = cache.put("PNG".getBytes(), "Cover_Cat.PNG", "image/png"); + assertEquals(Optional.of(id), cache.findIdByFilename("cover_cat.png", "image/")); + } + + @Test + @DisplayName("mime prefix excludes a non-image with the same name") + void mimePrefixExcludesNonImage() { + cache.put("PDF".getBytes(), "cover.pdf", "application/pdf"); + assertTrue(cache.findIdByFilename("cover.pdf", "image/").isEmpty()); + // Without the constraint it is found. + assertTrue(cache.findIdByFilename("cover.pdf", null).isPresent()); + } + + @Test + @DisplayName("disk fallback: a fresh cache with empty memory finds it via persisted meta") + void diskFallback() { + String id = cache.put("PNG".getBytes(), "gzh-cover.png", "image/png"); + // A brand-new instance over the same dir has nothing in memory yet. + GeneratedFileCache reopened = new GeneratedFileCache(dir); + assertEquals(Optional.of(id), reopened.findIdByFilename("gzh-cover.png", "image/")); + } + + @Test + @DisplayName("unknown filename and null/blank input → empty") + void misses() { + cache.put("PNG".getBytes(), "cover.png", "image/png"); + assertTrue(cache.findIdByFilename("nope.png", "image/").isEmpty()); + assertTrue(cache.findIdByFilename(null, "image/").isEmpty()); + assertTrue(cache.findIdByFilename(" ", "image/").isEmpty()); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/tool/guard/WorkspacePathGuardSandboxTest.java b/mateclaw-server/src/test/java/vip/mate/tool/guard/WorkspacePathGuardSandboxTest.java index 89d6f526..0021d37d 100644 --- a/mateclaw-server/src/test/java/vip/mate/tool/guard/WorkspacePathGuardSandboxTest.java +++ b/mateclaw-server/src/test/java/vip/mate/tool/guard/WorkspacePathGuardSandboxTest.java @@ -87,6 +87,20 @@ class WorkspacePathGuardSandboxTest { WorkspacePathGuard.validatePath(DEFAULT_ROOT + "/notes/new-file.txt")); } + @Test + @DisplayName("validatePath: a relative path resolves into the default root, not the process CWD (issue #494)") + void validatePathRelative_resolvesIntoRoot() { + // A plain relative path must land inside the sandbox and return a + // path rooted there — not one resolved against the JVM launch dir. + java.nio.file.Path resolved = WorkspacePathGuard.validatePath("./report.html"); + org.junit.jupiter.api.Assertions.assertTrue( + resolved.startsWith(java.nio.file.Paths.get(DEFAULT_ROOT)), + "relative path should resolve inside the root, got: " + resolved); + // A relative traversal that climbs out is still rejected. + assertThrows(IllegalArgumentException.class, () -> + WorkspacePathGuard.validatePath("../escape.txt")); + } + @Test @DisplayName("Per-conversation workspace still takes precedence over the default root") void conversationWorkspace_wins() { diff --git a/mateclaw-server/src/test/java/vip/mate/tool/guard/WorkspacePathGuardShellTest.java b/mateclaw-server/src/test/java/vip/mate/tool/guard/WorkspacePathGuardShellTest.java index 02a58a73..ad82b35c 100644 --- a/mateclaw-server/src/test/java/vip/mate/tool/guard/WorkspacePathGuardShellTest.java +++ b/mateclaw-server/src/test/java/vip/mate/tool/guard/WorkspacePathGuardShellTest.java @@ -311,6 +311,41 @@ class WorkspacePathGuardShellTest { WorkspacePathGuard.validateShellCommand("echo evil > /tmp/leak.txt")); } + // ==================== Filesystem-root tokens ==================== + + @Test + @DisplayName("Destructive command targeting the filesystem root (`rm -rf //`, `/.`, `/..`) → rejected") + void destructiveFilesystemRoot_blocked() { + // "//", "/." and "/.." all normalize to "/" — a delete aimed there must + // not be skipped by the root-token false-positive allowance. + assertThrows(IllegalArgumentException.class, () -> + WorkspacePathGuard.validateShellCommand("rm -rf //")); + assertThrows(IllegalArgumentException.class, () -> + WorkspacePathGuard.validateShellCommand("rm -rf /.")); + assertThrows(IllegalArgumentException.class, () -> + WorkspacePathGuard.validateShellCommand("rm -rf /..")); + } + + @Test + @DisplayName("sed empty replacement (s/pattern//) is not misread as a filesystem-root path") + void sedEmptyReplacement_pass() { + // The trailing "// inside the sed script produces a token that + // normalizes to "/" — allowed because the command is non-destructive. + assertDoesNotThrow(() -> + WorkspacePathGuard.validateShellCommand("sed 's/\"text\": \"//' data.json")); + } + + @Test + @DisplayName("Compound command mixing a destructive verb with a root-normalizing token fails closed") + void destructiveMixedWithRootToken_blocked() { + // The destructive flag is command-wide by design: a compound command + // that both deletes and carries a "/"-normalizing token is refused, + // trading a rare false positive for never skipping `rm ... //`. + assertThrows(IllegalArgumentException.class, () -> + WorkspacePathGuard.validateShellCommand( + "rm -f old.log && sed 's/\"x\": \"//' data.json")); + } + // ==================== Device-node negative cases ==================== @Test diff --git a/mateclaw-server/src/test/java/vip/mate/tool/guard/guardian/WorkspaceBoundaryGuardianTest.java b/mateclaw-server/src/test/java/vip/mate/tool/guard/guardian/WorkspaceBoundaryGuardianTest.java index 42347e92..f86334dc 100644 --- a/mateclaw-server/src/test/java/vip/mate/tool/guard/guardian/WorkspaceBoundaryGuardianTest.java +++ b/mateclaw-server/src/test/java/vip/mate/tool/guard/guardian/WorkspaceBoundaryGuardianTest.java @@ -5,17 +5,23 @@ import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.condition.DisabledOnOs; import org.junit.jupiter.api.condition.OS; +import org.junit.jupiter.api.io.TempDir; import vip.mate.tool.guard.WorkspacePathGuard; import vip.mate.tool.guard.model.GuardDecision; import vip.mate.tool.guard.model.GuardFinding; import vip.mate.tool.guard.model.GuardSeverity; import vip.mate.tool.guard.model.ToolInvocationContext; +import vip.mate.workspace.core.service.ChatUploadLocationResolver; +import java.nio.file.Files; +import java.nio.file.Path; import java.util.List; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; /** * Verifies that {@link WorkspaceBoundaryGuardian} turns a workspace-boundary @@ -28,7 +34,7 @@ class WorkspaceBoundaryGuardianTest { private static final String WORKSPACE = "/tmp/ws-boundary-guardian-test"; private static final String DEFAULT_ROOT = "/tmp/ws-boundary-default-root"; - private final WorkspaceBoundaryGuardian guardian = new WorkspaceBoundaryGuardian(); + private final WorkspaceBoundaryGuardian guardian = new WorkspaceBoundaryGuardian(null); @AfterEach void teardown() { @@ -47,6 +53,12 @@ class WorkspaceBoundaryGuardianTest { .withWorkspaceBasePath(basePath); } + private ToolInvocationContext read(String path, String basePath) { + String args = "{\"filePath\":\"" + path + "\"}"; + return ToolInvocationContext.of("read_file", args, "conv", "agent") + .withWorkspaceBasePath(basePath); + } + private ToolInvocationContext code(String language, String src, String basePath) { String args = "{\"language\":\"" + language + "\",\"code\":\"" + src.replace("\"", "\\\"") + "\"}"; @@ -144,6 +156,28 @@ class WorkspaceBoundaryGuardianTest { assertTrue(guardian.evaluate(write(WORKSPACE + "/notes.txt", WORKSPACE)).isEmpty()); } + @Test + @DisplayName("write_file with a relative path resolves against the workspace, not the process CWD (issue #494)") + void writeRelativePath_pass() { + // Regression for #494: a plain relative path like "./foo.html" was + // resolved against the JVM launch directory via toAbsolutePath(), so it + // fell outside the workspace whenever the server ran from elsewhere and + // tripped a spurious CRITICAL "工作区越界" block. It must resolve into + // the workspace and pass. + assertTrue(guardian.evaluate(write("./fiber-signal-architecture.html", WORKSPACE)).isEmpty()); + assertTrue(guardian.evaluate(write("notes/todo.md", WORKSPACE)).isEmpty()); + assertTrue(guardian.evaluate(write("report.txt", WORKSPACE)).isEmpty()); + } + + @Test + @DisplayName("write_file with a relative traversal that climbs out is still blocked") + void writeRelativeTraversal_blocked() { + // The fix must not weaken the boundary: "../escape.txt" normalizes to a + // path outside the workspace root and stays blocked. + assertBlocked(guardian.evaluate(write("../escape.txt", WORKSPACE))); + assertBlocked(guardian.evaluate(write("../../etc/evil.conf", WORKSPACE))); + } + // ==================== Default-root fallback & escape hatch ==================== @Test @@ -172,6 +206,68 @@ class WorkspaceBoundaryGuardianTest { ToolInvocationContext.of("web_search", "{}", "conv", "agent"))); } + // ==================== Chat-upload fallback scope ==================== + + @Test + @DisplayName("A stored chat-upload path outside the workspace is allowed via the DB-backed fallback") + void chatUpload_storedPath_pass(@TempDir Path uploadRoot) throws Exception { + Path stored = Files.createDirectories(uploadRoot.resolve("conv")) + .resolve("1777391026594_secret.txt"); + Files.writeString(stored, "attachment"); + + ChatUploadLocationResolver resolver = mock(ChatUploadLocationResolver.class); + when(resolver.resolveCandidateUploadRoots("conv")).thenReturn(List.of(uploadRoot)); + WorkspaceBoundaryGuardian g = new WorkspaceBoundaryGuardian(resolver); + + // The upload root sits outside the workspace, so the boundary check + // trips first; the fallback must clear it for the real stored path. + assertTrue(g.evaluate(read(stored.toString(), WORKSPACE)).isEmpty()); + } + + @Test + @DisplayName("An outside path merely sharing a basename with an attachment stays blocked") + void chatUpload_basenameCollision_blocked(@TempDir Path uploadRoot, @TempDir Path elsewhere) + throws Exception { + // Store an attachment whose "{millis}_{safeName}" name would basename-match + // a request for "secret.txt" — the fallback must not let that clear a + // violation for a path pointing somewhere else entirely. + Path uploadDir = Files.createDirectories(uploadRoot.resolve("conv")); + Files.writeString(uploadDir.resolve("1777391026594_secret.txt"), "attachment"); + Path outside = elsewhere.resolve("secret.txt"); + Files.writeString(outside, "not an attachment"); + + ChatUploadLocationResolver resolver = mock(ChatUploadLocationResolver.class); + when(resolver.resolveCandidateUploadRoots("conv")).thenReturn(List.of(uploadRoot)); + WorkspaceBoundaryGuardian g = new WorkspaceBoundaryGuardian(resolver); + + assertBlocked(g.evaluate(read(outside.toString(), WORKSPACE))); + } + + @Test + @DisplayName("A path inside a different conversation's upload dir stays blocked") + void chatUpload_otherConversation_blocked(@TempDir Path uploadRoot) throws Exception { + Path otherConvFile = Files.createDirectories(uploadRoot.resolve("other-conv")) + .resolve("1777391026594_secret.txt"); + Files.writeString(otherConvFile, "someone else's attachment"); + + ChatUploadLocationResolver resolver = mock(ChatUploadLocationResolver.class); + when(resolver.resolveCandidateUploadRoots("conv")).thenReturn(List.of(uploadRoot)); + WorkspaceBoundaryGuardian g = new WorkspaceBoundaryGuardian(resolver); + + assertBlocked(g.evaluate(read(otherConvFile.toString(), WORKSPACE))); + } + + @Test + @DisplayName("Resolver failure keeps the BLOCK finding (fail closed)") + void chatUpload_resolverFailure_blocked() { + ChatUploadLocationResolver resolver = mock(ChatUploadLocationResolver.class); + when(resolver.resolveCandidateUploadRoots("conv")) + .thenThrow(new RuntimeException("db down")); + WorkspaceBoundaryGuardian g = new WorkspaceBoundaryGuardian(resolver); + + assertBlocked(g.evaluate(read("/somewhere/else/file.txt", WORKSPACE))); + } + // ==================== Tool-result spill dir stays reachable (issue #403) ==================== @Test diff --git a/mateclaw-server/src/test/java/vip/mate/tool/mcp/runtime/McpClientManagerProgressWrapTest.java b/mateclaw-server/src/test/java/vip/mate/tool/mcp/runtime/McpClientManagerProgressWrapTest.java new file mode 100644 index 00000000..494a24b6 --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/tool/mcp/runtime/McpClientManagerProgressWrapTest.java @@ -0,0 +1,110 @@ +package vip.mate.tool.mcp.runtime; + +import com.fasterxml.jackson.databind.ObjectMapper; +import io.modelcontextprotocol.client.McpSyncClient; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.springframework.ai.tool.ToolCallback; +import org.springframework.ai.tool.definition.DefaultToolDefinition; +import org.springframework.ai.tool.definition.ToolDefinition; +import org.springframework.ai.tool.metadata.ToolMetadata; +import org.springframework.ai.chat.model.ToolContext; + +import java.util.List; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.Mockito.mock; + +/** + * Black-box regression suite verifying that the new + * {@link McpClientManager#wrapServerCallbacks(long, ToolCallback[], McpIdentityForwardService, + * String, String, McpSyncClient, ObjectMapper)} overload: + *
    + *
  1. does not break the existing null-mcpClient path
    2. + *
  2. wraps with {@link ProgressAwareMcpToolCallback} when mcpClient is provided
    3. + *
+ */ +class McpClientManagerProgressWrapTest { + + private static ToolCallback stub(String name) { + ToolDefinition def = DefaultToolDefinition.builder() + .name(name).description("").inputSchema("{}").build(); + return new ToolCallback() { + @Override public ToolDefinition getToolDefinition() { return def; } + @Override public ToolMetadata getToolMetadata() { return ToolCallback.super.getToolMetadata(); } + @Override public String call(String toolInput) { return name + ":" + toolInput; } + @Override public String call(String toolInput, ToolContext toolContext) { return call(toolInput); } + }; + } + + // ── Backward-compat: null McpSyncClient (same as before) ── + + @Test + @DisplayName("null McpSyncClient → no ProgressAwareMcpToolCallback wrapping") + void nullMcpClientNoProgressWrap() { + ToolCallback cb = stub("search"); + List wrapped = McpClientManager.wrapServerCallbacks(99L, + new ToolCallback[]{cb}, null, null, null, null, null); + + assertEquals(1, wrapped.size()); + assertInstanceOf(PrefixedNameToolCallback.class, wrapped.get(0)); + PrefixedNameToolCallback p = (PrefixedNameToolCallback) wrapped.get(0); + // Inner should be the original stub, NOT a ProgressAwareMcpToolCallback + assertFalse(p.getDelegate() instanceof ProgressAwareMcpToolCallback, + "should NOT wrap when mcpClient is null"); + } + + // ── With McpSyncClient → wraps ── + + @Test + @DisplayName("non-null McpSyncClient wraps with ProgressAwareMcpToolCallback") + void withMcpClientWrapsProgress() { + McpSyncClient client = mock(McpSyncClient.class); + ToolCallback cb = stub("long_task"); + ObjectMapper mapper = new ObjectMapper(); + + List wrapped = McpClientManager.wrapServerCallbacks(88L, + new ToolCallback[]{cb}, null, null, null, client, mapper); + + assertEquals(1, wrapped.size()); + assertInstanceOf(PrefixedNameToolCallback.class, wrapped.get(0)); + PrefixedNameToolCallback p = (PrefixedNameToolCallback) wrapped.get(0); + assertInstanceOf(ProgressAwareMcpToolCallback.class, p.getDelegate(), + "should wrap with ProgressAwareMcpToolCallback when mcpClient is provided"); + ProgressAwareMcpToolCallback prog = (ProgressAwareMcpToolCallback) p.getDelegate(); + assertEquals(cb, prog.getDelegate(), "original callback preserved as delegate"); + } + + @Test + @DisplayName("ProgressAwareMcpToolCallback sits outside IdentityForward but inside PrefixedName (correct chain)") + void chainOrder() { + McpSyncClient client = mock(McpSyncClient.class); + ToolCallback cb = stub("private_data"); + ObjectMapper mapper = new ObjectMapper(); + + // identitySvc=null → no identity wrapping + List wrapped = McpClientManager.wrapServerCallbacks(77L, + new ToolCallback[]{cb}, null, null, "my-server", client, mapper); + + assertEquals(1, wrapped.size()); + assertInstanceOf(PrefixedNameToolCallback.class, wrapped.get(0)); + PrefixedNameToolCallback p = (PrefixedNameToolCallback) wrapped.get(0); + assertInstanceOf(ProgressAwareMcpToolCallback.class, p.getDelegate()); + + ProgressAwareMcpToolCallback prog = (ProgressAwareMcpToolCallback) p.getDelegate(); + // IdentityForwarding is NOT wrapped because identitySvc=null; the raw stub IS the delegate + assertSame(cb, prog.getDelegate()); + } + + @Test + @DisplayName("existing two-arg wrapServerCallbacks overload still compiles and works") + void twoArgOverloadStillWorks() { + ToolCallback cb = stub("read_file"); + // This is the original API used by McpClientManagerWrapTest — must still work + List wrapped = McpClientManager.wrapServerCallbacks(66L, + new ToolCallback[]{cb}); + + assertEquals(1, wrapped.size()); + assertInstanceOf(PrefixedNameToolCallback.class, wrapped.get(0)); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/tool/mcp/runtime/McpClientManagerSnapshotTest.java b/mateclaw-server/src/test/java/vip/mate/tool/mcp/runtime/McpClientManagerSnapshotTest.java index 96c127c2..28d57543 100644 --- a/mateclaw-server/src/test/java/vip/mate/tool/mcp/runtime/McpClientManagerSnapshotTest.java +++ b/mateclaw-server/src/test/java/vip/mate/tool/mcp/runtime/McpClientManagerSnapshotTest.java @@ -38,7 +38,8 @@ class McpClientManagerSnapshotTest { void staleListToolsServesSnapshotAndRequestsReconnect() throws Exception { ApplicationEventPublisher publisher = mock(ApplicationEventPublisher.class); McpClientManager manager = new McpClientManager(publisher, - new McpIdentityForwardService(new McpIdentityForwardProperties())); + new McpIdentityForwardService(new McpIdentityForwardProperties()), + null, null); // A client whose connection went stale: every listTools() throws. McpSyncClient deadClient = mock(McpSyncClient.class); diff --git a/mateclaw-server/src/test/java/vip/mate/tool/mcp/runtime/McpProgressContextTest.java b/mateclaw-server/src/test/java/vip/mate/tool/mcp/runtime/McpProgressContextTest.java new file mode 100644 index 00000000..d0d21f90 --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/tool/mcp/runtime/McpProgressContextTest.java @@ -0,0 +1,148 @@ +package vip.mate.tool.mcp.runtime; + +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.*; + +/** + * White-box unit tests for {@link McpProgressContext}. + * Covers register → lookup → remove lifecycle, snapshot persistence, + * multi-conversation isolation, and concurrent safety. + */ +class McpProgressContextTest { + + private McpProgressContext ctx() { + return new McpProgressContext(); + } + + // ── Token Map ── + + @Test + @DisplayName("register → lookup returns same entry") + void registerAndLookup() { + McpProgressContext ctx = ctx(); + var entry = new McpProgressContext.ProgressEntry("conv_1", "call_1", "search"); + ctx.register("token-1", entry); + assertSame(entry, ctx.lookup("token-1")); + } + + @Test + @DisplayName("lookup for unregistered token returns null") + void lookupMissingReturnsNull() { + assertNull(ctx().lookup("nonexistent")); + } + + @Test + @DisplayName("remove makes subsequent lookup return null") + void removeThenLookupReturnsNull() { + McpProgressContext ctx = ctx(); + ctx.register("tok", new McpProgressContext.ProgressEntry("c", "t", "n")); + ctx.remove("tok"); + assertNull(ctx.lookup("tok")); + } + + @Test + @DisplayName("register overwrites existing entry for same token") + void registerOverwrites() { + McpProgressContext ctx = ctx(); + var first = new McpProgressContext.ProgressEntry("c1", "t1", "n1"); + var second = new McpProgressContext.ProgressEntry("c2", "t2", "n2"); + ctx.register("tok", first); + ctx.register("tok", second); + assertSame(second, ctx.lookup("tok")); + } + + @Test + @DisplayName("remove of non-existent token is no-op") + void removeNonexistentIsNoop() { + McpProgressContext ctx = ctx(); + assertDoesNotThrow(() -> ctx.remove("ghost")); + } + + // ── Snapshot Map ── + + @Test + @DisplayName("updateSnapshot stores and getSnapshots returns latest") + void snapshotStoreAndRetrieve() { + McpProgressContext ctx = ctx(); + ctx.updateSnapshot("conv_1", "call_a", "{\"percent\":30}"); + ctx.updateSnapshot("conv_1", "call_a", "{\"percent\":70}"); + + var snapshots = ctx.getSnapshots("conv_1"); + assertEquals(1, snapshots.size()); + assertEquals("{\"percent\":70}", snapshots.get("call_a")); + } + + @Test + @DisplayName("getSnapshots for unknown conversation returns empty map") + void snapshotsForUnknownConversation() { + assertTrue(ctx().getSnapshots("no_such_conv").isEmpty()); + } + + @Test + @DisplayName("removeSnapshot cleans up individual tool snapshot") + void removeSnapshot() { + McpProgressContext ctx = ctx(); + ctx.updateSnapshot("conv_1", "call_a", "{\"p\":50}"); + ctx.updateSnapshot("conv_1", "call_b", "{\"p\":80}"); + ctx.removeSnapshot("conv_1", "call_a"); + + var snapshots = ctx.getSnapshots("conv_1"); + assertEquals(1, snapshots.size()); + assertNull(snapshots.get("call_a")); + assertEquals("{\"p\":80}", snapshots.get("call_b")); + } + + @Test + @DisplayName("removeSnapshot for unknown keys is no-op") + void removeSnapshotNoop() { + assertDoesNotThrow(() -> { + McpProgressContext ctx = ctx(); + ctx.removeSnapshot("no_conv", "no_call"); + ctx.updateSnapshot("cv", "cl", "{}"); + ctx.removeSnapshot("cv", "other"); + assertEquals(1, ctx.getSnapshots("cv").size()); + }); + } + + @Test + @DisplayName("getSnapshots returns immutable copy") + void snapshotsImmutable() { + McpProgressContext ctx = ctx(); + ctx.updateSnapshot("c", "t", "{}"); + var snap = ctx.getSnapshots("c"); + assertThrows(UnsupportedOperationException.class, () -> snap.put("x", "y")); + } + + @Test + @DisplayName("multiple conversations isolated") + void multiConversationIsolation() { + McpProgressContext ctx = ctx(); + ctx.updateSnapshot("c1", "t1", "A"); + ctx.updateSnapshot("c2", "t2", "B"); + + assertEquals(1, ctx.getSnapshots("c1").size()); + assertEquals(1, ctx.getSnapshots("c2").size()); + assertEquals("A", ctx.getSnapshots("c1").get("t1")); + assertEquals("B", ctx.getSnapshots("c2").get("t2")); + } + + @Test + @DisplayName("register + snapshot lifecycle: full round-trip") + void fullRoundtrip() { + McpProgressContext ctx = ctx(); + var entry = new McpProgressContext.ProgressEntry("conv_x", "call_x", "long_task"); + ctx.register("pt-1", entry); + assertEquals(entry, ctx.lookup("pt-1")); + + ctx.updateSnapshot("conv_x", "call_x", "{\"percent\":33}"); + ctx.updateSnapshot("conv_x", "call_x", "{\"percent\":99}"); + assertEquals("{\"percent\":99}", ctx.getSnapshots("conv_x").get("call_x")); + + ctx.remove("pt-1"); + ctx.removeSnapshot("conv_x", "call_x"); + assertNull(ctx.lookup("pt-1")); + assertTrue(ctx.getSnapshots("conv_x").isEmpty()); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/tool/mcp/runtime/McpProgressRelayTest.java b/mateclaw-server/src/test/java/vip/mate/tool/mcp/runtime/McpProgressRelayTest.java new file mode 100644 index 00000000..3c095228 --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/tool/mcp/runtime/McpProgressRelayTest.java @@ -0,0 +1,134 @@ +package vip.mate.tool.mcp.runtime; + +import com.fasterxml.jackson.databind.ObjectMapper; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import vip.mate.channel.web.ChatStreamTracker; + +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.ArgumentMatchers.*; +import static org.mockito.Mockito.*; + +/** + * Integration test for the {@link McpProgressRelay} event listener. + * Verifies that {@link McpProgressEvent} → {@link ChatStreamTracker#broadcastObject} + * forwarding works correctly, including snapshot updates and skipBuffer=true. + */ +class McpProgressRelayTest { + + private ChatStreamTracker streamTracker; + private McpProgressContext progressContext; + private McpProgressRelay relay; + + @BeforeEach + void setUp() { + streamTracker = mock(ChatStreamTracker.class); + progressContext = new McpProgressContext(); + relay = new McpProgressRelay(streamTracker, progressContext, new ObjectMapper()); + } + + @Test + @DisplayName("relay forwards event to ChatStreamTracker with skipBuffer=true") + void forwardsEvent() { + McpProgressEvent event = new McpProgressEvent( + this, "conv_1", "call_abc", "long_task", 0.5, 1.0, "Processing..."); + + relay.onMcpProgress(event); + + verify(streamTracker).broadcastObject( + eq("conv_1"), + eq(McpProgressRelay.EVENT_TOOL_PROGRESS), + any(Object.class), + eq(true)); + } + + @Test + @DisplayName("relay updates progress snapshot") + void updatesSnapshot() { + McpProgressEvent event = new McpProgressEvent( + this, "conv_1", "call_abc", "task", 0.75, 1.0, "Almost done"); + + relay.onMcpProgress(event); + + var snapshots = progressContext.getSnapshots("conv_1"); + assertEquals(1, snapshots.size()); + String json = snapshots.get("call_abc"); + assertNotNull(json); + assertTrue(json.contains("\"percent\":75")); + assertTrue(json.contains("\"call_abc\"")); + } + + @Test + @DisplayName("streamTracker throws → relay logs warning, does not propagate") + void streamTrackerThrowsDoesNotPropagate() { + doThrow(new RuntimeException("SSE dead")).when(streamTracker) + .broadcastObject(any(), any(), any(), anyBoolean()); + + McpProgressEvent event = new McpProgressEvent( + this, "conv", "call", "tool", 0.0, null, "init"); + + // Should not throw + assertDoesNotThrow(() -> relay.onMcpProgress(event)); + } + + @Test + @DisplayName("null progress → broadcast still succeeds with 0.0") + void nullProgress() { + // This would be an edge case from MCP SDK; not expected but guarded + McpProgressEvent event = new McpProgressEvent( + this, "conv_2", "call_2", "task", 0.0, null, null); + + relay.onMcpProgress(event); + + verify(streamTracker).broadcastObject( + eq("conv_2"), + eq(McpProgressRelay.EVENT_TOOL_PROGRESS), + any(Object.class), + eq(true)); + } + + @Test + @DisplayName("stage inference: 0-5% → prepare, 5-95% → execute, 95%+ → finalize") + void stageInference() { + // Test via the relay that stage reflects in the broadcast data + McpProgressEvent event = new McpProgressEvent( + this, "c", "t", "task", 0.97, 1.0, "Finishing"); + + relay.onMcpProgress(event); + + verify(streamTracker).broadcastObject( + eq("c"), eq("tool_call_progress"), + argThat((Object data) -> { + if (data instanceof Map m) { + return "finalize".equals(m.get("stage")); + } + return false; + }), + eq(true)); + } + + @Test + @DisplayName("event constant matches frontend expectation") + void eventConstantCorrect() { + assertEquals("tool_call_progress", McpProgressRelay.EVENT_TOOL_PROGRESS, + "must match the SSE event name used in useChat.ts and ChatStreamTracker"); + } + + @Test + @DisplayName("multiple events for same tool call update snapshot idempotently") + void multipleEventsUpdateSameSnapshot() { + relay.onMcpProgress(new McpProgressEvent(this, "c", "t", "n", 0.2, 1.0, "A")); + relay.onMcpProgress(new McpProgressEvent(this, "c", "t", "n", 0.6, 1.0, "B")); + relay.onMcpProgress(new McpProgressEvent(this, "c", "t", "n", 0.99, 1.0, "C")); + + // Only 1 snapshot (latest) + var snapshots = progressContext.getSnapshots("c"); + assertEquals(1, snapshots.size()); + String json = snapshots.get("t"); + assertTrue(json.contains("\"percent\":99")); + assertTrue(json.contains("C")); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/tool/mcp/runtime/ProgressAwareMcpToolCallbackTest.java b/mateclaw-server/src/test/java/vip/mate/tool/mcp/runtime/ProgressAwareMcpToolCallbackTest.java new file mode 100644 index 00000000..e9cdd98c --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/tool/mcp/runtime/ProgressAwareMcpToolCallbackTest.java @@ -0,0 +1,171 @@ +package vip.mate.tool.mcp.runtime; + +import com.fasterxml.jackson.databind.ObjectMapper; +import io.modelcontextprotocol.client.McpSyncClient; +import io.modelcontextprotocol.spec.McpSchema; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.springframework.ai.chat.model.ToolContext; +import org.springframework.ai.tool.ToolCallback; +import org.springframework.ai.tool.definition.DefaultToolDefinition; +import org.springframework.ai.tool.definition.ToolDefinition; +import org.springframework.ai.tool.metadata.ToolMetadata; + +import java.util.List; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.*; + +/** + * White-box tests for {@link ProgressAwareMcpToolCallback}. + * + *

Covers the two code paths: + *

    + *
  1. progressToken present in ToolContext → direct McpSyncClient.callTool() with injected meta
    2. + *
  2. progressToken absent → delegates to inner callback (backward-compatible)
    3. + *
+ */ +class ProgressAwareMcpToolCallbackTest { + + private static final ObjectMapper MAPPER = new ObjectMapper(); + + private ToolCallback delegate; + private McpSyncClient mcpClient; + private ProgressAwareMcpToolCallback wrapper; + + @BeforeEach + void setUp() { + delegate = mock(ToolCallback.class); + mcpClient = mock(McpSyncClient.class); + when(delegate.getToolDefinition()).thenReturn( + DefaultToolDefinition.builder().name("search").description("desc").inputSchema("{}").build()); + when(delegate.getToolMetadata()).thenReturn(ToolMetadata.builder().build()); + wrapper = new ProgressAwareMcpToolCallback(delegate, mcpClient, "search", MAPPER); + } + + @Test + @DisplayName("getToolDefinition delegates to inner callback") + void delegatesGetToolDefinition() { + assertEquals("search", wrapper.getToolDefinition().name()); + verify(delegate).getToolDefinition(); + } + + @Test + @DisplayName("getToolMetadata delegates to inner callback") + void delegatesGetToolMetadata() { + assertNotNull(wrapper.getToolMetadata()); + verify(delegate).getToolMetadata(); + } + + @Test + @DisplayName("call(toolInput) without ToolContext delegates to inner") + void callWithoutToolContextDelegates() { + when(delegate.call("{}")).thenReturn("result"); + assertEquals("result", wrapper.call("{}")); + verify(delegate).call("{}"); + verifyNoInteractions(mcpClient); + } + + @Test + @DisplayName("call with ToolContext but without progressToken delegates to inner") + void callWithoutProgressTokenDelegates() { + ToolContext ctx = new ToolContext(Map.of()); + when(delegate.call("{}", ctx)).thenReturn("delegated"); + assertEquals("delegated", wrapper.call("{}", ctx)); + verify(delegate).call("{}", ctx); + verifyNoInteractions(mcpClient); + } + + @Test + @DisplayName("call with null ToolContext delegates to inner") + void callWithNullToolContextDelegates() { + when(delegate.call("{}", null)).thenReturn("null_ctx"); + assertEquals("null_ctx", wrapper.call("{}", (ToolContext) null)); + verify(delegate).call("{}", (ToolContext) null); + verifyNoInteractions(mcpClient); + } + + @Test + @DisplayName("call with progressToken in ToolContext calls McpSyncClient directly with meta injected") + void callWithProgressTokenUsesMcpClient() { + ToolContext ctx = new ToolContext(Map.of( + ProgressAwareMcpToolCallback.MCP_PROGRESS_TOKEN_KEY, "pt-uuid-123")); + McpSchema.TextContent textContent = new McpSchema.TextContent("mcp result"); + McpSchema.CallToolResult result = new McpSchema.CallToolResult(List.of(textContent), false); + when(mcpClient.callTool(any())).thenReturn(result); + + String output = wrapper.call("{\"q\":\"hello\"}", ctx); + + assertEquals("mcp result", output); + verify(mcpClient).callTool(any(McpSchema.CallToolRequest.class)); + verify(delegate, never()).call(any(), any()); + } + + @Test + @DisplayName("progressToken present but blank → delegates (edge case)") + void blankProgressTokenDelegates() { + ToolContext ctx = new ToolContext(Map.of( + ProgressAwareMcpToolCallback.MCP_PROGRESS_TOKEN_KEY, " ")); + when(delegate.call("{}", ctx)).thenReturn("fallback"); + assertEquals("fallback", wrapper.call("{}", ctx)); + verify(delegate).call("{}", ctx); + verifyNoInteractions(mcpClient); + } + + @Test + @DisplayName("McpSyncClient throws → falls back to delegate") + void mcpClientThrowsFallsBackToDelegate() { + ToolContext ctx = new ToolContext(Map.of( + ProgressAwareMcpToolCallback.MCP_PROGRESS_TOKEN_KEY, "tok")); + when(mcpClient.callTool(any())).thenThrow(new RuntimeException("connection lost")); + when(delegate.call(eq("{}"), any(ToolContext.class))).thenReturn("fallback result"); + + String output = wrapper.call("{}", ctx); + + assertEquals("fallback result", output); + verify(mcpClient).callTool(any()); + verify(delegate).call(eq("{}"), any()); + } + + @Test + @DisplayName("callTool succeeds with multi-text content concatenated") + void multiTextContentConcatenated() { + ToolContext ctx = new ToolContext(Map.of( + ProgressAwareMcpToolCallback.MCP_PROGRESS_TOKEN_KEY, "tok")); + McpSchema.CallToolResult result = new McpSchema.CallToolResult(List.of( + new McpSchema.TextContent("part1"), + new McpSchema.TextContent("part2")), false); + when(mcpClient.callTool(any())).thenReturn(result); + + assertEquals("part1part2", wrapper.call("{}", ctx)); + } + + @Test + @DisplayName("getDelegate returns inner callback (for ReturnDirect / IdentityForward detection)") + void getDelegateReturnsInner() { + assertSame(delegate, wrapper.getDelegate()); + } + + @Test + @DisplayName("parseArguments handles null input") + void parseArgumentsHandlesNull() { + ToolContext ctx = new ToolContext(Map.of( + ProgressAwareMcpToolCallback.MCP_PROGRESS_TOKEN_KEY, "tok")); + when(mcpClient.callTool(any())).thenReturn( + new McpSchema.CallToolResult(List.of(new McpSchema.TextContent("ok")), false)); + assertEquals("ok", wrapper.call(null, ctx)); + } + + @Test + @DisplayName("parseArguments handles blank input") + void parseArgumentsHandlesBlank() { + ToolContext ctx = new ToolContext(Map.of( + ProgressAwareMcpToolCallback.MCP_PROGRESS_TOKEN_KEY, "tok")); + when(mcpClient.callTool(any())).thenReturn( + new McpSchema.CallToolResult(List.of(new McpSchema.TextContent("ok")), false)); + assertEquals("ok", wrapper.call(" ", ctx)); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/wiki/service/WikiLinkServiceCrossKbTest.java b/mateclaw-server/src/test/java/vip/mate/wiki/service/WikiLinkServiceCrossKbTest.java new file mode 100644 index 00000000..11298bda --- /dev/null +++ b/mateclaw-server/src/test/java/vip/mate/wiki/service/WikiLinkServiceCrossKbTest.java @@ -0,0 +1,64 @@ +package vip.mate.wiki.service; + +import com.fasterxml.jackson.databind.ObjectMapper; +import org.junit.jupiter.api.Test; + +import java.util.List; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Coverage for cross-KB wikilink targets ({@code [[kbId/slug]]}): parsing and + * the broken-link exemption. A cross-KB target must never be flagged broken by + * the single-KB lint (existence can only be checked in the target KB), while a + * plain single-KB target keeps its exact slug/title resolution. + */ +class WikiLinkServiceCrossKbTest { + + private final WikiLinkService svc = new WikiLinkService(new ObjectMapper()); + + @Test + void parseCrossKb_recognisesNumericPrefix() { + WikiLinkService.CrossKbRef ref = svc.parseCrossKb("2055137662148763649/photosynthesis"); + assertNotNull(ref); + assertEquals(2055137662148763649L, ref.kbId()); + assertEquals("photosynthesis", ref.slug()); + } + + @Test + void parseCrossKb_ignoresPlainSlugAndTitle() { + assertNull(svc.parseCrossKb("photosynthesis")); + assertNull(svc.parseCrossKb("Energy Metabolism")); + // Slug-shaped but non-numeric prefix stays single-KB. + assertNull(svc.parseCrossKb("chapter/section")); + // Numeric prefix but empty slug is not a valid cross-KB ref. + assertNull(svc.parseCrossKb("123/")); + assertNull(svc.parseCrossKb(null)); + } + + @Test + void computeBrokenLinks_exemptsCrossKbTargets() { + Set outlinks = svc.extractOutlinks( + "See [[123/photosynthesis]] and [[missing-local]] and [[known-local]]."); + // Only this KB's own slug is resolvable. + Set resolvable = Set.of("known-local"); + List broken = svc.computeBrokenLinks(outlinks, resolvable); + // Cross-KB target exempt; local unknown flagged; local known resolves. + assertTrue(broken.contains("missing-local")); + assertFalse(broken.contains("123/photosynthesis"), + "cross-KB target must not be flagged broken by the single-KB lint"); + assertFalse(broken.contains("known-local")); + } + + @Test + void extractOutlinks_keepsCrossKbTargetVerbatim() { + Set outlinks = svc.extractOutlinks("Ref [[456/Some-Page|display]]."); + // Lowercased, alias stripped, prefix preserved. + assertTrue(outlinks.contains("456/some-page")); + } +} diff --git a/mateclaw-server/src/test/java/vip/mate/workspace/conversation/ConversationServiceCleanAttachmentFilesTest.java b/mateclaw-server/src/test/java/vip/mate/workspace/conversation/ConversationServiceCleanAttachmentFilesTest.java index 992a115f..852a42f3 100644 --- a/mateclaw-server/src/test/java/vip/mate/workspace/conversation/ConversationServiceCleanAttachmentFilesTest.java +++ b/mateclaw-server/src/test/java/vip/mate/workspace/conversation/ConversationServiceCleanAttachmentFilesTest.java @@ -17,7 +17,6 @@ import vip.mate.workspace.core.service.ChatUploadLocationResolver; import java.io.IOException; import java.nio.file.Files; -import java.nio.file.InvalidPathException; import java.nio.file.Path; import java.nio.file.Paths; import java.util.Comparator; @@ -27,27 +26,20 @@ import java.util.stream.Stream; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatCode; -import static org.assertj.core.api.Assertions.assertThatThrownBy; import static org.mockito.ArgumentMatchers.any; import static org.mockito.Mockito.when; /** - * Regression coverage for issue #36: deleting a CRON-task conversation throws - * {@code InvalidPathException} on Windows because the conversation id - * ("cron:<jobId>") contains a colon, which is illegal in Windows path - * segments. The exception bubbles out of the {@code @Transactional} - * {@code deleteConversation}, rolling back the row deletes and leaving the - * user unable to remove the entry. + * Regression coverage for issues #36 / #507: a conversation id like + * {@code cron:} or {@code wecom:} carries a colon, which is illegal + * in a Windows path segment. The old behaviour caught the resulting + * {@code InvalidPathException} and silently skipped cleanup; the fix instead + * sanitizes the id to a filesystem-safe segment so the attachment directory is + * both written and cleaned consistently on every OS. */ @ExtendWith(MockitoExtension.class) class ConversationServiceCleanAttachmentFilesTest { - /** NUL byte: rejected by Paths.get on every OS, so it portably triggers - * the same InvalidPathException branch the colon hits on Windows. Built - * via String.valueOf((char) 0) so the source text contains no embedded - * NUL (which would be invisible in diff tools). */ - private static final String UNREPRESENTABLE_ID = "bad" + (char) 0 + "id"; - @Mock private ConversationMapper conversationMapper; @Mock private MessageMapper messageMapper; @Mock private AgentMapper agentMapper; @@ -60,11 +52,11 @@ class ConversationServiceCleanAttachmentFilesTest { @BeforeEach void stubResolver() { - // cleanAttachmentFiles now resolves the upload root via the resolver. - // Point its candidate roots at the legacy default dir so both the - // happy-path and unrepresentable-id cases exercise the real filesystem. - when(chatUploadLocationResolver.resolveCandidateUploadRoots(any())) - .thenReturn(List.of(Paths.get("data", "chat-uploads"))); + // cleanAttachmentFiles now walks sanitized conversation dirs. Mirror the + // real resolver: {legacy-default-root}/{sanitizeSegment(id)}. + when(chatUploadLocationResolver.resolveCandidateConversationDirs(any())) + .thenAnswer(inv -> List.of(Paths.get("data", "chat-uploads") + .resolve(ChatUploadLocationResolver.sanitizeSegment(inv.getArgument(0))))); } @AfterEach @@ -79,28 +71,25 @@ class ConversationServiceCleanAttachmentFilesTest { } @Test - @DisplayName("conversation id that yields an unrepresentable path is skipped, not thrown") - void unrepresentablePathIdIsSkipped() { - // Precondition: confirm the id really does break Paths.resolve on - // this JDK / OS. If a future JDK ever accepts the NUL byte, the - // service-level assertion below would silently pass without - // exercising the catch branch we are guarding — fail loudly here - // instead. - assertThatThrownBy(() -> Paths.get("data", "chat-uploads").resolve(UNREPRESENTABLE_ID)) - .isInstanceOf(InvalidPathException.class); + @DisplayName("colon-bearing id (cron:jobId / wecom:xxx) is sanitized and its dir is cleaned, not skipped") + void colonIdIsSanitizedAndCleaned() throws IOException { + // The ':' would throw InvalidPathException as a raw Windows path segment; + // the service must sanitize it and clean the sanitized dir — never throw, + // never silently skip (the pre-fix bug from issue #36). + String convId = "cron:job-" + UUID.randomUUID(); + Path uploadRoot = Paths.get("data", "chat-uploads"); + createdDir = uploadRoot.resolve(ChatUploadLocationResolver.sanitizeSegment(convId)); + Files.createDirectories(createdDir); + Files.writeString(createdDir.resolve("a.txt"), "hello"); - // Without the catch in cleanAttachmentFiles, this would propagate - // InvalidPathException out of the @Transactional deleteConversation, - // rolling back the row deletes — the user-visible bug from issue #36. - assertThatCode(() -> service.cleanAttachmentFiles(UNREPRESENTABLE_ID)) - .doesNotThrowAnyException(); + assertThatCode(() -> service.cleanAttachmentFiles(convId)).doesNotThrowAnyException(); + assertThat(Files.exists(createdDir)).isFalse(); } @Test @DisplayName("legal id with a real attachment dir is still cleaned (happy path)") void legalIdHappyPathStillCleans() throws IOException { - // UUID-shaped id matches what the web channel actually uses, so the - // resolve() succeeds on every OS and the walk/delete loop runs. + // UUID-shaped id matches what the web channel uses; sanitize is a no-op. String convId = "test-" + UUID.randomUUID(); Path uploadRoot = Paths.get("data", "chat-uploads"); createdDir = uploadRoot.resolve(convId); diff --git a/mateclaw-server/src/test/java/vip/mate/workspace/core/service/ChatUploadLocationResolverTest.java b/mateclaw-server/src/test/java/vip/mate/workspace/core/service/ChatUploadLocationResolverTest.java index 900f7078..214fb224 100644 --- a/mateclaw-server/src/test/java/vip/mate/workspace/core/service/ChatUploadLocationResolverTest.java +++ b/mateclaw-server/src/test/java/vip/mate/workspace/core/service/ChatUploadLocationResolverTest.java @@ -200,4 +200,68 @@ class ChatUploadLocationResolverTest { assertThat(root).isEqualTo(tempDir.toAbsolutePath().normalize()); } + + // ==================== conversationId path safety (issue #507) ==================== + + @Test + @DisplayName("sanitizeSegment: colon (and other unsafe chars) → underscore; safe ids unchanged") + void sanitizeSegmentReplacesUnsafeChars() { + // IM-channel id with the ':' that breaks Windows paths. + assertThat(ChatUploadLocationResolver.sanitizeSegment("wecom:XuZhanFu")) + .isEqualTo("wecom_XuZhanFu"); + // Separator-laden id is fully flattened to a single safe segment. + assertThat(ChatUploadLocationResolver.sanitizeSegment("a/b\\c:d*e?")) + .isEqualTo("a_b_c_d_e_"); + // Already-safe ids (web / webchat / numeric) are a no-op. + assertThat(ChatUploadLocationResolver.sanitizeSegment("2055137662148763649")) + .isEqualTo("2055137662148763649"); + assertThat(ChatUploadLocationResolver.sanitizeSegment("conv-abc_1.2")) + .isEqualTo("conv-abc_1.2"); + assertThat(ChatUploadLocationResolver.sanitizeSegment(null)).isEmpty(); + } + + @Test + @DisplayName("resolveConversationDir: colon id lands under the sanitized segment (no InvalidPathException)") + void resolveConversationDirUsesSanitizedSegment() { + stubConversation("wecom:XuZhanFu", null, null); + when(workspaceService.getById(anyLong())).thenReturn(workspace(1L, null)); + + ChatUploadLocationResolver r = resolver(tempDir); + Path dir = r.resolveConversationDir("wecom:XuZhanFu"); + + assertThat(dir).isEqualTo(tempDir.toAbsolutePath().normalize().resolve("wecom_XuZhanFu")); + } + + @Test + @DisplayName("candidate conversation dirs: sanitized first, then raw id for backward compat") + void candidateConversationDirsIncludeSanitizedAndRaw() { + stubConversation("wecom:XuZhanFu", null, null); + when(workspaceService.getById(anyLong())).thenReturn(workspace(1L, null)); + + ChatUploadLocationResolver r = resolver(tempDir); + List dirs = r.resolveCandidateConversationDirs("wecom:XuZhanFu"); + Path base = tempDir.toAbsolutePath().normalize(); + + int sanitizedIdx = dirs.indexOf(base.resolve("wecom_XuZhanFu")); + assertThat(sanitizedIdx).isGreaterThanOrEqualTo(0); + // On a POSIX filesystem the raw ':' dir is a legal (legacy) candidate, + // ordered after the sanitized one. + boolean posix = !System.getProperty("os.name").toLowerCase().contains("win"); + if (posix) { + int rawIdx = dirs.indexOf(base.resolve("wecom:XuZhanFu")); + assertThat(rawIdx).isGreaterThan(sanitizedIdx); + } + } + + @Test + @DisplayName("candidate conversation dirs: safe id yields a single dir (no duplicate raw)") + void candidateConversationDirsNoDuplicateForSafeId() { + stubConversation("plainconv", null, null); + when(workspaceService.getById(anyLong())).thenReturn(workspace(1L, null)); + + ChatUploadLocationResolver r = resolver(tempDir); + List dirs = r.resolveCandidateConversationDirs("plainconv"); + + assertThat(dirs).containsExactly(tempDir.toAbsolutePath().normalize().resolve("plainconv")); + } } diff --git a/mateclaw-server/src/test/resources/test-bundles/sample/scripts/run.sh b/mateclaw-server/src/test/resources/test-bundles/sample/scripts/run.sh new file mode 100644 index 00000000..ba3306c4 --- /dev/null +++ b/mateclaw-server/src/test/resources/test-bundles/sample/scripts/run.sh @@ -0,0 +1,3 @@ +#!/bin/sh +# Test fixture supporting file for SkillBundleMaterializerTest. +echo "hello from sample bundle" diff --git a/mateclaw-ui/package.json b/mateclaw-ui/package.json index 939af2a1..c4b3aea1 100644 --- a/mateclaw-ui/package.json +++ b/mateclaw-ui/package.json @@ -1,6 +1,6 @@ { "name": "mateclaw-ui", - "version": "1.7.0", + "version": "1.8.0", "private": true, "type": "module", "description": "MateClaw - Personal AI Assistant Web Console", @@ -55,6 +55,7 @@ "rollup-plugin-visualizer": "^7.0.1", "tailwindcss": "^4.0.6", "typescript": "~5.7.2", + "unplugin-vue-components": "^32.1.0", "vite": "^7.3.1", "vitest": "^4.1.7", "vue-tsc": "^3.2.6" diff --git a/mateclaw-ui/pnpm-lock.yaml b/mateclaw-ui/pnpm-lock.yaml index dbf6e128..974a5750 100644 --- a/mateclaw-ui/pnpm-lock.yaml +++ b/mateclaw-ui/pnpm-lock.yaml @@ -126,6 +126,9 @@ importers: typescript: specifier: ~5.7.2 version: 5.7.3 + unplugin-vue-components: + specifier: ^32.1.0 + version: 32.1.0(esbuild@0.27.5)(rollup@4.60.1)(vite@7.3.1(@types/node@25.9.1)(jiti@2.6.1)(lightningcss@1.32.0))(vue@3.5.31(typescript@5.7.3)) vite: specifier: ^7.3.1 version: 7.3.1(@types/node@25.9.1)(jiti@2.6.1)(lightningcss@1.32.0) @@ -1096,6 +1099,10 @@ packages: resolution: {integrity: sha512-csJvb+6kEiQaqo1woTdSAuOWdN0WTLIydkKrBnS+V5gZz0oqBrp4kQ35519QgK6TpBThiG3V1vNSHlIkv4AglQ==} engines: {node: '>=22.0.0'} + chokidar@5.0.0: + resolution: {integrity: sha512-TQMmc3w+5AxjpL8iIiwebF73dRDF4fBIieAqGn9RGCWaEVwQ6Fb2cGe31Yns0RRIzii5goJ1Y7xbMwo1TxMplw==} + engines: {node: '>= 20.19.0'} + cliui@9.0.1: resolution: {integrity: sha512-k7ndgKhwoQveBL+/1tqGJYNz097I7WOvwbmmU2AR5+magtbjPWQTS1C5vzGkBC8Ym8UWRzfKUzUUqFLypY4Q+w==} engines: {node: '>=20'} @@ -1125,6 +1132,9 @@ packages: confbox@0.1.8: resolution: {integrity: sha512-RMtmw0iFkeR4YV+fUOSucriAQNb9g8zFR52MWCtl+cCZOFRNL6zeB395vPzFhEjjn4fMxXudmELnl/KF/WrK6w==} + confbox@0.2.4: + resolution: {integrity: sha512-ysOGlgTFbN2/Y6Cg3Iye8YKulHw+R2fNXHrgSmXISQdMnomY6eNDprVdW9R5xBguEqI954+S6709UyiO7B+6OQ==} + convert-source-map@2.0.0: resolution: {integrity: sha512-Kvp459HrV2FEJ1CAsi1Ku+MY3kasH19TFykTz2xWmMeq6bk2NU3XXvfJ+Q61m0xktWwt+1HSYf3JZsTms3aRJg==} @@ -1477,6 +1487,9 @@ packages: resolution: {integrity: sha512-knvyeauYhqjOYvQ66MznSMs83wmHrCycNEN6Ao+2AeYEfxUIkuiVxdEa1qlGEPK+We3n0THiDciYSsCcgW/DoA==} engines: {node: '>=12.0.0'} + exsolve@1.1.0: + resolution: {integrity: sha512-D+42+T12DdIlJM3uepa55qGiL3sYdLBOxIl2ifQCzCHz4c7eiolaHsi3BIqEr7JxBzxv2pYZQX9kw16ziMcEmw==} + fast-deep-equal@3.1.3: resolution: {integrity: sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==} @@ -1791,6 +1804,10 @@ packages: lit@3.3.2: resolution: {integrity: sha512-NF9zbsP79l4ao2SNrH3NkfmFgN/hBYSQo90saIVI1o5GpjAdCPVstVzO1MrLOakHoEhYkrtRjPK6Ob521aoYWQ==} + local-pkg@1.2.1: + resolution: {integrity: sha512-++gUqRDEvcnN6Zhqrr+y/CkVEHhlrR96vZn3nZZPYzMcBUyBtTKzB9NadClFIsIVSsu+3i9tfk/erqy9kAmt7Q==} + engines: {node: '>=14'} + locate-path@6.0.0: resolution: {integrity: sha512-iPZK6eYjbxRu3uB4/WZ3EsEIMJFMqAoopl3R+zuq0UjcAm/MO6KCweDgPfP3elTztoKP3KtnVHxTn2NHBSDVUw==} engines: {node: '>=10'} @@ -1958,6 +1975,9 @@ packages: pkg-types@1.3.1: resolution: {integrity: sha512-/Jm5M4RvtBFVkKWRu2BLUTNP8/M2a+UwuAX+ae4770q1qVGtfjG+WTCupoZixokjmHiry8uI+dlY8KXYV5HVVQ==} + pkg-types@2.3.1: + resolution: {integrity: sha512-y+ichcgc2LrADuhLNAx8DFjVfgz91pRxfZdI3UDhxHvcVEZsenLO+7XaU5vOp0u/7V/wZ+plyuQxtrDlZJ+yeg==} + points-on-curve@0.2.0: resolution: {integrity: sha512-0mYKnYYe9ZcqMCWhUjItv/oHjvgEsfKvnUTg8sAtnHr3GVy7rGkXCb6d5cSyqrWqL4k81b9CPg3urd+T7aop3A==} @@ -1994,10 +2014,17 @@ packages: resolution: {integrity: sha512-vYt7UD1U9Wg6138shLtLOvdAu+8DsC/ilFtEVHcH+wydcSpNE20AfSOduf6MkRFahL5FY7X1oU7nKVZFtfq8Fg==} engines: {node: '>=6'} + quansync@0.2.11: + resolution: {integrity: sha512-AifT7QEbW9Nri4tAwR5M/uzpBuqfZf+zwaEM/QkzEjj7NBuFD2rBuy0K3dE+8wltbezDV7JMA0WfnCPYRSYbXA==} + react@19.2.5: resolution: {integrity: sha512-llUJLzz1zTUBrskt2pwZgLq59AemifIftw4aB7JxOqf1HY2FDaGDxgwpAPVzHU1kdWabH7FauP4i1oEeer2WCA==} engines: {node: '>=0.10.0'} + readdirp@5.0.0: + resolution: {integrity: sha512-9u/XQ1pvrQtYyMpZe7DXKv2p5CNvyVwzUB6uhLAnQwHMSgKMBR62lc7AHljaeteeHXn11XTAaLLUVZYVZyuRBQ==} + engines: {node: '>= 20.19.0'} + resolve-from@4.0.0: resolution: {integrity: sha512-pb/MYmXstAkysRFx8piNI1tGFNQIFA3vkE3Gq4EuA1dF6gHp/+vgZqsCGJapvy8N3Q+4o7FwvquPJcnZ7RYy4g==} engines: {node: '>=4'} @@ -2120,6 +2147,10 @@ packages: resolution: {integrity: sha512-j2Zq4NyQYG5XMST4cbs02Ak8iJUdxRM0XI5QyxXuZOzKOINmWurp3smXu3y5wDcJrptwpSjgXHzIQxR0omXljQ==} engines: {node: '>=12.0.0'} + tinyglobby@0.2.17: + resolution: {integrity: sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g==} + engines: {node: '>=12.0.0'} + tinyrainbow@3.1.0: resolution: {integrity: sha512-Bf+ILmBgretUrdJxzXM0SgXLZ3XfiaUuOj/IKQHuTXip+05Xn+uyEYdVg0kYDipTBcLrCVyUzAPz7QmArb0mmw==} engines: {node: '>=14.0.0'} @@ -2150,6 +2181,53 @@ packages: undici-types@7.24.6: resolution: {integrity: sha512-WRNW+sJgj5OBN4/0JpHFqtqzhpbnV0GuB+OozA9gCL7a993SmU+1JBZCzLNxYsbMfIeDL+lTsphD5jN5N+n0zg==} + unplugin-utils@0.3.2: + resolution: {integrity: sha512-xVToRh2CTmLk2HnEG7ac4rl1MJTT3RFkpS8B++/SnB0kXvuaavD+n3m/vrzyWQOdJNSZQACnbz01pnppbwV5BA==} + engines: {node: '>=20.19.0'} + + unplugin-vue-components@32.1.0: + resolution: {integrity: sha512-YiUkSxuRjab18XFOrX5VsIxXzccrfmHVGsGeJgSgklb829DQmCy9E4vvDUE4tuvZZdxyFJZX0Oc4TPnnxiiMyg==} + engines: {node: '>=20.19.0'} + peerDependencies: + '@nuxt/kit': ^3.2.2 || ^4.0.0 + vue: ^3.0.0 + peerDependenciesMeta: + '@nuxt/kit': + optional: true + + unplugin@3.3.0: + resolution: {integrity: sha512-qa66K+crbfyE6JK10GjvbJeRrOsuC/JpbnHctfyp/i4oBTxWOzJfRZyDiOk1PtErMFRu8JhsU/wPvOdBNWe5Rg==} + engines: {node: ^20.19.0 || >=22.12.0} + peerDependencies: + '@farmfe/core': '*' + '@rspack/core': '*' + bun-types-no-globals: '*' + esbuild: '*' + rolldown: '*' + rollup: '*' + unloader: '*' + vite: '*' + webpack: '*' + peerDependenciesMeta: + '@farmfe/core': + optional: true + '@rspack/core': + optional: true + bun-types-no-globals: + optional: true + esbuild: + optional: true + rolldown: + optional: true + rollup: + optional: true + unloader: + optional: true + vite: + optional: true + webpack: + optional: true + update-browserslist-db@1.2.3: resolution: {integrity: sha512-Js0m9cx+qOgDxo0eMiFGEueWztz+d4+M3rGlmKPT+T4IS/jP4ylw3Nwpu6cpTTP8R1MAC1kF4VbdLt3ARf209w==} hasBin: true @@ -2312,6 +2390,9 @@ packages: typescript: optional: true + webpack-virtual-modules@0.6.2: + resolution: {integrity: sha512-66/V2i5hQanC51vBQKPH4aI8NMAcBW59FVBs+rC7eGHupMyfn34q7rZIE+ETlJ+XTevqfUhVVBgSUNSW2flEUQ==} + whatwg-mimetype@3.0.0: resolution: {integrity: sha512-nt+N2dzIutVRxARx1nghPKGv1xHikU7HKdfafKkLNLindmPU/ch3U31NOCGGA/dmPcmb1VlofO0vnKAcsm0o/Q==} engines: {node: '>=12'} @@ -3255,6 +3336,10 @@ snapshots: '@chevrotain/types': 12.0.0 '@chevrotain/utils': 12.0.0 + chokidar@5.0.0: + dependencies: + readdirp: 5.0.0 + cliui@9.0.1: dependencies: string-width: 7.2.0 @@ -3279,6 +3364,8 @@ snapshots: confbox@0.1.8: {} + confbox@0.2.4: {} + convert-source-map@2.0.0: {} copy-anything@4.0.5: @@ -3719,6 +3806,8 @@ snapshots: expect-type@1.3.0: {} + exsolve@1.1.0: {} + fast-deep-equal@3.1.3: {} fast-json-stable-stringify@2.1.0: {} @@ -3984,6 +4073,12 @@ snapshots: lit-element: 4.2.2 lit-html: 3.3.2 + local-pkg@1.2.1: + dependencies: + mlly: 1.8.2 + pkg-types: 2.3.1 + quansync: 0.2.11 + locate-path@6.0.0: dependencies: p-locate: 5.0.0 @@ -4149,6 +4244,12 @@ snapshots: mlly: 1.8.2 pathe: 2.0.3 + pkg-types@2.3.1: + dependencies: + confbox: 0.2.4 + exsolve: 1.1.0 + pathe: 2.0.3 + points-on-curve@0.2.0: {} points-on-path@0.2.1: @@ -4182,8 +4283,12 @@ snapshots: punycode@2.3.1: {} + quansync@0.2.11: {} + react@19.2.5: {} + readdirp@5.0.0: {} + resolve-from@4.0.0: {} rfdc@1.4.1: {} @@ -4302,6 +4407,11 @@ snapshots: fdir: 6.5.0(picomatch@4.0.4) picomatch: 4.0.4 + tinyglobby@0.2.17: + dependencies: + fdir: 6.5.0(picomatch@4.0.4) + picomatch: 4.0.4 + tinyrainbow@3.1.0: {} ts-dedent@2.2.0: {} @@ -4320,6 +4430,44 @@ snapshots: undici-types@7.24.6: {} + unplugin-utils@0.3.2: + dependencies: + pathe: 2.0.3 + picomatch: 4.0.4 + + unplugin-vue-components@32.1.0(esbuild@0.27.5)(rollup@4.60.1)(vite@7.3.1(@types/node@25.9.1)(jiti@2.6.1)(lightningcss@1.32.0))(vue@3.5.31(typescript@5.7.3)): + dependencies: + chokidar: 5.0.0 + local-pkg: 1.2.1 + magic-string: 0.30.21 + mlly: 1.8.2 + obug: 2.1.1 + picomatch: 4.0.4 + tinyglobby: 0.2.17 + unplugin: 3.3.0(esbuild@0.27.5)(rollup@4.60.1)(vite@7.3.1(@types/node@25.9.1)(jiti@2.6.1)(lightningcss@1.32.0)) + unplugin-utils: 0.3.2 + vue: 3.5.31(typescript@5.7.3) + transitivePeerDependencies: + - '@farmfe/core' + - '@rspack/core' + - bun-types-no-globals + - esbuild + - rolldown + - rollup + - unloader + - vite + - webpack + + unplugin@3.3.0(esbuild@0.27.5)(rollup@4.60.1)(vite@7.3.1(@types/node@25.9.1)(jiti@2.6.1)(lightningcss@1.32.0)): + dependencies: + '@jridgewell/remapping': 2.3.5 + picomatch: 4.0.4 + webpack-virtual-modules: 0.6.2 + optionalDependencies: + esbuild: 0.27.5 + rollup: 4.60.1 + vite: 7.3.1(@types/node@25.9.1)(jiti@2.6.1)(lightningcss@1.32.0) + update-browserslist-db@1.2.3(browserslist@4.28.2): dependencies: browserslist: 4.28.2 @@ -4440,6 +4588,8 @@ snapshots: optionalDependencies: typescript: 5.7.3 + webpack-virtual-modules@0.6.2: {} + whatwg-mimetype@3.0.0: {} which@2.0.2: diff --git a/mateclaw-ui/src/api/index.ts b/mateclaw-ui/src/api/index.ts index 5321165d..62f6e7fa 100644 --- a/mateclaw-ui/src/api/index.ts +++ b/mateclaw-ui/src/api/index.ts @@ -187,6 +187,13 @@ export const chatApi = { } // ==================== Conversation ==================== +// Content calendar (read-only) — produced 公众号 / 小红书 pieces + lifecycle status. +export const contentItemApi = { + list: (params?: { page?: number; size?: number; platform?: string; status?: string }) => + http.get('/content-items', { params }), + summary: () => http.get('/content-items/summary'), +} + export const conversationApi = { list: () => http.get('/conversations'), /** @@ -824,7 +831,10 @@ export const wikiApi = { listFailures: (limit = 100) => http.get<{ data: WikiFailureItem[] }>(`/wiki/admin/failures?limit=${limit}`), // Raw Materials - listRaw: (kbId: number) => http.get(`/wiki/knowledge-bases/${kbId}/raw`), + listRaw: ( + kbId: number, + filters?: { status?: string; sourceType?: string; keyword?: string; startTime?: string; endTime?: string }, + ) => http.get(`/wiki/knowledge-bases/${kbId}/raw`, filters ? { params: filters } : undefined), addRawText: (kbId: number, data: { title: string; content: string }) => http.post(`/wiki/knowledge-bases/${kbId}/raw/text`, data), uploadRaw: (kbId: number, formData: FormData, onProgress?: (pct: number) => void) => @@ -840,6 +850,16 @@ export const wikiApi = { http.post(`/wiki/knowledge-bases/${kbId}/raw/${rawId}/reprocess`), cancelRaw: (kbId: number, rawId: number) => http.post(`/wiki/knowledge-bases/${kbId}/raw/${rawId}/cancel`), + // Batch reprocess/delete. Select by explicit `ids` (Snowflake strings — never + // coerce to number) or by a `status` selector (e.g. retry all failed). + batchReprocessRaw: ( + kbId: number, + body: { ids?: (string | number)[]; status?: string; force?: boolean }, + ) => http.post(`/wiki/knowledge-bases/${kbId}/raw/batch/reprocess`, body), + batchDeleteRaw: ( + kbId: number, + body: { ids?: (string | number)[]; status?: string }, + ) => http.post(`/wiki/knowledge-bases/${kbId}/raw/batch/delete`, body), downloadRaw: (kbId: number, rawId: number) => http.get(`/wiki/knowledge-bases/${kbId}/raw/${rawId}/download`, { responseType: 'blob', diff --git a/mateclaw-ui/src/assets/main.css b/mateclaw-ui/src/assets/main.css index 072bb1d6..29b22af7 100644 --- a/mateclaw-ui/src/assets/main.css +++ b/mateclaw-ui/src/assets/main.css @@ -401,6 +401,20 @@ html.dark body::before { ================================================================ */ .markdown-body { line-height: 1.75; } +/* Inline preview for tool-generated images (render_html_image / image gen). + Rendered by useMarkdownRenderer.link(); click opens full-size in a new tab. */ +.markdown-body .markdown-generated-image { + display: block; + max-width: min(560px, 100%); + max-height: 640px; + height: auto; + margin: 12px 0; + border-radius: 10px; + border: 1px solid var(--mc-code-header-border); + object-fit: contain; + cursor: zoom-in; +} + /* headings */ .markdown-body h1, .markdown-body h2, diff --git a/mateclaw-ui/src/components/chat/ContextUsagePanel.vue b/mateclaw-ui/src/components/chat/ContextUsagePanel.vue new file mode 100644 index 00000000..f8f08db1 --- /dev/null +++ b/mateclaw-ui/src/components/chat/ContextUsagePanel.vue @@ -0,0 +1,161 @@ + + + + + diff --git a/mateclaw-ui/src/components/chat/ExecutionDetailDialog.vue b/mateclaw-ui/src/components/chat/ExecutionDetailDialog.vue index 02cfb7d3..8f341f9a 100644 --- a/mateclaw-ui/src/components/chat/ExecutionDetailDialog.vue +++ b/mateclaw-ui/src/components/chat/ExecutionDetailDialog.vue @@ -1,6 +1,6 @@ + diff --git a/mateclaw-ui/src/views/Security/AutoApproveGrants/index.vue b/mateclaw-ui/src/views/Security/AutoApproveGrants/index.vue index 4334d628..218ed8cb 100644 --- a/mateclaw-ui/src/views/Security/AutoApproveGrants/index.vue +++ b/mateclaw-ui/src/views/Security/AutoApproveGrants/index.vue @@ -292,9 +292,9 @@