The skill detail drawer could only show and edit SKILL.md; the bundle
files under scripts/ and references/ had no console surface, and
templates/ was readable by agents but absent from the canonical
store's bucket set.
- admin endpoints on /api/v1/skills/{id}/files: list (self-heals an
empty canonical store from on-disk files), read, upsert, delete.
Writes update the canonical row, materialize the workspace cache,
and re-resolve the skill so agents pick changes up immediately.
Path envelope enforces the three buckets and blocks traversal;
builtin skill files stay read-only; virtual skills own no files.
- templates/ becomes a first-class DB-persisted bucket shared across
the syncer, the workspace write/delete envelope, and prune guards.
- the agent-facing write_file action now mirrors into the canonical
store and re-resolves instead of writing only the local filesystem.
- SkillMarket detail drawer gains a Files tab: grouped list, viewer,
inline editor, create and delete, refetched on every entry.
Skills backed by a network service could show ready while the service
was unreachable from the current deployment (intranet-only address,
wrong network segment) — the failure only surfaced mid-task.
- endpoint requirement type: TCP-connect probe (1.5s timeout) of the
declared service address; accepts http(s)://host[:port][/path],
host:port, and bare-host forms
- URL-shaped check targets infer the endpoint type without an explicit
declaration; unparseable targets report UNKNOWN instead of missing
- probe results cached 60s per host:port so refresh passes stay cheap
and a VPN connect is picked up within a minute
- unreachable endpoints surface as setup-needed on the skill card,
pre-flight requirement rows, and the agent-facing catalog
The runtime resolved SKILL.md from the workspace directory while the
admin console read the skill_content column, so out-of-band file edits
(agent shell tools in a chat session) changed runtime behavior but never
showed up in the console, and a failed workspace export left agents
executing stale content the console claimed was current.
- SkillContentReconciler: three-way sync between the canonical DB column
and the convention-workspace file, anchored on a sidecar hash marker.
File-side edits ingest into the DB, DB-side edits materialize to the
file, two-sided conflicts resolve DB-wins with a backup.
- Skill detail GET performs a read-time reconcile and triggers a
single-skill rescan when the file side changed, so a console query is
always current without waiting for the runtime cache TTL.
- SkillMarket detail drawer refetches the row and runtime status on open
instead of rendering the page-load list snapshot.
Bundled skill scripts/ and references/ are now persisted to mate_skill_file during startup sync; a workspace missing its scripts directory is force-restored from the classpath bundle even when the SKILL.md version is unchanged; and builtin skills with neither DB rows nor on-disk files backfill from the classpath. Fixes installs performed from builds whose jar shipped without bundle scripts.
Send agent-generated files as native WeChat attachments via the iLink upload flow, and fix the wire protocol for file uploads: dedicated wire ObjectMapper (bypasses the global Long-to-String serializer), md5/len fields and encrypt_type on media items, channel_version 1.0.2, and explicit business-error handling on ret != 0. The weixin adapter now routes generated-file URLs through GeneratedFileScrubber, matching WeCom/Feishu behavior.
Fixes#307
Entity extraction previously constrained entity types but let the
model freely invent any relation between entities, producing noise
that diluted the entities a knowledge base actually cares about.
Adds an optional per-KB relation schema (subjectType/predicate/
objectType triples): when set, the extraction prompt is scoped to
only those relations, and a hard filter drops anything that slips
through before it is persisted. Empty/unset keeps the existing
open-vocabulary behaviour.
Windows-authored zips often store entry names in the local codepage (GBK)
without setting the ZIP UTF-8 flag, while file content stays UTF-8. The
previous fallback decoded the whole archive with one charset, so a single
GBK-named entry forced already-correct UTF-8 content to be re-decoded as
GBK, corrupting valid Chinese text into mojibake. Names and content now
each try UTF-8 first and fall back to GBK independently, per entry.
Global transformation templates (workspace_id IS NULL, e.g. the 7 built-in
starter packs made global by V165) were shared across every workspace but
not actually read-only: any workspace member could edit or delete them,
mutating/affecting all workspaces, with deletes unrecoverable (Flyway
seed runs once).
- Controller: reject update/delete of null-workspace templates with 403
(err.wiki.global_template_readonly); read/apply paths unchanged.
- Service: defense-in-depth — update/delete also reject global templates,
guarding non-HTTP callers (WikiTool LLM entry points). delete() now
checks the entity before deleting instead of deleting blindly.
- findByName: add deterministic ORDER BY (workspace_id IS NULL) ASC so a
workspace-local template wins over a same-named global one (was LIMIT 1
with no ordering). Consistent across H2/MySQL/Kingbase.
- i18n: new err.wiki.global_template_readonly (zh + en).
- Tests: +2 controller mock tests (403 on update/delete, no service write),
+2 E2E tests (global template stays intact; findByName prefers local).
Tests: 10/10 green (4 controller + 6 E2E).
buildWikiContext enumerated an agent's bound knowledge-base pages into
the system prompt capped only by maxContextChars (default 10000, sized
for large cloud models). On a small-context model a large KB therefore
consumed a big fixed slice of the window on every turn — the "tool token
estimate fills the context" report in #521 (the growth lands in the
system-prompt bucket, not the tool-schema bucket; wiki tool schemas are
fixed-size and do not scale with file count).
Add a budgeted buildWikiContext(agentId, budgetTokens) overload mirroring
buildRelevantContext: the page enumeration also stops once the estimated
token total exceeds the budget, appending the existing
'... and more (use wiki_list_pages)' hint. AgentGraphBuilder passes the
same prefix budget it already applies to the memory block; the legacy
Integer.MAX_VALUE path keeps chars-only behavior for large models.
Tests cover null-budget (all pages), token-budget truncation, and
zero-budget skip.
Custom (user-added) providers were hard-coded supportModelDiscovery=false in
createCustomProvider, so self-hosted OpenAI-compatible endpoints (vLLM /
Xinference / LocalAI / gateways) never surfaced the 'discover models' button —
users had to add every model id by hand.
- ModelProtocol: add per-protocol supportsSelfConfiguredDiscovery() + resolve()
helper (single source of truth for chat-model class and capability flags).
baseUrl+apiKey protocols (openai-compatible, dashscope-native, gemini-native,
anthropic-messages) => true; OAuth protocols => false. The flag is deliberately
narrower than 'can ever discover' (built-in ChatGPT-OAuth still discovers via
its OAuth session); javadoc warns against reusing it to gate the button.
- createCustomProvider: default supportModelDiscovery from the resolved protocol
instead of always false. Existing rows are unaffected (no migration).
- OpenAiModelsPath: new single source of truth for the models-listing path,
honoring an optional 'modelsPath' generateKwargs override (mirrors the existing
'completionsPath' override) for endpoints behind a reverse proxy / non-standard
prefix (e.g. /openai/v1/models) that would otherwise 404 on /v1/models.
Shared by BOTH discovery (ModelDiscoveryService) and the failover liveness
probe (OpenAiCompatibleListModelsProbe) so an override can't make a provider
discoverable yet still marked unhealthy by a probe hitting the wrong path.
- Tests: ModelProtocolTest (capability table + resolve fallback), OpenAiModelsPathTest
(path branch table + vendor cases + modelsPath override), and custom-provider
discovery-default assertions. Path-resolution coverage consolidated into
OpenAiModelsPathTest (was split across the discovery + probe test files).
- Docs: zh/en models.md note custom-provider discovery + modelsPath override.
Refs matevip/mateclaw#519
Give raw-material uploads a dedicated five-minute timeout and process file-picker and drag/drop uploads through a shared two-worker queue, so constrained uplinks no longer abort multipart requests at the global 30-second deadline.
Switch wiki processing jobs and page citations to application-assigned IDs: the PostgreSQL/Kingbase migrations define plain BIGINT primary keys without identity defaults, so database-generated keys fail on insert.
Eight entities (fact, fact contradiction, morning-card seen, wiki hot
cache / relation / transformation / transformation run / image caption
cache) declared IdType.AUTO while their PostgreSQL-compatible migrations
define the primary key as a plain BIGINT with no identity default.
MyBatis-Plus omits the id column from the generated INSERT under AUTO,
so every insert fails with a NOT NULL violation on those databases —
silently on paths that only log a warning. Switch them to snowflake
ASSIGN_ID, which works on all dialects since auto-increment columns
accept explicit values. Add a parameterized contract test pinning the
id strategy for all eight entities.
The lint script referenced eslint with --ext flags but the repo never had
an ESLint config file, so pnpm lint always failed. Add a flat config
(typescript-eslint recommended + vue essential) with legacy-code rules
downgraded to warnings, drop the flat-config-incompatible --ext flags,
and move pnpm build approvals from the no-longer-read
pnpm.onlyBuiltDependencies field to pnpm-workspace.yaml allowBuilds.
Give raw-material uploads a dedicated five-minute timeout and process file-picker and drag/drop uploads through a shared two-worker queue, so constrained uplinks no longer abort multipart requests at the global 30-second deadline.
Switch wiki processing jobs and page citations to application-assigned IDs: the PostgreSQL/Kingbase migrations define plain BIGINT primary keys without identity defaults, so database-generated keys fail on insert.
- Settings → System gains a 'default workspace storage path' item: validated
on save (absolute, creatable), applied immediately without restart, and
re-applied from the database on startup. Blank clears the override;
existing data is never migrated.
- Desktop local file/command tools get a renderer settings page (allowed
directory list with per-row delete, add via native picker, enable toggle,
tunnel status); the native dialog additionally gains a 'remove directory'
flow, fixing the whitelist that could only grow.
- System settings save surfaces backend validation errors as a toast.
Let skill Python scripts install packages from a configurable pip index instead of the default PyPI. docker-compose passes PIP_INDEX_URL / PIP_TRUSTED_HOST into the container; for the desktop app (host JVM, no Docker env) SkillScriptExecutionService falls back to mateclaw.pip.index-url / trusted-host Spring config and injects them into the subprocess, auto-deriving the trusted host for plain-HTTP LAN mirrors. The runtime image gains pip and a build toolchain (with the PEP 668 marker removed so on-the-fly installs work), and the script timeout ceiling is raised to accommodate large installs.
Wire MCP standard notifications/progress into the existing SSE stream so long-running MCP tool calls surface live progress instead of a bare spinner. A per-call progressToken maps back to (conversationId, toolCallId); ProgressAwareMcpToolCallback injects it into tools/call _meta and calls McpSyncClient directly (falling back to the delegate on error, and applying identity forwarding first). Progress events skip the ring buffer and are replayed from a latest-value snapshot on SSE reconnect. Frontend renders a gradient progress bar in ToolCallSegment when a running tool reports progress.
- Shell scan: a token normalizing to the filesystem root (//, /., /..) is
only skipped when the command carries no destructive verb; with
rm/rmdir/shred/srm present the scan fails closed, so 'rm -rf //' is
refused while sed empty replacements (s/pattern//) stay allowed.
- Chat-upload fallback: a boundary violation is waived only when the
requested path itself normalizes inside one of the conversation's
candidate upload directories; a basename match against a stored
attachment no longer clears the violation. Resolver/DB failures keep
the BLOCK finding. The unused candidate-roots resolve overload is
removed.
- Regression tests for destructive root tokens, sed allowance, the
fail-closed compound case, stored-upload-path allowance, basename
collisions, cross-conversation paths, and resolver failure.
Skip absolute-path tokens that normalize to the filesystem root in the shell boundary scan (shell syntax like sed's s/pattern// was misread as a path outside the workspace), and add a DB-backed chat-upload fallback in WorkspaceBoundaryGuardian: when a file-tool path triggers a boundary violation, resolve the conversation's real candidate upload roots so attachments stored in workspace-scoped directories are found even when the thread-local workspaceBasePath is null.
Adds a read-only GET /api/v1/settings/search-providers catalog (admin-gated, no secrets), grouped collapsible provider cards, and a schema-driven plugin config form. Breaks a SystemSettingService<->PluginManager circular dependency via parameter @Lazy (with a context smoke test), and fixes PluginManager.updateConfig to merge instead of overwrite so omitted/blank secret fields are preserved. Hardens plugin search-provider id validation (reject-not-trim, case-insensitive conflict) and insulates the provider bridge hot path from throwing plugin code.
- Remove "(issue #477)" internal planning references from shipped Java
(SearchProviderRegistry, PluginSearchBridge, and the two new tests) — code
should describe what it does, not point at issue trackers.
- Drop "openclaw" external-project attribution from the search provider
comments (SearchProviderRegistry, SearchCache, SearchQuery), restating them
as objective functional descriptions.
- requireSessionOwnership now also checks session.kbId() == path kbId (404 on
mismatch), so a research session started under one KB cannot be addressed via
another KB path even when the caller's key is bound to both — defense-in-depth
on top of the keyId ownership check.
- Drop internal "R7" / "review #446" markers from the controller Javadoc in
favour of functional wording.
- Import Set/Map/concurrent types and static any() instead of inline FQNs in the
new kb-open research/auth tests, per code style.
* feat(kb-open): Deep Research open API (start/SSE/status/cancel)
Implements the async Deep Research endpoint for the KB Open API (#443).
Research is a multi-step LLM pipeline (plan → retrieve+draft → compose)
that runs asynchronously and broadcasts progress via SSE.
Endpoints:
- POST /{kbId}/research start (returns sessionId + streamUrl)
- GET /{kbId}/research/{id}/stream SSE progress (?token= for EventSource)
- GET /{kbId}/research/{id}/status query status / final report
- POST /{kbId}/research/{id}/cancel cancel running session
Components:
- KbOpenResearchController: 4 endpoints, @RequireKbScope("kb:search")
- KbResearchSessionRegistry: in-memory session tracking with keyId
ownership (a caller can only query/cancel their own sessions)
Security:
- R7: SSE uses ?token= query param (KbOpenApiAuthFilter already supports
this fallback for EventSource which can't set Authorization headers)
- Session ownership: status/cancel/stream all verify keyId match
- Cancel checks session is RUNNING (409 otherwise)
Reuses existing WikiResearchService.research() + ChatStreamTracker for
the actual research pipeline and SSE broadcasting.
Tests (6 new, all green):
- KbResearchSessionRegistryTest: register/complete/fail/cancel lifecycle,
cancel-on-completed no-op, unknown session returns empty
Closes#443
* fix(kb-open-research): cooperative cancel, sticky terminal, TTL, concurrency cap
Review #446 — address all 4 job-lifecycle/cost blockers + nits:
1. Cooperative cancellation (was: cancel only flipped status, pipeline ran
to completion). Cancel endpoint now calls streamTracker.requestStop();
WikiResearchService.ensureNotCancelled() checks isStopRequested at each
stage boundary (plan→draft, draft→compose) and inside the parallel draft
fan-out — so cancel actually halts the expensive LLM calls, not just the
SSE stream. Throws ResearchCancelledException (caught locally, no error
broadcast).
2. Sticky CANCELLED terminal. complete()/fail() now no-op on a CANCELLED
session, so a user who cancelled never sees a COMPLETED report surface
via /status.
3. Session registry TTL. Terminal sessions get an updatedAt timestamp and
are evicted by a @Scheduled sweep after
mate.kbopen.research.session-ttl (default 30m). RUNNING sessions are
never evicted. Prevents unbounded memory growth.
4. Per-key concurrency cap. startIfAllowed() rejects new research when a
key already has mate.kbopen.research.max-concurrent-per-key (default 3)
RUNNING sessions → 429. Stops one key from spawning ~60 parallel
multi-step LLM pipelines per minute under the per-min rate limiter.
5. Inline FQN → import (controller LinkedHashMap, test List.of).
Nits (inherited from P0-A rebase):
- V162→V164, prefix VARCHAR(12), design doc moved to rfcs/.
- Design doc: kb:search scope row now documents it covers /research/**.
31 tests pass (12 registry incl. sticky-cancel/concurrency/TTL +
13 service + 4 rate limiter + 4 controller + ...).
* fix(kb-open): scope-limited ?token= SSE auth fallback in KbOpenApiAuthFilter
R7: the SSE progress stream (/research/{id}/stream) is consumed by browser
EventSource, which cannot set an Authorization header. The filter's
extractBearerToken() never read ?token= (still a TODO), so the SSE endpoint
was unreachable from the browser — the headline use case got 401.
Fix: accept ?token= ONLY on SSE stream paths (isSseStreamPath, suffix
/stream), reject it everywhere else so the API key does not leak into
access/proxy logs for normal calls (R5). Matches the JwtAuthFilter convention
(getRequestURI logs carry no query string).
Also bypass the per-minute rate limiter on the SSE path: EventSource
reconnects/heartbeats would otherwise burn the key's window and 429 its own
POST /research start. Rate limiting belongs on the cost-producing endpoints.
Tests (6 new, KbOpenApiAuthFilterTest):
- non-SSE: header passes, ?token= rejected (no authenticate call)
- SSE: ?token= authenticates, missing token → 401
- SSE: bypasses rate limiter; non-SSE still hits it
* fix(kb-open-research): make per-key concurrency cap atomic (no check-then-act race)
startIfAllowed() did stream-and-count then put() — not atomic. Two
concurrent starts for the same key could both pass the count check (both
see < cap) and both put, admitting more sessions than the cap. On the
virtual-thread start endpoint this is a real DoS/cost-bypass path.
Fix: maintain a per-key AtomicInteger running counter (runningPerKey),
incremented atomically on start (incrementAndGet + rollback on overflow)
and decremented on each RUNNING→terminal transition (complete/fail/cancel).
The counter is kept in lock-step with status==RUNNING; since terminal
states are sticky, each session decrements exactly once.
cancel() also rewritten to capture the pre-transition state cleanly (the
old return check relied on Map.computeIfPresent returning the new value,
which worked but read as 'before.status==CANCELLED').
Tests (+2): cancelled/failed release slot (counter consistency), and a
concurrent-start test (12 virtual threads, cap=3) asserting exactly cap
admits — would be flaky/fail under the old impl.
* refactor(kb-open-research): remove unused register() back-compat method
register() was left over from the initial impl — it bypassed the per-key
concurrency cap (no startIfAllowed check) and, after the atomic-counter fix,
incremented runningPerKey without any overflow rollback. With no production
caller (the start endpoint uses startIfAllowed), it only existed for tests to
set up a RUNNING session. Drop it and route the tests through startIfAllowed
so nothing can accidentally ship a path that ignores the cap.
After the workspace-aware chat-uploads change, the upload root became
absolute (the resolver normalizes via toAbsolutePath/normalize, and the
autoconfiguration rewrites baseDir to an absolute path). ChatController.upload
then set ChatUploadResponse.path to that absolute path — despite the inline
comment promising a relative path "to avoid exposing the server's absolute
path". The field is rendered into the LLM prompt ("附件: foo (path)") and
returned to the client, so this leaked the server filesystem layout into both
the prompt and the response, and broke portability if the deploy dir moves.
Extract toRelativeUploadPath(uploadRoot, convId, storedName) which makes the
path relative to the upload root's parent (preserving the trailing sub-dir
name, e.g. chat-uploads/{convId}/{storedName}) and normalizes separators to
'/'. Retrieval is unaffected: it goes through the basename-based
ChatUploadResolver and the /api/v1/chat/files/... URL, not this field.
Adds ChatControllerUploadPathTest (default root, absolute workspace-scoped
root, custom base-dir name) asserting the result is relative and leak-free.
Addresses the blocker item in #452.
Adversarial review of PR #464 found that classify() promoted an absent
channelType to the 'authenticated' trust branch, stamping an untrusted
ThreadLocal username (e.g. stale value on a reused thread, or internal
tasks like SkillConsolidation/Reflection that carry no channel) with
authenticated trust — contradicting the fail-closed contract the service
documents.
- classify(): channel==null/blank now resolves to NONE (no injection);
only the explicit 'web' channel may yield authenticated. Unrecognised
non-web channels downgrade to external, never authenticated.
- signingKey(): replace the one-shot keyParseAttempted latch with
lastAttemptedPem so a corrected/hot-reloaded PEM re-parses on the next
call without an app restart. Still fail-closed when PEM is unchanged.
- Tests: 4 new cases lock the regression (null+dirty-ThreadLocal->NONE,
blank->NONE, novel channel->external, self-heal after config fix).
- .gitignore: exclude local .codebase-memory/ agent index.
MCP+identity suite: 93/93 green.
Two SSRF hardenings on top of the private-network deployment mode:
1. Redirect / subresource re-validation. The SSRF guard previously ran only on
the initial navigation URL in the tool layer, so a public page that 302s to
169.254.169.254 (or a script fetch / img to a metadata IP) reached the target
unchecked — worse now that private-network mode exists. Install a per-context
request interceptor (BrowserLauncher.applyContextDefaults) that re-runs
UrlSafetyChecker on every http(s) request and aborts blocked ones. Non-network
schemes (data:/blob:/about:) pass through; unexpected checker faults fail open
so a transient error cannot wedge the page (the initial URL was already checked).
2. Allowlist can no longer open a cloud-metadata endpoint. Metadata hostnames and
IPs are now checked BEFORE the allowlist short-circuits, so an operator entry
like 169.254.0.0/16 or metadata.google.internal can never expose instance
metadata. Ordinary private-host allowlisting is unaffected (regression-tested).
Also correct the 192.0.0.192 comment (Oracle Cloud IMDS, not Azure).
The identity forwarded to opt-in MCP servers was a one-dimensional string
(ChatOrigin.requesterId): a MateClaw username for web logins, but a webchat
visitorId for visitors and an IM sender id for IM — indistinguishable to the
REST backend. The signed-token mode (d204b702) made this worse: an RS256
signature over an unauthenticated visitorId reads as "MateClaw authenticated
this user" to any backend that trusts the signature.
Introduce an identity-typing dimension at McpIdentityForwardService:
- classify() branches on ChatOrigin: authenticated (web login, sub=immutable
userId), anonymous (webchat visitor, trust=anonymous), external (IM sender,
trust=external), or none (cron/system → nothing injected, fail-closed).
- mint() adds `trust` and `channel_type` claims; plaintext value is prefixed
`trust:subject` so backends can tell the kinds apart without a JWT.
The immutable userId reaches resolve() without coupling it to the user store:
JwtAuthFilter stamps user.id into auth.setDetails() (both JWT and PAT paths),
and ChatController.memoryOrigin carries it on a new ChatOrigin.requesterUserId
field (only-add, per the record's evolution rule).
Resolves the webchat semantic mismatch raised in #459 and the "sub should be
an immutable user id" follow-up. 82 tests green (4 identity classes covered
with claim assertions + full ChatOrigin/MCP regression).
(cherry picked from commit b5d2cfbf98b39848d7139c743a0b81fea71e8ffe)
Replace java.util.Set.of / java.util.Map.of inline fully-qualified calls with
top-of-file imports per code style (test sources sync to the open-source repo).
* feat(mcp): forward authenticated user identity to opt-in STDIO MCP servers
A STDIO MCP server is one shared subprocess per configuration; its env is fixed
at spawn and STDIO has no per-request header channel, so per-user identity must
travel in-band with each tool call. Previously nothing carried it, so an MCP
server could not call its downstream REST backend on behalf of the acting user.
Inject the authenticated username (from ToolExecutionContext) into each tool
call's JSON arguments under the reserved key `__mateclaw_user__`, for servers an
operator explicitly opts in via `mateclaw.mcp.identity-forward.servers` (by name
or id). The MCP server reads/strips it and forwards on-behalf-of alongside its
own backend API key.
- McpIdentityForwardProperties: per-server opt-in allowlist (name or id).
- IdentityForwardingToolCallback: wraps an MCP callback, merges the username
into the args; injected by trusted code, overwrites any LLM-supplied value
(no spoofing); forwards unchanged when there is no user or args aren't an
object/are malformed.
- McpClientManager: captures server names; wraps opt-in servers' callbacks
inside the prefix wrapper (so name-prefixing / return-direct still see the raw
delegate). Non-opt-in servers are untouched — username never leaks to them.
- Tests: injection, LLM-value overwrite, empty/non-object/malformed inputs,
no-user passthrough, opt-in matching by id/name.
- Docs (zh/en mcp.md): opt-in config, `__mateclaw_user__` contract, FastMCP
Python skeleton, trust model.
Default off (empty allowlist) — zero behavior change for existing servers.
Plaintext username suits a trusted-network REST backend keyed by an API key;
a signed short-lived token is noted as the stronger-isolation follow-up.
* feat(mcp): add signed-token trust model for MCP identity forwarding
Plaintext username forwarding makes the REST backend trust an unverifiable
assertion from the (shared, LLM-adjacent) MCP service — a confused-deputy model.
Add an opt-in signed-token mode so identity crosses the trust boundary as a
short-lived RS256 JWT the backend can verify with a public key.
- McpIdentityForwardProperties: nested `token` config (enabled, issuer,
ttl-seconds, key-id, private-key-pem, audiences) + USER_ARG/TOKEN_ARG keys.
- McpIdentityForwardService: resolves the injection — plaintext username
(__mateclaw_user__) when token mode off, else a minted RS256 JWT
(__mateclaw_token__) with sub=user, aud=server, short exp, jti. Lazy key
parse; fail-closed when token mode is on but the key is missing/unparseable
(no silent downgrade to plaintext). Signs with MateClaw's private key so the
backend only needs the public key (cannot mint/impersonate).
- IdentityForwardingToolCallback: now delegates the what-to-inject decision to
the service (keyed by per-server audience); static withClaim() keeps the
JSON-merge logic (overwrites LLM-supplied key, leaves non-object/malformed
args untouched).
- McpClientManager: injects the service; passes service + audience through the
wrap path for opt-in servers only.
- Tests: token mint+verify (with an in-test RSA keypair, asserting sub/aud/iss/
exp/jti), plaintext mode, no-user and no-key fail-closed, audience resolution.
- Docs (zh/en): token config, key generation, claims, REST-side verification
example, public-key distribution + JWKS-endpoint follow-up.
Default unchanged: token.enabled=false → plaintext (back-compat); whole feature
still opt-in per server and off by default.
mate_memory_recall.filename is VARCHAR(256), but the snippet-level recall
tracker assembles the key as `path + '#' + H2-heading-slug`. When the LLM
writes an over-long daily-note heading (the summarize prompt placed no
length cap on the `##` title), the CJK-preserving slug pushes the filename
past the column, and writes fail with Data too long / string too long.
Three layers of defence, root cause + hard caps:
1. prompt (source) — summarize-system.txt now asks for short (≤30 chars)
`##` titles; details go in the body, not the heading.
2. slug cap (close to source) — MemoryRecallTracker.sanitizeSectionKey
caps the slug at MAX_SECTION_SLUG=200, leaving path+'#' well under 256.
3. write-side cap (catches every path) — MemoryRecallService.recordRecall
truncates filename to MAX_FILENAME_LENGTH=255 at the entry point, so
the select/insert/update branches share one value and the dup-key
concurrency fallback still matches. Covers trackActiveRetrieval too,
which bypasses sanitizeSectionKey.
Tests: MemoryRecallFilenameTruncationTest covers both caps (over-long CJK
heading, normal heading untouched, ascii slug, date prefix survives) plus
an end-to-end assertion that the stored value fits VARCHAR(256). Existing
memory-suite unit tests still green.
Replace the single inline org.assertj.core.api.Assertions.assertThat call
with the static import already used for assertThatThrownBy, per code style
(test sources sync to the open-source repo).
* feat(kb-open): P0-B 9 open API endpoints
Implements the 9 read-only KB Open API endpoints on top of the P0-A
auth skeleton (#441). Each returns an explicit DTO (A5: never raw
entities) and delegates assembly to service-layer methods that return
pure DTOs (A6: no HTTP coupling, MCP-ready).
Endpoints:
- GET /pages/{slug} entity card (mode=summary/full/section:{heading})
- POST /search hybrid retrieval (granularity=entity/chunk)
- POST /search/chunks chunk-level semantic search
- POST /pages/{slug}/traverse entity relation graph (depth ≤ 2)
- GET /pages/{slug}/trace provenance (page → chunk → raw)
- GET /taxonomy pageType/entityType/relationType enumeration
- GET /whats-new recent changes + stale pages
- GET /stats KB statistics
- GET /pages lightweight page list
Components:
- KbOpenApiController: 9 endpoints, each @RequireKbScope annotated
- KbOpenApiService: assembly layer (card, traverse BFS, metadata parsing)
- KbOpenApiDtos: all response DTOs as records (PageCard, TraceResult,
TaxonomyResult, KbStats, WhatsNewResult, TraverseResult, PageList)
Traverse (pragmatic version):
- depth ≤ 2 with explosion guard, predicate LIKE matching
- slug → pageId → mention → primaryEntity (salience-highest)
- neighbor nodes echo slug when available (R11)
- edge sourceHandle via evidenceChunkId → citing page
Tests (4 new, all green):
- KbOpenApiControllerTest: 404 on missing page/slug, delegation to service
Closes#442
* fix(kb-open): address review feedback on #445
BLOCKERS:
- stats.pagesWithLinks always returned 0 because listByKbId() nulls out
content. Switch to listByKbIdWithContent() so [[wiki link]] detection works.
- Test file: replace inline java.util.List.of() FQN with import + simple name
(sync-opensource would expose the unidiomatic style).
NITS (inherited from P0-A rebase):
- V162→V164, prefix VARCHAR(12), FQN imports, parseScopes trim, ?token=
fallback removal, design doc moved to rfcs/ — all now in ancestor commit
6fd62440.
EXTRA:
- whatsNew staleReason: hardcoded Chinese "上游 fact 页面变更" → English
"Upstream fact page changed" (external-facing API response).
* chore(wiki): drop RFC-012 prefix from progress field Javadocs (#449 nit)
Per #449 review (4825113234): the internal RFC-012 reference should not
appear in code. progressPhase/progressTotal/progressDone Javadocs still
carried the "RFC-012 M2 v2 UI:" prefix after #449's English translation
pass — drop it now that these lines are touched.
Zero behavior change.
* chore(kb-open): drop inline FQN in parseScopes (#444 nit)
Per #444 review (4825157096): parseScopes used
`.collect(java.util.stream.Collectors.toUnmodifiableSet())` while
`Collectors` is already imported at the top of the file. Use the simple
name. Zero behavior change.
@SpringBootTest + H2 coverage asserting that both listRecent and search exclude
a still-running sibling conversation (stream_status='running') and the caller's
own current conversation, so concurrent sessions of the same agent cannot leak
into each other's session_search results.
Every endpoint now binds its independent id param to an authorized KB: rawId/chunkId resolve-then-workspace-check, pageId is asserted to belong to the path kbId, and slugs stay kbId-scoped. Adds unit tests for same-KB/cross-KB/unknown cases.
Propagates structured error codes through the KB processing pipeline, surfaces silent sub-step warnings as a non-failure warning state, and adds a cross-KB failure center for aggregated visibility.
Lets an employee pin an ordered fallback chain of (provider, model) entries; the same provider may appear multiple times with different models. Build-time dedup keys on exact (provider, model).
Add unit coverage for the StageInstructions custom deserializer: plain-string
shorthand, full object with instructions+template, unknown-field skipping, and
both forms coexisting on one WikiPageTypeDef (backward compatibility).
- resolveAgentBasePath: the relative-override branch now normalizes the
resolved path and rejects values that escape the workspace root via "../"
(the absolute branch already did this), keeping attachment/media/tool I/O
contained when an agent's workspaceBasePath is a relative override.
- cleanAttachmentFiles: return early on a null/blank conversationId so a bare
upload root can never be walked and deleted wholesale.
- Translate the chat-upload Javadoc/comments to English (cleanAttachmentFiles,
BaseAgent image-path resolver) per code style.
- Add a resolver test for the relative-override escape fallback.
Strict OpenAI-compatible providers reject the /chat/completions request
with HTTP 400 when an assistant message in history carries a tool call
whose function.arguments is not parseable JSON. Normalize blank or
non-JSON arguments to "{}" at the send chokepoint so streaming,
history-replay, and older-persisted tool calls all stay well-formed.
* feat(sso): feishu OAuth2 single sign-on (ISSUE #405 P0)
Implements the SSO design (ISSUE #405) with feishu as the first IdP
and a generic OAuth2 provider abstraction for future dingtalk/wecom
extensions. SSO is disabled by default — existing deployments are
unaffected until mateclaw.sso.enabled=true.
Backend:
- SsoProvider interface + SsoUserInfo record: generic IdP abstraction
- FeishuSsoProvider: OAuth2 authorization-code flow (app_access_token
with Caffeine cache → user_access_token → user info). apiBase switches
between feishu.cn / larksuite.com by domain config.
- SsoProviderRegistry: conditional registration, lists enabled providers
- SsoStateService: HMAC-signed OAuth2 state + self-contained bind_token
JWT, both persisted to sso_state DB table for multi-node correctness.
State is one-time-consumable (conditional UPDATE), bind_token jti
anti-replay via PK insert. Hourly ShedLock purge (LambdaQuery + Java
time, works on all 3 dialects).
- SsoService: authorize/callback/bind, user mapping (union_id first →
external_id fallback), auto-create with concurrent idempotency
(DuplicateKeyException → rollback orphan user → re-query), link-only
mode issues bind_token for existing-account binding.
- SsoController: 4 endpoints (/providers, /authorize, /callback, /bind)
all permitAll.
- V159 migration (h2/mysql/kingbase): mate_user_external_identity,
sso_state, ALTER mate_user.password NULL (SSO-only users).
- AuthService: generateToken promoted to public; login() guards
password=null (SSO-only users cannot password-login).
- SecurityConfig: /auth/sso/** added to permitAll whitelist.
- LoginRateLimitFilter: expanded to cover /auth/sso/bind (brute-force
surface equivalent to /auth/login).
- application.yml: mateclaw.sso.* config block (all env-var driven).
Frontend:
- Login.vue: dynamic SSO buttons (only shown when providers configured),
OAuth2 callback detection (?sso=callback), link-only bind dialog,
shared applyLogin flow (localStorage + workspace + route).
- api/index.ts: ssoApi (providers, authorize, callback, bind).
Tests: SsoStateServiceTest (11) — state issue/verify/replay/tamper,
bind_token issue/verify/anti-replay/garbage. Regression: PAT (23) +
Approval resolve (13) all green.
Not in scope (P1/P2): link-only bind/unbind management endpoints,
user enable/disable endpoint, dingtalk/wecom providers, admin SSO
config page. Workspace assignment for auto-created users remains a
product decision (design doc §12 item 2).
* fix(sso): self-review fixes — P0 security + P1 quality
P0-1 BindRequired serialization: replaced the R.fail(200, Map.toString())
hack with a structured SsoCallbackResponse record. Controller no longer
catches an exception for a non-error path; frontend reads bindRequired
flag directly instead of regex-parsing a stringified map.
P0-2 createSsoUser unbounded recursion: added a retry flag — second
DuplicateKeyException (extreme race where identity was concurrently
deleted) now throws a 503 instead of recursing to stack overflow.
P0-3 state TTL not enforced: verifyState's conditional UPDATE now
includes created_at > cutoff, so a state unused for 5+ min is rejected
at consumption time, not just at the 1h purge. Without this the 5-min
window was advisory only.
P1-5 SsoStateService unused ObjectMapper: removed dead injection.
P1-6 audit JSON string concat: replaced with ObjectMapper serialization
(provider/externalId no longer risk breaking the JSON structure).
P1-7 LoginRateLimitFilter shared counter: documented the intentional
decision that login + bind share a per-IP counter (same brute-force
surface) with guidance on switching to per-path if finer isolation
is needed.
* feat(webchat): add approval resolve + replay for API-Key channel (ISSUE #413 P1)
Before this, a WebChat (API-Key) channel that hit a ToolGuard-protected
tool parked the turn in a pending approval the visitor could never
clear — it hung for 30 min until the GC timeout and the turn was
wasted. This PR closes the loop, mirroring the web ChatController.
A1 — no code change. tool_approval_requested already reaches the SDK
via ToolExecutionGuardHelper's streamTracker.broadcastObject (direct
SSE push, bypassing the StreamDelta path). Adding it to
forwardVisitorEvent would double-deliver; the default-drop is correct.
A2 — new /sessions/approve and /sessions/deny REST endpoints. Auth is
the existing visitorToken + conversationId ownership guard; the actor
is webchatUsername(visitorId), which resolves the 'no MateClaw
username' blocker noted in the old stopSession javadoc. Both broadcast
tool_approval_resolved so the SDK clears its banner in real time.
A3 — approve returns an SSE stream: resolveAndConsume (atomic DB +
metadata + memory), restoreChatOrigin (recovers the webchat origin
captured at createPending), then chatWithReplayStream replays the
tool call and continues the turn. Replay may re-trigger approvals,
which the existing tool_approval_requested direct push handles.
A4 — stopSession now sweeps pending approvals (denyAllByConversation)
and broadcasts each resolution, so stopping a stream no longer leaves
approvals lingering for the GC.
Tests: WebChatApprovalInteractionTest (7) — deny resolves + broadcasts,
deny auth/ownership guards, idempotent unknown-pending, stop sweep
clears pending, stop no-op when nothing pending.
Regression: WebChatStopStreamTest (5), WebChatArchivePinTest (6),
WebChatSchemaFieldsTest (5), WebChatWikiPageListTest (8),
ApprovalWorkflowServiceResolveTest (13), GcTest (7), RecoveryTest (7).
* fix(webchat): IDOR guard + SSE hang fix (PR #415 review)
Addresses all review feedback from mateaix:
P0 IDOR (security): /sessions/approve and /sessions/deny accepted a
client-supplied pendingId without cross-checking it belonged to the
caller's conversation. A visitor could resolve / replay another
visitor's guarded tool call. Fix: getPending(pendingId) then assert
conversationId matches before resolving. Added getPending delegate on
ApprovalWorkflowService so the webchat controller (which holds the
workflow facade) can do the precise lookup.
SSE hang: approveSession's already-resolved / error branches broadcast
'done' before streamTracker.register/attach, so the event had no
subscriber and the SSE hung to the 10-min timeout. Fix: register+attach
first, then resolveAndConsume. Removed the now-duplicate register/attach
in the replay branch.
Tests: +2 IDOR cases (cross-visitor pendingId rejected 404; mismatched
pendingId rejected 404). denyResolvesPending now asserts via getPending
(findPendingByConversation returns the earliest pending, polluted by
cross-test map state). denyUnknownPendingIsSafe updated to expect 404
(no longer leaks pendingId existence). Isolated IDOR victim/attacker
visitor IDs to avoid cross-test conversationId collisions.
Regression: WebChatStopStreamTest (5), WebChatArchivePinTest (6),
ApprovalWorkflowServiceResolveTest (13), GcTest (7), RecoveryTest (7).
* style(webchat): use simple ChatOrigin name in approveSession (PR #415 review)
Reviewer flagged fully-qualified inline types (ResolveOutcome was fixed
in the prior commit; ChatOrigin was missed). Add the import and switch
the 3 FQN references in approveSession to the simple name, matching the
ResolveOutcome cleanup. chatStream's pre-existing FQN usages are out of
this PR's scope and left untouched.
Before this, a workflow await_approval step whose approverChannels
pointed at feishu/wecom was effectively dead for IM interaction. Even
after PR #414 (B1) pushed the notice to the IM group, clicking the
card's Approve/Deny buttons did nothing useful:
- Identity check (requester==clicker) failed-closed: wf- approvals
have userId=null (system-initiated), so every click was rejected.
- Even if it passed, the synthetic /approve injection was a dead end:
the router routes by conversationId, but wf- ids use a synthetic
workflow:run:{runId} key that no IM conversation matches, so
findPendingByConversation returned null and the /approve was fed
to the LLM as plain text.
B3 fix: both ToolGuardCardHandlers now detect the wf- prefix and
resolve inline (approvalService.resolve), bypassing the synthetic
injection entirely. The WorkflowApprovalResolvedEvent published
inside resolve is picked up by ApprovalResumeBridge (activated in
PR #414 B2), which resumes the paused run. This mirrors the Web /
WebChat resolve path (PR #415).
Identity policy: any audience member may resolve a wf- approval.
The card only reaches channels declared in await_approval's
approverChannels, so whoever sees it is a designated approver.
Regular tool approvals keep the strict requester==clicker guard.
Tests:
- wecom ToolGuardCardHandlerTest: +2 wf- cases (inline resolve, no
synthetic injection; already-resolved renders expired). Existing 6
cases updated for the new 3-arg constructor.
- feishu FeishuCardDispatcherTest: updated for the new factory
constructor signature.
Regression: ApprovalWorkflowServiceResolveTest (13), GcTest (7),
RecoveryTest (7), feishu dispatcher (4), button value (7),
renderer (3+3) — all green.
