Workflow KnowledgeFS retrieval nodes ran through `retrieval-tests`, which
never wrote an AnswerTrace or an overview event, so their queries were
missing from the retrieval history and from the space overview; the history
also only ever showed each member their own traces.
- AnswerTrace gains a `source` (retrieval_test | workflow | service_api |
agent | mcp; migration 0051_answer_trace_source) derived from the
Capability v2 caller kind. The retrieval-tests route records one trace per
run (stages, evidence bundle, profile metadata), returns its id as
`answerTraceId`, and the workflow node's failed-retrieval capture attaches
to that trace instead of creating a second record.
- The quality trace list exposes and filters by `source`; traces from other
caller kinds are visible to any current reader of the space, and counts and
scores fall back to the evidence embedded in the trace when no bundle row
exists.
- Overview accounting: retrieval-tests and Research tasks now emit
`query.requested`, and the Research job state machine emits
`query.completed` / `query.failed` on terminal stages (wired for both the
in-process gateway and the durable runtime), so query volume, answer rate
and outcomes include every caller. Activity details keep `source` and
`taskKind`.
- Console and service trace routes accept a `source` filter; the retrieval
test page shows a source badge and an all / retrieval tests / workflow
filter, with translations for every locale.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015zw5G5SX3HmVfnZof6YWAc
- create website preview jobs before a knowledge space exists
- persist crawled page content in KnowledgeFS object storage
- import selected pages using preview job and page IDs
- validate preview ownership, expiry, and configuration fingerprint
- clean up preview content after cancellation, expiry, or consumption
- add namespace preview routes, capabilities, migrations, and contracts
- keep online document and drive flows unchanged
KnowledgeFS spaces used their own `knowledge_space_*` RBAC vocabulary, so
workspace roles configured for knowledge bases did not apply to them. Reuse
the legacy `dataset_*` permission points instead, following the mapping the
dataset console already applies:
- read -> dataset_readonly
- create -> dataset_create_and_management
- edit / document write -> dataset_edit
- delete -> dataset_delete
- access config -> dataset_access_config
- query / research / trace detail -> dataset_retrieval_recall
The enterprise `/knowledge-fs/permission-keys/batch` lookup now answers in
that vocabulary and is translated back into the product capabilities the
console exposes as `permission_keys`, so the web stays unchanged; the
historical `knowledge_space_*` keys remain accepted during the transition.
Workspace-level checks send the mapped dataset scene with
`resource_type=dataset`, and the RBAC-disabled fallback grants every dataset
point KnowledgeFS consults.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015zw5G5SX3HmVfnZof6YWAc
Legacy knowledge bases (`datasets`) and KnowledgeFS spaces
(`knowledge_fs_control_spaces`) live in different tables, so a dataset API
key binding could only ever name a legacy dataset and the KnowledgeFS service
API never consulted bindings at all: a key scoped to specific knowledge bases
still reached every KnowledgeFS space in the workspace.
- Bindings carry a `resource_type` (`dataset` | `knowledge_fs_space`) and a
nullable `control_space_id` (migration 9a4e7d1c2b60, existing rows stay
`dataset`), with a CHECK that exactly one id column matches the type.
- `dataset_api_key_service` exposes the key scope by kind, validates and
binds KnowledgeFS spaces (tenant-owned, not deleting/deleted), and cleans up
keys scoped only to a space when its deletion is requested. The orphan-key
cleanup is now NULL-safe so a space binding keeps a key alive.
- Legacy dataset routes only honour `dataset` bindings; the KnowledgeFS
authorization service only honours `knowledge_fs_space` bindings and raises
a scope error that the service API maps to 403 (unknown keys stay 401).
- Console key creation accepts `knowledge_space_ids`; list/create responses
return them next to `dataset_ids`.
- The web scope picker lists KnowledgeFS spaces alongside legacy datasets
when KnowledgeFS is enabled and submits the ids by kind; the scope column
counts both.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015zw5G5SX3HmVfnZof6YWAc
Every document-processing failure surfaced as "the document could not be
processed", and every data-source failure as "the source could not be read",
because the root cause was erased at three points: the compilation classifier
collapsed any coded-but-non-retryable error into DOCUMENT_COMPILATION_FAILED,
datasource runtime errors were reduced to per-operation constants, and the
console mapped dozens of codes onto a handful of generic sentences without the
stage or a support reference.
- knowledge-fs: keep the specific public code for any coded error in the
compilation classifier (retry follows the error or the catalog policy), tag
previously uncoded failures (semantic chunking output, embedding dimension,
PDF rendering, lease loss, generation model, unsupported file type) with
public codes, give embedding provider errors their own codes instead of the
parser's, alias datasource runtime codes, and register 17 catalog entries
(model response invalid, parser unsupported type, PDF render, crawl page
not found, provider timeout/unavailable/rejected, content too large, ...).
- api: extend the public error-code enum (including the missing
DOCUMENT_PARSER_TIMEOUT) and regenerate the contracts.
- web: per-code copy that says what happened and what to do, in all 24
locales, plus a "failed while <stage> · reference <id>" line under each
failed task in the background-task drawer and the document task list.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015zw5G5SX3HmVfnZof6YWAc
A retrieval test record could be flagged as a bad case any number of times,
piling duplicate open cases onto the quality page. Refuse to file another bad
case while the trace already has one that is open or replaying, and let the
retrieval test page know about it.
- knowledge-fs: `createBadCase` looks up an unresolved case for the trace
inside its transaction and throws `QualityBadCaseAlreadyOpenError`; the
create route answers 409 with `QUALITY_BAD_CASE_ALREADY_OPEN`. Quality trace
list items expose `openBadCaseId`.
- api: pass `open_bad_case_id` through the console trace response;
regenerate the contracts.
- web: show the saved-as-bad-case state instead of the flag action when the
selected trace already has an unresolved case, treat a 409 on flagging the
same way, and refresh the history so the state follows the server.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015zw5G5SX3HmVfnZof6YWAc
Query images attached to a retrieval test vanished as soon as the run's
history record was selected, and were never shown again after a reload,
because nothing persisted them: the gateway dropped `queryImages` when writing
`answer_traces`, the console trace/research responses had no image field, and
the composer cleared its images on every record selection.
- knowledge-fs: add a nullable JSON `query_images` column to `answer_traces`
(migration 0049 for postgres and tidb), persist and read it in the answer
trace repository, and expose `queryImages` on quality trace list items.
- api: enrich trace and research task responses with
`KnowledgeFSQueryImageResponse`, resolving file names and short-lived signed
preview URLs only for files the acting account still owns; regenerate the
console and service contracts.
- web: keep composer images scoped to the selected record like the query
text, remember images per run in this session, fall back to persisted
previews (with a labelled placeholder when the file is gone), show them in
the result panel and as a count on history records, and carry them into
retests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015zw5G5SX3HmVfnZof6YWAc
Bring the LLM-driven "automatic" metadata filtering mode from the legacy
Knowledge Retrieval node to the KnowledgeFS retrieval node
(knowledge-retrieval-v2).
Backend:
- Accept `automatic` in `metadata_filtering_mode` and add
`metadata_model_config` to the node entity.
- Add `automatic_metadata_filter.py`, which reuses the legacy metadata
filter prompt templates, invokes the configured model, parses the JSON
answer and normalises operators/values against the KnowledgeFS
metadata catalog before building custom metadata conditions.
- Resolve the shared metadata catalog across the selected spaces through
the new `list_metadata_fields` app execution capability (paginated),
fail closed on missing/unavailable models and catalog admission errors,
fail open on extraction errors, and expose the outcome via
`metrics.metadata_filtering` and `llm_usage`.
- Inject the default extractor from the node factory.
- Fix an unescaped brace in the legacy completion prompt template that
broke `.format()`.
Frontend:
- Offer disabled/automatic/manual modes, normalise unknown persisted
modes to disabled, store the automatic model and completion params,
render the model selector through the shared MetadataFilter, and
require a model when saving in automatic mode.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WnUM1CdV5LpcxZEQ2vu9Xc
- allow updating mutable source configuration fields
- enqueue sync after active business configuration changes
- pause scheduled sync while a source is disabled
- preserve and rebind sync policies across source version updates
- support website, online document, and online drive sources
