opensource/dify
2
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-09-05 00:31:19 +08:00
Code Issues Packages Projects Releases Wiki Activity

restore semantic document compilation pipeline

Browse Source
This commit is contained in:
Jyong 2026-08-13 13:10:39 -04:00
parent 0bc3a849ce
commit ffd2a7094d
40 changed files with 10753 additions and 301 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
api/knowledge-fs-contract.lock.json
View File

@ -1,6 +1,6 @@
{
"schemaVersion": 5,
"subtreeTree": "d89aa89a543c1c1a5b3d042881597d9af2a1a47b",
"subtreeTree": "73665ebbd8b6e07538c7d07f6983f17922dce439",
"openapiSha256": "47936a7d9ffdc27e2b2b8982a90e1936dc3bf59a64c316f452a6912ec1d2fcd6",
"capabilityV2AuthManifestSha256": "fc0a47e23cce12544882f0298522b4933002e892b84ce1815df7e81d36a7a0c7",
"capabilityV2AuthTestVectorSha256": "ae0de37b1ff05c40f905cf17a7b410d8971acacf64db07d5ee3d6fecfa559ce3",

103
knowledge-fs/.harness/changes/2026-08-13-semantic-document-compilation-restoration.md Normal file
View File

@ -0,0 +1,103 @@
# Semantic Document Compilation Restoration
## Summary
Restores the missing reasoning-model semantic compilation stage between parsing and indexing.
Parser output is normalized without mutating the stored parse artifact, the configured reasoning
model selects contiguous source ranges, and the service materializes authoritative chunk text from
those source elements. Outline, summaries, PageIndex, dense/full-text projections, and Graph facts
now share one immutable publication generation.
## Behavior and invariants
- Untrusted Unstructured `Title`/`Heading` classifications no longer become compulsory chunk
boundaries; explicit parser hierarchy remains trusted.
- Semantic requests are bounded by elements, windows, response bytes, chunks, entities, and
relations. Tables/images remain atomic at model-window boundaries.
- Model output supplies source ranges and metadata, never authoritative document text. Coverage,
order, terminal model identity, and Unicode grapheme limits fail closed.
- A compact immutable generation receipt makes completed retries provider-free and detects
conflicting or corrupt replay.
- Node rows and their receipt are persisted in one database transaction; all-excluded generations
are represented explicitly.
- New semantic compilations materialize quality-controlled joint Graph facts synchronously before
candidate publication. Graph facts remain generation- and source-node-scoped.
- Reasoning-profile migrations rebuild semantic nodes, outline, paths, PageIndex, search
projections, and Graph together. Embedding-only migrations clone the exact semantic node
generation and rerun only the affected projections/Graph materialization without another LLM
segmentation call.
- Existing published generations remain readable until candidate evaluation and publication CAS
succeed. No semantic profile silently falls back to legacy fixed-size chunking.
## Data model
- Adds paired PostgreSQL/TiDB migration `0043_semantic_generation_receipts`.
- Adds the schema/catalog entry, migration registry artifacts, migration runner expectations, and
receipt repository transaction support.
## Product diagnostics
- Document compilation continues to expose parsed, outline-built, nodes-generated,
projection-built, evaluated, and published checkpoints.
- Existing document-list failure hover behavior shows the actionable failure reason directly and
keeps the trace id as secondary support information.
- Document outlines are derived from semantic-node section paths and semantic summaries rather
than parser newline rendering.
## Rollout
1. Apply migration 0043 before starting the new API/workers.
2. Deploy API and worker runtime together; missing semantic/Graph runtime dependencies fail startup.
3. Canary new imports and compare chunk coherence, outline localization, retrieval recall, provider
calls, latency, and Graph provenance.
4. Rebuild existing documents through normal reindex/profile-migration candidate publication.
5. Drill publication-head rollback before removing the legacy read path.
The rollout is now executable through guarded, bounded commands:
- `semantic:rollout:static` verifies migration 0043 and its generated registry without network IO.
- `semantic:rollout:preflight` reads health, settings, documents, and failed reindex baselines.
- `semantic:rollout:canary` accepts only explicit document asset ids, polls every accepted job, and
verifies non-empty semantic outline provenance.
- `semantic:rollout:backfill` uses the existing bounded whole-space bulk reindex path.
- `semantic:rollout:rollback` submits a CAS-bound immutable document revision rollback and verifies
that the requested revision becomes active.
- Canary, backfill, and rollback require `SEMANTIC_ROLLOUT_APPLY=1` plus an exact
`semantic:<mode>:<space-id>` confirmation. The script never applies database DDL.
## Verification
Passed locally on 2026-08-13:
- `pnpm --dir knowledge-fs check`, including workspace typechecks/tests, OpenAPI and capability
exports, migration checks, retrieval/phase-4 evaluations, Swagger checks, workflow gates, and
static Docker/Compose smoke checks.
- `pnpm --dir knowledge-fs build` (12/12 tasks).
- `pnpm --dir knowledge-fs test` (22/22 tasks); the API suite passed 4,475 tests with 3 skipped,
API-app passed 252 tests, database passed 114 tests, and adapters passed 105 tests.
- `pnpm --dir knowledge-fs semantic:rollout:test` passed all 5 guarded rollout tests, including
static migration evidence, mutation confirmation, bounded preflight, explicit-document canary,
and package-script registration.
- Retrieval evaluation: recall 0.890, citation 0.880, no-answer rate 0.060, answer accuracy 0.890,
and faithfulness 0.910. Phase-4 regression deltas remained within their configured bounds.
- Repository CI coverage task passed. The semantic compilation change set now reports 90.01%
branch coverage. An additional informational full API coverage run reports 89.48% branches. The
API package is intentionally excluded from the repository CI coverage task; its remaining
historical-package gap stays a tracked cleanup gate before deleting the legacy compiler.
- Biome passed for every changed TypeScript/TSX/JSON source file, and `git diff --check` passed.
- KnowledgeFS contract generation/check passed after regenerating the contract lock from the exact
KnowledgeFS change set.
Known repository-wide baseline:
- `pnpm --dir knowledge-fs lint` is not globally green because of pre-existing Admin formatting
findings, generated capability/OpenAPI formatting findings, and the generated OpenAPI size limit.
None is in the semantic compiler change set; all changed files pass Biome. These unrelated files
were deliberately not rewritten in this iteration.
Not performed locally:
- Production migration, deployment smoke, canary imports, historical-document rebuild/backfill,
sampled shadow comparison, and publication-head rollback drill. Local static/preflight/canary
simulations and guardrail tests are complete, but these production executions remain SSC.7
operational gates and must complete before the legacy read/compiler path is removed.

21
knowledge-fs/.harness/docs/consolidated-iteration-plan.md
View File

@ -1,7 +1,7 @@
# KnowledgeFS Master Iteration Plan
> Created: 2026-06-24
> Updated: 2026-08-07
> Updated: 2026-08-13
> Source directory: `.harness/docs`
> Status: current executable master plan
> Rule: historical plans remain source records; this file is the first document to use
@ -28,6 +28,7 @@ and how each slice should be accepted and verified.
| `.harness/docs/pageindex-research-retrieval-v2-iteration-plan.md` | Document selection, book-like layered Research navigation, Value node queue, interactive/durable policies, degradation, budgets, and human-golden findability. | Active execution plan. Automatic Golden Question generation is explicitly excluded. |
| `.harness/docs/multimodal-knowledgefs-iteration-plan.md` | `DocumentMultimodalManifest`, table/image/code/page inventory, visual assets, thumbnails, VLM answer support, visual embeddings, Admin browser, evals. | Core functional capability implemented. Remaining work is external QA fixtures, provider conformance, and richer trace UX. |
| `.harness/docs/image-query-retrieval-iteration-plan.md` | Image-as-query support: query-side image visual embedding (`inputType: "query"`), gateway image transport with typed degradation, query images in VLM answering, and query image-to-text expansion for deep/research. | Implemented 2026-08-07 for the backend-only scope. Retrieval-time recognition of document images remains explicitly excluded (belongs to ingest-side enrichment reindex). |
| `.harness/docs/semantic-document-compilation-restoration-plan.md` | Restores the pre-monorepo profile-aware LLM semantic chunker and adds layout recomposition, durable semantic receipts, unified outline/index/Graph derivation, diagnostics, and rollout gates. | Active P0 regression-restoration track. It precedes further outline-quality work because current parser boundaries leak into chunks, outlines, and Graph inputs. |
| `.harness/docs/rag-platform-redesign-technical-selection.md` | Architecture source of truth and technology choices. | Updated to reflect this master plan, PageIndex-inspired outlines, native multimodal contracts, and visual indexing. |
| Dify prototype `/datasets` | Product UX target for dataset list/detail, overview readiness, sources, documents, evidence, quality, settings, agent access, and pipeline surfaces. | Used as the Admin/product parity target before deeper quality-only iteration. |
@ -88,7 +89,8 @@ The desired mode is:
| Queryable ingestion | Done | QI.1-QI.5: upload creates nodes, local compute runtime, local generator over nodes, evidence query smoke, Admin BFF upload smoke. | None for this plan. |
| Durable local runtime | Done | DLR.1-DLR.15: PostgreSQL executor, DB repository bundle, migrations, `.env`, durable smoke, application packaging, app Compose guardrails, API/Admin images and smoke gates. | None for this plan. |
| JuiceFS hardening | Done | JH.1-JH.7: manifests, commit ledger, artifact segments, consistency/cache contracts, sessions/leases, fsck/gc/status/stats, quota/projection hardening, Admin/MCP operator UX. | Keep docs/runbooks aligned when related behavior changes. |
| PageIndex outline | V2 backend verification 2026-08-06 | Deterministic schema/builder/repository/API/KnowledgeFS; document shortlist; root-to-leaf layered LLM lane; Value propagation and node queue; per-level/round replay-safe checkpoints; degradation/budget semantics; exact-generation human-golden findability and bounded repair queue. | Automatic Golden Question generation remains explicitly excluded until its product requirements are decided. |
| Semantic document compilation | Local implementation, regression, and rollout automation complete; production execution pending 2026-08-13 | SSC.0-SSC.6 restore profile-aware LLM semantic ranges, immutable source-derived text, durable generation receipts, unified outline/PageIndex/search/Graph derivation, and exact profile-migration replay. SSC.7 provides guarded static/preflight/canary/backfill/rollback commands and tests; the changed semantic chain is above 90% branch coverage. | Apply migration 0043 before workers, run production preflight/canary/backfill/shadow comparison/rollback, close the historical full-API branch gap, then retire the legacy final chunker. |
| PageIndex outline | V2 backend verification 2026-08-06 | Deterministic schema/builder/repository/API/KnowledgeFS; document shortlist; root-to-leaf layered LLM lane; Value propagation and node queue; per-level/round replay-safe checkpoints; degradation/budget semantics; exact-generation human-golden findability and bounded repair queue. | Rebase outline inputs onto restored semantic generations before additional outline-quality tuning. Automatic Golden Question generation remains explicitly excluded until its product requirements are decided. |
| Multimodal KnowledgeFS | Mostly done | Manifest, metadata normalization, asset extraction, PDF rasterization, thumbnails, KnowledgeFS descriptors, VLM answer providers, visual embeddings, visual retrieval metrics, Admin browser, eval utilities. | External QA fixtures, provider conformance packs, richer trace drill-downs. |
| Prototype product parity | Planned | Underlying APIs and data contracts exist in pieces across KnowledgeFS, SourceFS, EvidenceFS, retrieval, quality, and Admin. | Align Admin routes and workflow APIs with the prototype: dataset list/detail shell, sources, documents, evidence, quality, settings, agent access, and pipeline mode. |
| Admin integration | Active | AIR.1-AIR.2 done: upload/readiness/citation paths and local/Compose upstream wiring repaired. | AIR.3/AIR.4: preview panel audit and outline/multimodal trace UX. |
@ -101,13 +103,14 @@ Work should proceed in this order unless a production regression appears:
1. Documentation alignment and planning source of truth.
2. Prototype product surface parity for the dataset workspace.
3. PageIndex outline quality hardening.
4. Multimodal functional completion.
5. Admin integration honesty and trace drill-downs.
6. API/code-health closure.
7. Research-mode completeness over outline + multimodal + graph evidence.
8. Evaluation governance and CI regression hardening.
9. Optional provider, deployment, and adapter expansion.
3. Restore semantic document compilation (SSC.0-SSC.7).
4. PageIndex outline quality hardening on semantic generations.
5. Multimodal functional completion.
6. Admin integration honesty and trace drill-downs.
7. API/code-health closure.
8. Research-mode completeness over outline + multimodal + graph evidence.
9. Evaluation governance and CI regression hardening.
10. Optional provider, deployment, and adapter expansion.
Every slice should:

180
knowledge-fs/.harness/docs/semantic-document-compilation-restoration-plan.md Normal file
View File

@ -0,0 +1,180 @@
# Semantic Document Compilation Restoration Plan
> Created: 2026-08-13
> Status: Implementation and local regression complete; production rollout verification active
> Owner boundary: KnowledgeFS TypeScript ingestion/compiler, Dify Admin integration
> Historical implementation reference: standalone KnowledgeFS commit `b3aa9ce`
## 1. Why this restoration exists
The product contract requires document compilation to parse the source, reconstruct a trustworthy
reading order, ask the knowledge-space reasoning model for semantic boundaries, and derive the
outline, summaries, vector/full-text projections, and Graph facts from the same immutable semantic
generation.
The monorepo currently persists the parser artifact and sends it directly to the deterministic
1,200-grapheme chunker. The reasoning model enriches an already-built outline and later extracts
Graph facts, so parser heading mistakes become hard chunk boundaries. The semantic chunking
implementation previously shipped in the standalone KnowledgeFS repository was not carried into
the monorepo migration. This track restores that behavior and extends it with layout recomposition
so forms, invoices, tables, and multi-column documents do not preserve false parser boundaries.
## 2. Non-negotiable invariants
1. Raw parser output remains immutable and auditable.
2. Normalized elements retain source element ids, pages, byte offsets, and bounding boxes.
3. The reasoning model selects source ranges; it never supplies authoritative chunk text.
4. Every eligible source unit is covered exactly once by leaf chunks. Gaps, overlap, reordering,
invalid ids, and over-limit output fail closed.
5. Tables/images stay atomic unless a bounded deterministic safety split is unavoidable.
6. The model selection, capability identity, prompt version, input fingerprint, output fingerprint,
and window manifest are frozen with the candidate publication.
7. A retry reuses or proves the complete generation; it cannot append duplicate nodes or Graph rows.
8. Outline, summary, PageIndex, dense/FTS projections, and Graph facts bind to one publication
generation and publish atomically.
9. Provider/configuration failures remain actionable. No silent fallback to legacy character
chunking is allowed for a profile that requires semantic compilation.
10. All model/network/memory/database work is bounded and batch-oriented.
## 3. Target pipeline
```text
source bytes
-> immutable ParseArtifact
-> deterministic LayoutRecompositionArtifact
-> frozen reasoning-model SemanticSegmentationPlan
-> fail-closed coverage/provenance validation
-> immutable semantic KnowledgeNode generation + receipt
-> outline/summary + PageIndex
-> dense/FTS/metadata projections + Graph facts
-> candidate evaluation
-> atomic publication head CAS
```
## 4. Execution slices
| ID | Status | Slice | Required behavior | Regression gate |
|---|---|---|---|---|
| SSC.0 | Complete 2026-08-13 | Baseline and migration audit | Restore the historical design record, create redacted structured-document fixtures, map current compiler/publication contracts, and prove the existing deterministic path reproduces fragmented output. | Focused parser/chunker/compiler tests fail for the new semantic contract before implementation. |
| SSC.1 | Complete 2026-08-13 | Layout normalization and boundary recomposition | Reuse the parser's bounded coordinate normalization for vertical CJK/noise, keep canonical element order and tables, classify Unstructured heading confidence, and preserve complete source provenance without mutating the stored parse artifact. | Invoice/form, trusted/untrusted heading, native parser, table isolation, and bounded-element tests. Multi-column reading order remains the parser provider's responsibility and is not silently reordered downstream. |
| SSC.2 | Complete 2026-08-13 | LLM semantic plan | Restore the profile-aware semantic chunker, bounded windows/look-ahead, structured output, joint entity/relation extraction, terminal model identity verification, and deterministic text materialization. Extend its prompt input with normalized structural hints. | Provider-boundary tests for natural boundaries, Unicode, caps, invalid JSON, incomplete coverage, wrong model identity, response limits, and retry replay. |
| SSC.3 | Complete 2026-08-13 | Durable generation | Persist a compact immutable semantic generation receipt and complete node generation transactionally in bounded batches; add PostgreSQL/TiDB migration and schema/index guards. | Repository, migration replay, all-excluded, conflicting replay, batch, identity, and size-bound tests. |
| SSC.4 | Complete 2026-08-13 | Compiler integration | Freeze the reasoning profile at admission, run recomposition and segmentation before projections, resume safely from checkpoints, and keep the published generation readable until the candidate is complete. | Worker success/failure/resume, deletion fence, profile migration, publication CAS, and no-legacy-fallback tests. |
| SSC.5 | Complete 2026-08-13 | Unified derived artifacts | Build outline hierarchy/summaries, PageIndex, dense/FTS projections, and Graph facts from final semantic nodes. Joint facts are quality-controlled and replayed for embedding-only migrations without another LLM call. | Exact-generation outline/path/Graph tests, duplicate prevention, source-node provenance, and profile-migration regression fixtures. |
| SSC.6 | Complete 2026-08-13 | Product diagnostics | Preserve compilation stages and semantic provenance; expose actionable document failure text on status hover while keeping technical trace ids secondary. The document outline now consumes semantic-node section paths and summaries. | Existing Admin hover/component coverage plus API worker/outline provenance tests. |
| SSC.7 | Local automation complete; production execution pending | Rollout and cleanup | The semantic compiler and receipt are versioned and fail closed. Static migration evidence, read-only preflight, explicit-document canary, bounded backfill, task polling, outline verification, retrieval probing, and revision rollback are available through guarded operator commands. Rebuild existing documents through normal reindex/profile-migration candidate publication. Do not delete the legacy implementation until production comparison is accepted. | Full checks, contract lock, rollout-script tests, deployment smoke, sampled shadow comparison, reindex/rebuild idempotency, rollback, and mixed-version reads. |
## 4.1 Execution checkpoint and rollout order
The code path is implemented in dependency order. The remaining work is operational verification,
not an untracked compiler shortcut:
1. **Final local gates (complete 2026-08-13)** — formatting, typecheck, focused and full package
tests, CI coverage, database migration registry, OpenAPI/contract lock, and build are complete.
The unrelated repository-wide lint baseline and informational API branch-coverage gap are
recorded in the change note.
2. **Deploy schema first** — apply `0043_semantic_generation_receipts` to PostgreSQL/TiDB before
workers that can persist semantic receipts are started.
3. **Deploy API and workers together** — the runtime fails startup when semantic compilation is
configured without the reasoning provider or synchronous Graph materializer. This prevents a
mixed deployment from silently reverting to fixed-size chunking.
4. **Canary new imports** — compare chunk coherence, outline localization, Graph provenance,
provider calls, latency, and retrieval recall on synthetic structured documents and redacted
operator samples.
5. **Rebuild existing documents** — use the normal candidate reindex/profile-migration path.
Embedding-only changes clone the immutable semantic node generation; reasoning changes build a
new semantic generation. Publication remains an atomic head CAS.
6. **Rollback** — keep the previous published projection set readable until the candidate passes
evaluation. Roll back the publication head; never mutate or partially append to the old node
generation.
7. **Legacy cleanup** — remove deterministic final chunking only after production canaries,
existing-document rebuilds, and rollback drills pass. Until then it remains readable for old
generations but is not a fallback for newly admitted semantic profiles.
### Guarded rollout commands
Every mutating command requires both `SEMANTIC_ROLLOUT_APPLY=1` and an exact, space-scoped
confirmation string. Responses and polling are bounded. Tokens are read from the environment and
are never printed.
```bash
# Repository/migration-registry evidence only; no network or mutation.
pnpm --dir knowledge-fs semantic:rollout:static
# Read-only health, settings, document and failed-reindex baseline.
SEMANTIC_ROLLOUT_SPACE_ID=<space-uuid> \
SEMANTIC_ROLLOUT_API_BASE=<knowledge-fs-api> \
SEMANTIC_ROLLOUT_AUTH_TOKEN=<operator-token> \
pnpm --dir knowledge-fs semantic:rollout:preflight
# Explicit-document canary. Add SEMANTIC_ROLLOUT_QUERY for a Research retrieval assertion.
SEMANTIC_ROLLOUT_SPACE_ID=<space-uuid> \
SEMANTIC_ROLLOUT_DOCUMENT_IDS=<asset-uuid>[,<asset-uuid>...] \
SEMANTIC_ROLLOUT_APPLY=1 \
SEMANTIC_ROLLOUT_CONFIRM=semantic:canary:<space-uuid> \
pnpm --dir knowledge-fs semantic:rollout:canary
# Whole-space bounded backfill through the existing bulk-reindex/candidate publication path.
SEMANTIC_ROLLOUT_SPACE_ID=<space-uuid> \
SEMANTIC_ROLLOUT_APPLY=1 \
SEMANTIC_ROLLOUT_CONFIRM=semantic:backfill:<space-uuid> \
pnpm --dir knowledge-fs semantic:rollout:backfill
# Roll a logical document back to a known prior immutable revision and verify activation.
SEMANTIC_ROLLOUT_SPACE_ID=<space-uuid> \
SEMANTIC_ROLLOUT_ROLLBACK_DOCUMENT_ID=<logical-document-uuid> \
SEMANTIC_ROLLOUT_ROLLBACK_REVISION=<prior-revision> \
SEMANTIC_ROLLOUT_APPLY=1 \
SEMANTIC_ROLLOUT_CONFIRM=semantic:rollback:<space-uuid> \
pnpm --dir knowledge-fs semantic:rollout:rollback
```
Migration 0043 still must be applied by the deployment system before API/workers are rolled out;
the operator script verifies checked-in evidence but intentionally does not receive database
credentials or execute production DDL.
## 5. Redacted fixture matrix
The real customer invoice must never enter the repository. A generated fixture with equivalent
geometry and synthetic names/identifiers covers:
- one-page Chinese invoice/form with vertical labels and a line-item table;
- legitimate native headings versus low-confidence PDF title classifications;
- two-column reading order and cross-page continuation;
- a table larger than one model window;
- scanned/OCR text with repeated page noise;
- an empty document and a single over-limit atomic element;
- long CJK/emoji grapheme boundaries;
- model output with gaps, overlap, reordered ids, invented ids, and altered text attempts.
## 6. Acceptance thresholds
- Eligible source coverage: exactly 100%.
- Unproven generated source text: 0 bytes.
- Duplicate leaf coverage: 0.
- Orphan one-character CJK layout fragments in the invoice fixture: 0.
- Same immutable request retry: no new provider call after a complete receipt exists.
- Same generation retry: byte-identical node ids/text/offsets/ACL/provenance.
- Graph entity/relation rows: generation-scoped, source-linked, and duplicate-free.
- Failed candidate publication: previous published generation remains queryable.
- Every provider request/response, node count, window count, receipt size, and SQL batch is bounded.
- Every CI-enforced coverage package remains at or above its repository threshold. The semantic
compilation change set is at 90.01% branch coverage. The API package is currently excluded from
the repository CI coverage task; its informational full-suite historical baseline is now 89.48%
and must reach 90% before legacy compiler cleanup.
## 7. Verification cadence
Each slice follows RED -> GREEN -> REFACTOR and records its result under `.harness/changes`.
Targeted package tests run after each behavior change. Before completion run:
```text
pnpm --dir knowledge-fs check
pnpm --dir knowledge-fs build
pnpm --dir knowledge-fs lint
pnpm --dir knowledge-fs db:migrations:check
```
The monorepo KnowledgeFS contract lock is regenerated only after all tracked KnowledgeFS changes
and generated artifacts are final and staged. Any skipped gate and its reason must be recorded in
the change summary.

17
knowledge-fs/apps/api/src/document-compilation-runtime-options.test.ts
View File

@ -60,6 +60,18 @@ describe("createApiDocumentCompilationRuntime", () => {
parser: {} as never,
repositories: {},
}),
).toThrow("requires the Reasoning-model semantic chunker");
expect(() =>
createApiDocumentCompilationRuntime({
adapter,
compute: {} as never,
config,
embeddingResolver: undefined,
parser: {} as never,
repositories: {},
semanticChunker: {} as never,
}),
).toThrow("requires the per-space plugin embedding resolver");
expect(() =>
@ -71,6 +83,7 @@ describe("createApiDocumentCompilationRuntime", () => {
modelCapabilityPreflight: {} as never,
parser: {} as never,
repositories: {},
semanticChunker: {} as never,
}),
).toThrow("requires the atomic initial profile activation repository");
@ -83,6 +96,7 @@ describe("createApiDocumentCompilationRuntime", () => {
initialProfileActivations: {} as never,
parser: {} as never,
repositories: {},
semanticChunker: {} as never,
}),
).toThrow("requires model capability preflight");
@ -96,6 +110,7 @@ describe("createApiDocumentCompilationRuntime", () => {
modelCapabilityPreflight: {} as never,
parser: {} as never,
repositories: {},
semanticChunker: {} as never,
}),
).toThrow("requires database repository: artifacts");
});
@ -122,6 +137,7 @@ describe("createApiDocumentCompilationRuntime", () => {
assets: required(gateway.documentAssets),
attempts: required(databaseRepositories.documentCompilationAttempts),
chunks: required(gateway.documentChunks),
graph: required(gateway.graphIndex),
legacyBootstraps: required(databaseRepositories.legacySpacePublicationBootstraps),
pageIndexUpgradeBackfills: required(databaseRepositories.pageIndexUpgradeBackfills),
logicalDocuments: required(gateway.logicalDocuments),
@ -137,6 +153,7 @@ describe("createApiDocumentCompilationRuntime", () => {
settings: required(gateway.documentSettings),
tasks: required(gateway.documentProcessingTasks),
},
semanticChunker: {} as never,
});
expect(assembly).toMatchObject({

26
knowledge-fs/apps/api/src/document-compilation-runtime-options.ts
View File

@ -47,6 +47,7 @@ import {
type ParseArtifactRepository,
type ProjectionSetPublicationMemberRepository,
type ProjectionSetPublicationRepository,
type SemanticChunker,
createDatabaseDocumentCompilationCandidateValidator,
createDatabaseDocumentCompilationIndexOverrideResolver,
createDatabaseDocumentLogicalMutationReconciler,
@ -72,6 +73,7 @@ import {
createDurableDocumentCompilationJobStateMachine,
createFtsProjectionBuilder,
createIncrementalReindexer,
createJointSemanticGraphMaterializer,
createKnowledgeSpaceProfileMigrationRuntime,
createLegacySpacePublicationBootstrapRuntime,
createLegacySpacePublicationBootstrapService,
@ -173,6 +175,7 @@ export interface CreateApiDocumentCompilationRuntimeOptions {
})
| undefined;
readonly semanticMetrics?: DocumentSemanticEnrichmentOperationalMetrics | undefined;
readonly semanticChunker?: SemanticChunker | undefined;
readonly visual?:
| {
readonly model: string;
@ -253,6 +256,7 @@ export function createApiDocumentCompilationRuntime({
repositories: partialRepositories,
semantic,
semanticMetrics,
semanticChunker,
visual,
}: CreateApiDocumentCompilationRuntimeOptions): ApiDocumentCompilationRuntimeAssembly | undefined {
if (!config) {
@ -261,6 +265,9 @@ export function createApiDocumentCompilationRuntime({
if (!compute) {
throw new Error("Document compilation runtime requires an in-process compute runtime");
}
if (!semanticChunker) {
throw new Error("Document compilation runtime requires the Reasoning-model semantic chunker");
}
if (!embeddingResolver) {
throw new Error(
"Document compilation runtime requires the per-space plugin embedding resolver",
@ -363,7 +370,7 @@ export function createApiDocumentCompilationRuntime({
},
publications: repositories.publications,
versions: {
chunkerVersion: "knowledge-compute-chunker-v1",
chunkerVersion: "knowledge-llm-semantic-chunker-v1",
indexVersion: "knowledge-index-v1",
nodeSchemaVersion: 1,
parserPolicyVersion: "configured-parser-v1",
@ -402,6 +409,7 @@ export function createApiDocumentCompilationRuntime({
maxProjectionBatchSize: embeddingBatchSize,
nodes: repositories.nodes,
projections: repositories.projections,
semanticChunker,
...(visual
? {
visualBuilder: createVisualEmbeddingProjectionBuilder({
@ -417,6 +425,19 @@ export function createApiDocumentCompilationRuntime({
maxNodes: maxDocumentNodes,
maxSummaryChars: 2_000,
});
const jointSemanticGraph =
semanticChunker && repositories.graph
? createJointSemanticGraphMaterializer({
graph: repositories.graph,
maxEntitiesPerNode: semantic?.semanticEntityExtractionMaxEntitiesPerNode ?? 50,
maxNodesPerArtifact: maxDocumentNodes,
maxRelationsPerNode: semantic?.semanticRelationExtractionMaxRelationsPerNode ?? 50,
nodes: repositories.nodes,
})
: undefined;
if (semanticChunker && !jointSemanticGraph) {
throw new Error("Semantic document compilation requires the graph repository");
}
if (profileMigration && !outlineSummaryEnhancer) {
throw new Error(
"Profile migration runtime requires the profile-aware PageIndex Summary enhancer",
@ -441,12 +462,14 @@ export function createApiDocumentCompilationRuntime({
outlineSummaryEnhancer,
outlines: repositories.outlines,
pageIndexBuild,
paths: repositories.paths,
profiles: repositories.profiles,
projections: {
getMany: repositories.projections.getMany.bind(repositories.projections),
},
publications: repositories.publications,
reindexer,
...(jointSemanticGraph ? { semanticGraph: jointSemanticGraph } : {}),
snapshots: createDatabaseKnowledgeSpaceProfileMigrationCandidateSnapshotRepository({
database: adapter.database,
maxMembers: maxCandidateComponents,
@ -516,6 +539,7 @@ export function createApiDocumentCompilationRuntime({
failureManagement: "caller",
generateKnowledgePathId: randomUUID,
jobs,
...(jointSemanticGraph ? { jointSemanticGraph } : {}),
indexOverrides: documentIndexOverrides,
knowledgePaths: repositories.paths,
...(multimodal?.documentMultimodalImageVariantGenerator

6
knowledge-fs/apps/api/src/index.ts
View File

@ -21,6 +21,7 @@ import {
createKnowledgeSpaceOutlineSummaryEnhancer,
createLlmAnswerQueryGenerator,
createLlmAutoRetrievalModeResolver,
createLlmSemanticChunker,
createModelCapabilityPreflight,
createPageIndexFindabilityEvaluator,
createPageIndexLayeredTreeSearch,
@ -433,6 +434,10 @@ const documentOutlineSummaryEnhancer = createKnowledgeSpaceOutlineSummaryEnhance
modelRequestGate: ingestionModelRuntimeOptions.modelRequestGate,
providerFactory: profileReasoningCapability.providerFactory,
});
const documentSemanticChunker = createLlmSemanticChunker({
maxNodes: 20_000,
reasoningProviderFactory: profileReasoningCapability.providerFactory,
});
const relevanceTriageOptions = createApiRelevanceTriageOptions({
...(repositoryOptions.documentAssets ? { documentAssets: repositoryOptions.documentAssets } : {}),
...(repositoryOptions.documentOutlines
@ -590,6 +595,7 @@ const documentCompilationRuntime = createApiDocumentCompilationRuntime({
semanticExtractionMaxConcurrency: ingestionModelRuntimeOptions.semanticExtractionMaxConcurrency,
},
semanticMetrics: operationalMetrics.semanticEnrichment,
semanticChunker: documentSemanticChunker,
...(visualEmbeddingOptions
? {
visual: {

6
knowledge-fs/apps/api/src/migrate.test.ts
View File

@ -133,6 +133,8 @@ describe("runApiDatabaseMigrations", () => {
"insert",
"schema",
"insert",
"schema",
"insert",
]);
expect(migrationSql).toHaveLength(expectedPostgresMigrationIds.length);
expect(migrationSql[2]).toContain("-- Migration id: 0003_projection_set_publications\n");
@ -192,6 +194,10 @@ describe("runApiDatabaseMigrations", () => {
expect(migrationSql[39]).toContain("-- Migration id: 0040_knowledge_space_metadata\n");
expect(migrationSql[40]).toContain("-- Migration id: 0041_logical_document_availability\n");
expect(migrationSql[41]).toContain("-- Migration id: 0042_workflow_failed_retrieval_capture\n");
expect(migrationSql[42]).toContain("-- Migration id: 0043_semantic_generation_receipts\n");
expect(migrationSql[42]).toContain(
'CREATE TABLE IF NOT EXISTS "knowledge_node_generation_receipts"',
);
expect(closed).toBe(true);
});

8
knowledge-fs/package.json
View File

@ -12,7 +12,7 @@
"scripts": {
"build": "turbo run build",
"capability:export": "node --import tsx scripts/export-capability-v2-operations.mjs",
"check": "pnpm typecheck && pnpm test && pnpm openapi:export:test && pnpm test:coverage:ci && pnpm eval:regression && pnpm eval:phase4 && pnpm swagger:test && pnpm db:migrations:check && pnpm p9:bundle:test && pnpm ci:workflow:test && pnpm compose:middleware:config && pnpm compose:config && pnpm dify:compose:config && pnpm compose:middleware:test && pnpm compose:apps:test && pnpm docker:context:test && pnpm local:happy-path:test && pnpm docker:api:bundle-smoke:test && pnpm docker:admin:http-smoke:test && pnpm docker:apps:smoke:test",
"check": "pnpm typecheck && pnpm test && pnpm openapi:export:test && pnpm test:coverage:ci && pnpm eval:regression && pnpm eval:phase4 && pnpm swagger:test && pnpm db:migrations:check && pnpm p9:bundle:test && pnpm ci:workflow:test && pnpm compose:middleware:config && pnpm compose:config && pnpm dify:compose:config && pnpm compose:middleware:test && pnpm compose:apps:test && pnpm docker:context:test && pnpm local:happy-path:test && pnpm semantic:rollout:test && pnpm docker:api:bundle-smoke:test && pnpm docker:admin:http-smoke:test && pnpm docker:apps:smoke:test",
"ci:workflow:test": "node --test scripts/github-actions-workflow.test.mjs scripts/secret-scan.test.mjs",
"compose:config": "docker compose --env-file infra/local/.env.example -f infra/local/compose.yaml config",
"compose:apps:test": "node --test scripts/compose-apps.test.mjs",
@ -49,6 +49,12 @@
"p9:bundle:test": "node --test scripts/build-p9-removal-bundle.test.mjs",
"security:dependencies": "node scripts/audit-backend-dependencies.mjs",
"security:secrets": "node scripts/secret-scan.mjs",
"semantic:rollout:backfill": "SEMANTIC_ROLLOUT_MODE=backfill node scripts/semantic-compilation-rollout.mjs",
"semantic:rollout:canary": "SEMANTIC_ROLLOUT_MODE=canary node scripts/semantic-compilation-rollout.mjs",
"semantic:rollout:preflight": "SEMANTIC_ROLLOUT_MODE=preflight node scripts/semantic-compilation-rollout.mjs",
"semantic:rollout:rollback": "SEMANTIC_ROLLOUT_MODE=rollback node scripts/semantic-compilation-rollout.mjs",
"semantic:rollout:static": "node scripts/semantic-compilation-rollout.mjs",
"semantic:rollout:test": "node --test scripts/semantic-compilation-rollout.test.mjs",
"swagger": "node tools/swagger/server.mjs",
"swagger:test": "node --test tools/swagger/server.test.mjs",
"test": "turbo run test",

2
knowledge-fs/packages/adapters/src/migration-runner.test.ts
View File

@ -50,6 +50,7 @@ const documentSemanticEnrichmentMigrationId = "0039_document_semantic_enrichment
const knowledgeSpaceMetadataMigrationId = "0040_knowledge_space_metadata";
const logicalDocumentAvailabilityMigrationId = "0041_logical_document_availability";
const workflowFailedRetrievalCaptureMigrationId = "0042_workflow_failed_retrieval_capture";
const semanticGenerationReceiptsMigrationId = "0043_semantic_generation_receipts";
const migrationsAfterDurableDeletion = [
versionedSpaceProfilesMigrationId,
profilePublicationBindingsMigrationId,
@ -76,6 +77,7 @@ const migrationsAfterDurableDeletion = [
knowledgeSpaceMetadataMigrationId,
logicalDocumentAvailabilityMigrationId,
workflowFailedRetrievalCaptureMigrationId,
semanticGenerationReceiptsMigrationId,
] as const;
const migrationsAfterTidbBaselineRepair = [
spaceAccessControlMigrationId,

1
knowledge-fs/packages/api/package.json
View File

@ -21,6 +21,7 @@
"hono": "^4.12.25",
"jose": "^5.10.0",
"sharp": "^0.35.3",
"unicode-segmenter": "0.15.0",
"zod": "^3.24.1"
},
"devDependencies": {

90
knowledge-fs/packages/api/src/document-compilation-worker.test.ts
View File

@ -371,6 +371,12 @@ describe("createDocumentCompilationWorker lease integration", () => {
const semanticAdmissions: unknown[] = [];
let semanticCalls = 0;
let smokeCalls = 0;
const outlines = createInMemoryDocumentOutlineRepository({ maxOutlines: 2 });
const knowledgePaths = createInMemoryKnowledgePathRepository({
maxBatchSize: 10,
maxListLimit: 10,
maxPaths: 10,
});
const worker = createDocumentCompilationWorker({
assets,
candidateComposer: {
@ -412,13 +418,12 @@ describe("createDocumentCompilationWorker lease integration", () => {
"018f0d60-7a49-7cc2-9c1b-5b36f18f6a36",
"018f0d60-7a49-7cc2-9c1b-5b36f18f6a37",
"018f0d60-7a49-7cc2-9c1b-5b36f18f6a38",
"018f0d60-7a49-7cc2-9c1b-5b36f18f6a3b",
"018f0d60-7a49-7cc2-9c1b-5b36f18f6a3c",
"018f0d60-7a49-7cc2-9c1b-5b36f18f6a3d",
]),
jobs: compilationJobs,
knowledgePaths: createInMemoryKnowledgePathRepository({
maxBatchSize: 10,
maxListLimit: 10,
maxPaths: 10,
}),
knowledgePaths,
multimodalManifests: createInMemoryDocumentMultimodalManifestRepository({
maxManifests: 2,
}),
@ -428,7 +433,7 @@ describe("createDocumentCompilationWorker lease integration", () => {
maxNodes: 10,
maxSummaryChars: 200,
}),
outlines: createInMemoryDocumentOutlineRepository({ maxOutlines: 2 }),
outlines,
pageIndexBuild: {
materializeBuilding: async ({ outline }) => {
pageIndexBuildCalls += 1;
@ -460,6 +465,19 @@ describe("createDocumentCompilationWorker lease integration", () => {
artifact: input.parseArtifact,
nodeIds: ["018f0d60-7a49-7cc2-9c1b-5b36f18f6a39"],
nodesCreated: 1,
outlineArtifact: ParseArtifactSchema.parse({
...input.parseArtifact,
elements: [
{
id: "018f0d60-7a49-7cc2-9c1b-5b36f18f6a39",
metadata: { semanticSectionSummary: "发票身份、购买方和金额信息。" },
sectionPath: ["电子发票", "购买方与金额"],
text: "发票号码、购买方与价税合计",
type: "paragraph",
},
],
metadata: { semanticCompilation: { source: "llm-semantic-v1" } },
}),
projectionIds: ["018f0d60-7a49-7cc2-9c1b-5b36f18f6a3a"],
projectionsCreated: 1,
status: "rebuilt",
@ -471,6 +489,21 @@ describe("createDocumentCompilationWorker lease integration", () => {
semanticAdmissions.push(input);
},
},
jointSemanticGraph: {
materialize: async () => {
semanticCalls += 1;
return {
entitiesExtracted: 1,
graphEntityIds: ["018f0d60-7a49-7cc2-9c1b-5b36f18f6a3b"],
graphEntitiesIndexed: 1,
graphRelationIds: ["018f0d60-7a49-7cc2-9c1b-5b36f18f6a3c"],
graphRelationsIndexed: 1,
nodesScanned: 1,
semanticProviderCalls: 0,
semanticProviderCallsMaximum: 0,
};
},
},
semanticPostProcessor: {
process: async () => {
semanticCalls += 1;
@ -509,8 +542,18 @@ describe("createDocumentCompilationWorker lease integration", () => {
expect.objectContaining({
componentReceipt: {
documentOutlines: [expect.objectContaining({ generationId })],
graphEntities: [],
graphRelations: [],
graphEntities: [
{
componentKey: "018f0d60-7a49-7cc2-9c1b-5b36f18f6a3b",
generationId,
},
],
graphRelations: [
{
componentKey: "018f0d60-7a49-7cc2-9c1b-5b36f18f6a3c",
generationId,
},
],
indexProjections: [
{
componentKey: "018f0d60-7a49-7cc2-9c1b-5b36f18f6a3a",
@ -528,18 +571,33 @@ describe("createDocumentCompilationWorker lease integration", () => {
expect(smokeCalls).toBe(0);
expect(mutableEmbeddingReads).toBe(0);
expect(pageIndexBuildCalls).toBe(0);
expect(semanticAdmissions).toEqual([
expect(semanticAdmissions).toEqual([]);
expect(semanticCalls).toBe(1);
expect(reindexInputs[0]).toEqual(
expect.objectContaining({
documentAssetId: asset.id,
parseArtifactId: "018f0d60-7a49-7cc2-9c1b-5b36f18f6a02",
publicationGenerationId: generationId,
enableGraph: true,
language: "zh-CN",
retrievalProfile: expect.objectContaining({ revision: 4 }),
skipDense: true,
}),
]);
expect(semanticCalls).toBe(0);
expect(reindexInputs[0]).toEqual(expect.objectContaining({ language: "zh-CN" }));
);
expect(reindexInputs[0]).not.toHaveProperty("denseModel");
expect(reindexInputs[0]).not.toHaveProperty("embeddingProfile");
await expect(
outlines.getByDocumentVersion({
documentAssetId: asset.id,
publicationGenerationId: generationId,
version: asset.version,
}),
).resolves.toMatchObject({
metadata: expect.objectContaining({ builder: "semantic-knowledge-nodes" }),
nodes: [
expect.objectContaining({
sectionPath: ["电子发票"],
children: [expect.objectContaining({ sectionPath: ["电子发票", "购买方与金额"] })],
}),
],
});
await expect(
assets.get({ id: asset.id, knowledgeSpaceId: asset.knowledgeSpaceId }),
).resolves.toMatchObject({ parserStatus: "pending" });
@ -1342,7 +1400,9 @@ describe("createDocumentCompilationWorker lease integration", () => {
expect.objectContaining({
denseModel: frozenEmbeddingProfile.vectorSpaceId,
embeddingProfile: frozenEmbeddingProfile,
enableGraph: true,
projectionStatus: "building",
retrievalProfile: frozenRetrievalProfile,
tenantId: "tenant-1",
}),
]);

94
knowledge-fs/packages/api/src/document-compilation-worker.ts
View File

@ -49,6 +49,7 @@ import {
type DocumentPdfRasterizer,
rasterizeDocumentPdfMultimodalAssets,
} from "./document-pdf-rasterizer";
import type { JointSemanticGraphMaterializer } from "./document-semantic-enrichment-processor";
import { logDocumentUploadDiagnostic } from "./document-upload-diagnostics";
import type { IncrementalReindexer } from "./index-reindexer";
import type { KnowledgeFsOperationLeaseCoordinator } from "./knowledge-fs-operation-leases";
@ -89,6 +90,7 @@ export interface DocumentCompilationWorkerOptions {
readonly failureManagement?: "caller" | "worker" | undefined;
readonly generateKnowledgePathId?: (() => string) | undefined;
readonly jobs: DocumentCompilationJobStateMachine;
readonly jointSemanticGraph?: JointSemanticGraphMaterializer | undefined;
readonly knowledgePaths?: KnowledgePathRepository | undefined;
readonly multimodalImageVariantGenerator?: DocumentImageVariantGenerator | undefined;
readonly multimodalLocalAssetAllowlist?: readonly string[] | undefined;
@ -220,6 +222,7 @@ export function createDocumentCompilationWorker({
failureManagement = "worker",
generateKnowledgePathId,
jobs,
jointSemanticGraph,
knowledgePaths,
multimodalImageVariantGenerator,
multimodalLocalAssetAllowlist,
@ -418,6 +421,9 @@ export function createDocumentCompilationWorker({
let documentOutlineIds: readonly string[] = [];
let knowledgePathIds: readonly string[] = [];
let persistedManifest: DocumentMultimodalManifest;
const deferOutlineUntilSemanticNodes = Boolean(
publicationGenerationId && frozenRetrievalProfile,
);
if (resumeOutlineGeneration && publicationGenerationId) {
const [persistedOutline, resumedManifest] = await Promise.all([
outlines?.getByDocumentVersion({
@ -461,7 +467,7 @@ export function createDocumentCompilationWorker({
knowledgeSpaceId: input.knowledgeSpaceId,
...(publicationGenerationId ? { publicationGenerationId } : {}),
});
if (outlineBuilder && outlines) {
if (outlineBuilder && outlines && !deferOutlineUntilSemanticNodes) {
const deterministicOutline = outlineBuilder.build({
knowledgeSpaceId: input.knowledgeSpaceId,
parseArtifact: canonicalArtifact,
@ -504,8 +510,10 @@ export function createDocumentCompilationWorker({
}
await assertWritable();
persistedManifest = await multimodalManifests.upsert(multimodalManifest);
await assertWritable();
await jobs.advance(input.documentCompilationJobId, "outline_built");
if (!deferOutlineUntilSemanticNodes) {
await assertWritable();
await jobs.advance(input.documentCompilationJobId, "outline_built");
}
}
const resolvedEmbedding = frozenEmbeddingProfile
@ -531,6 +539,7 @@ export function createDocumentCompilationWorker({
? { excludedNodeOrdinals: documentIndexOverrides.excludedNodeOrdinals }
: {}),
...(frozenEmbeddingProfile ? { embeddingProfile: frozenEmbeddingProfile } : {}),
enableGraph: documentIndexOverrides.enableGraph !== false,
knowledgeSpaceId: input.knowledgeSpaceId,
...(documentIndexOverrides.language
? { language: documentIndexOverrides.language }
@ -541,11 +550,70 @@ export function createDocumentCompilationWorker({
publicationGenerationId || legacyStagedProjectionPublication ? "building" : "ready",
projectionVersion: input.version,
...(publicationGenerationId ? { publicationGenerationId } : {}),
...(frozenRetrievalProfile ? { retrievalProfile: frozenRetrievalProfile } : {}),
...(initialJob.stage === "outline_built" ? { resetFailedProjections: true } : {}),
...(signal ? { signal } : {}),
...(frozenRetrievalProfile && !resolvedEmbedding ? { skipDense: true as const } : {}),
tenantId: input.tenantId,
...(visualEmbeddingModel ? { visualModel: visualEmbeddingModel } : {}),
});
if (
deferOutlineUntilSemanticNodes &&
!resumeOutlineGeneration &&
publicationGenerationId
) {
if (
reindexResult.status !== "rebuilt" ||
!reindexResult.outlineArtifact ||
!outlineBuilder ||
!outlines
) {
throw new Error(
"Generation-scoped semantic compilation requires a semantic outline artifact",
);
}
const deterministicOutline = outlineBuilder.build({
knowledgeSpaceId: input.knowledgeSpaceId,
parseArtifact: reindexResult.outlineArtifact,
publicationGenerationId,
});
const outline = outlineSummaryEnhancer
? await outlineSummaryEnhancer.enhance({
outline: deterministicOutline,
parseArtifact: reindexResult.outlineArtifact,
retrievalProfile: frozenRetrievalProfile,
...(signal ? { signal } : {}),
tenantId: input.tenantId,
})
: deterministicOutline;
await assertWritable();
const persistedOutline = await outlines.upsert(outline);
if (documentIndexOverrides.enablePageIndex !== false) {
await assertWritable();
await pageIndexBuild?.materializeBuilding({
builtAt: persistedOutline.updatedAt ?? persistedOutline.createdAt,
outline: persistedOutline,
tenantId: input.tenantId,
});
}
documentOutlineIds = [persistedOutline.id];
if (knowledgePaths && generateKnowledgePathId) {
await assertWritable();
const persistedPaths = await knowledgePaths.upsertMany(
buildCompilationKnowledgePaths({
asset: activeAsset,
generateId: generateKnowledgePathId,
manifest: persistedManifest,
outline: persistedOutline,
publicationGenerationId,
tenantId: input.tenantId,
}),
);
knowledgePathIds = persistedPaths.map((path) => path.id);
}
await assertWritable();
await jobs.advance(input.documentCompilationJobId, "outline_built");
}
await assertWritable();
if (legacyStagedProjectionPublication && reindexResult.status === "rebuilt") {
stagedProjectionIds = [...(reindexResult.projectionIds ?? [])];
@ -580,8 +648,28 @@ export function createDocumentCompilationWorker({
await jobs.advance(input.documentCompilationJobId, "nodes_generated");
let graphEntityIds: readonly string[] = [];
let graphRelationIds: readonly string[] = [];
if (
jointSemanticGraph &&
publicationGenerationId &&
frozenRetrievalProfile &&
reindexResult.status === "rebuilt" &&
documentIndexOverrides.enableGraph !== false
) {
await assertWritable();
const semanticResult = await jointSemanticGraph.materialize({
createdAt: activeAsset.updatedAt ?? activeAsset.createdAt,
knowledgeSpaceId: input.knowledgeSpaceId,
parseArtifactId: canonicalArtifact.id,
publicationGenerationId,
retrievalProfile: frozenRetrievalProfile,
});
await assertWritable();
graphEntityIds = semanticResult.graphEntityIds;
graphRelationIds = semanticResult.graphRelationIds;
}
if (
semanticEnrichmentAdmission &&
!jointSemanticGraph &&
publicationGenerationId &&
frozenRetrievalProfile &&
reindexResult.status === "rebuilt" &&

172
knowledge-fs/packages/api/src/document-layout-recomposer.test.ts Normal file
View File

@ -0,0 +1,172 @@
import { type ParseArtifact, ParseArtifactSchema } from "@knowledge/core";
import { describe, expect, it } from "vitest";
import { recomposeDocumentLayoutForSemanticSegmentation } from "./document-layout-recomposer";
describe("document layout recomposition for semantic segmentation", () => {
it("removes unproven PDF title boundaries while preserving every source element", () => {
const input = artifact({
elements: [
element("title", "电子发票（普通发票）", ["电子发票（普通发票）"], "title-1"),
element("paragraph", "发票号码：26322000000000000000", ["电子发票（普通发票）"], "p-1"),
element(
"title",
"名称：示例人工智能有限公司",
["名称：示例人工智能有限公司"],
"false-title-1",
),
element(
"paragraph",
"统一社会信用代码：91320506EXAMPLE01",
["名称：示例人工智能有限公司"],
"p-2",
),
element("title", "91320506EXAMPLE02", ["91320506EXAMPLE02"], "false-title-2"),
element("table", "餐饮服务 | 1 | 533.96 | 6% | 32.04", ["91320506EXAMPLE02"], "table-1"),
element("paragraph", "合计：566.00", ["91320506EXAMPLE02"], "p-3"),
],
parser: "unstructured",
});
const result = recomposeDocumentLayoutForSemanticSegmentation(input);
expect(result.artifact.elements.map(({ id, text, type }) => ({ id, text, type }))).toEqual(
input.elements.map(({ id, text, type }) => ({ id, text, type })),
);
expect(result.artifact.elements.map((item) => item.sectionPath)).toEqual([
[],
[],
[],
[],
[],
[],
[],
]);
expect(result.artifact.elements[2]?.metadata.layoutRecomposition).toEqual({
boundaryPolicy: "reasoning-model",
originalSectionPath: ["名称：示例人工智能有限公司"],
originalType: "title",
reason: "unstructured-heading-without-hierarchy-evidence",
schemaVersion: 1,
});
expect(result.stats).toEqual({
elementsRecomposed: 7,
modelDecidedHeadingBoundaries: 3,
trustedHeadingBoundaries: 0,
});
expect(result.fingerprint).toMatch(/^sha256:[a-f0-9]{64}$/u);
});
it("preserves explicit Unstructured heading hierarchy and applies it to following content", () => {
const input = artifact({
elements: [
element("title", "第一章", ["第一章"], "chapter", { category_depth: 0 }),
element("heading", "范围", ["第一章", "范围"], "section", {
category_depth: 1,
parent_id: "chapter",
}),
element("paragraph", "适用范围正文。", ["第一章", "范围"], "paragraph"),
],
parser: "unstructured",
});
const result = recomposeDocumentLayoutForSemanticSegmentation(input);
expect(result.artifact.elements.map((item) => item.sectionPath)).toEqual([
["第一章"],
["第一章", "范围"],
["第一章", "范围"],
]);
expect(result.stats).toEqual({
elementsRecomposed: 3,
modelDecidedHeadingBoundaries: 0,
trustedHeadingBoundaries: 2,
});
});
it("leaves native parser section boundaries unchanged", () => {
const input = artifact({
elements: [
element("heading", "安装", ["安装"], "heading"),
element("paragraph", "安装正文。", ["安装"], "paragraph"),
],
parser: "native-markdown",
});
const result = recomposeDocumentLayoutForSemanticSegmentation(input);
expect(result.artifact).toEqual(input);
expect(result.stats).toEqual({
elementsRecomposed: 0,
modelDecidedHeadingBoundaries: 0,
trustedHeadingBoundaries: 0,
});
});
it("rejects artifacts that exceed the bounded element count", () => {
const input = artifact({
elements: [element("paragraph", "一", [], "one"), element("paragraph", "二", [], "two")],
parser: "unstructured",
});
expect(() => recomposeDocumentLayoutForSemanticSegmentation(input, { maxElements: 1 })).toThrow(
"Document layout recomposition exceeds maxElements=1",
);
expect(() => recomposeDocumentLayoutForSemanticSegmentation(input, { maxElements: 0 })).toThrow(
"maxElements must be at least 1",
);
});
it("trusts a non-empty parent id even when category depth is absent", () => {
const input = artifact({
elements: [
element("heading", "子章节", ["父章节", "子章节"], "child", {
parent_id: "parent",
}),
element("paragraph", "正文", ["父章节", "子章节"], "body"),
],
parser: "unstructured",
});
const result = recomposeDocumentLayoutForSemanticSegmentation(input, { maxElements: 2 });
expect(result.stats.trustedHeadingBoundaries).toBe(1);
expect(result.artifact.elements[1]?.sectionPath).toEqual(["父章节", "子章节"]);
});
});
function artifact({
elements,
parser,
}: {
readonly elements: ParseArtifact["elements"];
readonly parser: ParseArtifact["parser"];
}): ParseArtifact {
return ParseArtifactSchema.parse({
artifactHash: "a".repeat(64),
contentType: "text",
createdAt: "2026-08-13T00:00:00.000Z",
documentAssetId: "018f0d60-7a49-7cc2-9c1b-5b36f18f2c42",
elements,
id: "018f0d60-7a49-7cc2-9c1b-5b36f18f2c43",
metadata: { parserVersion: `${parser}@test` },
parser,
version: 1,
});
}
function element(
type: ParseArtifact["elements"][number]["type"],
text: string,
sectionPath: readonly string[],
id: string,
metadata: Record<string, unknown> = {},
): ParseArtifact["elements"][number] {
return {
id,
metadata,
pageNumber: 1,
sectionPath: [...sectionPath],
text,
type,
};
}

157
knowledge-fs/packages/api/src/document-layout-recomposer.ts Normal file
View File

@ -0,0 +1,157 @@
import { createHash } from "node:crypto";
import { type ParseArtifact, ParseArtifactSchema, stableJson } from "@knowledge/core";
const DEFAULT_MAX_ELEMENTS = 20_000;
const LAYOUT_RECOMPOSITION_SCHEMA_VERSION = 1;
export interface DocumentLayoutRecompositionOptions {
readonly maxElements?: number | undefined;
}
export interface DocumentLayoutRecompositionStats {
readonly elementsRecomposed: number;
readonly modelDecidedHeadingBoundaries: number;
readonly trustedHeadingBoundaries: number;
}
export interface DocumentLayoutRecompositionResult {
readonly artifact: ParseArtifact;
readonly fingerprint: string;
readonly stats: DocumentLayoutRecompositionStats;
}
/**
* Produces the bounded parser view consumed by semantic segmentation.
*
* Native Markdown/HTML headings are authored structure and remain authoritative. Unstructured
* `Title`/`Heading` labels are only hard boundaries when the provider supplied explicit hierarchy
* evidence (`category_depth` or `parent_id`). Other labels remain visible to the reasoning model as
* title/heading units, but cannot fragment its input merely because a layout classifier guessed
* that a company name, tax id, or form field was a heading.
*
* Element order, text, ids, pages, and source metadata are never rewritten. This keeps canonical
* byte offsets stable while making boundary confidence explicit and replayable.
*/
export function recomposeDocumentLayoutForSemanticSegmentation(
input: ParseArtifact,
{ maxElements = DEFAULT_MAX_ELEMENTS }: DocumentLayoutRecompositionOptions = {},
): DocumentLayoutRecompositionResult {
if (!Number.isSafeInteger(maxElements) || maxElements < 1) {
throw new Error("Document layout recomposition maxElements must be at least 1");
}
const artifact = ParseArtifactSchema.parse(input);
if (artifact.elements.length > maxElements) {
throw new Error(`Document layout recomposition exceeds maxElements=${maxElements}`);
}
if (artifact.parser !== "unstructured") {
const unchanged = ParseArtifactSchema.parse(artifact);
return {
artifact: unchanged,
fingerprint: layoutRecompositionFingerprint(unchanged),
stats: {
elementsRecomposed: 0,
modelDecidedHeadingBoundaries: 0,
trustedHeadingBoundaries: 0,
},
};
}
let currentTrustedPath: string[] = [];
let modelDecidedHeadingBoundaries = 0;
let trustedHeadingBoundaries = 0;
const elements = artifact.elements.map((element) => {
const heading = element.type === "title" || element.type === "heading";
if (heading && hasExplicitHierarchyEvidence(element.metadata)) {
currentTrustedPath = [...element.sectionPath];
trustedHeadingBoundaries += 1;
return {
...element,
metadata: { ...element.metadata },
sectionPath: [...currentTrustedPath],
};
}
if (heading) {
modelDecidedHeadingBoundaries += 1;
return {
...element,
metadata: {
...element.metadata,
layoutRecomposition: {
boundaryPolicy: "reasoning-model",
originalSectionPath: [...element.sectionPath],
originalType: element.type,
reason: "unstructured-heading-without-hierarchy-evidence",
schemaVersion: LAYOUT_RECOMPOSITION_SCHEMA_VERSION,
},
},
sectionPath: [...currentTrustedPath],
};
}
const sectionPathChanged = !sameStrings(element.sectionPath, currentTrustedPath);
return {
...element,
metadata: {
...element.metadata,
...(sectionPathChanged
? {
layoutRecomposition: {
boundaryPolicy: "reasoning-model",
originalSectionPath: [...element.sectionPath],
reason: "inherited-untrusted-heading-boundary",
schemaVersion: LAYOUT_RECOMPOSITION_SCHEMA_VERSION,
},
}
: {}),
},
sectionPath: [...currentTrustedPath],
};
});
const recomposed = ParseArtifactSchema.parse({ ...artifact, elements });
return {
artifact: recomposed,
fingerprint: layoutRecompositionFingerprint(recomposed),
stats: {
elementsRecomposed: recomposed.elements.length,
modelDecidedHeadingBoundaries,
trustedHeadingBoundaries,
},
};
}
function hasExplicitHierarchyEvidence(metadata: Readonly<Record<string, unknown>>): boolean {
const categoryDepth = metadata.category_depth;
const parentId = metadata.parent_id;
return (
(typeof categoryDepth === "number" && Number.isInteger(categoryDepth) && categoryDepth >= 0) ||
(typeof parentId === "string" && parentId.trim().length > 0)
);
}
function layoutRecompositionFingerprint(artifact: ParseArtifact): string {
return `sha256:${createHash("sha256")
.update(
stableJson({
artifactHash: artifact.artifactHash,
elements: artifact.elements.map((element) => ({
id: element.id,
metadata: element.metadata,
pageNumber: element.pageNumber,
sectionPath: element.sectionPath,
text: element.text,
type: element.type,
})),
schemaVersion: LAYOUT_RECOMPOSITION_SCHEMA_VERSION,
}),
)
.digest("hex")}`;
}
function sameStrings(left: readonly string[], right: readonly string[]): boolean {
return left.length === right.length && left.every((value, index) => value === right[index]);
}

42
knowledge-fs/packages/api/src/document-outline-builder.test.ts
View File

@ -97,6 +97,48 @@ describe("document outline builder", () => {
expect(outline.nodes[0]?.children[0]?.summary).toContain("dense, full-text, and graph");
});
it("preserves semantic-node lineage and model section summaries", () => {
const builder = createDocumentOutlineBuilder({
generateId: sequenceIds([
"018f0d60-7a49-7cc2-9c1b-5b36f18f2c50",
"018f0d60-7a49-7cc2-9c1b-5b36f18f2c51",
"018f0d60-7a49-7cc2-9c1b-5b36f18f2c52",
]),
maxElements: 20,
maxNodes: 10,
maxSummaryChars: 120,
now: () => createdAt,
});
const sourceNodeId = "018f0d60-7a49-7cc2-9c1b-5b36f18f2d01";
const semanticArtifact: ParseArtifact = {
...parseArtifact([
{
id: sourceNodeId,
metadata: {
semanticSectionSummary: "发票身份、购买方和价税合计。",
sourceKnowledgeNodeId: sourceNodeId,
},
sectionPath: ["电子发票", "购买方与金额"],
text: "发票号码、购买方、价税合计和开票人",
type: "paragraph",
},
]),
metadata: {
parserVersion: "native-markdown@1",
semanticCompilation: { source: "llm-semantic-v1" },
},
};
const outline = builder.build({ knowledgeSpaceId, parseArtifact: semanticArtifact });
expect(outline.metadata).toMatchObject({ builder: "semantic-knowledge-nodes" });
expect(outline.nodes[0]?.children[0]).toMatchObject({
sectionPath: ["电子发票", "购买方与金额"],
sourceNodeIds: [sourceNodeId],
summary: "发票身份、购买方和价税合计。",
});
});
it("falls back to a document node when parse output has no heading structure", () => {
const builder = createDocumentOutlineBuilder({
generateId: sequenceIds([

20
knowledge-fs/packages/api/src/document-outline-builder.ts
View File

@ -156,7 +156,9 @@ export function createDocumentOutlineBuilder({
id: buildGenerateId(),
knowledgeSpaceId,
metadata: {
builder: "deterministic-parse-artifact",
builder: artifact.metadata.semanticCompilation
? "semantic-knowledge-nodes"
: "deterministic-parse-artifact",
contentType: artifact.contentType,
parser: artifact.parser,
parserVersion: artifact.metadata.parserVersion,
@ -373,8 +375,22 @@ function applySpanToDraft(draft: OutlineNodeDraft, span: ElementSpan): void {
draft.sourceElementIds.push(span.element.id);
}
const sourceKnowledgeNodeId = span.element.metadata.sourceKnowledgeNodeId;
if (
typeof sourceKnowledgeNodeId === "string" &&
sourceKnowledgeNodeId.trim() &&
!draft.sourceNodeIds.includes(sourceKnowledgeNodeId)
) {
draft.sourceNodeIds.push(sourceKnowledgeNodeId);
}
if (span.element.type !== "heading" && span.element.type !== "title") {
draft.summaryTexts.push(span.text);
const semanticSectionSummary = span.element.metadata.semanticSectionSummary;
draft.summaryTexts.push(
typeof semanticSectionSummary === "string" && semanticSectionSummary.trim()
? semanticSectionSummary.trim()
: span.text,
);
}
if (!draft.titleLocation && (span.element.type === "heading" || span.element.type === "title")) {

495
knowledge-fs/packages/api/src/document-semantic-enrichment-processor.test.ts
View File

@ -1,13 +1,22 @@
import { type KnowledgeNode, KnowledgeNodeSchema } from "@knowledge/core";
import {
type KnowledgeNode,
KnowledgeNodeSchema,
type KnowledgeSpaceRetrievalProfile,
ParseArtifactSchema,
} from "@knowledge/core";
import { describe, expect, it } from "vitest";
import { createDocumentSemanticEnrichmentProcessor } from "./document-semantic-enrichment-processor";
import {
createDocumentSemanticEnrichmentProcessor,
createJointSemanticGraphMaterializer,
} from "./document-semantic-enrichment-processor";
import {
createInMemoryDocumentSemanticEnrichmentRepository,
createInMemoryDocumentSemanticExtractionCheckpointRepository,
} from "./document-semantic-enrichment-repository";
import { createInMemoryGraphIndexRepository } from "./graph-index-repository";
import { createInMemoryKnowledgeNodeRepository } from "./knowledge-node-repository";
import { createLlmSemanticChunker } from "./llm-semantic-chunker";
const tenantId = "tenant-1";
const knowledgeSpaceId = uuid(1);
@ -17,6 +26,162 @@ const publicationGenerationId = uuid(4);
const createdAt = "2026-08-09T10:00:00.000Z";
describe("createDocumentSemanticEnrichmentProcessor", () => {
it("validates processor and joint materializer bounds", () => {
const nodes = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 1,
maxListLimit: 1,
maxNodes: 1,
});
const graph = createInMemoryGraphIndexRepository({
maxBatchSize: 1,
maxEntities: 1,
maxRelations: 1,
});
const baseProcessor = {
checkpoints: createInMemoryDocumentSemanticExtractionCheckpointRepository(),
graph,
maxConcurrentBatches: 1,
maxEntitiesPerNode: 1,
maxNodesPerArtifact: 1,
maxOutputTokens: 1,
maxRelationsPerNode: 1,
nodes,
providerBatchSize: 1,
providerFactory: () => semanticProvider(),
};
for (const [name, override] of [
["maxConcurrentBatches", { maxConcurrentBatches: 0 }],
["maxEntitiesPerNode", { maxEntitiesPerNode: 0 }],
["maxNodesPerArtifact", { maxNodesPerArtifact: 0 }],
["maxOutputTokens", { maxOutputTokens: 0 }],
["maxRelationsPerNode", { maxRelationsPerNode: 0 }],
["providerBatchSize", { providerBatchSize: 0 }],
] as const) {
expect(() =>
createDocumentSemanticEnrichmentProcessor({ ...baseProcessor, ...override }),
).toThrow(`${name} must be at least 1`);
}
const baseMaterializer = {
graph,
maxEntitiesPerNode: 1,
maxNodesPerArtifact: 1,
maxRelationsPerNode: 1,
nodes,
};
for (const [name, override] of [
["maxEntitiesPerNode", { maxEntitiesPerNode: 0 }],
["maxNodesPerArtifact", { maxNodesPerArtifact: 0 }],
["maxRelationsPerNode", { maxRelationsPerNode: 0 }],
] as const) {
expect(() =>
createJointSemanticGraphMaterializer({ ...baseMaterializer, ...override }),
).toThrow(`${name} must be at least 1`);
}
});
it("returns an empty result without constructing a provider", async () => {
const nodes = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 1,
maxListLimit: 1,
maxNodes: 1,
});
const graph = createInMemoryGraphIndexRepository({
maxBatchSize: 1,
maxEntities: 1,
maxRelations: 1,
});
let providerFactoryCalls = 0;
const processor = createDocumentSemanticEnrichmentProcessor({
checkpoints: createInMemoryDocumentSemanticExtractionCheckpointRepository(),
graph,
maxConcurrentBatches: 1,
maxEntitiesPerNode: 1,
maxNodesPerArtifact: 1,
maxOutputTokens: 1,
maxRelationsPerNode: 1,
nodes,
providerBatchSize: 1,
providerFactory: () => {
providerFactoryCalls += 1;
return semanticProvider();
},
});
const expected = {
entitiesExtracted: 0,
graphEntityIds: [],
graphEntitiesIndexed: 0,
graphRelationIds: [],
graphRelationsIndexed: 0,
nodesScanned: 0,
semanticProviderCalls: 0,
semanticProviderCallsMaximum: 0,
};
await expect(processor.process(await semanticJob())).resolves.toEqual(expected);
await expect(
createJointSemanticGraphMaterializer({
graph,
maxEntitiesPerNode: 1,
maxNodesPerArtifact: 1,
maxRelationsPerNode: 1,
nodes,
}).materialize({
createdAt,
knowledgeSpaceId,
parseArtifactId,
publicationGenerationId,
retrievalProfile: (await semanticJob()).retrievalProfile,
}),
).resolves.toEqual(expected);
expect(providerFactoryCalls).toBe(0);
});
it("rejects an artifact page that exceeds the configured node bound", async () => {
const nodes = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 2,
maxListLimit: 2,
maxNodes: 2,
});
await nodes.createMany([knowledgeNode(0), knowledgeNode(1)]);
const graph = createInMemoryGraphIndexRepository({
maxBatchSize: 2,
maxEntities: 2,
maxRelations: 2,
});
const processor = createDocumentSemanticEnrichmentProcessor({
checkpoints: createInMemoryDocumentSemanticExtractionCheckpointRepository(),
graph,
maxConcurrentBatches: 1,
maxEntitiesPerNode: 1,
maxNodesPerArtifact: 1,
maxOutputTokens: 1,
maxRelationsPerNode: 1,
nodes,
providerBatchSize: 1,
providerFactory: () => semanticProvider(),
});
await expect(processor.process(await semanticJob())).rejects.toThrow(
"node count exceeds maxNodesPerArtifact=1",
);
await expect(
createJointSemanticGraphMaterializer({
graph,
maxEntitiesPerNode: 1,
maxNodesPerArtifact: 1,
maxRelationsPerNode: 1,
nodes,
}).materialize({
createdAt,
knowledgeSpaceId,
parseArtifactId,
publicationGenerationId,
retrievalProfile: (await semanticJob()).retrievalProfile,
}),
).rejects.toThrow("node count exceeds maxNodesPerArtifact=1");
});
it("batches 80 nodes into at most 20 requests, checkpoints them, and preserves published nodes", async () => {
const nodes = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 100,
@ -132,6 +297,7 @@ describe("createDocumentSemanticEnrichmentProcessor", () => {
});
await nodes.createMany([knowledgeNode(0)]);
const provider = semanticProvider({ entityCount: 1 });
let gatedRequests = 0;
const processor = createDocumentSemanticEnrichmentProcessor({
checkpoints: createInMemoryDocumentSemanticExtractionCheckpointRepository(),
graph: createInMemoryGraphIndexRepository({
@ -144,6 +310,12 @@ describe("createDocumentSemanticEnrichmentProcessor", () => {
maxNodesPerArtifact: 10,
maxOutputTokens: 1_500,
maxRelationsPerNode: 8,
modelRequestGate: {
run: async (request) => {
gatedRequests += 1;
return request();
},
},
nodes,
now: () => createdAt,
providerBatchSize: 8,
@ -156,6 +328,253 @@ describe("createDocumentSemanticEnrichmentProcessor", () => {
});
expect(provider.entityCalls).toBe(1);
expect(provider.relationCalls).toBe(0);
expect(gatedRequests).toBe(1);
});
it("fails closed when a checkpoint repository drops a completed batch", async () => {
const nodes = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 1,
maxListLimit: 1,
maxNodes: 1,
});
await nodes.createMany([knowledgeNode(0)]);
const processor = createDocumentSemanticEnrichmentProcessor({
checkpoints: {
getMany: async () => [],
putMany: async () => [],
},
graph: createInMemoryGraphIndexRepository({
maxBatchSize: 2,
maxEntities: 2,
maxRelations: 2,
}),
maxConcurrentBatches: 1,
maxEntitiesPerNode: 2,
maxNodesPerArtifact: 1,
maxOutputTokens: 1_500,
maxRelationsPerNode: 2,
nodes,
providerBatchSize: 1,
providerFactory: () => semanticProvider({ entityCount: 1 }),
});
await expect(processor.process(await semanticJob())).rejects.toThrow(
"checkpoint is incomplete",
);
});
it("rejects mixed, legacy, and frozen-model-mismatched semantic generations", async () => {
const graph = createInMemoryGraphIndexRepository({
maxBatchSize: 10,
maxEntities: 10,
maxRelations: 10,
});
const job = await semanticJob();
const validJoint = await jointKnowledgeNode(job.retrievalProfile);
const mixedNodes = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 10,
maxListLimit: 10,
maxNodes: 10,
});
await mixedNodes.createMany([validJoint, knowledgeNode(1)]);
const processorFor = (nodes: ReturnType<typeof createInMemoryKnowledgeNodeRepository>) =>
createDocumentSemanticEnrichmentProcessor({
checkpoints: createInMemoryDocumentSemanticExtractionCheckpointRepository(),
graph,
maxConcurrentBatches: 1,
maxEntitiesPerNode: 8,
maxNodesPerArtifact: 10,
maxOutputTokens: 1_500,
maxRelationsPerNode: 8,
nodes,
providerBatchSize: 8,
providerFactory: () => semanticProvider(),
});
await expect(processorFor(mixedNodes).process(job)).rejects.toThrow(
"refuses a mixed joint/legacy node generation",
);
const legacyNodes = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 10,
maxListLimit: 10,
maxNodes: 10,
});
await legacyNodes.createMany([knowledgeNode(0)]);
await expect(
createJointSemanticGraphMaterializer({
graph,
maxEntitiesPerNode: 8,
maxNodesPerArtifact: 10,
maxRelationsPerNode: 8,
nodes: legacyNodes,
}).materialize({
createdAt,
knowledgeSpaceId,
parseArtifactId,
publicationGenerationId,
retrievalProfile: job.retrievalProfile,
}),
).rejects.toThrow("refuses a legacy or invalid node generation");
const otherProfile: KnowledgeSpaceRetrievalProfile = {
...job.retrievalProfile,
reasoningModel: { model: "other", pluginId: "other-plugin", provider: "other-provider" },
};
const mismatchedJoint = await jointKnowledgeNode(otherProfile);
const mismatchedNodes = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 10,
maxListLimit: 10,
maxNodes: 10,
});
await mismatchedNodes.createMany([mismatchedJoint]);
await expect(processorFor(mismatchedNodes).process(job)).rejects.toThrow(
"joint metadata does not match the frozen reasoning model",
);
await expect(
createJointSemanticGraphMaterializer({
graph,
maxEntitiesPerNode: 8,
maxNodesPerArtifact: 10,
maxRelationsPerNode: 8,
nodes: mismatchedNodes,
}).materialize({
createdAt,
knowledgeSpaceId,
parseArtifactId,
publicationGenerationId,
retrievalProfile: job.retrievalProfile,
}),
).rejects.toThrow("metadata does not match the frozen reasoning model");
});
it("indexes joint semantic-chunk metadata without a second model request", async () => {
const nodes = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 10,
maxListLimit: 10,
maxNodes: 10,
});
let chunkingCalls = 0;
const semanticNodes = await createLlmSemanticChunker({
now: () => createdAt,
reasoningProviderFactory: () => ({
kind: "plugin-daemon",
async *stream(input) {
chunkingCalls += 1;
const user = input.messages.find((message) => message.role === "user");
const payload = JSON.parse(user?.content ?? "{}") as {
units: Array<{ id: string }>;
};
yield {
delta: JSON.stringify({
chunks: [
{
endUnitId: payload.units.at(-1)?.id,
entities: [
{ confidence: 0.98, id: "acme", text: "Acme", type: "organization" },
{ confidence: 0.97, id: "policy", text: "policy", type: "policy" },
],
relations: [
{
confidence: 0.96,
objectEntityId: "policy",
subjectEntityId: "acme",
type: "references",
},
],
sectionPath: ["Guide", "Renewal"],
sectionSummary: "Acme renewal policy.",
startUnitId: payload.units[0]?.id,
},
],
}),
type: "delta" as const,
};
yield {
finishReason: "stop",
metadata: { model: input.model, provider: "plugin-daemon" },
type: "done" as const,
};
},
}),
}).chunk({
knowledgeSpaceId,
parseArtifact: ParseArtifactSchema.parse({
artifactHash: "a".repeat(64),
contentType: "text",
createdAt,
documentAssetId,
elements: [
{
id: "element-1",
metadata: {},
sectionPath: ["Guide"],
text: "Acme follows the renewal policy.",
type: "paragraph",
},
],
id: parseArtifactId,
metadata: {},
parser: "native-markdown",
version: 1,
}),
publicationGenerationId,
retrievalProfile: (await semanticJob()).retrievalProfile,
tenantId,
});
await nodes.createMany(semanticNodes);
let enrichmentFactoryCalls = 0;
const graph = createInMemoryGraphIndexRepository({
maxBatchSize: 10,
maxEntities: 10,
maxRelations: 10,
});
const processor = createDocumentSemanticEnrichmentProcessor({
checkpoints: createInMemoryDocumentSemanticExtractionCheckpointRepository(),
graph,
maxConcurrentBatches: 1,
maxEntitiesPerNode: 8,
maxNodesPerArtifact: 10,
maxOutputTokens: 1_500,
maxRelationsPerNode: 8,
nodes,
now: () => createdAt,
providerBatchSize: 8,
providerFactory: () => {
enrichmentFactoryCalls += 1;
throw new Error("joint semantic nodes must not invoke enrichment provider");
},
});
await expect(processor.process(await semanticJob())).resolves.toMatchObject({
entitiesExtracted: 2,
graphEntitiesIndexed: 2,
graphRelationsIndexed: 1,
semanticProviderCalls: 0,
semanticProviderCallsMaximum: 0,
});
expect(chunkingCalls).toBe(1);
expect(enrichmentFactoryCalls).toBe(0);
await expect(
createJointSemanticGraphMaterializer({
graph,
maxEntitiesPerNode: 8,
maxNodesPerArtifact: 10,
maxRelationsPerNode: 8,
nodes,
now: () => createdAt,
}).materialize({
createdAt,
knowledgeSpaceId,
parseArtifactId,
publicationGenerationId,
retrievalProfile: (await semanticJob()).retrievalProfile,
}),
).resolves.toMatchObject({
graphEntityIds: [expect.any(String), expect.any(String)],
graphRelationIds: [expect.any(String)],
semanticProviderCalls: 0,
});
});
});
@ -219,6 +638,78 @@ function semanticProvider(options: { entityCount?: number; failEntityCall?: numb
return provider;
}
async function jointKnowledgeNode(
retrievalProfile: KnowledgeSpaceRetrievalProfile,
): Promise<KnowledgeNode> {
const nodes = await createLlmSemanticChunker({
now: () => createdAt,
reasoningProviderFactory: () => ({
kind: "plugin-daemon",
async *stream(input) {
const user = input.messages.find((message) => message.role === "user");
const payload = JSON.parse(user?.content ?? "{}") as {
units: Array<{ id: string }>;
};
yield {
delta: JSON.stringify({
chunks: [
{
endUnitId: payload.units.at(-1)?.id,
entities: [
{ confidence: 0.98, id: "acme", text: "Acme", type: "organization" },
{ confidence: 0.97, id: "policy", text: "policy", type: "policy" },
],
relations: [
{
confidence: 0.96,
objectEntityId: "policy",
subjectEntityId: "acme",
type: "references",
},
],
startUnitId: payload.units[0]?.id,
},
],
}),
type: "delta" as const,
};
yield {
finishReason: "stop",
metadata: { model: input.model, provider: "plugin-daemon" },
type: "done" as const,
};
},
}),
}).chunk({
knowledgeSpaceId,
parseArtifact: ParseArtifactSchema.parse({
artifactHash: "a".repeat(64),
contentType: "text",
createdAt,
documentAssetId,
elements: [
{
id: "joint-element",
metadata: {},
sectionPath: ["Guide"],
text: "Acme follows the renewal policy.",
type: "paragraph",
},
],
id: parseArtifactId,
metadata: {},
parser: "native-markdown",
version: 1,
}),
publicationGenerationId,
retrievalProfile,
tenantId,
});
const node = nodes[0];
if (!node) throw new Error("joint semantic node fixture is empty");
return node;
}
async function semanticJob() {
const repository = createInMemoryDocumentSemanticEnrichmentRepository({
generateLeaseToken: () => uuid(99),

255
knowledge-fs/packages/api/src/document-semantic-enrichment-processor.ts
View File

@ -3,6 +3,7 @@ import { createHash } from "node:crypto";
import {
type KnowledgeNode,
type KnowledgeSpaceModelSelection,
type KnowledgeSpaceRetrievalProfile,
PublicationGenerationIdSchema,
stableJson,
} from "@knowledge/core";
@ -21,7 +22,7 @@ import {
import { createExtractionQualityControlFlow } from "./extraction-quality-control-flow";
import type { GraphIndexRepository } from "./graph-index-repository";
import { createGraphIndexWriter } from "./graph-index-writer";
import { cloneJsonObject } from "./json-utils";
import { cloneJsonObject, isPlainObject } from "./json-utils";
import {
type KnowledgeNodeRepository,
cloneKnowledgeNode,
@ -35,6 +36,7 @@ import {
type RelationExtractionTextProvider,
createLlmRelationExtractionProvider,
} from "./llm-relation-extraction-provider";
import { hasValidLlmSemanticJointExtraction } from "./llm-semantic-chunker";
import { createRelationExtractionFlow } from "./relation-extraction-flow";
export type DocumentSemanticEnrichmentTextProvider = EntityExtractionTextProvider &
@ -42,7 +44,9 @@ export type DocumentSemanticEnrichmentTextProvider = EntityExtractionTextProvide
export interface DocumentSemanticEnrichmentProcessorResult {
readonly entitiesExtracted: number;
readonly graphEntityIds: readonly string[];
readonly graphEntitiesIndexed: number;
readonly graphRelationIds: readonly string[];
readonly graphRelationsIndexed: number;
readonly nodesScanned: number;
readonly semanticProviderCalls: number;
@ -53,6 +57,25 @@ export interface DocumentSemanticEnrichmentProcessor {
process(job: DocumentSemanticEnrichmentJob): Promise<DocumentSemanticEnrichmentProcessorResult>;
}
export interface JointSemanticGraphMaterializer {
materialize(input: {
readonly createdAt: string;
readonly knowledgeSpaceId: string;
readonly parseArtifactId: string;
readonly publicationGenerationId: string;
readonly retrievalProfile: KnowledgeSpaceRetrievalProfile;
}): Promise<DocumentSemanticEnrichmentProcessorResult>;
}
export interface JointSemanticGraphMaterializerOptions {
readonly graph: GraphIndexRepository;
readonly maxEntitiesPerNode: number;
readonly maxNodesPerArtifact: number;
readonly maxRelationsPerNode: number;
readonly nodes: Pick<KnowledgeNodeRepository, "listByArtifact">;
readonly now?: (() => string) | undefined;
}
export interface DocumentSemanticEnrichmentProcessorOptions {
readonly checkpoints: DocumentSemanticExtractionCheckpointRepository;
readonly graph: GraphIndexRepository;
@ -121,7 +144,9 @@ export function createDocumentSemanticEnrichmentProcessor({
if (page.items.length === 0) {
return {
entitiesExtracted: 0,
graphEntityIds: [],
graphEntitiesIndexed: 0,
graphRelationIds: [],
graphRelationsIndexed: 0,
nodesScanned: 0,
semanticProviderCalls: 0,
@ -133,6 +158,35 @@ export function createDocumentSemanticEnrichmentProcessor({
if (!selection) {
throw new Error("Document semantic enrichment requires a frozen reasoning model");
}
const originalNodes = page.items.map(cloneKnowledgeNode);
const jointSemanticNodes = originalNodes.filter(hasValidLlmSemanticJointExtraction);
if (jointSemanticNodes.length > 0) {
if (jointSemanticNodes.length !== originalNodes.length) {
throw new Error(
"Document semantic enrichment refuses a mixed joint/legacy node generation",
);
}
if (
jointSemanticNodes.some(
(node) => stableJson(jointSemanticModelSelection(node)) !== stableJson(selection),
)
) {
throw new Error(
"Document semantic enrichment joint metadata does not match the frozen reasoning model",
);
}
return indexPreparedSemanticNodes({
graph,
job,
maxEntitiesPerNode,
maxNodesPerArtifact,
maxRelationsPerNode,
nodes: jointSemanticNodes,
now,
semanticProviderCalls: 0,
semanticProviderCallsMaximum: 0,
});
}
let semanticProviderCalls = 0;
const resolvedProvider = providerFactory(selection);
const provider: DocumentSemanticEnrichmentTextProvider = {
@ -159,7 +213,6 @@ export function createDocumentSemanticEnrichmentProcessor({
publicationGenerationId: generationId,
tenantId: job.tenantId,
};
const originalNodes = page.items.map(cloneKnowledgeNode);
const entityCheckpoints = await completeEntityCheckpoints({
checkpoints,
maxConcurrentBatches,
@ -186,57 +239,175 @@ export function createDocumentSemanticEnrichmentProcessor({
tenantId: job.tenantId,
});
const relationNodes = applyStageCheckpoints(entityNodes, relationCheckpoints);
const qualityRepository = await temporaryNodeRepository(relationNodes);
const controlled = await createExtractionQualityControlFlow({
maxBatchSize: maxNodesPerArtifact,
maxEligibleEntitiesPerNode: maxEntitiesPerNode,
maxEligibleRelationsPerNode: maxRelationsPerNode,
nodes: qualityRepository,
now,
}).apply({
knowledgeSpaceId: job.knowledgeSpaceId,
nodeIds: relationNodes.map((node) => node.id),
publicationGenerationId: generationId,
});
if (controlled.missingNodeIds.length > 0) {
throw new Error("Document semantic enrichment quality stage lost immutable nodes");
}
const indexed = await createGraphIndexWriter({
extractionVersion: 1,
graph,
maxBatchSize: maxNodesPerArtifact,
nodes: qualityRepository,
// A generation retry must reproduce byte-identical immutable graph rows.
now: () => job.createdAt,
}).indexNodes({
knowledgeSpaceId: job.knowledgeSpaceId,
nodes: controlled.controlledNodes,
publicationGenerationId: generationId,
});
if (indexed.missingNodeIds.length > 0) {
throw new Error("Document semantic enrichment graph stage lost immutable nodes");
}
const eligibleRelationNodes = entityNodes.filter(
(node) => extractedEntitiesFromNodeMetadata(node).length >= 2,
).length;
return {
entitiesExtracted: controlled.controlledNodes.reduce(
(sum, node) => sum + extractedEntitiesFromNodeMetadata(node).length,
0,
),
graphEntitiesIndexed: indexed.stats.entitiesIndexed,
graphRelationsIndexed: indexed.stats.relationsIndexed,
nodesScanned: originalNodes.length,
return indexPreparedSemanticNodes({
graph,
job,
maxEntitiesPerNode,
maxNodesPerArtifact,
maxRelationsPerNode,
nodes: relationNodes,
now,
semanticProviderCalls,
semanticProviderCallsMaximum:
Math.ceil(originalNodes.length / providerBatchSize) +
Math.ceil(eligibleRelationNodes / providerBatchSize),
};
});
},
};
}
/** Materializes only the joint facts already frozen by semantic chunking; it never calls an LLM. */
export function createJointSemanticGraphMaterializer({
graph,
maxEntitiesPerNode,
maxNodesPerArtifact,
maxRelationsPerNode,
nodes,
now = () => new Date().toISOString(),
}: JointSemanticGraphMaterializerOptions): JointSemanticGraphMaterializer {
for (const [name, value] of Object.entries({
maxEntitiesPerNode,
maxNodesPerArtifact,
maxRelationsPerNode,
})) {
if (!Number.isSafeInteger(value) || value < 1) {
throw new Error(`Joint semantic Graph ${name} must be at least 1`);
}
}
return {
materialize: async (input) => {
const generationId = PublicationGenerationIdSchema.parse(input.publicationGenerationId);
const page = await nodes.listByArtifact({
knowledgeSpaceId: input.knowledgeSpaceId,
limit: maxNodesPerArtifact,
parseArtifactId: input.parseArtifactId,
publicationGenerationId: generationId,
});
if (page.nextCursor) {
throw new Error(
`Joint semantic Graph node count exceeds maxNodesPerArtifact=${maxNodesPerArtifact}`,
);
}
if (page.items.length === 0) {
return {
entitiesExtracted: 0,
graphEntityIds: [],
graphEntitiesIndexed: 0,
graphRelationIds: [],
graphRelationsIndexed: 0,
nodesScanned: 0,
semanticProviderCalls: 0,
semanticProviderCallsMaximum: 0,
};
}
const prepared = page.items.map(cloneKnowledgeNode);
if (prepared.some((node) => !hasValidLlmSemanticJointExtraction(node))) {
throw new Error("Joint semantic Graph refuses a legacy or invalid node generation");
}
const selection = input.retrievalProfile.reasoningModel;
if (
prepared.some(
(node) => stableJson(jointSemanticModelSelection(node)) !== stableJson(selection),
)
) {
throw new Error("Joint semantic Graph metadata does not match the frozen reasoning model");
}
return indexPreparedSemanticNodes({
graph,
job: {
createdAt: input.createdAt,
knowledgeSpaceId: input.knowledgeSpaceId,
publicationGenerationId: generationId,
},
maxEntitiesPerNode,
maxNodesPerArtifact,
maxRelationsPerNode,
nodes: prepared,
now,
semanticProviderCalls: 0,
semanticProviderCallsMaximum: 0,
});
},
};
}
function jointSemanticModelSelection(node: KnowledgeNode): unknown {
const semantic = node.metadata.semanticChunking;
return isPlainObject(semantic) ? semantic.modelSelection : undefined;
}
async function indexPreparedSemanticNodes({
graph,
job,
maxEntitiesPerNode,
maxNodesPerArtifact,
maxRelationsPerNode,
nodes,
now,
semanticProviderCalls,
semanticProviderCallsMaximum,
}: {
readonly graph: GraphIndexRepository;
readonly job: Pick<
DocumentSemanticEnrichmentJob,
"createdAt" | "knowledgeSpaceId" | "publicationGenerationId"
>;
readonly maxEntitiesPerNode: number;
readonly maxNodesPerArtifact: number;
readonly maxRelationsPerNode: number;
readonly nodes: readonly KnowledgeNode[];
readonly now: () => string;
readonly semanticProviderCalls: number;
readonly semanticProviderCallsMaximum: number;
}): Promise<DocumentSemanticEnrichmentProcessorResult> {
const generationId = PublicationGenerationIdSchema.parse(job.publicationGenerationId);
const qualityRepository = await temporaryNodeRepository(nodes);
const controlled = await createExtractionQualityControlFlow({
maxBatchSize: maxNodesPerArtifact,
maxEligibleEntitiesPerNode: maxEntitiesPerNode,
maxEligibleRelationsPerNode: maxRelationsPerNode,
nodes: qualityRepository,
now,
}).apply({
knowledgeSpaceId: job.knowledgeSpaceId,
nodeIds: nodes.map((node) => node.id),
publicationGenerationId: generationId,
});
if (controlled.missingNodeIds.length > 0) {
throw new Error("Document semantic enrichment quality stage lost immutable nodes");
}
const indexed = await createGraphIndexWriter({
extractionVersion: 1,
graph,
maxBatchSize: maxNodesPerArtifact,
nodes: qualityRepository,
now: () => job.createdAt,
}).indexNodes({
knowledgeSpaceId: job.knowledgeSpaceId,
nodes: controlled.controlledNodes,
publicationGenerationId: generationId,
});
if (indexed.missingNodeIds.length > 0) {
throw new Error("Document semantic enrichment graph stage lost immutable nodes");
}
return {
entitiesExtracted: controlled.controlledNodes.reduce(
(sum, node) => sum + extractedEntitiesFromNodeMetadata(node).length,
0,
),
graphEntityIds: indexed.entities.map((entity) => entity.id),
graphEntitiesIndexed: indexed.stats.entitiesIndexed,
graphRelationIds: indexed.relations.map((relation) => relation.id),
graphRelationsIndexed: indexed.stats.relationsIndexed,
nodesScanned: nodes.length,
semanticProviderCalls,
semanticProviderCallsMaximum,
};
}
async function completeEntityCheckpoints(input: {
readonly checkpoints: DocumentSemanticExtractionCheckpointRepository;
readonly maxConcurrentBatches: number;

2
knowledge-fs/packages/api/src/document-semantic-enrichment-runtime.test.ts
View File

@ -58,7 +58,9 @@ describe("createDocumentSemanticEnrichmentRuntime", () => {
const processor = {
process: vi.fn(async () => ({
entitiesExtracted: 4,
graphEntityIds: [uuid(31), uuid(32), uuid(33)],
graphEntitiesIndexed: 3,
graphRelationIds: [uuid(34)],
graphRelationsIndexed: 1,
nodesScanned: 8,
semanticProviderCalls: 2,

752
knowledge-fs/packages/api/src/index-reindexer-semantic.test.ts Normal file
View File

@ -0,0 +1,752 @@
import type { ComputeRuntime } from "@knowledge/compute";
import { KnowledgeNodeSchema, ParseArtifactSchema } from "@knowledge/core";
import { describe, expect, it } from "vitest";
import { createIncrementalReindexer } from "./index-reindexer";
import { createInMemoryKnowledgeNodeRepository } from "./knowledge-node-repository";
import { createLlmSemanticChunker } from "./llm-semantic-chunker";
import { createInMemoryParseArtifactRepository } from "./parse-artifact-repository";
const KNOWLEDGE_SPACE_ID = "018f0d60-7a49-7cc2-9c1b-5b36f18f2c40";
const DOCUMENT_ASSET_ID = "018f0d60-7a49-7cc2-9c1b-5b36f18f2c41";
const PARSE_ARTIFACT_ID = "018f0d60-7a49-7cc2-9c1b-5b36f18f2c42";
const GENERATION_A = "018f0d60-7a49-7cc2-9c1b-5b36f18f2ca1";
const GENERATION_B = "018f0d60-7a49-7cc2-9c1b-5b36f18f2ca2";
function parseArtifact() {
return ParseArtifactSchema.parse({
artifactHash: "a".repeat(64),
contentType: "text",
createdAt: "2026-08-13T12:00:00.000Z",
documentAssetId: DOCUMENT_ASSET_ID,
elements: [
{
id: "element-1",
sectionPath: ["Invoice"],
sourceLocation: { endOffset: 18, startOffset: 0 },
text: "Invoice buyer amount",
type: "paragraph",
},
],
id: PARSE_ARTIFACT_ID,
metadata: {},
parser: "native-markdown",
version: 1,
});
}
function computeRuntime(onChunk?: () => void): ComputeRuntime {
return {
chunkParseArtifact: (input) => {
onChunk?.();
return [
KnowledgeNodeSchema.parse({
artifactHash: input.parseArtifact.artifactHash,
documentAssetId: input.parseArtifact.documentAssetId,
endOffset: 18,
id: "018f0d60-7a49-7cc2-9c1b-5b36f18f2d42",
kind: "chunk",
knowledgeSpaceId: input.knowledgeSpaceId,
metadata: { chunkIndex: 0, elementIds: ["element-1"] },
parseArtifactId: input.parseArtifact.id,
permissionScope: input.permissionScope ? [...input.permissionScope] : undefined,
sourceLocation: { endOffset: 18, sectionPath: ["Invoice"], startOffset: 0 },
startOffset: 0,
text: "Invoice buyer amount",
}),
];
},
countApproxTokens: () => 1,
countTokens: () => 1,
diffText: () => ({ operations: [], stats: { delete: 0, equal: 0, insert: 0 } }),
packEvidence: () => ({ context: "", items: [], omitted: [], tokenBudget: 1, usedTokens: 0 }),
rrfFuse: () => [],
};
}
function retrievalProfile() {
return {
defaultMode: "research" as const,
reasoningModel: {
model: "reasoning-v1",
pluginId: "reasoning-plugin",
provider: "plugin-daemon",
},
rerank: { enabled: false as const },
revision: 1,
scoreThreshold: { enabled: false as const, stage: "mode-final" as const },
topK: 10,
};
}
function semanticChunker(onCall: () => void) {
return createLlmSemanticChunker({
maxChunkChars: 512,
maxWindowChars: 600,
now: () => "2026-08-13T12:00:00.000Z",
promptVersion: "semantic-reindex-v1",
reasoningProviderFactory: () => ({
kind: "plugin-daemon",
async *stream(input) {
onCall();
const user = input.messages.find((message) => message.role === "user");
const payload = JSON.parse(user?.content ?? "{}") as {
units: Array<{ id: string }>;
};
yield {
delta: JSON.stringify({
chunks: [
{
endUnitId: payload.units.at(-1)?.id,
entities: [
{
confidence: 0.99,
id: "invoice",
text: "Invoice",
type: "policy",
},
],
relations: [],
sectionPath: ["Invoice", "Buyer and amount"],
sectionSummary: "Invoice identity, buyer, and amount details.",
startUnitId: payload.units[0]?.id,
},
],
}),
type: "delta" as const,
};
yield {
finishReason: "stop",
metadata: { model: input.model, provider: "plugin-daemon" },
type: "done" as const,
};
},
}),
});
}
function echoSemanticChunker(onCall: () => void) {
return createLlmSemanticChunker({
maxChunkChars: 512,
maxWindowChars: 600,
now: () => "2026-08-13T12:00:00.000Z",
promptVersion: "semantic-reindex-v1",
reasoningProviderFactory: () => ({
kind: "plugin-daemon",
async *stream(input) {
onCall();
const user = input.messages.find((message) => message.role === "user");
const payload = JSON.parse(user?.content ?? "{}") as {
sectionPath: string[];
units: Array<{ id: string; type: string }>;
};
yield {
delta: JSON.stringify({
chunks: [
{
endUnitId: payload.units.at(-1)?.id,
entities: [],
relations: [],
sectionPath: payload.sectionPath,
...(payload.units[0]?.type === "paragraph"
? { sectionSummary: "Semantic paragraph summary." }
: {}),
startUnitId: payload.units[0]?.id,
},
],
}),
type: "delta" as const,
};
yield {
finishReason: "stop",
metadata: { model: input.model, provider: "plugin-daemon" },
type: "done" as const,
};
},
}),
});
}
function richParseArtifact() {
return ParseArtifactSchema.parse({
artifactHash: "b".repeat(64),
contentType: "structured",
createdAt: "2026-08-13T12:00:00.000Z",
documentAssetId: DOCUMENT_ASSET_ID,
elements: [
{
id: "paragraph",
metadata: {},
pageNumber: 1,
sectionPath: ["Rich"],
text: "Paragraph content.",
type: "paragraph",
},
{
id: "table",
metadata: { title: "Amounts" },
pageNumber: 2,
sectionPath: ["Rich"],
text: "Item | Amount",
type: "table",
},
{
id: "image",
metadata: { caption: "Receipt" },
sectionPath: ["Rich"],
text: "Receipt image",
type: "image",
},
],
id: "018f0d60-7a49-7cc2-9c1b-5b36f18f2c49",
metadata: {},
parser: "native-structured",
version: 1,
});
}
function compactCompletionSemanticChunker(onCall: () => void) {
return createLlmSemanticChunker({
maxChunkChars: 512,
maxWindowChars: 600,
reasoningProviderFactory: () => ({
async *stream(input) {
onCall();
const user = input.messages.find((message) => message.role === "user");
const payload = JSON.parse(user?.content ?? "{}") as {
units: Array<{ id: string }>;
};
yield {
delta: JSON.stringify({
chunks: payload.units.map((unit) => ({
endUnitId: unit.id,
entities: [],
relations: [],
startUnitId: unit.id,
})),
}),
type: "delta" as const,
};
yield { type: "done" as const };
},
}),
});
}
describe("incremental reindexer semantic generations", () => {
it("uses the frozen reasoning model and replays the durable generation without another call", async () => {
const nodes = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 4,
maxListLimit: 1,
maxNodes: 4,
});
let llmCalls = 0;
const reindexer = createIncrementalReindexer({
artifacts: createInMemoryParseArtifactRepository({ maxArtifacts: 4 }),
compute: {
...computeRuntime(),
chunkParseArtifact: () => {
throw new Error("deterministic chunker must not run");
},
},
maxNodeReplayPageSize: 1,
maxNodes: 4,
nodes,
semanticChunker: semanticChunker(() => llmCalls++),
});
const input = {
chunkConfig: { maxChunkChars: 512 },
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
parseArtifact: parseArtifact(),
projectionVersion: 1,
publicationGenerationId: GENERATION_A,
retrievalProfile: retrievalProfile(),
tenantId: "tenant-1",
} as const;
const first = await reindexer.reindex(input);
const replay = await reindexer.reindex(input);
expect(first).toMatchObject({ nodesCreated: 1, status: "rebuilt" });
expect(first).toMatchObject({
outlineArtifact: {
elements: [
expect.objectContaining({
sectionPath: ["Invoice", "Buyer and amount"],
text: "Invoice buyer amount",
}),
],
metadata: expect.objectContaining({ semanticCompilation: expect.any(Object) }),
},
});
expect(replay).toMatchObject({
nodeIds: first.status === "rebuilt" ? first.nodeIds : undefined,
nodesCreated: 1,
status: "rebuilt",
});
expect(llmCalls).toBe(1);
await expect(
nodes.getGenerationReceipt?.({
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
parseArtifactId: PARSE_ARTIFACT_ID,
publicationGenerationId: GENERATION_A,
}),
).resolves.toMatchObject({
documentChunkCount: 1,
storedNodeCount: 1,
windowManifest: [expect.objectContaining({ windowId: "window-000000" })],
});
});
it("persists and replays a fully excluded semantic result", async () => {
const nodes = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 4,
maxListLimit: 1,
maxNodes: 4,
});
let llmCalls = 0;
const reindexer = createIncrementalReindexer({
artifacts: createInMemoryParseArtifactRepository({ maxArtifacts: 4 }),
compute: computeRuntime(),
maxNodeReplayPageSize: 1,
maxNodes: 4,
nodes,
semanticChunker: semanticChunker(() => llmCalls++),
});
const input = {
chunkConfig: { maxChunkChars: 512 },
excludedNodeOrdinals: [0],
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
parseArtifact: parseArtifact(),
projectionVersion: 1,
publicationGenerationId: GENERATION_A,
retrievalProfile: retrievalProfile(),
tenantId: "tenant-1",
} as const;
await expect(reindexer.reindex(input)).resolves.toMatchObject({
nodeIds: [],
nodesCreated: 0,
status: "rebuilt",
});
await expect(reindexer.reindex(input)).resolves.toMatchObject({ nodesCreated: 0 });
expect(llmCalls).toBe(1);
});
it("clones an existing semantic node generation for projection-only migration", async () => {
const nodes = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 4,
maxListLimit: 4,
maxNodes: 4,
});
let deterministicCalls = 0;
const reindexer = createIncrementalReindexer({
artifacts: createInMemoryParseArtifactRepository({ maxArtifacts: 4 }),
compute: computeRuntime(() => deterministicCalls++),
maxNodeReplayPageSize: 1,
maxNodes: 4,
nodes,
});
const source = await reindexer.reindex({
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
parseArtifact: parseArtifact(),
projectionVersion: 1,
publicationGenerationId: GENERATION_A,
});
const target = await reindexer.reindex({
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
parseArtifact: parseArtifact(),
projectionVersion: 2,
publicationGenerationId: GENERATION_B,
reuseNodeGenerationId: GENERATION_A,
});
const targetReplay = await reindexer.reindex({
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
parseArtifact: parseArtifact(),
projectionVersion: 2,
publicationGenerationId: GENERATION_B,
reuseNodeGenerationId: GENERATION_A,
});
expect(source).toMatchObject({ nodesCreated: 1 });
expect(target).toMatchObject({ nodesCreated: 1 });
expect(targetReplay).toMatchObject({
nodeIds: target.status === "rebuilt" ? target.nodeIds : undefined,
nodesCreated: 1,
});
expect(deterministicCalls).toBe(1);
});
it("requires tenant and durable receipt capabilities for profile-scoped semantic generations", async () => {
const durableNodes = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 4,
maxListLimit: 4,
maxNodes: 4,
});
const baseOptions = {
artifacts: createInMemoryParseArtifactRepository({ maxArtifacts: 4 }),
compute: computeRuntime(),
maxNodes: 4,
semanticChunker: semanticChunker(() => undefined),
};
await expect(
createIncrementalReindexer({ ...baseOptions, nodes: durableNodes }).reindex({
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
parseArtifact: parseArtifact(),
projectionVersion: 1,
publicationGenerationId: GENERATION_A,
retrievalProfile: retrievalProfile(),
}),
).rejects.toThrow("tenantId is required");
const {
completeGenerationAtomically: _completeGenerationAtomically,
getGenerationReceipt: _getGenerationReceipt,
...nodesWithoutReceipts
} = durableNodes;
await expect(
createIncrementalReindexer({ ...baseOptions, nodes: nodesWithoutReceipts }).reindex({
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
parseArtifact: parseArtifact(),
projectionVersion: 1,
publicationGenerationId: GENERATION_A,
retrievalProfile: retrievalProfile(),
tenantId: "tenant-1",
}),
).rejects.toThrow("requires durable generation receipts");
});
it("persists complete semantic options and builds a typed outline across layout elements", async () => {
const nodes = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 10,
maxListLimit: 10,
maxNodes: 10,
});
let llmCalls = 0;
const reindexer = createIncrementalReindexer({
artifacts: createInMemoryParseArtifactRepository({ maxArtifacts: 4 }),
compute: computeRuntime(),
maxNodes: 10,
nodes,
semanticChunker: echoSemanticChunker(() => llmCalls++),
});
const input = {
chunkConfig: {
maxChunkChars: 512,
maxNodes: 10,
maxWindowChars: 600,
overlapChars: 0,
},
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
language: "zh-CN",
parseArtifact: richParseArtifact(),
permissionScope: ["tenant:one"],
projectionVersion: 1,
publicationGenerationId: GENERATION_A,
retrievalProfile: retrievalProfile(),
tenantId: "tenant-1",
} as const;
const first = await reindexer.reindex(input);
const replay = await reindexer.reindex(input);
expect(first).toMatchObject({
nodesCreated: 3,
outlineArtifact: {
elements: [
expect.objectContaining({
metadata: expect.objectContaining({
semanticSectionSummary: "Semantic paragraph summary.",
}),
pageNumber: 1,
type: "paragraph",
}),
expect.objectContaining({ pageNumber: 2, type: "table" }),
expect.objectContaining({ type: "image" }),
],
},
status: "rebuilt",
});
expect(
first.status === "rebuilt" ? first.outlineArtifact?.elements[2]?.pageNumber : 0,
).toBeUndefined();
expect(replay).toMatchObject({ nodesCreated: 3, status: "rebuilt" });
expect(llmCalls).toBe(3);
await expect(
nodes.getGenerationReceipt?.({
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
parseArtifactId: richParseArtifact().id,
publicationGenerationId: GENERATION_A,
}),
).resolves.toMatchObject({
language: "zh-CN",
permissionScope: ["tenant:one"],
semanticConfig: { maxNodes: 10, overlapChars: 0 },
});
});
it("supports semantic compilation without a publication generation", async () => {
let llmCalls = 0;
const reindexer = createIncrementalReindexer({
artifacts: createInMemoryParseArtifactRepository({ maxArtifacts: 2 }),
compute: computeRuntime(),
maxNodes: 4,
nodes: createInMemoryKnowledgeNodeRepository({
maxBatchSize: 4,
maxListLimit: 4,
maxNodes: 4,
}),
semanticChunker: echoSemanticChunker(() => llmCalls++),
});
await expect(
reindexer.reindex({
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
parseArtifact: parseArtifact(),
projectionVersion: 1,
retrievalProfile: retrievalProfile(),
tenantId: "tenant-1",
}),
).resolves.toMatchObject({ nodesCreated: 1, outlineArtifact: expect.any(Object) });
expect(llmCalls).toBe(1);
});
it("fails closed when durable semantic receipt capabilities are incomplete", async () => {
const durableNodes = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 4,
maxListLimit: 4,
maxNodes: 4,
});
const baseInput = {
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
parseArtifact: parseArtifact(),
projectionVersion: 1,
publicationGenerationId: GENERATION_A,
retrievalProfile: retrievalProfile(),
tenantId: "tenant-1",
} as const;
await expect(
createIncrementalReindexer({
artifacts: createInMemoryParseArtifactRepository({ maxArtifacts: 2 }),
compute: computeRuntime(),
maxNodes: 4,
nodes: durableNodes,
semanticChunker: {
chunk: async () => [],
},
}).reindex(baseInput),
).rejects.toThrow("must expose replay defaults");
const nodesWithoutAtomicPersistence = {
...durableNodes,
completeGenerationAtomically: async () => undefined,
} as unknown as typeof durableNodes;
await expect(
createIncrementalReindexer({
artifacts: createInMemoryParseArtifactRepository({ maxArtifacts: 2 }),
compute: computeRuntime(),
maxNodes: 4,
nodes: nodesWithoutAtomicPersistence,
semanticChunker: echoSemanticChunker(() => undefined),
}).reindex(baseInput),
).rejects.toThrow("requires atomic semantic generation receipts");
});
it("rejects semantic exclusions outside the canonical upper bound before model invocation", async () => {
let llmCalls = 0;
const reindexer = createIncrementalReindexer({
artifacts: createInMemoryParseArtifactRepository({ maxArtifacts: 2 }),
compute: computeRuntime(),
maxNodes: 4,
nodes: createInMemoryKnowledgeNodeRepository({
maxBatchSize: 4,
maxListLimit: 4,
maxNodes: 4,
}),
semanticChunker: echoSemanticChunker(() => llmCalls++),
});
await expect(
reindexer.reindex({
excludedNodeOrdinals: [-1],
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
parseArtifact: parseArtifact(),
projectionVersion: 1,
publicationGenerationId: GENERATION_A,
retrievalProfile: retrievalProfile(),
tenantId: "tenant-1",
}),
).rejects.toThrow("exclusions exceed the canonical chunk upper bound");
expect(llmCalls).toBe(0);
});
it("builds a multi-chunk receipt with a compact optional completion identity", async () => {
const nodes = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 4,
maxListLimit: 4,
maxNodes: 4,
});
let llmCalls = 0;
const source = ParseArtifactSchema.parse({
...parseArtifact(),
artifactHash: "c".repeat(64),
elements: [
{
id: "two-sentences",
metadata: {},
sectionPath: ["Receipt"],
text: "First sentence. Second sentence.",
type: "paragraph",
},
],
id: "018f0d60-7a49-7cc2-9c1b-5b36f18f2c50",
});
const reindexer = createIncrementalReindexer({
artifacts: createInMemoryParseArtifactRepository({ maxArtifacts: 2 }),
compute: computeRuntime(),
maxNodes: 4,
nodes,
semanticChunker: compactCompletionSemanticChunker(() => llmCalls++),
});
await expect(
reindexer.reindex({
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
parseArtifact: source,
projectionVersion: 1,
publicationGenerationId: GENERATION_A,
retrievalProfile: retrievalProfile(),
tenantId: "tenant-1",
}),
).resolves.toMatchObject({ nodesCreated: 2 });
expect(llmCalls).toBe(1);
await expect(
nodes.getGenerationReceipt?.({
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
parseArtifactId: source.id,
publicationGenerationId: GENERATION_A,
}),
).resolves.toMatchObject({
completionCatalog: [{ fingerprint: expect.stringMatching(/^sha256:/u) }],
documentChunkCount: 2,
windowManifest: [expect.objectContaining({ chunkRanges: expect.any(Array) })],
});
});
it("fails closed when a semantic chunker returns corrupt receipt markers", async () => {
const baseInput = {
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
parseArtifact: parseArtifact(),
projectionVersion: 1,
publicationGenerationId: GENERATION_A,
retrievalProfile: retrievalProfile(),
tenantId: "tenant-1",
} as const;
const cases: Array<{
error: string;
mutate: (
node: ReturnType<typeof KnowledgeNodeSchema.parse>,
) => ReturnType<typeof KnowledgeNodeSchema.parse>;
}> = [
{
error: "cannot build semantic window receipt from node marker",
mutate: (node) =>
KnowledgeNodeSchema.parse({
...node,
metadata: { ...node.metadata, semanticChunking: null },
}),
},
{
error: "cannot build semantic window receipt from node marker",
mutate: (node) => {
const marker = node.metadata.semanticChunking as Record<string, unknown>;
return KnowledgeNodeSchema.parse({
...node,
metadata: { ...node.metadata, semanticChunking: { ...marker, unitRange: null } },
});
},
},
{
error: "cannot build semantic window receipt from node marker",
mutate: (node) => {
const marker = node.metadata.semanticChunking as Record<string, unknown>;
return KnowledgeNodeSchema.parse({
...node,
metadata: { ...node.metadata, semanticChunking: { ...marker, windowId: " " } },
});
},
},
{
error: "cannot build semantic window receipt from node marker",
mutate: (node) => {
const marker = node.metadata.semanticChunking as Record<string, unknown>;
return KnowledgeNodeSchema.parse({
...node,
metadata: {
...node.metadata,
semanticChunking: { ...marker, inputFingerprint: "invalid" },
},
});
},
},
{
error: "cannot build semantic window receipt from node marker",
mutate: (node) =>
KnowledgeNodeSchema.parse({
...node,
metadata: { ...node.metadata, chunkIndex: 1 },
}),
},
{
error: "semantic completion identity is missing",
mutate: (node) => {
const marker = node.metadata.semanticChunking as Record<string, unknown>;
return KnowledgeNodeSchema.parse({
...node,
metadata: { ...node.metadata, semanticChunking: { ...marker, completion: null } },
});
},
},
{
error: "semantic completion actualModel is invalid",
mutate: (node) => {
const marker = node.metadata.semanticChunking as Record<string, unknown>;
const completion = marker.completion as Record<string, unknown>;
return KnowledgeNodeSchema.parse({
...node,
metadata: {
...node.metadata,
semanticChunking: {
...marker,
completion: { ...completion, actual: { model: " " } },
},
},
});
},
},
];
for (const testCase of cases) {
const base = echoSemanticChunker(() => undefined);
const corruptingChunker = {
...base,
chunk: async (input: Parameters<typeof base.chunk>[0]) =>
(await base.chunk(input)).map(testCase.mutate),
};
await expect(
createIncrementalReindexer({
artifacts: createInMemoryParseArtifactRepository({ maxArtifacts: 2 }),
compute: computeRuntime(),
maxNodes: 4,
nodes: createInMemoryKnowledgeNodeRepository({
maxBatchSize: 4,
maxListLimit: 4,
maxNodes: 4,
}),
semanticChunker: corruptingChunker,
}).reindex(baseInput),
).rejects.toThrow(testCase.error);
}
});
});

54
knowledge-fs/packages/api/src/index-reindexer.test.ts
View File

@ -775,6 +775,60 @@ describe("incremental reindexer", () => {
nodes,
}),
).toThrow("Incremental reindexer maxNodes must be at least 1");
expect(() =>
createIncrementalReindexer({
artifacts,
compute,
maxNodes: 4,
maxProjectionBatchSize: 0,
nodes,
}),
).toThrow("maxProjectionBatchSize must be at least 1");
expect(() =>
createIncrementalReindexer({
artifacts,
compute,
maxNodeReplayPageSize: 0,
maxNodes: 4,
nodes,
}),
).toThrow("maxNodeReplayPageSize must be at least 1");
const validating = createIncrementalReindexer({ artifacts, compute, maxNodes: 4, nodes });
const validInput = {
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
parseArtifact: parseArtifact({ artifactHash: "8".repeat(64) }),
projectionVersion: 1,
} as const;
await expect(validating.reindex({ ...validInput, knowledgeSpaceId: " " })).rejects.toThrow(
"knowledgeSpaceId is required",
);
await expect(validating.reindex({ ...validInput, projectionVersion: 0 })).rejects.toThrow(
"projectionVersion must be a positive integer",
);
await expect(
validating.reindex({ ...validInput, reuseNodeGenerationId: PUBLICATION_GENERATION_ID }),
).rejects.toThrow("reuseNodeGenerationId requires publicationGenerationId");
await expect(
validating.reindex({
...validInput,
publicationGenerationId: PUBLICATION_GENERATION_ID,
reuseNodeGenerationId: PUBLICATION_GENERATION_ID,
}),
).rejects.toThrow("source and target node generations must be different");
await expect(
validating.reindex({ ...validInput, denseModel: "dense", skipDense: true }),
).rejects.toThrow("skipDense cannot include dense model configuration");
await expect(
validating.reindex({ ...validInput, skipVisual: true, visualModel: "clip" }),
).rejects.toThrow("skipVisual cannot include visual model configuration");
await expect(
validating.reindex({
...validInput,
publicationGenerationId: "018f0d60-7a49-7cc2-9c1b-5b36f18f2c53",
reuseNodeGenerationId: PUBLICATION_GENERATION_ID,
}),
).rejects.toThrow("could not load source generation nodes");
let denseBuilds = 0;
await expect(

900
knowledge-fs/packages/api/src/index-reindexer.ts
View File

File diff suppressed because it is too large Load Diff

2
knowledge-fs/packages/api/src/index.ts
View File

@ -112,6 +112,7 @@ export * from "./document-multimodal-manifest-builder";
export * from "./document-multimodal-manifest-repository";
export * from "./document-pdf-rasterizer";
export * from "./document-outline-builder";
export * from "./document-layout-recomposer";
export * from "./document-outline-evaluation";
export * from "./document-outline-repository";
export * from "./document-offsets";
@ -238,6 +239,7 @@ export * from "./llm-community-summary-provider";
export * from "./llm-entity-extraction-provider";
export * from "./llm-multimodal-answer-provider";
export * from "./llm-relation-extraction-provider";
export * from "./llm-semantic-chunker";
export * from "./knowledge-fs-errors";
export * from "./knowledge-fs-handlers";
export * from "./knowledge-fs-command-registry";

504
knowledge-fs/packages/api/src/knowledge-node-generation-receipt-repository.test.ts Normal file
View File

@ -0,0 +1,504 @@
import type { KnowledgeNode } from "@knowledge/core";
import { KnowledgeNodeSchema } from "@knowledge/core";
import { describe, expect, it } from "vitest";
import {
type KnowledgeNodeGenerationReceipt,
KnowledgeNodeGenerationReceiptConflictError,
createInMemoryKnowledgeNodeRepository,
} from "./knowledge-node-repository";
import {
MAX_KNOWLEDGE_NODE_GENERATION_RECEIPT_BYTES,
knowledgeNodeGenerationReceiptSerializedBytes,
llmSemanticCompletionFingerprint,
maximumKnowledgeNodeGenerationReceiptSerializedBytes,
} from "./semantic-generation-receipt";
const KNOWLEDGE_SPACE_ID = "018f0d60-7a49-7cc2-9c1b-5b36f18f2c40";
const DOCUMENT_ASSET_ID = "018f0d60-7a49-7cc2-9c1b-5b36f18f2c41";
const PARSE_ARTIFACT_ID = "018f0d60-7a49-7cc2-9c1b-5b36f18f2c42";
const PUBLICATION_GENERATION_ID = "018f0d60-7a49-7cc2-9c1b-5b36f18f2ca1";
interface ReceiptPatch {
readonly path: readonly string[];
readonly value: unknown;
}
function knowledgeNode(chunkIndex: number): KnowledgeNode {
const startOffset = chunkIndex * 20;
return KnowledgeNodeSchema.parse({
artifactHash: "a".repeat(64),
documentAssetId: DOCUMENT_ASSET_ID,
endOffset: startOffset + 12,
id: `018f0d60-7a49-7cc2-9c1b-${(0x5b36f18f8a00 + chunkIndex).toString(16).padStart(12, "0")}`,
kind: "chunk",
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
metadata: { chunkIndex },
parseArtifactId: PARSE_ARTIFACT_ID,
permissionScope: ["tenant:tenant-1"],
publicationGenerationId: PUBLICATION_GENERATION_ID,
sourceLocation: { endOffset: startOffset + 12, sectionPath: [], startOffset },
startOffset,
text: `chunk ${chunkIndex}`,
});
}
function semanticGenerationReceipt(
overrides: Partial<KnowledgeNodeGenerationReceipt> = {},
): KnowledgeNodeGenerationReceipt {
const completion = {
actualModel: "reasoning-v1",
actualProvider: "plugin-daemon",
finishReason: "stop",
transportProvider: "plugin-daemon",
};
return {
artifactHash: "a".repeat(64),
completionCatalog: [
{ ...completion, fingerprint: llmSemanticCompletionFingerprint(completion) },
],
documentAssetId: DOCUMENT_ASSET_ID,
documentChunkCount: 1,
excludedNodeOrdinals: [0],
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
modelSelection: {
model: "reasoning-v1",
pluginId: "reasoning-plugin",
provider: "plugin-daemon",
},
parseArtifactId: PARSE_ARTIFACT_ID,
permissionScope: ["tenant:tenant-1"],
promptResponseFingerprint: `sha256:${"b".repeat(64)}`,
publicationGenerationId: PUBLICATION_GENERATION_ID,
requestFingerprint: `sha256:${"c".repeat(64)}`,
responseFingerprint: `sha256:${"d".repeat(64)}`,
schemaVersion: 1,
semanticConfig: {
maxChunkChars: 1_200,
maxNodes: 20_000,
maxWindowChars: 4_800,
overlapChars: 0,
promptVersion: "semantic-v2",
},
storedNodeCount: 0,
storedResponseFingerprint: `sha256:${"e".repeat(64)}`,
windowManifest: [
{
chunkRanges: [["u-000000-000000", "u-000000-000000"]],
committedUnitRange: ["u-000000-000000", "u-000000-000000"],
completionIndex: 0,
coreUnitRange: ["u-000000-000000", "u-000000-000000"],
firstChunkIndex: 0,
inputFingerprint: `sha256:${"1".repeat(64)}`,
responseFingerprint: `sha256:${"2".repeat(64)}`,
windowId: "window-000000",
},
],
...overrides,
};
}
function patch(path: string, value: unknown): ReceiptPatch {
return { path: path.split("."), value };
}
function applyReceiptPatch(target: Record<string, unknown>, input: ReceiptPatch): void {
const finalSegment = input.path.at(-1);
if (!finalSegment) throw new Error("Receipt patch path is required");
let cursor: unknown = target;
for (const segment of input.path.slice(0, -1)) {
cursor = Array.isArray(cursor)
? cursor[Number(segment)]
: isMutableRecord(cursor)
? cursor[segment]
: undefined;
if (cursor === undefined) throw new Error(`Receipt patch path is invalid: ${input.path}`);
}
if (Array.isArray(cursor)) {
cursor[Number(finalSegment)] = input.value;
return;
}
if (!isMutableRecord(cursor)) {
throw new Error(`Receipt patch target is invalid: ${input.path}`);
}
cursor[finalSegment] = input.value;
}
function isMutableRecord(value: unknown): value is Record<string, unknown> {
return value !== null && typeof value === "object" && !Array.isArray(value);
}
describe("KnowledgeNode semantic generation receipts", () => {
it("persists an all-excluded generation and replays it without node rows", async () => {
const repository = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 1,
maxListLimit: 1,
maxNodes: 2,
});
const receipt = semanticGenerationReceipt();
await expect(
repository.completeGenerationAtomically?.({ nodes: [], receipt }),
).resolves.toEqual({ nodes: [], receipt });
await expect(
repository.getGenerationReceipt?.({
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
parseArtifactId: PARSE_ARTIFACT_ID,
publicationGenerationId: PUBLICATION_GENERATION_ID,
}),
).resolves.toEqual(receipt);
await expect(
repository.completeGenerationAtomically?.({ nodes: [], receipt }),
).resolves.toEqual({ nodes: [], receipt });
});
it("rejects a conflicting replay for the same immutable generation", async () => {
const repository = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 1,
maxListLimit: 1,
maxNodes: 2,
});
await repository.completeGenerationAtomically?.({
nodes: [],
receipt: semanticGenerationReceipt(),
});
await expect(
repository.completeGenerationAtomically?.({
nodes: [],
receipt: semanticGenerationReceipt({ language: "zh-CN" }),
}),
).rejects.toBeInstanceOf(KnowledgeNodeGenerationReceiptConflictError);
});
it("atomically persists a complete immutable generation beyond ordinary batch size", async () => {
const repository = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 1,
maxListLimit: 2,
maxNodes: 2,
});
const nodes = [knowledgeNode(0), knowledgeNode(1)];
const baseWindow = semanticGenerationReceipt().windowManifest[0];
if (!baseWindow) throw new Error("semantic receipt fixture requires one window");
const receipt = semanticGenerationReceipt({
documentChunkCount: 2,
excludedNodeOrdinals: [],
storedNodeCount: 2,
windowManifest: [
{
...baseWindow,
chunkRanges: [
["u-000000-000000", "u-000000-000000"],
["u-000000-000001", "u-000000-000001"],
],
},
],
});
await expect(repository.upsertMany(nodes)).rejects.toThrow("maxBatchSize=1");
await expect(repository.completeGenerationAtomically?.({ nodes, receipt })).resolves.toEqual({
nodes,
receipt,
});
});
it("rejects receipts whose node count or identity does not match persisted nodes", async () => {
const repository = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 1,
maxListLimit: 1,
maxNodes: 2,
});
await expect(
repository.completeGenerationAtomically?.({
nodes: [knowledgeNode(0)],
receipt: semanticGenerationReceipt(),
}),
).rejects.toThrow("storedNodeCount does not match nodes");
const storedReceipt = semanticGenerationReceipt({
documentChunkCount: 1,
excludedNodeOrdinals: [],
storedNodeCount: 1,
});
for (const node of [
KnowledgeNodeSchema.parse({ ...knowledgeNode(0), artifactHash: "b".repeat(64) }),
KnowledgeNodeSchema.parse({
...knowledgeNode(0),
metadata: { ...knowledgeNode(0).metadata, chunkIndex: "invalid" },
}),
]) {
await expect(
repository.completeGenerationAtomically?.({ nodes: [node], receipt: storedReceipt }),
).rejects.toThrow("identity does not match nodes");
}
await expect(
repository.completeGenerationAtomically?.({
nodes: [KnowledgeNodeSchema.parse({ ...knowledgeNode(0), metadata: { chunkIndex: 1 } })],
receipt: storedReceipt,
}),
).rejects.toThrow("chunk indexes do not match nodes");
});
it("round-trips minimal terminal identity and a manifest without look-ahead", async () => {
const repository = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 1,
maxListLimit: 1,
maxNodes: 2,
});
const baseWindow = semanticGenerationReceipt().windowManifest[0];
if (!baseWindow) throw new Error("semantic receipt fixture requires one window");
const completion = {};
const receipt = semanticGenerationReceipt({
completionCatalog: [
{ fingerprint: llmSemanticCompletionFingerprint(completion), ...completion },
],
windowManifest: [{ ...baseWindow, lookAheadUnitRange: undefined }],
});
await expect(
repository.completeGenerationAtomically?.({ nodes: [], receipt }),
).resolves.toEqual({ nodes: [], receipt });
await expect(
repository.getGenerationReceipt?.({
knowledgeSpaceId: KNOWLEDGE_SPACE_ID,
parseArtifactId: PARSE_ARTIFACT_ID,
publicationGenerationId: "018f0d60-7a49-7cc2-9c1b-5b36f18f2ca2",
}),
).resolves.toBeNull();
});
it("enforces the durable receipt byte limit before persistence", async () => {
const base = semanticGenerationReceipt({ permissionScope: ["x"] });
const baseBytes = knowledgeNodeGenerationReceiptSerializedBytes(base);
const oversized = semanticGenerationReceipt({
permissionScope: [
`x${"y".repeat(MAX_KNOWLEDGE_NODE_GENERATION_RECEIPT_BYTES + 1 - baseBytes)}`,
],
});
const repository = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 1,
maxListLimit: 1,
maxNodes: 2,
});
await expect(
repository.completeGenerationAtomically?.({ nodes: [], receipt: oversized }),
).rejects.toThrow(`exceeds maxBytes=${MAX_KNOWLEDGE_NODE_GENERATION_RECEIPT_BYTES}`);
});
it("fails closed for malformed receipt envelopes, identities, windows, and ranges", async () => {
const repository = createInMemoryKnowledgeNodeRepository({
maxBatchSize: 2,
maxListLimit: 2,
maxNodes: 4,
});
const valid = semanticGenerationReceipt();
const invalidCases: Array<{
readonly expected: string;
readonly patches: readonly ReceiptPatch[];
}> = [
{ expected: "schemaVersion must be 1", patches: [patch("schemaVersion", 2)] },
{ expected: "artifactHash is invalid", patches: [patch("artifactHash", "bad")] },
{
expected: "maxChunkChars must be at least 1",
patches: [patch("semanticConfig.maxChunkChars", 0)],
},
{
expected: "maxNodes must be at least 1",
patches: [patch("semanticConfig.maxNodes", 0)],
},
{
expected: "maxWindowChars must be at least 1",
patches: [patch("semanticConfig.maxWindowChars", 0)],
},
{
expected: "overlapChars is invalid",
patches: [patch("semanticConfig.overlapChars", -1)],
},
{
expected: "overlapChars is invalid",
patches: [patch("semanticConfig.overlapChars", 1_200)],
},
{
expected: "semantic config is invalid",
patches: [patch("semanticConfig.maxWindowChars", 1_199)],
},
{
expected: "semantic config is invalid",
patches: [patch("semanticConfig.promptVersion", " ")],
},
{
expected: "node counts are invalid",
patches: [patch("documentChunkCount", -1)],
},
{
expected: "node counts are invalid",
patches: [patch("storedNodeCount", 2)],
},
{
expected: "exclusions are invalid",
patches: [patch("excludedNodeOrdinals", [-1])],
},
{
expected: "exclusions are invalid",
patches: [patch("excludedNodeOrdinals", [1])],
},
{
expected: "exclusions are invalid",
patches: [patch("documentChunkCount", 2), patch("excludedNodeOrdinals", [1, 0])],
},
{
expected: "fingerprint is invalid",
patches: [patch("requestFingerprint", "sha256:bad")],
},
{
expected: "permissionScope is invalid",
patches: [patch("permissionScope", [" "])],
},
{ expected: "language is invalid", patches: [patch("language", " ")] },
{
expected: "completion catalog is invalid",
patches: [patch("completionCatalog", "not-an-array")],
},
{
expected: "completion identity is invalid",
patches: [patch("completionCatalog", [null])],
},
{
expected: "actualModel is invalid",
patches: [patch("completionCatalog.0.actualModel", " ")],
},
{
expected: "actualProvider is invalid",
patches: [patch("completionCatalog.0.actualProvider", "x".repeat(256))],
},
{
expected: "finishReason is invalid",
patches: [patch("completionCatalog.0.finishReason", "x".repeat(65))],
},
{
expected: "transportProvider is invalid",
patches: [patch("completionCatalog.0.transportProvider", 42)],
},
{
expected: "completion identity is invalid",
patches: [patch("completionCatalog.0.fingerprint", `sha256:${"0".repeat(64)}`)],
},
{
expected: "completion identity is invalid",
patches: [
patch("completionCatalog", [valid.completionCatalog[0], valid.completionCatalog[0]]),
],
},
{
expected: "window manifest is incomplete",
patches: [patch("windowManifest", [])],
},
{
expected: "window manifest is incomplete",
patches: [patch("completionCatalog", [])],
},
{
expected: "window manifest is invalid",
patches: [patch("windowManifest", [null])],
},
{
expected: "window manifest is invalid",
patches: [patch("windowManifest.0.windowId", "window-x")],
},
{
expected: "window manifest is invalid",
patches: [patch("windowManifest.0.inputFingerprint", "bad")],
},
{
expected: "window manifest is invalid",
patches: [patch("windowManifest.0.responseFingerprint", "bad")],
},
{
expected: "window manifest is invalid",
patches: [patch("windowManifest.0.completionIndex", -1)],
},
{
expected: "window manifest is invalid",
patches: [patch("windowManifest.0.firstChunkIndex", 1)],
},
{
expected: "window manifest is invalid",
patches: [patch("windowManifest.0.chunkRanges", [])],
},
{
expected: "window unit range is invalid",
patches: [patch("windowManifest.0.coreUnitRange", ["bad", "bad"])],
},
{
expected: "window unit range is invalid",
patches: [patch("windowManifest.0.chunkRanges", [["u-000000-000000"]])],
},
{
expected: "window chunks do not cover the document",
patches: [
patch("documentChunkCount", 2),
patch("storedNodeCount", 1),
patch("excludedNodeOrdinals", [1]),
],
},
];
for (const { expected, patches } of invalidCases) {
const receipt = structuredClone(valid) as unknown as Record<string, unknown>;
for (const receiptPatch of patches) applyReceiptPatch(receipt, receiptPatch);
await expect(
repository.completeGenerationAtomically?.({
nodes: [],
receipt: receipt as unknown as KnowledgeNodeGenerationReceipt,
}),
).rejects.toThrow(expected);
}
});
it("computes exact empty and bounded receipt admission sizes", () => {
const empty = semanticGenerationReceipt({
completionCatalog: [],
documentChunkCount: 0,
excludedNodeOrdinals: [],
storedNodeCount: 0,
windowManifest: [],
});
expect(
maximumKnowledgeNodeGenerationReceiptSerializedBytes({
emptyReceipt: empty,
maximumChunkCount: 0,
maximumWindowCount: 0,
}),
).toBe(knowledgeNodeGenerationReceiptSerializedBytes(empty));
expect(
maximumKnowledgeNodeGenerationReceiptSerializedBytes({
emptyReceipt: empty,
maximumChunkCount: 2,
maximumWindowCount: 1,
}),
).toBeGreaterThan(knowledgeNodeGenerationReceiptSerializedBytes(empty));
expect(() =>
maximumKnowledgeNodeGenerationReceiptSerializedBytes({
emptyReceipt: semanticGenerationReceipt(),
maximumChunkCount: 1,
maximumWindowCount: 1,
}),
).toThrow("requires empty dynamic arrays");
for (const [maximumChunkCount, maximumWindowCount] of [
[-1, 0],
[0, -1],
[0, 1],
] as const) {
expect(() =>
maximumKnowledgeNodeGenerationReceiptSerializedBytes({
emptyReceipt: empty,
maximumChunkCount,
maximumWindowCount,
}),
).toThrow("admission bounds are invalid");
}
});
});

776
knowledge-fs/packages/api/src/knowledge-node-repository.ts
View File

@ -7,8 +7,11 @@ import type {
} from "@knowledge/core";
import {
KnowledgeNodeSchema,
KnowledgeSpaceModelSelectionSchema,
PUBLICATION_GENERATION_ID_SENTINEL,
PublicationGenerationIdSchema,
UuidSchema,
stableJson,
} from "@knowledge/core";
import { numberColumn, optionalStringColumn, stringColumn } from "./database-row-utils";
@ -23,7 +26,36 @@ import {
assertExactGenerationReplay,
assertInMemoryGenerationNotPublished,
} from "./generation-immutability";
import { cloneJsonObject, jsonObjectColumn, jsonStringArrayColumn } from "./json-utils";
import {
cloneJsonObject,
isPlainObject,
jsonObjectColumn,
jsonStringArrayColumn,
} from "./json-utils";
import {
type KnowledgeNodeGenerationCompletionReceipt,
type KnowledgeNodeGenerationReceipt,
type KnowledgeNodeGenerationUnitRangeReceipt,
type KnowledgeNodeGenerationWindowReceipt,
type KnowledgeNodeSemanticGenerationConfig,
MAX_KNOWLEDGE_NODE_GENERATION_RECEIPT_BYTES,
MAX_LLM_SEMANTIC_COMPLETION_IDENTITIES,
MAX_LLM_SEMANTIC_FINISH_REASON_CODE_POINTS,
MAX_LLM_SEMANTIC_TERMINAL_IDENTITY_CODE_POINTS,
MAX_LLM_SEMANTIC_UNIT_ID_CODE_POINTS,
MAX_LLM_SEMANTIC_WINDOWS,
MAX_LLM_SEMANTIC_WINDOW_ID_CODE_POINTS,
knowledgeNodeGenerationReceiptSerializedBytes,
llmSemanticCompletionFingerprint,
} from "./semantic-generation-receipt";
export type {
KnowledgeNodeGenerationCompletionReceipt,
KnowledgeNodeGenerationReceipt,
KnowledgeNodeGenerationUnitRangeReceipt,
KnowledgeNodeGenerationWindowReceipt,
KnowledgeNodeSemanticGenerationConfig,
} from "./semantic-generation-receipt";
export interface KnowledgeNodeCursor {
readonly id: string;
@ -99,12 +131,34 @@ export interface UpdateKnowledgeNodeMetadataManyInput {
readonly publicationGenerationId?: string | undefined;
}
export interface KnowledgeNodeGenerationReceiptLookupInput {
readonly knowledgeSpaceId: string;
readonly parseArtifactId: string;
readonly publicationGenerationId: string;
}
export interface CompleteKnowledgeNodeGenerationInput {
readonly nodes: readonly KnowledgeNode[];
readonly receipt: KnowledgeNodeGenerationReceipt;
}
export interface CompleteKnowledgeNodeGenerationResult {
readonly nodes: KnowledgeNode[];
readonly receipt: KnowledgeNodeGenerationReceipt;
}
export interface KnowledgeNodeRepository {
completeGenerationAtomically?(
input: CompleteKnowledgeNodeGenerationInput,
): Promise<CompleteKnowledgeNodeGenerationResult>;
createMany(nodes: readonly KnowledgeNode[]): Promise<KnowledgeNode[]>;
deleteByDocumentAsset(
input: DeleteKnowledgeNodesByDocumentAssetInput,
): Promise<DeleteKnowledgeNodesResult>;
get(input: KnowledgeNodeLookupInput): Promise<KnowledgeNode | null>;
getGenerationReceipt?(
input: KnowledgeNodeGenerationReceiptLookupInput,
): Promise<KnowledgeNodeGenerationReceipt | null>;
getMany(input: GetManyKnowledgeNodesInput): Promise<KnowledgeNode[]>;
/**
* Reads immutable evidence references by their globally unique ids without selecting a
@ -117,6 +171,11 @@ export interface KnowledgeNodeRepository {
): Promise<readonly string[]>;
listBySpace(input: ListKnowledgeNodesBySpaceInput): Promise<ListKnowledgeNodesBySpaceResult>;
updateMetadataMany(input: UpdateKnowledgeNodeMetadataManyInput): Promise<KnowledgeNode[]>;
/**
* Persists one immutable publication generation in a single repository transaction while the
* implementation may split SQL statements into repository-safe batches.
*/
upsertGenerationAtomically?(nodes: readonly KnowledgeNode[]): Promise<KnowledgeNode[]>;
upsertMany(nodes: readonly KnowledgeNode[]): Promise<KnowledgeNode[]>;
}
@ -151,6 +210,12 @@ export class KnowledgeNodeLogicalConflictError extends Error {
}
}
export class KnowledgeNodeGenerationReceiptConflictError extends Error {
constructor() {
super("Knowledge node generation receipt conflicts with the immutable persisted receipt");
}
}
export function createInMemoryKnowledgeNodeRepository({
maxBatchSize,
maxListLimit,
@ -160,8 +225,83 @@ export function createInMemoryKnowledgeNodeRepository({
validateKnowledgeNodeRepositoryBounds({ maxBatchSize, maxListLimit, maxNodes });
const nodes = new Map<string, KnowledgeNode>();
const generationReceipts = new Map<string, KnowledgeNodeGenerationReceipt>();
const upsertAtomically = (input: readonly KnowledgeNode[]): KnowledgeNode[] => {
const parsed = input.map((node) => cloneKnowledgeNode(KnowledgeNodeSchema.parse(node)));
validateKnowledgeNodeLogicalBatch(parsed);
const next = new Map(nodes);
const persisted: KnowledgeNode[] = [];
for (const node of parsed) {
const existingById = next.get(node.id);
if (existingById && !hasSameKnowledgeNodeOwnership(existingById, node)) {
throw new KnowledgeNodeOwnershipConflictError(node.id);
}
if (
existingById &&
knowledgeNodeLogicalIdentity(existingById) !== knowledgeNodeLogicalIdentity(node)
) {
throw new KnowledgeNodeLogicalConflictError();
}
const existingByLogicalIdentity = findKnowledgeNodeByLogicalIdentity(next.values(), node);
if (
existingById &&
existingByLogicalIdentity &&
existingById.id !== existingByLogicalIdentity.id
) {
throw new KnowledgeNodeLogicalConflictError();
}
if (
existingByLogicalIdentity &&
!hasSameKnowledgeNodeOwnership(existingByLogicalIdentity, node)
) {
throw new KnowledgeNodeOwnershipConflictError(node.id);
}
const existing = existingById ?? existingByLogicalIdentity;
if (existing && node.publicationGenerationId) {
assertExactGenerationReplay({
componentType: "knowledge-node",
incoming: node,
logicalKey: knowledgeNodeLogicalIdentity(node),
persisted: existing,
});
persisted.push(existing);
continue;
}
const stored = existing ? { ...node, id: existing.id } : node;
next.set(stored.id, cloneKnowledgeNode(stored));
persisted.push(stored);
}
if (next.size > maxNodes) {
throw new KnowledgeNodeCapacityExceededError(maxNodes);
}
nodes.clear();
for (const [id, node] of next) {
nodes.set(id, node);
}
return persisted.map(cloneKnowledgeNode);
};
return {
completeGenerationAtomically: async (input) => {
const receipt = validateKnowledgeNodeGenerationCompletion(input);
const key = knowledgeNodeGenerationReceiptKey(receipt);
const existingReceipt = generationReceipts.get(key);
if (existingReceipt && stableJson(existingReceipt) !== stableJson(receipt)) {
throw new KnowledgeNodeGenerationReceiptConflictError();
}
const persistedNodes = input.nodes.length > 0 ? upsertAtomically(input.nodes) : [];
generationReceipts.set(key, cloneKnowledgeNodeGenerationReceipt(receipt));
return {
nodes: persistedNodes,
receipt: cloneKnowledgeNodeGenerationReceipt(existingReceipt ?? receipt),
};
},
createMany: async (input) => {
validateKnowledgeNodeBatch(input, maxBatchSize);
const parsed = input.map((node) => cloneKnowledgeNode(KnowledgeNodeSchema.parse(node)));
@ -213,66 +353,13 @@ export function createInMemoryKnowledgeNodeRepository({
return persisted.map(cloneKnowledgeNode);
},
upsertGenerationAtomically: async (input) => {
validateKnowledgeNodeGenerationBatch(input);
return upsertAtomically(input);
},
upsertMany: async (input) => {
validateKnowledgeNodeBatch(input, maxBatchSize);
const parsed = input.map((node) => cloneKnowledgeNode(KnowledgeNodeSchema.parse(node)));
validateKnowledgeNodeLogicalBatch(parsed);
const next = new Map(nodes);
const persisted: KnowledgeNode[] = [];
for (const node of parsed) {
const existingById = next.get(node.id);
if (existingById && !hasSameKnowledgeNodeOwnership(existingById, node)) {
throw new KnowledgeNodeOwnershipConflictError(node.id);
}
if (
existingById &&
knowledgeNodeLogicalIdentity(existingById) !== knowledgeNodeLogicalIdentity(node)
) {
throw new KnowledgeNodeLogicalConflictError();
}
const existingByLogicalIdentity = findKnowledgeNodeByLogicalIdentity(next.values(), node);
if (
existingById &&
existingByLogicalIdentity &&
existingById.id !== existingByLogicalIdentity.id
) {
throw new KnowledgeNodeLogicalConflictError();
}
if (
existingByLogicalIdentity &&
!hasSameKnowledgeNodeOwnership(existingByLogicalIdentity, node)
) {
throw new KnowledgeNodeOwnershipConflictError(node.id);
}
const existing = existingById ?? existingByLogicalIdentity;
if (existing && node.publicationGenerationId) {
assertExactGenerationReplay({
componentType: "knowledge-node",
incoming: node,
logicalKey: knowledgeNodeLogicalIdentity(node),
persisted: existing,
});
persisted.push(existing);
continue;
}
const stored = existing ? { ...node, id: existing.id } : node;
next.set(stored.id, cloneKnowledgeNode(stored));
persisted.push(stored);
}
if (next.size > maxNodes) {
throw new KnowledgeNodeCapacityExceededError(maxNodes);
}
nodes.clear();
for (const [id, node] of next) {
nodes.set(id, node);
}
return persisted.map(cloneKnowledgeNode);
return upsertAtomically(input);
},
deleteByDocumentAsset: async ({ documentAssetId, knowledgeSpaceId, maxNodes }) => {
if (!Number.isInteger(maxNodes) || maxNodes < 1) {
@ -321,6 +408,11 @@ export function createInMemoryKnowledgeNodeRepository({
? cloneKnowledgeNode(node)
: null;
},
getGenerationReceipt: async (input) => {
const normalized = normalizeKnowledgeNodeGenerationReceiptLookup(input);
const receipt = generationReceipts.get(knowledgeNodeGenerationReceiptKey(normalized));
return receipt ? cloneKnowledgeNodeGenerationReceipt(receipt) : null;
},
getMany: async ({ ids, knowledgeSpaceId, publicationGenerationId }) => {
validateKnowledgeNodeBatchIds(ids, maxBatchSize);
const generation = normalizeKnowledgeNodeGeneration(publicationGenerationId);
@ -448,8 +540,38 @@ export function createDatabaseKnowledgeNodeRepository({
maxNodes: Number.MAX_SAFE_INTEGER,
});
const tableName = "knowledge_nodes";
const receiptTableName = "knowledge_node_generation_receipts";
return {
completeGenerationAtomically: async (input) => {
const receipt = validateKnowledgeNodeGenerationCompletion(input);
const parsedNodes = input.nodes.map((node) =>
cloneKnowledgeNode(KnowledgeNodeSchema.parse(node)),
);
validateKnowledgeNodeLogicalBatch(parsedNodes);
return database.transaction(async (transaction) => {
const persisted: KnowledgeNode[] = [];
for (const batch of chunkKnowledgeNodeBatch(parsedNodes, maxBatchSize)) {
persisted.push(
...(await databaseWriteKnowledgeNodeGroups({
database,
executor: transaction,
legacyMode: "upsert",
nodes: batch,
tableName,
})),
);
}
const persistedReceipt = await databaseWriteKnowledgeNodeGenerationReceipt({
database,
executor: transaction,
receipt,
tableName: receiptTableName,
});
return { nodes: persisted, receipt: persistedReceipt };
});
},
createMany: async (input) => {
validateKnowledgeNodeBatch(input, maxBatchSize);
const nodes = input.map((node) => cloneKnowledgeNode(KnowledgeNodeSchema.parse(node)));
@ -473,6 +595,27 @@ export function createDatabaseKnowledgeNodeRepository({
}),
);
},
upsertGenerationAtomically: async (input) => {
validateKnowledgeNodeGenerationBatch(input);
const nodes = input.map((node) => cloneKnowledgeNode(KnowledgeNodeSchema.parse(node)));
validateKnowledgeNodeLogicalBatch(nodes);
return database.transaction(async (transaction) => {
const persisted: KnowledgeNode[] = [];
for (const batch of chunkKnowledgeNodeBatch(nodes, maxBatchSize)) {
persisted.push(
...(await databaseWriteKnowledgeNodeGroups({
database,
executor: transaction,
legacyMode: "upsert",
nodes: batch,
tableName,
})),
);
}
return persisted;
});
},
upsertMany: async (input) => {
validateKnowledgeNodeBatch(input, maxBatchSize);
const nodes = input.map((node) => cloneKnowledgeNode(KnowledgeNodeSchema.parse(node)));
@ -589,6 +732,30 @@ export function createDatabaseKnowledgeNodeRepository({
return result.rows[0] ? mapKnowledgeNodeRow(result.rows[0]) : null;
},
getGenerationReceipt: async (input) => {
const normalized = normalizeKnowledgeNodeGenerationReceiptLookup(input);
const result = await database.execute({
maxRows: 1,
operation: "select",
params: [
normalized.knowledgeSpaceId,
normalized.publicationGenerationId,
normalized.parseArtifactId,
],
sql: `SELECT * FROM ${quoteDatabaseIdentifier(database, receiptTableName)} WHERE ${quoteDatabaseIdentifier(
database,
"knowledge_space_id",
)} = ${databasePlaceholder(database, 1)} AND ${quoteDatabaseIdentifier(
database,
"publication_generation_id",
)} = ${databasePlaceholder(database, 2)} AND ${quoteDatabaseIdentifier(
database,
"parse_artifact_id",
)} = ${databasePlaceholder(database, 3)} LIMIT 1;`,
tableName: receiptTableName,
});
return result.rows[0] ? mapKnowledgeNodeGenerationReceiptRow(result.rows[0]) : null;
},
getMany: async ({ ids, knowledgeSpaceId, publicationGenerationId }) => {
return databaseKnowledgeNodeGetMany(database, tableName, maxBatchSize, {
ids,
@ -801,6 +968,466 @@ export function knowledgeNodeCursor(node: KnowledgeNode): KnowledgeNodeCursor {
};
}
function validateKnowledgeNodeGenerationCompletion({
nodes,
receipt: inputReceipt,
}: CompleteKnowledgeNodeGenerationInput): KnowledgeNodeGenerationReceipt {
const receipt = validateKnowledgeNodeGenerationReceipt(inputReceipt);
if (nodes.length !== receipt.storedNodeCount) {
throw new Error("Knowledge node generation receipt storedNodeCount does not match nodes");
}
const expectedIndexes = Array.from(
{ length: receipt.documentChunkCount },
(_, index) => index,
).filter((index) => !receipt.excludedNodeOrdinals.includes(index));
const actualIndexes: number[] = [];
for (const candidate of nodes) {
const node = KnowledgeNodeSchema.parse(candidate);
if (
node.knowledgeSpaceId !== receipt.knowledgeSpaceId ||
node.documentAssetId !== receipt.documentAssetId ||
node.parseArtifactId !== receipt.parseArtifactId ||
node.publicationGenerationId !== receipt.publicationGenerationId ||
node.artifactHash !== receipt.artifactHash ||
!Number.isSafeInteger(node.metadata.chunkIndex)
) {
throw new Error("Knowledge node generation receipt identity does not match nodes");
}
actualIndexes.push(node.metadata.chunkIndex as number);
}
actualIndexes.sort((left, right) => left - right);
if (stableJson(actualIndexes) !== stableJson(expectedIndexes)) {
throw new Error("Knowledge node generation receipt chunk indexes do not match nodes");
}
return receipt;
}
function validateKnowledgeNodeGenerationReceipt(
input: KnowledgeNodeGenerationReceipt,
): KnowledgeNodeGenerationReceipt {
assertKnowledgeNodeGenerationReceiptSize(input);
const knowledgeSpaceId = UuidSchema.parse(input.knowledgeSpaceId);
const publicationGenerationId = PublicationGenerationIdSchema.parse(
input.publicationGenerationId,
);
const parseArtifactId = UuidSchema.parse(input.parseArtifactId);
const documentAssetId = UuidSchema.parse(input.documentAssetId);
if (input.schemaVersion !== 1) {
throw new Error("Knowledge node generation receipt schemaVersion must be 1");
}
if (!/^[a-f0-9]{64}$/u.test(input.artifactHash)) {
throw new Error("Knowledge node generation receipt artifactHash is invalid");
}
const semanticConfig = input.semanticConfig;
for (const [name, value] of [
["maxChunkChars", semanticConfig.maxChunkChars],
["maxNodes", semanticConfig.maxNodes],
["maxWindowChars", semanticConfig.maxWindowChars],
] as const) {
if (!Number.isSafeInteger(value) || value < 1) {
throw new Error(`Knowledge node generation receipt ${name} must be at least 1`);
}
}
if (
!Number.isSafeInteger(semanticConfig.overlapChars) ||
semanticConfig.overlapChars < 0 ||
semanticConfig.overlapChars >= semanticConfig.maxChunkChars
) {
throw new Error("Knowledge node generation receipt overlapChars is invalid");
}
if (
semanticConfig.maxWindowChars < semanticConfig.maxChunkChars ||
!semanticConfig.promptVersion.trim()
) {
throw new Error("Knowledge node generation receipt semantic config is invalid");
}
if (
!Number.isSafeInteger(input.documentChunkCount) ||
input.documentChunkCount < 0 ||
input.documentChunkCount > semanticConfig.maxNodes ||
!Number.isSafeInteger(input.storedNodeCount) ||
input.storedNodeCount < 0 ||
input.storedNodeCount > input.documentChunkCount
) {
throw new Error("Knowledge node generation receipt node counts are invalid");
}
const excludedNodeOrdinals = [...input.excludedNodeOrdinals];
if (
excludedNodeOrdinals.some(
(ordinal) =>
!Number.isSafeInteger(ordinal) || ordinal < 0 || ordinal >= input.documentChunkCount,
) ||
excludedNodeOrdinals.some((ordinal, index) => {
const previous = excludedNodeOrdinals[index - 1];
return previous !== undefined && ordinal <= previous;
}) ||
input.storedNodeCount !== input.documentChunkCount - excludedNodeOrdinals.length
) {
throw new Error("Knowledge node generation receipt exclusions are invalid");
}
for (const fingerprint of [
input.promptResponseFingerprint,
input.requestFingerprint,
input.responseFingerprint,
input.storedResponseFingerprint,
]) {
if (!/^sha256:[a-f0-9]{64}$/u.test(fingerprint)) {
throw new Error("Knowledge node generation receipt fingerprint is invalid");
}
}
const permissionScope = [...input.permissionScope];
if (permissionScope.some((scope) => typeof scope !== "string" || !scope.trim())) {
throw new Error("Knowledge node generation receipt permissionScope is invalid");
}
if (input.language !== undefined && !input.language.trim()) {
throw new Error("Knowledge node generation receipt language is invalid");
}
const modelSelection = KnowledgeSpaceModelSelectionSchema.parse(input.modelSelection);
const completionCatalog = validateKnowledgeNodeGenerationCompletionCatalog(
input.completionCatalog,
);
const windowManifest = validateKnowledgeNodeGenerationWindowManifest(
input.windowManifest,
input.documentChunkCount,
semanticConfig.maxNodes,
completionCatalog.length,
);
const receipt: KnowledgeNodeGenerationReceipt = {
artifactHash: input.artifactHash,
completionCatalog,
documentAssetId,
documentChunkCount: input.documentChunkCount,
excludedNodeOrdinals,
knowledgeSpaceId,
...(input.language === undefined ? {} : { language: input.language }),
modelSelection,
parseArtifactId,
permissionScope,
promptResponseFingerprint: input.promptResponseFingerprint,
publicationGenerationId,
requestFingerprint: input.requestFingerprint,
responseFingerprint: input.responseFingerprint,
schemaVersion: 1,
semanticConfig: {
maxChunkChars: semanticConfig.maxChunkChars,
maxNodes: semanticConfig.maxNodes,
maxWindowChars: semanticConfig.maxWindowChars,
overlapChars: semanticConfig.overlapChars,
promptVersion: semanticConfig.promptVersion,
},
storedNodeCount: input.storedNodeCount,
storedResponseFingerprint: input.storedResponseFingerprint,
windowManifest,
};
assertKnowledgeNodeGenerationReceiptSize(receipt);
return receipt;
}
function validateKnowledgeNodeGenerationCompletionCatalog(
input: readonly KnowledgeNodeGenerationCompletionReceipt[],
): KnowledgeNodeGenerationCompletionReceipt[] {
if (!Array.isArray(input) || input.length > MAX_LLM_SEMANTIC_COMPLETION_IDENTITIES) {
throw new Error("Knowledge node generation receipt completion catalog is invalid");
}
const fingerprints = new Set<string>();
const identities = new Set<string>();
return input.map((candidate) => {
if (!isPlainObject(candidate)) {
throw new Error("Knowledge node generation receipt completion identity is invalid");
}
const actualModel = optionalBoundedReceiptString(
candidate.actualModel,
"actualModel",
MAX_LLM_SEMANTIC_TERMINAL_IDENTITY_CODE_POINTS,
);
const actualProvider = optionalBoundedReceiptString(
candidate.actualProvider,
"actualProvider",
MAX_LLM_SEMANTIC_TERMINAL_IDENTITY_CODE_POINTS,
);
const finishReason = optionalBoundedReceiptString(
candidate.finishReason,
"finishReason",
MAX_LLM_SEMANTIC_FINISH_REASON_CODE_POINTS,
);
const transportProvider = optionalBoundedReceiptString(
candidate.transportProvider,
"transportProvider",
MAX_LLM_SEMANTIC_TERMINAL_IDENTITY_CODE_POINTS,
);
const identity = {
...(actualModel ? { actualModel } : {}),
...(actualProvider ? { actualProvider } : {}),
...(finishReason ? { finishReason } : {}),
...(transportProvider ? { transportProvider } : {}),
};
const fingerprint = llmSemanticCompletionFingerprint(identity);
if (
candidate.fingerprint !== fingerprint ||
fingerprints.has(fingerprint) ||
identities.has(stableJson(identity))
) {
throw new Error("Knowledge node generation receipt completion identity is invalid");
}
fingerprints.add(fingerprint);
identities.add(stableJson(identity));
return { fingerprint, ...identity };
});
}
function validateKnowledgeNodeGenerationWindowManifest(
input: readonly KnowledgeNodeGenerationWindowReceipt[],
documentChunkCount: number,
maxNodes: number,
completionCatalogLength: number,
): KnowledgeNodeGenerationWindowReceipt[] {
if (
!Array.isArray(input) ||
input.length > maxNodes ||
input.length > MAX_LLM_SEMANTIC_WINDOWS ||
(documentChunkCount > 0 && input.length === 0) ||
(documentChunkCount === 0 && input.length > 0) ||
(input.length > 0 && completionCatalogLength === 0) ||
completionCatalogLength > input.length
) {
throw new Error("Knowledge node generation receipt window manifest is incomplete");
}
const windowIds = new Set<string>();
let nextChunkIndex = 0;
const windows = input.map((window, windowIndex) => {
if (window === null || typeof window !== "object" || Array.isArray(window)) {
throw new Error("Knowledge node generation receipt window manifest is invalid");
}
if (
!isValidReceiptWindowId(window.windowId) ||
window.windowId !== `window-${windowIndex.toString().padStart(6, "0")}` ||
windowIds.has(window.windowId) ||
!/^sha256:[a-f0-9]{64}$/u.test(window.inputFingerprint) ||
!/^sha256:[a-f0-9]{64}$/u.test(window.responseFingerprint) ||
!Number.isSafeInteger(window.completionIndex) ||
window.completionIndex < 0 ||
window.completionIndex >= completionCatalogLength ||
!Number.isSafeInteger(window.firstChunkIndex) ||
window.firstChunkIndex !== nextChunkIndex ||
!Array.isArray(window.chunkRanges) ||
window.chunkRanges.length < 1 ||
window.chunkRanges.length > maxNodes
) {
throw new Error("Knowledge node generation receipt window manifest is invalid");
}
windowIds.add(window.windowId);
const coreUnitRange = validateKnowledgeNodeGenerationUnitRange(window.coreUnitRange);
const committedUnitRange = validateKnowledgeNodeGenerationUnitRange(window.committedUnitRange);
const lookAheadUnitRange =
window.lookAheadUnitRange === undefined
? undefined
: validateKnowledgeNodeGenerationUnitRange(window.lookAheadUnitRange);
const chunkRanges = window.chunkRanges.map(validateKnowledgeNodeGenerationUnitRange);
nextChunkIndex += chunkRanges.length;
return {
chunkRanges,
committedUnitRange,
completionIndex: window.completionIndex,
coreUnitRange,
firstChunkIndex: window.firstChunkIndex,
inputFingerprint: window.inputFingerprint,
...(lookAheadUnitRange ? { lookAheadUnitRange } : {}),
responseFingerprint: window.responseFingerprint,
windowId: window.windowId,
};
});
if (nextChunkIndex !== documentChunkCount) {
throw new Error("Knowledge node generation receipt window chunks do not cover the document");
}
return windows;
}
function validateKnowledgeNodeGenerationUnitRange(
input: KnowledgeNodeGenerationUnitRangeReceipt,
): KnowledgeNodeGenerationUnitRangeReceipt {
if (!Array.isArray(input) || input.length !== 2 || !input.every(isValidReceiptUnitId)) {
throw new Error("Knowledge node generation receipt window unit range is invalid");
}
return [input[0] as string, input[1] as string];
}
function isValidReceiptWindowId(value: unknown): value is string {
return (
typeof value === "string" &&
Array.from(value).length <= MAX_LLM_SEMANTIC_WINDOW_ID_CODE_POINTS &&
/^window-\d{6,}$/u.test(value)
);
}
function isValidReceiptUnitId(value: unknown): value is string {
return (
typeof value === "string" &&
Array.from(value).length <= MAX_LLM_SEMANTIC_UNIT_ID_CODE_POINTS &&
/^u-\d{6,}-\d{6,}$/u.test(value)
);
}
function optionalBoundedReceiptString(
value: unknown,
name: string,
maxCodePoints: number,
): string | undefined {
if (value === undefined) return undefined;
if (typeof value !== "string" || !value.trim() || Array.from(value).length > maxCodePoints) {
throw new Error(`Knowledge node generation receipt ${name} is invalid`);
}
return value.trim();
}
function assertKnowledgeNodeGenerationReceiptSize(value: unknown): void {
if (
knowledgeNodeGenerationReceiptSerializedBytes(value) >
MAX_KNOWLEDGE_NODE_GENERATION_RECEIPT_BYTES
) {
throw new Error(
`Knowledge node generation receipt exceeds maxBytes=${MAX_KNOWLEDGE_NODE_GENERATION_RECEIPT_BYTES}`,
);
}
}
function normalizeKnowledgeNodeGenerationReceiptLookup(
input: KnowledgeNodeGenerationReceiptLookupInput,
): KnowledgeNodeGenerationReceiptLookupInput {
return {
knowledgeSpaceId: UuidSchema.parse(input.knowledgeSpaceId),
parseArtifactId: UuidSchema.parse(input.parseArtifactId),
publicationGenerationId: PublicationGenerationIdSchema.parse(input.publicationGenerationId),
};
}
function knowledgeNodeGenerationReceiptKey(
input: KnowledgeNodeGenerationReceiptLookupInput,
): string {
return stableJson([input.knowledgeSpaceId, input.publicationGenerationId, input.parseArtifactId]);
}
function cloneKnowledgeNodeGenerationReceipt(
receipt: KnowledgeNodeGenerationReceipt,
): KnowledgeNodeGenerationReceipt {
return validateKnowledgeNodeGenerationReceipt(
JSON.parse(JSON.stringify(receipt)) as KnowledgeNodeGenerationReceipt,
);
}
async function databaseWriteKnowledgeNodeGenerationReceipt({
database,
executor,
receipt,
tableName,
}: {
readonly database: DatabaseAdapter;
readonly executor: DatabaseExecutor;
readonly receipt: KnowledgeNodeGenerationReceipt;
readonly tableName: string;
}): Promise<KnowledgeNodeGenerationReceipt> {
const columns = [
"knowledge_space_id",
"publication_generation_id",
"parse_artifact_id",
"document_asset_id",
"artifact_hash",
"document_chunk_count",
"stored_node_count",
"request_fingerprint",
"response_fingerprint",
"prompt_response_fingerprint",
"receipt",
] as const;
const params: readonly DatabaseQueryValue[] = [
receipt.knowledgeSpaceId,
receipt.publicationGenerationId,
receipt.parseArtifactId,
receipt.documentAssetId,
receipt.artifactHash,
receipt.documentChunkCount,
receipt.storedNodeCount,
receipt.requestFingerprint,
receipt.responseFingerprint,
receipt.promptResponseFingerprint,
JSON.stringify(receipt),
];
const conflictSql =
database.dialect === "postgres"
? " ON CONFLICT DO NOTHING RETURNING *"
: ` ON DUPLICATE KEY UPDATE ${quoteDatabaseIdentifier(
database,
"knowledge_space_id",
)} = ${quoteDatabaseIdentifier(database, "knowledge_space_id")}`;
const inserted = await executor.execute({
maxRows: 1,
operation: "insert",
params,
sql: `INSERT INTO ${quoteDatabaseIdentifier(database, tableName)} (${columns
.map((column) => quoteDatabaseIdentifier(database, column))
.join(", ")}) VALUES (${columns
.map((_, index) =>
index === columns.length - 1
? jsonInsertPlaceholder(database, index + 1, "receipt")
: databasePlaceholder(database, index + 1),
)
.join(", ")})${conflictSql};`,
tableName,
});
const row =
inserted.rows[0] ??
(
await executor.execute({
maxRows: 1,
operation: "select",
params: [
receipt.knowledgeSpaceId,
receipt.publicationGenerationId,
receipt.parseArtifactId,
],
sql: `SELECT * FROM ${quoteDatabaseIdentifier(database, tableName)} WHERE ${quoteDatabaseIdentifier(
database,
"knowledge_space_id",
)} = ${databasePlaceholder(database, 1)} AND ${quoteDatabaseIdentifier(
database,
"publication_generation_id",
)} = ${databasePlaceholder(database, 2)} AND ${quoteDatabaseIdentifier(
database,
"parse_artifact_id",
)} = ${databasePlaceholder(database, 3)} LIMIT 1;`,
tableName,
})
).rows[0];
if (!row) {
throw new Error("Knowledge node generation receipt was not persisted");
}
const persisted = mapKnowledgeNodeGenerationReceiptRow(row);
if (stableJson(persisted) !== stableJson(receipt)) {
throw new KnowledgeNodeGenerationReceiptConflictError();
}
return persisted;
}
function mapKnowledgeNodeGenerationReceiptRow(row: DatabaseRow): KnowledgeNodeGenerationReceipt {
const receipt = validateKnowledgeNodeGenerationReceipt(
jsonObjectColumn(row, "receipt") as unknown as KnowledgeNodeGenerationReceipt,
);
if (
stringColumn(row, "knowledge_space_id") !== receipt.knowledgeSpaceId ||
stringColumn(row, "publication_generation_id") !== receipt.publicationGenerationId ||
stringColumn(row, "parse_artifact_id") !== receipt.parseArtifactId ||
stringColumn(row, "document_asset_id") !== receipt.documentAssetId ||
stringColumn(row, "artifact_hash") !== receipt.artifactHash ||
numberColumn(row, "document_chunk_count") !== receipt.documentChunkCount ||
numberColumn(row, "stored_node_count") !== receipt.storedNodeCount ||
stringColumn(row, "request_fingerprint") !== receipt.requestFingerprint ||
stringColumn(row, "response_fingerprint") !== receipt.responseFingerprint ||
stringColumn(row, "prompt_response_fingerprint") !== receipt.promptResponseFingerprint
) {
throw new KnowledgeNodeGenerationReceiptConflictError();
}
return receipt;
}
async function databaseWriteKnowledgeNodeGroups({
database,
executor,
@ -1278,6 +1905,37 @@ function validateKnowledgeNodeBatch(nodes: readonly KnowledgeNode[], maxBatchSiz
}
}
function validateKnowledgeNodeGenerationBatch(nodes: readonly KnowledgeNode[]): void {
if (nodes.length < 1) {
throw new Error("Knowledge node generation batch must contain at least 1 node");
}
const first = KnowledgeNodeSchema.parse(nodes[0]);
if (!first.publicationGenerationId) {
throw new Error("Knowledge node generation batch requires publicationGenerationId");
}
for (const candidate of nodes) {
const node = KnowledgeNodeSchema.parse(candidate);
if (
node.publicationGenerationId !== first.publicationGenerationId ||
node.knowledgeSpaceId !== first.knowledgeSpaceId ||
node.parseArtifactId !== first.parseArtifactId
) {
throw new Error("Knowledge node generation batch must share one generation and artifact");
}
}
}
function chunkKnowledgeNodeBatch(
nodes: readonly KnowledgeNode[],
maxBatchSize: number,
): KnowledgeNode[][] {
const batches: KnowledgeNode[][] = [];
for (let start = 0; start < nodes.length; start += maxBatchSize) {
batches.push(nodes.slice(start, start + maxBatchSize));
}
return batches;
}
function validateKnowledgeNodeLogicalBatch(nodes: readonly KnowledgeNode[]): void {
const identities = new Set<string>();

338
knowledge-fs/packages/api/src/knowledge-space-profile-migration-candidate-builder.test.ts
View File

@ -3,11 +3,14 @@ import type {
DatabaseExecuteInput,
DatabaseExecuteResult,
DatabaseTransactionCallback,
DocumentAsset,
IndexProjection,
KnowledgePath,
} from "@knowledge/core";
import { describe, expect, it, vi } from "vitest";
import { deterministicChildId } from "./api-shared-utils";
import { buildDocumentOutlineKnowledgePath } from "./document-knowledge-paths";
import type { KnowledgeSpaceProfileMigrationRun } from "./knowledge-space-profile-migration";
import {
type ReplaceKnowledgeSpaceProfileMigrationCandidateSnapshotInput,
@ -334,12 +337,48 @@ describe("profile migration candidate builder", () => {
pageIndexSummaryOutlineRebuilt: true,
publicationStatus: "validating",
});
expect(fixture.reindex).toHaveBeenCalledOnce();
expect(fixture.reindex).toHaveBeenCalledWith(
expect.objectContaining({
denseModel: fixture.baseVectorSpaceId,
embeddingProfile: expect.objectContaining({ model: "embedding-v1" }),
retrievalProfile: expect.objectContaining({
reasoningModel: expect.objectContaining({ model: "reasoning-v2" }),
}),
}),
);
expect(fixture.buildOutline).toHaveBeenCalledWith(
expect.objectContaining({
parseArtifact: expect.objectContaining({
metadata: expect.objectContaining({ semanticCompilation: expect.any(Object) }),
}),
}),
);
expect(fixture.enhance).toHaveBeenCalledOnce();
expect(fixture.materialize).toHaveBeenCalledOnce();
expect(fixture.materializeGraph).toHaveBeenCalledWith(
expect.objectContaining({
createdAt: now,
publicationGenerationId: fixture.expectedGenerationId,
}),
);
expect(fixture.heartbeat.mock.calls.length).toBeGreaterThanOrEqual(5);
expect(fixture.candidateMembers()).toEqual(
expect.arrayContaining([
expect.objectContaining({ componentKey: pathId, componentType: "knowledge-path" }),
expect.objectContaining({
componentType: "knowledge-path",
generationId: fixture.expectedGenerationId,
}),
expect.objectContaining({
componentKey: fixture.rebuiltGraphEntityId,
componentType: "graph-entity",
generationId: fixture.expectedGenerationId,
}),
expect.objectContaining({
componentKey: fixture.rebuiltGraphRelationId,
componentType: "graph-relation",
generationId: fixture.expectedGenerationId,
}),
expect.objectContaining({
componentKey: fixture.rebuiltOutlineId,
componentType: "document-outline",
@ -376,12 +415,32 @@ describe("profile migration candidate builder", () => {
publicationStatus: "validating",
});
expect(fixture.reindex).toHaveBeenCalledOnce();
expect(fixture.reindex).toHaveBeenCalledWith(
expect.objectContaining({
reuseNodeGenerationId: baseGenerationId,
skipVisual: true,
retrievalProfile: expect.objectContaining({
reasoningModel: expect.objectContaining({ model: "reasoning-v1" }),
}),
}),
);
expect(fixture.materializeGraph).toHaveBeenCalledWith(
expect.objectContaining({
createdAt: now,
publicationGenerationId: fixture.expectedGenerationId,
}),
);
expect(fixture.candidateMembers()).toEqual(
expect.arrayContaining([
expect.objectContaining({ componentKey: pathId, componentType: "knowledge-path" }),
expect.objectContaining({ componentKey: fixture.visualProjectionId }),
expect.objectContaining({ componentKey: fixture.rebuiltFtsId }),
expect.objectContaining({ componentKey: fixture.rebuiltDenseId }),
expect.objectContaining({
componentKey: fixture.rebuiltGraphEntityId,
componentType: "graph-entity",
generationId: fixture.expectedGenerationId,
}),
]),
);
expect(
@ -740,6 +799,9 @@ function builderFixture(
const visualProjectionId = deterministicChildId(documentAssetId, "base-visual");
const rebuiltFtsId = deterministicChildId(runId, "rebuilt-fts");
const rebuiltDenseId = deterministicChildId(runId, "rebuilt-dense");
const baseGraphEntityId = deterministicChildId(documentAssetId, "base-graph-entity");
const rebuiltGraphEntityId = deterministicChildId(runId, "rebuilt-graph-entity");
const rebuiltGraphRelationId = deterministicChildId(runId, "rebuilt-graph-relation");
const expectedGenerationId = deterministicChildId(
runId,
`profile-migration:${scope === "full-vector-space" ? "vector-space" : "page-index"}:${documentAssetId}`,
@ -771,33 +833,55 @@ function builderFixture(
publicationGenerationId: baseGenerationId,
version: 1,
};
const asset: DocumentAsset = {
createdAt: now,
filename: "synthetic-invoice.pdf",
id: documentAssetId,
knowledgeSpaceId: spaceId,
metadata: { permissionScope: ["read"] },
mimeType: "application/pdf",
objectKey: "documents/synthetic-invoice.pdf",
parserStatus: "parsed",
sha256: digestA,
sizeBytes: 1_024,
version: 1,
};
const storedPaths = new Map<string, KnowledgePath[]>();
const baseOutlinePath = {
...buildDocumentOutlineKnowledgePath({
asset,
id: pathId,
publicationGenerationId: baseGenerationId,
tenantId,
}),
id: pathId,
};
storedPaths.set(baseGenerationId, [baseOutlinePath]);
let rebuiltOutline: typeof baseOutline | undefined;
let candidate: ProjectionSetPublication | undefined;
let candidateMembers: readonly ProjectionSetPublicationMember[] = [];
const projections = new Map<string, IndexProjection>();
const baseProjectionMembers: ProjectionSetPublicationMember[] = [];
if (scope === "full-vector-space") {
const baseProjections = [
projection(baseFtsId, baseGenerationId, "fts"),
projection(baseDenseId, baseGenerationId, "dense-vector", oldVectorSpaceId),
{
...projection(visualProjectionId, baseGenerationId, "dense-vector", "visual-model"),
metadata: {
documentAssetId,
multimodal: { vectorSpace: "visual" },
},
const baseProjections = [
projection(baseFtsId, baseGenerationId, "fts"),
projection(baseDenseId, baseGenerationId, "dense-vector", oldVectorSpaceId),
{
...projection(visualProjectionId, baseGenerationId, "dense-vector", "visual-model"),
metadata: {
documentAssetId,
multimodal: { vectorSpace: "visual" },
},
];
for (const item of baseProjections) projections.set(item.id, item);
baseProjectionMembers.push(
member("index-projection", baseFtsId, baseGenerationId),
member("index-projection", baseDenseId, baseGenerationId),
member("index-projection", visualProjectionId, baseGenerationId),
);
}
},
];
for (const item of baseProjections) projections.set(item.id, item);
const baseProjectionMembers: ProjectionSetPublicationMember[] = [
member("index-projection", baseFtsId, baseGenerationId),
member("index-projection", baseDenseId, baseGenerationId),
member("index-projection", visualProjectionId, baseGenerationId),
];
const baseMembers = [
member("document-outline", outlineId, baseGenerationId),
member("knowledge-path", pathId, baseGenerationId),
member("graph-entity", baseGraphEntityId, baseGenerationId),
...baseProjectionMembers,
];
const heartbeat = vi.fn(async () => undefined);
@ -805,7 +889,28 @@ function builderFixture(
...outline,
metadata: { summary: { model: "reasoning-v2" } },
}));
const buildOutline = vi.fn(
({ publicationGenerationId }: { readonly publicationGenerationId?: string }) => {
rebuiltOutline = {
...baseOutline,
id: rebuiltOutlineId,
metadata: {},
publicationGenerationId: publicationGenerationId ?? baseGenerationId,
};
return rebuiltOutline as never;
},
);
const materialize = vi.fn(async () => ({ status: "building" }) as never);
const materializeGraph = vi.fn(async () => ({
entitiesExtracted: 1,
graphEntityIds: [rebuiltGraphEntityId],
graphEntitiesIndexed: 1,
graphRelationIds: [rebuiltGraphRelationId],
graphRelationsIndexed: 1,
nodesScanned: 1,
semanticProviderCalls: 0,
semanticProviderCallsMaximum: 0,
}));
const reindex = vi.fn(
async ({ publicationGenerationId }: { readonly publicationGenerationId?: string }) => {
if (options.incompleteReindexReceipt) {
@ -819,13 +924,26 @@ function builderFixture(
}
if (!publicationGenerationId) throw new Error("generation missing");
projections.set(rebuiltFtsId, projection(rebuiltFtsId, publicationGenerationId, "fts"));
const targetVectorSpaceId =
scope === "full-vector-space" ? newVectorSpaceId : oldVectorSpaceId;
projections.set(
rebuiltDenseId,
projection(rebuiltDenseId, publicationGenerationId, "dense-vector", newVectorSpaceId),
projection(rebuiltDenseId, publicationGenerationId, "dense-vector", targetVectorSpaceId),
);
return {
artifact: {} as never,
nodeIds: [deterministicChildId(publicationGenerationId, "semantic-node")],
nodesCreated: 1,
outlineArtifact: {
artifactHash: digestA,
documentAssetId,
elements: [],
id: parseArtifactId,
metadata: { semanticCompilation: { source: "reasoning-v2" } },
parseVersion: "semantic-outline-v1",
parser: "semantic",
version: 1,
} as never,
projectionIds: [rebuiltFtsId, rebuiltDenseId],
projectionsCreated: 2,
status: "rebuilt" as const,
@ -881,8 +999,7 @@ function builderFixture(
}) as never,
},
assets: {
get: async () =>
({ id: documentAssetId, metadata: { permissionScope: ["read"] }, version: 1 }) as never,
get: async () => asset,
},
maxDocuments: 10,
maxMembers: 100,
@ -893,15 +1010,7 @@ function builderFixture(
},
now: () => now,
outlineBuilder: {
build: ({ publicationGenerationId }: { readonly publicationGenerationId?: string }) => {
rebuiltOutline = {
...baseOutline,
id: rebuiltOutlineId,
metadata: {},
publicationGenerationId: publicationGenerationId ?? baseGenerationId,
};
return rebuiltOutline as never;
},
build: buildOutline,
} as never,
outlineSummaryEnhancer: { enhance } as never,
outlines: {
@ -919,59 +1028,106 @@ function builderFixture(
hasCompleteBuild: async () => options.completePageIndex !== false,
materializeBuilding: materialize,
},
paths: {
listPhysicalDescendants: async ({ publicationGenerationId }) => ({
items: storedPaths.get(publicationGenerationId ?? "") ?? [],
}),
upsertMany: async (items) => {
for (const item of items) {
const generationId = item.publicationGenerationId ?? "";
const generation = storedPaths.get(generationId) ?? [];
const existing = generation.findIndex((path) => path.id === item.id);
if (existing >= 0) generation[existing] = item;
else generation.push(item);
storedPaths.set(generationId, generation);
}
return [...items];
},
},
profiles: {
getRevision: async ({ kind }) =>
getRevision: async ({ kind, revision }) =>
kind === "embedding"
? ({
id: "embedding-2",
revision: 2,
snapshot: {
dimension: 3072,
model: "embedding-v2",
pluginId: "plugin-embedding",
provider: "plugin-daemon",
revision: 2,
vectorSpaceId: newVectorSpaceId,
},
snapshotDigest: digestB,
state: "candidate",
} as never)
: ({
id: "retrieval-2",
revision: 2,
snapshot: {
defaultMode: "research",
reasoningModel: {
model: "reasoning-v2",
pluginId: "plugin-reasoning",
? revision === 1
? ({
id: "embedding-1",
revision: 1,
snapshot: {
dimension: 3072,
model: "embedding-v1",
pluginId: "plugin-embedding",
provider: "plugin-daemon",
revision: 1,
vectorSpaceId: oldVectorSpaceId,
},
rerank: { enabled: false },
snapshotDigest: digestA,
state: "active",
} as never)
: ({
id: "embedding-2",
revision: 2,
scoreThreshold: { enabled: false, stage: "mode-final" },
topK: 12,
},
snapshotDigest: digestB,
state: "candidate",
} as never),
snapshot: {
dimension: 3072,
model: "embedding-v2",
pluginId: "plugin-embedding",
provider: "plugin-daemon",
revision: 2,
vectorSpaceId: newVectorSpaceId,
},
snapshotDigest: digestB,
state: "candidate",
} as never)
: revision === 1
? ({
id: "retrieval-1",
revision: 1,
snapshot: {
defaultMode: "research",
reasoningModel: {
model: "reasoning-v1",
pluginId: "plugin-reasoning",
provider: "plugin-daemon",
},
rerank: { enabled: false },
revision: 1,
scoreThreshold: { enabled: false, stage: "mode-final" },
topK: 12,
},
snapshotDigest: digestA,
state: "active",
} as never)
: ({
id: "retrieval-2",
revision: 2,
snapshot: {
defaultMode: "research",
reasoningModel: {
model: "reasoning-v2",
pluginId: "plugin-reasoning",
provider: "plugin-daemon",
},
rerank: { enabled: false },
revision: 2,
scoreThreshold: { enabled: false, stage: "mode-final" },
topK: 12,
},
snapshotDigest: digestB,
state: "candidate",
} as never),
},
projections: {
getMany: async ({ ids }) => ids.flatMap((id) => projections.get(id) ?? []),
},
publications,
reindexer: { reindex } as never,
semanticGraph: { materialize: materializeGraph },
snapshots: { replace },
});
const input = {
...(scope === "full-vector-space"
? {
baseEmbeddingProfile: {
id: "embedding-1",
revision: 1,
snapshotDigest: digestA,
},
}
: {}),
baseEmbeddingProfile: {
id: "embedding-1",
revision: 1,
snapshotDigest: digestA,
},
basePublication: {
fingerprint: baseFingerprint,
headRevision: 3,
@ -992,6 +1148,8 @@ function builderFixture(
};
return {
baseDenseId,
baseVectorSpaceId: oldVectorSpaceId,
buildOutline,
builder,
candidateMembers: () => candidateMembers,
candidateStatus: () => candidate?.status,
@ -1000,9 +1158,12 @@ function builderFixture(
heartbeat,
input,
materialize,
materializeGraph,
published: () => basePublication,
rebuiltDenseId,
rebuiltFtsId,
rebuiltGraphEntityId,
rebuiltGraphRelationId,
rebuiltOutlineId,
reindex,
replace,
@ -1021,15 +1182,25 @@ describe("profile migration structural evaluator", () => {
expect(result).toMatchObject({ passed: true });
});
it("accepts a Research-only reasoning Summary/Outline/PageIndex rebuild without Graph or FTS", async () => {
it("accepts a reasoning rebuild with one semantic generation for path, outline, FTS, and dense", async () => {
const rebuiltGenerationId = deterministicChildId(
runId,
`profile-migration:page-index:${documentAssetId}`,
);
const candidateMembers = [
member("document-outline", outlineId, rebuiltGenerationId),
member("knowledge-path", pathId, rebuiltGenerationId),
member("index-projection", ftsId, rebuiltGenerationId),
member("index-projection", denseId, rebuiltGenerationId),
];
const result = await evaluator({
baseMembers: [member("document-outline", outlineId, baseGenerationId)],
candidateMembers: [member("document-outline", outlineId, rebuiltGenerationId)],
candidateMembers,
outlineGenerationId: rebuiltGenerationId,
projections: [
projection(ftsId, rebuiltGenerationId, "fts"),
projection(denseId, rebuiltGenerationId, "dense-vector", vectorSpaceId),
],
summaryModel: "reasoning-v2",
}).evaluate({
candidate: candidateResult(),
@ -1062,7 +1233,7 @@ describe("profile migration structural evaluator", () => {
expect(result).toMatchObject({ passed: true });
});
it("fails closed when a reasoning candidate drops a non-outline Graph/path member", async () => {
it("fails closed when a reasoning candidate drops a preserved multimodal member", async () => {
const rebuiltGenerationId = deterministicChildId(
runId,
`profile-migration:page-index:${documentAssetId}`,
@ -1070,7 +1241,7 @@ describe("profile migration structural evaluator", () => {
const result = await evaluator({
baseMembers: [
member("document-outline", outlineId, baseGenerationId),
member("knowledge-path", pathId, baseGenerationId),
member("multimodal-manifest", pathId, baseGenerationId),
],
candidateMembers: [member("document-outline", outlineId, rebuiltGenerationId)],
outlineGenerationId: rebuiltGenerationId,
@ -1108,21 +1279,21 @@ function evaluator(input: {
},
pageIndexBuild: { hasCompleteBuild: async () => true },
profiles: {
getRevision: async ({ kind }) =>
getRevision: async ({ kind, revision }) =>
kind === "embedding"
? ({
id: "embedding-2",
revision: 1,
id: revision === 1 ? "embedding-1" : "embedding-2",
revision,
snapshot: {
dimension: 3072,
model: "embedding-v2",
model: revision === 1 ? "embedding-v1" : "embedding-v2",
pluginId: "plugin-embedding",
provider: "plugin-daemon",
revision: 1,
revision,
vectorSpaceId,
},
snapshotDigest: digestB,
state: "candidate",
snapshotDigest: revision === 1 ? digestA : digestB,
state: revision === 1 ? "active" : "candidate",
} as never)
: ({
id: "retrieval-2",
@ -1156,10 +1327,11 @@ function migrationRun(
return {
accessChannel: "interactive",
basePublication: { fingerprint: baseFingerprint, headRevision: 3, id: basePublicationId },
baseEmbeddingProfile: { id: "embedding-1", revision: 1, snapshotDigest: digestA },
baseRetrievalProfile: { id: "retrieval-1", revision: 1, snapshotDigest: digestA },
candidateProfile: {
id: changedKind === "embedding" ? "embedding-2" : "retrieval-2",
revision: changedKind === "embedding" ? 1 : 2,
revision: 2,
snapshotDigest: digestB,
},
candidatePublicationFingerprint: candidateResult().publicationFingerprint,

735
knowledge-fs/packages/api/src/knowledge-space-profile-migration-candidate-builder.ts
View File

@ -4,7 +4,9 @@ import {
type DatabaseAdapter,
type DatabaseQueryValue,
DateTimeSchema,
type DocumentOutline,
type IndexProjection,
type KnowledgePath,
type KnowledgeSpaceEmbeddingProfile,
KnowledgeSpaceEmbeddingProfileSchema,
type KnowledgeSpaceRetrievalProfile,
@ -19,12 +21,18 @@ import {
import { deterministicChildId } from "./api-shared-utils";
import { databasePlaceholder, quoteDatabaseIdentifier } from "./database-sql-utils";
import type { DocumentAssetRepository } from "./document-asset-repository";
import {
buildDocumentOutlineKnowledgePath,
buildDocumentSectionKnowledgePaths,
} from "./document-knowledge-paths";
import type { DocumentOutlineBuilder } from "./document-outline-builder";
import type { DocumentOutlineRepository } from "./document-outline-repository";
import type { DocumentOutlineSummaryEnhancer } from "./document-outline-summary-enhancer";
import type { JointSemanticGraphMaterializer } from "./document-semantic-enrichment-processor";
import type { IndexProjectionRepository } from "./index-projection-repository";
import type { IncrementalReindexer } from "./index-reindexer";
import { isPlainObject } from "./json-utils";
import type { KnowledgePathRepository } from "./knowledge-path-repository";
import { lockKnowledgeSpaceForDeletionAdmission } from "./knowledge-space-deletion-admission";
import type {
KnowledgeSpaceProfileMigrationProfileReference,
@ -259,6 +267,8 @@ export interface RepositoryKnowledgeSpaceProfileMigrationCandidateBuilderOptions
readonly assets: Pick<DocumentAssetRepository, "get">;
readonly maxDocuments: number;
readonly maxMembers: number;
readonly maxPathReadPageSize?: number | undefined;
readonly maxPathsPerDocument?: number | undefined;
readonly maxProjectionBatchSize: number;
readonly members: Pick<ProjectionSetPublicationMemberRepository, "listByFingerprint">;
readonly now?: (() => string) | undefined;
@ -269,6 +279,9 @@ export interface RepositoryKnowledgeSpaceProfileMigrationCandidateBuilderOptions
PublishedPageIndexBuildRepository,
"hasCompleteBuild" | "materializeBuilding"
>;
readonly paths?:
| Pick<KnowledgePathRepository, "listPhysicalDescendants" | "upsertMany">
| undefined;
readonly profiles: Pick<KnowledgeSpaceProfileRepository, "getRevision">;
readonly projections: Required<Pick<IndexProjectionRepository, "getMany">>;
readonly publications: Pick<
@ -276,6 +289,7 @@ export interface RepositoryKnowledgeSpaceProfileMigrationCandidateBuilderOptions
"createCandidate" | "getByFingerprint" | "getPublished" | "validate"
>;
readonly reindexer: Pick<IncrementalReindexer, "reindex">;
readonly semanticGraph?: JointSemanticGraphMaterializer | undefined;
readonly snapshots: KnowledgeSpaceProfileMigrationCandidateSnapshotRepository;
}
@ -303,6 +317,8 @@ export function createRepositoryKnowledgeSpaceProfileMigrationCandidateBuilder({
assets,
maxDocuments,
maxMembers,
maxPathReadPageSize = 100,
maxPathsPerDocument = 20_000,
maxProjectionBatchSize,
members,
now = () => new Date().toISOString(),
@ -310,14 +326,18 @@ export function createRepositoryKnowledgeSpaceProfileMigrationCandidateBuilder({
outlineSummaryEnhancer,
outlines,
pageIndexBuild,
paths,
profiles,
projections,
publications,
reindexer,
semanticGraph,
snapshots,
}: RepositoryKnowledgeSpaceProfileMigrationCandidateBuilderOptions): KnowledgeSpaceProfileMigrationCandidateBuilder {
positiveInteger(maxDocuments, "maxDocuments");
positiveInteger(maxMembers, "maxMembers");
positiveInteger(maxPathReadPageSize, "maxPathReadPageSize");
positiveInteger(maxPathsPerDocument, "maxPathsPerDocument");
positiveInteger(maxProjectionBatchSize, "maxProjectionBatchSize");
const loadBase = async (
@ -457,11 +477,69 @@ export function createRepositoryKnowledgeSpaceProfileMigrationCandidateBuilder({
return buildResult(candidate, { successorMembersCloned: true }, requireValidating);
}
if (input.rebuildScope === "full-page-index-summary-outline") {
if (!paths) {
throw candidateError(
"PROFILE_MIGRATION_REASONING_REBUILD_UNAVAILABLE",
"Reasoning migration requires outline-derived KnowledgeFS path persistence",
);
}
if (!input.baseEmbeddingProfile) {
throw candidateError(
"PROFILE_MIGRATION_REASONING_REBUILD_UNAVAILABLE",
"Reasoning migration requires the frozen active embedding profile",
);
}
const baseProjectionMembers = base.members.filter(
(member) => member.componentType === "index-projection",
);
const baseProjections = await loadProjections(
projections,
baseProjectionMembers.map((member) => member.componentKey),
input.knowledgeSpaceId,
maxProjectionBatchSize,
);
const preservedProjectionIds = new Set(
baseProjections
.filter((projection) => !isOrdinarySearchProjection(projection))
.map((projection) => projection.id),
);
const baseDerivedPathIds = await resolveBaseOutlineDerivedPathIds({
documents: base.documents,
maxPaths: maxPathsPerDocument,
pageSize: maxPathReadPageSize,
paths,
tenantId: input.tenantId,
});
const expectedCandidatePathGenerations = new Set(
base.documents.map((document) =>
migrationGenerationId(input.runId, "page-index", document.documentAssetId),
),
);
assertSameMemberSnapshot(
base.members.filter((member) => member.componentType !== "document-outline"),
candidateMembers.filter((member) => member.componentType !== "document-outline"),
base.members.filter(
(member) =>
member.componentType !== "document-outline" &&
!(semanticGraph && isGraphMember(member)) &&
!(
member.componentType === "knowledge-path" &&
baseDerivedPathIds.has(member.componentKey)
) &&
(member.componentType !== "index-projection" ||
preservedProjectionIds.has(member.componentKey)),
),
candidateMembers.filter(
(member) =>
member.componentType !== "document-outline" &&
!(semanticGraph && isGraphMember(member)) &&
!(
member.componentType === "knowledge-path" &&
expectedCandidatePathGenerations.has(member.generationId)
) &&
(member.componentType !== "index-projection" ||
preservedProjectionIds.has(member.componentKey)),
),
"PROFILE_MIGRATION_PAGE_INDEX_REBUILD_INCOMPLETE",
"Reasoning migration changed or dropped a non-outline publication member",
"Reasoning migration changed or dropped a preserved publication member",
);
const candidateOutlineMembers = candidateMembers.filter(
(member) => member.componentType === "document-outline",
@ -480,9 +558,31 @@ export function createRepositoryKnowledgeSpaceProfileMigrationCandidateBuilder({
"candidate",
);
const profile = KnowledgeSpaceRetrievalProfileSchema.parse(retrieval.snapshot);
const embedding = await requireProfile(
profiles,
input,
"embedding",
input.baseEmbeddingProfile,
"active",
);
const embeddingProfile = KnowledgeSpaceEmbeddingProfileSchema.parse(embedding.snapshot);
const outlinesByDocument = groupByDocument(
candidateMembers.filter((member) => member.componentType === "document-outline"),
);
const candidateProjectionMembers = candidateMembers.filter(
(member) => member.componentType === "index-projection",
);
const candidateProjections = await loadProjections(
projections,
candidateProjectionMembers.map((member) => member.componentKey),
input.knowledgeSpaceId,
maxProjectionBatchSize,
);
const candidateProjectionById = new Map(
candidateProjections.map((projection) => [projection.id, projection]),
);
const candidateProjectionsByDocument = groupByDocument(candidateProjectionMembers);
const candidateGraphByDocument = groupByDocument(candidateMembers.filter(isGraphMember));
for (const document of base.documents) {
const expectedGeneration = migrationGenerationId(
input.runId,
@ -510,6 +610,84 @@ export function createRepositoryKnowledgeSpaceProfileMigrationCandidateBuilder({
`Document ${document.documentAssetId} PageIndex Summary/Outline rebuild is incomplete`,
);
}
if (
(candidateGraphByDocument.get(document.documentAssetId) ?? []).some(
(member) => member.generationId !== expectedGeneration,
)
) {
throw candidateError(
"PROFILE_MIGRATION_REASONING_REBUILD_INCOMPLETE",
`Document ${document.documentAssetId} Graph lineage is incomplete`,
);
}
const expectedPaths = buildOutlineDerivedPaths({
asset: document.asset,
outline,
publicationGenerationId: expectedGeneration,
tenantId: input.tenantId,
});
await assertOutlineDerivedPathClosure({
expected: expectedPaths,
maxPaths: maxPathsPerDocument,
members: candidateMembers.filter(
(member) =>
member.componentType === "knowledge-path" &&
expectedPaths.some((path) => path.id === member.componentKey),
),
pageSize: maxPathReadPageSize,
paths,
});
const ordinary = (candidateProjectionsByDocument.get(document.documentAssetId) ?? [])
.filter((member) => !preservedProjectionIds.has(member.componentKey))
.map((member) => {
const projection = candidateProjectionById.get(member.componentKey);
if (
!projection ||
!isOrdinarySearchProjection(projection) ||
projection.publicationGenerationId !== expectedGeneration ||
member.generationId !== expectedGeneration ||
projectionDocumentAssetId(projection) !== document.documentAssetId ||
projection.status !== "ready"
) {
throw candidateError(
"PROFILE_MIGRATION_REASONING_REBUILD_INCOMPLETE",
`Document ${document.documentAssetId} semantic projection lineage is incomplete`,
);
}
return projection;
});
const fts = ordinary.filter((projection) => projection.type === "fts");
const dense = ordinary.filter((projection) => projection.type === "dense-vector");
if (
fts.length < 1 ||
dense.length !== fts.length ||
dense.some((projection) => projection.model !== embeddingProfile.vectorSpaceId)
) {
throw candidateError(
"PROFILE_MIGRATION_REASONING_REBUILD_INCOMPLETE",
`Document ${document.documentAssetId} semantic search projection closure is incomplete`,
);
}
}
const baseDocumentIds = new Set(base.documents.map((document) => document.documentAssetId));
if (
candidateMembers.some(
(member) =>
isGraphMember(member) &&
(!member.documentAssetId || !baseDocumentIds.has(member.documentAssetId)),
) ||
candidateProjectionMembers.some(
(member) =>
!preservedProjectionIds.has(member.componentKey) &&
(!member.documentAssetId ||
!baseDocumentIds.has(member.documentAssetId) ||
!isOrdinarySearchProjection(candidateProjectionById.get(member.componentKey))),
)
) {
throw candidateError(
"PROFILE_MIGRATION_REASONING_REBUILD_INCOMPLETE",
"Reasoning migration candidate contains an extra or unowned search projection",
);
}
return buildResult(candidate, { pageIndexSummaryOutlineRebuilt: true }, requireValidating);
}
@ -526,8 +704,14 @@ export function createRepositoryKnowledgeSpaceProfileMigrationCandidateBuilder({
(member) => member.componentType === "index-projection",
);
assertSameMemberSnapshot(
base.members.filter((member) => member.componentType !== "index-projection"),
candidateMembers.filter((member) => member.componentType !== "index-projection"),
base.members.filter(
(member) =>
member.componentType !== "index-projection" && !(semanticGraph && isGraphMember(member)),
),
candidateMembers.filter(
(member) =>
member.componentType !== "index-projection" && !(semanticGraph && isGraphMember(member)),
),
"PROFILE_MIGRATION_VECTOR_REBUILD_INCOMPLETE",
"Embedding migration changed or dropped a non-index publication member",
);
@ -562,6 +746,7 @@ export function createRepositoryKnowledgeSpaceProfileMigrationCandidateBuilder({
);
const projectionsById = new Map(loaded.map((projection) => [projection.id, projection]));
const membersByDocument = groupByDocument(projectionMembers);
const graphByDocument = groupByDocument(candidateMembers.filter(isGraphMember));
for (const document of base.documents) {
const expectedGeneration = migrationGenerationId(
input.runId,
@ -587,6 +772,16 @@ export function createRepositoryKnowledgeSpaceProfileMigrationCandidateBuilder({
}
return projection;
});
if (
(graphByDocument.get(document.documentAssetId) ?? []).some(
(member) => member.generationId !== expectedGeneration,
)
) {
throw candidateError(
"PROFILE_MIGRATION_VECTOR_REBUILD_INCOMPLETE",
`Document ${document.documentAssetId} Graph lineage is incomplete`,
);
}
const baseOwned = baseProjectionMembers
.filter((member) => member.documentAssetId === document.documentAssetId)
.flatMap((member) => {
@ -620,6 +815,11 @@ export function createRepositoryKnowledgeSpaceProfileMigrationCandidateBuilder({
}
const baseDocumentIds = new Set(base.documents.map((document) => document.documentAssetId));
if (
candidateMembers.some(
(member) =>
isGraphMember(member) &&
(!member.documentAssetId || !baseDocumentIds.has(member.documentAssetId)),
) ||
projectionMembers.some(
(member) =>
!preservedProjectionIds.has(member.componentKey) &&
@ -691,6 +891,18 @@ export function createRepositoryKnowledgeSpaceProfileMigrationCandidateBuilder({
if (input.rebuildScope === "clone-publication") {
nextMembers = base.members.map(memberInput);
} else if (input.rebuildScope === "full-page-index-summary-outline") {
if (!paths) {
throw candidateError(
"PROFILE_MIGRATION_REASONING_REBUILD_UNAVAILABLE",
"Reasoning migration requires outline-derived KnowledgeFS path persistence",
);
}
if (!input.baseEmbeddingProfile) {
throw candidateError(
"PROFILE_MIGRATION_REASONING_REBUILD_UNAVAILABLE",
"Reasoning migration requires the frozen active embedding profile",
);
}
const retrieval = await requireProfile(
profiles,
input,
@ -699,52 +911,12 @@ export function createRepositoryKnowledgeSpaceProfileMigrationCandidateBuilder({
"candidate",
);
const retrievalProfile = KnowledgeSpaceRetrievalProfileSchema.parse(retrieval.snapshot);
const rebuilt: KnowledgeSpaceProfileMigrationCandidateMemberInput[] = [];
for (const document of base.documents) {
await input.execution?.heartbeat();
const generationId = migrationGenerationId(
input.runId,
"page-index",
document.documentAssetId,
);
const deterministicOutline = outlineBuilder.build({
knowledgeSpaceId: input.knowledgeSpaceId,
parseArtifact: document.artifact,
publicationGenerationId: generationId,
});
const enhanced = await outlineSummaryEnhancer.enhance({
outline: deterministicOutline,
parseArtifact: document.artifact,
retrievalProfile,
tenantId: input.tenantId,
});
const outline = await outlines.upsert(enhanced);
await pageIndexBuild.materializeBuilding({
builtAt: outline.updatedAt ?? outline.createdAt,
outline,
tenantId: input.tenantId,
});
rebuilt.push({
componentKey: outline.id,
componentType: "document-outline",
documentAssetId: document.documentAssetId,
generationId,
});
await input.execution?.heartbeat();
}
nextMembers = [
...base.members
.filter((member) => member.componentType !== "document-outline")
.map(memberInput),
...rebuilt,
];
} else {
const embedding = await requireProfile(
profiles,
input,
"embedding",
input.candidateProfile,
"candidate",
input.baseEmbeddingProfile,
"active",
);
const embeddingProfile = KnowledgeSpaceEmbeddingProfileSchema.parse(embedding.snapshot);
const baseProjectionMembers = base.members.filter(
@ -761,7 +933,185 @@ export function createRepositoryKnowledgeSpaceProfileMigrationCandidateBuilder({
.filter((projection) => !isOrdinarySearchProjection(projection))
.map((projection) => projection.id),
);
const baseDerivedPathIds = await resolveBaseOutlineDerivedPathIds({
documents: base.documents,
maxPaths: maxPathsPerDocument,
pageSize: maxPathReadPageSize,
paths,
tenantId: input.tenantId,
});
const rebuilt: KnowledgeSpaceProfileMigrationCandidateMemberInput[] = [];
for (const document of base.documents) {
await input.execution?.heartbeat();
const generationId = migrationGenerationId(
input.runId,
"page-index",
document.documentAssetId,
);
const reindexResult = await reindexer.reindex({
denseModel: embeddingProfile.vectorSpaceId,
embeddingProfile,
enableGraph: true,
knowledgeSpaceId: input.knowledgeSpaceId,
parseArtifact: document.artifact,
permissionScope: stringArray(document.asset.metadata.permissionScope),
projectionStatus: "ready",
projectionVersion: document.asset.version,
publicationGenerationId: generationId,
retrievalProfile,
skipVisual: true,
tenantId: input.tenantId,
});
if (
reindexResult.status !== "rebuilt" ||
!reindexResult.outlineArtifact ||
!reindexResult.projectionIds ||
reindexResult.projectionIds.length === 0 ||
reindexResult.projectionIds.length !== reindexResult.projectionsCreated ||
(reindexResult.nodeIds?.length ?? 0) !== reindexResult.nodesCreated
) {
throw candidateError(
"PROFILE_MIGRATION_REASONING_REBUILD_INCOMPLETE",
`Document ${document.documentAssetId} did not produce a complete semantic generation receipt`,
);
}
const deterministicOutline = outlineBuilder.build({
knowledgeSpaceId: input.knowledgeSpaceId,
parseArtifact: reindexResult.outlineArtifact,
publicationGenerationId: generationId,
});
const enhanced = await outlineSummaryEnhancer.enhance({
outline: deterministicOutline,
parseArtifact: reindexResult.outlineArtifact,
retrievalProfile,
tenantId: input.tenantId,
});
const outline = await outlines.upsert(enhanced);
await pageIndexBuild.materializeBuilding({
builtAt: outline.updatedAt ?? outline.createdAt,
outline,
tenantId: input.tenantId,
});
const rebuiltPaths = buildOutlineDerivedPaths({
asset: document.asset,
outline,
publicationGenerationId: generationId,
tenantId: input.tenantId,
});
await persistOutlineDerivedPaths({
batchSize: maxProjectionBatchSize,
expected: rebuiltPaths,
paths,
});
rebuilt.push(
{
componentKey: outline.id,
componentType: "document-outline",
documentAssetId: document.documentAssetId,
generationId,
},
...rebuiltPaths.map((path) => ({
componentKey: path.id,
componentType: "knowledge-path" as const,
documentAssetId: document.documentAssetId,
generationId,
})),
);
rebuilt.push(
...reindexResult.projectionIds.map((componentKey) => ({
componentKey,
componentType: "index-projection" as const,
documentAssetId: document.documentAssetId,
generationId,
})),
);
if (semanticGraph) {
const graph = await semanticGraph.materialize({
createdAt: candidate.createdAt,
knowledgeSpaceId: input.knowledgeSpaceId,
parseArtifactId: document.artifact.id,
publicationGenerationId: generationId,
retrievalProfile,
});
rebuilt.push(
...graph.graphEntityIds.map((componentKey) => ({
componentKey,
componentType: "graph-entity" as const,
documentAssetId: document.documentAssetId,
generationId,
})),
...graph.graphRelationIds.map((componentKey) => ({
componentKey,
componentType: "graph-relation" as const,
documentAssetId: document.documentAssetId,
generationId,
})),
);
}
await input.execution?.heartbeat();
}
nextMembers = [
...base.members
.filter(
(member) =>
member.componentType !== "document-outline" &&
!(semanticGraph && isGraphMember(member)) &&
!(
member.componentType === "knowledge-path" &&
baseDerivedPathIds.has(member.componentKey)
) &&
(member.componentType !== "index-projection" ||
preservedProjectionIds.has(member.componentKey)),
)
.map(memberInput),
...rebuilt,
];
} else {
const embedding = await requireProfile(
profiles,
input,
"embedding",
input.candidateProfile,
"candidate",
);
const embeddingProfile = KnowledgeSpaceEmbeddingProfileSchema.parse(embedding.snapshot);
const retrieval = await requireProfile(
profiles,
input,
"retrieval",
input.baseRetrievalProfile,
"active",
);
const retrievalProfile = KnowledgeSpaceRetrievalProfileSchema.parse(retrieval.snapshot);
const baseProjectionMembers = base.members.filter(
(member) => member.componentType === "index-projection",
);
const baseProjections = await loadProjections(
projections,
baseProjectionMembers.map((member) => member.componentKey),
input.knowledgeSpaceId,
maxProjectionBatchSize,
);
const baseProjectionById = new Map(
baseProjections.map((projection) => [projection.id, projection]),
);
const ordinaryNodeGenerationsByDocument = new Map<string, Set<string>>();
for (const member of baseProjectionMembers) {
if (!member.documentAssetId) continue;
const projection = baseProjectionById.get(member.componentKey);
if (!projection || !isOrdinarySearchProjection(projection)) continue;
const generations =
ordinaryNodeGenerationsByDocument.get(member.documentAssetId) ?? new Set<string>();
generations.add(member.generationId);
ordinaryNodeGenerationsByDocument.set(member.documentAssetId, generations);
}
const preservedProjectionIds = new Set(
baseProjections
.filter((projection) => !isOrdinarySearchProjection(projection))
.map((projection) => projection.id),
);
const rebuilt: KnowledgeSpaceProfileMigrationCandidateMemberInput[] = [];
const rebuiltGraph: KnowledgeSpaceProfileMigrationCandidateMemberInput[] = [];
for (const document of base.documents) {
await input.execution?.heartbeat();
const generationId = migrationGenerationId(
@ -769,6 +1119,15 @@ export function createRepositoryKnowledgeSpaceProfileMigrationCandidateBuilder({
"vector-space",
document.documentAssetId,
);
const sourceNodeGenerations =
ordinaryNodeGenerationsByDocument.get(document.documentAssetId) ?? new Set<string>();
if (sourceNodeGenerations.size !== 1) {
throw candidateError(
"PROFILE_MIGRATION_VECTOR_REBUILD_INCOMPLETE",
`Document ${document.documentAssetId} must have exactly one reusable ordinary node generation`,
);
}
const reuseNodeGenerationId = [...sourceNodeGenerations][0] as string;
const result = await reindexer.reindex({
denseModel: embeddingProfile.vectorSpaceId,
embeddingProfile,
@ -778,6 +1137,9 @@ export function createRepositoryKnowledgeSpaceProfileMigrationCandidateBuilder({
projectionStatus: "ready",
projectionVersion: document.asset.version,
publicationGenerationId: generationId,
retrievalProfile,
reuseNodeGenerationId,
skipVisual: true,
tenantId: input.tenantId,
});
if (
@ -799,17 +1161,42 @@ export function createRepositoryKnowledgeSpaceProfileMigrationCandidateBuilder({
generationId,
})),
);
if (semanticGraph) {
const graph = await semanticGraph.materialize({
createdAt: candidate.createdAt,
knowledgeSpaceId: input.knowledgeSpaceId,
parseArtifactId: document.artifact.id,
publicationGenerationId: generationId,
retrievalProfile,
});
rebuiltGraph.push(
...graph.graphEntityIds.map((componentKey) => ({
componentKey,
componentType: "graph-entity" as const,
documentAssetId: document.documentAssetId,
generationId,
})),
...graph.graphRelationIds.map((componentKey) => ({
componentKey,
componentType: "graph-relation" as const,
documentAssetId: document.documentAssetId,
generationId,
})),
);
}
await input.execution?.heartbeat();
}
nextMembers = [
...base.members
.filter(
(member) =>
member.componentType !== "index-projection" ||
preservedProjectionIds.has(member.componentKey),
!(semanticGraph && isGraphMember(member)) &&
(member.componentType !== "index-projection" ||
preservedProjectionIds.has(member.componentKey)),
)
.map(memberInput),
...rebuilt,
...rebuiltGraph,
];
}
if (nextMembers.length > maxMembers) {
@ -894,10 +1281,22 @@ export function createRepositoryKnowledgeSpaceProfileMigrationEvaluator({
assertSameMemberSnapshot(baseMembers, candidateMembers);
} else if (run.rebuildScope === "full-page-index-summary-outline") {
assertSameMemberSnapshot(
baseMembers.filter((member) => member.componentType !== "document-outline"),
candidateMembers.filter((member) => member.componentType !== "document-outline"),
baseMembers.filter(
(member) =>
member.componentType !== "document-outline" &&
member.componentType !== "index-projection" &&
member.componentType !== "knowledge-path" &&
!isGraphMember(member),
),
candidateMembers.filter(
(member) =>
member.componentType !== "document-outline" &&
member.componentType !== "index-projection" &&
member.componentType !== "knowledge-path" &&
!isGraphMember(member),
),
"PROFILE_MIGRATION_PAGE_INDEX_REBUILD_INCOMPLETE",
"Reasoning evaluation found a changed or missing non-outline publication member",
"Reasoning evaluation found a changed or missing preserved publication member",
);
if (
candidateMembers.filter((member) => member.componentType === "document-outline")
@ -910,8 +1309,12 @@ export function createRepositoryKnowledgeSpaceProfileMigrationEvaluator({
}
} else {
assertSameMemberSnapshot(
baseMembers.filter((member) => member.componentType !== "index-projection"),
candidateMembers.filter((member) => member.componentType !== "index-projection"),
baseMembers.filter(
(member) => member.componentType !== "index-projection" && !isGraphMember(member),
),
candidateMembers.filter(
(member) => member.componentType !== "index-projection" && !isGraphMember(member),
),
"PROFILE_MIGRATION_VECTOR_REBUILD_INCOMPLETE",
"Embedding evaluation found a changed or missing non-index publication member",
);
@ -928,6 +1331,29 @@ export function createRepositoryKnowledgeSpaceProfileMigrationEvaluator({
) {
return failedEvaluation("candidate document ownership differs from the frozen base");
}
if (run.rebuildScope === "full-page-index-summary-outline") {
for (const documentAssetId of candidateDocuments) {
const expectedGeneration = migrationGenerationId(run.id, "page-index", documentAssetId);
if (
!candidateMembers.some(
(member) =>
member.componentType === "knowledge-path" &&
member.documentAssetId === documentAssetId &&
member.generationId === expectedGeneration,
) ||
candidateMembers.some(
(member) =>
isGraphMember(member) &&
member.documentAssetId === documentAssetId &&
member.generationId !== expectedGeneration,
)
) {
return failedEvaluation(
`document ${documentAssetId} semantic path or Graph generation is incomplete`,
);
}
}
}
if (baseMembers.length === 0 && candidateMembers.length === 0) {
return {
passed: true,
@ -1054,6 +1480,15 @@ export function createRepositoryKnowledgeSpaceProfileMigrationEvaluator({
) {
return failedEvaluation(`projection ${member.componentKey} lineage is invalid`);
}
if (
reasoningProfile !== undefined &&
isOrdinarySearchProjection(projection) &&
member.generationId !== migrationGenerationId(run.id, "page-index", documentAssetId)
) {
return failedEvaluation(
`document ${documentAssetId} contains a stale reasoning search projection`,
);
}
if (projection.type === "fts") {
ftsProjections += 1;
ftsCount += 1;
@ -1352,6 +1787,198 @@ function isOrdinarySearchProjection(projection: IndexProjection | undefined): bo
);
}
function isGraphMember(member: Pick<ProjectionSetPublicationMember, "componentType">): boolean {
return member.componentType === "graph-entity" || member.componentType === "graph-relation";
}
function buildOutlineDerivedPaths({
asset,
outline,
publicationGenerationId,
tenantId,
}: {
readonly asset: CandidateDocument["asset"];
readonly outline: DocumentOutline;
readonly publicationGenerationId: string;
readonly tenantId: string;
}): readonly KnowledgePath[] {
let sequence = 0;
const generateId = () =>
deterministicChildId(publicationGenerationId, `reasoning-path-seed:${sequence++}`);
const derived = [
buildDocumentOutlineKnowledgePath({
asset,
id: generateId(),
publicationGenerationId,
tenantId,
}),
...buildDocumentSectionKnowledgePaths({
asset,
generateId,
outline,
publicationGenerationId,
tenantId,
}),
];
if (
new Set(derived.map((path) => path.id)).size !== derived.length ||
new Set(derived.map((path) => path.virtualPath)).size !== derived.length
) {
throw candidateError(
"PROFILE_MIGRATION_REASONING_PATH_REBUILD_INCOMPLETE",
`Document ${asset.id} produced duplicate outline-derived KnowledgeFS paths`,
);
}
return derived;
}
async function persistOutlineDerivedPaths({
batchSize,
expected,
paths,
}: {
readonly batchSize: number;
readonly expected: readonly KnowledgePath[];
readonly paths: Pick<KnowledgePathRepository, "upsertMany">;
}): Promise<void> {
const persisted: KnowledgePath[] = [];
for (const batch of batches(expected, batchSize)) {
persisted.push(...(await paths.upsertMany(batch)));
}
assertExactKnowledgePaths(expected, persisted);
}
async function assertOutlineDerivedPathClosure({
expected,
maxPaths,
members,
pageSize,
paths,
}: {
readonly expected: readonly KnowledgePath[];
readonly maxPaths: number;
readonly members: readonly Pick<
ProjectionSetPublicationMember,
"componentKey" | "componentType" | "documentAssetId" | "generationId"
>[];
readonly pageSize: number;
readonly paths: Pick<KnowledgePathRepository, "listPhysicalDescendants">;
}): Promise<void> {
assertSameMemberSnapshot(
expected.map((path) => ({
componentKey: path.id,
componentType: "knowledge-path" as const,
documentAssetId: path.targetId,
generationId: path.publicationGenerationId as string,
})),
members,
"PROFILE_MIGRATION_REASONING_PATH_REBUILD_INCOMPLETE",
`Document ${expected[0]?.targetId ?? "unknown"} outline-derived path membership is incomplete`,
);
const stored = await listDocumentGenerationPaths({
anchor: expected[0] as KnowledgePath,
maxPaths,
pageSize,
paths,
});
const expectedVirtualPaths = new Set(expected.map((path) => path.virtualPath));
assertExactKnowledgePaths(
expected,
stored.filter((path) => expectedVirtualPaths.has(path.virtualPath)),
);
}
async function listDocumentGenerationPaths({
anchor,
maxPaths,
pageSize,
paths,
}: {
readonly anchor: KnowledgePath;
readonly maxPaths: number;
readonly pageSize: number;
readonly paths: Pick<KnowledgePathRepository, "listPhysicalDescendants">;
}): Promise<readonly KnowledgePath[]> {
const parentPath = anchor.virtualPath.replace(/\/outline\.json$/u, "");
const matched: KnowledgePath[] = [];
let cursor: Awaited<ReturnType<KnowledgePathRepository["listPhysicalDescendants"]>>["nextCursor"];
do {
const page = await paths.listPhysicalDescendants({
...(cursor ? { cursor } : {}),
knowledgeSpaceId: anchor.knowledgeSpaceId,
limit: Math.min(pageSize, maxPaths - matched.length),
parentPath,
publicationGenerationId: anchor.publicationGenerationId,
viewName: anchor.viewName,
});
matched.push(...page.items);
cursor = page.nextCursor;
if (cursor && matched.length >= maxPaths) {
throw candidateError(
"PROFILE_MIGRATION_REASONING_PATH_REBUILD_INCOMPLETE",
`Document ${anchor.targetId} path count exceeds ${maxPaths}`,
);
}
} while (cursor);
return matched;
}
async function resolveBaseOutlineDerivedPathIds({
documents,
maxPaths,
pageSize,
paths,
tenantId,
}: {
readonly documents: readonly CandidateDocument[];
readonly maxPaths: number;
readonly pageSize: number;
readonly paths: Pick<KnowledgePathRepository, "listPhysicalDescendants">;
readonly tenantId: string;
}): Promise<ReadonlySet<string>> {
const ids = new Set<string>();
for (const document of documents) {
const expected = buildOutlineDerivedPaths({
asset: document.asset,
outline: document.baseOutline,
publicationGenerationId: PublicationGenerationIdSchema.parse(
document.baseOutline.publicationGenerationId,
),
tenantId,
});
const resolved = await listDocumentGenerationPaths({
anchor: expected[0] as KnowledgePath,
maxPaths,
pageSize,
paths,
});
for (const path of resolved) {
const contentKind = path.metadata.contentKind;
if (
path.targetId === document.documentAssetId &&
(contentKind === "document-outline" || contentKind === "document-section")
) {
ids.add(path.id);
}
}
}
return ids;
}
function assertExactKnowledgePaths(
expected: readonly KnowledgePath[],
actual: readonly KnowledgePath[],
): void {
const left = expected.map((path) => stableJson(path)).sort();
const right = actual.map((path) => stableJson(path)).sort();
if (left.length !== right.length || left.some((value, index) => value !== right[index])) {
throw candidateError(
"PROFILE_MIGRATION_REASONING_PATH_REBUILD_INCOMPLETE",
"Outline-derived KnowledgeFS path receipt is incomplete or incompatible",
);
}
}
function failedEvaluation(reason: string): KnowledgeSpaceProfileMigrationEvaluationResult {
return { passed: false, summary: { reason: reason.slice(0, 512) } };
}

2020
knowledge-fs/packages/api/src/llm-semantic-chunker.test.ts Normal file
View File

File diff suppressed because it is too large Load Diff

2326
knowledge-fs/packages/api/src/llm-semantic-chunker.ts Normal file
View File

File diff suppressed because it is too large Load Diff

169
knowledge-fs/packages/api/src/semantic-generation-receipt.ts Normal file
View File

@ -0,0 +1,169 @@
import { createHash } from "node:crypto";
import type { KnowledgeSpaceModelSelection } from "@knowledge/core";
import { stableJson } from "@knowledge/core";
export const MAX_KNOWLEDGE_NODE_GENERATION_RECEIPT_BYTES = 4 * 1024 * 1024;
export const MAX_KNOWLEDGE_NODE_GENERATION_RECEIPT_DATABASE_TEXT_BYTES = 8 * 1024 * 1024;
export const MAX_LLM_SEMANTIC_WINDOWS = 4_096;
export const MAX_LLM_SEMANTIC_COMPLETION_IDENTITIES = 64;
export const MAX_LLM_SEMANTIC_WINDOW_ID_CODE_POINTS = 32;
export const MAX_LLM_SEMANTIC_UNIT_ID_CODE_POINTS = 32;
export const MAX_LLM_SEMANTIC_TERMINAL_IDENTITY_CODE_POINTS = 255;
export const MAX_LLM_SEMANTIC_FINISH_REASON_CODE_POINTS = 64;
const SHA256_FINGERPRINT = `sha256:${"f".repeat(64)}`;
const MAX_WINDOW_ID = `window-${"9".repeat(
MAX_LLM_SEMANTIC_WINDOW_ID_CODE_POINTS - "window-".length,
)}`;
const MAX_UNIT_ID = `u-${"9".repeat(14)}-${"9".repeat(15)}`;
const MAX_TERMINAL_IDENTITY = "\u{1f600}".repeat(MAX_LLM_SEMANTIC_TERMINAL_IDENTITY_CODE_POINTS);
const MAX_FINISH_REASON = "\u{1f600}".repeat(MAX_LLM_SEMANTIC_FINISH_REASON_CODE_POINTS);
export interface KnowledgeNodeSemanticGenerationConfig {
readonly maxChunkChars: number;
readonly maxNodes: number;
readonly maxWindowChars: number;
readonly overlapChars: number;
readonly promptVersion: string;
}
export interface KnowledgeNodeGenerationCompletionReceipt {
readonly actualModel?: string | undefined;
readonly actualProvider?: string | undefined;
readonly fingerprint: string;
readonly finishReason?: string | undefined;
readonly transportProvider?: string | undefined;
}
export type KnowledgeNodeGenerationUnitRangeReceipt = readonly [
startUnitId: string,
endUnitId: string,
];
export interface KnowledgeNodeGenerationWindowReceipt {
readonly chunkRanges: readonly KnowledgeNodeGenerationUnitRangeReceipt[];
readonly committedUnitRange: KnowledgeNodeGenerationUnitRangeReceipt;
readonly completionIndex: number;
readonly coreUnitRange: KnowledgeNodeGenerationUnitRangeReceipt;
readonly firstChunkIndex: number;
readonly inputFingerprint: string;
readonly lookAheadUnitRange?: KnowledgeNodeGenerationUnitRangeReceipt | undefined;
/** Opaque hash of the complete generated semantic response payload for this window. */
readonly responseFingerprint: string;
readonly windowId: string;
}
/**
* Durable proof that semantic generation completed even when editorial exclusions persist no node
* rows. Window entries deliberately contain only canonical replay fields; completion identities
* are de-duplicated in a bounded catalog.
*/
export interface KnowledgeNodeGenerationReceipt {
readonly artifactHash: string;
readonly completionCatalog: readonly KnowledgeNodeGenerationCompletionReceipt[];
readonly documentAssetId: string;
readonly documentChunkCount: number;
readonly excludedNodeOrdinals: readonly number[];
readonly knowledgeSpaceId: string;
readonly language?: string | undefined;
readonly modelSelection: KnowledgeSpaceModelSelection;
readonly parseArtifactId: string;
readonly permissionScope: readonly string[];
readonly promptResponseFingerprint: string;
readonly publicationGenerationId: string;
readonly requestFingerprint: string;
readonly responseFingerprint: string;
readonly schemaVersion: 1;
readonly semanticConfig: KnowledgeNodeSemanticGenerationConfig;
readonly storedNodeCount: number;
readonly storedResponseFingerprint: string;
readonly windowManifest: readonly KnowledgeNodeGenerationWindowReceipt[];
}
export function llmSemanticCompletionFingerprint(
entry: Omit<KnowledgeNodeGenerationCompletionReceipt, "fingerprint">,
): string {
return `sha256:${createHash("sha256")
.update(
stableJson({
...(entry.actualModel ? { actualModel: entry.actualModel } : {}),
...(entry.actualProvider ? { actualProvider: entry.actualProvider } : {}),
...(entry.finishReason ? { finishReason: entry.finishReason } : {}),
...(entry.transportProvider ? { transportProvider: entry.transportProvider } : {}),
}),
)
.digest("hex")}`;
}
export function knowledgeNodeGenerationReceiptSerializedBytes(value: unknown): number {
return new TextEncoder().encode(stableJson(value)).byteLength;
}
/**
* Exact upper bound for an admitted receipt. The caller supplies an envelope with the real ACL,
* language, model selection, prompt version and exclusions plus empty dynamic arrays. Dynamic
* bytes use the repository-enforced identifier/terminal caps, one chunk range per possible node,
* and the maximum bounded completion catalog.
*/
export function maximumKnowledgeNodeGenerationReceiptSerializedBytes({
emptyReceipt,
maximumChunkCount,
maximumWindowCount,
}: {
readonly emptyReceipt: KnowledgeNodeGenerationReceipt;
readonly maximumChunkCount: number;
readonly maximumWindowCount: number;
}): number {
if (emptyReceipt.completionCatalog.length !== 0 || emptyReceipt.windowManifest.length !== 0) {
throw new Error("Semantic generation receipt admission requires empty dynamic arrays");
}
if (
!Number.isSafeInteger(maximumChunkCount) ||
maximumChunkCount < 0 ||
!Number.isSafeInteger(maximumWindowCount) ||
maximumWindowCount < 0 ||
maximumWindowCount > MAX_LLM_SEMANTIC_WINDOWS ||
maximumWindowCount > maximumChunkCount
) {
throw new Error("Semantic generation receipt admission bounds are invalid");
}
const completionCount = Math.min(maximumWindowCount, MAX_LLM_SEMANTIC_COMPLETION_IDENTITIES);
const maximumCompletion: KnowledgeNodeGenerationCompletionReceipt = {
actualModel: MAX_TERMINAL_IDENTITY,
actualProvider: MAX_TERMINAL_IDENTITY,
fingerprint: SHA256_FINGERPRINT,
finishReason: MAX_FINISH_REASON,
transportProvider: MAX_TERMINAL_IDENTITY,
};
const completionBytes = knowledgeNodeGenerationReceiptSerializedBytes(maximumCompletion);
const completionCatalogBytes = arraySerializedBytes(completionCount, completionBytes);
const maximumWindow: KnowledgeNodeGenerationWindowReceipt = {
chunkRanges: [],
committedUnitRange: [MAX_UNIT_ID, MAX_UNIT_ID],
completionIndex: Math.max(0, completionCount - 1),
coreUnitRange: [MAX_UNIT_ID, MAX_UNIT_ID],
firstChunkIndex: Math.max(0, maximumChunkCount - 1),
inputFingerprint: SHA256_FINGERPRINT,
lookAheadUnitRange: [MAX_UNIT_ID, MAX_UNIT_ID],
responseFingerprint: SHA256_FINGERPRINT,
windowId: MAX_WINDOW_ID,
};
const emptyWindowBytes = knowledgeNodeGenerationReceiptSerializedBytes(maximumWindow);
const rangeBytes = knowledgeNodeGenerationReceiptSerializedBytes([MAX_UNIT_ID, MAX_UNIT_ID]);
// Every admitted window has at least one chunk. Across all window chunk arrays, JSON contributes
// one bracket/comma byte per window plus (rangeBytes + separator) per possible document chunk.
const allChunkRangeArraysBytes = maximumWindowCount + maximumChunkCount * (rangeBytes + 1);
const windowObjectsBytes = maximumWindowCount * (emptyWindowBytes - 2) + allChunkRangeArraysBytes;
const windowManifestBytes =
maximumWindowCount === 0 ? 2 : 2 + windowObjectsBytes + (maximumWindowCount - 1);
const emptyReceiptBytes = knowledgeNodeGenerationReceiptSerializedBytes(emptyReceipt);
return emptyReceiptBytes - 4 + completionCatalogBytes + windowManifestBytes;
}
function arraySerializedBytes(itemCount: number, itemBytes: number): number {
return itemCount === 0 ? 2 : 2 + itemCount * itemBytes + (itemCount - 1);
}

44
knowledge-fs/packages/database/migrations/0043_semantic_generation_receipts.postgres.sql Normal file
View File

@ -0,0 +1,44 @@
-- Knowledge Platform schema migration
-- Migration id: 0043_semantic_generation_receipts
-- Dialect: postgres
CREATE TABLE IF NOT EXISTS "knowledge_node_generation_receipts" (
"knowledge_space_id" UUID NOT NULL,
"publication_generation_id" UUID NOT NULL,
"parse_artifact_id" UUID NOT NULL,
"document_asset_id" UUID NOT NULL,
"artifact_hash" VARCHAR(64) NOT NULL,
"document_chunk_count" INTEGER NOT NULL,
"stored_node_count" INTEGER NOT NULL,
"request_fingerprint" VARCHAR(71) NOT NULL,
"response_fingerprint" VARCHAR(71) NOT NULL,
"prompt_response_fingerprint" VARCHAR(71) NOT NULL,
"receipt" JSONB NOT NULL,
PRIMARY KEY ("knowledge_space_id", "publication_generation_id", "parse_artifact_id"),
CONSTRAINT "knowledge_node_generation_receipts_counts_ck" CHECK (
"document_chunk_count" >= 0 AND "stored_node_count" >= 0
AND "stored_node_count" <= "document_chunk_count"
),
CONSTRAINT "knowledge_node_generation_receipts_hashes_ck" CHECK (
"artifact_hash" ~ '^[a-f0-9]{64}$'
AND "request_fingerprint" ~ '^sha256:[a-f0-9]{64}$'
AND "response_fingerprint" ~ '^sha256:[a-f0-9]{64}$'
AND "prompt_response_fingerprint" ~ '^sha256:[a-f0-9]{64}$'
),
CONSTRAINT "knowledge_node_generation_receipts_json_ck"
CHECK (jsonb_typeof("receipt") = 'object'),
CONSTRAINT "knowledge_node_generation_receipts_bytes_ck"
CHECK (octet_length("receipt"::text) <= 8388608),
CONSTRAINT "knowledge_node_generation_receipts_pub_gen_nonzero_ck"
CHECK ("publication_generation_id" <> '00000000-0000-0000-0000-000000000000'::uuid),
FOREIGN KEY ("knowledge_space_id")
REFERENCES "knowledge_spaces" ("id") ON DELETE CASCADE,
FOREIGN KEY ("document_asset_id")
REFERENCES "document_assets" ("id") ON DELETE CASCADE,
FOREIGN KEY ("parse_artifact_id")
REFERENCES "parse_artifacts" ("id") ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS "knowledge_node_generation_receipts_document_idx"
ON "knowledge_node_generation_receipts"
("knowledge_space_id", "document_asset_id", "publication_generation_id", "parse_artifact_id");

46
knowledge-fs/packages/database/migrations/0043_semantic_generation_receipts.tidb.sql Normal file
View File

@ -0,0 +1,46 @@
-- Knowledge Platform schema migration
-- Migration id: 0043_semantic_generation_receipts
-- Dialect: tidb
CREATE TABLE IF NOT EXISTS `knowledge_node_generation_receipts` (
`knowledge_space_id` CHAR(36) NOT NULL,
`publication_generation_id` CHAR(36) NOT NULL,
`parse_artifact_id` CHAR(36) NOT NULL,
`document_asset_id` CHAR(36) NOT NULL,
`artifact_hash` VARCHAR(64) NOT NULL,
`document_chunk_count` INT NOT NULL,
`stored_node_count` INT NOT NULL,
`request_fingerprint` VARCHAR(71) NOT NULL,
`response_fingerprint` VARCHAR(71) NOT NULL,
`prompt_response_fingerprint` VARCHAR(71) NOT NULL,
`receipt` JSON NOT NULL,
PRIMARY KEY (`knowledge_space_id`, `publication_generation_id`, `parse_artifact_id`),
CONSTRAINT `knowledge_node_generation_receipts_counts_ck` CHECK (
`document_chunk_count` >= 0 AND `stored_node_count` >= 0
AND `stored_node_count` <= `document_chunk_count`
),
CONSTRAINT `knowledge_node_generation_receipts_hashes_ck` CHECK (
`artifact_hash` REGEXP '^[a-f0-9]{64}$'
AND `request_fingerprint` REGEXP '^sha256:[a-f0-9]{64}$'
AND `response_fingerprint` REGEXP '^sha256:[a-f0-9]{64}$'
AND `prompt_response_fingerprint` REGEXP '^sha256:[a-f0-9]{64}$'
),
CONSTRAINT `knowledge_node_generation_receipts_json_ck`
CHECK (JSON_TYPE(`receipt`) = 'OBJECT'),
CONSTRAINT `knowledge_node_generation_receipts_bytes_ck`
CHECK (OCTET_LENGTH(CAST(`receipt` AS CHAR)) <= 8388608),
CONSTRAINT `knowledge_node_generation_receipts_pub_gen_nonzero_ck` CHECK (
`publication_generation_id` REGEXP '^[0-9A-Fa-f]{8}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{12}$'
AND `publication_generation_id` <> '00000000-0000-0000-0000-000000000000'
),
FOREIGN KEY (`knowledge_space_id`)
REFERENCES `knowledge_spaces` (`id`) ON DELETE CASCADE,
FOREIGN KEY (`document_asset_id`)
REFERENCES `document_assets` (`id`) ON DELETE CASCADE,
FOREIGN KEY (`parse_artifact_id`)
REFERENCES `parse_artifacts` (`id`) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS `knowledge_node_generation_receipts_document_idx`
ON `knowledge_node_generation_receipts`
(`knowledge_space_id`, `document_asset_id`, `publication_generation_id`, `parse_artifact_id`);

2
knowledge-fs/packages/database/src/migration-artifacts.generated.ts
View File

@ -87,4 +87,6 @@ export const migrationArtifacts = [
{ content: "-- Knowledge Platform schema migration\n-- Migration id: 0041_logical_document_availability\n-- Dialect: tidb\n-- Adds document-scoped availability without mutating source or physical index state.\n\nALTER TABLE `logical_documents`\n ADD COLUMN IF NOT EXISTS `enabled` BOOLEAN NOT NULL DEFAULT TRUE,\n ADD COLUMN IF NOT EXISTS `disabled_at` TIMESTAMP NULL,\n ADD COLUMN IF NOT EXISTS `disabled_by_subject_id` VARCHAR(255) NULL;\n\nALTER TABLE `logical_documents`\n ADD CONSTRAINT `logical_documents_availability_ck`\n CHECK (\n (`enabled` AND `disabled_at` IS NULL AND `disabled_by_subject_id` IS NULL)\n OR (NOT `enabled` AND `disabled_at` IS NOT NULL AND `disabled_by_subject_id` IS NOT NULL)\n );\n", path: "packages/database/migrations/0041_logical_document_availability.tidb.sql" },
{ content: "-- Knowledge Platform schema migration\n-- Migration id: 0042_workflow_failed_retrieval_capture\n-- Dialect: postgres\n-- Workflow empty-retrieval events retain their admitted Capability provenance and frozen scope.\n\nALTER TABLE \"failed_queries\"\n ADD COLUMN IF NOT EXISTS \"capability_grant_id\" UUID;\n\nALTER TABLE \"failed_queries\"\n DROP CONSTRAINT IF EXISTS \"failed_queries_permission_binding_ck\";\n\nALTER TABLE \"failed_queries\"\n ADD CONSTRAINT \"failed_queries_permission_binding_ck\" CHECK (\n (\"tenant_id\" IS NULL AND \"capability_grant_id\" IS NULL\n AND \"requested_by_subject_id\" IS NULL AND \"access_channel\" IS NULL\n AND \"permission_snapshot_id\" IS NULL AND \"permission_snapshot_revision\" IS NULL\n AND \"required_permission_scope\" IS NULL AND \"revision\" IS NULL)\n OR (\"tenant_id\" IS NOT NULL AND \"capability_grant_id\" IS NOT NULL\n AND \"requested_by_subject_id\" IS NULL AND \"access_channel\" IS NULL\n AND \"permission_snapshot_id\" IS NULL AND \"permission_snapshot_revision\" IS NULL\n AND \"required_permission_scope\" IS NOT NULL\n AND jsonb_typeof(\"required_permission_scope\") = 'array'\n AND \"revision\" IS NOT NULL AND \"revision\" >= 1)\n OR (\"tenant_id\" IS NOT NULL AND \"capability_grant_id\" IS NULL\n AND \"requested_by_subject_id\" IS NOT NULL\n AND \"access_channel\" IS NOT NULL\n AND \"access_channel\" IN ('interactive', 'service_api', 'mcp', 'agent')\n AND \"permission_snapshot_id\" IS NOT NULL\n AND \"permission_snapshot_revision\" IS NOT NULL\n AND \"permission_snapshot_revision\" >= 1\n AND \"required_permission_scope\" IS NOT NULL\n AND jsonb_typeof(\"required_permission_scope\") = 'array'\n AND \"revision\" IS NOT NULL AND \"revision\" >= 1)\n );\n\nDO $kfs_0042_failed_query_capability_fk$\nBEGIN\n IF NOT EXISTS (\n SELECT 1 FROM pg_constraint\n WHERE conname = 'failed_queries_capability_grant_fk'\n AND conrelid = 'failed_queries'::regclass\n ) THEN\n ALTER TABLE \"failed_queries\"\n ADD CONSTRAINT \"failed_queries_capability_grant_fk\"\n FOREIGN KEY (\"tenant_id\", \"knowledge_space_id\", \"capability_grant_id\")\n REFERENCES \"capability_grants\" (\"tenant_id\", \"knowledge_space_id\", \"grant_id\")\n ON DELETE RESTRICT;\n END IF;\nEND\n$kfs_0042_failed_query_capability_fk$;\n\nCREATE INDEX IF NOT EXISTS \"failed_queries_capability_grant_idx\"\n ON \"failed_queries\" (\"tenant_id\", \"knowledge_space_id\", \"capability_grant_id\");\n", path: "packages/database/migrations/0042_workflow_failed_retrieval_capture.postgres.sql" },
{ content: "-- Knowledge Platform schema migration\n-- Migration id: 0042_workflow_failed_retrieval_capture\n-- Dialect: tidb\n-- Workflow empty-retrieval events retain their admitted Capability provenance and frozen scope.\n\nALTER TABLE `failed_queries`\n ADD COLUMN IF NOT EXISTS `capability_grant_id` CHAR(36) NULL;\n\nSET @fq_0042_binding_ck_exists = (\n SELECT COUNT(*) FROM information_schema.tidb_check_constraints\n WHERE constraint_schema = DATABASE()\n AND table_name = 'failed_queries'\n AND constraint_name = 'failed_queries_permission_binding_ck'\n);\nSET @fq_0042_binding_ck_drop_sql = IF(\n @fq_0042_binding_ck_exists > 0,\n 'ALTER TABLE `failed_queries` DROP CONSTRAINT `failed_queries_permission_binding_ck`',\n 'SELECT 1'\n);\nPREPARE fq_0042_binding_ck_drop_stmt FROM @fq_0042_binding_ck_drop_sql;\nEXECUTE fq_0042_binding_ck_drop_stmt;\nDEALLOCATE PREPARE fq_0042_binding_ck_drop_stmt;\n\nALTER TABLE `failed_queries`\n MODIFY COLUMN `permission_binding_complete` TINYINT GENERATED ALWAYS AS (\n CASE WHEN\n (`tenant_id` IS NULL AND `capability_grant_id` IS NULL\n AND `requested_by_subject_id` IS NULL AND `access_channel` IS NULL\n AND `permission_snapshot_id` IS NULL AND `permission_snapshot_revision` IS NULL\n AND `required_permission_scope` IS NULL AND `revision` IS NULL)\n OR (`tenant_id` IS NOT NULL AND `capability_grant_id` IS NOT NULL\n AND `requested_by_subject_id` IS NULL AND `access_channel` IS NULL\n AND `permission_snapshot_id` IS NULL AND `permission_snapshot_revision` IS NULL\n AND `required_permission_scope` IS NOT NULL\n AND JSON_TYPE(`required_permission_scope`) = 'ARRAY'\n AND `revision` IS NOT NULL AND `revision` >= 1)\n OR (`tenant_id` IS NOT NULL AND `capability_grant_id` IS NULL\n AND `requested_by_subject_id` IS NOT NULL\n AND `access_channel` IN ('interactive', 'service_api', 'mcp', 'agent')\n AND `permission_snapshot_id` IS NOT NULL\n AND `permission_snapshot_revision` IS NOT NULL\n AND `permission_snapshot_revision` >= 1\n AND `required_permission_scope` IS NOT NULL\n AND JSON_TYPE(`required_permission_scope`) = 'ARRAY'\n AND `revision` IS NOT NULL AND `revision` >= 1)\n THEN 1 ELSE 0\n END\n ) VIRTUAL;\n\nALTER TABLE `failed_queries`\n ADD CONSTRAINT `failed_queries_permission_binding_ck`\n CHECK (`permission_binding_complete` = 1);\n\nSET @fq_0042_capability_fk_exists = (\n SELECT COUNT(*) FROM information_schema.table_constraints\n WHERE table_schema = DATABASE()\n AND table_name = 'failed_queries'\n AND constraint_name = 'failed_queries_capability_grant_fk'\n);\nSET @fq_0042_capability_fk_sql = IF(\n @fq_0042_capability_fk_exists = 0,\n 'ALTER TABLE `failed_queries` ADD CONSTRAINT `failed_queries_capability_grant_fk` FOREIGN KEY (`tenant_id`, `knowledge_space_id`, `capability_grant_id`) REFERENCES `capability_grants` (`tenant_id`, `knowledge_space_id`, `grant_id`) ON DELETE RESTRICT',\n 'SELECT 1'\n);\nPREPARE fq_0042_capability_fk_stmt FROM @fq_0042_capability_fk_sql;\nEXECUTE fq_0042_capability_fk_stmt;\nDEALLOCATE PREPARE fq_0042_capability_fk_stmt;\n\nCREATE INDEX IF NOT EXISTS `failed_queries_capability_grant_idx`\n ON `failed_queries` (`tenant_id`, `knowledge_space_id`, `capability_grant_id`);\n", path: "packages/database/migrations/0042_workflow_failed_retrieval_capture.tidb.sql" },
{ content: "-- Knowledge Platform schema migration\n-- Migration id: 0043_semantic_generation_receipts\n-- Dialect: postgres\n\nCREATE TABLE IF NOT EXISTS \"knowledge_node_generation_receipts\" (\n \"knowledge_space_id\" UUID NOT NULL,\n \"publication_generation_id\" UUID NOT NULL,\n \"parse_artifact_id\" UUID NOT NULL,\n \"document_asset_id\" UUID NOT NULL,\n \"artifact_hash\" VARCHAR(64) NOT NULL,\n \"document_chunk_count\" INTEGER NOT NULL,\n \"stored_node_count\" INTEGER NOT NULL,\n \"request_fingerprint\" VARCHAR(71) NOT NULL,\n \"response_fingerprint\" VARCHAR(71) NOT NULL,\n \"prompt_response_fingerprint\" VARCHAR(71) NOT NULL,\n \"receipt\" JSONB NOT NULL,\n PRIMARY KEY (\"knowledge_space_id\", \"publication_generation_id\", \"parse_artifact_id\"),\n CONSTRAINT \"knowledge_node_generation_receipts_counts_ck\" CHECK (\n \"document_chunk_count\" >= 0 AND \"stored_node_count\" >= 0\n AND \"stored_node_count\" <= \"document_chunk_count\"\n ),\n CONSTRAINT \"knowledge_node_generation_receipts_hashes_ck\" CHECK (\n \"artifact_hash\" ~ '^[a-f0-9]{64}$'\n AND \"request_fingerprint\" ~ '^sha256:[a-f0-9]{64}$'\n AND \"response_fingerprint\" ~ '^sha256:[a-f0-9]{64}$'\n AND \"prompt_response_fingerprint\" ~ '^sha256:[a-f0-9]{64}$'\n ),\n CONSTRAINT \"knowledge_node_generation_receipts_json_ck\"\n CHECK (jsonb_typeof(\"receipt\") = 'object'),\n CONSTRAINT \"knowledge_node_generation_receipts_bytes_ck\"\n CHECK (octet_length(\"receipt\"::text) <= 8388608),\n CONSTRAINT \"knowledge_node_generation_receipts_pub_gen_nonzero_ck\"\n CHECK (\"publication_generation_id\" <> '00000000-0000-0000-0000-000000000000'::uuid),\n FOREIGN KEY (\"knowledge_space_id\")\n REFERENCES \"knowledge_spaces\" (\"id\") ON DELETE CASCADE,\n FOREIGN KEY (\"document_asset_id\")\n REFERENCES \"document_assets\" (\"id\") ON DELETE CASCADE,\n FOREIGN KEY (\"parse_artifact_id\")\n REFERENCES \"parse_artifacts\" (\"id\") ON DELETE CASCADE\n);\n\nCREATE INDEX IF NOT EXISTS \"knowledge_node_generation_receipts_document_idx\"\n ON \"knowledge_node_generation_receipts\"\n (\"knowledge_space_id\", \"document_asset_id\", \"publication_generation_id\", \"parse_artifact_id\");\n", path: "packages/database/migrations/0043_semantic_generation_receipts.postgres.sql" },
{ content: "-- Knowledge Platform schema migration\n-- Migration id: 0043_semantic_generation_receipts\n-- Dialect: tidb\n\nCREATE TABLE IF NOT EXISTS `knowledge_node_generation_receipts` (\n `knowledge_space_id` CHAR(36) NOT NULL,\n `publication_generation_id` CHAR(36) NOT NULL,\n `parse_artifact_id` CHAR(36) NOT NULL,\n `document_asset_id` CHAR(36) NOT NULL,\n `artifact_hash` VARCHAR(64) NOT NULL,\n `document_chunk_count` INT NOT NULL,\n `stored_node_count` INT NOT NULL,\n `request_fingerprint` VARCHAR(71) NOT NULL,\n `response_fingerprint` VARCHAR(71) NOT NULL,\n `prompt_response_fingerprint` VARCHAR(71) NOT NULL,\n `receipt` JSON NOT NULL,\n PRIMARY KEY (`knowledge_space_id`, `publication_generation_id`, `parse_artifact_id`),\n CONSTRAINT `knowledge_node_generation_receipts_counts_ck` CHECK (\n `document_chunk_count` >= 0 AND `stored_node_count` >= 0\n AND `stored_node_count` <= `document_chunk_count`\n ),\n CONSTRAINT `knowledge_node_generation_receipts_hashes_ck` CHECK (\n `artifact_hash` REGEXP '^[a-f0-9]{64}$'\n AND `request_fingerprint` REGEXP '^sha256:[a-f0-9]{64}$'\n AND `response_fingerprint` REGEXP '^sha256:[a-f0-9]{64}$'\n AND `prompt_response_fingerprint` REGEXP '^sha256:[a-f0-9]{64}$'\n ),\n CONSTRAINT `knowledge_node_generation_receipts_json_ck`\n CHECK (JSON_TYPE(`receipt`) = 'OBJECT'),\n CONSTRAINT `knowledge_node_generation_receipts_bytes_ck`\n CHECK (OCTET_LENGTH(CAST(`receipt` AS CHAR)) <= 8388608),\n CONSTRAINT `knowledge_node_generation_receipts_pub_gen_nonzero_ck` CHECK (\n `publication_generation_id` REGEXP '^[0-9A-Fa-f]{8}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{12}$'\n AND `publication_generation_id` <> '00000000-0000-0000-0000-000000000000'\n ),\n FOREIGN KEY (`knowledge_space_id`)\n REFERENCES `knowledge_spaces` (`id`) ON DELETE CASCADE,\n FOREIGN KEY (`document_asset_id`)\n REFERENCES `document_assets` (`id`) ON DELETE CASCADE,\n FOREIGN KEY (`parse_artifact_id`)\n REFERENCES `parse_artifacts` (`id`) ON DELETE CASCADE\n);\n\nCREATE INDEX IF NOT EXISTS `knowledge_node_generation_receipts_document_idx`\n ON `knowledge_node_generation_receipts`\n (`knowledge_space_id`, `document_asset_id`, `publication_generation_id`, `parse_artifact_id`);\n", path: "packages/database/migrations/0043_semantic_generation_receipts.tidb.sql" },
] as const satisfies readonly MigrationArtifact[];

4
knowledge-fs/packages/database/src/migration-file.test.ts
View File

@ -144,6 +144,8 @@ describe("migration file rendering", () => {
"packages/database/migrations/0041_logical_document_availability.tidb.sql",
"packages/database/migrations/0042_workflow_failed_retrieval_capture.postgres.sql",
"packages/database/migrations/0042_workflow_failed_retrieval_capture.tidb.sql",
"packages/database/migrations/0043_semantic_generation_receipts.postgres.sql",
"packages/database/migrations/0043_semantic_generation_receipts.tidb.sql",
]);
const workflowCapturePostgres = artifacts.find(
(artifact) =>
@ -892,6 +894,7 @@ describe("migration file rendering", () => {
"packages/database/migrations/0040_knowledge_space_metadata.postgres.sql",
"packages/database/migrations/0041_logical_document_availability.postgres.sql",
"packages/database/migrations/0042_workflow_failed_retrieval_capture.postgres.sql",
"packages/database/migrations/0043_semantic_generation_receipts.postgres.sql",
]);
expect(
getPendingMigrationArtifacts({
@ -938,6 +941,7 @@ describe("migration file rendering", () => {
"0040_knowledge_space_metadata",
"0041_logical_document_availability",
"0042_workflow_failed_retrieval_capture",
"0043_semantic_generation_receipts",
],
dialect: "postgres",
}),

34
knowledge-fs/packages/database/src/schema.test.ts
View File

@ -49,6 +49,7 @@ describe("database schema catalog", () => {
"knowledge_fs_leases",
"retrieval_execution_leases",
"knowledge_nodes",
"knowledge_node_generation_receipts",
"index_projections",
"index_projection_fts_postings",
"tidb_fts_posting_backfills",
@ -119,6 +120,39 @@ describe("database schema catalog", () => {
]);
});
it("models immutable semantic generation receipts with bounded JSON and cascade ownership", () => {
const schema = getDatabaseSchema();
const table = findTable(schema, "knowledge_node_generation_receipts");
expect(table.primaryKey).toEqual([
"knowledge_space_id",
"publication_generation_id",
"parse_artifact_id",
]);
expect(table.checkConstraints?.map((constraint) => constraint.name)).toEqual(
expect.arrayContaining([
"knowledge_node_generation_receipts_counts_ck",
"knowledge_node_generation_receipts_hashes_ck",
"knowledge_node_generation_receipts_json_ck",
"knowledge_node_generation_receipts_bytes_ck",
"knowledge_node_generation_receipts_pub_gen_nonzero_ck",
]),
);
expect(table.foreignKeys).toEqual(
expect.arrayContaining([
expect.objectContaining({ referencedTable: "knowledge_spaces", onDelete: "CASCADE" }),
expect.objectContaining({ referencedTable: "document_assets", onDelete: "CASCADE" }),
expect.objectContaining({ referencedTable: "parse_artifacts", onDelete: "CASCADE" }),
]),
);
expect(findIndex(schema, "knowledge_node_generation_receipts_document_idx").columns).toEqual([
"knowledge_space_id",
"document_asset_id",
"publication_generation_id",
"parse_artifact_id",
]);
});
it("models durable bulk task history with exact authorization provenance", () => {
const schema = getDatabaseSchema();
const table = findTable(schema, "bulk_operations");

86
knowledge-fs/packages/database/src/schema.ts
View File

@ -2306,6 +2306,80 @@ const tables = [
),
],
},
{
name: "knowledge_node_generation_receipts",
checkConstraints: [
publicationGenerationCheck(
"knowledge_node_generation_receipts_pub_gen_nonzero_ck",
"publication_generation_id",
false,
),
{
expression: {
postgres:
'"document_chunk_count" >= 0 AND "stored_node_count" >= 0 AND "stored_node_count" <= "document_chunk_count"',
tidb: "`document_chunk_count` >= 0 AND `stored_node_count` >= 0 AND `stored_node_count` <= `document_chunk_count`",
},
name: "knowledge_node_generation_receipts_counts_ck",
},
{
expression: {
postgres:
"\"artifact_hash\" ~ '^[a-f0-9]{64}$' AND \"request_fingerprint\" ~ '^sha256:[a-f0-9]{64}$' AND \"response_fingerprint\" ~ '^sha256:[a-f0-9]{64}$' AND \"prompt_response_fingerprint\" ~ '^sha256:[a-f0-9]{64}$'",
tidb: "`artifact_hash` REGEXP '^[a-f0-9]{64}$' AND `request_fingerprint` REGEXP '^sha256:[a-f0-9]{64}$' AND `response_fingerprint` REGEXP '^sha256:[a-f0-9]{64}$' AND `prompt_response_fingerprint` REGEXP '^sha256:[a-f0-9]{64}$'",
},
name: "knowledge_node_generation_receipts_hashes_ck",
},
{
expression: {
postgres: "jsonb_typeof(\"receipt\") = 'object'",
tidb: "JSON_TYPE(`receipt`) = 'OBJECT'",
},
name: "knowledge_node_generation_receipts_json_ck",
},
{
expression: {
postgres: 'octet_length("receipt"::text) <= 8388608',
tidb: "OCTET_LENGTH(CAST(`receipt` AS CHAR)) <= 8388608",
},
name: "knowledge_node_generation_receipts_bytes_ck",
},
],
foreignKeys: [
{
columns: ["knowledge_space_id"],
onDelete: "CASCADE",
referencedColumns: ["id"],
referencedTable: "knowledge_spaces",
},
{
columns: ["document_asset_id"],
onDelete: "CASCADE",
referencedColumns: ["id"],
referencedTable: "document_assets",
},
{
columns: ["parse_artifact_id"],
onDelete: "CASCADE",
referencedColumns: ["id"],
referencedTable: "parse_artifacts",
},
],
columns: [
idColumn("knowledge_space_id"),
idColumn("publication_generation_id"),
idColumn("parse_artifact_id"),
idColumn("document_asset_id"),
varcharColumn("artifact_hash", 64),
integerColumn("document_chunk_count"),
integerColumn("stored_node_count"),
varcharColumn("request_fingerprint", 71),
varcharColumn("response_fingerprint", 71),
varcharColumn("prompt_response_fingerprint", 71),
jsonColumn("receipt"),
],
primaryKey: ["knowledge_space_id", "publication_generation_id", "parse_artifact_id"],
},
{
name: "index_projections",
checkConstraints: [
@ -6746,6 +6820,18 @@ const indexes = [
postgres: "GIN",
},
},
{
columns: [
"knowledge_space_id",
"document_asset_id",
"publication_generation_id",
"parse_artifact_id",
],
name: "knowledge_node_generation_receipts_document_idx",
purpose:
"Delete immutable semantic-generation receipts for one tombstoned document without a table scan",
tableName: "knowledge_node_generation_receipts",
},
{
columns: ["knowledge_space_id", "id", "version"],
name: "document_assets_space_id_version_uq",

3
knowledge-fs/pnpm-lock.yaml generated
View File

@ -172,6 +172,9 @@ importers:
sharp:
specifier: 0.35.3
version: 0.35.3(@types/node@22.19.18)
unicode-segmenter:
specifier: 0.15.0
version: 0.15.0
zod:
specifier: ^3.24.1
version: 3.25.76

374
knowledge-fs/scripts/semantic-compilation-rollout.mjs Normal file
View File

@ -0,0 +1,374 @@
#!/usr/bin/env node
import { readFile } from "node:fs/promises";
const textDecoder = new TextDecoder();
const mode = process.env.SEMANTIC_ROLLOUT_MODE?.trim() || "static";
const supportedModes = new Set(["backfill", "canary", "preflight", "rollback", "static"]);
const mutatingModes = new Set(["backfill", "canary", "rollback"]);
const apiBase = normalizeBaseUrl(process.env.SEMANTIC_ROLLOUT_API_BASE ?? "http://127.0.0.1:8788");
const token = process.env.SEMANTIC_ROLLOUT_AUTH_TOKEN?.trim() || "dev-token";
const maxJsonBytes = positiveInteger(
process.env.SEMANTIC_ROLLOUT_MAX_JSON_BYTES ?? "1048576",
"SEMANTIC_ROLLOUT_MAX_JSON_BYTES",
);
const maxPolls = positiveInteger(
process.env.SEMANTIC_ROLLOUT_MAX_POLLS ?? "120",
"SEMANTIC_ROLLOUT_MAX_POLLS",
);
const pollIntervalMs = nonnegativeInteger(
process.env.SEMANTIC_ROLLOUT_POLL_INTERVAL_MS ?? "2000",
"SEMANTIC_ROLLOUT_POLL_INTERVAL_MS",
);
if (!supportedModes.has(mode)) {
throw new Error(`Unsupported SEMANTIC_ROLLOUT_MODE=${mode}`);
}
const staticEvidence = await verifyStaticEvidence();
if (mode === "static") {
printResult({ mode, staticEvidence });
process.exit(0);
}
const knowledgeSpaceId = requiredUuid(
process.env.SEMANTIC_ROLLOUT_SPACE_ID,
"SEMANTIC_ROLLOUT_SPACE_ID",
);
if (mutatingModes.has(mode)) {
assertMutationConfirmation(mode, knowledgeSpaceId);
}
const preflight = await runPreflight(knowledgeSpaceId);
if (mode === "preflight") {
printResult({ knowledgeSpaceId, mode, preflight, staticEvidence });
process.exit(0);
}
if (mode === "canary") {
const documentIds = requiredUuidList(
process.env.SEMANTIC_ROLLOUT_DOCUMENT_IDS,
"SEMANTIC_ROLLOUT_DOCUMENT_IDS",
);
const result = await reindexDocuments({ documentIds, knowledgeSpaceId });
await verifyOutlines(knowledgeSpaceId, documentIds);
const retrieval = await verifyRetrievalIfConfigured(knowledgeSpaceId);
printResult({ knowledgeSpaceId, mode, preflight, result, retrieval, staticEvidence });
process.exit(0);
}
if (mode === "backfill") {
const result = await reindexDocuments({ all: true, knowledgeSpaceId });
const documentIds = result.items
.map((item) => item?.asset?.id)
.filter((value) => typeof value === "string");
await verifyOutlines(knowledgeSpaceId, documentIds);
const retrieval = await verifyRetrievalIfConfigured(knowledgeSpaceId);
printResult({ knowledgeSpaceId, mode, preflight, result, retrieval, staticEvidence });
process.exit(0);
}
const rollback = await rollbackDocument(knowledgeSpaceId);
printResult({ knowledgeSpaceId, mode, preflight, rollback, staticEvidence });
async function verifyStaticEvidence() {
const expectedMigrationId = "0043_semantic_generation_receipts";
const paths = [
`packages/database/migrations/${expectedMigrationId}.postgres.sql`,
`packages/database/migrations/${expectedMigrationId}.tidb.sql`,
];
for (const path of paths) {
const source = await readFile(new URL(`../${path}`, import.meta.url), "utf8");
if (
!source.includes(expectedMigrationId) ||
!source.includes("knowledge_node_generation_receipts")
) {
throw new Error(`Semantic receipt migration evidence is incomplete in ${path}`);
}
}
const registry = await readFile(
new URL("../packages/database/src/migration-artifacts.generated.ts", import.meta.url),
"utf8",
);
if (!registry.includes(expectedMigrationId)) {
throw new Error(
"Generated migration registry does not contain semantic receipt migration 0043",
);
}
return { migrationId: expectedMigrationId, registry: "present" };
}
async function runPreflight(spaceId) {
const encoded = encodeURIComponent(spaceId);
const [health, settings, documents, tasks] = await Promise.all([
requestJson("/health", { expectedStatus: 200, method: "GET" }),
requestJson(`/knowledge-spaces/${encoded}/settings`, {
expectedStatus: 200,
method: "GET",
}),
requestJson(`/knowledge-spaces/${encoded}/documents?limit=100`, {
expectedStatus: 200,
method: "GET",
}),
requestJson(`/knowledge-spaces/${encoded}/background-tasks?limit=50`, {
expectedStatus: 200,
method: "GET",
}),
]);
if (!health || typeof health !== "object") throw new Error("KnowledgeFS health is invalid");
if (health.components?.database === false || health.components?.objectStorage === false) {
throw new Error("KnowledgeFS durable dependencies are unhealthy");
}
if (!settings || typeof settings !== "object")
throw new Error("KnowledgeFS settings are invalid");
if (!Array.isArray(documents.items)) throw new Error("KnowledgeFS document list is invalid");
if (!Array.isArray(tasks.items)) throw new Error("KnowledgeFS background-task list is invalid");
const activeFailures = tasks.items.filter(
(task) => task?.state === "failed" && task?.operation === "document_reindex",
);
return {
activeReindexFailures: activeFailures.length,
configurationState: settings.configurationState ?? settings.configuration_state ?? "unknown",
documentCount: documents.items.length,
healthComponents: health.components ?? {},
};
}
async function reindexDocuments(input) {
const body = input.all ? { all: true } : { documentIds: input.documentIds };
const result = await requestJson(
`/knowledge-spaces/${encodeURIComponent(input.knowledgeSpaceId)}/documents/bulk/reindex`,
{
body: JSON.stringify(body),
expectedStatus: 202,
headers: { "content-type": "application/json" },
method: "POST",
},
);
if (!Array.isArray(result.items) || typeof result.bulkJobId !== "string") {
throw new Error("Bulk reindex response is invalid");
}
const rejected = result.items.filter((item) => item?.status !== "queued");
if (rejected.length > 0) {
throw new Error(`Bulk reindex rejected ${rejected.length} document(s)`);
}
for (const item of result.items) {
const statusUrl = requiredString(item.statusUrl, "bulk reindex statusUrl");
await pollTask(statusUrl);
}
return {
bulkJobId: result.bulkJobId,
documentsQueued: result.items.length,
items: result.items,
};
}
async function rollbackDocument(spaceId) {
const documentId = requiredUuid(
process.env.SEMANTIC_ROLLOUT_ROLLBACK_DOCUMENT_ID,
"SEMANTIC_ROLLOUT_ROLLBACK_DOCUMENT_ID",
);
const targetRevision = positiveInteger(
process.env.SEMANTIC_ROLLOUT_ROLLBACK_REVISION,
"SEMANTIC_ROLLOUT_ROLLBACK_REVISION",
);
const encodedSpace = encodeURIComponent(spaceId);
const encodedDocument = encodeURIComponent(documentId);
const current = await requestJson(
`/knowledge-spaces/${encodedSpace}/logical-documents/${encodedDocument}`,
{ expectedStatus: 200, method: "GET" },
);
const expectedActiveRevision = positiveInteger(
current.activeRevision,
"logical document activeRevision",
);
const expectedRowVersion = nonnegativeInteger(current.rowVersion, "logical document rowVersion");
const task = await requestJson(
`/knowledge-spaces/${encodedSpace}/documents/${encodedDocument}/revisions/${targetRevision}/rollback`,
{
body: JSON.stringify({ expectedActiveRevision, expectedRowVersion }),
expectedStatus: 202,
headers: { "content-type": "application/json" },
method: "POST",
},
);
const taskId = requiredUuid(task.id, "rollback task id");
await pollTask(
`/knowledge-spaces/${encodedSpace}/documents/${encodedDocument}/processing-tasks/${encodeURIComponent(taskId)}`,
);
const restored = await requestJson(
`/knowledge-spaces/${encodedSpace}/logical-documents/${encodedDocument}`,
{ expectedStatus: 200, method: "GET" },
);
if (restored.activeRevision !== targetRevision) {
throw new Error(
`Rollback completed without activating revision ${targetRevision}; observed ${String(restored.activeRevision)}`,
);
}
return { documentId, fromRevision: expectedActiveRevision, taskId, toRevision: targetRevision };
}
async function verifyOutlines(spaceId, documentIds) {
for (const documentId of documentIds) {
const outline = await requestJson(
`/knowledge-spaces/${encodeURIComponent(spaceId)}/documents/${encodeURIComponent(documentId)}/outline`,
{ expectedStatus: 200, method: "GET" },
);
if (!Array.isArray(outline.nodes) || outline.nodes.length === 0) {
throw new Error(`Semantic outline is empty for document ${documentId}`);
}
if (
outline.nodes.some(
(node) => !Array.isArray(node?.sectionPath) || !Array.isArray(node?.sourceNodeIds),
)
) {
throw new Error(`Semantic outline provenance is incomplete for document ${documentId}`);
}
}
}
async function verifyRetrievalIfConfigured(spaceId) {
const query = process.env.SEMANTIC_ROLLOUT_QUERY?.trim();
if (!query) return { checked: false };
const result = await requestJson(
`/knowledge-spaces/${encodeURIComponent(spaceId)}/retrieval-tests`,
{
body: JSON.stringify({ includeText: true, mode: "research", query }),
expectedStatus: 200,
headers: { "content-type": "application/json" },
method: "POST",
},
);
if (!Array.isArray(result.items)) throw new Error("Retrieval verification response is invalid");
const minimumItems = nonnegativeInteger(
process.env.SEMANTIC_ROLLOUT_MIN_RETRIEVAL_ITEMS ?? "1",
"SEMANTIC_ROLLOUT_MIN_RETRIEVAL_ITEMS",
);
if (result.items.length < minimumItems) {
throw new Error(
`Retrieval verification returned ${result.items.length} item(s); expected at least ${minimumItems}`,
);
}
return { checked: true, itemCount: result.items.length, mode: result.mode ?? "research" };
}
async function pollTask(path) {
for (let poll = 1; poll <= maxPolls; poll += 1) {
const task = await requestJson(path, { expectedStatus: 200, method: "GET" });
const terminalState = task.state ?? task.stage;
if (["completed", "published", "ready", "smoke_eval_passed"].includes(terminalState)) {
return task;
}
if (["canceled", "failed"].includes(terminalState)) {
throw new Error(`Rollout task ${path} ended in ${terminalState}: ${task.errorMessage ?? ""}`);
}
if (poll < maxPolls) await delay(pollIntervalMs);
}
throw new Error(`Rollout task ${path} exceeded SEMANTIC_ROLLOUT_MAX_POLLS=${maxPolls}`);
}
async function requestJson(path, options) {
const response = await fetch(new URL(path, apiBase), {
body: options.body,
headers: { authorization: `Bearer ${token}`, ...(options.headers ?? {}) },
method: options.method,
});
const payload = await readBoundedJson(response);
const expected = Array.isArray(options.expectedStatus)
? options.expectedStatus
: [options.expectedStatus];
if (!expected.includes(response.status)) {
throw new Error(
`${options.method} ${path} returned ${response.status}: ${JSON.stringify(payload)}`,
);
}
return payload;
}
async function readBoundedJson(response) {
if (!response.body) return {};
const reader = response.body.getReader();
const chunks = [];
let totalBytes = 0;
try {
while (true) {
const { done, value } = await reader.read();
if (done) break;
totalBytes += value.byteLength;
if (totalBytes > maxJsonBytes) {
throw new Error(
`Rollout response exceeded SEMANTIC_ROLLOUT_MAX_JSON_BYTES=${maxJsonBytes}`,
);
}
chunks.push(value);
}
} finally {
reader.releaseLock();
}
const bytes = new Uint8Array(totalBytes);
let offset = 0;
for (const chunk of chunks) {
bytes.set(chunk, offset);
offset += chunk.byteLength;
}
const text = textDecoder.decode(bytes);
return text ? JSON.parse(text) : {};
}
function assertMutationConfirmation(selectedMode, spaceId) {
if (process.env.SEMANTIC_ROLLOUT_APPLY !== "1") {
throw new Error(`SEMANTIC_ROLLOUT_APPLY=1 is required for ${selectedMode}`);
}
const expected = `semantic:${selectedMode}:${spaceId}`;
if (process.env.SEMANTIC_ROLLOUT_CONFIRM !== expected) {
throw new Error(`SEMANTIC_ROLLOUT_CONFIRM must equal ${expected}`);
}
}
function requiredUuid(value, name) {
const normalized = requiredString(value, name);
if (
!/^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/iu.test(normalized)
) {
throw new Error(`${name} must be a UUID`);
}
return normalized;
}
function requiredUuidList(value, name) {
const items = requiredString(value, name)
.split(",")
.map((item) => requiredUuid(item.trim(), name));
return [...new Set(items)];
}
function requiredString(value, name) {
if (typeof value !== "string" || !value.trim()) throw new Error(`${name} is required`);
return value.trim();
}
function positiveInteger(value, name) {
const parsed = typeof value === "number" ? value : Number.parseInt(value ?? "", 10);
if (!Number.isSafeInteger(parsed) || parsed < 1) throw new Error(`${name} must be at least 1`);
return parsed;
}
function nonnegativeInteger(value, name) {
const parsed = typeof value === "number" ? value : Number.parseInt(value ?? "", 10);
if (!Number.isSafeInteger(parsed) || parsed < 0) {
throw new Error(`${name} must be a non-negative integer`);
}
return parsed;
}
function normalizeBaseUrl(value) {
const normalized = requiredString(value, "SEMANTIC_ROLLOUT_API_BASE");
return normalized.endsWith("/") ? normalized : `${normalized}/`;
}
function delay(milliseconds) {
return new Promise((resolve) => setTimeout(resolve, milliseconds));
}
function printResult(value) {
console.log(JSON.stringify(value, null, 2));
}

186
knowledge-fs/scripts/semantic-compilation-rollout.test.mjs Normal file
View File

@ -0,0 +1,186 @@
import assert from "node:assert/strict";
import { spawn } from "node:child_process";
import { readFile } from "node:fs/promises";
import { createServer } from "node:http";
import { test } from "node:test";
const rootPackage = JSON.parse(await readFile(new URL("../package.json", import.meta.url), "utf8"));
const scriptUrl = new URL("./semantic-compilation-rollout.mjs", import.meta.url);
const spaceId = "018f0d60-7a49-7cc2-9c1b-5b36f18f2c40";
const documentId = "018f0d60-7a49-7cc2-9c1b-5b36f18f2c41";
test("static rollout evidence is safe and includes migration 0043", async () => {
const result = await runScript({ SEMANTIC_ROLLOUT_MODE: "static" });
assert.equal(result.code, 0, result.stderr);
const payload = JSON.parse(result.stdout);
assert.equal(payload.mode, "static");
assert.equal(payload.staticEvidence.migrationId, "0043_semantic_generation_receipts");
});
test("mutating rollout modes require an exact space-scoped confirmation", async () => {
const result = await runScript({
SEMANTIC_ROLLOUT_DOCUMENT_IDS: documentId,
SEMANTIC_ROLLOUT_MODE: "canary",
SEMANTIC_ROLLOUT_SPACE_ID: spaceId,
});
assert.notEqual(result.code, 0);
assert.match(result.stderr, /SEMANTIC_ROLLOUT_APPLY=1 is required/u);
});
test("preflight is read-only, bounded, and reports durable state", async (context) => {
const requests = [];
const server = createServer((request, response) => {
requests.push({ method: request.method, url: request.url });
response.setHeader("content-type", "application/json");
if (request.url === "/health") {
response.end(JSON.stringify({ components: { database: true, objectStorage: true } }));
return;
}
if (request.url?.endsWith("/settings")) {
response.end(JSON.stringify({ configurationState: "active" }));
return;
}
if (request.url?.includes("/documents?")) {
response.end(JSON.stringify({ items: [{ id: documentId }] }));
return;
}
if (request.url?.includes("/background-tasks?")) {
response.end(JSON.stringify({ items: [] }));
return;
}
response.statusCode = 404;
response.end(JSON.stringify({ error: "not found" }));
});
await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
context.after(() => new Promise((resolve) => server.close(resolve)));
const address = server.address();
assert(address && typeof address === "object");
const result = await runScript({
SEMANTIC_ROLLOUT_API_BASE: `http://127.0.0.1:${address.port}`,
SEMANTIC_ROLLOUT_MODE: "preflight",
SEMANTIC_ROLLOUT_SPACE_ID: spaceId,
});
assert.equal(result.code, 0, result.stderr);
const payload = JSON.parse(result.stdout);
assert.equal(payload.preflight.configurationState, "active");
assert.equal(payload.preflight.documentCount, 1);
assert.deepEqual(
requests.map((request) => request.method),
["GET", "GET", "GET", "GET"],
);
});
test("canary reindexes only explicit documents and verifies the published outline", async (context) => {
const requests = [];
const server = createServer(async (request, response) => {
let body = "";
for await (const chunk of request) body += chunk;
requests.push({ body, method: request.method, url: request.url });
response.setHeader("content-type", "application/json");
if (request.url === "/health") {
response.end(JSON.stringify({ components: { database: true, objectStorage: true } }));
return;
}
if (request.url?.endsWith("/settings")) {
response.end(JSON.stringify({ configurationState: "active" }));
return;
}
if (request.url?.includes("/documents?")) {
response.end(JSON.stringify({ items: [{ id: documentId }] }));
return;
}
if (request.url?.includes("/background-tasks?")) {
response.end(JSON.stringify({ items: [] }));
return;
}
if (request.method === "POST" && request.url?.endsWith("/documents/bulk/reindex")) {
response.statusCode = 202;
response.end(
JSON.stringify({
bulkJobId: "bulk-canary",
items: [
{
asset: { id: documentId },
status: "queued",
statusUrl: "/jobs/canary",
},
],
}),
);
return;
}
if (request.url === "/jobs/canary") {
response.end(JSON.stringify({ stage: "published" }));
return;
}
if (request.url?.endsWith(`/documents/${documentId}/outline`)) {
response.end(
JSON.stringify({
nodes: [{ sectionPath: ["Canary"], sourceNodeIds: ["node-1"] }],
}),
);
return;
}
response.statusCode = 404;
response.end(JSON.stringify({ error: "not found" }));
});
await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
context.after(() => new Promise((resolve) => server.close(resolve)));
const address = server.address();
assert(address && typeof address === "object");
const result = await runScript({
SEMANTIC_ROLLOUT_API_BASE: `http://127.0.0.1:${address.port}`,
SEMANTIC_ROLLOUT_APPLY: "1",
SEMANTIC_ROLLOUT_CONFIRM: `semantic:canary:${spaceId}`,
SEMANTIC_ROLLOUT_DOCUMENT_IDS: documentId,
SEMANTIC_ROLLOUT_MAX_POLLS: "1",
SEMANTIC_ROLLOUT_MODE: "canary",
SEMANTIC_ROLLOUT_SPACE_ID: spaceId,
});
assert.equal(result.code, 0, result.stderr);
const payload = JSON.parse(result.stdout);
assert.equal(payload.result.documentsQueued, 1);
const mutation = requests.find((request) => request.method === "POST");
assert.deepEqual(JSON.parse(mutation?.body ?? "{}"), { documentIds: [documentId] });
});
test("package scripts expose explicit rollout phases and keep their tests in check", () => {
assert.equal(
rootPackage.scripts["semantic:rollout:static"],
"node scripts/semantic-compilation-rollout.mjs",
);
assert.match(
rootPackage.scripts["semantic:rollout:preflight"],
/SEMANTIC_ROLLOUT_MODE=preflight/u,
);
assert.match(rootPackage.scripts["semantic:rollout:canary"], /SEMANTIC_ROLLOUT_MODE=canary/u);
assert.match(rootPackage.scripts["semantic:rollout:backfill"], /SEMANTIC_ROLLOUT_MODE=backfill/u);
assert.match(rootPackage.scripts["semantic:rollout:rollback"], /SEMANTIC_ROLLOUT_MODE=rollback/u);
assert.equal(
rootPackage.scripts["semantic:rollout:test"],
"node --test scripts/semantic-compilation-rollout.test.mjs",
);
assert.match(rootPackage.scripts.check, /semantic:rollout:test/u);
});
function runScript(extraEnv) {
return new Promise((resolve) => {
const child = spawn(process.execPath, [scriptUrl.pathname], {
env: { ...process.env, ...extraEnv },
stdio: ["ignore", "pipe", "pipe"],
});
let stdout = "";
let stderr = "";
child.stdout.setEncoding("utf8");
child.stderr.setEncoding("utf8");
child.stdout.on("data", (chunk) => {
stdout += chunk;
});
child.stderr.on("data", (chunk) => {
stderr += chunk;
});
child.on("close", (code) => resolve({ code, stderr, stdout }));
});
}