mirror of
https://github.com/langgenius/dify.git
synced 2026-09-08 02:43:49 +08:00
chore: migrate knowledge-fs source tree
Import the committed KnowledgeFS snapshot dc4072ee302317145612087ce7440851dc329fd0 under knowledge-fs/ without its Git history, local IDE settings, or build artifacts.
This commit is contained in:
parent
873ef2b592
commit
4ee43b8afc
@ -8,6 +8,7 @@
|
||||
**/*.pyc
|
||||
**/.mypy_cache
|
||||
**/.ruff_cache
|
||||
knowledge-fs/
|
||||
.git
|
||||
.github
|
||||
*.md
|
||||
|
||||
5
.gitignore
vendored
5
.gitignore
vendored
@ -30,6 +30,11 @@ share/python-wheels/
|
||||
*.egg
|
||||
MANIFEST
|
||||
|
||||
# KnowledgeFS is an independently rooted TypeScript workspace. Its admin `lib`
|
||||
# directory contains source files rather than Python build output.
|
||||
!/knowledge-fs/apps/admin/lib/
|
||||
!/knowledge-fs/apps/admin/lib/**
|
||||
|
||||
# PyInstaller
|
||||
# Usually these files are written by a python script from a template
|
||||
# before PyInstaller builds the exe, so as to inject date/other infos into it.
|
||||
|
||||
7
knowledge-fs/.codex/AGENTS.md
Normal file
7
knowledge-fs/.codex/AGENTS.md
Normal file
@ -0,0 +1,7 @@
|
||||
# Agent Development Requirements
|
||||
|
||||
> This file in `.harness` folder records user-level development requirements that every agent must follow.
|
||||
|
||||
## Context Requirements
|
||||
|
||||
- Treat `.harness` as the complete information base for this project.
|
||||
1
knowledge-fs/.codex/config.toml
Normal file
1
knowledge-fs/.codex/config.toml
Normal file
@ -0,0 +1 @@
|
||||
sandbox_mode = "danger-full-access"
|
||||
15
knowledge-fs/.dockerignore
Normal file
15
knowledge-fs/.dockerignore
Normal file
@ -0,0 +1,15 @@
|
||||
.git
|
||||
.github
|
||||
.harness
|
||||
.next
|
||||
.turbo
|
||||
coverage
|
||||
dist
|
||||
node_modules
|
||||
**/.next
|
||||
**/.turbo
|
||||
**/coverage
|
||||
**/dist
|
||||
**/node_modules
|
||||
.env
|
||||
.env.*
|
||||
88
knowledge-fs/.env.example
Normal file
88
knowledge-fs/.env.example
Normal file
@ -0,0 +1,88 @@
|
||||
POSTGRES_DB=knowledge_fs
|
||||
POSTGRES_PASSWORD=knowledge_fs
|
||||
POSTGRES_PORT=5432
|
||||
POSTGRES_USER=knowledge_fs
|
||||
DATABASE_URL=postgresql://knowledge_fs:knowledge_fs@127.0.0.1:5432/knowledge_fs
|
||||
KNOWLEDGE_DATABASE_REPOSITORIES=
|
||||
DURABLE_DELETION_ENABLED=off
|
||||
DURABLE_DELETION_WRITER_FENCE_VERSION=
|
||||
# Keep this stable for the lifetime of deletion idempotency ledgers; do not rotate silently.
|
||||
DURABLE_DELETION_HMAC_KEY_BASE64=
|
||||
|
||||
MINIO_ACCESS_KEY=knowledge
|
||||
MINIO_API_PORT=9000
|
||||
MINIO_BUCKET=knowledge-fs
|
||||
MINIO_CONSOLE_PORT=9001
|
||||
MINIO_ENDPOINT=http://127.0.0.1:9000
|
||||
MINIO_REGION=us-east-1
|
||||
MINIO_ROOT_PASSWORD=knowledge-secret
|
||||
MINIO_ROOT_USER=knowledge
|
||||
MINIO_SECRET_KEY=knowledge-secret
|
||||
|
||||
R2_ACCESS_KEY_ID=
|
||||
R2_ACCOUNT_ID=
|
||||
R2_BUCKET=
|
||||
R2_REGION=auto
|
||||
R2_SECRET_ACCESS_KEY=
|
||||
|
||||
UNSTRUCTURED_PORT=8000
|
||||
UNSTRUCTURED_API_URL=http://127.0.0.1:8000
|
||||
UNSTRUCTURED_API_KEY=
|
||||
UNSTRUCTURED_MAX_RESPONSE_BYTES=
|
||||
UNSTRUCTURED_MAX_RETRIES=
|
||||
UNSTRUCTURED_RETRY_DELAY_MS=
|
||||
|
||||
OPENAI_API_KEY=
|
||||
ANTHROPIC_API_KEY=
|
||||
GEMINI_API_KEY=
|
||||
KNOWLEDGE_EMBEDDING_PROVIDER=
|
||||
KNOWLEDGE_EMBEDDING_MODEL=
|
||||
KNOWLEDGE_ENTITY_EXTRACTION_PROVIDER=
|
||||
KNOWLEDGE_ENTITY_EXTRACTION_MODEL=
|
||||
KNOWLEDGE_ENTITY_EXTRACTION_MAX_ENTITIES_PER_NODE=
|
||||
KNOWLEDGE_ENTITY_EXTRACTION_MAX_NODES_PER_RUN=
|
||||
KNOWLEDGE_ENTITY_EXTRACTION_MAX_OUTPUT_TOKENS=
|
||||
KNOWLEDGE_RELATION_EXTRACTION_MODEL=
|
||||
KNOWLEDGE_RELATION_EXTRACTION_MAX_RELATIONS_PER_NODE=
|
||||
KNOWLEDGE_RELATION_EXTRACTION_MAX_OUTPUT_TOKENS=
|
||||
KNOWLEDGE_COMMUNITY_SUMMARY_MODEL=
|
||||
KNOWLEDGE_COMMUNITY_SUMMARY_MAX_OUTPUT_TOKENS=
|
||||
# Graph-expanded retrieval (deep/research modes). On by default when the graph
|
||||
# repository is wired; set KNOWLEDGE_GRAPH_EXPANSION=off to disable. Tuning
|
||||
# knobs fall back to built-in defaults (in parentheses) when unset.
|
||||
KNOWLEDGE_GRAPH_EXPANSION=
|
||||
KNOWLEDGE_GRAPH_EXPANSION_MAX_DEPTH= # traversal hops, 1-2 (2)
|
||||
KNOWLEDGE_GRAPH_EXPANSION_FANOUT= # neighbors expanded per node (20)
|
||||
KNOWLEDGE_GRAPH_EXPANSION_MAX_SEED_ENTITIES= # seeds taken from base hits (5)
|
||||
KNOWLEDGE_GRAPH_EXPANSION_MAX_TRAVERSAL_NODES= # traversal node budget (50)
|
||||
KNOWLEDGE_GRAPH_EXPANSION_GRAPH_TOP_K= # entity names used to re-retrieve (10)
|
||||
KNOWLEDGE_GRAPH_EXPANSION_GRAPH_BOOST= # fusion weight of graph hits (0.2)
|
||||
KNOWLEDGE_GRAPH_EXPANSION_TIMEOUT_MS= # traversal time budget (250)
|
||||
# Scheduled source sync. Sources opt in via metadata.syncPolicy —
|
||||
# {"everyHours": 6} or {"dailyAt": ["03:00"], "utcOffset": "+08:00"}.
|
||||
# The scheduler is on by default (set KNOWLEDGE_SOURCE_SYNC=off to disable) and
|
||||
# is multi-replica safe: per-source runs are serialized by an atomic DB claim.
|
||||
KNOWLEDGE_SOURCE_SYNC=
|
||||
KNOWLEDGE_SOURCE_SYNC_TICK_MS= # scheduler tick interval (60000)
|
||||
KNOWLEDGE_SOURCE_SYNC_MAX_SOURCES_PER_TICK= # sources scanned per tick (200)
|
||||
# Answer synthesis LLM (opt-in). Unset/off => extractive evidence answers.
|
||||
# Set to openai|anthropic|gemini to let that provider write grounded answers.
|
||||
KNOWLEDGE_ANSWER_PROVIDER=
|
||||
KNOWLEDGE_ANSWER_MODEL=
|
||||
KNOWLEDGE_ANSWER_MAX_OUTPUT_TOKENS=
|
||||
# Gateway span tracing. off (default) | console (one JSON line per span) |
|
||||
# otlp (OTLP/HTTP JSON to an OpenTelemetry collector).
|
||||
KNOWLEDGE_TRACING=
|
||||
KNOWLEDGE_TRACING_OTLP_ENDPOINT= # e.g. http://localhost:4318/v1/traces
|
||||
KNOWLEDGE_TRACING_OTLP_HEADERS= # optional JSON object, e.g. {"authorization":"Bearer …"}
|
||||
KNOWLEDGE_TRACING_SERVICE_NAME= # resource service.name (knowledge-fs-api)
|
||||
KNOWLEDGE_TRACING_FLUSH_MS= # export batch interval (5000)
|
||||
|
||||
API_PORT=8788
|
||||
ADMIN_PORT=3000
|
||||
KNOWLEDGE_API_BASE_URL=http://localhost:8788
|
||||
NEXT_PUBLIC_API_BASE_URL=http://localhost:8788
|
||||
|
||||
KNOWLEDGE_DEV_AUTH_TOKEN=dev-token
|
||||
KNOWLEDGE_DEV_SUBJECT_ID=dev-user
|
||||
KNOWLEDGE_DEV_TENANT_ID=tenant-dev
|
||||
90
knowledge-fs/.github/workflows/ci.yml
vendored
Normal file
90
knowledge-fs/.github/workflows/ci.yml
vendored
Normal file
@ -0,0 +1,90 @@
|
||||
name: GitHub Flow
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches:
|
||||
- main
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
CI: true
|
||||
DOCKER_BUILDKIT: "1"
|
||||
|
||||
jobs:
|
||||
quality:
|
||||
name: Quality gates
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Setup pnpm
|
||||
uses: pnpm/action-setup@v6
|
||||
|
||||
- name: Setup Node
|
||||
uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version: 22
|
||||
cache: pnpm
|
||||
cache-dependency-path: pnpm-lock.yaml
|
||||
|
||||
- name: Install dependencies
|
||||
run: pnpm install --frozen-lockfile
|
||||
|
||||
- name: Check
|
||||
run: pnpm check
|
||||
|
||||
- name: Build
|
||||
run: pnpm build
|
||||
|
||||
- name: Lint
|
||||
run: pnpm lint
|
||||
|
||||
- name: Test app compose contract
|
||||
run: pnpm compose:apps:test
|
||||
|
||||
- name: Validate compose config
|
||||
run: pnpm compose:config
|
||||
|
||||
- name: Validate app compose config
|
||||
run: docker compose --env-file infra/local/.env.example -f infra/local/compose.yaml --profile apps config
|
||||
|
||||
docker-image:
|
||||
name: Production API image
|
||||
runs-on: ubuntu-latest
|
||||
needs: [quality]
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Setup pnpm
|
||||
uses: pnpm/action-setup@v6
|
||||
|
||||
- name: Setup Node
|
||||
uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version: 22
|
||||
cache: pnpm
|
||||
cache-dependency-path: pnpm-lock.yaml
|
||||
|
||||
- name: Build API Docker image
|
||||
run: pnpm docker:api:build
|
||||
|
||||
- name: Build Admin Docker image
|
||||
run: pnpm docker:admin:build
|
||||
|
||||
- name: Smoke isolated API bundle
|
||||
run: pnpm docker:api:bundle-smoke
|
||||
|
||||
- name: Smoke Admin Docker HTTP homepage
|
||||
run: pnpm docker:admin:http-smoke
|
||||
11
knowledge-fs/.gitignore
vendored
Normal file
11
knowledge-fs/.gitignore
vendored
Normal file
@ -0,0 +1,11 @@
|
||||
node_modules/
|
||||
dist/
|
||||
.turbo/
|
||||
.next/
|
||||
coverage/
|
||||
**/coverage/
|
||||
*.tsbuildinfo
|
||||
.env
|
||||
.env.*
|
||||
infra/local/.env
|
||||
!.env.example
|
||||
92
knowledge-fs/.harness/agents/development-requirements.md
Normal file
92
knowledge-fs/.harness/agents/development-requirements.md
Normal file
@ -0,0 +1,92 @@
|
||||
# Agent Development Requirements
|
||||
|
||||
> This file records user-level development requirements that every agent must follow.
|
||||
|
||||
## Context Requirements
|
||||
|
||||
- Treat `.harness` as the complete information base for this project.
|
||||
- While the temporary planning documents exist, every development round must read and carry:
|
||||
- `.harness/docs/TEMP-task-document.md`
|
||||
- `.harness/docs/TEMP-progress-document.md`
|
||||
- Update `.harness/docs/TEMP-progress-document.md` when work completes, blocks, or context/token limits approach.
|
||||
- Delete the temporary task and progress documents only after the full project development cycle is complete.
|
||||
- After those temporary documents have been intentionally deleted, continue maintenance from `.harness/docs/iteration-plan.md`, `.harness/changes`, and this requirements file instead of recreating temporary docs.
|
||||
|
||||
## Traceability Requirements
|
||||
|
||||
- All code, configuration, architecture, test, and documentation changes must be summarized under `.harness/changes`.
|
||||
- Each change summary must explain:
|
||||
- What changed.
|
||||
- Why it changed.
|
||||
- How it was verified.
|
||||
- Any known risks or follow-up work.
|
||||
- Do not rely only on git history for traceability.
|
||||
|
||||
## Review Cadence
|
||||
|
||||
- After every 10 implementation commits, pause forward feature iteration and review project health before continuing.
|
||||
- The review must cover:
|
||||
- Whether the technical direction still follows `.harness` architecture decisions.
|
||||
- Performance risks such as N+1 query paths, missing indexes, unbounded memory, repeated database round-trips, and large-object buffering.
|
||||
- Unit/integration test coverage and whether new behavior was developed with TDD.
|
||||
- CI/build/lint/test health.
|
||||
- Whether `.harness/changes` and temporary progress documents are complete.
|
||||
- Record the review checkpoint commit and findings in `.harness/docs/TEMP-progress-document.md`.
|
||||
- If the temporary progress document has already been intentionally deleted after project completion, record the review checkpoint and findings in `.harness/changes` instead.
|
||||
- Fix high-priority review findings before continuing regular iteration.
|
||||
|
||||
## TDD Requirements
|
||||
|
||||
- Follow `.harness/skills/test-driven-development/SKILL.md` for all logic, bug fixes, and behavior changes.
|
||||
- Write or update tests before implementing new behavior whenever the change is behavioral.
|
||||
- Prefer state-based tests over implementation-detail interaction tests.
|
||||
- Keep tests DAMP: each test should read as a clear behavioral specification.
|
||||
- Use the test pyramid:
|
||||
- Mostly small unit tests.
|
||||
- Focused integration tests for API, database, filesystem, and provider boundaries.
|
||||
- Limited E2E tests for critical user flows.
|
||||
- Project coverage must stay at or above 90% for lines, statements, branches, and functions.
|
||||
- Coverage gates are mandatory for packages that contain behavior.
|
||||
- If a package has no behavioral code yet, record that explicitly in the change summary.
|
||||
|
||||
## Performance Requirements
|
||||
|
||||
- This project has extremely high performance requirements.
|
||||
- Code must be designed to avoid:
|
||||
- N+1 queries.
|
||||
- Missing or unused indexes on high-traffic query paths.
|
||||
- Repeated database round-trips for data that can be batched.
|
||||
- Unbounded result sets or unbounded in-memory accumulation.
|
||||
- Memory leaks from long-lived references, global mutable caches, or uncapped buffers.
|
||||
- Query waterfalls between API, retrieval, evidence, and trace loading.
|
||||
- Database-facing work must document expected access patterns and required indexes.
|
||||
- Retrieval and KnowledgeFS paths must batch related entity loads by ids instead of looping over per-row queries.
|
||||
- New list/read APIs must include explicit pagination, limits, and stable ordering.
|
||||
- Cache usage must be bounded and version-aware.
|
||||
- Performance-sensitive behavior should have guard tests where practical, such as tests that verify required indexes or batched access contracts exist.
|
||||
- If a performance trade-off is accepted temporarily, record it in `.harness/changes` with a follow-up.
|
||||
|
||||
## Code Review Regression Guardrails
|
||||
|
||||
- Do not add new broad responsibilities to `packages/api/src/index.ts`; prefer focused modules for repositories, workflows, route helpers, and provider logic.
|
||||
- Do not reintroduce route-handler `context: any` assertions; if Hono/OpenAPI inference becomes too deep, isolate it behind a typed helper and keep validated request bodies/params/query values locally typed.
|
||||
- Runtime adapters must report their real backing implementation through `kind`; in-memory fallbacks must not masquerade as S3, R2, KV, or other durable services.
|
||||
- Production container entrypoints must run compiled JavaScript as a non-root user; `tsx` is allowed for development only.
|
||||
- Large object access should prefer streaming-capable adapter contracts; all eager byte reads must keep explicit size caps.
|
||||
- Admin UI network paths must go through the BFF allowlist instead of direct browser-to-API form actions.
|
||||
- BFF/API client request and response bodies must enforce byte limits while streaming or chunk-reading; do not call `arrayBuffer()`, `json()`, or `text()` on untrusted bodies before bounds are checked.
|
||||
- SQL identifier rendering must escape dialect quote characters, and generated migrations must remain deterministic through `pnpm db:migrations:check`.
|
||||
- New database relationships should declare foreign keys or document why a relationship is intentionally application-managed.
|
||||
- Cache and queue adapters must bound retained entries and retained bytes where applicable, and idempotency indexes must be cleaned up with terminal job lifecycle transitions.
|
||||
- SSE parsers must preserve multi-line `data:` semantics and avoid exposing raw provider payloads or credentials in errors/logs.
|
||||
- User-visible labels and utility formatting should be covered by tests for singular/plural grammar and rounding edge cases.
|
||||
|
||||
## Verification Requirements
|
||||
|
||||
- Before reporting completion, run the relevant verification commands.
|
||||
- For the current TypeScript-only workspace, the default verification set is:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- Record any skipped verification and the reason in both the progress document and the relevant change summary.
|
||||
- If the temporary progress document has already been intentionally deleted after project completion, record skipped verification only in the relevant `.harness/changes` summary.
|
||||
@ -0,0 +1,50 @@
|
||||
# Auth Subject Middleware
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added the shared `AuthSubject` contract for `subjectId`, `tenantId`, and scopes.
|
||||
- Added Bearer-token auth verifier support:
|
||||
- `createJwtAuthVerifier` validates signed JWTs with `jose`.
|
||||
- `createStaticAuthVerifier` supports tests and local/dev injection.
|
||||
- Protected `/knowledge-spaces` routes while keeping `/health` and `/openapi.json` public.
|
||||
- Removed client-supplied `tenantId` from KnowledgeSpace create/list API inputs.
|
||||
- Hardened in-memory KnowledgeSpace repository get/update/delete operations with tenant-scoped ids.
|
||||
|
||||
## Why
|
||||
|
||||
KnowledgeSpace CRUD must not trust tenant identity supplied by callers. This slice establishes the server-side subject boundary required before durable database execution and later permission filtering.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- RED: Gateway tests first asserted 401/403 responses, server-derived tenant ids, cross-tenant isolation, and JWT subject derivation before implementation.
|
||||
- GREEN: Implemented subject schema, JWT/static verifiers, auth middleware, route wiring, and tenant-scoped repository methods.
|
||||
- REFACTOR: Kept auth scope checks centralized in middleware and retained explicit bounded list behavior.
|
||||
|
||||
## Security / Performance Notes
|
||||
|
||||
- Business routes now require authenticated subjects and route tenant scope from trusted middleware context.
|
||||
- Cross-tenant reads, updates, and deletes return 404 instead of leaking resource existence.
|
||||
- List requests remain bounded by explicit `limit` and tenant-scoped filtering.
|
||||
- This is still an in-memory repository skeleton; durable database execution must use indexed tenant/id or tenant/slug access paths.
|
||||
|
||||
## Verification
|
||||
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm --filter @knowledge/core test:coverage`
|
||||
- `pnpm lint`
|
||||
- Full verification passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- JWT support uses a shared secret verifier only; JWKS/OIDC discovery remains a later integration.
|
||||
- Local app wiring currently has no dev auth verifier, so protected business routes correctly return 401 until a dev auth mode is intentionally added.
|
||||
@ -0,0 +1,51 @@
|
||||
# 2026-05-08 Bounded Cache Adapter
|
||||
|
||||
## Summary
|
||||
|
||||
- Expanded the core `CacheAdapter` contract beyond health checks.
|
||||
- Added a bounded in-memory cache adapter.
|
||||
- Wired Node and Cloudflare platform adapter skeletons to the cache contract.
|
||||
- Added TDD coverage for set/get/delete, TTL expiry, stats cleanup, max entry bounds, oldest-entry eviction, and byte-copy isolation.
|
||||
|
||||
## Files Added Or Updated
|
||||
|
||||
- `packages/core/src/platform-adapter.ts`
|
||||
- `packages/core/src/platform-adapter.test.ts`
|
||||
- `packages/adapters/src/cache.ts`
|
||||
- `packages/adapters/src/cache.test.ts`
|
||||
- `packages/adapters/src/cloudflare.ts`
|
||||
- `packages/adapters/src/node.ts`
|
||||
- `packages/adapters/src/index.ts`
|
||||
- `.harness/docs/TEMP-progress-document.md`
|
||||
- `.harness/changes/2026-05-08-bounded-cache-adapter.md`
|
||||
|
||||
## Why
|
||||
|
||||
The platform will rely on version-aware caches for retrieval, evidence, generation, provider health, and rate-limit coordination. Cache implementations must be bounded to avoid long-running memory growth.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- `maxEntries` is mandatory and must be at least 1.
|
||||
- Cache values are copied on set/get so callers cannot mutate retained state.
|
||||
- TTL expiry removes stale bytes.
|
||||
- Stats collection purges expired entries before reporting memory usage.
|
||||
- Eviction uses `Map` insertion order to remove the oldest key without sorting.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- RED: Added `packages/adapters/src/cache.test.ts`, then ran `pnpm --filter @knowledge/adapters test`.
|
||||
- The test failed because `./cache` did not exist.
|
||||
- GREEN: Added the cache contract and memory implementation.
|
||||
- REFACTOR: Replaced sort-based eviction with insertion-order eviction and added extra coverage for bounded configuration and expiry cleanup.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/adapters test`: passed.
|
||||
- `pnpm --filter @knowledge/adapters test:coverage`: passed.
|
||||
- `packages/adapters`: above 90% for lines, statements, branches, and functions.
|
||||
- `pnpm --filter @knowledge/core test:coverage`: passed.
|
||||
|
||||
## Known Risks And Follow-Up
|
||||
|
||||
- This is a bounded memory cache contract implementation, not the final Cloudflare KV or Redis adapter.
|
||||
- Future KV/Redis adapters must preserve TTL, bounded access, and byte-copy semantics where applicable.
|
||||
@ -0,0 +1,57 @@
|
||||
# Bounded Database Query Planner Contract
|
||||
|
||||
## What Changed
|
||||
|
||||
- Expanded `DatabaseAdapter` with bounded read planning methods:
|
||||
- `planListRows(input)`
|
||||
- `planBatchGetRows(input)`
|
||||
- Added shared query-plan input/output types in `@knowledge/core`.
|
||||
- Added list-plan validation for:
|
||||
- required positive integer `limit`
|
||||
- `maxListLimit`
|
||||
- declared table/index/columns
|
||||
- explicit covering index prefixes
|
||||
- stable `orderBy`
|
||||
- cursor shape and single-direction cursor paging
|
||||
- Added primary-key batch read planning with:
|
||||
- non-empty id lists
|
||||
- `maxBatchIds`
|
||||
- primary-key-only batch columns
|
||||
- Added dialect-specific SQL plan rendering for PostgreSQL and TiDB placeholders/quoting.
|
||||
- Added tests for success paths, bounded failures, index-prefix failures, cursor failures, unknown schema objects, and primary-key batch guardrails.
|
||||
|
||||
## Why
|
||||
|
||||
The project has strict performance requirements and will later introduce real Drizzle/SQL client execution. This slice establishes the adapter contract first, so database reads cannot be designed as unbounded scans, per-row waterfalls, or index-ambiguous list calls.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- RED: Added database planner tests to `packages/adapters/src/database.test.ts`.
|
||||
- The first run failed because `planListRows`, `planBatchGetRows`, and planner bounds did not exist.
|
||||
- GREEN: Implemented planner methods and validation in `packages/adapters/src/database.ts`.
|
||||
- REFACTOR: Added extra guard tests to lift coverage and make error behavior explicit.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- List plans require explicit `limit` and reject limits above `maxListLimit`.
|
||||
- List plans require a declared index and verify the query uses the leading index columns.
|
||||
- Batch reads use one primary-key `IN (...)` statement instead of one query per id.
|
||||
- Batch id counts are capped by `maxBatchIds`.
|
||||
- This still does not open a database connection, so no runtime query round-trips were introduced.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/adapters test -- src/database.test.ts`: passed.
|
||||
- `pnpm --filter @knowledge/adapters test:coverage`: passed.
|
||||
- `packages/adapters`: 98.6% lines/statements, 95.91% branches, 100% functions.
|
||||
- `pnpm --filter @knowledge/adapters typecheck`: passed.
|
||||
- `pnpm --filter @knowledge/core typecheck`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `pnpm check`: passed.
|
||||
- `pnpm build`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- The planner emits SQL strings for contract validation only; real Drizzle/SQL execution is still a later slice.
|
||||
- Cursor planning currently supports one direction across ordered columns. Mixed-direction cursor paging should only be added with a dedicated test and SQL strategy.
|
||||
@ -0,0 +1,52 @@
|
||||
# 2026-05-08 Bounded Job Queue Adapter
|
||||
|
||||
## Summary
|
||||
|
||||
- Expanded the core `JobQueueAdapter` contract beyond health checks.
|
||||
- Added a bounded inline job queue adapter for local and skeleton runtimes.
|
||||
- Wired Node and Cloudflare platform adapter skeletons to the inline queue contract.
|
||||
- Added TDD coverage for enqueue/dequeue/complete/fail, bounded batch size, bounded queued jobs, idempotency, delayed jobs, retry scheduling, and payload clone isolation.
|
||||
|
||||
## Files Added Or Updated
|
||||
|
||||
- `packages/core/src/platform-adapter.ts`
|
||||
- `packages/core/src/platform-adapter.test.ts`
|
||||
- `packages/adapters/src/job-queue.ts`
|
||||
- `packages/adapters/src/job-queue.test.ts`
|
||||
- `packages/adapters/src/cloudflare.ts`
|
||||
- `packages/adapters/src/node.ts`
|
||||
- `packages/adapters/src/index.ts`
|
||||
- `.harness/docs/TEMP-progress-document.md`
|
||||
- `.harness/changes/2026-05-08-bounded-job-queue-adapter.md`
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 1 requires adapter contracts for job coordination. The platform needs a bounded queue abstraction before ingestion, parsing, indexing, and background evaluation can safely schedule work.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- `maxQueuedJobs` is mandatory and must be at least 1.
|
||||
- `maxBatchSize` is mandatory and must be at least 1.
|
||||
- `dequeue` rejects unbounded and oversized batch requests.
|
||||
- Idempotency keys prevent duplicate active jobs for the same logical work.
|
||||
- Payloads are cloned on enqueue and dequeue so callers cannot mutate retained queue state.
|
||||
- The implementation avoids `delete` on hot mutable job records to preserve object shape stability.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- RED: Added `packages/adapters/src/job-queue.test.ts`, then ran `pnpm --filter @knowledge/adapters test`.
|
||||
- The test failed because `./job-queue` did not exist.
|
||||
- GREEN: Added the job queue contract and inline implementation.
|
||||
- REFACTOR: Fixed strict optional field handling, removed `delete`, and formatted with Biome.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/adapters test`: passed.
|
||||
- `pnpm --filter @knowledge/adapters typecheck`: passed.
|
||||
- `pnpm --filter @knowledge/adapters test:coverage`: passed.
|
||||
- `packages/adapters`: above 90% for lines, statements, branches, and functions.
|
||||
|
||||
## Known Risks And Follow-Up
|
||||
|
||||
- This is a bounded inline queue implementation, not the final Cloudflare Queues or pg-boss adapter.
|
||||
- Future real queue adapters must preserve bounded dequeue semantics, idempotency behavior, and payload immutability at the adapter boundary.
|
||||
@ -0,0 +1,47 @@
|
||||
# Checked-In Database Migration Artifacts
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added deterministic initial schema migration artifacts for PostgreSQL and TiDB.
|
||||
- Updated migration rendering to use a stable migration id instead of a generated timestamp.
|
||||
- Added `getInitialSchemaMigrationArtifacts()` and `findMigrationArtifactDrift()` to keep generated SQL artifacts tied to the schema catalog.
|
||||
- Added `pnpm db:migrations:write` and `pnpm db:migrations:check`.
|
||||
- Added migration drift checking to root `pnpm check`.
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 1 requires migrations to compile for PostgreSQL and TiDB dialect targets. Checked-in generated SQL artifacts make the current schema auditable, while the drift check prevents catalog changes from silently leaving migration files stale.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- RED: Updated `packages/database/src/migration-file.test.ts` to require stable migration ids and checked-in artifact metadata.
|
||||
- RED: `pnpm db:migrations:check` failed because the script did not exist, then failed again because the SQL artifacts were missing.
|
||||
- GREEN: Added artifact rendering, drift detection, the migration CLI script, root scripts, and generated SQL files.
|
||||
- REFACTOR: Kept filesystem CLI code outside `src` so package coverage remains focused on library behavior.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- Drift checking performs a bounded comparison over the known migration artifact list.
|
||||
- No database connections or live migrations run in this slice.
|
||||
- Existing schema indexes remain the source of truth for high-traffic access patterns and are preserved in the generated SQL.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm db:migrations:write`: passed.
|
||||
- `pnpm db:migrations:check`: passed.
|
||||
- `pnpm --filter @knowledge/database test:coverage`: passed.
|
||||
- `packages/database`: 100% lines/statements/branches/functions.
|
||||
- `pnpm check`: passed.
|
||||
- `pnpm build`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
- `pnpm wasm:build`: passed.
|
||||
- `pnpm compose:config`: passed.
|
||||
- `docker compose --profile apps config`: passed.
|
||||
- `git diff --check`: passed.
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- This slice does not execute migrations against PostgreSQL or TiDB.
|
||||
- Drizzle definitions remain deferred; the schema catalog is the migration source for now.
|
||||
- Live database migration tests can be added once container-backed integration tests are accepted.
|
||||
@ -0,0 +1,45 @@
|
||||
# CI Workflow And WASM Build Gate
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added a GitHub Actions CI workflow for pull requests and pushes to `main`.
|
||||
- CI now runs TypeScript checks, Vitest coverage gates, Rust checks/tests, WASM build, lint, and Docker Compose config validation.
|
||||
- Added root `wasm:build` script using the pinned local `wasm-pack` package.
|
||||
- Added `wasm-pack` `0.14.0` as a root dev dependency and updated `pnpm-lock.yaml`.
|
||||
- Added `rust-toolchain.toml` to make the Rust stable toolchain and `wasm32-unknown-unknown` target explicit for local and CI runs.
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 1 requires PR-level CI/CD and a real `wasm-pack build` gate for the placeholder Rust WASM crate. This turns the local verification chain into repeatable CI checks without adding deployment, publishing, or live container integration yet.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- RED: `test -f .github/workflows/ci.yml` failed because no CI workflow existed.
|
||||
- RED: `pnpm wasm:build` failed because the root script did not exist.
|
||||
- GREEN: Added the workflow, script, pinned `wasm-pack` dependency, lockfile update, and Rust toolchain file.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- CI validates existing bounded adapter and database planner tests, including coverage gates above 90%.
|
||||
- The workflow validates Compose rendering only; it does not start local databases or object stores.
|
||||
- WASM build output remains ignored via `pkg/`, avoiding tracked generated artifact churn.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm install --frozen-lockfile`: passed.
|
||||
- `pnpm wasm:build`: passed.
|
||||
- `pnpm check`: passed.
|
||||
- `packages/adapters`: 96.82% lines/statements, 96.12% branches, 100% functions.
|
||||
- `packages/api`, `packages/core`, and `packages/database`: 100% lines/statements/branches/functions.
|
||||
- `pnpm build`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
- `pnpm compose:config`: passed.
|
||||
- `docker compose --profile apps config`: passed.
|
||||
- `git diff --check`: passed.
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- The CI workflow does not deploy, publish Docker images, or run wrangler deployment.
|
||||
- Live MinIO/PostgreSQL/Unstructured integration remains deferred to a later container-backed test slice.
|
||||
- On macOS arm64, `wasm-pack` may compile `wasm-bindgen-cli` on first run if no prebuilt binary is available; the cache absorbs that after the first run.
|
||||
@ -0,0 +1,54 @@
|
||||
# 2026-05-08 Core Domain Models
|
||||
|
||||
## Summary
|
||||
|
||||
- Added first-sprint core domain model schemas in `packages/core`.
|
||||
- Covered the contract surface for:
|
||||
- `KnowledgeSpace`
|
||||
- `Source`
|
||||
- `DocumentAsset`
|
||||
- `ParseArtifact`
|
||||
- `KnowledgeNode`
|
||||
- `IndexProjection`
|
||||
- `KnowledgePath`
|
||||
- `EvidenceBundle`
|
||||
- `AnswerTrace`
|
||||
- Exported the model contracts from `@knowledge/core`.
|
||||
- Added TDD tests before implementation and confirmed the RED step failed because `./models` did not exist.
|
||||
|
||||
## Files Added Or Updated
|
||||
|
||||
- `packages/core/src/models.ts`
|
||||
- `packages/core/src/models.test.ts`
|
||||
- `packages/core/src/index.ts`
|
||||
- `.harness/docs/TEMP-progress-document.md`
|
||||
- `.harness/changes/2026-05-08-core-domain-models.md`
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 1 requires core data model scaffolding before database migrations and adapter work can be implemented safely.
|
||||
|
||||
The Zod domain models establish runtime-validated contracts for the metadata, document, artifact, node, projection, virtual path, evidence, and trace entities described in the `.harness` architecture documents.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- RED: Added `packages/core/src/models.test.ts`, then ran `pnpm --filter @knowledge/core test`.
|
||||
- The test failed because `./models` did not exist.
|
||||
- GREEN: Added `packages/core/src/models.ts` and exported it from `packages/core/src/index.ts`.
|
||||
- REFACTOR: Ran Biome formatting and full verification.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/core test`: passed.
|
||||
- `pnpm --filter @knowledge/core test:coverage`: passed.
|
||||
- `packages/core`: 100% lines, statements, branches, and functions.
|
||||
- `pnpm check`: passed.
|
||||
- `pnpm build`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
|
||||
## Known Risks And Follow-Up
|
||||
|
||||
- These are domain-level Zod contracts, not Drizzle database tables yet.
|
||||
- Next Sprint 1 work should add database schema/migration scaffolding for PostgreSQL/TiDB or a database-neutral schema mapping layer.
|
||||
- Object storage adapter contract tests and local S3-compatible skeleton remain pending.
|
||||
@ -0,0 +1,50 @@
|
||||
# Database Capability Descriptors
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `DatabaseCapabilities` to `@knowledge/core`.
|
||||
- Added `getCapabilities()` to `DatabaseAdapter`.
|
||||
- Added schema adapter capability descriptors for:
|
||||
- dense vector support
|
||||
- full-text support
|
||||
- native CJK full-text behavior
|
||||
- recursive CTE support
|
||||
- concurrent vector and full-text retrieval
|
||||
- estimated vector/FTS p99 latency
|
||||
- max vector scale
|
||||
- SQL permission filtering
|
||||
- publication strategy
|
||||
- Wired Node/PostgreSQL and Cloudflare/TiDB skeletons through the shared capability contract.
|
||||
- Added tests proving PostgreSQL and TiDB descriptors differ where planner behavior will need to branch.
|
||||
|
||||
## Why
|
||||
|
||||
The architecture requires retrieval planning to adapt by backend. TiDB and PostgreSQL both serve as the unified database, but they differ in CJK full-text behavior, practical vector scale, and latency assumptions. Exposing this through `DatabaseAdapter` prevents future retrieval code from hard-coding backend conditionals outside the adapter layer.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- RED: Added capability descriptor expectations to `packages/adapters/src/database.test.ts`.
|
||||
- The first run failed because `getCapabilities()` did not exist.
|
||||
- GREEN: Added the core contract and schema adapter descriptors.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- Capability descriptors make latency and scale assumptions explicit before retrieval planner work begins.
|
||||
- Permission filtering is declared as `sql-where`, preserving the requirement to filter in the database before ranking/evidence assembly.
|
||||
- No runtime database calls were introduced.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/adapters test:coverage`: passed.
|
||||
- `packages/adapters`: 98.69% lines/statements, 95.97% branches, 100% functions.
|
||||
- `pnpm --filter @knowledge/adapters typecheck`: passed.
|
||||
- `pnpm --filter @knowledge/core typecheck`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `pnpm check`: passed.
|
||||
- `pnpm build`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- Latency estimates are static planning assumptions until real PostgreSQL/TiDB benchmarks are added.
|
||||
- Retrieval planner implementation should consume these descriptors instead of branching directly on adapter kind.
|
||||
@ -0,0 +1,37 @@
|
||||
# 2026-05-08 Database Migration File Renderer
|
||||
|
||||
## Summary
|
||||
|
||||
- Added deterministic migration file rendering for the database schema catalog.
|
||||
- Supports PostgreSQL and TiDB dialect output.
|
||||
- Keeps migration rendering pure and side-effect free.
|
||||
- Added tests that ensure table creation statements appear before indexes.
|
||||
|
||||
## Files Added Or Updated
|
||||
|
||||
- `packages/database/src/migration-file.ts`
|
||||
- `packages/database/src/migration-file.test.ts`
|
||||
- `packages/database/src/index.ts`
|
||||
- `.harness/docs/TEMP-progress-document.md`
|
||||
- `.harness/changes/2026-05-08-database-migration-file-renderer.md`
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 1 requires migrations to compile for PostgreSQL and TiDB dialect targets. The schema catalog already renders SQL statements; this slice adds deterministic full migration text suitable for future generated migration files.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- RED: Added `packages/database/src/migration-file.test.ts`, then ran `pnpm --filter @knowledge/database test`.
|
||||
- The test failed because `./migration-file` did not exist.
|
||||
- GREEN: Added `renderMigrationFile` and exported it from `@knowledge/database`.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/database test:coverage`: passed.
|
||||
- `packages/database`: 100% lines, statements, branches, and functions.
|
||||
- `pnpm --filter @knowledge/database typecheck`: passed.
|
||||
|
||||
## Known Risks And Follow-Up
|
||||
|
||||
- This renderer does not write files yet; it only provides deterministic migration text.
|
||||
- A later dev-env or CI slice can write rendered migrations into tracked files if the project chooses checked-in SQL artifacts.
|
||||
@ -0,0 +1,77 @@
|
||||
# 2026-05-08 Initial Skeleton And Process Requirements
|
||||
|
||||
## Summary
|
||||
|
||||
- Added the initial TypeScript monorepo skeleton.
|
||||
- Added Hono API, Next.js Admin, core, API, and adapter package boundaries.
|
||||
- Added a Rust compute crate placeholder for future WASM-only pure compute modules.
|
||||
- Added temporary task and progress documents for multi-round continuity.
|
||||
- Recorded agent-level requirements for traceability and TDD.
|
||||
- Added coverage gates for current behavioral TypeScript packages.
|
||||
|
||||
## Files Added Or Updated
|
||||
|
||||
- Root workspace/config:
|
||||
- `package.json`
|
||||
- `pnpm-workspace.yaml`
|
||||
- `pnpm-lock.yaml`
|
||||
- `turbo.json`
|
||||
- `tsconfig.base.json`
|
||||
- `biome.json`
|
||||
- `.gitignore`
|
||||
- Apps:
|
||||
- `apps/api`
|
||||
- `apps/admin`
|
||||
- Packages:
|
||||
- `packages/core`
|
||||
- `packages/api`
|
||||
- `packages/adapters`
|
||||
- Rust:
|
||||
- `Cargo.toml`
|
||||
- `Cargo.lock`
|
||||
- `crates/knowledge_compute`
|
||||
- Harness:
|
||||
- `.harness/docs/TEMP-task-document.md`
|
||||
- `.harness/docs/TEMP-progress-document.md`
|
||||
- `.harness/agents/development-requirements.md`
|
||||
- `.harness/changes/2026-05-08-initial-skeleton-and-process.md`
|
||||
|
||||
## Why
|
||||
|
||||
The project needs a concrete foundation aligned with `.harness/docs/iteration-plan.md` and `.harness/docs/rag-platform-redesign-technical-selection.md`.
|
||||
|
||||
The user also required:
|
||||
|
||||
- All changes to be traceable under `.harness/changes`.
|
||||
- Project requirements to be recorded under `.harness/agents`.
|
||||
- TDD according to `.harness/skills/test-driven-development/SKILL.md`.
|
||||
- Test coverage above 90%.
|
||||
|
||||
## Verification
|
||||
|
||||
Initial skeleton verification completed before this change summary was created:
|
||||
|
||||
- `pnpm install`: passed.
|
||||
- `pnpm check`: passed.
|
||||
- `pnpm build`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
|
||||
After adding coverage gates:
|
||||
|
||||
- `pnpm test:coverage`: passed.
|
||||
- `packages/core`: 100% lines, statements, branches, and functions.
|
||||
- `packages/adapters`: 100% lines, statements, branches, and functions.
|
||||
- `packages/api`: 100% lines, statements, branches, and functions.
|
||||
- `pnpm check`: passed, including coverage gates.
|
||||
- `pnpm lint`: passed.
|
||||
|
||||
## Known Risks And Follow-Up
|
||||
|
||||
- `wasm-pack` is not installed. Current Rust verification uses `cargo check --workspace` and `cargo test --workspace`.
|
||||
- App packages are shell placeholders and currently have no behavioral tests.
|
||||
- Coverage gates currently apply to behavioral TypeScript packages:
|
||||
- `packages/core`
|
||||
- `packages/api`
|
||||
- `packages/adapters`
|
||||
- Future packages with behavior must add tests and coverage gates before being considered complete.
|
||||
@ -0,0 +1,47 @@
|
||||
# Knowledge Space CRUD API Skeleton
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added OpenAPI-backed KnowledgeSpace CRUD routes to the Hono gateway:
|
||||
- `POST /knowledge-spaces`
|
||||
- `GET /knowledge-spaces`
|
||||
- `GET /knowledge-spaces/{id}`
|
||||
- `PATCH /knowledge-spaces/{id}`
|
||||
- `DELETE /knowledge-spaces/{id}`
|
||||
- Added a bounded in-memory `KnowledgeSpaceRepository` skeleton for Sprint 2 API behavior before durable database execution lands.
|
||||
- Added tenant-scoped slug uniqueness, explicit list limits, stable slug cursor pagination, and repository capacity bounds.
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 2 starts moving the gateway from health/OpenAPI scaffolding toward real platform resources. KnowledgeSpace CRUD is the first tenant-scoped resource boundary and sets the shape for later database-backed handlers.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- RED: Gateway tests first referenced `createInMemoryKnowledgeSpaceRepository()` and the new CRUD routes before implementation.
|
||||
- GREEN: Added the repository skeleton, OpenAPI route schemas, and route handlers.
|
||||
- REFACTOR: Kept list operations bounded and tenant-scoped from the first implementation.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- List requests require explicit `tenantId` and bounded `limit`.
|
||||
- The in-memory skeleton enforces `maxSpaces` and `maxListLimit`.
|
||||
- Tenant slug lookup is linear only inside the bounded skeleton; durable database implementation should use the existing `knowledge_spaces_tenant_slug_uq` unique index.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`: passed.
|
||||
- `pnpm --filter @knowledge/api typecheck`: passed.
|
||||
- `pnpm --filter @knowledge/api test:coverage`: passed.
|
||||
- `pnpm check`: passed.
|
||||
- `pnpm build`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
- `pnpm wasm:build`: passed.
|
||||
- `pnpm compose:config`: passed.
|
||||
- `docker compose --profile apps config`: passed.
|
||||
- `git diff --check`: passed.
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- This is an in-memory API skeleton. The next durable step should replace repository persistence with database adapter execution backed by existing schema/index guarantees.
|
||||
- Auth middleware is still pending; this slice accepts `tenantId` in the request body/query until server-side subject attachment lands.
|
||||
@ -0,0 +1,45 @@
|
||||
# Local Development Environment Scaffold
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `compose.yaml` for local development services:
|
||||
- PostgreSQL with pgvector.
|
||||
- MinIO S3-compatible object storage.
|
||||
- Self-hosted Unstructured API.
|
||||
- Optional API and Admin app containers behind the `apps` profile.
|
||||
- Added `.env.example` with local-only defaults.
|
||||
- Added `infra/local/README.md` with local workflow commands and service notes.
|
||||
- Added root package scripts:
|
||||
- `pnpm compose:config`
|
||||
- `pnpm dev:infra`
|
||||
- `pnpm dev:stack`
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 1 includes a local development environment for PostgreSQL + pgvector, MinIO, Unstructured, and app startup. This scaffold gives developers a reproducible local stack while keeping runtime secrets in ignored `.env` files.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- This is environment configuration, not runtime behavior, so no RED unit test was required.
|
||||
- Validation used Docker Compose config rendering plus the normal repository verification suite.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- PostgreSQL uses the pgvector image so vector/FTS/database-as-search-engine work can be developed locally without adding another search service.
|
||||
- App containers depend on healthy infrastructure services, avoiding startup waterfalls caused by unavailable dependencies.
|
||||
- Persistent named volumes avoid repeated dependency/database bootstrap work between local runs.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm compose:config`: passed.
|
||||
- `docker compose --profile apps config`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `pnpm build`: passed.
|
||||
- `pnpm check`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- Container images are not digest-pinned yet; pinning should happen before production-like CI or release workflows.
|
||||
- The app containers still use in-memory adapter skeletons until real PostgreSQL/MinIO adapters are implemented.
|
||||
- MinIO bucket initialization is not automated yet.
|
||||
@ -0,0 +1,38 @@
|
||||
# MinIO Object Storage Integration Smoke
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added a live MinIO object-storage integration smoke test for the Node platform adapter.
|
||||
- Added root `pnpm test:minio` and package-level adapter `test:minio` scripts.
|
||||
- Documented the local command flow in `infra/local/README.md`.
|
||||
|
||||
## Why
|
||||
|
||||
The object-storage adapter now has fake-client contract coverage and runtime wiring. A separate live smoke test verifies the same adapter path against a real S3-compatible MinIO endpoint without making normal CI depend on local containers.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- RED: `pnpm test:minio` failed because no script existed.
|
||||
- GREEN: Added an explicit integration script and a guarded integration test that only runs live when `RUN_MINIO_INTEGRATION=1`.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- The smoke test uses a single bounded object and explicit list limit.
|
||||
- The live test is opt-in and is not part of default `pnpm check`, avoiding container startup overhead in normal CI.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/adapters test -- src/object-storage.integration.test.ts`: passed with the live test skipped when `RUN_MINIO_INTEGRATION` is unset.
|
||||
- `docker compose up -d minio minio-bootstrap`: not run successfully because the local Docker daemon is unavailable in this environment.
|
||||
- `pnpm check`: passed.
|
||||
- `pnpm build`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
- `pnpm wasm:build`: passed.
|
||||
- `pnpm compose:config`: passed.
|
||||
- `docker compose --profile apps config`: passed.
|
||||
- `git diff --check`: passed.
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- Run `docker compose up -d minio minio-bootstrap && pnpm test:minio` in an environment with Docker daemon available to exercise the live path.
|
||||
@ -0,0 +1,51 @@
|
||||
# 2026-05-08 Object Storage Adapter Contract
|
||||
|
||||
## Summary
|
||||
|
||||
- Expanded the core `ObjectStorageAdapter` contract beyond health checks.
|
||||
- Added a bounded in-memory object storage adapter for local/standalone tests and skeleton use.
|
||||
- Wired Cloudflare and Node platform adapter skeletons to the object storage implementation.
|
||||
- Added TDD coverage for object write/read/head/delete, bounded listing, pagination cursors, and max object size rejection.
|
||||
|
||||
## Files Added Or Updated
|
||||
|
||||
- `packages/core/src/platform-adapter.ts`
|
||||
- `packages/core/src/platform-adapter.test.ts`
|
||||
- `packages/adapters/src/object-storage.ts`
|
||||
- `packages/adapters/src/object-storage.test.ts`
|
||||
- `packages/adapters/src/cloudflare.ts`
|
||||
- `packages/adapters/src/node.ts`
|
||||
- `packages/adapters/src/index.ts`
|
||||
- `.harness/docs/TEMP-progress-document.md`
|
||||
- `.harness/changes/2026-05-08-object-storage-adapter-contract.md`
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 1 requires an object storage adapter before document upload and ingestion can safely store immutable document assets.
|
||||
|
||||
The implementation keeps performance constraints visible:
|
||||
|
||||
- Object writes enforce a configured `maxObjectBytes` cap.
|
||||
- Object listing requires an explicit positive limit.
|
||||
- Pagination uses stable key cursors.
|
||||
- Reads return byte copies so callers cannot mutate adapter-held state.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- RED: Added `packages/adapters/src/object-storage.test.ts`, then ran `pnpm --filter @knowledge/adapters test`.
|
||||
- The test failed because `./object-storage` did not exist.
|
||||
- GREEN: Added the object storage contract and memory adapter implementation.
|
||||
- REFACTOR: Fixed strict optional typing, added unbounded-list rejection coverage, and ran formatting.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/adapters test`: passed.
|
||||
- `pnpm --filter @knowledge/adapters test:coverage`: passed.
|
||||
- `packages/adapters`: 100% lines, statements, branches, and functions.
|
||||
- `pnpm --filter @knowledge/core test:coverage`: passed.
|
||||
- `packages/core`: 100% lines, statements, branches, and functions.
|
||||
|
||||
## Known Risks And Follow-Up
|
||||
|
||||
- This is a bounded memory adapter and contract skeleton, not the final R2/S3 client implementation.
|
||||
- Future R2/S3 adapters must preserve the same bounded listing semantics and avoid loading unbounded object lists into memory.
|
||||
@ -0,0 +1,49 @@
|
||||
# Object Storage Runtime Wiring
|
||||
|
||||
## What Changed
|
||||
|
||||
- Extended the Node platform factory with runtime object storage configuration:
|
||||
- Complete `MINIO_ENDPOINT`, `MINIO_BUCKET`, `MINIO_ACCESS_KEY`, and `MINIO_SECRET_KEY` env selects the S3-compatible adapter.
|
||||
- Missing MinIO configuration keeps the bounded memory adapter fallback.
|
||||
- Optional injected S3 client supports tests without real network calls.
|
||||
- Extended the Cloudflare platform factory with R2-compatible configuration:
|
||||
- Complete `R2_ACCOUNT_ID`, `R2_BUCKET`, `R2_ACCESS_KEY_ID`, and `R2_SECRET_ACCESS_KEY` env selects the R2 S3-compatible adapter.
|
||||
- Missing R2 configuration keeps the bounded memory adapter fallback.
|
||||
- Added a one-shot `minio-bootstrap` compose service that creates `${MINIO_BUCKET:-knowledge-fs}`.
|
||||
- Updated the API compose dependency so app startup waits for `minio-bootstrap` to complete successfully.
|
||||
- Updated `.env.example`, `pnpm dev:infra`, and `infra/local/README.md` for MinIO/R2 runtime wiring.
|
||||
- Made API gateway tests pass an explicit empty env so unit tests remain independent from developer machine object-storage credentials.
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 1 needs the S3-compatible object storage adapter to be reachable from real platform factories while preserving simple local and CI defaults. This keeps no-credential environments on memory storage, but lets the local compose API switch to MinIO when the provided env is present.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- RED: Added platform factory tests proving complete MinIO/R2 env should route object storage through an injected fake S3 client.
|
||||
- The first test run failed because both factories still returned memory-backed object storage.
|
||||
- GREEN: Wired Node and Cloudflare factories to create configured S3 clients when env is complete.
|
||||
- REFACTOR: Kept incomplete-env fallback tests explicit and isolated gateway tests from ambient shell env.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- Runtime wiring preserves the existing S3 adapter guardrails: bounded object size, explicit list limits, continuation cursors, and byte-copy isolation.
|
||||
- Factory tests use injected fake clients, so they do not perform network calls or depend on external services.
|
||||
- The API now waits for bucket bootstrap completion, avoiding startup retries or repeated failed object-storage calls caused by a missing bucket.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/adapters test -- src/adapters.test.ts`: passed.
|
||||
- `pnpm check`: passed.
|
||||
- `packages/adapters`: 96.82% lines/statements, 96.12% branches, 100% functions.
|
||||
- `pnpm build`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
- `pnpm compose:config`: passed.
|
||||
- `docker compose --profile apps config`: passed.
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- This slice validates compose rendering but does not start a live MinIO container.
|
||||
- Live MinIO integration tests should be added once the project is ready to depend on local container startup in CI.
|
||||
- Cloudflare Workers binding-specific configuration can be refined later; this slice uses explicit R2 S3-compatible env keys.
|
||||
@ -0,0 +1,72 @@
|
||||
# 2026-05-08 Performance Requirements And Database Schema
|
||||
|
||||
## Summary
|
||||
|
||||
- Recorded the user's high-performance requirements under `.harness/agents`.
|
||||
- Added a new `@knowledge/database` package.
|
||||
- Added a database schema catalog for PostgreSQL and TiDB.
|
||||
- Added SQL renderers for table and index scaffolding.
|
||||
- Added performance guard tests that ensure required high-traffic query indexes exist.
|
||||
|
||||
## Files Added Or Updated
|
||||
|
||||
- `.harness/agents/development-requirements.md`
|
||||
- `.harness/docs/TEMP-task-document.md`
|
||||
- `.harness/docs/TEMP-progress-document.md`
|
||||
- `.harness/changes/2026-05-08-performance-requirements-and-database-schema.md`
|
||||
- `packages/database/package.json`
|
||||
- `packages/database/tsconfig.json`
|
||||
- `packages/database/vitest.config.ts`
|
||||
- `packages/database/src/index.ts`
|
||||
- `packages/database/src/schema.ts`
|
||||
- `packages/database/src/schema.test.ts`
|
||||
- `pnpm-lock.yaml`
|
||||
|
||||
## Why
|
||||
|
||||
The project has extremely high performance requirements. Database-facing development must avoid N+1 queries, repeated database round-trips, unbounded loads, memory leaks, and missing indexes.
|
||||
|
||||
The schema catalog makes table structure and required access-pattern indexes auditable before runtime query code is introduced.
|
||||
|
||||
## Performance Guardrails Added
|
||||
|
||||
- Tenant/space resolution indexes.
|
||||
- Source listing by space/status.
|
||||
- Document asset listing by space/source/version.
|
||||
- Ingestion status listing by space/status/created time.
|
||||
- Document deduplication by space/hash/version.
|
||||
- Parse artifact lookup by asset/version and artifact hash.
|
||||
- Knowledge node batch loading by space/asset/kind.
|
||||
- Knowledge node source-order lookup by artifact/offset.
|
||||
- Permission-scope retrieval filtering index.
|
||||
- Projection lookup by space/type/status and node/type/version.
|
||||
- KnowledgeFS virtual path unique lookup.
|
||||
- Evidence bundle lookup by trace and answerability state.
|
||||
- Answer trace listing by space/created time.
|
||||
- Trace-step loading by trace/started time.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- Added schema catalog tests before relying on the database package in other code.
|
||||
- Added a failing test for nullable optional domain relationships:
|
||||
- `source_id`
|
||||
- `trace_id`
|
||||
- `evidence_bundle_id`
|
||||
- Fixed schema rendering so optional relationships are nullable.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/database test`: passed.
|
||||
- `pnpm --filter @knowledge/database test:coverage`: passed.
|
||||
- `packages/database`: 100% lines, statements, branches, and functions.
|
||||
- `pnpm install`: passed.
|
||||
- `pnpm check`: passed.
|
||||
- `pnpm build`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
|
||||
## Known Risks And Follow-Up
|
||||
|
||||
- This is schema and migration rendering scaffolding, not a live database adapter yet.
|
||||
- The next database step should decide whether to layer Drizzle definitions on top of this catalog or keep generated SQL as the migration source.
|
||||
- Runtime repository methods must use batched access and pagination; this package currently prevents missing index regressions but does not execute queries.
|
||||
@ -0,0 +1,51 @@
|
||||
# Review Fix Performance Guardrails
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added bounded terminal retention for the inline job queue.
|
||||
- Added cumulative job stats that do not require retaining every completed or failed job.
|
||||
- Added memory object storage bounds for object count and total retained bytes.
|
||||
- Added metadata clone isolation for memory object storage.
|
||||
- Added S3 read-size enforcement for externally oversized objects and streamed bodies.
|
||||
- Added primary-key tie-breakers to non-unique keyset pagination indexes.
|
||||
- Added database planner validation requiring unique ordering or an `id` tie-breaker.
|
||||
- Recorded the 10-commit project health review cadence in `.harness/agents/development-requirements.md`.
|
||||
|
||||
## Why
|
||||
|
||||
The review at checkpoint commit `9c6714f` found several long-running performance risks: terminal jobs could accumulate without bound, S3 reads could buffer external large objects, memory object storage could grow without count or total-byte limits, and keyset pagination could skip or repeat rows when ordered by non-unique columns.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- RED: Added failing inline job queue tests for terminal retention, idempotency cleanup, and invalid retention config.
|
||||
- RED: Added failing object storage tests for metadata isolation, memory object/byte bounds, and oversized S3 reads.
|
||||
- RED: Added failing database planner and schema tests for `id` tie-breaker requirements.
|
||||
- GREEN: Implemented bounded behavior and planner/schema guardrails.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- Inline job queues now bound retained terminal history while preserving cumulative stats.
|
||||
- Memory object storage now has bounded object and byte budgets.
|
||||
- S3-compatible reads reject oversized objects before or during buffering.
|
||||
- Database pagination now requires deterministic cursor ordering backed by declared indexes.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/adapters test -- src/job-queue.test.ts src/object-storage.test.ts src/database.test.ts`: passed.
|
||||
- `pnpm --filter @knowledge/database test -- src/schema.test.ts src/migration-file.test.ts`: passed.
|
||||
- `pnpm db:migrations:write`: passed.
|
||||
- `pnpm db:migrations:check`: passed.
|
||||
- `pnpm check`: passed.
|
||||
- `pnpm build`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
- `pnpm wasm:build`: passed.
|
||||
- `pnpm compose:config`: passed.
|
||||
- `docker compose --profile apps config`: passed.
|
||||
- `git diff --check`: passed.
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- Inline queue retention is an in-memory skeleton; durable queue backends still need real platform implementations.
|
||||
- Memory object storage remains a bounded fallback and should not replace MinIO/R2 for real deployments.
|
||||
- The next iteration should add a live MinIO integration smoke test.
|
||||
@ -0,0 +1,50 @@
|
||||
# S3-Compatible Object Storage Adapter
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `createS3ObjectStorageAdapter` in `@knowledge/adapters`.
|
||||
- Added `@aws-sdk/client-s3` to the adapters package.
|
||||
- Implemented the full `ObjectStorageAdapter` contract on top of S3-compatible commands:
|
||||
- `PutObjectCommand`
|
||||
- `GetObjectCommand`
|
||||
- `HeadObjectCommand`
|
||||
- `DeleteObjectCommand`
|
||||
- `ListObjectsV2Command`
|
||||
- `HeadBucketCommand`
|
||||
- Added support for injected S3 clients so tests and future runtime wiring can provide MinIO, R2, or custom S3-compatible clients.
|
||||
- Kept the existing memory object storage adapter unchanged for current skeleton defaults.
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 1 requires a real object storage adapter surface for R2/MinIO-compatible deployments. The project can now target MinIO and Cloudflare R2 through the same S3-compatible adapter while keeping local tests independent from a running object storage service.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- RED: Extended `packages/adapters/src/object-storage.test.ts` to reference `createS3ObjectStorageAdapter`.
|
||||
- The first run failed because the S3 adapter factory did not exist.
|
||||
- GREEN: Added the S3-compatible implementation and SDK dependency.
|
||||
- REFACTOR: Added body/error/default mapping tests to keep adapter coverage above the project threshold.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- `putObject` rejects payloads above `maxObjectBytes` before sending an S3 command.
|
||||
- `listObjects` requires an explicit positive `limit` and uses S3 continuation cursors.
|
||||
- `getObject` returns copied bytes so callers cannot mutate adapter-returned buffers.
|
||||
- No real network integration test was added in this slice, avoiding external-service dependency in CI.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/adapters test -- src/object-storage.test.ts`: passed.
|
||||
- `pnpm --filter @knowledge/adapters test:coverage`: passed.
|
||||
- `packages/adapters`: 98.8% lines/statements, 95.07% branches, 100% functions.
|
||||
- `pnpm --filter @knowledge/adapters typecheck`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `pnpm check`: passed.
|
||||
- `pnpm build`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- Real MinIO/R2 runtime wiring is still pending.
|
||||
- MinIO bucket bootstrap is still pending.
|
||||
- Integration tests against a live MinIO container should be added once local service wiring is in place.
|
||||
@ -0,0 +1,47 @@
|
||||
# Schema Database Adapter Contract
|
||||
|
||||
## What Changed
|
||||
|
||||
- Expanded `DatabaseAdapter` from a health-only placeholder into an explicit schema contract:
|
||||
- `dialect`
|
||||
- `getSchemaSummary()`
|
||||
- `renderMigrationSql()`
|
||||
- `checkPerformanceIndexes()`
|
||||
- Added `createSchemaDatabaseAdapter` in `@knowledge/adapters`.
|
||||
- Wired Node/Docker skeletons to PostgreSQL schema behavior.
|
||||
- Wired Cloudflare skeletons to TiDB schema behavior.
|
||||
- Added adapter tests for PostgreSQL/TiDB migration rendering, performance index checks, and schema summary clone isolation.
|
||||
- Added `@knowledge/database` as an explicit adapter package dependency.
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 1 requires the platform adapter layer to expose meaningful database behavior before real query/runtime adapters are introduced. This keeps the current skeleton honest: it still does not open database connections, but it now advertises the actual schema, dialect-specific migration SQL, and required performance indexes through one shared contract.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- RED: Added `packages/adapters/src/database.test.ts`, then ran `pnpm --filter @knowledge/adapters test`.
|
||||
- The first failure confirmed `./database` did not exist.
|
||||
- GREEN: Added the schema-backed database adapter and wired platform skeletons to it.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- The adapter exposes `checkPerformanceIndexes()` so high-traffic access-pattern indexes stay visible through the platform layer.
|
||||
- Schema summaries are cloned before returning to callers, preventing accidental retained-state mutation.
|
||||
- This slice introduces no runtime database queries, so there are no new query round-trips or N+1 paths.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/adapters test`: passed.
|
||||
- `pnpm --filter @knowledge/adapters test:coverage`: passed.
|
||||
- `packages/adapters`: 97.95% lines/statements, 95.2% branches, 100% functions.
|
||||
- `pnpm --filter @knowledge/adapters typecheck`: passed.
|
||||
- `pnpm --filter @knowledge/core typecheck`: passed.
|
||||
- `pnpm check`: passed.
|
||||
- `pnpm build`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- This is still a schema-backed skeleton, not a live PostgreSQL/TiDB connection adapter.
|
||||
- The next database slice should define bounded query execution/list contracts before adding real Drizzle or SQL client integration.
|
||||
@ -0,0 +1,51 @@
|
||||
# Database-Backed KnowledgeSpace Repository
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added a minimal `DatabaseAdapter.execute(input)` contract with bounded row execution metadata.
|
||||
- Added injected executor support to the schema database adapter while preserving schema, migration, planner, capability, and health behavior.
|
||||
- Added `createDatabaseKnowledgeSpaceRepository()` for tenant-scoped KnowledgeSpace CRUD through parameterized SQL.
|
||||
- Kept the gateway default on the bounded in-memory repository until runtime database driver wiring is added.
|
||||
|
||||
## Why
|
||||
|
||||
KnowledgeSpace CRUD had authenticated tenant scope but still only had in-memory persistence. This slice adds the execution boundary needed to move CRUD toward real PostgreSQL/TiDB-backed storage without introducing live driver dependencies yet.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- RED: Adapter tests first asserted `database.execute()` behavior before it existed.
|
||||
- RED: Gateway tests first referenced `createDatabaseKnowledgeSpaceRepository()` before implementation.
|
||||
- GREEN: Added core execution types, schema adapter executor injection, and the database-backed repository.
|
||||
- REFACTOR: Kept SQL parameterized, read execution bounded by `maxRows`, and tenant filters on every id-based operation.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- All database reads use explicit `maxRows`.
|
||||
- KnowledgeSpace list uses tenant-scoped slug keyset pagination and reads only `limit + 1`.
|
||||
- Database repository operations use parameter arrays rather than string-interpolating user input.
|
||||
- Cross-tenant get/update/delete paths filter by `tenant_id` and return not-found semantics.
|
||||
|
||||
## Verification
|
||||
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/adapters test -- src/database.test.ts`
|
||||
- `pnpm --filter @knowledge/adapters test:coverage`
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm --filter @knowledge/core typecheck`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm --filter @knowledge/adapters typecheck`
|
||||
- `pnpm lint`
|
||||
- Full verification passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- No real PostgreSQL/TiDB driver is wired in this slice; executors are injected for tests and future runtime wiring.
|
||||
- TiDB non-returning write behavior is covered through a follow-up read, but live driver integration still needs its own smoke/integration tests.
|
||||
@ -0,0 +1,46 @@
|
||||
# Document Upload API
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added authenticated `POST /knowledge-spaces/{id}/documents` upload support.
|
||||
- Added bounded in-memory and database-backed `DocumentAssetRepository` implementations.
|
||||
- Stored uploaded bytes through the platform object-storage adapter with tenant/space/document object-key isolation.
|
||||
- Added OpenAPI coverage for the multipart upload route.
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 2 needs the first document ingestion boundary on top of authenticated tenant-scoped KnowledgeSpace CRUD. This slice persists uploaded objects and creates pending `DocumentAsset` records without starting parsing, queueing, or external provider work.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- RED: Gateway tests first referenced the upload route and document-asset repositories before implementation.
|
||||
- GREEN: Added upload request validation, object storage writes, SHA-256 calculation, pending asset creation, and database SQL wiring.
|
||||
- REFACTOR: Kept upload buffering bounded by `maxUploadBytes` and added cleanup for object writes when asset persistence fails.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- Uploads are size checked before buffering; the current `Uint8Array` object-storage contract remains bounded by `maxUploadBytes`.
|
||||
- The route performs one tenant-scoped KnowledgeSpace lookup before upload and one document-asset insert after object storage.
|
||||
- Database-backed asset creation uses parameterized SQL and explicit `maxRows`.
|
||||
- Default in-memory asset storage is capped by `maxAssets` to avoid unbounded retention.
|
||||
|
||||
## Verification
|
||||
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- Full verification passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- This slice supports a single file per request only; batch upload and version overwrite behavior are deferred.
|
||||
- Parser dispatch, job enqueueing, and Unstructured/native parser integration are deferred to ingestion iterations.
|
||||
- Real production database driver wiring is still a separate runtime integration slice.
|
||||
@ -0,0 +1,50 @@
|
||||
# Parser Adapter Contracts
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added the `@knowledge/parsers` package.
|
||||
- Added shared parser contracts for `ParserAdapter` and `ParseDocumentInput`.
|
||||
- Added native Markdown and HTML parsers that emit existing `ParseArtifact` / `ParseElement` core models.
|
||||
- Added an Unstructured API client skeleton using `fetch` and the legacy `/general/v0/general` partition endpoint.
|
||||
- Added a parser router that prefers native Markdown/HTML and falls back to Unstructured for complex or unknown document types.
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 2 needs parser boundaries before the synchronous MVP ingestion path can connect document upload to parse artifact persistence. This slice establishes parser contracts and parser selection without coupling parsing into the upload route yet.
|
||||
|
||||
## TDD Notes
|
||||
|
||||
- RED: Added `packages/parsers/src/parser.test.ts` before `src/index.ts` existed and confirmed the suite failed on missing parser factories.
|
||||
- GREEN: Implemented Markdown, HTML, Unstructured, and router behavior until the package tests passed.
|
||||
- REFACTOR: Added coverage for parser bounds, empty native elements, Unstructured response failures, and response-size guardrails.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- Native parser input is bounded by `maxInputBytes`, defaulting to `10 MiB`.
|
||||
- Parser output is bounded by `maxElements`, defaulting to `20,000`.
|
||||
- Unstructured response reads are bounded by `maxResponseBytes`, defaulting to `5 MiB`, with both `content-length` precheck and post-read byte validation.
|
||||
- Parser routing avoids external Unstructured calls for Markdown, plaintext, HTML, and XHTML documents.
|
||||
|
||||
## Verification
|
||||
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/parsers test`
|
||||
- `pnpm --filter @knowledge/parsers test:coverage`
|
||||
- `pnpm --filter @knowledge/parsers build`
|
||||
- `pnpm lint`
|
||||
- Full verification passed:
|
||||
- `pnpm install`
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- This slice does not persist parse artifacts or update document parser status.
|
||||
- This slice does not connect parser dispatch to document upload; that belongs to the synchronous MVP ingestion iteration.
|
||||
- Unstructured integration is fake-fetch tested only; live container smoke remains a future integration test.
|
||||
@ -0,0 +1,30 @@
|
||||
# Synchronous Document Ingestion
|
||||
|
||||
## Summary
|
||||
|
||||
- Connected the authenticated document upload route to parser execution and parse artifact persistence.
|
||||
- Added bounded in-memory and database-backed `ParseArtifactRepository` implementations.
|
||||
- Extended `DocumentAssetRepository` with scoped asset lookup and parser status updates.
|
||||
|
||||
## Behavior
|
||||
|
||||
- Successful uploads now parse the uploaded bytes directly, persist a `ParseArtifact`, update the asset to `parsed`, and return the updated `DocumentAsset`.
|
||||
- Parser or artifact persistence failures keep the raw uploaded object, best-effort mark the asset as `failed`, and return `Document parsing failed`.
|
||||
- Asset persistence failures still delete the just-uploaded object to avoid orphaned storage.
|
||||
- The default parser supports native Markdown/HTML and fails closed for complex formats when Unstructured is not configured.
|
||||
|
||||
## Performance And Safety
|
||||
|
||||
- The ingestion path reuses the already buffered upload bytes and does not read the object back from object storage.
|
||||
- Database repositories use parameterized SQL with explicit `maxRows`.
|
||||
- Document status updates are scoped by `id` and `knowledge_space_id`.
|
||||
- In-memory parse artifact storage is bounded and clone-isolated.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED confirmed with failing gateway tests for missing parse artifact repository and document status update support.
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api build`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
|
||||
@ -0,0 +1,40 @@
|
||||
# 10-Commit Health Review After `0105450`
|
||||
|
||||
## What Changed
|
||||
|
||||
- Completed the required project health review after the 10th implementation commit following checkpoint `0105450`.
|
||||
- Reviewed implementation commits from `326371a` through `292d012`, covering Sprint 3 chunking/indexing/retrieval and the initial Sprint 4 KnowledgeFS resource model.
|
||||
- Updated the temporary progress document so the next implementation count starts from checkpoint `292d012`.
|
||||
|
||||
## Findings
|
||||
|
||||
- Technical direction remains aligned with `.harness` architecture: Rust is still limited to pure compute, TypeScript owns IO/orchestration/repositories, and database/search paths stay behind adapter/repository boundaries.
|
||||
- Performance guardrails remain healthy:
|
||||
- KnowledgeNode and KnowledgePath writes are batched or single-row writes without per-item read waterfalls.
|
||||
- Retrieval runs dense and FTS searches in parallel and joins citation metadata in-query, avoiding post-retrieval N+1 lookups.
|
||||
- New list paths use explicit limits, `limit + 1`, stable keyset cursors, and catalog-backed indexes.
|
||||
- Embedding providers and WASM compute paths enforce bounded inputs/outputs.
|
||||
- Test and coverage health remains green. The latest full verification passed with coverage gates above 90%.
|
||||
- `.harness/changes` traceability is complete for each reviewed implementation slice.
|
||||
- No high-priority defects were found that require remediation before continuing.
|
||||
|
||||
## Verification
|
||||
|
||||
- Reviewed latest successful verification from `292d012`:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
- Additional review scans:
|
||||
- `git log --oneline 0105450..HEAD`
|
||||
- `rg` scans for TODO/FIXME/unbounded/N+1 markers and new indexed access paths.
|
||||
|
||||
## Known Risks And Follow-Up
|
||||
|
||||
- Live service integration is still limited by local Docker availability; compose rendering passes, but live MinIO/PostgreSQL smoke should run in an environment with Docker daemon access.
|
||||
- Basic RRF remains TypeScript MVP logic; later Phase 2 retrieval hardening can move fusion/evidence packing deeper into WASM as planned.
|
||||
- No further feature implementation should be counted against checkpoint `0105450`; the next review cadence starts from `292d012`.
|
||||
@ -0,0 +1,48 @@
|
||||
# 10-Commit Health Review: 50d3a26 to 32ed484
|
||||
|
||||
## What Changed
|
||||
|
||||
- Completed the required 10-implementation-commit health review after checkpoint `50d3a26`.
|
||||
- Reviewed implementation commits:
|
||||
- `fde38b3` Add evidence prompt templates
|
||||
- `c790a69` Add SSE query streaming
|
||||
- `d59cb91` Add generation cost tracking
|
||||
- `2f5db59` Add citation normalization
|
||||
- `980d49b` Add generation cache and skip path
|
||||
- `c313400` Add KnowledgeFS grep endpoint
|
||||
- `59d4133` Add KnowledgeFS find endpoint
|
||||
- `5e0a47a` Add WASM text diff
|
||||
- `c2adeb5` Add KnowledgeFS diff and open_node
|
||||
- `32ed484` Harden CommandRegistry guardrails
|
||||
- Noted that `e6adbfe` was a review-remediation commit and remains excluded from the feature implementation count according to the previous review note.
|
||||
|
||||
## Findings
|
||||
|
||||
- No high-priority defects or technical-direction drift were found.
|
||||
- TypeScript still owns orchestration, IO, HTTP, CommandRegistry, KnowledgeFS, generation, and cache behavior.
|
||||
- Rust remains limited to pure WASM compute for text diff and existing compute primitives.
|
||||
- Performance-sensitive paths remain bounded:
|
||||
- KnowledgeFS grep batches node hydration with `getMany`.
|
||||
- KnowledgeFS find is path-scoped and explicitly limited.
|
||||
- KnowledgeFS diff reuses bounded content reads and the WASM diff guardrails.
|
||||
- `open_node` uses one tenant-scoped node lookup.
|
||||
- CommandRegistry validates cost estimates before handlers run.
|
||||
- Generation cache keys remain versioned and avoid raw query/evidence text.
|
||||
- Test and coverage health passed. API branch coverage is currently 90.04%, which passes but leaves little margin for the next API-heavy slice.
|
||||
|
||||
## Verification
|
||||
|
||||
- Latest full verification before this review passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Follow-Up
|
||||
|
||||
- Continue Sprint 8 with SourceFS mount inspection tools.
|
||||
- Add enough branch coverage in the next API-heavy slice to keep a healthier buffer above the 90% API branch threshold.
|
||||
@ -0,0 +1,32 @@
|
||||
# 10-Commit Health Review: f950b59 to 50d3a26
|
||||
|
||||
## Scope
|
||||
|
||||
- Reviewed implementation commits after checkpoint `f950b59` through checkpoint `50d3a26`.
|
||||
- Covered answerability, permission filtering, metadata filters, EvidenceBundle caching, AnswerTrace recording, trace API, LLM provider contracts, LLM routing, WASM evidence packing, and context-window packing.
|
||||
|
||||
## Findings
|
||||
|
||||
- Found one performance issue in the LLM provider boundary: provider responses were read with `response.text()` before enforcing `maxResponseBytes`.
|
||||
- No high-priority tenant isolation, N+1, repeated database query, missing explicit read limit, cache key leakage, or WASM pure-compute drift was found in the reviewed range.
|
||||
|
||||
## Remediation
|
||||
|
||||
- Fixed the LLM provider response reader to stream and cancel oversized responses immediately.
|
||||
- Added a regression test for streaming oversized provider responses.
|
||||
|
||||
## Technical Direction
|
||||
|
||||
- The architecture remains aligned with `.harness`: TypeScript owns API/orchestration, Rust/WASM stays pure compute, and database-facing access remains repository-bound.
|
||||
- Retrieval and trace database paths continue to use parameterized SQL, explicit `maxRows`, and bounded fanout.
|
||||
- LLM routing and context-window packing remain provider-agnostic and avoid network/database work during route selection and budget calculation.
|
||||
|
||||
## Test And CI Health
|
||||
|
||||
- Focused generation tests passed after remediation.
|
||||
- Full workspace verification passed with `pnpm check`, `pnpm build`, `pnpm lint`, `cargo test --workspace`, `pnpm wasm:build`, `pnpm compose:config`, `docker compose --profile apps config`, and `git diff --check`.
|
||||
|
||||
## Residual Risks
|
||||
|
||||
- Live provider streaming over real network connections still needs integration coverage when production LLM runtime wiring lands.
|
||||
- Context packing currently uses approximate WASM token counting and should be revisited when model-specific tokenizer support is introduced.
|
||||
@ -0,0 +1,61 @@
|
||||
# 10-Commit Health Review After Checkpoint 32ed484
|
||||
|
||||
## Review Scope
|
||||
|
||||
- Reviewed implementation commits after checkpoint `32ed484`:
|
||||
- `8669890` Add SourceFS mount inspection tools
|
||||
- `d62c683` Add safe shell planner executor
|
||||
- `e9151c3` Complete MCP KnowledgeFS tools
|
||||
- `7fd9a5a` Add MCP retrieval and shell tools
|
||||
- `5c8ff39` Add gateway rate limiting
|
||||
- `a66bf24` Add provider degradation flags
|
||||
- `9074e7e` Add component health endpoint
|
||||
- `0e20ade` Initialize Admin Console shell
|
||||
- `460fb33` Add Admin shared API client
|
||||
- `7f8c688` Define Admin UI BFF constraints
|
||||
- Reviewed follow-up remediation commit:
|
||||
- `92f4e22` Bound Admin SSE response reads
|
||||
|
||||
## Findings
|
||||
|
||||
- Found one actionable performance issue:
|
||||
- Admin `streamQuery()` buffered full SSE responses with `response.text()`, which could retain unbounded generation output in memory.
|
||||
|
||||
## Fixes Applied
|
||||
|
||||
- Added `maxSseBytes` to the Admin shared API client.
|
||||
- Replaced full-response SSE buffering with chunked bounded reads.
|
||||
- Added TDD coverage for oversized SSE response rejection.
|
||||
|
||||
## Health Check Result
|
||||
|
||||
- Technical direction remains aligned:
|
||||
- Hono owns platform and core runtime behavior.
|
||||
- Next.js Admin remains a UI shell with a thin BFF proxy only.
|
||||
- Safe shell execution remains CommandRegistry-bound rather than host-shell execution.
|
||||
- Performance guardrails are healthy after remediation:
|
||||
- New SourceFS, MCP, rate limiting, and BFF paths use explicit limits or bounded forwarding.
|
||||
- Admin SSE reads now have an explicit byte cap.
|
||||
- No new N+1 database access path was introduced in this review window.
|
||||
- Test and CI health remain healthy:
|
||||
- TDD red/green was recorded for the remediation.
|
||||
- Full local verification passed after the fix.
|
||||
- Traceability is complete:
|
||||
- Every implementation slice in the review window has a `.harness/changes` record.
|
||||
- The remediation has a dedicated `.harness/changes` record.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Next Cadence
|
||||
|
||||
- Latest reviewed code checkpoint: `92f4e22`.
|
||||
- The next mandatory project health review is due after 10 new implementation commits following `92f4e22`.
|
||||
@ -0,0 +1,52 @@
|
||||
# 10-Commit Health Review: b7ac774
|
||||
|
||||
## Scope
|
||||
|
||||
- Review checkpoint: `b7ac774`.
|
||||
- Reviewed implementation commits after checkpoint `292d012`:
|
||||
- `3e9594e` Add filesystem path namespaces
|
||||
- `b473f65` Add ResourceMount model
|
||||
- `7a1b8b6` Add CommandRegistry contract
|
||||
- `c460dee` Add KnowledgeFS ls and tree endpoints
|
||||
- `e4107b8` Add KnowledgeFS cat and stat endpoints
|
||||
- `a11c529` Add MCP server skeleton
|
||||
- `227ac78` Add golden question CRUD
|
||||
- `ccf36f2` Add retrieval evaluation MVP
|
||||
- `fd730d8` Add phase 1 end-to-end integration test
|
||||
- `b7ac774` Add standalone API Docker image
|
||||
|
||||
## Findings
|
||||
|
||||
- No high-priority correctness, architecture, performance, or test-coverage issues were found.
|
||||
- Technical direction remains aligned with `.harness`: TypeScript owns orchestration, Hono owns Gateway/MCP/OpenAPI behavior, and Rust remains limited to pure WASM compute.
|
||||
- KnowledgeFS route implementations use explicit limits, stable cursor pagination, and database indexes on `(knowledge_space_id, view_type, view_name, virtual_path, id)`.
|
||||
- Golden question CRUD and retrieval evaluation paths are tenant/space-scoped and bounded by explicit list limits, `maxQuestions`, and `maxTopK`.
|
||||
- Retrieval evaluation batches embeddings for golden questions, avoiding per-question embedding N+1 calls.
|
||||
- MCP tools are schema-bounded and dispatch through existing KnowledgeFS/search handler boundaries.
|
||||
- Standalone Docker work adds a real Hono server entrypoint without moving business logic out of the gateway.
|
||||
|
||||
## Verification Reviewed
|
||||
|
||||
- Recent slices recorded and passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
- Additional review smoke:
|
||||
- `PORT=8799 pnpm --filter @knowledge/api-app start`
|
||||
- `curl http://127.0.0.1:8799/health`
|
||||
|
||||
## Residual Risks
|
||||
|
||||
- Docker daemon is unavailable in the current environment, so live `pnpm docker:api:build` and container startup remain unverified here.
|
||||
- API branch coverage is passing but close to the floor at about 90.06%; future API slices should add enough branch coverage to create more margin.
|
||||
- The standalone image currently runs TypeScript through `tsx`; a later production packaging pass should emit or bundle runtime artifacts into a slimmer image.
|
||||
|
||||
## Follow-Up
|
||||
|
||||
- Next implementation cadence starts from checkpoint `b7ac774`.
|
||||
- Next feature work should enter Phase 2 Sprint 5 hybrid retrieval hardening.
|
||||
@ -0,0 +1,69 @@
|
||||
# 10-Commit Health Review After f73b3e2
|
||||
|
||||
## Summary
|
||||
|
||||
- Completed the mandatory health review after 10 implementation commits following checkpoint `92f4e22`.
|
||||
- Review checkpoint commit: `f73b3e2 Add Cloudflare job queue adapter`.
|
||||
- Found and remediated Cloudflare job delivery reliability issues before continuing feature iteration.
|
||||
|
||||
## Review Scope
|
||||
|
||||
- Technical direction against `.harness` architecture.
|
||||
- Performance and reliability guardrails: unbounded memory, N+1/repeated database access, queue delivery loss, large payloads, and cache/key versioning.
|
||||
- TDD and package coverage health.
|
||||
- CI/build/lint/test health.
|
||||
- `.harness/changes` and temporary progress document completeness.
|
||||
|
||||
## Findings
|
||||
|
||||
- **High priority: Cloudflare retry did not re-deliver jobs.**
|
||||
- `createCloudflareJobQueueAdapter().retry()` requeued state but did not send a new Cloudflare Queue message.
|
||||
- Impact: retried jobs could remain queued in state without a delivery event.
|
||||
- Fix: retry now persists the queued state and sends a compact Queue message with delay derived from `runAfter`.
|
||||
- **High priority: Cloudflare enqueue delivery failure left orphan queued state.**
|
||||
- `enqueue()` persisted queued state before `queue.send()`. If `send()` failed, the job remained queued even though no Queue message existed.
|
||||
- Impact: job could be stuck until manual intervention.
|
||||
- Fix: enqueue now cancels the job, persists terminal state, clears local idempotency mapping, and rethrows the delivery error.
|
||||
|
||||
## Health Assessment
|
||||
|
||||
- Architecture remains aligned:
|
||||
- Hono/API orchestration stays TypeScript-first.
|
||||
- Cloudflare-specific behavior remains behind adapter contracts.
|
||||
- Rust remains pure compute only.
|
||||
- Next.js remains Admin UI/BFF only.
|
||||
- Performance posture remains acceptable:
|
||||
- Job queue operations retain explicit bounds for batch size, lease duration, active queue size, and terminal retention.
|
||||
- Queue messages carry job identifiers and type metadata only; raw document payloads are not sent.
|
||||
- No new database query paths or N+1 risks were introduced in this cadence.
|
||||
- Test posture remains acceptable:
|
||||
- New behavior was added RED first.
|
||||
- Coverage gates remain above 90%.
|
||||
- Review remediation added regression tests for retry re-delivery and enqueue delivery failure.
|
||||
- Traceability is complete:
|
||||
- Each implementation slice has a `.harness/changes` record.
|
||||
- `TEMP-progress-document.md` records commit counts and verification.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/adapters test -- src/adapters.test.ts` failed because retry did not send another Queue message and delivery failure left queued state.
|
||||
- Focused remediation verification:
|
||||
- `pnpm --filter @knowledge/adapters test -- src/adapters.test.ts`
|
||||
- `pnpm --filter @knowledge/adapters typecheck`
|
||||
- `pnpm --filter @knowledge/adapters test:coverage`
|
||||
- `pnpm lint`
|
||||
- Full remediation verification:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Next Cadence
|
||||
|
||||
- After the remediation commit, the latest reviewed checkpoint becomes that remediation commit.
|
||||
- The next 10 implementation commits after the remediation checkpoint must pause for another health review before feature iteration continues.
|
||||
@ -0,0 +1,58 @@
|
||||
# 10-Commit Health Review: f950b59
|
||||
|
||||
## Scope
|
||||
|
||||
- Reviewed checkpoint: `f950b59`.
|
||||
- Previous checkpoint: `b7ac774`.
|
||||
- Reviewed implementation commits:
|
||||
- `93e192e` Add mixed-language FTS normalization
|
||||
- `c4a3509` Add WASM RRF fusion
|
||||
- `3441e8b` Add retrieval planner mode router
|
||||
- `891873b` Optimize hybrid recall planning
|
||||
- `aea05e4` Add reranker provider interface
|
||||
- `cbd9562` Integrate reranking into retrieval runtime
|
||||
- `197d8e7` Add query normalization cache
|
||||
- `3451a73` Add retrieval strategy comparison
|
||||
- `60e58ac` Add EvidenceBundle contract
|
||||
- `f950b59` Add EvidenceBundle assembly
|
||||
|
||||
## Findings
|
||||
|
||||
- No high-priority code defects requiring immediate remediation were found.
|
||||
- Technical direction remains aligned:
|
||||
- Rust remains limited to pure compute for RRF fusion.
|
||||
- TypeScript owns provider calls, retrieval planning, evaluation, caching, and EvidenceBundle assembly.
|
||||
- Core model changes remain Zod contracts without IO.
|
||||
- Performance boundaries remain acceptable:
|
||||
- Retrieval database paths use explicit `topK` and `maxRows`.
|
||||
- Hybrid recall still runs dense and FTS searches in parallel.
|
||||
- Reranking is capped by `maxRerankCandidates`.
|
||||
- Query normalization cache keys do not include raw query text and use TTL-backed cache writes.
|
||||
- EvidenceBundle assembly consumes already retrieved/reranked candidates and performs no DB/object/cache lookups.
|
||||
- Test health remains acceptable:
|
||||
- Full verification passed on the latest implementation slice.
|
||||
- Package coverage gates remain above 90%.
|
||||
- `.harness/changes` contains a trace entry for each implementation slice.
|
||||
|
||||
## Residual Risks
|
||||
|
||||
- API branch coverage is still close to the 90% floor, so future API slices should add branch coverage while implementing behavior.
|
||||
- Retrieval strategy comparison intentionally runs bounded dense-only and FTS-only baseline reads in addition to hybrid evaluation; future CI regression jobs must keep golden question page size explicit and small.
|
||||
- Real provider/runtime wiring for reranking remains deferred.
|
||||
|
||||
## Verification Reference
|
||||
|
||||
- Latest full verification before this review passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Next Cadence
|
||||
|
||||
- Next review cadence starts from checkpoint `f950b59`.
|
||||
- The next 10 implementation commits after `f950b59` must pause for another health review before feature iteration continues.
|
||||
@ -0,0 +1,34 @@
|
||||
# 10-Commit Health Review
|
||||
|
||||
## Summary
|
||||
|
||||
- Completed the required project health review after the 10th implementation commit following checkpoint `9c6714f`.
|
||||
- New reviewed checkpoint: `0105450`.
|
||||
|
||||
## Review Scope
|
||||
|
||||
- Technical direction: Sprint 2 API, auth, upload, parser, ingestion, read APIs, and trace hooks remain aligned with the TypeScript-first gateway and adapter-boundary architecture.
|
||||
- Performance boundaries: current hot paths retain explicit upload/object/parser bounds, tenant-scoped indexed database access, no object-storage readback during synchronous ingestion, and no newly introduced N+1 database paths.
|
||||
- Trace safety: request and ingestion spans contain bounded metadata only; JWTs, file bodies, document text, object bodies, filenames, and stack traces are not recorded.
|
||||
- Testing and coverage: latest full verification passed with API package coverage above 90% and workspace checks green.
|
||||
- Traceability: the basic trace hook slice is recorded under `.harness/changes`, and this review updates the temporary progress documents to start the next cadence from checkpoint `0105450`.
|
||||
|
||||
## Findings
|
||||
|
||||
- Documentation drift found and fixed: `TEMP-progress-document.md` still described checkpoint `9c6714f` as the latest reviewed checkpoint and reported the previous 9-commit count.
|
||||
- No code defects requiring immediate remediation were found in this review.
|
||||
|
||||
## Verification
|
||||
|
||||
- Review used the already-passed verification from the trace hook slice:
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
- Post-review documentation diff check:
|
||||
- `git diff --check`
|
||||
@ -0,0 +1,44 @@
|
||||
# Admin Retrieval UI
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `apps/admin/lib/retrieval-preview.ts`.
|
||||
- Added a bounded retrieval preview helper that turns query SSE events into Admin UI state.
|
||||
- Added retrieval preview UI for:
|
||||
- streaming answer
|
||||
- inline citations
|
||||
- confidence
|
||||
- freshness
|
||||
- Added tests for bounded answer accumulation and citation truncation.
|
||||
|
||||
## Why
|
||||
|
||||
- Phase 2 Sprint 9 requires the Admin Console to expose retrieval workflows with streaming answer preview, citations, confidence, and freshness before the trace viewer work starts.
|
||||
|
||||
## Performance And Safety Notes
|
||||
|
||||
- `createRetrievalPreview()` rejects answers that exceed `maxAnswerChars`.
|
||||
- Citations are capped by `maxCitations`.
|
||||
- The helper consumes already-received SSE events and does not call Hono, database, cache, object storage, or providers.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/admin test -- lib/retrieval-preview.test.ts app/page.test.tsx` failed because the preview helper and retrieval UI fields were missing.
|
||||
- Focused verification:
|
||||
- `pnpm --filter @knowledge/admin test -- lib/retrieval-preview.test.ts app/page.test.tsx`
|
||||
- `pnpm --filter @knowledge/admin typecheck`
|
||||
- `pnpm lint`
|
||||
- Full verification:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- The retrieval panel currently renders a static preview state. Live browser-side streaming interaction should wire this to `createAdminApiClient().streamQuery()` in the next retrieval UI refinement.
|
||||
@ -0,0 +1,47 @@
|
||||
# Admin Shared Hono API Client
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `apps/admin/lib/api-client.ts`.
|
||||
- Added a shared fetch client for:
|
||||
- `GET /health`
|
||||
- `GET /knowledge-spaces`
|
||||
- `POST /queries` SSE
|
||||
- Added typed response parsing for health and KnowledgeSpace list responses.
|
||||
- Added compact SSE parsing for Admin query streaming.
|
||||
- Wired the Admin page API base display through the shared client helper.
|
||||
- Added tests with fake fetch for request URLs, auth headers, response parsing, SSE parsing, and bounded input rejection.
|
||||
|
||||
## Why
|
||||
|
||||
- Phase 2 Sprint 9 requires the Admin Console to consume Hono APIs through shared/generated client boundaries before live UI workflows are built.
|
||||
|
||||
## Performance And Safety Notes
|
||||
|
||||
- Client methods enforce bounded list limits and query byte size before network calls.
|
||||
- Auth tokens are only placed in request headers and are not included in errors, logs, or cache keys.
|
||||
- The SSE parser only returns structured event/data pairs and rejects malformed JSON.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/admin test` failed because `apps/admin/lib/api-client.ts` was missing.
|
||||
- Focused verification:
|
||||
- `pnpm --filter @knowledge/admin test`
|
||||
- `pnpm --filter @knowledge/admin typecheck`
|
||||
- `pnpm --filter @knowledge/admin build`
|
||||
- `pnpm lint`
|
||||
- Full verification:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- This is a hand-written shared client slice. OpenAPI code generation can replace or generate this boundary in a later pass if desired.
|
||||
- Upload UI, trace viewer, and live retrieval UI will consume this client in the next Sprint 9 slices.
|
||||
@ -0,0 +1,40 @@
|
||||
# Admin Retrieval Trace Viewer
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `apps/admin/lib/trace-summary.ts`.
|
||||
- Added a bounded trace summary helper for route, recall candidates, filters, rerank, and evidence.
|
||||
- Updated the Admin trace panel to render the trace summary fields alongside recent document context.
|
||||
- Added tests for summary mapping and oversized trace-step rejection.
|
||||
|
||||
## Why
|
||||
|
||||
- Phase 2 Sprint 9 requires a retrieval trace viewer so operators can inspect route selection, recall, filtering, reranking, and evidence behavior.
|
||||
|
||||
## Performance And Safety Notes
|
||||
|
||||
- Trace summaries reject inputs above `maxSteps`.
|
||||
- Summary formatting only uses low-cardinality scalar attributes.
|
||||
- The UI helper performs no database, cache, object-storage, provider, or network calls.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/admin test -- lib/trace-summary.test.ts app/page.test.tsx` failed because the trace summary helper and UI fields were missing.
|
||||
- Focused verification:
|
||||
- `pnpm --filter @knowledge/admin test -- lib/trace-summary.test.ts app/page.test.tsx`
|
||||
- `pnpm --filter @knowledge/admin typecheck`
|
||||
- `pnpm lint`
|
||||
- Full verification:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- The trace viewer currently renders a static preview summary. A later slice should wire it to the Hono trace API through the shared Admin client.
|
||||
@ -0,0 +1,50 @@
|
||||
# Admin UI BFF Constraints
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `apps/admin/lib/bff.ts`.
|
||||
- Added a thin Admin BFF proxy boundary for Next.js route handlers.
|
||||
- Added `apps/admin/app/api/bff/[...path]/route.ts` to delegate UI-friendly requests to Hono APIs.
|
||||
- Added a strict allowlist for proxied Hono routes:
|
||||
- health and OpenAPI reads
|
||||
- KnowledgeSpace CRUD
|
||||
- document upload/read/artifact read
|
||||
- query streaming
|
||||
- trace reads
|
||||
- Added request body bounds and request/response header allowlists.
|
||||
- Added forbidden-import scanning for Admin source files so UI/BFF code does not import core runtime packages directly.
|
||||
|
||||
## Why
|
||||
|
||||
- Phase 2 Sprint 9 requires any Next.js BFF route to stay thin, UI-only, and delegated to Hono rather than owning knowledge, retrieval, ingestion, job, permission, provider, database, or adapter logic.
|
||||
|
||||
## Performance And Safety Notes
|
||||
|
||||
- The BFF buffers only bounded non-GET request bodies before forwarding.
|
||||
- Cookies and other ambient browser headers are not forwarded to Hono; only `accept`, `authorization`, `content-type`, and `x-trace-id` are allowed.
|
||||
- Upstream responses only expose `cache-control`, `content-type`, and `x-trace-id`.
|
||||
- Route path segments are validated before proxying and unknown paths return `404`.
|
||||
- Upstream fetch failures map to `502` without leaking stack traces or request details.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/admin test -- lib/bff.test.ts` failed because `apps/admin/lib/bff.ts` did not exist.
|
||||
- Focused verification:
|
||||
- `pnpm --filter @knowledge/admin test -- lib/bff.test.ts`
|
||||
- `pnpm --filter @knowledge/admin typecheck`
|
||||
- `pnpm --filter @knowledge/admin build`
|
||||
- Full verification:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- This BFF boundary intentionally does not proxy large uploads by default; upload UI can continue calling the Hono API directly for large files.
|
||||
- The forbidden-import scan is test-enforced rather than a dedicated lint plugin; a later pass can promote it into a standalone CI script if Admin grows substantially.
|
||||
@ -0,0 +1,47 @@
|
||||
# Admin Upload UI And Health Report
|
||||
|
||||
## What Changed
|
||||
|
||||
- Extended the Admin shared API client with:
|
||||
- `uploadDocument()`
|
||||
- `getDocument()`
|
||||
- `getParseArtifact()`
|
||||
- Added client-side upload bounds through `maxUploadBytes`.
|
||||
- Added typed parsing for `DocumentAsset` and `ParseArtifact` API responses.
|
||||
- Added `apps/admin/lib/document-health.ts` for parse status, node count, quality risks, size labels, and publish readiness.
|
||||
- Updated the Admin Console upload panel to render a multipart upload form.
|
||||
- Updated the publish readiness panel to show parser status, node count, quality risks, parse element count, and file size.
|
||||
|
||||
## Why
|
||||
|
||||
- Phase 2 Sprint 9 requires the Admin Console to move from a shell toward upload intake and health reporting while still consuming Hono APIs through shared client boundaries.
|
||||
|
||||
## Performance And Safety Notes
|
||||
|
||||
- Uploads are rejected before network calls when `File.size` exceeds `maxUploadBytes`.
|
||||
- Parse artifact and document response parsing clones metadata and element arrays into UI-owned objects.
|
||||
- Health report risk lists are explicitly capped with `maxRisks`.
|
||||
- The UI health report is computed locally from already-loaded asset/artifact summaries and does not introduce database, object-storage, or provider calls.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/admin test -- lib/api-client.test.ts lib/document-health.test.ts app/page.test.tsx` failed because upload client methods, document health helper, and parser-status UI were missing.
|
||||
- Focused verification:
|
||||
- `pnpm --filter @knowledge/admin test -- lib/api-client.test.ts lib/document-health.test.ts app/page.test.tsx`
|
||||
- `pnpm --filter @knowledge/admin typecheck`
|
||||
- `pnpm lint`
|
||||
- Full verification:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- The visible form is still an operational UI scaffold; authenticated browser-side submission wiring and live document state refresh should follow in the next Admin UI slices.
|
||||
- Admin package coverage is not yet promoted into the root `test:coverage` gate; existing behavioral packages remain above 90%.
|
||||
@ -0,0 +1,47 @@
|
||||
# Answerability States
|
||||
|
||||
## Summary
|
||||
|
||||
- Added a rule-based answerability evaluator for Sprint 6 EvidenceBundle flow.
|
||||
- EvidenceBundle assembly now uses the shared evaluator by default.
|
||||
|
||||
## Changes
|
||||
|
||||
- Added `createAnswerabilityEvaluator()` in `@knowledge/api`.
|
||||
- Added configurable rules:
|
||||
- `minFinalScore`
|
||||
- `minItems`
|
||||
- Added answerability outcomes:
|
||||
- `answerable`
|
||||
- `partial`
|
||||
- `not-enough-evidence`
|
||||
- `conflict`
|
||||
- `permission-limited`
|
||||
- Integrated the evaluator with `createEvidenceBundleAssembler()`.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- The evaluator is pure in-memory logic over already bounded EvidenceBundle items.
|
||||
- No database, object storage, cache, network, or provider calls were introduced.
|
||||
|
||||
## Verification
|
||||
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm lint`
|
||||
- Full verification passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Review Cadence
|
||||
|
||||
- This slice will be implementation commit 1 after review checkpoint `f950b59`.
|
||||
- The next 10-commit review is not due yet.
|
||||
@ -0,0 +1,47 @@
|
||||
# AnswerTrace Recording
|
||||
|
||||
## Summary
|
||||
|
||||
- Added AnswerTrace recording boundaries for Sprint 6.
|
||||
- Trace persistence records normalize, route, recall, filter, rerank, and evidence stages as bounded steps.
|
||||
|
||||
## Changes
|
||||
|
||||
- Added `AnswerTraceRepository`.
|
||||
- Added `createInMemoryAnswerTraceRepository()`.
|
||||
- Added `createDatabaseAnswerTraceRepository()`.
|
||||
- Added `createAnswerTraceRecorder()`.
|
||||
- Database persistence writes:
|
||||
- one `answer_traces` row
|
||||
- one batched `answer_trace_steps` insert for all steps
|
||||
- Database reads use explicit `maxRows` and stable step ordering.
|
||||
- In-memory persistence enforces `maxTraces` and `maxSteps`.
|
||||
- Recorder enforces `maxSteps` before persistence and returns `AnswerTraceSchema`-validated output.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- Step writes are batched rather than one database call per step.
|
||||
- Trace reads use indexed `trace_id + started_at + id` ordering.
|
||||
- Step metadata is JSON only; raw document bytes and document text are not written.
|
||||
|
||||
## Verification
|
||||
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm lint`
|
||||
- Full verification passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Review Cadence
|
||||
|
||||
- This slice will be implementation commit 5 after review checkpoint `f950b59`.
|
||||
- The next 10-commit review is not due yet.
|
||||
@ -0,0 +1,33 @@
|
||||
# Basic Hybrid Retrieval
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added a database-backed hybrid retrieval repository for ready `index_projections`.
|
||||
- Dense retrieval runs one bounded parameterized vector query per request.
|
||||
- FTS retrieval runs one bounded parameterized database-native full-text query per request.
|
||||
- Added a basic hybrid retriever that executes dense and FTS searches in parallel and fuses candidates with reciprocal rank fusion.
|
||||
- Added tests for PostgreSQL pgvector SQL, TiDB vector/FTS SQL, bounded `topK`, bounded `limit`, query-vector validation, parameterized SQL, and fused duplicate candidates.
|
||||
|
||||
## Why
|
||||
|
||||
- Sprint 3 requires a first retrieval path after dense and FTS projections are available.
|
||||
- The implementation keeps retrieval inside the `DatabaseAdapter` boundary and avoids per-node query waterfalls.
|
||||
- Explicit `topK`, `limit`, and `maxRows` guardrails prevent unbounded reads and in-memory accumulation.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`: passed.
|
||||
- `pnpm --filter @knowledge/api test:coverage`: passed.
|
||||
- `pnpm check`: passed.
|
||||
- `pnpm build`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
- `pnpm wasm:build`: passed.
|
||||
- `pnpm compose:config`: passed.
|
||||
- `docker compose --profile apps config`: passed.
|
||||
- `git diff --check`: passed.
|
||||
|
||||
## Known Risks And Follow-Up
|
||||
|
||||
- RRF currently runs in TypeScript as the basic MVP path; the iteration plan allows later WASM RRF hardening.
|
||||
- Results currently return fused node/projection ids and source labels; citation/source-location enrichment is the next Sprint 3 slice.
|
||||
@ -0,0 +1,42 @@
|
||||
# Basic Ingestion Trace Hooks
|
||||
|
||||
## Summary
|
||||
|
||||
- Added lightweight request and ingestion tracing hooks to the Knowledge Gateway.
|
||||
- Kept tracing injectable and no-op by default so runtime deployments do not need an SDK/exporter yet.
|
||||
- Added an in-memory trace recorder for deterministic API tests.
|
||||
|
||||
## Behavior
|
||||
|
||||
- Every gateway request now receives or propagates a `traceId` and returns it in the `x-trace-id` response header.
|
||||
- HTTP request spans record bounded attributes: method, normalized route, status code, trace id, and tenant id after authentication.
|
||||
- Document upload ingestion records step spans for space lookup, upload read/hash, object put, asset create, parser parse, artifact create, status update, and cleanup when needed.
|
||||
- Uploaded `DocumentAsset` metadata and persisted `ParseArtifact` metadata include the request `traceId`.
|
||||
|
||||
## Performance And Safety
|
||||
|
||||
- Default tracing is no-op and does not allocate persistent runtime state.
|
||||
- The test recorder records only bounded span metadata.
|
||||
- Trace attributes intentionally avoid JWTs, uploaded file bytes, object bodies, full document text, filenames, and exception stack traces.
|
||||
- Ingestion continues to reuse the already bounded uploaded bytes buffer instead of rereading object storage.
|
||||
|
||||
## Review Cadence
|
||||
|
||||
- This implementation commit is the 10th implementation commit after review checkpoint `9c6714f`.
|
||||
- Feature iteration must pause after commit and push until a project health review is completed.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED confirmed with failing API tests for missing `x-trace-id`, missing trace recorder events, and missing trace id metadata.
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- Full verification passed:
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
@ -0,0 +1,46 @@
|
||||
# Blue-Green Index Publication
|
||||
|
||||
## Summary
|
||||
|
||||
- Added the first blue-green publication boundary for `IndexProjection` versions.
|
||||
- Candidate projection versions can now be evaluated with a status summary, published to active `ready`, or rolled back to `failed`.
|
||||
|
||||
## Changes
|
||||
|
||||
- Extended `IndexProjectionRepository` with:
|
||||
- `summarizeVersion(input)` for bounded status-count evaluation.
|
||||
- `publishVersion(input)` to promote candidate `building` rows to `ready` and mark previous ready rows `stale`.
|
||||
- `rollbackVersion(input)` to mark candidate `building` rows `failed`.
|
||||
- Implemented the behavior for both bounded in-memory and database-backed repositories.
|
||||
- Database-backed publication uses parameterized `UPDATE` and aggregate `SELECT` statements.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- Retrieval still reads only `status = "ready"` rows through `listReadyBySpace()`.
|
||||
- Publishing does not rewrite vectors or text payloads; it flips compact status fields only.
|
||||
- Rollback leaves the active ready version untouched and marks only matching candidate rows failed.
|
||||
- Database queries are tenant/space and projection-type scoped, avoiding cross-space publication.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts` failed because `summarizeVersion()` did not exist.
|
||||
- Focused verification:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm lint`
|
||||
- Full verification:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Commit Tracking
|
||||
|
||||
- This slice is implementation commit 6 after reviewed checkpoint `3b9b4d8` once committed and pushed.
|
||||
- The next 10-commit health review is not yet due.
|
||||
@ -0,0 +1,41 @@
|
||||
# Cache Polish: Model, Permission, And Index Versions
|
||||
|
||||
## Summary
|
||||
|
||||
- Added version-aware embedding and rerank provider cache wrappers.
|
||||
- Added a KnowledgeFS path resolution cache boundary with permission snapshot and path index version in the key.
|
||||
- Kept cache keys digest-based so raw query text, document text, and virtual paths are not exposed in cache keys.
|
||||
|
||||
## Changes
|
||||
|
||||
- Added `createCachedEmbeddingProvider()` in `@knowledge/embeddings`.
|
||||
- Added `createCachedRerankerProvider()` in `@knowledge/embeddings`.
|
||||
- Added `createKnowledgePathResolutionCache()` in `@knowledge/api`.
|
||||
- Added focused cache tests covering model/tokenizer version changes, malformed cache entry recovery, clone isolation, permission snapshot ordering, path index version isolation, and bounded path input.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- Embedding cache keys include provider kind, model id, model version, tokenizer version, input type, and text digests.
|
||||
- Rerank cache keys include provider kind, model id, model version, query digest, topN, document ids, document metadata digest, and document text digests.
|
||||
- Path resolution cache keys include knowledge space id, permission snapshot, path index version, and virtual path digest.
|
||||
- Cache entry size, TTL, cache version, and path byte bounds are validated.
|
||||
- Malformed or stale cache entries are ignored and refreshed instead of throwing runtime errors.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/embeddings test -- src/embedding.test.ts` failed because cached provider factories did not exist.
|
||||
- `pnpm --filter @knowledge/api test -- src/cache-polish.test.ts` failed because `createKnowledgePathResolutionCache()` did not exist.
|
||||
- Focused verification:
|
||||
- `pnpm --filter @knowledge/embeddings test -- src/embedding.test.ts`
|
||||
- `pnpm --filter @knowledge/api test -- src/cache-polish.test.ts`
|
||||
- `pnpm --filter @knowledge/embeddings test:coverage`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm --filter @knowledge/embeddings typecheck`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm lint`
|
||||
|
||||
## Commit Tracking
|
||||
|
||||
- This slice is review checkpoint `92f4e22` + implementation commit 5 after commit and push.
|
||||
- The next 10-commit health review is not yet due.
|
||||
@ -0,0 +1,45 @@
|
||||
# CI Retrieval Regression Evaluation
|
||||
|
||||
## Summary
|
||||
|
||||
- Added a deterministic retrieval regression gate for CI and local checks.
|
||||
- The gate compares current recall, citation hit rate, no-answer rate, and question count against checked-in thresholds and optional baseline deltas.
|
||||
- Wired `pnpm eval:regression` into the root `check` script and GitHub Actions.
|
||||
|
||||
## Changes
|
||||
|
||||
- Added `createRetrievalRegressionGate()` in `@knowledge/api`.
|
||||
- Added `packages/api/scripts/retrieval-regression-gate.ts` for CI-friendly report evaluation.
|
||||
- Added `.harness/evaluation/retrieval-regression-report.json` as the current deterministic baseline/current fixture.
|
||||
- Exported retrieval regression types and factory from `@knowledge/api`.
|
||||
- Updated `.github/workflows/ci.yml` with an explicit retrieval regression evaluation step.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- Threshold and metric inputs are validated before evaluation.
|
||||
- Failure output is bounded with `maxFailures` to keep CI logs predictable.
|
||||
- The fixture is static and local; no network, database, or unbounded retrieval work runs during CI.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/api test -- src/retrieval-regression.test.ts` failed because `./retrieval-regression` did not exist.
|
||||
- `pnpm eval:regression` failed because the root script did not exist.
|
||||
- GREEN focused verification:
|
||||
- `pnpm --filter @knowledge/api test -- src/retrieval-regression.test.ts`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm eval:regression`
|
||||
- Full verification:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Commit Tracking
|
||||
|
||||
- This slice is review checkpoint `92f4e22` + implementation commit 4 after commit and push.
|
||||
- The next 10-commit health review is not yet due.
|
||||
@ -0,0 +1,41 @@
|
||||
# Citation Normalization
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `createCitationNormalizer()` to `@knowledge/generation`.
|
||||
- Normalizes generated answer citation markers against packed evidence items.
|
||||
- Removes orphan markers from answer text while reporting them for traceability.
|
||||
- Maps valid markers to bounded citation metadata: marker, node id, score, and cloned source citations.
|
||||
- Added safety guards for maximum answer bytes, maximum citation count, and duplicate evidence markers.
|
||||
- Added focused generation tests for valid marker mapping, orphan cleanup, byte bounds, citation-count bounds, and duplicate marker rejection.
|
||||
|
||||
## Why
|
||||
|
||||
Generation output needs a deterministic, bounded post-processing step before API responses or future cache entries rely on citations. This keeps answer text from exposing unsupported citation markers and gives downstream trace/UI layers a structured citation list tied to the EvidenceBundle packing result.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- The normalizer is pure in-memory compute over already-packed evidence and the generated answer text.
|
||||
- It performs no provider, database, cache, object-storage, or trace round-trips.
|
||||
- Answer text and citation count are explicitly bounded to prevent unbounded regex scans or retained citation arrays.
|
||||
- Returned citation metadata is cloned so callers cannot mutate retained evidence citation state.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/generation test -- src/generation.test.ts`: passed.
|
||||
- `pnpm --filter @knowledge/generation typecheck`: passed.
|
||||
- `pnpm --filter @knowledge/generation test:coverage`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `pnpm check`: passed.
|
||||
- `pnpm build`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
- `pnpm wasm:build`: passed.
|
||||
- `pnpm compose:config`: passed.
|
||||
- `docker compose --profile apps config`: passed.
|
||||
- `git diff --check`: passed.
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- Citation normalization is currently exposed as a reusable generation utility and is not yet wired into the SSE query route. The next generation/cache slices should decide the final response-envelope shape for normalized citations.
|
||||
- Marker syntax is intentionally narrow (`[E<number>]`) to match the evidence packer output; future prompt templates that introduce new citation formats must version the normalizer or add explicit tests.
|
||||
@ -0,0 +1,31 @@
|
||||
# Citation Source Location Response
|
||||
|
||||
## What Changed
|
||||
|
||||
- Extended hybrid retrieval results with citation source-location data.
|
||||
- Each fused retrieval item now includes document asset id, document version, artifact hash, page number, section path, and start/end offsets.
|
||||
- Dense and FTS retrieval queries now join `index_projections` to `knowledge_nodes` and `parse_artifacts` in the same bounded query.
|
||||
- Added tests proving citation fields are returned, PostgreSQL SQL joins the required tables, TiDB retrieval still uses vector/FTS primitives, and guardrails remain covered.
|
||||
|
||||
## Why
|
||||
|
||||
- Sprint 3 requires retrieval output to be citation-ready before EvidenceBundle and answer generation work begins.
|
||||
- Joining node/artifact metadata during retrieval avoids N+1 lookups after candidate recall.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`: passed.
|
||||
- `pnpm --filter @knowledge/api test:coverage`: passed.
|
||||
- `pnpm check`: passed.
|
||||
- `pnpm build`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
- `pnpm wasm:build`: passed.
|
||||
- `pnpm compose:config`: passed.
|
||||
- `docker compose --profile apps config`: passed.
|
||||
- `git diff --check`: passed.
|
||||
|
||||
## Known Risks And Follow-Up
|
||||
|
||||
- This slice exposes citation metadata through the retrieval boundary only; no public query API or EvidenceBundle persistence is added yet.
|
||||
- Citation scoring and evidence packing remain later Phase 2/Sprint work.
|
||||
@ -0,0 +1,47 @@
|
||||
# CJK/English FTS Tuning
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `normalizeMixedLanguageFtsText()` to `@knowledge/api`.
|
||||
- `createFtsProjectionBuilder()` now stores normalized FTS text instead of raw node text.
|
||||
- FTS projection metadata now records `ftsLanguageStrategy: "mixed-cjk-latin-v1"`.
|
||||
- PostgreSQL and TiDB FTS retrieval now pass normalized query parameters into database-native FTS SQL.
|
||||
- Added tests covering:
|
||||
- Mixed Chinese/English normalization such as `合同ABC-123续约 terms`.
|
||||
- English normalization and punctuation-only empty normalization.
|
||||
- FTS projection metadata strategy and normalized text.
|
||||
- PostgreSQL and TiDB retrieval query parameter normalization.
|
||||
|
||||
## Why
|
||||
|
||||
Phase 2 Sprint 5 starts retrieval quality hardening. Mixed CJK/English content is a known risk because PostgreSQL `simple` FTS and TiDB FULLTEXT do not expose identical tokenization behavior. A shared normalization step makes both indexing and query parameters more predictable before live backend-specific pg_jieba/pg_bigm or TiDB parser configuration is wired.
|
||||
|
||||
## Performance And Safety
|
||||
|
||||
- Normalization runs once per node during FTS projection build and once per FTS query.
|
||||
- Retrieval still uses one bounded parameterized FTS SQL query with explicit `topK` and `maxRows`.
|
||||
- No unbounded scans or additional database round trips were introduced.
|
||||
- Query text is still passed as a parameter, not interpolated into SQL.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED confirmed with `pnpm --filter @knowledge/api test -- src/gateway.test.ts`; tests failed because the normalizer and normalized FTS behavior did not exist.
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- Full verification passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks And Follow-Up
|
||||
|
||||
- This is a portable fallback strategy, not a substitute for live PostgreSQL pg_jieba/pg_bigm or TiDB parser validation.
|
||||
- Existing FTS rows created before this change would need projection rebuild to get normalized `ftsText`.
|
||||
- Future retrieval optimization should compare raw FTS, normalized FTS, and backend-native CJK parser behavior with golden questions.
|
||||
@ -0,0 +1,50 @@
|
||||
# Cloudflare Job Queue Adapter
|
||||
|
||||
## Summary
|
||||
|
||||
- Added a Cloudflare-oriented `JobQueueAdapter` implementation that wraps the durable contract with Queue delivery and Durable Object-like state persistence boundaries.
|
||||
- Wired the Cloudflare platform factory to accept injectable Queue binding and state store test doubles.
|
||||
- Kept a no-op local skeleton path for development and tests when real Cloudflare bindings are not available.
|
||||
|
||||
## Changes
|
||||
|
||||
- Added `createCloudflareJobQueueAdapter()` in `@knowledge/adapters`.
|
||||
- Added portable binding contracts:
|
||||
- `CloudflareQueueBinding` with `send(body, options)`.
|
||||
- `CloudflareJobStateStore` with `put(jobId, record)`.
|
||||
- `enqueue()` now stores job state and sends a bounded queue message containing job id, type, attempts, and idempotency key.
|
||||
- `runAfter` maps to Cloudflare Queue `delaySeconds`.
|
||||
- `lease()`, `dequeue()`, `heartbeat()`, `fail()`, `retry()`, `complete()`, and `cancel()` persist updated job state.
|
||||
- Duplicate idempotency-key enqueues return the existing job without sending a duplicate Queue message.
|
||||
- `createCloudflarePlatformAdapter()` accepts `jobQueue` and `jobStateStore` injection for tests and future Workers runtime wiring.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- Queue messages do not include raw payload bytes or large document content.
|
||||
- State persistence receives clone-isolated `JobRecord` snapshots from the underlying bounded queue.
|
||||
- The local no-op binding path is explicitly a skeleton; real Cloudflare Queues and Durable Objects binding configuration remains a later runtime/deployment slice.
|
||||
- Bounded lease, batch, queued-job, and terminal-retention limits are inherited from the durable inline implementation.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/adapters test -- src/adapters.test.ts` failed because `./cloudflare-job-queue` did not exist.
|
||||
- Focused verification:
|
||||
- `pnpm --filter @knowledge/adapters test -- src/adapters.test.ts`
|
||||
- `pnpm --filter @knowledge/adapters typecheck`
|
||||
- `pnpm --filter @knowledge/adapters test:coverage`
|
||||
- `pnpm lint`
|
||||
- Full verification:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Commit Tracking
|
||||
|
||||
- This slice is review checkpoint `92f4e22` + implementation commit 10 after commit and push.
|
||||
- A mandatory 10-commit health review must run immediately after this commit is pushed.
|
||||
@ -0,0 +1,36 @@
|
||||
# CommandRegistry Contract
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `packages/core/src/command-registry.ts` with an allowlisted KnowledgeFS command registry.
|
||||
- Added command contracts for handlers, resource/node overrides, permission checks, cost estimation, trace hooks, cache policy, and degradation policy.
|
||||
- Exported the registry through `@knowledge/core`.
|
||||
- Added TDD coverage for:
|
||||
- allowlisted command registration and validated execution,
|
||||
- resource type and node-kind handler overrides,
|
||||
- duplicate/max registry bounds,
|
||||
- unauthorized, invalid, unsupported, missing, and unsafe command execution,
|
||||
- command failure trace events,
|
||||
- summary clone isolation.
|
||||
|
||||
## Why It Changed
|
||||
|
||||
- Sprint 4 needs a central command dispatch boundary before implementing safe shell-style KnowledgeFS commands.
|
||||
- The `.harness` architecture requires safe shell behavior to be an allowlisted dispatcher over registered KnowledgeFS commands, never host shell execution.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- The registry is explicitly bounded with `maxCommands`.
|
||||
- Execution validates input through command schemas before invoking handlers.
|
||||
- Cost estimation is part of the contract so future commands can expose bounded read/scan expectations before expensive execution paths are added.
|
||||
- No host process execution, database queries, or network IO were introduced in this slice.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/core test -- src/command-registry.test.ts`
|
||||
- `pnpm --filter @knowledge/core test:coverage`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- The registry is a contract and in-memory dispatcher only; actual `ls`, `tree`, `cat`, and other KnowledgeFS command handlers still need to be implemented over bounded repositories.
|
||||
- Future safe shell parsing must dispatch only through this registry and preserve the same allowlist.
|
||||
@ -0,0 +1,45 @@
|
||||
# Component Health Endpoint
|
||||
|
||||
## What Changed
|
||||
|
||||
- Extended `/health` to report gateway component health for:
|
||||
- parser
|
||||
- embedding
|
||||
- reranker
|
||||
- LLM
|
||||
- Added `componentHealth` injection to `createKnowledgeGateway()`.
|
||||
- Component health sources may expose either `health()` or `models()`.
|
||||
- Adapter health remains reported for database, object storage, cache, and jobs.
|
||||
|
||||
## Why
|
||||
|
||||
- Phase 2 Sprint 8 requires `/health` to report DB, object store, cache, parser, embedding, reranker, and LLM readiness.
|
||||
|
||||
## Performance And Safety Notes
|
||||
|
||||
- Component health probes run concurrently with platform health.
|
||||
- Probe failures are converted to `false` and do not throw out of the endpoint.
|
||||
- `/health.ok` continues to reflect platform adapter health, while optional provider readiness is visible in `components`.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts` failed because parser/embedding/reranker/LLM components were missing from `/health`.
|
||||
- Focused verification:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm lint`
|
||||
- Full verification:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- Production provider health can later be backed by real provider clients, circuit-breaker state, or cached health probes.
|
||||
@ -0,0 +1,37 @@
|
||||
# Context Window Packing
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `ComputeRuntime.packEvidence()` as the TypeScript wrapper for WASM `packEvidenceJson`.
|
||||
- Added Zod validation for packed evidence context, included items, omitted items, and token accounting.
|
||||
- Added `createContextWindowPacker()` to `@knowledge/generation`.
|
||||
- Context packing now splits a model context window into system prompt tokens, evidence token budget, output tokens, and safety margin.
|
||||
- The packer calls WASM evidence packing only after budget validation succeeds.
|
||||
|
||||
## Why
|
||||
|
||||
- Sprint 7 generation needs context-window budgeting before prompt templates and SSE generation can safely send evidence to LLM providers.
|
||||
- Budget splitting keeps model output reservations and system prompts from accidentally crowding evidence beyond a model context limit.
|
||||
|
||||
## Performance And Safety Notes
|
||||
|
||||
- Budget computation is in-memory and bounded.
|
||||
- Invalid context windows fail before invoking the evidence packer.
|
||||
- `ComputeRuntime.packEvidence()` validates all WASM output before returning it to TypeScript callers.
|
||||
- The implementation does not add database, object-storage, cache, network, or job access.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/compute test -- src/compute.test.ts`
|
||||
- `pnpm --filter @knowledge/compute typecheck`
|
||||
- `pnpm --filter @knowledge/compute test:coverage`
|
||||
- `pnpm --filter @knowledge/generation test -- src/generation.test.ts`
|
||||
- `pnpm --filter @knowledge/generation typecheck`
|
||||
- `pnpm --filter @knowledge/generation test:coverage`
|
||||
|
||||
Full workspace verification is recorded in `TEMP-progress-document.md` after completion.
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- This slice does not yet build evidence-driven prompt templates or stream query responses.
|
||||
- The next workflow step is a mandatory 10-commit project health review before continuing feature iteration.
|
||||
@ -0,0 +1,50 @@
|
||||
# Degradation Flags
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added configured hybrid retrieval degradation policies:
|
||||
- dense failure can degrade to FTS-only retrieval.
|
||||
- FTS failure can degrade to dense-only retrieval.
|
||||
- reranker failure can degrade by skipping rerank and returning fused candidates.
|
||||
- Added low-cardinality `degradationFlags` to retrieval metrics when fallback paths are used.
|
||||
- Added LLM router fallback policies for primary provider failures.
|
||||
- LLM generate and stream paths now mark fallback routing metadata as degraded with fallback source provider and error class.
|
||||
- Validation now rejects invalid LLM fallback provider, model, and output-token settings.
|
||||
|
||||
## Why
|
||||
|
||||
- Phase 2 Sprint 8 requires provider failures to degrade through configured fallback paths instead of always failing closed.
|
||||
|
||||
## Performance And Safety Notes
|
||||
|
||||
- Retrieval legs still execute concurrently and remain bounded by existing topK/planner limits.
|
||||
- Degradation metrics store only fixed flag strings and error class names; raw queries, prompts, provider payloads, and stack traces are not recorded.
|
||||
- LLM stream fallback only occurs before any primary stream output is emitted, avoiding mixed-provider partial answers.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts` failed because dense retrieval failures still escaped.
|
||||
- `pnpm --filter @knowledge/generation test -- src/generation.test.ts` failed because LLM fallback policies were not implemented.
|
||||
- Focused verification:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/generation test -- src/generation.test.ts`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm --filter @knowledge/generation typecheck`
|
||||
- `pnpm --filter @knowledge/generation test:coverage`
|
||||
- `pnpm lint`
|
||||
- Full verification:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- Provider health scoring/circuit breaking is still a future runtime concern.
|
||||
- Embedding query creation fallback should be wired when the production query embedding runtime is introduced.
|
||||
@ -0,0 +1,47 @@
|
||||
# Dense Vector Projection
|
||||
|
||||
## Summary
|
||||
|
||||
- Added the Sprint 3 dense vector projection boundary.
|
||||
- Projection building now batches `KnowledgeNode` text through the embedding provider and persists `IndexProjection` rows.
|
||||
|
||||
## Behavior
|
||||
|
||||
- Added bounded in-memory and database-backed `IndexProjectionRepository` implementations.
|
||||
- Added `createDenseVectorProjectionBuilder()` to embed a node batch once and create ready `dense-vector` projections.
|
||||
- Projection metadata records dense vector, dimension, embedding provider, model version, artifact hash, document asset id, and parse artifact id.
|
||||
- Database persistence writes `dense_vector` separately from JSON metadata and keeps the core `IndexProjection` model stable.
|
||||
- Added ready-projection listing by `knowledgeSpaceId + type + status` with stable `nodeId + id` keyset pagination.
|
||||
|
||||
## Performance And Safety
|
||||
|
||||
- Dense projection building uses one embedding call per node batch, avoiding N+1 provider requests.
|
||||
- Repository writes use one parameterized batch insert and explicit `maxRows`.
|
||||
- List reads require explicit limits, use `limit + 1`, and use a stable keyset cursor.
|
||||
- Schema index `index_projections_space_type_status_idx` now includes `node_id` and `id` for stable ready-projection pagination.
|
||||
- Migration artifacts now include nullable dialect-specific `dense_vector` storage.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED confirmed with database schema tests failing for missing vector storage and stable projection index columns.
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/database test -- src/schema.test.ts`
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm --filter @knowledge/database test:coverage`
|
||||
- `pnpm db:migrations:write`
|
||||
- `pnpm db:migrations:check`
|
||||
- Full verification passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Follow-Up
|
||||
|
||||
- FTS projection is the next Sprint 3 slice.
|
||||
- Runtime database drivers still need live pgvector/TiDB integration tests before relying on vector search in production.
|
||||
@ -0,0 +1,51 @@
|
||||
# DocumentCompilationJob State Machine
|
||||
|
||||
## Summary
|
||||
|
||||
- Added the Phase 3 durable ingestion `DocumentCompilationJob` state machine.
|
||||
- The state machine starts document compilation work through `JobQueueAdapter` and enforces the ordered pipeline:
|
||||
`queued -> parsed -> nodes_generated -> projection_built -> smoke_eval_passed -> published`.
|
||||
|
||||
## Changes
|
||||
|
||||
- Added `packages/api/src/document-compilation-job.ts`.
|
||||
- Added `DocumentCompilationJob`, stage types, repository contract, and in-memory bounded repository.
|
||||
- Added `createDocumentCompilationJobStateMachine()` with:
|
||||
- `start()` to create a compilation job and enqueue `document.compile` work.
|
||||
- `advance()` to enforce ordered stage transitions.
|
||||
- `fail()` to mark failed and notify the underlying queue with optional retry time.
|
||||
- `cancel()` to mark canceled and notify the underlying queue.
|
||||
- `get()` returning clone-isolated state.
|
||||
- Exported the state machine boundary from `@knowledge/api`.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- Repository capacity is bounded by `maxJobs`.
|
||||
- Stage transitions are strict and terminal states cannot be advanced.
|
||||
- Queue payloads are compact identifiers and version fields only; no raw document bytes or parsed content are enqueued.
|
||||
- Idempotency key includes tenant, space, document asset, and version.
|
||||
- Returned records are clone-isolated.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/api test -- src/document-compilation-job.test.ts` failed because `./document-compilation-job` did not exist.
|
||||
- Focused verification:
|
||||
- `pnpm --filter @knowledge/api test -- src/document-compilation-job.test.ts`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm lint`
|
||||
- Full verification:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Commit Tracking
|
||||
|
||||
- This slice is implementation commit 2 after reviewed checkpoint `3b9b4d8` once committed and pushed.
|
||||
- The next 10-commit health review is not yet due.
|
||||
@ -0,0 +1,44 @@
|
||||
# Document Compilation Job Status APIs
|
||||
|
||||
## Summary
|
||||
|
||||
- Added Phase 3 Sprint 10 status and cancel APIs for durable document compilation jobs.
|
||||
- The APIs are protected by existing auth middleware, tenant-scoped, and backed by the injected `DocumentCompilationJobStateMachine`.
|
||||
|
||||
## Changes
|
||||
|
||||
- Added `GET /jobs/{id}` to read a document compilation job.
|
||||
- Added `DELETE /jobs/{id}` to cancel a non-terminal document compilation job.
|
||||
- Added OpenAPI schemas and path entries for job status/cancel.
|
||||
- Added `/jobs` auth, rate-limit, and trace route wiring.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- `GET /jobs/{id}` requires `knowledge-spaces:read` or `knowledge-spaces:*`.
|
||||
- `DELETE /jobs/{id}` requires `knowledge-spaces:write` or `knowledge-spaces:*`.
|
||||
- Cross-tenant job access returns 404 to avoid leaking resource existence.
|
||||
- Missing job runtime returns 503 instead of silently pretending job state exists.
|
||||
- Cancel delegates to the state machine and underlying queue, preserving terminal-state protection.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts` failed because `/jobs/{id}` returned 404 before the routes existed.
|
||||
- Focused verification:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- Full verification:
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Commit Tracking
|
||||
|
||||
- This slice is implementation commit 4 after reviewed checkpoint `3b9b4d8` once committed and pushed.
|
||||
- The next 10-commit health review is not yet due.
|
||||
@ -0,0 +1,29 @@
|
||||
# Document Ingestion Read APIs
|
||||
|
||||
## Summary
|
||||
|
||||
- Added authenticated document ingestion read boundaries after upload and parse.
|
||||
- Exposed tenant-scoped document status and parse artifact reads through the Knowledge Gateway.
|
||||
|
||||
## Behavior
|
||||
|
||||
- `GET /knowledge-spaces/{id}/documents/{documentId}` returns a scoped `DocumentAsset`.
|
||||
- `GET /knowledge-spaces/{id}/documents/{documentId}/parse-artifacts/{version}` returns a scoped `ParseArtifact`.
|
||||
- Both routes require `knowledge-spaces:read` or `knowledge-spaces:*`.
|
||||
- Missing spaces, missing documents, missing artifact versions, and cross-tenant access all return 404.
|
||||
|
||||
## Performance And Safety
|
||||
|
||||
- Read paths use existing scoped repository lookups rather than list scans.
|
||||
- Database-backed repositories continue to use parameterized SQL with explicit `maxRows: 1`.
|
||||
- Artifact reads verify the tenant-scoped space and document before loading the artifact by indexed asset/version.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED confirmed with failing API tests for missing OpenAPI paths and missing read routes.
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api build`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm lint`
|
||||
|
||||
@ -0,0 +1,48 @@
|
||||
# Durable Document Ingestion Migration
|
||||
|
||||
## Summary
|
||||
|
||||
- Added the first durable ingestion upload path for Phase 3 Sprint 10.
|
||||
- When a `DocumentCompilationJobStateMachine` is injected, document upload now stores the raw object and pending `DocumentAsset`, starts a durable `document.compile` job, and returns `202 Accepted` with a status URL instead of parsing synchronously on the request path.
|
||||
|
||||
## Changes
|
||||
|
||||
- Exported a `DocumentCompilationJobStateMachine` interface from `packages/api/src/document-compilation-job.ts`.
|
||||
- Extended `createKnowledgeGateway()` with an optional `documentCompilationJobs` dependency.
|
||||
- Added a durable upload branch that:
|
||||
- Reuses existing auth, tenant-scoped space lookup, upload bounds, object storage, and asset persistence.
|
||||
- Keeps `DocumentAsset.parserStatus` as `pending`.
|
||||
- Starts a compact durable compilation job with tenant, space, document asset, and version ids only.
|
||||
- Returns `202 Accepted`, `Location`, `statusUrl`, and minimal compilation job status.
|
||||
- Kept the default gateway path synchronous for current local/dev compatibility when no durable state machine is configured.
|
||||
- Added an OpenAPI `202` response schema for accepted durable uploads.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- The request path does not parse bytes or create parse artifacts in durable mode.
|
||||
- Queue payloads remain compact and do not include raw file bytes, text content, JWTs, or object bodies.
|
||||
- Existing upload size bounds, object key isolation, SHA-256 metadata, and tenant scoping are preserved.
|
||||
- If durable job start fails after asset creation, the asset is best-effort marked `failed` and the raw object cleanup path is reused.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts` failed because durable upload still returned `201` and parsed synchronously.
|
||||
- Focused verification:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- Full verification:
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Commit Tracking
|
||||
|
||||
- This slice is implementation commit 3 after reviewed checkpoint `3b9b4d8` once committed and pushed.
|
||||
- The next 10-commit health review is not yet due.
|
||||
@ -0,0 +1,57 @@
|
||||
# Durable JobQueue Contract
|
||||
|
||||
## Summary
|
||||
|
||||
- Extended the existing bounded inline `JobQueueAdapter` contract for Phase 3 durable ingestion.
|
||||
- Added lease, heartbeat, retry, cancel, and status semantics while preserving existing enqueue/dequeue/complete/fail compatibility.
|
||||
- Kept inline behavior bounded for local tests, Cloudflare skeletons, and Standalone skeletons until real Cloudflare Queues / pg-boss adapters are implemented.
|
||||
|
||||
## Changes
|
||||
|
||||
- Extended core job queue types:
|
||||
- `JobStatus` now includes `canceled`.
|
||||
- `JobRecord` now includes optional lease, heartbeat, and cancel timestamps.
|
||||
- `JobQueueAdapter` now exposes `lease()`, `heartbeat()`, `retry()`, `cancel()`, and `status()`.
|
||||
- `JobQueueStats` now includes cumulative `canceled` count.
|
||||
- Updated `createInlineJobQueueAdapter()`:
|
||||
- Added bounded `maxLeaseMs` validation.
|
||||
- `lease()` assigns `leaseExpiresAt` and recovers expired running jobs.
|
||||
- `heartbeat()` only extends the active worker lease.
|
||||
- `retry()` requeues non-terminal jobs with optional `runAfter`.
|
||||
- `cancel()` moves jobs to terminal `canceled` state and participates in bounded terminal retention.
|
||||
- `status()` returns clone-isolated snapshots or `null` for missing/pruned jobs.
|
||||
- Kept internal job records on a stable field shape and avoided `delete` for performance.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- Lease requests require explicit bounded `limit` and `leaseMs`.
|
||||
- Expired lease recovery reuses the same FIFO map walk as dequeue and does not scan terminal jobs as candidates.
|
||||
- Terminal retention remains bounded by `maxRetainedJobs`.
|
||||
- Payload/status snapshots preserve clone isolation.
|
||||
- The inline adapter remains a skeleton/local implementation; real Cloudflare and pg-boss adapters are still separate Phase 3 tasks.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/adapters test -- src/job-queue.test.ts` failed because `queue.lease` did not exist.
|
||||
- Focused verification:
|
||||
- `pnpm --filter @knowledge/adapters test -- src/job-queue.test.ts`
|
||||
- `pnpm --filter @knowledge/core test -- src/platform-adapter.test.ts`
|
||||
- `pnpm --filter @knowledge/adapters typecheck`
|
||||
- `pnpm --filter @knowledge/core typecheck`
|
||||
- `pnpm --filter @knowledge/adapters test:coverage`
|
||||
- `pnpm lint`
|
||||
- Full verification:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Commit Tracking
|
||||
|
||||
- This slice is review checkpoint `92f4e22` + implementation commit 9 after commit and push.
|
||||
- The next implementation commit will trigger the mandatory 10-commit health review after it is committed and pushed.
|
||||
@ -0,0 +1,43 @@
|
||||
# Embedding Provider Interface
|
||||
|
||||
## Summary
|
||||
|
||||
- Added `@knowledge/embeddings` as the provider boundary for external embedding APIs.
|
||||
- This completes Sprint 3 task `1.3.5` without wiring embeddings into index projection persistence yet.
|
||||
|
||||
## Behavior
|
||||
|
||||
- Added `EmbeddingProvider`, `EmbeddingModelInfo`, dense embedding result, and optional sparse vector contracts.
|
||||
- Added OpenAI-compatible, Voyage-compatible, and Cohere-compatible provider factories using injected `fetch`.
|
||||
- Added deterministic `createStaticEmbeddingProvider()` for local tests and future offline skeleton paths.
|
||||
- Provider calls validate model support, batch size, per-text byte limits, response size, response schema, vector count, vector dimension, and sparse-vector shape.
|
||||
|
||||
## Performance And Safety
|
||||
|
||||
- Embedding requests are batched through one provider call instead of per-node request loops.
|
||||
- Inputs require explicit non-empty bounded batches and bounded text bytes.
|
||||
- Provider responses are read with a cumulative byte limit before JSON parsing.
|
||||
- Dense/sparse vectors and model descriptors are cloned on return to avoid retaining mutable caller state.
|
||||
- API keys are only placed in request headers and are not included in return metadata.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED confirmed with embedding tests failing because `packages/embeddings/src/index.ts` did not exist.
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/embeddings test -- src/embedding.test.ts`
|
||||
- `pnpm --filter @knowledge/embeddings test:coverage`
|
||||
- `pnpm --filter @knowledge/embeddings typecheck`
|
||||
- Full verification passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Follow-Up
|
||||
|
||||
- Dense vector projection persistence remains the next Sprint 3 slice.
|
||||
- Runtime environment wiring for real provider credentials should stay separate from the provider contract.
|
||||
@ -0,0 +1,52 @@
|
||||
# EvidenceBundle Assembly
|
||||
|
||||
## Summary
|
||||
|
||||
- Added the first EvidenceBundle assembly boundary for Sprint 6.
|
||||
- Reranked hybrid retrieval candidates can now be converted into structured core `EvidenceBundle` objects.
|
||||
- This slice is pure TypeScript assembly logic and does not introduce database, object storage, network, or cache IO.
|
||||
|
||||
## Changes
|
||||
|
||||
- Added `createEvidenceBundleAssembler()` to `@knowledge/api`.
|
||||
- Added assembly options:
|
||||
- `maxItems`
|
||||
- `maxMissingEvidence`
|
||||
- `generateId`
|
||||
- `now`
|
||||
- Added assembly behavior for:
|
||||
- score breakdowns
|
||||
- citations with source offsets
|
||||
- freshness metadata
|
||||
- structured conflicts
|
||||
- projection/source metadata
|
||||
- missing expected evidence
|
||||
- basic answerability state inference pending the dedicated answerability slice
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- Assembly is bounded by `maxItems` and `maxMissingEvidence`.
|
||||
- It consumes already retrieved/reranked candidates and performs no N+1 database lookups.
|
||||
- It validates through `EvidenceBundleSchema` before returning, keeping downstream generation/cache layers on a stable contract.
|
||||
|
||||
## Verification
|
||||
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm lint`
|
||||
- Full verification passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Review Cadence
|
||||
|
||||
- This slice will be implementation commit 10 after review checkpoint `b7ac774`.
|
||||
- A 10-commit health review is required immediately after this commit is pushed.
|
||||
@ -0,0 +1,51 @@
|
||||
# EvidenceBundle Cache
|
||||
|
||||
## Summary
|
||||
|
||||
- Added a bounded EvidenceBundle cache boundary backed by `CacheAdapter`.
|
||||
- Cache keys include query, permission, strategy, metadata filters, and index projection inputs.
|
||||
|
||||
## Changes
|
||||
|
||||
- Added `createEvidenceBundleCache()`.
|
||||
- Added `EvidenceBundleCacheKeyInput` and `EvidenceBundleCache` contracts.
|
||||
- Cache key inputs include:
|
||||
- `knowledgeSpaceId`
|
||||
- query digest
|
||||
- sorted permission snapshot
|
||||
- retrieval strategy
|
||||
- strategy version
|
||||
- index projection fingerprint
|
||||
- metadata filters
|
||||
- Cache keys use SHA-256 and do not include raw query text.
|
||||
- Cached payloads are validated with `EvidenceBundleSchema`.
|
||||
- Cache hits return clone-isolated `EvidenceBundle` objects.
|
||||
- Corrupt cache payloads return cache miss instead of a half-valid bundle.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- Uses the existing bounded `CacheAdapter`; no database or provider calls were introduced.
|
||||
- Query text is hashed before key construction to avoid long or sensitive cache keys.
|
||||
- Permission snapshots are sorted so equivalent snapshots share the same cache entry.
|
||||
|
||||
## Verification
|
||||
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm lint`
|
||||
- Full verification passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Review Cadence
|
||||
|
||||
- This slice will be implementation commit 4 after review checkpoint `f950b59`.
|
||||
- The next 10-commit review is not due yet.
|
||||
@ -0,0 +1,49 @@
|
||||
# EvidenceBundle Contract
|
||||
|
||||
## Summary
|
||||
|
||||
- Expanded the core EvidenceBundle contract for Sprint 6 evidence assembly.
|
||||
- Evidence items now carry structured scores, citations, conflicts, freshness, and metadata.
|
||||
- Missing evidence is now structured instead of a list of free-form strings.
|
||||
|
||||
## Changes
|
||||
|
||||
- Added `EvidenceScoresSchema`.
|
||||
- Added `EvidenceFreshnessSchema`.
|
||||
- Added `EvidenceConflictSchema`.
|
||||
- Added `MissingEvidenceSchema`.
|
||||
- Extended `CitationSchema` with optional `artifactHash`, `startOffset`, and `endOffset`.
|
||||
- Extended `EvidenceItemSchema` with:
|
||||
- `scores`
|
||||
- `citations`
|
||||
- `conflicts`
|
||||
- `freshness`
|
||||
- `metadata`
|
||||
- Updated `EvidenceBundleSchema.missingEvidence` to use structured missing-evidence entries.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- This slice is a contract-only change; it introduces no database reads, network calls, or runtime loops.
|
||||
- Structured evidence metadata prepares the next assembly layer to avoid re-querying citation and score context later.
|
||||
|
||||
## Verification
|
||||
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/core test -- src/models.test.ts`
|
||||
- `pnpm --filter @knowledge/core typecheck`
|
||||
- `pnpm --filter @knowledge/core test:coverage`
|
||||
- `pnpm lint`
|
||||
- Full verification passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Review Cadence
|
||||
|
||||
- This slice will be implementation commit 9 after review checkpoint `b7ac774`.
|
||||
- The next implementation commit after this slice will trigger the required 10-commit health review.
|
||||
@ -0,0 +1,39 @@
|
||||
# Evidence-Driven Prompt Templates
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `createEvidencePromptTemplateRegistry()` to `@knowledge/generation`.
|
||||
- Added versioned, mode-specific default templates for `fast`, `deep`, and `research` answer generation.
|
||||
- Prompt rendering now returns LLM messages plus template metadata: template id/version, mode, answerability state, evidence counts, omitted counts, and used evidence tokens.
|
||||
- Added bounded prompt inputs with explicit query and evidence-context byte limits.
|
||||
- Added validation for duplicate template modes, blank template ids/versions, unsupported modes, empty rendered messages, invalid roles, and blank message content.
|
||||
|
||||
## Why
|
||||
|
||||
- Sprint 7 generation needs a stable prompt layer between context-window packing and SSE query generation.
|
||||
- Versioned templates make prompt strategy changes auditable and cacheable in later generation cache work.
|
||||
|
||||
## Performance And Safety Notes
|
||||
|
||||
- Rendering is pure in-memory string assembly with no database, cache, object storage, provider, filesystem, or network access.
|
||||
- Query and evidence context are bounded before messages are produced.
|
||||
- Prompt metadata records low-cardinality template state without storing provider credentials or raw request bodies.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/generation test -- src/generation.test.ts`
|
||||
- `pnpm --filter @knowledge/generation typecheck`
|
||||
- `pnpm --filter @knowledge/generation test:coverage`
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- The next slice should wire the rendered messages into the streaming query endpoint.
|
||||
- Citation normalization and generation cache are still planned follow-up slices.
|
||||
@ -0,0 +1,42 @@
|
||||
# Expanded CommandRegistry Guardrails
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added runtime validation for command definitions:
|
||||
- Commands must support at least one resource type.
|
||||
- Supported resource types must be unique.
|
||||
- Cache policy `maxBytes` and `ttlSeconds` must be positive integers when present.
|
||||
- Added runtime validation for command cost estimates:
|
||||
- `estimatedBytes`, `estimatedRows`, and `estimatedMs` must be non-negative finite numbers when present.
|
||||
- Changed execution order so cost estimation is validated before the command handler runs.
|
||||
- Extended trace hook events with `durationMs` and successful `cost` details.
|
||||
- Added TDD coverage for successful trace metadata, invalid cost estimates not invoking handlers, and invalid command definitions.
|
||||
|
||||
## Why
|
||||
|
||||
- Sprint 8 requires the CommandRegistry to be the safe dispatch boundary for KnowledgeFS, SourceFS, MCP, and later safe-shell execution.
|
||||
- Bad command metadata should fail before expensive IO, database access, or side effects begin.
|
||||
- Trace hooks need bounded, low-cardinality timing/cost metadata so later command logs and shell tooling can reason about command behavior.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED:
|
||||
- `pnpm --filter @knowledge/core test -- src/command-registry.test.ts` failed because trace metadata and definition/cost validation were missing.
|
||||
- GREEN focused verification:
|
||||
- `pnpm --filter @knowledge/core test -- src/command-registry.test.ts`
|
||||
- `pnpm --filter @knowledge/core test:coverage`
|
||||
- `pnpm --filter @knowledge/core typecheck`
|
||||
- Full verification:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks And Follow-Up
|
||||
|
||||
- This slice hardens the core registry contract only; SourceFS inspection tools and safe-shell pipeline parsing remain separate Sprint 8 slices.
|
||||
- This commit reaches the 10-commit cadence after checkpoint `50d3a26`, so feature iteration must pause for a project health review immediately after commit and push.
|
||||
49
knowledge-fs/.harness/changes/2026-05-11-fts-projection.md
Normal file
49
knowledge-fs/.harness/changes/2026-05-11-fts-projection.md
Normal file
@ -0,0 +1,49 @@
|
||||
# FTS Projection
|
||||
|
||||
## Summary
|
||||
|
||||
- Added the Sprint 3 full-text projection boundary.
|
||||
- `KnowledgeNode` text can now be persisted as ready `fts` `IndexProjection` rows for database-native exact search.
|
||||
|
||||
## Behavior
|
||||
|
||||
- Added `createFtsProjectionBuilder()` to convert bounded node batches into ready FTS projections.
|
||||
- Projection metadata records FTS text, parser marker, artifact hash, document asset id, and parse artifact id.
|
||||
- Database-backed projection persistence now writes `fts_document` separately from JSON metadata.
|
||||
- PostgreSQL insert SQL uses `to_tsvector('simple', $n)` while TiDB keeps text in a FULLTEXT-indexed column.
|
||||
|
||||
## Performance And Safety
|
||||
|
||||
- FTS projection building is batch-based and reuses the existing `IndexProjectionRepository`.
|
||||
- Database writes remain a single parameterized batch insert.
|
||||
- Schema now includes database-native FTS storage:
|
||||
- PostgreSQL `tsvector` with GIN index.
|
||||
- TiDB `TEXT` with FULLTEXT index.
|
||||
- Ready projection listing remains bounded and keyset-paginated by `node_id + id`.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED confirmed with database schema tests failing because `index_projections_fts_document_idx` was missing.
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/database test -- src/schema.test.ts`
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm db:migrations:write`
|
||||
- `pnpm db:migrations:check`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm --filter @knowledge/database test:coverage`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm lint`
|
||||
- Full verification passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Follow-Up
|
||||
|
||||
- Basic hybrid retrieval is the next Sprint 3 slice.
|
||||
- Future retrieval SQL should use the new FTS storage through bounded search methods rather than scanning projection metadata JSON.
|
||||
@ -0,0 +1,42 @@
|
||||
# Generation Cache And Skip Path
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `createGenerationCache()` to `@knowledge/generation`.
|
||||
- Added deterministic generation cache keys for the evidence/template/model/version/parameter combination.
|
||||
- Added `createGenerationSkipPath()` and `GenerationModelUnavailableError`.
|
||||
- Added skip behavior that returns the cloned `EvidenceBundle` when generation budget is exhausted or the selected model is unavailable.
|
||||
- Added cache-hit behavior that returns cached `GenerateTextResult` before calling a provider.
|
||||
- Added tests for cache hits/misses, session-context bypass, malformed/oversized cache entries, cache-key validation, budget skip, model-unavailable skip, and non-skippable provider errors.
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 7 requires generation reuse when the same evidence, prompt template, model version, and generation parameters are used. It also needs a safe fallback path when budget or model availability prevents answer synthesis, so callers can still receive the underlying EvidenceBundle instead of failing the whole query.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- Cache keys are content-addressed and do not include raw query text or raw evidence text.
|
||||
- Cache entries are bounded by `maxEntryBytes` and TTL-bound by `ttlMs`.
|
||||
- Session-context prompts bypass cache reads and writes to avoid unsafe reuse.
|
||||
- Skip decisions happen before provider calls, avoiding unnecessary network and token spend.
|
||||
- The skip path returns cloned EvidenceBundles and cached results so callers cannot mutate retained state.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/generation test -- src/generation.test.ts`: passed.
|
||||
- `pnpm --filter @knowledge/generation typecheck`: passed.
|
||||
- `pnpm --filter @knowledge/generation test:coverage`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `pnpm check`: passed.
|
||||
- `pnpm build`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
- `pnpm wasm:build`: passed.
|
||||
- `pnpm compose:config`: passed.
|
||||
- `docker compose --profile apps config`: passed.
|
||||
- `git diff --check`: passed.
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- The cache/skip utilities are not yet wired into the `/queries` SSE route. A later query-runtime slice should combine retrieval, prompt rendering, cache lookup, streaming generation, citation normalization, and final response metadata.
|
||||
- Cache invalidation currently relies on versioned cache keys and TTL. When production index publication and deletion workflows land, they should invalidate or namespace affected generation caches alongside evidence caches.
|
||||
@ -0,0 +1,40 @@
|
||||
# Generation Cost Tracking
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `createGenerationCostTracker()` to `@knowledge/generation`.
|
||||
- Added model/provider pricing configuration with a required `priceVersion`.
|
||||
- Added `withGenerationCostTracking()` provider wrapper that annotates non-streaming results and streaming terminal events with cost metadata when provider usage is available.
|
||||
- Added cost metadata fields for prompt tokens, completion/output tokens, total tokens, input/output USD cost, total USD cost, provider, model, currency, and pricing version.
|
||||
- Added validation for blank price versions, duplicate prices, missing prices, invalid pricing values, and invalid token usage.
|
||||
|
||||
## Why
|
||||
|
||||
- Sprint 7 requires generated responses to carry retrieval/generation cost breakdowns before query streaming can be production-ready.
|
||||
- Keeping the calculation in the generation package allows API routes, SSE generators, and future caches to share one deterministic cost boundary.
|
||||
|
||||
## Performance And Safety Notes
|
||||
|
||||
- Cost estimation is pure arithmetic over returned provider usage and does not add network, database, cache, filesystem, or object-storage work.
|
||||
- Pricing lookup is a single in-memory map lookup keyed by provider/model.
|
||||
- The wrapper preserves provider streaming behavior and only annotates the terminal `done` event.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/generation test -- src/generation.test.ts`
|
||||
- `pnpm --filter @knowledge/generation typecheck`
|
||||
- `pnpm --filter @knowledge/generation test:coverage`
|
||||
- `pnpm lint`
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- Live provider pricing should be updated through configuration before production use.
|
||||
- The query generator wiring still needs to expose this cost metadata through `POST /queries` terminal SSE events.
|
||||
@ -0,0 +1,51 @@
|
||||
# Golden Question CRUD
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `GoldenQuestionSchema` to `@knowledge/core`.
|
||||
- Added `golden_questions` to the schema catalog with PostgreSQL/TiDB migration artifacts.
|
||||
- Added bounded in-memory and database-backed `GoldenQuestionRepository` implementations.
|
||||
- Added authenticated API routes:
|
||||
- `POST /knowledge-spaces/{id}/golden-questions`
|
||||
- `GET /knowledge-spaces/{id}/golden-questions`
|
||||
- `GET /knowledge-spaces/{id}/golden-questions/{questionId}`
|
||||
- `PATCH /knowledge-spaces/{id}/golden-questions/{questionId}`
|
||||
- `DELETE /knowledge-spaces/{id}/golden-questions/{questionId}`
|
||||
- Added OpenAPI schemas for golden question request and response bodies.
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 4 requires golden question CRUD before the retrieval evaluation MVP can compute recall and citation metrics. Golden questions store human-labeled expected evidence ids and are scoped to a KnowledgeSpace, giving the next evaluation slice a durable source of truth.
|
||||
|
||||
## Performance And Safety
|
||||
|
||||
- List operations require explicit `limit` and use stable keyset pagination by `created_at, id`.
|
||||
- Database reads, updates, and deletes are filtered by `knowledge_space_id`.
|
||||
- Database operations use parameter arrays and never interpolate question text or evidence ids into SQL.
|
||||
- Repository fallbacks are bounded by `maxQuestions` and `maxListLimit`.
|
||||
- API routes validate the authenticated subject's tenant-scoped KnowledgeSpace before accessing golden question rows.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED confirmed with `pnpm --filter @knowledge/api test -- src/gateway.test.ts`; tests failed because golden question routes and repository factories did not exist.
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm --filter @knowledge/core test:coverage`
|
||||
- `pnpm --filter @knowledge/database test:coverage`
|
||||
- `pnpm db:migrations:check`
|
||||
- Full verification passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks And Follow-Up
|
||||
|
||||
- This slice only manages golden questions. It does not run retrieval evaluation yet.
|
||||
- Expected evidence ids currently reference ids generically; the evaluation MVP will define whether they must be node ids, document ids, or mixed evidence ids.
|
||||
- No Admin Console UI is included in this slice; UI work is scheduled later.
|
||||
@ -0,0 +1,43 @@
|
||||
# IndexProjection Versioning
|
||||
|
||||
## Summary
|
||||
|
||||
- Added the first Phase 3 Sprint 10 projection-versioning guardrail.
|
||||
- Dense vector and FTS projection builders can now create non-active `building` candidate versions while existing `ready` projections remain the only rows returned to retrieval callers.
|
||||
|
||||
## Changes
|
||||
|
||||
- Added `ProjectionBuildStatus` with supported build statuses `ready` and `building`.
|
||||
- Extended dense vector and FTS projection build inputs with optional `status`.
|
||||
- Kept default build status as `ready` for existing local/dev and retrieval tests.
|
||||
- Added candidate-version coverage proving a version 2 `building` projection for the same node does not overwrite or appear alongside the version 1 `ready` projection.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- Candidate builds use new projection rows and ids; active ready rows are not overwritten.
|
||||
- `listReadyBySpace()` continues to filter `status === "ready"`, so retrieval does not accidentally read candidate projections.
|
||||
- Runtime validation rejects unsupported build statuses.
|
||||
- Existing bounded batch, bounded list, clone isolation, and database parameterization behavior remains unchanged.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts` failed because candidate version 2 projections were still created with `status: "ready"`.
|
||||
- Focused verification:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- Full verification:
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Commit Tracking
|
||||
|
||||
- This slice is implementation commit 5 after reviewed checkpoint `3b9b4d8` once committed and pushed.
|
||||
- The next 10-commit health review is not yet due.
|
||||
@ -0,0 +1,36 @@
|
||||
# KnowledgeNode Repository
|
||||
|
||||
## Summary
|
||||
|
||||
- Added bounded in-memory and database-backed `KnowledgeNode` repositories.
|
||||
- This completes the Sprint 3 persistence boundary for chunk output without adding API routes yet.
|
||||
|
||||
## Behavior
|
||||
|
||||
- Added `KnowledgeNodeRepository` with `createMany(nodes)` and `listByArtifact({ parseArtifactId, limit, cursor? })`.
|
||||
- In-memory persistence supports bounded batch writes, total node capacity, clone isolation, and stable artifact listing.
|
||||
- Database-backed persistence writes nodes in a single parameterized batch insert and lists by `parse_artifact_id`, `start_offset`, and `id`.
|
||||
- Pagination uses an explicit `{ startOffset, id }` keyset cursor and reads `limit + 1` rows to compute `nextCursor`.
|
||||
|
||||
## Performance And Safety
|
||||
|
||||
- Batch writes reject empty input and batches larger than `maxBatchSize`.
|
||||
- Listing requires explicit `limit` and rejects values above `maxListLimit`.
|
||||
- Database reads pass explicit `maxRows` and rely on the existing `knowledge_nodes_artifact_offset_idx`.
|
||||
- User/node text is kept in SQL params and is not interpolated into SQL strings.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED confirmed with API tests failing because `createInMemoryKnowledgeNodeRepository` did not exist.
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- Full verification passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
@ -0,0 +1,38 @@
|
||||
# KnowledgeFS `cat` and `stat` Endpoints
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added authenticated `GET /knowledge-spaces/{id}/fs/cat`.
|
||||
- Added authenticated `GET /knowledge-spaces/{id}/fs/stat`.
|
||||
- Wired both commands through `CommandRegistry`.
|
||||
- Added exact path resolution over `KnowledgePathRepository.get()`.
|
||||
- Extended `KnowledgeNodeRepository` with tenant/space-scoped `get({ id, knowledgeSpaceId })`.
|
||||
- Implemented `cat` for:
|
||||
- document paths backed by bounded object storage reads,
|
||||
- knowledge node paths backed by direct node lookup.
|
||||
- Implemented `stat` for:
|
||||
- path metadata,
|
||||
- document asset size, hash, MIME type, parser status, and version.
|
||||
- Added OpenAPI schemas and tests for document/node cat, stat, missing paths, missing targets, unsupported cat targets, and database-backed node lookup.
|
||||
|
||||
## Why It Changed
|
||||
|
||||
- Sprint 4 requires `cat` and `stat` after `ls/tree` so agents can inspect KnowledgeFS leaves without jumping into semantic retrieval.
|
||||
- The implementation keeps filesystem semantics behind the same command registry used by `ls/tree`.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- Exact reads perform one path lookup plus at most one target lookup.
|
||||
- Database-backed node reads use parameterized SQL with `knowledge_space_id + id` and `maxRows: 1`.
|
||||
- Object content reads stay behind `ObjectStorageAdapter`, which already enforces bounded object reads.
|
||||
- No list scan, N+1 query loop, or host shell execution path was introduced.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- `cat` currently supports document objects and knowledge nodes. Parse artifacts, table JSON/HTML, page files, and metadata-specific virtual files remain follow-up work.
|
||||
- Binary document rendering is not specialized yet; this slice returns decoded text for object-backed document paths.
|
||||
@ -0,0 +1,40 @@
|
||||
# KnowledgeFS Diff And Open Node
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `diffText(input)` to `@knowledge/compute` and optional `diffTextJson` support on the injected WASM module.
|
||||
- Added validated `TextDiff` output contracts for merged `equal`, `insert`, and `delete` operations.
|
||||
- Added authenticated `GET /knowledge-spaces/{id}/fs/diff`.
|
||||
- Added authenticated `GET /knowledge-spaces/{id}/fs/open_node`.
|
||||
- Registered `diff` and `open_node` in the KnowledgeFS `CommandRegistry`.
|
||||
- Added OpenAPI path coverage and API tests for successful diff, citation-ready node fetch, cross-tenant hiding, missing nodes, missing diff paths, and missing compute runtime.
|
||||
|
||||
## Why
|
||||
|
||||
- Sprint 8 requires version diff and citation-ready node fetches before completing the KnowledgeFS command surface.
|
||||
- Diff stays behind the TypeScript compute boundary so API code does not depend directly on Rust/WASM details.
|
||||
- `open_node` uses a single tenant-scoped node lookup and returns source-location citation data that later MCP and shell tools can reuse.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED:
|
||||
- `pnpm --filter @knowledge/compute test -- src/compute.test.ts` failed because `runtime.diffText` was missing.
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts` failed because `/fs/diff` and `/fs/open_node` were not implemented.
|
||||
- GREEN focused verification:
|
||||
- `pnpm --filter @knowledge/compute test:coverage`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm --filter @knowledge/generation typecheck`
|
||||
- Full verification:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks And Follow-Up
|
||||
|
||||
- Diff currently compares already-readable path content in memory and relies on object-storage and WASM diff bounds; very large semantic diff flows should add streaming or pre-windowing before lifting those bounds.
|
||||
- The next slice should continue with the expanded CommandRegistry/MCP command exposure and should trigger the required 10-commit health review after one more implementation commit.
|
||||
@ -0,0 +1,39 @@
|
||||
# KnowledgeFS Find Endpoint
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added authenticated `GET /knowledge-spaces/{id}/fs/find`.
|
||||
- Added `find` to the KnowledgeFS command registry.
|
||||
- Added scoped path search filters for `resourceType`, `nameContains`, and metadata key/value.
|
||||
- Reused KnowledgeFS list response entries so callers get familiar resource path metadata.
|
||||
- Added tests for metadata-filtered results, pagination, empty filters, and invalid metadata filter shape.
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 8 requires a metadata-aware KnowledgeFS find command for agents and future MCP tools. It lets callers discover resources under a physical KnowledgeFS path without unbounded listing or host shell access.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- The endpoint requires explicit `limit`.
|
||||
- Search is tenant-scoped through the KnowledgeSpace lookup and path-scoped through physical KnowledgeFS descendants.
|
||||
- Results use existing stable KnowledgePath cursors.
|
||||
- Filtering is bounded by the path repository page size and never scans the whole workspace in one call.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`: passed.
|
||||
- `pnpm --filter @knowledge/api typecheck`: passed.
|
||||
- `pnpm --filter @knowledge/api test:coverage`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `pnpm check`: passed.
|
||||
- `pnpm build`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
- `pnpm wasm:build`: passed.
|
||||
- `pnpm compose:config`: passed.
|
||||
- `docker compose --profile apps config`: passed.
|
||||
- `git diff --check`: passed.
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- Current filtering is bounded in the command layer. A later database-backed find path should push resource type/name/metadata predicates into indexed SQL for very large workspaces.
|
||||
@ -0,0 +1,42 @@
|
||||
# KnowledgeFS Grep Endpoint
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added authenticated `GET /knowledge-spaces/{id}/fs/grep`.
|
||||
- Added `grep` to the KnowledgeFS command registry.
|
||||
- Added `KnowledgeFsGrepResult` and match response schemas.
|
||||
- Extended `KnowledgeNodeRepository` with `getMany()` so grep can batch node hydration.
|
||||
- Implemented scoped grep over physical KnowledgeFS descendants and node text.
|
||||
- Added tests for tenant-scoped, paginated grep matches and no-match pagination behavior.
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 8 starts by completing agent-facing KnowledgeFS search. `grep` gives callers a bounded way to inspect exact text matches under a KnowledgeFS physical path without using raw host shell commands.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- The endpoint requires explicit `limit`.
|
||||
- Path enumeration uses existing stable KnowledgePath cursors.
|
||||
- Node hydration is batched with `getMany()` instead of one repository call per path.
|
||||
- The search is scoped by tenant-validated KnowledgeSpace id and physical KnowledgeFS path.
|
||||
- The implementation is bounded by the path repository list limit and optional `timeoutMs`.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`: passed.
|
||||
- `pnpm --filter @knowledge/api typecheck`: passed.
|
||||
- `pnpm --filter @knowledge/api test:coverage`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `pnpm check`: passed.
|
||||
- `pnpm build`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
- `pnpm wasm:build`: passed.
|
||||
- `pnpm compose:config`: passed.
|
||||
- `docker compose --profile apps config`: passed.
|
||||
- `git diff --check`: passed.
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- The current in-memory/default command uses bounded path enumeration plus batched node reads. A later database-runtime wiring slice should replace this with a database FTS-backed grep repository for large corpora.
|
||||
- Snippets currently return the matching node text. Future UX work may add configurable context windows and highlighted ranges.
|
||||
@ -0,0 +1,40 @@
|
||||
# KnowledgeFS `ls` and `tree` Endpoints
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added authenticated `GET /knowledge-spaces/{id}/fs/ls`.
|
||||
- Added authenticated `GET /knowledge-spaces/{id}/fs/tree`.
|
||||
- Wired both endpoints through the core `CommandRegistry` with `ls` and `tree` command handlers.
|
||||
- Extended `KnowledgePathRepository` with bounded `listPhysicalDescendants()` for prefix-scoped physical view traversal.
|
||||
- Implemented in-memory and database-backed descendant listing over the existing `knowledge_paths_space_view_path_idx` access pattern.
|
||||
- Added OpenAPI response schemas for KnowledgeFS list and tree responses.
|
||||
- Added tests for:
|
||||
- OpenAPI path exposure,
|
||||
- tenant-scoped `ls` and `tree`,
|
||||
- bounded list/tree limits,
|
||||
- missing tenant-space hiding,
|
||||
- invalid path and cursor rejection,
|
||||
- repository prefix listing with parameterized SQL and explicit `maxRows`.
|
||||
|
||||
## Why It Changed
|
||||
|
||||
- Sprint 4 requires paginated KnowledgeFS directory listing before later `cat`, `stat`, MCP, and safe shell work can reuse the same filesystem command surface.
|
||||
- The implementation keeps route behavior behind `CommandRegistry` instead of adding scattered command semantics.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- `ls` and `tree` require explicit `limit`.
|
||||
- Repository reads use `limit + 1` with `maxRows` to detect truncation without unbounded reads.
|
||||
- Database-backed prefix traversal remains indexed by `knowledge_space_id`, `view_type`, `view_name`, `virtual_path`, and `id`.
|
||||
- SQL uses parameter arrays; user-provided paths/cursors are not interpolated into SQL.
|
||||
- The current tree endpoint builds only from a bounded descendant page, not from a full corpus scan.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- Directory pagination currently follows underlying `KnowledgePath` cursor semantics; later large-directory work may add directory-level cursors to avoid repeated virtual directory entries across pages.
|
||||
- `cat` and `stat` endpoints remain the next Sprint 4 slice.
|
||||
@ -0,0 +1,41 @@
|
||||
# KnowledgeFS Resource Model
|
||||
|
||||
## What Changed
|
||||
|
||||
- Extended `KnowledgePath` with physical/semantic view metadata:
|
||||
- `viewType`
|
||||
- `viewName`
|
||||
- `metadata`
|
||||
- Added checked-in PostgreSQL/TiDB migration columns for KnowledgeFS path views.
|
||||
- Added `knowledge_paths_space_view_path_idx` on `knowledge_space_id + view_type + view_name + virtual_path + id`.
|
||||
- Added bounded in-memory and database-backed `KnowledgePathRepository` implementations.
|
||||
- Added physical-view listing with explicit limits and stable `virtualPath + id` keyset pagination.
|
||||
|
||||
## Why
|
||||
|
||||
- Sprint 3/Phase 1 needs KnowledgeFS virtual path records that can expose physical views such as `/by-source`, `/by-type`, `/by-time`, and `/by-owner`.
|
||||
- The repository boundary gives later filesystem commands an indexed, tenant-scoped path model without per-resource query waterfalls.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/core test -- src/models.test.ts`: passed.
|
||||
- `pnpm --filter @knowledge/database test -- src/schema.test.ts`: passed.
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`: passed.
|
||||
- `pnpm db:migrations:write`: passed.
|
||||
- `pnpm db:migrations:check`: passed.
|
||||
- `pnpm --filter @knowledge/api test:coverage`: passed.
|
||||
- `pnpm --filter @knowledge/core test:coverage`: passed.
|
||||
- `pnpm --filter @knowledge/database test:coverage`: passed.
|
||||
- `pnpm check`: passed.
|
||||
- `pnpm build`: passed.
|
||||
- `pnpm lint`: passed.
|
||||
- `cargo test --workspace`: passed.
|
||||
- `pnpm wasm:build`: passed.
|
||||
- `pnpm compose:config`: passed.
|
||||
- `docker compose --profile apps config`: passed.
|
||||
- `git diff --check`: passed.
|
||||
|
||||
## Known Risks And Follow-Up
|
||||
|
||||
- This slice models and persists KnowledgeFS paths; it does not yet implement SourceFS/EvidenceFS namespaces, `ResourceMount`, or filesystem commands.
|
||||
- The next implementation commit reaches the 10-commit review cadence after checkpoint `0105450`, so feature iteration must pause for health review after commit and push.
|
||||
@ -0,0 +1,37 @@
|
||||
# LLM Provider Interface
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `@knowledge/generation` as the provider boundary for answer generation.
|
||||
- Added provider-agnostic contracts for messages, generation results, model descriptors, usage metadata, and streaming events.
|
||||
- Added `createOpenAIChatProvider()` for OpenAI-compatible chat completions.
|
||||
- Added `createAnthropicMessagesProvider()` for Anthropic/Claude-compatible messages.
|
||||
- Added `createStaticLlmProvider()` for deterministic tests and local fallback.
|
||||
- Added coverage gates for the new package.
|
||||
|
||||
## Why
|
||||
|
||||
- Sprint 7 needs LLM generation behind one TypeScript interface before routing, evidence packing, prompt templates, streaming query APIs, and cost tracking can be wired.
|
||||
- The interface keeps provider details out of retrieval/evidence code and preserves the project rule that external API providers stay behind TypeScript adapters.
|
||||
|
||||
## Performance And Safety Notes
|
||||
|
||||
- Message count, message byte size, output token budget, and provider response byte size are bounded.
|
||||
- Streaming and non-streaming paths share the same input validation.
|
||||
- Provider responses are Zod-validated and fail closed on non-2xx responses, malformed JSON, malformed payloads, oversized responses, and malformed stream events.
|
||||
- The stream event surface emits only deltas and bounded metadata; it does not include API keys, raw request headers, JWTs, or full request bodies.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm install`
|
||||
- `pnpm --filter @knowledge/generation test -- src/generation.test.ts`
|
||||
- `pnpm --filter @knowledge/generation typecheck`
|
||||
- `pnpm --filter @knowledge/generation test:coverage`
|
||||
- `pnpm lint`
|
||||
|
||||
Full workspace verification is recorded in `TEMP-progress-document.md` after completion.
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- This slice implements provider contracts and SDK-free HTTP wiring only; generation routing, evidence packing, prompt templates, SSE query endpoints, cost tracking, and citation normalization remain later Sprint 7 work.
|
||||
- The HTTP payloads intentionally target stable OpenAI-compatible chat completions and Anthropic-compatible messages shapes; live provider integration tests are deferred until runtime secrets and provider selection are wired.
|
||||
33
knowledge-fs/.harness/changes/2026-05-11-llm-routing.md
Normal file
33
knowledge-fs/.harness/changes/2026-05-11-llm-routing.md
Normal file
@ -0,0 +1,33 @@
|
||||
# LLM Routing
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `createLlmRouter()` to `@knowledge/generation`.
|
||||
- Added fast/deep/research route modes with configurable provider, model, output-token ceiling, and optional temperature.
|
||||
- Added routed non-streaming generation that annotates result metadata with route mode, policy version, and provider key.
|
||||
- Added routed streaming generation that passes delta events through and annotates terminal events with routing metadata.
|
||||
- Added tests for valid routing, stream routing, and invalid policy configuration.
|
||||
|
||||
## Why
|
||||
|
||||
- Sprint 7 requires generation model selection to be policy-driven before evidence packing, prompt templates, streaming query APIs, and cost tracking are wired.
|
||||
- Retrieval/generation modes should choose provider/model policy centrally rather than scattering model conditionals through gateway code.
|
||||
|
||||
## Performance And Safety Notes
|
||||
|
||||
- Route selection is pure in-memory config lookup and does not perform network, database, or object-storage work.
|
||||
- Policy output-token ceilings are applied before provider calls; caller-provided lower limits are allowed, but higher limits are clamped to policy.
|
||||
- Invalid policies fail during router creation where possible, and missing mode policies fail before any provider call.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/generation test -- src/generation.test.ts`
|
||||
- `pnpm --filter @knowledge/generation typecheck`
|
||||
- `pnpm --filter @knowledge/generation test:coverage`
|
||||
|
||||
Full workspace verification is recorded in `TEMP-progress-document.md` after completion.
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- Runtime configuration wiring is not included yet; current router is an injectable library boundary.
|
||||
- Later Sprint 7 work must connect this router to evidence packing, prompt templates, SSE query endpoints, and generation cost tracking.
|
||||
@ -0,0 +1,50 @@
|
||||
# MCP KnowledgeFS Tools
|
||||
|
||||
## What Changed
|
||||
|
||||
- Expanded `createKnowledgeMcpServer()` from `knowledge.fs.ls`, `knowledge.fs.cat`, and `knowledge.search` to the full KnowledgeFS tool set:
|
||||
- `knowledge.fs.ls`
|
||||
- `knowledge.fs.tree`
|
||||
- `knowledge.fs.cat`
|
||||
- `knowledge.fs.grep`
|
||||
- `knowledge.fs.find`
|
||||
- `knowledge.fs.stat`
|
||||
- `knowledge.fs.diff`
|
||||
- `knowledge.fs.open_node`
|
||||
- `knowledge.search`
|
||||
- Added MCP input schemas for grep, find, diff, and open_node.
|
||||
- Reused the MCP filesystem list bound for `ls`, `tree`, `grep`, and `find`.
|
||||
- Extended MCP tests to verify registration, structured tool output, dispatch to injected handlers, unknown-tool rejection, and limit rejection.
|
||||
|
||||
## Why
|
||||
|
||||
- Phase 2 Sprint 8 requires MCP KnowledgeFS tools so agents can use the same filesystem command semantics through MCP as through the API and CommandRegistry boundaries.
|
||||
|
||||
## Performance And Safety Notes
|
||||
|
||||
- MCP list-like tools require explicit positive limits.
|
||||
- `maxFsListLimit` protects list, tree, grep, and find from unbounded result windows.
|
||||
- MCP handlers stay injected and do not perform direct storage/database work; the actual filesystem behavior remains behind existing repository and CommandRegistry boundaries.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/api test -- src/mcp.test.ts` failed because only `knowledge.fs.ls`, `knowledge.fs.cat`, and `knowledge.search` were registered.
|
||||
- Focused verification:
|
||||
- `pnpm --filter @knowledge/api test -- src/mcp.test.ts`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm lint`
|
||||
- Full verification:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- This slice completes MCP KnowledgeFS tools only. MCP retrieval evidence and shell tools remain in the next Sprint 8 MCP slice.
|
||||
@ -0,0 +1,41 @@
|
||||
# MCP Retrieval And Shell Tools
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added MCP `knowledge.fetch_evidence` with bounded `topK` validation.
|
||||
- Added MCP `knowledge.shell.plan` and `knowledge.shell.execute` tools.
|
||||
- MCP retrieval and shell tools dispatch to injected handlers; MCP remains a schema/guard boundary and does not perform retrieval, storage, or host shell execution directly.
|
||||
- Extended MCP tests to cover tool registration, structured evidence output, shell plan/execute output, unknown-tool rejection, `topK` bounds, and invalid shell command input.
|
||||
|
||||
## Why
|
||||
|
||||
- Phase 2 Sprint 8 requires retrieval and shell MCP tools so agents can access evidence and safe shell ergonomics through the same MCP server surface.
|
||||
|
||||
## Performance And Safety Notes
|
||||
|
||||
- `knowledge.fetch_evidence` reuses `maxSearchTopK` to prevent unbounded evidence fanout through MCP.
|
||||
- Shell MCP inputs require a non-empty bounded command string.
|
||||
- Shell execution remains delegated to the SafeShell boundary; no host shell or process execution is introduced.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/api test -- src/mcp.test.ts` failed because `knowledge.fetch_evidence`, `knowledge.shell.plan`, and `knowledge.shell.execute` were not registered.
|
||||
- Focused verification:
|
||||
- `pnpm --filter @knowledge/api test -- src/mcp.test.ts`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm lint`
|
||||
- Full verification:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- This slice exposes MCP hooks only. Gateway runtime wiring from authenticated subjects to per-request SafeShell instances can be added when MCP transport/auth integration is finalized.
|
||||
@ -0,0 +1,39 @@
|
||||
# MCP Server Skeleton
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `@modelcontextprotocol/sdk` to `@knowledge/api`.
|
||||
- Added `createKnowledgeMcpServer()` with an SDK-backed `McpServer` instance.
|
||||
- Registered Phase 1 MCP tools:
|
||||
- `knowledge.fs.ls`
|
||||
- `knowledge.fs.cat`
|
||||
- `knowledge.search`
|
||||
- Added a deterministic `listTools()` / `callTool()` wrapper for focused tests and future transport wiring.
|
||||
- Added bounded Zod validation for knowledge space ids, KnowledgeFS paths, list limits, and search `topK`.
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 4 requires MCP and OpenAPI to be first-class access surfaces. This slice establishes the MCP contract without creating a second data access implementation. Tool calls delegate to injected KnowledgeFS/search handlers so the gateway can reuse existing tenant-scoped, bounded repository paths.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED confirmed with `pnpm --filter @knowledge/api test -- src/mcp.test.ts`; tests failed because `createKnowledgeMcpServer` did not exist.
|
||||
- GREEN focused verification passed:
|
||||
- `pnpm --filter @knowledge/api test -- src/mcp.test.ts`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm --filter @knowledge/api build`
|
||||
- Full verification passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks And Follow-Up
|
||||
|
||||
- This slice does not expose an HTTP or stdio MCP transport yet. It only creates the registered server boundary for later runtime wiring.
|
||||
- `knowledge.search` currently delegates to an injected search handler; later retrieval planner work should map MCP search directly onto the production retrieval runtime.
|
||||
- MCP auth/session binding is not implemented in this slice. Transport wiring must preserve tenant subject and permission checks.
|
||||
53
knowledge-fs/.harness/changes/2026-05-11-metadata-filters.md
Normal file
53
knowledge-fs/.harness/changes/2026-05-11-metadata-filters.md
Normal file
@ -0,0 +1,53 @@
|
||||
# Metadata Filters
|
||||
|
||||
## Summary
|
||||
|
||||
- Added retrieval metadata filters for Sprint 6.
|
||||
- Filters are applied before RRF fusion and reranking.
|
||||
|
||||
## Changes
|
||||
|
||||
- Added `RetrievalMetadataFilters` to hybrid retrieval inputs.
|
||||
- Supported filters:
|
||||
- `documentTypes`
|
||||
- `sourceIds`
|
||||
- `createdAfter` / `createdBefore`
|
||||
- `entities`
|
||||
- `tags`
|
||||
- `languages`
|
||||
- `freshnessStatuses`
|
||||
- `nodeKinds`
|
||||
- Passed filters through dense-vector and FTS repository calls.
|
||||
- Extended retrieval SQL to join `document_assets` in the same bounded query.
|
||||
- Pushed indexed filters for node kind, document MIME type, source id, and document created-at range into SQL.
|
||||
- Added bounded in-memory candidate filtering before fusion/reranking for metadata fields and safety.
|
||||
- Added `metadataFilteredCandidates` metric when filters remove candidates.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- Retrieval remains two bounded database queries: dense and FTS.
|
||||
- No post-retrieval N+1 document lookups were introduced.
|
||||
- SQL-pushed filters use joined document/node fields already available on the retrieval path.
|
||||
- Metadata fallback filtering runs only over bounded candidate arrays before fusion and reranking.
|
||||
|
||||
## Verification
|
||||
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm lint`
|
||||
- Full verification passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Review Cadence
|
||||
|
||||
- This slice will be implementation commit 3 after review checkpoint `f950b59`.
|
||||
- The next 10-commit review is not due yet.
|
||||
@ -0,0 +1,49 @@
|
||||
# Next.js Admin Console Initialization
|
||||
|
||||
## What Changed
|
||||
|
||||
- Replaced the Admin placeholder page with an operational KnowledgeFS Admin shell.
|
||||
- Added dashboard sections for:
|
||||
- System health
|
||||
- Upload intake
|
||||
- Publish readiness
|
||||
- Retrieval workspace
|
||||
- Trace review
|
||||
- Added responsive global CSS for a dense admin/workflow UI.
|
||||
- Added a render test for the Admin home page.
|
||||
- Added `next.config.ts` with standalone output.
|
||||
- Changed the Admin build script from TypeScript-only checking to `next build`.
|
||||
- Ignored TypeScript incremental build info generated by Next/tsc.
|
||||
|
||||
## Why
|
||||
|
||||
- Phase 2 Sprint 9 requires the Next.js Admin Console to run in dev and bundle for Standalone deployment.
|
||||
|
||||
## Performance And Safety Notes
|
||||
|
||||
- The page is currently a static server-rendered shell with no client-side state or extra client components.
|
||||
- The Admin bundle is generated with Next standalone output for later Docker/Standalone packaging.
|
||||
- UI text stays operational and low-cardinality; no API tokens, JWTs, document contents, or query traces are embedded.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/admin test` failed because the page still rendered placeholder content.
|
||||
- Focused verification:
|
||||
- `pnpm --filter @knowledge/admin test`
|
||||
- `pnpm --filter @knowledge/admin typecheck`
|
||||
- `pnpm --filter @knowledge/admin build`
|
||||
- `curl -fsS http://127.0.0.1:3001 | rg "KnowledgeFS Admin|System health|Upload intake|Retrieval workspace|Trace review"`
|
||||
- Full verification:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- The shell is static in this slice. API client wiring and live data are the next Sprint 9 tasks.
|
||||
@ -0,0 +1,41 @@
|
||||
# Optimized Hybrid Recall
|
||||
|
||||
## What Changed
|
||||
|
||||
- Extended `createBasicHybridRetriever()` with optional planner and fusion runtime injection.
|
||||
- Planner-backed retrieval now uses mode-aware dense/FTS fanout for recall while keeping final output bounded by request `limit`.
|
||||
- Added a WASM-compatible RRF fusion path using ranked dense and FTS lists.
|
||||
- Added retrieval `plan` and latency/candidate `metrics` to retriever results.
|
||||
- Kept the existing local RRF fusion path as a fallback when no fusion runtime is injected.
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 5 needs production retrieval behavior where recall fanout is larger than final answer count, mode choices are explicit, and fusion can use the Rust/WASM RRF primitive added in the previous slice.
|
||||
|
||||
## Performance And Safety Notes
|
||||
|
||||
- Dense and FTS repository searches still run in parallel with `Promise.all`.
|
||||
- All search fanout values come from a bounded `RetrievalPlan`; repository calls receive explicit `topK`/`maxRows`.
|
||||
- Fusion input is bounded with max lists, max items per list, max output items, and max JSON input bytes.
|
||||
- Metrics expose candidate counts and elapsed durations without raw queries, document text, vectors, or credentials.
|
||||
- No additional database round trips were introduced.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- Runtime wiring still needs to provide a real WASM compute module in application composition.
|
||||
- Reranker provider integration is still a separate Sprint 5 slice.
|
||||
- Evaluation comparison between dense-only, FTS-only, and hybrid modes is still pending.
|
||||
@ -0,0 +1,42 @@
|
||||
# Permission Filtering
|
||||
|
||||
## Summary
|
||||
|
||||
- Added permission-aware filtering to hybrid retrieval.
|
||||
- Unauthorized candidates are removed before RRF fusion and reranking.
|
||||
|
||||
## Changes
|
||||
|
||||
- Extended retrieval candidates with node `permissionScope`.
|
||||
- Extended hybrid retrieval input with optional caller `permissionScope`.
|
||||
- Selected `knowledge_nodes.permission_scope` in dense-vector and FTS retrieval SQL through the existing joined query.
|
||||
- Filtered protected candidates before WASM/native RRF fusion.
|
||||
- Preserved public candidates whose node permission scope is empty.
|
||||
- Added `permissionFilteredCandidates` to retrieval metrics when filtering removes candidates.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- Filtering runs over already bounded dense and FTS candidate arrays.
|
||||
- No extra database query, object storage read, cache read, or provider call was introduced.
|
||||
- Retrieval SQL still uses the same bounded joins and explicit `maxRows` limits.
|
||||
|
||||
## Verification
|
||||
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- Full verification passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Review Cadence
|
||||
|
||||
- This slice will be implementation commit 2 after review checkpoint `f950b59`.
|
||||
- The next 10-commit review is not due yet.
|
||||
@ -0,0 +1,49 @@
|
||||
# pg-boss Job Queue Adapter
|
||||
|
||||
## Summary
|
||||
|
||||
- Added a Standalone-oriented `pg-boss` job queue adapter boundary for Phase 3 durable ingestion.
|
||||
- Wired the Node platform factory to use the pg-boss adapter when a boss client is injected, while keeping the bounded inline fallback for local/dev without a database-backed job runtime.
|
||||
|
||||
## Changes
|
||||
|
||||
- Added `createPgBossJobQueueAdapter()` in `@knowledge/adapters`.
|
||||
- Added portable `PgBossClient` contract with `send`, optional `complete`, optional `fail`, and optional `cancel`.
|
||||
- `enqueue()` sends compact pg-boss payloads with job id, type, attempts, and optional idempotency key.
|
||||
- `runAfter` maps to pg-boss `startAfter`; `idempotencyKey` maps to `singletonKey`.
|
||||
- `retry()` re-delivers jobs through pg-boss and updates the external job id.
|
||||
- `complete()`, `fail()`, and `cancel()` forward lifecycle calls to pg-boss when the corresponding client method exists.
|
||||
- Added `externalJobId` to `JobRecord` for adapter-level provider job correlation.
|
||||
- `createNodePlatformAdapter()` accepts `jobBoss` injection and switches `jobs.kind` to `pg-boss`.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- pg-boss messages do not include raw document bytes or large payload bodies.
|
||||
- Duplicate idempotency-key enqueue calls do not send duplicate pg-boss jobs.
|
||||
- Initial delivery failure cancels the inline state and clears local idempotency mapping.
|
||||
- Retry delivery failure fails closed instead of leaving queued work without a delivery event.
|
||||
- Inline fallback remains bounded by batch, queue, lease, and retention limits.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/adapters test -- src/adapters.test.ts` failed because `./pg-boss-job-queue` did not exist.
|
||||
- Focused verification:
|
||||
- `pnpm --filter @knowledge/adapters test -- src/adapters.test.ts`
|
||||
- `pnpm --filter @knowledge/adapters typecheck`
|
||||
- `pnpm --filter @knowledge/adapters test:coverage`
|
||||
- `pnpm lint`
|
||||
- Full verification:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Commit Tracking
|
||||
|
||||
- This slice is implementation commit 1 after reviewed checkpoint `3b9b4d8` once committed and pushed.
|
||||
- The next 10-commit health review is not yet due.
|
||||
@ -0,0 +1,52 @@
|
||||
# Phase 1 End-to-End Integration Test
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `packages/api/src/phase1-e2e.test.ts`.
|
||||
- The E2E test covers the Phase 1 critical path:
|
||||
- Authenticated PDF upload through the Hono gateway.
|
||||
- Parser-produced `ParseArtifact` persistence.
|
||||
- WASM runtime chunking through `@knowledge/compute`.
|
||||
- `KnowledgeNode` persistence.
|
||||
- Dense vector and FTS projection building.
|
||||
- Hybrid retrieval with citation source location.
|
||||
- Golden-question retrieval evaluation over the retrieved evidence.
|
||||
- Added `@knowledge/compute` as an API package dev dependency for the integration test.
|
||||
- Updated dense projection building to pass `inputType: "search_document"` explicitly when embedding document chunks.
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 4 requires a minimum end-to-end proof that the Phase 1 components can work together before moving into packaging and later production-hardening work. The test locks down the upload -> parse -> chunk -> index -> retrieve -> cite loop without requiring live Docker services or external parser/embedding providers.
|
||||
|
||||
## Performance And Safety
|
||||
|
||||
- The integration test keeps all repositories bounded.
|
||||
- Upload uses the existing bounded gateway upload path.
|
||||
- Chunking passes through the WASM runtime JSON boundary and validates `KnowledgeNodeSchema`.
|
||||
- Dense indexing embeds all stored nodes in one batch instead of per-node calls.
|
||||
- Retrieval work remains bounded by explicit `topK` and `limit`.
|
||||
- The E2E retrieval fixture uses an in-memory projection scan over one bounded fixture and does not introduce a production query path.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED confirmed with `pnpm --filter @knowledge/api test -- src/phase1-e2e.test.ts`; the new test failed before `@knowledge/compute` was declared in `@knowledge/api`.
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/api test -- src/phase1-e2e.test.ts src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- Full verification passed:
|
||||
- `pnpm install`
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks And Follow-Up
|
||||
|
||||
- This is an in-process E2E test using fake parser and embedding providers. A live Docker/MinIO/PostgreSQL/Unstructured smoke remains a later environment-dependent gate.
|
||||
- The test exercises the TypeScript WASM runtime boundary with a deterministic module fixture rather than loading the generated WASM package directly.
|
||||
- The next Sprint 4 item is standalone Docker image packaging.
|
||||
@ -0,0 +1,30 @@
|
||||
# Production Deployment Guide
|
||||
|
||||
## Summary
|
||||
|
||||
- Added `docs/production-deployment.md` for Phase 2 Sprint 9 production polish.
|
||||
- Documented SaaS and Standalone deployment shapes while preserving the required Next.js Admin / Hono API separation.
|
||||
- Linked the guide from the root README.
|
||||
|
||||
## Changes
|
||||
|
||||
- Added release gates for TypeScript, coverage, retrieval regression, Rust, WASM, Compose config, and diff hygiene.
|
||||
- Documented runtime configuration for auth, object storage, database/cache, Unstructured, and Admin Console.
|
||||
- Covered Cloudflare Pages + Workers deployment expectations for SaaS, including R2/KV/TiDB/Unstructured service provisioning and expected `wrangler` shape.
|
||||
- Covered Docker Compose deployment for Standalone, including API/Admin service separation, MinIO bucket bootstrap, PostgreSQL migrations, and smoke checks.
|
||||
- Added rollback and operational guardrails for bounded reads, cache keys, tenant safety, traces/logging, and retrieval regression gates.
|
||||
- Recorded current deployment automation gaps so the guide does not imply unsupported runtime wiring already exists.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `test -f docs/production-deployment.md` failed because the production deployment guide did not exist.
|
||||
- Documentation verification:
|
||||
- `rg -n "Cloudflare Pages|Cloudflare Workers|wrangler|Standalone|Docker Compose|Hono API|Next.js Admin" docs/production-deployment.md README.md`
|
||||
- `pnpm lint`
|
||||
- `git diff --check`
|
||||
|
||||
## Commit Tracking
|
||||
|
||||
- This slice is review checkpoint `92f4e22` + implementation commit 8 after commit and push.
|
||||
- The next 10-commit health review is not yet due.
|
||||
@ -0,0 +1,51 @@
|
||||
# Query Normalization Cache
|
||||
|
||||
## Summary
|
||||
|
||||
- Added a bounded query normalization cache for Phase 2 Sprint 5 retrieval hardening.
|
||||
- The cache reuses the shared `CacheAdapter` contract and stores normalized query metadata behind strategy-versioned SHA-256 keys.
|
||||
- Raw query text is intentionally excluded from cache keys.
|
||||
|
||||
## Changes
|
||||
|
||||
- Added `createQueryNormalizationCache()` to `@knowledge/api`.
|
||||
- Normalization output includes:
|
||||
- `normalizedQuery`
|
||||
- `queryLanguage`
|
||||
- `strategyVersion`
|
||||
- `cacheHit`
|
||||
- Cache entries are written with explicit TTL.
|
||||
- Cache reads validate payload shape and reject corrupted entries.
|
||||
- Inputs and config are bounded:
|
||||
- empty query rejected
|
||||
- oversized query rejected by `maxQueryBytes`
|
||||
- invalid `ttlMs`, `maxQueryBytes`, and blank `strategyVersion` rejected
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- Cache keys are deterministic and low-cardinality by strategy version plus digest.
|
||||
- The implementation avoids retaining raw query text in keys.
|
||||
- Query inputs are byte-limited before normalization.
|
||||
- Cache values are small JSON payloads and inherit adapter-level TTL/size bounds.
|
||||
|
||||
## Verification
|
||||
|
||||
- Focused verification passed:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm lint`
|
||||
- Full verification passed:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Review Cadence
|
||||
|
||||
- This slice will be implementation commit 7 after review checkpoint `b7ac774`.
|
||||
- The 10-commit review checkpoint is not due yet.
|
||||
45
knowledge-fs/.harness/changes/2026-05-11-rate-limiting.md
Normal file
45
knowledge-fs/.harness/changes/2026-05-11-rate-limiting.md
Normal file
@ -0,0 +1,45 @@
|
||||
# Rate Limiting
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added a `RateLimiter` contract to the API package.
|
||||
- Added `createNoopRateLimiter()` as the default gateway behavior so unconfigured runtimes keep existing behavior.
|
||||
- Added `createInMemoryRateLimiter()` with bounded key storage, fixed-window counters, per-tool overrides, and explicit config validation.
|
||||
- Gateway protected routes now apply rate limiting after auth by `{ tenantId, subjectId, tool }`.
|
||||
- Rate-limited requests return `429`, a `retry-after` header, and structured metadata.
|
||||
- Public `/health` and `/openapi.json` remain outside limiter checks.
|
||||
|
||||
## Why
|
||||
|
||||
- Phase 2 Sprint 8 requires per-tenant, per-agent, per-tool rate limits for retrieval, KnowledgeFS, document, query, and CRUD surfaces.
|
||||
|
||||
## Performance And Safety Notes
|
||||
|
||||
- In-memory limiter state is bounded with `maxKeys`.
|
||||
- Expired windows are pruned before adding new keys to avoid unbounded memory growth.
|
||||
- Limiter keys use tenant, subject, and normalized low-cardinality tool names; raw paths, queries, JWTs, and request bodies are not stored.
|
||||
- The limiter runs after auth so anonymous traffic still receives `401` instead of consuming tenant-scoped quota.
|
||||
|
||||
## Verification
|
||||
|
||||
- RED first:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts` failed because `createInMemoryRateLimiter` was missing.
|
||||
- Focused verification:
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm lint`
|
||||
- Full verification:
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `pnpm lint`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- This slice adds no Redis/KV backend. Distributed rate limiting remains a runtime adapter follow-up.
|
||||
- Rate-limit OpenAPI response schemas can be expanded once route-level API docs are consolidated.
|
||||
@ -0,0 +1,39 @@
|
||||
# Reranker Provider Interface
|
||||
|
||||
## What Changed
|
||||
|
||||
- Added `RerankerProvider`, rerank input/output types, and reranker model metadata to `@knowledge/embeddings`.
|
||||
- Added Cohere-compatible and Voyage-compatible HTTP reranker providers.
|
||||
- Added deterministic static reranker for local tests and fallback behavior.
|
||||
- Added tests for request mapping, response mapping, clone isolation, input bounds, provider failures, malformed payloads, oversized responses, and duplicate result indexes.
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 5 needs a provider-agnostic reranking boundary before retrieval runtime can rerank expanded hybrid recall candidates. The existing embeddings package already owns external model provider contracts, so reranking now follows the same shape and verification style.
|
||||
|
||||
## Performance And Safety Notes
|
||||
|
||||
- Rerank requests are bounded by document count and per-document byte size before `fetch` is called.
|
||||
- Provider responses are read with an explicit byte ceiling.
|
||||
- Result indexes are validated against the original bounded document list, and duplicates are rejected.
|
||||
- Returned documents and metadata are clone-isolated so callers cannot mutate provider-retained state.
|
||||
- The implementation does not add database calls or runtime-global caches.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/embeddings test -- src/embedding.test.ts`
|
||||
- `pnpm --filter @knowledge/embeddings test:coverage`
|
||||
- `pnpm --filter @knowledge/embeddings typecheck`
|
||||
- `pnpm lint`
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- Retrieval runtime does not yet call the reranker provider; that is the next Sprint 5 integration slice.
|
||||
- Provider-specific request/response variants may need expansion as concrete production models are selected.
|
||||
@ -0,0 +1,41 @@
|
||||
# Reranking Retrieval Integration
|
||||
|
||||
## What Changed
|
||||
|
||||
- Integrated optional `RerankerProvider` support into `createBasicHybridRetriever()`.
|
||||
- Added `rerankerModel` and `maxRerankCandidates` configuration with validation.
|
||||
- Changed planned retrieval to preserve a bounded fusion window for reranking before final output limiting.
|
||||
- Added rerank score/original retrieval score metadata and rerank latency/candidate metrics.
|
||||
- Added tests proving reranking reorders planned hybrid candidates and does not issue unbounded rerank requests.
|
||||
|
||||
## Why
|
||||
|
||||
Sprint 5 requires the production retrieval flow to expand recall, fuse candidates, then rerank a bounded candidate window before returning final evidence.
|
||||
|
||||
## Performance And Safety Notes
|
||||
|
||||
- Dense and FTS searches still run in parallel.
|
||||
- Reranking only runs when a reranker is configured and the retrieval plan asks for a rerank candidate window.
|
||||
- `maxRerankCandidates` caps provider payload size independently of planner fanout.
|
||||
- Reranker documents contain only bounded candidate text and low-cardinality metadata, not vectors or credentials.
|
||||
- No additional database round trips are introduced.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pnpm --filter @knowledge/api test -- src/gateway.test.ts`
|
||||
- `pnpm --filter @knowledge/api test:coverage`
|
||||
- `pnpm --filter @knowledge/api typecheck`
|
||||
- `pnpm lint`
|
||||
- `pnpm check`
|
||||
- `pnpm build`
|
||||
- `cargo test --workspace`
|
||||
- `pnpm wasm:build`
|
||||
- `pnpm compose:config`
|
||||
- `docker compose --profile apps config`
|
||||
- `git diff --check`
|
||||
|
||||
## Known Risks / Follow-Up
|
||||
|
||||
- Runtime app composition still needs to wire a real reranker provider from environment configuration.
|
||||
- Reranker input text currently prefers FTS text and falls back to section path when full node text is not present in retrieval metadata.
|
||||
- Dense-only candidates may need richer text hydration once retrieval repositories expose node text directly.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user