KnowledgeFS document-ingestion performance runbook

This runbook covers the bounded-concurrency, batched, resumable document-compilation pipeline. KnowledgeFS remains a Dify-dependent service: every model request is routed through Dify's model runtime and plugin-daemon; KnowledgeFS does not own provider credentials.

Runtime shape

The searchable publication path is:

parse and normalize the source;
build the deterministic outline;
generate bottom-up outline summaries in bounded batches;
chunk and build FTS/dense/PageIndex projections;
durably admit optional graph work and publish the searchable generation;
after the exact outline generation is published, extract entities/relations asynchronously and attach graph rows to that generation.

Outline summaries, embeddings, entity extraction, and relation extraction share one process-wide model-request gate. Outline and semantic extraction also retain smaller per-document bounds. Graph failure therefore degrades graph retrieval only; it cannot roll back an already searchable dense/FTS/PageIndex generation.

Graph entity and relation upserts are physically separate database operations. Published graph reads therefore use the durable semantic-job success state as a visibility barrier: no new async graph row is queryable while its job is pending, running, retrying, failed, or superseded. A jobless legacy synchronous graph generation remains readable for backward compatibility.

Capacity settings

Variable	Default	Valid range	Meaning
`KNOWLEDGE_MODEL_RUNTIME_GLOBAL_CONCURRENCY`	16	1–128	Total concurrent ingestion-time Dify model requests per KnowledgeFS API replica, including embedding transport requests.
`KNOWLEDGE_OUTLINE_SUMMARY_MAX_CONCURRENCY`	8	1–32	Maximum outline-summary provider batches scheduled by one document.
`KNOWLEDGE_OUTLINE_SUMMARY_BATCH_SIZE`	8	1–32	Maximum outline nodes in one structured summary request.
`KNOWLEDGE_OUTLINE_SUMMARY_BATCH_MAX_INPUT_CHARS`	32000	1–200000	Maximum aggregate prompt material in one outline-summary batch.
`KNOWLEDGE_SEMANTIC_EXTRACTION_BATCH_SIZE`	8	1–32	Maximum nodes in one entity or relation extraction request.
`KNOWLEDGE_SEMANTIC_EXTRACTION_MAX_CONCURRENCY`	4	1–32	Maximum semantic batches scheduled by one graph-enrichment job.
`KNOWLEDGE_EMBEDDING_REQUEST_CONCURRENCY`	2	1–8	Maximum 16-text embedding transport requests scheduled by one embedding call.

The effective process limit is always the global limit, even when the sum of per-document limits is higher. Across R API replicas, the theoretical provider pressure is R × global concurrency, so scale replicas and the global value together. Start with defaults. Increase one step at a time only when queue wait is sustained, provider 429/timeout rates remain flat, plugin-daemon CPU/memory is healthy, and database connections have headroom.

Do not raise KNOWLEDGE_EMBEDDING_REQUEST_CONCURRENCY to compensate for slow individual provider requests. It improves multi-batch documents but also increases high-dimensional response memory.

Progress contract

Document processing task responses retain the existing durable stage and progressPercent fields and add:

phase: the work expected after the last durable checkpoint;
activeOperations: bounded operation names such as outline_summary, chunk, fts_index, and embedding;
semanticEnrichment: optional graph state (not_scheduled, pending, running, ready, failed, or disabled) plus safe completion counters when available.

The chunk/FTS/embedding work remains one durable chunking_indexing phase because the current reindexer publishes those projections atomically. activeOperations names its components without pretending that a partial projection is a resumable publication checkpoint. SSE progress events and the unified background-task response expose the same additive metadata.

Structured metrics

The API emits aggregation-only JSON lines. They intentionally contain no tenant, space, document, model credential, URL, object key, prompt, source text, or free-form provider error.

Event	Important dimensions
`knowledge_fs.ingestion_model_concurrency.metric`	lifecycle, active/queued requests, limit, queue wait milliseconds
`knowledge_fs.outline_summary.metric`	outcome, duration, node count, provider calls, checkpoint hits, bounded failure kind
`knowledge_fs.embedding_request.metric`	outcome, duration, text count, concurrency limit, bounded failure kind
`knowledge_fs.semantic_enrichment.metric`	outcome, duration, queue wait, execution attempt, provider calls, nodes scanned, degraded flag, bounded failure kind
`knowledge_fs.durable_task.metric`	compilation/research lifecycle and terminal outcome

Alert on a sustained increase in queueWaitMs, failureKind=rate_limited, failureKind=timeout, terminal document compilation failures, or semantic_enrichment.metric degraded=true. A failed semantic-enrichment job means ordinary search is available but graph-derived retrieval may be incomplete.

Retry and recovery

Outline checkpoints are scoped by tenant, space, document generation, node, input fingerprint, model route, and prompt version. A retry reuses exact successful nodes and recomputes only misses.
Semantic entity/relation checkpoints are generation- and fingerprint-scoped. Relation extraction is skipped for nodes with fewer than two eligible entities.
Embedding batches preserve input order and abort sibling transport work after the first failure. A document retry reuses exact generation-scoped ready projections through the compilation repositories.
Semantic jobs wait for their generation to become published without consuming a model execution attempt. An unrelated document advancing the Space head does not invalidate an in-flight job; only a failed/canceled/superseded compilation or a successful generation absent from the current publication supersedes it.

Do not delete checkpoint rows to retry a normal transient provider failure. Use the existing task retry operation; immutable-key mismatches are evidence of a code/schema defect and should be investigated before data is changed.

Rollout

Apply KnowledgeFS migrations before starting the new API image. Migrations 0038 and 0039 add outline-summary checkpoints and durable semantic-enrichment jobs/checkpoints.
Start one API replica with the default values and verify readiness/migration checks.
Import a representative small and large document. Confirm searchable publication completes before semantic enrichment reaches ready.
Observe the four ingestion metric families for at least one normal workload window.
Roll out remaining replicas. Remember that the global limit is per process.
Increase concurrency only after comparing provider request latency, 429/timeout rate, plugin-daemon memory, and database pool saturation with the previous window.

Rollback

Prefer a configuration rollback before a code rollback:

set outline, semantic, and embedding concurrency to 1 to serialize new provider work;
leave the global gate enabled and lower it if the provider is overloaded;
disable graph in the affected document/space settings when graph enrichment must stop; dense, FTS, and PageIndex publication remains available;
deploy the prior application image only after confirming it tolerates migrations 0038/0039.

The new tables are additive. Do not drop them during an application rollback: retained checkpoints and queued jobs are required for a safe forward recovery. If an old image cannot understand the additive task response fields, its clients may ignore them; existing fields are unchanged.

7.7 KiB Raw Blame History Unescape Escape