dify/knowledge-fs

KnowledgeFS

KnowledgeFS is Dify's backend knowledge runtime. It provides tenant-scoped ingestion, parsing, indexing, retrieval, KnowledgeFS commands, MCP tools, durable jobs, traces, and evaluation APIs.

KnowledgeFS is not an independently deployable product. It must run with the Dify API:

• Dify owns model and datasource plugin credentials.
• Dify creates model and datasource plugin instances and performs plugin invocation.
• Dify owns physical object storage through its configured STORAGE_TYPE.
• KnowledgeFS reaches those capabilities only through the authenticated Dify inner API.
• KNOWLEDGE_INTEGRATED_MODE_ENABLED is a Workspace rollout/cutover gate; it never selects a different runtime or credential owner.

Runtime architecture

Dify API
  ├─ model manager / plugin daemon
  ├─ datasource plugins
  ├─ unified object storage
  └─ authenticated inner API
          │
          ▼
KnowledgeFS API
  ├─ document compilation and retrieval
  ├─ KnowledgeFS / MCP command surfaces
  ├─ PostgreSQL repositories and durable jobs
  └─ optional Unstructured parser dependency

Main directories:

apps/api/                              KnowledgeFS backend entrypoint
packages/api/                          Hono gateway, repositories, retrieval, jobs, auth
packages/adapters/                     Database, Dify storage, cache, and queue adapters
packages/dify-model-runtime-client/    Bounded Dify model inner-API client
packages/dify-datasource-runtime-client/
                                       Bounded Dify datasource inner-API client
packages/core/                         Shared contracts and schemas
packages/database/                     Schema catalog and SQL migrations
packages/compute/                      Bounded pure TypeScript compute
packages/parsers/                      Native and Unstructured parser adapters
infra/local/                           Developer harness; requires a running Dify API
infra/kubernetes/                      Inert Dify integration baseline

The repository still contains reusable lower-level adapters and an optional local Admin test harness. They are development assets, not alternative production deployment modes.

Required production configuration

The canonical Dify Compose service loads docker/envs/core-services/knowledge-fs.env.example. Operator-owned inputs are limited to:

• DATABASE_URL
• DIFY_INNER_API_URL and DIFY_INNER_API_KEY, injected by Dify Compose
• UNSTRUCTURED_API_URL and optional UNSTRUCTURED_API_KEY
• bounded Poppler PDF image rasterization settings (the API image includes pdftoppm)
• KnowledgeFS capability/JWKS and document-compilation rollout settings

Do not configure storage-provider credentials, model-provider keys, datasource credentials, or a direct Plugin Daemon endpoint in KnowledgeFS. The Dify inner key must match INNER_API_KEY_FOR_PLUGIN.

See production deployment and the operator manual.

Development

Prerequisites:

• Node.js 22+
• pnpm 10.33.0 through Corepack
• Docker
• A reachable Dify API

Install dependencies:

corepack enable
pnpm install
cp infra/local/.env.example infra/local/.env

Set DIFY_INNER_API_URL and DIFY_INNER_API_KEY in the ignored local env, then start the local database and parser:

pnpm dev:infra
pnpm local:db:migrate

Run the backend from source:

pnpm dev:api

For the optional local Admin test harness:

pnpm --filter @knowledge/admin dev

Run the bounded local smoke after Dify and the local processes are available:

pnpm local:happy-path

It validates health, workspace bootstrap, Markdown upload, parse artifacts, and query evidence. Use LOCAL_SMOKE_ADMIN_BASE when the Admin harness is not on its default port. Other useful forms:

LOCAL_SMOKE_RUN_MIGRATIONS=1 pnpm local:happy-path
pnpm local:happy-path:durable
pnpm local:happy-path:api

The durable smoke requires database health and Dify-backed object-storage health. The API-only smoke skips the Admin BFF.

See the local developer guide for details.

Validation

pnpm typecheck
pnpm test
pnpm lint:backend
pnpm openapi:export:test
pnpm db:migrations:check
pnpm compose:middleware:test
pnpm compose:apps:test
pnpm dify:compose:config
git diff --check

Build the backend production bundle or image:

pnpm --filter @knowledge/api-app build:prod
pnpm docker:api:build
pnpm docker:api:bundle-smoke

The isolated image smoke proves the bundle can boot and remains unhealthy while Dify is absent; a Dify Compose/Kubernetes smoke is required to validate the real inner API, storage, database, models, and datasources.

API and design references

• API reference
• Production deployment
• Operator manual
• Project overview
• Kubernetes Dify integration baseline
• OpenAPI snapshot