dify/api
Jyong c8aa2d4f59 feat(knowledge-fs): report specific, actionable background task failures
Every document-processing failure surfaced as "the document could not be
processed", and every data-source failure as "the source could not be read",
because the root cause was erased at three points: the compilation classifier
collapsed any coded-but-non-retryable error into DOCUMENT_COMPILATION_FAILED,
datasource runtime errors were reduced to per-operation constants, and the
console mapped dozens of codes onto a handful of generic sentences without the
stage or a support reference.

- knowledge-fs: keep the specific public code for any coded error in the
  compilation classifier (retry follows the error or the catalog policy), tag
  previously uncoded failures (semantic chunking output, embedding dimension,
  PDF rendering, lease loss, generation model, unsupported file type) with
  public codes, give embedding provider errors their own codes instead of the
  parser's, alias datasource runtime codes, and register 17 catalog entries
  (model response invalid, parser unsupported type, PDF render, crawl page
  not found, provider timeout/unavailable/rejected, content too large, ...).
- api: extend the public error-code enum (including the missing
  DOCUMENT_PARSER_TIMEOUT) and regenerate the contracts.
- web: per-code copy that says what happened and what to do, in all 24
  locales, plus a "failed while <stage> · reference <id>" line under each
  failed task in the background-task drawer and the document task list.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015zw5G5SX3HmVfnZof6YWAc
2026-09-03 03:09:29 -04:00
..
.idea
.vscode
clients feat: Enterprise Home Snapshot — runtime endpoints and dify-agent adapter (#40996) 2026-08-21 10:39:36 +00:00
commands Merge remote-tracking branch 'origin/main' into deploy/konwledge 2026-08-28 14:50:39 +08:00
configs fix(knowledge-fs): align retrieval test timeout 2026-09-01 13:51:15 -04:00
constants Merge remote-tracking branch 'origin/main' into deploy/konwledge 2026-08-27 10:06:48 +08:00
context
contexts
controllers Merge remote-tracking branch 'origin/main' into deploy/konwledge 2026-09-02 21:56:53 +08:00
core fix(workflow): let the KnowledgeFS retrieval node run on text or image alone 2026-09-03 01:52:19 -04:00
dev Merge remote-tracking branch 'origin/main' into deploy/konwledge 2026-09-01 09:45:49 +08:00
docker
enterprise fix: add missing metrics back and enhance typing (#40935) 2026-08-21 09:46:45 +00:00
enums fix(api): scope hosted credits by tenant plan (#41131) 2026-08-24 06:35:42 +00:00
events Merge remote-tracking branch 'origin/main' into deploy/konwledge 2026-08-27 10:06:48 +08:00
extensions Merge remote-tracking branch 'origin/main' into deploy/konwledge 2026-09-02 10:31:10 +08:00
factories fix(api): align Service API OpenAPI contract (#41248) 2026-08-25 10:54:14 +00:00
fields refactor(api): remove unused workflow archived logs endpoint (#41345) 2026-08-27 03:48:57 +00:00
libs Merge remote-tracking branch 'origin/main' into deploy/konwledge 2026-09-02 21:56:53 +08:00
machinery refactor(api): require active workspace in request context (#41280) 2026-09-02 06:54:47 +00:00
migrations Merge remote-tracking branch 'origin/main' into deploy/konwledge 2026-09-02 21:56:53 +08:00
models Merge remote-tracking branch 'origin/main' into deploy/konwledge 2026-09-02 21:56:53 +08:00
openapi/markdown Merge remote-tracking branch 'origin/main' into deploy/konwledge 2026-09-02 21:56:53 +08:00
providers fix(api): preserve tracing configuration boundaries (#41403) 2026-08-27 22:11:32 +08:00
repositories Merge remote-tracking branch 'origin/main' into deploy/konwledge 2026-09-02 21:56:53 +08:00
schedule fix: fall back to the batch API when the global plugin manifest is empty (#41107) 2026-08-24 03:33:23 +00:00
services feat(knowledge-fs): report specific, actionable background task failures 2026-09-03 03:09:29 -04:00
tasks Merge remote-tracking branch 'origin/main' into deploy/konwledge 2026-09-02 10:31:10 +08:00
templates
tests fix(workflow): let the KnowledgeFS retrieval node run on text or image alone 2026-09-03 01:52:19 -04:00
.dockerignore
.env.example Merge remote-tracking branch 'origin/main' into deploy/konwledge 2026-08-28 14:50:39 +08:00
.importlinter refactor(api): decouple system features and web adapters (#40772) 2026-09-01 11:50:52 +00:00
.ruff.toml
AGENTS.md
app_factory.py Merge remote-tracking branch 'origin/main' into deploy/konwledge 2026-08-12 10:33:29 +08:00
app.py
celery_entrypoint.py
celery_healthcheck.py
cnt_base.sh
conftest.py
dify_app.py
Dockerfile chore: upgrade Node.js runtime to 24 LTS (#41573) 2026-09-01 06:57:38 +00:00
Dockerfile.dockerignore
gunicorn.conf.py
knowledge-fs-contract.lock.json feat(knowledge-fs): report specific, actionable background task failures 2026-09-03 03:09:29 -04:00
knowledge-fs-product-operation-gaps.json
knowledge-fs-product-operations.json perf(knowledge-fs): harden cross-format document ingestion 2026-08-31 07:48:35 -04:00
pyproject.toml chore(deps-dev): bump nltk from 3.10.0 to 3.10.3 in /api (#41602) 2026-09-02 01:20:41 +00:00
pyrefly-local-excludes.txt
pytest.ini
README.md
uv.lock fix(dify-agent): harden E2B binding lifecycle (#41623) 2026-09-02 09:47:24 +00:00

Dify Backend API

Setup and Run

Important

In the v1.3.0 release, poetry has been replaced with uv as the package manager for Dify API backend service.

uv and pnpm are required to run the setup and development commands below.

The scripts resolve paths relative to their location, so you can run them from anywhere.

  1. Run setup (copies env files and installs dependencies).

    ./dev/setup
    
  2. Review api/.env, web/.env.local, and docker/middleware.env values (see the SECRET_KEY note below).

  3. Start middleware (PostgreSQL/Redis/Weaviate).

    ./dev/start-docker-compose
    
  4. Start backend (runs migrations first).

    ./dev/start-api
    
  5. Start Dify web service.

    ./dev/start-web
    

    ./dev/setup and ./dev/start-web install JavaScript dependencies through the repository root workspace, so you do not need a separate cd web && pnpm install step.

  6. Set up your application by visiting http://localhost:3000.

  7. Start the worker service (async and scheduler tasks, runs from api).

    ./dev/start-worker
    
  8. Start Celery Beat when scheduled tasks are needed. This is required when KNOWLEDGE_FS_LIFECYCLE_WORKER_ENABLED=true so provisioning and deletion outbox commands are dispatched to the lifecycle worker.

    ./dev/start-beat
    

Environment notes

Important

When the frontend and backend run on different subdomains, set COOKIE_DOMAIN to the sites top-level domain (e.g., example.com). The frontend and backend must be under the same top-level domain in order to share authentication cookies.

  • Generate a SECRET_KEY in the .env file.

    bash for Linux

    sed -i "/^SECRET_KEY=/c\\SECRET_KEY=$(openssl rand -base64 42)" .env
    

    bash for Mac

    secret_key=$(openssl rand -base64 42)
    sed -i '' "/^SECRET_KEY=/c\\
    SECRET_KEY=${secret_key}" .env
    

Testing

  1. Install dependencies for both the backend and the test environment

    cd api
    uv sync --group dev
    
  2. Run the tests locally with mocked system environment variables in tool.pytest_env section in pyproject.toml, more can check Claude.md

    cd api
    uv run pytest                           # Run all tests
    uv run pytest tests/unit_tests/         # Unit tests only
    uv run pytest tests/integration_tests/  # Integration tests
    
    # Code quality
    ./dev/reformat               # Run all formatters and linters
    uv run ruff check --fix ./   # Fix linting issues
    uv run ruff format ./        # Format code
    uv run pyrefly check         # Type checking
    

Generate TS stub

uv run dev/generate_swagger_specs.py --output-dir openapi

use https://jsontotable.org/openapi-to-typescript to convert to typescript