diff --git a/api/README.md b/api/README.md index 794b05d3af..9d89b490b0 100644 --- a/api/README.md +++ b/api/README.md @@ -1,6 +1,6 @@ # Dify Backend API -## Usage +## Setup and Run > [!IMPORTANT] > @@ -8,48 +8,77 @@ > [`uv`](https://docs.astral.sh/uv/) as the package manager > for Dify API backend service. -1. Start the docker-compose stack +`uv` and `pnpm` are required to run the setup and development commands below. - The backend require some middleware, including PostgreSQL, Redis, and Weaviate, which can be started together using `docker-compose`. +### Using scripts (recommended) + +The scripts resolve paths relative to their location, so you can run them from anywhere. + +1. Run setup (copies env files and installs dependencies). ```bash - cd ../docker - cp middleware.env.example middleware.env - # change the profile to mysql if you are not using postgres,change the profile to other vector database if you are not using weaviate - docker compose -f docker-compose.middleware.yaml --profile postgresql --profile weaviate -p dify up -d - cd ../api + ./dev/setup ``` -1. Copy `.env.example` to `.env` +1. Review `api/.env`, `web/.env.local`, and `docker/middleware.env` values (see the `SECRET_KEY` note below). - ```cli - cp .env.example .env +1. Start middleware (PostgreSQL/Redis/Weaviate). + + ```bash + ./dev/start-docker-compose ``` -> [!IMPORTANT] -> -> When the frontend and backend run on different subdomains, set COOKIE_DOMAIN to the site’s top-level domain (e.g., `example.com`). The frontend and backend must be under the same top-level domain in order to share authentication cookies. +1. Start backend (runs migrations first). -1. Generate a `SECRET_KEY` in the `.env` file. - - bash for Linux - - ```bash for Linux - sed -i "/^SECRET_KEY=/c\SECRET_KEY=$(openssl rand -base64 42)" .env + ```bash + ./dev/start-api ``` - bash for Mac +1. Start Dify [web](../web) service. - ```bash for Mac - secret_key=$(openssl rand -base64 42) - sed -i '' "/^SECRET_KEY=/c\\ - SECRET_KEY=${secret_key}" .env + ```bash + ./dev/start-web ``` -1. Create environment. +1. Set up your application by visiting `http://localhost:3000`. - Dify API service uses [UV](https://docs.astral.sh/uv/) to manage dependencies. - First, you need to add the uv package manager, if you don't have it already. +1. Optional: start the worker service (async tasks, runs from `api`). + + ```bash + ./dev/start-worker + ``` + +1. Optional: start Celery Beat (scheduled tasks). + + ```bash + ./dev/start-beat + ``` + +### Manual commands + +
+Show manual setup and run steps + +These commands assume you start from the repository root. + +1. Start the docker-compose stack. + + The backend requires middleware, including PostgreSQL, Redis, and Weaviate, which can be started together using `docker-compose`. + + ```bash + cp docker/middleware.env.example docker/middleware.env + # Use mysql or another vector database profile if you are not using postgres/weaviate. + docker compose -f docker/docker-compose.middleware.yaml --profile postgresql --profile weaviate -p dify up -d + ``` + +1. Copy env files. + + ```bash + cp api/.env.example api/.env + cp web/.env.example web/.env.local + ``` + +1. Install UV if needed. ```bash pip install uv @@ -57,60 +86,96 @@ brew install uv ``` -1. Install dependencies +1. Install API dependencies. ```bash - uv sync --dev + cd api + uv sync --group dev ``` -1. Run migrate - - Before the first launch, migrate the database to the latest version. +1. Install web dependencies. ```bash + cd web + pnpm install + cd .. + ``` + +1. Start backend (runs migrations first, in a new terminal). + + ```bash + cd api uv run flask db upgrade - ``` - -1. Start backend - - ```bash uv run flask run --host 0.0.0.0 --port=5001 --debug ``` -1. Start Dify [web](../web) service. +1. Start Dify [web](../web) service (in a new terminal). -1. Setup your application by visiting `http://localhost:3000`. + ```bash + cd web + pnpm dev:inspect + ``` -1. If you need to handle and debug the async tasks (e.g. dataset importing and documents indexing), please start the worker service. +1. Set up your application by visiting `http://localhost:3000`. -```bash -uv run celery -A app.celery worker -P threads -c 2 --loglevel INFO -Q dataset,priority_dataset,priority_pipeline,pipeline,mail,ops_trace,app_deletion,plugin,workflow_storage,conversation,workflow,schedule_poller,schedule_executor,triggered_workflow_dispatcher,trigger_refresh_executor,retention -``` +1. Optional: start the worker service (async tasks, in a new terminal). -Additionally, if you want to debug the celery scheduled tasks, you can run the following command in another terminal to start the beat service: + ```bash + cd api + uv run celery -A app.celery worker -P threads -c 2 --loglevel INFO -Q dataset,priority_dataset,priority_pipeline,pipeline,mail,ops_trace,app_deletion,plugin,workflow_storage,conversation,workflow,schedule_poller,schedule_executor,triggered_workflow_dispatcher,trigger_refresh_executor,retention + ``` -```bash -uv run celery -A app.celery beat -``` +1. Optional: start Celery Beat (scheduled tasks, in a new terminal). + + ```bash + cd api + uv run celery -A app.celery beat + ``` + +
+ +### Environment notes + +> [!IMPORTANT] +> +> When the frontend and backend run on different subdomains, set COOKIE_DOMAIN to the site’s top-level domain (e.g., `example.com`). The frontend and backend must be under the same top-level domain in order to share authentication cookies. + +- Generate a `SECRET_KEY` in the `.env` file. + + bash for Linux + + ```bash + sed -i "/^SECRET_KEY=/c\\SECRET_KEY=$(openssl rand -base64 42)" .env + ``` + + bash for Mac + + ```bash + secret_key=$(openssl rand -base64 42) + sed -i '' "/^SECRET_KEY=/c\\ + SECRET_KEY=${secret_key}" .env + ``` ## Testing 1. Install dependencies for both the backend and the test environment ```bash - uv sync --dev + cd api + uv sync --group dev ``` 1. Run the tests locally with mocked system environment variables in `tool.pytest_env` section in `pyproject.toml`, more can check [Claude.md](../CLAUDE.md) ```bash + cd api uv run pytest # Run all tests uv run pytest tests/unit_tests/ # Unit tests only uv run pytest tests/integration_tests/ # Integration tests # Code quality - ../dev/reformat # Run all formatters and linters - uv run ruff check --fix ./ # Fix linting issues - uv run ruff format ./ # Format code - uv run basedpyright . # Type checking + ./dev/reformat # Run all formatters and linters + uv run ruff check --fix ./ # Fix linting issues + uv run ruff format ./ # Format code + uv run basedpyright . # Type checking ``` diff --git a/dev/setup b/dev/setup new file mode 100755 index 0000000000..399c8f28a5 --- /dev/null +++ b/dev/setup @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(dirname "$(realpath "$0")")" +ROOT="$(dirname "$SCRIPT_DIR")" + +API_ENV_EXAMPLE="$ROOT/api/.env.example" +API_ENV="$ROOT/api/.env" +WEB_ENV_EXAMPLE="$ROOT/web/.env.example" +WEB_ENV="$ROOT/web/.env.local" +MIDDLEWARE_ENV_EXAMPLE="$ROOT/docker/middleware.env.example" +MIDDLEWARE_ENV="$ROOT/docker/middleware.env" + +# 1) Copy api/.env.example -> api/.env +cp "$API_ENV_EXAMPLE" "$API_ENV" + +# 2) Copy web/.env.example -> web/.env.local +cp "$WEB_ENV_EXAMPLE" "$WEB_ENV" + +# 3) Copy docker/middleware.env.example -> docker/middleware.env +cp "$MIDDLEWARE_ENV_EXAMPLE" "$MIDDLEWARE_ENV" + +# 4) Install deps +cd "$ROOT/api" +uv sync --group dev + +cd "$ROOT/web" +pnpm install diff --git a/dev/start-api b/dev/start-api index 0b50ad0d0a..bdfa58eca6 100755 --- a/dev/start-api +++ b/dev/start-api @@ -3,8 +3,9 @@ set -x SCRIPT_DIR="$(dirname "$(realpath "$0")")" -cd "$SCRIPT_DIR/.." +cd "$SCRIPT_DIR/../api" +uv run flask db upgrade -uv --directory api run \ +uv run \ flask run --host 0.0.0.0 --port=5001 --debug diff --git a/dev/start-docker-compose b/dev/start-docker-compose new file mode 100755 index 0000000000..9652be169d --- /dev/null +++ b/dev/start-docker-compose @@ -0,0 +1,8 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(dirname "$(realpath "$0")")" +ROOT="$(dirname "$SCRIPT_DIR")" + +cd "$ROOT/docker" +docker compose -f docker-compose.middleware.yaml --profile postgresql --profile weaviate -p dify up -d diff --git a/dev/start-worker b/dev/start-worker index 7876620188..3e48065631 100755 --- a/dev/start-worker +++ b/dev/start-worker @@ -83,7 +83,7 @@ while [[ $# -gt 0 ]]; do done SCRIPT_DIR="$(dirname "$(realpath "$0")")" -cd "$SCRIPT_DIR/.." +cd "$SCRIPT_DIR/../api" if [[ -n "${ENV_FILE}" ]]; then if [[ ! -f "${ENV_FILE}" ]]; then @@ -123,6 +123,6 @@ echo " Concurrency: ${CONCURRENCY}" echo " Pool: ${POOL}" echo " Log Level: ${LOGLEVEL}" -uv --directory api run \ +uv run \ celery -A app.celery worker \ -P ${POOL} -c ${CONCURRENCY} --loglevel ${LOGLEVEL} -Q ${QUEUES} diff --git a/dev/update-uv b/dev/update-uv index 189bd1f6b1..35d723fe20 100755 --- a/dev/update-uv +++ b/dev/update-uv @@ -4,7 +4,7 @@ set -e set -o pipefail -SCRIPT_DIR="$(dirname "$0")" +SCRIPT_DIR="$(dirname "$(realpath "$0")")" REPO_ROOT="$(dirname "${SCRIPT_DIR}")" # rely on `poetry` in path