diff --git a/.agents/skills/frontend-code-review/references/accessibility-ui.md b/.agents/skills/frontend-code-review/references/accessibility-ui.md index 92764786b15..eb9cdd47728 100644 --- a/.agents/skills/frontend-code-review/references/accessibility-ui.md +++ b/.agents/skills/frontend-code-review/references/accessibility-ui.md @@ -2,6 +2,8 @@ Accessibility findings are first-class review findings. Treat broken keyboard access, missing accessible names, focus loss, and unreachable popup content as correctness bugs, not polish. +## Review Evidence + Before finalizing UI or accessibility findings, fetch the latest Web Interface Guidelines as a required baseline: ```text @@ -21,13 +23,26 @@ Flag: - Clickable `div` or `span` used for actions. - Router navigation implemented with button or `onClick` when a `Link` / `` is the real semantic element. -- Icon-only buttons without `aria-label` or `aria-labelledby`. +- Icon-only controls without an accessible name; follow the naming rules below and the Dify UI `IconButton` contract. - Decorative icons missing `aria-hidden="true"`. - Images without `alt`; use `alt=""` only when truly decorative. - Heading levels that skip hierarchy in page-level content. Prefer semantic HTML before ARIA. +## Accessible Names And Descriptions + +Read [Accessible names and descriptions] when a change affects labels, ARIA naming, help/error relationships, or hidden text. That document owns the shared implementation and review contract. + +Flag violations supported by the final rendered behavior: + +- Missing or insufficient names, redundant name overrides, or naming attributes prohibited by the element's role. +- Overrides that omit visible label wording or suppress necessary descendant information. +- Broken or stale label/description references, including relationships lost when responsive content or overlays unmount. +- Descriptions used instead of names, repeated help text, or essential structured content available only as a flattened description. + +Inspect the computed name and description in the relevant state. Matching an `aria-label` string to nearby text alone does not prove redundancy. + ## Keyboard And Focus Flag: @@ -56,7 +71,7 @@ Flag: - Placeholder text used as the only label. - Password managers accidentally triggered on non-auth fields because autocomplete is missing or wrong. -Prefer visible labels. If visible surrounding text already labels the control, use a visually hidden label or a precise `aria-label`. +Prefer visible labels and associate them through the appropriate field primitive, a native `label`, or `aria-labelledby`. Do not duplicate an existing label with hidden text or `aria-label`; follow [Accessible names and descriptions] when no suitable visible label exists. ## Disabled, Loading, And Async States @@ -107,3 +122,5 @@ Flag: - Images without dimensions. - Loading copy using `...` instead of `…`. - Hardcoded dates, times, numbers, or currency formats instead of `Intl.*`. + +[Accessible names and descriptions]: ../../../../packages/dify-ui/docs/accessible-names-and-descriptions.md diff --git a/.agents/skills/frontend-code-review/references/component-architecture.md b/.agents/skills/frontend-code-review/references/component-architecture.md index 42b7b0bfbb4..3ece06099d9 100644 --- a/.agents/skills/frontend-code-review/references/component-architecture.md +++ b/.agents/skills/frontend-code-review/references/component-architecture.md @@ -25,7 +25,9 @@ Flag: - A page/tab-level section component becoming the data owner without needing a shared snapshot or shared loading/error/empty UI. - Feature code promoted to shared only because it appears once or might be reused later. -Accept repeated TanStack Query calls in siblings when each component independently consumes the data. Cache deduplication is not a reason to hoist by itself. +Accept repeated TanStack Query hooks for the same key and input in Client Component siblings under one QueryClient; +shared cache is not a reason to hoist. Separate Server Component QueryClients do not share it, so request-level +deduplication needs an identified request-local cache or verified framework or transport owner. ## Component Boundaries diff --git a/.agents/skills/frontend-code-review/references/data-query-contracts.md b/.agents/skills/frontend-code-review/references/data-query-contracts.md index db2e2017d27..c68d7e0d1c4 100644 --- a/.agents/skills/frontend-code-review/references/data-query-contracts.md +++ b/.agents/skills/frontend-code-review/references/data-query-contracts.md @@ -22,10 +22,22 @@ Flag: - Fake fallback IDs or placeholder inputs used to force a query to run. - Query results copied into local state for rendering. - Shared query behavior such as invalidation, stale defaults, or retry rules reimplemented at call sites. -- `prefetchQuery` treated as a hard gate or as returning data/errors to the caller. +- Deprecated imperative reads such as `fetchQuery`, `prefetchQuery`, `ensureQueryData`, or their infinite variants when + the current `query` or `infiniteQuery` contract applies. Use `useQuery(consoleQuery.xxx.queryOptions(...))` or `useQuery(marketplaceQuery.xxx.queryOptions(...))` directly unless a feature hook performs real orchestration. +For imperative access, treat the choices as independent dimensions: + +- `query` or `infiniteQuery` resolves the generated query and returns its data. +- `staleTime` decides whether cached data satisfies this call: `0` treats it as stale, a finite value accepts a freshness + window, `Infinity` accepts it until invalidation, and `'static'` accepts available data even after invalidation. +- `select` projects the resolved value without replacing cached query-function data. An imperative query defaults to no + retries when `retry` is not configured; `enabled` is observer-only, so guard before a conditional call. +- `await` blocks the current flow, `return` transfers the Promise to the caller, and `void` discards the result without + handling rejection. Handle rejection before discarding a potentially rejecting Promise; use `.catch(noop)` only for + intentional silence or feedback owned elsewhere, and preserve rejection for hard gates. + ## Mutations Flag: @@ -44,12 +56,18 @@ Flag: - Request-time auth, setup, workspace role, or tenant decisions moved into static `next.config redirects()`. - Dynamic role gates depending on `workspaces.current` implemented as static path redirects. -- Authorization logic depending on soft `prefetchQuery`. +- Authorization logic depending on an imperative query whose rejection is swallowed. - Removing a client fallback before server API unavailable behavior is defined. - Global placeholder query contracts introduced to solve a route-local Suspense issue. - Branding-sensitive UI reading placeholder defaults without checking pending/placeholder state. +- A Server Component rendering or passing an imperative query result that the browser can independently revalidate, + leaving server and client output with different owners. +- A non-blocking Server Component query without pending-query dehydration, Next-compatible error redaction, a + `HydrationBoundary` covering the same-key client consumer, or an explicit Suspense and SSR-content decision. -Separate hard gates from soft prefetches. `fetchQuery` can be a server decision boundary; `prefetchQuery` is cache warmup. +Hard gates await `query` or `infiniteQuery` and preserve rejection; soft prefetches handle failure at the fallback owner. +Treat Server Components as prefetch-and-dehydrate owners by default, rendering returned data only under exclusive server +ownership or a freshness contract that prevents server/client drift. ## Workspace And Tenant diff --git a/.agents/skills/how-to-write-component/references/data.md b/.agents/skills/how-to-write-component/references/data.md index 3bda645f44a..7307f25a6ba 100644 --- a/.agents/skills/how-to-write-component/references/data.md +++ b/.agents/skills/how-to-write-component/references/data.md @@ -20,7 +20,7 @@ Read this document when a component consumes generated contracts, nullable API v - When a query supplies several sibling owners, shared derived facts, or workflow commands, model it as a graph node with field selectors. Do not keep it in a page and pass its data, query key, observer methods, and status fields back down as a deconstructed query object. - For missing required input, branch the whole generated input with `skipToken`. Add `enabled` only for an independent execution condition; do not put `skipToken` inside a placeholder payload or coerce IDs to empty strings. - Return generated `queryOptions()`, `infiniteOptions()`, or `mutationOptions()` directly from TanStack Query atoms. Pass supported options into the generated call instead of spreading into a parallel object. -- Share the exact options between prefetch and render when they represent the same request. Do not extract option helpers merely to reuse input construction. +- For the same logical request, preserve key, input, operation, and result contracts. Share the exact options between prefetch and render only when transport, context, and cache policy are shared; imperative and observer freshness may differ. Do not extract option helpers merely to reuse input construction. - In Jotai-backed components, consume field-level selectors or named facts rather than the complete query result unless observer methods such as `refetch` or an infinite-scroll field group are part of that owner's contract. - Keep observer methods inside the component that owns the query surface or behind a named graph command. A parent must not pass raw `refetch`, query keys, or invalidation details merely because a descendant action needs fresh data. - Treat `refetch()` as an explicit execution command: it can run a query whose observer currently has `enabled: false`. Retry and refresh commands must recheck the permission, identity, or availability condition before including conditionally enabled queries. @@ -39,12 +39,21 @@ Read this document when a component consumes generated contracts, nullable API v ## Prefetch And Hidden Surfaces - Prefetch expensive secondary content from the trigger or menu-open event when it benefits the visible path. Do not mount hidden subscribers solely to warm the cache. -- `prefetchQuery` is cache warmup, not an authorization or availability gate. Use a hard fetch boundary when the server must decide whether rendering may proceed. +- Use `query` or `infiniteQuery` for imperative access. `staleTime` defines cache acceptance; `select` projects the return + value without replacing cached query-function data. Imperative queries default to no retries when `retry` is not + configured, and observer-only `enabled` does not prevent an imperative call. +- `await` blocks, `return` transfers the Promise, and `void` discards its value without handling rejection. Handle + rejection before discarding a potentially rejecting Promise; use `.catch(noop)` only for intentional silence or + feedback owned elsewhere. Hard server gates await the query and preserve rejection. ## SSR, Authentication, And Workspace - Static configuration owns path-invariant routing. Request-dependent authentication, setup, role, and tenant decisions belong to SSR or runtime decision boundaries. - Distinguish soft SSR cache warming from authoritative decisions. Prefetched or placeholder data must not grant access or represent successful availability. +- Treat Server Components as query prefetch-and-dehydrate owners by default. Do not render or pass an imperative query + result when a browser observer can revalidate the same data unless ownership and freshness explicitly prevent drift. +- Non-blocking RSC streaming requires pending-query dehydration without redacting Next.js server errors, a + `HydrationBoundary` around the same-key client consumer, and Suspense when that content must be server-rendered. - Never reuse tenant-scoped state after switching workspaces. Discard it at the switch boundary or isolate it by workspace identity. - Do not make product or authorization decisions from bootstrap defaults. Wait for authoritative data, or render an explicit loading or error state. - Keep loading and Suspense behavior inside the feature that owns the request. Do not add fake global data merely to bypass that boundary. diff --git a/.agents/skills/how-to-write-component/references/ownership.md b/.agents/skills/how-to-write-component/references/ownership.md index 712d7ddc3d9..9a60f60c2d3 100644 --- a/.agents/skills/how-to-write-component/references/ownership.md +++ b/.agents/skills/how-to-write-component/references/ownership.md @@ -19,6 +19,7 @@ Read this document when auditing, adding, moving, splitting, or refactoring Reac - A page or feature root may wire route identity, providers, layout, navigation, and genuine cross-surface coordination. It must not call a child-specific state or query hook merely to assemble that child's props. - Keep child contracts at the ownership boundary: stable domain identity, a small immutable snapshot, placement options, or named cross-boundary commands. Do not pass an internal state machine as separate `data`, `pending`, `error`, `retry`, `open`, setter, and callback props when the parent does not use them, and do not hide the same fan-out in a props bag or hook result object. - Repeated TanStack Query calls in siblings are acceptable when each sibling independently consumes the data; the cache already deduplicates requests. +- Repeated TanStack Query hooks for the same key and input in Client Component siblings under one QueryClient share that cache. Separate Server Component QueryClients do not, so request-level deduplication needs an identified request-local cache or verified framework or transport owner. - Treat parent input according to what the child boundary does with it: - If the child only renders or performs a light local decision, pass the snapshot as props. It does not become a new state owner. - If the child builds queries, dialogs, mutations, derivations, commands, or a reset lifecycle around a stable identity or snapshot, give that boundary a feature-local state file. This does not by itself require a new module or directory. @@ -26,6 +27,7 @@ Read this document when auditing, adding, moving, splitting, or refactoring Reac - One pass-through layer is acceptable for stable identity and placement. It is not permission to relay workflow state and handlers through an unrelated component. - Route identity may pass once from a framework route into its feature boundary. If multiple descendants, queries, facts, or commands need it, bridge it into the feature graph and stop passing it as props. - A query snapshot may cross once as immutable display input. Query keys, observer methods, retry/loading/error groups, and invalidation mechanics belong to the query surface or feature graph and must not be decomposed into props. +- Do not replace prop drilling with one large view-model hook. Move each query, derived value, and handler to the concrete owner that consumes it. - Keep source selection, defaults, validation, dirty checks, and payload shaping beside the workflow that owns submission. ## Boundaries diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 467d75ad153..bbe309600e5 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -1,7 +1,7 @@ name: '🕷️ Bug report' description: Report errors or unexpected behavior labels: - - bug + - 🐞 bug body: - type: checkboxes attributes: diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml index 05cad470e7c..364598db372 100644 --- a/.github/ISSUE_TEMPLATE/feature_request.yml +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -1,7 +1,7 @@ name: '⭐ Feature or enhancement request' description: Propose something new. labels: - - enhancement + - 💪 enhancement body: - type: checkboxes attributes: diff --git a/.github/actions/setup-web/action.yml b/.github/actions/setup-web/action.yml index dbc20df50af..69565020d79 100644 --- a/.github/actions/setup-web/action.yml +++ b/.github/actions/setup-web/action.yml @@ -4,13 +4,13 @@ description: Set up Node.js, Vite+, pnpm, and web dependencies runs: using: composite steps: - - name: Setup pnpm - uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6.0.10 + - name: Setup pnpm and Node.js + uses: pnpm/setup@703c52620218391530e48b9e8870d5c0082e1b9b # v2.1.0 with: - run_install: false + install: false - name: Setup Vite+ uses: voidzero-dev/setup-vp@1b32467adbe183473499fd9d5d372c3ed9641754 # v1.18.0 with: - node-version-file: package.json + node-manager: false cache: true run-install: true diff --git a/.github/workflows/api-tests.yml b/.github/workflows/api-tests.yml index 1b1afe96529..4e2abe4c433 100644 --- a/.github/workflows/api-tests.yml +++ b/.github/workflows/api-tests.yml @@ -71,10 +71,10 @@ jobs: retention-days: 1 api-integration: - name: API Integration Tests - runs-on: depot-ubuntu-24.04 + name: API Integration Tests (${{ matrix.python-version }}, shard ${{ matrix.shardIndex }}/${{ matrix.shardTotal }}) + runs-on: depot-ubuntu-24.04-4 env: - COVERAGE_FILE: coverage-integration + COVERAGE_FILE: coverage-integration-${{ matrix.python-version }}-${{ matrix.shardIndex }} STORAGE_TYPE: opendal OPENDAL_SCHEME: fs OPENDAL_FS_ROOT: /tmp/dify-storage @@ -82,9 +82,17 @@ jobs: run: shell: bash strategy: + fail-fast: false matrix: python-version: - '3.12' + shardIndex: + - 1 + - 2 + - 3 + - 4 + shardTotal: + - 4 steps: - name: Checkout code @@ -112,6 +120,8 @@ jobs: -p no:benchmark \ --start-middleware \ -n auto \ + --shard-index "${{ matrix.shardIndex }}" \ + --shard-total "${{ matrix.shardTotal }}" \ --timeout "${PYTEST_TIMEOUT:-180}" \ api/tests/integration_tests/workflow \ api/tests/integration_tests/tools \ @@ -120,8 +130,9 @@ jobs: - name: Upload integration coverage data uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: - name: api-coverage-integration - path: coverage-integration + name: api-coverage-integration-${{ matrix.python-version }}-${{ matrix.shardIndex }} + path: ${{ env.COVERAGE_FILE }} + if-no-files-found: error retention-days: 1 api-coverage: @@ -171,12 +182,15 @@ jobs: echo "" >> "$GITHUB_STEP_SUMMARY" unit_coverage="$(find coverage-data -type f -name coverage-unit -print -quit)" - integration_coverage="$(find coverage-data -type f -name coverage-integration -print -quit)" + mapfile -t integration_coverage < <(find coverage-data -type f -name 'coverage-integration-*' -print | sort) : "${unit_coverage:?coverage-unit artifact not found}" - : "${integration_coverage:?coverage-integration artifact not found}" + if [[ "${#integration_coverage[@]}" -ne 4 ]]; then + echo "expected 4 integration coverage artifacts, found ${#integration_coverage[@]}" >&2 + exit 1 + fi report_file="$(mktemp)" - uv run --project api coverage combine "$unit_coverage" "$integration_coverage" + uv run --project api coverage combine "$unit_coverage" "${integration_coverage[@]}" uv run --project api coverage report --show-missing | tee "$report_file" echo "Summary: \`$(tail -n 1 "$report_file")\`" >> "$GITHUB_STEP_SUMMARY" { diff --git a/.github/workflows/cli-e2e.yml b/.github/workflows/cli-e2e.yml index 51c5485ce0f..b58e94d9b0d 100644 --- a/.github/workflows/cli-e2e.yml +++ b/.github/workflows/cli-e2e.yml @@ -87,10 +87,9 @@ jobs: with: bun-version: latest - - uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6.0.10 + - uses: pnpm/setup@703c52620218391530e48b9e8870d5c0082e1b9b # v2.1.0 with: - package_json_field: packageManager - run_install: false + install: false - name: Install CLI dependencies working-directory: cli @@ -131,11 +130,6 @@ jobs: - uses: oven-sh/setup-bun@v2 with: bun-version: latest - - uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6.0.10 - with: - package_json_field: packageManager - run_install: false - - run: pnpm install --frozen-lockfile - run: pnpm tree:gen - name: Run framework + output + error-handling @@ -181,11 +175,6 @@ jobs: - uses: oven-sh/setup-bun@v2 with: bun-version: latest - - uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6.0.10 - with: - package_json_field: packageManager - run_install: false - - run: pnpm install --frozen-lockfile - run: pnpm tree:gen - name: Run discovery suite @@ -247,11 +236,6 @@ jobs: - uses: oven-sh/setup-bun@v2 with: bun-version: latest - - uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6.0.10 - with: - package_json_field: packageManager - run_install: false - - run: pnpm install --frozen-lockfile - run: pnpm tree:gen - name: 'Run run/${{ matrix.name }}' @@ -312,11 +296,6 @@ jobs: - uses: oven-sh/setup-bun@v2 with: bun-version: latest - - uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6.0.10 - with: - package_json_field: packageManager - run_install: false - - run: pnpm install --frozen-lockfile - run: pnpm tree:gen - name: Run auth/login + status + whoami @@ -371,11 +350,6 @@ jobs: - uses: oven-sh/setup-bun@v2 with: bun-version: latest - - uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6.0.10 - with: - package_json_field: packageManager - run_install: false - - run: pnpm install --frozen-lockfile - run: pnpm tree:gen - name: Run use / devices / logout / agent (serial) diff --git a/.github/workflows/main-ci.yml b/.github/workflows/main-ci.yml index 6023aa7fae2..b924fd18936 100644 --- a/.github/workflows/main-ci.yml +++ b/.github/workflows/main-ci.yml @@ -37,8 +37,6 @@ jobs: # Check which paths were changed to determine which tests to run check-changes: name: Check Changed Files - needs: pre_job - if: needs.pre_job.outputs.should_skip != 'true' runs-on: depot-ubuntu-24.04 outputs: api-changed: ${{ steps.changes.outputs.api }} @@ -49,6 +47,11 @@ jobs: migration-changed: ${{ steps.changes.outputs.migration }} sandbox-runtime-changed: ${{ steps.changes.outputs.sandbox-runtime }} dify-agent-changed: ${{ steps.changes.outputs.dify-agent }} + python-style-changed: ${{ steps.changes.outputs.python-style }} + dify-agent-style-changed: ${{ steps.changes.outputs.dify-agent-style }} + web-style-changed: ${{ steps.changes.outputs.web-style }} + ts-common-style-changed: ${{ steps.changes.outputs.ts-common-style }} + superlinter-changed: ${{ steps.changes.outputs.superlinter }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3 @@ -150,6 +153,58 @@ jobs: - 'docker/generate_docker_compose' - 'docker/ssrf_proxy/**' - 'docker/volumes/sandbox/conf/**' + python-style: + - 'api/**' + - 'scripts/ast_grep_guard.py' + - 'scripts/check_no_new_getattr.py' + - 'scripts/check_no_new_controller_sqlalchemy.py' + - 'scripts/lint_controller_sqlalchemy.py' + - 'scripts/ast_grep_rules/no_new_getattr.yml' + - 'scripts/ast_grep_rules/no_new_controller_sqlalchemy.yml' + - '.github/workflows/style.yml' + - '.github/workflows/main-ci.yml' + dify-agent-style: + - 'dify-agent/**' + - '.github/workflows/style.yml' + - '.github/workflows/main-ci.yml' + web-style: + - 'web/**' + - 'e2e/**' + - 'sdks/nodejs-client/**' + - 'packages/**' + - 'package.json' + - 'pnpm-lock.yaml' + - 'pnpm-workspace.yaml' + - 'knip.config.ts' + - 'scripts/check-web-production-unused-after-knip-fix.mjs' + - '.github/workflows/style.yml' + - '.github/actions/setup-web/**' + ts-common-style: + - 'web/**' + - 'cli/**' + - 'e2e/**' + - 'sdks/nodejs-client/**' + - 'packages/**' + - 'package.json' + - 'pnpm-lock.yaml' + - 'pnpm-workspace.yaml' + - 'vite.config.ts' + - 'lint.config.ts' + - 'eslint.config.mjs' + - 'knip.config.ts' + - 'scripts/check-web-production-unused-after-knip-fix.mjs' + - 'oxlint-suppressions.json' + - 'eslint-suppressions.json' + - '.vscode/**' + - '.github/**' + superlinter: + - '**.sh' + - '**.yaml' + - '**.yml' + - '**Dockerfile' + - 'dev/**' + - '.editorconfig' + - '.vite-hooks/**' # Run tests in parallel while always emitting stable required checks. api-tests-run: @@ -396,10 +451,19 @@ jobs: style-check: name: Style Check - needs: pre_job + if: ${{ always() }} + needs: + - pre_job + - check-changes uses: ./.github/workflows/style.yml with: base-rev: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha }} + classification-succeeded: ${{ needs.check-changes.result == 'success' }} + run-python-style: ${{ needs.check-changes.outputs.python-style-changed == 'true' }} + run-dify-agent-style: ${{ needs.check-changes.outputs.dify-agent-style-changed == 'true' }} + run-web-style: ${{ needs.check-changes.outputs.web-style-changed == 'true' }} + run-ts-common-style: ${{ needs.check-changes.outputs.ts-common-style-changed == 'true' }} + run-superlinter: ${{ needs.check-changes.outputs.superlinter-changed == 'true' }} skip-checks: ${{ needs.pre_job.outputs.should_skip == 'true' }} vdb-tests-run: diff --git a/.github/workflows/style.yml b/.github/workflows/style.yml index ca9d93d136e..11fb97b6963 100644 --- a/.github/workflows/style.yml +++ b/.github/workflows/style.yml @@ -6,6 +6,30 @@ on: base-rev: required: true type: string + classification-succeeded: + description: Whether the caller successfully classified the changed files. + required: true + type: boolean + run-python-style: + description: Run Python style checks for the changed files. + required: true + type: boolean + run-dify-agent-style: + description: Run Dify Agent style checks for the changed files. + required: true + type: boolean + run-web-style: + description: Run Web style checks for the changed files. + required: true + type: boolean + run-ts-common-style: + description: Run shared TypeScript style checks for the changed files. + required: true + type: boolean + run-superlinter: + description: Run SuperLinter for the changed files. + required: true + type: boolean skip-checks: description: Create the required check runs without repeating previously successful work. required: false @@ -20,72 +44,72 @@ permissions: jobs: python-style: name: Python Style - if: ${{ !inputs.skip-checks }} + if: ${{ !inputs.skip-checks && (!inputs.classification-succeeded || inputs.run-python-style || inputs.run-dify-agent-style) }} runs-on: depot-ubuntu-24.04 steps: + - name: Verify changed-file classification + if: ${{ !inputs.classification-succeeded }} + run: | + echo "Changed-file classification failed; refusing to skip Python style checks." >&2 + exit 1 + - name: Checkout code uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false fetch-depth: 0 - - name: Check changed files - id: changed-files - uses: tj-actions/changed-files@9426d40962ed5378910ee2e21d5f8c6fcbf2dd96 # v47.0.6 - with: - files: | - api/** - scripts/ast_grep_guard.py - scripts/check_no_new_getattr.py - scripts/check_no_new_controller_sqlalchemy.py - scripts/lint_controller_sqlalchemy.py - scripts/ast_grep_rules/no_new_getattr.yml - scripts/ast_grep_rules/no_new_controller_sqlalchemy.yml - .github/workflows/style.yml - .github/workflows/main-ci.yml - - name: Setup UV and Python - if: steps.changed-files.outputs.any_changed == 'true' uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 with: enable-cache: false python-version: '3.12' - cache-dependency-glob: api/uv.lock + cache-dependency-glob: | + api/uv.lock + dify-agent/uv.lock - name: Install dependencies - if: steps.changed-files.outputs.any_changed == 'true' + if: inputs.run-python-style run: uv sync --project api --dev - name: Run Import Linter - if: steps.changed-files.outputs.any_changed == 'true' + if: inputs.run-python-style run: uv run --directory api --dev lint-imports - name: Run Response Contract Linter - if: steps.changed-files.outputs.any_changed == 'true' + if: inputs.run-python-style run: uv run --project api --dev python api/dev/lint_response_contracts.py --fail-on-mismatch - name: Run No New Getattr Guard - if: steps.changed-files.outputs.any_changed == 'true' + if: inputs.run-python-style run: uv run --project api python scripts/check_no_new_getattr.py --base-rev "${{ inputs.base-rev }}" - name: Run No New Controller SQLAlchemy Guard - if: steps.changed-files.outputs.any_changed == 'true' + if: inputs.run-python-style run: uv run --project api python scripts/check_no_new_controller_sqlalchemy.py --base-rev "${{ inputs.base-rev }}" - name: Run Type Checks - if: steps.changed-files.outputs.any_changed == 'true' + if: inputs.run-python-style env: PYREFLY_OUTPUT_FORMAT: github run: make type-check-core + - name: Run Dify Agent Ruff + if: inputs.run-dify-agent-style + run: make -C dify-agent check + + - name: Run Dify Agent Type Checks + if: inputs.run-dify-agent-style + run: make -C dify-agent typecheck + - name: Dotenv check - if: steps.changed-files.outputs.any_changed == 'true' + if: inputs.run-python-style run: uv run --project api dotenv-linter ./api/.env.example ./web/.env.example web-style: name: Web Style - if: ${{ !inputs.skip-checks }} + if: ${{ !inputs.skip-checks && (!inputs.classification-succeeded || inputs.run-web-style) }} runs-on: depot-ubuntu-24.04 defaults: run: @@ -95,126 +119,84 @@ jobs: pull-requests: read steps: + - name: Verify changed-file classification + if: ${{ !inputs.classification-succeeded }} + working-directory: . + run: | + echo "Changed-file classification failed; refusing to skip Web style checks." >&2 + exit 1 + - name: Checkout code uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false - - name: Check changed files - id: changed-files - uses: tj-actions/changed-files@9426d40962ed5378910ee2e21d5f8c6fcbf2dd96 # v47.0.6 - with: - files: | - web/** - e2e/** - sdks/nodejs-client/** - packages/** - package.json - pnpm-lock.yaml - pnpm-workspace.yaml - knip.config.ts - scripts/check-web-production-unused-after-knip-fix.mjs - .github/workflows/style.yml - .github/actions/setup-web/** - - name: Setup web environment - if: steps.changed-files.outputs.any_changed == 'true' uses: ./.github/actions/setup-web - name: Web tsslint - if: steps.changed-files.outputs.any_changed == 'true' env: NODE_OPTIONS: --max-old-space-size=4096 run: vp run lint:tss - name: Web dead code check - if: steps.changed-files.outputs.any_changed == 'true' working-directory: . run: vp run knip - name: Web dead code check production - if: steps.changed-files.outputs.any_changed == 'true' working-directory: . run: vp run knip:production - name: Web production unused declarations check - if: steps.changed-files.outputs.any_changed == 'true' working-directory: . run: vp run knip:production-unused-check ts-common-style: name: TS Common - if: ${{ !inputs.skip-checks }} + if: ${{ !inputs.skip-checks && (!inputs.classification-succeeded || inputs.run-ts-common-style) }} runs-on: depot-ubuntu-24.04-4 permissions: checks: write pull-requests: read steps: + - name: Verify changed-file classification + if: ${{ !inputs.classification-succeeded }} + run: | + echo "Changed-file classification failed; refusing to skip TypeScript style checks." >&2 + exit 1 + - name: Checkout code uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false - - name: Check changed files - id: changed-files - uses: tj-actions/changed-files@9426d40962ed5378910ee2e21d5f8c6fcbf2dd96 # v47.0.6 - with: - files: | - web/** - cli/** - e2e/** - sdks/nodejs-client/** - packages/** - package.json - pnpm-lock.yaml - pnpm-workspace.yaml - vite.config.ts - lint.config.ts - eslint.config.mjs - knip.config.ts - scripts/check-web-production-unused-after-knip-fix.mjs - oxlint-suppressions.json - eslint-suppressions.json - .vscode/** - .github/** - - name: Setup web environment - if: steps.changed-files.outputs.any_changed == 'true' uses: ./.github/actions/setup-web - name: Static check - if: steps.changed-files.outputs.any_changed == 'true' run: pnpm -w check superlinter: name: SuperLinter - if: ${{ !inputs.skip-checks }} + if: ${{ !inputs.skip-checks && (!inputs.classification-succeeded || inputs.run-superlinter) }} runs-on: depot-ubuntu-24.04 steps: + - name: Verify changed-file classification + if: ${{ !inputs.classification-succeeded }} + run: | + echo "Changed-file classification failed; refusing to skip SuperLinter." >&2 + exit 1 + - name: Checkout code uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 0 persist-credentials: false - - name: Check changed files - id: changed-files - uses: tj-actions/changed-files@9426d40962ed5378910ee2e21d5f8c6fcbf2dd96 # v47.0.6 - with: - files: | - **.sh - **.yaml - **.yml - **Dockerfile - dev/** - .editorconfig - .vite-hooks/** - - name: Super-linter uses: super-linter/super-linter/slim@4ce20838b8ab83717e78138c5b3a1407148e0918 # v8.7.0 - if: steps.changed-files.outputs.any_changed == 'true' env: BASH_SEVERITY: warning DEFAULT_BRANCH: origin/main diff --git a/api/.importlinter b/api/.importlinter index 53da8e4c6c5..3933ee443ed 100644 --- a/api/.importlinter +++ b/api/.importlinter @@ -203,6 +203,7 @@ forbidden_modules = name = Account application services and contracts are framework and persistence neutral type = forbidden source_modules = + services.account_access_service services.account_avatar_service services.account_change_email_ports services.account_change_email_service @@ -215,11 +216,14 @@ source_modules = services.account_initialization_service services.account_integration_service services.account_login_service + services.account_oauth_service services.account_password_service services.account_ports services.account_profile_service + services.entities.account_access_entities services.entities.account_entities services.entities.account_login_entities + services.entities.account_oauth_entities services.entities.auth_audit_entities forbidden_modules = configs @@ -421,6 +425,22 @@ forbidden_modules = sqlalchemy werkzeug +[importlinter:contract:file-grant-service-boundary] +name = File grant application service is framework and persistence neutral +type = forbidden +source_modules = + services.file_grant_service + services.entities.file_grant_entities +forbidden_modules = + configs + controllers + extensions + flask + models + repositories + sqlalchemy + werkzeug + [importlinter:contract:account-activation-service-boundary] name = Account activation application service is framework and persistence neutral type = forbidden diff --git a/api/README.md b/api/README.md index 4082250cc9d..5c175dea7f6 100644 --- a/api/README.md +++ b/api/README.md @@ -98,7 +98,7 @@ The scripts resolve paths relative to their location, so you can run them from a uv run pytest tests/integration_tests/ # Integration tests # Code quality - ./dev/reformat # Run all formatters and linters + ../dev/reformat # Run all formatters and linters uv run ruff check --fix ./ # Fix linting issues uv run ruff format ./ # Format code uv run pyrefly check # Type checking diff --git a/api/conftest.py b/api/conftest.py index 350b0306016..e967a723d2a 100644 --- a/api/conftest.py +++ b/api/conftest.py @@ -33,6 +33,18 @@ ensure_backend_test_environment(_REPO_ROOT) def pytest_addoption(parser: pytest.Parser) -> None: group = parser.getgroup("dify") + group.addoption( + "--shard-index", + type=int, + default=1, + help="One-based index of the test shard to run.", + ) + group.addoption( + "--shard-total", + type=int, + default=1, + help="Total number of test shards.", + ) group.addoption( "--start-middleware", action="store_true", @@ -58,9 +70,33 @@ def pytest_addoption(parser: pytest.Parser) -> None: def pytest_configure(config: pytest.Config) -> None: + shard_index = config.getoption("shard_index") + shard_total = config.getoption("shard_total") + if shard_total < 1: + raise pytest.UsageError("--shard-total must be at least 1") + if not 1 <= shard_index <= shard_total: + raise pytest.UsageError("--shard-index must be between 1 and --shard-total") + config.stash[_DIFY_COMPOSE_STACKS_KEY] = [] +def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item]) -> None: + """Select a deterministic, evenly sized slice of collected tests for this shard.""" + shard_index = config.getoption("shard_index") + shard_total = config.getoption("shard_total") + if shard_total == 1: + return + + selected: list[pytest.Item] = [] + deselected: list[pytest.Item] = [] + for item_index, item in enumerate(items): + target = selected if item_index % shard_total == shard_index - 1 else deselected + target.append(item) + + config.hook.pytest_deselected(items=deselected) + items[:] = selected + + def pytest_sessionstart(session: pytest.Session) -> None: config = session.config if hasattr(config, "workerinput"): diff --git a/api/constants/languages.py b/api/constants/languages.py index 8c1ff455363..3ae6773a149 100644 --- a/api/constants/languages.py +++ b/api/constants/languages.py @@ -8,6 +8,7 @@ language_timezone_mapping = { "de-DE": "Europe/Berlin", "ja-JP": "Asia/Tokyo", "ko-KR": "Asia/Seoul", + "lo-LA": "Asia/Vientiane", "ru-RU": "Europe/Moscow", "it-IT": "Europe/Rome", "uk-UA": "Europe/Kyiv", diff --git a/api/controllers/console/app/statistic.py b/api/controllers/console/app/statistic.py index 48635166e3a..2ebf60bdcfe 100644 --- a/api/controllers/console/app/statistic.py +++ b/api/controllers/console/app/statistic.py @@ -1,31 +1,22 @@ +from datetime import datetime from decimal import Decimal -import sqlalchemy as sa -from flask import abort from flask_restx import Resource from pydantic import BaseModel, Field, field_validator +from werkzeug.exceptions import BadRequest from controllers.common.schema import query_params_from_model, register_response_schema_models, register_schema_models from controllers.console import console_ns from controllers.console.app.wraps import get_app_model -from controllers.console.wraps import ( - RBACPermission, - RBACResourceScope, - account_initialization_required, - model_validate, - rbac_permission_required, - setup_required, - with_current_user, -) -from core.app.entities.app_invoke_entities import InvokeFrom -from extensions.ext_database import db +from controllers.console.flask_admission import console_account_admission +from controllers.console.wraps import RBACPermission, RBACResourceScope, model_validate +from extensions.ext_application_services import application_services from fields.base import ResponseModel from libs.datetime_utils import parse_time_range -from libs.helper import convert_datetime_to_date, dump_response -from libs.login import login_required -from models import AppMode -from models.account import Account -from models.model import App +from libs.helper import dump_response +from libs.login import current_account_with_tenant +from machinery.context import RequestContext +from models.model import App, AppMode class StatisticTimeRangeQuery(BaseModel): @@ -142,6 +133,20 @@ register_response_schema_models( ) +def _resolve_statistic_time_range( + req_data: StatisticTimeRangeQuery, +) -> tuple[datetime | None, datetime | None, str]: + timezone = current_account_with_tenant().account.timezone + assert timezone is not None + + try: + start_date, end_date = parse_time_range(req_data.start, req_data.end, timezone) + except ValueError as error: + raise BadRequest(str(error)) from error + + return start_date, end_date, timezone + + @console_ns.route("/apps//statistics/daily-messages") class DailyMessageStatistic(Resource): @console_ns.doc("get_daily_message_statistics") @@ -152,52 +157,20 @@ class DailyMessageStatistic(Resource): "Daily message statistics retrieved successfully", console_ns.models[DailyMessageStatisticResponse.__name__], ) - @setup_required - @login_required - @account_initialization_required - @with_current_user - @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_MONITOR) + @console_account_admission( + rbac_resource_scope=RBACResourceScope.APP, + rbac_permission=RBACPermission.APP_MONITOR, + ) @get_app_model @model_validate(StatisticTimeRangeQuery) - def get(self, req_data: StatisticTimeRangeQuery, account: Account, app_model: App): - - converted_created_at = convert_datetime_to_date("created_at") - sql_query = f"""SELECT - {converted_created_at} AS date, - COUNT(*) AS message_count -FROM - messages -WHERE - app_id = :app_id - AND invoke_from != :invoke_from""" - assert account.timezone is not None - arg_dict: dict[str, object] = { - "tz": account.timezone, - "app_id": app_model.id, - "invoke_from": InvokeFrom.DEBUGGER, - } - - try: - start_datetime_utc, end_datetime_utc = parse_time_range(req_data.start, req_data.end, account.timezone) - except ValueError as e: - abort(400, description=str(e)) - - if start_datetime_utc: - sql_query += " AND created_at >= :start" - arg_dict["start"] = start_datetime_utc - - if end_datetime_utc: - sql_query += " AND created_at < :end" - arg_dict["end"] = end_datetime_utc - - sql_query += " GROUP BY date ORDER BY date" - - response_data = [] - - with db.engine.begin() as conn: - rs = conn.execute(sa.text(sql_query), arg_dict) - for i in rs: - response_data.append({"date": str(i.date), "message_count": i.message_count}) + def get(self, req_data: StatisticTimeRangeQuery, _request_context: RequestContext, app_model: App): + start_date, end_date, timezone = _resolve_statistic_time_range(req_data) + response_data = application_services().app_statistics.get_daily_messages( + app_id=app_model.id, + start_date=start_date, + end_date=end_date, + timezone=timezone, + ) return dump_response(DailyMessageStatisticResponse, {"data": response_data}) @@ -212,51 +185,20 @@ class DailyConversationStatistic(Resource): "Daily conversation statistics retrieved successfully", console_ns.models[DailyConversationStatisticResponse.__name__], ) - @setup_required - @login_required - @account_initialization_required - @with_current_user - @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_MONITOR) + @console_account_admission( + rbac_resource_scope=RBACResourceScope.APP, + rbac_permission=RBACPermission.APP_MONITOR, + ) @get_app_model @model_validate(StatisticTimeRangeQuery) - def get(self, req_data: StatisticTimeRangeQuery, account: Account, app_model: App): - - converted_created_at = convert_datetime_to_date("created_at") - sql_query = f"""SELECT - {converted_created_at} AS date, - COUNT(DISTINCT conversation_id) AS conversation_count -FROM - messages -WHERE - app_id = :app_id - AND invoke_from != :invoke_from""" - assert account.timezone is not None - arg_dict: dict[str, object] = { - "tz": account.timezone, - "app_id": app_model.id, - "invoke_from": InvokeFrom.DEBUGGER, - } - - try: - start_datetime_utc, end_datetime_utc = parse_time_range(req_data.start, req_data.end, account.timezone) - except ValueError as e: - abort(400, description=str(e)) - - if start_datetime_utc: - sql_query += " AND created_at >= :start" - arg_dict["start"] = start_datetime_utc - - if end_datetime_utc: - sql_query += " AND created_at < :end" - arg_dict["end"] = end_datetime_utc - - sql_query += " GROUP BY date ORDER BY date" - - response_data = [] - with db.engine.begin() as conn: - rs = conn.execute(sa.text(sql_query), arg_dict) - for i in rs: - response_data.append({"date": str(i.date), "conversation_count": i.conversation_count}) + def get(self, req_data: StatisticTimeRangeQuery, _request_context: RequestContext, app_model: App): + start_date, end_date, timezone = _resolve_statistic_time_range(req_data) + response_data = application_services().app_statistics.get_daily_conversations( + app_id=app_model.id, + start_date=start_date, + end_date=end_date, + timezone=timezone, + ) return dump_response(DailyConversationStatisticResponse, {"data": response_data}) @@ -271,52 +213,20 @@ class DailyTerminalsStatistic(Resource): "Daily terminal statistics retrieved successfully", console_ns.models[DailyTerminalStatisticResponse.__name__], ) - @setup_required - @login_required - @account_initialization_required - @with_current_user - @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_MONITOR) + @console_account_admission( + rbac_resource_scope=RBACResourceScope.APP, + rbac_permission=RBACPermission.APP_MONITOR, + ) @get_app_model @model_validate(StatisticTimeRangeQuery) - def get(self, req_data: StatisticTimeRangeQuery, account: Account, app_model: App): - - converted_created_at = convert_datetime_to_date("created_at") - sql_query = f"""SELECT - {converted_created_at} AS date, - COUNT(DISTINCT messages.from_end_user_id) AS terminal_count -FROM - messages -WHERE - app_id = :app_id - AND invoke_from != :invoke_from""" - assert account.timezone is not None - arg_dict: dict[str, object] = { - "tz": account.timezone, - "app_id": app_model.id, - "invoke_from": InvokeFrom.DEBUGGER, - } - - try: - start_datetime_utc, end_datetime_utc = parse_time_range(req_data.start, req_data.end, account.timezone) - except ValueError as e: - abort(400, description=str(e)) - - if start_datetime_utc: - sql_query += " AND created_at >= :start" - arg_dict["start"] = start_datetime_utc - - if end_datetime_utc: - sql_query += " AND created_at < :end" - arg_dict["end"] = end_datetime_utc - - sql_query += " GROUP BY date ORDER BY date" - - response_data = [] - - with db.engine.begin() as conn: - rs = conn.execute(sa.text(sql_query), arg_dict) - for i in rs: - response_data.append({"date": str(i.date), "terminal_count": i.terminal_count}) + def get(self, req_data: StatisticTimeRangeQuery, _request_context: RequestContext, app_model: App): + start_date, end_date, timezone = _resolve_statistic_time_range(req_data) + response_data = application_services().app_statistics.get_daily_terminals( + app_id=app_model.id, + start_date=start_date, + end_date=end_date, + timezone=timezone, + ) return dump_response(DailyTerminalStatisticResponse, {"data": response_data}) @@ -331,55 +241,20 @@ class DailyTokenCostStatistic(Resource): "Daily token cost statistics retrieved successfully", console_ns.models[DailyTokenCostStatisticResponse.__name__], ) - @setup_required - @login_required - @account_initialization_required - @with_current_user - @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_MONITOR) + @console_account_admission( + rbac_resource_scope=RBACResourceScope.APP, + rbac_permission=RBACPermission.APP_MONITOR, + ) @get_app_model @model_validate(StatisticTimeRangeQuery) - def get(self, req_data: StatisticTimeRangeQuery, account: Account, app_model: App): - - converted_created_at = convert_datetime_to_date("created_at") - sql_query = f"""SELECT - {converted_created_at} AS date, - (SUM(messages.message_tokens) + SUM(messages.answer_tokens)) AS token_count, - SUM(total_price) AS total_price -FROM - messages -WHERE - app_id = :app_id - AND invoke_from != :invoke_from""" - assert account.timezone is not None - arg_dict: dict[str, object] = { - "tz": account.timezone, - "app_id": app_model.id, - "invoke_from": InvokeFrom.DEBUGGER, - } - - try: - start_datetime_utc, end_datetime_utc = parse_time_range(req_data.start, req_data.end, account.timezone) - except ValueError as e: - abort(400, description=str(e)) - - if start_datetime_utc: - sql_query += " AND created_at >= :start" - arg_dict["start"] = start_datetime_utc - - if end_datetime_utc: - sql_query += " AND created_at < :end" - arg_dict["end"] = end_datetime_utc - - sql_query += " GROUP BY date ORDER BY date" - - response_data = [] - - with db.engine.begin() as conn: - rs = conn.execute(sa.text(sql_query), arg_dict) - for i in rs: - response_data.append( - {"date": str(i.date), "token_count": i.token_count, "total_price": i.total_price, "currency": "USD"} - ) + def get(self, req_data: StatisticTimeRangeQuery, _request_context: RequestContext, app_model: App): + start_date, end_date, timezone = _resolve_statistic_time_range(req_data) + response_data = application_services().app_statistics.get_daily_token_costs( + app_id=app_model.id, + start_date=start_date, + end_date=end_date, + timezone=timezone, + ) return dump_response(DailyTokenCostStatisticResponse, {"data": response_data}) @@ -394,71 +269,20 @@ class AverageSessionInteractionStatistic(Resource): "Average session interaction statistics retrieved successfully", console_ns.models[AverageSessionInteractionStatisticResponse.__name__], ) - @setup_required - @login_required - @account_initialization_required - @with_current_user - @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_MONITOR) + @console_account_admission( + rbac_resource_scope=RBACResourceScope.APP, + rbac_permission=RBACPermission.APP_MONITOR, + ) @get_app_model(mode=[AppMode.CHAT, AppMode.AGENT_CHAT, AppMode.ADVANCED_CHAT, AppMode.AGENT]) @model_validate(StatisticTimeRangeQuery) - def get(self, req_data: StatisticTimeRangeQuery, account: Account, app_model: App): - - converted_created_at = convert_datetime_to_date("c.created_at") - sql_query = f"""SELECT - {converted_created_at} AS date, - AVG(subquery.message_count) AS interactions -FROM - ( - SELECT - m.conversation_id, - COUNT(m.id) AS message_count - FROM - conversations c - JOIN - messages m - ON c.id = m.conversation_id - WHERE - c.app_id = :app_id - AND m.invoke_from != :invoke_from""" - assert account.timezone is not None - arg_dict: dict[str, object] = { - "tz": account.timezone, - "app_id": app_model.id, - "invoke_from": InvokeFrom.DEBUGGER, - } - - try: - start_datetime_utc, end_datetime_utc = parse_time_range(req_data.start, req_data.end, account.timezone) - except ValueError as e: - abort(400, description=str(e)) - - if start_datetime_utc: - sql_query += " AND c.created_at >= :start" - arg_dict["start"] = start_datetime_utc - - if end_datetime_utc: - sql_query += " AND c.created_at < :end" - arg_dict["end"] = end_datetime_utc - - sql_query += """ - GROUP BY m.conversation_id - ) subquery -LEFT JOIN - conversations c - ON c.id = subquery.conversation_id -GROUP BY - date -ORDER BY - date""" - - response_data = [] - - with db.engine.begin() as conn: - rs = conn.execute(sa.text(sql_query), arg_dict) - for i in rs: - response_data.append( - {"date": str(i.date), "interactions": float(i.interactions.quantize(Decimal("0.01")))} - ) + def get(self, req_data: StatisticTimeRangeQuery, _request_context: RequestContext, app_model: App): + start_date, end_date, timezone = _resolve_statistic_time_range(req_data) + response_data = application_services().app_statistics.get_average_session_interactions( + app_id=app_model.id, + start_date=start_date, + end_date=end_date, + timezone=timezone, + ) return dump_response(AverageSessionInteractionStatisticResponse, {"data": response_data}) @@ -473,61 +297,20 @@ class UserSatisfactionRateStatistic(Resource): "User satisfaction rate statistics retrieved successfully", console_ns.models[UserSatisfactionRateStatisticResponse.__name__], ) - @setup_required - @login_required - @account_initialization_required - @with_current_user - @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_MONITOR) + @console_account_admission( + rbac_resource_scope=RBACResourceScope.APP, + rbac_permission=RBACPermission.APP_MONITOR, + ) @get_app_model @model_validate(StatisticTimeRangeQuery) - def get(self, req_data: StatisticTimeRangeQuery, account: Account, app_model: App): - - converted_created_at = convert_datetime_to_date("m.created_at") - sql_query = f"""SELECT - {converted_created_at} AS date, - COUNT(m.id) AS message_count, - COUNT(mf.id) AS feedback_count -FROM - messages m -LEFT JOIN - message_feedbacks mf - ON mf.message_id=m.id AND mf.rating='like' -WHERE - m.app_id = :app_id - AND m.invoke_from != :invoke_from""" - assert account.timezone is not None - arg_dict: dict[str, object] = { - "tz": account.timezone, - "app_id": app_model.id, - "invoke_from": InvokeFrom.DEBUGGER, - } - - try: - start_datetime_utc, end_datetime_utc = parse_time_range(req_data.start, req_data.end, account.timezone) - except ValueError as e: - abort(400, description=str(e)) - - if start_datetime_utc: - sql_query += " AND m.created_at >= :start" - arg_dict["start"] = start_datetime_utc - - if end_datetime_utc: - sql_query += " AND m.created_at < :end" - arg_dict["end"] = end_datetime_utc - - sql_query += " GROUP BY date ORDER BY date" - - response_data = [] - - with db.engine.begin() as conn: - rs = conn.execute(sa.text(sql_query), arg_dict) - for i in rs: - response_data.append( - { - "date": str(i.date), - "rate": round((i.feedback_count * 1000 / i.message_count) if i.message_count > 0 else 0, 2), - } - ) + def get(self, req_data: StatisticTimeRangeQuery, _request_context: RequestContext, app_model: App): + start_date, end_date, timezone = _resolve_statistic_time_range(req_data) + response_data = application_services().app_statistics.get_user_satisfaction_rates( + app_id=app_model.id, + start_date=start_date, + end_date=end_date, + timezone=timezone, + ) return dump_response(UserSatisfactionRateStatisticResponse, {"data": response_data}) @@ -542,52 +325,20 @@ class AverageResponseTimeStatistic(Resource): "Average response time statistics retrieved successfully", console_ns.models[AverageResponseTimeStatisticResponse.__name__], ) - @setup_required - @login_required - @account_initialization_required - @with_current_user - @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_MONITOR) + @console_account_admission( + rbac_resource_scope=RBACResourceScope.APP, + rbac_permission=RBACPermission.APP_MONITOR, + ) @get_app_model(mode=AppMode.COMPLETION) @model_validate(StatisticTimeRangeQuery) - def get(self, req_data: StatisticTimeRangeQuery, account: Account, app_model: App): - - converted_created_at = convert_datetime_to_date("created_at") - sql_query = f"""SELECT - {converted_created_at} AS date, - AVG(provider_response_latency) AS latency -FROM - messages -WHERE - app_id = :app_id - AND invoke_from != :invoke_from""" - assert account.timezone is not None - arg_dict: dict[str, object] = { - "tz": account.timezone, - "app_id": app_model.id, - "invoke_from": InvokeFrom.DEBUGGER, - } - - try: - start_datetime_utc, end_datetime_utc = parse_time_range(req_data.start, req_data.end, account.timezone) - except ValueError as e: - abort(400, description=str(e)) - - if start_datetime_utc: - sql_query += " AND created_at >= :start" - arg_dict["start"] = start_datetime_utc - - if end_datetime_utc: - sql_query += " AND created_at < :end" - arg_dict["end"] = end_datetime_utc - - sql_query += " GROUP BY date ORDER BY date" - - response_data = [] - - with db.engine.begin() as conn: - rs = conn.execute(sa.text(sql_query), arg_dict) - for i in rs: - response_data.append({"date": str(i.date), "latency": round(i.latency * 1000, 4)}) + def get(self, req_data: StatisticTimeRangeQuery, _request_context: RequestContext, app_model: App): + start_date, end_date, timezone = _resolve_statistic_time_range(req_data) + response_data = application_services().app_statistics.get_average_response_times( + app_id=app_model.id, + start_date=start_date, + end_date=end_date, + timezone=timezone, + ) return dump_response(AverageResponseTimeStatisticResponse, {"data": response_data}) @@ -602,54 +353,19 @@ class TokensPerSecondStatistic(Resource): "Tokens per second statistics retrieved successfully", console_ns.models[TokensPerSecondStatisticResponse.__name__], ) - @setup_required - @login_required - @account_initialization_required - @with_current_user - @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_MONITOR) + @console_account_admission( + rbac_resource_scope=RBACResourceScope.APP, + rbac_permission=RBACPermission.APP_MONITOR, + ) @get_app_model @model_validate(StatisticTimeRangeQuery) - def get(self, req_data: StatisticTimeRangeQuery, account: Account, app_model: App): - - converted_created_at = convert_datetime_to_date("created_at") - sql_query = f"""SELECT - {converted_created_at} AS date, - CASE - WHEN SUM(provider_response_latency) = 0 THEN 0 - ELSE (SUM(answer_tokens) / SUM(provider_response_latency)) - END as tokens_per_second -FROM - messages -WHERE - app_id = :app_id - AND invoke_from != :invoke_from""" - assert account.timezone is not None - arg_dict: dict[str, object] = { - "tz": account.timezone, - "app_id": app_model.id, - "invoke_from": InvokeFrom.DEBUGGER, - } - - try: - start_datetime_utc, end_datetime_utc = parse_time_range(req_data.start, req_data.end, account.timezone) - except ValueError as e: - abort(400, description=str(e)) - - if start_datetime_utc: - sql_query += " AND created_at >= :start" - arg_dict["start"] = start_datetime_utc - - if end_datetime_utc: - sql_query += " AND created_at < :end" - arg_dict["end"] = end_datetime_utc - - sql_query += " GROUP BY date ORDER BY date" - - response_data = [] - - with db.engine.begin() as conn: - rs = conn.execute(sa.text(sql_query), arg_dict) - for i in rs: - response_data.append({"date": str(i.date), "tps": round(i.tokens_per_second, 4)}) + def get(self, req_data: StatisticTimeRangeQuery, _request_context: RequestContext, app_model: App): + start_date, end_date, timezone = _resolve_statistic_time_range(req_data) + response_data = application_services().app_statistics.get_tokens_per_second( + app_id=app_model.id, + start_date=start_date, + end_date=end_date, + timezone=timezone, + ) return dump_response(TokensPerSecondStatisticResponse, {"data": response_data}) diff --git a/api/controllers/console/app/workflow_app_log.py b/api/controllers/console/app/workflow_app_log.py index 78db9c346d7..e74abd03f66 100644 --- a/api/controllers/console/app/workflow_app_log.py +++ b/api/controllers/console/app/workflow_app_log.py @@ -4,29 +4,25 @@ from typing import Any from dateutil.parser import isoparse from flask_restx import Resource from pydantic import BaseModel, Field, field_validator -from sqlalchemy.orm import sessionmaker from controllers.common.schema import query_params_from_model, register_schema_models from controllers.console import console_ns from controllers.console.app.wraps import get_app_model +from controllers.console.flask_admission import console_account_admission from controllers.console.wraps import ( RBACPermission, RBACResourceScope, - account_initialization_required, model_validate, - rbac_permission_required, - setup_required, ) -from extensions.ext_database import db +from extensions.ext_application_services import application_services from fields.base import ResponseModel from fields.end_user_fields import SimpleEndUser from fields.member_fields import SimpleAccount from graphon.enums import WorkflowExecutionStatus -from libs.helper import to_timestamp -from libs.login import login_required +from libs.helper import dump_response, to_timestamp +from machinery.context import RequestContext from models import App from models.model import AppMode -from services.workflow_app_service import WorkflowAppService class WorkflowAppLogQuery(BaseModel): @@ -136,34 +132,32 @@ class WorkflowAppLogApi(Resource): "Workflow app logs retrieved successfully", console_ns.models[WorkflowAppLogPaginationResponse.__name__], ) - @setup_required - @login_required - @account_initialization_required - @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_LOG_AND_ANNOTATION) + @console_account_admission( + rbac_resource_scope=RBACResourceScope.APP, + rbac_permission=RBACPermission.APP_LOG_AND_ANNOTATION, + ) @get_app_model(mode=[AppMode.WORKFLOW]) @model_validate(WorkflowAppLogQuery) - def get(self, req_data: WorkflowAppLogQuery, app_model: App): + def get( + self, + req_data: WorkflowAppLogQuery, + _request_context: RequestContext, + app_model: App, + ): """ Get workflow app logs """ - - # get paginate workflow app logs - workflow_app_service = WorkflowAppService() - with sessionmaker(db.engine, expire_on_commit=False).begin() as session: - workflow_app_log_pagination = workflow_app_service.get_paginate_workflow_app_logs( - session=session, - app_model=app_model, - keyword=req_data.keyword, - status=req_data.status, - created_at_before=req_data.created_at__before, - created_at_after=req_data.created_at__after, - page=req_data.page, - limit=req_data.limit, - detail=req_data.detail, - created_by_end_user_session_id=req_data.created_by_end_user_session_id, - created_by_account=req_data.created_by_account, - ) - - return WorkflowAppLogPaginationResponse.model_validate( - workflow_app_log_pagination, from_attributes=True - ).model_dump(mode="json") + result = application_services().workflow_app_logs.list_logs( + tenant_id=app_model.tenant_id, + app_id=app_model.id, + keyword=req_data.keyword, + status=req_data.status, + created_at_before=req_data.created_at__before, + created_at_after=req_data.created_at__after, + page=req_data.page, + limit=req_data.limit, + detail=req_data.detail, + created_by_end_user_session_id=req_data.created_by_end_user_session_id, + created_by_account=req_data.created_by_account, + ) + return dump_response(WorkflowAppLogPaginationResponse, result) diff --git a/api/controllers/console/app/workflow_run.py b/api/controllers/console/app/workflow_run.py index 26606e1c174..8a4b179439c 100644 --- a/api/controllers/console/app/workflow_run.py +++ b/api/controllers/console/app/workflow_run.py @@ -3,26 +3,19 @@ from uuid import UUID from flask_restx import Resource from pydantic import BaseModel, Field, field_validator -from sqlalchemy.orm import sessionmaker from configs import dify_config from controllers.common.errors import NotFoundError from controllers.common.schema import query_params_from_model, register_response_schema_models, register_schema_models from controllers.console import console_ns from controllers.console.app.wraps import get_app_model +from controllers.console.flask_admission import console_account_admission from controllers.console.wraps import ( RBACPermission, RBACResourceScope, - account_initialization_required, model_validate, - rbac_permission_required, - setup_required, - with_current_tenant_id, - with_current_user, ) -from core.workflow.human_input_forms import load_form_tokens_by_form_id as _load_form_tokens_by_form_id -from core.workflow.nodes.human_input.pause_reason import HumanInputRequired -from extensions.ext_database import db +from extensions.ext_application_services import application_services from fields.base import ResponseModel from fields.workflow_run_fields import ( AdvancedChatWorkflowRunPaginationResponse, @@ -32,14 +25,11 @@ from fields.workflow_run_fields import ( WorkflowRunNodeExecutionResponse, WorkflowRunPaginationResponse, ) -from graphon.enums import WorkflowExecutionStatus from libs.custom_inputs import time_duration -from libs.helper import uuid_value -from libs.login import login_required -from models import Account, App, AppMode, WorkflowRunTriggeredFrom -from models.workflow import WorkflowRun -from repositories.factory import DifyAPIRepositoryFactory -from services.workflow_run_service import WorkflowRunListArgs, WorkflowRunService +from libs.helper import dump_response, uuid_value +from machinery.context import RequestContext +from models import App, AppMode, WorkflowRunTriggeredFrom +from services.workflow_run_service import WorkflowRunListArgs def _build_backstage_input_url(form_token: str | None) -> str | None: @@ -51,10 +41,6 @@ def _build_backstage_input_url(form_token: str | None) -> str | None: return f"{base_url.rstrip('/')}/form/{form_token}" -# Workflow run status choices for filtering -WORKFLOW_RUN_STATUS_CHOICES = ["running", "succeeded", "failed", "stopped", "partial-succeeded"] - - class WorkflowRunListQuery(BaseModel): last_id: str | None = Field(default=None, description="Last run ID for pagination") limit: int = Field(default=20, ge=1, le=100, description="Number of items per page (1-100)") @@ -96,6 +82,19 @@ class WorkflowRunCountQuery(BaseModel): return time_duration(value) +def _workflow_run_list_args(req_data: WorkflowRunListQuery) -> WorkflowRunListArgs: + args: WorkflowRunListArgs = {"limit": req_data.limit} + if req_data.last_id is not None: + args["last_id"] = req_data.last_id + if req_data.status is not None: + args["status"] = req_data.status + return args + + +def _triggered_from(value: str | None) -> WorkflowRunTriggeredFrom: + return WorkflowRunTriggeredFrom(value) if value else WorkflowRunTriggeredFrom.DEBUGGING + + class HumanInputPauseTypeResponse(ResponseModel): type: Literal["human_input"] form_id: str @@ -143,37 +142,24 @@ class AdvancedChatAppWorkflowRunListApi(Resource): "Workflow runs retrieved successfully", console_ns.models[AdvancedChatWorkflowRunPaginationResponse.__name__], ) - @setup_required - @login_required - @account_initialization_required - @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_CREATE_AND_MANAGEMENT) + @console_account_admission( + rbac_resource_scope=RBACResourceScope.APP, + rbac_permission=RBACPermission.APP_CREATE_AND_MANAGEMENT, + ) @get_app_model(mode=[AppMode.ADVANCED_CHAT]) @model_validate(WorkflowRunListQuery) - def get(self, req_data: WorkflowRunListQuery, app_model: App): + def get(self, req_data: WorkflowRunListQuery, request_context: RequestContext, app_model: App): """ Get advanced chat app workflow run list """ - args: WorkflowRunListArgs = {"limit": req_data.limit} - if req_data.last_id is not None: - args["last_id"] = req_data.last_id - if req_data.status is not None: - args["status"] = req_data.status - - # Default to DEBUGGING if not specified - triggered_from = ( - WorkflowRunTriggeredFrom(req_data.triggered_from) - if req_data.triggered_from - else WorkflowRunTriggeredFrom.DEBUGGING + result = application_services().workflow_runs.get_paginate_advanced_chat_workflow_runs( + request_context, + app_id=app_model.id, + args=_workflow_run_list_args(req_data), + triggered_from=_triggered_from(req_data.triggered_from), ) - workflow_run_service = WorkflowRunService() - result = workflow_run_service.get_paginate_advanced_chat_workflow_runs( - app_model=app_model, args=args, triggered_from=triggered_from - ) - - return AdvancedChatWorkflowRunPaginationResponse.model_validate(result, from_attributes=True).model_dump( - mode="json" - ) + return dump_response(AdvancedChatWorkflowRunPaginationResponse, result) @console_ns.route("/apps//advanced-chat/workflow-runs/count") @@ -187,34 +173,25 @@ class AdvancedChatAppWorkflowRunCountApi(Resource): "Workflow runs count retrieved successfully", console_ns.models[WorkflowRunCountResponse.__name__], ) - @setup_required - @login_required - @account_initialization_required - @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_CREATE_AND_MANAGEMENT) + @console_account_admission( + rbac_resource_scope=RBACResourceScope.APP, + rbac_permission=RBACPermission.APP_CREATE_AND_MANAGEMENT, + ) @get_app_model(mode=[AppMode.ADVANCED_CHAT]) @model_validate(WorkflowRunCountQuery) - def get(self, req_data: WorkflowRunCountQuery, app_model: App): + def get(self, req_data: WorkflowRunCountQuery, request_context: RequestContext, app_model: App): """ Get advanced chat workflow runs count statistics """ - args = req_data.model_dump(exclude_none=True) - - # Default to DEBUGGING if not specified - triggered_from = ( - WorkflowRunTriggeredFrom(req_data.triggered_from) - if req_data.triggered_from - else WorkflowRunTriggeredFrom.DEBUGGING + result = application_services().workflow_runs.get_workflow_runs_count( + request_context, + app_id=app_model.id, + status=req_data.status, + time_range=req_data.time_range, + triggered_from=_triggered_from(req_data.triggered_from), ) - workflow_run_service = WorkflowRunService() - result = workflow_run_service.get_workflow_runs_count( - app_model=app_model, - status=args.get("status"), - time_range=args.get("time_range"), - triggered_from=triggered_from, - ) - - return WorkflowRunCountResponse.model_validate(result).model_dump(mode="json") + return dump_response(WorkflowRunCountResponse, result) @console_ns.route("/apps//workflow-runs") @@ -228,35 +205,24 @@ class WorkflowRunListApi(Resource): "Workflow runs retrieved successfully", console_ns.models[WorkflowRunPaginationResponse.__name__], ) - @setup_required - @login_required - @account_initialization_required - @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_CREATE_AND_MANAGEMENT) + @console_account_admission( + rbac_resource_scope=RBACResourceScope.APP, + rbac_permission=RBACPermission.APP_CREATE_AND_MANAGEMENT, + ) @get_app_model(mode=[AppMode.ADVANCED_CHAT, AppMode.WORKFLOW]) @model_validate(WorkflowRunListQuery) - def get(self, req_data: WorkflowRunListQuery, app_model: App): + def get(self, req_data: WorkflowRunListQuery, request_context: RequestContext, app_model: App): """ Get workflow run list """ - args: WorkflowRunListArgs = {"limit": req_data.limit} - if req_data.last_id is not None: - args["last_id"] = req_data.last_id - if req_data.status is not None: - args["status"] = req_data.status - - # Default to DEBUGGING for workflow if not specified (backward compatibility) - triggered_from = ( - WorkflowRunTriggeredFrom(req_data.triggered_from) - if req_data.triggered_from - else WorkflowRunTriggeredFrom.DEBUGGING + result = application_services().workflow_runs.get_paginate_workflow_runs( + request_context, + app_id=app_model.id, + args=_workflow_run_list_args(req_data), + triggered_from=_triggered_from(req_data.triggered_from), ) - workflow_run_service = WorkflowRunService() - result = workflow_run_service.get_paginate_workflow_runs( - app_model=app_model, args=args, triggered_from=triggered_from - ) - - return WorkflowRunPaginationResponse.model_validate(result, from_attributes=True).model_dump(mode="json") + return dump_response(WorkflowRunPaginationResponse, result) @console_ns.route("/apps//workflow-runs/count") @@ -270,34 +236,25 @@ class WorkflowRunCountApi(Resource): "Workflow runs count retrieved successfully", console_ns.models[WorkflowRunCountResponse.__name__], ) - @setup_required - @login_required - @account_initialization_required - @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_CREATE_AND_MANAGEMENT) + @console_account_admission( + rbac_resource_scope=RBACResourceScope.APP, + rbac_permission=RBACPermission.APP_CREATE_AND_MANAGEMENT, + ) @get_app_model(mode=[AppMode.ADVANCED_CHAT, AppMode.WORKFLOW]) @model_validate(WorkflowRunCountQuery) - def get(self, req_data: WorkflowRunCountQuery, app_model: App): + def get(self, req_data: WorkflowRunCountQuery, request_context: RequestContext, app_model: App): """ Get workflow runs count statistics """ - args = req_data.model_dump(exclude_none=True) - - # Default to DEBUGGING for workflow if not specified (backward compatibility) - triggered_from = ( - WorkflowRunTriggeredFrom(req_data.triggered_from) - if req_data.triggered_from - else WorkflowRunTriggeredFrom.DEBUGGING + result = application_services().workflow_runs.get_workflow_runs_count( + request_context, + app_id=app_model.id, + status=req_data.status, + time_range=req_data.time_range, + triggered_from=_triggered_from(req_data.triggered_from), ) - workflow_run_service = WorkflowRunService() - result = workflow_run_service.get_workflow_runs_count( - app_model=app_model, - status=args.get("status"), - time_range=args.get("time_range"), - triggered_from=triggered_from, - ) - - return WorkflowRunCountResponse.model_validate(result).model_dump(mode="json") + return dump_response(WorkflowRunCountResponse, result) @console_ns.route("/apps//workflow-runs/") @@ -311,23 +268,24 @@ class WorkflowRunDetailApi(Resource): console_ns.models[WorkflowRunDetailResponse.__name__], ) @console_ns.response(404, "Workflow run not found") - @setup_required - @login_required - @account_initialization_required - @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_CREATE_AND_MANAGEMENT) + @console_account_admission( + rbac_resource_scope=RBACResourceScope.APP, + rbac_permission=RBACPermission.APP_CREATE_AND_MANAGEMENT, + ) @get_app_model(mode=[AppMode.ADVANCED_CHAT, AppMode.WORKFLOW]) - def get(self, app_model: App, run_id: UUID): + def get(self, request_context: RequestContext, app_model: App, run_id: UUID): """ Get workflow run detail """ - run_id_str = str(run_id) - - workflow_run_service = WorkflowRunService() - workflow_run = workflow_run_service.get_workflow_run(app_model=app_model, run_id=run_id_str) + workflow_run = application_services().workflow_runs.get_workflow_run( + request_context, + app_id=app_model.id, + run_id=str(run_id), + ) if workflow_run is None: raise NotFoundError("Workflow run not found") - return WorkflowRunDetailResponse.model_validate(workflow_run, from_attributes=True).model_dump(mode="json") + return dump_response(WorkflowRunDetailResponse, workflow_run) @console_ns.route("/apps//workflow-runs//node-executions") @@ -341,28 +299,22 @@ class WorkflowRunNodeExecutionListApi(Resource): console_ns.models[WorkflowRunNodeExecutionListResponse.__name__], ) @console_ns.response(404, "Workflow run not found") - @setup_required - @login_required - @account_initialization_required - @with_current_user - @rbac_permission_required(RBACResourceScope.APP, RBACPermission.APP_CREATE_AND_MANAGEMENT) + @console_account_admission( + rbac_resource_scope=RBACResourceScope.APP, + rbac_permission=RBACPermission.APP_CREATE_AND_MANAGEMENT, + ) @get_app_model(mode=[AppMode.ADVANCED_CHAT, AppMode.WORKFLOW]) - def get(self, current_user: Account, app_model: App, run_id: UUID): + def get(self, request_context: RequestContext, app_model: App, run_id: UUID): """ Get workflow run node execution list """ - run_id_str = str(run_id) - - workflow_run_service = WorkflowRunService() - node_executions = workflow_run_service.get_workflow_run_node_executions( - app_model=app_model, - run_id=run_id_str, - user=current_user, + node_executions = application_services().workflow_runs.get_workflow_run_node_executions( + request_context, + app_id=app_model.id, + run_id=str(run_id), ) - return WorkflowRunNodeExecutionListResponse.model_validate( - {"data": node_executions}, from_attributes=True - ).model_dump(mode="json") + return dump_response(WorkflowRunNodeExecutionListResponse, {"data": node_executions}) @console_ns.route("/workflow//pause-details") @@ -378,11 +330,8 @@ class ConsoleWorkflowPauseDetailsApi(Resource): console_ns.models[WorkflowPauseDetailsResponse.__name__], ) @console_ns.response(404, "Workflow run not found") - @setup_required - @login_required - @account_initialization_required - @with_current_tenant_id - def get(self, current_tenant_id: str, workflow_run_id: str): + @console_account_admission() + def get(self, request_context: RequestContext, workflow_run_id: str): """ Get workflow pause details. @@ -391,51 +340,31 @@ class ConsoleWorkflowPauseDetailsApi(Resource): Returns information about why and where the workflow is paused. """ - # Query WorkflowRun to determine if workflow is suspended - session_maker = sessionmaker(bind=db.engine) - workflow_run_repo = DifyAPIRepositoryFactory.create_api_workflow_run_repository(session_maker=session_maker) - - workflow_run = db.session.get(WorkflowRun, workflow_run_id) - if not workflow_run: + details = application_services().workflow_runs.get_pause_details( + request_context, + workflow_run_id=workflow_run_id, + ) + if details is None: raise NotFoundError("Workflow run not found") - if workflow_run.tenant_id != current_tenant_id: - raise NotFoundError("Workflow run not found") - - # Check if workflow is suspended - is_paused = workflow_run.status == WorkflowExecutionStatus.PAUSED - if not is_paused: - empty_response = WorkflowPauseDetailsResponse(paused_at=None, paused_nodes=[]) - return empty_response.model_dump(mode="json"), 200 - - pause_entity = workflow_run_repo.get_workflow_pause(workflow_run_id) - pause_reasons = pause_entity.get_pause_reasons() if pause_entity else [] - form_tokens_by_form_id = _load_form_tokens_by_form_id( - [reason.form_id for reason in pause_reasons if isinstance(reason, HumanInputRequired)] + return ( + dump_response( + WorkflowPauseDetailsResponse, + { + "paused_at": details.paused_at.isoformat() + "Z" if details.paused_at else None, + "paused_nodes": [ + { + "node_id": node.node_id, + "node_title": node.node_title, + "pause_type": { + "type": "human_input", + "form_id": node.form_id, + "backstage_input_url": _build_backstage_input_url(node.form_token), + }, + } + for node in details.paused_nodes + ], + }, + ), + 200, ) - - # Build response - paused_at = pause_entity.paused_at if pause_entity else None - paused_nodes: list[PausedNodeResponse] = [] - - for reason in pause_reasons: - if isinstance(reason, HumanInputRequired): - paused_nodes.append( - PausedNodeResponse( - node_id=reason.node_id, - node_title=reason.node_title, - pause_type=HumanInputPauseTypeResponse( - type="human_input", - form_id=reason.form_id, - backstage_input_url=_build_backstage_input_url(form_tokens_by_form_id.get(reason.form_id)), - ), - ) - ) - else: - raise AssertionError("unimplemented.") - - response = WorkflowPauseDetailsResponse( - paused_at=paused_at.isoformat() + "Z" if paused_at else None, - paused_nodes=paused_nodes, - ) - return response.model_dump(mode="json"), 200 diff --git a/api/controllers/console/auth/oauth.py b/api/controllers/console/auth/oauth.py index 58f13f33511..5ff5a86c3d5 100644 --- a/api/controllers/console/auth/oauth.py +++ b/api/controllers/console/auth/oauth.py @@ -1,47 +1,51 @@ -import logging import urllib.parse -import httpx -from flask import current_app, redirect, request +from flask import redirect, request from flask_restx import Resource from pydantic import BaseModel, Field -from werkzeug.exceptions import Unauthorized from werkzeug.wrappers import Response from configs import dify_config from constants.languages import languages from controllers.common.fields import RedirectResponse -from controllers.common.schema import query_params_from_model, register_response_schema_model, register_schema_models +from controllers.common.schema import query_params_from_model, register_response_schema_models, register_schema_models from controllers.console.error import AccountInFreezeError, EmailDomainSuspendedError -from enums import DeploymentEdition -from extensions.ext_database import db -from libs.datetime_utils import naive_utc_now -from libs.helper import extract_remote_ip +from controllers.console.wraps import model_validate, setup_required, social_oauth_login_enabled +from extensions.ext_application_services import application_services +from fields.base import ResponseModel +from libs.helper import dump_response, extract_remote_ip from libs.helper import timezone as validate_timezone_string -from libs.oauth import GitHubOAuth, GoogleOAuth, OAuthUserInfo, decode_oauth_state +from libs.oauth import decode_oauth_state from libs.token import ( set_access_token_to_cookie, set_csrf_token_to_cookie, set_refresh_token_to_cookie, ) -from models import Account, AccountStatus -from services.account_service import AccountService, RegisterService, TenantService -from services.billing_service import BillingService -from services.errors.account import ( - AccountNotFoundError, - AccountRegisterError, - SeatsLimitExceededError, +from services.account_errors import ( + AccountEmailDomainSuspendedError, + AccountEmailFrozenError, + InvalidOAuthInvitationError, + InvalidOAuthProviderError, + OAuthAccountBannedError, + OAuthAccountNotFoundError, + OAuthIdentityLockUnavailableError, + OAuthInvitationAccountMismatchError, + OAuthProviderAuthorizationError, + OAuthProviderRequestError, + OAuthRegistrationError, + OAuthSeatsLimitExceededError, + OAuthWorkspaceCreationNotAllowedError, ) -from services.errors.account import ( - EmailDomainSuspendedError as EmailDomainSuspendedRegistrationError, +from services.entities.account_entities import AccountSessionTokens +from services.entities.account_oauth_entities import ( + OAuthAuthorizationRequest, + OAuthCallbackCommand, + OAuthCallbackResult, + OAuthInvitationResult, ) -from services.errors.workspace import WorkSpaceNotAllowedCreateError, WorkSpaceNotFoundError -from services.system_feature_service import SystemFeatureService from .. import console_ns -logger = logging.getLogger(__name__) - class OAuthLoginQuery(BaseModel): invite_token: str | None = Field(default=None, description="Optional invitation token") @@ -55,31 +59,12 @@ class OAuthCallbackQuery(BaseModel): state: str | None = Field(default=None, description="OAuth state parameter") +class OAuthErrorResponse(ResponseModel): + error: str = Field(description="OAuth error message") + + register_schema_models(console_ns, OAuthLoginQuery, OAuthCallbackQuery) -register_response_schema_model(console_ns, RedirectResponse) - - -def get_oauth_providers(): - with current_app.app_context(): - if not dify_config.GITHUB_CLIENT_ID or not dify_config.GITHUB_CLIENT_SECRET: - github_oauth = None - else: - github_oauth = GitHubOAuth( - client_id=dify_config.GITHUB_CLIENT_ID, - client_secret=dify_config.GITHUB_CLIENT_SECRET, - redirect_uri=dify_config.CONSOLE_API_URL + "/console/api/oauth/authorize/github", - ) - if not dify_config.GOOGLE_CLIENT_ID or not dify_config.GOOGLE_CLIENT_SECRET: - google_oauth = None - else: - google_oauth = GoogleOAuth( - client_id=dify_config.GOOGLE_CLIENT_ID, - client_secret=dify_config.GOOGLE_CLIENT_SECRET, - redirect_uri=dify_config.CONSOLE_API_URL + "/console/api/oauth/authorize/google", - ) - - OAUTH_PROVIDERS = {"github": github_oauth, "google": google_oauth} - return OAUTH_PROVIDERS +register_response_schema_models(console_ns, RedirectResponse, OAuthErrorResponse) def _validated_timezone(value: str | None) -> str | None: @@ -97,22 +82,33 @@ def _validated_language(value: str | None) -> str | None: return None -def _url_origin(url: str) -> tuple[str, str, int] | None: - parsed_url = urllib.parse.urlsplit(url) - if parsed_url.scheme not in {"http", "https"} or parsed_url.hostname is None: - return None - - try: - port = parsed_url.port - except ValueError: - return None - - if port is None: - port = 443 if parsed_url.scheme == "https" else 80 - return parsed_url.scheme, parsed_url.hostname, port +def _preferred_interface_language() -> str | None: + preferred_lang = request.accept_languages.best_match(languages) + if preferred_lang and preferred_lang in languages: + return preferred_lang + return None -def _get_redirect_target(redirect_url: str | None) -> str: +def _redirect_with_console_session(tokens: AccountSessionTokens, target_url: str) -> Response: + """Attach application-issued Console session cookies to a redirect response.""" + response = redirect(target_url) + set_access_token_to_cookie(request, response, tokens.access_token) + set_refresh_token_to_cookie(request, response, tokens.refresh_token) + set_csrf_token_to_cookie(request, response, tokens.csrf_token) + return response + + +def _oauth_callback_target(result: OAuthCallbackResult, requested_redirect: str | None) -> str: + if isinstance(result, OAuthInvitationResult): + query = urllib.parse.urlencode({"invite_token": result.invite_token}) + return f"{dify_config.CONSOLE_WEB_URL}/signin/invite-settings?{query}" + + target_url = _safe_console_redirect_target(requested_redirect) + query_char = "&" if "?" in target_url else "?" + return f"{target_url}{query_char}oauth_new_user={str(result.oauth_new_user).lower()}" + + +def _safe_console_redirect_target(redirect_url: str | None) -> str: if not redirect_url: return dify_config.CONSOLE_WEB_URL @@ -127,28 +123,22 @@ def _get_redirect_target(redirect_url: str | None) -> str: return dify_config.CONSOLE_WEB_URL -def _preferred_interface_language(language: str | None = None) -> str: - if language: - return language - - preferred_lang = request.accept_languages.best_match(languages) - if preferred_lang and preferred_lang in languages: - return preferred_lang - return languages[0] +def _url_origin(url: str) -> tuple[str, str, int] | None: + parsed_url = urllib.parse.urlsplit(url) + if parsed_url.scheme not in {"http", "https"} or parsed_url.hostname is None: + return None + try: + port = parsed_url.port + except ValueError: + return None + if port is None: + port = 443 if parsed_url.scheme == "https" else 80 + return parsed_url.scheme, parsed_url.hostname, port -def _redirect_with_console_session(account: Account, target_url: str) -> Response: - """Create a console session and attach its cookies to a redirect response.""" - token_pair = AccountService.login( - account=account, - session=db.session(), - ip_address=extract_remote_ip(request), - ) - response = redirect(target_url) - set_access_token_to_cookie(request, response, token_pair.access_token) - set_refresh_token_to_cookie(request, response, token_pair.refresh_token) - set_csrf_token_to_cookie(request, response, token_pair.csrf_token) - return response +def _signin_redirect(message: str, **params: str) -> Response: + query = urllib.parse.urlencode({"message": message, **params}) + return redirect(f"{dify_config.CONSOLE_WEB_URL}/signin?{query}") @console_ns.route("/oauth/login/") @@ -158,24 +148,23 @@ class OAuthLogin(Resource): @console_ns.doc(params={"provider": "OAuth provider name (github/google)"}) @console_ns.doc(params=query_params_from_model(OAuthLoginQuery)) @console_ns.response(302, "Redirect to OAuth authorization URL", console_ns.models[RedirectResponse.__name__]) - @console_ns.response(400, "Invalid provider") - def get(self, provider: str): - invite_token = request.args.get("invite_token") or None - timezone = _validated_timezone(request.args.get("timezone") or None) - language = _validated_language(request.args.get("language") or None) - redirect_url = request.args.get("redirect_url") or None - OAUTH_PROVIDERS = get_oauth_providers() - with current_app.app_context(): - oauth_provider = OAUTH_PROVIDERS.get(provider) - if not oauth_provider: - return {"error": "Invalid provider"}, 400 - - auth_url = oauth_provider.get_authorization_url( - invite_token=invite_token, - timezone=timezone, - language=language, - redirect_url=redirect_url, - ) + @console_ns.response(400, "Invalid provider", console_ns.models[OAuthErrorResponse.__name__]) + @setup_required + @social_oauth_login_enabled + @model_validate(OAuthLoginQuery) + def get(self, req_data: OAuthLoginQuery, provider: str): + try: + auth_url = application_services().accounts.oauth.start_authorization( + provider, + OAuthAuthorizationRequest( + invite_token=req_data.invite_token or None, + timezone=_validated_timezone(req_data.timezone), + language=_validated_language(req_data.language), + redirect_url=req_data.redirect_url or None, + ), + ) + except InvalidOAuthProviderError: + return dump_response(OAuthErrorResponse, {"error": "Invalid provider"}), 400 return redirect(auth_url) @@ -186,161 +175,53 @@ class OAuthCallback(Resource): @console_ns.doc(params={"provider": "OAuth provider name (github/google)"}) @console_ns.doc(params=query_params_from_model(OAuthCallbackQuery)) @console_ns.response(302, "Redirect to console with access token", console_ns.models[RedirectResponse.__name__]) - @console_ns.response(400, "OAuth process failed") - def get(self, provider: str): - OAUTH_PROVIDERS = get_oauth_providers() - with current_app.app_context(): - oauth_provider = OAUTH_PROVIDERS.get(provider) - if not oauth_provider: - return {"error": "Invalid provider"}, 400 - - code = request.args.get("code") - state = request.args.get("state") - oauth_state = decode_oauth_state(state) - invite_token = oauth_state.get("invite_token") - timezone = _validated_timezone(oauth_state.get("timezone")) - language = _validated_language(oauth_state.get("language")) - redirect_url = oauth_state.get("redirect_url") - - if not code: - return {"error": "Authorization code is required"}, 400 - + @console_ns.response(400, "OAuth process failed", console_ns.models[OAuthErrorResponse.__name__]) + @setup_required + @social_oauth_login_enabled + @model_validate(OAuthCallbackQuery) + def get(self, req_data: OAuthCallbackQuery, provider: str): + oauth_state = decode_oauth_state(req_data.state) try: - token = oauth_provider.get_access_token(code) - user_info = oauth_provider.get_user_info(token) - except httpx.RequestError as e: - error_text = str(e) - if isinstance(e, httpx.HTTPStatusError): - error_text = e.response.text - logger.exception("An error occurred during the OAuth process with %s: %s", provider, error_text) - return {"error": "OAuth process failed"}, 400 - except ValueError as e: - logger.warning("OAuth error with %s", provider, exc_info=True) - return redirect(f"{dify_config.CONSOLE_WEB_URL}/signin?message={urllib.parse.quote(str(e))}") - - if invite_token and RegisterService.is_valid_invite_token(invite_token): - invitation = RegisterService.get_invitation_if_token_valid( - None, - None, - invite_token, - session=db.session(), + result = application_services().accounts.oauth.complete_authorization( + OAuthCallbackCommand( + provider=provider, + code=req_data.code, + invite_token=oauth_state.get("invite_token"), + timezone=_validated_timezone(oauth_state.get("timezone")), + language=_validated_language(oauth_state.get("language")), + browser_language=_preferred_interface_language(), + ip_address=extract_remote_ip(request), + ) ) - if not invitation: - return redirect(f"{dify_config.CONSOLE_WEB_URL}/signin?message=Invalid invitation token.") - if invitation["data"]["email"].lower() != user_info.email.lower(): - message = "This invitation was sent to another account. Please sign in with the invited account." - query = urllib.parse.urlencode({"message": message, "invite_token": invite_token}) - return redirect(f"{dify_config.CONSOLE_WEB_URL}/signin?{query}") - - account = invitation["account"] - if account.status == AccountStatus.BANNED: - return redirect(f"{dify_config.CONSOLE_WEB_URL}/signin?message=Account is banned.") - - AccountService.link_account_integrate(provider, user_info.id, account, session=db.session()) - target_url = f"{dify_config.CONSOLE_WEB_URL}/signin/invite-settings?invite_token={invite_token}" - return _redirect_with_console_session(account, target_url) - - try: - account, oauth_new_user = _generate_account( - provider, - user_info, - timezone=timezone, - language=language, - ip_address=extract_remote_ip(request), + except InvalidOAuthProviderError: + return dump_response(OAuthErrorResponse, {"error": "Invalid provider"}), 400 + except (OAuthProviderRequestError, OAuthIdentityLockUnavailableError): + return dump_response(OAuthErrorResponse, {"error": "OAuth process failed"}), 400 + except OAuthProviderAuthorizationError as exc: + return _signin_redirect(exc.description) + except InvalidOAuthInvitationError: + return _signin_redirect("Invalid invitation token.") + except OAuthInvitationAccountMismatchError as exc: + return _signin_redirect( + "This invitation was sent to another account. Please sign in with the invited account.", + invite_token=exc.invite_token, ) - except AccountNotFoundError: - return redirect(f"{dify_config.CONSOLE_WEB_URL}/signin?message=Account not found.") - except (WorkSpaceNotFoundError, WorkSpaceNotAllowedCreateError): - return redirect( - f"{dify_config.CONSOLE_WEB_URL}/signin" - "?message=Workspace not found, please contact system admin to invite you to join in a workspace." + except OAuthAccountBannedError: + return _signin_redirect("Account is banned.") + except OAuthAccountNotFoundError: + return _signin_redirect("Account not found.") + except OAuthWorkspaceCreationNotAllowedError: + return _signin_redirect( + "Workspace not found, please contact system admin to invite you to join in a workspace." ) - except SeatsLimitExceededError: - return redirect(f"{dify_config.CONSOLE_WEB_URL}/signin?message=Licensed seats limit exceeded.") - except EmailDomainSuspendedRegistrationError: - return redirect(f"{dify_config.CONSOLE_WEB_URL}/signin?message={EmailDomainSuspendedError.description}") - except AccountRegisterError as exc: - return redirect(f"{dify_config.CONSOLE_WEB_URL}/signin?message={exc.description}") + except OAuthSeatsLimitExceededError: + return _signin_redirect("Licensed seats limit exceeded.") + except AccountEmailDomainSuspendedError: + return _signin_redirect(EmailDomainSuspendedError.description or "") + except AccountEmailFrozenError: + return _signin_redirect(AccountInFreezeError.description or "") + except OAuthRegistrationError as exc: + return _signin_redirect(exc.description) - # Check account status - if account.status == AccountStatus.BANNED: - return redirect(f"{dify_config.CONSOLE_WEB_URL}/signin?message=Account is banned.") - - if account.status == AccountStatus.PENDING: - account.status = AccountStatus.ACTIVE - account.initialized_at = naive_utc_now() - db.session.commit() - - try: - TenantService.create_owner_tenant_if_not_exist(account, session=db.session()) - except Unauthorized: - return redirect(f"{dify_config.CONSOLE_WEB_URL}/signin?message=Workspace not found.") - except WorkSpaceNotAllowedCreateError: - return redirect( - f"{dify_config.CONSOLE_WEB_URL}/signin" - "?message=Workspace not found, please contact system admin to invite you to join in a workspace." - ) - - target_url = _get_redirect_target(redirect_url) - query_char = "&" if "?" in target_url else "?" - target_url = f"{target_url}{query_char}oauth_new_user={str(oauth_new_user).lower()}" - return _redirect_with_console_session(account, target_url) - - -def _get_account_by_openid_or_email(provider: str, user_info: OAuthUserInfo) -> Account | None: - account: Account | None = Account.get_by_openid(provider, user_info.id) - - if not account: - account = AccountService.get_account_by_email_with_case_fallback(user_info.email, session=db.session()) - - return account - - -def _generate_account( - provider: str, - user_info: OAuthUserInfo, - timezone: str | None = None, - language: str | None = None, - ip_address: str | None = None, -) -> tuple[Account, bool]: - # Get account by openid or email. - account = _get_account_by_openid_or_email(provider, user_info) - oauth_new_user = False - - if account: - tenants = TenantService.get_join_tenants(account, session=db.session()) - if not tenants: - if not SystemFeatureService.is_workspace_creation_allowed(): - raise WorkSpaceNotAllowedCreateError() - else: - TenantService.create_owner_tenant(account, session=db.session()) - - if not account: - normalized_email = user_info.email.lower() - oauth_new_user = True - if not SystemFeatureService.is_registration_allowed(): - if dify_config.DEPLOYMENT_EDITION == DeploymentEdition.CLOUD: - freeze_type = BillingService.get_email_freeze_type(normalized_email) - if freeze_type: - if freeze_type == "email_domain_suspended": - raise EmailDomainSuspendedRegistrationError() - raise AccountRegisterError(description=AccountInFreezeError.description or "") - raise AccountRegisterError(description=("Invalid email or password")) - account_name = user_info.name or "Dify" - interface_language = _preferred_interface_language(language) - account = RegisterService.register( - email=normalized_email, - name=account_name, - password=None, - open_id=user_info.id, - provider=provider, - language=interface_language, - timezone=timezone, - ip_address=ip_address, - session=db.session(), - ) - - # Link account - AccountService.link_account_integrate(provider, user_info.id, account, session=db.session()) - - return account, oauth_new_user + target_url = _oauth_callback_target(result, oauth_state.get("redirect_url")) + return _redirect_with_console_session(result.tokens, target_url) diff --git a/api/controllers/console/explore/recommended_app.py b/api/controllers/console/explore/recommended_app.py index 1125c0bc580..1fdf61f049d 100644 --- a/api/controllers/console/explore/recommended_app.py +++ b/api/controllers/console/explore/recommended_app.py @@ -7,17 +7,17 @@ from pydantic import BaseModel, Field, computed_field, field_validator from controllers.common.schema import query_params_from_model, register_response_schema_models, register_schema_models from controllers.console import console_ns from controllers.console.explore.error import RecommendedAppNotFoundError -from controllers.console.wraps import account_initialization_required, model_validate, with_current_user +from controllers.console.flask_admission import console_account_admission +from controllers.console.wraps import model_validate from extensions.ext_application_services import application_services from fields.base import ResponseModel from libs.helper import build_icon_url, dump_response -from libs.login import login_required -from models import Account +from machinery.context import RequestContext from services.recommended_app_query_service import RecommendedAppNotFoundError as RecommendedAppQueryNotFoundError class RecommendedAppsQuery(BaseModel): - language: str | None = Field(default=None, description="Language code for recommended app localization") + language: str = Field(default="en-US", description="Language code for recommended app localization") class RecommendedAppInfoResponse(ResponseModel): @@ -97,16 +97,13 @@ register_response_schema_models( class RecommendedAppListApi(Resource): @console_ns.doc(params=query_params_from_model(RecommendedAppsQuery)) @console_ns.response(200, "Success", console_ns.models[RecommendedAppListResponse.__name__]) - @login_required - @account_initialization_required - @with_current_user + @console_account_admission() @model_validate(RecommendedAppsQuery) - def get(self, req_data: RecommendedAppsQuery, current_user: Account): + def get(self, req_data: RecommendedAppsQuery, _request_context: RequestContext): return dump_response( RecommendedAppListResponse, application_services().recommended_app_queries.list_recommended( - requested_language=req_data.language, - interface_language=current_user.interface_language, + language=req_data.language, ), ) @@ -115,16 +112,13 @@ class RecommendedAppListApi(Resource): class LearnDifyAppListApi(Resource): @console_ns.doc(params=query_params_from_model(RecommendedAppsQuery)) @console_ns.response(200, "Success", console_ns.models[LearnDifyAppListResponse.__name__]) - @login_required - @account_initialization_required - @with_current_user + @console_account_admission() @model_validate(RecommendedAppsQuery) - def get(self, req_data: RecommendedAppsQuery, current_user: Account): + def get(self, req_data: RecommendedAppsQuery, _request_context: RequestContext): return dump_response( LearnDifyAppListResponse, application_services().recommended_app_queries.list_learn_dify( - requested_language=req_data.language, - interface_language=current_user.interface_language, + language=req_data.language, ), ) @@ -133,9 +127,8 @@ class LearnDifyAppListApi(Resource): class RecommendedAppApi(Resource): @console_ns.response(200, "Success", console_ns.models[RecommendedAppDetailResponse.__name__]) @console_ns.response(404, "Recommended app not found") - @login_required - @account_initialization_required - def get(self, app_id: UUID): + @console_account_admission() + def get(self, _request_context: RequestContext, app_id: UUID): try: result = application_services().recommended_app_queries.get_detail(str(app_id)) except RecommendedAppQueryNotFoundError: diff --git a/api/controllers/console/notification.py b/api/controllers/console/notification.py index 080080bb361..03ebbc2ba6d 100644 --- a/api/controllers/console/notification.py +++ b/api/controllers/console/notification.py @@ -2,7 +2,7 @@ from flask_restx import Resource from pydantic import BaseModel, Field from controllers.common.fields import SimpleResultResponse -from controllers.common.schema import register_response_schema_models, register_schema_models +from controllers.common.schema import query_params_from_model, register_response_schema_models, register_schema_models from controllers.console import console_ns from controllers.console.flask_admission import console_account_admission from controllers.console.wraps import model_validate @@ -17,6 +17,10 @@ class DismissNotificationPayload(BaseModel): notification_id: str = Field(...) +class NotificationQuery(BaseModel): + language: str = Field(default="en-US", description="Notification language") + + class NotificationItemResponse(ResponseModel): notification_id: str | None = None frequency: str | None = None @@ -32,17 +36,19 @@ class NotificationResponse(ResponseModel): notifications: list[NotificationItemResponse] -register_schema_models(console_ns, DismissNotificationPayload) +register_schema_models(console_ns, DismissNotificationPayload, NotificationQuery) register_response_schema_models(console_ns, SimpleResultResponse, NotificationResponse) @console_ns.route("/notification") class NotificationApi(Resource): @console_ns.doc("get_notification") + @console_ns.doc(params=query_params_from_model(NotificationQuery)) @console_ns.doc( description=( "Return the active in-product notification for the current user " - "in their interface language (falls back to English if unavailable). " + "in the requested language (defaults to English when omitted). " + "Unavailable translations fall back to English, then the first available content. " "The notification is NOT marked as seen here; call POST /notification/dismiss " "when the user explicitly closes the modal." ), @@ -53,8 +59,9 @@ class NotificationApi(Resource): ) @console_ns.response(200, "Success", console_ns.models[NotificationResponse.__name__]) @console_account_admission(editions=frozenset({DeploymentEdition.CLOUD})) - def get(self, request_context: RequestContext): - result = application_services().notifications.get_active(request_context) + @model_validate(NotificationQuery) + def get(self, query: NotificationQuery, request_context: RequestContext): + result = application_services().notifications.get_active(request_context, query.language) return dump_response(NotificationResponse, result), 200 diff --git a/api/controllers/console/socketio/workflow.py b/api/controllers/console/socketio/workflow.py index 545af703dda..4569310bbe4 100644 --- a/api/controllers/console/socketio/workflow.py +++ b/api/controllers/console/socketio/workflow.py @@ -93,12 +93,10 @@ def handle_collaboration_event(sid, data): 1. mouse_move 2. vars_and_features_update 3. sync_request (ask leader to update graph) - 4. app_state_update - 5. mcp_server_update - 6. workflow_update - 7. comments_update - 8. node_panel_presence - 9. graph_view_state (session reports tab visibility; drives leader election) + 4. workflow_update + 5. comments_update + 6. node_panel_presence + 7. graph_view_state (session reports tab visibility; drives leader election) """ return collaboration_service.relay_collaboration_event(sid, data) diff --git a/api/controllers/console/workspace/models.py b/api/controllers/console/workspace/models.py index 54d021251fd..02b5a68f9c8 100644 --- a/api/controllers/console/workspace/models.py +++ b/api/controllers/console/workspace/models.py @@ -349,6 +349,8 @@ class ModelProviderModelCredentialApi(Resource): ) @setup_required @login_required + @is_admin_or_owner_required + @rbac_permission_required(RBACResourceScope.WORKSPACE, RBACPermission.CREDENTIAL_MANAGE, resource_required=False) @account_initialization_required @with_current_user @with_current_tenant_id diff --git a/api/controllers/console/wraps.py b/api/controllers/console/wraps.py index dd681a3b5e5..a56a6de5b14 100644 --- a/api/controllers/console/wraps.py +++ b/api/controllers/console/wraps.py @@ -356,6 +356,16 @@ def email_password_login_enabled[**P, R](view: Callable[P, R]) -> Callable[P, R] return decorated +def social_oauth_login_enabled[**P, R](view: Callable[P, R]) -> Callable[P, R]: + @wraps(view) + def decorated(*args: P.args, **kwargs: P.kwargs): + if not dify_config.ENABLE_SOCIAL_OAUTH_LOGIN: + abort(403) + return view(*args, **kwargs) + + return decorated + + def enable_change_email[**P, R](view: Callable[P, R]) -> Callable[P, R]: @wraps(view) def decorated(*args: P.args, **kwargs: P.kwargs): diff --git a/api/controllers/files/__init__.py b/api/controllers/files/__init__.py index f8976b86b9f..42b3761b92d 100644 --- a/api/controllers/files/__init__.py +++ b/api/controllers/files/__init__.py @@ -14,12 +14,13 @@ api = ExternalApi( files_ns = Namespace("files", description="File operations", path="/") -from . import image_preview, tool_files, upload +from . import appdeploy_files, image_preview, tool_files, upload api.add_namespace(files_ns) __all__ = [ "api", + "appdeploy_files", "bp", "files_ns", "image_preview", diff --git a/api/controllers/files/appdeploy_files.py b/api/controllers/files/appdeploy_files.py new file mode 100644 index 00000000000..265884cf7df --- /dev/null +++ b/api/controllers/files/appdeploy_files.py @@ -0,0 +1,401 @@ +"""File endpoints reached with an AppDeploy file grant. + +Upload, remote-upload, produce, and resolve authenticate with a Bearer grant; +content authenticates with a per-file token in the query string, because an +```` cannot carry a header. + +They stay on the public ``files`` blueprint rather than moving under +``inner_api``, whose contract is a fully trusted caller holding the master key. +None of the five meets it: ``content`` is a browser-facing surface, ``upload`` +carries an end user's payload, and ``produced`` and ``resolve`` are called by +workers executing third-party plugin code. The one genuinely server-to-server +step, minting the grant, already lives in ``inner_api``. +""" + +from __future__ import annotations + +from typing import IO +from urllib.parse import quote +from uuid import UUID + +from flask import Response, request +from flask_restx import Resource +from pydantic import BaseModel, Field, HttpUrl, ValidationError +from werkzeug.datastructures import FileStorage + +import services +from controllers.common.errors import ( + BlockedFileExtensionError, + FilenameNotExistsError, + FileTooLargeError, + NoFileUploadedError, + RemoteFileUploadError, + TooManyFilesError, + UnsupportedFileTypeError, +) +from controllers.common.schema import query_params_from_model, register_response_schema_models, register_schema_models +from controllers.files import files_ns +from controllers.files.wraps import FileGrantInvalidError, GrantedFileNotFoundError, file_grant_required +from extensions.ext_application_services import application_services +from fields.base import ResponseModel +from fields.file_fields import FileResponse +from fields.file_grant_fields import ResolvedFileResponse +from libs.exception import BaseHTTPException +from libs.helper import dump_response +from services.entities.file_grant_entities import ( + FileContent, + FileGrantClaims, + FileGrantContext, + FileGrantScope, + FileKind, + FileRef, + StoredUpload, +) +from services.errors.file_grant import ( + EndUserNotFoundError, + InvalidFileGrantError, + RemoteFileUnavailableError, + TooManyFileRefsError, +) +from services.file_grant_service import MAX_FILE_GRANT_REFS + +# Everything outside this whitelist is served as an attachment. Produced files +# carry a plugin-declared MIME type, so SVG and the rest of the XML family must +# never render in the viewer's origin. +INLINE_MIME_TYPES = frozenset({"image/png", "image/jpeg", "image/gif", "image/webp"}) + + +class InvalidFileRequestError(BaseHTTPException): + error_code = "invalid_request" + description = "The file request is malformed." + code = 400 + + +class FileRefPayload(BaseModel): + id: str + kind: FileKind + + +class RemoteFileUploadPayload(BaseModel): + url: HttpUrl = Field(description="Remote file URL to fetch and store") + + +class FileResolvePayload(BaseModel): + files: list[FileRefPayload] = Field(default_factory=list, max_length=MAX_FILE_GRANT_REFS) + + +class FileContentQuery(BaseModel): + token: str = Field(description="Signed content token scoped to this file") + + +class RemoteFileUploadResponse(FileResponse): + """Dify's upload shape plus the ``url`` key its remote-upload clients read. + + Dify has no service-api remote upload for this endpoint to stand in for, and + the web and console one it does have answers under ``url``. Carrying both + names lets a client of either move over untouched; they hold one URL. + """ + + url: str + + +class ProducedFileResponse(ResponseModel): + id: str + name: str + size: int + mime_type: str | None = None + url: str + internal_url: str + + +class FileResolveResponse(ResponseModel): + files: list[ResolvedFileResponse] + + +register_schema_models(files_ns, RemoteFileUploadPayload, FileResolvePayload) +register_response_schema_models( + files_ns, FileResponse, RemoteFileUploadResponse, ProducedFileResponse, FileResolveResponse +) + + +@files_ns.route("/appdeploy/upload") +class GrantedFileUploadApi(Resource): + """Store one uploaded file against the grant's end user.""" + + @file_grant_required(FileGrantScope.UPLOAD) + @files_ns.doc("grant_upload_file") + @files_ns.doc( + responses={ + 201: "File uploaded", + 400: "No file uploaded, the file has no name, or its extension is blocked", + 401: "Invalid grant", + 403: "Grant lacks the upload scope", + 413: "File too large", + 415: "Unsupported file type", + } + ) + @files_ns.response(201, "File uploaded", files_ns.models[FileResponse.__name__]) + def post(self, grant: FileGrantClaims): + upload = _single_upload() + upload_file = _store_upload( + grant, + filename=upload.filename or "", + stream=upload.stream, + mimetype=upload.mimetype, + ) + return _granted_file_response(upload_file), 201 + + +@files_ns.route("/appdeploy/remote-upload") +class GrantedRemoteFileUploadApi(Resource): + """Fetch a remote URL through the SSRF-safe fetcher and store it.""" + + @file_grant_required(FileGrantScope.UPLOAD) + @files_ns.doc("grant_upload_remote_file") + @files_ns.expect(files_ns.models[RemoteFileUploadPayload.__name__]) + @files_ns.doc( + responses={ + 201: "Remote file uploaded", + 400: "Invalid URL, unfetchable remote file, or a blocked extension", + 401: "Invalid grant", + 403: "Grant lacks the upload scope", + 413: "File too large", + 415: "Unsupported file type", + } + ) + @files_ns.response(201, "Remote file uploaded", files_ns.models[RemoteFileUploadResponse.__name__]) + def post(self, grant: FileGrantClaims): + try: + payload = RemoteFileUploadPayload.model_validate(files_ns.payload or {}) + except ValidationError as exc: + raise InvalidFileRequestError(str(exc)) from exc + + try: + upload_file = application_services().file_grants.store_remote_upload( + context=_grant_context(grant), + url=str(payload.url), + ) + except RemoteFileUnavailableError as exc: + raise RemoteFileUploadError(f"Failed to fetch file from {payload.url}") from exc + except EndUserNotFoundError as exc: + raise GrantedFileNotFoundError() from exc + except services.errors.file.FileTooLargeError as exc: + raise FileTooLargeError(exc.description) from exc + except services.errors.file.BlockedFileExtensionError as exc: + raise BlockedFileExtensionError(exc.description) from exc + except services.errors.file.UnsupportedFileTypeError: + raise UnsupportedFileTypeError() + return _remote_granted_file_response(upload_file), 201 + + +@files_ns.route("/appdeploy/produced") +class ProducedFileApi(Resource): + """Store one file produced by a running workflow node.""" + + @file_grant_required(FileGrantScope.PRODUCE) + @files_ns.doc("grant_upload_produced_file") + @files_ns.doc( + responses={ + 201: "Produced file stored", + 400: "No file uploaded", + 401: "Invalid grant", + 403: "Grant lacks the produce scope", + 413: "File too large", + } + ) + @files_ns.response(201, "Produced file stored", files_ns.models[ProducedFileResponse.__name__]) + def post(self, grant: FileGrantClaims): + upload = _single_upload() + try: + tool_file, access = application_services().file_grants.store_produced( + context=_grant_context(grant), + filename=upload.filename, + stream=upload.stream, + mimetype=upload.mimetype, + ) + except services.errors.file.FileTooLargeError as exc: + raise FileTooLargeError(exc.description) + except EndUserNotFoundError as exc: + raise GrantedFileNotFoundError() from exc + + return ProducedFileResponse( + id=tool_file.id, + name=tool_file.name, + size=tool_file.size, + mime_type=tool_file.mime_type, + url=access.external_url, + internal_url=access.internal_url, + ).model_dump(mode="json"), 201 + + +@files_ns.route("/appdeploy/resolve") +class GrantedFileResolveApi(Resource): + """Re-check ownership and sign fresh URLs at the moment of use.""" + + @file_grant_required(FileGrantScope.RESOLVE) + @files_ns.doc("grant_resolve_files") + @files_ns.expect(files_ns.models[FileResolvePayload.__name__]) + @files_ns.doc( + responses={ + 200: "Files resolved", + 400: "Malformed request", + 401: "Invalid grant", + 403: "Grant lacks the resolve scope", + } + ) + @files_ns.response(200, "Files resolved", files_ns.models[FileResolveResponse.__name__]) + def post(self, grant: FileGrantClaims): + try: + payload = FileResolvePayload.model_validate(files_ns.payload or {}) + except ValidationError as exc: + raise InvalidFileRequestError(str(exc)) from exc + + refs = [FileRef(id=ref.id, kind=ref.kind) for ref in payload.files] + try: + resolved = application_services().file_grants.resolve_file_access( + context=_grant_context(grant), + refs=refs, + ) + except EndUserNotFoundError as exc: + raise GrantedFileNotFoundError() from exc + except TooManyFileRefsError as exc: + raise InvalidFileRequestError(str(exc)) from exc + + return FileResolveResponse( + files=[ResolvedFileResponse.from_resolved(ref.id, file) for ref, file in zip(refs, resolved, strict=True)] + ).model_dump(mode="json") + + +@files_ns.route("/appdeploy//content") +class GrantedFileContentApi(Resource): + """Stream one file's bytes to a holder of its content token.""" + + @files_ns.doc("grant_file_content") + @files_ns.doc(params=query_params_from_model(FileContentQuery)) + @files_ns.doc( + responses={ + 200: "File stream returned", + 401: "Invalid or expired content token", + 404: "File not found", + } + ) + def get(self, file_id: UUID): + try: + query = FileContentQuery.model_validate(request.args.to_dict(flat=True)) + except ValidationError as exc: + raise FileGrantInvalidError() from exc + + try: + content = application_services().file_grants.load_content( + token=query.token, + requested_file_id=str(file_id), + ) + except InvalidFileGrantError as exc: + raise FileGrantInvalidError() from exc + if content is None: + raise GrantedFileNotFoundError() + + return _content_response(content) + + +def _content_response(content: FileContent) -> Response: + mime_type = _normalized_mime_type(content.mime_type) + inline = mime_type in INLINE_MIME_TYPES + + response = Response( + content.stream, + mimetype=mime_type if inline else "application/octet-stream", + direct_passthrough=True, + headers={}, + ) + response.headers["X-Content-Type-Options"] = "nosniff" + if not inline: + encoded_filename = quote(content.name or "") + response.headers["Content-Disposition"] = f"attachment; filename*=UTF-8''{encoded_filename}" + response.headers["Content-Type"] = "application/octet-stream" + # The sibling preview endpoints advertise `Accept-Ranges` for audio and video. + # Every such type downloads here, so the hint could only ever ride on a + # response no player will seek. + if content.size > 0: + response.headers["Content-Length"] = str(content.size) + return response + + +def _normalized_mime_type(mime_type: str | None) -> str: + return mime_type.split(";", 1)[0].strip().lower() if mime_type else "" + + +def _single_upload() -> FileStorage: + if "file" not in request.files: + raise NoFileUploadedError() + if len(request.files) > 1: + raise TooManyFilesError() + + upload = request.files["file"] + if not upload.filename: + raise FilenameNotExistsError() + return upload + + +def _store_upload( + grant: FileGrantClaims, + *, + filename: str, + stream: IO[bytes], + mimetype: str, +) -> StoredUpload: + try: + return application_services().file_grants.store_upload( + context=_grant_context(grant), + filename=filename, + stream=stream, + mimetype=mimetype, + ) + except EndUserNotFoundError as exc: + raise GrantedFileNotFoundError() from exc + except services.errors.file.FileTooLargeError as exc: + raise FileTooLargeError(exc.description) + except services.errors.file.BlockedFileExtensionError as exc: + raise BlockedFileExtensionError(exc.description) from exc + except services.errors.file.UnsupportedFileTypeError: + raise UnsupportedFileTypeError() + + +def _grant_context(grant: FileGrantClaims) -> FileGrantContext: + return FileGrantContext(tenant_id=grant.tenant_id, app_id=grant.app_id, end_user_id=grant.sub) + + +def _granted_file_response(upload_file: StoredUpload) -> dict[str, object]: + """Answer an upload exactly as dify's own upload endpoints answer it. + + A client moving off ``POST /v1/files/upload`` must not have to read a second + shape, so the same model reads the same ``upload_files`` row: every key dify + leaves null for such a row is null here too. Only the value of ``source_url`` + is ours. Dify signs a ``file-preview`` URL there and this channel signs a + content-token URL, which keeps that key's promise of a signed URL that + retrieves the file while keeping the grant its only way in. + """ + + signed_url, _ = application_services().file_grants.content_urls(file_id=upload_file.id, kind=FileKind.UPLOAD) + return dump_response(FileResponse, upload_file) | {"source_url": signed_url} + + +def _remote_granted_file_response(upload_file: StoredUpload) -> dict[str, object]: + """Answer a remote upload with the upload shape plus dify's ``url`` key. + + Reuses the URL already signed for ``source_url`` rather than signing a + second one, so the two keys are one value under the two names dify's two + kinds of client look for. + """ + + response = _granted_file_response(upload_file) + return response | {"url": response["source_url"]} + + +__all__ = [ + "GrantedFileContentApi", + "GrantedFileResolveApi", + "GrantedFileUploadApi", + "GrantedRemoteFileUploadApi", + "ProducedFileApi", +] diff --git a/api/controllers/files/wraps.py b/api/controllers/files/wraps.py new file mode 100644 index 00000000000..b72925028d8 --- /dev/null +++ b/api/controllers/files/wraps.py @@ -0,0 +1,57 @@ +"""Bearer authentication for the AppDeploy file grant endpoints.""" + +from collections.abc import Callable +from functools import wraps + +from flask import request + +from extensions.ext_application_services import application_services +from libs.exception import BaseHTTPException +from services.entities.file_grant_entities import FileGrantClaims, FileGrantScope + + +class FileGrantInvalidError(BaseHTTPException): + error_code = "grant_invalid" + description = "The file grant is missing, malformed, or expired." + code = 401 + + +class FileGrantScopeDeniedError(BaseHTTPException): + error_code = "grant_scope_denied" + description = "The file grant does not carry the required scope." + code = 403 + + +class GrantedFileNotFoundError(BaseHTTPException): + error_code = "file_not_found" + description = "File not found." + code = 404 + + +def file_grant_required[**P, R](scope: FileGrantScope) -> Callable[[Callable[P, R]], Callable[P, R]]: + """Require a valid file grant carrying ``scope`` and inject its claims.""" + + def decorator(view: Callable[P, R]) -> Callable[P, R]: + @wraps(view) + def decorated(*args: P.args, **kwargs: P.kwargs) -> R: + kwargs["grant"] = _authenticated_claims(scope) + return view(*args, **kwargs) + + return decorated + + return decorator + + +def _authenticated_claims(scope: FileGrantScope) -> FileGrantClaims: + scheme, _, token = request.headers.get("Authorization", "").partition(" ") + if scheme.lower() != "bearer" or not token: + raise FileGrantInvalidError() + + claims = application_services().file_grants.decode_grant(token) + if claims is None: + raise FileGrantInvalidError() + + if scope not in claims.scopes: + raise FileGrantScopeDeniedError() + + return claims diff --git a/api/controllers/inner_api/__init__.py b/api/controllers/inner_api/__init__.py index 27be2a50d7e..0db1b6b37f0 100644 --- a/api/controllers/inner_api/__init__.py +++ b/api/controllers/inner_api/__init__.py @@ -21,6 +21,7 @@ from .agent import files as _agent_files from .agent import llm as _agent_llm from .agent import tools as _agent_tools from .app import dsl as _app_dsl +from .app import file_grants as _app_file_grants from .knowledge import retrieval as _knowledge_retrieval from .knowledge_fs import storage as _knowledge_fs_storage from .plugin import agent_config as _agent_config @@ -36,6 +37,7 @@ __all__ = [ "_agent_llm", "_agent_tools", "_app_dsl", + "_app_file_grants", "_knowledge_fs_storage", "_knowledge_retrieval", "_mail", diff --git a/api/controllers/inner_api/app/file_grants.py b/api/controllers/inner_api/app/file_grants.py new file mode 100644 index 00000000000..0acc9754c73 --- /dev/null +++ b/api/controllers/inner_api/app/file_grants.py @@ -0,0 +1,203 @@ +"""Mint AppDeploy file grants for the enterprise control plane. + +This is the only endpoint that asserts an AppDeploy identity: the application +service upserts the subject's ``EndUser`` row, validates the files the caller +claims to reference, and signs a short-lived grant. +""" + +from __future__ import annotations + +from flask_restx import Resource +from pydantic import BaseModel, Field, ValidationError + +from controllers.common.schema import register_response_schema_models, register_schema_models +from controllers.console.wraps import setup_required +from controllers.files.wraps import GrantedFileNotFoundError +from controllers.inner_api import inner_api_ns +from controllers.inner_api.wraps import enterprise_inner_api_only +from extensions.ext_application_services import application_services +from fields.base import ResponseModel +from fields.file_grant_fields import ResolvedFileResponse +from libs.exception import BaseHTTPException +from services.entities.file_grant_entities import FileGrantMintRequest, FileGrantScope, FileKind, FileRef +from services.file_grant_service import ( + AppNotFoundError, + EndUserNotFoundError, + TooManyFileRefsError, +) +from services.file_grant_service import ( + GrantedFileNotFoundError as ServiceGrantedFileNotFoundError, +) +from services.file_grant_service import ( + GrantTtlTooLongError as ServiceGrantTtlTooLongError, +) +from services.file_grant_service import ( + InvalidGrantRequestError as ServiceInvalidGrantRequestError, +) +from services.file_grant_service import ( + InvalidSubjectError as ServiceInvalidSubjectError, +) + + +class InvalidGrantRequestError(BaseHTTPException): + error_code = "invalid_request" + description = "The file grant request is malformed." + code = 400 + + +class GrantTtlTooLongError(BaseHTTPException): + error_code = "grant_ttl_too_long" + description = "The requested file grant lifetime exceeds its allowed window." + code = 400 + + +class InvalidSubjectError(BaseHTTPException): + error_code = "invalid_subject" + description = "The subject is empty or contains a NUL byte." + code = 400 + + +class GrantAppNotFoundError(BaseHTTPException): + error_code = "app_not_found" + description = "App not found." + code = 404 + + +class FileGrantFileRef(BaseModel): + id: str + kind: FileKind + + +class FileGrantMintPayload(BaseModel): + tenant_id: str + app_id: str + subject: str + is_anonymous: bool = False + scopes: list[FileGrantScope] + ttl_seconds: int = Field(gt=0) + # Recorded by the enterprise caller's own audit log; it never enters the + # grant because a rotated key must not strand the files it uploaded. + actor_key_digest: str | None = None + file_ids: list[FileGrantFileRef] = Field(default_factory=list) + optional_file_ids: list[FileGrantFileRef] = Field(default_factory=list) + run_deadline: int | None = None + + +class FileGrantLimits(ResponseModel): + file_size_limit: int + image_file_size_limit: int + audio_file_size_limit: int + video_file_size_limit: int + workflow_file_upload_limit: int + batch_count_limit: int + + +class FileGrantFileMetadata(ResponseModel): + id: str + kind: FileKind + name: str + size: int + extension: str + mime_type: str | None = None + + +class FileGrantMintResponse(ResponseModel): + grant: str + expires_at: int + limits: FileGrantLimits + files: list[FileGrantFileMetadata] + optional_files: list[ResolvedFileResponse] + + +register_schema_models(inner_api_ns, FileGrantMintPayload) +register_response_schema_models(inner_api_ns, FileGrantMintResponse) + + +@inner_api_ns.route("/enterprise/file-grants") +class EnterpriseFileGrantApi(Resource): + """Assert one AppDeploy subject and sign a grant for it.""" + + @setup_required + @enterprise_inner_api_only + @inner_api_ns.doc("enterprise_mint_file_grant") + @inner_api_ns.expect(inner_api_ns.models[FileGrantMintPayload.__name__]) + @inner_api_ns.response( + 200, + "File grant minted", + inner_api_ns.models[FileGrantMintResponse.__name__], + ) + @inner_api_ns.doc( + responses={ + 400: "Invalid request, subject, or grant TTL", + 404: "App not found, or a required file is not owned by the subject", + } + ) + def post(self): + try: + payload = FileGrantMintPayload.model_validate(inner_api_ns.payload or {}) + except ValidationError as exc: + raise InvalidGrantRequestError(str(exc)) from exc + + try: + result = application_services().file_grants.mint( + FileGrantMintRequest( + tenant_id=payload.tenant_id, + app_id=payload.app_id, + subject=payload.subject, + is_anonymous=payload.is_anonymous, + scopes=tuple(payload.scopes), + ttl_seconds=payload.ttl_seconds, + file_refs=tuple(FileRef(id=ref.id, kind=ref.kind) for ref in payload.file_ids), + optional_file_refs=tuple(FileRef(id=ref.id, kind=ref.kind) for ref in payload.optional_file_ids), + run_deadline=payload.run_deadline, + ) + ) + except AppNotFoundError as exc: + raise GrantAppNotFoundError() from exc + except ServiceGrantTtlTooLongError as exc: + raise GrantTtlTooLongError() from exc + except ServiceInvalidSubjectError as exc: + raise InvalidSubjectError() from exc + except (ServiceInvalidGrantRequestError, TooManyFileRefsError) as exc: + raise InvalidGrantRequestError(str(exc)) from exc + except (EndUserNotFoundError, ServiceGrantedFileNotFoundError) as exc: + raise GrantedFileNotFoundError() from exc + + return FileGrantMintResponse( + grant=result.grant, + expires_at=result.expires_at, + limits=FileGrantLimits( + file_size_limit=result.limits.file_size_limit, + image_file_size_limit=result.limits.image_file_size_limit, + audio_file_size_limit=result.limits.audio_file_size_limit, + video_file_size_limit=result.limits.video_file_size_limit, + workflow_file_upload_limit=result.limits.workflow_file_upload_limit, + batch_count_limit=result.limits.batch_count_limit, + ), + files=[ + FileGrantFileMetadata( + id=file.id, + kind=file.kind, + name=file.name, + size=file.size, + extension=file.extension, + mime_type=file.mime_type, + ) + for file in result.files + ], + optional_files=[ + ResolvedFileResponse.from_resolved(ref.id, access) + for ref, access in zip( + payload.optional_file_ids, + result.optional_files, + strict=True, + ) + ], + ).model_dump(mode="json") + + +__all__ = [ + "EnterpriseFileGrantApi", + "FileGrantMintPayload", + "FileGrantMintResponse", +] diff --git a/api/controllers/openapi/_contract.py b/api/controllers/openapi/_contract.py index a7dcf9093da..aa92cdf411d 100644 --- a/api/controllers/openapi/_contract.py +++ b/api/controllers/openapi/_contract.py @@ -5,8 +5,9 @@ reference — emitting the Swagger schema AND doing the runtime validation/ serialisation — so the advertised and enforced contracts can't drift. Validation failures map to a single shape: 422. -They must sit BELOW ``@auth_router.guard`` so auth runs before validation and the -``view.__wrapped__`` unit-test seam unwraps exactly the guard layer. +They must sit below route admission (or a direct ``@auth_router.guard``) so +authentication runs before validation and the ``view.__wrapped__`` unit-test +seam can bypass the outer admission layer. """ from __future__ import annotations diff --git a/api/controllers/openapi/account.py b/api/controllers/openapi/account.py index f1b02ef115e..6ddd9a14ef2 100644 --- a/api/controllers/openapi/account.py +++ b/api/controllers/openapi/account.py @@ -1,12 +1,11 @@ from __future__ import annotations from datetime import UTC, datetime +from uuid import UUID from flask_restx import Resource -from sqlalchemy.orm import Session -from werkzeug.exceptions import NotFound +from werkzeug.exceptions import NotFound, Unauthorized -from controllers.common.session import with_session from controllers.openapi import openapi_ns from controllers.openapi._contract import accepts, returns from controllers.openapi._models import ( @@ -18,114 +17,77 @@ from controllers.openapi._models import ( SessionRow, WorkspacePayload, ) -from controllers.openapi.auth.composition import auth_router -from controllers.openapi.auth.data import AuthData -from extensions.ext_redis import redis_client -from libs.oauth_bearer import ( - Scope, - TokenType, - get_auth_ctx, -) -from libs.rate_limit import ( - LIMIT_ME_PER_ACCOUNT, - enforce, -) -from services.account_service import AccountService, TenantService -from services.oauth_device_flow import ( - list_active_sessions, - revoke_oauth_token, - token_belongs_to_subject, -) +from controllers.openapi.flask_admission import openapi_account_admission +from extensions.ext_application_services import application_services +from libs.oauth_bearer import Scope +from libs.rate_limit import LIMIT_ME_PER_ACCOUNT +from machinery.context import AccountRequestContext +from services.account_errors import AccountNotFoundError, AccountSessionNotFoundError +from services.entities.account_access_entities import AccountSessionSnapshot, AccountWorkspaceSnapshot +from services.entities.account_entities import AccountSnapshot @openapi_ns.route("/account") class AccountApi(Resource): - @auth_router.guard(scope=Scope.FULL, allowed_token_types=frozenset({TokenType.OAUTH_ACCOUNT})) + @openapi_account_admission(scope=Scope.FULL, rate_limit=LIMIT_ME_PER_ACCOUNT) @returns(200, AccountResponse, description="Account info") - @with_session(write=False) - def get(self, session: Session, *, auth_data: AuthData): - enforce(LIMIT_ME_PER_ACCOUNT, key=f"account:{auth_data.account_id}") - - account_id_str = str(auth_data.account_id) if auth_data.account_id else None - account = AccountService.get_account_by_id(account_id_str, session=session) if account_id_str else None - memberships = TenantService.get_account_memberships(account_id_str, session=session) if account_id_str else [] - default_ws_id = _pick_default_workspace(memberships) - + def get(self, request_context: AccountRequestContext): + try: + snapshot = application_services().accounts.access.get(request_context) + except AccountNotFoundError: + raise Unauthorized("account not found") from None return AccountResponse( subject_type="account", - subject_email=account.email if account else None, - account=_account_payload(account) if account else None, - workspaces=[_workspace_payload(m) for m in memberships], - default_workspace_id=default_ws_id, + subject_email=snapshot.account.email, + account=_account_payload(snapshot.account), + workspaces=[_workspace_payload(workspace) for workspace in snapshot.workspaces], + default_workspace_id=snapshot.default_workspace_id, ) @openapi_ns.route("/account/sessions/self") class AccountSessionsSelfApi(Resource): - @auth_router.guard(scope=Scope.FULL, allowed_token_types=frozenset({TokenType.OAUTH_ACCOUNT})) + @openapi_account_admission(scope=Scope.FULL) @returns(200, RevokeResponse, description="Session revoked") - @with_session - def delete(self, session: Session, *, auth_data: AuthData): - revoke_oauth_token(redis_client, str(auth_data.token_id), session=session) + def delete(self, request_context: AccountRequestContext): + application_services().accounts.access.revoke_current_session(request_context) return RevokeResponse(status="revoked") @openapi_ns.route("/account/sessions") class AccountSessionsApi(Resource): - @auth_router.guard(scope=Scope.FULL, allowed_token_types=frozenset({TokenType.OAUTH_ACCOUNT})) + @openapi_account_admission(scope=Scope.FULL) @returns(200, SessionListResponse, description="Session list") @accepts(query=SessionListQuery) - @with_session(write=False) - def get(self, session: Session, *, auth_data: AuthData, query: SessionListQuery): - # SessionListQuery enforces the advertised bounds (extra='forbid', page>=1, - # 1<=limit<=MAX_PAGE_LIMIT) so the server rejects out-of-range paging rather - # than silently coercing (e.g. page=0 -> empty slice). - ctx = get_auth_ctx() - now = datetime.now(UTC) - page = query.page - limit = query.limit - - all_rows = list_active_sessions(ctx, now, session=session) - - total = len(all_rows) - sliced = all_rows[(page - 1) * limit : page * limit] - - items = [ - SessionRow( - id=str(r.id), - prefix=r.prefix, - client_id=r.client_id, - device_label=r.device_label, - created_at=_iso(r.created_at), - last_used_at=_iso(r.last_used_at), - expires_at=_iso(r.expires_at), - ) - for r in sliced - ] - + def get(self, request_context: AccountRequestContext, *, query: SessionListQuery): + page = application_services().accounts.access.list_sessions( + request_context, + page=query.page, + limit=query.limit, + ) return SessionListResponse( - page=page, - limit=limit, - total=total, - has_more=page * limit < total, - data=items, + page=page.page, + limit=page.limit, + total=page.total, + has_more=page.has_more, + data=[_session_row(session) for session in page.items], ) @openapi_ns.route("/account/sessions/") class AccountSessionByIdApi(Resource): - @auth_router.guard(scope=Scope.FULL, allowed_token_types=frozenset({TokenType.OAUTH_ACCOUNT})) + @openapi_account_admission(scope=Scope.FULL) @returns(200, RevokeResponse, description="Session revoked") - @with_session - def delete(self, session: Session, session_id: str, *, auth_data: AuthData): - ctx = get_auth_ctx() - - # 404 (not 403) on cross-subject so the endpoint doesn't leak - # token IDs that belong to other subjects. - if not token_belongs_to_subject(session_id, ctx, session=session): - raise NotFound("session not found") - - revoke_oauth_token(redis_client, session_id, session=session) + def delete(self, request_context: AccountRequestContext, session_id: str): + try: + token_id = str(UUID(session_id)) + except ValueError: + raise NotFound("session not found") from None + try: + application_services().accounts.access.revoke_session(request_context, token_id=token_id) + except AccountSessionNotFoundError: + # Do not reveal whether a token ID belongs to another account. + raise NotFound("session not found") from None return RevokeResponse(status="revoked") @@ -137,19 +99,21 @@ def _iso(dt: datetime | None) -> str | None: return dt.isoformat().replace("+00:00", "Z") -def _pick_default_workspace(memberships) -> str | None: - if not memberships: - return None - for join, tenant in memberships: - if getattr(join, "current", False): - return str(tenant.id) - return str(memberships[0][1].id) +def _session_row(session: AccountSessionSnapshot) -> SessionRow: + return SessionRow( + id=session.id, + prefix=session.prefix, + client_id=session.client_id, + device_label=session.device_label, + created_at=_iso(session.created_at), + last_used_at=_iso(session.last_used_at), + expires_at=_iso(session.expires_at), + ) -def _workspace_payload(row) -> WorkspacePayload: - join, tenant = row - return WorkspacePayload(id=str(tenant.id), name=tenant.name, role=getattr(join, "role", "")) +def _workspace_payload(workspace: AccountWorkspaceSnapshot) -> WorkspacePayload: + return WorkspacePayload(id=workspace.id, name=workspace.name, role=workspace.role) -def _account_payload(account) -> AccountPayload: - return AccountPayload(id=str(account.id), email=account.email, name=account.name) +def _account_payload(account: AccountSnapshot) -> AccountPayload: + return AccountPayload(id=account.id, email=account.email, name=account.name) diff --git a/api/controllers/openapi/auth/pipeline.py b/api/controllers/openapi/auth/pipeline.py index 6a68e5c91fe..9f14105a942 100644 --- a/api/controllers/openapi/auth/pipeline.py +++ b/api/controllers/openapi/auth/pipeline.py @@ -119,8 +119,8 @@ class PipelineRouter: """Entry point for openapi auth. `guard()` is the decorator that endpoints attach to. It applies - global gates (edition, token type) then dispatches to the matching - `PipelineRoute` for the token type. + global gates (edition, license, token type) then dispatches to the + matching `PipelineRoute` for the token type. """ def __init__(self, routes: dict[TokenType, PipelineRoute]) -> None: @@ -132,6 +132,7 @@ class PipelineRouter: scope: Scope | None = None, allowed_token_types: frozenset[TokenType] | None = None, edition: frozenset[DeploymentEdition] | None = None, + require_valid_enterprise_license: bool = False, workspace_membership: bool = False, allowed_roles: frozenset[TenantAccountRole] | None = None, rbac: RBACRequirement | None = None, @@ -140,6 +141,7 @@ class PipelineRouter: scope=scope, allowed_token_types=allowed_token_types, edition=edition, + require_valid_enterprise_license=require_valid_enterprise_license, workspace_membership=workspace_membership, allowed_roles=allowed_roles, rbac=rbac, @@ -151,6 +153,7 @@ class PipelineRouter: scope: Scope | None = None, allowed_token_types: frozenset[TokenType] | None = None, edition: frozenset[DeploymentEdition] | None = None, + require_valid_enterprise_license: bool = False, allowed_roles: frozenset[TenantAccountRole] | None = None, rbac: RBACRequirement | None = None, ) -> Callable: @@ -158,6 +161,7 @@ class PipelineRouter: scope=scope, allowed_token_types=allowed_token_types, edition=edition, + require_valid_enterprise_license=require_valid_enterprise_license, workspace_membership=True, allowed_roles=allowed_roles, rbac=rbac, @@ -169,6 +173,7 @@ class PipelineRouter: scope: Scope | None, allowed_token_types: frozenset[TokenType] | None, edition: frozenset[DeploymentEdition] | None, + require_valid_enterprise_license: bool, workspace_membership: bool, allowed_roles: frozenset[TenantAccountRole] | None, rbac: RBACRequirement | None, @@ -183,6 +188,7 @@ class PipelineRouter: scope=scope, allowed_token_types=allowed_token_types, edition=edition, + require_valid_enterprise_license=require_valid_enterprise_license, workspace_membership=workspace_membership, allowed_roles=allowed_roles, rbac=rbac, @@ -201,6 +207,7 @@ class PipelineRouter: scope: Scope | None, allowed_token_types: frozenset[TokenType] | None, edition: frozenset[DeploymentEdition] | None, + require_valid_enterprise_license: bool, workspace_membership: bool = False, allowed_roles: frozenset[TenantAccountRole] | None = None, rbac: RBACRequirement | None = None, @@ -210,7 +217,9 @@ class PipelineRouter: raise NotFound() license_checked = False - if edition is not None and DeploymentEdition.ENTERPRISE in edition: + if dify_config.DEPLOYMENT_EDITION == DeploymentEdition.ENTERPRISE and ( + require_valid_enterprise_license or (edition is not None and DeploymentEdition.ENTERPRISE in edition) + ): _check_license() license_checked = True diff --git a/api/controllers/openapi/flask_admission.py b/api/controllers/openapi/flask_admission.py new file mode 100644 index 00000000000..7e485c70db6 --- /dev/null +++ b/api/controllers/openapi/flask_admission.py @@ -0,0 +1,88 @@ +"""Flask adapter for account-authenticated OpenAPI admission.""" + +from __future__ import annotations + +from collections.abc import Callable +from functools import wraps +from typing import Concatenate + +from flask import Response, request +from werkzeug.exceptions import Unauthorized + +from controllers.openapi.auth.composition import auth_router +from controllers.openapi.auth.data import AuthData +from core.logging.context import get_request_id, get_trace_id +from enums import DeploymentEdition +from libs.oauth_bearer import Scope, TokenType +from libs.rate_limit import RateLimit, enforce +from machinery.context import AccountRequestContext +from models.account import Account, AccountStatus + + +def openapi_account_admission[T, **P, R]( + *, + scope: Scope, + editions: frozenset[DeploymentEdition] | None = None, + require_initialized: bool = True, + require_valid_enterprise_license: bool = True, + rate_limit: RateLimit | None = None, +) -> Callable[ + [Callable[Concatenate[T, AccountRequestContext, P], R]], + Callable[Concatenate[T, P], R | Response], +]: + """Authenticate an account bearer and inject framework-neutral identity. + + Client-version admission remains attached to the OpenAPI blueprint so it + can also reject requests for removed routes. Edition and Enterprise + license checks are delegated to the shared auth router before the stable + context is constructed. + """ + + def decorator( + view: Callable[Concatenate[T, AccountRequestContext, P], R], + ) -> Callable[Concatenate[T, P], R | Response]: + @wraps(view) + def inject_request_context( + self: T, + /, + *args: P.args, + **kwargs: P.kwargs, + ) -> R: + auth_data = kwargs.pop("auth_data", None) + if not isinstance(auth_data, AuthData): + raise RuntimeError("OpenAPI auth pipeline did not provide valid AuthData") + account = auth_data.caller + if not isinstance(account, Account) or auth_data.account_id is None: + raise Unauthorized("account not found") + if require_initialized and account.status == AccountStatus.UNINITIALIZED: + raise Unauthorized("account not initialized") + + account_id = str(auth_data.account_id) + if rate_limit is not None: + enforce(rate_limit, key=f"account:{account_id}") + + context = AccountRequestContext( + request_id=get_request_id(), + trace_id=get_trace_id() or request.headers.get("X-Trace-Id"), + account_id=account_id, + access_token_id=str(auth_data.token_id) if auth_data.token_id is not None else None, + ) + return view(self, context, *args, **kwargs) + + authenticated = auth_router.guard( + scope=scope, + allowed_token_types=frozenset({TokenType.OAUTH_ACCOUNT}), + edition=editions, + require_valid_enterprise_license=require_valid_enterprise_license, + )(inject_request_context) + + # Keep one stable test seam: one ``__wrapped__`` skips route admission + # and reaches input parsing/response handling. Client-version admission + # stays blueprint-wide by design. + @wraps(view) + def admitted(self: T, /, *args: P.args, **kwargs: P.kwargs) -> R | Response: + return authenticated(self, *args, **kwargs) + + return admitted + + return decorator diff --git a/api/controllers/service_api/app/workflow.py b/api/controllers/service_api/app/workflow.py index 33a2f3a4b64..d40438e6fa3 100644 --- a/api/controllers/service_api/app/workflow.py +++ b/api/controllers/service_api/app/workflow.py @@ -48,6 +48,7 @@ from core.errors.error import ( ) from core.helper.trace_id_helper import get_external_trace_id, get_trace_session_id, omit_trace_session_id_from_payload from enums import CloudPlan, DeploymentEdition +from extensions.ext_application_services import application_services from extensions.ext_database import db from extensions.ext_redis import redis_client from fields.base import ResponseModel @@ -71,7 +72,6 @@ from services.errors.app import ( TriggerWorkflowServiceModeUnavailableError as TriggerWorkflowServiceModeUnavailableServiceError, ) from services.errors.llm import InvokeRateLimitError -from services.workflow_app_service import WorkflowAppService logger = logging.getLogger(__name__) @@ -614,20 +614,16 @@ class WorkflowAppLogApi(Resource): created_at_before = isoparse(args.created_at__before) if args.created_at__before else None created_at_after = isoparse(args.created_at__after) if args.created_at__after else None - # get paginate workflow app logs - workflow_app_service = WorkflowAppService() - with sessionmaker(db.engine).begin() as session: - workflow_app_log_pagination = workflow_app_service.get_paginate_workflow_app_logs( - session=session, - app_model=app_model, - keyword=args.keyword, - status=status, - created_at_before=created_at_before, - created_at_after=created_at_after, - page=args.page, - limit=args.limit, - created_by_end_user_session_id=args.created_by_end_user_session_id, - created_by_account=args.created_by_account, - ) - - return dump_response(WorkflowAppLogPaginationResponse, workflow_app_log_pagination) + result = application_services().workflow_app_logs.list_logs( + tenant_id=app_model.tenant_id, + app_id=app_model.id, + keyword=args.keyword, + status=status, + created_at_before=created_at_before, + created_at_after=created_at_after, + page=args.page, + limit=args.limit, + created_by_end_user_session_id=args.created_by_end_user_session_id, + created_by_account=args.created_by_account, + ) + return dump_response(WorkflowAppLogPaginationResponse, result) diff --git a/api/controllers/service_api/dataset/dataset.py b/api/controllers/service_api/dataset/dataset.py index 09a404b9eff..32035f539b1 100644 --- a/api/controllers/service_api/dataset/dataset.py +++ b/api/controllers/service_api/dataset/dataset.py @@ -430,10 +430,11 @@ class DatasetListApi(DatasetApiResource): query_params["tag_ids"] = request.args.getlist("tag_ids") query = DatasetListQuery.model_validate(query_params) # provider = request.args.get("provider", default="vendor") + effective_limit = min(query.limit, 100) datasets, total = DatasetService.get_datasets( query.page, - query.limit, + effective_limit, session, tenant_id, current_user, @@ -467,8 +468,8 @@ class DatasetListApi(DatasetApiResource): item["embedding_available"] = True response = { "data": data, - "has_more": len(datasets) == query.limit, - "limit": query.limit, + "has_more": query.page * effective_limit < total, + "limit": effective_limit, "total": total, "page": query.page, } diff --git a/api/controllers/service_api/dataset/document.py b/api/controllers/service_api/dataset/document.py index 5049c2a3782..a2288a99c43 100644 --- a/api/controllers/service_api/dataset/document.py +++ b/api/controllers/service_api/dataset/document.py @@ -1015,8 +1015,9 @@ class DocumentListApi(DatasetApiResource): query = query.order_by(desc(Document.created_at), desc(Document.position)) + effective_limit = min(query_params.limit, 100) paginated_documents = paginate_query( - query, session=session, page=query_params.page, per_page=query_params.limit, max_per_page=100 + query, session=session, page=query_params.page, per_page=effective_limit, max_per_page=100 ) documents = paginated_documents.items @@ -1029,8 +1030,8 @@ class DocumentListApi(DatasetApiResource): response = { "data": document_responses(documents, session=session), - "has_more": len(documents) == query_params.limit, - "limit": query_params.limit, + "has_more": query_params.page * effective_limit < paginated_documents.total, + "limit": effective_limit, "total": paginated_documents.total, "page": query_params.page, } diff --git a/api/controllers/service_api/dataset/segment.py b/api/controllers/service_api/dataset/segment.py index 7585e5f3dc1..921c59a0942 100644 --- a/api/controllers/service_api/dataset/segment.py +++ b/api/controllers/service_api/dataset/segment.py @@ -281,7 +281,7 @@ class SegmentApi(DatasetApiResource): use_defaults_for_malformed_ints=True, ) page = args.page - limit = args.limit + limit = min(args.limit, 100) dataset_id_str = str(dataset_id) dataset = session.scalar( select(Dataset).where(Dataset.tenant_id == tenant_id, Dataset.id == dataset_id_str).limit(1) @@ -331,7 +331,7 @@ class SegmentApi(DatasetApiResource): "data": segment_responses_with_summaries(segments, summaries, session=session), "doc_form": document.doc_form, "total": total, - "has_more": len(segments) == limit, + "has_more": page * limit < total, "limit": limit, "page": page, } diff --git a/api/controllers/web/app.py b/api/controllers/web/app.py index 1e985b77cc1..929cd6ae51a 100644 --- a/api/controllers/web/app.py +++ b/api/controllers/web/app.py @@ -26,6 +26,7 @@ from libs.passport import PassportService from libs.token import extract_webapp_passport from models.model import App, EndUser from services.app_definition_query_service import AppDefinitionNotPublishedError, AppDefinitionUnavailableError +from services.web_passport_service import WebAppAuthType from services.webapp_access_query_service import ( WebAppAccessAppNotFoundError, WebAppAccessReferenceRequiredError, @@ -184,12 +185,16 @@ class AppWebAuthPermission(Resource): if not tk: raise Unauthorized("Access token is missing.") decoded = PassportService().verify(tk) - user_id = decoded.get("user_id", "visitor") except Unauthorized: raise WebAppAuthRequiredError() from None except Exception: logger.exception("Unexpected error during auth verification") raise + if decoded.get("auth_type") != WebAppAuthType.INTERNAL: + raise WebAppAuthRequiredError() + user_id = decoded.get("user_id") + if not user_id: + raise WebAppAuthRequiredError() try: is_allowed = webapp_access.is_user_allowed(user_id=str(user_id), app_id=app_id) diff --git a/api/controllers/web/wraps.py b/api/controllers/web/wraps.py index 69e0edb059f..d6aac7e97c6 100644 --- a/api/controllers/web/wraps.py +++ b/api/controllers/web/wraps.py @@ -19,6 +19,7 @@ from models.model import App, EndUser, Site from services.app_service import AppService from services.enterprise.enterprise_service import EnterpriseService, WebAppAccessMode, WebAppSettings from services.system_feature_service import SystemFeatureService +from services.web_passport_gateways import resolve_web_app_auth_type from services.webapp_auth_service import WebAppAuthService @@ -133,6 +134,14 @@ def _validate_user_accessibility( if not webapp_settings: raise WebAppAuthRequiredError("Web app settings not found.") + auth_type = decoded.get("auth_type") + if not auth_type: + raise WebAppAuthRequiredError("Missing auth_type in the token.") + + expected_auth_type = resolve_web_app_auth_type(webapp_settings.access_mode) + if auth_type != expected_auth_type: + raise WebAppAuthRequiredError() + if WebAppAuthService.is_app_require_permission_check( access_mode=webapp_settings.access_mode, session=db.session() ): @@ -140,10 +149,7 @@ def _validate_user_accessibility( if not EnterpriseService.WebAppAuth.is_user_allowed_to_access_webapp(user_id, app_id): raise WebAppAuthAccessDeniedError() - auth_type = decoded.get("auth_type") granted_at = decoded.get("granted_at") - if not auth_type: - raise WebAppAuthAccessDeniedError("Missing auth_type in the token.") if not granted_at: raise WebAppAuthAccessDeniedError("Missing granted_at in the token.") # check if sso has been updated diff --git a/api/core/rag/index_processor/index_processor_base.py b/api/core/rag/index_processor/index_processor_base.py index 7af2c517b84..06bacd68669 100644 --- a/api/core/rag/index_processor/index_processor_base.py +++ b/api/core/rag/index_processor/index_processor_base.py @@ -190,7 +190,16 @@ class BaseIndexProcessor(ABC): upload_file_id_list.append(upload_file_id) continue if current_user: - upload_file_id = self._download_image(image.split(" ")[0], current_user) + image_url = image.split(" ")[0] + try: + parsed_url = urlparse(image_url) + except ValueError: + logging.debug("Skipping malformed image reference: %s", image_url) + continue + if parsed_url.scheme not in {"http", "https"} or not parsed_url.netloc: + logging.debug("Skipping non-HTTP image reference: %s", image_url) + continue + upload_file_id = self._download_image(image_url, current_user) if upload_file_id: upload_file_id_list.append(upload_file_id) diff --git a/api/core/tools/tool_manager.py b/api/core/tools/tool_manager.py index 0f33b104914..fc02ecaf259 100644 --- a/api/core/tools/tool_manager.py +++ b/api/core/tools/tool_manager.py @@ -42,7 +42,7 @@ from core.tools.entities.tool_entities import ( ToolProviderType, emoji_icon_adapter, ) -from core.tools.errors import ToolProviderNotFoundError +from core.tools.errors import ToolProviderCredentialValidationError, ToolProviderNotFoundError from core.tools.mcp_tool.provider import MCPToolProviderController from core.tools.mcp_tool.tool import MCPTool from core.tools.plugin_tool.provider import PluginToolProviderController @@ -233,7 +233,10 @@ class ToolManager: builtin_provider = None logger.info("Error getting builtin provider %s:%s", credential_id, e, exc_info=True) if builtin_provider is None: - raise ToolProviderNotFoundError(f"provider has been deleted: {credential_id}") + raise ToolProviderCredentialValidationError( + f"Tool credential {credential_id} has been deleted. " + "Select or authorize another credential." + ) if builtin_provider is None: with Session(db.engine) as session: @@ -247,7 +250,10 @@ class ToolManager: .order_by(BuiltinToolProvider.is_default.desc(), BuiltinToolProvider.created_at.asc()) ) if builtin_provider is None: - raise ToolProviderNotFoundError(f"no default provider for {provider_id}") + raise ToolProviderCredentialValidationError( + f"No workspace credential is configured for tool provider {provider_id}. " + "Authorize the provider or select a credential." + ) else: builtin_provider = db.session.scalar( select(BuiltinToolProvider) @@ -259,7 +265,10 @@ class ToolManager: ) if builtin_provider is None: - raise ToolProviderNotFoundError(f"builtin provider {provider_id} not found") + raise ToolProviderCredentialValidationError( + f"No credential is configured for built-in tool provider {provider_id}. " + "Authorize the provider or select a credential." + ) from core.helper.credential_utils import runtime_check_credential_policy_compliance @@ -294,15 +303,24 @@ class ToolManager: system_credentials = BuiltinToolManageService.get_oauth_client(tenant_id, provider_id) oauth_handler = OAuthHandler() - refreshed_credentials = oauth_handler.refresh_credentials( - tenant_id=tenant_id, - user_id=builtin_provider.user_id, - plugin_id=tool_provider.plugin_id, - provider=provider_name, - redirect_uri=redirect_uri, - system_credentials=system_credentials or {}, - credentials=decrypted_credentials, - ) + try: + refreshed_credentials = oauth_handler.refresh_credentials( + tenant_id=tenant_id, + user_id=builtin_provider.user_id, + plugin_id=tool_provider.plugin_id, + provider=provider_name, + redirect_uri=redirect_uri, + system_credentials=system_credentials or {}, + credentials=decrypted_credentials, + ) + except Exception as exc: + logger.warning( + "Failed to refresh OAuth credentials for tool provider %s", provider_id, exc_info=True + ) + raise ToolProviderCredentialValidationError( + f"OAuth credential for tool provider {provider_id} could not be refreshed. " + "Reauthorize or select another credential." + ) from exc # update the credentials builtin_provider.encrypted_credentials = json.dumps( encrypter.encrypt(refreshed_credentials.credentials) diff --git a/api/core/workflow/nodes/agent/message_transformer.py b/api/core/workflow/nodes/agent/message_transformer.py index 4a145c84928..891cf4d0d77 100644 --- a/api/core/workflow/nodes/agent/message_transformer.py +++ b/api/core/workflow/nodes/agent/message_transformer.py @@ -27,6 +27,7 @@ from services.tools.builtin_tools_manage_service import BuiltinToolManageService from .events import AgentLogEvent from .exceptions import AgentNodeError, AgentVariableTypeError, ToolFileNotFoundError +from .think_tags import ThinkStreamState _file_access_controller = DatabaseFileAccessController() @@ -54,7 +55,7 @@ class AgentMessageTransformer: conversation_id=conversation_id, ) - text = "" + think_state = ThinkStreamState() files: list[File] = [] json_list: list[dict | list] = [] @@ -125,10 +126,10 @@ class AgentMessageTransformer: ) elif message.type == ToolInvokeMessage.MessageType.TEXT: assert isinstance(message.message, ToolInvokeMessage.TextMessage) - text += message.message.text + chunk = think_state.feed_text(message.message.text) yield StreamChunkEvent( selector=[node_id, "text"], - chunk=message.message.text, + chunk=chunk, is_final=False, ) elif message.type == ToolInvokeMessage.MessageType.JSON: @@ -152,8 +153,9 @@ class AgentMessageTransformer: linked_file = self._file_from_link_message(message=message, tenant_id=tenant_id) if linked_file is not None: files.append(linked_file) + yield from self._close_open_think(think_state=think_state, node_id=node_id) stream_text = f"{'File' if linked_file is not None else 'Link'}: {message.message.text}\n" - text += stream_text + think_state.feed_text(stream_text) yield StreamChunkEvent( selector=[node_id, "text"], chunk=stream_text, @@ -249,6 +251,8 @@ class AgentMessageTransformer: else: agent_logs.append(agent_log) + yield from self._close_open_think(think_state=think_state, node_id=node_id) + yield agent_log json_output: list[dict[str, Any] | list[Any]] = [] @@ -271,6 +275,8 @@ class AgentMessageTransformer: else: json_output.append({"data": []}) + yield from self._close_open_think(think_state=think_state, node_id=node_id) + yield StreamChunkEvent( selector=[node_id, "text"], chunk="", @@ -288,7 +294,7 @@ class AgentMessageTransformer: node_run_result=NodeRunResult( status=WorkflowNodeExecutionStatus.SUCCEEDED, outputs={ - "text": text, + "text": think_state.text, "usage": jsonable_encoder(llm_usage), "files": ArrayFileSegment(value=files), "json": json_output, @@ -304,6 +310,17 @@ class AgentMessageTransformer: ) ) + @staticmethod + def _close_open_think(*, think_state: ThinkStreamState, node_id: str) -> Generator[StreamChunkEvent, None, None]: + closed = think_state.close_if_open() + if closed is None: + return + yield StreamChunkEvent( + selector=[node_id, "text"], + chunk=closed, + is_final=False, + ) + @staticmethod def _file_from_link_message(*, message: ToolInvokeMessage, tenant_id: str) -> File | None: if not isinstance(message.message, ToolInvokeMessage.TextMessage): diff --git a/api/core/workflow/nodes/agent/think_tags.py b/api/core/workflow/nodes/agent/think_tags.py new file mode 100644 index 00000000000..fd6e741084c --- /dev/null +++ b/api/core/workflow/nodes/agent/think_tags.py @@ -0,0 +1,109 @@ +"""Normalize unclosed ```` tags in agent/workflow text streams. + +Reasoning models (GLM, DeepSeek, etc.) wrap chain-of-thought in ```` +tags. A tool call often interrupts generation before ```` is emitted, +so a later ```` or the final answer stays nested in the still-open tag +and the UI renders the reply as thinking. See https://github.com/langgenius/dify/issues/41558 +""" + +from __future__ import annotations + +THINK_OPEN = "" +THINK_CLOSE = "" + + +def has_unclosed_think(text: str) -> bool: + """Return True when a ```` is still open in ``text``.""" + _, inside, _ = _walk(text, inside=False, content_since_open=False, close_at_end=False) + return inside + + +def close_unclosed_think_tags(text: str) -> str: + """Insert missing ```` before a nested open tag and at end of text.""" + repaired, _, _ = _walk(text, inside=False, content_since_open=False, close_at_end=True) + return repaired + + +def normalize_think_chunk( + chunk: str, + *, + inside: bool, + content_since_open: bool = False, +) -> tuple[str, bool, bool]: + """Repair nested opens in a stream chunk and return updated open state. + + Does not append a trailing ```` — the caller closes on tool-call + interruptions and at end of stream. + """ + return _walk(chunk, inside=inside, content_since_open=content_since_open, close_at_end=False) + + +class ThinkStreamState: + """Track concatenated text and whether a ```` block is still open.""" + + def __init__(self) -> None: + self.text = "" + self.inside = False + self.content_since_open = False + + def feed_text(self, chunk: str) -> str: + repaired, self.inside, self.content_since_open = normalize_think_chunk( + chunk, + inside=self.inside, + content_since_open=self.content_since_open, + ) + self.text += repaired + return repaired + + def close_if_open(self) -> str | None: + if not self.inside: + return None + self.inside = False + self.content_since_open = False + self.text += THINK_CLOSE + return THINK_CLOSE + + +def _walk( + text: str, + *, + inside: bool, + content_since_open: bool, + close_at_end: bool, +) -> tuple[str, bool, bool]: + parts: list[str] = [] + i = 0 + length = len(text) + open_len = len(THINK_OPEN) + close_len = len(THINK_CLOSE) + + while i < length: + if text.startswith(THINK_OPEN, i): + if inside and content_since_open: + parts.append(THINK_CLOSE) + elif inside: + i += open_len + continue + parts.append(THINK_OPEN) + inside = True + content_since_open = False + i += open_len + continue + if text.startswith(THINK_CLOSE, i): + parts.append(THINK_CLOSE) + inside = False + content_since_open = False + i += close_len + continue + ch = text[i] + parts.append(ch) + if inside and not ch.isspace(): + content_since_open = True + i += 1 + + if close_at_end and inside: + parts.append(THINK_CLOSE) + inside = False + content_since_open = False + + return "".join(parts), inside, content_since_open diff --git a/api/events/event_handlers/sync_plugin_trigger_when_app_created.py b/api/events/event_handlers/sync_plugin_trigger_when_app_created.py index 68be37dfdbc..cc6e851ad87 100644 --- a/api/events/event_handlers/sync_plugin_trigger_when_app_created.py +++ b/api/events/event_handlers/sync_plugin_trigger_when_app_created.py @@ -1,6 +1,6 @@ import logging -from events.app_event import app_draft_workflow_was_synced +from events.app_event import app_draft_workflow_was_synced, app_published_workflow_was_updated from models.model import App, AppMode from models.workflow import Workflow from services.trigger.trigger_service import TriggerService @@ -9,14 +9,26 @@ logger = logging.getLogger(__name__) @app_draft_workflow_was_synced.connect -def handle(sender, synced_draft_workflow: Workflow, **kwargs): +@app_published_workflow_was_updated.connect +def handle( + sender, + synced_draft_workflow: Workflow | None = None, + published_workflow: Workflow | None = None, + **kwargs, +): """ - While creating a workflow or updating a workflow, we may need to sync - its plugin trigger relationships in DB. + Sync plugin trigger relationships when a draft changes or is published. + + The published workflow must be reconciled as well because production trigger + dispatch relies on these relationships, while debug dispatch does not. """ app: App = sender if app.mode != AppMode.WORKFLOW.value: # only handle workflow app, chatflow is not supported yet return - TriggerService.sync_plugin_trigger_relationships(app, synced_draft_workflow) + workflow = published_workflow if published_workflow is not None else synced_draft_workflow + if workflow is None: + return + + TriggerService.sync_plugin_trigger_relationships(app, workflow) diff --git a/api/extensions/ext_application_services.py b/api/extensions/ext_application_services.py index 65b948ccb9f..d4ed5580e9c 100644 --- a/api/extensions/ext_application_services.py +++ b/api/extensions/ext_application_services.py @@ -1,9 +1,11 @@ """Composition root for application services used by transport adapters.""" import json +import time from collections.abc import Mapping from dataclasses import dataclass from datetime import UTC, datetime +from functools import partial from typing import cast from uuid import uuid4 @@ -14,35 +16,52 @@ from sqlalchemy.orm import Session, sessionmaker from configs import dify_config from constants.dsl_version import CURRENT_APP_DSL_VERSION +from constants.languages import languages from core.db.session_factory import get_session_maker from core.helper.ssrf_proxy import ssrf_proxy from core.schemas.schema_manager import SchemaManager +from core.tools.tool_file_manager import ToolFileManager from enums import DeploymentEdition, WebAppAccessMode from extensions.ext_redis import RedisClientWrapper, redis_client -from libs.datetime_utils import naive_utc_now +from extensions.ext_storage import storage +from libs.datetime_utils import naive_utc_now, utc_now from libs.helper import RateLimiter +from libs.oauth import GitHubOAuth, GoogleOAuth +from libs.oauth_bearer import invalidate_oauth_token_cache from libs.passport import PassportService from repositories.account_activation_repository import SQLAlchemyAccountActivationRepository from repositories.account_integration_repository import SQLAlchemyAccountIntegrationRepository +from repositories.account_oauth_repository import ( + AccountServiceOAuthAccountRegistrationGateway, + AccountServiceOAuthSessionGateway, + AccountServiceOAuthWorkspaceGateway, + RegisterServiceOAuthInvitationGateway, +) from repositories.account_repository import SQLAlchemyAccountRepository from repositories.app_definition_query_repository import AppDefinitionQueryRepository from repositories.app_site_command_repository import AppSiteCommandRepository +from repositories.app_statistic_query_repository import AppStatisticQueryRepository from repositories.data_source_api_key_auth_repository import SQLAlchemyDataSourceApiKeyAuthBindingRepository from repositories.data_source_oauth_binding_repository import SQLAlchemyDataSourceOAuthBindingRepository from repositories.explore_banner_query_repository import ExploreBannerQueryRepository from repositories.factory import DifyAPIRepositoryFactory +from repositories.file_grant_repository import FileGrantRepository from repositories.installation_state_repository import InstallationStateRepository +from repositories.oauth_access_token_repository import SQLAlchemyOAuthAccessTokenRepository from repositories.oauth_server_repository import RedisOAuthServerTokenRepository, SQLAlchemyOAuthServerRepository from repositories.recommended_app_catalog_repository import DatabaseRecommendedAppCatalogRepository +from repositories.sqlalchemy_api_workflow_run_repository import DifyAPISQLAlchemyWorkflowRunRepository from repositories.step_by_step_tour_repository import SQLAlchemyStepByStepTourStateRepository from repositories.tag_repository import TagRepository from repositories.trial_app_query_repository import TrialAppQueryRepository from repositories.trial_app_usage_repository import TrialAppUsageRepository from repositories.web_passport_repository import WebPassportRepository from repositories.webapp_access_query_repository import WebAppAccessQueryRepository +from repositories.workflow_app_log_query_repository import WorkflowAppLogQueryRepository from repositories.workflow_run_archive_repository import WorkflowRunArchiveBundleQueryRepository from repositories.workspace_member_query_repository import WorkspaceMemberQueryRepository from repositories.workspace_query_repository import WorkspaceQueryRepository +from services.account_access_service import AccountAccessService from services.account_activation_service import AccountActivationService from services.account_adapters import ( BillingAccountActivationEligibility, @@ -103,11 +122,18 @@ from services.account_login_adapters import ( TurnstileHumanVerificationGateway, ) from services.account_login_service import ConsoleAuthenticationService +from services.account_oauth_adapters import ( + DeploymentOAuthPolicyGateway, + DifyOAuthProviderGateway, + RedisOAuthAccountClaimLock, +) +from services.account_oauth_service import AccountOAuthService, OAuthProviderGateway from services.account_password_hasher import DefaultAccountPasswordHasher from services.account_password_service import AccountPasswordService from services.account_profile_service import AccountProfileService from services.app_definition_query_service import AppDefinitionQueryService from services.app_site_service import AppSiteService +from services.app_statistic_query import AppStatisticQuery from services.auth.data_source_api_key_auth_gateways import ( ProviderApiKeyAuthCredentialValidator, TenantApiKeyAuthCredentialEncryptor, @@ -118,10 +144,13 @@ from services.billing_service import BillingService from services.compliance_download_service import ComplianceDownloadService from services.data_source_oauth_service import DataSourceOAuthService, InvalidDataSourceOAuthProviderError from services.enterprise.enterprise_service import EnterpriseService +from services.entities.file_grant_entities import FileGrantLimits from services.errors.enterprise import EnterpriseServiceError from services.explore_banner_query_service import ExploreBannerQueryService from services.feature_query_service import FeatureQueryService from services.feature_service_gateway import FeatureServiceGateway +from services.file_grant_gateways import FileGrantFileGateway, FileGrantRemoteFileGateway, FileGrantTokenGateway +from services.file_grant_service import FileGrantService from services.file_service import FileService from services.init_validation_service import InitValidationService from services.inner_mail_service import InnerMailService @@ -160,6 +189,8 @@ from services.webapp_access_query_service import ( WebAppAccessQueryService, WebAppAccessUnavailableError, ) +from services.workflow_app_log_query_service import WorkflowAppLogQueryService +from services.workflow_run_service import WorkflowRunService from services.workflow_statistic_query_service import WorkflowStatisticQueryService from services.workspace_member_query_service import WorkspaceMemberQueryService from services.workspace_member_role_resolver import DeploymentWorkspaceMemberRoleResolver @@ -190,6 +221,7 @@ def _is_user_allowed_to_access_webapp(user_id: str, app_id: str) -> bool: @dataclass(frozen=True, slots=True) class AccountServices: + access: AccountAccessService authentication: ConsoleAuthenticationService avatar: AccountAvatarService change_email: AccountChangeEmailService @@ -200,6 +232,7 @@ class AccountServices: forgot_password: AccountForgotPasswordService initialization: AccountInitializationService integrations: AccountIntegrationService + oauth: AccountOAuthService password: AccountPasswordService profile: AccountProfileService @@ -210,6 +243,7 @@ class ApplicationServices: account_activation: AccountActivationService app_definitions: AppDefinitionQueryService app_sites: AppSiteService + app_statistics: AppStatisticQuery billing_portal: BillingPortalService compliance_downloads: ComplianceDownloadService data_source_api_key_auth: DataSourceApiKeyAuthService @@ -220,6 +254,7 @@ class ApplicationServices: schema_definitions: SchemaDefinitionService setup: SetupService feature_queries: FeatureQueryService + file_grants: FileGrantService files: FileService oauth_server: OAuthServerService init_validation: InitValidationService @@ -230,8 +265,10 @@ class ApplicationServices: remote_files: RemoteFileService trial_app_usage: TrialAppUsageRecorder workflow_run_archives: WorkflowRunArchiveService + workflow_runs: WorkflowRunService workspace_queries: WorkspaceQueryService workspace_member_queries: WorkspaceMemberQueryService + workflow_app_logs: WorkflowAppLogQueryService inner_mail: InnerMailService web_passport: WebPassportService tags: TagApplicationService @@ -278,6 +315,86 @@ def _build_oauth_server_service( ) +def _build_file_grant_service(*, database_client: sessionmaker[Session]) -> FileGrantService: + repository = FileGrantRepository(session_factory=database_client) + return FileGrantService( + repository=repository, + files=FileGrantFileGateway( + load_end_user=repository.get_end_user, + subject_exists=repository.subject_exists, + file_service=FileService(session_factory=database_client), + tool_files=ToolFileManager(), + storage=storage, + ), + remote_files=FileGrantRemoteFileGateway(), + tokens=FileGrantTokenGateway( + secret_key=dify_config.SECRET_KEY, + external_files_url=dify_config.FILES_URL, + internal_files_url=dify_config.INTERNAL_FILES_URL or dify_config.FILES_URL, + content_token_ttl_seconds=dify_config.FILES_ACCESS_TIMEOUT, + now=lambda: int(time.time()), + ), + limits=FileGrantLimits( + file_size_limit=dify_config.UPLOAD_FILE_SIZE_LIMIT, + image_file_size_limit=dify_config.UPLOAD_IMAGE_FILE_SIZE_LIMIT, + audio_file_size_limit=dify_config.UPLOAD_AUDIO_FILE_SIZE_LIMIT, + video_file_size_limit=dify_config.UPLOAD_VIDEO_FILE_SIZE_LIMIT, + workflow_file_upload_limit=dify_config.WORKFLOW_FILE_UPLOAD_LIMIT, + batch_count_limit=dify_config.UPLOAD_FILE_BATCH_LIMIT, + ), + now=lambda: int(time.time()), + ) + + +def _build_account_oauth_service( + *, + database_client: sessionmaker[Session], + deployment_edition: DeploymentEdition, + redis: RedisClientWrapper, + accounts: SQLAlchemyAccountRepository, + integrations: SQLAlchemyAccountIntegrationRepository, + memberships: WorkspaceQueryRepository, +) -> AccountOAuthService: + providers: dict[str, OAuthProviderGateway] = {} + if dify_config.GITHUB_CLIENT_ID and dify_config.GITHUB_CLIENT_SECRET: + providers["github"] = DifyOAuthProviderGateway( + provider_name="github", + client=GitHubOAuth( + client_id=dify_config.GITHUB_CLIENT_ID, + client_secret=dify_config.GITHUB_CLIENT_SECRET, + redirect_uri=dify_config.CONSOLE_API_URL + "/console/api/oauth/authorize/github", + ), + ) + if dify_config.GOOGLE_CLIENT_ID and dify_config.GOOGLE_CLIENT_SECRET: + providers["google"] = DifyOAuthProviderGateway( + provider_name="google", + client=GoogleOAuth( + client_id=dify_config.GOOGLE_CLIENT_ID, + client_secret=dify_config.GOOGLE_CLIENT_SECRET, + redirect_uri=dify_config.CONSOLE_API_URL + "/console/api/oauth/authorize/google", + ), + ) + + policy = DeploymentOAuthPolicyGateway( + billing_enabled=deployment_edition == DeploymentEdition.CLOUD, + ) + return AccountOAuthService( + providers=providers, + accounts=accounts, + integrations=integrations, + memberships=memberships, + invitations=RegisterServiceOAuthInvitationGateway(session_factory=database_client), + account_claims=RedisOAuthAccountClaimLock(client=redis), + registration=AccountServiceOAuthAccountRegistrationGateway(session_factory=database_client), + workspaces=AccountServiceOAuthWorkspaceGateway(session_factory=database_client), + sessions=AccountServiceOAuthSessionGateway(session_factory=database_client), + registration_policy=policy, + workspace_policy=policy, + supported_languages=languages, + now=naive_utc_now, + ) + + def build_application_services( *, database_client: sessionmaker[Session], @@ -306,8 +423,19 @@ def build_application_services( invitation_tokens = RedisInvitationTokenStore(redis=redis) activation_accounts = SQLAlchemyAccountActivationRepository(session_factory=database_client) account_provisioning = SQLAlchemyConsoleAuthProvisioningGateway(session_factory=database_client) + workflow_run_repository = DifyAPISQLAlchemyWorkflowRunRepository(session_maker=database_client) + workflow_node_execution_repository = DifyAPIRepositoryFactory.create_api_workflow_node_execution_repository( + session_maker=database_client + ) return ApplicationServices( accounts=AccountServices( + access=AccountAccessService( + accounts=accounts, + workspaces=workspace_query_repository, + sessions=SQLAlchemyOAuthAccessTokenRepository(session_factory=database_client), + invalidate_token_cache=partial(invalidate_oauth_token_cache, redis), + now=utc_now, + ), authentication=ConsoleAuthenticationService( accounts=accounts, workspaces=workspace_query_repository, @@ -426,6 +554,14 @@ def build_application_services( now=naive_utc_now, ), integrations=AccountIntegrationService(integrations=integrations), + oauth=_build_account_oauth_service( + database_client=database_client, + deployment_edition=deployment_edition, + redis=redis, + accounts=accounts, + integrations=integrations, + memberships=workspace_query_repository, + ), password=AccountPasswordService( accounts=accounts, passwords=passwords, @@ -455,6 +591,7 @@ def build_application_services( app_sites=AppSiteService( sites=AppSiteCommandRepository(session_factory=database_client), ), + app_statistics=AppStatisticQueryRepository(session_factory=database_client), billing_portal=BillingPortalService( accounts=accounts, get_subscription=BillingService.get_subscription, @@ -502,6 +639,7 @@ def build_application_services( features=feature_gateway, app_dsl_version=CURRENT_APP_DSL_VERSION, ), + file_grants=_build_file_grant_service(database_client=database_client), files=file_service, oauth_server=_build_oauth_server_service(database_client=database_client, redis=redis), init_validation=InitValidationService( @@ -510,7 +648,6 @@ def build_application_services( expected_password=initialization_password, ), notifications=NotificationService( - accounts=accounts, notifications=BillingNotificationGateway(), ), step_by_step_tour=StepByStepTourService( @@ -537,6 +674,10 @@ def build_application_services( dispatcher=dispatch_workflow_run_archive_download_task, sign_download_url=sign_workflow_run_archive_download_url, ), + workflow_runs=WorkflowRunService( + workflow_runs=workflow_run_repository, + node_executions=workflow_node_execution_repository, + ), workspace_queries=WorkspaceQueryService( workspaces=workspace_query_repository, plans=DeploymentWorkspacePlanGateway(), @@ -547,6 +688,9 @@ def build_application_services( ), roles=DeploymentWorkspaceMemberRoleResolver(), ), + workflow_app_logs=WorkflowAppLogQueryService( + logs=WorkflowAppLogQueryRepository(session_factory=database_client), + ), inner_mail=InnerMailService(dispatch=enqueue_inner_mail), web_passport=WebPassportService( passports=WebPassportRepository( diff --git a/api/fields/file_grant_fields.py b/api/fields/file_grant_fields.py new file mode 100644 index 00000000000..b0c550bb425 --- /dev/null +++ b/api/fields/file_grant_fields.py @@ -0,0 +1,49 @@ +"""Response DTOs shared by the AppDeploy file grant surfaces.""" + +from __future__ import annotations + +from fields.base import ResponseModel +from services.entities.file_grant_entities import FileKind, ResolvedFileAccess + + +class ResolvedFileResponse(ResponseModel): + """One requested file reference, either resolved or accounted for. + + The resolve endpoint and optional mint references answer item by item so one + missing history file cannot fail a whole run. A file that exists but belongs + to another owner is reported exactly like one that never existed. + """ + + id: str + ok: bool + kind: FileKind | None = None + name: str | None = None + size: int | None = None + extension: str | None = None + mime_type: str | None = None + url: str | None = None + internal_url: str | None = None + error: str | None = None + + @classmethod + def from_resolved(cls, file_id: str, access: ResolvedFileAccess | None) -> ResolvedFileResponse: + """Describe one resolved reference without performing infrastructure work.""" + + if access is None: + return cls(id=file_id, ok=False, error="not_found") + + file = access.file + return cls( + id=file.id, + ok=True, + kind=file.kind, + name=file.name, + size=file.size, + extension=file.extension, + mime_type=file.mime_type, + url=access.external_url, + internal_url=access.internal_url, + ) + + +__all__ = ["ResolvedFileResponse"] diff --git a/api/libs/datetime_utils.py b/api/libs/datetime_utils.py index d962d81e78a..a158bb1d5ea 100644 --- a/api/libs/datetime_utils.py +++ b/api/libs/datetime_utils.py @@ -17,11 +17,16 @@ class _NowFunction(Protocol): _now_func: _NowFunction = datetime.datetime.now +def utc_now() -> datetime.datetime: + """Return a timezone-aware datetime representing the current UTC time.""" + return _now_func(datetime.UTC) + + def naive_utc_now() -> datetime.datetime: """Return a naive datetime object (without timezone information) representing current UTC time. """ - return _now_func(datetime.UTC).replace(tzinfo=None) + return utc_now().replace(tzinfo=None) def ensure_naive_utc(dt: datetime.datetime) -> datetime.datetime: diff --git a/api/libs/oauth_bearer.py b/api/libs/oauth_bearer.py index ed17503bb3b..89ede4b8a12 100644 --- a/api/libs/oauth_bearer.py +++ b/api/libs/oauth_bearer.py @@ -318,6 +318,14 @@ AUDIT_OAUTH_EXPIRED = "oauth.token_expired" ScopeVariant = Literal["account", "external_sso"] +class _TokenCacheClient(Protocol): + def delete(self, *names: str | bytes) -> object: ... + + +def invalidate_oauth_token_cache(client: _TokenCacheClient, token_hash: str) -> None: + client.delete(TOKEN_CACHE_KEY_FMT.format(hash=token_hash)) + + class OAuthAccessTokenResolver: """``.for_account()`` / ``.for_external_sso()`` are variant-scoped views sharing DB + cache plumbing. @@ -385,7 +393,7 @@ class OAuthAccessTokenResolver: row_id, extra={"audit": True, "token_id": str(row_id)}, ) - self._redis.delete(self._cache_key(token_hash)) + invalidate_oauth_token_cache(self._redis, token_hash) self.cache_set_negative(token_hash) diff --git a/api/machinery/context.py b/api/machinery/context.py index 5d6f5436619..4facf6caf9b 100644 --- a/api/machinery/context.py +++ b/api/machinery/context.py @@ -8,3 +8,12 @@ class RequestContext(NamedTuple): trace_id: str | None account_id: str active_workspace_id: str + + +class AccountRequestContext(NamedTuple): + """Stable identity for account-scoped use cases that do not require a workspace.""" + + request_id: str + trace_id: str | None + account_id: str + access_token_id: str | None = None diff --git a/api/models/account.py b/api/models/account.py index 822b784a2fa..ed7bc2258b2 100644 --- a/api/models/account.py +++ b/api/models/account.py @@ -187,15 +187,6 @@ class Account(UserMixin, TypeBase): def get_status(self) -> AccountStatus: return self.status - @classmethod - def get_by_openid(cls, provider: str, open_id: str): - account_integrate = db.session.execute( - select(AccountIntegrate).where(AccountIntegrate.provider == provider, AccountIntegrate.open_id == open_id) - ).scalar_one_or_none() - if account_integrate: - return db.session.scalar(select(Account).where(Account.id == account_integrate.account_id)) - return None - # check current_user.current_tenant.current_role in ['admin', 'owner'] @property def is_admin_or_owner(self): @@ -312,7 +303,7 @@ class TenantAccountJoin(TypeBase): ) tenant_id: Mapped[str] = mapped_column(StringUUID) account_id: Mapped[str] = mapped_column(StringUUID) - current: Mapped[bool] = mapped_column(sa.Boolean, server_default=sa.text("false"), default=False) + current: Mapped[bool] = mapped_column(sa.Boolean, server_default=sa.false(), default=False) role: Mapped[TenantAccountRole] = mapped_column( EnumText(TenantAccountRole, length=16), server_default="normal", default=TenantAccountRole.NORMAL ) diff --git a/api/models/agent.py b/api/models/agent.py index d981194c714..eb1fafae98a 100644 --- a/api/models/agent.py +++ b/api/models/agent.py @@ -189,13 +189,13 @@ class Agent(DefaultFieldsMixin, Base): workflow_node_id: Mapped[str | None] = mapped_column(String(255), nullable=True) active_config_snapshot_id: Mapped[str | None] = mapped_column(StringUUID, nullable=True) active_config_has_model: Mapped[bool] = mapped_column( - sa.Boolean, nullable=False, default=False, server_default=sa.text("false") + sa.Boolean, nullable=False, default=False, server_default=sa.false() ) active_config_is_published: Mapped[bool] = mapped_column( sa.Boolean, nullable=False, default=False, - server_default=sa.text("false"), + server_default=sa.false(), comment=( "Whether the normal shared Agent draft has been published into the active config snapshot. " "User-scoped debug drafts do not affect this flag." diff --git a/api/models/comment.py b/api/models/comment.py index 222284f42b3..565169126f7 100644 --- a/api/models/comment.py +++ b/api/models/comment.py @@ -64,7 +64,7 @@ class WorkflowComment(TypeBase): resolved_at: Mapped[datetime | None] = mapped_column(sa.DateTime, default=None) resolved_by: Mapped[str | None] = mapped_column(StringUUID, default=None) - resolved: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("false"), default=False) + resolved: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.false(), default=False) # Relationships replies: Mapped[list[WorkflowCommentReply]] = relationship( lambda: WorkflowCommentReply, back_populates="comment", cascade="all, delete-orphan", init=False diff --git a/api/models/credential_permission.py b/api/models/credential_permission.py index effa78635b1..969a2e32d0d 100644 --- a/api/models/credential_permission.py +++ b/api/models/credential_permission.py @@ -45,9 +45,7 @@ class CredentialPermission(TypeBase): credential_type: Mapped[str] = mapped_column(String(40), nullable=False) account_id: Mapped[str] = mapped_column(StringUUID, nullable=False) tenant_id: Mapped[str] = mapped_column(StringUUID, nullable=False) - has_permission: Mapped[bool] = mapped_column( - sa.Boolean, nullable=False, server_default=sa.text("true"), default=True - ) + has_permission: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.true(), default=True) created_at: Mapped[datetime] = mapped_column( DateTime, nullable=False, server_default=func.current_timestamp(), init=False ) diff --git a/api/models/dataset.py b/api/models/dataset.py index ddcc8020ed8..327c9503420 100644 --- a/api/models/dataset.py +++ b/api/models/dataset.py @@ -202,15 +202,15 @@ class Dataset(Base): collection_binding_id = mapped_column(StringUUID, nullable=True) retrieval_model = mapped_column(AdjustedJSON, nullable=True) summary_index_setting = mapped_column(AdjustedJSON, nullable=True) - built_in_field_enabled = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("false")) + built_in_field_enabled = mapped_column(sa.Boolean, nullable=False, server_default=sa.false()) icon_info = mapped_column(AdjustedJSON, nullable=True) runtime_mode = mapped_column( EnumText(DatasetRuntimeMode, length=255), nullable=True, server_default=sa.text("'general'") ) pipeline_id = mapped_column(StringUUID, nullable=True) chunk_structure = mapped_column(sa.String(255), nullable=True) - enable_api = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("true")) - is_multimodal = mapped_column(sa.Boolean, default=False, nullable=False, server_default=sa.text("false")) + enable_api = mapped_column(sa.Boolean, nullable=False, server_default=sa.true()) + is_multimodal = mapped_column(sa.Boolean, default=False, nullable=False, server_default=sa.false()) def get_total_documents(self, *, session: Session) -> int: return self.get_document_count(session=session) @@ -560,7 +560,7 @@ class Document(Base): completed_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True) # pause - is_paused: Mapped[bool | None] = mapped_column(sa.Boolean, nullable=True, server_default=sa.text("false")) + is_paused: Mapped[bool | None] = mapped_column(sa.Boolean, nullable=True, server_default=sa.false()) paused_by = mapped_column(StringUUID, nullable=True) paused_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True) @@ -572,10 +572,10 @@ class Document(Base): indexing_status = mapped_column( EnumText(IndexingStatus, length=255), nullable=False, server_default=sa.text("'waiting'") ) - enabled: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("true")) + enabled: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.true()) disabled_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True) disabled_by = mapped_column(StringUUID, nullable=True) - archived: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("false")) + archived: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.false()) archived_reason = mapped_column(String(255), nullable=True) archived_by = mapped_column(StringUUID, nullable=True) archived_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True) @@ -588,7 +588,7 @@ class Document(Base): EnumText(IndexStructureType, length=255), nullable=False, server_default=sa.text("'text_model'") ) doc_language = mapped_column(String(255), nullable=True) - need_summary: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("false")) + need_summary: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.false()) DATA_SOURCES = ["upload_file", "notion_import", "website_crawl"] @@ -622,10 +622,6 @@ class Document(Base): return data_source_info_dict return {} - @property - def data_source_detail_dict(self) -> dict[str, Any]: - return self.get_data_source_detail_dict(session=db.session()) - def get_data_source_detail_dict(self, *, session: Session) -> dict[str, Any]: if self.data_source_info: if self.data_source_type == "upload_file": @@ -665,10 +661,6 @@ class Document(Base): return session.get(DatasetProcessRule, self.dataset_process_rule_id) return None - @property - def dataset(self) -> Dataset | None: - return self.get_dataset(session=db.session()) - def get_dataset(self, *, session: Session) -> Dataset | None: """Load the owning dataset with the caller-owned database session.""" return session.get(Dataset, self.dataset_id) @@ -694,10 +686,6 @@ class Document(Base): or 0 ) - @property - def uploader(self): - return self.get_uploader(session=db.session()) - def get_uploader(self, *, session: Session) -> str | None: user = session.scalar(select(Account).where(Account.id == self.created_by)) return user.name if user else None @@ -710,10 +698,6 @@ class Document(Base): def last_update_date(self): return self.updated_at - @property - def doc_metadata_details(self) -> list[DocMetadataDetailItem] | None: - return self.get_doc_metadata_details(session=db.session()) - def get_doc_metadata_details(self, *, session: Session) -> list[DocMetadataDetailItem] | None: if self.doc_metadata: document_metadatas = session.scalars( @@ -917,7 +901,7 @@ class DocumentSegment(TypeBase): # indexing fields index_node_id: Mapped[str | None] = mapped_column(String(255), nullable=True, default=None) index_node_hash: Mapped[str | None] = mapped_column(String(255), nullable=True, default=None) - enabled: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("true"), default=True) + enabled: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.true(), default=True) answer: Mapped[str | None] = mapped_column(LongText, nullable=True, default=None) keywords: Mapped[Any] = mapped_column(sa.JSON, nullable=True, default=None) disabled_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True, default=None) @@ -1378,7 +1362,7 @@ class TidbAuthBinding(TypeBase): tenant_id: Mapped[str | None] = mapped_column(StringUUID, nullable=True) cluster_id: Mapped[str] = mapped_column(String(255), nullable=False) cluster_name: Mapped[str] = mapped_column(String(255), nullable=False) - active: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("false")) + active: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.false()) status: Mapped[TidbAuthBindingStatus] = mapped_column( EnumText(TidbAuthBindingStatus, length=255), nullable=False, server_default=sa.text("'CREATING'") ) @@ -1429,9 +1413,7 @@ class DatasetPermission(TypeBase): dataset_id: Mapped[str] = mapped_column(StringUUID, nullable=False) account_id: Mapped[str] = mapped_column(StringUUID, nullable=False) tenant_id: Mapped[str] = mapped_column(StringUUID, nullable=False) - has_permission: Mapped[bool] = mapped_column( - sa.Boolean, nullable=False, server_default=sa.text("true"), default=True - ) + has_permission: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.true(), default=True) created_at: Mapped[datetime] = mapped_column( DateTime, nullable=False, server_default=func.current_timestamp(), init=False ) @@ -1547,7 +1529,7 @@ class DatasetAutoDisableLog(TypeBase): tenant_id: Mapped[str] = mapped_column(StringUUID, nullable=False) dataset_id: Mapped[str] = mapped_column(StringUUID, nullable=False) document_id: Mapped[str] = mapped_column(StringUUID, nullable=False) - notified: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("false"), default=False) + notified: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.false(), default=False) created_at: Mapped[datetime] = mapped_column( DateTime, nullable=False, server_default=sa.func.current_timestamp(), init=False ) @@ -1703,10 +1685,8 @@ class Pipeline(TypeBase): name: Mapped[str] = mapped_column(sa.String(255), nullable=False) description: Mapped[str] = mapped_column(LongText, nullable=False, default=sa.text("''")) workflow_id: Mapped[str | None] = mapped_column(StringUUID, nullable=True, default=None) - is_public: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("false"), default=False) - is_published: Mapped[bool] = mapped_column( - sa.Boolean, nullable=False, server_default=sa.text("false"), default=False - ) + is_public: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.false(), default=False) + is_published: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.false(), default=False) created_by: Mapped[str | None] = mapped_column(StringUUID, nullable=True, default=None) created_at: Mapped[datetime] = mapped_column( sa.DateTime, nullable=False, server_default=func.current_timestamp(), init=False @@ -1830,7 +1810,7 @@ class DocumentSegmentSummary(TypeBase): default=SummaryStatus.GENERATING, ) error: Mapped[str | None] = mapped_column(LongText, nullable=True, default=None) - enabled: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("true"), default=True) + enabled: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.true(), default=True) disabled_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True, default=None) disabled_by: Mapped[str | None] = mapped_column(StringUUID, nullable=True, default=None) created_at: Mapped[datetime] = mapped_column( diff --git a/api/models/enums.py b/api/models/enums.py index ff8fb422a51..d86838a119d 100644 --- a/api/models/enums.py +++ b/api/models/enums.py @@ -208,6 +208,7 @@ class InvokeFrom(StrEnum): class EndUserType(StrEnum): """Persisted type values for the ``end_users.type`` column.""" + APP_DEPLOY = "app-deploy" BROWSER = "browser" MCP = "mcp" OPENAPI = "openapi" diff --git a/api/models/model.py b/api/models/model.py index 9fd53e554ef..050b63ee0b3 100644 --- a/api/models/model.py +++ b/api/models/model.py @@ -435,9 +435,9 @@ class App(Base): enable_api: Mapped[bool] = mapped_column(sa.Boolean) api_rpm: Mapped[int] = mapped_column(sa.Integer, server_default=sa.text("0")) api_rph: Mapped[int] = mapped_column(sa.Integer, server_default=sa.text("0")) - is_demo: Mapped[bool] = mapped_column(sa.Boolean, server_default=sa.text("false")) - is_public: Mapped[bool] = mapped_column(sa.Boolean, server_default=sa.text("false")) - is_universal: Mapped[bool] = mapped_column(sa.Boolean, server_default=sa.text("false")) + is_demo: Mapped[bool] = mapped_column(sa.Boolean, server_default=sa.false()) + is_public: Mapped[bool] = mapped_column(sa.Boolean, server_default=sa.false()) + is_universal: Mapped[bool] = mapped_column(sa.Boolean, server_default=sa.false()) tracing = mapped_column(LongText, nullable=True) max_active_requests: Mapped[int | None] created_by = mapped_column(StringUUID, nullable=True) @@ -447,7 +447,7 @@ class App(Base): updated_at: Mapped[datetime] = mapped_column( sa.DateTime, nullable=False, server_default=func.current_timestamp(), onupdate=func.current_timestamp() ) - use_icon_as_answer_icon: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("false")) + use_icon_as_answer_icon: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.false()) @property def desc_or_prompt(self) -> str: @@ -989,12 +989,8 @@ class RecommendedApp(TypeBase): custom_disclaimer: Mapped[str] = mapped_column(LongText, default="") position: Mapped[int] = mapped_column(sa.Integer, nullable=False, default=0) is_listed: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, default=True) - is_learn_dify: Mapped[bool] = mapped_column( - sa.Boolean, nullable=False, server_default=sa.text("false"), default=False - ) - is_cloud_only: Mapped[bool] = mapped_column( - sa.Boolean, nullable=False, server_default=sa.text("false"), default=False - ) + is_learn_dify: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.false(), default=False) + is_cloud_only: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.false(), default=False) install_count: Mapped[int] = mapped_column(sa.Integer, nullable=False, default=0) language: Mapped[str] = mapped_column( String(255), @@ -1034,7 +1030,7 @@ class InstalledApp(TypeBase): app_id: Mapped[str] = mapped_column(StringUUID, nullable=False) app_owner_tenant_id: Mapped[str] = mapped_column(StringUUID, nullable=False) position: Mapped[int] = mapped_column(sa.Integer, nullable=False, default=0) - is_pinned: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("false"), default=False) + is_pinned: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.false(), default=False) last_used_at: Mapped[datetime | None] = mapped_column(sa.DateTime, nullable=True, default=None) created_at: Mapped[datetime] = mapped_column( sa.DateTime, nullable=False, server_default=func.current_timestamp(), init=False @@ -1239,7 +1235,7 @@ class Conversation(Base): lambda: MessageAnnotation, backref="conversation", lazy="select", passive_deletes="all" ) - is_deleted: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("false")) + is_deleted: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.false()) @property def inputs(self) -> dict[str, Any]: @@ -1598,7 +1594,7 @@ class Message(Base): updated_at: Mapped[datetime] = mapped_column( sa.DateTime, nullable=False, server_default=func.current_timestamp(), onupdate=func.current_timestamp() ) - agent_based: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("false")) + agent_based: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.false()) workflow_run_id: Mapped[str | None] = mapped_column(StringUUID) app_mode: Mapped[AppMode | None] = mapped_column(EnumText(AppMode, length=255), nullable=True) @@ -2249,9 +2245,7 @@ class EndUser(Base, UserMixin): type: Mapped[EndUserType] = mapped_column(EnumText(EndUserType, length=255), nullable=False) external_user_id = mapped_column(String(255), nullable=True) name = mapped_column(String(255)) - _is_anonymous: Mapped[bool] = mapped_column( - "is_anonymous", sa.Boolean, nullable=False, server_default=sa.text("true") - ) + _is_anonymous: Mapped[bool] = mapped_column("is_anonymous", sa.Boolean, nullable=False, server_default=sa.true()) @property @override @@ -2366,17 +2360,15 @@ class Site(TypeBase): customize_domain: Mapped[str | None] = mapped_column(String(255), nullable=True, default=None) chat_color_theme: Mapped[str | None] = mapped_column(String(255), nullable=True, default=None) - prompt_public: Mapped[bool] = mapped_column( - sa.Boolean, nullable=False, server_default=sa.text("false"), default=False - ) + prompt_public: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.false(), default=False) chat_color_theme_inverted: Mapped[bool] = mapped_column( - sa.Boolean, nullable=False, server_default=sa.text("false"), default=False + sa.Boolean, nullable=False, server_default=sa.false(), default=False ) show_workflow_steps: Mapped[bool] = mapped_column( - sa.Boolean, nullable=False, server_default=sa.text("true"), default=True + sa.Boolean, nullable=False, server_default=sa.true(), default=True ) use_icon_as_answer_icon: Mapped[bool] = mapped_column( - sa.Boolean, nullable=False, server_default=sa.text("false"), default=False + sa.Boolean, nullable=False, server_default=sa.false(), default=False ) custom_disclaimer: Mapped[str] = mapped_column(LongText, nullable=False, default="") status: Mapped[AppStatus] = mapped_column( @@ -2535,7 +2527,7 @@ class UploadFile(TypeBase): # 3. Avoid relying on these fields for logic, as their values may not always be accurate. # # `used` may indicate whether the file has been utilized by another service. - used: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("false")) + used: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.false()) # The `created_by_role` field indicates whether the file was created by an `Account` or an `EndUser`. # Its value is derived from the `CreatorUserRole` enumeration. @@ -2846,7 +2838,7 @@ class TraceAppConfig(TypeBase): onupdate=func.current_timestamp(), init=False, ) - is_active: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("true"), default=True) + is_active: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.true(), default=True) @property def tracing_config_dict(self) -> dict[str, Any]: diff --git a/api/models/oauth.py b/api/models/oauth.py index 84be7eb37ce..b3effe1f5e2 100644 --- a/api/models/oauth.py +++ b/api/models/oauth.py @@ -45,7 +45,7 @@ class DatasourceProvider(TypeBase): encrypted_credentials: Mapped[dict[str, Any]] = mapped_column(AdjustedJSON, nullable=False) user_id: Mapped[str | None] = mapped_column(StringUUID, nullable=True, default=None) avatar_url: Mapped[str] = mapped_column(LongText, nullable=True, default="default") - is_default: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("false"), default=False) + is_default: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.false(), default=False) expires_at: Mapped[int] = mapped_column(sa.Integer, nullable=False, server_default="-1", default=-1) visibility: Mapped[PermissionEnum] = mapped_column( EnumText(PermissionEnum, length=40), diff --git a/api/models/onboarding.py b/api/models/onboarding.py index 3495d91274e..e8d06070fc9 100644 --- a/api/models/onboarding.py +++ b/api/models/onboarding.py @@ -32,7 +32,7 @@ class AccountStepByStepTourState(TypeBase): ) account_id: Mapped[str] = mapped_column(StringUUID, nullable=False) first_workspace_id: Mapped[str | None] = mapped_column(StringUUID, nullable=True, default=None) - skipped: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("false"), default=False) + skipped: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.false(), default=False) completed_task_ids: Mapped[list[str]] = mapped_column(AdjustedJSON, nullable=False, default_factory=list) manually_enabled_workspace_ids: Mapped[list[str]] = mapped_column( AdjustedJSON, diff --git a/api/models/snippet.py b/api/models/snippet.py index b6fbc4ed824..4a9320e51f6 100644 --- a/api/models/snippet.py +++ b/api/models/snippet.py @@ -47,7 +47,7 @@ class CustomizedSnippet(Base): workflow_id: Mapped[str | None] = mapped_column(StringUUID, nullable=True) # State flags - is_published: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("false")) + is_published: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.false()) version: Mapped[int] = mapped_column(sa.Integer, nullable=False, server_default=sa.text("1")) use_count: Mapped[int] = mapped_column(sa.Integer, nullable=False, server_default=sa.text("0")) diff --git a/api/models/source.py b/api/models/source.py index 8fce7df205d..c8cc0eb6508 100644 --- a/api/models/source.py +++ b/api/models/source.py @@ -36,7 +36,7 @@ class DataSourceOauthBinding(TypeBase): onupdate=func.current_timestamp(), init=False, ) - disabled: Mapped[bool] = mapped_column(sa.Boolean, nullable=True, server_default=sa.text("false"), default=False) + disabled: Mapped[bool] = mapped_column(sa.Boolean, nullable=True, server_default=sa.false(), default=False) class DataSourceApiKeyAuthBindingDict(TypedDict): @@ -75,7 +75,7 @@ class DataSourceApiKeyAuthBinding(TypeBase): onupdate=func.current_timestamp(), init=False, ) - disabled: Mapped[bool] = mapped_column(sa.Boolean, nullable=True, server_default=sa.text("false"), default=False) + disabled: Mapped[bool] = mapped_column(sa.Boolean, nullable=True, server_default=sa.false(), default=False) def to_dict(self) -> DataSourceApiKeyAuthBindingDict: result: DataSourceApiKeyAuthBindingDict = { diff --git a/api/models/tools.py b/api/models/tools.py index c8468925aab..67b4929da1b 100644 --- a/api/models/tools.py +++ b/api/models/tools.py @@ -62,7 +62,7 @@ class ToolOAuthTenantClient(TypeBase): tenant_id: Mapped[str] = mapped_column(StringUUID, nullable=False) plugin_id: Mapped[str] = mapped_column(String(255), nullable=False) provider: Mapped[str] = mapped_column(String(255), nullable=False) - enabled: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("true"), init=False) + enabled: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.true(), init=False) # oauth params of the tool provider encrypted_oauth_params: Mapped[str] = mapped_column(LongText, nullable=False, init=False) @@ -109,7 +109,7 @@ class BuiltinToolProvider(TypeBase): onupdate=func.current_timestamp(), init=False, ) - is_default: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("false"), default=False) + is_default: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.false(), default=False) # credential type, e.g., "api-key", "oauth2" credential_type: Mapped[CredentialType] = mapped_column( EnumText(CredentialType, length=32), diff --git a/api/models/trigger.py b/api/models/trigger.py index fe61a479495..4da9314d7df 100644 --- a/api/models/trigger.py +++ b/api/models/trigger.py @@ -199,7 +199,7 @@ class TriggerOAuthTenantClient(TypeBase): tenant_id: Mapped[str] = mapped_column(StringUUID, nullable=False) plugin_id: Mapped[str] = mapped_column(String(255), nullable=False) provider: Mapped[str] = mapped_column(String(255), nullable=False) - enabled: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.text("true"), default=True) + enabled: Mapped[bool] = mapped_column(sa.Boolean, nullable=False, server_default=sa.true(), default=True) # oauth params of the trigger provider encrypted_oauth_params: Mapped[str] = mapped_column(LongText, nullable=False, default="{}") created_at: Mapped[datetime] = mapped_column( diff --git a/api/openapi/markdown/console-openapi.md b/api/openapi/markdown/console-openapi.md index 60315c22924..4eb329385f5 100644 --- a/api/openapi/markdown/console-openapi.md +++ b/api/openapi/markdown/console-openapi.md @@ -6570,7 +6570,7 @@ Check if dataset is in use | Name | Located in | Description | Required | Schema | | ---- | ---------- | ----------- | -------- | ------ | -| language | query | Language code for recommended app localization | No | string | +| language | query | Language code for recommended app localization | No | string,
**Default:** en-US | #### Responses @@ -6583,7 +6583,7 @@ Check if dataset is in use | Name | Located in | Description | Required | Schema | | ---- | ---------- | ----------- | -------- | ------ | -| language | query | Language code for recommended app localization | No | string | +| language | query | Language code for recommended app localization | No | string,
**Default:** en-US | #### Responses @@ -9307,7 +9307,13 @@ Claim a workspace-staged upload. Multipart file bodies remain accepted as a lega | 302 | Redirect to OAuth callback page | ### [GET] /notification -Return the active in-product notification for the current user in their interface language (falls back to English if unavailable). The notification is NOT marked as seen here; call POST /notification/dismiss when the user explicitly closes the modal. +Return the active in-product notification for the current user in the requested language (defaults to English when omitted). Unavailable translations fall back to English, then the first available content. The notification is NOT marked as seen here; call POST /notification/dismiss when the user explicitly closes the modal. + +#### Parameters + +| Name | Located in | Description | Required | Schema | +| ---- | ---------- | ----------- | -------- | ------ | +| language | query | Notification language | No | string,
**Default:** en-US | #### Responses @@ -9377,7 +9383,7 @@ Handle OAuth callback and complete login process | Code | Description | Schema | | ---- | ----------- | ------ | | 302 | Redirect to console with access token | **application/json**: [RedirectResponse](#redirectresponse)
| -| 400 | OAuth process failed | | +| 400 | OAuth process failed | **application/json**: [OAuthErrorResponse](#oautherrorresponse)
| ### [GET] /oauth/data-source/binding/{provider} Bind OAuth data source with authorization code @@ -9466,7 +9472,7 @@ Initiate OAuth login process | Code | Description | Schema | | ---- | ----------- | ------ | | 302 | Redirect to OAuth authorization URL | **application/json**: [RedirectResponse](#redirectresponse)
| -| 400 | Invalid provider | | +| 400 | Invalid provider | **application/json**: [OAuthErrorResponse](#oautherrorresponse)
| ### [GET] /oauth/plugin/{provider_id}/datasource/callback #### Parameters @@ -24403,6 +24409,12 @@ Coarse node-level status used by Inspector to pick a banner. | title | string | | Yes | | title_pic_url | string | | Yes | +#### NotificationQuery + +| Name | Type | Description | Required | +| ---- | ---- | ----------- | -------- | +| language | string,
**Default:** en-US | Notification language | No | + #### NotificationResponse | Name | Type | Description | Required | @@ -24514,6 +24526,12 @@ Coarse node-level status used by Inspector to pick a banner. | ---- | ---- | ----------- | -------- | | result | string | Operation result | Yes | +#### OAuthErrorResponse + +| Name | Type | Description | Required | +| ---- | ---- | ----------- | -------- | +| error | string | OAuth error message | Yes | + #### OAuthLoginQuery | Name | Type | Description | Required | @@ -26046,7 +26064,7 @@ Model class for provider quota configuration. | Name | Type | Description | Required | | ---- | ---- | ----------- | -------- | -| language | string | Language code for recommended app localization | No | +| language | string,
**Default:** en-US | Language code for recommended app localization | No | #### RedirectResponse diff --git a/api/repositories/account_integration_repository.py b/api/repositories/account_integration_repository.py index 949d6a2e408..5b1abd39419 100644 --- a/api/repositories/account_integration_repository.py +++ b/api/repositories/account_integration_repository.py @@ -14,6 +14,15 @@ class SQLAlchemyAccountIntegrationRepository(AccountIntegrationRepository): def __init__(self, session_factory: sessionmaker[Session]) -> None: self._session_factory = session_factory + @override + def find_account_id(self, *, provider: str, open_id: str) -> str | None: + with self._session_factory() as session: + return session.scalar( + select(AccountIntegrate.account_id) + .where(AccountIntegrate.provider == provider, AccountIntegrate.open_id == open_id) + .limit(1) + ) + @override def list_for_account(self, account_id: str) -> list[AccountIntegrationSnapshot]: with self._session_factory() as session: @@ -23,3 +32,27 @@ class SQLAlchemyAccountIntegrationRepository(AccountIntegrationRepository): ) ).all() return [AccountIntegrationSnapshot(provider=row.provider, created_at=row.created_at) for row in rows] + + @override + def link(self, account_id: str, *, provider: str, open_id: str) -> None: + with self._session_factory.begin() as session: + integration = session.scalar( + select(AccountIntegrate) + .where( + AccountIntegrate.account_id == account_id, + AccountIntegrate.provider == provider, + ) + .limit(1) + ) + if integration is None: + session.add( + AccountIntegrate( + account_id=account_id, + provider=provider, + open_id=open_id, + encrypted_token="", + ) + ) + return + integration.open_id = open_id + integration.encrypted_token = "" diff --git a/api/repositories/account_oauth_repository.py b/api/repositories/account_oauth_repository.py new file mode 100644 index 00000000000..3f43cfeb54e --- /dev/null +++ b/api/repositories/account_oauth_repository.py @@ -0,0 +1,128 @@ +"""Persistence-backed gateways for Console account OAuth sign-in.""" + +from typing import override + +from sqlalchemy.orm import Session, sessionmaker + +from libs.datetime_utils import naive_utc_now +from models.account import Account, AccountStatus +from services.account_errors import ( + AccountEmailDomainSuspendedError, + OAuthAccountNotFoundError, + OAuthRegistrationError, + OAuthSeatsLimitExceededError, + OAuthWorkspaceCreationNotAllowedError, +) +from services.account_oauth_service import ( + OAuthAccountRegistrationGateway, + OAuthInvitationGateway, + OAuthSessionGateway, + OAuthWorkspaceGateway, +) +from services.account_service import AccountService, RegisterService, TenantService +from services.enterprise.enterprise_service import try_join_default_workspace +from services.entities.account_entities import AccountSessionTokens +from services.entities.account_oauth_entities import ( + OAuthAccountRegistration, + OAuthInvitation, +) +from services.errors.account import AccountRegisterError, EmailDomainSuspendedError, SeatsLimitExceededError +from services.errors.workspace import WorkSpaceNotAllowedCreateError, WorkspacesLimitExceededError + + +class RegisterServiceOAuthInvitationGateway(OAuthInvitationGateway): + def __init__(self, *, session_factory: sessionmaker[Session]) -> None: + self._session_factory = session_factory + + @override + def resolve(self, invite_token: str) -> OAuthInvitation | None: + with self._session_factory() as session: + invitation = RegisterService.get_invitation_if_token_valid( + None, + None, + invite_token, + session=session, + ) + if invitation is None: + return None + account = invitation["account"] + return OAuthInvitation( + account_id=account.id, + account_email=account.email, + account_status=account.status.value, + ) + + +class AccountServiceOAuthAccountRegistrationGateway(OAuthAccountRegistrationGateway): + def __init__(self, *, session_factory: sessionmaker[Session]) -> None: + self._session_factory = session_factory + + @override + def register(self, registration: OAuthAccountRegistration) -> str: + with self._session_factory() as session: + try: + account = AccountService.create_account( + email=registration.email, + name=registration.name, + interface_language=registration.language, + password=None, + timezone=registration.timezone, + ip_address=registration.ip_address, + check_normalized_email=True, + session=session, + ) + account.status = AccountStatus.ACTIVE + account.initialized_at = naive_utc_now() + session.commit() + except EmailDomainSuspendedError as exc: + raise AccountEmailDomainSuspendedError from exc + except SeatsLimitExceededError as exc: + raise OAuthSeatsLimitExceededError from exc + except AccountRegisterError as exc: + raise OAuthRegistrationError(exc.description) from exc + except Exception as exc: + session.rollback() + raise OAuthRegistrationError(f"Registration failed: {exc}") from exc + return account.id + + +class AccountServiceOAuthWorkspaceGateway(OAuthWorkspaceGateway): + """Adapt account workspace operations to the OAuth application port.""" + + def __init__(self, *, session_factory: sessionmaker[Session]) -> None: + self._session_factory = session_factory + + @override + def create_owner_workspace(self, account_id: str) -> None: + with self._session_factory() as session: + account = session.get(Account, account_id) + if account is None: + raise OAuthAccountNotFoundError + try: + TenantService.create_owner_tenant(account, session=session) + except (WorkSpaceNotAllowedCreateError, WorkspacesLimitExceededError) as exc: + raise OAuthWorkspaceCreationNotAllowedError from exc + + @override + def try_join_default_workspace(self, account_id: str) -> None: + try_join_default_workspace(account_id) + + +class AccountServiceOAuthSessionGateway(OAuthSessionGateway): + """Adapt Console session issuance to the OAuth application port.""" + + def __init__(self, *, session_factory: sessionmaker[Session]) -> None: + self._session_factory = session_factory + + @override + def login(self, account_id: str, *, ip_address: str) -> AccountSessionTokens: + with self._session_factory() as session: + account = session.get(Account, account_id) + if account is None: + raise OAuthAccountNotFoundError + token_pair = AccountService.login(account=account, session=session, ip_address=ip_address) + return AccountSessionTokens( + access_token=token_pair.access_token, + refresh_token=token_pair.refresh_token, + csrf_token=token_pair.csrf_token, + ) diff --git a/api/repositories/account_repository.py b/api/repositories/account_repository.py index 1f1c988fb83..65c352384d6 100644 --- a/api/repositories/account_repository.py +++ b/api/repositories/account_repository.py @@ -1,5 +1,6 @@ """SQLAlchemy implementation of the account persistence port.""" +from datetime import datetime from typing import override from sqlalchemy import case, delete, select @@ -45,6 +46,15 @@ class SQLAlchemyAccountRepository(AccountRepository, ConsoleAuthAccountRepositor account = session.execute(select(Account).where(Account.email == email.lower())).scalar_one_or_none() return self._to_snapshot(account) if account is not None else None + @override + def activate_pending(self, account_id: str, *, initialized_at: datetime) -> None: + with self._session_factory.begin() as session: + account = session.get(Account, account_id) + if account is None or account.status != AccountStatus.PENDING: + return + account.status = AccountStatus.ACTIVE + account.initialized_at = initialized_at + @override def get_credentials(self, account_id: str) -> AccountCredentials | None: with self._session_factory() as session: diff --git a/api/repositories/api_workflow_run_repository.py b/api/repositories/api_workflow_run_repository.py index b3225794db9..f08782153b0 100644 --- a/api/repositories/api_workflow_run_repository.py +++ b/api/repositories/api_workflow_run_repository.py @@ -41,7 +41,6 @@ from typing import Protocol, TypedDict from sqlalchemy.orm import Session -from core.repositories.factory import WorkflowExecutionRepository from core.workflow.nodes.human_input.pause_reason import PauseReason as DifyPauseReason from graphon.entities.pause_reason import PauseReason as GraphonPauseReason from graphon.enums import WorkflowType @@ -82,7 +81,7 @@ class WorkflowRunCleanupRef: created_at: datetime -class APIWorkflowRunRepository(WorkflowExecutionRepository, Protocol): +class APIWorkflowRunRepository(Protocol): """ Protocol for service-layer WorkflowRun repository operations. diff --git a/api/repositories/app_statistic_query_repository.py b/api/repositories/app_statistic_query_repository.py new file mode 100644 index 00000000000..d20502c4bf2 --- /dev/null +++ b/api/repositories/app_statistic_query_repository.py @@ -0,0 +1,401 @@ +"""Database read model for app monitoring statistics.""" + +from datetime import datetime +from decimal import Decimal +from typing import override + +import sqlalchemy as sa +from sqlalchemy.engine import RowMapping +from sqlalchemy.orm import Session, sessionmaker + +from core.app.entities.app_invoke_entities import InvokeFrom +from libs.helper import convert_datetime_to_date +from services.app_statistic_query import ( + AppStatisticQuery, + AverageResponseTimeStatisticRecord, + AverageSessionInteractionStatisticRecord, + DailyConversationStatisticRecord, + DailyMessageStatisticRecord, + DailyTerminalStatisticRecord, + DailyTokenCostStatisticRecord, + TokensPerSecondStatisticRecord, + UserSatisfactionRateStatisticRecord, +) + + +def _append_time_range( + sql_query: str, + parameters: dict[str, object], + *, + column: str, + start_date: datetime | None, + end_date: datetime | None, +) -> str: + if start_date is not None: + sql_query += f" AND {column} >= :start_date" + parameters["start_date"] = start_date + + if end_date is not None: + sql_query += f" AND {column} < :end_date" + parameters["end_date"] = end_date + + return sql_query + + +class AppStatisticQueryRepository(AppStatisticQuery): + def __init__(self, *, session_factory: sessionmaker[Session]) -> None: + self._session_factory = session_factory + + def _execute(self, sql_query: str, parameters: dict[str, object]) -> tuple[RowMapping, ...]: + with self._session_factory() as session: + return tuple(session.execute(sa.text(sql_query), parameters).mappings()) + + @staticmethod + def _parameters( + *, + app_id: str, + timezone: str, + ) -> dict[str, object]: + return { + "tz": timezone, + "app_id": app_id, + "excluded_invoke_from": InvokeFrom.DEBUGGER, + } + + @override + def get_daily_messages( + self, + *, + app_id: str, + start_date: datetime | None, + end_date: datetime | None, + timezone: str, + ) -> tuple[DailyMessageStatisticRecord, ...]: + converted_created_at = convert_datetime_to_date("created_at") + sql_query = f"""SELECT + {converted_created_at} AS date, + COUNT(*) AS message_count +FROM + messages +WHERE + app_id = :app_id + AND invoke_from != :excluded_invoke_from""" + parameters = self._parameters( + app_id=app_id, + timezone=timezone, + ) + sql_query = _append_time_range( + sql_query, + parameters, + column="created_at", + start_date=start_date, + end_date=end_date, + ) + sql_query += " GROUP BY date ORDER BY date" + + return tuple( + DailyMessageStatisticRecord(date=str(row["date"]), message_count=row["message_count"]) + for row in self._execute(sql_query, parameters) + ) + + @override + def get_daily_conversations( + self, + *, + app_id: str, + start_date: datetime | None, + end_date: datetime | None, + timezone: str, + ) -> tuple[DailyConversationStatisticRecord, ...]: + converted_created_at = convert_datetime_to_date("created_at") + sql_query = f"""SELECT + {converted_created_at} AS date, + COUNT(DISTINCT conversation_id) AS conversation_count +FROM + messages +WHERE + app_id = :app_id + AND invoke_from != :excluded_invoke_from""" + parameters = self._parameters( + app_id=app_id, + timezone=timezone, + ) + sql_query = _append_time_range( + sql_query, + parameters, + column="created_at", + start_date=start_date, + end_date=end_date, + ) + sql_query += " GROUP BY date ORDER BY date" + + return tuple( + DailyConversationStatisticRecord( + date=str(row["date"]), + conversation_count=row["conversation_count"], + ) + for row in self._execute(sql_query, parameters) + ) + + @override + def get_daily_terminals( + self, + *, + app_id: str, + start_date: datetime | None, + end_date: datetime | None, + timezone: str, + ) -> tuple[DailyTerminalStatisticRecord, ...]: + converted_created_at = convert_datetime_to_date("created_at") + sql_query = f"""SELECT + {converted_created_at} AS date, + COUNT(DISTINCT messages.from_end_user_id) AS terminal_count +FROM + messages +WHERE + app_id = :app_id + AND invoke_from != :excluded_invoke_from""" + parameters = self._parameters( + app_id=app_id, + timezone=timezone, + ) + sql_query = _append_time_range( + sql_query, + parameters, + column="created_at", + start_date=start_date, + end_date=end_date, + ) + sql_query += " GROUP BY date ORDER BY date" + + return tuple( + DailyTerminalStatisticRecord(date=str(row["date"]), terminal_count=row["terminal_count"]) + for row in self._execute(sql_query, parameters) + ) + + @override + def get_daily_token_costs( + self, + *, + app_id: str, + start_date: datetime | None, + end_date: datetime | None, + timezone: str, + ) -> tuple[DailyTokenCostStatisticRecord, ...]: + converted_created_at = convert_datetime_to_date("created_at") + sql_query = f"""SELECT + {converted_created_at} AS date, + (SUM(messages.message_tokens) + SUM(messages.answer_tokens)) AS token_count, + SUM(total_price) AS total_price +FROM + messages +WHERE + app_id = :app_id + AND invoke_from != :excluded_invoke_from""" + parameters = self._parameters( + app_id=app_id, + timezone=timezone, + ) + sql_query = _append_time_range( + sql_query, + parameters, + column="created_at", + start_date=start_date, + end_date=end_date, + ) + sql_query += " GROUP BY date ORDER BY date" + + return tuple( + DailyTokenCostStatisticRecord( + date=str(row["date"]), + token_count=int(row["token_count"]) if row["token_count"] is not None else None, + total_price=row["total_price"], + currency="USD", + ) + for row in self._execute(sql_query, parameters) + ) + + @override + def get_average_session_interactions( + self, + *, + app_id: str, + start_date: datetime | None, + end_date: datetime | None, + timezone: str, + ) -> tuple[AverageSessionInteractionStatisticRecord, ...]: + converted_created_at = convert_datetime_to_date("c.created_at") + sql_query = f"""SELECT + {converted_created_at} AS date, + AVG(subquery.message_count) AS interactions +FROM + ( + SELECT + m.conversation_id, + COUNT(m.id) AS message_count + FROM + conversations c + JOIN + messages m + ON c.id = m.conversation_id + WHERE + c.app_id = :app_id + AND m.invoke_from != :excluded_invoke_from""" + parameters = self._parameters( + app_id=app_id, + timezone=timezone, + ) + sql_query = _append_time_range( + sql_query, + parameters, + column="c.created_at", + start_date=start_date, + end_date=end_date, + ) + sql_query += """ + GROUP BY m.conversation_id + ) subquery +LEFT JOIN + conversations c + ON c.id = subquery.conversation_id +GROUP BY + date +ORDER BY + date""" + + return tuple( + AverageSessionInteractionStatisticRecord( + date=str(row["date"]), + interactions=float(row["interactions"].quantize(Decimal("0.01"))), + ) + for row in self._execute(sql_query, parameters) + ) + + @override + def get_user_satisfaction_rates( + self, + *, + app_id: str, + start_date: datetime | None, + end_date: datetime | None, + timezone: str, + ) -> tuple[UserSatisfactionRateStatisticRecord, ...]: + converted_created_at = convert_datetime_to_date("m.created_at") + sql_query = f"""SELECT + {converted_created_at} AS date, + COUNT(m.id) AS message_count, + COUNT(mf.id) AS feedback_count +FROM + messages m +LEFT JOIN + message_feedbacks mf + ON mf.message_id=m.id AND mf.rating='like' +WHERE + m.app_id = :app_id + AND m.invoke_from != :excluded_invoke_from""" + parameters = self._parameters( + app_id=app_id, + timezone=timezone, + ) + sql_query = _append_time_range( + sql_query, + parameters, + column="m.created_at", + start_date=start_date, + end_date=end_date, + ) + sql_query += " GROUP BY date ORDER BY date" + + return tuple( + UserSatisfactionRateStatisticRecord( + date=str(row["date"]), + rate=float( + round( + row["feedback_count"] * 1000 / row["message_count"] if row["message_count"] > 0 else 0, + 2, + ) + ), + ) + for row in self._execute(sql_query, parameters) + ) + + @override + def get_average_response_times( + self, + *, + app_id: str, + start_date: datetime | None, + end_date: datetime | None, + timezone: str, + ) -> tuple[AverageResponseTimeStatisticRecord, ...]: + converted_created_at = convert_datetime_to_date("created_at") + sql_query = f"""SELECT + {converted_created_at} AS date, + AVG(provider_response_latency) AS latency +FROM + messages +WHERE + app_id = :app_id + AND invoke_from != :excluded_invoke_from""" + parameters = self._parameters( + app_id=app_id, + timezone=timezone, + ) + sql_query = _append_time_range( + sql_query, + parameters, + column="created_at", + start_date=start_date, + end_date=end_date, + ) + sql_query += " GROUP BY date ORDER BY date" + + return tuple( + AverageResponseTimeStatisticRecord( + date=str(row["date"]), + latency=round(row["latency"] * 1000, 4), + ) + for row in self._execute(sql_query, parameters) + ) + + @override + def get_tokens_per_second( + self, + *, + app_id: str, + start_date: datetime | None, + end_date: datetime | None, + timezone: str, + ) -> tuple[TokensPerSecondStatisticRecord, ...]: + converted_created_at = convert_datetime_to_date("created_at") + sql_query = f"""SELECT + {converted_created_at} AS date, + CASE + WHEN SUM(provider_response_latency) = 0 THEN 0 + ELSE (SUM(answer_tokens) / SUM(provider_response_latency)) + END as tokens_per_second +FROM + messages +WHERE + app_id = :app_id + AND invoke_from != :excluded_invoke_from""" + parameters = self._parameters( + app_id=app_id, + timezone=timezone, + ) + sql_query = _append_time_range( + sql_query, + parameters, + column="created_at", + start_date=start_date, + end_date=end_date, + ) + sql_query += " GROUP BY date ORDER BY date" + + return tuple( + TokensPerSecondStatisticRecord( + date=str(row["date"]), + tps=round(row["tokens_per_second"], 4), + ) + for row in self._execute(sql_query, parameters) + ) diff --git a/api/repositories/file_grant_repository.py b/api/repositories/file_grant_repository.py new file mode 100644 index 00000000000..98ff0fa8b16 --- /dev/null +++ b/api/repositories/file_grant_repository.py @@ -0,0 +1,213 @@ +from __future__ import annotations + +import os +from collections.abc import Sequence + +from sqlalchemy import select +from sqlalchemy.orm import Session, sessionmaker + +from models.enums import CreatorUserRole, EndUserType +from models.model import App, EndUser, UploadFile +from models.tools import ToolFile +from services.entities.file_grant_entities import ( + FileContentRecord, + FileGrantContext, + FileKind, + FileRef, + ResolvedFile, +) + + +class FileGrantRepository: + def __init__(self, *, session_factory: sessionmaker[Session]) -> None: + self._session_factory = session_factory + + def get_or_create_subject( + self, + *, + tenant_id: str, + app_id: str, + session_id: str, + external_user_id: str, + is_anonymous: bool, + ) -> str | None: + with self._session_factory() as session: + end_user = session.scalar( + self._subject_statement( + tenant_id=tenant_id, + app_id=app_id, + session_id=session_id, + require_app=True, + ).limit(1) + ) + if end_user is not None: + return end_user.id + + with self._session_factory.begin() as session: + app = session.scalar(select(App).where(App.id == app_id, App.tenant_id == tenant_id).with_for_update()) + if app is None: + return None + + end_user = session.scalar( + self._subject_statement( + tenant_id=tenant_id, + app_id=app_id, + session_id=session_id, + require_app=False, + ).limit(1) + ) + if end_user is None: + end_user = EndUser( + tenant_id=tenant_id, + app_id=app_id, + type=EndUserType.APP_DEPLOY, + is_anonymous=is_anonymous, + session_id=session_id, + external_user_id=external_user_id, + ) + session.add(end_user) + session.flush() + return end_user.id + + def subject_exists(self, context: FileGrantContext) -> bool: + with self._session_factory() as session: + return ( + session.scalar( + select(EndUser.id) + .where( + EndUser.id == context.end_user_id, + EndUser.tenant_id == context.tenant_id, + EndUser.app_id == context.app_id, + EndUser.type == EndUserType.APP_DEPLOY, + ) + .limit(1) + ) + is not None + ) + + def get_end_user(self, context: FileGrantContext) -> EndUser | None: + with self._session_factory(expire_on_commit=False) as session: + return session.scalar( + select(EndUser) + .where( + EndUser.id == context.end_user_id, + EndUser.tenant_id == context.tenant_id, + EndUser.app_id == context.app_id, + EndUser.type == EndUserType.APP_DEPLOY, + ) + .limit(1) + ) + + def resolve_owned_files( + self, + *, + context: FileGrantContext, + refs: Sequence[FileRef], + ) -> list[ResolvedFile | None]: + upload_ids = {ref.id for ref in refs if ref.kind == FileKind.UPLOAD} + tool_ids = {ref.id for ref in refs if ref.kind == FileKind.TOOL} + + with self._session_factory() as session: + uploads = self._load_uploads(session, context=context, file_ids=upload_ids) + tool_files = self._load_tool_files(session, context=context, file_ids=tool_ids) + + return [uploads.get(ref.id) if ref.kind == FileKind.UPLOAD else tool_files.get(ref.id) for ref in refs] + + def get_content_record(self, *, file_id: str, kind: FileKind) -> FileContentRecord | None: + with self._session_factory() as session: + match kind: + case FileKind.UPLOAD: + upload_file = session.scalar(select(UploadFile).where(UploadFile.id == file_id).limit(1)) + if upload_file is None: + return None + return FileContentRecord( + name=upload_file.name, + size=upload_file.size, + mime_type=upload_file.mime_type, + storage_key=upload_file.key, + ) + case FileKind.TOOL: + tool_file = session.scalar(select(ToolFile).where(ToolFile.id == file_id).limit(1)) + if tool_file is None: + return None + return FileContentRecord( + name=tool_file.name or "", + size=tool_file.size, + mime_type=tool_file.mimetype, + storage_key=tool_file.file_key, + ) + + @staticmethod + def _subject_statement(*, tenant_id: str, app_id: str, session_id: str, require_app: bool): + statement = select(EndUser) + if require_app: + statement = statement.join(App, App.id == EndUser.app_id) + predicates = [ + EndUser.tenant_id == tenant_id, + EndUser.app_id == app_id, + EndUser.session_id == session_id, + EndUser.type == EndUserType.APP_DEPLOY, + ] + if require_app: + predicates.append(App.tenant_id == tenant_id) + return statement.where(*predicates) + + @staticmethod + def _load_uploads( + session: Session, + *, + context: FileGrantContext, + file_ids: set[str], + ) -> dict[str, ResolvedFile]: + if not file_ids: + return {} + rows = session.scalars( + select(UploadFile).where( + UploadFile.id.in_(file_ids), + UploadFile.tenant_id == context.tenant_id, + UploadFile.created_by_role == CreatorUserRole.END_USER, + UploadFile.created_by == context.end_user_id, + ) + ).all() + return { + row.id: ResolvedFile( + id=row.id, + kind=FileKind.UPLOAD, + name=row.name, + size=row.size, + extension=row.extension, + mime_type=row.mime_type, + ) + for row in rows + } + + @staticmethod + def _load_tool_files( + session: Session, + *, + context: FileGrantContext, + file_ids: set[str], + ) -> dict[str, ResolvedFile]: + if not file_ids: + return {} + rows = session.scalars( + select(ToolFile).where( + ToolFile.id.in_(file_ids), + ToolFile.tenant_id == context.tenant_id, + ToolFile.user_id == context.end_user_id, + ) + ).all() + return { + row.id: ResolvedFile( + id=row.id, + kind=FileKind.TOOL, + name=row.name or "", + size=row.size, + extension=os.path.splitext(row.name or "")[1].lstrip(".").lower(), + mime_type=row.mimetype, + ) + for row in rows + } + + +__all__ = ["FileGrantRepository"] diff --git a/api/repositories/oauth_access_token_repository.py b/api/repositories/oauth_access_token_repository.py new file mode 100644 index 00000000000..87ba4704093 --- /dev/null +++ b/api/repositories/oauth_access_token_repository.py @@ -0,0 +1,89 @@ +"""SQLAlchemy persistence for account-scoped OAuth access sessions.""" + +from __future__ import annotations + +from datetime import datetime +from typing import override + +from sqlalchemy import func, select, update +from sqlalchemy.orm import Session, sessionmaker + +from models.oauth import OAuthAccessToken +from services.account_ports import AccountSessionRepository +from services.entities.account_access_entities import ( + AccountSessionRevocation, + AccountSessionSnapshot, +) + + +class SQLAlchemyOAuthAccessTokenRepository(AccountSessionRepository): + def __init__(self, *, session_factory: sessionmaker[Session]) -> None: + self._session_factory = session_factory + + @override + def list_active( + self, + *, + account_id: str, + active_at: datetime, + offset: int, + limit: int, + ) -> tuple[int, tuple[AccountSessionSnapshot, ...]]: + predicates = ( + OAuthAccessToken.account_id == account_id, + OAuthAccessToken.revoked_at.is_(None), + OAuthAccessToken.token_hash.is_not(None), + OAuthAccessToken.expires_at > active_at, + ) + with self._session_factory() as session: + total = session.scalar(select(func.count()).select_from(OAuthAccessToken).where(*predicates)) or 0 + rows = session.scalars( + select(OAuthAccessToken) + .where(*predicates) + .order_by(OAuthAccessToken.created_at.desc(), OAuthAccessToken.id.desc()) + .offset(offset) + .limit(limit) + ).all() + return int(total), tuple(self._to_snapshot(row) for row in rows) + + @override + def revoke( + self, + *, + account_id: str, + token_id: str, + revoked_at: datetime, + ) -> AccountSessionRevocation: + with self._session_factory.begin() as session: + row = session.execute( + select(OAuthAccessToken.account_id, OAuthAccessToken.token_hash) + .where(OAuthAccessToken.id == token_id) + .with_for_update() + ).one_or_none() + if row is None or row.account_id != account_id: + return AccountSessionRevocation(owned=False) + + token_hash = row.token_hash + if token_hash is not None: + session.execute( + update(OAuthAccessToken) + .where( + OAuthAccessToken.id == token_id, + OAuthAccessToken.account_id == account_id, + OAuthAccessToken.revoked_at.is_(None), + ) + .values(revoked_at=revoked_at, token_hash=None) + ) + return AccountSessionRevocation(owned=True, token_hash=token_hash) + + @staticmethod + def _to_snapshot(row: OAuthAccessToken) -> AccountSessionSnapshot: + return AccountSessionSnapshot( + id=str(row.id), + prefix=row.prefix, + client_id=row.client_id, + device_label=row.device_label, + created_at=row.created_at, + last_used_at=row.last_used_at, + expires_at=row.expires_at, + ) diff --git a/api/repositories/sqlalchemy_api_workflow_run_repository.py b/api/repositories/sqlalchemy_api_workflow_run_repository.py index bc0a494384d..aa858282df1 100644 --- a/api/repositories/sqlalchemy_api_workflow_run_repository.py +++ b/api/repositories/sqlalchemy_api_workflow_run_repository.py @@ -22,10 +22,10 @@ Implementation Notes: import json import logging import uuid -from collections.abc import Callable, Sequence +from collections.abc import Callable, Mapping, Sequence from datetime import datetime from decimal import Decimal -from typing import Any, cast, override +from typing import Any, NamedTuple, cast, override import sqlalchemy as sa from pydantic import ValidationError @@ -33,6 +33,7 @@ from sqlalchemy import and_, delete, func, null, or_, select, tuple_ from sqlalchemy.engine import CursorResult from sqlalchemy.orm import Session, selectinload, sessionmaker +from core.workflow.human_input_forms import load_form_tokens_by_form_id from core.workflow.nodes.human_input.entities import FormDefinition from core.workflow.nodes.human_input.pause_reason import ( HumanInputRequired, @@ -55,6 +56,7 @@ from libs.datetime_utils import naive_utc_now from libs.helper import convert_datetime_to_date from libs.infinite_scroll_pagination import InfiniteScrollPagination from libs.time_parser import get_time_threshold +from models import Message from models.enums import WorkflowRunTriggeredFrom from models.human_input import HumanInputForm, HumanInputFormRecipient from models.workflow import WorkflowAppLog, WorkflowArchiveLog, WorkflowPause, WorkflowPauseReason, WorkflowRun @@ -76,6 +78,18 @@ logger = logging.getLogger(__name__) _HITL_REASON_TYPES = frozenset({PauseReasonType.LEGACY_HUMAN_INPUT_REQUIRED, PauseReasonType.HITL_REQUIRED}) +class WorkflowRunMessageRef(NamedTuple): + message_id: str + conversation_id: str + + +class WorkflowRunPauseRecord(NamedTuple): + status: WorkflowExecutionStatus + paused_at: datetime | None + reasons: tuple[DifyPauseReason, ...] + form_tokens: Mapping[str, str] + + class _WorkflowRunError(Exception): pass @@ -184,6 +198,31 @@ class DifyAPISQLAlchemyWorkflowRunRepository(APIWorkflowRunRepository): """ self._session_maker = session_maker + def get_message_refs( + self, + *, + app_id: str, + workflow_run_ids: Sequence[str], + ) -> dict[str, WorkflowRunMessageRef]: + if not workflow_run_ids: + return {} + + stmt = select(Message.workflow_run_id, Message.id, Message.conversation_id).where( + Message.app_id == app_id, + Message.workflow_run_id.in_(workflow_run_ids), + ) + with self._session_maker() as session: + rows = session.execute(stmt).all() + + messages_by_run_id: dict[str, WorkflowRunMessageRef] = {} + for workflow_run_id, message_id, conversation_id in rows: + if workflow_run_id is not None: + messages_by_run_id.setdefault( + workflow_run_id, + WorkflowRunMessageRef(message_id=message_id, conversation_id=conversation_id), + ) + return messages_by_run_id + @override def get_paginated_workflow_runs( self, @@ -1152,6 +1191,48 @@ class DifyAPISQLAlchemyWorkflowRunRepository(APIWorkflowRunRepository): pause_reasons=pause_reasons, ) + def get_pause_record( + self, + *, + workspace_id: str, + workflow_run_id: str, + ) -> WorkflowRunPauseRecord | None: + stmt = ( + select(WorkflowRun) + .options(selectinload(WorkflowRun.pause)) + .where( + WorkflowRun.tenant_id == workspace_id, + WorkflowRun.id == workflow_run_id, + ) + ) + with self._session_maker() as session: + workflow_run = session.scalar(stmt) + if workflow_run is None: + return None + if workflow_run.status != WorkflowExecutionStatus.PAUSED: + return WorkflowRunPauseRecord( + status=workflow_run.status, + paused_at=None, + reasons=(), + form_tokens={}, + ) + + pause_model = workflow_run.pause + if pause_model is None: + reasons: tuple[DifyPauseReason, ...] = () + else: + reason_models = self._get_reasons_by_pause_id(session, pause_model.id) + reasons = tuple(self._hydrate_pause_reasons(session, reason_models)) + form_ids = [reason.form_id for reason in reasons if isinstance(reason, HumanInputRequired)] + form_tokens = load_form_tokens_by_form_id(form_ids, session=session) + + return WorkflowRunPauseRecord( + status=workflow_run.status, + paused_at=pause_model.created_at if pause_model is not None else None, + reasons=reasons, + form_tokens=form_tokens, + ) + @override def resume_workflow_pause( self, diff --git a/api/repositories/workflow_app_log_query_repository.py b/api/repositories/workflow_app_log_query_repository.py new file mode 100644 index 00000000000..440337ede47 --- /dev/null +++ b/api/repositories/workflow_app_log_query_repository.py @@ -0,0 +1,261 @@ +import uuid +from datetime import datetime +from enum import Enum +from typing import override + +from sqlalchemy import and_, func, literal, or_, select +from sqlalchemy.engine import RowMapping +from sqlalchemy.orm import Session, aliased, sessionmaker + +from graphon.enums import WorkflowExecutionStatus +from libs.helper import escape_like_pattern +from models import Account, EndUser, TenantAccountJoin, WorkflowAppLog, WorkflowRun +from models.enums import CreatorUserRole +from models.trigger import WorkflowTriggerLog +from services.workflow_app_log_query_service import ( + WorkflowAppLogAccount, + WorkflowAppLogEndUser, + WorkflowAppLogItem, + WorkflowAppLogPage, + WorkflowAppLogQuery, + WorkflowAppLogRunSummary, +) + + +class WorkflowAppLogQueryRepository(WorkflowAppLogQuery): + def __init__(self, *, session_factory: sessionmaker[Session]) -> None: + self._session_factory = session_factory + + @override + def get_paginated( + self, + *, + tenant_id: str, + app_id: str, + keyword: str | None = None, + status: WorkflowExecutionStatus | None = None, + created_at_before: datetime | None = None, + created_at_after: datetime | None = None, + page: int = 1, + limit: int = 20, + detail: bool = False, + created_by_end_user_session_id: str | None = None, + created_by_account: str | None = None, + ) -> WorkflowAppLogPage: + with self._session_factory() as session: + workflow_run = aliased(WorkflowRun) + stmt = ( + select( + WorkflowAppLog.id.label("log_id"), + WorkflowAppLog.created_from.label("log_created_from"), + WorkflowAppLog.created_by_role.label("log_created_by_role"), + WorkflowAppLog.created_at.label("log_created_at"), + workflow_run.id.label("run_id"), + workflow_run.version.label("run_version"), + workflow_run.status.label("run_status"), + workflow_run.triggered_from.label("run_triggered_from"), + workflow_run.error.label("run_error"), + workflow_run.elapsed_time.label("run_elapsed_time"), + workflow_run.total_tokens.label("run_total_tokens"), + workflow_run.total_steps.label("run_total_steps"), + workflow_run.created_at.label("run_created_at"), + workflow_run.finished_at.label("run_finished_at"), + workflow_run.exceptions_count.label("run_exceptions_count"), + ) + .select_from(WorkflowAppLog) + .outerjoin( + workflow_run, + and_( + workflow_run.id == WorkflowAppLog.workflow_run_id, + workflow_run.tenant_id == WorkflowAppLog.tenant_id, + workflow_run.app_id == WorkflowAppLog.app_id, + ), + ) + .where( + WorkflowAppLog.tenant_id == tenant_id, + WorkflowAppLog.app_id == app_id, + ) + ) + + if detail: + workflow_trigger_log = aliased(WorkflowTriggerLog) + stmt = stmt.outerjoin( + workflow_trigger_log, + and_( + workflow_trigger_log.tenant_id == tenant_id, + workflow_trigger_log.app_id == app_id, + workflow_trigger_log.workflow_run_id == WorkflowAppLog.workflow_run_id, + ), + ).add_columns(workflow_trigger_log.trigger_metadata.label("trigger_metadata")) + else: + stmt = stmt.add_columns(literal(None).label("trigger_metadata")) + + if keyword: + escaped_keyword = escape_like_pattern(keyword[:30]) + keyword_like_value = f"%{escaped_keyword}%" + run_creator = aliased(EndUser) + keyword_conditions = ( + workflow_run.inputs.ilike(keyword_like_value, escape="\\"), + workflow_run.outputs.ilike(keyword_like_value, escape="\\"), + select(1) + .select_from(run_creator) + .where( + workflow_run.created_by_role == CreatorUserRole.END_USER, + run_creator.id == workflow_run.created_by, + run_creator.session_id.ilike(keyword_like_value, escape="\\"), + ) + .exists(), + ) + keyword_uuid = self._safe_parse_uuid(keyword) + if keyword_uuid is not None: + stmt = stmt.where(or_(*keyword_conditions, workflow_run.id == keyword_uuid)) + else: + stmt = stmt.where(or_(*keyword_conditions)) + + if status is not None: + stmt = stmt.where(workflow_run.status == status) + if created_at_before is not None: + stmt = stmt.where(WorkflowAppLog.created_at <= created_at_before) + if created_at_after is not None: + stmt = stmt.where(WorkflowAppLog.created_at >= created_at_after) + + if created_by_end_user_session_id: + log_creator = aliased(EndUser) + stmt = stmt.where( + WorkflowAppLog.created_by_role == CreatorUserRole.END_USER, + select(1) + .select_from(log_creator) + .where( + log_creator.id == WorkflowAppLog.created_by, + log_creator.session_id == created_by_end_user_session_id, + ) + .exists(), + ) + if created_by_account: + account_id = session.scalar( + select(Account.id) + .join(TenantAccountJoin, TenantAccountJoin.account_id == Account.id) + .where( + Account.email == created_by_account, + TenantAccountJoin.tenant_id == tenant_id, + ) + ) + if account_id is None: + raise ValueError(f"Account not found: {created_by_account}") + + stmt = stmt.where( + WorkflowAppLog.created_by_role == CreatorUserRole.ACCOUNT, + WorkflowAppLog.created_by == account_id, + ) + + total = session.scalar(select(func.count()).select_from(stmt.subquery())) or 0 + + actor_account = aliased(Account) + actor_end_user = aliased(EndUser) + paginated_stmt = ( + stmt.outerjoin( + actor_account, + and_( + WorkflowAppLog.created_by_role == CreatorUserRole.ACCOUNT, + actor_account.id == WorkflowAppLog.created_by, + ), + ) + .outerjoin( + actor_end_user, + and_( + WorkflowAppLog.created_by_role == CreatorUserRole.END_USER, + actor_end_user.id == WorkflowAppLog.created_by, + ), + ) + .add_columns( + actor_account.id.label("account_id"), + actor_account.name.label("account_name"), + actor_account.email.label("account_email"), + actor_end_user.id.label("end_user_id"), + actor_end_user.type.label("end_user_type"), + actor_end_user.session_id.label("end_user_session_id"), + ) + .order_by(WorkflowAppLog.created_at.desc()) + .offset((page - 1) * limit) + .limit(limit) + ) + rows = session.execute(paginated_stmt).mappings().all() + + return WorkflowAppLogPage( + page=page, + limit=limit, + total=total, + has_more=total > page * limit, + data=tuple(self._to_item(row=row, detail=detail) for row in rows), + ) + + @staticmethod + def _to_item(*, row: RowMapping, detail: bool) -> WorkflowAppLogItem: + run_id = row["run_id"] + workflow_run = ( + WorkflowAppLogRunSummary( + id=run_id, + version=row["run_version"], + status=WorkflowAppLogQueryRepository._enum_value(row["run_status"]), + triggered_from=WorkflowAppLogQueryRepository._enum_value(row["run_triggered_from"]), + error=row["run_error"], + elapsed_time=row["run_elapsed_time"], + total_tokens=row["run_total_tokens"], + total_steps=row["run_total_steps"], + created_at=row["run_created_at"], + finished_at=row["run_finished_at"], + exceptions_count=row["run_exceptions_count"], + ) + if run_id is not None + else None + ) + + account_id = row["account_id"] + account = ( + WorkflowAppLogAccount( + id=account_id, + name=row["account_name"], + email=row["account_email"], + ) + if account_id is not None + else None + ) + end_user_id = row["end_user_id"] + end_user = ( + WorkflowAppLogEndUser( + id=end_user_id, + type=WorkflowAppLogQueryRepository._enum_value(row["end_user_type"]), + is_anonymous=False, + session_id=row["end_user_session_id"], + ) + if end_user_id is not None + else None + ) + + return WorkflowAppLogItem( + id=row["log_id"], + workflow_run=workflow_run, + details={"trigger_metadata": row["trigger_metadata"]} if detail else None, + created_from=WorkflowAppLogQueryRepository._enum_value(row["log_created_from"]), + created_by_role=WorkflowAppLogQueryRepository._enum_value(row["log_created_by_role"]), + created_by_account=account, + created_by_end_user=end_user, + created_at=row["log_created_at"], + ) + + @staticmethod + def _enum_value(value: Enum | str | None) -> str: + if value is None: + raise ValueError("Required enum value is missing") + if isinstance(value, str): + return value + return str(value.value) + + @staticmethod + def _safe_parse_uuid(value: str) -> uuid.UUID | None: + if len(value) < 32: + return None + try: + return uuid.UUID(value) + except ValueError: + return None diff --git a/api/repositories/workspace_query_repository.py b/api/repositories/workspace_query_repository.py index 9af32673fa5..8e83d158c47 100644 --- a/api/repositories/workspace_query_repository.py +++ b/api/repositories/workspace_query_repository.py @@ -7,11 +7,17 @@ from sqlalchemy.orm import Session, sessionmaker from models.account import Tenant, TenantAccountJoin, TenantStatus from services.account_login_service import ConsoleAuthWorkspaceQuery -from services.account_ports import AccountWorkspaceMembershipQuery +from services.account_ports import AccountWorkspaceMembershipQuery, AccountWorkspaceSnapshotQuery +from services.entities.account_access_entities import AccountWorkspaceSnapshot from services.workspace_query_service import WorkspaceQuery, WorkspaceRecord -class WorkspaceQueryRepository(WorkspaceQuery, AccountWorkspaceMembershipQuery, ConsoleAuthWorkspaceQuery): +class WorkspaceQueryRepository( + WorkspaceQuery, + AccountWorkspaceMembershipQuery, + AccountWorkspaceSnapshotQuery, + ConsoleAuthWorkspaceQuery, +): def __init__(self, session_factory: sessionmaker[Session]) -> None: self._session_factory = session_factory @@ -52,6 +58,35 @@ class WorkspaceQueryRepository(WorkspaceQuery, AccountWorkspaceMembershipQuery, with self._session_factory() as session: return tuple(session.scalars(stmt).all()) + @override + def list_account_access_workspaces(self, account_id: str) -> tuple[AccountWorkspaceSnapshot, ...]: + """List every membership for the OpenAPI account identity response. + + Unlike the Console workspace picker, the identity response preserves + its existing behavior of including archived memberships. + """ + stmt = ( + select( + Tenant.id, + Tenant.name, + TenantAccountJoin.role, + TenantAccountJoin.current, + ) + .join(TenantAccountJoin, TenantAccountJoin.tenant_id == Tenant.id) + .where(TenantAccountJoin.account_id == account_id) + .order_by(Tenant.created_at.asc(), Tenant.id.asc()) + ) + with self._session_factory() as session: + return tuple( + AccountWorkspaceSnapshot( + id=workspace_id, + name=name, + role=role.value, + current=current, + ) + for workspace_id, name, role, current in session.execute(stmt).all() + ) + @override def has_active_for_account(self, account_id: str) -> bool: stmt = ( @@ -65,3 +100,7 @@ class WorkspaceQueryRepository(WorkspaceQuery, AccountWorkspaceMembershipQuery, ) with self._session_factory() as session: return session.scalar(stmt) is not None + + @override + def has_active_membership(self, account_id: str) -> bool: + return self.has_active_for_account(account_id) diff --git a/api/services/account_access_service.py b/api/services/account_access_service.py new file mode 100644 index 00000000000..9fb3a768cb3 --- /dev/null +++ b/api/services/account_access_service.py @@ -0,0 +1,77 @@ +"""Application service for account identity and access-session use cases.""" + +from __future__ import annotations + +from collections.abc import Callable +from datetime import datetime + +from machinery.context import AccountRequestContext +from services.account_errors import AccountNotFoundError, AccountSessionNotFoundError +from services.account_ports import ( + AccountSessionRepository, + AccountSnapshotQuery, + AccountTokenCacheInvalidator, + AccountWorkspaceSnapshotQuery, +) +from services.entities.account_access_entities import AccountAccessSnapshot, AccountSessionPage + + +class AccountAccessService: + def __init__( + self, + *, + accounts: AccountSnapshotQuery, + workspaces: AccountWorkspaceSnapshotQuery, + sessions: AccountSessionRepository, + invalidate_token_cache: AccountTokenCacheInvalidator, + now: Callable[[], datetime], + ) -> None: + self._accounts = accounts + self._workspaces = workspaces + self._sessions = sessions + self._invalidate_token_cache = invalidate_token_cache + self._now = now + + def get(self, context: AccountRequestContext) -> AccountAccessSnapshot: + account = self._accounts.get(context.account_id) + if account is None: + raise AccountNotFoundError + + workspaces = tuple(self._workspaces.list_account_access_workspaces(context.account_id)) + default_workspace_id = next((workspace.id for workspace in workspaces if workspace.current), None) + if default_workspace_id is None and workspaces: + default_workspace_id = workspaces[0].id + + return AccountAccessSnapshot( + account=account, + workspaces=workspaces, + default_workspace_id=default_workspace_id, + ) + + def list_sessions(self, context: AccountRequestContext, *, page: int, limit: int) -> AccountSessionPage: + total, sessions = self._sessions.list_active( + account_id=context.account_id, + active_at=self._now(), + offset=(page - 1) * limit, + limit=limit, + ) + return AccountSessionPage(page=page, limit=limit, total=total, items=tuple(sessions)) + + def revoke_current_session(self, context: AccountRequestContext) -> None: + if context.access_token_id is None: + raise RuntimeError("OpenAPI account admission did not resolve an access token") + self._revoke(context, token_id=context.access_token_id, require_owned=False) + + def revoke_session(self, context: AccountRequestContext, *, token_id: str) -> None: + self._revoke(context, token_id=token_id, require_owned=True) + + def _revoke(self, context: AccountRequestContext, *, token_id: str, require_owned: bool) -> None: + revocation = self._sessions.revoke( + account_id=context.account_id, + token_id=token_id, + revoked_at=self._now(), + ) + if require_owned and not revocation.owned: + raise AccountSessionNotFoundError + if revocation.token_hash is not None: + self._invalidate_token_cache(revocation.token_hash) diff --git a/api/services/account_errors.py b/api/services/account_errors.py index b64b245ddac..9ff9d5ec73a 100644 --- a/api/services/account_errors.py +++ b/api/services/account_errors.py @@ -9,6 +9,10 @@ class AccountNotFoundError(AccountApplicationError): """The admitted account no longer exists.""" +class AccountSessionNotFoundError(AccountApplicationError): + """The requested access session is not owned by the admitted account.""" + + class CurrentAccountPasswordIncorrectError(AccountApplicationError): """The supplied current password does not match the account credential.""" @@ -161,6 +165,62 @@ class EducationDiscountPausedError(AccountApplicationError): """Education discount activation is temporarily paused.""" +class InvalidOAuthProviderError(AccountApplicationError): + """The requested Console OAuth provider is unavailable.""" + + +class OAuthProviderRequestError(AccountApplicationError): + """The remote OAuth provider could not complete the request.""" + + +class OAuthProviderAuthorizationError(AccountApplicationError): + """The remote OAuth provider rejected the authorization exchange.""" + + def __init__(self, description: str) -> None: + super().__init__(description) + self.description = description + + +class OAuthIdentityLockUnavailableError(AccountApplicationError): + """The OAuth account claim could not be acquired or its lease was lost.""" + + +class InvalidOAuthInvitationError(AccountApplicationError): + """The OAuth callback references an invitation that can no longer be resolved.""" + + +class OAuthInvitationAccountMismatchError(AccountApplicationError): + """The OAuth identity does not own the account referenced by the invitation.""" + + def __init__(self, invite_token: str) -> None: + super().__init__(invite_token) + self.invite_token = invite_token + + +class OAuthAccountBannedError(AccountApplicationError): + """The OAuth identity resolves to a banned Console account.""" + + +class OAuthAccountNotFoundError(AccountApplicationError): + """An account disappeared while the OAuth use case was running.""" + + +class OAuthWorkspaceCreationNotAllowedError(AccountApplicationError): + """Workspace policy prevents provisioning a workspace for the OAuth account.""" + + +class OAuthSeatsLimitExceededError(AccountApplicationError): + """Account registration would exceed the licensed seat limit.""" + + +class OAuthRegistrationError(AccountApplicationError): + """The OAuth account could not be registered.""" + + def __init__(self, description: str) -> None: + super().__init__(description) + self.description = description + + class EducationRateLimitExceededError(AccountApplicationError): """Too many education verification or activation requests were made.""" diff --git a/api/services/account_oauth_adapters.py b/api/services/account_oauth_adapters.py new file mode 100644 index 00000000000..c9586958311 --- /dev/null +++ b/api/services/account_oauth_adapters.py @@ -0,0 +1,203 @@ +"""Infrastructure gateways for Console account OAuth sign-in.""" + +import logging +from collections.abc import Generator +from contextlib import AbstractContextManager, contextmanager +from hashlib import sha256 +from threading import Event, Thread +from typing import Protocol, override + +import httpx +from redis import RedisError +from redis.exceptions import LockError + +from extensions.ext_redis import RedisClientWrapper +from libs.oauth import OAuth +from services.account_errors import ( + OAuthIdentityLockUnavailableError, + OAuthProviderAuthorizationError, + OAuthProviderRequestError, +) +from services.account_oauth_service import ( + OAuthAccountClaimLease, + OAuthAccountClaimLock, + OAuthProviderGateway, + OAuthRegistrationPolicyGateway, + OAuthWorkspacePolicyGateway, +) +from services.billing_service import BillingService +from services.entities.account_oauth_entities import OAuthAuthorizationRequest, OAuthIdentity +from services.system_feature_service import SystemFeatureService + +logger = logging.getLogger(__name__) + +_OAUTH_ACCOUNT_CLAIM_LOCK_PREFIX = "oauth:account-claim:" +_OAUTH_ACCOUNT_CLAIM_LOCK_TIMEOUT_SECONDS = 60 +_OAUTH_ACCOUNT_CLAIM_LOCK_BLOCKING_TIMEOUT_SECONDS = 10 +_OAUTH_ACCOUNT_CLAIM_LOCK_RENEW_INTERVAL_SECONDS = 20 +_OAUTH_ACCOUNT_CLAIM_LOCK_HEARTBEAT_JOIN_TIMEOUT_SECONDS = 2 + + +class _RedisLock(Protocol): + def acquire(self) -> bool: ... + + def reacquire(self) -> bool: ... + + def release(self) -> None: ... + + +class _RedisOAuthAccountClaimLease(OAuthAccountClaimLease): + def __init__(self, *, locks: tuple[_RedisLock, ...], lost: Event) -> None: + self._locks = locks + self._lost = lost + + @override + def ensure_owned(self) -> None: + if self._lost.is_set(): + raise OAuthIdentityLockUnavailableError + try: + for lock in self._locks: + lock.reacquire() + except (LockError, RedisError) as exc: + self._lost.set() + raise OAuthIdentityLockUnavailableError from exc + except Exception as exc: + self._lost.set() + raise OAuthIdentityLockUnavailableError from exc + + def mark_lost(self) -> None: + self._lost.set() + + +class DifyOAuthProviderGateway(OAuthProviderGateway): + def __init__(self, *, provider_name: str, client: OAuth) -> None: + self._provider_name = provider_name + self._client = client + + @override + def get_authorization_url(self, request: OAuthAuthorizationRequest) -> str: + return self._client.get_authorization_url( + invite_token=request.invite_token, + timezone=request.timezone, + language=request.language, + redirect_url=request.redirect_url, + ) + + @override + def get_identity(self, code: str) -> OAuthIdentity: + try: + token = self._client.get_access_token(code) + user_info = self._client.get_user_info(token) + except httpx.HTTPError as exc: + error_text = exc.response.text if isinstance(exc, httpx.HTTPStatusError) else str(exc) + logger.exception( + "An error occurred during the OAuth process with %s: %s", + self._provider_name, + error_text, + ) + raise OAuthProviderRequestError from exc + except ValueError as exc: + logger.warning("OAuth error with %s", self._provider_name, exc_info=True) + raise OAuthProviderAuthorizationError(str(exc)) from exc + return OAuthIdentity(id=user_info.id, name=user_info.name, email=user_info.email) + + +class RedisOAuthAccountClaimLock(OAuthAccountClaimLock): + def __init__(self, *, client: RedisClientWrapper) -> None: + self._client = client + + @override + def acquire(self, *, provider: str, open_id: str, email: str) -> AbstractContextManager[OAuthAccountClaimLease]: + return self._acquire( + lock_names=( + self._lock_name("identity", provider, open_id), + self._lock_name("email", email), + ) + ) + + @override + def acquire_account(self, account_id: str) -> AbstractContextManager[OAuthAccountClaimLease]: + return self._acquire(lock_names=(self._lock_name("account", account_id),)) + + @contextmanager + def _acquire(self, *, lock_names: tuple[str, ...]) -> Generator[OAuthAccountClaimLease, None, None]: + sorted_lock_names = sorted(set(lock_names)) + locks: list[_RedisLock] = [] + try: + for lock_name in sorted_lock_names: + lock = self._client.lock( + lock_name, + timeout=_OAUTH_ACCOUNT_CLAIM_LOCK_TIMEOUT_SECONDS, + blocking_timeout=_OAUTH_ACCOUNT_CLAIM_LOCK_BLOCKING_TIMEOUT_SECONDS, + thread_local=False, + ) + if not lock.acquire(): + raise OAuthIdentityLockUnavailableError + locks.append(lock) + except (LockError, RedisError) as exc: + self._release(locks) + raise OAuthIdentityLockUnavailableError from exc + except OAuthIdentityLockUnavailableError: + self._release(locks) + raise + + stop_heartbeat = Event() + lease = _RedisOAuthAccountClaimLease(locks=tuple(locks), lost=Event()) + heartbeat = Thread( + target=self._renew_while_held, + args=(lease, stop_heartbeat), + daemon=True, + name=f"OAuthAccountClaimLock({sha256(''.join(sorted_lock_names).encode()).hexdigest()[:12]})", + ) + heartbeat.start() + try: + yield lease + lease.ensure_owned() + finally: + stop_heartbeat.set() + heartbeat.join(timeout=_OAUTH_ACCOUNT_CLAIM_LOCK_HEARTBEAT_JOIN_TIMEOUT_SECONDS) + if heartbeat.is_alive(): + logger.warning("OAuth account claim lock heartbeat did not stop before release") + self._release(locks) + + @staticmethod + def _lock_name(kind: str, *parts: str) -> str: + digest = sha256("\0".join((kind, *parts)).encode()).hexdigest() + return f"{_OAUTH_ACCOUNT_CLAIM_LOCK_PREFIX}{digest}" + + @staticmethod + def _release(locks: list[_RedisLock]) -> None: + for lock in reversed(locks): + try: + lock.release() + except (LockError, RedisError): + logger.warning("Failed to release OAuth account claim lock", exc_info=True) + + @staticmethod + def _renew_while_held(lease: _RedisOAuthAccountClaimLease, stop_heartbeat: Event) -> None: + while not stop_heartbeat.wait(_OAUTH_ACCOUNT_CLAIM_LOCK_RENEW_INTERVAL_SECONDS): + try: + lease.ensure_owned() + except OAuthIdentityLockUnavailableError: + lease.mark_lost() + logger.error("OAuth account claim lock ownership was lost; stop renewing", exc_info=True) + return + + +class DeploymentOAuthPolicyGateway(OAuthRegistrationPolicyGateway, OAuthWorkspacePolicyGateway): + def __init__(self, *, billing_enabled: bool) -> None: + self._billing_enabled = billing_enabled + + @override + def is_registration_allowed(self) -> bool: + return SystemFeatureService.is_registration_allowed() + + @override + def get_freeze_type(self, email: str) -> str | None: + if not self._billing_enabled: + return None + return BillingService.get_email_freeze_type(email) + + @override + def is_creation_allowed(self) -> bool: + return SystemFeatureService.is_workspace_creation_allowed() diff --git a/api/services/account_oauth_service.py b/api/services/account_oauth_service.py new file mode 100644 index 00000000000..3a8354b6e7e --- /dev/null +++ b/api/services/account_oauth_service.py @@ -0,0 +1,283 @@ +"""Application service for Console account OAuth sign-in.""" + +from collections.abc import Callable, Mapping, Sequence +from contextlib import AbstractContextManager +from datetime import datetime +from typing import Protocol + +from services.account_email import normalize_email +from services.account_errors import ( + AccountEmailDomainSuspendedError, + AccountEmailFrozenError, + InvalidOAuthInvitationError, + InvalidOAuthProviderError, + OAuthAccountBannedError, + OAuthAccountNotFoundError, + OAuthInvitationAccountMismatchError, + OAuthRegistrationError, + OAuthWorkspaceCreationNotAllowedError, +) +from services.account_ports import AccountIntegrationRepository, AccountRepository, AccountWorkspaceMembershipQuery +from services.entities.account_entities import AccountSessionTokens, AccountSnapshot +from services.entities.account_oauth_entities import ( + OAuthAccountRegistration, + OAuthAuthorizationRequest, + OAuthCallbackCommand, + OAuthCallbackResult, + OAuthIdentity, + OAuthInvitation, + OAuthInvitationResult, + OAuthSignInResult, +) + +_BANNED_ACCOUNT_STATUS = "banned" +_PENDING_ACCOUNT_STATUS = "pending" + + +class OAuthProviderGateway(Protocol): + def get_authorization_url(self, request: OAuthAuthorizationRequest) -> str: ... + + def get_identity(self, code: str) -> OAuthIdentity: ... + + +class OAuthInvitationGateway(Protocol): + def resolve(self, invite_token: str) -> OAuthInvitation | None: ... + + +class OAuthAccountClaimLease(Protocol): + def ensure_owned(self) -> None: ... + + +class OAuthAccountClaimLock(Protocol): + def acquire(self, *, provider: str, open_id: str, email: str) -> AbstractContextManager[OAuthAccountClaimLease]: ... + + def acquire_account(self, account_id: str) -> AbstractContextManager[OAuthAccountClaimLease]: ... + + +class OAuthAccountRegistrationGateway(Protocol): + def register(self, registration: OAuthAccountRegistration) -> str: ... + + +class OAuthWorkspaceGateway(Protocol): + def create_owner_workspace(self, account_id: str) -> None: ... + + def try_join_default_workspace(self, account_id: str) -> None: ... + + +class OAuthSessionGateway(Protocol): + def login(self, account_id: str, *, ip_address: str) -> AccountSessionTokens: ... + + +class OAuthRegistrationPolicyGateway(Protocol): + def is_registration_allowed(self) -> bool: ... + + def get_freeze_type(self, email: str) -> str | None: ... + + +class OAuthWorkspacePolicyGateway(Protocol): + def is_creation_allowed(self) -> bool: ... + + +class AccountOAuthService: + def __init__( + self, + *, + providers: Mapping[str, OAuthProviderGateway], + accounts: AccountRepository, + integrations: AccountIntegrationRepository, + memberships: AccountWorkspaceMembershipQuery, + invitations: OAuthInvitationGateway, + account_claims: OAuthAccountClaimLock, + registration: OAuthAccountRegistrationGateway, + workspaces: OAuthWorkspaceGateway, + sessions: OAuthSessionGateway, + registration_policy: OAuthRegistrationPolicyGateway, + workspace_policy: OAuthWorkspacePolicyGateway, + supported_languages: Sequence[str], + now: Callable[[], datetime], + ) -> None: + self._providers = dict(providers) + self._accounts = accounts + self._integrations = integrations + self._memberships = memberships + self._invitations = invitations + self._account_claims = account_claims + self._registration = registration + self._workspaces = workspaces + self._sessions = sessions + self._registration_policy = registration_policy + self._workspace_policy = workspace_policy + self._supported_languages = tuple(supported_languages) + self._now = now + + def start_authorization(self, provider: str, request: OAuthAuthorizationRequest) -> str: + return self._provider(provider).get_authorization_url(request) + + def complete_authorization(self, command: OAuthCallbackCommand) -> OAuthCallbackResult: + provider = self._provider(command.provider) + identity = provider.get_identity(command.code) + identity_email_key = self._identity_email_key(identity.email) + + with self._account_claims.acquire( + provider=command.provider, + open_id=identity.id, + email=identity_email_key, + ) as identity_claim: + return self._complete_claimed_authorization(command, identity, identity_claim) + + def _complete_claimed_authorization( + self, + command: OAuthCallbackCommand, + identity: OAuthIdentity, + identity_claim: OAuthAccountClaimLease, + ) -> OAuthCallbackResult: + if command.invite_token is not None: + return self._complete_invitation(command, identity, identity_claim) + + account = self._resolve_account(command.provider, identity) + oauth_new_user = account is None + if account is None: + identity_claim.ensure_owned() + account = self._register_account(command, identity) + identity_claim.ensure_owned() + + self._ensure_account_can_login(account) + identity_claim.ensure_owned() + self._integrations.link(account.id, provider=command.provider, open_id=identity.id) + identity_claim.ensure_owned() + with self._account_claims.acquire_account(account.id) as account_claim: + if oauth_new_user: + self._provision_new_account_workspaces(account.id, account_claim) + else: + self._provision_owner_workspace_if_required(account.id, account_claim) + if account.status == _PENDING_ACCOUNT_STATUS: + account_claim.ensure_owned() + self._accounts.activate_pending(account.id, initialized_at=self._now()) + account_claim.ensure_owned() + + identity_claim.ensure_owned() + tokens = self._sessions.login(account.id, ip_address=command.ip_address) + return OAuthSignInResult(tokens=tokens, oauth_new_user=oauth_new_user) + + def _provision_new_account_workspaces( + self, + account_id: str, + account_claim: OAuthAccountClaimLease, + ) -> None: + if self._memberships.has_active_membership(account_id): + account_claim.ensure_owned() + self._workspaces.try_join_default_workspace(account_id) + account_claim.ensure_owned() + return + + creation_error = OAuthWorkspaceCreationNotAllowedError() + account_claim.ensure_owned() + if self._workspace_policy.is_creation_allowed(): + account_claim.ensure_owned() + try: + self._workspaces.create_owner_workspace(account_id) + except OAuthWorkspaceCreationNotAllowedError as exc: + creation_error = exc + else: + account_claim.ensure_owned() + self._workspaces.try_join_default_workspace(account_id) + account_claim.ensure_owned() + return + + account_claim.ensure_owned() + self._workspaces.try_join_default_workspace(account_id) + account_claim.ensure_owned() + if self._memberships.has_active_membership(account_id): + return + raise creation_error + + def _provision_owner_workspace_if_required( + self, + account_id: str, + account_claim: OAuthAccountClaimLease, + ) -> None: + if self._memberships.has_active_membership(account_id): + return + account_claim.ensure_owned() + if not self._workspace_policy.is_creation_allowed(): + raise OAuthWorkspaceCreationNotAllowedError + account_claim.ensure_owned() + self._workspaces.create_owner_workspace(account_id) + account_claim.ensure_owned() + + def _complete_invitation( + self, + command: OAuthCallbackCommand, + identity: OAuthIdentity, + identity_claim: OAuthAccountClaimLease, + ) -> OAuthCallbackResult: + invite_token = command.invite_token + if invite_token is None: + raise AssertionError("invitation completion requires a token") + invitation = self._invitations.resolve(invite_token) + if invitation is None: + raise InvalidOAuthInvitationError + if self._normalize_email(invitation.account_email) != self._normalize_email(identity.email): + raise OAuthInvitationAccountMismatchError(invite_token) + if invitation.account_status == _BANNED_ACCOUNT_STATUS: + raise OAuthAccountBannedError + + identity_claim.ensure_owned() + self._integrations.link(invitation.account_id, provider=command.provider, open_id=identity.id) + identity_claim.ensure_owned() + tokens = self._sessions.login(invitation.account_id, ip_address=command.ip_address) + return OAuthInvitationResult(tokens=tokens, invite_token=invite_token) + + def _register_account(self, command: OAuthCallbackCommand, identity: OAuthIdentity) -> AccountSnapshot: + normalized_email = self._normalize_email(identity.email) + if not self._registration_policy.is_registration_allowed(): + freeze_type = self._registration_policy.get_freeze_type(normalized_email) + if freeze_type == "email_domain_suspended": + raise AccountEmailDomainSuspendedError + if freeze_type: + raise AccountEmailFrozenError + raise OAuthRegistrationError("Invalid email or password") + + language = command.language or command.browser_language + if language not in self._supported_languages: + language = self._supported_languages[0] + account_id = self._registration.register( + OAuthAccountRegistration( + email=normalized_email, + name=identity.name or "Dify", + language=language, + timezone=command.timezone, + ip_address=command.ip_address, + ) + ) + account = self._accounts.get(account_id) + if account is None: + raise OAuthAccountNotFoundError + return account + + def _resolve_account(self, provider: str, identity: OAuthIdentity) -> AccountSnapshot | None: + account_id = self._integrations.find_account_id(provider=provider, open_id=identity.id) + if account_id is not None: + account = self._accounts.get(account_id) + if account is not None: + return account + return self._accounts.find_by_email(identity.email) + + @staticmethod + def _normalize_email(email: str) -> str: + return email.strip().lower() + + @staticmethod + def _identity_email_key(email: str) -> str: + return normalize_email(email.strip()) + + @staticmethod + def _ensure_account_can_login(account: AccountSnapshot) -> None: + if account.status == _BANNED_ACCOUNT_STATUS: + raise OAuthAccountBannedError + + def _provider(self, provider: str) -> OAuthProviderGateway: + gateway = self._providers.get(provider) + if gateway is None: + raise InvalidOAuthProviderError + return gateway diff --git a/api/services/account_ports.py b/api/services/account_ports.py index 78792664127..bc0b5d1eeb9 100644 --- a/api/services/account_ports.py +++ b/api/services/account_ports.py @@ -1,8 +1,14 @@ -"""Persistence ports used by account application services.""" +"""Ports used by account application services.""" from collections.abc import Sequence +from datetime import datetime from typing import Protocol +from services.entities.account_access_entities import ( + AccountSessionRevocation, + AccountSessionSnapshot, + AccountWorkspaceSnapshot, +) from services.entities.account_entities import ( AccountCredentials, AccountDeletionChallenge, @@ -16,11 +22,17 @@ from services.entities.account_entities import ( ) +class AccountSnapshotQuery(Protocol): + def get(self, account_id: str) -> AccountSnapshot | None: ... + + class AccountRepository(Protocol): def get(self, account_id: str) -> AccountSnapshot | None: ... def find_by_email(self, email: str) -> AccountSnapshot | None: ... + def activate_pending(self, account_id: str, *, initialized_at: datetime) -> None: ... + def get_credentials(self, account_id: str) -> AccountCredentials | None: ... def update_profile(self, account_id: str, changes: AccountProfileChanges) -> AccountSnapshot | None: ... @@ -42,12 +54,45 @@ class AccountRepository(Protocol): class AccountIntegrationRepository(Protocol): + def find_account_id(self, *, provider: str, open_id: str) -> str | None: ... + def list_for_account(self, account_id: str) -> list[AccountIntegrationSnapshot]: ... + def link(self, account_id: str, *, provider: str, open_id: str) -> None: ... + class AccountWorkspaceMembershipQuery(Protocol): def list_ids_for_account(self, account_id: str) -> Sequence[str]: ... + def has_active_membership(self, account_id: str) -> bool: ... + + +class AccountWorkspaceSnapshotQuery(Protocol): + def list_account_access_workspaces(self, account_id: str) -> Sequence[AccountWorkspaceSnapshot]: ... + + +class AccountSessionRepository(Protocol): + def list_active( + self, + *, + account_id: str, + active_at: datetime, + offset: int, + limit: int, + ) -> tuple[int, Sequence[AccountSessionSnapshot]]: ... + + def revoke( + self, + *, + account_id: str, + token_id: str, + revoked_at: datetime, + ) -> AccountSessionRevocation: ... + + +class AccountTokenCacheInvalidator(Protocol): + def __call__(self, token_hash: str) -> None: ... + class AccountAvatarFileGateway(Protocol): def get_owned_signed_url(self, *, account_id: str, upload_file_id: str) -> str | None: ... diff --git a/api/services/account_service.py b/api/services/account_service.py index 0e0378c9996..32067a44e8b 100644 --- a/api/services/account_service.py +++ b/api/services/account_service.py @@ -72,7 +72,6 @@ from services.errors.account import ( CannotOperateSelfError, EmailDomainSuspendedError, InvalidActionError, - LinkAccountIntegrateError, MemberNotInTenantError, NoPermissionError, RefreshTokenNotFoundError, @@ -322,9 +321,9 @@ class AccountService: @staticmethod def get_account_by_id(account_id: str, *, session: Session) -> Account | None: """Plain ``Account`` getter — no banned check, no tenant rotation, - no ``last_active_at`` write. Use this from read-only identity - endpoints (``/openapi/v1/account``) where ``load_user``'s - side-effects (current-tenant assignment, commit) are unwanted. + no ``last_active_at`` write. Use this from authentication and read + paths where ``load_user``'s current-tenant assignment and commit are + unwanted. ``session`` is injected by the caller so this service stays free of a Flask-scoped session import. @@ -514,35 +513,6 @@ class AccountService: return account - @staticmethod - def link_account_integrate(provider: str, open_id: str, account: Account, *, session: Session): - """Link account integrate""" - try: - # Query whether there is an existing binding record for the same provider - account_integrate: AccountIntegrate | None = session.scalar( - select(AccountIntegrate) - .where(AccountIntegrate.account_id == account.id, AccountIntegrate.provider == provider) - .limit(1) - ) - - if account_integrate: - # If it exists, update the record - account_integrate.open_id = open_id - account_integrate.encrypted_token = "" # todo - account_integrate.updated_at = naive_utc_now() - else: - # If it does not exist, create a new record - account_integrate = AccountIntegrate( - account_id=account.id, provider=provider, open_id=open_id, encrypted_token="" - ) - session.add(account_integrate) - - session.commit() - logger.info("Account %s linked %s account %s.", account.id, provider, open_id) - except Exception as e: - logger.exception("Failed to link %s account %s to Account %s", provider, open_id, account.id) - raise LinkAccountIntegrateError("Failed to link account.") from e - @staticmethod def update_account_email(account: Account, email: str, session: Session) -> Account: """Update account email""" @@ -1261,36 +1231,12 @@ class TenantService: ).all() ) - @staticmethod - def get_account_memberships(account_id: str, *, session: Session) -> list[Row[tuple[TenantAccountJoin, Tenant]]]: - """Return ``(TenantAccountJoin, Tenant)`` rows for every workspace - the account belongs to. Unlike :meth:`get_join_tenants` this keeps - the join row so callers can read ``role``/``current`` alongside the - tenant — used by ``/openapi/v1/account`` to render workspace - membership + pick the default workspace. - - ``session`` is injected by the caller so this service stays free - of a Flask-scoped session import. - - No tenant-status filter: parity with the legacy controller query - (the openapi identity endpoint listed all joined tenants). - """ - return ( - session.query(TenantAccountJoin, Tenant) - .join(Tenant, Tenant.id == TenantAccountJoin.tenant_id) - .filter(TenantAccountJoin.account_id == account_id) - .all() - ) - @staticmethod def get_workspaces_for_account(account_id: str, *, session: Session) -> list[Row[tuple[Tenant, TenantAccountJoin]]]: """``(Tenant, TenantAccountJoin)`` rows for every workspace the account belongs to, ordered by ``Tenant.created_at`` ASC — the - canonical ordering for ``/openapi/v1/workspaces``. - - Distinct from :meth:`get_account_memberships`: tuple order is - flipped (tenant first) and rows are sorted, so the workspace - listing is stable across requests. + canonical ordering for ``/openapi/v1/workspaces``. Rows keep the + tenant first so callers can serialize the workspace directly. """ return list( session.execute( @@ -1864,8 +1810,6 @@ class RegisterService: email: str, name: str, password: str | None = None, - open_id: str | None = None, - provider: str | None = None, language: str | None = None, status: AccountStatus | None = None, is_setup: bool | None = False, @@ -1894,9 +1838,6 @@ class RegisterService: account.status = status or AccountStatus.ACTIVE account.initialized_at = naive_utc_now() - if open_id is not None and provider is not None: - AccountService.link_account_integrate(provider, open_id, account, session=session) - if ( SystemFeatureService.is_workspace_creation_allowed() and create_workspace_required @@ -2049,11 +1990,6 @@ class RegisterService: redis_client.setex(cls._get_invitation_token_key(token), expiry_hours * 60 * 60, json.dumps(invitation_data)) return token - @classmethod - def is_valid_invite_token(cls, token: str) -> bool: - data = redis_client.get(cls._get_invitation_token_key(token)) - return data is not None - @classmethod def revoke_token(cls, workspace_id: str | None, email: str | None, token: str): if workspace_id and email: diff --git a/api/services/app_statistic_query.py b/api/services/app_statistic_query.py new file mode 100644 index 00000000000..3b1913b6bca --- /dev/null +++ b/api/services/app_statistic_query.py @@ -0,0 +1,122 @@ +"""Application boundary for app monitoring statistics.""" + +from collections.abc import Sequence +from datetime import datetime +from decimal import Decimal +from typing import NamedTuple, Protocol + + +class DailyMessageStatisticRecord(NamedTuple): + date: str + message_count: int + + +class DailyConversationStatisticRecord(NamedTuple): + date: str + conversation_count: int + + +class DailyTerminalStatisticRecord(NamedTuple): + date: str + terminal_count: int + + +class DailyTokenCostStatisticRecord(NamedTuple): + date: str + token_count: int | None + total_price: Decimal | None + currency: str | None + + +class AverageSessionInteractionStatisticRecord(NamedTuple): + date: str + interactions: float + + +class UserSatisfactionRateStatisticRecord(NamedTuple): + date: str + rate: float + + +class AverageResponseTimeStatisticRecord(NamedTuple): + date: str + latency: float + + +class TokensPerSecondStatisticRecord(NamedTuple): + date: str + tps: float + + +class AppStatisticQuery(Protocol): + def get_daily_messages( + self, + *, + app_id: str, + start_date: datetime | None, + end_date: datetime | None, + timezone: str, + ) -> Sequence[DailyMessageStatisticRecord]: ... + + def get_daily_conversations( + self, + *, + app_id: str, + start_date: datetime | None, + end_date: datetime | None, + timezone: str, + ) -> Sequence[DailyConversationStatisticRecord]: ... + + def get_daily_terminals( + self, + *, + app_id: str, + start_date: datetime | None, + end_date: datetime | None, + timezone: str, + ) -> Sequence[DailyTerminalStatisticRecord]: ... + + def get_daily_token_costs( + self, + *, + app_id: str, + start_date: datetime | None, + end_date: datetime | None, + timezone: str, + ) -> Sequence[DailyTokenCostStatisticRecord]: ... + + def get_average_session_interactions( + self, + *, + app_id: str, + start_date: datetime | None, + end_date: datetime | None, + timezone: str, + ) -> Sequence[AverageSessionInteractionStatisticRecord]: ... + + def get_user_satisfaction_rates( + self, + *, + app_id: str, + start_date: datetime | None, + end_date: datetime | None, + timezone: str, + ) -> Sequence[UserSatisfactionRateStatisticRecord]: ... + + def get_average_response_times( + self, + *, + app_id: str, + start_date: datetime | None, + end_date: datetime | None, + timezone: str, + ) -> Sequence[AverageResponseTimeStatisticRecord]: ... + + def get_tokens_per_second( + self, + *, + app_id: str, + start_date: datetime | None, + end_date: datetime | None, + timezone: str, + ) -> Sequence[TokensPerSecondStatisticRecord]: ... diff --git a/api/services/dataset_service.py b/api/services/dataset_service.py index fb203de91d6..8d2c138cf3d 100644 --- a/api/services/dataset_service.py +++ b/api/services/dataset_service.py @@ -3854,7 +3854,8 @@ class SegmentService: logger.exception("Failed to regenerate summary for segment %s", segment.id) # Don't fail the entire update if summary regeneration fails # update multimodel vector index - VectorService.update_multimodel_vector(segment, args.attachment_ids or [], dataset, session=session) + if args.attachment_ids is not None: + VectorService.update_multimodel_vector(segment, args.attachment_ids, dataset, session=session) except Exception as e: logger.exception("update segment index failed") segment.enabled = False diff --git a/api/services/end_user_service.py b/api/services/end_user_service.py index c15e9949abb..a01d3a8128f 100644 --- a/api/services/end_user_service.py +++ b/api/services/end_user_service.py @@ -63,6 +63,11 @@ class EndUserService: EndUser.tenant_id == tenant_id, EndUser.app_id == app_id, EndUser.session_id == user_id, + # An AppDeploy row is never a legacy row this could upgrade: + # the type was added after the split, and FileGrantService + # reads its rows by type. Retyping one here would hide it + # from that read and strand the files it owns. + EndUser.type != EndUserType.APP_DEPLOY, ) .order_by( # Prioritize records with matching type (0 = match, 1 = no match) diff --git a/api/services/entities/account_access_entities.py b/api/services/entities/account_access_entities.py new file mode 100644 index 00000000000..a59db492259 --- /dev/null +++ b/api/services/entities/account_access_entities.py @@ -0,0 +1,52 @@ +"""Framework-neutral data contracts for account identity and access sessions.""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime + +from services.entities.account_entities import AccountSnapshot + + +@dataclass(frozen=True, slots=True) +class AccountWorkspaceSnapshot: + id: str + name: str + role: str + current: bool + + +@dataclass(frozen=True, slots=True) +class AccountAccessSnapshot: + account: AccountSnapshot + workspaces: tuple[AccountWorkspaceSnapshot, ...] + default_workspace_id: str | None + + +@dataclass(frozen=True, slots=True) +class AccountSessionSnapshot: + id: str + prefix: str + client_id: str + device_label: str + created_at: datetime | None + last_used_at: datetime | None + expires_at: datetime | None + + +@dataclass(frozen=True, slots=True) +class AccountSessionPage: + page: int + limit: int + total: int + items: tuple[AccountSessionSnapshot, ...] + + @property + def has_more(self) -> bool: + return self.page * self.limit < self.total + + +@dataclass(frozen=True, slots=True) +class AccountSessionRevocation: + owned: bool + token_hash: str | None = None diff --git a/api/services/entities/account_entities.py b/api/services/entities/account_entities.py index 3e4293871e8..cfc637b09e1 100644 --- a/api/services/entities/account_entities.py +++ b/api/services/entities/account_entities.py @@ -1,4 +1,4 @@ -"""Framework-neutral contracts for Console account use cases.""" +"""Framework-neutral contracts shared by account use cases.""" from __future__ import annotations diff --git a/api/services/entities/account_oauth_entities.py b/api/services/entities/account_oauth_entities.py new file mode 100644 index 00000000000..bfe66b99e63 --- /dev/null +++ b/api/services/entities/account_oauth_entities.py @@ -0,0 +1,62 @@ +"""Framework-neutral contracts for Console account OAuth sign-in.""" + +from dataclasses import dataclass + +from services.entities.account_entities import AccountSessionTokens as _AccountSessionTokens + + +@dataclass(frozen=True, slots=True) +class OAuthAuthorizationRequest: + invite_token: str | None = None + timezone: str | None = None + language: str | None = None + redirect_url: str | None = None + + +@dataclass(frozen=True, slots=True) +class OAuthIdentity: + id: str + name: str + email: str + + +@dataclass(frozen=True, slots=True) +class OAuthCallbackCommand: + provider: str + code: str + invite_token: str | None + timezone: str | None + language: str | None + browser_language: str | None + ip_address: str + + +@dataclass(frozen=True, slots=True) +class OAuthInvitation: + account_id: str + account_email: str + account_status: str + + +@dataclass(frozen=True, slots=True) +class OAuthAccountRegistration: + email: str + name: str + language: str + timezone: str | None + ip_address: str + + +@dataclass(frozen=True, slots=True) +class OAuthSignInResult: + tokens: _AccountSessionTokens + oauth_new_user: bool + + +@dataclass(frozen=True, slots=True) +class OAuthInvitationResult: + tokens: _AccountSessionTokens + invite_token: str + + +type OAuthCallbackResult = OAuthSignInResult | OAuthInvitationResult diff --git a/api/services/entities/file_grant_entities.py b/api/services/entities/file_grant_entities.py new file mode 100644 index 00000000000..5d7ffb44d7d --- /dev/null +++ b/api/services/entities/file_grant_entities.py @@ -0,0 +1,139 @@ +from __future__ import annotations + +from collections.abc import Iterable +from dataclasses import dataclass +from datetime import datetime +from enum import StrEnum + +from pydantic import BaseModel + + +class FileKind(StrEnum): + UPLOAD = "upload" + TOOL = "tool" + + +class FileGrantScope(StrEnum): + UPLOAD = "upload" + RESOLVE = "resolve" + PRODUCE = "produce" + + +class FileGrantClaims(BaseModel): + sub: str + tenant_id: str + app_id: str + scopes: list[FileGrantScope] + exp: int + + +class FileContentClaims(BaseModel): + kind: FileKind + file_id: str + exp: int + + +@dataclass(frozen=True, slots=True) +class FileGrantContext: + tenant_id: str + app_id: str + end_user_id: str + + +@dataclass(frozen=True, slots=True) +class FileRef: + id: str + kind: FileKind + + +@dataclass(frozen=True, slots=True) +class ResolvedFile: + id: str + kind: FileKind + name: str + size: int + extension: str + mime_type: str | None + + +@dataclass(frozen=True, slots=True) +class ResolvedFileAccess: + file: ResolvedFile + external_url: str + internal_url: str + + +@dataclass(frozen=True, slots=True) +class FileContentRecord: + name: str + size: int + mime_type: str | None + storage_key: str + + +@dataclass(frozen=True, slots=True) +class FileContent: + name: str + size: int + mime_type: str | None + stream: Iterable[bytes] + + +@dataclass(frozen=True, slots=True) +class StoredUpload: + id: str + name: str + size: int + extension: str | None + mime_type: str | None + created_by: str | None + created_at: datetime | None + tenant_id: str | None + source_url: str + + +@dataclass(frozen=True, slots=True) +class StoredProducedFile: + id: str + name: str + size: int + mime_type: str | None + + +@dataclass(frozen=True, slots=True) +class RemoteFile: + filename: str + mimetype: str + content: bytes + + +@dataclass(frozen=True, slots=True) +class FileGrantLimits: + file_size_limit: int + image_file_size_limit: int + audio_file_size_limit: int + video_file_size_limit: int + workflow_file_upload_limit: int + batch_count_limit: int + + +@dataclass(frozen=True, slots=True) +class FileGrantMintRequest: + tenant_id: str + app_id: str + subject: str + is_anonymous: bool + scopes: tuple[FileGrantScope, ...] + ttl_seconds: int + file_refs: tuple[FileRef, ...] + optional_file_refs: tuple[FileRef, ...] + run_deadline: int | None + + +@dataclass(frozen=True, slots=True) +class FileGrantMintResult: + grant: str + expires_at: int + limits: FileGrantLimits + files: tuple[ResolvedFile, ...] + optional_files: tuple[ResolvedFileAccess | None, ...] diff --git a/api/services/errors/account.py b/api/services/errors/account.py index fc1d6772174..aa9084efb30 100644 --- a/api/services/errors/account.py +++ b/api/services/errors/account.py @@ -38,10 +38,6 @@ class AccountNotLinkTenantError(BaseServiceError): pass -class LinkAccountIntegrateError(BaseServiceError): - pass - - class TenantNotFoundError(BaseServiceError): pass diff --git a/api/services/errors/file_grant.py b/api/services/errors/file_grant.py new file mode 100644 index 00000000000..17f08db9115 --- /dev/null +++ b/api/services/errors/file_grant.py @@ -0,0 +1,34 @@ +class AppNotFoundError(Exception): + pass + + +class EndUserNotFoundError(Exception): + pass + + +class InvalidGrantRequestError(Exception): + pass + + +class InvalidFileGrantError(Exception): + pass + + +class GrantTtlTooLongError(Exception): + pass + + +class GrantedFileNotFoundError(Exception): + pass + + +class InvalidSubjectError(Exception): + pass + + +class RemoteFileUnavailableError(Exception): + pass + + +class TooManyFileRefsError(Exception): + pass diff --git a/api/services/errors/workspace.py b/api/services/errors/workspace.py index 577238507f8..18ff8b6eccd 100644 --- a/api/services/errors/workspace.py +++ b/api/services/errors/workspace.py @@ -5,9 +5,5 @@ class WorkSpaceNotAllowedCreateError(BaseServiceError): pass -class WorkSpaceNotFoundError(BaseServiceError): - pass - - class WorkspacesLimitExceededError(BaseServiceError): pass diff --git a/api/services/explore_banner_query_service.py b/api/services/explore_banner_query_service.py index 8ef4f3827f4..4eb204f7d85 100644 --- a/api/services/explore_banner_query_service.py +++ b/api/services/explore_banner_query_service.py @@ -7,6 +7,8 @@ from collections.abc import Sequence from datetime import datetime from typing import Any, NamedTuple, Protocol +from constants.languages import languages + _DEFAULT_LANGUAGE = "en-US" @@ -37,6 +39,7 @@ class ExploreBannerQueryService: if not self._enabled: return () + language = language if language in languages else _DEFAULT_LANGUAGE banners = tuple(self._banners.list_enabled(language)) if banners or language == _DEFAULT_LANGUAGE: return banners diff --git a/api/services/file_grant_gateways.py b/api/services/file_grant_gateways.py new file mode 100644 index 00000000000..3fdf2493ee8 --- /dev/null +++ b/api/services/file_grant_gateways.py @@ -0,0 +1,327 @@ +from __future__ import annotations + +import mimetypes +import os +import re +import urllib.parse +from collections.abc import Callable, Sequence +from typing import IO, cast +from uuid import uuid4 + +import httpx +import jwt +from pydantic import ValidationError + +from core.file import remote_fetcher +from core.helper import ssrf_proxy +from core.tools.tool_file_manager import ToolFileManager, resolve_extension +from extensions.ext_storage import Storage +from models.model import EndUser +from services.entities.file_grant_entities import ( + FileContent, + FileContentClaims, + FileContentRecord, + FileGrantClaims, + FileGrantContext, + FileGrantScope, + FileKind, + RemoteFile, + StoredProducedFile, + StoredUpload, +) +from services.errors.file import FileTooLargeError +from services.errors.file_grant import EndUserNotFoundError +from services.file_service import FileService + +FILE_GRANT_AUDIENCE = "dify-files" +FILE_CONTENT_AUDIENCE = "dify-files-content" +_ALGORITHM = "HS256" + + +class FileGrantTokenGateway: + def __init__( + self, + *, + secret_key: str, + external_files_url: str, + internal_files_url: str, + content_token_ttl_seconds: int, + now: Callable[[], int], + ) -> None: + self._secret_key = secret_key + self._external_files_url = external_files_url + self._internal_files_url = internal_files_url + self._content_token_ttl_seconds = content_token_ttl_seconds + self._now = now + + def issue_grant( + self, + *, + context: FileGrantContext, + scopes: Sequence[FileGrantScope], + ttl_seconds: int, + ) -> tuple[str, int]: + expires_at = self._now() + ttl_seconds + token = jwt.encode( + { + "aud": FILE_GRANT_AUDIENCE, + "sub": context.end_user_id, + "tenant_id": context.tenant_id, + "app_id": context.app_id, + "scopes": [str(scope) for scope in scopes], + "exp": expires_at, + }, + self._secret_key, + algorithm=_ALGORITHM, + ) + return token, expires_at + + def decode_grant(self, token: str) -> FileGrantClaims | None: + payload = self._decode( + token, audience=FILE_GRANT_AUDIENCE, required=["exp", "sub", "tenant_id", "app_id", "scopes"] + ) + if payload is None: + return None + try: + return FileGrantClaims.model_validate(payload) + except ValidationError: + return None + + def issue_content_urls(self, *, file_id: str, kind: FileKind) -> tuple[str, str]: + external_token = self._issue_content_token(file_id=file_id, kind=kind) + internal_token = self._issue_content_token(file_id=file_id, kind=kind) + path = f"/files/appdeploy/{file_id}/content" + return ( + f"{self._external_files_url}{path}?token={external_token}", + f"{self._internal_files_url}{path}?token={internal_token}", + ) + + def decode_content_token(self, token: str) -> FileContentClaims | None: + payload = self._decode( + token, + audience=FILE_CONTENT_AUDIENCE, + required=["exp", "kind", "file_id"], + ) + if payload is None: + return None + try: + return FileContentClaims.model_validate(payload) + except ValidationError: + return None + + def _issue_content_token(self, *, file_id: str, kind: FileKind) -> str: + return jwt.encode( + { + "aud": FILE_CONTENT_AUDIENCE, + "kind": str(kind), + "file_id": file_id, + "nonce": os.urandom(8).hex(), + "exp": self._now() + self._content_token_ttl_seconds, + }, + self._secret_key, + algorithm=_ALGORITHM, + ) + + def _decode(self, token: str, *, audience: str, required: list[str]) -> dict[str, object] | None: + try: + return cast( + dict[str, object], + jwt.decode( + token, + self._secret_key, + algorithms=[_ALGORITHM], + audience=audience, + options={"require": ["aud", *required]}, + ), + ) + except jwt.PyJWTError: + return None + + +class FileGrantFileGateway: + def __init__( + self, + *, + load_end_user: Callable[[FileGrantContext], EndUser | None], + subject_exists: Callable[[FileGrantContext], bool], + file_service: FileService, + tool_files: ToolFileManager, + storage: Storage, + ) -> None: + self._load_end_user = load_end_user + self._subject_exists = subject_exists + self._file_service = file_service + self._tool_files = tool_files + self._storage = storage + + def store_upload_stream( + self, + *, + context: FileGrantContext, + filename: str, + stream: IO[bytes], + mimetype: str, + ) -> StoredUpload: + if not self._subject_exists(context): + raise EndUserNotFoundError(context.end_user_id) + extension = os.path.splitext(filename)[1].lstrip(".").lower() + limit = FileService.file_size_limit(extension=extension) + content = stream.read(limit + 1) + if len(content) > limit: + raise FileTooLargeError(f"File size exceeded. The limit is {limit} bytes.") + return self.store_upload( + context=context, + filename=filename, + content=content, + mimetype=mimetype, + ) + + def store_upload( + self, + *, + context: FileGrantContext, + filename: str, + content: bytes, + mimetype: str, + source_url: str = "", + ) -> StoredUpload: + end_user = self._load_end_user(context) + if end_user is None: + raise EndUserNotFoundError(context.end_user_id) + upload = self._file_service.upload_file( + filename=filename, + content=content, + mimetype=mimetype, + user=end_user, + source_url=source_url, + ) + return StoredUpload( + id=upload.id, + name=upload.name, + size=upload.size, + extension=upload.extension, + mime_type=upload.mime_type, + created_by=upload.created_by, + created_at=upload.created_at, + tenant_id=upload.tenant_id, + source_url=upload.source_url, + ) + + def store_produced( + self, + *, + context: FileGrantContext, + filename: str | None, + stream: IO[bytes], + mimetype: str, + ) -> StoredProducedFile: + extension = resolve_extension(filename=filename, mimetype=mimetype).lstrip(".").lower() + limit = FileService.file_size_limit(extension=extension) + content = stream.read(limit + 1) + if len(content) > limit: + raise FileTooLargeError(f"File size exceeded. The limit is {limit} bytes.") + stored = self._tool_files.create_file_by_raw( + user_id=context.end_user_id, + tenant_id=context.tenant_id, + conversation_id=None, + file_binary=content, + mimetype=mimetype, + filename=filename, + ) + return StoredProducedFile( + id=stored.id, + name=stored.name or "", + size=stored.size, + mime_type=stored.mimetype, + ) + + def open_content(self, record: FileContentRecord) -> FileContent: + return FileContent( + name=record.name, + size=record.size, + mime_type=record.mime_type, + stream=self._storage.load(record.storage_key, stream=True), + ) + + +class FileGrantRemoteFileGateway: + def fetch(self, url: str) -> RemoteFile | None: + try: + metadata = remote_fetcher.make_request("HEAD", url=url, follow_redirects=True) + if metadata.status_code != httpx.codes.OK: + metadata.close() + metadata = remote_fetcher.make_request( + "GET", + url=url, + timeout=3, + follow_redirects=True, + stream_response=True, + ) + if metadata.status_code != httpx.codes.OK: + metadata.close() + return None + + filename, extension, mimetype = self._file_info(metadata) + limit = FileService.file_size_limit(extension=extension) + declared_size = self._declared_size(metadata) + if declared_size is not None and declared_size > limit: + metadata.close() + raise FileTooLargeError(f"File size exceeded. The limit is {limit} bytes.") + + if metadata.request.method == "HEAD": + metadata.close() + response = remote_fetcher.make_request( + "GET", + url=url, + timeout=3, + follow_redirects=True, + stream_response=True, + ) + if response.status_code != httpx.codes.OK: + response.close() + return None + else: + response = metadata + + try: + buffered = ssrf_proxy.buffer_response(response, max_response_bytes=limit) + except ssrf_proxy.ResponseTooLargeError as exc: + raise FileTooLargeError(f"File size exceeded. The limit is {limit} bytes.") from exc + return RemoteFile(filename=filename, mimetype=mimetype, content=buffered.content) + except (httpx.RequestError, ssrf_proxy.UnsupportedResponseEncodingError): + return None + + @staticmethod + def _file_info(response: httpx.Response) -> tuple[str, str, str]: + parsed_url = urllib.parse.urlparse(str(response.url)) + filename = urllib.parse.unquote(os.path.basename(parsed_url.path)) + if not filename: + content_disposition = response.headers.get("Content-Disposition", "") + filename_match = re.search(r'filename="?([^";]+)', content_disposition) + filename = filename_match.group(1) if filename_match else uuid4().hex + extension = os.path.splitext(filename)[1].lstrip(".").lower() + mimetype = ( + mimetypes.guess_type(filename)[0] + or response.headers.get("Content-Type", "").split(";", 1)[0].strip() + or "application/octet-stream" + ) + return filename, extension, mimetype + + @staticmethod + def _declared_size(response: httpx.Response) -> int | None: + value = response.headers.get("Content-Length") + if value is None: + return None + try: + return int(value) + except ValueError: + return None + + +__all__ = [ + "FILE_CONTENT_AUDIENCE", + "FILE_GRANT_AUDIENCE", + "FileGrantFileGateway", + "FileGrantRemoteFileGateway", + "FileGrantTokenGateway", +] diff --git a/api/services/file_grant_service.py b/api/services/file_grant_service.py new file mode 100644 index 00000000000..1cf542f278c --- /dev/null +++ b/api/services/file_grant_service.py @@ -0,0 +1,314 @@ +from __future__ import annotations + +import base64 +import hashlib +from collections.abc import Callable, Sequence +from typing import IO, Protocol + +from services.entities.file_grant_entities import ( + FileContent, + FileContentClaims, + FileContentRecord, + FileGrantClaims, + FileGrantContext, + FileGrantLimits, + FileGrantMintRequest, + FileGrantMintResult, + FileGrantScope, + FileKind, + FileRef, + RemoteFile, + ResolvedFile, + ResolvedFileAccess, + StoredProducedFile, + StoredUpload, +) +from services.errors.file_grant import ( + AppNotFoundError, + EndUserNotFoundError, + GrantedFileNotFoundError, + GrantTtlTooLongError, + InvalidFileGrantError, + InvalidGrantRequestError, + InvalidSubjectError, + RemoteFileUnavailableError, + TooManyFileRefsError, +) + +MAX_SESSION_GRANT_TTL_SECONDS = 7200 +MAX_WORKFLOW_EXECUTION_SECONDS = 24 * 60 * 60 +RUN_GRANT_EXPIRY_GRACE_SECONDS = 5 * 60 +MAX_RUN_GRANT_TTL_SECONDS = MAX_WORKFLOW_EXECUTION_SECONDS + RUN_GRANT_EXPIRY_GRACE_SECONDS +MAX_FILE_GRANT_REFS = 100 + + +class FileGrantRepository(Protocol): + def get_or_create_subject( + self, + *, + tenant_id: str, + app_id: str, + session_id: str, + external_user_id: str, + is_anonymous: bool, + ) -> str | None: ... + + def subject_exists(self, context: FileGrantContext) -> bool: ... + + def resolve_owned_files( + self, + *, + context: FileGrantContext, + refs: Sequence[FileRef], + ) -> list[ResolvedFile | None]: ... + + def get_content_record(self, *, file_id: str, kind: FileKind) -> FileContentRecord | None: ... + + +class FileGrantFiles(Protocol): + def store_upload_stream( + self, + *, + context: FileGrantContext, + filename: str, + stream: IO[bytes], + mimetype: str, + ) -> StoredUpload: ... + + def store_upload( + self, + *, + context: FileGrantContext, + filename: str, + content: bytes, + mimetype: str, + source_url: str = "", + ) -> StoredUpload: ... + + def store_produced( + self, + *, + context: FileGrantContext, + filename: str | None, + stream: IO[bytes], + mimetype: str, + ) -> StoredProducedFile: ... + + def open_content(self, record: FileContentRecord) -> FileContent: ... + + +class FileGrantTokens(Protocol): + def issue_grant( + self, + *, + context: FileGrantContext, + scopes: Sequence[FileGrantScope], + ttl_seconds: int, + ) -> tuple[str, int]: ... + + def decode_grant(self, token: str) -> FileGrantClaims | None: ... + + def issue_content_urls(self, *, file_id: str, kind: FileKind) -> tuple[str, str]: ... + + def decode_content_token(self, token: str) -> FileContentClaims | None: ... + + +class FileGrantRemoteFiles(Protocol): + def fetch(self, url: str) -> RemoteFile | None: ... + + +class FileGrantService: + def __init__( + self, + *, + repository: FileGrantRepository, + files: FileGrantFiles, + tokens: FileGrantTokens, + remote_files: FileGrantRemoteFiles, + limits: FileGrantLimits, + now: Callable[[], int], + ) -> None: + self._repository = repository + self._files = files + self._tokens = tokens + self._remote_files = remote_files + self._limits = limits + self._now = now + + @staticmethod + def session_id_for_subject(subject: str) -> str: + digest = hashlib.sha256(subject.encode()).digest() + return base64.urlsafe_b64encode(digest).decode().rstrip("=") + + def mint(self, request: FileGrantMintRequest) -> FileGrantMintResult: + self._validate_subject(request.subject) + self._validate_ref_count((*request.file_refs, *request.optional_file_refs)) + ttl_seconds = self._effective_ttl_seconds(request) + + end_user_id = self._repository.get_or_create_subject( + tenant_id=request.tenant_id, + app_id=request.app_id, + session_id=self.session_id_for_subject(request.subject), + external_user_id=request.subject[:255], + is_anonymous=request.is_anonymous, + ) + if end_user_id is None: + raise AppNotFoundError(request.app_id) + + context = FileGrantContext(request.tenant_id, request.app_id, end_user_id) + strict_file_count = len(request.file_refs) + resolved_files = self._repository.resolve_owned_files( + context=context, + refs=(*request.file_refs, *request.optional_file_refs), + ) + strict_files = resolved_files[:strict_file_count] + if any(file is None for file in strict_files): + raise GrantedFileNotFoundError() + optional_files = resolved_files[strict_file_count:] + grant, expires_at = self._tokens.issue_grant( + context=context, + scopes=request.scopes, + ttl_seconds=ttl_seconds, + ) + return FileGrantMintResult( + grant=grant, + expires_at=expires_at, + limits=self._limits, + files=tuple(file for file in strict_files if file is not None), + optional_files=tuple(self._with_access(file) if file is not None else None for file in optional_files), + ) + + def decode_grant(self, token: str) -> FileGrantClaims | None: + return self._tokens.decode_grant(token) + + def store_upload( + self, + *, + context: FileGrantContext, + filename: str, + stream: IO[bytes], + mimetype: str, + ) -> StoredUpload: + return self._files.store_upload_stream( + context=context, + filename=filename, + stream=stream, + mimetype=mimetype, + ) + + def store_remote_upload(self, *, context: FileGrantContext, url: str) -> StoredUpload: + if not self._repository.subject_exists(context): + raise EndUserNotFoundError(context.end_user_id) + remote_file = self._remote_files.fetch(url) + if remote_file is None: + raise RemoteFileUnavailableError(url) + return self._files.store_upload( + context=context, + filename=remote_file.filename, + content=remote_file.content, + mimetype=remote_file.mimetype, + source_url=url, + ) + + def store_produced( + self, + *, + context: FileGrantContext, + filename: str | None, + stream: IO[bytes], + mimetype: str, + ) -> tuple[StoredProducedFile, ResolvedFileAccess]: + if not self._repository.subject_exists(context): + raise EndUserNotFoundError(context.end_user_id) + stored = self._files.store_produced( + context=context, + filename=filename, + stream=stream, + mimetype=mimetype, + ) + file = ResolvedFile(stored.id, FileKind.TOOL, stored.name, stored.size, "", stored.mime_type) + return stored, self._with_access(file) + + def resolve_files( + self, + *, + context: FileGrantContext, + refs: Sequence[FileRef], + ) -> list[ResolvedFile | None]: + self._validate_ref_count(refs) + if not self._repository.subject_exists(context): + raise EndUserNotFoundError(context.end_user_id) + return self._repository.resolve_owned_files(context=context, refs=refs) + + def resolve_file_access( + self, + *, + context: FileGrantContext, + refs: Sequence[FileRef], + ) -> list[ResolvedFileAccess | None]: + return [ + self._with_access(file) if file is not None else None + for file in self.resolve_files(context=context, refs=refs) + ] + + def load_content(self, *, token: str, requested_file_id: str) -> FileContent | None: + claims = self._tokens.decode_content_token(token) + if claims is None: + raise InvalidFileGrantError() + if claims.file_id != requested_file_id: + return None + record = self._repository.get_content_record(file_id=requested_file_id, kind=claims.kind) + return self._files.open_content(record) if record is not None else None + + def content_urls(self, *, file_id: str, kind: FileKind) -> tuple[str, str]: + return self._tokens.issue_content_urls(file_id=file_id, kind=kind) + + def _with_access(self, file: ResolvedFile) -> ResolvedFileAccess: + external_url, internal_url = self._tokens.issue_content_urls(file_id=file.id, kind=file.kind) + return ResolvedFileAccess(file=file, external_url=external_url, internal_url=internal_url) + + def _effective_ttl_seconds(self, request: FileGrantMintRequest) -> int: + if request.run_deadline is None: + if request.ttl_seconds > MAX_SESSION_GRANT_TTL_SECONDS: + raise GrantTtlTooLongError() + return request.ttl_seconds + + now = self._now() + if FileGrantScope.PRODUCE not in request.scopes: + raise InvalidGrantRequestError("A run deadline requires the produce scope.") + if request.run_deadline <= now: + raise InvalidGrantRequestError("The run deadline has expired.") + if request.run_deadline > now + MAX_WORKFLOW_EXECUTION_SECONDS: + raise InvalidGrantRequestError("The run deadline exceeds the workflow execution limit.") + if request.ttl_seconds > MAX_RUN_GRANT_TTL_SECONDS: + raise GrantTtlTooLongError() + return min(request.ttl_seconds, request.run_deadline - now + RUN_GRANT_EXPIRY_GRACE_SECONDS) + + @staticmethod + def _validate_subject(subject: str) -> None: + if not subject.strip() or "\x00" in subject: + raise InvalidSubjectError() + + @staticmethod + def _validate_ref_count(refs: Sequence[FileRef]) -> None: + if len(refs) > MAX_FILE_GRANT_REFS: + raise TooManyFileRefsError(f"A file grant request may contain at most {MAX_FILE_GRANT_REFS} references.") + + +__all__ = [ + "MAX_FILE_GRANT_REFS", + "MAX_RUN_GRANT_TTL_SECONDS", + "MAX_SESSION_GRANT_TTL_SECONDS", + "MAX_WORKFLOW_EXECUTION_SECONDS", + "RUN_GRANT_EXPIRY_GRACE_SECONDS", + "AppNotFoundError", + "EndUserNotFoundError", + "FileGrantService", + "GrantTtlTooLongError", + "GrantedFileNotFoundError", + "InvalidFileGrantError", + "InvalidGrantRequestError", + "InvalidSubjectError", + "RemoteFileUnavailableError", + "TooManyFileRefsError", +] diff --git a/api/services/file_service.py b/api/services/file_service.py index 5d1261c0575..418d1e4b2f6 100644 --- a/api/services/file_service.py +++ b/api/services/file_service.py @@ -131,21 +131,32 @@ class FileService: file_size: int, default_file_size_limit: int | None = None, ) -> bool: + return file_size <= FileService.file_size_limit( + extension=extension, + default_file_size_limit=default_file_size_limit, + ) + + @staticmethod + def file_size_limit( + *, + extension: str, + default_file_size_limit: int | None = None, + ) -> int: + """Return the size an extension is allowed, in bytes.""" + if extension in IMAGE_EXTENSIONS: - file_size_limit = dify_config.UPLOAD_IMAGE_FILE_SIZE_LIMIT * 1024 * 1024 + file_size_limit = dify_config.UPLOAD_IMAGE_FILE_SIZE_LIMIT elif extension in VIDEO_EXTENSIONS: - file_size_limit = dify_config.UPLOAD_VIDEO_FILE_SIZE_LIMIT * 1024 * 1024 + file_size_limit = dify_config.UPLOAD_VIDEO_FILE_SIZE_LIMIT elif extension in AUDIO_EXTENSIONS: - file_size_limit = dify_config.UPLOAD_AUDIO_FILE_SIZE_LIMIT * 1024 * 1024 + file_size_limit = dify_config.UPLOAD_AUDIO_FILE_SIZE_LIMIT else: # Context-specific uploads may override the default limit without changing media-specific limits. file_size_limit = ( - (default_file_size_limit if default_file_size_limit is not None else dify_config.UPLOAD_FILE_SIZE_LIMIT) - * 1024 - * 1024 + default_file_size_limit if default_file_size_limit is not None else dify_config.UPLOAD_FILE_SIZE_LIMIT ) - return file_size <= file_size_limit + return file_size_limit * 1024 * 1024 def get_file_base64(self, file_id: str) -> str: with self._session_maker(expire_on_commit=False) as session: diff --git a/api/services/notification_service.py b/api/services/notification_service.py index 13236ef16ed..1d20f01f14a 100644 --- a/api/services/notification_service.py +++ b/api/services/notification_service.py @@ -2,8 +2,8 @@ from typing import Protocol +from constants.languages import languages from machinery.context import RequestContext -from services.account_ports import AccountRepository from services.entities.notification_entities import ( AccountNotification, AccountNotificationBatch, @@ -22,20 +22,15 @@ class NotificationGateway(Protocol): class NotificationService: - def __init__(self, *, accounts: AccountRepository, notifications: NotificationGateway) -> None: - self._accounts = accounts + def __init__(self, *, notifications: NotificationGateway) -> None: self._notifications = notifications - def get_active(self, context: RequestContext) -> NotificationResult: + def get_active(self, context: RequestContext, language: str) -> NotificationResult: batch = self._notifications.get_active(context.account_id) if not batch.should_show: return NotificationResult(should_show=False, notifications=()) - account = self._accounts.get(context.account_id) - if account is None: - raise RuntimeError("Console account admission resolved an unknown account") - language = account.interface_language or _FALLBACK_LANGUAGE - + language = language if language in languages else _FALLBACK_LANGUAGE notifications = tuple(self._localize(notification, language) for notification in batch.notifications) return NotificationResult(should_show=bool(notifications), notifications=notifications) diff --git a/api/services/oauth_device_flow.py b/api/services/oauth_device_flow.py index 9e59b8c326a..f3967026aad 100644 --- a/api/services/oauth_device_flow.py +++ b/api/services/oauth_device_flow.py @@ -10,12 +10,12 @@ import uuid from dataclasses import asdict, dataclass, field from datetime import UTC, datetime, timedelta from enum import StrEnum -from typing import Any, NotRequired, TypedDict +from typing import NotRequired, TypedDict -from sqlalchemy import and_, func, select, update +from sqlalchemy import func, select, update from sqlalchemy.orm import Session -from libs.oauth_bearer import TOKEN_CACHE_KEY_FMT, AuthContext, SubjectType +from libs.oauth_bearer import SubjectType, invalidate_oauth_token_cache from models.oauth import OAuthAccessToken logger = logging.getLogger(__name__) @@ -382,7 +382,7 @@ def mint_oauth_token( ) if outcome.rotated and outcome.old_hash: - redis_client.delete(TOKEN_CACHE_KEY_FMT.format(hash=outcome.old_hash)) + invalidate_oauth_token_cache(redis_client, outcome.old_hash) return MintResult(token=token, token_id=outcome.token_id, expires_at=expires_at) @@ -487,70 +487,3 @@ def oauth_ttl_days(tenant_id: str | None = None) -> int: logger.warning("%s=%d above max %d; clamping", _TTL_ENV_VAR, value, MAX_TTL_DAYS) return MAX_TTL_DAYS return value - - -def subject_match_clauses(ctx: AuthContext) -> tuple[Any, ...]: - if ctx.subject_type == SubjectType.ACCOUNT: - return (OAuthAccessToken.account_id == str(ctx.account_id),) - return ( - OAuthAccessToken.subject_email == ctx.subject_email, - OAuthAccessToken.subject_issuer == ctx.subject_issuer, - OAuthAccessToken.account_id.is_(None), - ) - - -def list_active_sessions(ctx: AuthContext, now: datetime, *, session: Session) -> list[OAuthAccessToken]: - return list( - session.execute( - select(OAuthAccessToken) - .where( - and_( - *subject_match_clauses(ctx), - OAuthAccessToken.revoked_at.is_(None), - OAuthAccessToken.token_hash.is_not(None), - OAuthAccessToken.expires_at > now, - ) - ) - .order_by(OAuthAccessToken.created_at.desc()) - ) - .scalars() - .all() - ) - - -def token_belongs_to_subject(token_id: str, ctx: AuthContext, *, session: Session) -> bool: - row = session.execute( - select(OAuthAccessToken.id).where( - and_( - OAuthAccessToken.id == token_id, - *subject_match_clauses(ctx), - ) - ) - ).first() - return row is not None - - -def revoke_oauth_token(redis_client: Any, token_id: str, *, session: Session) -> None: - row = ( - session.query(OAuthAccessToken.token_hash) - .filter( - OAuthAccessToken.id == token_id, - OAuthAccessToken.revoked_at.is_(None), - ) - .one_or_none() - ) - pre_revoke_hash = row[0] if row else None - - stmt = ( - update(OAuthAccessToken) - .where( - OAuthAccessToken.id == token_id, - OAuthAccessToken.revoked_at.is_(None), - ) - .values(revoked_at=datetime.now(UTC), token_hash=None) - ) - session.execute(stmt) - session.commit() - - if pre_revoke_hash: - redis_client.delete(TOKEN_CACHE_KEY_FMT.format(hash=pre_revoke_hash)) diff --git a/api/services/recommended_app_query_service.py b/api/services/recommended_app_query_service.py index 5b79f40d505..f0da2259797 100644 --- a/api/services/recommended_app_query_service.py +++ b/api/services/recommended_app_query_service.py @@ -5,6 +5,8 @@ from typing import NamedTuple, Protocol from constants.languages import languages +_DEFAULT_LANGUAGE = "en-US" + class RecommendedAppInfoRecord(NamedTuple): id: str @@ -116,10 +118,9 @@ class RecommendedAppQueryService: def list_recommended( self, *, - requested_language: str | None, - interface_language: str | None, + language: str, ) -> RecommendedAppListResult: - language = self._resolve_language(requested_language, interface_language) + language = language if language in languages else _DEFAULT_LANGUAGE page = self._catalog.list_recommended(language) return RecommendedAppListResult( @@ -130,10 +131,9 @@ class RecommendedAppQueryService: def list_learn_dify( self, *, - requested_language: str | None, - interface_language: str | None, + language: str, ) -> LearnDifyAppListResult: - language = self._resolve_language(requested_language, interface_language) + language = language if language in languages else _DEFAULT_LANGUAGE page = self._catalog.list_learn_dify(language) return LearnDifyAppListResult(recommended_apps=self._with_trial_status(page.recommended_apps)) @@ -176,11 +176,3 @@ class RecommendedAppQueryService: ) for app in apps ) - - @staticmethod - def _resolve_language(requested_language: str | None, interface_language: str | None) -> str: - if requested_language and requested_language in languages: - return requested_language - if interface_language: - return interface_language - return languages[0] diff --git a/api/services/skill_management_service.py b/api/services/skill_management_service.py index 92d3de2f2ea..4d4c4299b79 100644 --- a/api/services/skill_management_service.py +++ b/api/services/skill_management_service.py @@ -3747,14 +3747,22 @@ class SkillManagementService: @staticmethod def _strip_single_root(paths: list[str]) -> dict[str, str]: - if not paths: - return {} - first_segments = {path.split("/", 1)[0] for path in paths if "/" in path} - root = next(iter(first_segments)) if len(first_segments) == 1 else None - if root is None or f"{root}/{_SKILL_MD}" not in paths or _SKILL_MD in paths: + if not paths or _SKILL_MD in paths: return {path: path for path in paths} - stripped = {path: path.removeprefix(f"{root}/") for path in paths} - return stripped + # Identify the root by the unique top-level `/SKILL.md`, rather than + # requiring every path to share one first segment: tools like macOS Finder's + # "Compress" add a sibling `__MACOSX/` metadata folder that must not defeat + # stripping of the real skill folder. Entries outside the detected root + # (like `__MACOSX/...`) are dropped rather than passed through, matching + # skill_package_service._normalize_members(ignore_outside_selected_root=True). + skill_md_roots = { + path.split("/", 1)[0] for path in paths if path.count("/") == 1 and path.endswith(f"/{_SKILL_MD}") + } + if len(skill_md_roots) != 1: + return {path: path for path in paths} + root = next(iter(skill_md_roots)) + prefix = f"{root}/" + return {path: path.removeprefix(prefix) for path in paths if path == root or path.startswith(prefix)} def _draft_payload_from_zip( self, @@ -3775,6 +3783,8 @@ class SkillManagementService: skill_md_content = "" for info in infos: raw_path = normalize_skill_file_path(info.filename.strip("/")) + if raw_path not in path_map: + continue path = normalize_skill_file_path(path_map[raw_path]) if info.is_dir(): items.append(SkillDraftTreeItemPayload(path=path, kind=SkillFileKind.DIRECTORY)) diff --git a/api/services/trigger/trigger_subscription_builder_service.py b/api/services/trigger/trigger_subscription_builder_service.py index 0901e5ccf7f..d10d96d2258 100644 --- a/api/services/trigger/trigger_subscription_builder_service.py +++ b/api/services/trigger/trigger_subscription_builder_service.py @@ -267,7 +267,11 @@ class TriggerSubscriptionBuilderService: credentials=subscription_builder.credentials, credential_type=credential_type, credential_expires_at=subscription_builder.credential_expires_at or -1, - expires_at=subscription_builder.expires_at, + expires_at=( + subscription.expires_at + if subscription.expires_at is not None + else (subscription_builder.expires_at or -1) + ), ) # Delete the builder after successful subscription creation diff --git a/api/services/web_passport_gateways.py b/api/services/web_passport_gateways.py index f6cf423302a..774a864e74a 100644 --- a/api/services/web_passport_gateways.py +++ b/api/services/web_passport_gateways.py @@ -10,6 +10,16 @@ from services.enterprise.enterprise_service import PERMISSION_CHECK_MODES, WebAp from services.web_passport_service import WebAppAuthType, WebPassportUnauthorizedError +def resolve_web_app_auth_type(access_mode: str) -> WebAppAuthType: + if access_mode == WebAppAccessMode.PUBLIC: + return WebAppAuthType.PUBLIC + if access_mode in PERMISSION_CHECK_MODES: + return WebAppAuthType.INTERNAL + if access_mode == WebAppAccessMode.SSO_VERIFIED: + return WebAppAuthType.EXTERNAL + raise ValueError(f"Unsupported web app access mode: {access_mode}") + + class DeploymentWebPassportAuthGateway: def __init__( self, @@ -25,13 +35,7 @@ class DeploymentWebPassportAuthGateway: def get_app_auth_type(self, app_id: str) -> WebAppAuthType: access_mode = self._get_app_access_mode(app_id).access_mode - if access_mode == WebAppAccessMode.PUBLIC: - return WebAppAuthType.PUBLIC - if access_mode in PERMISSION_CHECK_MODES: - return WebAppAuthType.INTERNAL - if access_mode == WebAppAccessMode.SSO_VERIFIED: - return WebAppAuthType.EXTERNAL - raise ValueError(f"Unsupported web app access mode: {access_mode}") + return resolve_web_app_auth_type(access_mode) class PassportTokenGateway: diff --git a/api/services/workflow_app_log_query_service.py b/api/services/workflow_app_log_query_service.py new file mode 100644 index 00000000000..4eb1df365d8 --- /dev/null +++ b/api/services/workflow_app_log_query_service.py @@ -0,0 +1,174 @@ +import json +from collections.abc import Mapping +from dataclasses import dataclass, replace +from datetime import datetime +from typing import Any, Protocol + +from core.plugin.plugin_service import PluginService +from graphon.enums import WorkflowExecutionStatus +from models.enums import AppTriggerType +from services.workflow.entities import TriggerMetadata + + +@dataclass(frozen=True, slots=True) +class WorkflowAppLogAccount: + id: str + name: str + email: str + + +@dataclass(frozen=True, slots=True) +class WorkflowAppLogEndUser: + id: str + type: str + is_anonymous: bool + session_id: str | None + + +@dataclass(frozen=True, slots=True) +class WorkflowAppLogRunSummary: + id: str + version: str | None + status: str + triggered_from: str + error: str | None + elapsed_time: float | None + total_tokens: int | None + total_steps: int | None + created_at: datetime | None + finished_at: datetime | None + exceptions_count: int | None + + +@dataclass(frozen=True, slots=True) +class WorkflowAppLogItem: + id: str + workflow_run: WorkflowAppLogRunSummary | None + details: dict[str, Any] | None + created_from: str + created_by_role: str + created_by_account: WorkflowAppLogAccount | None + created_by_end_user: WorkflowAppLogEndUser | None + created_at: datetime + + +@dataclass(frozen=True, slots=True) +class WorkflowAppLogPage: + page: int + limit: int + total: int + has_more: bool + data: tuple[WorkflowAppLogItem, ...] + + +class WorkflowAppLogQuery(Protocol): + def get_paginated( + self, + *, + tenant_id: str, + app_id: str, + keyword: str | None = None, + status: WorkflowExecutionStatus | None = None, + created_at_before: datetime | None = None, + created_at_after: datetime | None = None, + page: int = 1, + limit: int = 20, + detail: bool = False, + created_by_end_user_session_id: str | None = None, + created_by_account: str | None = None, + ) -> WorkflowAppLogPage: ... + + +class WorkflowAppLogQueryService: + def __init__( + self, + *, + logs: WorkflowAppLogQuery, + ) -> None: + self._logs = logs + + def list_logs( + self, + *, + tenant_id: str, + app_id: str, + keyword: str | None = None, + status: WorkflowExecutionStatus | None = None, + created_at_before: datetime | None = None, + created_at_after: datetime | None = None, + page: int = 1, + limit: int = 20, + detail: bool = False, + created_by_end_user_session_id: str | None = None, + created_by_account: str | None = None, + ) -> WorkflowAppLogPage: + result = self._logs.get_paginated( + tenant_id=tenant_id, + app_id=app_id, + keyword=keyword, + status=status, + created_at_before=created_at_before, + created_at_after=created_at_after, + page=page, + limit=limit, + detail=detail, + created_by_end_user_session_id=created_by_end_user_session_id, + created_by_account=created_by_account, + ) + + return WorkflowAppLogPage( + page=result.page, + limit=result.limit, + total=result.total, + has_more=result.has_more, + data=tuple(self._resolve_details(tenant_id=tenant_id, item=item) for item in result.data), + ) + + def _resolve_details( + self, + *, + tenant_id: str, + item: WorkflowAppLogItem, + ) -> WorkflowAppLogItem: + if item.details is None: + return item + + return replace( + item, + details={ + "trigger_metadata": self._resolve_trigger_metadata( + tenant_id, + item.details.get("trigger_metadata"), + ) + }, + ) + + @staticmethod + def _resolve_trigger_metadata( + tenant_id: str, + value: str | Mapping[str, Any] | None, + ) -> dict[str, Any]: + metadata = WorkflowAppLogQueryService._safe_json_loads(value) + if not metadata: + return {} + + trigger_metadata = TriggerMetadata.model_validate(metadata) + if trigger_metadata.type == AppTriggerType.TRIGGER_PLUGIN: + icon = metadata.get("icon_filename") + icon_dark = metadata.get("icon_dark_filename") + metadata["icon"] = PluginService.get_plugin_icon_url(tenant_id=tenant_id, filename=icon) if icon else None + metadata["icon_dark"] = ( + PluginService.get_plugin_icon_url(tenant_id=tenant_id, filename=icon_dark) if icon_dark else None + ) + return metadata + + @staticmethod + def _safe_json_loads(value: Any) -> Any: + if not value: + return None + if isinstance(value, str): + try: + return json.loads(value) + except Exception: + return None + return value diff --git a/api/services/workflow_app_service.py b/api/services/workflow_app_service.py deleted file mode 100644 index 453c130e1aa..00000000000 --- a/api/services/workflow_app_service.py +++ /dev/null @@ -1,231 +0,0 @@ -import json -import uuid -from datetime import datetime -from typing import Any, TypedDict - -from sqlalchemy import and_, func, or_, select -from sqlalchemy.orm import Session - -from core.plugin.plugin_service import PluginService -from graphon.enums import WorkflowExecutionStatus -from models import Account, App, EndUser, TenantAccountJoin, WorkflowAppLog, WorkflowRun -from models.enums import AppTriggerType, CreatorUserRole -from models.trigger import WorkflowTriggerLog -from services.workflow.entities import TriggerMetadata - - -class LogViewDetails(TypedDict): - trigger_metadata: dict[str, Any] | None - - -# Since the workflow_app_log table has exceeded 100 million records, we use an additional details field to extend it -class LogView: - """Lightweight wrapper for WorkflowAppLog with computed details. - - - Exposes `details_` for marshalling to `details` in API response - - Resolves the account/end-user accessors through the session it was built with - - Proxies all other attributes to the underlying `WorkflowAppLog` - """ - - def __init__(self, log: WorkflowAppLog, details: LogViewDetails | None, session: Session): - self.log = log - self.details_ = details - self._session = session - - @property - def details(self) -> LogViewDetails | None: - return self.details_ - - @property - def created_by_account(self) -> Account | None: - return self.log.created_by_account(self._session) - - @property - def created_by_end_user(self) -> EndUser | None: - return self.log.created_by_end_user(self._session) - - def __getattr__(self, name): - return getattr(self.log, name) - - -class WorkflowAppService: - def get_paginate_workflow_app_logs( - self, - *, - session: Session, - app_model: App, - keyword: str | None = None, - status: WorkflowExecutionStatus | None = None, - created_at_before: datetime | None = None, - created_at_after: datetime | None = None, - page: int = 1, - limit: int = 20, - detail: bool = False, - created_by_end_user_session_id: str | None = None, - created_by_account: str | None = None, - ): - """ - Get paginate workflow app logs using SQLAlchemy 2.0 style - :param session: SQLAlchemy session - :param app_model: app model - :param keyword: search keyword - :param status: filter by status - :param created_at_before: filter logs created before this timestamp - :param created_at_after: filter logs created after this timestamp - :param page: page number - :param limit: items per page - :param detail: whether to return detailed logs - :param created_by_end_user_session_id: filter by end user session id - :param created_by_account: filter by account email - :return: Pagination object - """ - # Build base statement using SQLAlchemy 2.0 style - stmt = select(WorkflowAppLog).where( - WorkflowAppLog.tenant_id == app_model.tenant_id, WorkflowAppLog.app_id == app_model.id - ) - - if detail: - # Simple left join by workflow_run_id to fetch trigger_metadata - stmt = stmt.outerjoin( - WorkflowTriggerLog, - and_( - WorkflowTriggerLog.tenant_id == app_model.tenant_id, - WorkflowTriggerLog.app_id == app_model.id, - WorkflowTriggerLog.workflow_run_id == WorkflowAppLog.workflow_run_id, - ), - ).add_columns(WorkflowTriggerLog.trigger_metadata) - - if keyword or status: - stmt = stmt.join(WorkflowRun, WorkflowRun.id == WorkflowAppLog.workflow_run_id) - # Join to workflow run for filtering when needed. - - if keyword: - from libs.helper import escape_like_pattern - - # Escape special characters in keyword to prevent SQL injection via LIKE wildcards - escaped_keyword = escape_like_pattern(keyword[:30]) - keyword_like_val = f"%{escaped_keyword}%" - keyword_conditions = [ - WorkflowRun.inputs.ilike(keyword_like_val, escape="\\"), - WorkflowRun.outputs.ilike(keyword_like_val, escape="\\"), - # filter keyword by end user session id if created by end user role - and_( - WorkflowRun.created_by_role == "end_user", - EndUser.session_id.ilike(keyword_like_val, escape="\\"), - ), - ] - - # filter keyword by workflow run id - keyword_uuid = self._safe_parse_uuid(keyword) - if keyword_uuid: - keyword_conditions.append(WorkflowRun.id == keyword_uuid) - - stmt = stmt.outerjoin( - EndUser, - and_(WorkflowRun.created_by == EndUser.id, WorkflowRun.created_by_role == CreatorUserRole.END_USER), - ).where(or_(*keyword_conditions)) - - if status: - stmt = stmt.where(WorkflowRun.status == status) - - # Add time-based filtering - if created_at_before: - stmt = stmt.where(WorkflowAppLog.created_at <= created_at_before) - - if created_at_after: - stmt = stmt.where(WorkflowAppLog.created_at >= created_at_after) - - # Filter by end user session id or account email - if created_by_end_user_session_id: - stmt = stmt.join( - EndUser, - and_( - WorkflowAppLog.created_by == EndUser.id, - WorkflowAppLog.created_by_role == CreatorUserRole.END_USER, - EndUser.session_id == created_by_end_user_session_id, - ), - ) - if created_by_account: - account = session.scalar( - select(Account) - .join(TenantAccountJoin, TenantAccountJoin.account_id == Account.id) - .where( - Account.email == created_by_account, - TenantAccountJoin.tenant_id == app_model.tenant_id, - ) - ) - if not account: - raise ValueError(f"Account not found: {created_by_account}") - - stmt = stmt.join( - Account, - and_( - WorkflowAppLog.created_by == Account.id, - WorkflowAppLog.created_by_role == CreatorUserRole.ACCOUNT, - Account.id == account.id, - ), - ) - - stmt = stmt.order_by(WorkflowAppLog.created_at.desc()) - - # Get total count using the same filters - count_stmt = select(func.count()).select_from(stmt.subquery()) - total = session.scalar(count_stmt) or 0 - - # Apply pagination limits - offset_stmt = stmt.offset((page - 1) * limit).limit(limit) - - # wrapper moved to module scope as `LogView` - - # Execute query and get items - if detail: - rows = session.execute(offset_stmt).all() - items = [ - LogView(log, {"trigger_metadata": self.handle_trigger_metadata(app_model.tenant_id, meta_val)}, session) - for log, meta_val in rows - ] - else: - items = [LogView(log, None, session) for log in session.scalars(offset_stmt).all()] - return { - "page": page, - "limit": limit, - "total": total, - "has_more": total > page * limit, - "data": items, - } - - def handle_trigger_metadata(self, tenant_id: str, meta_val: str | None) -> dict[str, Any]: - metadata: dict[str, Any] | None = self._safe_json_loads(meta_val) - if not metadata: - return {} - trigger_metadata = TriggerMetadata.model_validate(metadata) - if trigger_metadata.type == AppTriggerType.TRIGGER_PLUGIN: - icon = metadata.get("icon_filename") - icon_dark = metadata.get("icon_dark_filename") - metadata["icon"] = PluginService.get_plugin_icon_url(tenant_id=tenant_id, filename=icon) if icon else None - metadata["icon_dark"] = ( - PluginService.get_plugin_icon_url(tenant_id=tenant_id, filename=icon_dark) if icon_dark else None - ) - return metadata - - @staticmethod - def _safe_json_loads(val): - if not val: - return None - if isinstance(val, str): - try: - return json.loads(val) - except Exception: - return None - return val - - @staticmethod - def _safe_parse_uuid(value: str): - # fast check - if len(value) < 32: - return None - - try: - return uuid.UUID(value) - except ValueError: - return None diff --git a/api/services/workflow_run_service.py b/api/services/workflow_run_service.py index 8618b343ae6..dac000065cd 100644 --- a/api/services/workflow_run_service.py +++ b/api/services/workflow_run_service.py @@ -1,22 +1,19 @@ import threading +from dataclasses import dataclass +from datetime import datetime from typing import TypedDict -from sqlalchemy import Engine, select -from sqlalchemy.orm import sessionmaker - import contexts -from extensions.ext_database import db +from core.workflow.nodes.human_input.pause_reason import HumanInputRequired +from graphon.enums import WorkflowExecutionStatus from libs.infinite_scroll_pagination import InfiniteScrollPagination +from machinery.context import RequestContext from models import ( - Account, - App, - EndUser, - Message, WorkflowRun, WorkflowRunTriggeredFrom, ) -from repositories.api_workflow_run_repository import APIWorkflowRunRepository -from repositories.factory import DifyAPIRepositoryFactory +from repositories.api_workflow_node_execution_repository import DifyAPIWorkflowNodeExecutionRepository +from repositories.sqlalchemy_api_workflow_run_repository import DifyAPISQLAlchemyWorkflowRunRepository from services.workflow_node_execution_trace_service import ( WorkflowNodeExecutionTrace, assemble_workflow_node_execution_traces, @@ -31,34 +28,43 @@ class WorkflowRunListArgs(TypedDict, total=False): status: str +@dataclass(frozen=True, slots=True) +class WorkflowRunPausedNode: + node_id: str + node_title: str + form_id: str + form_token: str | None + + +@dataclass(frozen=True, slots=True) +class WorkflowRunPauseDetails: + paused_at: datetime | None + paused_nodes: tuple[WorkflowRunPausedNode, ...] + + class WorkflowRunService: - _session_factory: sessionmaker - _workflow_run_repo: APIWorkflowRunRepository - - def __init__(self, session_factory: Engine | sessionmaker | None = None): - """Initialize WorkflowRunService with repository dependencies.""" - match session_factory: - case None: - session_factory = sessionmaker(bind=db.engine, expire_on_commit=False) - case Engine(): - session_factory = sessionmaker(bind=session_factory, expire_on_commit=False) - - self._session_factory = session_factory - self._node_execution_service_repo = DifyAPIRepositoryFactory.create_api_workflow_node_execution_repository( - self._session_factory - ) - self._workflow_run_repo = DifyAPIRepositoryFactory.create_api_workflow_run_repository(self._session_factory) + def __init__( + self, + *, + workflow_runs: DifyAPISQLAlchemyWorkflowRunRepository, + node_executions: DifyAPIWorkflowNodeExecutionRepository, + ) -> None: + self._workflow_runs = workflow_runs + self._node_executions = node_executions def get_paginate_advanced_chat_workflow_runs( self, - app_model: App, + context: RequestContext, + *, + app_id: str, args: WorkflowRunListArgs, triggered_from: WorkflowRunTriggeredFrom = WorkflowRunTriggeredFrom.DEBUGGING, ) -> InfiniteScrollPagination: """ Get advanced chat app workflow run list - :param app_model: app model + :param context: admitted Console request context + :param app_id: app id :param args: request args :param triggered_from: workflow run triggered from (default: DEBUGGING for preview runs) """ @@ -73,35 +79,29 @@ class WorkflowRunService: def __getattr__(self, item): return getattr(self._workflow_run, item) - pagination = self.get_paginate_workflow_runs(app_model, args, triggered_from) + pagination = self.get_paginate_workflow_runs( + context, + app_id=app_id, + args=args, + triggered_from=triggered_from, + ) # Batch-load the associated Message for every run in a single query to avoid # an N+1 pattern: the deprecated WorkflowRun.message property issues one query # per run. The filter matches that property exactly (app_id + workflow_run_id). workflow_runs = pagination.data run_ids = [workflow_run.id for workflow_run in workflow_runs] - messages_by_run_id: dict[str, Message] = {} - if run_ids: - with self._session_factory() as session: - messages = session.scalars( - select(Message).where( - Message.app_id == app_model.id, - Message.workflow_run_id.in_(run_ids), - ) - ).all() - for loaded_message in messages: - run_id = loaded_message.workflow_run_id - if run_id is None: - continue - # setdefault mirrors scalar()'s single-row-per-run semantics. - messages_by_run_id.setdefault(run_id, loaded_message) + messages_by_run_id = self._workflow_runs.get_message_refs( + app_id=app_id, + workflow_run_ids=run_ids, + ) with_message_workflow_runs = [] for workflow_run in workflow_runs: message = messages_by_run_id.get(workflow_run.id) with_message_workflow_run = WorkflowWithMessage(workflow_run=workflow_run) if message: - with_message_workflow_run.message_id = message.id + with_message_workflow_run.message_id = message.message_id with_message_workflow_run.conversation_id = message.conversation_id with_message_workflow_runs.append(with_message_workflow_run) @@ -111,14 +111,17 @@ class WorkflowRunService: def get_paginate_workflow_runs( self, - app_model: App, + context: RequestContext, + *, + app_id: str, args: WorkflowRunListArgs, triggered_from: WorkflowRunTriggeredFrom = WorkflowRunTriggeredFrom.DEBUGGING, ) -> InfiniteScrollPagination: """ Get workflow run list - :param app_model: app model + :param context: admitted Console request context + :param app_id: app id :param args: request args :param triggered_from: workflow run triggered from (default: DEBUGGING) """ @@ -126,31 +129,34 @@ class WorkflowRunService: last_id = args.get("last_id") status = args.get("status") - return self._workflow_run_repo.get_paginated_workflow_runs( - tenant_id=app_model.tenant_id, - app_id=app_model.id, + return self._workflow_runs.get_paginated_workflow_runs( + tenant_id=context.active_workspace_id, + app_id=app_id, triggered_from=triggered_from, limit=limit, last_id=last_id, status=status, ) - def get_workflow_run(self, app_model: App, run_id: str) -> WorkflowRun | None: + def get_workflow_run(self, context: RequestContext, *, app_id: str, run_id: str) -> WorkflowRun | None: """ Get workflow run detail - :param app_model: app model + :param context: admitted Console request context + :param app_id: app id :param run_id: workflow run id """ - return self._workflow_run_repo.get_workflow_run_by_id( - tenant_id=app_model.tenant_id, - app_id=app_model.id, + return self._workflow_runs.get_workflow_run_by_id( + tenant_id=context.active_workspace_id, + app_id=app_id, run_id=run_id, ) def get_workflow_runs_count( self, - app_model: App, + context: RequestContext, + *, + app_id: str, status: str | None = None, time_range: str | None = None, triggered_from: WorkflowRunTriggeredFrom = WorkflowRunTriggeredFrom.DEBUGGING, @@ -158,15 +164,16 @@ class WorkflowRunService: """ Get workflow runs count statistics - :param app_model: app model + :param context: admitted Console request context + :param app_id: app id :param status: optional status filter :param time_range: optional time range filter (e.g., "7d", "4h", "30m", "30s") :param triggered_from: workflow run triggered from (default: DEBUGGING) :return: dict with total and status counts """ - return self._workflow_run_repo.get_workflow_runs_count( - tenant_id=app_model.tenant_id, - app_id=app_model.id, + return self._workflow_runs.get_workflow_runs_count( + tenant_id=context.active_workspace_id, + app_id=app_id, triggered_from=triggered_from, status=status, time_range=time_range, @@ -174,14 +181,15 @@ class WorkflowRunService: def get_workflow_run_node_executions( self, - app_model: App, + context: RequestContext, + *, + app_id: str, run_id: str, - user: Account | EndUser, ) -> list[WorkflowNodeExecutionTrace]: """ Get workflow run node execution list """ - workflow_run = self.get_workflow_run(app_model, run_id) + workflow_run = self.get_workflow_run(context, app_id=app_id, run_id=run_id) contexts.plugin_tool_providers.set({}) contexts.plugin_tool_providers_lock.set(threading.Lock()) @@ -189,14 +197,43 @@ class WorkflowRunService: if not workflow_run: return [] - # Get tenant_id from user - tenant_id = user.tenant_id if isinstance(user, EndUser) else user.current_tenant_id - if tenant_id is None: - raise ValueError("User tenant_id cannot be None") - - node_executions = self._node_execution_service_repo.get_executions_by_workflow_run( - tenant_id=tenant_id, - app_id=app_model.id, + node_executions = self._node_executions.get_executions_by_workflow_run( + tenant_id=context.active_workspace_id, + app_id=app_id, workflow_run_id=run_id, ) - return assemble_workflow_node_execution_traces(node_executions, self._node_execution_service_repo) + return assemble_workflow_node_execution_traces(node_executions, self._node_executions) + + def get_pause_details( + self, + context: RequestContext, + *, + workflow_run_id: str, + ) -> WorkflowRunPauseDetails | None: + pause_record = self._workflow_runs.get_pause_record( + workspace_id=context.active_workspace_id, + workflow_run_id=workflow_run_id, + ) + if pause_record is None: + return None + if pause_record.status != WorkflowExecutionStatus.PAUSED: + return WorkflowRunPauseDetails(paused_at=None, paused_nodes=()) + + human_input_reasons: list[HumanInputRequired] = [] + for reason in pause_record.reasons: + if not isinstance(reason, HumanInputRequired): + raise NotImplementedError(f"Pause details do not support {type(reason).__name__}") + human_input_reasons.append(reason) + + return WorkflowRunPauseDetails( + paused_at=pause_record.paused_at, + paused_nodes=tuple( + WorkflowRunPausedNode( + node_id=reason.node_id, + node_title=reason.node_title, + form_id=reason.form_id, + form_token=pause_record.form_tokens.get(reason.form_id), + ) + for reason in human_input_reasons + ), + ) diff --git a/api/tests/test_containers_integration_tests/controllers/openapi/conftest.py b/api/tests/test_containers_integration_tests/controllers/openapi/conftest.py index d06da5bc018..2047ee765d1 100644 --- a/api/tests/test_containers_integration_tests/controllers/openapi/conftest.py +++ b/api/tests/test_containers_integration_tests/controllers/openapi/conftest.py @@ -1,8 +1,7 @@ from __future__ import annotations import uuid -from collections.abc import Callable, Generator -from contextlib import contextmanager +from collections.abc import Callable from typing import Literal from unittest.mock import patch @@ -12,7 +11,8 @@ from flask import Flask from sqlalchemy.orm import Session from controllers.openapi.auth.data import AuthData -from libs.oauth_bearer import AuthContext, Scope, SubjectType, TokenType, reset_auth_ctx, set_auth_ctx +from libs.oauth_bearer import Scope, TokenType +from machinery.context import AccountRequestContext from models import Account, Tenant from services.account_service import AccountService, TenantService from tests.test_containers_integration_tests.helpers import generate_valid_password @@ -91,34 +91,14 @@ def auth_for( ) -@contextmanager -def account_auth_context( +def request_context_for( account: Account, *, - token_id: uuid.UUID, - client_id: str = "integration-cli", -) -> Generator[AuthContext]: - """Publish an account ``AuthContext`` for handlers that read ``get_auth_ctx()``. - - The auth pipeline normally sets this ContextVar; the integration suite - bypasses the pipeline via ``inspect.unwrap``, so endpoints that resolve the - caller through ``get_auth_ctx()`` (the ``/account/sessions*`` family) need it - set explicitly. Resets on exit so the worker thread can't leak identity. - """ - ctx = AuthContext( - subject_type=SubjectType.ACCOUNT, - subject_email=account.email, - subject_issuer=None, - account_id=uuid.UUID(str(account.id)), - client_id=client_id, - scopes=frozenset({Scope.FULL}), - token_id=token_id, - token_type=TokenType.OAUTH_ACCOUNT, - expires_at=None, - token_hash="integration-test", + token_id: uuid.UUID | None = None, +) -> AccountRequestContext: + return AccountRequestContext( + request_id="integration-request", + trace_id="integration-trace", + account_id=str(account.id), + access_token_id=str(token_id) if token_id is not None else None, ) - reset_token = set_auth_ctx(ctx) - try: - yield ctx - finally: - reset_auth_ctx(reset_token) diff --git a/api/tests/test_containers_integration_tests/controllers/openapi/test_account.py b/api/tests/test_containers_integration_tests/controllers/openapi/test_account.py index 0a04c4fdad0..d2181b8398d 100644 --- a/api/tests/test_containers_integration_tests/controllers/openapi/test_account.py +++ b/api/tests/test_containers_integration_tests/controllers/openapi/test_account.py @@ -9,20 +9,21 @@ from sqlalchemy.orm import Session from controllers.openapi.account import AccountApi from models import Account from models.account import TenantAccountRole -from tests.test_containers_integration_tests.controllers.openapi.conftest import add_tenant_for_account, auth_for +from tests.test_containers_integration_tests.controllers.openapi.conftest import ( + add_tenant_for_account, + request_context_for, +) class TestAccountInfo: - def test_returns_account_and_owner_workspace( - self, app: Flask, db_session_with_containers: Session, make_account: Callable[..., Account] - ) -> None: + def test_returns_account_and_owner_workspace(self, app: Flask, make_account: Callable[..., Account]) -> None: account = make_account() owner_tenant = account.current_tenant assert owner_tenant is not None api = AccountApi() with app.test_request_context("/openapi/v1/account"): - result = unwrap(api.get)(api, db_session_with_containers, auth_data=auth_for(account)) + result = unwrap(api.get)(api, request_context_for(account)) assert result.subject_type == "account" assert result.subject_email == account.email @@ -47,7 +48,7 @@ class TestAccountInfo: api = AccountApi() with app.test_request_context("/openapi/v1/account"): - result = unwrap(api.get)(api, db_session_with_containers, auth_data=auth_for(account)) + result = unwrap(api.get)(api, request_context_for(account)) assert {w.id for w in result.workspaces} == {owner_tenant.id, second.id} roles = {w.id: w.role for w in result.workspaces} diff --git a/api/tests/test_containers_integration_tests/controllers/openapi/test_account_sessions.py b/api/tests/test_containers_integration_tests/controllers/openapi/test_account_sessions.py index 9e6acab5543..d2e339c565e 100644 --- a/api/tests/test_containers_integration_tests/controllers/openapi/test_account_sessions.py +++ b/api/tests/test_containers_integration_tests/controllers/openapi/test_account_sessions.py @@ -18,7 +18,7 @@ from controllers.openapi.account import ( from extensions.ext_redis import redis_client from models import Account from services.oauth_device_flow import PREFIX_OAUTH_ACCOUNT, MintResult, mint_oauth_token -from tests.test_containers_integration_tests.controllers.openapi.conftest import account_auth_context, auth_for +from tests.test_containers_integration_tests.controllers.openapi.conftest import request_context_for def _mint_account_token( @@ -51,13 +51,11 @@ class TestSessionList: api = AccountSessionsApi() with app.test_request_context("/openapi/v1/account/sessions"): - with account_auth_context(account, token_id=mint.token_id): - result = unwrap(api.get)( - api, - db_session_with_containers, - auth_data=auth_for(account, token_id=mint.token_id), - query=SessionListQuery(), - ) + result = unwrap(api.get)( + api, + request_context_for(account, token_id=mint.token_id), + query=SessionListQuery(), + ) assert result.total == 1 row = result.data[0] @@ -76,13 +74,11 @@ class TestSessionList: api = AccountSessionsApi() with app.test_request_context("/openapi/v1/account/sessions"): - with account_auth_context(account, token_id=mine.token_id): - result = unwrap(api.get)( - api, - db_session_with_containers, - auth_data=auth_for(account, token_id=mine.token_id), - query=SessionListQuery(), - ) + result = unwrap(api.get)( + api, + request_context_for(account, token_id=mine.token_id), + query=SessionListQuery(), + ) assert {row.id for row in result.data} == {str(mine.token_id)} @@ -96,23 +92,18 @@ class TestSessionRevoke: revoke_api = AccountSessionsSelfApi() with app.test_request_context("/openapi/v1/account/sessions/self", method="DELETE"): - with account_auth_context(account, token_id=mint.token_id): - result = unwrap(revoke_api.delete)( - revoke_api, db_session_with_containers, auth_data=auth_for(account, token_id=mint.token_id) - ) + result = unwrap(revoke_api.delete)(revoke_api, request_context_for(account, token_id=mint.token_id)) assert result.status == "revoked" # Revocation persisted: the real list path no longer returns it. list_api = AccountSessionsApi() with app.test_request_context("/openapi/v1/account/sessions"): - with account_auth_context(account, token_id=mint.token_id): - listing = unwrap(list_api.get)( - list_api, - db_session_with_containers, - auth_data=auth_for(account, token_id=mint.token_id), - query=SessionListQuery(), - ) + listing = unwrap(list_api.get)( + list_api, + request_context_for(account, token_id=mint.token_id), + query=SessionListQuery(), + ) assert listing.total == 0 def test_revoke_by_id_for_own_session( @@ -124,13 +115,11 @@ class TestSessionRevoke: api = AccountSessionByIdApi() with app.test_request_context(f"/openapi/v1/account/sessions/{session_id}", method="DELETE"): - with account_auth_context(account, token_id=mint.token_id): - result = unwrap(api.delete)( - api, - db_session_with_containers, - session_id=session_id, - auth_data=auth_for(account, token_id=mint.token_id), - ) + result = unwrap(api.delete)( + api, + request_context_for(account, token_id=mint.token_id), + session_id=session_id, + ) assert result.status == "revoked" @@ -146,11 +135,9 @@ class TestSessionRevoke: api = AccountSessionByIdApi() session_id = str(foreign.token_id) with app.test_request_context(f"/openapi/v1/account/sessions/{session_id}", method="DELETE"): - with account_auth_context(outsider, token_id=uuid4()): - with pytest.raises(NotFound): - unwrap(api.delete)( - api, - db_session_with_containers, - session_id=session_id, - auth_data=auth_for(outsider, token_id=uuid4()), - ) + with pytest.raises(NotFound): + unwrap(api.delete)( + api, + request_context_for(outsider, token_id=uuid4()), + session_id=session_id, + ) diff --git a/api/tests/test_containers_integration_tests/controllers/web/test_human_input_form.py b/api/tests/test_containers_integration_tests/controllers/web/test_human_input_form.py index 363197e6c74..d80e3fe31ff 100644 --- a/api/tests/test_containers_integration_tests/controllers/web/test_human_input_form.py +++ b/api/tests/test_containers_integration_tests/controllers/web/test_human_input_form.py @@ -2,7 +2,6 @@ from __future__ import annotations import json from datetime import UTC, datetime, timedelta -from typing import override from uuid import uuid4 import pytest @@ -22,7 +21,6 @@ from core.workflow.nodes.human_input.entities import ( ) from core.workflow.nodes.human_input.enums import HumanInputFormKind, HumanInputFormStatus, ValueSourceType from core.workflow.nodes.human_input.pause_reason import HumanInputRequired -from graphon.entities import WorkflowExecution from graphon.enums import WorkflowExecutionStatus from graphon.runtime import GraphRuntimeState, VariablePool from models.account import Account, Tenant, TenantAccountJoin, TenantAccountRole @@ -40,14 +38,6 @@ from repositories.sqlalchemy_api_workflow_run_repository import DifyAPISQLAlchem from services.entities.feature_entities import FeatureModel -class _TestWorkflowRunRepository(DifyAPISQLAlchemyWorkflowRunRepository): - """Concrete repository for tests where save() is not under test.""" - - @override - def save(self, execution: WorkflowExecution) -> None: - return None - - def _create_app_with_site(session: Session) -> tuple[App, Account]: tenant = Tenant(name="Test Tenant") account = Account(name="Tester", email=f"tester-{uuid4()}@example.com") @@ -218,7 +208,9 @@ def test_get_human_input_form_resolves_runtime_select_options( ) engine = db_session_with_containers.get_bind() assert isinstance(engine, Engine) - workflow_run_repo = _TestWorkflowRunRepository(session_maker=sessionmaker(bind=engine, expire_on_commit=False)) + workflow_run_repo = DifyAPISQLAlchemyWorkflowRunRepository( + session_maker=sessionmaker(bind=engine, expire_on_commit=False) + ) workflow_run_repo.create_workflow_pause( workflow_run_id=workflow_run.id, state_owner_user_id=account.id, diff --git a/api/tests/test_containers_integration_tests/controllers/web/test_wraps.py b/api/tests/test_containers_integration_tests/controllers/web/test_wraps.py index 9a143ab3bc3..5bda3738af5 100644 --- a/api/tests/test_containers_integration_tests/controllers/web/test_wraps.py +++ b/api/tests/test_containers_integration_tests/controllers/web/test_wraps.py @@ -91,8 +91,8 @@ class TestValidateUserAccessibility: def test_missing_auth_type_raises(self) -> None: decoded = {"user_id": "u1", "granted_at": 1} - settings = SimpleNamespace(access_mode="public") - with pytest.raises(WebAppAuthAccessDeniedError, match="auth_type"): + settings = SimpleNamespace(access_mode="private") + with pytest.raises(WebAppAuthRequiredError, match="auth_type"): _validate_user_accessibility( decoded=decoded, app_code="code", @@ -103,7 +103,7 @@ class TestValidateUserAccessibility: def test_missing_granted_at_raises(self) -> None: decoded = {"user_id": "u1", "auth_type": "external"} - settings = SimpleNamespace(access_mode="public") + settings = SimpleNamespace(access_mode="sso_verified") with pytest.raises(WebAppAuthAccessDeniedError, match="granted_at"): _validate_user_accessibility( decoded=decoded, @@ -121,7 +121,7 @@ class TestValidateUserAccessibility: mock_sso_time.return_value = datetime.now(UTC) old_granted = int((datetime.now(UTC) - timedelta(hours=1)).timestamp()) decoded = {"user_id": "u1", "auth_type": "external", "granted_at": old_granted} - settings = SimpleNamespace(access_mode="public") + settings = SimpleNamespace(access_mode="sso_verified") with pytest.raises(WebAppAuthAccessDeniedError, match="SSO settings"): _validate_user_accessibility( decoded=decoded, @@ -139,7 +139,7 @@ class TestValidateUserAccessibility: mock_workspace_sso.return_value = datetime.now(UTC) old_granted = int((datetime.now(UTC) - timedelta(hours=1)).timestamp()) decoded = {"user_id": "u1", "auth_type": "internal", "granted_at": old_granted} - settings = SimpleNamespace(access_mode="public") + settings = SimpleNamespace(access_mode="private") with pytest.raises(WebAppAuthAccessDeniedError, match="SSO settings"): _validate_user_accessibility( decoded=decoded, @@ -157,7 +157,7 @@ class TestValidateUserAccessibility: mock_sso_time.return_value = datetime.now(UTC) - timedelta(hours=2) recent_granted = int(datetime.now(UTC).timestamp()) decoded = {"user_id": "u1", "auth_type": "external", "granted_at": recent_granted} - settings = SimpleNamespace(access_mode="public") + settings = SimpleNamespace(access_mode="sso_verified") _validate_user_accessibility( decoded=decoded, app_code="code", @@ -172,8 +172,8 @@ class TestValidateUserAccessibility: def test_permission_check_denies_unauthorized_user( self, mock_perm: MagicMock, mock_app_id: MagicMock, mock_allowed: MagicMock ) -> None: - decoded = {"user_id": "u1", "auth_type": "external", "granted_at": int(datetime.now(UTC).timestamp())} - settings = SimpleNamespace(access_mode="internal") + decoded = {"user_id": "u1", "auth_type": "internal", "granted_at": int(datetime.now(UTC).timestamp())} + settings = SimpleNamespace(access_mode="private") with pytest.raises(WebAppAuthAccessDeniedError): _validate_user_accessibility( decoded=decoded, @@ -183,6 +183,37 @@ class TestValidateUserAccessibility: webapp_settings=settings, ) + @pytest.mark.parametrize( + ("access_mode", "auth_type"), + [ + pytest.param("private", "external", id="private-rejects-external"), + pytest.param("private_all", "external", id="private-all-rejects-external"), + pytest.param("sso_verified", "internal", id="sso-verified-rejects-internal"), + ], + ) + @patch("controllers.web.wraps.EnterpriseService.WebAppAuth.is_user_allowed_to_access_webapp") + @patch("controllers.web.wraps.WebAppAuthService.is_app_require_permission_check") + def test_auth_type_must_match_current_access_mode( + self, + mock_permission_check: MagicMock, + mock_allowed: MagicMock, + access_mode: str, + auth_type: str, + ) -> None: + decoded = {"user_id": "u1", "auth_type": auth_type, "granted_at": int(datetime.now(UTC).timestamp())} + + with pytest.raises(WebAppAuthRequiredError): + _validate_user_accessibility( + decoded=decoded, + app_code="code", + app_web_auth_enabled=True, + system_webapp_auth_enabled=True, + webapp_settings=SimpleNamespace(access_mode=access_mode), + ) + + mock_permission_check.assert_not_called() + mock_allowed.assert_not_called() + class TestDecodeJwtToken: @pytest.fixture diff --git a/api/tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py b/api/tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py index 2d5e3ebcca9..56da1c948d1 100644 --- a/api/tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py +++ b/api/tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py @@ -23,7 +23,7 @@ from unittest.mock import Mock import pytest from sqlalchemy import Engine, delete, select -from sqlalchemy.orm import Session +from sqlalchemy.orm import Session, sessionmaker from core.app.app_config.entities import WorkflowUIBasedAppConfig from core.app.entities.app_invoke_entities import InvokeFrom, WorkflowAppGenerateEntity @@ -46,6 +46,8 @@ from models import Account from models import WorkflowPause as WorkflowPauseModel from models.model import AppMode, UploadFile from models.workflow import Workflow, WorkflowRun +from repositories.factory import DifyAPIRepositoryFactory +from repositories.sqlalchemy_api_workflow_run_repository import DifyAPISQLAlchemyWorkflowRunRepository from services.file_service import FileService from services.workflow_run_service import WorkflowRunService @@ -99,7 +101,14 @@ class TestPauseStatePersistenceLayerTestContainers: @pytest.fixture def workflow_run_service(self, engine: Engine, file_service: FileService): """Create WorkflowRunService instance with TestContainers engine and FileService.""" - return WorkflowRunService(engine) + session_factory = sessionmaker(bind=engine, expire_on_commit=False) + workflow_runs = DifyAPISQLAlchemyWorkflowRunRepository(session_maker=session_factory) + return WorkflowRunService( + workflow_runs=workflow_runs, + node_executions=DifyAPIRepositoryFactory.create_api_workflow_node_execution_repository( + session_maker=session_factory + ), + ) @pytest.fixture(autouse=True) def setup_test_data(self, db_session_with_containers: Session, file_service, workflow_run_service): @@ -403,7 +412,7 @@ class TestPauseStatePersistenceLayerTestContainers: layer.on_event(event) # Assert - Retrieve and verify - pause_entity = self.workflow_run_service._workflow_run_repo.get_workflow_pause(self.test_workflow_run_id) + pause_entity = self.workflow_run_service._workflow_runs.get_workflow_pause(self.test_workflow_run_id) assert pause_entity is not None assert pause_entity.workflow_execution_id == self.test_workflow_run_id assert pause_entity.get_pause_reasons() == event.reasons @@ -542,7 +551,7 @@ class TestPauseStatePersistenceLayerTestContainers: assert pause_model is not None # Verify the state owner is the workflow creator - pause_entity = self.workflow_run_service._workflow_run_repo.get_workflow_pause(different_workflow_run.id) + pause_entity = self.workflow_run_service._workflow_runs.get_workflow_pause(different_workflow_run.id) assert pause_entity is not None resumption_context = WorkflowResumptionContext.loads(pause_entity.get_state().decode()) assert resumption_context.get_generate_entity().workflow_execution_id == different_workflow_run.id diff --git a/api/tests/test_containers_integration_tests/models/test_account.py b/api/tests/test_containers_integration_tests/models/test_account.py index 1f1c4a4ede1..676bf013bcd 100644 --- a/api/tests/test_containers_integration_tests/models/test_account.py +++ b/api/tests/test_containers_integration_tests/models/test_account.py @@ -8,7 +8,6 @@ Also absorbs unit_tests/models/test_account.py role helper coverage. Covers: - Account.current_tenant setter (sets _current_tenant and role from TenantAccountJoin) - Account.set_tenant_id (resolves tenant + role from real join row) -- Account.get_by_openid (AccountIntegrate lookup then Account fetch) - Tenant.get_accounts (returns accounts linked via TenantAccountJoin) """ @@ -20,9 +19,9 @@ import pytest from sqlalchemy import delete from sqlalchemy.orm import Session -from models.account import Account, AccountIntegrate, Tenant, TenantAccountJoin, TenantAccountRole +from models.account import Account, Tenant, TenantAccountJoin, TenantAccountRole -TrackedRow = Account | AccountIntegrate | Tenant | TenantAccountJoin +TrackedRow = Account | Tenant | TenantAccountJoin def _cleanup_tracked_rows(db_session: Session, tracked: list[TrackedRow]) -> None: @@ -172,37 +171,6 @@ class TestAccountSetTenantId(_DBTrackingTestBase): assert account._current_tenant is None -class TestAccountGetByOpenId(_DBTrackingTestBase): - """Integration tests for Account.get_by_openid class method.""" - - def test_get_by_openid_returns_account_when_integrate_exists(self, db_session_with_containers: Session) -> None: - """get_by_openid returns the Account when a matching AccountIntegrate row exists.""" - account = self._create_account(db_session_with_containers, email_prefix="openid") - provider = "google" - open_id = f"google_{uuid4()}" - - integrate = AccountIntegrate( - account_id=account.id, - provider=provider, - open_id=open_id, - encrypted_token="token", - ) - db_session_with_containers.add(integrate) - db_session_with_containers.flush() - self._tracked.append(integrate) - - result = Account.get_by_openid(provider, open_id) - - assert result is not None - assert result.id == account.id - - def test_get_by_openid_returns_none_when_no_integrate_exists(self) -> None: - """get_by_openid returns None when no AccountIntegrate row matches.""" - result = Account.get_by_openid("github", f"github_{uuid4()}") - - assert result is None - - class TestTenantGetAccounts(_DBTrackingTestBase): """Integration tests for Tenant.get_accounts method.""" diff --git a/api/tests/test_containers_integration_tests/repositories/test_sqlalchemy_workflow_run_cleanup_repository.py b/api/tests/test_containers_integration_tests/repositories/test_sqlalchemy_workflow_run_cleanup_repository.py index be659fac184..7fc557f9b11 100644 --- a/api/tests/test_containers_integration_tests/repositories/test_sqlalchemy_workflow_run_cleanup_repository.py +++ b/api/tests/test_containers_integration_tests/repositories/test_sqlalchemy_workflow_run_cleanup_repository.py @@ -4,13 +4,11 @@ from __future__ import annotations from dataclasses import dataclass, field from datetime import datetime, timedelta -from typing import override from uuid import uuid4 from sqlalchemy import Engine, select from sqlalchemy.orm import Session, sessionmaker -from graphon.entities import WorkflowExecution from graphon.entities.pause_reason import PauseReasonType from graphon.enums import WorkflowExecutionStatus, WorkflowType from models.enums import CreatorUserRole, WorkflowRunTriggeredFrom @@ -18,14 +16,6 @@ from models.workflow import WorkflowAppLog, WorkflowAppLogCreatedFrom, WorkflowP from repositories.sqlalchemy_api_workflow_run_repository import DifyAPISQLAlchemyWorkflowRunRepository -class _TestWorkflowRunRepository(DifyAPISQLAlchemyWorkflowRunRepository): - """Concrete repository for tests where save() is not under test.""" - - @override - def save(self, execution: WorkflowExecution) -> None: - return None - - @dataclass class _TestScope: """Per-test identifiers for rows created by cleanup repository tests.""" @@ -39,7 +29,7 @@ class _TestScope: def _repository(db_session_with_containers: Session) -> DifyAPISQLAlchemyWorkflowRunRepository: engine = db_session_with_containers.get_bind() assert isinstance(engine, Engine) - return _TestWorkflowRunRepository(session_maker=sessionmaker(bind=engine, expire_on_commit=False)) + return DifyAPISQLAlchemyWorkflowRunRepository(session_maker=sessionmaker(bind=engine, expire_on_commit=False)) def _create_workflow_run( diff --git a/api/tests/test_containers_integration_tests/services/test_account_oauth_identity_lock.py b/api/tests/test_containers_integration_tests/services/test_account_oauth_identity_lock.py new file mode 100644 index 00000000000..e0c40d40576 --- /dev/null +++ b/api/tests/test_containers_integration_tests/services/test_account_oauth_identity_lock.py @@ -0,0 +1,64 @@ +"""Redis-backed integration coverage for Console OAuth account-claim leases.""" + +import time +from hashlib import sha256 +from uuid import uuid4 + +import pytest + +from extensions.ext_redis import redis_client +from services import account_oauth_adapters +from services.account_errors import OAuthIdentityLockUnavailableError +from services.account_oauth_adapters import RedisOAuthAccountClaimLock + + +@pytest.mark.usefixtures("flask_app_with_containers") +def test_account_claim_locks_remain_exclusive_beyond_their_initial_ttl(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(account_oauth_adapters, "_OAUTH_ACCOUNT_CLAIM_LOCK_TIMEOUT_SECONDS", 0.5) + monkeypatch.setattr(account_oauth_adapters, "_OAUTH_ACCOUNT_CLAIM_LOCK_RENEW_INTERVAL_SECONDS", 0.1) + monkeypatch.setattr(account_oauth_adapters, "_OAUTH_ACCOUNT_CLAIM_LOCK_BLOCKING_TIMEOUT_SECONDS", 0.2) + provider = "github" + open_id = f"identity-{uuid4().hex}" + email = f"account-{uuid4().hex}@example.com" + identity_digest = sha256("\0".join(("identity", provider, open_id)).encode()).hexdigest() + email_digest = sha256("\0".join(("email", email)).encode()).hexdigest() + lock_names = [ + f"oauth:account-claim:{identity_digest}", + f"oauth:account-claim:{email_digest}", + ] + account_claims = RedisOAuthAccountClaimLock(client=redis_client) + contenders = [ + redis_client.lock(lock_name, timeout=1, blocking=False, thread_local=False) for lock_name in lock_names + ] + + with account_claims.acquire(provider=provider, open_id=open_id, email=email): + time.sleep(1.2) + assert all(contender.acquire(blocking=False) is False for contender in contenders) + + for contender in contenders: + assert contender.acquire(blocking=False) is True + contender.release() + + +@pytest.mark.usefixtures("flask_app_with_containers") +def test_different_providers_with_the_same_email_contend_for_one_claim(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(account_oauth_adapters, "_OAUTH_ACCOUNT_CLAIM_LOCK_BLOCKING_TIMEOUT_SECONDS", 0.1) + email = f"account-{uuid4().hex}@example.com" + account_claims = RedisOAuthAccountClaimLock(client=redis_client) + + with account_claims.acquire(provider="github", open_id=f"github-{uuid4().hex}", email=email): + with pytest.raises(OAuthIdentityLockUnavailableError): + with account_claims.acquire(provider="google", open_id=f"google-{uuid4().hex}", email=email): + raise AssertionError("same-email claim body must not run concurrently") + + +@pytest.mark.usefixtures("flask_app_with_containers") +def test_final_account_claim_serializes_different_provider_identities(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(account_oauth_adapters, "_OAUTH_ACCOUNT_CLAIM_LOCK_BLOCKING_TIMEOUT_SECONDS", 0.1) + account_claims = RedisOAuthAccountClaimLock(client=redis_client) + account_id = f"account-{uuid4().hex}" + + with account_claims.acquire_account(account_id): + with pytest.raises(OAuthIdentityLockUnavailableError): + with account_claims.acquire_account(account_id): + raise AssertionError("same-account claim body must not run concurrently") diff --git a/api/tests/test_containers_integration_tests/services/test_account_service.py b/api/tests/test_containers_integration_tests/services/test_account_service.py index e2a0ff364d0..e6b8dab1c5a 100644 --- a/api/tests/test_containers_integration_tests/services/test_account_service.py +++ b/api/tests/test_containers_integration_tests/services/test_account_service.py @@ -280,86 +280,6 @@ class TestAccountService: session=db_session_with_containers, ) - def test_link_account_integrate_new_provider( - self, db_session_with_containers: Session, mock_external_service_dependencies - ): - """ - Test linking account with new OAuth provider. - """ - fake = Faker() - email = fake.email() - name = fake.name() - # Setup mocks - mock_external_service_dependencies["feature_service"].is_registration_allowed.return_value = True - mock_external_service_dependencies["billing_service"].is_email_in_freeze.return_value = False - - # Create account - account = AccountService.create_account( - email=email, - name=name, - interface_language="en-US", - password=None, - session=db_session_with_containers, - ) - - # Link with new provider - AccountService.link_account_integrate( - "new-google", "google_open_id_123", account, session=db_session_with_containers - ) - - # Verify integration was created - from models import AccountIntegrate - - integration = ( - db_session_with_containers.query(AccountIntegrate) - .filter_by(account_id=account.id, provider="new-google") - .first() - ) - assert integration is not None - assert integration.open_id == "google_open_id_123" - - def test_link_account_integrate_existing_provider( - self, db_session_with_containers: Session, mock_external_service_dependencies - ): - """ - Test linking account with existing provider (should update). - """ - fake = Faker() - email = fake.email() - name = fake.name() - # Setup mocks - mock_external_service_dependencies["feature_service"].is_registration_allowed.return_value = True - mock_external_service_dependencies["billing_service"].is_email_in_freeze.return_value = False - - # Create account - account = AccountService.create_account( - email=email, - name=name, - interface_language="en-US", - password=None, - session=db_session_with_containers, - ) - - # Link with provider first time - AccountService.link_account_integrate( - "exists-google", "google_open_id_123", account, session=db_session_with_containers - ) - - # Link with same provider but different open_id (should update) - AccountService.link_account_integrate( - "exists-google", "google_open_id_456", account, session=db_session_with_containers - ) - - # Verify integration was updated - from models import AccountIntegrate - - integration = ( - db_session_with_containers.query(AccountIntegrate) - .filter_by(account_id=account.id, provider="exists-google") - .first() - ) - assert integration.open_id == "google_open_id_456" - def test_update_login_info(self, db_session_with_containers: Session, mock_external_service_dependencies): """ Test updating login information. @@ -1967,52 +1887,6 @@ class TestRegisterService: assert account.current_tenant is not None assert account.current_tenant.name == f"{name}'s Workspace" - def test_register_with_oauth(self, db_session_with_containers: Session, mock_external_service_dependencies): - """ - Test account registration with OAuth integration. - """ - fake = Faker() - email = fake.email() - name = fake.name() - open_id = fake.uuid4() - provider = fake.random_element(elements=("google", "github", "microsoft")) - language = fake.random_element(elements=("en-US", "zh-CN")) - # Setup mocks - mock_external_service_dependencies["feature_service"].is_registration_allowed.return_value = True - mock_external_service_dependencies["feature_service"].is_workspace_creation_allowed.return_value = True - mock_external_service_dependencies[ - "feature_service" - ].get_system_features.return_value.license.workspaces.is_available.return_value = True - mock_external_service_dependencies["billing_service"].is_email_in_freeze.return_value = False - - # Execute registration with OAuth - account = RegisterService.register( - email=email, - name=name, - password=None, - open_id=open_id, - provider=provider, - language=language, - session=db_session_with_containers, - ) - - # Verify account was created - assert account.email == email - assert account.name == name - assert account.status == "active" - assert account.initialized_at is not None - - # Verify OAuth integration was created - from models import AccountIntegrate - - integration = ( - db_session_with_containers.query(AccountIntegrate) - .filter_by(account_id=account.id, provider=provider) - .first() - ) - assert integration is not None - assert integration.open_id == open_id - def test_register_with_pending_status( self, db_session_with_containers: Session, mock_external_service_dependencies ): @@ -2510,52 +2384,6 @@ class TestRegisterService: assert invitation_data["email"] == account.email assert invitation_data["workspace_id"] == tenant.id - def test_is_valid_invite_token_valid(self, db_session_with_containers: Session, mock_external_service_dependencies): - """ - Test validation of valid invite token. - """ - fake = Faker() - tenant_name = fake.company() - email = fake.email() - name = fake.name() - password = generate_valid_password(fake) - # Setup mocks - mock_external_service_dependencies["feature_service"].is_registration_allowed.return_value = True - mock_external_service_dependencies["billing_service"].is_email_in_freeze.return_value = False - - # Create tenant and account - tenant = TenantService.create_tenant(name=tenant_name, session=db_session_with_containers) - account = AccountService.create_account( - email=email, - name=name, - interface_language="en-US", - password=password, - session=db_session_with_containers, - ) - - # Generate a real token - token = RegisterService.generate_invite_token(tenant, account) - - # Execute validation - is_valid = RegisterService.is_valid_invite_token(token) - - # Verify token is valid - assert is_valid is True - - def test_is_valid_invite_token_invalid( - self, db_session_with_containers: Session, mock_external_service_dependencies - ): - """ - Test validation of invalid invite token. - """ - fake = Faker() - invalid_token = fake.uuid4() - # Execute validation with non-existent token - is_valid = RegisterService.is_valid_invite_token(invalid_token) - - # Verify token is invalid - assert is_valid is False - def test_revoke_token_with_workspace_and_email( self, db_session_with_containers: Session, mock_external_service_dependencies ): diff --git a/api/tests/test_containers_integration_tests/services/test_workflow_app_service.py b/api/tests/test_containers_integration_tests/services/test_workflow_app_service.py index 617c27f41a6..e31cf7d80ad 100644 --- a/api/tests/test_containers_integration_tests/services/test_workflow_app_service.py +++ b/api/tests/test_containers_integration_tests/services/test_workflow_app_service.py @@ -7,22 +7,48 @@ from unittest.mock import patch import pytest from faker import Faker -from sqlalchemy.orm import Session +from sqlalchemy.orm import Session, sessionmaker from graphon.enums import WorkflowExecutionStatus from models import EndUser, Workflow, WorkflowAppLog, WorkflowRun from models.enums import CreatorUserRole, EndUserType from models.workflow import WorkflowAppLogCreatedFrom +from repositories.workflow_app_log_query_repository import WorkflowAppLogQueryRepository from services.account_service import AccountService, TenantService - -# Delay import of AppService to avoid circular dependency -# from services.app_service import AppService, CreateAppParams -from services.workflow_app_service import WorkflowAppService +from services.workflow_app_log_query_service import WorkflowAppLogQueryService from tests.test_containers_integration_tests.helpers import generate_valid_password -class TestWorkflowAppService: - """Integration tests for WorkflowAppService using testcontainers.""" +class _WorkflowAppLogTestClient: + def __init__(self, session: Session) -> None: + session_factory = sessionmaker(bind=session.get_bind(), expire_on_commit=False) + self._service = WorkflowAppLogQueryService( + logs=WorkflowAppLogQueryRepository(session_factory=session_factory), + ) + + def get_paginate_workflow_app_logs(self, *, session: Session, app_model, **kwargs): + assert session.get_bind() is not None + result = self._service.list_logs( + tenant_id=app_model.tenant_id, + app_id=app_model.id, + **kwargs, + ) + return { + "page": result.page, + "limit": result.limit, + "total": result.total, + "has_more": result.has_more, + "data": list(result.data), + } + + +def _workflow_run(log): + assert log.workflow_run is not None + return log.workflow_run + + +class TestWorkflowAppLogQueryService: + """Integration tests for workflow app log queries using testcontainers.""" @pytest.fixture def mock_external_service_dependencies(self): @@ -247,7 +273,7 @@ class TestWorkflowAppService: ) # Act: Execute the method under test - service = WorkflowAppService() + service = _WorkflowAppLogTestClient(db_session_with_containers) result = service.get_paginate_workflow_app_logs( session=db_session_with_containers, app_model=app, page=1, limit=20 ) @@ -263,10 +289,8 @@ class TestWorkflowAppService: # Verify the returned data log_entry = result["data"][0] assert log_entry.id == workflow_app_log.id - assert log_entry.tenant_id == app.tenant_id - assert log_entry.app_id == app.id - assert log_entry.workflow_id == workflow.id - assert log_entry.workflow_run_id == workflow_run.id + returned_run = _workflow_run(log_entry) + assert returned_run.id == workflow_run.id # Verify database state @@ -293,7 +317,7 @@ class TestWorkflowAppService: db_session_with_containers.commit() # Act: Execute the method under test with keyword search - service = WorkflowAppService() + service = _WorkflowAppLogTestClient(db_session_with_containers) result = service.get_paginate_workflow_app_logs( session=db_session_with_containers, app_model=app, keyword="test_keyword", page=1, limit=20 ) @@ -305,7 +329,7 @@ class TestWorkflowAppService: # Verify the returned data contains the searched keyword log_entry = result["data"][0] - assert log_entry.workflow_run_id == workflow_run.id + assert _workflow_run(log_entry).id == workflow_run.id # Test with non-matching keyword result_no_match = service.get_paginate_workflow_app_logs( @@ -331,7 +355,7 @@ class TestWorkflowAppService: app, account = self._create_test_app_and_account(db_session_with_containers, mock_external_service_dependencies) workflow, _, _ = self._create_test_workflow_data(db_session_with_containers, app, account) - service = WorkflowAppService() + service = _WorkflowAppLogTestClient(db_session_with_containers) # Test 1: Search with % character workflow_run_1 = WorkflowRun( @@ -373,7 +397,7 @@ class TestWorkflowAppService: # Should find the workflow_run_1 entry assert result["total"] >= 1 assert len(result["data"]) >= 1 - assert any(log.workflow_run_id == workflow_run_1.id for log in result["data"]) + assert any(_workflow_run(log).id == workflow_run_1.id for log in result["data"]) # Test 2: Search with _ character workflow_run_2 = WorkflowRun( @@ -415,7 +439,7 @@ class TestWorkflowAppService: # Should find the workflow_run_2 entry assert result["total"] >= 1 assert len(result["data"]) >= 1 - assert any(log.workflow_run_id == workflow_run_2.id for log in result["data"]) + assert any(_workflow_run(log).id == workflow_run_2.id for log in result["data"]) # Test 3: Search with % should NOT match 100% (verifies escaping works correctly) workflow_run_4 = WorkflowRun( @@ -459,7 +483,7 @@ class TestWorkflowAppService: assert result["total"] >= 1 assert len(result["data"]) >= 1 # Verify that we found workflow_run_1 (50% discount) but not workflow_run_4 (100% different) - found_run_ids = [log.workflow_run_id for log in result["data"]] + found_run_ids = [_workflow_run(log).id for log in result["data"]] assert workflow_run_1.id in found_run_ids assert workflow_run_4.id not in found_run_ids @@ -535,7 +559,7 @@ class TestWorkflowAppService: workflow_app_logs.append(workflow_app_log) # Act & Assert: Test filtering by different statuses - service = WorkflowAppService() + service = _WorkflowAppLogTestClient(db_session_with_containers) # Test succeeded status filter result_succeeded = service.get_paginate_workflow_app_logs( @@ -641,7 +665,7 @@ class TestWorkflowAppService: workflow_app_logs.append(workflow_app_log) # Act & Assert: Test time-based filtering - service = WorkflowAppService() + service = _WorkflowAppLogTestClient(db_session_with_containers) # Test filtering logs created after 2 hours ago result_after = service.get_paginate_workflow_app_logs( @@ -746,7 +770,7 @@ class TestWorkflowAppService: workflow_app_logs.append(workflow_app_log) # Act & Assert: Test pagination - service = WorkflowAppService() + service = _WorkflowAppLogTestClient(db_session_with_containers) # Test first page with limit 10 result_page1 = service.get_paginate_workflow_app_logs( @@ -916,7 +940,7 @@ class TestWorkflowAppService: workflow_app_logs.append(workflow_app_log) # Act & Assert: Test user role filtering - service = WorkflowAppService() + service = _WorkflowAppLogTestClient(db_session_with_containers) # Test filtering by end user session ID result_session_filter = service.get_paginate_workflow_app_logs( @@ -1048,14 +1072,14 @@ class TestWorkflowAppService: db_session_with_containers.commit() # Act & Assert: Test UUID keyword search - service = WorkflowAppService() + service = _WorkflowAppLogTestClient(db_session_with_containers) # Test searching by workflow run UUID result_uuid_search = service.get_paginate_workflow_app_logs( session=db_session_with_containers, app_model=app, keyword=workflow_run_id, page=1, limit=20 ) assert result_uuid_search["total"] == 1 - assert result_uuid_search["data"][0].workflow_run_id == workflow_run_id + assert _workflow_run(result_uuid_search["data"][0]).id == workflow_run_id # Test searching by partial UUID (should not match) partial_uuid = workflow_run_id[:8] @@ -1136,7 +1160,7 @@ class TestWorkflowAppService: db_session_with_containers.commit() # Act & Assert: Test edge cases - service = WorkflowAppService() + service = _WorkflowAppLogTestClient(db_session_with_containers) # Test with page 1 (normal case) result_page_one = service.get_paginate_workflow_app_logs( @@ -1180,7 +1204,7 @@ class TestWorkflowAppService: app, account = self._create_test_app_and_account(db_session_with_containers, mock_external_service_dependencies) # Act & Assert: Test empty results - service = WorkflowAppService() + service = _WorkflowAppLogTestClient(db_session_with_containers) # Test with no workflow logs result_no_logs = service.get_paginate_workflow_app_logs( @@ -1291,7 +1315,7 @@ class TestWorkflowAppService: db_session_with_containers.commit() - service = WorkflowAppService() + service = _WorkflowAppLogTestClient(db_session_with_containers) # Test complex combination: keyword + status + time range + pagination result_complex = service.get_paginate_workflow_app_logs( @@ -1391,7 +1415,7 @@ class TestWorkflowAppService: db_session_with_containers.commit() - service = WorkflowAppService() + service = _WorkflowAppLogTestClient(db_session_with_containers) # Test performance with large dataset and pagination import time @@ -1442,14 +1466,18 @@ class TestWorkflowAppService: db_session_with_containers, mock_external_service_dependencies ) app1 = self._create_test_app(db_session_with_containers, tenant1, account1) - workflow1, _, _ = self._create_test_workflow_data(db_session_with_containers, app1, account1) + workflow1, workflow_run1, _ = self._create_test_workflow_data(db_session_with_containers, app1, account1) # Create second tenant and app tenant2, account2 = self._create_test_tenant_and_account( db_session_with_containers, mock_external_service_dependencies ) app2 = self._create_test_app(db_session_with_containers, tenant2, account2) - workflow2, _, _ = self._create_test_workflow_data(db_session_with_containers, app2, account2) + workflow2, workflow_run2, _ = self._create_test_workflow_data(db_session_with_containers, app2, account2) + run_ids_by_app = { + app1.id: {workflow_run1.id}, + app2.id: {workflow_run2.id}, + } # Create logs for both tenants for i, (app, workflow, account) in enumerate([(app1, workflow1, account1), (app2, workflow2, account2)]): @@ -1476,6 +1504,7 @@ class TestWorkflowAppService: ) db_session_with_containers.add(workflow_run) db_session_with_containers.flush() + run_ids_by_app[app.id].add(workflow_run.id) log = WorkflowAppLog( tenant_id=app.tenant_id, @@ -1492,7 +1521,7 @@ class TestWorkflowAppService: db_session_with_containers.commit() - service = WorkflowAppService() + service = _WorkflowAppLogTestClient(db_session_with_containers) # Test tenant isolation: tenant1 should only see its own logs result_tenant1 = service.get_paginate_workflow_app_logs( @@ -1500,9 +1529,7 @@ class TestWorkflowAppService: ) assert result_tenant1["total"] == 4 # 3 new logs + 1 from _create_test_workflow_data - for log in result_tenant1["data"]: - assert log.tenant_id == app1.tenant_id - assert log.app_id == app1.id + assert {_workflow_run(log).id for log in result_tenant1["data"]} == run_ids_by_app[app1.id] # Test tenant isolation: tenant2 should only see its own logs result_tenant2 = service.get_paginate_workflow_app_logs( @@ -1510,9 +1537,7 @@ class TestWorkflowAppService: ) assert result_tenant2["total"] == 4 # 3 new logs + 1 from _create_test_workflow_data - for log in result_tenant2["data"]: - assert log.tenant_id == app2.tenant_id - assert log.app_id == app2.id + assert {_workflow_run(log).id for log in result_tenant2["data"]} == run_ids_by_app[app2.id] # Test cross-tenant search should not work result_cross_tenant = service.get_paginate_workflow_app_logs( @@ -1530,7 +1555,7 @@ class TestWorkflowAppService: self, db_session_with_containers: Session, mock_external_service_dependencies ): app, account = self._create_test_app_and_account(db_session_with_containers, mock_external_service_dependencies) - service = WorkflowAppService() + service = _WorkflowAppLogTestClient(db_session_with_containers) with pytest.raises(ValueError, match="Account not found: nonexistent@example.com"): service.get_paginate_workflow_app_logs( @@ -1543,7 +1568,7 @@ class TestWorkflowAppService: self, db_session_with_containers: Session, mock_external_service_dependencies ): app, account = self._create_test_app_and_account(db_session_with_containers, mock_external_service_dependencies) - service = WorkflowAppService() + service = _WorkflowAppLogTestClient(db_session_with_containers) workflow, workflow_run, _log = self._create_test_workflow_data(db_session_with_containers, app, account) result = service.get_paginate_workflow_app_logs( diff --git a/api/tests/test_containers_integration_tests/services/test_workflow_run_service.py b/api/tests/test_containers_integration_tests/services/test_workflow_run_service.py index 263f41e5144..8d37aaa4e8c 100644 --- a/api/tests/test_containers_integration_tests/services/test_workflow_run_service.py +++ b/api/tests/test_containers_integration_tests/services/test_workflow_run_service.py @@ -5,13 +5,17 @@ from unittest.mock import patch import pytest from faker import Faker -from sqlalchemy.orm import Session +from sqlalchemy import Engine +from sqlalchemy.orm import Session, sessionmaker -from models.enums import ConversationFromSource, CreatorUserRole, EndUserType +from machinery.context import RequestContext +from models.enums import ConversationFromSource, CreatorUserRole from models.model import ( Message, ) from models.workflow import WorkflowRun +from repositories.factory import DifyAPIRepositoryFactory +from repositories.sqlalchemy_api_workflow_run_repository import DifyAPISQLAlchemyWorkflowRunRepository from services.account_service import AccountService, TenantService from services.app_service import AppService, CreateAppParams from services.workflow_run_service import WorkflowRunService @@ -21,6 +25,19 @@ from tests.test_containers_integration_tests.helpers import generate_valid_passw class TestWorkflowRunService: """Integration tests for WorkflowRunService using testcontainers.""" + @pytest.fixture + def workflow_run_service(self, db_session_with_containers: Session) -> WorkflowRunService: + engine = db_session_with_containers.get_bind() + assert isinstance(engine, Engine) + session_factory = sessionmaker(bind=engine, expire_on_commit=False) + workflow_runs = DifyAPISQLAlchemyWorkflowRunRepository(session_maker=session_factory) + return WorkflowRunService( + workflow_runs=workflow_runs, + node_executions=DifyAPIRepositoryFactory.create_api_workflow_node_execution_repository( + session_maker=session_factory + ), + ) + @pytest.fixture def mock_external_service_dependencies(self): """Mock setup for external service dependencies.""" @@ -50,6 +67,15 @@ class TestWorkflowRunService: "account_feature_service": mock_account_feature_service, } + @staticmethod + def _request_context(app, account) -> RequestContext: + return RequestContext( + request_id="request-1", + trace_id="trace-1", + account_id=account.id, + active_workspace_id=app.tenant_id, + ) + def _create_test_app_and_account(self, db_session_with_containers: Session, mock_external_service_dependencies): """ Helper method to create a test app and account for testing. @@ -196,7 +222,10 @@ class TestWorkflowRunService: return message def test_get_paginate_workflow_runs_success( - self, db_session_with_containers: Session, mock_external_service_dependencies + self, + db_session_with_containers: Session, + mock_external_service_dependencies, + workflow_run_service: WorkflowRunService, ): """ Test successful pagination of workflow runs with debugging trigger. @@ -218,9 +247,12 @@ class TestWorkflowRunService: workflow_runs.append(workflow_run) # Act: Execute the method under test - workflow_run_service = WorkflowRunService() args = {"limit": 3, "last_id": None} - result = workflow_run_service.get_paginate_workflow_runs(app, args) + result = workflow_run_service.get_paginate_workflow_runs( + self._request_context(app, account), + app_id=app.id, + args=args, + ) # Assert: Verify the expected outcomes assert result is not None @@ -238,7 +270,10 @@ class TestWorkflowRunService: assert workflow_run.tenant_id == app.tenant_id def test_get_paginate_workflow_runs_with_last_id( - self, db_session_with_containers: Session, mock_external_service_dependencies + self, + db_session_with_containers: Session, + mock_external_service_dependencies, + workflow_run_service: WorkflowRunService, ): """ Test pagination of workflow runs with last_id parameter. @@ -261,9 +296,12 @@ class TestWorkflowRunService: workflow_runs.append(workflow_run) # Act: Execute the method under test with last_id - workflow_run_service = WorkflowRunService() args = {"limit": 2, "last_id": workflow_runs[1].id} - result = workflow_run_service.get_paginate_workflow_runs(app, args) + result = workflow_run_service.get_paginate_workflow_runs( + self._request_context(app, account), + app_id=app.id, + args=args, + ) # Assert: Verify the expected outcomes assert result is not None @@ -281,7 +319,10 @@ class TestWorkflowRunService: assert workflow_run.tenant_id == app.tenant_id def test_get_paginate_workflow_runs_default_limit( - self, db_session_with_containers: Session, mock_external_service_dependencies + self, + db_session_with_containers: Session, + mock_external_service_dependencies, + workflow_run_service: WorkflowRunService, ): """ Test pagination of workflow runs with default limit. @@ -299,9 +340,12 @@ class TestWorkflowRunService: workflow_run = self._create_test_workflow_run(db_session_with_containers, app, account, "debugging") # Act: Execute the method under test without limit - workflow_run_service = WorkflowRunService() args = {} # No limit specified - result = workflow_run_service.get_paginate_workflow_runs(app, args) + result = workflow_run_service.get_paginate_workflow_runs( + self._request_context(app, account), + app_id=app.id, + args=args, + ) # Assert: Verify the expected outcomes assert result is not None @@ -319,7 +363,10 @@ class TestWorkflowRunService: assert workflow_run_result.tenant_id == app.tenant_id def test_get_paginate_advanced_chat_workflow_runs_success( - self, db_session_with_containers: Session, mock_external_service_dependencies + self, + db_session_with_containers: Session, + mock_external_service_dependencies, + workflow_run_service: WorkflowRunService, ): """ Test successful pagination of advanced chat workflow runs with message information. @@ -344,9 +391,12 @@ class TestWorkflowRunService: workflow_runs.append(workflow_run) # Act: Execute the method under test - workflow_run_service = WorkflowRunService() args = {"limit": 2, "last_id": None} - result = workflow_run_service.get_paginate_advanced_chat_workflow_runs(app, args) + result = workflow_run_service.get_paginate_advanced_chat_workflow_runs( + self._request_context(app, account), + app_id=app.id, + args=args, + ) # Assert: Verify the expected outcomes assert result is not None @@ -364,7 +414,12 @@ class TestWorkflowRunService: assert workflow_run.app_id == app.id assert workflow_run.tenant_id == app.tenant_id - def test_get_workflow_run_success(self, db_session_with_containers: Session, mock_external_service_dependencies): + def test_get_workflow_run_success( + self, + db_session_with_containers: Session, + mock_external_service_dependencies, + workflow_run_service: WorkflowRunService, + ): """ Test successful retrieval of workflow run by ID. @@ -381,8 +436,11 @@ class TestWorkflowRunService: workflow_run = self._create_test_workflow_run(db_session_with_containers, app, account, "debugging") # Act: Execute the method under test - workflow_run_service = WorkflowRunService() - result = workflow_run_service.get_workflow_run(app, workflow_run.id) + result = workflow_run_service.get_workflow_run( + self._request_context(app, account), + app_id=app.id, + run_id=workflow_run.id, + ) # Assert: Verify the expected outcomes assert result is not None @@ -394,7 +452,12 @@ class TestWorkflowRunService: assert result.type == "chat" assert result.version == "1.0.0" - def test_get_workflow_run_not_found(self, db_session_with_containers: Session, mock_external_service_dependencies): + def test_get_workflow_run_not_found( + self, + db_session_with_containers: Session, + mock_external_service_dependencies, + workflow_run_service: WorkflowRunService, + ): """ Test workflow run retrieval when run ID does not exist. @@ -411,14 +474,20 @@ class TestWorkflowRunService: non_existent_id = str(uuid.uuid4()) # Act: Execute the method under test - workflow_run_service = WorkflowRunService() - result = workflow_run_service.get_workflow_run(app, non_existent_id) + result = workflow_run_service.get_workflow_run( + self._request_context(app, account), + app_id=app.id, + run_id=non_existent_id, + ) # Assert: Verify the expected outcomes assert result is None def test_get_workflow_run_node_executions_success( - self, db_session_with_containers: Session, mock_external_service_dependencies + self, + db_session_with_containers: Session, + mock_external_service_dependencies, + workflow_run_service: WorkflowRunService, ): """ Test successful retrieval of workflow run node executions. @@ -487,8 +556,11 @@ class TestWorkflowRunService: db_session_with_containers.commit() # Act: Execute the method under test - workflow_run_service = WorkflowRunService() - result = workflow_run_service.get_workflow_run_node_executions(app, workflow_run.id, account) + result = workflow_run_service.get_workflow_run_node_executions( + self._request_context(app, account), + app_id=app.id, + run_id=workflow_run.id, + ) # Assert: Verify the expected outcomes assert result is not None @@ -507,7 +579,10 @@ class TestWorkflowRunService: assert node_execution.node_id.startswith("node_") def test_get_workflow_run_node_executions_empty( - self, db_session_with_containers: Session, mock_external_service_dependencies + self, + db_session_with_containers: Session, + mock_external_service_dependencies, + workflow_run_service: WorkflowRunService, ): """ Test getting node executions for a workflow run with no executions. @@ -521,7 +596,6 @@ class TestWorkflowRunService: account_service = AccountService() tenant_service = TenantService() app_service = AppService() - workflow_run_service = WorkflowRunService() # Create account and tenant account = account_service.create_account( @@ -549,9 +623,9 @@ class TestWorkflowRunService: # Act: Get node executions result = workflow_run_service.get_workflow_run_node_executions( - app_model=app, + self._request_context(app, account), + app_id=app.id, run_id=workflow_run.id, - user=account, ) # Assert: Verify empty result @@ -559,7 +633,10 @@ class TestWorkflowRunService: assert len(result) == 0 def test_get_workflow_run_node_executions_invalid_workflow_run_id( - self, db_session_with_containers: Session, mock_external_service_dependencies + self, + db_session_with_containers: Session, + mock_external_service_dependencies, + workflow_run_service: WorkflowRunService, ): """ Test getting node executions with invalid workflow run ID. @@ -573,7 +650,6 @@ class TestWorkflowRunService: account_service = AccountService() tenant_service = TenantService() app_service = AppService() - workflow_run_service = WorkflowRunService() # Create account and tenant account = account_service.create_account( @@ -601,137 +677,11 @@ class TestWorkflowRunService: # Act: Get node executions with invalid ID result = workflow_run_service.get_workflow_run_node_executions( - app_model=app, + self._request_context(app, account), + app_id=app.id, run_id=invalid_workflow_run_id, - user=account, ) # Assert: Verify empty result assert result is not None assert len(result) == 0 - - def test_get_workflow_run_node_executions_database_error( - self, db_session_with_containers: Session, mock_external_service_dependencies - ): - """ - Test getting node executions when database encounters an error. - - This test verifies: - - Proper error handling when database operations fail - - Graceful degradation in error scenarios - - Error propagation to calling code - """ - # Arrange: Setup test data - account_service = AccountService() - tenant_service = TenantService() - app_service = AppService() - workflow_run_service = WorkflowRunService() - - # Create account and tenant - account = account_service.create_account( - email="test@example.com", - name="Test User", - password="password123", - interface_language="en-US", - session=db_session_with_containers, - ) - TenantService.create_owner_tenant_if_not_exist(account, name="test_tenant", session=db_session_with_containers) - tenant = account.current_tenant - - # Create app - app_args = CreateAppParams( - name="Test App", - mode="chat", - icon_type="emoji", - icon="🚀", - icon_background="#4ECDC4", - ) - app = app_service.create_app(tenant.id, app_args, account, session=db_session_with_containers) - - # Create workflow run - workflow_run = self._create_test_workflow_run(db_session_with_containers, app, account, "debugging") - - # Mock database error by closing the session - db_session_with_containers.close() - - # Act & Assert: Verify error handling - with pytest.raises((Exception, RuntimeError)): - workflow_run_service.get_workflow_run_node_executions( - app_model=app, - run_id=workflow_run.id, - user=account, - ) - - def test_get_workflow_run_node_executions_end_user( - self, db_session_with_containers: Session, mock_external_service_dependencies - ): - """ - Test node execution retrieval for end user. - - This test verifies: - - Proper handling of end user vs account user - - Correct tenant ID extraction for end users - - Repository method calls with proper parameters - """ - # Arrange: Create test data - fake = Faker() - app, account = self._create_test_app_and_account(db_session_with_containers, mock_external_service_dependencies) - - # Create workflow run - workflow_run = self._create_test_workflow_run(db_session_with_containers, app, account, "debugging") - - # Create end user - from models.model import EndUser - - end_user = EndUser( - tenant_id=app.tenant_id, - app_id=app.id, - type=EndUserType.BROWSER, - is_anonymous=False, - session_id=str(uuid.uuid4()), - external_user_id=str(uuid.uuid4()), - name=fake.name(), - ) - db_session_with_containers.add(end_user) - db_session_with_containers.commit() - - # Create node execution - from models.workflow import WorkflowNodeExecutionModel - - node_execution = WorkflowNodeExecutionModel( - tenant_id=app.tenant_id, - app_id=app.id, - workflow_id=workflow_run.workflow_id, - triggered_from="workflow-run", - workflow_run_id=workflow_run.id, - index=0, - node_id="node_0", - node_type="llm", - title="Node 0", - inputs=json.dumps({"input": "test_input"}), - process_data=json.dumps({"process": "test_process"}), - status="succeeded", - elapsed_time=0.5, - execution_metadata=json.dumps({"tokens": 50}), - created_by_role=CreatorUserRole.END_USER, - created_by=end_user.id, - created_at=datetime.now(UTC), - ) - db_session_with_containers.add(node_execution) - db_session_with_containers.commit() - - # Act: Execute the method under test - workflow_run_service = WorkflowRunService() - result = workflow_run_service.get_workflow_run_node_executions(app, workflow_run.id, end_user) - - # Assert: Verify the expected outcomes - assert result is not None - assert len(result) == 1 - - # Verify node execution properties - node_exec = result[0] - assert node_exec.tenant_id == app.tenant_id - assert node_exec.app_id == app.id - assert node_exec.workflow_run_id == workflow_run.id - assert node_exec.created_by == end_user.id - assert node_exec.created_by_role == CreatorUserRole.END_USER diff --git a/api/tests/unit_tests/commands/test_check_no_new_getattr.py b/api/tests/unit_tests/commands/test_check_no_new_getattr.py index a63569c706f..a846955a770 100644 --- a/api/tests/unit_tests/commands/test_check_no_new_getattr.py +++ b/api/tests/unit_tests/commands/test_check_no_new_getattr.py @@ -228,19 +228,6 @@ def test_style_workflow_wires_no_new_getattr_guard() -> None: assert checkout_step is not None assert "fetch-depth: 0" in checkout_step.group("step") - changed_files_step = re.search( - r"(?ms)^ - name: Check changed files\n.*?^ files: \|\n(?P(?:^ \S[^\n]*\n)+)", - job_text, - ) - assert changed_files_step is not None - - files_block = changed_files_step.group("files") - assert "api/**\n" in files_block - assert "scripts/check_no_new_getattr.py\n" in files_block - assert "scripts/ast_grep_rules/no_new_getattr.yml\n" in files_block - assert ".github/workflows/style.yml\n" in files_block - assert ".github/workflows/main-ci.yml\n" in files_block - guard_command = 'scripts/check_no_new_getattr.py --base-rev "${{ inputs.base-rev }}"' assert guard_command in job_text @@ -251,6 +238,7 @@ def test_style_workflow_wires_no_new_getattr_guard() -> None: ) assert guard_step is not None + assert "if: inputs.run-python-style" in guard_step.group("step") assert "GITHUB_BASE_SHA" not in guard_step.group("step") @@ -266,17 +254,22 @@ def test_main_ci_passes_style_base_rev_input() -> None: "base-rev: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha }}" in style_job.group("job") ) - - api_filter = re.search( - r"(?ms)^ api:\n(?P(?:^ - '[^']+'\n)+)", - workflow, + assert "run-python-style: ${{ needs.check-changes.outputs.python-style-changed == 'true' }}" in style_job.group( + "job" ) - assert api_filter is not None - filter_text = api_filter.group("filter") - assert "scripts/check_no_new_getattr.py" in filter_text - assert "scripts/ast_grep_rules/no_new_getattr.yml" in filter_text - assert ".github/workflows/style.yml" in filter_text - assert ".github/workflows/main-ci.yml" in filter_text + + for filter_name in ("api", "python-style"): + path_filter = re.search( + rf"(?ms)^ {re.escape(filter_name)}:\n(?P(?:^ - '[^']+'\n)+)", + workflow, + ) + assert path_filter is not None, filter_name + filter_text = path_filter.group("filter") + assert "api/**" in filter_text + assert "scripts/check_no_new_getattr.py" in filter_text + assert "scripts/ast_grep_rules/no_new_getattr.yml" in filter_text + assert ".github/workflows/style.yml" in filter_text + assert ".github/workflows/main-ci.yml" in filter_text def test_base_rev_mode_passes_when_only_legacy_getattr_exists(tmp_path: Path) -> None: diff --git a/api/tests/unit_tests/controllers/console/app/test_app_apis.py b/api/tests/unit_tests/controllers/console/app/test_app_apis.py index 6d574657017..0e25b43928e 100644 --- a/api/tests/unit_tests/controllers/console/app/test_app_apis.py +++ b/api/tests/unit_tests/controllers/console/app/test_app_apis.py @@ -573,21 +573,36 @@ class TestWorkflowAppLogEndpoints: def test_workflow_app_log_api_get(self, database_app: Flask, monkeypatch: pytest.MonkeyPatch) -> None: api = workflow_app_log_module.WorkflowAppLogApi() method = unwrap(api.get) - - def fake_get_paginate(self, *, session: Session, **_kwargs): - assert session.get_bind() is db.engine - return {"page": 1, "limit": 20, "total": 0, "has_more": False, "data": []} - - monkeypatch.setattr( - workflow_app_log_module.WorkflowAppService, - "get_paginate_workflow_app_logs", - fake_get_paginate, - ) + workflow_app_logs = MagicMock() + workflow_app_logs.list_logs.return_value = { + "page": 1, + "limit": 20, + "total": 0, + "has_more": False, + "data": [], + } + services = MagicMock(workflow_app_logs=workflow_app_logs) + monkeypatch.setattr(workflow_app_log_module, "application_services", lambda: services) + context = RequestContext("request-1", None, USER_ID, TENANT_ID) + app_model = _make_app("app-1") with database_app.test_request_context("/?page=1&limit=20"): - result = method(api, WorkflowAppLogQuery(page=1, limit=20), app_model=_make_app("app-1")) + result = method(api, WorkflowAppLogQuery(page=1, limit=20), context, app_model=app_model) assert result == {"page": 1, "limit": 20, "total": 0, "has_more": False, "data": []} + workflow_app_logs.list_logs.assert_called_once_with( + tenant_id=app_model.tenant_id, + app_id=app_model.id, + keyword=None, + status=None, + created_at_before=None, + created_at_after=None, + page=1, + limit=20, + detail=False, + created_by_end_user_session_id=None, + created_by_account=None, + ) class TestWorkflowDraftVariableEndpoints: diff --git a/api/tests/unit_tests/controllers/console/app/test_statistic_api.py b/api/tests/unit_tests/controllers/console/app/test_statistic_api.py index 2228069b40c..ed77c3ea5df 100644 --- a/api/tests/unit_tests/controllers/console/app/test_statistic_api.py +++ b/api/tests/unit_tests/controllers/console/app/test_statistic_api.py @@ -1,255 +1,198 @@ from __future__ import annotations +from collections.abc import Callable +from datetime import UTC, datetime from decimal import Decimal from inspect import unwrap from types import SimpleNamespace from typing import Any +from unittest.mock import MagicMock import pytest from flask import Flask from werkzeug.exceptions import BadRequest from controllers.console.app import statistic as statistic_module -from models.account import Account +from machinery.context import RequestContext from models.model import App +from services.app_statistic_query import ( + AppStatisticQuery, + AverageResponseTimeStatisticRecord, + AverageSessionInteractionStatisticRecord, + DailyConversationStatisticRecord, + DailyMessageStatisticRecord, + DailyTerminalStatisticRecord, + DailyTokenCostStatisticRecord, + TokensPerSecondStatisticRecord, + UserSatisfactionRateStatisticRecord, +) -def _account() -> Account: - account = Account(name="Statistics Tester", email="statistics-tester@example.com", timezone="UTC") - account.id = "account-1" - return account +def _request_context() -> RequestContext: + return RequestContext( + request_id="request-1", + trace_id="trace-1", + account_id="account-1", + active_workspace_id="tenant-1", + ) def _app_model() -> App: return App(id="app-1", tenant_id="tenant-1", name="Statistics App") -class _ConnContext: - def __init__(self, rows): - self._rows = rows - - def __enter__(self): - return self - - def __exit__(self, exc_type, exc, tb): - return False - - def execute(self, _query, _args): - return self._rows - - -def _install_db(monkeypatch: pytest.MonkeyPatch, rows) -> None: - engine = SimpleNamespace(begin=lambda: _ConnContext(rows)) - monkeypatch.setattr(statistic_module, "db", SimpleNamespace(engine=engine)) - - -def _install_common(monkeypatch: pytest.MonkeyPatch) -> None: +def _install_dependencies( + monkeypatch: pytest.MonkeyPatch, + statistics: MagicMock, + *, + time_range: tuple[datetime | None, datetime | None] = (None, None), +) -> None: monkeypatch.setattr( statistic_module, - "parse_time_range", - lambda *_args, **_kwargs: (None, None), + "application_services", + lambda: SimpleNamespace(app_statistics=statistics), ) - monkeypatch.setattr(statistic_module, "convert_datetime_to_date", lambda field: field) + monkeypatch.setattr( + statistic_module, + "current_account_with_tenant", + lambda: SimpleNamespace(account=SimpleNamespace(timezone="UTC")), + ) + monkeypatch.setattr(statistic_module, "parse_time_range", lambda *_args, **_kwargs: time_range) -def _json_payload(response: Any) -> dict[str, Any]: +def _invoke( + app: Flask, + resource_type: type, + *, + start: str | None = None, + end: str | None = None, +) -> dict[str, Any]: + resource = resource_type() + method = unwrap(resource.get) + with app.test_request_context("/console/api/apps/app-1/statistics", method="GET"): + response = method( + resource, + statistic_module.StatisticTimeRangeQuery(start=start, end=end), + _request_context(), + app_model=_app_model(), + ) return response if isinstance(response, dict) else response.get_json() -def test_daily_message_statistic_returns_rows(app: Flask, monkeypatch: pytest.MonkeyPatch) -> None: - api = statistic_module.DailyMessageStatistic() - method = unwrap(api.get) +@pytest.mark.parametrize( + ("resource_type", "query_call_getter", "record", "expected"), + [ + pytest.param( + statistic_module.DailyMessageStatistic, + lambda query: query.get_daily_messages, + DailyMessageStatisticRecord(date="2024-01-01", message_count=3), + {"date": "2024-01-01", "message_count": 3}, + id="daily-messages", + ), + pytest.param( + statistic_module.DailyConversationStatistic, + lambda query: query.get_daily_conversations, + DailyConversationStatisticRecord(date="2024-01-02", conversation_count=5), + {"date": "2024-01-02", "conversation_count": 5}, + id="daily-conversations", + ), + pytest.param( + statistic_module.DailyTerminalsStatistic, + lambda query: query.get_daily_terminals, + DailyTerminalStatisticRecord(date="2024-01-03", terminal_count=7), + {"date": "2024-01-03", "terminal_count": 7}, + id="daily-terminals", + ), + pytest.param( + statistic_module.DailyTokenCostStatistic, + lambda query: query.get_daily_token_costs, + DailyTokenCostStatisticRecord( + date="2024-01-04", + token_count=10, + total_price=Decimal("0.25"), + currency="USD", + ), + {"date": "2024-01-04", "token_count": 10, "total_price": "0.25", "currency": "USD"}, + id="daily-token-costs", + ), + pytest.param( + statistic_module.AverageSessionInteractionStatistic, + lambda query: query.get_average_session_interactions, + AverageSessionInteractionStatisticRecord(date="2024-01-05", interactions=2.5), + {"date": "2024-01-05", "interactions": 2.5}, + id="average-session-interactions", + ), + pytest.param( + statistic_module.UserSatisfactionRateStatistic, + lambda query: query.get_user_satisfaction_rates, + UserSatisfactionRateStatisticRecord(date="2024-01-06", rate=100.0), + {"date": "2024-01-06", "rate": 100.0}, + id="user-satisfaction-rate", + ), + pytest.param( + statistic_module.AverageResponseTimeStatistic, + lambda query: query.get_average_response_times, + AverageResponseTimeStatisticRecord(date="2024-01-07", latency=1234.0), + {"date": "2024-01-07", "latency": 1234.0}, + id="average-response-time", + ), + pytest.param( + statistic_module.TokensPerSecondStatistic, + lambda query: query.get_tokens_per_second, + TokensPerSecondStatisticRecord(date="2024-01-08", tps=15.5), + {"date": "2024-01-08", "tps": 15.5}, + id="tokens-per-second", + ), + ], +) +def test_statistic_endpoint_delegates_to_statistic_query( + app: Flask, + monkeypatch: pytest.MonkeyPatch, + resource_type: type, + query_call_getter: Callable[[MagicMock], MagicMock], + record: tuple, + expected: dict[str, Any], +) -> None: + statistics = MagicMock(spec=AppStatisticQuery) + query_call = query_call_getter(statistics) + query_call.return_value = [record] + _install_dependencies(monkeypatch, statistics) - rows = [SimpleNamespace(date="2024-01-01", message_count=3)] - _install_common(monkeypatch) - _install_db(monkeypatch, rows) - - with app.test_request_context("/console/api/apps/app-1/statistics/daily-messages", method="GET"): - response = method( - api, - SimpleNamespace(start=None, end=None), - _account(), - app_model=_app_model(), - ) - - assert _json_payload(response) == {"data": [{"date": "2024-01-01", "message_count": 3}]} + assert _invoke(app, resource_type) == {"data": [expected]} + query_call.assert_called_once_with( + app_id="app-1", + start_date=None, + end_date=None, + timezone="UTC", + ) -def test_daily_conversation_statistic_returns_rows(app: Flask, monkeypatch: pytest.MonkeyPatch) -> None: - api = statistic_module.DailyConversationStatistic() - method = unwrap(api.get) +def test_statistic_endpoint_passes_time_range(app: Flask, monkeypatch: pytest.MonkeyPatch) -> None: + statistics = MagicMock(spec=AppStatisticQuery) + statistics.get_daily_messages.return_value = [] + start_date = datetime(2024, 1, 1, tzinfo=UTC) + end_date = datetime(2024, 1, 2, tzinfo=UTC) + _install_dependencies(monkeypatch, statistics, time_range=(start_date, end_date)) - rows = [SimpleNamespace(date="2024-01-02", conversation_count=5)] - _install_common(monkeypatch) - _install_db(monkeypatch, rows) - - with app.test_request_context("/console/api/apps/app-1/statistics/daily-conversations", method="GET"): - response = method( - api, - SimpleNamespace(start=None, end=None), - _account(), - app_model=_app_model(), - ) - - assert _json_payload(response) == {"data": [{"date": "2024-01-02", "conversation_count": 5}]} + assert _invoke(app, statistic_module.DailyMessageStatistic, start="start", end="end") == {"data": []} + statistics.get_daily_messages.assert_called_once_with( + app_id="app-1", + start_date=start_date, + end_date=end_date, + timezone="UTC", + ) -def test_daily_token_cost_statistic_returns_rows(app: Flask, monkeypatch: pytest.MonkeyPatch) -> None: - api = statistic_module.DailyTokenCostStatistic() - method = unwrap(api.get) - - rows = [SimpleNamespace(date="2024-01-03", token_count=10, total_price=0.25, currency="USD")] - _install_common(monkeypatch) - _install_db(monkeypatch, rows) - - with app.test_request_context("/console/api/apps/app-1/statistics/token-costs", method="GET"): - response = method( - api, - SimpleNamespace(start=None, end=None), - _account(), - app_model=_app_model(), - ) - - data = _json_payload(response) - assert len(data["data"]) == 1 - assert data["data"][0]["date"] == "2024-01-03" - assert data["data"][0]["token_count"] == 10 - assert Decimal(data["data"][0]["total_price"]) == Decimal("0.25") - - -def test_daily_terminals_statistic_returns_rows(app: Flask, monkeypatch: pytest.MonkeyPatch) -> None: - api = statistic_module.DailyTerminalsStatistic() - method = unwrap(api.get) - - rows = [SimpleNamespace(date="2024-01-04", terminal_count=7)] - _install_common(monkeypatch) - _install_db(monkeypatch, rows) - - with app.test_request_context("/console/api/apps/app-1/statistics/daily-end-users", method="GET"): - response = method( - api, - SimpleNamespace(start=None, end=None), - _account(), - app_model=_app_model(), - ) - - assert _json_payload(response) == {"data": [{"date": "2024-01-04", "terminal_count": 7}]} - - -def test_average_session_interaction_statistic_requires_chat_mode(app: Flask, monkeypatch: pytest.MonkeyPatch) -> None: - """Test that AverageSessionInteractionStatistic is limited to chat/agent modes.""" - # This just verifies the decorator is applied correctly - # Actual endpoint testing would require complex JOIN mocking - api = statistic_module.AverageSessionInteractionStatistic() - method = unwrap(api.get) - assert callable(method) - - -def test_daily_message_statistic_with_invalid_time_range(app: Flask, monkeypatch: pytest.MonkeyPatch) -> None: - api = statistic_module.DailyMessageStatistic() - method = unwrap(api.get) - - def mock_parse(*args, **kwargs): - raise ValueError("Invalid time range") - - _install_db(monkeypatch, []) - monkeypatch.setattr(statistic_module, "parse_time_range", mock_parse) - monkeypatch.setattr(statistic_module, "convert_datetime_to_date", lambda field: field) - - with app.test_request_context("/console/api/apps/app-1/statistics/daily-messages", method="GET"): - with pytest.raises(BadRequest): - method( - api, - SimpleNamespace(start=None, end=None), - _account(), - app_model=_app_model(), - ) - - -def test_daily_message_statistic_multiple_rows(app: Flask, monkeypatch: pytest.MonkeyPatch) -> None: - api = statistic_module.DailyMessageStatistic() - method = unwrap(api.get) - - rows = [ - SimpleNamespace(date="2024-01-01", message_count=10), - SimpleNamespace(date="2024-01-02", message_count=15), - SimpleNamespace(date="2024-01-03", message_count=12), - ] - _install_common(monkeypatch) - _install_db(monkeypatch, rows) - - with app.test_request_context("/console/api/apps/app-1/statistics/daily-messages", method="GET"): - response = method( - api, - SimpleNamespace(start=None, end=None), - _account(), - app_model=_app_model(), - ) - - data = _json_payload(response) - assert len(data["data"]) == 3 - - -def test_daily_message_statistic_empty_result(app: Flask, monkeypatch: pytest.MonkeyPatch) -> None: - api = statistic_module.DailyMessageStatistic() - method = unwrap(api.get) - - _install_common(monkeypatch) - _install_db(monkeypatch, []) - - with app.test_request_context("/console/api/apps/app-1/statistics/daily-messages", method="GET"): - response = method( - api, - SimpleNamespace(start=None, end=None), - _account(), - app_model=_app_model(), - ) - - assert _json_payload(response) == {"data": []} - - -def test_daily_conversation_statistic_with_time_range(app: Flask, monkeypatch: pytest.MonkeyPatch) -> None: - api = statistic_module.DailyConversationStatistic() - method = unwrap(api.get) - - rows = [SimpleNamespace(date="2024-01-02", conversation_count=5)] - _install_db(monkeypatch, rows) +def test_statistic_endpoint_rejects_invalid_time_range(app: Flask, monkeypatch: pytest.MonkeyPatch) -> None: + statistics = MagicMock(spec=AppStatisticQuery) + _install_dependencies(monkeypatch, statistics) monkeypatch.setattr( statistic_module, "parse_time_range", - lambda *_args, **_kwargs: ("s", "e"), + MagicMock(side_effect=ValueError("Invalid time range")), ) - monkeypatch.setattr(statistic_module, "convert_datetime_to_date", lambda field: field) - with app.test_request_context("/console/api/apps/app-1/statistics/daily-conversations", method="GET"): - response = method( - api, - SimpleNamespace(start=None, end=None), - _account(), - app_model=_app_model(), - ) + with pytest.raises(BadRequest, match="Invalid time range"): + _invoke(app, statistic_module.DailyMessageStatistic) - assert _json_payload(response) == {"data": [{"date": "2024-01-02", "conversation_count": 5}]} - - -def test_daily_token_cost_with_multiple_currencies(app: Flask, monkeypatch: pytest.MonkeyPatch) -> None: - api = statistic_module.DailyTokenCostStatistic() - method = unwrap(api.get) - - rows = [ - SimpleNamespace(date="2024-01-01", token_count=100, total_price=Decimal("0.50"), currency="USD"), - SimpleNamespace(date="2024-01-02", token_count=200, total_price=Decimal("1.00"), currency="USD"), - ] - _install_common(monkeypatch) - _install_db(monkeypatch, rows) - - with app.test_request_context("/console/api/apps/app-1/statistics/token-costs", method="GET"): - response = method( - api, - SimpleNamespace(start=None, end=None), - _account(), - app_model=_app_model(), - ) - - data = _json_payload(response) - assert len(data["data"]) == 2 + statistics.get_daily_messages.assert_not_called() diff --git a/api/tests/unit_tests/controllers/console/app/test_workflow_pause_details_api.py b/api/tests/unit_tests/controllers/console/app/test_workflow_pause_details_api.py index a1be8254d63..c5872bbf434 100644 --- a/api/tests/unit_tests/controllers/console/app/test_workflow_pause_details_api.py +++ b/api/tests/unit_tests/controllers/console/app/test_workflow_pause_details_api.py @@ -1,193 +1,111 @@ -"""Console workflow pause-detail tests backed by persisted workflow execution state.""" +"""Controller tests for Console workflow pause details.""" from __future__ import annotations -import inspect -from dataclasses import dataclass from datetime import datetime +from inspect import unwrap +from types import SimpleNamespace from unittest.mock import Mock -from uuid import uuid4 import pytest from flask import Flask -from sqlalchemy.engine import Engine -from sqlalchemy.orm import Session from controllers.common.errors import NotFoundError from controllers.console.app import workflow_run as workflow_run_module -from core.workflow.nodes.human_input.entities import ParagraphInputConfig, UserActionConfig -from core.workflow.nodes.human_input.pause_reason import HumanInputRequired -from graphon.enums import WorkflowExecutionStatus -from models.enums import CreatorUserRole, WorkflowRunTriggeredFrom -from models.workflow import WorkflowPause, WorkflowRun, WorkflowType +from machinery.context import RequestContext +from services.workflow_run_service import WorkflowRunPauseDetails, WorkflowRunPausedNode from tests.unit_tests.config_override import apply_config_overrides -@dataclass(frozen=True) -class _Database: - engine: Engine - session: Session - - -def _persist_run( - session: Session, - *, - run_id: str, - tenant_id: str, - status: WorkflowExecutionStatus, - paused: bool = False, -) -> WorkflowRun: - workflow_id = str(uuid4()) - workflow_run = WorkflowRun( - id=run_id, - tenant_id=tenant_id, - app_id=str(uuid4()), - workflow_id=workflow_id, - type=WorkflowType.WORKFLOW, - triggered_from=WorkflowRunTriggeredFrom.DEBUGGING, - version="draft", - graph="{}", - inputs="{}", - status=status, - created_by_role=CreatorUserRole.ACCOUNT, - created_by=str(uuid4()), - created_at=datetime(2024, 1, 1, 12, 0, 0), +def _request_context(*, workspace_id: str = "tenant-1") -> RequestContext: + return RequestContext( + request_id="request-1", + trace_id="trace-1", + account_id="account-1", + active_workspace_id=workspace_id, ) - session.add(workflow_run) - if paused: - session.add( - WorkflowPause( - workflow_id=workflow_id, - workflow_run_id=run_id, - state_object_key="workflow-pauses/state.json", - ) - ) - session.commit() - return workflow_run -class _PauseEntity: - def __init__(self, paused_at: datetime, reasons: list[HumanInputRequired]): - self.paused_at = paused_at - self._reasons = reasons - - def get_pause_reasons(self): - return self._reasons - - -def test_pause_details_returns_backstage_input_url( - app: Flask, monkeypatch: pytest.MonkeyPatch, sqlite_session: Session -) -> None: - apply_config_overrides(monkeypatch, APP_WEB_URL="https://web.example.com") - - tenant_id = str(uuid4()) - run_id = str(uuid4()) - _persist_run( - sqlite_session, - run_id=run_id, - tenant_id=tenant_id, - status=WorkflowExecutionStatus.PAUSED, - paused=True, - ) +def _mock_application_services(monkeypatch: pytest.MonkeyPatch, workflow_runs: Mock) -> None: monkeypatch.setattr( workflow_run_module, - "db", - _Database(engine=sqlite_session.get_bind(), session=sqlite_session), + "application_services", + lambda: SimpleNamespace(workflow_runs=workflow_runs), ) - reason = HumanInputRequired( - form_id="form-1", - form_content="content", - inputs=[ParagraphInputConfig(output_variable_name="name")], - actions=[UserActionConfig(id="approve", title="Approve")], - node_id="node-1", - node_title="Ask Name", - ) - pause_entity = _PauseEntity(paused_at=datetime(2024, 1, 1, 12, 0, 0), reasons=[reason]) - repo = Mock() - repo.get_workflow_pause.return_value = pause_entity - monkeypatch.setattr( - workflow_run_module.DifyAPIRepositoryFactory, - "create_api_workflow_run_repository", - lambda *_, **__: repo, - ) - monkeypatch.setattr( - workflow_run_module, - "_load_form_tokens_by_form_id", - lambda _form_ids: {"form-1": "backstage-token"}, +def test_pause_details_returns_backstage_input_url(app: Flask, monkeypatch: pytest.MonkeyPatch) -> None: + apply_config_overrides(monkeypatch, APP_WEB_URL="https://web.example.com/") + workflow_runs = Mock() + workflow_runs.get_pause_details.return_value = WorkflowRunPauseDetails( + paused_at=datetime(2024, 1, 1, 12, 0, 0), + paused_nodes=( + WorkflowRunPausedNode( + node_id="node-1", + node_title="Ask Name", + form_id="form-1", + form_token="backstage-token", + ), + ), ) + _mock_application_services(monkeypatch, workflow_runs) + request_context = _request_context() - with app.test_request_context(f"/console/api/workflow/{run_id}/pause-details", method="GET"): - handler = inspect.unwrap(workflow_run_module.ConsoleWorkflowPauseDetailsApi.get) - response, status = handler( - workflow_run_module.ConsoleWorkflowPauseDetailsApi(), - tenant_id, - workflow_run_id=run_id, - ) + api = workflow_run_module.ConsoleWorkflowPauseDetailsApi() + handler = unwrap(api.get) + with app.test_request_context("/console/api/workflow/run-1/pause-details", method="GET"): + response, status = handler(api, request_context, workflow_run_id="run-1") assert status == 200 - assert response["paused_at"] == "2024-01-01T12:00:00Z" - assert response["paused_nodes"][0]["node_id"] == "node-1" - assert response["paused_nodes"][0]["pause_type"]["type"] == "human_input" - assert ( - response["paused_nodes"][0]["pause_type"]["backstage_input_url"] - == "https://web.example.com/form/backstage-token" - ) - assert "pending_human_inputs" not in response - - -def test_pause_details_tenant_isolation(app: Flask, monkeypatch: pytest.MonkeyPatch, sqlite_session: Session) -> None: - apply_config_overrides(monkeypatch, APP_WEB_URL="https://web.example.com") - - run_id = str(uuid4()) - _persist_run( - sqlite_session, - run_id=run_id, - tenant_id=str(uuid4()), - status=WorkflowExecutionStatus.PAUSED, - paused=True, - ) - monkeypatch.setattr( - workflow_run_module, - "db", - _Database(engine=sqlite_session.get_bind(), session=sqlite_session), + assert response == { + "paused_at": "2024-01-01T12:00:00Z", + "paused_nodes": [ + { + "node_id": "node-1", + "node_title": "Ask Name", + "pause_type": { + "type": "human_input", + "form_id": "form-1", + "backstage_input_url": "https://web.example.com/form/backstage-token", + }, + } + ], + } + workflow_runs.get_pause_details.assert_called_once_with( + request_context, + workflow_run_id="run-1", ) - handler = inspect.unwrap(workflow_run_module.ConsoleWorkflowPauseDetailsApi.get) - with app.test_request_context(f"/console/api/workflow/{run_id}/pause-details", method="GET"): - with pytest.raises(NotFoundError): - handler( - workflow_run_module.ConsoleWorkflowPauseDetailsApi(), - str(uuid4()), - workflow_run_id=run_id, - ) - -def test_pause_details_returns_empty_response_for_non_paused_run( - app: Flask, monkeypatch: pytest.MonkeyPatch, sqlite_session: Session +def test_pause_details_maps_missing_or_inaccessible_run_to_not_found( + app: Flask, monkeypatch: pytest.MonkeyPatch ) -> None: - tenant_id = str(uuid4()) - run_id = str(uuid4()) - _persist_run( - sqlite_session, - run_id=run_id, - tenant_id=tenant_id, - status=WorkflowExecutionStatus.RUNNING, - ) - monkeypatch.setattr( - workflow_run_module, - "db", - _Database(engine=sqlite_session.get_bind(), session=sqlite_session), + workflow_runs = Mock() + workflow_runs.get_pause_details.return_value = None + _mock_application_services(monkeypatch, workflow_runs) + request_context = _request_context(workspace_id="other-tenant") + api = workflow_run_module.ConsoleWorkflowPauseDetailsApi() + handler = unwrap(api.get) + + with app.test_request_context("/console/api/workflow/run-1/pause-details", method="GET"): + with pytest.raises(NotFoundError, match="Workflow run not found"): + handler(api, request_context, workflow_run_id="run-1") + + workflow_runs.get_pause_details.assert_called_once_with( + request_context, + workflow_run_id="run-1", ) - with app.test_request_context(f"/console/api/workflow/{run_id}/pause-details", method="GET"): - handler = inspect.unwrap(workflow_run_module.ConsoleWorkflowPauseDetailsApi.get) - response, status = handler( - workflow_run_module.ConsoleWorkflowPauseDetailsApi(), - tenant_id, - workflow_run_id=run_id, - ) + +def test_pause_details_returns_empty_response_for_non_paused_run(app: Flask, monkeypatch: pytest.MonkeyPatch) -> None: + workflow_runs = Mock() + workflow_runs.get_pause_details.return_value = WorkflowRunPauseDetails(paused_at=None, paused_nodes=()) + _mock_application_services(monkeypatch, workflow_runs) + + api = workflow_run_module.ConsoleWorkflowPauseDetailsApi() + handler = unwrap(api.get) + with app.test_request_context("/console/api/workflow/run-1/pause-details", method="GET"): + response, status = handler(api, _request_context(), workflow_run_id="run-1") assert status == 200 assert response == {"paused_at": None, "paused_nodes": []} diff --git a/api/tests/unit_tests/controllers/console/app/test_workflow_run_api.py b/api/tests/unit_tests/controllers/console/app/test_workflow_run_api.py index c1176d4155b..a9686f0cccf 100644 --- a/api/tests/unit_tests/controllers/console/app/test_workflow_run_api.py +++ b/api/tests/unit_tests/controllers/console/app/test_workflow_run_api.py @@ -3,15 +3,20 @@ from __future__ import annotations import json from datetime import UTC, datetime from inspect import unwrap +from types import SimpleNamespace from typing import Any +from unittest.mock import Mock import pytest from flask import Flask from flask_restx import marshal from sqlalchemy.orm import Session +from controllers.common.errors import NotFoundError from controllers.console.app import workflow_run as workflow_run_module +from extensions.ext_database import db from graphon.enums import WorkflowExecutionStatus, WorkflowNodeExecutionStatus +from machinery.context import RequestContext from models import Account, App, AppMode from models.enums import CreatorUserRole, WorkflowRunTriggeredFrom from models.model import IconType @@ -58,6 +63,23 @@ def _app() -> App: ) +def _request_context(*, workspace_id: str = "tenant-1") -> RequestContext: + return RequestContext( + request_id="request-1", + trace_id="trace-1", + account_id="account-1", + active_workspace_id=workspace_id, + ) + + +def _mock_application_services(monkeypatch: pytest.MonkeyPatch, workflow_runs: Mock) -> None: + monkeypatch.setattr( + workflow_run_module, + "application_services", + lambda: SimpleNamespace(workflow_runs=workflow_runs), + ) + + def _workflow_run_summary(session: Session, **overrides: object) -> WorkflowRun: created_at = datetime(2026, 1, 2, 3, 4, 5, tzinfo=UTC) workflow_run = WorkflowRun( @@ -129,17 +151,15 @@ def test_workflow_run_list_returns_frontend_history_contract( ) -> None: _account(sqlite_session) workflow_run = _workflow_run_summary(sqlite_session) - - class WorkflowRunService: - def get_paginate_workflow_runs(self, **_kwargs): - return { - "limit": 10, - "has_more": False, - "data": [workflow_run], - } - - monkeypatch.setattr(workflow_run_module, "WorkflowRunService", WorkflowRunService) - monkeypatch.setattr(workflow_run_module.db, "session", sqlite_session) + workflow_runs = Mock() + workflow_runs.get_paginate_workflow_runs.return_value = { + "limit": 10, + "has_more": False, + "data": [workflow_run], + } + _mock_application_services(monkeypatch, workflow_runs) + monkeypatch.setattr(db, "session", sqlite_session) + request_context = _request_context() api = workflow_run_module.WorkflowRunListApi() handler = unwrap(api.get) @@ -148,6 +168,7 @@ def test_workflow_run_list_returns_frontend_history_contract( payload = handler( api, workflow_run_module.WorkflowRunListQuery(limit=10), + request_context, app_model=_app(), ) @@ -168,6 +189,12 @@ def test_workflow_run_list_returns_frontend_history_contract( "exceptions_count": 0, "retry_index": 0, } + workflow_runs.get_paginate_workflow_runs.assert_called_once_with( + request_context, + app_id="app-1", + args={"limit": 10}, + triggered_from=WorkflowRunTriggeredFrom.DEBUGGING, + ) def test_advanced_chat_workflow_run_list_keeps_message_fields( @@ -179,17 +206,15 @@ def test_advanced_chat_workflow_run_list_keeps_message_fields( conversation_id="conversation-1", message_id="message-1", ) - - class WorkflowRunService: - def get_paginate_advanced_chat_workflow_runs(self, **_kwargs): - return { - "limit": 1, - "has_more": True, - "data": [workflow_run], - } - - monkeypatch.setattr(workflow_run_module, "WorkflowRunService", WorkflowRunService) - monkeypatch.setattr(workflow_run_module.db, "session", sqlite_session) + workflow_runs = Mock() + workflow_runs.get_paginate_advanced_chat_workflow_runs.return_value = { + "limit": 1, + "has_more": True, + "data": [workflow_run], + } + _mock_application_services(monkeypatch, workflow_runs) + monkeypatch.setattr(db, "session", sqlite_session) + request_context = _request_context() api = workflow_run_module.AdvancedChatAppWorkflowRunListApi() handler = unwrap(api.get) @@ -198,6 +223,7 @@ def test_advanced_chat_workflow_run_list_keeps_message_fields( payload = handler( api, workflow_run_module.WorkflowRunListQuery(limit=1), + request_context, app_model=_app(), ) @@ -205,6 +231,52 @@ def test_advanced_chat_workflow_run_list_keeps_message_fields( assert response["data"][0]["conversation_id"] == "conversation-1" assert response["data"][0]["message_id"] == "message-1" + workflow_runs.get_paginate_advanced_chat_workflow_runs.assert_called_once_with( + request_context, + app_id="app-1", + args={"limit": 1}, + triggered_from=WorkflowRunTriggeredFrom.DEBUGGING, + ) + + +def test_workflow_run_count_passes_filters_to_application_service(app: Flask, monkeypatch: pytest.MonkeyPatch) -> None: + workflow_runs = Mock() + workflow_runs.get_workflow_runs_count.return_value = { + "total": 2, + "running": 0, + "succeeded": 2, + "failed": 0, + "stopped": 0, + "partial-succeeded": 0, + } + _mock_application_services(monkeypatch, workflow_runs) + request_context = _request_context() + query = workflow_run_module.WorkflowRunCountQuery( + status="succeeded", + time_range="7d", + triggered_from="app-run", + ) + + api = workflow_run_module.WorkflowRunCountApi() + handler = unwrap(api.get) + with app.test_request_context("/apps/app-1/workflow-runs/count", method="GET"): + payload = handler(api, query, request_context, app_model=_app()) + + assert payload == { + "total": 2, + "running": 0, + "succeeded": 2, + "failed": 0, + "stopped": 0, + "partial_succeeded": 0, + } + workflow_runs.get_workflow_runs_count.assert_called_once_with( + request_context, + app_id="app-1", + status="succeeded", + time_range="7d", + triggered_from=WorkflowRunTriggeredFrom.APP_RUN, + ) def test_workflow_run_detail_returns_frontend_detail_contract( @@ -212,19 +284,17 @@ def test_workflow_run_detail_returns_frontend_detail_contract( ) -> None: _account(sqlite_session) workflow_run = _workflow_run_summary(sqlite_session) - - class WorkflowRunService: - def get_workflow_run(self, **_kwargs): - return workflow_run - - monkeypatch.setattr(workflow_run_module, "WorkflowRunService", WorkflowRunService) - monkeypatch.setattr(workflow_run_module.db, "session", sqlite_session) + workflow_runs = Mock() + workflow_runs.get_workflow_run.return_value = workflow_run + _mock_application_services(monkeypatch, workflow_runs) + monkeypatch.setattr(db, "session", sqlite_session) + request_context = _request_context() api = workflow_run_module.WorkflowRunDetailApi() handler = unwrap(api.get) with app.test_request_context("/apps/app-1/workflow-runs/run-1", method="GET"): - payload = handler(api, app_model=_app(), run_id="run-1") + payload = handler(api, request_context, app_model=_app(), run_id="run-1") response = _serialize_200_response(api.get, payload) @@ -246,26 +316,48 @@ def test_workflow_run_detail_returns_frontend_detail_contract( "finished_at": 1767323045, "exceptions_count": 0, } + workflow_runs.get_workflow_run.assert_called_once_with( + request_context, + app_id="app-1", + run_id="run-1", + ) + + +def test_workflow_run_detail_maps_missing_run_to_not_found(app: Flask, monkeypatch: pytest.MonkeyPatch) -> None: + workflow_runs = Mock() + workflow_runs.get_workflow_run.return_value = None + _mock_application_services(monkeypatch, workflow_runs) + request_context = _request_context(workspace_id="tenant-2") + api = workflow_run_module.WorkflowRunDetailApi() + handler = unwrap(api.get) + + with app.test_request_context("/apps/app-1/workflow-runs/run-1", method="GET"): + with pytest.raises(NotFoundError, match="Workflow run not found"): + handler(api, request_context, app_model=_app(), run_id="run-1") + + workflow_runs.get_workflow_run.assert_called_once_with( + request_context, + app_id="app-1", + run_id="run-1", + ) def test_workflow_run_node_executions_return_frontend_trace_contract( app: Flask, monkeypatch: pytest.MonkeyPatch, sqlite_session: Session ) -> None: - account = _account(sqlite_session) + _account(sqlite_session) execution = _workflow_run_node_execution(sqlite_session) - - class WorkflowRunService: - def get_workflow_run_node_executions(self, **_kwargs): - return [execution] - - monkeypatch.setattr(workflow_run_module, "WorkflowRunService", WorkflowRunService) - monkeypatch.setattr(workflow_run_module.db, "session", sqlite_session) + workflow_runs = Mock() + workflow_runs.get_workflow_run_node_executions.return_value = [execution] + _mock_application_services(monkeypatch, workflow_runs) + monkeypatch.setattr(db, "session", sqlite_session) + request_context = _request_context() api = workflow_run_module.WorkflowRunNodeExecutionListApi() handler = unwrap(api.get) with app.test_request_context("/apps/app-1/workflow-runs/run-1/node-executions", method="GET"): - payload = handler(api, account, app_model=_app(), run_id="run-1") + payload = handler(api, request_context, app_model=_app(), run_id="run-1") response = _serialize_200_response(api.get, payload) @@ -297,3 +389,8 @@ def test_workflow_run_node_executions_return_frontend_trace_contract( } ] } + workflow_runs.get_workflow_run_node_executions.assert_called_once_with( + request_context, + app_id="app-1", + run_id="run-1", + ) diff --git a/api/tests/unit_tests/controllers/console/auth/test_oauth.py b/api/tests/unit_tests/controllers/console/auth/test_oauth.py index 3a0d1b24310..38472867522 100644 --- a/api/tests/unit_tests/controllers/console/auth/test_oauth.py +++ b/api/tests/unit_tests/controllers/console/auth/test_oauth.py @@ -1,735 +1,303 @@ -"""Unit tests for OAuth controller endpoints.""" - -from __future__ import annotations - -from unittest.mock import ANY, MagicMock, patch +from collections.abc import Callable +from dataclasses import dataclass, field +from types import SimpleNamespace +from typing import NoReturn import pytest from flask import Flask -from sqlalchemy.orm import Session, scoped_session, sessionmaker +from werkzeug.exceptions import Forbidden, UnprocessableEntity -from controllers.console.auth.oauth import ( - OAuthCallback, - OAuthLogin, - _generate_account, - _get_account_by_openid_or_email, - get_oauth_providers, -) +from controllers.console import wraps as console_wraps +from controllers.console.auth import oauth as oauth_controller +from controllers.console.auth.oauth import OAuthCallback, OAuthLogin from enums import DeploymentEdition -from libs.oauth import OAuthUserInfo, encode_oauth_state -from models.account import Account, AccountIntegrate, AccountStatus, Tenant -from services.errors.account import AccountRegisterError -from services.errors.account import ( - EmailDomainSuspendedError as EmailDomainSuspendedRegistrationError, +from libs.oauth import encode_oauth_state +from services.account_errors import ( + InvalidOAuthProviderError, + OAuthIdentityLockUnavailableError, + OAuthInvitationAccountMismatchError, + OAuthProviderRequestError, + OAuthRegistrationError, ) -from tests.unit_tests.config_override import config_overrides_context +from services.entities.account_entities import AccountSessionTokens +from services.entities.account_oauth_entities import ( + OAuthAuthorizationRequest, + OAuthCallbackCommand, + OAuthCallbackResult, + OAuthInvitationResult, + OAuthSignInResult, +) +from services.system_feature_service import SystemFeatureService + +CONSOLE_WEB_URL = "https://console.example.com" + + +@dataclass +class FakeOAuthService: + authorization_url: str = "https://provider.example/authorize" + callback_result: OAuthCallbackResult = OAuthSignInResult( + tokens=AccountSessionTokens("access-token", "refresh-token", "csrf-token"), + oauth_new_user=False, + ) + authorization_error: Exception | None = None + callback_error: Exception | None = None + authorization_calls: list[tuple[str, OAuthAuthorizationRequest]] = field(default_factory=list) + callback_calls: list[OAuthCallbackCommand] = field(default_factory=list) + + def start_authorization(self, provider: str, request: OAuthAuthorizationRequest) -> str: + self.authorization_calls.append((provider, request)) + if self.authorization_error is not None: + raise self.authorization_error + return self.authorization_url + + def complete_authorization(self, command: OAuthCallbackCommand) -> OAuthCallbackResult: + self.callback_calls.append(command) + if self.callback_error is not None: + raise self.callback_error + return self.callback_result @pytest.fixture(autouse=True) -def _oauth_config(config_overrides) -> None: - config_overrides(CONSOLE_WEB_URL="http://localhost:3000") - - -class TestGetOAuthProviders: - @pytest.mark.parametrize( - ("github_config", "google_config", "expected_github", "expected_google"), - [ - # Both providers configured - ( - {"id": "github_id", "secret": "github_secret"}, - {"id": "google_id", "secret": "google_secret"}, - True, - True, - ), - # Only GitHub configured - ({"id": "github_id", "secret": "github_secret"}, {"id": None, "secret": None}, True, False), - # Only Google configured - ({"id": None, "secret": None}, {"id": "google_id", "secret": "google_secret"}, False, True), - # No providers configured - ({"id": None, "secret": None}, {"id": None, "secret": None}, False, False), - ], +def _oauth_admission( + config_overrides: Callable[..., None], +) -> None: + config_overrides( + DEPLOYMENT_EDITION=DeploymentEdition.CLOUD, + ENABLE_SOCIAL_OAUTH_LOGIN=True, + CONSOLE_WEB_URL=CONSOLE_WEB_URL, ) - def test_should_configure_oauth_providers_correctly( - self, app: Flask, github_config, google_config, expected_github, expected_google, config_overrides + + +def _install_service(monkeypatch: pytest.MonkeyPatch, service: FakeOAuthService) -> None: + services = SimpleNamespace(accounts=SimpleNamespace(oauth=service)) + monkeypatch.setattr(oauth_controller, "application_services", lambda: services) + + +def test_login_parses_input_and_delegates_to_application_service( + app: Flask, + monkeypatch: pytest.MonkeyPatch, +) -> None: + service = FakeOAuthService() + _install_service(monkeypatch, service) + + with app.test_request_context( + "/oauth/login/github?invite_token=invite&timezone=Asia%2FShanghai&language=zh-Hans&redirect_url=%2Fapps" ): - config_overrides( - GITHUB_CLIENT_ID=github_config["id"], - GITHUB_CLIENT_SECRET=github_config["secret"], - GOOGLE_CLIENT_ID=google_config["id"], - GOOGLE_CLIENT_SECRET=google_config["secret"], - CONSOLE_API_URL="http://localhost", - ) + response = OAuthLogin().get("github") - with app.app_context(): - providers = get_oauth_providers() - - assert (providers["github"] is not None) == expected_github - assert (providers["google"] is not None) == expected_google - - -class TestOAuthLogin: - @pytest.fixture - def resource(self): - return OAuthLogin() - - @pytest.fixture - def mock_oauth_provider(self): - provider = MagicMock() - provider.get_authorization_url.return_value = "https://github.com/login/oauth/authorize?..." - return provider - - @pytest.mark.parametrize( - ("invite_token", "expected_token"), - [ - (None, None), - ("test_invite_token", "test_invite_token"), - ("", None), - ], - ) - @patch("controllers.console.auth.oauth.get_oauth_providers") - @patch("controllers.console.auth.oauth.redirect") - def test_should_handle_oauth_login_with_various_tokens( - self, - mock_redirect, - mock_get_providers, - resource: OAuthLogin, - app: Flask, - mock_oauth_provider, - invite_token, - expected_token, - ): - mock_get_providers.return_value = {"github": mock_oauth_provider, "google": None} - - query_string = f"invite_token={invite_token}" if invite_token else "" - with app.test_request_context(f"/auth/oauth/github?{query_string}"): - resource.get("github") - - mock_oauth_provider.get_authorization_url.assert_called_once_with( - invite_token=expected_token, - timezone=None, - language=None, - redirect_url=None, - ) - mock_redirect.assert_called_once_with("https://github.com/login/oauth/authorize?...") - - @patch("controllers.console.auth.oauth.get_oauth_providers") - @patch("controllers.console.auth.oauth.redirect") - def test_should_pass_timezone_to_oauth_state( - self, - mock_redirect, - mock_get_providers, - resource: OAuthLogin, - app: Flask, - mock_oauth_provider, - ): - mock_get_providers.return_value = {"github": mock_oauth_provider, "google": None} - - with app.test_request_context("/auth/oauth/github?timezone=Asia/Shanghai"): - resource.get("github") - - mock_oauth_provider.get_authorization_url.assert_called_once_with( - invite_token=None, - timezone="Asia/Shanghai", - language=None, - redirect_url=None, - ) - mock_redirect.assert_called_once_with("https://github.com/login/oauth/authorize?...") - - @patch("controllers.console.auth.oauth.get_oauth_providers") - @patch("controllers.console.auth.oauth.redirect") - def test_should_pass_language_to_oauth_state( - self, - mock_redirect, - mock_get_providers, - resource: OAuthLogin, - app: Flask, - mock_oauth_provider, - ): - mock_get_providers.return_value = {"github": mock_oauth_provider, "google": None} - - with app.test_request_context("/auth/oauth/github?language=zh-Hans"): - resource.get("github") - - mock_oauth_provider.get_authorization_url.assert_called_once_with( - invite_token=None, - timezone=None, - language="zh-Hans", - redirect_url=None, - ) - mock_redirect.assert_called_once_with("https://github.com/login/oauth/authorize?...") - - @pytest.mark.parametrize( - ("provider", "expected_error"), - [ - ("invalid_provider", "Invalid provider"), - ("github", "Invalid provider"), # When GitHub is not configured - ("google", "Invalid provider"), # When Google is not configured - ], - ) - @patch("controllers.console.auth.oauth.get_oauth_providers") - def test_should_return_error_for_invalid_providers( - self, mock_get_providers, resource, app, provider, expected_error - ): - mock_get_providers.return_value = {"github": None, "google": None} - - with app.test_request_context(f"/auth/oauth/{provider}"): - response, status_code = resource.get(provider) - - assert status_code == 400 - assert response["error"] == expected_error - - -class TestOAuthCallback: - @pytest.fixture - def resource(self): - return OAuthCallback() - - @pytest.fixture - def oauth_setup(self): - """Common OAuth setup for callback tests""" - oauth_provider = MagicMock() - oauth_provider.get_access_token.return_value = "access_token" - oauth_provider.get_user_info.return_value = OAuthUserInfo(id="123", name="Test User", email="test@example.com") - - account = Account(name="Test User", email="test@example.com", status=AccountStatus.ACTIVE) - account.id = "123" - - token_pair = MagicMock() - token_pair.access_token = "jwt_access_token" - token_pair.refresh_token = "jwt_refresh_token" - - return {"provider": oauth_provider, "account": account, "token_pair": token_pair} - - @patch("controllers.console.auth.oauth.get_oauth_providers") - @patch("controllers.console.auth.oauth._generate_account") - @patch("controllers.console.auth.oauth.AccountService") - @patch("controllers.console.auth.oauth.TenantService") - @patch("controllers.console.auth.oauth.redirect") - def test_should_handle_successful_oauth_callback( - self, - mock_redirect, - mock_tenant_service, - mock_account_service, - mock_generate_account, - mock_get_providers, - resource: OAuthCallback, - app: Flask, - oauth_setup, - ): - mock_get_providers.return_value = {"github": oauth_setup["provider"]} - mock_generate_account.return_value = (oauth_setup["account"], True) - mock_account_service.login.return_value = oauth_setup["token_pair"] - - with ( - patch("controllers.console.auth.oauth.extract_remote_ip", return_value="203.0.113.10"), - app.test_request_context("/auth/oauth/github/callback?code=test_code"), - ): - resource.get("github") - - oauth_setup["provider"].get_access_token.assert_called_once_with("test_code") - oauth_setup["provider"].get_user_info.assert_called_once_with("access_token") - mock_generate_account.assert_called_once_with( + assert response.status_code == 302 + assert response.headers["Location"] == "https://provider.example/authorize" + assert service.authorization_calls == [ + ( "github", - oauth_setup["provider"].get_user_info.return_value, - timezone=None, - language=None, + OAuthAuthorizationRequest( + invite_token="invite", + timezone="Asia/Shanghai", + language="zh-Hans", + redirect_url="/apps", + ), + ) + ] + + +def test_login_returns_adapter_error_for_unknown_provider( + app: Flask, + monkeypatch: pytest.MonkeyPatch, +) -> None: + service = FakeOAuthService(authorization_error=InvalidOAuthProviderError()) + _install_service(monkeypatch, service) + + with app.test_request_context("/oauth/login/unknown"): + payload, status = OAuthLogin().get("unknown") + + assert status == 400 + assert payload == {"error": "Invalid provider"} + + +def test_oauth_admission_does_not_query_enterprise_features( + app: Flask, + monkeypatch: pytest.MonkeyPatch, + config_overrides: Callable[..., None], +) -> None: + config_overrides(DEPLOYMENT_EDITION=DeploymentEdition.ENTERPRISE) + monkeypatch.setattr(console_wraps, "_is_setup_completed", lambda: True) + service = FakeOAuthService() + _install_service(monkeypatch, service) + + def unexpected_feature_query() -> NoReturn: + raise AssertionError("OAuth admission must not query Enterprise features") + + monkeypatch.setattr(SystemFeatureService, "get_license", unexpected_feature_query) + + with app.test_request_context("/oauth/login/github"): + response = OAuthLogin().get("github") + + assert response.status_code == 302 + assert service.authorization_calls + + +def test_callback_passes_stable_values_and_serializes_session_cookies( + app: Flask, + monkeypatch: pytest.MonkeyPatch, +) -> None: + service = FakeOAuthService() + _install_service(monkeypatch, service) + cookie_calls: list[tuple[str, str]] = [] + monkeypatch.setattr(oauth_controller, "extract_remote_ip", lambda _request: "203.0.113.10") + monkeypatch.setattr( + oauth_controller, + "set_access_token_to_cookie", + lambda _request, _response, token: cookie_calls.append(("access", token)), + ) + monkeypatch.setattr( + oauth_controller, + "set_refresh_token_to_cookie", + lambda _request, _response, token: cookie_calls.append(("refresh", token)), + ) + monkeypatch.setattr( + oauth_controller, + "set_csrf_token_to_cookie", + lambda _request, _response, token: cookie_calls.append(("csrf", token)), + ) + state = encode_oauth_state( + invite_token="invite", + timezone="Asia/Shanghai", + language="zh-Hans", + redirect_url="/apps", + ) + + with app.test_request_context( + f"/oauth/authorize/github?code=code-1&state={state}", + headers={"Accept-Language": "en-US,en;q=0.9"}, + ): + response = OAuthCallback().get("github") + + assert response.status_code == 302 + assert response.headers["Location"] == "/apps?oauth_new_user=false" + assert service.callback_calls == [ + OAuthCallbackCommand( + provider="github", + code="code-1", + invite_token="invite", + timezone="Asia/Shanghai", + language="zh-Hans", + browser_language="en-US", ip_address="203.0.113.10", ) - mock_redirect.assert_called_once_with("http://localhost:3000?oauth_new_user=true") - - @pytest.mark.parametrize( - ("service_error", "expected_message"), - [ - ( - EmailDomainSuspendedRegistrationError(), - "This email domain has been suspended.", - ), - (AccountRegisterError("This email account is frozen."), "This email account is frozen."), - ], - ) - @patch("controllers.console.auth.oauth.get_oauth_providers") - @patch("controllers.console.auth.oauth._generate_account") - @patch("controllers.console.auth.oauth.redirect") - def test_should_translate_registration_freeze_errors( - self, - mock_redirect, - mock_generate_account, - mock_get_providers, - resource: OAuthCallback, - app: Flask, - oauth_setup, - service_error, - expected_message, - ): - mock_get_providers.return_value = {"github": oauth_setup["provider"]} - mock_generate_account.side_effect = service_error - - with app.test_request_context("/auth/oauth/github/callback?code=test_code"): - resource.get("github") - - mock_redirect.assert_called_once_with(f"http://localhost:3000/signin?message={expected_message}") - - @pytest.mark.parametrize( - ("exception", "expected_error"), - [ - (Exception("OAuth error"), "OAuth process failed"), - (ValueError("Invalid token"), "OAuth process failed"), - (KeyError("Missing key"), "OAuth process failed"), - ], - ) - @patch("controllers.console.auth.oauth.get_oauth_providers") - def test_should_handle_oauth_exceptions( - self, mock_get_providers, resource: OAuthCallback, app: Flask, exception, expected_error - ): - # Import the real requests module to create a proper exception - import httpx - - request_exception = httpx.RequestError("OAuth error") - request_exception.response = MagicMock() - request_exception.response.text = str(exception) - - mock_oauth_provider = MagicMock() - mock_oauth_provider.get_access_token.side_effect = request_exception - mock_get_providers.return_value = {"github": mock_oauth_provider} - - with app.test_request_context("/auth/oauth/github/callback?code=test_code"): - response, status_code = resource.get("github") - - assert status_code == 400 - assert response["error"] == expected_error - - @patch("controllers.console.auth.oauth.get_oauth_providers") - @patch("controllers.console.auth.oauth.RegisterService") - @patch("controllers.console.auth.oauth.AccountService") - @patch("controllers.console.auth.oauth.redirect") - def test_invitation_comparison_is_case_insensitive( - self, - mock_redirect, - mock_account_service, - mock_register_service, - mock_get_providers, - resource: OAuthCallback, - app: Flask, - oauth_setup, - ): - oauth_setup["provider"].get_user_info.return_value = OAuthUserInfo( - id="123", name="Test User", email="User@Example.com" - ) - mock_get_providers.return_value = {"github": oauth_setup["provider"]} - mock_register_service.is_valid_invite_token.return_value = True - mock_register_service.get_invitation_if_token_valid.return_value = { - "account": oauth_setup["account"], - "data": {"email": "user@example.com"}, - "tenant": Tenant(name="Invited Workspace"), - } - mock_account_service.login.return_value = oauth_setup["token_pair"] - - state = encode_oauth_state(invite_token="invite123", timezone="Asia/Shanghai") - with app.test_request_context(f"/auth/oauth/github/callback?code=test_code&state={state}"): - resource.get("github") - - mock_register_service.get_invitation_if_token_valid.assert_called_once_with( - None, None, "invite123", session=ANY - ) - mock_redirect.assert_called_once_with("http://localhost:3000/signin/invite-settings?invite_token=invite123") - - @pytest.mark.parametrize( - ("account_status", "expected_redirect"), - [ - (AccountStatus.BANNED, "http://localhost:3000/signin?message=Account is banned."), - # CLOSED status: Currently NOT handled, will proceed to login (security issue) - # This documents actual behavior. See test_defensive_check_for_closed_account_status for details - ( - AccountStatus.CLOSED.value, - "http://localhost:3000?oauth_new_user=false", - ), - ], - ) - @patch("controllers.console.auth.oauth.AccountService") - @patch("controllers.console.auth.oauth.TenantService") - @patch("controllers.console.auth.oauth.get_oauth_providers") - @patch("controllers.console.auth.oauth._generate_account") - @patch("controllers.console.auth.oauth.redirect") - def test_should_redirect_based_on_account_status( - self, - mock_redirect, - mock_generate_account, - mock_get_providers, - mock_tenant_service, - mock_account_service, - resource: OAuthCallback, - app: Flask, - oauth_setup, - account_status, - expected_redirect, - ): - - mock_get_providers.return_value = {"github": oauth_setup["provider"]} - - account = Account(name="Test User", email="test@example.com", status=account_status) - account.id = "123" - mock_generate_account.return_value = (account, False) - - # Mock login for CLOSED status - mock_token_pair = MagicMock() - mock_token_pair.access_token = "jwt_access_token" - mock_token_pair.refresh_token = "jwt_refresh_token" - mock_token_pair.csrf_token = "csrf_token" - mock_account_service.login.return_value = mock_token_pair - - with app.test_request_context("/auth/oauth/github/callback?code=test_code"): - resource.get("github") - - mock_redirect.assert_called_once_with(expected_redirect) - - @patch("controllers.console.auth.oauth.get_oauth_providers") - @patch("controllers.console.auth.oauth._generate_account") - @patch("controllers.console.auth.oauth.TenantService") - @patch("controllers.console.auth.oauth.AccountService") - def test_should_activate_pending_account( - self, - mock_account_service, - mock_tenant_service, - mock_generate_account, - mock_get_providers, - resource: OAuthCallback, - app: Flask, - oauth_setup, - ): - mock_get_providers.return_value = {"github": oauth_setup["provider"]} - - mock_account = Account(name="Test User", email="test@example.com", status=AccountStatus.PENDING) - mock_generate_account.return_value = (mock_account, False) - - mock_token_pair = MagicMock() - mock_token_pair.access_token = "jwt_access_token" - mock_token_pair.refresh_token = "jwt_refresh_token" - mock_token_pair.csrf_token = "csrf_token" - mock_account_service.login.return_value = mock_token_pair - - with app.test_request_context("/auth/oauth/github/callback?code=test_code"): - resource.get("github") - - assert mock_account.status == AccountStatus.ACTIVE - assert mock_account.initialized_at is not None - - @patch("controllers.console.auth.oauth.get_oauth_providers") - @patch("controllers.console.auth.oauth._generate_account") - @patch("controllers.console.auth.oauth.TenantService") - @patch("controllers.console.auth.oauth.AccountService") - @patch("controllers.console.auth.oauth.redirect") - def test_defensive_check_for_closed_account_status( - self, - mock_redirect, - mock_account_service, - mock_tenant_service, - mock_generate_account, - mock_get_providers, - resource: OAuthCallback, - app: Flask, - oauth_setup, - ): - """Defensive test for CLOSED account status handling in OAuth callback. - - This is a defensive test documenting expected security behavior for CLOSED accounts. - - Current behavior: CLOSED status is NOT checked, allowing closed accounts to login. - Expected behavior: CLOSED accounts should be rejected like BANNED accounts. - - Context: - - AccountStatus.CLOSED is defined in the enum but never used in production - - No production service path sets accounts to CLOSED - - Account deletion uses external service instead of status change - - All authentication services (OAuth, password, email) don't check CLOSED status - - TODO: If CLOSED status is implemented in the future: - 1. Update OAuth callback to check for CLOSED status - 2. Add similar checks to all authentication services for consistency - 3. Update this test to verify the rejection behavior - - Security consideration: Until properly implemented, CLOSED status provides no protection. - """ - # Setup - mock_get_providers.return_value = {"github": oauth_setup["provider"]} - - # Create account with CLOSED status - closed_account = Account(name="Closed Account", email="closed@example.com", status=AccountStatus.CLOSED) - closed_account.id = "123" - mock_generate_account.return_value = (closed_account, False) - - # Mock successful login (current behavior) - mock_token_pair = MagicMock() - mock_token_pair.access_token = "jwt_access_token" - mock_token_pair.refresh_token = "jwt_refresh_token" - mock_token_pair.csrf_token = "csrf_token" - mock_account_service.login.return_value = mock_token_pair - - # Execute OAuth callback - with app.test_request_context("/auth/oauth/github/callback?code=test_code"): - resource.get("github") - - # Verify current behavior: login succeeds (this is NOT ideal) - mock_redirect.assert_called_once_with("http://localhost:3000?oauth_new_user=false") - mock_account_service.login.assert_called_once() - - # Document expected behavior in comments: - # Expected: mock_redirect.assert_called_once_with( - # "http://localhost:3000/signin?message=Account is closed." - # ) - # Expected: mock_account_service.login.assert_not_called() + ] + assert cookie_calls == [ + ("access", "access-token"), + ("refresh", "refresh-token"), + ("csrf", "csrf-token"), + ] -class TestAccountGeneration: - @pytest.fixture - def user_info(self): - return OAuthUserInfo(id="123", name="Test User", email="test@example.com") +@pytest.mark.parametrize( + ("redirect_url", "expected"), + [ + ("https://console.example.com/apps", "https://console.example.com/apps?oauth_new_user=false"), + ("https://console.example.com.malicious.example/apps", f"{CONSOLE_WEB_URL}?oauth_new_user=false"), + ("//malicious.example.com/apps", f"{CONSOLE_WEB_URL}?oauth_new_user=false"), + ("///malicious.example.com/apps", f"{CONSOLE_WEB_URL}?oauth_new_user=false"), + (r"\\malicious.example.com/apps", f"{CONSOLE_WEB_URL}?oauth_new_user=false"), + ], +) +def test_callback_serializes_safe_redirect_target( + app: Flask, + monkeypatch: pytest.MonkeyPatch, + redirect_url: str, + expected: str, +) -> None: + service = FakeOAuthService() + _install_service(monkeypatch, service) + state = encode_oauth_state(redirect_url=redirect_url) - @pytest.fixture - def mock_account(self) -> Account: - return Account(name="Test User", email="test@example.com") + with app.test_request_context(f"/oauth/authorize/github?code=code-1&state={state}"): + response = OAuthCallback().get("github") - @patch("controllers.console.auth.oauth.AccountService.get_account_by_email_with_case_fallback") - def test_should_get_account_by_openid_or_email( - self, - mock_get_account, - app: Flask, - user_info: OAuthUserInfo, - sqlite_session: Session, - ): - account = Account(name="Test User", email="test@example.com") - sqlite_session.add(account) - sqlite_session.flush() - sqlite_session.add( - AccountIntegrate( - account_id=account.id, - provider="github", - open_id="123", - encrypted_token="encrypted-token", - ) - ) - sqlite_session.commit() - database_session = scoped_session(sessionmaker(bind=sqlite_session.get_bind(), expire_on_commit=False)) + assert response.headers["Location"] == expected - with patch("controllers.console.auth.oauth.db.session", database_session), app.test_request_context("/"): - # Test OpenID found - result = _get_account_by_openid_or_email("github", user_info) - assert result is not None - assert result.id == account.id - mock_get_account.assert_not_called() - # Test fallback to email lookup - mock_get_account.return_value = account +def test_callback_serializes_invitation_completion_target( + app: Flask, + monkeypatch: pytest.MonkeyPatch, +) -> None: + tokens = AccountSessionTokens("access-token", "refresh-token", "csrf-token") + service = FakeOAuthService(callback_result=OAuthInvitationResult(tokens=tokens, invite_token="invite token")) + _install_service(monkeypatch, service) - result = _get_account_by_openid_or_email("google", user_info) - assert result is account - mock_get_account.assert_called_once() - database_session.remove() + with app.test_request_context("/oauth/authorize/github?code=code-1"): + response = OAuthCallback().get("github") - @pytest.mark.parametrize( - ("allow_register", "existing_account", "should_create"), - [ - (True, None, True), # New account creation allowed - (True, "existing", False), # Existing account - (False, None, False), # Registration not allowed - ], - ) - @patch("controllers.console.auth.oauth._get_account_by_openid_or_email") - @patch("controllers.console.auth.oauth.SystemFeatureService") - @patch("controllers.console.auth.oauth.RegisterService") - @patch("controllers.console.auth.oauth.AccountService") - @patch("controllers.console.auth.oauth.TenantService") - def test_should_handle_account_generation_scenarios( - self, - mock_tenant_service: MagicMock, - mock_account_service: MagicMock, - mock_register_service: MagicMock, - mock_feature_service: MagicMock, - mock_get_account: MagicMock, - app: Flask, - user_info: OAuthUserInfo, - mock_account, - allow_register, - existing_account, - should_create, - ): - mock_get_account.return_value = mock_account if existing_account else None - mock_feature_service.is_registration_allowed.return_value = allow_register - mock_register_service.register.return_value = mock_account + assert response.headers["Location"] == f"{CONSOLE_WEB_URL}/signin/invite-settings?invite_token=invite+token" - with app.test_request_context(headers={"Accept-Language": "en-US,en;q=0.9"}): - if not allow_register and not existing_account: - with pytest.raises(AccountRegisterError): - _generate_account("github", user_info) - else: - result, oauth_new_user = _generate_account("github", user_info) - assert result == mock_account - assert oauth_new_user == should_create - if should_create: - mock_register_service.register.assert_called_once_with( - email="test@example.com", - name="Test User", - password=None, - open_id="123", - provider="github", - language="en-US", - timezone=None, - ip_address=None, - session=ANY, - ) - else: - mock_register_service.register.assert_not_called() +def test_callback_rejects_missing_code_before_service_call( + app: Flask, + monkeypatch: pytest.MonkeyPatch, +) -> None: + service = FakeOAuthService() + _install_service(monkeypatch, service) - @pytest.mark.parametrize( - ("freeze_type", "expected_error"), - [ - ("email_domain_suspended", EmailDomainSuspendedRegistrationError), - ("freeze", AccountRegisterError), - ], - ) - @config_overrides_context(DEPLOYMENT_EDITION=DeploymentEdition.CLOUD) - @patch("controllers.console.auth.oauth.BillingService.get_email_freeze_type") - @patch("controllers.console.auth.oauth._get_account_by_openid_or_email", return_value=None) - @patch("controllers.console.auth.oauth.SystemFeatureService") - def test_should_reject_registration_for_frozen_email( - self, - mock_feature_service, - mock_get_account, - mock_get_freeze_type, - freeze_type, - expected_error, - app: Flask, - user_info: OAuthUserInfo, - ): - mock_feature_service.is_registration_allowed.return_value = False - mock_get_freeze_type.return_value = freeze_type + with app.test_request_context("/oauth/authorize/github"), pytest.raises(UnprocessableEntity): + OAuthCallback().get("github") - with app.test_request_context("/"): - with pytest.raises(expected_error): - _generate_account("github", user_info) + assert service.callback_calls == [] - mock_get_freeze_type.assert_called_once_with("test@example.com") - @patch("controllers.console.auth.oauth._get_account_by_openid_or_email", return_value=None) - @patch("controllers.console.auth.oauth.SystemFeatureService") - @patch("controllers.console.auth.oauth.RegisterService") - @patch("controllers.console.auth.oauth.AccountService") - @patch("controllers.console.auth.oauth.TenantService") - def test_should_register_with_lowercase_email( - self, - mock_tenant_service: MagicMock, - mock_account_service: MagicMock, - mock_register_service: MagicMock, - mock_feature_service: MagicMock, - mock_get_account: MagicMock, - app: Flask, - ): - user_info = OAuthUserInfo(id="123", name="Test User", email="Upper@Example.com") - mock_feature_service.is_registration_allowed.return_value = True - mock_register_service.register.return_value = Account(name="Test User", email="upper@example.com") +@pytest.mark.parametrize("error", [OAuthProviderRequestError(), OAuthIdentityLockUnavailableError()]) +def test_callback_maps_oauth_processing_error_to_bad_request( + app: Flask, + monkeypatch: pytest.MonkeyPatch, + error: Exception, +) -> None: + service = FakeOAuthService(callback_error=error) + _install_service(monkeypatch, service) - with app.test_request_context(headers={"Accept-Language": "en-US"}): - _generate_account("github", user_info) + with app.test_request_context("/oauth/authorize/github?code=code-1"): + payload, status = OAuthCallback().get("github") - mock_register_service.register.assert_called_once_with( - email="upper@example.com", - name="Test User", - password=None, - open_id="123", - provider="github", - language="en-US", - timezone=None, - ip_address=None, - session=ANY, - ) + assert status == 400 + assert payload == {"error": "OAuth process failed"} - @patch("controllers.console.auth.oauth._get_account_by_openid_or_email", return_value=None) - @patch("controllers.console.auth.oauth.SystemFeatureService") - @patch("controllers.console.auth.oauth.RegisterService") - @patch("controllers.console.auth.oauth.AccountService") - @patch("controllers.console.auth.oauth.TenantService") - def test_should_register_with_browser_timezone( - self, - mock_tenant_service: MagicMock, - mock_account_service: MagicMock, - mock_register_service: MagicMock, - mock_feature_service: MagicMock, - mock_get_account: MagicMock, - app: Flask, - user_info: OAuthUserInfo, - ): - mock_feature_service.is_registration_allowed.return_value = True - mock_register_service.register.return_value = Account(name="Test User", email="test@example.com") - with app.test_request_context(headers={"Accept-Language": "zh-Hans,zh;q=0.9"}): - _generate_account("github", user_info, timezone="Asia/Shanghai") +@pytest.mark.parametrize( + ("error", "expected_query"), + [ + ( + OAuthInvitationAccountMismatchError("invite-token"), + "message=This+invitation+was+sent+to+another+account.+Please+sign+in+with+the+invited+account." + "&invite_token=invite-token", + ), + (OAuthRegistrationError("Registration failed"), "message=Registration+failed"), + ], +) +def test_callback_serializes_application_errors_as_signin_redirects( + app: Flask, + monkeypatch: pytest.MonkeyPatch, + error: Exception, + expected_query: str, +) -> None: + service = FakeOAuthService(callback_error=error) + _install_service(monkeypatch, service) - mock_register_service.register.assert_called_once_with( - email="test@example.com", - name="Test User", - password=None, - open_id="123", - provider="github", - language="zh-Hans", - timezone="Asia/Shanghai", - ip_address=None, - session=ANY, - ) + with app.test_request_context("/oauth/authorize/github?code=code-1"): + response = OAuthCallback().get("github") - @patch("controllers.console.auth.oauth._get_account_by_openid_or_email", return_value=None) - @patch("controllers.console.auth.oauth.SystemFeatureService") - @patch("controllers.console.auth.oauth.RegisterService") - @patch("controllers.console.auth.oauth.AccountService") - @patch("controllers.console.auth.oauth.TenantService") - def test_should_register_with_state_language( - self, - mock_tenant_service: MagicMock, - mock_account_service: MagicMock, - mock_register_service: MagicMock, - mock_feature_service: MagicMock, - mock_get_account: MagicMock, - app: Flask, - user_info: OAuthUserInfo, - ): - mock_feature_service.is_registration_allowed.return_value = True - mock_register_service.register.return_value = Account(name="Test User", email="test@example.com") + assert response.status_code == 302 + assert response.headers["Location"] == f"{CONSOLE_WEB_URL}/signin?{expected_query}" - with app.test_request_context(headers={"Accept-Language": "en-US,en;q=0.9"}): - _generate_account("github", user_info, language="zh-Hans") - mock_register_service.register.assert_called_once_with( - email="test@example.com", - name="Test User", - password=None, - open_id="123", - provider="github", - language="zh-Hans", - timezone=None, - ip_address=None, - session=ANY, - ) +def test_oauth_admission_rejects_disabled_social_login( + app: Flask, + monkeypatch: pytest.MonkeyPatch, + config_overrides: Callable[..., None], +) -> None: + service = FakeOAuthService() + _install_service(monkeypatch, service) + config_overrides(ENABLE_SOCIAL_OAUTH_LOGIN=False) - @patch("controllers.console.auth.oauth._get_account_by_openid_or_email") - @patch("controllers.console.auth.oauth.TenantService") - @patch("controllers.console.auth.oauth.SystemFeatureService") - @patch("controllers.console.auth.oauth.AccountService") - def test_should_create_workspace_for_account_without_tenant( - self, - mock_account_service: MagicMock, - mock_feature_service: MagicMock, - mock_tenant_service: MagicMock, - mock_get_account: MagicMock, - app: Flask, - user_info: OAuthUserInfo, - mock_account, - ): - mock_get_account.return_value = mock_account - mock_tenant_service.get_join_tenants.return_value = [] - mock_feature_service.is_workspace_creation_allowed.return_value = True + with app.test_request_context("/oauth/login/github"), pytest.raises(Forbidden): + OAuthLogin().get("github") - with app.test_request_context(headers={"Accept-Language": "en-US,en;q=0.9"}): - result, oauth_new_user = _generate_account("github", user_info) - - assert result == mock_account - assert oauth_new_user is False - mock_tenant_service.create_owner_tenant.assert_called_once_with(mock_account, session=ANY) + assert service.authorization_calls == [] diff --git a/api/tests/unit_tests/controllers/console/auth/test_oauth_redirect.py b/api/tests/unit_tests/controllers/console/auth/test_oauth_redirect.py deleted file mode 100644 index e3910d4a348..00000000000 --- a/api/tests/unit_tests/controllers/console/auth/test_oauth_redirect.py +++ /dev/null @@ -1,190 +0,0 @@ -import urllib.parse -from unittest.mock import ANY, MagicMock, patch - -import pytest -from flask import Flask - -from controllers.console.auth.oauth import OAuthCallback, OAuthLogin -from libs.oauth import OAuthUserInfo, encode_oauth_state -from models.account import Account, AccountStatus, Tenant -from tests.unit_tests.config_override import config_overrides_context - -REDIRECT_URL = "/apps?category=workflow" -CONSOLE_WEB_URL = "https://console.example.com" - - -@pytest.fixture -def app() -> Flask: - app = Flask(__name__) - app.config["TESTING"] = True - return app - - -def test_oauth_login_passes_relative_redirect_url_through(app: Flask) -> None: - oauth_provider = MagicMock() - oauth_provider.get_authorization_url.return_value = "https://accounts.google.com/o/oauth2/v2/auth?state=..." - query = urllib.parse.urlencode({"redirect_url": REDIRECT_URL}) - - with ( - patch("controllers.console.auth.oauth.get_oauth_providers", return_value={"google": oauth_provider}), - app.test_request_context(f"/oauth/login/google?{query}"), - ): - response = OAuthLogin().get("google") - - oauth_provider.get_authorization_url.assert_called_once_with( - invite_token=None, - timezone=None, - language=None, - redirect_url=REDIRECT_URL, - ) - assert response.status_code == 302 - assert response.headers["Location"] == "https://accounts.google.com/o/oauth2/v2/auth?state=..." - - -@pytest.mark.parametrize( - ("redirect_url", "expected_target_url"), - [ - (REDIRECT_URL, REDIRECT_URL), - (f"{CONSOLE_WEB_URL}{REDIRECT_URL}", f"{CONSOLE_WEB_URL}{REDIRECT_URL}"), - ("https://console.example.com.malicious.example/apps", CONSOLE_WEB_URL), - ("//malicious.example.com/apps", CONSOLE_WEB_URL), - ("///malicious.example.com/apps", CONSOLE_WEB_URL), - (r"\\malicious.example.com/apps", CONSOLE_WEB_URL), - ], -) -@pytest.mark.parametrize("oauth_new_user", [False, True]) -def test_oauth_callback_validates_redirect_url_and_appends_new_user_flag( - app: Flask, - redirect_url: str, - expected_target_url: str, - oauth_new_user: bool, -) -> None: - oauth_provider = MagicMock() - oauth_provider.get_access_token.return_value = "google-access-token" - oauth_provider.get_user_info.return_value = OAuthUserInfo( - id="google-user-123", - name="Test User", - email="test@example.com", - ) - account = Account(name="Test User", email="test@example.com", status=AccountStatus.ACTIVE) - token_pair = MagicMock() - token_pair.access_token = "dify-access-token" - token_pair.refresh_token = "dify-refresh-token" - token_pair.csrf_token = "dify-csrf-token" - state = encode_oauth_state(redirect_url=redirect_url) - - with ( - patch("controllers.console.auth.oauth.get_oauth_providers", return_value={"google": oauth_provider}), - config_overrides_context(CONSOLE_WEB_URL=CONSOLE_WEB_URL), - patch("controllers.console.auth.oauth._generate_account", return_value=(account, oauth_new_user)), - patch("controllers.console.auth.oauth.TenantService.create_owner_tenant_if_not_exist"), - patch("controllers.console.auth.oauth.AccountService.login", return_value=token_pair), - patch("controllers.console.auth.oauth.set_access_token_to_cookie"), - patch("controllers.console.auth.oauth.set_refresh_token_to_cookie"), - patch("controllers.console.auth.oauth.set_csrf_token_to_cookie"), - app.test_request_context(f"/oauth/authorize/google?code=test-code&state={state}"), - ): - response = OAuthCallback().get("google") - - assert response.status_code == 302 - query_char = "&" if "?" in expected_target_url else "?" - assert response.headers["Location"] == ( - f"{expected_target_url}{query_char}oauth_new_user={str(oauth_new_user).lower()}" - ) - - -def test_oauth_callback_with_invitation_establishes_console_session(app: Flask) -> None: - oauth_provider = MagicMock() - oauth_provider.get_access_token.return_value = "google-access-token" - oauth_provider.get_user_info.return_value = OAuthUserInfo( - id="google-user-123", - name="Test User", - email="Invitee@Example.com", - ) - account = Account(name="Test User", email="invitee@example.com", status=AccountStatus.ACTIVE) - token_pair = MagicMock() - token_pair.access_token = "dify-access-token" - token_pair.refresh_token = "dify-refresh-token" - token_pair.csrf_token = "dify-csrf-token" - state = encode_oauth_state(invite_token="invite-token") - - with ( - patch("controllers.console.auth.oauth.get_oauth_providers", return_value={"google": oauth_provider}), - config_overrides_context(CONSOLE_WEB_URL=CONSOLE_WEB_URL), - patch("controllers.console.auth.oauth.RegisterService") as register_service, - patch("controllers.console.auth.oauth.AccountService.link_account_integrate") as link_account, - patch("controllers.console.auth.oauth.AccountService.login", return_value=token_pair) as login, - patch("controllers.console.auth.oauth.TenantService.create_owner_tenant_if_not_exist") as create_workspace, - patch("controllers.console.auth.oauth.set_access_token_to_cookie") as set_access_cookie, - patch("controllers.console.auth.oauth.set_refresh_token_to_cookie") as set_refresh_cookie, - patch("controllers.console.auth.oauth.set_csrf_token_to_cookie") as set_csrf_cookie, - app.test_request_context(f"/oauth/authorize/google?code=test-code&state={state}"), - ): - register_service.is_valid_invite_token.return_value = True - register_service.get_invitation_if_token_valid.return_value = { - "account": account, - "data": { - "account_id": "account-id", - "email": "invitee@example.com", - "workspace_id": "workspace-id", - }, - "tenant": Tenant(name="Invited Workspace"), - } - - response = OAuthCallback().get("google") - - assert response.status_code == 302 - assert response.headers["Location"] == (f"{CONSOLE_WEB_URL}/signin/invite-settings?invite_token=invite-token") - link_account.assert_called_once_with("google", "google-user-123", account, session=ANY) - login.assert_called_once_with(account=account, session=ANY, ip_address=ANY) - create_workspace.assert_not_called() - set_access_cookie.assert_called_once_with(ANY, response, "dify-access-token") - set_refresh_cookie.assert_called_once_with(ANY, response, "dify-refresh-token") - set_csrf_cookie.assert_called_once_with(ANY, response, "dify-csrf-token") - - -def test_oauth_callback_with_invitation_rejects_another_account(app: Flask) -> None: - oauth_provider = MagicMock() - oauth_provider.get_access_token.return_value = "google-access-token" - oauth_provider.get_user_info.return_value = OAuthUserInfo( - id="google-user-123", - name="Test User", - email="another@example.com", - ) - account = Account(name="Test User", email="another@example.com", status=AccountStatus.ACTIVE) - state = encode_oauth_state(invite_token="invite-token") - - with ( - patch("controllers.console.auth.oauth.get_oauth_providers", return_value={"google": oauth_provider}), - config_overrides_context(CONSOLE_WEB_URL=CONSOLE_WEB_URL), - patch("controllers.console.auth.oauth.RegisterService") as register_service, - patch("controllers.console.auth.oauth.AccountService.link_account_integrate") as link_account, - patch("controllers.console.auth.oauth.AccountService.login") as login, - patch("controllers.console.auth.oauth.set_access_token_to_cookie") as set_access_cookie, - patch("controllers.console.auth.oauth.set_refresh_token_to_cookie") as set_refresh_cookie, - patch("controllers.console.auth.oauth.set_csrf_token_to_cookie") as set_csrf_cookie, - app.test_request_context(f"/oauth/authorize/google?code=test-code&state={state}"), - ): - register_service.is_valid_invite_token.return_value = True - register_service.get_invitation_if_token_valid.return_value = { - "account": account, - "data": { - "account_id": "account-id", - "email": "invitee@example.com", - "workspace_id": "workspace-id", - }, - "tenant": Tenant(name="Invited Workspace"), - } - - response = OAuthCallback().get("google") - - query = urllib.parse.parse_qs(urllib.parse.urlparse(response.headers["Location"]).query) - assert response.status_code == 302 - assert query["message"] == ["This invitation was sent to another account. Please sign in with the invited account."] - assert query["invite_token"] == ["invite-token"] - link_account.assert_not_called() - login.assert_not_called() - register_service.revoke_token.assert_not_called() - set_access_cookie.assert_not_called() - set_refresh_cookie.assert_not_called() - set_csrf_cookie.assert_not_called() diff --git a/api/tests/unit_tests/controllers/console/auth/test_oauth_timezone.py b/api/tests/unit_tests/controllers/console/auth/test_oauth_timezone.py deleted file mode 100644 index f4a332305cf..00000000000 --- a/api/tests/unit_tests/controllers/console/auth/test_oauth_timezone.py +++ /dev/null @@ -1,131 +0,0 @@ -from unittest.mock import ANY, MagicMock, patch - -import pytest -from flask import Flask - -from controllers.console.auth.oauth import OAuthLogin, _generate_account -from enums import DeploymentEdition -from libs.oauth import OAuthUserInfo -from models.account import Account -from services.errors.account import AccountRegisterError - - -@pytest.fixture -def app() -> Flask: - app = Flask(__name__) - app.config["TESTING"] = True - return app - - -@patch("controllers.console.auth.oauth.redirect") -@patch("controllers.console.auth.oauth.get_oauth_providers") -def test_oauth_login_passes_language_and_timezone_to_authorization_url( - mock_get_oauth_providers, - mock_redirect, - app: Flask, -): - oauth_provider = MagicMock() - oauth_provider.get_authorization_url.return_value = "https://github.com/login/oauth/authorize?state=..." - mock_get_oauth_providers.return_value = {"github": oauth_provider} - - with app.test_request_context("/oauth/login/github?language=zh-Hans&timezone=Asia/Shanghai"): - OAuthLogin().get("github") - - oauth_provider.get_authorization_url.assert_called_once_with( - invite_token=None, - timezone="Asia/Shanghai", - language="zh-Hans", - redirect_url=None, - ) - mock_redirect.assert_called_once_with("https://github.com/login/oauth/authorize?state=...") - - -@patch("controllers.console.auth.oauth.AccountService.link_account_integrate") -@patch("controllers.console.auth.oauth.RegisterService") -@patch("controllers.console.auth.oauth.SystemFeatureService") -@patch("controllers.console.auth.oauth._get_account_by_openid_or_email", return_value=None) -def test_generate_account_registers_with_browser_timezone( - mock_get_account, - mock_feature_service, - mock_register_service, - mock_link_account, - app: Flask, -): - account = Account(name="Test User", email="user@example.com") - mock_register_service.register.return_value = account - mock_feature_service.is_registration_allowed.return_value = True - user_info = OAuthUserInfo(id="github-123", name="Test User", email="User@Example.com") - - with app.test_request_context(headers={"Accept-Language": "zh-Hans,zh;q=0.9"}): - result, oauth_new_user = _generate_account( - "github", user_info, timezone="Asia/Shanghai", ip_address="203.0.113.10" - ) - - assert result is account - assert oauth_new_user is True - mock_register_service.register.assert_called_once_with( - email="user@example.com", - name="Test User", - password=None, - open_id="github-123", - provider="github", - language="zh-Hans", - timezone="Asia/Shanghai", - ip_address="203.0.113.10", - session=ANY, - ) - mock_link_account.assert_called_once_with("github", "github-123", account, session=ANY) - - -@patch("controllers.console.auth.oauth.AccountService.link_account_integrate") -@patch("controllers.console.auth.oauth.RegisterService") -@patch("controllers.console.auth.oauth.SystemFeatureService") -@patch("controllers.console.auth.oauth._get_account_by_openid_or_email", return_value=None) -def test_generate_account_prefers_state_language_over_accept_language( - mock_get_account, - mock_feature_service, - mock_register_service, - mock_link_account, - app: Flask, -): - account = Account(name="Test User", email="user@example.com") - mock_register_service.register.return_value = account - mock_feature_service.is_registration_allowed.return_value = True - user_info = OAuthUserInfo(id="github-123", name="Test User", email="User@Example.com") - - with app.test_request_context(headers={"Accept-Language": "en-US,en;q=0.9"}): - _generate_account("github", user_info, language="zh-Hans") - - mock_register_service.register.assert_called_once_with( - email="user@example.com", - name="Test User", - password=None, - open_id="github-123", - provider="github", - language="zh-Hans", - timezone=None, - ip_address=None, - session=ANY, - ) - mock_link_account.assert_called_once_with("github", "github-123", account, session=ANY) - - -@patch("controllers.console.auth.oauth.RegisterService") -@patch("controllers.console.auth.oauth.SystemFeatureService") -@patch("controllers.console.auth.oauth._get_account_by_openid_or_email", return_value=None) -def test_generate_account_rejects_new_user_when_registration_disabled( - mock_get_account, - mock_feature_service, - mock_register_service, - app: Flask, - config_overrides, -): - mock_feature_service.is_registration_allowed.return_value = False - config_overrides(DEPLOYMENT_EDITION=DeploymentEdition.COMMUNITY) - user_info = OAuthUserInfo(id="github-123", name="Test User", email="user@example.com") - - with app.test_request_context(headers={"Accept-Language": "en-US,en;q=0.9"}): - with pytest.raises(AccountRegisterError): - _generate_account("github", user_info) - - mock_register_service.register.assert_not_called() diff --git a/api/tests/unit_tests/controllers/console/explore/test_banner.py b/api/tests/unit_tests/controllers/console/explore/test_banner.py index 5c44ea4b8f5..9ade4ba82ad 100644 --- a/api/tests/unit_tests/controllers/console/explore/test_banner.py +++ b/api/tests/unit_tests/controllers/console/explore/test_banner.py @@ -110,7 +110,7 @@ class TestExploreBannerQueryService: assert service.list_for_language("fr-FR") == (record,) assert banners.requested_languages == ["fr-FR"] - def test_falls_back_to_en_us(self) -> None: + def test_falls_back_to_en_us_when_requested_translation_is_missing(self) -> None: record = _record(title="fallback") banners = FakeExploreBannerQuery({"en-US": (record,)}) service = ExploreBannerQueryService(banners=banners, enabled=True) @@ -118,6 +118,14 @@ class TestExploreBannerQueryService: assert service.list_for_language("es-ES") == (record,) assert banners.requested_languages == ["es-ES", "en-US"] + def test_invalid_language_uses_en_us(self) -> None: + record = _record(title="fallback") + banners = FakeExploreBannerQuery({"en-US": (record,)}) + service = ExploreBannerQueryService(banners=banners, enabled=True) + + assert service.list_for_language("invalid") == (record,) + assert banners.requested_languages == ["en-US"] + def test_does_not_repeat_default_language_query(self) -> None: banners = FakeExploreBannerQuery() service = ExploreBannerQueryService(banners=banners, enabled=True) diff --git a/api/tests/unit_tests/controllers/console/explore/test_recommended_app.py b/api/tests/unit_tests/controllers/console/explore/test_recommended_app.py index fccbba38d6d..e281a22caac 100644 --- a/api/tests/unit_tests/controllers/console/explore/test_recommended_app.py +++ b/api/tests/unit_tests/controllers/console/explore/test_recommended_app.py @@ -7,7 +7,7 @@ from flask import Flask from pydantic import ValidationError import controllers.console.explore.recommended_app as module -from models import Account +from machinery.context import RequestContext from models.model import AppMode, IconType from services.recommended_app_query_service import ( LearnDifyAppListResult, @@ -21,11 +21,13 @@ from services.recommended_app_query_service import ( ) -def make_account(interface_language: str | None) -> Account: - account = Account(name="Test User", email="user@example.com") - account.id = "account-1" - account.interface_language = interface_language - return account +def _request_context() -> RequestContext: + return RequestContext( + request_id="request-1", + trace_id="trace-1", + account_id="account-1", + active_workspace_id="workspace-1", + ) class TestRecommendedAppListApi: @@ -44,17 +46,16 @@ class TestRecommendedAppListApi: return_value=SimpleNamespace(recommended_app_queries=queries), ), ): - result = method(api, module.RecommendedAppsQuery(language="en-US"), make_account("fr-FR")) + result = method(api, module.RecommendedAppsQuery(language="en-US"), _request_context()) queries.list_recommended.assert_called_once_with( - requested_language="en-US", - interface_language="fr-FR", + language="en-US", ) assert result == {"recommended_apps": [], "categories": []} class TestLearnDifyAppListApi: - def test_get_with_language_param(self, app: Flask) -> None: + def test_get_uses_default_language(self, app: Flask) -> None: api = module.LearnDifyAppListApi() method = unwrap(api.get) @@ -62,18 +63,17 @@ class TestLearnDifyAppListApi: queries.list_learn_dify.return_value = LearnDifyAppListResult(recommended_apps=()) with ( - app.test_request_context("/", query_string={"language": "en-US"}), + app.test_request_context("/"), patch.object( module, "application_services", return_value=SimpleNamespace(recommended_app_queries=queries), ), ): - result = method(api, module.RecommendedAppsQuery(language="en-US"), make_account("fr-FR")) + result = method(api, module.RecommendedAppsQuery(), _request_context()) queries.list_learn_dify.assert_called_once_with( - requested_language="en-US", - interface_language="fr-FR", + language="en-US", ) assert result == {"recommended_apps": []} @@ -102,7 +102,7 @@ class TestRecommendedAppApi: return_value=SimpleNamespace(recommended_app_queries=queries), ), ): - result = method(api, "11111111-1111-1111-1111-111111111111") + result = method(api, _request_context(), "11111111-1111-1111-1111-111111111111") queries.get_detail.assert_called_once_with("11111111-1111-1111-1111-111111111111") assert result == { @@ -130,7 +130,7 @@ class TestRecommendedAppApi: ), ): with pytest.raises(module.RecommendedAppNotFoundError) as exc_info: - method(api, "11111111-1111-1111-1111-111111111111") + method(api, _request_context(), "11111111-1111-1111-1111-111111111111") assert exc_info.value.data == { "code": "recommended_app_not_found", diff --git a/api/tests/unit_tests/controllers/console/test_document_detail_api_data_source_info.py b/api/tests/unit_tests/controllers/console/test_document_detail_api_data_source_info.py index 9507fb4a75c..fac5aa79ae7 100644 --- a/api/tests/unit_tests/controllers/console/test_document_detail_api_data_source_info.py +++ b/api/tests/unit_tests/controllers/console/test_document_detail_api_data_source_info.py @@ -7,6 +7,7 @@ and data_source_detail_dict for all data_source_type values, including "local_fi import json from typing import Literal, NotRequired, TypedDict +from unittest.mock import MagicMock from models.dataset import Document @@ -116,7 +117,7 @@ class TestDocumentDetailDataSourceInfo: ) # data_source_detail_dict should return raw data for notion_import - detail_result = document.data_source_detail_dict + detail_result = document.get_data_source_detail_dict(session=MagicMock()) assert detail_result == notion_data # Test website_crawl @@ -127,7 +128,7 @@ class TestDocumentDetailDataSourceInfo: ) # data_source_detail_dict should return raw data for website_crawl - detail_result = document.data_source_detail_dict + detail_result = document.get_data_source_detail_dict(session=MagicMock()) assert detail_result == website_data def test_local_file_data_source_detail_dict_without_db(self): @@ -139,5 +140,5 @@ class TestDocumentDetailDataSourceInfo: ) # Should return empty dict for local_file type (handled in the model) - detail_result = document.data_source_detail_dict + detail_result = document.get_data_source_detail_dict(session=MagicMock()) assert detail_result == {} diff --git a/api/tests/unit_tests/controllers/console/test_notification.py b/api/tests/unit_tests/controllers/console/test_notification.py index 48843d1af8a..8a37c06c1a0 100644 --- a/api/tests/unit_tests/controllers/console/test_notification.py +++ b/api/tests/unit_tests/controllers/console/test_notification.py @@ -2,6 +2,9 @@ from inspect import unwrap from types import SimpleNamespace from unittest.mock import Mock, patch +import pytest +from flask import Flask + from controllers.console.notification import ( DismissNotificationPayload, NotificationApi, @@ -20,7 +23,15 @@ def _request_context() -> RequestContext: ) -def test_get_notification_delegates_and_serializes_result() -> None: +@pytest.mark.parametrize( + ("query_string", "expected_language"), + [({}, "en-US"), ({"language": "zh-Hans"}, "zh-Hans")], +) +def test_get_notification_validates_language_query_and_serializes_result( + app: Flask, + query_string: dict[str, str], + expected_language: str, +) -> None: service = Mock() service.get_active.return_value = NotificationResult( should_show=True, @@ -38,10 +49,13 @@ def test_get_notification_delegates_and_serializes_result() -> None: ) services = SimpleNamespace(notifications=service) api = NotificationApi() - method = unwrap(api.get) + method = api.get.__wrapped__ context = _request_context() - with patch("controllers.console.notification.application_services", return_value=services): + with ( + app.test_request_context("/notification", query_string=query_string), + patch("controllers.console.notification.application_services", return_value=services), + ): result, status = method(api, context) assert status == 200 @@ -59,7 +73,7 @@ def test_get_notification_delegates_and_serializes_result() -> None: } ], } - service.get_active.assert_called_once_with(context) + service.get_active.assert_called_once_with(context, expected_language) def test_dismiss_notification_delegates_with_stable_account_context() -> None: diff --git a/api/tests/unit_tests/controllers/console/test_workspace_credential_mutation_permissions.py b/api/tests/unit_tests/controllers/console/test_workspace_credential_mutation_permissions.py index e6082a2e8b8..9714a7cceb9 100644 --- a/api/tests/unit_tests/controllers/console/test_workspace_credential_mutation_permissions.py +++ b/api/tests/unit_tests/controllers/console/test_workspace_credential_mutation_permissions.py @@ -9,6 +9,7 @@ from controllers.console.agent.roster import AgentAppApi from controllers.console.datasets.data_source import DataSourceApi from controllers.console.datasets.rag_pipeline.datasource_auth import DatasourceAuth from controllers.console.workspace.model_providers import ModelProviderCredentialApi +from controllers.console.workspace.models import ModelProviderModelCredentialApi from controllers.console.workspace.tool_providers import ToolBuiltinProviderAddApi, ToolOAuthCustomClient @@ -37,6 +38,7 @@ def test_workspace_credential_mutations_require_management_permission( "method", [ ModelProviderCredentialApi.get, + ModelProviderModelCredentialApi.get, ], ) def test_model_provider_credential_get_requires_admin_and_rbac( diff --git a/api/tests/unit_tests/controllers/console/workspace/test_workspace.py b/api/tests/unit_tests/controllers/console/workspace/test_workspace.py index 76a87aeac66..2bf35fc6917 100644 --- a/api/tests/unit_tests/controllers/console/workspace/test_workspace.py +++ b/api/tests/unit_tests/controllers/console/workspace/test_workspace.py @@ -201,10 +201,12 @@ class TestWorkspaceQueryRepository: TenantAccountJoin( tenant_id=earlier.id, account_id="account-1", + current=True, last_opened_at=last_opened_at, ), TenantAccountJoin(tenant_id=later.id, account_id="account-1"), TenantAccountJoin(tenant_id=archived.id, account_id="account-1"), + TenantAccountJoin(tenant_id=archived.id, account_id="account-3"), TenantAccountJoin(tenant_id=other_account.id, account_id="account-2"), ] ) @@ -213,6 +215,7 @@ class TestWorkspaceQueryRepository: repository = WorkspaceQueryRepository(workspace_session.session_factory) result = repository.list_for_account("account-1") membership_ids = repository.list_ids_for_account("account-1") + access_workspaces = repository.list_account_access_workspaces("account-1") assert repository.has_active_for_account("account-1") is True assert repository.has_active_for_account("missing-account") is False @@ -233,6 +236,12 @@ class TestWorkspaceQueryRepository: ), ) assert set(membership_ids) == {earlier.id, later.id, archived.id} + access_by_id = {workspace.id: workspace for workspace in access_workspaces} + assert set(access_by_id) == {earlier.id, later.id, archived.id} + assert access_by_id[earlier.id].current is True + assert access_by_id[earlier.id].role == "normal" + assert repository.has_active_membership("account-1") is True + assert repository.has_active_membership("account-3") is False class TestDeploymentWorkspacePlanGateway: diff --git a/api/tests/unit_tests/controllers/files/test_appdeploy_files.py b/api/tests/unit_tests/controllers/files/test_appdeploy_files.py new file mode 100644 index 00000000000..b9e7d6e289e --- /dev/null +++ b/api/tests/unit_tests/controllers/files/test_appdeploy_files.py @@ -0,0 +1,833 @@ +"""Tests for the file endpoints reached with an AppDeploy file grant.""" + +import time +from collections.abc import Callable, Iterator +from datetime import datetime +from io import BytesIO +from types import SimpleNamespace +from typing import IO, cast +from unittest.mock import MagicMock, patch +from uuid import UUID + +import jwt +import pytest +from flask import Flask +from sqlalchemy import update +from sqlalchemy.engine import Engine +from sqlalchemy.orm import Session, sessionmaker + +from controllers.common.errors import ( + BlockedFileExtensionError, + FilenameNotExistsError, + FileTooLargeError, + NoFileUploadedError, + TooManyFilesError, +) +from controllers.files.appdeploy_files import ( + GrantedFileContentApi, + GrantedFileResolveApi, + GrantedFileUploadApi, + GrantedRemoteFileUploadApi, + InvalidFileRequestError, + ProducedFileApi, +) +from controllers.files.wraps import FileGrantInvalidError, GrantedFileNotFoundError +from extensions.ext_application_services import _build_file_grant_service +from extensions.storage.storage_type import StorageType +from libs.datetime_utils import naive_utc_now +from models.enums import CreatorUserRole, EndUserType +from models.model import EndUser, UploadFile +from models.tools import ToolFile +from services.entities.file_grant_entities import FileGrantContext, FileGrantScope, FileKind, RemoteFile +from services.errors.file import FileTooLargeError as FileTooLargeServiceError +from services.file_grant_gateways import FILE_CONTENT_AUDIENCE, FileGrantFileGateway +from services.file_grant_service import MAX_FILE_GRANT_REFS, FileGrantService +from tests.unit_tests.file_grant_test_utils import issue_file_grant + +CONTROLLER_MODULE = "controllers.files.appdeploy_files" + +SECRET_KEY = "file-grant-test-secret-long-enough-for-hs256" +TENANT_ID = "11111111-1111-4111-8111-111111111111" +OTHER_TENANT_ID = "1a1a1a1a-1111-4111-8111-111111111111" +APP_ID = "22222222-2222-4222-8222-222222222222" +FILE_ID = UUID("66666666-6666-4666-8666-666666666666") +UPLOAD_FILE_ID = "77777777-7777-4777-8777-777777777777" +UPLOADED_AT = datetime(2026, 8, 20, 12, 0) + +# What ``POST /v1/files/upload`` answers with. The grant channel is a drop-in +# for it, so the whole key set travels together and a key dify leaves null must +# still be present and null here. +DIFY_UPLOAD_RESPONSE_KEYS = frozenset( + { + "id", + "reference", + "name", + "size", + "extension", + "mime_type", + "created_by", + "created_at", + "preview_url", + "source_url", + "original_url", + "user_id", + "tenant_id", + "conversation_id", + "file_key", + } +) + +# The subset dify's own web upload client reads, from +# ``web/app/components/base/file-uploader/utils.ts``. +WEB_CLIENT_KEYS = frozenset( + { + "id", + "name", + "size", + "extension", + "mime_type", + "created_by", + "created_at", + "preview_url", + "source_url", + } +) + + +@pytest.fixture(autouse=True) +def granted_config(config_overrides: Callable[..., None]) -> None: + config_overrides( + SECRET_KEY=SECRET_KEY, + FILES_URL="https://files.example.com", + INTERNAL_FILES_URL="http://dify-api.dify.svc:5001", + FILES_ACCESS_TIMEOUT=300, + ) + + +@pytest.fixture +def sqlite_db(sqlite_engine: Engine) -> Iterator[FileGrantService]: + service = _build_file_grant_service(database_client=sessionmaker(bind=sqlite_engine, expire_on_commit=False)) + services = SimpleNamespace(file_grants=service) + with ( + patch(f"{CONTROLLER_MODULE}.application_services", return_value=services), + patch("controllers.files.wraps.application_services", return_value=services), + ): + yield service + + +@pytest.fixture +def file_gateway(sqlite_db: FileGrantService) -> FileGrantFileGateway: + return cast(FileGrantFileGateway, sqlite_db._files) + + +@pytest.fixture +def end_user(sqlite_session: Session) -> EndUser: + record = EndUser( + tenant_id=TENANT_ID, + app_id=APP_ID, + type=EndUserType.APP_DEPLOY, + is_anonymous=True, + session_id="seeded", + external_user_id="adp1.seeded", + ) + sqlite_session.add(record) + sqlite_session.commit() + return record + + +def _bearer(*scopes: FileGrantScope, end_user_id: str, tenant_id: str = TENANT_ID) -> dict[str, str]: + token, _ = issue_file_grant( + end_user_id=end_user_id, + tenant_id=tenant_id, + app_id=APP_ID, + scopes=scopes, + ttl_seconds=600, + ) + return {"Authorization": f"Bearer {token}"} + + +def _content_token(*, file_id: str, kind: FileKind, expires_in: int = 300) -> str: + return jwt.encode( + { + "aud": FILE_CONTENT_AUDIENCE, + "kind": str(kind), + "file_id": file_id, + "nonce": "0011223344556677", + "exp": int(time.time()) + expires_in, + }, + SECRET_KEY, + algorithm="HS256", + ) + + +def _stub_upload_file(**overrides: object) -> SimpleNamespace: + """Stand in for the ``upload_files`` row ``FileService`` hands back.""" + + return SimpleNamespace( + **{ + "id": UPLOAD_FILE_ID, + "name": "report.pdf", + "size": 2048, + "extension": "pdf", + "mime_type": "application/pdf", + "tenant_id": TENANT_ID, + "created_by": "99999999-9999-4999-8999-999999999999", + "created_at": UPLOADED_AT, + "source_url": "", + **overrides, + } + ) + + +def _persist_upload_file(session: Session, *, owner_id: str, tenant_id: str = TENANT_ID) -> UploadFile: + upload_file = UploadFile( + tenant_id=tenant_id, + storage_type=StorageType.OPENDAL, + key="upload_files/report.pdf", + name="report.pdf", + size=2048, + extension="pdf", + mime_type="application/pdf", + created_by=owner_id, + created_by_role=CreatorUserRole.END_USER, + created_at=naive_utc_now(), + used=False, + ) + session.add(upload_file) + session.commit() + return upload_file + + +def _persist_tool_file(session: Session, *, owner_id: str, mimetype: str = "image/png") -> ToolFile: + tool_file = ToolFile( + user_id=owner_id, + tenant_id=TENANT_ID, + conversation_id=None, + file_key="tools/chart.png", + mimetype=mimetype, + name="chart.png", + size=64, + ) + session.add(tool_file) + session.commit() + return tool_file + + +def test_upload_stores_the_file_for_the_grant_end_user( + app: Flask, end_user: EndUser, file_gateway: FileGrantFileGateway +) -> None: + with ( + patch.object(file_gateway, "store_upload", wraps=file_gateway.store_upload) as store_upload, + patch.object(file_gateway._file_service, "upload_file", return_value=_stub_upload_file()), + ): + with app.test_request_context( + "/files/appdeploy/upload", + method="POST", + headers=_bearer(FileGrantScope.UPLOAD, end_user_id=end_user.id), + data={"file": (BytesIO(b"pdf-bytes"), "report.pdf")}, + content_type="multipart/form-data", + ): + body, status = GrantedFileUploadApi().post() + + assert status == 201 + assert body["id"] == UPLOAD_FILE_ID + assert body["extension"] == "pdf" + assert store_upload.call_args.kwargs["context"].end_user_id == end_user.id + + +def test_upload_answers_in_dify_s_own_upload_shape( + app: Flask, end_user: EndUser, file_gateway: FileGrantFileGateway +) -> None: + """A client moving off ``POST /v1/files/upload`` must not meet a second shape.""" + + with patch.object( + file_gateway._file_service, + "upload_file", + return_value=_stub_upload_file(created_by=end_user.id), + ): + with app.test_request_context( + "/files/appdeploy/upload", + method="POST", + headers=_bearer(FileGrantScope.UPLOAD, end_user_id=end_user.id), + data={"file": (BytesIO(b"pdf-bytes"), "report.pdf")}, + content_type="multipart/form-data", + ): + body, _ = GrantedFileUploadApi().post() + + source_url = body.pop("source_url") + assert body == { + "id": UPLOAD_FILE_ID, + "reference": None, + "name": "report.pdf", + "size": 2048, + "extension": "pdf", + "mime_type": "application/pdf", + "created_by": end_user.id, + "created_at": int(UPLOADED_AT.timestamp()), + "preview_url": None, + "original_url": None, + "user_id": None, + "tenant_id": TENANT_ID, + "conversation_id": None, + "file_key": None, + } + assert source_url.startswith(f"https://files.example.com/files/appdeploy/{UPLOAD_FILE_ID}/content?token=") + + +def test_upload_carries_every_key_dify_s_web_client_reads( + app: Flask, end_user: EndUser, file_gateway: FileGrantFileGateway +) -> None: + with patch.object(file_gateway._file_service, "upload_file", return_value=_stub_upload_file()): + with app.test_request_context( + "/files/appdeploy/upload", + method="POST", + headers=_bearer(FileGrantScope.UPLOAD, end_user_id=end_user.id), + data={"file": (BytesIO(b"pdf-bytes"), "report.pdf")}, + content_type="multipart/form-data", + ): + body, _ = GrantedFileUploadApi().post() + + assert set(body) >= WEB_CLIENT_KEYS + + +@pytest.mark.usefixtures("sqlite_db") +def test_upload_rejects_a_grant_from_another_tenant(app: Flask, end_user: EndUser) -> None: + with app.test_request_context( + "/files/appdeploy/upload", + method="POST", + headers=_bearer(FileGrantScope.UPLOAD, end_user_id=end_user.id, tenant_id=OTHER_TENANT_ID), + data={"file": (BytesIO(b"pdf-bytes"), "report.pdf")}, + content_type="multipart/form-data", + ): + with pytest.raises(GrantedFileNotFoundError): + GrantedFileUploadApi().post() + + +@pytest.mark.usefixtures("sqlite_db") +def test_upload_applies_the_shared_per_extension_size_limit( + app: Flask, end_user: EndUser, config_overrides: Callable[..., None] +) -> None: + config_overrides(UPLOAD_FILE_SIZE_LIMIT=1) + + with app.test_request_context( + "/files/appdeploy/upload", + method="POST", + headers=_bearer(FileGrantScope.UPLOAD, end_user_id=end_user.id), + data={"file": (BytesIO(b"0" * (1024 * 1024 + 1)), "big.bin")}, + content_type="multipart/form-data", + ): + with pytest.raises(FileTooLargeError): + GrantedFileUploadApi().post() + + +@pytest.mark.usefixtures("sqlite_db") +def test_upload_rejects_a_blacklisted_extension( + app: Flask, end_user: EndUser, config_overrides: Callable[..., None] +) -> None: + config_overrides(inner_UPLOAD_FILE_EXTENSION_BLACKLIST="exe") + + with app.test_request_context( + "/files/appdeploy/upload", + method="POST", + headers=_bearer(FileGrantScope.UPLOAD, end_user_id=end_user.id), + data={"file": (BytesIO(b"MZ"), "payload.exe")}, + content_type="multipart/form-data", + ): + with pytest.raises(BlockedFileExtensionError): + GrantedFileUploadApi().post() + + +@pytest.mark.usefixtures("sqlite_db") +def test_upload_rejects_a_request_carrying_no_file(app: Flask, end_user: EndUser) -> None: + with app.test_request_context( + "/files/appdeploy/upload", + method="POST", + headers=_bearer(FileGrantScope.UPLOAD, end_user_id=end_user.id), + data={"note": "no file here"}, + content_type="multipart/form-data", + ): + with pytest.raises(NoFileUploadedError): + GrantedFileUploadApi().post() + + +@pytest.mark.usefixtures("sqlite_db") +def test_upload_rejects_a_batch_of_files(app: Flask, end_user: EndUser) -> None: + with app.test_request_context( + "/files/appdeploy/upload", + method="POST", + headers=_bearer(FileGrantScope.UPLOAD, end_user_id=end_user.id), + data={ + "file": (BytesIO(b"first"), "first.pdf"), + "second": (BytesIO(b"second"), "second.pdf"), + }, + content_type="multipart/form-data", + ): + with pytest.raises(TooManyFilesError): + GrantedFileUploadApi().post() + + +@pytest.mark.usefixtures("sqlite_db") +def test_upload_rejects_a_file_without_a_name(app: Flask, end_user: EndUser) -> None: + with app.test_request_context( + "/files/appdeploy/upload", + method="POST", + headers=_bearer(FileGrantScope.UPLOAD, end_user_id=end_user.id), + data={"file": (BytesIO(b"nameless"), "")}, + content_type="multipart/form-data", + ): + with pytest.raises(FilenameNotExistsError): + GrantedFileUploadApi().post() + + +def test_remote_upload_fetches_through_the_ssrf_safe_fetcher( + app: Flask, end_user: EndUser, sqlite_db: FileGrantService, file_gateway: FileGrantFileGateway +) -> None: + url = "https://example.com/docs/report.pdf" + + with ( + patch.object( + sqlite_db._remote_files, + "fetch", + return_value=RemoteFile(filename="report.pdf", mimetype="application/pdf", content=b"pdf-bytes"), + ) as fetch, + patch.object(file_gateway._file_service, "upload_file", return_value=_stub_upload_file()) as upload_file, + ): + with app.test_request_context( + "/files/appdeploy/remote-upload", + method="POST", + headers=_bearer(FileGrantScope.UPLOAD, end_user_id=end_user.id), + json={"url": url}, + ): + with patch(f"{CONTROLLER_MODULE}.files_ns") as files_ns: + files_ns.payload = {"url": url} + body, status = GrantedRemoteFileUploadApi().post() + + assert status == 201 + assert body["id"] == UPLOAD_FILE_ID + fetch.assert_called_once_with(url) + kwargs = upload_file.call_args.kwargs + assert kwargs["source_url"] == url + assert kwargs["content"] == b"pdf-bytes" + assert kwargs["user"].id == end_user.id + + +def test_remote_upload_answers_in_the_upload_shape_plus_dify_s_url_key( + app: Flask, end_user: EndUser, sqlite_db: FileGrantService, file_gateway: FileGrantFileGateway +) -> None: + """Dify's own remote upload answers under ``url``, so this one answers under both.""" + + url = "https://example.com/docs/report.pdf" + with ( + patch.object( + sqlite_db._remote_files, + "fetch", + return_value=RemoteFile(filename="report.pdf", mimetype="application/pdf", content=b"pdf-bytes"), + ), + patch.object( + file_gateway._file_service, + "upload_file", + return_value=_stub_upload_file(source_url=url), + ), + ): + with app.test_request_context( + "/files/appdeploy/remote-upload", + method="POST", + headers=_bearer(FileGrantScope.UPLOAD, end_user_id=end_user.id), + json={"url": url}, + ): + with patch(f"{CONTROLLER_MODULE}.files_ns") as files_ns: + files_ns.payload = {"url": url} + body, _ = GrantedRemoteFileUploadApi().post() + + assert set(body) == DIFY_UPLOAD_RESPONSE_KEYS | {"url"} + # The row records where the bytes came from; the response hands back the + # signed URL that fetches them, exactly as dify's own remote upload does. + assert body["source_url"].startswith(f"https://files.example.com/files/appdeploy/{UPLOAD_FILE_ID}/content?token=") + # One URL under both names, not two signings of the same file. + assert body["url"] == body["source_url"] + + +def test_remote_upload_honours_the_size_precheck(app: Flask, end_user: EndUser, sqlite_db: FileGrantService) -> None: + url = "https://example.com/docs/huge.pdf" + with patch.object( + sqlite_db._remote_files, + "fetch", + side_effect=FileTooLargeServiceError("too large"), + ): + with app.test_request_context( + "/files/appdeploy/remote-upload", + method="POST", + headers=_bearer(FileGrantScope.UPLOAD, end_user_id=end_user.id), + json={"url": url}, + ): + with patch(f"{CONTROLLER_MODULE}.files_ns") as files_ns: + files_ns.payload = {"url": url} + with pytest.raises(FileTooLargeError): + GrantedRemoteFileUploadApi().post() + + +def test_produced_stores_a_tool_file_and_returns_both_urls( + app: Flask, end_user: EndUser, file_gateway: FileGrantFileGateway +) -> None: + tool_file = SimpleNamespace( + id="88888888-8888-4888-8888-888888888888", + name="chart.png", + size=64, + mimetype="image/png", + ) + + with patch.object(file_gateway._tool_files, "create_file_by_raw", return_value=tool_file) as create_file: + with app.test_request_context( + "/files/appdeploy/produced", + method="POST", + headers=_bearer(FileGrantScope.PRODUCE, end_user_id=end_user.id), + data={"file": (BytesIO(b"0" * 16), "chart.png")}, + content_type="multipart/form-data", + ): + body, status = ProducedFileApi().post() + + assert status == 201 + kwargs = create_file.call_args.kwargs + assert kwargs["user_id"] == end_user.id + assert kwargs["tenant_id"] == TENANT_ID + assert kwargs["conversation_id"] is None + assert body["url"].startswith(f"https://files.example.com/files/appdeploy/{tool_file.id}/content?token=") + assert body["internal_url"].startswith( + f"http://dify-api.dify.svc:5001/files/appdeploy/{tool_file.id}/content?token=" + ) + + +def test_produced_rejects_a_grant_whose_subject_was_deleted(app: Flask, file_gateway: FileGrantFileGateway) -> None: + with patch.object(file_gateway._tool_files, "create_file_by_raw") as create_file: + with app.test_request_context( + "/files/appdeploy/produced", + method="POST", + headers=_bearer( + FileGrantScope.PRODUCE, + end_user_id="99999999-9999-4999-8999-999999999999", + ), + data={"file": (BytesIO(b"content"), "chart.png")}, + content_type="multipart/form-data", + ): + with pytest.raises(GrantedFileNotFoundError): + ProducedFileApi().post() + + create_file.assert_not_called() + + +@pytest.fixture +def one_megabyte_image_limit(config_overrides: Callable[..., None]) -> int: + """Hold images to one mebibyte while every other kind stays far above it.""" + + config_overrides( + UPLOAD_FILE_SIZE_LIMIT=64, + UPLOAD_IMAGE_FILE_SIZE_LIMIT=1, + UPLOAD_VIDEO_FILE_SIZE_LIMIT=64, + UPLOAD_AUDIO_FILE_SIZE_LIMIT=64, + ) + return 1024 * 1024 + + +def test_produced_accepts_a_file_of_exactly_the_per_extension_limit( + app: Flask, end_user: EndUser, file_gateway: FileGrantFileGateway, one_megabyte_image_limit: int +) -> None: + with patch.object( + file_gateway._tool_files, + "create_file_by_raw", + return_value=SimpleNamespace( + id="88888888-8888-4888-8888-888888888888", + name="chart.png", + size=one_megabyte_image_limit, + mimetype="image/png", + ), + ) as create_file: + with app.test_request_context( + "/files/appdeploy/produced", + method="POST", + headers=_bearer(FileGrantScope.PRODUCE, end_user_id=end_user.id), + data={"file": (BytesIO(b"0" * one_megabyte_image_limit), "chart.png")}, + content_type="multipart/form-data", + ): + _, status = ProducedFileApi().post() + + assert status == 201 + assert len(create_file.call_args.kwargs["file_binary"]) == one_megabyte_image_limit + + +def test_produced_rejects_a_file_one_byte_over_the_per_extension_limit( + app: Flask, end_user: EndUser, file_gateway: FileGrantFileGateway, one_megabyte_image_limit: int +) -> None: + """``create_file_by_raw`` has no limit of its own, so nothing else would stop this.""" + + with patch.object(file_gateway._tool_files, "create_file_by_raw") as create_file: + with app.test_request_context( + "/files/appdeploy/produced", + method="POST", + headers=_bearer(FileGrantScope.PRODUCE, end_user_id=end_user.id), + data={"file": (BytesIO(b"0" * (one_megabyte_image_limit + 1)), "chart.png")}, + content_type="multipart/form-data", + ): + with pytest.raises(FileTooLargeError) as raised: + ProducedFileApi().post() + + assert raised.value.code == 413 + create_file.assert_not_called() + + +class _CountingStream: + """A body that reports how much of itself a reader actually pulled.""" + + def __init__(self, size: int) -> None: + self._remaining = size + self.bytes_read = 0 + + def read(self, size: int = -1) -> bytes: + served = self._remaining if size < 0 else min(size, self._remaining) + self._remaining -= served + self.bytes_read += served + return b"0" * served + + +def test_produced_stops_reading_an_oversized_body_at_the_per_extension_limit( + end_user: EndUser, + sqlite_db: FileGrantService, + file_gateway: FileGrantFileGateway, + one_megabyte_image_limit: int, +) -> None: + """The caller is a worker running plugin code with no proxy body limit in front of it.""" + + stream = _CountingStream(one_megabyte_image_limit * 64) + + with patch.object(file_gateway._tool_files, "create_file_by_raw") as create_file: + with pytest.raises(FileTooLargeServiceError): + sqlite_db.store_produced( + context=FileGrantContext(TENANT_ID, APP_ID, end_user.id), + filename="chart.png", + stream=cast(IO[bytes], stream), + mimetype="image/png", + ) + + assert stream.bytes_read == one_megabyte_image_limit + 1 + create_file.assert_not_called() + + +@pytest.mark.usefixtures("sqlite_db") +def test_resolve_signs_urls_per_item_and_hides_foreign_files( + app: Flask, end_user: EndUser, sqlite_session: Session +) -> None: + owned = _persist_upload_file(sqlite_session, owner_id=end_user.id) + foreign = _persist_upload_file(sqlite_session, owner_id="00000000-0000-4000-8000-000000000000") + payload = { + "files": [ + {"id": owned.id, "kind": "upload"}, + {"id": foreign.id, "kind": "upload"}, + ] + } + + with app.test_request_context( + "/files/appdeploy/resolve", + method="POST", + headers=_bearer(FileGrantScope.RESOLVE, end_user_id=end_user.id), + json=payload, + ): + with patch(f"{CONTROLLER_MODULE}.files_ns") as files_ns: + files_ns.payload = payload + body = GrantedFileResolveApi().post() + + resolved, hidden = body["files"] + assert resolved["ok"] is True + assert resolved["kind"] == "upload" + assert resolved["extension"] == "pdf" + assert resolved["url"].startswith(f"https://files.example.com/files/appdeploy/{owned.id}/content?token=") + assert resolved["internal_url"].startswith( + f"http://dify-api.dify.svc:5001/files/appdeploy/{owned.id}/content?token=" + ) + assert hidden == { + "id": foreign.id, + "ok": False, + "kind": None, + "name": None, + "size": None, + "extension": None, + "mime_type": None, + "url": None, + "internal_url": None, + "error": "not_found", + } + + +@pytest.mark.usefixtures("sqlite_db") +def test_resolve_answers_a_mixed_batch_item_by_item(app: Flask, end_user: EndUser, sqlite_session: Session) -> None: + owned_upload = _persist_upload_file(sqlite_session, owner_id=end_user.id) + owned_tool = _persist_tool_file(sqlite_session, owner_id=end_user.id) + missing_id = "44444444-4444-4444-8444-444444444444" + payload = { + "files": [ + {"id": owned_upload.id, "kind": "upload"}, + {"id": missing_id, "kind": "upload"}, + # A real file, but looked up in the wrong table. + {"id": owned_upload.id, "kind": "tool"}, + {"id": owned_tool.id, "kind": "tool"}, + ] + } + + with app.test_request_context( + "/files/appdeploy/resolve", + method="POST", + headers=_bearer(FileGrantScope.RESOLVE, end_user_id=end_user.id), + json=payload, + ): + with patch(f"{CONTROLLER_MODULE}.files_ns") as files_ns: + files_ns.payload = payload + body = GrantedFileResolveApi().post() + + assert [(file["id"], file["ok"], file["kind"], file["error"]) for file in body["files"]] == [ + (owned_upload.id, True, "upload", None), + (missing_id, False, None, "not_found"), + (owned_upload.id, False, None, "not_found"), + (owned_tool.id, True, "tool", None), + ] + + +@pytest.mark.usefixtures("sqlite_db") +def test_resolve_returns_an_empty_batch_unchanged(app: Flask, end_user: EndUser) -> None: + payload: dict[str, object] = {"files": []} + + with app.test_request_context( + "/files/appdeploy/resolve", + method="POST", + headers=_bearer(FileGrantScope.RESOLVE, end_user_id=end_user.id), + json=payload, + ): + with patch(f"{CONTROLLER_MODULE}.files_ns") as files_ns: + files_ns.payload = payload + assert GrantedFileResolveApi().post() == {"files": []} + + +def test_resolve_rejects_an_unbounded_batch(app: Flask, end_user: EndUser, sqlite_db: FileGrantService) -> None: + del sqlite_db + payload = { + "files": [ + {"id": f"00000000-0000-4000-8000-{index:012d}", "kind": "upload"} + for index in range(MAX_FILE_GRANT_REFS + 1) + ] + } + with app.test_request_context( + "/files/appdeploy/resolve", + method="POST", + headers=_bearer(FileGrantScope.RESOLVE, end_user_id=end_user.id), + json=payload, + ): + with patch(f"{CONTROLLER_MODULE}.files_ns") as files_ns: + files_ns.payload = payload + with pytest.raises(InvalidFileRequestError): + GrantedFileResolveApi().post() + + +@pytest.fixture +def stored_bytes(file_gateway: FileGrantFileGateway) -> Iterator[MagicMock]: + with patch.object(file_gateway._storage, "load", return_value=iter([b"file-bytes"])) as load: + yield load + + +@pytest.mark.usefixtures("sqlite_db", "stored_bytes") +@pytest.mark.parametrize( + ("mime_type", "expected_content_type", "expects_attachment"), + [ + ("image/png", "image/png", False), + ("image/jpeg", "image/jpeg", False), + ("image/gif", "image/gif", False), + ("image/webp", "image/webp", False), + # Case and parameters are normalized before the whitelist is consulted. + ("IMAGE/PNG; charset=binary", "image/png", False), + ("application/pdf", "application/octet-stream", True), + ("image/svg+xml", "application/octet-stream", True), + ("text/html", "application/octet-stream", True), + ("application/xhtml+xml", "application/octet-stream", True), + ("text/javascript", "application/octet-stream", True), + ("audio/mpeg", "application/octet-stream", True), + ("video/mp4", "application/octet-stream", True), + ], +) +def test_content_disposition_follows_the_inline_whitelist( + app: Flask, + sqlite_session: Session, + mime_type: str, + expected_content_type: str, + expects_attachment: bool, +) -> None: + tool_file = _persist_tool_file(sqlite_session, owner_id="anyone", mimetype=mime_type) + token = _content_token(file_id=tool_file.id, kind=FileKind.TOOL) + + with app.test_request_context(f"/files/appdeploy/{tool_file.id}/content", query_string={"token": token}): + response = GrantedFileContentApi().get(UUID(tool_file.id)) + + assert response.headers["X-Content-Type-Options"] == "nosniff" + assert response.headers["Content-Type"].startswith(expected_content_type) + assert ("Content-Disposition" in response.headers) is expects_attachment + if expects_attachment: + assert response.headers["Content-Disposition"] == "attachment; filename*=UTF-8''chart.png" + # Range is never honoured here, so the hint must not be advertised either. + assert "Accept-Ranges" not in response.headers + assert response.headers["Content-Length"] == "64" + + +@pytest.mark.usefixtures("sqlite_db", "stored_bytes") +def test_content_downloads_a_file_with_no_recorded_mime_type(app: Flask, sqlite_session: Session) -> None: + upload_file = _persist_upload_file(sqlite_session, owner_id="anyone") + sqlite_session.execute(update(UploadFile).where(UploadFile.id == upload_file.id).values(mime_type=None)) + sqlite_session.commit() + token = _content_token(file_id=upload_file.id, kind=FileKind.UPLOAD) + + with app.test_request_context(f"/files/appdeploy/{upload_file.id}/content", query_string={"token": token}): + response = GrantedFileContentApi().get(UUID(upload_file.id)) + + assert response.headers["X-Content-Type-Options"] == "nosniff" + assert response.headers["Content-Type"].startswith("application/octet-stream") + assert response.headers["Content-Disposition"] == "attachment; filename*=UTF-8''report.pdf" + + +@pytest.mark.usefixtures("sqlite_db", "stored_bytes") +def test_content_rejects_an_expired_token(app: Flask, sqlite_session: Session) -> None: + tool_file = _persist_tool_file(sqlite_session, owner_id="anyone") + token = _content_token(file_id=tool_file.id, kind=FileKind.TOOL, expires_in=-1) + + with app.test_request_context(f"/files/appdeploy/{tool_file.id}/content", query_string={"token": token}): + with pytest.raises(FileGrantInvalidError): + GrantedFileContentApi().get(UUID(tool_file.id)) + + +@pytest.mark.usefixtures("sqlite_db", "stored_bytes") +def test_content_rejects_a_token_minted_for_another_file(app: Flask, sqlite_session: Session) -> None: + tool_file = _persist_tool_file(sqlite_session, owner_id="anyone") + token = _content_token(file_id=str(FILE_ID), kind=FileKind.TOOL) + + with app.test_request_context(f"/files/appdeploy/{tool_file.id}/content", query_string={"token": token}): + with pytest.raises(GrantedFileNotFoundError): + GrantedFileContentApi().get(UUID(tool_file.id)) + + +@pytest.mark.usefixtures("sqlite_db", "stored_bytes") +def test_content_rejects_a_token_naming_the_wrong_table(app: Flask, sqlite_session: Session) -> None: + tool_file = _persist_tool_file(sqlite_session, owner_id="anyone") + token = _content_token(file_id=tool_file.id, kind=FileKind.UPLOAD) + + with app.test_request_context(f"/files/appdeploy/{tool_file.id}/content", query_string={"token": token}): + with pytest.raises(GrantedFileNotFoundError): + GrantedFileContentApi().get(UUID(tool_file.id)) + + +@pytest.mark.usefixtures("sqlite_db", "stored_bytes") +def test_content_rejects_a_file_grant_replayed_as_a_content_token(app: Flask, sqlite_session: Session) -> None: + tool_file = _persist_tool_file(sqlite_session, owner_id="anyone") + grant, _ = issue_file_grant( + end_user_id="anyone", + tenant_id=TENANT_ID, + app_id=APP_ID, + scopes=[FileGrantScope.RESOLVE], + ttl_seconds=600, + ) + + with app.test_request_context(f"/files/appdeploy/{tool_file.id}/content", query_string={"token": grant}): + with pytest.raises(FileGrantInvalidError): + GrantedFileContentApi().get(UUID(tool_file.id)) diff --git a/api/tests/unit_tests/controllers/files/test_file_grant_wraps.py b/api/tests/unit_tests/controllers/files/test_file_grant_wraps.py new file mode 100644 index 00000000000..893001f54df --- /dev/null +++ b/api/tests/unit_tests/controllers/files/test_file_grant_wraps.py @@ -0,0 +1,151 @@ +"""Tests for the Bearer file-grant decorator.""" + +import time +from collections.abc import Callable +from types import SimpleNamespace +from typing import cast + +import jwt +import pytest +from flask import Flask + +from controllers.files.wraps import FileGrantInvalidError, FileGrantScopeDeniedError, file_grant_required +from libs.passport import PassportService +from services.entities.file_grant_entities import FileGrantClaims, FileGrantScope +from services.file_grant_gateways import FILE_GRANT_AUDIENCE +from tests.unit_tests.file_grant_test_utils import issue_file_grant, token_gateway + +SECRET_KEY = "file-grant-test-secret-long-enough-for-hs256" +TENANT_ID = "11111111-1111-4111-8111-111111111111" +APP_ID = "22222222-2222-4222-8222-222222222222" +END_USER_ID = "55555555-5555-4555-8555-555555555555" + + +@pytest.fixture(autouse=True) +def granted_config(config_overrides: Callable[..., None]) -> None: + config_overrides(SECRET_KEY=SECRET_KEY) + + +@pytest.fixture(autouse=True) +def file_grant_service(granted_config: None, monkeypatch: pytest.MonkeyPatch) -> None: + del granted_config + service = SimpleNamespace(decode_grant=token_gateway().decode_grant) + monkeypatch.setattr( + "controllers.files.wraps.application_services", + lambda: SimpleNamespace(file_grants=service), + ) + + +@file_grant_required(FileGrantScope.UPLOAD) +def _view(grant: FileGrantClaims) -> FileGrantClaims: + return grant + + +def _call(app: Flask, authorization: str | None) -> FileGrantClaims: + headers: dict[str, str] = {"Authorization": authorization} if authorization is not None else {} + with app.test_request_context("/", method="POST", headers=headers): + return cast(Callable[[], FileGrantClaims], _view)() + + +def _grant(*scopes: FileGrantScope, ttl_seconds: int = 600) -> str: + token, _ = issue_file_grant( + end_user_id=END_USER_ID, + tenant_id=TENANT_ID, + app_id=APP_ID, + scopes=scopes, + ttl_seconds=ttl_seconds, + ) + return token + + +def test_valid_grant_is_injected_into_the_view(app: Flask) -> None: + claims = _call(app, f"Bearer {_grant(FileGrantScope.UPLOAD, FileGrantScope.RESOLVE)}") + + assert claims.sub == END_USER_ID + assert claims.tenant_id == TENANT_ID + assert claims.app_id == APP_ID + assert claims.scopes == [FileGrantScope.UPLOAD, FileGrantScope.RESOLVE] + + +def test_missing_authorization_is_rejected(app: Flask) -> None: + with pytest.raises(FileGrantInvalidError): + _call(app, None) + + +def test_non_bearer_authorization_is_rejected(app: Flask) -> None: + with pytest.raises(FileGrantInvalidError): + _call(app, f"Basic {_grant(FileGrantScope.UPLOAD)}") + + +def test_webapp_passport_cannot_be_replayed_as_a_grant(app: Flask) -> None: + """The passport is signed with the same key, so only ``aud`` separates them.""" + + passport = PassportService().issue( + { + "iss": "SELF_HOSTED", + "sub": "Web API Passport", + "app_id": APP_ID, + "end_user_id": END_USER_ID, + "exp": int(time.time()) + 600, + } + ) + + with pytest.raises(FileGrantInvalidError): + _call(app, f"Bearer {passport}") + + +def test_content_token_audience_is_not_accepted_as_a_grant(app: Flask) -> None: + content_token = jwt.encode( + { + "aud": "dify-files-content", + "kind": "upload", + "file_id": "66666666-6666-4666-8666-666666666666", + "exp": int(time.time()) + 600, + }, + SECRET_KEY, + algorithm="HS256", + ) + + with pytest.raises(FileGrantInvalidError): + _call(app, f"Bearer {content_token}") + + +def test_expired_grant_is_rejected(app: Flask) -> None: + expired = jwt.encode( + { + "aud": FILE_GRANT_AUDIENCE, + "sub": END_USER_ID, + "tenant_id": TENANT_ID, + "app_id": APP_ID, + "scopes": ["upload"], + "exp": int(time.time()) - 1, + }, + SECRET_KEY, + algorithm="HS256", + ) + + with pytest.raises(FileGrantInvalidError): + _call(app, f"Bearer {expired}") + + +def test_grant_signed_with_another_key_is_rejected(app: Flask) -> None: + forged = jwt.encode( + { + "aud": FILE_GRANT_AUDIENCE, + "sub": END_USER_ID, + "tenant_id": TENANT_ID, + "app_id": APP_ID, + "scopes": ["upload"], + "exp": int(time.time()) + 600, + }, + "some-other-secret-long-enough-for-hs256-signing", + algorithm="HS256", + ) + + with pytest.raises(FileGrantInvalidError): + _call(app, f"Bearer {forged}") + + +def test_grant_without_the_required_scope_is_denied(app: Flask) -> None: + with pytest.raises(FileGrantScopeDeniedError): + _call(app, f"Bearer {_grant(FileGrantScope.RESOLVE, FileGrantScope.PRODUCE)}") diff --git a/api/tests/unit_tests/controllers/inner_api/app/test_file_grants.py b/api/tests/unit_tests/controllers/inner_api/app/test_file_grants.py new file mode 100644 index 00000000000..97b5e1adffc --- /dev/null +++ b/api/tests/unit_tests/controllers/inner_api/app/test_file_grants.py @@ -0,0 +1,493 @@ +"""Tests for the AppDeploy file grant minting endpoint.""" + +import inspect +import os +import time +from collections.abc import Callable, Iterator +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import patch + +import jwt +import pytest +from flask import Flask +from sqlalchemy import select +from sqlalchemy.engine import Engine +from sqlalchemy.orm import Session, sessionmaker + +from controllers.files.wraps import GrantedFileNotFoundError +from controllers.inner_api.app.file_grants import ( + EnterpriseFileGrantApi, + GrantAppNotFoundError, + GrantTtlTooLongError, + InvalidGrantRequestError, + InvalidSubjectError, +) +from extensions.ext_application_services import _build_file_grant_service +from extensions.storage.storage_type import StorageType +from libs.datetime_utils import naive_utc_now +from models.enums import CreatorUserRole, EndUserType +from models.model import App, EndUser, UploadFile +from models.tools import ToolFile +from services import end_user_service +from services.end_user_service import EndUserService +from services.file_grant_gateways import FILE_GRANT_AUDIENCE +from services.file_grant_service import ( + MAX_RUN_GRANT_TTL_SECONDS, + MAX_SESSION_GRANT_TTL_SECONDS, + MAX_WORKFLOW_EXECUTION_SECONDS, + RUN_GRANT_EXPIRY_GRACE_SECONDS, + FileGrantService, +) + +CONTROLLER_MODULE = "controllers.inner_api.app.file_grants" + +SECRET_KEY = "file-grant-test-secret-long-enough-for-hs256" +TENANT_ID = "11111111-1111-4111-8111-111111111111" +APP_ID = "22222222-2222-4222-8222-222222222222" +SUBJECT = "adp1.dGVzdC1zdWJqZWN0" + + +@pytest.fixture +def granted_config(config_overrides: Callable[..., None]) -> None: + config_overrides( + SECRET_KEY=SECRET_KEY, + FILES_URL="https://files.example.com", + INTERNAL_FILES_URL="http://dify-api.dify.svc:5001", + FILES_ACCESS_TIMEOUT=300, + ) + + +@pytest.fixture +def seeded_app(sqlite_session: Session) -> App: + app_model = App( + id=APP_ID, + tenant_id=TENANT_ID, + name="deployed app", + mode="workflow", + enable_site=True, + enable_api=True, + ) + sqlite_session.add(app_model) + sqlite_session.commit() + return app_model + + +def _mint(app: Flask, payload: dict[str, object]) -> dict[str, object]: + handler = EnterpriseFileGrantApi() + with app.test_request_context("/", method="POST", json=payload): + with patch(f"{CONTROLLER_MODULE}.inner_api_ns") as mock_ns: + mock_ns.payload = payload + return inspect.unwrap(handler.post)(handler) + + +def _subject_of(response: dict[str, object]) -> str: + grant = response["grant"] + assert isinstance(grant, str) + return str(jwt.decode(grant, SECRET_KEY, algorithms=["HS256"], audience=FILE_GRANT_AUDIENCE)["sub"]) + + +def _payload(**overrides: object) -> dict[str, object]: + return { + "tenant_id": TENANT_ID, + "app_id": APP_ID, + "subject": SUBJECT, + "is_anonymous": True, + "scopes": ["upload"], + "ttl_seconds": 600, + } | overrides + + +def _persist_upload_file(session: Session, *, owner_id: str, tenant_id: str = TENANT_ID) -> UploadFile: + upload_file = UploadFile( + tenant_id=tenant_id, + storage_type=StorageType.OPENDAL, + key="upload_files/report.pdf", + name="report.pdf", + size=2048, + extension="pdf", + mime_type="application/pdf", + created_by=owner_id, + created_by_role=CreatorUserRole.END_USER, + created_at=naive_utc_now(), + used=False, + ) + session.add(upload_file) + session.commit() + return upload_file + + +def _persist_tool_file(session: Session, *, owner_id: str, tenant_id: str = TENANT_ID) -> ToolFile: + tool_file = ToolFile( + user_id=owner_id, + tenant_id=tenant_id, + conversation_id=None, + file_key="tools/chart.png", + mimetype="image/png", + name="chart.png", + size=64, + ) + session.add(tool_file) + session.commit() + return tool_file + + +@pytest.fixture +def sqlite_db(sqlite_engine: Engine, granted_config: None) -> Iterator[None]: + del granted_config + service = _build_file_grant_service(database_client=sessionmaker(bind=sqlite_engine, expire_on_commit=False)) + services = SimpleNamespace(file_grants=service) + with patch(f"{CONTROLLER_MODULE}.application_services", return_value=services): + yield + + +@pytest.mark.usefixtures("granted_config", "seeded_app", "sqlite_db") +def test_mint_creates_exactly_one_end_user_and_reuses_it(app: Flask, sqlite_session: Session) -> None: + first = _mint(app, _payload()) + second = _mint(app, _payload(ttl_seconds=900)) + + end_users = list(sqlite_session.scalars(select(EndUser).where(EndUser.tenant_id == TENANT_ID)).all()) + assert len(end_users) == 1 + assert end_users[0].type == EndUserType.APP_DEPLOY + assert end_users[0].session_id == FileGrantService.session_id_for_subject(SUBJECT) + assert end_users[0].external_user_id == SUBJECT + + first_grant = first["grant"] + second_grant = second["grant"] + assert isinstance(first_grant, str) + assert isinstance(second_grant, str) + first_claims = jwt.decode(first_grant, SECRET_KEY, algorithms=["HS256"], audience=FILE_GRANT_AUDIENCE) + second_claims = jwt.decode(second_grant, SECRET_KEY, algorithms=["HS256"], audience=FILE_GRANT_AUDIENCE) + assert first_claims["sub"] == second_claims["sub"] == end_users[0].id + assert first_claims["tenant_id"] == TENANT_ID + assert first_claims["app_id"] == APP_ID + assert first_claims["scopes"] == ["upload"] + first_expires_at = first["expires_at"] + second_expires_at = second["expires_at"] + assert isinstance(first_expires_at, int) + assert isinstance(second_expires_at, int) + assert second_expires_at > first_expires_at + + +@pytest.mark.usefixtures("granted_config", "seeded_app", "sqlite_db") +def test_mint_returns_dify_upload_limits(app: Flask, config_overrides: Callable[..., None]) -> None: + config_overrides( + UPLOAD_FILE_SIZE_LIMIT=15, + UPLOAD_IMAGE_FILE_SIZE_LIMIT=10, + UPLOAD_AUDIO_FILE_SIZE_LIMIT=50, + UPLOAD_VIDEO_FILE_SIZE_LIMIT=100, + WORKFLOW_FILE_UPLOAD_LIMIT=10, + UPLOAD_FILE_BATCH_LIMIT=5, + ) + + response = _mint(app, _payload()) + + assert response["limits"] == { + "file_size_limit": 15, + "image_file_size_limit": 10, + "audio_file_size_limit": 50, + "video_file_size_limit": 100, + "workflow_file_upload_limit": 10, + "batch_count_limit": 5, + } + + +@pytest.mark.usefixtures("granted_config", "seeded_app", "sqlite_db") +def test_mint_rejects_ttl_over_the_cap_before_touching_identity(app: Flask, sqlite_session: Session) -> None: + with pytest.raises(GrantTtlTooLongError): + _mint(app, _payload(ttl_seconds=MAX_SESSION_GRANT_TTL_SECONDS + 1)) + + assert sqlite_session.scalars(select(EndUser)).all() == [] + + +@pytest.mark.usefixtures("granted_config", "seeded_app", "sqlite_db") +def test_mint_accepts_a_session_ttl_exactly_at_the_cap(app: Flask) -> None: + before = int(time.time()) + + response = _mint(app, _payload(ttl_seconds=MAX_SESSION_GRANT_TTL_SECONDS)) + + expires_at = response["expires_at"] + assert isinstance(expires_at, int) + assert MAX_SESSION_GRANT_TTL_SECONDS <= expires_at - before <= MAX_SESSION_GRANT_TTL_SECONDS + 5 + + +@pytest.mark.usefixtures("granted_config", "seeded_app", "sqlite_db") +def test_mint_accepts_a_run_ttl_until_deadline_plus_grace(app: Flask) -> None: + now = int(time.time()) + run_duration = MAX_RUN_GRANT_TTL_SECONDS - RUN_GRANT_EXPIRY_GRACE_SECONDS + + response = _mint( + app, + _payload( + scopes=["resolve", "produce"], + ttl_seconds=MAX_RUN_GRANT_TTL_SECONDS, + run_deadline=now + run_duration, + ), + ) + + expires_at = response["expires_at"] + assert isinstance(expires_at, int) + assert MAX_RUN_GRANT_TTL_SECONDS <= expires_at - now <= MAX_RUN_GRANT_TTL_SECONDS + 5 + + +@pytest.mark.usefixtures("granted_config", "seeded_app", "sqlite_db") +def test_mint_rejects_a_run_ttl_past_deadline_grace(app: Flask) -> None: + now = int(time.time()) + + with pytest.raises(GrantTtlTooLongError): + _mint( + app, + _payload( + scopes=["resolve", "produce"], + ttl_seconds=MAX_RUN_GRANT_TTL_SECONDS + 1, + run_deadline=now + MAX_WORKFLOW_EXECUTION_SECONDS, + ), + ) + + +@pytest.mark.usefixtures("granted_config", "seeded_app", "sqlite_db") +def test_mint_rejects_an_expired_run_deadline(app: Flask) -> None: + with pytest.raises(InvalidGrantRequestError): + _mint( + app, + _payload(scopes=["resolve", "produce"], ttl_seconds=1, run_deadline=int(time.time()) - 1), + ) + + +@pytest.mark.usefixtures("granted_config", "seeded_app", "sqlite_db") +def test_mint_rejects_a_run_deadline_without_produce_scope(app: Flask) -> None: + with pytest.raises(InvalidGrantRequestError): + _mint(app, _payload(run_deadline=int(time.time()) + 60)) + + +@pytest.mark.usefixtures("granted_config", "seeded_app", "sqlite_db") +def test_mint_rejects_a_run_deadline_beyond_the_workflow_limit(app: Flask) -> None: + with pytest.raises(InvalidGrantRequestError): + _mint( + app, + _payload( + scopes=["resolve", "produce"], + run_deadline=int(time.time()) + MAX_WORKFLOW_EXECUTION_SECONDS + 1, + ), + ) + + +@pytest.mark.usefixtures("granted_config", "seeded_app", "sqlite_db") +def test_mint_caps_a_run_grant_at_deadline_plus_grace(app: Flask) -> None: + now = int(time.time()) + run_deadline = now + 60 + + response = _mint( + app, + _payload( + scopes=["resolve", "produce"], + ttl_seconds=1200, + run_deadline=run_deadline, + ), + ) + + expires_at = response["expires_at"] + assert isinstance(expires_at, int) + assert ( + run_deadline + RUN_GRANT_EXPIRY_GRACE_SECONDS + <= expires_at + <= (run_deadline + RUN_GRANT_EXPIRY_GRACE_SECONDS + 1) + ) + + +@pytest.mark.usefixtures("granted_config", "seeded_app", "sqlite_db") +@pytest.mark.parametrize("ttl_seconds", [0, -1]) +def test_mint_rejects_a_non_positive_ttl(app: Flask, ttl_seconds: int) -> None: + with pytest.raises(InvalidGrantRequestError): + _mint(app, _payload(ttl_seconds=ttl_seconds)) + + +@pytest.mark.usefixtures("granted_config", "seeded_app", "sqlite_db") +@pytest.mark.parametrize("subject", ["", " ", "\t\n", "adp1.with\x00nul", "\x00"]) +def test_mint_rejects_an_unusable_subject(app: Flask, subject: str, sqlite_session: Session) -> None: + """A NUL would reach ``external_user_id`` verbatim and blow up on PostgreSQL.""" + + with pytest.raises(InvalidSubjectError): + _mint(app, _payload(subject=subject)) + + assert sqlite_session.scalars(select(EndUser)).all() == [] + + +@pytest.mark.usefixtures("granted_config", "seeded_app", "sqlite_db") +def test_mint_folds_an_oversized_subject_into_one_identity(app: Flask, sqlite_session: Session) -> None: + """``external_user_id`` truncates at 255, so ``session_id`` is what keeps them apart.""" + + long_subject = "adp1." + "s" * 4000 + sibling = long_subject[:-1] + "t" + + first = _mint(app, _payload(subject=long_subject)) + again = _mint(app, _payload(subject=long_subject)) + other = _mint(app, _payload(subject=sibling)) + + end_users = sqlite_session.scalars(select(EndUser).order_by(EndUser.created_at)).all() + assert len(end_users) == 2 + assert all(len(end_user.external_user_id) == 255 for end_user in end_users) + assert _subject_of(first) == _subject_of(again) != _subject_of(other) + + +@pytest.mark.usefixtures("granted_config", "sqlite_db") +def test_mint_rejects_unknown_app(app: Flask) -> None: + with pytest.raises(GrantAppNotFoundError): + _mint(app, _payload()) + + +@pytest.mark.usefixtures("granted_config", "seeded_app", "sqlite_db") +def test_mint_returns_strict_metadata_without_urls(app: Flask, sqlite_session: Session) -> None: + owner_id = _subject_of(_mint(app, _payload())) + upload_file = _persist_upload_file(sqlite_session, owner_id=owner_id) + tool_file = _persist_tool_file(sqlite_session, owner_id=owner_id) + + response = _mint( + app, + _payload(file_ids=[{"id": upload_file.id, "kind": "upload"}, {"id": tool_file.id, "kind": "tool"}]), + ) + + assert response["files"] == [ + { + "id": upload_file.id, + "kind": "upload", + "name": "report.pdf", + "size": 2048, + "extension": "pdf", + "mime_type": "application/pdf", + }, + { + "id": tool_file.id, + "kind": "tool", + "name": "chart.png", + "size": 64, + "extension": "png", + "mime_type": "image/png", + }, + ] + + +@pytest.mark.usefixtures("granted_config", "seeded_app", "sqlite_db") +def test_mint_fails_the_whole_strict_batch_on_one_miss(app: Flask, sqlite_session: Session) -> None: + owner_id = _subject_of(_mint(app, _payload())) + upload_file = _persist_upload_file(sqlite_session, owner_id=owner_id) + + with pytest.raises(GrantedFileNotFoundError): + _mint( + app, + _payload( + file_ids=[ + {"id": upload_file.id, "kind": "upload"}, + {"id": "33333333-3333-4333-8333-333333333333", "kind": "upload"}, + ] + ), + ) + + +@pytest.mark.usefixtures("granted_config", "seeded_app", "sqlite_db") +def test_mint_hides_files_owned_by_another_subject(app: Flask, sqlite_session: Session) -> None: + other_owner = _subject_of(_mint(app, _payload(subject="adp1.other"))) + foreign_file = _persist_upload_file(sqlite_session, owner_id=other_owner) + + with pytest.raises(GrantedFileNotFoundError): + _mint(app, _payload(file_ids=[{"id": foreign_file.id, "kind": "upload"}])) + + +@pytest.mark.usefixtures("granted_config", "seeded_app", "sqlite_db") +def test_mint_reports_optional_files_item_by_item(app: Flask, sqlite_session: Session) -> None: + owner_id = _subject_of(_mint(app, _payload())) + upload_file = _persist_upload_file(sqlite_session, owner_id=owner_id) + missing_id = "44444444-4444-4444-8444-444444444444" + + response = _mint( + app, + _payload( + optional_file_ids=[ + {"id": upload_file.id, "kind": "upload"}, + {"id": missing_id, "kind": "tool"}, + ] + ), + ) + + optional_files = response["optional_files"] + assert isinstance(optional_files, list) + present, absent = optional_files + assert present["ok"] is True + assert present["name"] == "report.pdf" + assert present["url"].startswith(f"https://files.example.com/files/appdeploy/{upload_file.id}/content?token=") + assert present["internal_url"].startswith( + f"http://dify-api.dify.svc:5001/files/appdeploy/{upload_file.id}/content?token=" + ) + assert absent == { + "id": missing_id, + "ok": False, + "kind": None, + "name": None, + "size": None, + "extension": None, + "mime_type": None, + "url": None, + "internal_url": None, + "error": "not_found", + } + assert response["files"] == [] + + +@pytest.mark.usefixtures("seeded_app") +def test_end_user_service_never_retypes_an_app_deploy_row( + sqlite_engine: Engine, + sqlite_session: Session, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Retyping would hide the row from the grant read and strand its files.""" + + subject = "subject-that-also-reaches-the-service-api" + session_id = FileGrantService.session_id_for_subject(subject) + owner = EndUser( + tenant_id=TENANT_ID, + app_id=APP_ID, + type=EndUserType.APP_DEPLOY, + is_anonymous=True, + session_id=session_id, + external_user_id=session_id, + ) + sqlite_session.add(owner) + sqlite_session.commit() + owner_id = owner.id + monkeypatch.setattr(end_user_service, "db", SimpleNamespace(engine=sqlite_engine)) + + EndUserService.get_or_create_end_user_by_type(EndUserType.SERVICE_API, TENANT_ID, APP_ID, session_id) + + sqlite_session.expire_all() + persisted_owner = sqlite_session.get(EndUser, owner_id) + assert persisted_owner is not None + assert persisted_owner.type == EndUserType.APP_DEPLOY + + +SKIPPED_DIRECTORY_NAMES = frozenset({".git", ".venv", "__pycache__", "migrations", "node_modules", "tests"}) + + +def test_app_deploy_end_users_have_exactly_one_writer() -> None: + """``end_users`` has no unique constraint, so a second writer would fork identities. + + ``end_user_service`` names the type only to exclude it from the legacy retype; + the behavioural guard above is what holds that exclusion in place. + """ + + api_root = Path(__file__).resolve().parents[5] + assert api_root.name == "api" + + referencing_modules: set[str] = set() + for directory, subdirectories, filenames in os.walk(api_root): + subdirectories[:] = [name for name in subdirectories if name not in SKIPPED_DIRECTORY_NAMES] + for filename in filenames: + if not filename.endswith(".py"): + continue + path = Path(directory) / filename + if "EndUserType.APP_DEPLOY" in path.read_text(encoding="utf-8"): + referencing_modules.add(path.relative_to(api_root).as_posix()) + + assert referencing_modules == { + "repositories/file_grant_repository.py", + "services/end_user_service.py", + } diff --git a/api/tests/unit_tests/controllers/openapi/conftest.py b/api/tests/unit_tests/controllers/openapi/conftest.py index 3a900011f54..9f60b5979a4 100644 --- a/api/tests/unit_tests/controllers/openapi/conftest.py +++ b/api/tests/unit_tests/controllers/openapi/conftest.py @@ -18,6 +18,7 @@ def _stub_execute( scope=None, allowed_token_types=None, edition=None, + require_valid_enterprise_license=False, workspace_membership=False, allowed_roles=None, rbac=None, diff --git a/api/tests/unit_tests/controllers/openapi/test_account.py b/api/tests/unit_tests/controllers/openapi/test_account.py index d4b28fa554e..a623d29ca69 100644 --- a/api/tests/unit_tests/controllers/openapi/test_account.py +++ b/api/tests/unit_tests/controllers/openapi/test_account.py @@ -2,13 +2,12 @@ import builtins import sys -import uuid from types import SimpleNamespace import pytest from flask import Flask from flask.views import MethodView -from werkzeug.exceptions import UnprocessableEntity +from werkzeug.exceptions import NotFound, UnprocessableEntity from controllers.openapi import bp as openapi_bp from controllers.openapi.account import ( @@ -17,8 +16,8 @@ from controllers.openapi.account import ( AccountSessionsApi, AccountSessionsSelfApi, ) -from controllers.openapi.auth.data import AuthData -from libs.oauth_bearer import Scope, TokenType +from machinery.context import AccountRequestContext +from services.entities.account_access_entities import AccountSessionPage if not hasattr(builtins, "MethodView"): builtins.MethodView = MethodView # type: ignore[attr-defined] @@ -88,96 +87,47 @@ def test_session_by_id_dispatches_to_correct_class(openapi_app: Flask): assert "DELETE" in rule.methods -def test_subject_match_for_account_filters_by_account_id(): - """Account subject scopes queries via account_id.""" - import uuid as _uuid - - from libs.oauth_bearer import AuthContext, SubjectType, TokenType - from services.oauth_device_flow import subject_match_clauses - - aid = _uuid.uuid4() - ctx = AuthContext( - subject_type=SubjectType.ACCOUNT, - subject_email="user@example.com", - subject_issuer="dify:account", - account_id=aid, - client_id="difyctl", - scopes=frozenset({"full"}), - token_id=_uuid.uuid4(), - token_type=TokenType.OAUTH_ACCOUNT, - expires_at=None, - token_hash="h1", - verified_tenants={}, - ) - clauses = subject_match_clauses(ctx) - # One predicate, on account_id - assert len(clauses) == 1 - assert "account_id" in str(clauses[0]) - - -def test_subject_match_for_external_sso_filters_by_email_and_issuer(): - """External SSO subject scopes via (subject_email, subject_issuer) - AND account_id IS NULL — so a same-email account row from a - federated tenant cannot be revoked through an SSO bearer. - """ - import uuid as _uuid - - from libs.oauth_bearer import AuthContext, SubjectType, TokenType - from services.oauth_device_flow import subject_match_clauses - - ctx = AuthContext( - subject_type=SubjectType.EXTERNAL_SSO, - subject_email="sso@partner.com", - subject_issuer="https://idp.partner.com", - account_id=None, - client_id="difyctl", - scopes=frozenset({"apps:run"}), - token_id=_uuid.uuid4(), - token_type=TokenType.OAUTH_EXTERNAL_SSO, - expires_at=None, - token_hash="h1", - verified_tenants={}, - ) - clauses = subject_match_clauses(ctx) - assert len(clauses) == 3 - rendered = " ".join(str(c) for c in clauses) - assert "subject_email" in rendered - assert "subject_issuer" in rendered - assert "account_id IS NULL" in rendered +def test_session_by_id_rejects_malformed_uuid(app: Flask) -> None: + api = AccountSessionByIdApi() + with app.test_request_context("/openapi/v1/account/sessions/not-a-uuid", method="DELETE"): + with pytest.raises(NotFound, match="session not found"): + api.delete.__wrapped__(api, _request_context(), session_id="not-a-uuid") # --- GET /account/sessions query validation (the handler routes ?page/?limit through -# SessionListQuery so the server enforces the bounds the contract advertises). The auth ctx and -# DB read are stubbed so these exercise only the validation + paging path; __wrapped__ skips the -# auth guard, which is covered separately in auth/. --- +# SessionListQuery so the server enforces the bounds the contract advertises). The application +# service is replaced with a small fake so these exercise only parsing and serialization; +# __wrapped__ skips the complete Admission boundary. --- _ACCOUNT_MOD = "controllers.openapi.account" -def _session_auth_data() -> AuthData: - return AuthData( - token_type=TokenType.OAUTH_ACCOUNT, - account_id=uuid.uuid4(), - token_hash="test", - token_id=uuid.uuid4(), - scopes=frozenset({Scope.FULL}), - required_scope=Scope.FULL, - allowed_roles=None, +def _request_context() -> AccountRequestContext: + return AccountRequestContext( + request_id="request-1", + trace_id="trace-1", + account_id="account-1", + access_token_id="token-1", ) -def _stub_session_deps(monkeypatch: pytest.MonkeyPatch, rows): +class _SessionListService: + def list_sessions(self, _context: AccountRequestContext, *, page: int, limit: int) -> AccountSessionPage: + return AccountSessionPage(page=page, limit=limit, total=0, items=()) + + +def _stub_account_service(monkeypatch: pytest.MonkeyPatch) -> None: mod = sys.modules[_ACCOUNT_MOD] - monkeypatch.setattr(mod, "get_auth_ctx", lambda: SimpleNamespace()) - monkeypatch.setattr(mod, "list_active_sessions", lambda *args, **kwargs: rows) + services = SimpleNamespace(accounts=SimpleNamespace(access=_SessionListService())) + monkeypatch.setattr(mod, "application_services", lambda: services) def test_sessions_list_valid_query_parses_page_and_limit(app: Flask, monkeypatch: pytest.MonkeyPatch): """A valid ?page&limit round-trips through SessionListQuery into the response envelope.""" api = AccountSessionsApi() - _stub_session_deps(monkeypatch, []) + _stub_account_service(monkeypatch) with app.test_request_context("/openapi/v1/account/sessions?page=2&limit=5"): - body, status = api.get.__wrapped__(api, auth_data=_session_auth_data()) + body, status = api.get.__wrapped__(api, _request_context()) assert status == 200 assert body["page"] == 2 assert body["limit"] == 5 @@ -188,9 +138,9 @@ def test_sessions_list_valid_query_parses_page_and_limit(app: Flask, monkeypatch def test_sessions_list_defaults_when_query_omitted(app: Flask, monkeypatch: pytest.MonkeyPatch): """No query → the model's defaults (page=1, limit=100) drive the envelope.""" api = AccountSessionsApi() - _stub_session_deps(monkeypatch, []) + _stub_account_service(monkeypatch) with app.test_request_context("/openapi/v1/account/sessions"): - body, status = api.get.__wrapped__(api, auth_data=_session_auth_data()) + body, status = api.get.__wrapped__(api, _request_context()) assert status == 200 assert body["page"] == 1 assert body["limit"] == 100 @@ -210,7 +160,7 @@ def test_sessions_list_defaults_when_query_omitted(app: Flask, monkeypatch: pyte def test_sessions_list_rejects_out_of_bounds_query(app: Flask, monkeypatch: pytest.MonkeyPatch, query): """Out-of-range / unknown query params raise 422 instead of being silently coerced.""" api = AccountSessionsApi() - _stub_session_deps(monkeypatch, []) + _stub_account_service(monkeypatch) with app.test_request_context(f"/openapi/v1/account/sessions?{query}"): with pytest.raises(UnprocessableEntity): - api.get.__wrapped__(api, auth_data=_session_auth_data()) + api.get.__wrapped__(api, _request_context()) diff --git a/api/tests/unit_tests/controllers/openapi/test_flask_admission.py b/api/tests/unit_tests/controllers/openapi/test_flask_admission.py new file mode 100644 index 00000000000..fb18eea75ae --- /dev/null +++ b/api/tests/unit_tests/controllers/openapi/test_flask_admission.py @@ -0,0 +1,116 @@ +from __future__ import annotations + +from collections.abc import Callable +from functools import wraps +from uuid import UUID + +import pytest +from flask import Flask +from werkzeug.exceptions import Unauthorized + +from controllers.openapi import flask_admission +from controllers.openapi.auth.data import AuthData +from enums import DeploymentEdition +from libs.oauth_bearer import Scope, TokenType +from libs.rate_limit import LIMIT_ME_PER_ACCOUNT +from machinery.context import AccountRequestContext +from models.account import Account, AccountStatus + + +def _auth_data(*, status: AccountStatus = AccountStatus.ACTIVE) -> AuthData: + account = Account(name="Ada", email="ada@example.com", status=status) + account.id = "11111111-1111-1111-1111-111111111111" + return AuthData( + token_type=TokenType.OAUTH_ACCOUNT, + account_id=UUID(account.id), + token_hash="hash-1", + token_id=UUID("22222222-2222-2222-2222-222222222222"), + scopes=frozenset({Scope.FULL}), + caller=account, + ) + + +def _install_fake_transport( + monkeypatch: pytest.MonkeyPatch, + auth_data: AuthData, + captured: dict[str, object], +) -> None: + def guard(**requirements: object) -> Callable[[Callable[..., object]], Callable[..., object]]: + captured.update(requirements) + + def decorator(view: Callable[..., object]) -> Callable[..., object]: + @wraps(view) + def admitted(*args: object, **kwargs: object) -> object: + return view(*args, auth_data=auth_data, **kwargs) + + return admitted + + return decorator + + monkeypatch.setattr(flask_admission.auth_router, "guard", guard) + + +def test_admission_builds_stable_request_context( + monkeypatch: pytest.MonkeyPatch, +) -> None: + captured: dict[str, object] = {} + limited: list[tuple[object, str]] = [] + _install_fake_transport(monkeypatch, _auth_data(), captured) + monkeypatch.setattr(flask_admission, "get_request_id", lambda: "request-1") + monkeypatch.setattr(flask_admission, "get_trace_id", lambda: "trace-1") + monkeypatch.setattr(flask_admission, "enforce", lambda spec, *, key: limited.append((spec, key))) + + @flask_admission.openapi_account_admission( + scope=Scope.FULL, + editions=frozenset({DeploymentEdition.ENTERPRISE}), + rate_limit=LIMIT_ME_PER_ACCOUNT, + ) + def view(_self: object, context: AccountRequestContext) -> AccountRequestContext: + return context + + with Flask(__name__).test_request_context("/openapi/v1/account"): + context = view(object()) + + assert context == AccountRequestContext( + request_id="request-1", + trace_id="trace-1", + account_id="11111111-1111-1111-1111-111111111111", + access_token_id="22222222-2222-2222-2222-222222222222", + ) + assert captured == { + "scope": Scope.FULL, + "allowed_token_types": frozenset({TokenType.OAUTH_ACCOUNT}), + "edition": frozenset({DeploymentEdition.ENTERPRISE}), + "require_valid_enterprise_license": True, + } + assert limited == [(LIMIT_ME_PER_ACCOUNT, "account:11111111-1111-1111-1111-111111111111")] + + +def test_admission_rejects_uninitialized_account(monkeypatch: pytest.MonkeyPatch) -> None: + _install_fake_transport(monkeypatch, _auth_data(status=AccountStatus.UNINITIALIZED), {}) + + @flask_admission.openapi_account_admission(scope=Scope.FULL) + def view(_self: object, _context: AccountRequestContext) -> None: + raise AssertionError("view must not run") + + with Flask(__name__).test_request_context("/openapi/v1/account"): + with pytest.raises(Unauthorized, match="account not initialized"): + view(object()) + + +def test_admission_rejects_missing_auth_data(monkeypatch: pytest.MonkeyPatch) -> None: + def guard(**_requirements: object) -> Callable[[Callable[..., object]], Callable[..., object]]: + def decorator(view: Callable[..., object]) -> Callable[..., object]: + return view + + return decorator + + monkeypatch.setattr(flask_admission.auth_router, "guard", guard) + + @flask_admission.openapi_account_admission(scope=Scope.FULL) + def view(_self: object, _context: AccountRequestContext) -> None: + raise AssertionError("view must not run") + + with Flask(__name__).test_request_context("/openapi/v1/account"): + with pytest.raises(RuntimeError, match="did not provide valid AuthData"): + view(object()) diff --git a/api/tests/unit_tests/controllers/service_api/app/test_workflow.py b/api/tests/unit_tests/controllers/service_api/app/test_workflow.py index 9dd436d8795..1c44c2397db 100644 --- a/api/tests/unit_tests/controllers/service_api/app/test_workflow.py +++ b/api/tests/unit_tests/controllers/service_api/app/test_workflow.py @@ -53,6 +53,7 @@ from models import Account from models.enums import CreatorUserRole, WorkflowRunTriggeredFrom from models.model import App, AppMode, EndUser from models.workflow import WorkflowAppLog, WorkflowAppLogCreatedFrom, WorkflowRun, WorkflowType +from repositories.workflow_app_log_query_repository import WorkflowAppLogQueryRepository from services.app_generate_service import AppGenerateService from services.billing_service import BillingService from services.errors.app import ( @@ -63,7 +64,7 @@ from services.errors.app import ( TriggerWorkflowServiceModeUnavailableError as TriggerWorkflowServiceModeUnavailableServiceError, ) from services.errors.llm import InvokeRateLimitError -from services.workflow_app_service import WorkflowAppService +from services.workflow_app_log_query_service import WorkflowAppLogQueryService def _default_workflow_inputs() -> dict[str, object]: @@ -162,8 +163,11 @@ def _persist_workflow_log( app_id: str, ) -> None: workflow_run_id = "log-run-1" + account = Account(name="Log Account", email="log-account@example.com") + account.id = "account-1" sqlite_session.add_all( [ + account, _make_workflow_run( run_id=workflow_run_id, tenant_id=tenant_id, @@ -206,7 +210,11 @@ def _expected_workflow_log_pagination_payload() -> dict[str, object]: "details": None, "created_from": "service-api", "created_by_role": "account", - "created_by_account": None, + "created_by_account": { + "id": "account-1", + "name": "Log Account", + "email": "log-account@example.com", + }, "created_by_end_user": None, "created_at": int(datetime(2026, 1, 1, 1, 0, 3).timestamp()), } @@ -214,6 +222,18 @@ def _expected_workflow_log_pagination_payload() -> dict[str, object]: } +def _stub_workflow_app_logs(monkeypatch: pytest.MonkeyPatch) -> MagicMock: + workflow_app_logs = MagicMock() + workflow_app_logs.list_logs.return_value = _expected_workflow_log_pagination_payload() + services = SimpleNamespace(workflow_app_logs=workflow_app_logs) + monkeypatch.setattr( + sys.modules["controllers.service_api.app.workflow"], + "application_services", + lambda: services, + ) + return workflow_app_logs + + class TestWorkflowRunPayload: """Test suite for WorkflowRunPayload Pydantic model.""" @@ -355,46 +375,6 @@ class TestWorkflowRunResponse: } -class TestWorkflowAppService: - """Test WorkflowAppService interface.""" - - def test_service_exists(self): - """Test WorkflowAppService class exists.""" - service = WorkflowAppService() - assert service is not None - - def test_get_paginate_workflow_app_logs_method_exists(self): - """Test get_paginate_workflow_app_logs method exists.""" - assert hasattr(WorkflowAppService, "get_paginate_workflow_app_logs") - assert callable(WorkflowAppService.get_paginate_workflow_app_logs) - - @pytest.mark.parametrize("sqlite_session", [(WorkflowAppLog,)], indirect=True) - def test_get_paginate_workflow_app_logs_returns_pagination(self, sqlite_session: Session): - """Test pagination returns committed logs scoped to the requested app.""" - log = _make_workflow_app_log() - sqlite_session.add(log) - sqlite_session.commit() - service = WorkflowAppService() - result = service.get_paginate_workflow_app_logs( - session=sqlite_session, - app_model=_make_app_model(), - keyword=None, - status=None, - created_at_before=None, - created_at_after=None, - page=1, - limit=20, - created_by_end_user_session_id=None, - created_by_account=None, - ) - - assert result["page"] == 1 - assert result["limit"] == 20 - assert result["total"] == 1 - assert result["has_more"] is False - assert [item.id for item in result["data"]] == [log.id] - - class TestWorkflowExecutionStatus: """Test WorkflowExecutionStatus enum.""" @@ -847,20 +827,26 @@ class TestWorkflowTaskStopApi: class TestWorkflowAppLogApi: - @pytest.mark.parametrize("sqlite_session", [(WorkflowRun, WorkflowAppLog, Account)], indirect=True) def test_success( self, app: Flask, monkeypatch: pytest.MonkeyPatch, - sqlite_engine: Engine, sqlite_session: Session, + sqlite_session_factory: sessionmaker[Session], ) -> None: - _persist_workflow_log(sqlite_session, tenant_id="tenant-1", app_id="a1") - _bind_sqlite_database(monkeypatch, sqlite_engine, sqlite_session) - api = WorkflowAppLogApi() handler = unwrap(api.get) app_model = _make_app_model(app_id="a1") + _persist_workflow_log(sqlite_session, tenant_id=app_model.tenant_id, app_id=app_model.id) + + workflow_app_logs = WorkflowAppLogQueryService( + logs=WorkflowAppLogQueryRepository(session_factory=sqlite_session_factory), + ) + monkeypatch.setattr( + sys.modules["controllers.service_api.app.workflow"], + "application_services", + lambda: SimpleNamespace(workflow_app_logs=workflow_app_logs), + ) with app.test_request_context("/workflows/logs", method="GET"): response = handler(api, app_model=app_model) @@ -994,18 +980,14 @@ class TestWorkflowAppLogApiGet: ``get`` is wrapped by ``@validate_app_token``. """ - @pytest.mark.parametrize("sqlite_session", [(WorkflowRun, WorkflowAppLog, Account)], indirect=True) def test_get_workflow_logs_success( self, app: Flask, workflow_app: App, monkeypatch: pytest.MonkeyPatch, - sqlite_engine: Engine, - sqlite_session: Session, ): """Test successful workflow log retrieval.""" - _persist_workflow_log(sqlite_session, tenant_id=workflow_app.tenant_id, app_id=workflow_app.id) - _bind_sqlite_database(monkeypatch, sqlite_engine, sqlite_session) + _stub_workflow_app_logs(monkeypatch) from controllers.service_api.app.workflow import WorkflowAppLogApi diff --git a/api/tests/unit_tests/controllers/service_api/dataset/test_dataset_apis.py b/api/tests/unit_tests/controllers/service_api/dataset/test_dataset_apis.py index 581f95d8ce5..8b3c7d42c79 100644 --- a/api/tests/unit_tests/controllers/service_api/dataset/test_dataset_apis.py +++ b/api/tests/unit_tests/controllers/service_api/dataset/test_dataset_apis.py @@ -257,6 +257,66 @@ class TestDatasetListApiGet: False, ) + @patch("controllers.service_api.dataset.dataset.create_plugin_provider_manager") + @patch("controllers.service_api.dataset.dataset.DatasetService") + def test_list_datasets_has_more_false_on_last_page_exact_limit( + self, + mock_dataset_svc: MagicMock, + mock_provider_mgr: MagicMock, + app: Flask, + account: Account, + tenant: Tenant, + controller_session: Session, + ) -> None: + """A full last page must set has_more false instead of forcing another fetch.""" + from controllers.service_api.dataset.dataset import DatasetListApi + + page_size = 20 + dataset = make_dataset(controller_session, tenant, account) + mock_dataset_svc.get_datasets.return_value = ([dataset] * page_size, page_size) + mock_provider_mgr.return_value.get_configurations.return_value.get_models.return_value = list[object]() + + with app.test_request_context(f"/datasets?page=1&limit={page_size}", method="GET"): + api = DatasetListApi() + response, status = unwrap(api.get)(api, controller_session, tenant_id=tenant.id) + + assert status == 200 + assert response["has_more"] is False + assert response["limit"] == page_size + assert response["total"] == page_size + assert response["page"] == 1 + + @patch("controllers.service_api.dataset.dataset.create_plugin_provider_manager") + @patch("controllers.service_api.dataset.dataset.DatasetService") + def test_list_datasets_has_more_true_when_limit_exceeds_cap( + self, + mock_dataset_svc: MagicMock, + mock_provider_mgr: MagicMock, + app: Flask, + account: Account, + tenant: Tenant, + controller_session: Session, + ) -> None: + """limit>100 still reports remaining rows after the server cap of 100.""" + from controllers.service_api.dataset.dataset import DatasetListApi + + returned_count = 100 + total = 150 + dataset = make_dataset(controller_session, tenant, account) + mock_dataset_svc.get_datasets.return_value = ([dataset] * returned_count, total) + mock_provider_mgr.return_value.get_configurations.return_value.get_models.return_value = list[object]() + + with app.test_request_context("/datasets?page=1&limit=200", method="GET"): + api = DatasetListApi() + response, status = unwrap(api.get)(api, controller_session, tenant_id=tenant.id) + + assert status == 200 + assert response["has_more"] is True + assert response["limit"] == 100 + assert response["total"] == total + assert response["page"] == 1 + assert mock_dataset_svc.get_datasets.call_args.args[1] == 100 + class TestDatasetListApiPost: """Test suite for DatasetListApi.post() endpoint.""" diff --git a/api/tests/unit_tests/controllers/service_api/dataset/test_dataset_segment.py b/api/tests/unit_tests/controllers/service_api/dataset/test_dataset_segment.py index fc5d540ce72..ca2fe02c02d 100644 --- a/api/tests/unit_tests/controllers/service_api/dataset/test_dataset_segment.py +++ b/api/tests/unit_tests/controllers/service_api/dataset/test_dataset_segment.py @@ -969,22 +969,28 @@ class TestSegmentPagination: assert limit >= 1 assert limit <= 100 - def test_has_more_calculation(self): - """Test has_more pagination flag calculation.""" - segments_count = 20 + def test_has_more_false_on_last_page_exact_limit(self): + """Last page that fills the limit exactly must not claim more rows.""" + page = 1 limit = 20 + total = 20 + effective_limit = min(limit, 100) - has_more = segments_count == limit - assert has_more is True - - def test_no_more_when_incomplete_page(self): - """Test has_more is False for incomplete page.""" - segments_count = 15 - limit = 20 - - has_more = segments_count == limit + has_more = page * effective_limit < total + assert effective_limit == 20 assert has_more is False + def test_has_more_true_when_limit_exceeds_cap_with_remaining_rows(self): + """Capped pages must still report remaining rows after the first 100.""" + page = 1 + limit = 200 + total = 150 + effective_limit = min(limit, 100) + + has_more = page * effective_limit < total + assert effective_limit == 100 + assert has_more is True + # ============================================================================= # API Endpoint Tests @@ -1070,6 +1076,91 @@ class TestSegmentApiGet(SQLiteEndpointTest): mock_dump_segments.assert_called_once_with([mock_segment], {}, session=ANY) assert isinstance(mock_dump_segments.call_args.kwargs["session"], Session) + @patch("controllers.service_api.dataset.segment.segment_responses_with_summaries") + @patch("controllers.service_api.dataset.segment.SummaryIndexService.get_segments_summaries") + @patch("controllers.service_api.dataset.segment.SegmentService") + @patch("controllers.service_api.dataset.segment.DocumentService") + @patch("controllers.service_api.dataset.segment.current_account_with_tenant") + def test_list_segments_has_more_false_on_last_page_exact_limit( + self, + mock_account_fn, + mock_doc_svc, + mock_seg_svc, + mock_get_summaries, + mock_dump_segments, + app: Flask, + mock_tenant, + mock_dataset, + mock_segment, + ): + """A full last page must set has_more false instead of forcing another fetch.""" + mock_account_fn.return_value = (_account(), mock_tenant.id) + self._persist_dataset(mock_dataset, mock_tenant.id) + mock_doc_svc.get_document.return_value = _document_for_dataset( + mock_dataset, doc_form=IndexStructureType.PARAGRAPH_INDEX + ) + page_size = 20 + segments = [mock_segment] * page_size + mock_seg_svc.get_segments.return_value = (segments, page_size) + mock_get_summaries.return_value = {} + mock_dump_segments.return_value = [_segment_response_dict() for _ in range(page_size)] + + with app.test_request_context( + f"/datasets/{mock_dataset.id}/documents/doc-id/segments?page=1&limit={page_size}", + method="GET", + ): + api = SegmentApi() + response, status = api.get(tenant_id=mock_tenant.id, dataset_id=mock_dataset.id, document_id="doc-id") + + assert status == 200 + assert response["has_more"] is False + assert response["limit"] == page_size + assert response["total"] == page_size + assert response["page"] == 1 + + @patch("controllers.service_api.dataset.segment.segment_responses_with_summaries") + @patch("controllers.service_api.dataset.segment.SummaryIndexService.get_segments_summaries") + @patch("controllers.service_api.dataset.segment.SegmentService") + @patch("controllers.service_api.dataset.segment.DocumentService") + @patch("controllers.service_api.dataset.segment.current_account_with_tenant") + def test_list_segments_has_more_true_when_limit_exceeds_cap( + self, + mock_account_fn, + mock_doc_svc, + mock_seg_svc, + mock_get_summaries, + mock_dump_segments, + app: Flask, + mock_tenant, + mock_dataset, + mock_segment, + ): + """limit>100 still reports remaining rows after the server cap of 100.""" + mock_account_fn.return_value = (_account(), mock_tenant.id) + self._persist_dataset(mock_dataset, mock_tenant.id) + mock_doc_svc.get_document.return_value = _document_for_dataset( + mock_dataset, doc_form=IndexStructureType.PARAGRAPH_INDEX + ) + returned_count = 100 + total = 150 + segments = [mock_segment] * returned_count + mock_seg_svc.get_segments.return_value = (segments, total) + mock_get_summaries.return_value = {} + mock_dump_segments.return_value = [_segment_response_dict() for _ in range(returned_count)] + + with app.test_request_context( + f"/datasets/{mock_dataset.id}/documents/doc-id/segments?page=1&limit=200", + method="GET", + ): + api = SegmentApi() + response, status = api.get(tenant_id=mock_tenant.id, dataset_id=mock_dataset.id, document_id="doc-id") + + assert status == 200 + assert response["has_more"] is True + assert response["limit"] == 100 + assert response["total"] == total + assert response["page"] == 1 + @patch("controllers.service_api.dataset.segment.current_account_with_tenant") def test_list_segments_dataset_not_found(self, mock_account_fn, app, mock_tenant, mock_dataset): """Test 404 when dataset not found.""" diff --git a/api/tests/unit_tests/controllers/service_api/dataset/test_document.py b/api/tests/unit_tests/controllers/service_api/dataset/test_document.py index a2e8d760feb..c297b95987e 100644 --- a/api/tests/unit_tests/controllers/service_api/dataset/test_document.py +++ b/api/tests/unit_tests/controllers/service_api/dataset/test_document.py @@ -1189,6 +1189,76 @@ class TestDocumentListApi(SQLiteControllerTest): assert "data_source_info_dict" not in response["data"][0] assert "doc_metadata_details" not in response["data"][0] + @patch("controllers.service_api.dataset.document.paginate_query") + @patch("controllers.service_api.dataset.document.DocumentService") + def test_list_documents_has_more_false_on_last_page_exact_limit( + self, mock_doc_svc, mock_paginate, app: Flask, mock_tenant, mock_dataset + ): + """A full last page must set has_more false instead of forcing another fetch.""" + self._persist_dataset(mock_dataset) + page_size = 20 + documents = [ + make_serializable_document( + id=f"doc-{index}", + name=f"Document {index}", + tenant_id=mock_tenant, + dataset_id=mock_dataset.id, + ) + for index in range(page_size) + ] + mock_paginate.return_value = _PaginationRecord(items=documents, total=page_size) + mock_doc_svc.enrich_documents_with_summary_index_status.return_value = None + + with app.test_request_context( + f"/datasets/{mock_dataset.id}/documents?page=1&limit={page_size}", + method="GET", + ): + api = DocumentListApi() + response = inspect.unwrap(type(api).get)( + api, self.session, tenant_id=mock_tenant, dataset_id=mock_dataset.id + ) + + assert response["has_more"] is False + assert response["limit"] == page_size + assert response["total"] == page_size + assert response["page"] == 1 + + @patch("controllers.service_api.dataset.document.paginate_query") + @patch("controllers.service_api.dataset.document.DocumentService") + def test_list_documents_has_more_true_when_limit_exceeds_cap( + self, mock_doc_svc, mock_paginate, app: Flask, mock_tenant, mock_dataset + ): + """limit>100 still reports remaining rows after the server cap of 100.""" + self._persist_dataset(mock_dataset) + returned_count = 100 + total = 150 + documents = [ + make_serializable_document( + id=f"doc-{index}", + name=f"Document {index}", + tenant_id=mock_tenant, + dataset_id=mock_dataset.id, + ) + for index in range(returned_count) + ] + mock_paginate.return_value = _PaginationRecord(items=documents, total=total) + mock_doc_svc.enrich_documents_with_summary_index_status.return_value = None + + with app.test_request_context( + f"/datasets/{mock_dataset.id}/documents?page=1&limit=200", + method="GET", + ): + api = DocumentListApi() + response = inspect.unwrap(type(api).get)( + api, self.session, tenant_id=mock_tenant, dataset_id=mock_dataset.id + ) + + assert response["has_more"] is True + assert response["limit"] == 100 + assert response["total"] == total + assert response["page"] == 1 + assert mock_paginate.call_args.kwargs["per_page"] == 100 + def test_list_documents_dataset_not_found(self, app: Flask, mock_tenant, mock_dataset): """Test 404 when dataset not found.""" # Arrange diff --git a/api/tests/unit_tests/controllers/web/test_app.py b/api/tests/unit_tests/controllers/web/test_app.py index 005c23e7f25..9b1247bf431 100644 --- a/api/tests/unit_tests/controllers/web/test_app.py +++ b/api/tests/unit_tests/controllers/web/test_app.py @@ -216,18 +216,17 @@ class TestAppWebAuthPermission: extract_passport.assert_not_called() @pytest.mark.parametrize( - ("decoded", "expected_user_id", "allowed"), + ("user_id", "allowed"), [ - pytest.param({"user_id": "user-1"}, "user-1", True, id="identified-user"), - pytest.param({}, "visitor", False, id="visitor-fallback"), + pytest.param("user-1", True, id="allowed-user"), + pytest.param("user-2", False, id="denied-user"), ], ) @patch("controllers.web.app.application_services") def test_checks_private_app_permission( self, application_services: MagicMock, - decoded: dict[str, str], - expected_user_id: str, + user_id: str, allowed: bool, app: Flask, ) -> None: @@ -241,14 +240,41 @@ class TestAppWebAuthPermission: patch("controllers.web.app.extract_webapp_passport", return_value="passport") as extract_passport, patch("controllers.web.app.PassportService") as passport_service, ): - passport_service.return_value.verify.return_value = decoded + passport_service.return_value.verify.return_value = {"user_id": user_id, "auth_type": "internal"} result = AppWebAuthPermission().get() assert result == {"result": allowed} webapp_access.requires_permission_check.assert_called_once_with("app-1") extract_passport.assert_called_once() passport_service.return_value.verify.assert_called_once_with("passport") - webapp_access.is_user_allowed.assert_called_once_with(user_id=expected_user_id, app_id="app-1") + webapp_access.is_user_allowed.assert_called_once_with(user_id=user_id, app_id="app-1") + + @pytest.mark.parametrize( + "decoded", + [ + pytest.param({}, id="missing-auth-type"), + pytest.param({"auth_type": "internal"}, id="missing-user-id"), + pytest.param({"user_id": "sso_external_user", "auth_type": "external"}, id="external-auth-type"), + ], + ) + @patch("controllers.web.app.application_services") + def test_private_app_requires_internal_identity( + self, application_services: MagicMock, decoded: dict[str, str], app: Flask + ) -> None: + webapp_access = MagicMock() + webapp_access.requires_permission_check.return_value = True + application_services.return_value = SimpleNamespace(webapp_access=webapp_access) + + with ( + app.test_request_context("/webapp/permission?appId=app-1", headers={"X-App-Code": "code1"}), + patch("controllers.web.app.extract_webapp_passport", return_value="passport"), + patch("controllers.web.app.PassportService") as passport_service, + ): + passport_service.return_value.verify.return_value = decoded + with pytest.raises(WebAppAuthRequiredError): + AppWebAuthPermission().get() + + webapp_access.is_user_allowed.assert_not_called() @pytest.mark.parametrize("failing_method", ["requires_permission_check", "is_user_allowed"]) @patch("controllers.web.app.application_services") @@ -264,7 +290,7 @@ class TestAppWebAuthPermission: application_services.return_value = SimpleNamespace(webapp_access=webapp_access) passport_service = MagicMock() - passport_service.return_value.verify.return_value = {"user_id": "user-1"} + passport_service.return_value.verify.return_value = {"user_id": "user-1", "auth_type": "internal"} with ( app.test_request_context("/webapp/permission?appId=app-1", headers={"X-App-Code": "code1"}), patch("controllers.web.app.extract_webapp_passport", return_value="passport"), diff --git a/api/tests/unit_tests/core/agent/test_base_agent_runner.py b/api/tests/unit_tests/core/agent/test_base_agent_runner.py index 7411164512c..c6808ed39f6 100644 --- a/api/tests/unit_tests/core/agent/test_base_agent_runner.py +++ b/api/tests/unit_tests/core/agent/test_base_agent_runner.py @@ -1,670 +1,693 @@ import json +from datetime import datetime from decimal import Decimal -from unittest.mock import MagicMock import pytest from pytest_mock import MockerFixture +from sqlalchemy.orm import Session import core.agent.base_agent_runner as module -import models.model as model_module from core.agent.base_agent_runner import BaseAgentRunner +from core.agent.entities import AgentEntity, AgentToolEntity +from core.app.app_config.entities import ( + AppAdditionalFeatures, + DatasetEntity, + DatasetRetrieveConfigEntity, + EasyUIBasedAppModelConfigFrom, + ModelConfigEntity, + PromptTemplateEntity, +) +from core.app.apps.agent_chat.app_config_manager import AgentChatAppConfig +from core.app.apps.base_app_queue_manager import AppQueueManager +from core.app.entities.app_invoke_entities import ( + AgentChatAppGenerateEntity, + InvokeFrom, + ModelConfigWithCredentialsEntity, +) +from core.model_manager import ModelInstance +from core.tools.__base.tool import Tool +from core.tools.__base.tool_runtime import ToolRuntime +from core.tools.entities.common_entities import I18nObject +from core.tools.entities.tool_entities import ( + ToolDescription, + ToolEntity, + ToolIdentity, + ToolProviderType, +) +from core.tools.utils.dataset_retriever.dataset_retriever_base_tool import DatasetRetrieverBaseTool +from core.tools.utils.dataset_retriever_tool import DatasetRetrieverTool +from extensions.ext_storage import storage +from extensions.storage.storage_type import StorageType +from graphon.file import FileTransferMethod, FileType +from graphon.model_runtime.entities import LLMUsage, PromptMessageTool +from graphon.model_runtime.model_providers.base.large_language_model import LargeLanguageModel +from models.enums import ConversationFromSource, CreatorUserRole, MessageFileBelongsTo +from models.model import AppMode, AppModelConfig, Conversation, Message, MessageAgentThought, MessageFile, UploadFile -# ========================================================== -# Fixtures -# ========================================================== + +def _message( + *, + message_id: str = "msg_current", + conversation_id: str = "conv1", + query: str = "hello", + answer: str = "", +) -> Message: + message = Message( + id=message_id, + app_id="app1", + conversation_id=conversation_id, + query=query, + message={"role": "user", "content": query}, + answer=answer, + message_unit_price=Decimal(0), + answer_unit_price=Decimal(0), + currency="USD", + from_source=ConversationFromSource.CONSOLE, + from_account_id="user", + ) + message._inputs = {} + return message + + +def _conversation(*, app_model_config_id: str | None = None) -> Conversation: + conversation = Conversation( + id="conv1", + app_id="app1", + app_model_config_id=app_model_config_id, + mode=AppMode.AGENT_CHAT, + name="Conversation", + from_source=ConversationFromSource.CONSOLE, + from_account_id="user", + is_deleted=False, + ) + conversation._inputs = {} + return conversation + + +def _thought( + *, + thought_id: str, + message_id: str = "m1", + tool: str | None = None, + tool_input: str | None = None, + observation: str | None = None, + thought: str = "thinking", +) -> MessageAgentThought: + row = MessageAgentThought( + message_id=message_id, + position=1, + created_by_role=CreatorUserRole.ACCOUNT, + created_by="user", + thought=thought, + tool=tool, + tool_input=tool_input, + observation=observation, + tool_labels_str="{}", + tool_meta_str="{}", + ) + row.id = thought_id + return row + + +def _persist_history(session: Session, *messages: Message, thoughts: list[MessageAgentThought] | None = None) -> None: + session.add_all([_conversation(), *messages, *(thoughts or [])]) + session.commit() + + +def _app_config( + *, + simple_prompt_template: str | None = "", + agent: AgentEntity | None = None, + dataset: DatasetEntity | None = None, + additional_features: AppAdditionalFeatures | None = None, +) -> AgentChatAppConfig: + return AgentChatAppConfig( + tenant_id="tenant", + app_id="app1", + app_mode=AppMode.AGENT_CHAT, + app_model_config_from=EasyUIBasedAppModelConfigFrom.ARGS, + app_model_config_dict={}, + model=ModelConfigEntity(provider="provider", model="model"), + prompt_template=PromptTemplateEntity( + prompt_type=PromptTemplateEntity.PromptType.SIMPLE, + simple_prompt_template=simple_prompt_template, + ), + agent=agent, + dataset=dataset, + additional_features=additional_features, + ) + + +def _app_generate( + *, + app_config: AgentChatAppConfig | None = None, + files: list[str] | None = None, +) -> AgentChatAppGenerateEntity: + """Build the real generate entity with only the fields used by these unit tests.""" + + return AgentChatAppGenerateEntity.model_construct( + task_id="task", + app_config=app_config or _app_config(), + inputs={}, + files=files or [], + user_id="user", + stream=False, + invoke_from=InvokeFrom.DEBUGGER, + ) + + +def _agent_tool(tool_name: str) -> AgentToolEntity: + return AgentToolEntity( + provider_type=ToolProviderType.BUILT_IN, + provider_id="provider", + tool_name=tool_name, + ) + + +def _agent(*tools: AgentToolEntity) -> AgentEntity: + return AgentEntity( + provider="provider", + model="model", + strategy=AgentEntity.Strategy.FUNCTION_CALLING, + tools=list(tools), + ) + + +def _tool_entity(name: str) -> ToolEntity: + return ToolEntity( + identity=ToolIdentity( + author="author", + name=name, + label=I18nObject(en_US=name), + provider="provider", + ), + description=ToolDescription( + human=I18nObject(en_US="Description"), + llm="desc", + ), + ) + + +def _dataset_tool(mocker: MockerFixture, name: str) -> DatasetRetrieverTool: + return DatasetRetrieverTool( + entity=_tool_entity(name), + runtime=ToolRuntime(tenant_id="tenant"), + retrieval_tool=mocker.Mock(spec=DatasetRetrieverBaseTool), + ) @pytest.fixture -def mock_db_session(mocker: MockerFixture): - session = mocker.MagicMock() - mocker.patch.object(module.db, "session", session) - return session +def database_session(sqlite_session: Session, monkeypatch: pytest.MonkeyPatch) -> Session: + """Bind legacy global-session writes to a real SQLite Session.""" + + monkeypatch.setattr(module.db, "session", sqlite_session) + return sqlite_session @pytest.fixture -def runner(mocker: MockerFixture): - r = BaseAgentRunner.__new__(BaseAgentRunner) - r.tenant_id = "tenant" - r.user_id = "user" - r.agent_thought_count = 0 - r.message = mocker.MagicMock(id="msg_current", conversation_id="conv1") - r.app_config = mocker.MagicMock() - r.app_config.app_id = "app1" - r.app_config.agent = None - r.dataset_tools = [] - r.application_generate_entity = mocker.MagicMock(invoke_from="test") - r._current_thoughts = [] - return r +def runner(sqlite_session: Session, mocker: MockerFixture) -> BaseAgentRunner: + app_config = _app_config() + llm = mocker.Mock(spec=LargeLanguageModel) + llm.get_model_schema.return_value = None + model_instance = mocker.Mock( + spec=ModelInstance, + model_type_instance=llm, + model_name="model", + credentials={}, + ) - -# ========================================================== -# _repack_app_generate_entity -# ========================================================== + return BaseAgentRunner( + session=sqlite_session, + tenant_id="tenant", + application_generate_entity=_app_generate(app_config=app_config), + conversation=_conversation(), + app_config=app_config, + model_config=ModelConfigWithCredentialsEntity.model_construct(), + config=_agent(), + queue_manager=mocker.Mock(spec=AppQueueManager), + message=_message(), + user_id="user", + model_instance=model_instance, + ) class TestRepack: - def test_sets_empty_if_none(self, runner: BaseAgentRunner, mocker: MockerFixture): - entity = mocker.MagicMock() - entity.app_config.prompt_template.simple_prompt_template = None + def test_sets_empty_if_none(self, runner: BaseAgentRunner) -> None: + entity = _app_generate(app_config=_app_config(simple_prompt_template=None)) result = runner._repack_app_generate_entity(entity) assert result.app_config.prompt_template.simple_prompt_template == "" - def test_keeps_existing(self, runner: BaseAgentRunner, mocker: MockerFixture): - entity = mocker.MagicMock() - entity.app_config.prompt_template.simple_prompt_template = "abc" + def test_keeps_existing(self, runner: BaseAgentRunner) -> None: + entity = _app_generate(app_config=_app_config(simple_prompt_template="abc")) result = runner._repack_app_generate_entity(entity) assert result.app_config.prompt_template.simple_prompt_template == "abc" -# ========================================================== -# update_prompt_message_tool -# ========================================================== - - -class TestUpdatePromptTool: - def test_replaces_prompt_tool_parameters_with_tool_schema(self, runner: BaseAgentRunner, mocker: MockerFixture): - tool = mocker.MagicMock() - schema = { - "type": "object", - "properties": {"p1": {"type": "string", "description": "desc"}}, - "required": ["p1"], - } - tool.get_llm_parameters_json_schema.return_value = schema - - prompt_tool = mocker.MagicMock() - prompt_tool.parameters = {"properties": {}, "required": []} - - result = runner.update_prompt_message_tool(tool, prompt_tool) - assert result.parameters == schema - - -# ========================================================== -# create_agent_thought -# ========================================================== - - -class TestCreateAgentThought: - def test_with_files(self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture): - mock_thought = mocker.MagicMock(id=10) - mocker.patch.object(module, "MessageAgentThought", return_value=mock_thought) - - result = runner.create_agent_thought("m", "msg", "tool", "input", ["f1"]) - assert result == "10" - assert runner.agent_thought_count == 1 - mock_db_session.add.assert_called_once_with(mock_thought) - mock_db_session.commit.assert_called_once_with() - mock_db_session.close.assert_called_once_with() - - def test_without_files(self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture): - mock_thought = mocker.MagicMock(id=11) - mocker.patch.object(module, "MessageAgentThought", return_value=mock_thought) - - result = runner.create_agent_thought("m", "msg", "tool", "input", []) - assert result == "11" - - -# ========================================================== -# save_agent_thought -# ========================================================== - - -class TestSaveAgentThought: - def setup_agent(self, mocker: MockerFixture): - agent = mocker.MagicMock() - agent.tool = "tool1;tool2" - agent.tool_labels = {} - agent.thought = "" - return agent - - def test_not_found(self, runner: BaseAgentRunner, mock_db_session): - mock_db_session.scalar.return_value = None - with pytest.raises(ValueError): - runner.save_agent_thought("id", None, None, None, None, None, None, [], None) - - def test_full_update(self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture): - agent = self.setup_agent(mocker) - mock_db_session.scalar.return_value = agent - - mock_label = mocker.MagicMock() - mock_label.to_dict.return_value = {"en_US": "label"} - mocker.patch.object(module.ToolManager, "get_tool_label", return_value=mock_label) - - usage = mocker.MagicMock( - prompt_tokens=1, - prompt_price_unit=Decimal("0.1"), - prompt_unit_price=Decimal("0.1"), - completion_tokens=2, - completion_price_unit=Decimal("0.2"), - completion_unit_price=Decimal("0.2"), - total_tokens=3, - total_price=Decimal("0.3"), - ) - - runner.save_agent_thought( - "id", - "tool1;tool2", - {"a": 1}, - "thought", - {"b": 2}, - {"meta": 1}, - "answer", - ["f1"], - usage, - ) - - assert agent.answer == "answer" - assert agent.tokens == 3 - assert "tool1" in json.loads(agent.tool_labels_str) - mock_db_session.commit.assert_called_once_with() - mock_db_session.close.assert_called_once_with() - - def test_label_fallback_when_none(self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture): - agent = self.setup_agent(mocker) - agent.tool = "unknown_tool" - mock_db_session.scalar.return_value = agent - mocker.patch.object(module.ToolManager, "get_tool_label", return_value=None) - - runner.save_agent_thought("id", None, None, None, None, None, None, [], None) - labels = json.loads(agent.tool_labels_str) - assert "unknown_tool" in labels - - def test_json_failure_paths(self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture): - agent = self.setup_agent(mocker) - mock_db_session.scalar.return_value = agent - - bad_obj = MagicMock() - bad_obj.__str__.return_value = "bad" - - runner.save_agent_thought( - "id", - None, - bad_obj, - None, - bad_obj, - bad_obj, - None, - [], - None, - ) - - assert mock_db_session.commit.called - - def test_messages_ids_none(self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture): - agent = self.setup_agent(mocker) - mock_db_session.scalar.return_value = agent - runner.save_agent_thought("id", None, None, None, None, None, None, None, None) - assert mock_db_session.commit.called - - def test_success_dict_serialization(self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture): - agent = self.setup_agent(mocker) - mock_db_session.scalar.return_value = agent - - runner.save_agent_thought( - "id", - None, - {"a": 1}, - None, - {"b": 2}, - None, - None, - [], - None, - ) - - assert isinstance(agent.tool_input, str) - assert isinstance(agent.observation, str) - - -# ========================================================== -# organize_agent_user_prompt -# ========================================================== - - -class TestOrganizeUserPrompt: - def test_no_files(self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture): - caller_session = mocker.MagicMock() - caller_session.scalars.return_value.all.return_value = [] - msg = mocker.MagicMock(id="1", query="hello", app_model_config=None) - result = runner.organize_agent_user_prompt(msg, session=caller_session) - assert result.content == "hello" - assert mock_db_session.mock_calls == [] - - def test_with_files_no_config(self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture): - mock_db_session.scalars.return_value.all.return_value = [mocker.MagicMock()] - msg = mocker.MagicMock(id="1", query="hello", app_model_config=None) - msg.app_model_config_with_session.return_value = None - result = runner.organize_agent_user_prompt(msg, session=mock_db_session) - assert result.content == "hello" - - def test_image_detail_low_fallback(self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture): - mock_db_session.scalars.return_value.all.return_value = [mocker.MagicMock()] - file_config = mocker.MagicMock() - file_config.image_config = mocker.MagicMock(detail=None) - mocker.patch.object(module.FileUploadConfigManager, "convert", return_value=file_config) - mocker.patch.object(module.file_factory, "build_from_message_files", return_value=[]) - - msg = mocker.MagicMock(id="1", query="hello") - app_model_config = mocker.MagicMock() - app_model_config.app_id = "app1" - app_model_config.to_dict.return_value = {} - msg.app_model_config_with_session.return_value = app_model_config - load_annotation_reply_config = mocker.patch.object( - module, "load_annotation_reply_config", return_value={"enabled": False} - ) - - result = runner.organize_agent_user_prompt(msg, session=mock_db_session) - assert result.content == "hello" - load_annotation_reply_config.assert_called_once_with(mock_db_session, "app1") - app_model_config.to_dict.assert_called_once_with(annotation_reply={"enabled": False}) - - -# ========================================================== -# organize_agent_history -# ========================================================== - - -class TestOrganizeHistory: - def test_empty(self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture): - caller_session = mocker.MagicMock() - caller_session.execute.return_value.scalars.return_value.all.return_value = [] - mocker.patch.object(module, "extract_thread_messages", return_value=[]) - result = runner.organize_agent_history([], session=caller_session) - assert result == [] - assert mock_db_session.mock_calls == [] - - def test_with_answer_only(self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture): - msg = mocker.MagicMock(id="m1", answer="ans", agent_thoughts=[], app_model_config=None) - msg.agent_thoughts_with_session.return_value = [] - msg.app_model_config_with_session.return_value = None - mock_db_session.execute.return_value.scalars.return_value.all.return_value = [msg] - mocker.patch.object(module, "extract_thread_messages", return_value=[msg]) - result = runner.organize_agent_history([], session=mock_db_session) - assert any(isinstance(x, module.AssistantPromptMessage) for x in result) - - def test_skip_current_message(self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture): - msg = mocker.MagicMock(id="msg_current", agent_thoughts=[], answer="ans", app_model_config=None) - mock_db_session.execute.return_value.scalars.return_value.all.return_value = [msg] - mocker.patch.object(module, "extract_thread_messages", return_value=[msg]) - result = runner.organize_agent_history([], session=mock_db_session) - assert result == [] - - def test_with_tool_calls_invalid_json(self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture): - thought = mocker.MagicMock( - tool="tool1", - tool_input="invalid", - observation="invalid", - thought="thinking", - ) - msg = mocker.MagicMock(id="m2", agent_thoughts=[thought], answer=None, app_model_config=None) - msg.agent_thoughts_with_session.return_value = [thought] - msg.app_model_config_with_session.return_value = None - - mock_db_session.execute.return_value.scalars.return_value.all.return_value = [msg] - mocker.patch.object(module, "extract_thread_messages", return_value=[msg]) - mocker.patch("uuid.uuid4", return_value="uuid") - - result = runner.organize_agent_history([], session=mock_db_session) - assert isinstance(result, list) - - def test_empty_tool_name_split(self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture): - thought = mocker.MagicMock(tool=";", thought="thinking") - msg = mocker.MagicMock(id="m5", agent_thoughts=[thought], answer=None, app_model_config=None) - msg.agent_thoughts_with_session.return_value = [thought] - msg.app_model_config_with_session.return_value = None - - mock_db_session.execute.return_value.scalars.return_value.all.return_value = [msg] - mocker.patch.object(module, "extract_thread_messages", return_value=[msg]) - result = runner.organize_agent_history([], session=mock_db_session) - assert isinstance(result, list) - - def test_valid_json_tool_flow(self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture): - thought = mocker.MagicMock( - tool="tool1", - tool_input=json.dumps({"tool1": {"x": 1}}), - observation=json.dumps({"tool1": "obs"}), - thought="thinking", - ) - - msg = mocker.MagicMock( - id="m100", - agent_thoughts=[thought], - answer=None, - app_model_config=None, - ) - msg.agent_thoughts_with_session.return_value = [thought] - msg.app_model_config_with_session.return_value = None - - mock_db_session.execute.return_value.scalars.return_value.all.return_value = [msg] - mocker.patch.object(module, "extract_thread_messages", return_value=[msg]) - mocker.patch("uuid.uuid4", return_value="uuid") - - result = runner.organize_agent_history([], session=mock_db_session) - assert isinstance(result, list) - - -# ========================================================== -# _convert_tool_to_prompt_message_tool (new coverage) -# ========================================================== - - -class TestConvertToolToPromptMessageTool: - def test_basic_conversion(self, runner: BaseAgentRunner, mocker: MockerFixture): - tool = mocker.MagicMock(tool_name="tool1") - - tool_entity = mocker.MagicMock() - tool_entity.entity.description.llm = "desc" - schema = { - "type": "object", - "properties": {"param1": {"type": "string", "description": "desc"}}, - "required": ["param1"], - } - tool_entity.get_llm_parameters_json_schema.return_value = schema - - mocker.patch.object(module.ToolManager, "get_agent_tool_runtime", return_value=tool_entity) - mocker.patch.object(module, "PromptMessageTool", side_effect=lambda **kw: MagicMock(**kw)) - - prompt_tool, entity = runner._convert_tool_to_prompt_message_tool(tool) - assert entity == tool_entity - assert prompt_tool.parameters == schema - - -# ========================================================== -# _init_prompt_tools additional branches -# ========================================================== - - -class TestInitPromptToolsExtended: - def test_agent_tool_branch(self, runner: BaseAgentRunner, mocker: MockerFixture): - agent_tool = mocker.MagicMock(tool_name="agent_tool") - runner.app_config.agent = mocker.MagicMock(tools=[agent_tool]) - mocker.patch.object(runner, "_convert_tool_to_prompt_message_tool", return_value=(MagicMock(), "entity")) - - tools, prompts = runner._init_prompt_tools() - assert "agent_tool" in tools - - def test_exception_in_conversion(self, runner: BaseAgentRunner, mocker: MockerFixture): - agent_tool = mocker.MagicMock(tool_name="bad_tool") - runner.app_config.agent = mocker.MagicMock(tools=[agent_tool]) - mocker.patch.object(runner, "_convert_tool_to_prompt_message_tool", side_effect=Exception) - - tools, prompts = runner._init_prompt_tools() - assert tools == {} - - -# ========================================================== -# Additional Coverage Tests (DO NOT MODIFY EXISTING TESTS) -# ========================================================== - - -class TestAdditionalCoverage: - def test_save_agent_thought_existing_labels(self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture): - agent = mocker.MagicMock() - agent.tool = "tool1" - agent.tool_labels = {"tool1": {"en_US": "existing"}} - agent.thought = "" - mock_db_session.scalar.return_value = agent - - runner.save_agent_thought("id", None, None, None, None, None, None, [], None) - labels = json.loads(agent.tool_labels_str) - assert labels["tool1"]["en_US"] == "existing" - - def test_save_agent_thought_tool_meta_string(self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture): - agent = mocker.MagicMock() - agent.tool = "tool1" - agent.tool_labels = {} - agent.thought = "" - mock_db_session.scalar.return_value = agent +def test_update_prompt_tool_replaces_parameters(runner: BaseAgentRunner, mocker: MockerFixture) -> None: + tool = mocker.Mock(spec=Tool) + schema = { + "type": "object", + "properties": {"p1": {"type": "string", "description": "desc"}}, + "required": ["p1"], + } + tool.get_llm_parameters_json_schema.return_value = schema + prompt_tool = PromptMessageTool(name="tool", description="", parameters={"properties": {}, "required": []}) + + result = runner.update_prompt_message_tool(tool, prompt_tool) + + assert result.parameters == schema + + +@pytest.mark.parametrize(("files", "expected_files"), [(["f1"], '["f1"]'), ([], "")]) +def test_create_agent_thought_persists_row( + runner: BaseAgentRunner, + database_session: Session, + files: list[str], + expected_files: str, +) -> None: + thought_id = runner.create_agent_thought("message-1", "message", "tool", "input", files) + + stored = database_session.get(MessageAgentThought, thought_id) + assert stored is not None + assert stored.message_id == "message-1" + assert stored.message_files == expected_files + assert stored.position == 1 + assert runner.agent_thought_count == 1 + + +def _persist_thought(session: Session, *, tool: str = "tool1;tool2") -> MessageAgentThought: + thought = _thought(thought_id="thought-1", tool=tool, thought="") + session.add(thought) + session.commit() + return thought + + +def test_save_agent_thought_rejects_missing_row(runner: BaseAgentRunner, database_session: Session) -> None: + assert database_session.get(MessageAgentThought, "missing") is None + with pytest.raises(ValueError, match="agent thought not found"): + runner.save_agent_thought("missing", None, None, None, None, None, None, [], None) + + +def test_save_agent_thought_full_update( + runner: BaseAgentRunner, + database_session: Session, + mocker: MockerFixture, +) -> None: + thought = _persist_thought(database_session) + label = I18nObject(en_US="label") + mocker.patch.object(module.ToolManager, "get_tool_label", return_value=label) + usage = LLMUsage( + prompt_tokens=1, + prompt_price_unit=Decimal("0.1"), + prompt_unit_price=Decimal("0.1"), + prompt_price=Decimal("0.1"), + completion_tokens=2, + completion_price_unit=Decimal("0.2"), + completion_unit_price=Decimal("0.2"), + completion_price=Decimal("0.2"), + total_tokens=3, + total_price=Decimal("0.3"), + currency="USD", + latency=0, + ) + + runner.save_agent_thought( + thought.id, + "tool1;tool2", + {"a": 1}, + "thought", + {"b": 2}, + {"meta": 1}, + "answer", + ["f1"], + usage, + ) + + stored = database_session.get(MessageAgentThought, thought.id) + assert stored is not None + assert stored.answer == "answer" + assert stored.tokens == 3 + assert stored.tool_input == '{"a": 1}' + assert stored.observation == '{"b": 2}' + assert stored.message_files == '["f1"]' + assert "tool1" in json.loads(stored.tool_labels_str) + + +def test_save_agent_thought_uses_label_fallback( + runner: BaseAgentRunner, + database_session: Session, + mocker: MockerFixture, +) -> None: + thought = _persist_thought(database_session, tool="unknown_tool") + mocker.patch.object(module.ToolManager, "get_tool_label", return_value=None) + + runner.save_agent_thought(thought.id, None, None, None, None, None, None, [], None) + + stored = database_session.get(MessageAgentThought, thought.id) + assert stored is not None + assert json.loads(stored.tool_labels_str)["unknown_tool"]["en_US"] == "unknown_tool" + + +def test_save_agent_thought_preserves_existing_labels( + runner: BaseAgentRunner, + database_session: Session, +) -> None: + thought = _persist_thought(database_session, tool="tool1") + thought.tool_labels_str = json.dumps({"tool1": {"en_US": "existing"}}) + database_session.commit() + + runner.save_agent_thought(thought.id, None, None, None, None, None, None, [], None) + + stored = database_session.get(MessageAgentThought, thought.id) + assert stored is not None + assert json.loads(stored.tool_labels_str)["tool1"]["en_US"] == "existing" + + +def test_save_agent_thought_serialization_fallbacks( + runner: BaseAgentRunner, + database_session: Session, + mocker: MockerFixture, +) -> None: + thought = _persist_thought(database_session, tool="tool1;;") + mocker.patch.object(module.ToolManager, "get_tool_label", return_value=None) + tool_input = {"a": 1} + observation = {"b": 2} + tool_meta = {"c": 3} + real_dumps = json.dumps + + def dumps_side_effect(value, *args, **kwargs): + if value in (tool_input, observation, tool_meta) and kwargs.get("ensure_ascii") is False: + raise TypeError("fail") + return real_dumps(value, *args, **kwargs) + + mocker.patch.object(module.json, "dumps", side_effect=dumps_side_effect) + + runner.save_agent_thought( + thought.id, + "tool1;;", + tool_input, + None, + observation, + tool_meta, + None, + [], + None, + ) + + stored = database_session.get(MessageAgentThought, thought.id) + assert stored is not None + assert isinstance(stored.tool_input, str) + assert isinstance(stored.observation, str) + assert isinstance(stored.tool_meta_str, str) + assert "" not in json.loads(stored.tool_labels_str) + + +@pytest.mark.parametrize("messages_ids", [None, []]) +def test_save_agent_thought_accepts_empty_message_ids( + runner: BaseAgentRunner, + database_session: Session, + messages_ids: list[str] | None, +) -> None: + thought = _persist_thought(database_session) + runner.save_agent_thought(thought.id, None, None, None, None, "meta_string", None, messages_ids, None) # type: ignore[arg-type] + + stored = database_session.get(MessageAgentThought, thought.id) + assert stored is not None + assert stored.tool_meta_str == "meta_string" + + +def _message_file( + *, + message_id: str = "m1", + belongs_to: MessageFileBelongsTo = MessageFileBelongsTo.ASSISTANT, + upload_file_id: str = "upload-1", +) -> MessageFile: + return MessageFile( + message_id=message_id, + type=FileType.IMAGE, + transfer_method=FileTransferMethod.LOCAL_FILE, + created_by_role=CreatorUserRole.ACCOUNT, + created_by="user", + belongs_to=belongs_to, + upload_file_id=upload_file_id, + ) + + +def _file_enabled_app_model_config() -> AppModelConfig: + return AppModelConfig( + app_id="app1", + file_upload=json.dumps( + { + "enabled": True, + "allowed_file_types": [FileType.IMAGE], + "allowed_file_extensions": [".png"], + "allowed_file_upload_methods": [FileTransferMethod.LOCAL_FILE], + "number_limits": 1, + "image": {"detail": "low"}, + } + ), + ) + + +def _upload_file() -> UploadFile: + return UploadFile( + tenant_id="tenant", + storage_type=StorageType.LOCAL, + key="image.png", + name="image.png", + size=1, + extension="png", + mime_type="image/png", + created_by_role=CreatorUserRole.ACCOUNT, + created_by="user", + created_at=datetime.now(), + used=False, + ) + + +def test_organize_user_prompt_without_files(runner: BaseAgentRunner, sqlite_session: Session) -> None: + result = runner.organize_agent_user_prompt(_message(message_id="m1"), session=sqlite_session) + assert result.content == "hello" + + +def test_organize_user_prompt_with_files_but_no_config(runner: BaseAgentRunner, sqlite_session: Session) -> None: + sqlite_session.add(_message_file()) + sqlite_session.commit() + + result = runner.organize_agent_user_prompt(_message(message_id="m1"), session=sqlite_session) + + assert result.content == "hello" + + +def test_organize_user_prompt_uses_file_config( + runner: BaseAgentRunner, + sqlite_session: Session, +) -> None: + config = _file_enabled_app_model_config() + sqlite_session.add_all([config, _conversation(app_model_config_id=config.id), _message_file()]) + sqlite_session.commit() + + result = runner.organize_agent_user_prompt( + _message(message_id="m1"), + session=sqlite_session, + ) + + assert result.content == "hello" + + +def test_organize_user_prompt_builds_file_content( + runner: BaseAgentRunner, + sqlite_session: Session, + mocker: MockerFixture, +) -> None: + config = _file_enabled_app_model_config() + upload_file = _upload_file() + message_file = _message_file( + belongs_to=MessageFileBelongsTo.USER, + upload_file_id=upload_file.id, + ) + sqlite_session.add_all([config, _conversation(app_model_config_id=config.id), upload_file, message_file]) + sqlite_session.commit() + mocker.patch.object(storage, "load", return_value=b"image") + + result = runner.organize_agent_user_prompt( + _message(message_id="m1"), + session=sqlite_session, + ) + + assert isinstance(result.content, list) + assert isinstance(result.content[0], module.ImagePromptMessageContent) + assert isinstance(result.content[-1], module.TextPromptMessageContent) + + +def test_organize_history_empty_preserves_system_prompt(runner: BaseAgentRunner, sqlite_session: Session) -> None: + system_message = module.SystemPromptMessage(content="sys") + result = runner.organize_agent_history([system_message], session=sqlite_session) + assert result == [system_message] + + +def test_organize_history_with_answer_only(runner: BaseAgentRunner, sqlite_session: Session) -> None: + _persist_history(sqlite_session, _message(message_id="m1", answer="answer")) + result = runner.organize_agent_history([], session=sqlite_session) + assert any(isinstance(item, module.AssistantPromptMessage) and item.content == "answer" for item in result) + - runner.save_agent_thought("id", None, None, None, None, "meta_string", None, [], None) - assert agent.tool_meta_str == "meta_string" - - def test_convert_dataset_retriever_tool(self, runner: BaseAgentRunner, mocker: MockerFixture): - ds_tool = mocker.MagicMock() - ds_tool.entity.identity.name = "ds" - ds_tool.entity.description.llm = "desc" - - param = mocker.MagicMock() - param.name = "query" - param.llm_description = "desc" - param.required = True - - ds_tool.get_runtime_parameters.return_value = [param] - - mocker.patch.object(module, "PromptMessageTool", side_effect=lambda **kw: MagicMock(**kw)) - - prompt = runner._convert_dataset_retriever_tool_to_prompt_message_tool(ds_tool) - assert prompt is not None - - def test_organize_user_prompt_with_file_objects( - self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture - ): - mock_db_session.scalars.return_value.all.return_value = [mocker.MagicMock()] - - file_config = mocker.MagicMock() - file_config.image_config = mocker.MagicMock(detail=None) - - mocker.patch.object(module.FileUploadConfigManager, "convert", return_value=file_config) - mocker.patch.object(module.file_factory, "build_from_message_files", return_value=["file1"]) - mocker.patch.object(module.file_manager, "to_prompt_message_content", return_value=mocker.MagicMock()) - - mocker.patch.object(module, "UserPromptMessage", side_effect=lambda **kw: MagicMock(**kw)) - mocker.patch.object(module, "TextPromptMessageContent", side_effect=lambda **kw: MagicMock(**kw)) - - msg = mocker.MagicMock(id="1", query="hello") - app_model_config = mocker.MagicMock() - app_model_config.app_id = "app1" - app_model_config.to_dict.return_value = {} - msg.app_model_config_with_session.return_value = app_model_config - mocker.patch.object(module, "load_annotation_reply_config", return_value={"enabled": False}) - - result = runner.organize_agent_user_prompt(msg, session=mock_db_session) - assert result is not None - - def test_organize_history_without_tool_names(self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture): - thought = mocker.MagicMock(tool=None, thought="thinking") - msg = mocker.MagicMock(id="m3", agent_thoughts=[thought], answer=None, app_model_config=None) - msg.agent_thoughts_with_session.return_value = [thought] - msg.app_model_config_with_session.return_value = None - - mock_db_session.execute.return_value.scalars.return_value.all.return_value = [msg] - mocker.patch.object(module, "extract_thread_messages", return_value=[msg]) - - result = runner.organize_agent_history([], session=mock_db_session) - assert isinstance(result, list) - - def test_organize_history_multiple_tools_split( - self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture - ): - thought = mocker.MagicMock( - tool="tool1;tool2", - tool_input=json.dumps({"tool1": {}, "tool2": {}}), - observation=json.dumps({"tool1": "o1", "tool2": "o2"}), - thought="thinking", - ) - msg = mocker.MagicMock(id="m4", agent_thoughts=[thought], answer=None, app_model_config=None) - msg.agent_thoughts_with_session.return_value = [thought] - msg.app_model_config_with_session.return_value = None - - mock_db_session.execute.return_value.scalars.return_value.all.return_value = [msg] - mocker.patch.object(module, "extract_thread_messages", return_value=[msg]) - mocker.patch("uuid.uuid4", return_value="uuid") - - result = runner.organize_agent_history([], session=mock_db_session) - assert isinstance(result, list) - - -class TestConvertDatasetRetrieverTool: - def test_required_param_added(self, runner: BaseAgentRunner, mocker: MockerFixture): - ds_tool = mocker.MagicMock() - ds_tool.entity.identity.name = "ds" - ds_tool.entity.description.llm = "desc" - - param = mocker.MagicMock() - param.name = "query" - param.llm_description = "desc" - param.required = True - - ds_tool.get_runtime_parameters.return_value = [param] - - mocker.patch.object(module, "PromptMessageTool", side_effect=lambda **kw: MagicMock(**kw)) - - prompt = runner._convert_dataset_retriever_tool_to_prompt_message_tool(ds_tool) - - assert prompt is not None - - -class TestBaseAgentRunnerInit: - def test_init_sets_stream_tool_call_and_files(self, mocker: MockerFixture): - caller_session = mocker.MagicMock() - caller_session.scalar.return_value = 2 - global_session = mocker.MagicMock() - mocker.patch.object(model_module.db, "session", global_session) - organize_agent_history = mocker.patch.object(BaseAgentRunner, "organize_agent_history", return_value=[]) - get_dataset_tools = mocker.patch.object( - module.DatasetRetrieverTool, "get_dataset_tools", return_value=["ds_tool"] - ) - - llm = mocker.MagicMock() - llm.get_model_schema.return_value = mocker.MagicMock( - features=[module.ModelFeature.STREAM_TOOL_CALL, module.ModelFeature.VISION] - ) - model_instance = mocker.MagicMock(model_type_instance=llm, model="m", credentials="c") - - app_config = mocker.MagicMock() - app_config.app_id = "app1" - app_config.agent = None - app_config.dataset = mocker.MagicMock(dataset_ids=["d1"], retrieve_config={"k": "v"}) - app_config.additional_features = mocker.MagicMock(show_retrieve_source=True) - - app_generate = mocker.MagicMock(invoke_from="test", inputs={}, files=["file1"]) - message = mocker.MagicMock(id="msg1", conversation_id="conv1") - - runner = BaseAgentRunner( - session=caller_session, - tenant_id="tenant", - application_generate_entity=app_generate, - conversation=mocker.MagicMock(), - app_config=app_config, - model_config=mocker.MagicMock(), - config=mocker.MagicMock(), - queue_manager=mocker.MagicMock(), - message=message, - user_id="user", - model_instance=model_instance, - ) - - assert runner.stream_tool_call is True - assert runner.files == ["file1"] - assert runner.dataset_tools == ["ds_tool"] - assert runner.agent_thought_count == 2 - organize_agent_history.assert_called_once_with(session=caller_session, prompt_messages=[]) - assert get_dataset_tools.call_args.kwargs["session"] is caller_session - assert global_session.mock_calls == [] - - -class TestBaseAgentRunnerCoverage: - def test_init_prompt_tools_adds_dataset_tools(self, runner: BaseAgentRunner, mocker: MockerFixture): - dataset_tool = mocker.MagicMock() - dataset_tool.entity.identity.name = "ds" - runner.dataset_tools = [dataset_tool] - - mocker.patch.object(runner, "_convert_dataset_retriever_tool_to_prompt_message_tool", return_value=MagicMock()) - - tools, prompt_tools = runner._init_prompt_tools() - - assert tools["ds"] == dataset_tool - assert len(prompt_tools) == 1 - - def test_save_agent_thought_json_dumps_fallbacks( - self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture - ): - agent = mocker.MagicMock() - agent.tool = "tool1" - agent.tool_labels = {} - agent.thought = "" - mock_db_session.scalar.return_value = agent - - mocker.patch.object(module.ToolManager, "get_tool_label", return_value=None) - - tool_input = {"a": 1} - observation = {"b": 2} - tool_meta = {"c": 3} - - real_dumps = json.dumps - - def dumps_side_effect(value, *args, **kwargs): - if value in (tool_input, observation, tool_meta) and kwargs.get("ensure_ascii") is False: - raise TypeError("fail") - return real_dumps(value, *args, **kwargs) - - mocker.patch.object(module.json, "dumps", side_effect=dumps_side_effect) - - runner.save_agent_thought( - "id", - "tool1", - tool_input, - None, - observation, - tool_meta, - None, - [], - None, - ) - - assert isinstance(agent.tool_input, str) - assert isinstance(agent.observation, str) - assert isinstance(agent.tool_meta_str, str) - - def test_save_agent_thought_skips_empty_tool_name( - self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture - ): - agent = mocker.MagicMock() - agent.tool = "tool1;;" - agent.tool_labels = {} - agent.thought = "" - mock_db_session.scalar.return_value = agent - - mocker.patch.object(module.ToolManager, "get_tool_label", return_value=None) - - runner.save_agent_thought("id", None, None, None, None, None, None, [], None) - - labels = json.loads(agent.tool_labels_str) - assert "" not in labels - - def test_organize_history_includes_system_prompt( - self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture - ): - mock_db_session.execute.return_value.scalars.return_value.all.return_value = [] - mocker.patch.object(module, "extract_thread_messages", return_value=[]) - - system_message = module.SystemPromptMessage(content="sys") - - result = runner.organize_agent_history([system_message], session=mock_db_session) - - assert system_message in result - - def test_organize_history_tool_inputs_and_observation_none( - self, runner: BaseAgentRunner, mock_db_session, mocker: MockerFixture - ): - thought = mocker.MagicMock( - tool="tool1", - tool_input=None, - observation=None, - thought="thinking", - ) - msg = mocker.MagicMock(id="m6", agent_thoughts=[thought], answer=None, app_model_config=None) - msg.agent_thoughts_with_session.return_value = [thought] - - mock_db_session.execute.return_value.scalars.return_value.all.return_value = [msg] - mocker.patch.object(module, "extract_thread_messages", return_value=[msg]) - mocker.patch("uuid.uuid4", return_value="uuid") - - mocker.patch.object( - runner, - "organize_agent_user_prompt", - return_value=module.UserPromptMessage(content="user"), - ) - - result = runner.organize_agent_history([], session=mock_db_session) - - assert any(isinstance(item, module.ToolPromptMessage) for item in result) +def test_organize_history_skips_current_message(runner: BaseAgentRunner, sqlite_session: Session) -> None: + _persist_history(sqlite_session, _message(message_id="msg_current", answer="answer")) + result = runner.organize_agent_history([], session=sqlite_session) + assert result == [] + + +@pytest.mark.parametrize( + ("tool", "tool_input", "observation"), + [ + ("tool1", "invalid", "invalid"), + (";", None, None), + ("tool1", None, None), + ("tool1;tool2", json.dumps({"tool1": {}, "tool2": {}}), json.dumps({"tool1": "o1", "tool2": "o2"})), + ], +) +def test_organize_history_reconstructs_tool_flows( + runner: BaseAgentRunner, + sqlite_session: Session, + tool: str, + tool_input: str | None, + observation: str | None, +) -> None: + message = _message(message_id="m2") + thought = _thought( + thought_id="thought-1", + message_id=message.id, + tool=tool, + tool_input=tool_input, + observation=observation, + ) + _persist_history(sqlite_session, message, thoughts=[thought]) + + result = runner.organize_agent_history([], session=sqlite_session) + + assert isinstance(result, list) + assert any(isinstance(item, module.AssistantPromptMessage) for item in result) + + +def test_organize_history_without_tool_name(runner: BaseAgentRunner, sqlite_session: Session) -> None: + message = _message(message_id="m3") + thought = _thought(thought_id="thought-1", message_id=message.id, tool=None) + _persist_history(sqlite_session, message, thoughts=[thought]) + + result = runner.organize_agent_history([], session=sqlite_session) + + assert any(isinstance(item, module.AssistantPromptMessage) and item.content == "thinking" for item in result) + + +def test_convert_tool_to_prompt_message_tool(runner: BaseAgentRunner, mocker: MockerFixture) -> None: + tool = _agent_tool("tool1") + tool_entity = mocker.Mock(spec=Tool, entity=_tool_entity("tool1")) + schema = { + "type": "object", + "properties": {"param1": {"type": "string", "description": "desc"}}, + "required": ["param1"], + } + tool_entity.get_llm_parameters_json_schema.return_value = schema + mocker.patch.object(module.ToolManager, "get_agent_tool_runtime", return_value=tool_entity) + + prompt_tool, entity = runner._convert_tool_to_prompt_message_tool(tool) + + assert entity is tool_entity + assert prompt_tool.parameters == schema + + +def test_convert_dataset_retriever_tool(runner: BaseAgentRunner, mocker: MockerFixture) -> None: + dataset_tool = _dataset_tool(mocker, "ds") + + prompt = runner._convert_dataset_retriever_tool_to_prompt_message_tool(dataset_tool) + + assert prompt.name == "ds" + assert prompt.parameters["required"] == ["query"] + + +def test_init_prompt_tools_adds_agent_and_dataset_tools(runner: BaseAgentRunner, mocker: MockerFixture) -> None: + agent_tool = _agent_tool("agent_tool") + agent_runtime = mocker.Mock(spec=Tool, entity=_tool_entity("agent_tool")) + agent_runtime.get_llm_parameters_json_schema.return_value = {"type": "object", "properties": {}} + mocker.patch.object(module.ToolManager, "get_agent_tool_runtime", return_value=agent_runtime) + dataset_tool = _dataset_tool(mocker, "dataset_tool") + runner.app_config.agent = _agent(agent_tool) + runner.dataset_tools = [dataset_tool] + + tools, prompts = runner._init_prompt_tools() + + assert tools == {"agent_tool": agent_runtime, "dataset_tool": dataset_tool} + assert len(prompts) == 2 + + +def test_init_prompt_tools_skips_deleted_agent_tool(runner: BaseAgentRunner, mocker: MockerFixture) -> None: + agent_tool = _agent_tool("bad_tool") + runner.app_config.agent = _agent(agent_tool) + mocker.patch.object(module.ToolManager, "get_agent_tool_runtime", side_effect=Exception) + + tools, prompts = runner._init_prompt_tools() + + assert tools == {} + assert prompts == [] + + +def test_init_uses_real_session_for_count_and_dependencies( + sqlite_session: Session, + mocker: MockerFixture, +) -> None: + sqlite_session.add_all( + [ + _thought(thought_id="thought-1", message_id="msg1"), + _thought(thought_id="thought-2", message_id="msg1"), + _thought(thought_id="decoy-thought", message_id="other-message"), + ] + ) + sqlite_session.commit() + get_dataset_tools = mocker.patch.object( + module.DatasetRetrieverTool, + "get_dataset_tools", + return_value=["ds_tool"], + ) + llm = mocker.Mock(spec=LargeLanguageModel) + llm.get_model_schema.return_value = mocker.Mock( + features=[module.ModelFeature.STREAM_TOOL_CALL, module.ModelFeature.VISION] + ) + model_instance = mocker.Mock( + spec=ModelInstance, + model_type_instance=llm, + model_name="m", + credentials="c", + ) + app_config = _app_config( + dataset=DatasetEntity( + dataset_ids=["d1"], + retrieve_config=DatasetRetrieveConfigEntity( + retrieve_strategy=DatasetRetrieveConfigEntity.RetrieveStrategy.MULTIPLE, + ), + ), + additional_features=AppAdditionalFeatures(show_retrieve_source=True), + ) + app_generate = _app_generate(app_config=app_config, files=["file1"]) + message = _message(message_id="msg1") + + initialized = BaseAgentRunner( + session=sqlite_session, + tenant_id="tenant", + application_generate_entity=app_generate, + conversation=_conversation(), + app_config=app_config, + model_config=ModelConfigWithCredentialsEntity.model_construct(), + config=_agent(), + queue_manager=mocker.Mock(spec=AppQueueManager), + message=message, + user_id="user", + model_instance=model_instance, + ) + + assert initialized.stream_tool_call is True + assert initialized.files == ["file1"] + assert initialized.dataset_tools == ["ds_tool"] + assert initialized.agent_thought_count == 2 + assert initialized.history_prompt_messages == [] + assert get_dataset_tools.call_args.kwargs["session"] is sqlite_session diff --git a/api/tests/unit_tests/core/rag/indexing/test_index_processor_base.py b/api/tests/unit_tests/core/rag/indexing/test_index_processor_base.py index 6dc66e5ff59..6dc50ba8a08 100644 --- a/api/tests/unit_tests/core/rag/indexing/test_index_processor_base.py +++ b/api/tests/unit_tests/core/rag/indexing/test_index_processor_base.py @@ -206,6 +206,27 @@ class TestBaseIndexProcessor: assert files == [] + def test_get_content_files_skips_invalid_remote_image_references( + self, processor: _ForwardingBaseIndexProcessor, unbound_session: Session + ) -> None: + document = Document(page_content="ignored", metadata={"document_id": "doc-1", "dataset_id": "ds-1"}) + images = [ + "document_images/image.png", + "//example.com/image.png", + "data:image/png;base64,AAAA", + "ftp://example.com/image.png", + "http://[invalid", + ] + + with ( + patch.object(processor, "_extract_markdown_images", return_value=images), + patch.object(processor, "_download_image") as mock_image_download, + ): + files = processor._get_content_files(document, current_user=Mock(), session=unbound_session) + + assert files == [] + mock_image_download.assert_not_called() + def test_get_content_files_ignores_missing_upload_records( self, processor: _ForwardingBaseIndexProcessor, sqlite_session: Session ) -> None: diff --git a/api/tests/unit_tests/core/tools/test_tool_manager.py b/api/tests/unit_tests/core/tools/test_tool_manager.py index a299fe465dc..eeac6f35365 100644 --- a/api/tests/unit_tests/core/tools/test_tool_manager.py +++ b/api/tests/unit_tests/core/tools/test_tool_manager.py @@ -25,7 +25,7 @@ from core.tools.entities.tool_entities import ( ToolParameter, ToolProviderType, ) -from core.tools.errors import ToolProviderNotFoundError +from core.tools.errors import ToolProviderCredentialValidationError, ToolProviderNotFoundError from core.tools.plugin_tool.provider import PluginToolProviderController from core.tools.tool_manager import ToolManager from models.base import TypeBase @@ -399,7 +399,49 @@ def test_get_tool_runtime_builtin_refreshes_expired_oauth_credentials( cache.delete.assert_called_once() -def test_get_tool_runtime_builtin_plugin_provider_deleted_raises( +def test_get_tool_runtime_builtin_maps_oauth_refresh_failure_to_credential_error( + monkeypatch: pytest.MonkeyPatch, tool_database: _ToolDatabase +): + tool = Mock() + controller = SimpleNamespace( + get_tool=Mock(return_value=tool), + need_credentials=True, + get_credentials_schema_by_type=Mock(return_value=[]), + ) + tenant_id = "00000000-0000-0000-0000-000000000001" + builtin_provider = _builtin_provider( + provider_id="00000000-0000-0000-0000-000000000002", + tenant_id=tenant_id, + credential_type=CredentialType.OAUTH2, + expires_at=1, + ) + tool_database.session.add(builtin_provider) + tool_database.session.commit() + monkeypatch.setattr("core.tools.tool_manager.db", tool_database) + + encrypter = Mock() + encrypter.decrypt.return_value = {"token": "expired"} + with ( + patch.object(ToolManager, "get_builtin_provider", return_value=controller), + patch("core.tools.tool_manager.create_provider_encrypter", return_value=(encrypter, Mock())), + patch("core.tools.tool_manager.time.time", return_value=1000), + patch( + "services.tools.builtin_tools_manage_service.BuiltinToolManageService.get_oauth_client", + return_value={"client_id": "id"}, + ), + patch("core.plugin.impl.oauth.OAuthHandler") as oauth_handler_cls, + ): + oauth_handler_cls.return_value.refresh_credentials.side_effect = ValueError("refresh token revoked") + with pytest.raises(ToolProviderCredentialValidationError, match="could not be refreshed"): + ToolManager.get_tool_runtime( + provider_type=ToolProviderType.BUILT_IN, + provider_id="time", + tool_name="weekday", + tenant_id=tenant_id, + ) + + +def test_get_tool_runtime_builtin_plugin_credential_deleted_raises( monkeypatch: pytest.MonkeyPatch, tool_database: _ToolDatabase ): plugin_controller = object.__new__(PluginToolProviderController) @@ -409,7 +451,7 @@ def test_get_tool_runtime_builtin_plugin_provider_deleted_raises( monkeypatch.setattr("core.tools.tool_manager.db", tool_database) with patch.object(ToolManager, "get_builtin_provider", return_value=plugin_controller): - with pytest.raises(ToolProviderNotFoundError, match="provider has been deleted"): + with pytest.raises(ToolProviderCredentialValidationError, match="credential .* has been deleted"): ToolManager.get_tool_runtime( provider_type=ToolProviderType.BUILT_IN, provider_id="time", @@ -419,6 +461,44 @@ def test_get_tool_runtime_builtin_plugin_provider_deleted_raises( ) +def test_get_tool_runtime_builtin_plugin_without_workspace_credential_raises( + monkeypatch: pytest.MonkeyPatch, tool_database: _ToolDatabase +): + plugin_controller = object.__new__(PluginToolProviderController) + plugin_controller.entity = SimpleNamespace(credentials_schema=[{"name": "k"}], oauth_schema=None) + plugin_controller.get_tool = Mock(return_value=Mock()) + plugin_controller.get_credentials_schema_by_type = Mock(return_value=[]) + + monkeypatch.setattr("core.tools.tool_manager.db", tool_database) + with patch.object(ToolManager, "get_builtin_provider", return_value=plugin_controller): + with pytest.raises(ToolProviderCredentialValidationError, match="No workspace credential is configured"): + ToolManager.get_tool_runtime( + provider_type=ToolProviderType.BUILT_IN, + provider_id="langgenius/dify-gmail/dify-gmail", + tool_name="send_draft", + tenant_id="00000000-0000-0000-0000-000000000001", + ) + + +def test_get_tool_runtime_hardcoded_provider_without_credential_raises( + monkeypatch: pytest.MonkeyPatch, tool_database: _ToolDatabase +): + controller = SimpleNamespace( + get_tool=Mock(return_value=Mock()), + need_credentials=True, + ) + + monkeypatch.setattr("core.tools.tool_manager.db", tool_database) + with patch.object(ToolManager, "get_builtin_provider", return_value=controller): + with pytest.raises(ToolProviderCredentialValidationError, match="No credential is configured"): + ToolManager.get_tool_runtime( + provider_type=ToolProviderType.BUILT_IN, + provider_id="legacy-provider", + tool_name="legacy-tool", + tenant_id="00000000-0000-0000-0000-000000000001", + ) + + def test_get_tool_runtime_api_path(): api_tool = Mock() api_tool.fork_tool_runtime.return_value = "api-runtime" diff --git a/api/tests/unit_tests/core/workflow/nodes/agent/test_message_transformer.py b/api/tests/unit_tests/core/workflow/nodes/agent/test_message_transformer.py index 5e7c553b5f6..26204b040ca 100644 --- a/api/tests/unit_tests/core/workflow/nodes/agent/test_message_transformer.py +++ b/api/tests/unit_tests/core/workflow/nodes/agent/test_message_transformer.py @@ -11,7 +11,7 @@ from core.workflow.nodes.agent.exceptions import ToolFileNotFoundError from core.workflow.nodes.agent.message_transformer import AgentMessageTransformer from graphon.enums import BuiltinNodeTypes from graphon.file import File, FileTransferMethod, FileType -from graphon.node_events import StreamCompletedEvent +from graphon.node_events import StreamChunkEvent, StreamCompletedEvent from graphon.variables.segments import ArrayFileSegment @@ -194,3 +194,80 @@ def test_transform_keeps_plain_link_as_text() -> None: assert text == "Link: https://dify.ai\n" assert files.value == [] + + +def _text(content: str) -> ToolInvokeMessage: + return ToolInvokeMessage( + type=ToolInvokeMessage.MessageType.TEXT, + message=ToolInvokeMessage.TextMessage(text=content), + ) + + +def _log(*, message_id: str = "log-1", label: str = "ROUND 1") -> ToolInvokeMessage: + return ToolInvokeMessage( + type=ToolInvokeMessage.MessageType.LOG, + message=ToolInvokeMessage.LogMessage( + id=message_id, + label=label, + status=ToolInvokeMessage.LogMessage.LogStatus.START, + data={}, + ), + ) + + +def test_transform_closes_think_tag_before_nested_open() -> None: + text, _ = _run_transform( + [ + _text("first pass\n"), + _text("second pass"), + ] + ) + + assert text == "first pass\nsecond pass" + + +def test_transform_closes_think_tag_when_tool_log_interrupts() -> None: + text, _ = _run_transform( + [ + _text("need to search"), + _log(label="CALL search"), + _text("search result summary"), + ] + ) + + assert text == "need to searchsearch result summary" + + +def test_transform_streams_close_tag_before_post_tool_text() -> None: + events = list( + AgentMessageTransformer().transform( + messages=_message_stream( + [ + _text("need to search"), + _log(label="CALL search"), + _text("visible reply"), + ] + ), + tool_info={}, + parameters_for_log={}, + user_id="user-id", + tenant_id="tenant-id", + conversation_id=None, + node_type=BuiltinNodeTypes.AGENT, + node_id="node-id", + node_execution_id="execution-id", + ) + ) + text_chunks = [ + event.chunk for event in events if isinstance(event, StreamChunkEvent) and event.selector == ["node-id", "text"] + ] + + assert text_chunks == ["need to search", "", "visible reply", ""] + + +def test_close_unclosed_think_tags_helper() -> None: + from core.workflow.nodes.agent.think_tags import close_unclosed_think_tags, has_unclosed_think + + assert has_unclosed_think("open") + assert not has_unclosed_think("opendone") + assert close_unclosed_think_tags("a\nb") == "a\nb" diff --git a/api/tests/unit_tests/core/workflow/nodes/agent_v2/test_dify_tools_builder.py b/api/tests/unit_tests/core/workflow/nodes/agent_v2/test_dify_tools_builder.py index f13032325ec..87ce8ae065c 100644 --- a/api/tests/unit_tests/core/workflow/nodes/agent_v2/test_dify_tools_builder.py +++ b/api/tests/unit_tests/core/workflow/nodes/agent_v2/test_dify_tools_builder.py @@ -839,6 +839,7 @@ def test_credential_validation_error_maps_to_credential_invalid(): with pytest.raises(WorkflowAgentDifyToolsBuildError) as exc_info: _build(builder, _standard_tools_payload()) assert exc_info.value.error_code == "agent_tool_credential_invalid" + assert "credential validation failed" in str(exc_info.value) def test_generic_value_error_maps_to_config_invalid(): diff --git a/api/tests/unit_tests/events/event_handlers/test_sync_plugin_trigger_when_app_created.py b/api/tests/unit_tests/events/event_handlers/test_sync_plugin_trigger_when_app_created.py new file mode 100644 index 00000000000..a9d631cf7dd --- /dev/null +++ b/api/tests/unit_tests/events/event_handlers/test_sync_plugin_trigger_when_app_created.py @@ -0,0 +1,31 @@ +from types import SimpleNamespace +from typing import cast +from unittest.mock import patch + +from events.event_handlers.sync_plugin_trigger_when_app_created import handle +from models.model import AppMode +from models.workflow import Workflow + + +def test_syncs_plugin_trigger_relationships_from_published_workflow() -> None: + app = SimpleNamespace(mode=AppMode.WORKFLOW.value) + published_workflow = cast(Workflow, object()) + + with patch( + "events.event_handlers.sync_plugin_trigger_when_app_created.TriggerService.sync_plugin_trigger_relationships" + ) as sync_relationships: + handle(app, published_workflow=published_workflow) + + sync_relationships.assert_called_once_with(app, published_workflow) + + +def test_keeps_draft_workflow_relationship_sync() -> None: + app = SimpleNamespace(mode=AppMode.WORKFLOW.value) + draft_workflow = cast(Workflow, object()) + + with patch( + "events.event_handlers.sync_plugin_trigger_when_app_created.TriggerService.sync_plugin_trigger_relationships" + ) as sync_relationships: + handle(app, synced_draft_workflow=draft_workflow) + + sync_relationships.assert_called_once_with(app, draft_workflow) diff --git a/api/tests/unit_tests/extensions/test_ext_application_services.py b/api/tests/unit_tests/extensions/test_ext_application_services.py index d0c60c0f4e2..1601d420982 100644 --- a/api/tests/unit_tests/extensions/test_ext_application_services.py +++ b/api/tests/unit_tests/extensions/test_ext_application_services.py @@ -21,8 +21,17 @@ from models.account import Account from models.model import AccountTrialAppRecord, DifySetup from repositories.account_activation_repository import SQLAlchemyAccountActivationRepository from repositories.account_integration_repository import SQLAlchemyAccountIntegrationRepository +from repositories.account_oauth_repository import ( + AccountServiceOAuthAccountRegistrationGateway, + AccountServiceOAuthSessionGateway, + AccountServiceOAuthWorkspaceGateway, + RegisterServiceOAuthInvitationGateway, +) from repositories.account_repository import SQLAlchemyAccountRepository from repositories.app_site_command_repository import AppSiteCommandRepository +from repositories.app_statistic_query_repository import AppStatisticQueryRepository +from repositories.sqlalchemy_api_workflow_run_repository import DifyAPISQLAlchemyWorkflowRunRepository +from repositories.workflow_app_log_query_repository import WorkflowAppLogQueryRepository from repositories.workflow_run_archive_repository import WorkflowRunArchiveBundleQueryRepository from services import account_forgot_password_service, recommended_app_catalog_gateway from services.account_adapters import ( @@ -44,6 +53,10 @@ from services.account_forgot_password_adapters import ( RedisForgotPasswordSecurityGateway, RedisForgotPasswordTokenGateway, ) +from services.account_oauth_adapters import ( + DeploymentOAuthPolicyGateway, + RedisOAuthAccountClaimLock, +) from services.app_site_service import AppSiteService from services.auth.data_source_api_key_auth_service import DataSourceApiKeyAuthService from services.billing_portal_service import BillingPortalService @@ -58,6 +71,8 @@ from services.retention.workflow_run.archive_download_task_cache import Workflow from services.retention.workflow_run.archive_log_service import WorkflowRunArchiveService from services.tag_application_service import TagApplicationService from services.webapp_access_query_service import WebAppAccessUnavailableError +from services.workflow_app_log_query_service import WorkflowAppLogQueryService +from services.workflow_run_service import WorkflowRunService from services.workflow_statistic_query_service import WorkflowStatisticQueryService from tests.unit_tests.config_override import apply_config_overrides @@ -251,6 +266,51 @@ def test_build_application_services_wires_app_site_boundary( assert services.app_sites._sites._session_factory is sqlite_session_factory +def test_build_application_services_wires_workflow_app_log_boundary( + sqlite_session_factory: sessionmaker[Session], +) -> None: + services = ext_application_services.build_application_services( + database_client=sqlite_session_factory, + deployment_edition=DeploymentEdition.COMMUNITY, + initialization_password="", + redis=MagicMock(spec=RedisClientWrapper), + ) + + assert isinstance(services.workflow_app_logs, WorkflowAppLogQueryService) + assert isinstance(services.workflow_app_logs._logs, WorkflowAppLogQueryRepository) + assert services.workflow_app_logs._logs._session_factory is sqlite_session_factory + + +def test_build_application_services_wires_app_statistic_boundary( + sqlite_session_factory: sessionmaker[Session], +) -> None: + services = ext_application_services.build_application_services( + database_client=sqlite_session_factory, + deployment_edition=DeploymentEdition.COMMUNITY, + initialization_password="", + redis=MagicMock(spec=RedisClientWrapper), + ) + + assert isinstance(services.app_statistics, AppStatisticQueryRepository) + assert services.app_statistics._session_factory is sqlite_session_factory + + +def test_build_application_services_wires_workflow_run_service( + sqlite_session_factory: sessionmaker[Session], +) -> None: + services = ext_application_services.build_application_services( + database_client=sqlite_session_factory, + deployment_edition=DeploymentEdition.COMMUNITY, + initialization_password="", + redis=MagicMock(spec=RedisClientWrapper), + ) + + workflow_runs = services.workflow_runs + assert isinstance(workflow_runs, WorkflowRunService) + assert isinstance(workflow_runs._workflow_runs, DifyAPISQLAlchemyWorkflowRunRepository) + assert workflow_runs._workflow_runs._session_maker is sqlite_session_factory + + def test_build_application_services_wires_billing_service( sqlite_session: Session, sqlite_session_factory: sessionmaker[Session], @@ -424,12 +484,23 @@ def test_build_application_services_wires_account_profile_repository( assert services.accounts.deletion._accounts is accounts assert services.accounts.authentication._accounts is accounts assert services.accounts.authentication._workspaces is services.workspace_queries._workspaces - assert services.notifications._accounts is accounts assert services.step_by_step_tour._accounts is accounts assert services.accounts.deletion._memberships is services.workspace_queries._workspaces integrations = services.accounts.integrations._integrations assert isinstance(integrations, SQLAlchemyAccountIntegrationRepository) assert integrations._session_factory is sqlite_session_factory + oauth = services.accounts.oauth + assert oauth._accounts is accounts + assert oauth._integrations is integrations + assert oauth._memberships is services.workspace_queries._workspaces + assert isinstance(oauth._invitations, RegisterServiceOAuthInvitationGateway) + assert isinstance(oauth._account_claims, RedisOAuthAccountClaimLock) + assert isinstance(oauth._registration, AccountServiceOAuthAccountRegistrationGateway) + assert isinstance(oauth._workspaces, AccountServiceOAuthWorkspaceGateway) + assert isinstance(oauth._sessions, AccountServiceOAuthSessionGateway) + assert oauth._sessions is not oauth._workspaces + assert isinstance(oauth._registration_policy, DeploymentOAuthPolicyGateway) + assert oauth._workspace_policy is oauth._registration_policy avatar_files = services.accounts.avatar._files assert isinstance(avatar_files, SQLAlchemyAccountAvatarFileGateway) assert avatar_files._session_factory is sqlite_session_factory @@ -700,14 +771,12 @@ def test_build_application_services_wires_dynamic_recommended_catalog( ) with patch.object(recommended_app_catalog_gateway.Path, "read_text", return_value=builtin_payload): result = services.recommended_app_queries.list_recommended( - requested_language="en-US", - interface_language=None, + language="en-US", ) assert result.recommended_apps apply_config_overrides(monkeypatch, HOSTED_FETCH_APP_TEMPLATES_MODE="invalid") with pytest.raises(ValueError, match="invalid fetch recommended apps mode: invalid"): services.recommended_app_queries.list_recommended( - requested_language="en-US", - interface_language=None, + language="en-US", ) diff --git a/api/tests/unit_tests/file_grant_test_utils.py b/api/tests/unit_tests/file_grant_test_utils.py new file mode 100644 index 00000000000..0fb7eae9408 --- /dev/null +++ b/api/tests/unit_tests/file_grant_test_utils.py @@ -0,0 +1,31 @@ +import time +from collections.abc import Sequence + +from configs import dify_config +from services.entities.file_grant_entities import FileGrantContext, FileGrantScope +from services.file_grant_gateways import FileGrantTokenGateway + + +def token_gateway() -> FileGrantTokenGateway: + return FileGrantTokenGateway( + secret_key=dify_config.SECRET_KEY, + external_files_url=dify_config.FILES_URL, + internal_files_url=dify_config.INTERNAL_FILES_URL or dify_config.FILES_URL, + content_token_ttl_seconds=dify_config.FILES_ACCESS_TIMEOUT, + now=lambda: int(time.time()), + ) + + +def issue_file_grant( + *, + end_user_id: str, + tenant_id: str, + app_id: str, + scopes: Sequence[FileGrantScope], + ttl_seconds: int, +) -> tuple[str, int]: + return token_gateway().issue_grant( + context=FileGrantContext(tenant_id=tenant_id, app_id=app_id, end_user_id=end_user_id), + scopes=scopes, + ttl_seconds=ttl_seconds, + ) diff --git a/api/tests/unit_tests/libs/test_datetime_utils.py b/api/tests/unit_tests/libs/test_datetime_utils.py index e2dce54d1e0..a4229d20777 100644 --- a/api/tests/unit_tests/libs/test_datetime_utils.py +++ b/api/tests/unit_tests/libs/test_datetime_utils.py @@ -4,7 +4,19 @@ from unittest.mock import patch import pytest import pytz -from libs.datetime_utils import naive_utc_now, parse_time_range, to_utc_timestamp +from libs.datetime_utils import naive_utc_now, parse_time_range, to_utc_timestamp, utc_now + + +def test_utc_now(monkeypatch: pytest.MonkeyPatch): + expected = datetime.datetime(2026, 8, 26, 12, tzinfo=datetime.UTC) + + def _now_func(tz: datetime.timezone | None) -> datetime.datetime: + return expected.astimezone(tz) + + monkeypatch.setattr("libs.datetime_utils._now_func", _now_func) + + assert utc_now() == expected + assert utc_now().tzinfo is datetime.UTC def test_naive_utc_now(monkeypatch: pytest.MonkeyPatch): diff --git a/api/tests/unit_tests/models/test_end_user_type.py b/api/tests/unit_tests/models/test_end_user_type.py index 222945814b4..af6407c7b7c 100644 --- a/api/tests/unit_tests/models/test_end_user_type.py +++ b/api/tests/unit_tests/models/test_end_user_type.py @@ -15,6 +15,7 @@ API_ROOT = Path(__file__).resolve().parents[3] def test_end_user_type_covers_persisted_creation_values(): assert {member.value for member in EndUserType} == { + "app-deploy", "browser", "mcp", "openapi", diff --git a/api/tests/unit_tests/pyrefly.toml b/api/tests/unit_tests/pyrefly.toml index c36941f0469..42abc4e6e29 100644 --- a/api/tests/unit_tests/pyrefly.toml +++ b/api/tests/unit_tests/pyrefly.toml @@ -863,7 +863,6 @@ project-excludes = [ "services/test_webhook_service.py", "services/test_webhook_service_additional.py", "services/test_website_service.py", - "services/test_workflow_app_service_metadata.py", "services/test_workflow_collaboration_service.py", "services/test_workflow_comment_service.py", "services/test_workflow_generator_service.py", diff --git a/api/tests/unit_tests/repositories/test_account_repository.py b/api/tests/unit_tests/repositories/test_account_repository.py index df182e62818..f41c2fadda4 100644 --- a/api/tests/unit_tests/repositories/test_account_repository.py +++ b/api/tests/unit_tests/repositories/test_account_repository.py @@ -195,6 +195,52 @@ def test_account_repository_finds_email_with_lowercase_fallback( assert account.email == "account@example.com" +def test_account_repositories_resolve_oauth_identity_and_email_fallback( + sqlite_session: Session, + sqlite_session_factory: sessionmaker[Session], +) -> None: + _persist_account(sqlite_session) + sqlite_session.add( + AccountIntegrate( + account_id="account-1", + provider="github", + open_id="github-user", + encrypted_token="", + ) + ) + sqlite_session.commit() + account_repository = SQLAlchemyAccountRepository(sqlite_session_factory) + integration_repository = SQLAlchemyAccountIntegrationRepository(sqlite_session_factory) + + oauth_account_id = integration_repository.find_account_id(provider="github", open_id="github-user") + oauth_account = account_repository.get(oauth_account_id) if oauth_account_id is not None else None + email_account = account_repository.find_by_email("ACCOUNT@Example.com") + + assert oauth_account is not None + assert oauth_account.id == "account-1" + assert email_account is not None + assert email_account.id == "account-1" + + +def test_account_repository_activates_only_pending_account( + sqlite_session: Session, + sqlite_session_factory: sessionmaker[Session], +) -> None: + account = _persist_account(sqlite_session) + account.status = AccountStatus.PENDING + sqlite_session.commit() + initialized_at = datetime(2026, 8, 24, 12, 0) + repository = SQLAlchemyAccountRepository(sqlite_session_factory) + + repository.activate_pending("account-1", initialized_at=initialized_at) + + sqlite_session.expire_all() + persisted = sqlite_session.get(Account, "account-1") + assert persisted is not None + assert persisted.status == AccountStatus.ACTIVE + assert persisted.initialized_at == initialized_at + + def test_account_repository_fails_closed_for_duplicate_email( sqlite_session: Session, sqlite_session_factory: sessionmaker[Session], @@ -234,6 +280,24 @@ def test_account_integration_repository_lists_integrations( assert integrations[0].provider == "github" +def test_account_integration_repository_upserts_provider_binding( + sqlite_session: Session, + sqlite_session_factory: sessionmaker[Session], +) -> None: + _persist_account(sqlite_session) + repository = SQLAlchemyAccountIntegrationRepository(sqlite_session_factory) + + repository.link("account-1", provider="github", open_id="first-user") + repository.link("account-1", provider="github", open_id="second-user") + + sqlite_session.expire_all() + integrations = list(sqlite_session.query(AccountIntegrate).all()) + assert len(integrations) == 1 + assert integrations[0].account_id == "account-1" + assert integrations[0].provider == "github" + assert integrations[0].open_id == "second-user" + + def test_account_repository_initializes_account_and_consumes_invitation_atomically( sqlite_session: Session, sqlite_session_factory: sessionmaker[Session], diff --git a/api/tests/unit_tests/repositories/test_app_statistic_query_repository.py b/api/tests/unit_tests/repositories/test_app_statistic_query_repository.py new file mode 100644 index 00000000000..b8fb51e3dad --- /dev/null +++ b/api/tests/unit_tests/repositories/test_app_statistic_query_repository.py @@ -0,0 +1,155 @@ +from datetime import UTC, datetime +from decimal import Decimal +from typing import cast, override + +import pytest +from sqlalchemy.engine import RowMapping +from sqlalchemy.orm import Session, sessionmaker + +from core.app.entities.app_invoke_entities import InvokeFrom +from repositories import app_statistic_query_repository as repository_module +from repositories.app_statistic_query_repository import AppStatisticQueryRepository +from services.app_statistic_query import ( + AverageResponseTimeStatisticRecord, + AverageSessionInteractionStatisticRecord, + DailyConversationStatisticRecord, + DailyMessageStatisticRecord, + DailyTerminalStatisticRecord, + DailyTokenCostStatisticRecord, + TokensPerSecondStatisticRecord, + UserSatisfactionRateStatisticRecord, +) + + +class _RecordingRepository(AppStatisticQueryRepository): + def __init__(self) -> None: + super().__init__(session_factory=cast(sessionmaker[Session], object())) + self.rows: tuple[RowMapping, ...] = () + self.calls: list[tuple[str, dict[str, object]]] = [] + + @override + def _execute(self, sql_query: str, parameters: dict[str, object]) -> tuple[RowMapping, ...]: + self.calls.append((sql_query, parameters.copy())) + return self.rows + + +def _row(**values: object) -> RowMapping: + return cast(RowMapping, values) + + +def test_app_statistic_repository_maps_results_and_preserves_query_scope( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(repository_module, "convert_datetime_to_date", lambda field: field) + repository = _RecordingRepository() + start_date = datetime(2024, 1, 1, tzinfo=UTC) + end_date = datetime(2024, 1, 2, tzinfo=UTC) + + repository.rows = (_row(date="2024-01-01", message_count=2),) + assert repository.get_daily_messages( + app_id="app-1", + start_date=start_date, + end_date=end_date, + timezone="Asia/Shanghai", + ) == (DailyMessageStatisticRecord(date="2024-01-01", message_count=2),) + + repository.rows = (_row(date="2024-01-01", conversation_count=3),) + assert repository.get_daily_conversations( + app_id="app-1", + start_date=start_date, + end_date=end_date, + timezone="Asia/Shanghai", + ) == (DailyConversationStatisticRecord(date="2024-01-01", conversation_count=3),) + + repository.rows = (_row(date="2024-01-01", terminal_count=4),) + assert repository.get_daily_terminals( + app_id="app-1", + start_date=start_date, + end_date=end_date, + timezone="Asia/Shanghai", + ) == (DailyTerminalStatisticRecord(date="2024-01-01", terminal_count=4),) + + repository.rows = (_row(date="2024-01-01", token_count=Decimal(5), total_price=Decimal("0.25")),) + assert repository.get_daily_token_costs( + app_id="app-1", + start_date=start_date, + end_date=end_date, + timezone="Asia/Shanghai", + ) == ( + DailyTokenCostStatisticRecord( + date="2024-01-01", + token_count=5, + total_price=Decimal("0.25"), + currency="USD", + ), + ) + + repository.rows = (_row(date="2024-01-01", interactions=Decimal("2.345")),) + assert repository.get_average_session_interactions( + app_id="app-1", + start_date=start_date, + end_date=end_date, + timezone="Asia/Shanghai", + ) == (AverageSessionInteractionStatisticRecord(date="2024-01-01", interactions=2.34),) + + repository.rows = (_row(date="2024-01-01", message_count=10, feedback_count=1),) + assert repository.get_user_satisfaction_rates( + app_id="app-1", + start_date=start_date, + end_date=end_date, + timezone="Asia/Shanghai", + ) == (UserSatisfactionRateStatisticRecord(date="2024-01-01", rate=100.0),) + + repository.rows = (_row(date="2024-01-01", latency=1.234),) + assert repository.get_average_response_times( + app_id="app-1", + start_date=start_date, + end_date=end_date, + timezone="Asia/Shanghai", + ) == (AverageResponseTimeStatisticRecord(date="2024-01-01", latency=1234.0),) + + repository.rows = (_row(date="2024-01-01", tokens_per_second=15.55555),) + assert repository.get_tokens_per_second( + app_id="app-1", + start_date=start_date, + end_date=end_date, + timezone="Asia/Shanghai", + ) == (TokensPerSecondStatisticRecord(date="2024-01-01", tps=15.5556),) + + assert len(repository.calls) == 8 + for sql_query, parameters in repository.calls: + assert "created_at >= :start_date" in sql_query + assert "created_at < :end_date" in sql_query + assert parameters == { + "tz": "Asia/Shanghai", + "app_id": "app-1", + "excluded_invoke_from": InvokeFrom.DEBUGGER, + "start_date": start_date, + "end_date": end_date, + } + + +def test_daily_messages_omit_time_range_when_not_provided( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(repository_module, "convert_datetime_to_date", lambda field: field) + repository = _RecordingRepository() + + assert ( + repository.get_daily_messages( + app_id="app-1", + start_date=None, + end_date=None, + timezone="Asia/Shanghai", + ) + == () + ) + + sql_query, parameters = repository.calls[0] + assert ":start_date" not in sql_query + assert ":end_date" not in sql_query + assert parameters == { + "tz": "Asia/Shanghai", + "app_id": "app-1", + "excluded_invoke_from": InvokeFrom.DEBUGGER, + } diff --git a/api/tests/unit_tests/repositories/test_file_grant_repository.py b/api/tests/unit_tests/repositories/test_file_grant_repository.py new file mode 100644 index 00000000000..b27353a622b --- /dev/null +++ b/api/tests/unit_tests/repositories/test_file_grant_repository.py @@ -0,0 +1,75 @@ +from sqlalchemy import Engine, event +from sqlalchemy.orm import Session, sessionmaker + +from extensions.storage.storage_type import StorageType +from libs.datetime_utils import naive_utc_now +from models.enums import CreatorUserRole +from models.model import UploadFile +from models.tools import ToolFile +from repositories.file_grant_repository import FileGrantRepository +from services.entities.file_grant_entities import FileGrantContext, FileKind, FileRef + + +def test_resolve_owned_files_uses_one_query_per_file_kind( + sqlite_engine: Engine, + sqlite_session_factory: sessionmaker[Session], +) -> None: + end_user_id = "11111111-1111-4111-8111-111111111111" + tenant_id = "22222222-2222-4222-8222-222222222222" + with sqlite_session_factory.begin() as session: + upload = UploadFile( + tenant_id=tenant_id, + storage_type=StorageType.OPENDAL, + key="upload_files/report.pdf", + name="report.pdf", + size=10, + extension="pdf", + mime_type="application/pdf", + created_by=end_user_id, + created_by_role=CreatorUserRole.END_USER, + created_at=naive_utc_now(), + used=False, + ) + tool_file = ToolFile( + user_id=end_user_id, + tenant_id=tenant_id, + conversation_id=None, + file_key="tools/chart.png", + mimetype="image/png", + name="chart.png", + size=20, + ) + session.add_all([upload, tool_file]) + session.flush() + upload_id = upload.id + tool_file_id = tool_file.id + + statements: list[str] = [] + + def record_statement( + _connection: object, + _cursor: object, + statement: str, + _parameters: object, + _context: object, + _executemany: object, + ) -> None: + statements.append(statement) + + event.listen(sqlite_engine, "before_cursor_execute", record_statement) + try: + refs = tuple( + FileRef(id=upload_id, kind=FileKind.UPLOAD) + if index % 2 == 0 + else FileRef(id=tool_file_id, kind=FileKind.TOOL) + for index in range(100) + ) + resolved = FileGrantRepository(session_factory=sqlite_session_factory).resolve_owned_files( + context=FileGrantContext(tenant_id, "app-1", end_user_id), + refs=refs, + ) + finally: + event.remove(sqlite_engine, "before_cursor_execute", record_statement) + + assert len(statements) == 2 + assert [file.kind if file is not None else None for file in resolved] == [ref.kind for ref in refs] diff --git a/api/tests/unit_tests/repositories/test_oauth_access_token_repository.py b/api/tests/unit_tests/repositories/test_oauth_access_token_repository.py new file mode 100644 index 00000000000..3fd15186d84 --- /dev/null +++ b/api/tests/unit_tests/repositories/test_oauth_access_token_repository.py @@ -0,0 +1,126 @@ +from __future__ import annotations + +from datetime import UTC, datetime, timedelta + +import pytest +from sqlalchemy.orm import Session, sessionmaker + +from models.oauth import OAuthAccessToken +from repositories.oauth_access_token_repository import SQLAlchemyOAuthAccessTokenRepository + +ACCOUNT_ID = "11111111-1111-1111-1111-111111111111" +OTHER_ACCOUNT_ID = "22222222-2222-2222-2222-222222222222" +TOKEN_ID = "33333333-3333-3333-3333-333333333333" +OTHER_TOKEN_ID = "44444444-4444-4444-4444-444444444444" +NOW = datetime(2026, 8, 25, 12, tzinfo=UTC) + + +def _token( + *, + token_id: str = TOKEN_ID, + account_id: str | None = ACCOUNT_ID, + token_hash: str | None = "live-hash", + expires_at: datetime | None = None, + revoked_at: datetime | None = None, + created_at: datetime | None = None, +) -> OAuthAccessToken: + token = OAuthAccessToken( + subject_email="user@example.com", + subject_issuer="dify:account" if account_id is not None else "https://idp.example.com", + account_id=account_id, + client_id="difyctl", + device_label="test-device", + prefix="dfoa_" if account_id is not None else "dfoe_", + token_hash=token_hash, + expires_at=expires_at or NOW + timedelta(days=1), + revoked_at=revoked_at, + ) + token.id = token_id + if created_at is not None: + token.created_at = created_at + return token + + +@pytest.mark.parametrize("sqlite_session", [(OAuthAccessToken,)], indirect=True) +def test_list_active_is_account_scoped_and_database_paginated( + sqlite_session: Session, + sqlite_session_factory: sessionmaker[Session], +) -> None: + sqlite_session.add_all( + [ + _token(token_id=TOKEN_ID, created_at=NOW - timedelta(minutes=1)), + _token(token_id=OTHER_TOKEN_ID, created_at=NOW - timedelta(minutes=2)), + _token( + token_id="55555555-5555-5555-5555-555555555555", + expires_at=NOW - timedelta(seconds=1), + ), + _token( + token_id="66666666-6666-6666-6666-666666666666", + token_hash=None, + revoked_at=NOW - timedelta(seconds=1), + ), + _token(token_id="77777777-7777-7777-7777-777777777777", account_id=OTHER_ACCOUNT_ID), + _token(token_id="88888888-8888-8888-8888-888888888888", account_id=None), + ] + ) + sqlite_session.commit() + repository = SQLAlchemyOAuthAccessTokenRepository(session_factory=sqlite_session_factory) + + total, rows = repository.list_active(account_id=ACCOUNT_ID, active_at=NOW, offset=1, limit=1) + + assert total == 2 + assert [row.id for row in rows] == [OTHER_TOKEN_ID] + + +@pytest.mark.parametrize("sqlite_session", [(OAuthAccessToken,)], indirect=True) +def test_revoke_returns_hash_and_persists_revocation( + sqlite_session: Session, + sqlite_session_factory: sessionmaker[Session], +) -> None: + sqlite_session.add(_token()) + sqlite_session.commit() + repository = SQLAlchemyOAuthAccessTokenRepository(session_factory=sqlite_session_factory) + + result = repository.revoke(account_id=ACCOUNT_ID, token_id=TOKEN_ID, revoked_at=NOW) + + assert result.owned is True + assert result.token_hash == "live-hash" + sqlite_session.expire_all() + persisted = sqlite_session.get(OAuthAccessToken, TOKEN_ID) + assert persisted is not None + assert persisted.token_hash is None + assert persisted.revoked_at is not None + + +@pytest.mark.parametrize("sqlite_session", [(OAuthAccessToken,)], indirect=True) +def test_revoke_is_idempotent_for_an_owned_session( + sqlite_session: Session, + sqlite_session_factory: sessionmaker[Session], +) -> None: + sqlite_session.add(_token(token_hash=None, revoked_at=NOW - timedelta(minutes=1))) + sqlite_session.commit() + repository = SQLAlchemyOAuthAccessTokenRepository(session_factory=sqlite_session_factory) + + result = repository.revoke(account_id=ACCOUNT_ID, token_id=TOKEN_ID, revoked_at=NOW) + + assert result.owned is True + assert result.token_hash is None + + +@pytest.mark.parametrize("sqlite_session", [(OAuthAccessToken,)], indirect=True) +def test_revoke_does_not_disclose_another_accounts_session( + sqlite_session: Session, + sqlite_session_factory: sessionmaker[Session], +) -> None: + sqlite_session.add(_token(account_id=OTHER_ACCOUNT_ID)) + sqlite_session.commit() + repository = SQLAlchemyOAuthAccessTokenRepository(session_factory=sqlite_session_factory) + + result = repository.revoke(account_id=ACCOUNT_ID, token_id=TOKEN_ID, revoked_at=NOW) + + assert result.owned is False + assert result.token_hash is None + sqlite_session.expire_all() + persisted = sqlite_session.get(OAuthAccessToken, TOKEN_ID) + assert persisted is not None + assert persisted.token_hash == "live-hash" diff --git a/api/tests/unit_tests/repositories/test_sqlalchemy_api_workflow_run_repository.py b/api/tests/unit_tests/repositories/test_sqlalchemy_api_workflow_run_repository.py index 76383d88b9d..0606ccd23c8 100644 --- a/api/tests/unit_tests/repositories/test_sqlalchemy_api_workflow_run_repository.py +++ b/api/tests/unit_tests/repositories/test_sqlalchemy_api_workflow_run_repository.py @@ -2,19 +2,25 @@ from __future__ import annotations import logging from datetime import UTC, datetime +from decimal import Decimal from unittest.mock import Mock, patch import pytest -from sqlalchemy.orm import Session +from sqlalchemy.orm import Session, sessionmaker from core.workflow.nodes.human_input.entities import FormDefinition, ParagraphInputConfig, UserActionConfig from core.workflow.nodes.human_input.enums import FormInputType from core.workflow.nodes.human_input.pause_reason import HumanInputRequired from graphon.entities.pause_reason import HitlRequired, PauseReasonType +from graphon.enums import WorkflowExecutionStatus, WorkflowType +from models import Message +from models.enums import ConversationFromSource, CreatorUserRole, WorkflowRunTriggeredFrom from models.human_input import HumanInputForm, HumanInputFormRecipient, RecipientType -from models.workflow import WorkflowPause, WorkflowPauseReason +from models.workflow import WorkflowPause, WorkflowPauseReason, WorkflowRun from repositories.sqlalchemy_api_workflow_run_repository import ( DifyAPISQLAlchemyWorkflowRunRepository, + WorkflowRunMessageRef, + WorkflowRunPauseRecord, _build_human_input_required_reason, _PrivateWorkflowPauseEntity, ) @@ -159,6 +165,133 @@ def test_private_workflow_pause_entity_preserves_list_shaped_pause_reasons() -> assert result == pause_reasons +def _message(*, message_id: str, app_id: str, workflow_run_id: str, conversation_id: str) -> Message: + message = Message( + app_id=app_id, + conversation_id=conversation_id, + query="query", + message={"role": "user", "content": "query"}, + answer="answer", + message_unit_price=Decimal("0.0001"), + answer_unit_price=Decimal("0.0001"), + currency="USD", + from_source=ConversationFromSource.API, + ) + message.id = message_id + message._inputs = {} + message.workflow_run_id = workflow_run_id + return message + + +def _workflow_run(*, run_id: str, tenant_id: str, status: WorkflowExecutionStatus) -> WorkflowRun: + return WorkflowRun( + id=run_id, + tenant_id=tenant_id, + app_id="app-1", + workflow_id="workflow-1", + type=WorkflowType.WORKFLOW, + triggered_from=WorkflowRunTriggeredFrom.DEBUGGING, + version="1", + graph="{}", + inputs="{}", + status=status, + created_by_role=CreatorUserRole.ACCOUNT, + created_by="account-1", + ) + + +def test_get_message_refs_filters_by_app_and_returns_lightweight_records( + sqlite_session: Session, + sqlite_session_factory: sessionmaker[Session], +) -> None: + sqlite_session.add_all( + [ + _message(message_id="msg-1", app_id="app-1", workflow_run_id="run-1", conversation_id="conv-1"), + _message(message_id="msg-2", app_id="app-2", workflow_run_id="run-2", conversation_id="conv-2"), + ] + ) + sqlite_session.commit() + repository = DifyAPISQLAlchemyWorkflowRunRepository(session_maker=sqlite_session_factory) + + result = repository.get_message_refs( + app_id="app-1", + workflow_run_ids=["run-1", "run-2"], + ) + + assert result == { + "run-1": WorkflowRunMessageRef(message_id="msg-1", conversation_id="conv-1"), + } + + +def test_get_pause_record_scopes_the_workflow_run_to_the_workspace( + sqlite_session: Session, + sqlite_session_factory: sessionmaker[Session], +) -> None: + sqlite_session.add( + _workflow_run( + run_id="run-1", + tenant_id="tenant-1", + status=WorkflowExecutionStatus.SUCCEEDED, + ) + ) + sqlite_session.commit() + repository = DifyAPISQLAlchemyWorkflowRunRepository(session_maker=sqlite_session_factory) + + assert repository.get_pause_record(workspace_id="tenant-2", workflow_run_id="run-1") is None + assert repository.get_pause_record( + workspace_id="tenant-1", + workflow_run_id="run-1", + ) == WorkflowRunPauseRecord( + status=WorkflowExecutionStatus.SUCCEEDED, + paused_at=None, + reasons=(), + form_tokens={}, + ) + + +def test_get_pause_record_loads_reasons_and_tokens_in_one_repository_call( + sqlite_session: Session, + sqlite_session_factory: sessionmaker[Session], +) -> None: + workflow_run = _workflow_run( + run_id="run-1", + tenant_id="tenant-1", + status=WorkflowExecutionStatus.PAUSED, + ) + pause = WorkflowPause( + workflow_id=workflow_run.workflow_id, + workflow_run_id=workflow_run.id, + state_object_key="pause-state", + ) + pause.id = "pause-1" + reason = WorkflowPauseReason( + pause_id=pause.id, + type_=PauseReasonType.HITL_REQUIRED, + form_id="form-1", + node_id="node-1", + ) + recipient = HumanInputFormRecipient( + form_id="form-1", + delivery_id="delivery-1", + recipient_type=RecipientType.CONSOLE, + recipient_payload="{}", + access_token="form-token", + ) + sqlite_session.add_all([workflow_run, pause, reason, recipient]) + sqlite_session.commit() + repository = DifyAPISQLAlchemyWorkflowRunRepository(session_maker=sqlite_session_factory) + + result = repository.get_pause_record(workspace_id="tenant-1", workflow_run_id="run-1") + + assert result is not None + assert result.status == WorkflowExecutionStatus.PAUSED + assert result.paused_at == pause.created_at + assert len(result.reasons) == 1 + assert isinstance(result.reasons[0], HumanInputRequired) + assert result.reasons[0].form_id == "form-1" + assert result.form_tokens == {"form-1": "form-token"} + + def test_delete_pause_model_deletes_record_when_state_object_delete_fails( caplog: pytest.LogCaptureFixture, ) -> None: diff --git a/api/tests/unit_tests/repositories/test_workflow_app_log_query_repository.py b/api/tests/unit_tests/repositories/test_workflow_app_log_query_repository.py new file mode 100644 index 00000000000..2de5e684a02 --- /dev/null +++ b/api/tests/unit_tests/repositories/test_workflow_app_log_query_repository.py @@ -0,0 +1,304 @@ +import json +from datetime import UTC, datetime, timedelta + +import pytest +from sqlalchemy.orm import Session, sessionmaker + +from graphon.enums import WorkflowExecutionStatus +from models.account import Account, Tenant, TenantAccountJoin, TenantAccountRole +from models.enums import ( + AppTriggerType, + CreatorUserRole, + EndUserType, + WorkflowRunTriggeredFrom, + WorkflowTriggerStatus, +) +from models.model import EndUser +from models.trigger import WorkflowTriggerLog +from models.workflow import WorkflowAppLog, WorkflowAppLogCreatedFrom, WorkflowRun, WorkflowType +from repositories.workflow_app_log_query_repository import WorkflowAppLogQueryRepository +from services.workflow_app_log_query_service import WorkflowAppLogAccount, WorkflowAppLogEndUser + + +def _log( + log_id: str, + *, + created_by: str, + created_by_role: CreatorUserRole, + created_at: datetime, +) -> WorkflowAppLog: + log = WorkflowAppLog( + tenant_id="tenant-1", + app_id="app-1", + workflow_id="workflow-1", + workflow_run_id=f"run-{log_id}", + created_from=WorkflowAppLogCreatedFrom.SERVICE_API, + created_by_role=created_by_role, + created_by=created_by, + ) + log.id = log_id + log.created_at = created_at + return log + + +def _run( + run_id: str, + *, + tenant_id: str = "tenant-1", + app_id: str = "app-1", + triggered_from: WorkflowRunTriggeredFrom = WorkflowRunTriggeredFrom.APP_RUN, +) -> WorkflowRun: + created_at = datetime(2026, 1, 1, tzinfo=UTC) + return WorkflowRun( + id=run_id, + tenant_id=tenant_id, + app_id=app_id, + workflow_id="workflow-1", + type=WorkflowType.WORKFLOW, + triggered_from=triggered_from, + version="2026-01-01", + graph=json.dumps({"nodes": [], "edges": []}), + inputs=json.dumps({"input": "value"}), + status=WorkflowExecutionStatus.SUCCEEDED, + outputs=json.dumps({"output": "value"}), + error=None, + elapsed_time=0.5, + total_tokens=10, + total_steps=2, + created_by_role=CreatorUserRole.ACCOUNT, + created_by="actor-1", + created_at=created_at, + finished_at=created_at + timedelta(seconds=1), + exceptions_count=0, + ) + + +def test_get_paginated_returns_detached_actor_records_by_role_when_ids_overlap( + sqlite_session: Session, + sqlite_session_factory: sessionmaker[Session], +) -> None: + created_at = datetime(2026, 1, 1) + tenant = Tenant(name="Workspace") + tenant.id = "tenant-1" + account = Account(name="Account", email="account@example.com") + account.id = "actor-1" + end_user = EndUser( + id="actor-1", + tenant_id=tenant.id, + app_id="app-1", + type=EndUserType.BROWSER, + session_id="session-1", + ) + sqlite_session.add_all( + [ + tenant, + account, + end_user, + TenantAccountJoin( + tenant_id=tenant.id, + account_id=account.id, + role=TenantAccountRole.OWNER, + ), + _log( + "account-log", + created_by=account.id, + created_by_role=CreatorUserRole.ACCOUNT, + created_at=created_at, + ), + _log( + "end-user-log", + created_by=end_user.id, + created_by_role=CreatorUserRole.END_USER, + created_at=created_at + timedelta(seconds=1), + ), + ] + ) + sqlite_session.commit() + sqlite_session.close() + + result = WorkflowAppLogQueryRepository(session_factory=sqlite_session_factory).get_paginated( + tenant_id=tenant.id, + app_id="app-1", + ) + + assert result.total == 2 + assert [record.id for record in result.data] == ["end-user-log", "account-log"] + by_id = {record.id: record for record in result.data} + assert by_id["account-log"].created_by_account == WorkflowAppLogAccount( + id=account.id, + name=account.name, + email=account.email, + ) + assert by_id["account-log"].created_by_end_user is None + assert by_id["end-user-log"].created_by_account is None + assert by_id["end-user-log"].created_by_end_user == WorkflowAppLogEndUser( + id=end_user.id, + type=end_user.type.value, + is_anonymous=False, + session_id=end_user.session_id, + ) + + +def test_get_paginated_preserves_missing_account_filter_error( + sqlite_session_factory: sessionmaker[Session], +) -> None: + repository = WorkflowAppLogQueryRepository(session_factory=sqlite_session_factory) + + with pytest.raises(ValueError, match=r"^Account not found: missing@example\.com$"): + repository.get_paginated( + tenant_id="tenant-1", + app_id="app-1", + created_by_account="missing@example.com", + ) + + +def test_enum_value_rejects_missing_required_value() -> None: + with pytest.raises(ValueError, match="Required enum value is missing"): + WorkflowAppLogQueryRepository._enum_value(None) + + +def test_get_paginated_projects_plugin_workflow_run_summary( + sqlite_session: Session, + sqlite_session_factory: sessionmaker[Session], +) -> None: + log = _log( + "plugin-log", + created_by="actor-1", + created_by_role=CreatorUserRole.ACCOUNT, + created_at=datetime(2026, 1, 1, tzinfo=UTC), + ) + run = _run(log.workflow_run_id, triggered_from=WorkflowRunTriggeredFrom.PLUGIN) + sqlite_session.add_all([run, log]) + sqlite_session.commit() + + result = WorkflowAppLogQueryRepository(session_factory=sqlite_session_factory).get_paginated( + tenant_id="tenant-1", + app_id="app-1", + ) + + assert result.total == 1 + summary = result.data[0].workflow_run + assert summary is not None + assert summary.id == run.id + assert summary.status == WorkflowExecutionStatus.SUCCEEDED.value + assert summary.triggered_from == WorkflowRunTriggeredFrom.PLUGIN.value + assert summary.version == run.version + assert summary.total_tokens == run.total_tokens + + +def test_get_paginated_keeps_log_when_workflow_run_is_missing( + sqlite_session: Session, + sqlite_session_factory: sessionmaker[Session], +) -> None: + log = _log( + "orphan-log", + created_by="actor-1", + created_by_role=CreatorUserRole.ACCOUNT, + created_at=datetime(2026, 1, 1, tzinfo=UTC), + ) + sqlite_session.add(log) + sqlite_session.commit() + + repository = WorkflowAppLogQueryRepository(session_factory=sqlite_session_factory) + result = repository.get_paginated(tenant_id="tenant-1", app_id="app-1") + + assert result.total == 1 + assert result.data[0].workflow_run is None + assert ( + repository.get_paginated( + tenant_id="tenant-1", + app_id="app-1", + status=WorkflowExecutionStatus.SUCCEEDED, + ).total + == 0 + ) + + +def test_get_paginated_includes_trigger_metadata_only_with_detail( + sqlite_session: Session, + sqlite_session_factory: sessionmaker[Session], +) -> None: + log = _log( + "trigger-log", + created_by="actor-1", + created_by_role=CreatorUserRole.ACCOUNT, + created_at=datetime(2026, 1, 1, tzinfo=UTC), + ) + trigger_metadata = json.dumps({"type": AppTriggerType.TRIGGER_SCHEDULE.value}) + sqlite_session.add_all( + [ + _run(log.workflow_run_id), + log, + WorkflowTriggerLog( + tenant_id="tenant-1", + app_id="app-1", + workflow_id="workflow-1", + workflow_run_id=log.workflow_run_id, + root_node_id=None, + trigger_metadata=trigger_metadata, + trigger_type=AppTriggerType.TRIGGER_SCHEDULE, + trigger_data="{}", + inputs="{}", + outputs=None, + status=WorkflowTriggerStatus.SUCCEEDED, + error=None, + queue_name="default", + celery_task_id=None, + created_by_role=CreatorUserRole.ACCOUNT, + created_by="actor-1", + retry_count=0, + ), + ] + ) + sqlite_session.commit() + + repository = WorkflowAppLogQueryRepository(session_factory=sqlite_session_factory) + + assert repository.get_paginated(tenant_id="tenant-1", app_id="app-1").data[0].details is None + assert repository.get_paginated( + tenant_id="tenant-1", + app_id="app-1", + detail=True, + ).data[0].details == {"trigger_metadata": trigger_metadata} + + +@pytest.mark.parametrize( + ("run_tenant_id", "run_app_id"), + [ + ("tenant-2", "app-1"), + ("tenant-1", "app-2"), + ], +) +def test_get_paginated_does_not_attach_workflow_run_from_another_scope( + run_tenant_id: str, + run_app_id: str, + sqlite_session: Session, + sqlite_session_factory: sessionmaker[Session], +) -> None: + log = _log( + "mismatched-log", + created_by="actor-1", + created_by_role=CreatorUserRole.ACCOUNT, + created_at=datetime(2026, 1, 1, tzinfo=UTC), + ) + sqlite_session.add_all( + [ + _run(log.workflow_run_id, tenant_id=run_tenant_id, app_id=run_app_id), + log, + ] + ) + sqlite_session.commit() + + repository = WorkflowAppLogQueryRepository(session_factory=sqlite_session_factory) + result = repository.get_paginated(tenant_id="tenant-1", app_id="app-1") + + assert result.total == 1 + assert result.data[0].workflow_run is None + assert ( + repository.get_paginated( + tenant_id="tenant-1", + app_id="app-1", + status=WorkflowExecutionStatus.SUCCEEDED, + ).total + == 0 + ) diff --git a/api/tests/unit_tests/services/test_account_access_service.py b/api/tests/unit_tests/services/test_account_access_service.py new file mode 100644 index 00000000000..7121d8cc1da --- /dev/null +++ b/api/tests/unit_tests/services/test_account_access_service.py @@ -0,0 +1,173 @@ +from __future__ import annotations + +from dataclasses import dataclass, field +from datetime import UTC, datetime + +import pytest + +from machinery.context import AccountRequestContext +from services.account_access_service import AccountAccessService +from services.account_errors import AccountNotFoundError, AccountSessionNotFoundError +from services.entities.account_access_entities import ( + AccountSessionRevocation, + AccountSessionSnapshot, + AccountWorkspaceSnapshot, +) +from services.entities.account_entities import AccountSnapshot + +NOW = datetime(2026, 8, 25, 12, tzinfo=UTC) + + +def _context(*, token_id: str | None = "token-1") -> AccountRequestContext: + return AccountRequestContext("request-1", "trace-1", "account-1", token_id) + + +def _account() -> AccountSnapshot: + return AccountSnapshot( + id="account-1", + name="Ada", + email="ada@example.com", + avatar=None, + is_password_set=False, + interface_language=None, + interface_theme=None, + timezone=None, + last_login_at=None, + last_login_ip=None, + status="active", + initialized_at=None, + created_at=NOW, + ) + + +@dataclass +class _Accounts: + account: AccountSnapshot | None = field(default_factory=_account) + + def get(self, account_id: str) -> AccountSnapshot | None: + assert account_id == "account-1" + return self.account + + +@dataclass +class _Workspaces: + items: tuple[AccountWorkspaceSnapshot, ...] = () + + def list_account_access_workspaces(self, account_id: str) -> tuple[AccountWorkspaceSnapshot, ...]: + assert account_id == "account-1" + return self.items + + +@dataclass +class _Sessions: + items: tuple[AccountSessionSnapshot, ...] = () + total: int = 0 + revocation: AccountSessionRevocation = AccountSessionRevocation(owned=True) + list_call: tuple[str, datetime, int, int] | None = None + revoke_call: tuple[str, str, datetime] | None = None + + def list_active( + self, + *, + account_id: str, + active_at: datetime, + offset: int, + limit: int, + ) -> tuple[int, tuple[AccountSessionSnapshot, ...]]: + self.list_call = (account_id, active_at, offset, limit) + return self.total, self.items + + def revoke(self, *, account_id: str, token_id: str, revoked_at: datetime) -> AccountSessionRevocation: + self.revoke_call = (account_id, token_id, revoked_at) + return self.revocation + + +@dataclass +class _TokenCache: + invalidated: list[str] = field(default_factory=list) + + def __call__(self, token_hash: str) -> None: + self.invalidated.append(token_hash) + + +def _service( + *, + accounts: _Accounts | None = None, + workspaces: _Workspaces | None = None, + sessions: _Sessions | None = None, + token_cache: _TokenCache | None = None, +) -> AccountAccessService: + return AccountAccessService( + accounts=accounts or _Accounts(), + workspaces=workspaces or _Workspaces(), + sessions=sessions or _Sessions(), + invalidate_token_cache=token_cache or _TokenCache(), + now=lambda: NOW, + ) + + +def test_get_prefers_current_workspace_as_default() -> None: + workspaces = _Workspaces( + items=( + AccountWorkspaceSnapshot("workspace-1", "First", "normal", False), + AccountWorkspaceSnapshot("workspace-2", "Current", "owner", True), + ) + ) + + snapshot = _service(workspaces=workspaces).get(_context()) + + assert snapshot.account.email == "ada@example.com" + assert snapshot.workspaces == workspaces.items + assert snapshot.default_workspace_id == "workspace-2" + + +def test_get_falls_back_to_first_workspace() -> None: + workspaces = _Workspaces( + items=( + AccountWorkspaceSnapshot("workspace-1", "First", "normal", False), + AccountWorkspaceSnapshot("workspace-2", "Second", "owner", False), + ) + ) + + assert _service(workspaces=workspaces).get(_context()).default_workspace_id == "workspace-1" + + +def test_get_raises_when_admitted_account_disappeared() -> None: + with pytest.raises(AccountNotFoundError): + _service(accounts=_Accounts(account=None)).get(_context()) + + +def test_list_sessions_delegates_database_pagination() -> None: + sessions = _Sessions(total=12) + + page = _service(sessions=sessions).list_sessions(_context(), page=3, limit=5) + + assert sessions.list_call == ("account-1", NOW, 10, 5) + assert page.page == 3 + assert page.total == 12 + assert page.has_more is False + + +def test_revoke_current_session_invalidates_live_token_cache() -> None: + sessions = _Sessions(revocation=AccountSessionRevocation(owned=True, token_hash="hash-1")) + cache = _TokenCache() + + _service(sessions=sessions, token_cache=cache).revoke_current_session(_context()) + + assert sessions.revoke_call == ("account-1", "token-1", NOW) + assert cache.invalidated == ["hash-1"] + + +def test_revoke_foreign_session_does_not_invalidate_cache() -> None: + sessions = _Sessions(revocation=AccountSessionRevocation(owned=False)) + cache = _TokenCache() + + with pytest.raises(AccountSessionNotFoundError): + _service(sessions=sessions, token_cache=cache).revoke_session(_context(), token_id="foreign") + + assert cache.invalidated == [] + + +def test_revoke_current_requires_admitted_token_id() -> None: + with pytest.raises(RuntimeError, match="did not resolve an access token"): + _service().revoke_current_session(_context(token_id=None)) diff --git a/api/tests/unit_tests/services/test_account_oauth_adapters.py b/api/tests/unit_tests/services/test_account_oauth_adapters.py new file mode 100644 index 00000000000..18d5e1068bd --- /dev/null +++ b/api/tests/unit_tests/services/test_account_oauth_adapters.py @@ -0,0 +1,318 @@ +from threading import Event +from typing import override + +import httpx +import pytest +from redis.exceptions import LockNotOwnedError +from sqlalchemy import func, select +from sqlalchemy.orm import Session, sessionmaker + +from libs.oauth import JsonObject, OAuth, OAuthUserInfo +from models.account import Account, AccountStatus, Tenant, TenantAccountJoin +from repositories import account_oauth_repository +from repositories.account_oauth_repository import ( + AccountServiceOAuthAccountRegistrationGateway, + AccountServiceOAuthWorkspaceGateway, +) +from services import account_oauth_adapters +from services.account_errors import ( + OAuthIdentityLockUnavailableError, + OAuthProviderAuthorizationError, + OAuthProviderRequestError, + OAuthWorkspaceCreationNotAllowedError, +) +from services.account_oauth_adapters import DifyOAuthProviderGateway, RedisOAuthAccountClaimLock +from services.account_service import AccountService, TenantService +from services.entities.account_oauth_entities import OAuthAccountRegistration, OAuthAuthorizationRequest, OAuthIdentity +from services.errors.workspace import WorkspacesLimitExceededError + + +class StubOAuthClient(OAuth): + def __init__(self) -> None: + super().__init__("client-id", "client-secret", "https://api.example/callback") + self.authorization_args: tuple[str | None, str | None, str | None, str | None] | None = None + self.access_codes: list[str] = [] + self.user_tokens: list[str] = [] + self.failure: Exception | None = None + + @override + def get_authorization_url( + self, + invite_token: str | None = None, + timezone: str | None = None, + language: str | None = None, + redirect_url: str | None = None, + ) -> str: + self.authorization_args = (invite_token, timezone, language, redirect_url) + return "https://provider.example/authorize" + + @override + def get_access_token(self, code: str) -> str: + self.access_codes.append(code) + if self.failure is not None: + raise self.failure + return "provider-token" + + @override + def get_user_info(self, token: str) -> OAuthUserInfo: + self.user_tokens.append(token) + return OAuthUserInfo(id="provider-user", name="User", email="user@example.com") + + @override + def get_raw_user_info(self, token: str) -> JsonObject: + raise AssertionError(token) + + @override + def _transform_user_info(self, raw_info: JsonObject) -> OAuthUserInfo: + raise AssertionError(raw_info) + + +class StubRedisLock: + def __init__(self, *, acquire_result: bool = True, reacquire_error: Exception | None = None) -> None: + self.acquire_result = acquire_result + self.reacquire_error = reacquire_error + self.acquire_calls = 0 + self.reacquire_calls = 0 + self.release_calls = 0 + self.reacquired = Event() + + def acquire(self) -> bool: + self.acquire_calls += 1 + return self.acquire_result + + def reacquire(self) -> bool: + self.reacquire_calls += 1 + self.reacquired.set() + if self.reacquire_error is not None: + raise self.reacquire_error + return True + + def release(self) -> None: + self.release_calls += 1 + + +class StubRedisClient: + def __init__(self, *locks: StubRedisLock) -> None: + self._locks = locks + self.lock_calls: list[tuple[str, float | None, float | None, bool]] = [] + + def lock( + self, + name: str, + timeout: float | None = None, + sleep: float = 0.1, + blocking: bool = True, + blocking_timeout: float | None = None, + thread_local: bool = True, + ) -> StubRedisLock: + del sleep, blocking + self.lock_calls.append((name, timeout, blocking_timeout, thread_local)) + return self._locks[len(self.lock_calls) - 1] + + +def test_account_claim_lock_uses_redis_without_exposing_identity_or_email() -> None: + locks = (StubRedisLock(), StubRedisLock()) + client = StubRedisClient(*locks) + account_claims = RedisOAuthAccountClaimLock(client=client) # type: ignore[arg-type] + + with account_claims.acquire(provider="github", open_id="provider-user", email="user@example.com"): + assert all(lock.acquire_calls == 1 for lock in locks) + + assert all(lock.release_calls == 1 for lock in locks) + assert len(client.lock_calls) == 2 + for lock_name, timeout, blocking_timeout, thread_local in client.lock_calls: + assert lock_name.startswith("oauth:account-claim:") + assert "provider-user" not in lock_name + assert "user@example.com" not in lock_name + assert timeout == 60 + assert blocking_timeout == 10 + assert thread_local is False + assert [call[0] for call in client.lock_calls] == sorted(call[0] for call in client.lock_calls) + + +def test_account_claim_lock_hashes_final_account_id() -> None: + lock = StubRedisLock() + client = StubRedisClient(lock) + account_claims = RedisOAuthAccountClaimLock(client=client) # type: ignore[arg-type] + + with account_claims.acquire_account("account-1"): + assert lock.acquire_calls == 1 + + assert lock.release_calls == 1 + assert len(client.lock_calls) == 1 + lock_name, _, _, _ = client.lock_calls[0] + assert lock_name.startswith("oauth:account-claim:") + assert "account-1" not in lock_name + + +def test_account_claim_lock_renews_both_leases_while_the_flow_is_running(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(account_oauth_adapters, "_OAUTH_ACCOUNT_CLAIM_LOCK_RENEW_INTERVAL_SECONDS", 0.01) + locks = (StubRedisLock(), StubRedisLock()) + client = StubRedisClient(*locks) + account_claims = RedisOAuthAccountClaimLock(client=client) # type: ignore[arg-type] + + with account_claims.acquire(provider="github", open_id="provider-user", email="user@example.com"): + assert all(lock.reacquired.wait(timeout=1) for lock in locks) + + assert all(lock.reacquire_calls >= 1 for lock in locks) + assert all(lock.release_calls == 1 for lock in locks) + + +def test_account_claim_lease_notifies_caller_when_heartbeat_loses_ownership( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(account_oauth_adapters, "_OAUTH_ACCOUNT_CLAIM_LOCK_RENEW_INTERVAL_SECONDS", 0.01) + lock = StubRedisLock(reacquire_error=LockNotOwnedError("lease lost")) + client = StubRedisClient(lock) + account_claims = RedisOAuthAccountClaimLock(client=client) # type: ignore[arg-type] + writes: list[str] = [] + + def use_lost_lease() -> None: + with account_claims.acquire_account("account-1") as lease: + assert lock.reacquired.wait(timeout=1) + lease.ensure_owned() + writes.append("must-not-run") + + with pytest.raises(OAuthIdentityLockUnavailableError): + use_lost_lease() + + assert writes == [] + assert lock.release_calls == 1 + + +def test_account_claim_lock_releases_partial_acquisition_on_failure() -> None: + first_lock = StubRedisLock() + failed_lock = StubRedisLock(acquire_result=False) + client = StubRedisClient(first_lock, failed_lock) + account_claims = RedisOAuthAccountClaimLock(client=client) # type: ignore[arg-type] + + with pytest.raises(OAuthIdentityLockUnavailableError): + with account_claims.acquire(provider="github", open_id="provider-user", email="user@example.com"): + raise AssertionError("lock body must not run") + + assert first_lock.release_calls == 1 + assert failed_lock.release_calls == 0 + + +def test_registration_gateway_creates_only_the_account( + sqlite_session_factory: sessionmaker[Session], + monkeypatch: pytest.MonkeyPatch, +) -> None: + calls: list[tuple[str, str, str, str | None, str | None, bool]] = [] + + def create_account( + email: str, + name: str, + interface_language: str, + password: str | None = None, + interface_theme: str = "light", + is_setup: bool | None = False, + timezone: str | None = None, + ip_address: str | None = None, + check_normalized_email: bool = False, + *, + session: Session, + ) -> Account: + del password, interface_theme, is_setup + calls.append((email, name, interface_language, timezone, ip_address, check_normalized_email)) + account = Account( + email=email, + name=name, + interface_language=interface_language, + timezone=timezone, + last_login_ip=ip_address, + ) + session.add(account) + session.flush() + return account + + def unexpected_default_workspace_join(account_id: str) -> None: + raise AssertionError(account_id) + + monkeypatch.setattr(AccountService, "create_account", create_account) + monkeypatch.setattr(account_oauth_repository, "try_join_default_workspace", unexpected_default_workspace_join) + gateway = AccountServiceOAuthAccountRegistrationGateway(session_factory=sqlite_session_factory) + + account_id = gateway.register( + OAuthAccountRegistration( + email="user@example.com", + name="User", + language="en-US", + timezone="Asia/Singapore", + ip_address="203.0.113.10", + ) + ) + + assert calls == [("user@example.com", "User", "en-US", "Asia/Singapore", "203.0.113.10", True)] + with sqlite_session_factory() as session: + account = session.get(Account, account_id) + assert account is not None + assert account.status == AccountStatus.ACTIVE + assert account.initialized_at is not None + assert session.scalar(select(func.count()).select_from(Tenant)) == 0 + assert session.scalar(select(func.count()).select_from(TenantAccountJoin)) == 0 + + +def test_workspace_gateway_maps_workspace_quota_failure( + sqlite_session_factory: sessionmaker[Session], + monkeypatch: pytest.MonkeyPatch, +) -> None: + with sqlite_session_factory.begin() as session: + account = Account(name="User", email="user@example.com") + session.add(account) + session.flush() + account_id = account.id + + def raise_workspace_limit(account: Account, *, session: Session) -> None: + assert account.id == account_id + assert session.get(Account, account_id) is account + raise WorkspacesLimitExceededError + + monkeypatch.setattr(TenantService, "create_owner_tenant", raise_workspace_limit) + gateway = AccountServiceOAuthWorkspaceGateway(session_factory=sqlite_session_factory) + + with pytest.raises(OAuthWorkspaceCreationNotAllowedError): + gateway.create_owner_workspace(account_id) + + +def test_provider_gateway_adapts_authorization_and_identity_contracts() -> None: + client = StubOAuthClient() + gateway = DifyOAuthProviderGateway(provider_name="github", client=client) + request = OAuthAuthorizationRequest( + invite_token="invite", + timezone="Asia/Shanghai", + language="zh-Hans", + redirect_url="/apps", + ) + + authorization_url = gateway.get_authorization_url(request) + identity = gateway.get_identity("authorization-code") + + assert authorization_url == "https://provider.example/authorize" + assert client.authorization_args == ("invite", "Asia/Shanghai", "zh-Hans", "/apps") + assert client.access_codes == ["authorization-code"] + assert client.user_tokens == ["provider-token"] + assert identity == OAuthIdentity("provider-user", "User", "user@example.com") + + +def test_provider_gateway_translates_transport_failure() -> None: + client = StubOAuthClient() + client.failure = httpx.ConnectError("provider unavailable") + gateway = DifyOAuthProviderGateway(provider_name="github", client=client) + + with pytest.raises(OAuthProviderRequestError) as raised: + gateway.get_identity("authorization-code") + + assert raised.value.__cause__ is client.failure + + +def test_provider_gateway_translates_provider_rejection() -> None: + client = StubOAuthClient() + client.failure = ValueError("invalid authorization code") + gateway = DifyOAuthProviderGateway(provider_name="github", client=client) + + with pytest.raises(OAuthProviderAuthorizationError) as raised: + gateway.get_identity("authorization-code") + + assert raised.value.description == "invalid authorization code" + assert raised.value.__cause__ is client.failure diff --git a/api/tests/unit_tests/services/test_account_oauth_service.py b/api/tests/unit_tests/services/test_account_oauth_service.py new file mode 100644 index 00000000000..d2231064107 --- /dev/null +++ b/api/tests/unit_tests/services/test_account_oauth_service.py @@ -0,0 +1,762 @@ +from _thread import LockType +from collections.abc import Callable, Generator +from concurrent.futures import ThreadPoolExecutor +from contextlib import contextmanager +from dataclasses import dataclass, field +from datetime import datetime +from threading import Barrier, Lock +from typing import NoReturn + +import pytest + +from services.account_errors import ( + AccountEmailDomainSuspendedError, + AccountEmailFrozenError, + InvalidOAuthInvitationError, + InvalidOAuthProviderError, + OAuthAccountBannedError, + OAuthIdentityLockUnavailableError, + OAuthInvitationAccountMismatchError, + OAuthRegistrationError, + OAuthWorkspaceCreationNotAllowedError, +) +from services.account_oauth_service import AccountOAuthService +from services.entities.account_entities import AccountSessionTokens, AccountSnapshot +from services.entities.account_oauth_entities import ( + OAuthAccountRegistration, + OAuthAuthorizationRequest, + OAuthCallbackCommand, + OAuthIdentity, + OAuthInvitation, + OAuthInvitationResult, + OAuthSignInResult, +) + +NOW = datetime(2026, 8, 24, 12, 0) + + +def _account( + *, + account_id: str = "account-1", + email: str = "user@example.com", + status: str = "active", +) -> AccountSnapshot: + return AccountSnapshot( + id=account_id, + name="User", + email=email, + avatar=None, + is_password_set=False, + interface_language="en-US", + interface_theme="light", + timezone="UTC", + last_login_at=None, + last_login_ip=None, + status=status, + initialized_at=None, + created_at=NOW, + ) + + +class FakeProvider: + def __init__(self, identity: OAuthIdentity | None = None) -> None: + self.identity = identity or OAuthIdentity(id="provider-user", name="User", email="user@example.com") + self.authorization_requests: list[OAuthAuthorizationRequest] = [] + self.codes: list[str] = [] + self.identity_hook: Callable[[], None] | None = None + + def get_authorization_url(self, request: OAuthAuthorizationRequest) -> str: + self.authorization_requests.append(request) + return "https://provider.example/authorize" + + def get_identity(self, code: str) -> OAuthIdentity: + self.codes.append(code) + if self.identity_hook is not None: + self.identity_hook() + return self.identity + + +@dataclass +class FakeAccounts: + email_account: AccountSnapshot | None = None + stored: dict[str, AccountSnapshot] = field(default_factory=dict) + get_calls: list[str] = field(default_factory=list) + email_lookups: list[str] = field(default_factory=list) + activations: list[tuple[str, datetime]] = field(default_factory=list) + + def get(self, account_id: str) -> AccountSnapshot | None: + self.get_calls.append(account_id) + return self.stored.get(account_id) + + def find_by_email(self, email: str) -> AccountSnapshot | None: + self.email_lookups.append(email) + return self.email_account + + def activate_pending(self, account_id: str, *, initialized_at: datetime) -> None: + self.activations.append((account_id, initialized_at)) + + def get_credentials(self, account_id: str) -> NoReturn: + raise AssertionError(account_id) + + def update_profile(self, account_id: str, changes: object) -> NoReturn: + raise AssertionError((account_id, changes)) + + def update_password(self, account_id: str, password: object) -> NoReturn: + raise AssertionError((account_id, password)) + + def initialize( + self, + account_id: str, + initialization: object, + *, + invitation_code: str | None, + workspace_id: str | None, + ) -> NoReturn: + raise AssertionError((account_id, initialization, invitation_code, workspace_id)) + + def email_exists(self, email: str) -> bool: + raise AssertionError(email) + + def reset_email(self, account_id: str, *, expected_old_email: str, new_email: str) -> NoReturn: + raise AssertionError((account_id, expected_old_email, new_email)) + + +@dataclass +class FakeIntegrations: + accounts: FakeAccounts + account_ids_by_identity: dict[tuple[str, str], str] = field(default_factory=dict) + identity_lookups: list[tuple[str, str]] = field(default_factory=list) + links: list[tuple[str, str, str]] = field(default_factory=list) + + def find_account_id(self, *, provider: str, open_id: str) -> str | None: + self.identity_lookups.append((provider, open_id)) + return self.account_ids_by_identity.get((provider, open_id)) + + def list_for_account(self, account_id: str) -> NoReturn: + raise AssertionError(account_id) + + def link(self, account_id: str, *, provider: str, open_id: str) -> None: + self.links.append((account_id, provider, open_id)) + self.account_ids_by_identity[(provider, open_id)] = account_id + account = self.accounts.get(account_id) or self.accounts.email_account + if account is not None: + self.accounts.email_account = account + + +@dataclass +class FakeAccountClaimLease: + lost: bool = False + checks: int = 0 + + def ensure_owned(self) -> None: + self.checks += 1 + if self.lost: + raise OAuthIdentityLockUnavailableError + + +@dataclass +class FakeAccountClaims: + claims: list[tuple[str, str, str]] = field(default_factory=list) + account_ids: list[str] = field(default_factory=list) + identity_leases: list[FakeAccountClaimLease] = field(default_factory=list) + account_leases: list[FakeAccountClaimLease] = field(default_factory=list) + lose_identity_on_acquire: bool = False + _locks: dict[str, LockType] = field(default_factory=dict, repr=False) + _registry_lock: LockType = field(default_factory=Lock, repr=False) + + @contextmanager + def acquire(self, *, provider: str, open_id: str, email: str) -> Generator[FakeAccountClaimLease, None, None]: + self.claims.append((provider, open_id, email)) + lease = FakeAccountClaimLease(lost=self.lose_identity_on_acquire) + self.identity_leases.append(lease) + with self._acquire_keys((f"email:{email}", f"identity:{provider}:{open_id}")): + yield lease + lease.ensure_owned() + + @contextmanager + def acquire_account(self, account_id: str) -> Generator[FakeAccountClaimLease, None, None]: + self.account_ids.append(account_id) + lease = FakeAccountClaimLease() + self.account_leases.append(lease) + with self._acquire_keys((f"account:{account_id}",)): + yield lease + lease.ensure_owned() + + @contextmanager + def _acquire_keys(self, keys: tuple[str, ...]) -> Generator[None, None, None]: + with self._registry_lock: + locks = [self._locks.setdefault(key, Lock()) for key in sorted(keys)] + for lock in locks: + lock.acquire() + try: + yield + finally: + for lock in reversed(locks): + lock.release() + + +@dataclass +class FakeMemberships: + workspace_ids: tuple[str, ...] = ("workspace-1",) + account_ids: list[str] = field(default_factory=list) + check_hook: Callable[[], None] | None = None + + def list_ids_for_account(self, account_id: str) -> tuple[str, ...]: + self.account_ids.append(account_id) + return self.workspace_ids + + def has_active_membership(self, account_id: str) -> bool: + self.account_ids.append(account_id) + if self.check_hook is not None: + self.check_hook() + return bool(self.workspace_ids) + + +@dataclass +class FakeInvitations: + invitation: OAuthInvitation | None = None + resolutions: list[str] = field(default_factory=list) + + def resolve(self, invite_token: str) -> OAuthInvitation | None: + self.resolutions.append(invite_token) + return self.invitation + + +@dataclass +class FakeRegistration: + account_id: str = "new-account" + registrations: list[OAuthAccountRegistration] = field(default_factory=list) + registration_hook: Callable[[], None] | None = None + + def register(self, registration: OAuthAccountRegistration) -> str: + self.registrations.append(registration) + if self.registration_hook is not None: + self.registration_hook() + return self.account_id + + +@dataclass +class FakeRuntime: + memberships: FakeMemberships + integrations: FakeIntegrations + created_accounts: list[str] = field(default_factory=list) + default_workspace_accounts: list[str] = field(default_factory=list) + workspace_operations: list[tuple[str, str]] = field(default_factory=list) + logins: list[tuple[str, str]] = field(default_factory=list) + default_workspace_id: str | None = None + workspace_creation_error: Exception | None = None + + def create_owner_workspace(self, account_id: str) -> None: + self._assert_identity_linked(account_id) + if self.workspace_creation_error is not None: + raise self.workspace_creation_error + self.created_accounts.append(account_id) + self.workspace_operations.append(("owner", account_id)) + self.memberships.workspace_ids = (*self.memberships.workspace_ids, f"owner-{account_id}") + + def try_join_default_workspace(self, account_id: str) -> None: + self._assert_identity_linked(account_id) + self.default_workspace_accounts.append(account_id) + self.workspace_operations.append(("default", account_id)) + if self.default_workspace_id is not None: + self.memberships.workspace_ids = (*self.memberships.workspace_ids, self.default_workspace_id) + + def login(self, account_id: str, *, ip_address: str) -> AccountSessionTokens: + self.logins.append((account_id, ip_address)) + return AccountSessionTokens("access", "refresh", "csrf") + + def _assert_identity_linked(self, account_id: str) -> None: + if not any(linked_account_id == account_id for linked_account_id, _, _ in self.integrations.links): + raise AssertionError(f"workspace provisioning preceded identity link for {account_id}") + + +@dataclass +class FakePolicy: + registration_allowed: bool = True + creation_allowed: bool = True + freeze_type: str | None = None + freeze_lookups: list[str] = field(default_factory=list) + + def is_registration_allowed(self) -> bool: + return self.registration_allowed + + def get_freeze_type(self, email: str) -> str | None: + self.freeze_lookups.append(email) + return self.freeze_type + + def is_creation_allowed(self) -> bool: + return self.creation_allowed + + +@dataclass +class Harness: + service: AccountOAuthService + provider: FakeProvider + providers: dict[str, FakeProvider] + accounts: FakeAccounts + integrations: FakeIntegrations + account_claims: FakeAccountClaims + memberships: FakeMemberships + invitations: FakeInvitations + registration: FakeRegistration + runtime: FakeRuntime + policy: FakePolicy + + +def _harness( + *, + identity: OAuthIdentity | None = None, + additional_identities: dict[str, OAuthIdentity] | None = None, +) -> Harness: + provider = FakeProvider(identity) + providers = {"github": provider} + providers.update( + {name: FakeProvider(additional_identity) for name, additional_identity in (additional_identities or {}).items()} + ) + accounts = FakeAccounts() + integrations = FakeIntegrations(accounts=accounts) + account_claims = FakeAccountClaims() + memberships = FakeMemberships() + invitations = FakeInvitations() + registration = FakeRegistration() + runtime = FakeRuntime(memberships=memberships, integrations=integrations) + policy = FakePolicy() + service = AccountOAuthService( + providers=providers, + accounts=accounts, + integrations=integrations, + memberships=memberships, + invitations=invitations, + account_claims=account_claims, + registration=registration, + workspaces=runtime, + sessions=runtime, + registration_policy=policy, + workspace_policy=policy, + supported_languages=("en-US", "zh-Hans"), + now=lambda: NOW, + ) + return Harness( + service=service, + provider=provider, + providers=providers, + accounts=accounts, + integrations=integrations, + account_claims=account_claims, + memberships=memberships, + invitations=invitations, + registration=registration, + runtime=runtime, + policy=policy, + ) + + +def _bind_identity( + harness: Harness, + account: AccountSnapshot, + *, + provider: str = "github", + open_id: str = "provider-user", +) -> None: + harness.accounts.stored[account.id] = account + harness.integrations.account_ids_by_identity[(provider, open_id)] = account.id + + +def _command(**overrides: object) -> OAuthCallbackCommand: + values: dict[str, object] = { + "provider": "github", + "code": "code-1", + "invite_token": None, + "timezone": None, + "language": None, + "browser_language": "en-US", + "ip_address": "203.0.113.10", + } + values.update(overrides) + return OAuthCallbackCommand(**values) # type: ignore[arg-type] + + +def test_start_authorization_delegates_to_configured_provider() -> None: + harness = _harness() + request = OAuthAuthorizationRequest(invite_token="invite", timezone="Asia/Shanghai") + + result = harness.service.start_authorization("github", request) + + assert result == "https://provider.example/authorize" + assert harness.provider.authorization_requests == [request] + + +def test_unknown_provider_is_rejected_before_any_account_work() -> None: + harness = _harness() + + with pytest.raises(InvalidOAuthProviderError): + harness.service.complete_authorization(_command(provider="unknown")) + + assert harness.integrations.identity_lookups == [] + + +def test_existing_account_login_uses_repositories_and_runtime_gateways() -> None: + harness = _harness() + _bind_identity(harness, _account()) + + result = harness.service.complete_authorization(_command()) + + assert isinstance(result, OAuthSignInResult) + assert result.oauth_new_user is False + assert harness.integrations.identity_lookups == [("github", "provider-user")] + assert harness.accounts.get_calls[0] == "account-1" + assert harness.accounts.email_lookups == [] + assert harness.integrations.links == [("account-1", "github", "provider-user")] + assert harness.runtime.created_accounts == [] + assert harness.runtime.logins == [("account-1", "203.0.113.10")] + assert harness.registration.registrations == [] + + +def test_existing_account_without_workspace_obeys_creation_policy() -> None: + harness = _harness() + harness.accounts.email_account = _account() + harness.memberships.workspace_ids = () + harness.policy.creation_allowed = False + + with pytest.raises(OAuthWorkspaceCreationNotAllowedError): + harness.service.complete_authorization(_command()) + + assert harness.integrations.links == [("account-1", "github", "provider-user")] + assert harness.runtime.created_accounts == [] + + +def test_existing_account_without_active_workspace_creates_owner_workspace() -> None: + harness = _harness() + harness.accounts.email_account = _account() + harness.memberships.workspace_ids = () + + harness.service.complete_authorization(_command()) + + assert harness.runtime.created_accounts == ["account-1"] + + +def test_new_account_registration_normalizes_email_and_prefers_state_language() -> None: + identity = OAuthIdentity(id="provider-user", name="", email="User@Example.com") + harness = _harness(identity=identity) + harness.accounts.stored["new-account"] = _account(account_id="new-account", email="user@example.com") + harness.memberships.workspace_ids = () + harness.runtime.default_workspace_id = "enterprise-default" + + result = harness.service.complete_authorization( + _command(language="zh-Hans", browser_language="en-US", timezone="Asia/Shanghai") + ) + + assert isinstance(result, OAuthSignInResult) + assert result.oauth_new_user is True + assert harness.registration.registrations == [ + OAuthAccountRegistration( + email="user@example.com", + name="Dify", + language="zh-Hans", + timezone="Asia/Shanghai", + ip_address="203.0.113.10", + ) + ] + assert harness.memberships.account_ids == ["new-account"] + assert harness.runtime.created_accounts == ["new-account"] + assert harness.runtime.default_workspace_accounts == ["new-account"] + assert harness.runtime.workspace_operations == [("owner", "new-account"), ("default", "new-account")] + assert harness.memberships.workspace_ids == ("owner-new-account", "enterprise-default") + assert harness.integrations.links == [("new-account", "github", "provider-user")] + + +def test_new_account_workspace_provisioning_obeys_the_same_policy_as_existing_accounts() -> None: + harness = _harness() + harness.accounts.stored["new-account"] = _account(account_id="new-account") + harness.memberships.workspace_ids = () + harness.policy.creation_allowed = False + + with pytest.raises(OAuthWorkspaceCreationNotAllowedError): + harness.service.complete_authorization(_command()) + + assert len(harness.registration.registrations) == 1 + assert harness.runtime.created_accounts == [] + assert harness.runtime.default_workspace_accounts == ["new-account"] + assert harness.integrations.links == [("new-account", "github", "provider-user")] + + +def test_new_account_uses_default_workspace_fallback_when_creation_is_disallowed() -> None: + harness = _harness() + harness.accounts.stored["new-account"] = _account(account_id="new-account") + harness.memberships.workspace_ids = () + harness.runtime.default_workspace_id = "enterprise-default" + harness.policy.creation_allowed = False + + result = harness.service.complete_authorization(_command()) + + assert isinstance(result, OAuthSignInResult) + assert result.oauth_new_user is True + assert harness.runtime.default_workspace_accounts == ["new-account"] + assert harness.memberships.account_ids == ["new-account", "new-account"] + assert harness.memberships.workspace_ids == ("enterprise-default",) + assert harness.runtime.created_accounts == [] + assert harness.runtime.workspace_operations == [("default", "new-account")] + + +def test_new_account_default_workspace_membership_bypasses_personal_workspace_quota() -> None: + harness = _harness() + harness.accounts.stored["new-account"] = _account(account_id="new-account") + harness.memberships.workspace_ids = () + harness.runtime.default_workspace_id = "enterprise-default" + harness.runtime.workspace_creation_error = OAuthWorkspaceCreationNotAllowedError() + + harness.service.complete_authorization(_command()) + + assert harness.runtime.default_workspace_accounts == ["new-account"] + assert harness.memberships.workspace_ids == ("enterprise-default",) + assert harness.runtime.created_accounts == [] + assert harness.runtime.workspace_operations == [("default", "new-account")] + + +def test_concurrent_callbacks_claim_identity_before_creating_account_or_workspace() -> None: + harness = _harness() + harness.accounts.stored["new-account"] = _account(account_id="new-account") + harness.memberships.workspace_ids = () + identity_resolved = Barrier(2) + + def synchronize_callbacks() -> None: + identity_resolved.wait(timeout=5) + + harness.provider.identity_hook = synchronize_callbacks + with ThreadPoolExecutor(max_workers=2) as executor: + futures = [executor.submit(harness.service.complete_authorization, _command()) for _ in range(2)] + results = [future.result(timeout=5) for future in futures] + + assert len(harness.registration.registrations) == 1 + assert harness.account_claims.claims == [ + ("github", "provider-user", "user@example.com"), + ("github", "provider-user", "user@example.com"), + ] + assert harness.integrations.links == [ + ("new-account", "github", "provider-user"), + ("new-account", "github", "provider-user"), + ] + assert harness.runtime.default_workspace_accounts == ["new-account"] + assert harness.runtime.created_accounts == ["new-account"] + assert sorted(result.oauth_new_user for result in results if isinstance(result, OAuthSignInResult)) == [False, True] + + +def test_concurrent_provider_callbacks_claim_normalized_email_before_registration() -> None: + harness = _harness( + identity=OAuthIdentity("github-user", "User", "Shared.User+github@GoogleMail.com"), + additional_identities={"google": OAuthIdentity("google-user", "User", "shareduser@gmail.COM")}, + ) + harness.accounts.stored["new-account"] = _account(account_id="new-account", email="shared@example.com") + harness.memberships.workspace_ids = () + identity_resolved = Barrier(2) + + def synchronize_callbacks() -> None: + identity_resolved.wait(timeout=5) + + harness.providers["github"].identity_hook = synchronize_callbacks + harness.providers["google"].identity_hook = synchronize_callbacks + commands = [ + _command(provider="github"), + _command(provider="google"), + ] + with ThreadPoolExecutor(max_workers=2) as executor: + futures = [executor.submit(harness.service.complete_authorization, command) for command in commands] + results = [future.result(timeout=5) for future in futures] + + assert len(harness.registration.registrations) == 1 + assert sorted(harness.account_claims.claims) == [ + ("github", "github-user", "shareduser@gmail.com"), + ("google", "google-user", "shareduser@gmail.com"), + ] + assert sorted(harness.integrations.links) == [ + ("new-account", "github", "github-user"), + ("new-account", "google", "google-user"), + ] + assert harness.runtime.default_workspace_accounts == ["new-account"] + assert harness.runtime.created_accounts == ["new-account"] + assert sorted(result.oauth_new_user for result in results if isinstance(result, OAuthSignInResult)) == [False, True] + + +def test_concurrent_provider_callbacks_for_one_account_serialize_workspace_provisioning() -> None: + harness = _harness( + identity=OAuthIdentity("github-user", "User", "github@example.com"), + additional_identities={"google": OAuthIdentity("google-user", "User", "google@example.com")}, + ) + account = _account(email="primary@example.com") + _bind_identity(harness, account, provider="github", open_id="github-user") + _bind_identity(harness, account, provider="google", open_id="google-user") + harness.memberships.workspace_ids = () + identity_resolved = Barrier(2) + + def synchronize_callbacks() -> None: + identity_resolved.wait(timeout=5) + + harness.providers["github"].identity_hook = synchronize_callbacks + harness.providers["google"].identity_hook = synchronize_callbacks + commands = [ + _command(provider="github"), + _command(provider="google"), + ] + with ThreadPoolExecutor(max_workers=2) as executor: + futures = [executor.submit(harness.service.complete_authorization, command) for command in commands] + results = [future.result(timeout=5) for future in futures] + + assert harness.registration.registrations == [] + assert sorted(harness.account_claims.claims) == [ + ("github", "github-user", "github@example.com"), + ("google", "google-user", "google@example.com"), + ] + assert harness.account_claims.account_ids == ["account-1", "account-1"] + assert harness.runtime.created_accounts == ["account-1"] + assert all(isinstance(result, OAuthSignInResult) and not result.oauth_new_user for result in results) + + +def test_lost_identity_claim_stops_before_registration() -> None: + harness = _harness() + harness.account_claims.lose_identity_on_acquire = True + + with pytest.raises(OAuthIdentityLockUnavailableError): + harness.service.complete_authorization(_command()) + + assert harness.registration.registrations == [] + assert harness.integrations.links == [] + assert harness.runtime.created_accounts == [] + + +def test_identity_claim_lost_during_registration_stops_follow_up_writes() -> None: + harness = _harness() + harness.accounts.stored["new-account"] = _account(account_id="new-account") + + def lose_identity_claim() -> None: + harness.account_claims.identity_leases[0].lost = True + + harness.registration.registration_hook = lose_identity_claim + + with pytest.raises(OAuthIdentityLockUnavailableError): + harness.service.complete_authorization(_command()) + + assert len(harness.registration.registrations) == 1 + assert harness.integrations.links == [] + assert harness.runtime.default_workspace_accounts == [] + assert harness.runtime.created_accounts == [] + + +def test_lost_account_claim_stops_before_workspace_creation() -> None: + harness = _harness() + _bind_identity(harness, _account()) + harness.memberships.workspace_ids = () + + def lose_account_claim() -> None: + harness.account_claims.account_leases[0].lost = True + + harness.memberships.check_hook = lose_account_claim + + with pytest.raises(OAuthIdentityLockUnavailableError): + harness.service.complete_authorization(_command()) + + assert harness.runtime.created_accounts == [] + assert harness.runtime.logins == [] + + +@pytest.mark.parametrize( + ("freeze_type", "expected_error"), + [ + ("email_domain_suspended", AccountEmailDomainSuspendedError), + ("freeze", AccountEmailFrozenError), + (None, OAuthRegistrationError), + ], +) +def test_disabled_registration_applies_account_policy( + freeze_type: str | None, + expected_error: type[Exception], +) -> None: + harness = _harness() + harness.policy.registration_allowed = False + harness.policy.freeze_type = freeze_type + + with pytest.raises(expected_error): + harness.service.complete_authorization(_command()) + + assert harness.policy.freeze_lookups == ["user@example.com"] + assert harness.registration.registrations == [] + + +def test_pending_account_is_activated_through_repository() -> None: + harness = _harness() + _bind_identity(harness, _account(status="pending")) + + harness.service.complete_authorization(_command()) + + assert harness.accounts.activations == [("account-1", NOW)] + + +def test_pending_account_is_not_activated_when_workspace_creation_is_disallowed() -> None: + harness = _harness() + _bind_identity(harness, _account(status="pending")) + harness.memberships.workspace_ids = () + harness.policy.creation_allowed = False + + with pytest.raises(OAuthWorkspaceCreationNotAllowedError): + harness.service.complete_authorization(_command()) + + assert harness.accounts.activations == [] + assert harness.runtime.logins == [] + + +def test_pending_account_is_not_activated_when_workspace_creation_fails() -> None: + harness = _harness() + _bind_identity(harness, _account(status="pending")) + harness.memberships.workspace_ids = () + harness.runtime.workspace_creation_error = RuntimeError("workspace quota exceeded") + + with pytest.raises(RuntimeError, match="workspace quota exceeded"): + harness.service.complete_authorization(_command()) + + assert harness.accounts.activations == [] + assert harness.runtime.logins == [] + + +def test_valid_invitation_links_and_logs_in_invited_account() -> None: + harness = _harness(identity=OAuthIdentity("provider-user", "User", "Invitee@Example.com")) + harness.invitations.invitation = OAuthInvitation("invited-account", "invitee@example.com", "active") + + result = harness.service.complete_authorization(_command(invite_token="invite-token")) + + assert isinstance(result, OAuthInvitationResult) + assert result.invite_token == "invite-token" + assert harness.integrations.links == [("invited-account", "github", "provider-user")] + assert harness.runtime.logins == [("invited-account", "203.0.113.10")] + assert harness.integrations.identity_lookups == [] + assert harness.invitations.resolutions == ["invite-token"] + + +def test_resolvable_invitation_requires_matching_email() -> None: + harness = _harness() + harness.invitations.invitation = OAuthInvitation("invited-account", "other@example.com", "active") + + with pytest.raises(OAuthInvitationAccountMismatchError) as raised: + harness.service.complete_authorization(_command(invite_token="invite-token")) + + assert raised.value.invite_token == "invite-token" + assert harness.integrations.links == [] + + +def test_stale_invitation_is_rejected() -> None: + harness = _harness() + + with pytest.raises(InvalidOAuthInvitationError): + harness.service.complete_authorization(_command(invite_token="invite-token")) + + assert harness.invitations.resolutions == ["invite-token"] + assert harness.integrations.identity_lookups == [] + assert harness.registration.registrations == [] + + +def test_banned_account_is_rejected_before_writes() -> None: + harness = _harness() + _bind_identity(harness, _account(status="banned")) + + with pytest.raises(OAuthAccountBannedError): + harness.service.complete_authorization(_command()) + + assert harness.integrations.links == [] diff --git a/api/tests/unit_tests/services/test_account_service.py b/api/tests/unit_tests/services/test_account_service.py index 0c90e62ee2a..5f049c1609a 100644 --- a/api/tests/unit_tests/services/test_account_service.py +++ b/api/tests/unit_tests/services/test_account_service.py @@ -2032,59 +2032,6 @@ class TestRegisterService: mock_join_default_workspace.assert_called_once_with(mock_account.id) - def test_register_with_oauth( - self, sqlite_session: Session, mock_external_service_dependencies: _MockDependencies - ) -> None: - """Test account registration with OAuth integration.""" - # Setup mocks - mock_external_service_dependencies["feature_service"].is_registration_allowed.return_value = True - mock_external_service_dependencies["feature_service"].is_workspace_creation_allowed.return_value = True - mock_external_service_dependencies[ - "feature_service" - ].get_system_features.return_value.license.workspaces.is_available.return_value = True - mock_external_service_dependencies["billing_service"].is_email_in_freeze.return_value = False - - # Mock AccountService.create_account and link_account_integrate - mock_account = TestAccountAssociatedDataFactory.create_account_mock() - with ( - patch("services.account_service.AccountService.create_account") as mock_create_account, - patch("services.account_service.AccountService.link_account_integrate") as mock_link_account, - ): - mock_create_account.return_value = mock_account - - # Mock TenantService methods - with ( - patch("services.account_service.TenantService.create_tenant") as mock_create_tenant, - patch("services.account_service.TenantService.create_tenant_member") as mock_create_member, - patch("services.account_service.tenant_was_created") as mock_event, - ): - mock_tenant = Tenant(name="Test User's Workspace") - sqlite_session.add(mock_tenant) - sqlite_session.flush() - mock_create_tenant.return_value = mock_tenant - mock_create_member.side_effect = lambda tenant, account, session, role: session.add( - TenantAccountJoin( - tenant_id=tenant.id, - account_id=account.id, - role=TenantAccountRole(role), - ) - ) - - # Execute test - result = RegisterService.register( - email="test@example.com", - name="Test User", - password=None, - open_id="oauth123", - provider="google", - language="en-US", - session=sqlite_session, - ) - - # Verify results - assert result == mock_account - mock_link_account.assert_called_once_with("google", "oauth123", mock_account, session=sqlite_session) - def test_register_with_pending_status( self, sqlite_session: Session, mock_external_service_dependencies: _MockDependencies ) -> None: @@ -2698,30 +2645,6 @@ class TestRegisterService: assert stored_data["role"] == "admin" assert stored_data["requires_setup"] is True - def test_is_valid_invite_token_valid(self, mock_redis_dependencies: MagicMock) -> None: - """Test checking valid invite token.""" - # Setup mock - mock_redis_dependencies.get.return_value = b'{"test": "data"}' - - # Execute test - result = RegisterService.is_valid_invite_token("valid-token") - - # Verify results - assert result is True - mock_redis_dependencies.get.assert_called_once_with("member_invite:token:valid-token") - - def test_is_valid_invite_token_invalid(self, mock_redis_dependencies: MagicMock) -> None: - """Test checking invalid invite token.""" - # Setup mock - mock_redis_dependencies.get.return_value = None - - # Execute test - result = RegisterService.is_valid_invite_token("invalid-token") - - # Verify results - assert result is False - mock_redis_dependencies.get.assert_called_once_with("member_invite:token:invalid-token") - def test_revoke_token_with_workspace_and_email(self, mock_redis_dependencies: MagicMock) -> None: """Test revoking token with workspace ID and email.""" # Execute test @@ -3012,22 +2935,6 @@ class TestSessionInjectedGetters: def test_account_belongs_to_tenant_false_when_no_join(self, sqlite_session: Session) -> None: assert TenantService.account_belongs_to_tenant("user-1", "tenant-1", session=sqlite_session) is False - def test_get_account_memberships_returns_join_tenant_pairs(self, sqlite_session: Session) -> None: - """Returns every ``(TenantAccountJoin, Tenant)`` pair for an account.""" - tenant = Tenant(name="Joined Workspace") - other_tenant = Tenant(name="Other Workspace") - sqlite_session.add_all([tenant, other_tenant]) - sqlite_session.flush() - join = self._add_tenant_account_join(sqlite_session, tenant, "user-123", TenantAccountRole.NORMAL, current=True) - self._add_tenant_account_join(sqlite_session, other_tenant, "other-user", TenantAccountRole.NORMAL) - sqlite_session.commit() - - out = TenantService.get_account_memberships("user-123", session=sqlite_session) - - assert len(out) == 1 - assert out[0][0] is join - assert out[0][1] is tenant - def test_get_workspaces_for_account_uses_session_execute(self, sqlite_session: Session) -> None: """The list endpoint orders by ``Tenant.created_at``; the helper returns ``(Tenant, TenantAccountJoin)`` rows in that order. diff --git a/api/tests/unit_tests/services/test_dataset_service_segment.py b/api/tests/unit_tests/services/test_dataset_service_segment.py index 5598361f977..1d50e5a487a 100644 --- a/api/tests/unit_tests/services/test_dataset_service_segment.py +++ b/api/tests/unit_tests/services/test_dataset_service_segment.py @@ -677,6 +677,46 @@ class TestSegmentServiceMutations: assert result is refreshed_segment assert segment.keywords == ["new"] vector_service.update_segment_vector.assert_called_once_with(["new"], segment, dataset, session=session) + vector_service.update_multimodel_vector.assert_not_called() + + def test_update_segment_omits_attachment_ids_leaves_existing_bindings_unchanged(self, account_context): + session = MagicMock() + segment = _make_segment(content="same content") + document = _make_document(doc_form=IndexStructureType.PARAGRAPH_INDEX, word_count=20) + dataset = _make_dataset() + refreshed_segment = SimpleNamespace(id=segment.id) + args = SegmentUpdateArgs(content="same content") + + with ( + patch("services.dataset_service.redis_client") as mock_redis, + patch("services.dataset_service.VectorService") as vector_service, + ): + mock_redis.get.return_value = None + session.get.return_value = refreshed_segment + + result = SegmentService.update_segment(args, segment, document, dataset, session) + + assert result is refreshed_segment + vector_service.update_multimodel_vector.assert_not_called() + + def test_update_segment_explicit_empty_attachment_ids_clears_bindings(self, account_context): + session = MagicMock() + segment = _make_segment(content="same content") + document = _make_document(doc_form=IndexStructureType.PARAGRAPH_INDEX, word_count=20) + dataset = _make_dataset() + refreshed_segment = SimpleNamespace(id=segment.id) + args = SegmentUpdateArgs(content="same content", attachment_ids=[]) + + with ( + patch("services.dataset_service.redis_client") as mock_redis, + patch("services.dataset_service.VectorService") as vector_service, + ): + mock_redis.get.return_value = None + session.get.return_value = refreshed_segment + + result = SegmentService.update_segment(args, segment, document, dataset, session) + + assert result is refreshed_segment vector_service.update_multimodel_vector.assert_called_once_with(segment, [], dataset, session=session) def test_update_segment_regenerates_child_chunks_and_updates_manual_summary(self, account_context): @@ -724,7 +764,7 @@ class TestSegmentServiceMutations: session=session, ) update_summary.assert_called_once_with(segment, dataset, "new summary", session=session) - vector_service.update_multimodel_vector.assert_called_once_with(segment, [], dataset, session=session) + vector_service.update_multimodel_vector.assert_not_called() def test_update_segment_auto_regenerates_summary_after_content_change(self, account_context): session = MagicMock() @@ -763,7 +803,7 @@ class TestSegmentServiceMutations: assert document.word_count == 18 vector_service.update_segment_vector.assert_called_once_with(["kw-1"], segment, dataset, session=session) generate_summary.assert_called_once_with(segment, dataset, {"enable": True}, session=session) - vector_service.update_multimodel_vector.assert_called_once_with(segment, [], dataset, session=session) + vector_service.update_multimodel_vector.assert_not_called() def test_update_segment_regenerates_summary_when_manual_summary_is_unchanged(self, account_context): session = MagicMock() @@ -799,7 +839,7 @@ class TestSegmentServiceMutations: assert result is refreshed_segment generate_summary.assert_called_once_with(segment, dataset, {"enable": True}, session=session) update_summary.assert_not_called() - vector_service.update_multimodel_vector.assert_called_once_with(segment, [], dataset, session=session) + vector_service.update_multimodel_vector.assert_not_called() def test_delete_segment_removes_index_and_updates_document_word_count(self): session = MagicMock() @@ -1012,7 +1052,7 @@ class TestSegmentServiceAdditionalRegenerationBranches: assert segment.word_count == len("question") + len("new answer") assert document.word_count == 20 + (len("question") + len("new answer") - 8) vector_service.update_segment_vector.assert_not_called() - vector_service.update_multimodel_vector.assert_called_once_with(segment, [], dataset, session=session) + vector_service.update_multimodel_vector.assert_not_called() def test_update_segment_content_change_uses_answer_when_counting_tokens_for_qa_segments(self, account_context): session = MagicMock() @@ -1049,7 +1089,7 @@ class TestSegmentServiceAdditionalRegenerationBranches: assert segment.tokens == 21 assert segment.word_count == len("new question") + len("new answer") vector_service.update_segment_vector.assert_called_once_with(["kw-1"], segment, dataset, session=session) - vector_service.update_multimodel_vector.assert_called_once_with(segment, [], dataset, session=session) + vector_service.update_multimodel_vector.assert_not_called() def test_update_segment_content_change_parent_child_uses_default_embedding_and_ignores_summary_failures( self, account_context @@ -1107,7 +1147,7 @@ class TestSegmentServiceAdditionalRegenerationBranches: session=session, ) update_summary.assert_called_once_with(segment, dataset, "new summary", session=session) - vector_service.update_multimodel_vector.assert_called_once_with(segment, [], dataset, session=session) + vector_service.update_multimodel_vector.assert_not_called() def test_update_segment_same_content_parent_child_marks_segment_error_for_non_high_quality_dataset( self, account_context diff --git a/api/tests/unit_tests/services/test_file_grant_gateways.py b/api/tests/unit_tests/services/test_file_grant_gateways.py new file mode 100644 index 00000000000..2b909f2ca03 --- /dev/null +++ b/api/tests/unit_tests/services/test_file_grant_gateways.py @@ -0,0 +1,90 @@ +from collections.abc import Callable +from unittest.mock import patch + +import httpx +import pytest + +from services.errors.file import FileTooLargeError +from services.file_grant_gateways import FileGrantRemoteFileGateway + + +def test_remote_file_gateway_bounds_a_get_without_content_length( + config_overrides: Callable[..., None], +) -> None: + config_overrides(UPLOAD_FILE_SIZE_LIMIT=1) + url = "https://example.com/report.pdf" + head = httpx.Response(200, request=httpx.Request("HEAD", url)) + download = httpx.Response( + 200, + content=b"0" * (1024 * 1024 + 1), + request=httpx.Request("GET", url), + ) + + with patch("services.file_grant_gateways.remote_fetcher.make_request", side_effect=[head, download]) as request: + with pytest.raises(FileTooLargeError): + FileGrantRemoteFileGateway().fetch(url) + + assert request.call_args_list[1].kwargs["stream_response"] is True + assert download.is_closed + + +def test_remote_file_gateway_uses_the_actual_body_size_when_content_length_is_incorrect( + config_overrides: Callable[..., None], +) -> None: + config_overrides(UPLOAD_FILE_SIZE_LIMIT=1) + url = "https://example.com/report.pdf" + head = httpx.Response( + 200, + headers={"Content-Length": "1"}, + request=httpx.Request("HEAD", url), + ) + download = httpx.Response( + 200, + headers={"Content-Length": "1"}, + content=b"0" * (1024 * 1024 + 1), + request=httpx.Request("GET", url), + ) + + with patch("services.file_grant_gateways.remote_fetcher.make_request", side_effect=[head, download]): + with pytest.raises(FileTooLargeError): + FileGrantRemoteFileGateway().fetch(url) + + assert download.is_closed + + +def test_remote_file_gateway_rejects_encoded_content_that_cannot_be_safely_bounded( + config_overrides: Callable[..., None], +) -> None: + config_overrides(UPLOAD_FILE_SIZE_LIMIT=1) + url = "https://example.com/report.pdf" + head = httpx.Response(200, request=httpx.Request("HEAD", url)) + download = httpx.Response( + 200, + headers={"Content-Encoding": "gzip"}, + stream=httpx.ByteStream(b"compressed"), + request=httpx.Request("GET", url), + ) + + with patch("services.file_grant_gateways.remote_fetcher.make_request", side_effect=[head, download]): + assert FileGrantRemoteFileGateway().fetch(url) is None + + assert download.is_closed + + +def test_remote_file_gateway_returns_bounded_content(config_overrides: Callable[..., None]) -> None: + config_overrides(UPLOAD_FILE_SIZE_LIMIT=1) + url = "https://example.com/report.pdf" + head = httpx.Response( + 200, + headers={"Content-Length": "9", "Content-Type": "application/pdf"}, + request=httpx.Request("HEAD", url), + ) + download = httpx.Response(200, content=b"pdf-bytes", request=httpx.Request("GET", url)) + + with patch("services.file_grant_gateways.remote_fetcher.make_request", side_effect=[head, download]): + file = FileGrantRemoteFileGateway().fetch(url) + + assert file is not None + assert file.filename == "report.pdf" + assert file.mimetype == "application/pdf" + assert file.content == b"pdf-bytes" diff --git a/api/tests/unit_tests/services/test_file_grant_service.py b/api/tests/unit_tests/services/test_file_grant_service.py new file mode 100644 index 00000000000..741eb9471d4 --- /dev/null +++ b/api/tests/unit_tests/services/test_file_grant_service.py @@ -0,0 +1,147 @@ +from io import BytesIO +from unittest.mock import MagicMock + +import pytest + +from services.entities.file_grant_entities import ( + FileGrantContext, + FileGrantLimits, + FileGrantMintRequest, + FileGrantScope, + FileKind, + FileRef, + ResolvedFile, +) +from services.errors.file_grant import EndUserNotFoundError, GrantTtlTooLongError +from services.file_grant_service import MAX_SESSION_GRANT_TTL_SECONDS, FileGrantService + + +def _service() -> tuple[FileGrantService, MagicMock, MagicMock, MagicMock, MagicMock]: + repository = MagicMock() + repository.get_or_create_subject.return_value = "end-user-1" + repository.subject_exists.return_value = True + repository.resolve_owned_files.return_value = list[ResolvedFile | None]() + files = MagicMock() + tokens = MagicMock() + tokens.issue_grant.return_value = ("grant", 1600) + remote_files = MagicMock() + service = FileGrantService( + repository=repository, + files=files, + tokens=tokens, + remote_files=remote_files, + limits=FileGrantLimits(15, 10, 50, 100, 10, 5), + now=lambda: 1000, + ) + return service, repository, files, tokens, remote_files + + +def _mint_request( + *, + ttl_seconds: int = 600, + file_refs: tuple[FileRef, ...] = (), + optional_file_refs: tuple[FileRef, ...] = (), +) -> FileGrantMintRequest: + return FileGrantMintRequest( + tenant_id="tenant-1", + app_id="app-1", + subject="subject-1", + is_anonymous=True, + scopes=(FileGrantScope.UPLOAD,), + ttl_seconds=ttl_seconds, + file_refs=file_refs, + optional_file_refs=optional_file_refs, + run_deadline=None, + ) + + +def test_mint_rejects_an_invalid_ttl_before_persistence() -> None: + service, repository, _files, tokens, _remote_files = _service() + + with pytest.raises(GrantTtlTooLongError): + service.mint(_mint_request(ttl_seconds=MAX_SESSION_GRANT_TTL_SECONDS + 1)) + + repository.get_or_create_subject.assert_not_called() + tokens.issue_grant.assert_not_called() + + +def test_mint_orchestrates_identity_resolution_and_token_issuance() -> None: + service, repository, _files, tokens, _remote_files = _service() + + result = service.mint(_mint_request()) + + assert result.grant == "grant" + repository.get_or_create_subject.assert_called_once() + tokens.issue_grant.assert_called_once_with( + context=FileGrantContext("tenant-1", "app-1", "end-user-1"), + scopes=(FileGrantScope.UPLOAD,), + ttl_seconds=600, + ) + + +def test_mint_resolves_required_and_optional_files_in_one_batch() -> None: + service, repository, _files, tokens, _remote_files = _service() + required_ref = FileRef(id="upload-1", kind=FileKind.UPLOAD) + optional_ref = FileRef(id="tool-1", kind=FileKind.TOOL) + required_file = ResolvedFile("upload-1", FileKind.UPLOAD, "report.pdf", 10, "pdf", "application/pdf") + optional_file = ResolvedFile("tool-1", FileKind.TOOL, "chart.png", 20, "png", "image/png") + repository.resolve_owned_files.return_value = [required_file, optional_file] + tokens.issue_content_urls.return_value = ("https://files/tool-1", "http://files/tool-1") + + result = service.mint( + _mint_request( + file_refs=(required_ref,), + optional_file_refs=(optional_ref,), + ) + ) + + repository.resolve_owned_files.assert_called_once_with( + context=FileGrantContext("tenant-1", "app-1", "end-user-1"), + refs=(required_ref, optional_ref), + ) + assert result.files == (required_file,) + assert result.optional_files[0] is not None + assert result.optional_files[0].file == optional_file + + +def test_store_produced_rejects_a_deleted_subject_before_reading_the_file() -> None: + service, repository, files, _tokens, _remote_files = _service() + repository.subject_exists.return_value = False + stream = BytesIO(b"produced content") + + with pytest.raises(EndUserNotFoundError): + service.store_produced( + context=FileGrantContext("tenant-1", "app-1", "deleted-user"), + filename="result.txt", + stream=stream, + mimetype="text/plain", + ) + + assert stream.tell() == 0 + files.store_produced.assert_not_called() + + +def test_store_remote_upload_rejects_a_deleted_subject_before_fetching() -> None: + service, repository, _files, _tokens, remote_files = _service() + repository.subject_exists.return_value = False + + with pytest.raises(EndUserNotFoundError): + service.store_remote_upload( + context=FileGrantContext("tenant-1", "app-1", "deleted-user"), + url="https://example.com/report.pdf", + ) + + remote_files.fetch.assert_not_called() + + +def test_resolve_rejects_a_deleted_subject_before_querying_files() -> None: + service, repository, _files, _tokens, _remote_files = _service() + repository.subject_exists.return_value = False + + with pytest.raises(EndUserNotFoundError): + service.resolve_files( + context=FileGrantContext("tenant-1", "app-1", "deleted-user"), + refs=(), + ) + + repository.resolve_owned_files.assert_not_called() diff --git a/api/tests/unit_tests/services/test_file_service.py b/api/tests/unit_tests/services/test_file_service.py index 49367183d9c..470235cf610 100644 --- a/api/tests/unit_tests/services/test_file_service.py +++ b/api/tests/unit_tests/services/test_file_service.py @@ -258,6 +258,20 @@ class TestFileService: is False ) + def test_file_size_limit(self, config_overrides: Callable[..., None]): + config_overrides( + UPLOAD_IMAGE_FILE_SIZE_LIMIT=10, + UPLOAD_VIDEO_FILE_SIZE_LIMIT=20, + UPLOAD_AUDIO_FILE_SIZE_LIMIT=30, + UPLOAD_FILE_SIZE_LIMIT=5, + ) + + assert FileService.file_size_limit(extension="jpg") == 10 * 1024 * 1024 + assert FileService.file_size_limit(extension="mp4") == 20 * 1024 * 1024 + assert FileService.file_size_limit(extension="mp3") == 30 * 1024 * 1024 + assert FileService.file_size_limit(extension="txt") == 5 * 1024 * 1024 + assert FileService.file_size_limit(extension="txt", default_file_size_limit=7) == 7 * 1024 * 1024 + def test_get_file_base64_success(self, file_service: FileService, db_session: Session): self._persist_upload_file(db_session, key="test_key") diff --git a/api/tests/unit_tests/services/test_notification_service.py b/api/tests/unit_tests/services/test_notification_service.py index 3be7f08a6f7..18b3dfef93f 100644 --- a/api/tests/unit_tests/services/test_notification_service.py +++ b/api/tests/unit_tests/services/test_notification_service.py @@ -1,11 +1,4 @@ -from datetime import datetime -from unittest.mock import Mock - -import pytest - from machinery.context import RequestContext -from services.account_ports import AccountRepository -from services.entities.account_entities import AccountSnapshot from services.entities.notification_entities import ( AccountNotification, AccountNotificationBatch, @@ -39,30 +32,6 @@ class NotificationGatewayStub: self.dismissals.append((notification_id, account_id)) -def _account(language: str | None = "zh-Hans") -> AccountSnapshot: - return AccountSnapshot( - id="account-1", - name="Account", - email="account@example.com", - avatar=None, - is_password_set=False, - interface_language=language, - interface_theme="light", - timezone="UTC", - last_login_at=None, - last_login_ip=None, - status="active", - initialized_at=None, - created_at=datetime(2026, 1, 1), - ) - - -def _accounts(account: AccountSnapshot | None) -> Mock: - accounts = Mock(spec=AccountRepository) - accounts.get.return_value = account - return accounts - - def _notification(contents: dict[str, NotificationContent]) -> AccountNotification: return AccountNotification( notification_id="notification-1", @@ -71,19 +40,18 @@ def _notification(contents: dict[str, NotificationContent]) -> AccountNotificati ) -def test_get_active_localizes_notification_for_account_language() -> None: +def test_get_active_localizes_notification_for_requested_language() -> None: chinese = NotificationContent("zh-Hans", "标题", "副标题", "正文", "zh.png") english = NotificationContent("en-US", "Title", "Subtitle", "Body", "en.png") gateway = NotificationGatewayStub( AccountNotificationBatch(True, (_notification({"zh-Hans": chinese, "en-US": english}),)) ) - service = NotificationService(accounts=_accounts(_account()), notifications=gateway) + service = NotificationService(notifications=gateway) - result = service.get_active(_context()) + result = service.get_active(_context(), "zh-Hans") - assert result == NotificationResult( - should_show=True, - notifications=(NotificationItem("notification-1", "once", "zh-Hans", "标题", "副标题", "正文", "zh.png"),), + assert result.notifications == ( + NotificationItem("notification-1", "once", "zh-Hans", "标题", "副标题", "正文", "zh.png"), ) assert gateway.get_account_ids == ["account-1"] @@ -91,47 +59,48 @@ def test_get_active_localizes_notification_for_account_language() -> None: def test_get_active_falls_back_to_english() -> None: english = NotificationContent("en-US", "Title", "Subtitle", "Body", "en.png") gateway = NotificationGatewayStub(AccountNotificationBatch(True, (_notification({"en-US": english}),))) - service = NotificationService(accounts=_accounts(_account("fr-FR")), notifications=gateway) + service = NotificationService(notifications=gateway) - result = service.get_active(_context()) + result = service.get_active(_context(), "fr-FR") assert result.notifications[0].lang == "en-US" assert result.notifications[0].title == "Title" -def test_get_active_skips_account_query_when_gateway_says_not_to_show() -> None: - accounts = _accounts(None) - service = NotificationService( - accounts=accounts, - notifications=NotificationGatewayStub(AccountNotificationBatch(False, ())), +def test_get_active_falls_back_to_english_for_unsupported_language() -> None: + unsupported = NotificationContent("xx-YY", "Unknown", "Unknown", "Unknown", "unknown.png") + english = NotificationContent("en-US", "Title", "Subtitle", "Body", "en.png") + gateway = NotificationGatewayStub( + AccountNotificationBatch(True, (_notification({"xx-YY": unsupported, "en-US": english}),)) ) + service = NotificationService(notifications=gateway) - result = service.get_active(_context()) + result = service.get_active(_context(), "xx-YY") + + assert result.notifications[0].lang == "en-US" + assert result.notifications[0].title == "Title" + + +def test_get_active_returns_empty_when_gateway_says_not_to_show() -> None: + service = NotificationService(notifications=NotificationGatewayStub(AccountNotificationBatch(False, ()))) + + result = service.get_active(_context(), "zh-Hans") assert result == NotificationResult(False, ()) - accounts.get.assert_not_called() def test_get_active_uses_empty_content_when_notification_has_no_translations() -> None: gateway = NotificationGatewayStub(AccountNotificationBatch(True, (_notification({}),))) - service = NotificationService(accounts=_accounts(_account(None)), notifications=gateway) + service = NotificationService(notifications=gateway) - result = service.get_active(_context()) + result = service.get_active(_context(), "") assert result.notifications == (NotificationItem("notification-1", "once", "en-US", "", "", "", ""),) -def test_get_active_rejects_unknown_admitted_account() -> None: - gateway = NotificationGatewayStub(AccountNotificationBatch(True, (_notification({}),))) - service = NotificationService(accounts=_accounts(None), notifications=gateway) - - with pytest.raises(RuntimeError, match="unknown account"): - service.get_active(_context()) - - def test_dismiss_delegates_identifiers_to_gateway() -> None: gateway = NotificationGatewayStub(AccountNotificationBatch(False, ())) - service = NotificationService(accounts=_accounts(_account()), notifications=gateway) + service = NotificationService(notifications=gateway) service.dismiss(_context(), "notification-1") diff --git a/api/tests/unit_tests/services/test_oauth_device_flow.py b/api/tests/unit_tests/services/test_oauth_device_flow.py deleted file mode 100644 index 35c5a691672..00000000000 --- a/api/tests/unit_tests/services/test_oauth_device_flow.py +++ /dev/null @@ -1,219 +0,0 @@ -from __future__ import annotations - -import uuid -from datetime import UTC, datetime, timedelta -from unittest.mock import MagicMock - -import pytest -from sqlalchemy.orm import Session - -from libs.oauth_bearer import TOKEN_CACHE_KEY_FMT, AuthContext, SubjectType, TokenType -from models.oauth import OAuthAccessToken -from services.oauth_device_flow import ( - list_active_sessions, - revoke_oauth_token, - subject_match_clauses, - token_belongs_to_subject, -) - -ACCOUNT_ID = uuid.UUID("11111111-1111-1111-1111-111111111111") -OTHER_ACCOUNT_ID = uuid.UUID("22222222-2222-2222-2222-222222222222") -TOKEN_ID = uuid.UUID("33333333-3333-3333-3333-333333333333") -OTHER_TOKEN_ID = uuid.UUID("44444444-4444-4444-4444-444444444444") - - -def _token( - *, - token_id: uuid.UUID = TOKEN_ID, - account_id: uuid.UUID | None = ACCOUNT_ID, - subject_email: str = "user@example.com", - subject_issuer: str = "dify:account", - token_hash: str | None = "live-hash", - expires_at: datetime | None = None, - revoked_at: datetime | None = None, - created_at: datetime | None = None, -) -> OAuthAccessToken: - token = OAuthAccessToken( - subject_email=subject_email, - subject_issuer=subject_issuer, - account_id=str(account_id) if account_id is not None else None, - client_id="difyctl", - device_label="test-device", - prefix="dfoa_" if account_id is not None else "dfoe_", - token_hash=token_hash, - expires_at=expires_at or datetime.now(UTC) + timedelta(days=1), - revoked_at=revoked_at, - ) - token.id = str(token_id) - if created_at is not None: - token.created_at = created_at - return token - - -def _account_ctx(*, account_id: uuid.UUID = ACCOUNT_ID) -> AuthContext: - return AuthContext( - subject_type=SubjectType.ACCOUNT, - subject_email="user@example.com", - subject_issuer="dify:account", - account_id=account_id, - client_id="difyctl", - scopes=frozenset({"full"}), - token_id=uuid.uuid4(), - token_type=TokenType.OAUTH_ACCOUNT, - expires_at=None, - token_hash="h1", - verified_tenants={}, - ) - - -def _sso_ctx() -> AuthContext: - return AuthContext( - subject_type=SubjectType.EXTERNAL_SSO, - subject_email="sso@partner.com", - subject_issuer="https://idp.partner.com", - account_id=None, - client_id="difyctl", - scopes=frozenset({"apps:run"}), - token_id=uuid.uuid4(), - token_type=TokenType.OAUTH_EXTERNAL_SSO, - expires_at=None, - token_hash="h1", - verified_tenants={}, - ) - - -# --------------------------------------------------------------------------- -# subject_match_clauses -# --------------------------------------------------------------------------- - - -def test_subject_match_clauses_account_matches_only_account_id(): - clauses = subject_match_clauses(_account_ctx()) - assert len(clauses) == 1 - assert "account_id" in str(clauses[0]) - - -def test_subject_match_clauses_external_sso_requires_null_account_id(): - """External SSO must additionally require ``account_id IS NULL`` so a - same-email account-flow row from a federated tenant cannot be - enumerated/revoked through an SSO bearer. - """ - clauses = subject_match_clauses(_sso_ctx()) - assert len(clauses) == 3 - rendered = " ".join(str(c) for c in clauses) - assert "subject_email" in rendered - assert "subject_issuer" in rendered - assert "account_id IS NULL" in rendered - - -# --------------------------------------------------------------------------- -# revoke_oauth_token -# --------------------------------------------------------------------------- - - -@pytest.mark.parametrize("sqlite_session", [(OAuthAccessToken,)], indirect=True) -def test_revoke_oauth_token_invalidates_redis_cache_when_live_hash_seen(sqlite_session: Session): - """Happy path: snapshot finds a live ``token_hash`` → UPDATE runs + - Redis cache entry is DEL'd so the next bearer probe re-reads the now - revoked row from DB. - """ - sqlite_session.add(_token()) - sqlite_session.commit() - - redis = MagicMock() - - revoke_oauth_token(redis, str(TOKEN_ID), session=sqlite_session) - - assert not sqlite_session.in_transaction() - persisted = sqlite_session.get(OAuthAccessToken, str(TOKEN_ID)) - assert persisted is not None - assert persisted.token_hash is None - assert persisted.revoked_at is not None - redis.delete.assert_called_once_with(TOKEN_CACHE_KEY_FMT.format(hash="live-hash")) - - -@pytest.mark.parametrize("sqlite_session", [(OAuthAccessToken,)], indirect=True) -def test_revoke_oauth_token_is_idempotent_when_already_revoked(sqlite_session: Session): - """Second call (or race-loser): no live hash → UPDATE still runs (it - is itself idempotent thanks to ``WHERE revoked_at IS NULL``) but the - Redis invalidation is skipped because there's no cache entry to - drop. - """ - revoked_at = datetime.now(UTC) - timedelta(minutes=1) - sqlite_session.add(_token(token_hash=None, revoked_at=revoked_at)) - sqlite_session.commit() - - redis = MagicMock() - - revoke_oauth_token(redis, str(TOKEN_ID), session=sqlite_session) - - assert not sqlite_session.in_transaction() - persisted = sqlite_session.get(OAuthAccessToken, str(TOKEN_ID)) - assert persisted is not None - assert persisted.token_hash is None - assert persisted.revoked_at is not None - redis.delete.assert_not_called() - - -# --------------------------------------------------------------------------- -# list_active_sessions / token_belongs_to_subject -# --------------------------------------------------------------------------- - - -@pytest.mark.parametrize("sqlite_session", [(OAuthAccessToken,)], indirect=True) -def test_list_active_sessions_returns_only_live_subject_tokens(sqlite_session: Session): - """Only live, hashed rows for the authenticated subject are returned newest-first.""" - - now = datetime.now(UTC) - active_new = _token(token_id=TOKEN_ID, created_at=now - timedelta(minutes=1)) - active_old = _token(token_id=OTHER_TOKEN_ID, created_at=now - timedelta(minutes=2)) - expired = _token( - token_id=uuid.UUID(int=5), - expires_at=now - timedelta(seconds=1), - created_at=now - timedelta(minutes=3), - ) - revoked = _token( - token_id=uuid.UUID(int=6), - token_hash=None, - revoked_at=now - timedelta(seconds=1), - created_at=now - timedelta(minutes=4), - ) - hashless = _token( - token_id=uuid.UUID(int=7), - token_hash=None, - created_at=now - timedelta(minutes=5), - ) - other_account = _token( - token_id=uuid.UUID(int=8), - account_id=OTHER_ACCOUNT_ID, - created_at=now - timedelta(minutes=6), - ) - external_sso = _token( - token_id=uuid.UUID(int=9), - account_id=None, - subject_email="user@example.com", - subject_issuer="https://idp.example.com", - created_at=now - timedelta(minutes=7), - ) - sqlite_session.add_all([active_new, active_old, expired, revoked, hashless, other_account, external_sso]) - sqlite_session.commit() - - out = list_active_sessions(_account_ctx(), now, session=sqlite_session) - - assert [token.id for token in out] == [str(TOKEN_ID), str(OTHER_TOKEN_ID)] - - -@pytest.mark.parametrize("sqlite_session", [(OAuthAccessToken,)], indirect=True) -def test_token_belongs_to_subject_true_when_row_present(sqlite_session: Session): - sqlite_session.add(_token()) - sqlite_session.commit() - - assert token_belongs_to_subject(str(TOKEN_ID), _account_ctx(), session=sqlite_session) is True - - -@pytest.mark.parametrize("sqlite_session", [(OAuthAccessToken,)], indirect=True) -def test_token_belongs_to_subject_false_for_other_account(sqlite_session: Session): - sqlite_session.add(_token(account_id=OTHER_ACCOUNT_ID)) - sqlite_session.commit() - - assert token_belongs_to_subject(str(TOKEN_ID), _account_ctx(), session=sqlite_session) is False diff --git a/api/tests/unit_tests/services/test_recommended_app_query_service.py b/api/tests/unit_tests/services/test_recommended_app_query_service.py index 09d661b6754..3e762b12167 100644 --- a/api/tests/unit_tests/services/test_recommended_app_query_service.py +++ b/api/tests/unit_tests/services/test_recommended_app_query_service.py @@ -2,7 +2,6 @@ from unittest.mock import MagicMock import pytest -from constants.languages import languages from services.recommended_app_query_service import ( RecommendedAppCatalogPage, RecommendedAppDetailRecord, @@ -83,17 +82,14 @@ def test_is_previewable_falls_back_to_catalog(expected: bool) -> None: @pytest.mark.parametrize( - ("requested_language", "interface_language", "expected"), + ("language", "expected"), [ - ("en-US", "fr-FR", "en-US"), - ("invalid", "fr-FR", "fr-FR"), - (None, "custom-language", "custom-language"), - (None, None, languages[0]), + ("fr-FR", "fr-FR"), + ("invalid", "en-US"), ], ) def test_list_recommended_resolves_language( - requested_language: str | None, - interface_language: str | None, + language: str, expected: str, ) -> None: catalog = MagicMock() @@ -101,8 +97,7 @@ def test_list_recommended_resolves_language( service, _ = _service(catalog=catalog) service.list_recommended( - requested_language=requested_language, - interface_language=interface_language, + language=language, ) catalog.list_recommended.assert_called_once_with(expected) @@ -113,7 +108,7 @@ def test_list_recommended_disables_upstream_trial_without_querying_trial_apps() catalog.list_recommended.return_value = _page("app-1") service, trial_apps = _service(catalog=catalog) - result = service.list_recommended(requested_language="en-US", interface_language=None) + result = service.list_recommended(language="en-US") assert result.recommended_apps[0].can_trial is False trial_apps.existing_ids.assert_not_called() @@ -130,7 +125,7 @@ def test_list_recommended_enriches_trial_status_in_one_bulk_query() -> None: trial_enabled=True, ) - result = service.list_recommended(requested_language="en-US", interface_language=None) + result = service.list_recommended(language="en-US") assert [app.can_trial for app in result.recommended_apps] == [True, False] trial_apps.existing_ids.assert_called_once_with(["app-1", "app-2"]) @@ -141,9 +136,9 @@ def test_list_learn_dify_does_not_return_categories() -> None: catalog.list_learn_dify.return_value = _page(categories=("ignored",)) service, _ = _service(catalog=catalog) - result = service.list_learn_dify(requested_language="invalid", interface_language="fr-FR") + result = service.list_learn_dify(language="invalid") - catalog.list_learn_dify.assert_called_once_with("fr-FR") + catalog.list_learn_dify.assert_called_once_with("en-US") assert result.recommended_apps == () assert not hasattr(result, "categories") diff --git a/api/tests/unit_tests/services/test_skill_management_service.py b/api/tests/unit_tests/services/test_skill_management_service.py index 8e9fd12529b..62807577ed0 100644 --- a/api/tests/unit_tests/services/test_skill_management_service.py +++ b/api/tests/unit_tests/services/test_skill_management_service.py @@ -2664,6 +2664,32 @@ def test_import_skill_package_creates_draft_and_rejects_name_conflicts() -> None assert exc_info.value.code == "skill_name_conflict" +def test_import_skill_package_strips_root_alongside_macos_metadata_folder() -> None: + package = io.BytesIO() + with zipfile.ZipFile(package, "w") as archive: + archive.writestr( + "expense-sop/SKILL.md", + "---\nname: expense-sop\ndescription: Expenses\n---\n# Expenses", + ) + archive.writestr("expense-sop/references/policy.md", "Policy") + # macOS Finder "Compress" adds an AppleDouble metadata sibling folder + # for archives whose source files carry extended attributes. + archive.writestr("__MACOSX/expense-sop/._SKILL.md", b"\x00") + + service = SkillManagementService(tool_file_manager=_FakeToolFileManager()) + imported = service.import_skill( + tenant_id=TENANT, + user_id=USER, + payload=SkillImportPayload(content=package.getvalue(), filename="expense-sop.zip"), + ) + + assert imported["name"] == "expense-sop" + imported_paths = [item["path"] for item in imported["files"]] + assert "SKILL.md" in imported_paths + assert "references/policy.md" in imported_paths + assert not any(path.startswith("__MACOSX") for path in imported_paths) + + def test_import_skill_package_rejects_missing_frontmatter_description() -> None: package = io.BytesIO() with zipfile.ZipFile(package, "w") as archive: diff --git a/api/tests/unit_tests/services/test_trigger_subscription_builder_service.py b/api/tests/unit_tests/services/test_trigger_subscription_builder_service.py index 6316de75792..1eb28c3dcd2 100644 --- a/api/tests/unit_tests/services/test_trigger_subscription_builder_service.py +++ b/api/tests/unit_tests/services/test_trigger_subscription_builder_service.py @@ -5,7 +5,7 @@ from unittest.mock import Mock, patch import pytest from core.plugin.entities.plugin_daemon import CredentialType -from core.trigger.entities.entities import SubscriptionBuilder, SubscriptionBuilderUpdater +from core.trigger.entities.entities import Subscription, SubscriptionBuilder, SubscriptionBuilderUpdater from core.trigger.trigger_manager import TriggerManager from models.provider_ids import TriggerProviderID from services.trigger.trigger_subscription_builder_service import TriggerSubscriptionBuilderService @@ -249,3 +249,50 @@ def test_process_validation_endpoint_uses_the_public_capability() -> None: assert str(provider_call["provider_id"]) == str(PROVIDER_ID) controller.dispatch.assert_called_once() append_log.assert_called_once() + + +def test_update_and_build_persists_provider_subscription_expires_at() -> None: + """Authorized create must store the plugin lease, not the builder's -1 placeholder.""" + builder = subscription_builder().model_copy( + update={ + "credential_type": CredentialType.OAUTH2, + "credentials": {"token": "x"}, + "credential_expires_at": 1_787_560_741, + } + ) + assert builder.expires_at == -1 + real_lease = 1_787_560_681 + provider_subscription = Subscription( + expires_at=real_lease, + endpoint="https://dify.example.com/triggers/plugin/builder-1", + parameters={"label_ids": ["INBOX"]}, + properties={"topic_name": "projects/p/topics/t"}, + ) + + with ( + patch.object(TriggerManager, "get_trigger_provider", return_value=Mock()), + patch.object(TriggerSubscriptionBuilderService, "acquire_builder_lock", return_value=nullcontext()), + patch.object(TriggerSubscriptionBuilderService, "get_subscription_builder", return_value=builder), + patch.object(TriggerManager, "subscribe_trigger", return_value=provider_subscription) as subscribe_trigger, + patch( + "services.trigger.trigger_subscription_builder_service.TriggerProviderService.add_trigger_subscription" + ) as add_subscription, + patch("services.trigger.trigger_subscription_builder_service.redis_client.setex"), + patch("services.trigger.trigger_subscription_builder_service.redis_client.delete"), + ): + TriggerSubscriptionBuilderService.update_and_build_builder( + tenant_id="tenant-1", + user_id="user-1", + provider_id=PROVIDER_ID, + subscription_builder_id=builder.id, + subscription_builder_updater=SubscriptionBuilderUpdater(), + ) + + subscribe_trigger.assert_called_once() + add_subscription.assert_called_once() + persisted = add_subscription.call_args.kwargs + assert persisted["expires_at"] == real_lease + assert persisted["expires_at"] != -1 + assert persisted["properties"] == {"topic_name": "projects/p/topics/t"} + assert persisted["credential_expires_at"] == 1_787_560_741 + assert persisted["credentials"] == {"token": "x"} diff --git a/api/tests/unit_tests/services/test_workflow_app_log_query_service.py b/api/tests/unit_tests/services/test_workflow_app_log_query_service.py new file mode 100644 index 00000000000..f5a3c61492e --- /dev/null +++ b/api/tests/unit_tests/services/test_workflow_app_log_query_service.py @@ -0,0 +1,119 @@ +import json +import uuid +from collections.abc import Mapping +from datetime import UTC, datetime +from unittest.mock import MagicMock, patch + +import pytest + +from models.enums import AppTriggerType +from repositories.workflow_app_log_query_repository import WorkflowAppLogQueryRepository +from services.workflow_app_log_query_service import ( + WorkflowAppLogItem, + WorkflowAppLogPage, + WorkflowAppLogQueryService, +) + + +def _service_with_metadata( + value: str | Mapping[str, object] | None, + *, + detail: bool = True, +) -> WorkflowAppLogQueryService: + logs = MagicMock() + logs.get_paginated.return_value = WorkflowAppLogPage( + page=1, + limit=20, + total=1, + has_more=False, + data=( + WorkflowAppLogItem( + id="log-1", + workflow_run=None, + details={"trigger_metadata": value} if detail else None, + created_from="web-app", + created_by_role="account", + created_by_account=None, + created_by_end_user=None, + created_at=datetime(2026, 1, 1, tzinfo=UTC), + ), + ), + ) + return WorkflowAppLogQueryService(logs=logs) + + +def test_list_logs_preserves_missing_workflow_run_and_resolves_empty_metadata() -> None: + service = _service_with_metadata(None) + + result = service.list_logs(tenant_id="tenant-1", app_id="app-1", detail=True) + + assert result.data[0].details == {"trigger_metadata": {}} + assert result.data[0].workflow_run is None + + +def test_list_logs_enriches_plugin_icons() -> None: + metadata = { + "type": AppTriggerType.TRIGGER_PLUGIN.value, + "icon_filename": "light.png", + "icon_dark_filename": "dark.png", + } + service = _service_with_metadata(json.dumps(metadata)) + + with patch( + "services.workflow_app_log_query_service.PluginService.get_plugin_icon_url", + side_effect=["https://cdn/light.png", "https://cdn/dark.png"], + ) as get_icon_url: + result = service.list_logs(tenant_id="tenant-1", app_id="app-1", detail=True) + + details = result.data[0].details + assert details is not None + trigger_metadata = details["trigger_metadata"] + assert trigger_metadata["icon"] == "https://cdn/light.png" + assert trigger_metadata["icon_dark"] == "https://cdn/dark.png" + assert get_icon_url.call_count == 2 + + +def test_list_logs_does_not_fetch_icons_for_non_plugin_metadata() -> None: + service = _service_with_metadata(json.dumps({"type": AppTriggerType.TRIGGER_WEBHOOK.value})) + + with patch("services.workflow_app_log_query_service.PluginService.get_plugin_icon_url") as get_icon_url: + result = service.list_logs(tenant_id="tenant-1", app_id="app-1", detail=True) + + assert result.data[0].details == {"trigger_metadata": {"type": AppTriggerType.TRIGGER_WEBHOOK.value}} + get_icon_url.assert_not_called() + + +def test_list_logs_does_not_resolve_metadata_without_detail() -> None: + service = _service_with_metadata("not-json", detail=False) + + result = service.list_logs(tenant_id="tenant-1", app_id="app-1") + + assert result.data[0].details is None + + +@pytest.mark.parametrize( + ("value", "expected"), + [ + (None, None), + ("", None), + ('{"k":"v"}', {"k": "v"}), + ("not-json", None), + ({"raw": True}, {"raw": True}), + ], +) +def test_safe_json_loads(value: object, expected: object) -> None: + assert WorkflowAppLogQueryService._safe_json_loads(value) == expected + + +def test_safe_parse_uuid_rejects_short_and_invalid_values() -> None: + assert WorkflowAppLogQueryRepository._safe_parse_uuid("short") is None + assert WorkflowAppLogQueryRepository._safe_parse_uuid("x" * 40) is None + + +def test_safe_parse_uuid_accepts_uuid() -> None: + raw = str(uuid.uuid4()) + + result = WorkflowAppLogQueryRepository._safe_parse_uuid(raw) + + assert result is not None + assert str(result) == raw diff --git a/api/tests/unit_tests/services/test_workflow_app_service_metadata.py b/api/tests/unit_tests/services/test_workflow_app_service_metadata.py deleted file mode 100644 index c51de1abaa9..00000000000 --- a/api/tests/unit_tests/services/test_workflow_app_service_metadata.py +++ /dev/null @@ -1,111 +0,0 @@ -"""Unit tests for workflow app log views and trigger metadata helpers.""" - -import json -import uuid -from unittest.mock import MagicMock, patch - -import pytest -from sqlalchemy.orm import Session - -from models.account import Account -from models.enums import AppTriggerType, CreatorUserRole -from models.workflow import WorkflowAppLog, WorkflowAppLogCreatedFrom -from services.workflow_app_service import LogView, WorkflowAppService - - -class TestLogView: - def test_details_and_proxy_attributes(self) -> None: - log = WorkflowAppLog( - tenant_id="tenant-1", - app_id="app-1", - workflow_id="workflow-1", - workflow_run_id="run-1", - created_from=WorkflowAppLogCreatedFrom.WEB_APP, - created_by_role=CreatorUserRole.ACCOUNT, - created_by="account-1", - ) - log.id = "log-1" - - view = LogView(log=log, details={"trigger_metadata": {"type": "plugin"}}, session=MagicMock()) - - assert view.details == {"trigger_metadata": {"type": "plugin"}} - assert view.id == "log-1" - - def test_account_accessors_resolve_via_wrapped_session(self, sqlite_session: Session) -> None: - account = Account(name="Test Account", email="test@example.com") - sqlite_session.add(account) - sqlite_session.flush() - log = WorkflowAppLog( - tenant_id="tenant-1", - app_id="app-1", - workflow_id="workflow-1", - workflow_run_id="run-1", - created_from=WorkflowAppLogCreatedFrom.WEB_APP, - created_by_role=CreatorUserRole.ACCOUNT, - created_by=account.id, - ) - - view = LogView(log=log, details=None, session=sqlite_session) - - resolved = view.created_by_account - assert resolved is not None - assert resolved.id == account.id - assert view.created_by_end_user is None - - -class TestHandleTriggerMetadata: - def test_returns_empty_dict_when_metadata_missing(self) -> None: - assert WorkflowAppService().handle_trigger_metadata("tenant-1", None) == {} - - def test_enriches_plugin_icons(self) -> None: - metadata = { - "type": AppTriggerType.TRIGGER_PLUGIN.value, - "icon_filename": "light.png", - "icon_dark_filename": "dark.png", - } - with patch( - "services.workflow_app_service.PluginService.get_plugin_icon_url", - side_effect=["https://cdn/light.png", "https://cdn/dark.png"], - ) as mock_icon: - result = WorkflowAppService().handle_trigger_metadata("tenant-1", json.dumps(metadata)) - - assert result["icon"] == "https://cdn/light.png" - assert result["icon_dark"] == "https://cdn/dark.png" - assert mock_icon.call_count == 2 - - def test_non_plugin_metadata_without_icon_lookup(self) -> None: - metadata = {"type": AppTriggerType.TRIGGER_WEBHOOK.value} - with patch("services.workflow_app_service.PluginService.get_plugin_icon_url") as mock_icon: - result = WorkflowAppService().handle_trigger_metadata("tenant-1", json.dumps(metadata)) - - assert result["type"] == AppTriggerType.TRIGGER_WEBHOOK.value - mock_icon.assert_not_called() - - -class TestSafeJsonLoads: - @pytest.mark.parametrize( - ("value", "expected"), - [ - (None, None), - ("", None), - ('{"k":"v"}', {"k": "v"}), - ("not-json", None), - ({"raw": True}, {"raw": True}), - ], - ) - def test_handles_various_inputs(self, value, expected) -> None: - assert WorkflowAppService._safe_json_loads(value) == expected - - -class TestSafeParseUuid: - def test_returns_none_for_short_or_invalid_values(self) -> None: - assert WorkflowAppService._safe_parse_uuid("short") is None - assert WorkflowAppService._safe_parse_uuid("x" * 40) is None - - def test_returns_uuid_for_valid_string(self) -> None: - raw = str(uuid.uuid4()) - - result = WorkflowAppService._safe_parse_uuid(raw) - - assert result is not None - assert str(result) == raw diff --git a/api/tests/unit_tests/services/test_workflow_run_service.py b/api/tests/unit_tests/services/test_workflow_run_service.py index c1aa0fd9cea..b71829dcf7c 100644 --- a/api/tests/unit_tests/services/test_workflow_run_service.py +++ b/api/tests/unit_tests/services/test_workflow_run_service.py @@ -1,62 +1,38 @@ -"""Workflow-run service tests with real SQLite-bound session factories.""" +"""Unit tests for the Console workflow-run application service.""" -from decimal import Decimal from types import SimpleNamespace -from typing import Any from unittest.mock import MagicMock import pytest -from sqlalchemy import Engine, event -from sqlalchemy.orm import Session, sessionmaker from graphon.enums import WorkflowExecutionStatus -from models import Account, App, EndUser, Message, WorkflowRun, WorkflowRunTriggeredFrom, WorkflowType -from models.account import Tenant -from models.enums import ConversationFromSource, CreatorUserRole, EndUserType -from models.model import AppMode +from machinery.context import RequestContext +from models import WorkflowRun, WorkflowRunTriggeredFrom, WorkflowType +from models.enums import CreatorUserRole +from repositories.sqlalchemy_api_workflow_run_repository import WorkflowRunMessageRef from services import workflow_run_service as service_module from services.workflow_run_service import WorkflowRunService @pytest.fixture -def repository_factory_mocks(monkeypatch: pytest.MonkeyPatch) -> tuple[MagicMock, MagicMock, Any]: - node_repo = MagicMock() - workflow_run_repo = MagicMock() - factory = SimpleNamespace( - create_api_workflow_node_execution_repository=MagicMock(return_value=node_repo), - create_api_workflow_run_repository=MagicMock(return_value=workflow_run_repo), - ) - monkeypatch.setattr(service_module, "DifyAPIRepositoryFactory", factory) - return node_repo, workflow_run_repo, factory +def service_dependencies() -> tuple[MagicMock, MagicMock]: + return MagicMock(), MagicMock() -def _app_model(*, app_id: str = "app-1", tenant_id: str = "tenant-1") -> App: - return App( - id=app_id, - tenant_id=tenant_id, - name="Workflow App", - mode=AppMode.ADVANCED_CHAT, - enable_site=False, - enable_api=False, +def _service(dependencies: tuple[MagicMock, MagicMock]) -> WorkflowRunService: + node_executions, workflow_runs = dependencies + return WorkflowRunService( + workflow_runs=workflow_runs, + node_executions=node_executions, ) -def _account(*, account_id: str = "account-1", current_tenant_id: str | None = "tenant-1") -> Account: - account = Account(name="Workflow User", email=f"{account_id}@example.com") - account.id = account_id - if current_tenant_id is not None: - tenant = Tenant(name="Workflow Tenant") - tenant.id = current_tenant_id - account._current_tenant = tenant - return account - - -def _end_user(*, end_user_id: str = "end-user-1", tenant_id: str = "tenant-1") -> EndUser: - return EndUser( - id=end_user_id, - tenant_id=tenant_id, - type=EndUserType.SERVICE_API, - session_id=f"session-{end_user_id}", +def _request_context(*, workspace_id: str = "tenant-1") -> RequestContext: + return RequestContext( + request_id="request-1", + trace_id="trace-1", + account_id="account-1", + active_workspace_id=workspace_id, ) @@ -79,87 +55,37 @@ def _workflow_run( ) -def _message(*, message_id: str, workflow_run_id: str, conversation_id: str) -> Message: - message = Message( - app_id="app-1", - conversation_id=conversation_id, - query="query", - message={"role": "user", "content": "query"}, - answer="answer", - message_unit_price=Decimal("0.0001"), - answer_unit_price=Decimal("0.0001"), - currency="USD", - from_source=ConversationFromSource.API, - ) - message.id = message_id - message._inputs = {} - message.workflow_run_id = workflow_run_id - return message +def test_init_keeps_injected_dependencies( + service_dependencies: tuple[MagicMock, MagicMock], +) -> None: + node_executions, workflow_runs = service_dependencies + service = _service(service_dependencies) -class TestWorkflowRunServiceInitialization: - def test___init___should_create_sessionmaker_from_db_engine_when_session_factory_missing( - self, - monkeypatch: pytest.MonkeyPatch, - repository_factory_mocks: tuple[MagicMock, MagicMock, Any], - sqlite_engine: Engine, - ) -> None: - monkeypatch.setattr(service_module, "db", SimpleNamespace(engine=sqlite_engine)) - - service = WorkflowRunService() - - assert isinstance(service._session_factory, sessionmaker) - assert service._session_factory.kw["bind"] is sqlite_engine - assert service._session_factory.kw["expire_on_commit"] is False - - def test___init___should_create_sessionmaker_when_engine_is_provided( - self, - repository_factory_mocks: tuple[MagicMock, MagicMock, Any], - sqlite_engine: Engine, - ) -> None: - service = WorkflowRunService(session_factory=sqlite_engine) - - assert isinstance(service._session_factory, sessionmaker) - assert service._session_factory.kw["bind"] is sqlite_engine - assert service._session_factory.kw["expire_on_commit"] is False - - def test___init___should_keep_provided_sessionmaker_and_create_repositories( - self, - repository_factory_mocks: tuple[MagicMock, MagicMock, Any], - sqlite_session_factory: sessionmaker[Session], - ) -> None: - node_repo, workflow_run_repo, factory = repository_factory_mocks - - service = WorkflowRunService(session_factory=sqlite_session_factory) - - assert service._session_factory is sqlite_session_factory - assert service._node_execution_service_repo is node_repo - assert service._workflow_run_repo is workflow_run_repo - factory.create_api_workflow_node_execution_repository.assert_called_once_with(sqlite_session_factory) - factory.create_api_workflow_run_repository.assert_called_once_with(sqlite_session_factory) + assert service._workflow_runs is workflow_runs + assert service._node_executions is node_executions class TestWorkflowRunServiceQueries: def test_get_paginate_workflow_runs_should_forward_filters_and_parse_limit( self, - repository_factory_mocks: tuple[MagicMock, MagicMock, Any], - sqlite_session_factory: sessionmaker[Session], + service_dependencies: tuple[MagicMock, MagicMock], ) -> None: - _, workflow_run_repo, _ = repository_factory_mocks - service = WorkflowRunService(session_factory=sqlite_session_factory) - app_model = _app_model(tenant_id="tenant-1", app_id="app-1") + _, workflow_runs = service_dependencies + service = _service(service_dependencies) expected = MagicMock(name="pagination") - workflow_run_repo.get_paginated_workflow_runs.return_value = expected + workflow_runs.get_paginated_workflow_runs.return_value = expected args = {"limit": "7", "last_id": "last-1", "status": "succeeded"} result = service.get_paginate_workflow_runs( - app_model=app_model, + _request_context(workspace_id="tenant-1"), + app_id="app-1", args=args, triggered_from=WorkflowRunTriggeredFrom.APP_RUN, ) assert result is expected - workflow_run_repo.get_paginated_workflow_runs.assert_called_once_with( + workflow_runs.get_paginated_workflow_runs.assert_called_once_with( tenant_id="tenant-1", app_id="app-1", triggered_from=WorkflowRunTriggeredFrom.APP_RUN, @@ -168,25 +94,26 @@ class TestWorkflowRunServiceQueries: status="succeeded", ) - @pytest.mark.parametrize("sqlite_session", [(Message,)], indirect=True) def test_get_paginate_advanced_chat_workflow_runs_should_attach_message_fields_when_message_exists( self, - repository_factory_mocks: tuple[MagicMock, MagicMock, Any], + service_dependencies: tuple[MagicMock, MagicMock], monkeypatch: pytest.MonkeyPatch, - sqlite_session_factory: sessionmaker[Session], - sqlite_session: Session, ) -> None: - service = WorkflowRunService(session_factory=sqlite_session_factory) - app_model = _app_model(tenant_id="tenant-1", app_id="app-1") + _, workflow_runs = service_dependencies + service = _service(service_dependencies) run_with_message = _workflow_run(status=WorkflowExecutionStatus.RUNNING) run_without_message = _workflow_run(run_id="run-2") pagination = SimpleNamespace(data=[run_with_message, run_without_message]) monkeypatch.setattr(service, "get_paginate_workflow_runs", MagicMock(return_value=pagination)) + workflow_runs.get_message_refs.return_value = { + "run-1": WorkflowRunMessageRef(message_id="msg-1", conversation_id="conv-1") + } - sqlite_session.add(_message(message_id="msg-1", conversation_id="conv-1", workflow_run_id="run-1")) - sqlite_session.commit() - - result = service.get_paginate_advanced_chat_workflow_runs(app_model=app_model, args={"limit": "2"}) + result = service.get_paginate_advanced_chat_workflow_runs( + _request_context(), + app_id="app-1", + args={"limit": "2"}, + ) assert result is pagination assert len(result.data) == 2 @@ -195,56 +122,24 @@ class TestWorkflowRunServiceQueries: assert result.data[0].status == "running" assert not hasattr(result.data[1], "message_id") assert result.data[1].id == "run-2" - - @pytest.mark.parametrize("sqlite_session", [(Message,)], indirect=True) - def test_get_paginate_advanced_chat_workflow_runs_batch_loads_messages_without_n_plus_one( - self, - repository_factory_mocks: tuple[MagicMock, MagicMock, Any], - monkeypatch: pytest.MonkeyPatch, - sqlite_session_factory: sessionmaker[Session], - sqlite_session: Session, - ) -> None: - """Messages must load with a constant query count regardless of run count. - - Previously the deprecated WorkflowRun.message property issued one query per - run (N+1); they are now batch-loaded in a single query. - """ - service = WorkflowRunService(session_factory=sqlite_session_factory) - app_model = _app_model(tenant_id="tenant-1", app_id="app-1") - runs = [_workflow_run(run_id=f"run-{i}") for i in range(5)] - pagination = SimpleNamespace(data=runs) - monkeypatch.setattr(service, "get_paginate_workflow_runs", MagicMock(return_value=pagination)) - - message_query_count = 0 - - def count_message_query(*_args: object) -> None: - nonlocal message_query_count - message_query_count += 1 - - engine = sqlite_session.get_bind() - event.listen(engine, "before_cursor_execute", count_message_query) - try: - service.get_paginate_advanced_chat_workflow_runs(app_model=app_model, args={}) - finally: - event.remove(engine, "before_cursor_execute", count_message_query) - assert all(not hasattr(run, "message_id") for run in runs) - assert message_query_count == 1 + workflow_runs.get_message_refs.assert_called_once_with( + app_id="app-1", + workflow_run_ids=["run-1", "run-2"], + ) def test_get_workflow_run_should_delegate_to_repository_by_tenant_and_app( self, - repository_factory_mocks: tuple[MagicMock, MagicMock, Any], - sqlite_session_factory: sessionmaker[Session], + service_dependencies: tuple[MagicMock, MagicMock], ) -> None: - _, workflow_run_repo, _ = repository_factory_mocks - service = WorkflowRunService(session_factory=sqlite_session_factory) - app_model = _app_model(tenant_id="tenant-1", app_id="app-1") + _, workflow_runs = service_dependencies + service = _service(service_dependencies) expected = _workflow_run() - workflow_run_repo.get_workflow_run_by_id.return_value = expected + workflow_runs.get_workflow_run_by_id.return_value = expected - result = service.get_workflow_run(app_model=app_model, run_id="run-1") + result = service.get_workflow_run(_request_context(), app_id="app-1", run_id="run-1") assert result is expected - workflow_run_repo.get_workflow_run_by_id.assert_called_once_with( + workflow_runs.get_workflow_run_by_id.assert_called_once_with( tenant_id="tenant-1", app_id="app-1", run_id="run-1", @@ -252,24 +147,23 @@ class TestWorkflowRunServiceQueries: def test_get_workflow_runs_count_should_forward_optional_filters( self, - repository_factory_mocks: tuple[MagicMock, MagicMock, Any], - sqlite_session_factory: sessionmaker[Session], + service_dependencies: tuple[MagicMock, MagicMock], ) -> None: - _, workflow_run_repo, _ = repository_factory_mocks - service = WorkflowRunService(session_factory=sqlite_session_factory) - app_model = _app_model(tenant_id="tenant-1", app_id="app-1") + _, workflow_runs = service_dependencies + service = _service(service_dependencies) expected = {"total": 3, "succeeded": 2} - workflow_run_repo.get_workflow_runs_count.return_value = expected + workflow_runs.get_workflow_runs_count.return_value = expected result = service.get_workflow_runs_count( - app_model=app_model, + _request_context(), + app_id="app-1", status="succeeded", time_range="7d", triggered_from=WorkflowRunTriggeredFrom.APP_RUN, ) assert result == expected - workflow_run_repo.get_workflow_runs_count.assert_called_once_with( + workflow_runs.get_workflow_runs_count.assert_called_once_with( tenant_id="tenant-1", app_id="app-1", triggered_from=WorkflowRunTriggeredFrom.APP_RUN, @@ -279,83 +173,44 @@ class TestWorkflowRunServiceQueries: def test_get_workflow_run_node_executions_should_return_empty_list_when_run_not_found( self, - repository_factory_mocks: tuple[MagicMock, MagicMock, Any], + service_dependencies: tuple[MagicMock, MagicMock], monkeypatch: pytest.MonkeyPatch, - sqlite_session_factory: sessionmaker[Session], ) -> None: - service = WorkflowRunService(session_factory=sqlite_session_factory) + service = _service(service_dependencies) monkeypatch.setattr(service, "get_workflow_run", MagicMock(return_value=None)) - app_model = _app_model(app_id="app-1") - user = _account(current_tenant_id="tenant-1") - result = service.get_workflow_run_node_executions(app_model=app_model, run_id="run-1", user=user) + result = service.get_workflow_run_node_executions( + _request_context(), + app_id="app-1", + run_id="run-1", + ) assert result == [] - def test_get_workflow_run_node_executions_should_use_end_user_tenant_id( + def test_get_workflow_run_node_executions_should_use_request_workspace( self, - repository_factory_mocks: tuple[MagicMock, MagicMock, Any], + service_dependencies: tuple[MagicMock, MagicMock], monkeypatch: pytest.MonkeyPatch, - sqlite_session_factory: sessionmaker[Session], ) -> None: - node_repo, _, _ = repository_factory_mocks - service = WorkflowRunService(session_factory=sqlite_session_factory) + node_executions, _ = service_dependencies + service = _service(service_dependencies) monkeypatch.setattr(service, "get_workflow_run", MagicMock(return_value=_workflow_run())) - user = _end_user(tenant_id="tenant-end-user") - app_model = _app_model(app_id="app-1") - expected_executions = [SimpleNamespace(id="exec-1")] - expected_traces = [SimpleNamespace(id="exec-1:retry:1")] - node_repo.get_executions_by_workflow_run.return_value = expected_executions - mock_assemble = MagicMock(return_value=expected_traces) - monkeypatch.setattr(service_module, "assemble_workflow_node_execution_traces", mock_assemble) - - result = service.get_workflow_run_node_executions(app_model=app_model, run_id="run-1", user=user) - - assert result == expected_traces - node_repo.get_executions_by_workflow_run.assert_called_once_with( - tenant_id="tenant-end-user", - app_id="app-1", - workflow_run_id="run-1", - ) - mock_assemble.assert_called_once_with(expected_executions, node_repo) - - def test_get_workflow_run_node_executions_should_use_account_current_tenant_id( - self, - repository_factory_mocks: tuple[MagicMock, MagicMock, Any], - monkeypatch: pytest.MonkeyPatch, - sqlite_session_factory: sessionmaker[Session], - ) -> None: - node_repo, _, _ = repository_factory_mocks - service = WorkflowRunService(session_factory=sqlite_session_factory) - monkeypatch.setattr(service, "get_workflow_run", MagicMock(return_value=_workflow_run())) - app_model = _app_model(app_id="app-1") - user = _account(current_tenant_id="tenant-account") expected_executions = [SimpleNamespace(id="exec-1"), SimpleNamespace(id="exec-2")] expected_traces = [SimpleNamespace(id="exec-1:retry:1"), SimpleNamespace(id="exec-1")] - node_repo.get_executions_by_workflow_run.return_value = expected_executions + node_executions.get_executions_by_workflow_run.return_value = expected_executions mock_assemble = MagicMock(return_value=expected_traces) monkeypatch.setattr(service_module, "assemble_workflow_node_execution_traces", mock_assemble) - result = service.get_workflow_run_node_executions(app_model=app_model, run_id="run-1", user=user) + result = service.get_workflow_run_node_executions( + _request_context(workspace_id="tenant-context"), + app_id="app-1", + run_id="run-1", + ) assert result == expected_traces - node_repo.get_executions_by_workflow_run.assert_called_once_with( - tenant_id="tenant-account", + node_executions.get_executions_by_workflow_run.assert_called_once_with( + tenant_id="tenant-context", app_id="app-1", workflow_run_id="run-1", ) - mock_assemble.assert_called_once_with(expected_executions, node_repo) - - def test_get_workflow_run_node_executions_should_raise_when_resolved_tenant_id_is_none( - self, - repository_factory_mocks: tuple[MagicMock, MagicMock, Any], - monkeypatch: pytest.MonkeyPatch, - sqlite_session_factory: sessionmaker[Session], - ) -> None: - service = WorkflowRunService(session_factory=sqlite_session_factory) - monkeypatch.setattr(service, "get_workflow_run", MagicMock(return_value=_workflow_run())) - app_model = _app_model(app_id="app-1") - user = _account(current_tenant_id=None) - - with pytest.raises(ValueError, match="tenant_id cannot be None"): - service.get_workflow_run_node_executions(app_model=app_model, run_id="run-1", user=user) + mock_assemble.assert_called_once_with(expected_executions, node_executions) diff --git a/api/tests/unit_tests/services/test_workflow_run_service_pause.py b/api/tests/unit_tests/services/test_workflow_run_service_pause.py index 5b24cfd8a66..3552cbd7c9d 100644 --- a/api/tests/unit_tests/services/test_workflow_run_service_pause.py +++ b/api/tests/unit_tests/services/test_workflow_run_service_pause.py @@ -1,57 +1,112 @@ -"""Tests for the session lifecycle owned by ``WorkflowRunService``.""" +"""Tests for Console workflow pause details.""" -from unittest.mock import create_autospec, patch +from datetime import datetime +from unittest.mock import MagicMock import pytest -from sqlalchemy import Engine, text -from sqlalchemy.orm import Session, sessionmaker -from repositories.api_workflow_run_repository import APIWorkflowRunRepository -from services.workflow_run_service import WorkflowRunService +from core.workflow.nodes.human_input.pause_reason import HumanInputRequired +from graphon.entities.pause_reason import SchedulingPause +from graphon.enums import WorkflowExecutionStatus +from machinery.context import RequestContext +from repositories.sqlalchemy_api_workflow_run_repository import WorkflowRunPauseRecord +from services.workflow_run_service import ( + WorkflowRunPauseDetails, + WorkflowRunPausedNode, + WorkflowRunService, +) @pytest.fixture -def sqlite_session_factory(sqlite_engine: Engine) -> sessionmaker[Session]: - """Return a real factory whose sessions are bound to the isolated SQLite engine.""" - return sessionmaker(bind=sqlite_engine, expire_on_commit=False) +def workflow_runs() -> MagicMock: + return MagicMock() -@pytest.fixture -def workflow_run_repository(): - """Keep the repository boundary mocked while exercising real session construction.""" - return create_autospec(APIWorkflowRunRepository) +def _service(workflow_runs: MagicMock) -> WorkflowRunService: + return WorkflowRunService( + workflow_runs=workflow_runs, + node_executions=MagicMock(), + ) -def test_init_with_session_factory( - sqlite_session_factory: sessionmaker[Session], workflow_run_repository: APIWorkflowRunRepository -) -> None: - with patch("services.workflow_run_service.DifyAPIRepositoryFactory", autospec=True) as repository_factory: - repository_factory.create_api_workflow_run_repository.return_value = workflow_run_repository - - service = WorkflowRunService(sqlite_session_factory) - - assert service._session_factory is sqlite_session_factory - repository_factory.create_api_workflow_run_repository.assert_called_once_with(sqlite_session_factory) - with service._session_factory() as session: - assert session.scalar(text("SELECT 1")) == 1 +def _request_context(*, workspace_id: str = "tenant-1") -> RequestContext: + return RequestContext( + request_id="request-1", + trace_id="trace-1", + account_id="account-1", + active_workspace_id=workspace_id, + ) -def test_init_with_engine_creates_bound_session_factory( - sqlite_engine: Engine, workflow_run_repository: APIWorkflowRunRepository -) -> None: - with patch("services.workflow_run_service.DifyAPIRepositoryFactory", autospec=True) as repository_factory: - repository_factory.create_api_workflow_run_repository.return_value = workflow_run_repository +def test_get_pause_details_returns_none_when_run_is_not_found(workflow_runs: MagicMock) -> None: + workflow_runs.get_pause_record.return_value = None - service = WorkflowRunService(sqlite_engine) + result = _service(workflow_runs).get_pause_details(_request_context(), workflow_run_id="run-1") - assert service._session_factory.kw["bind"] is sqlite_engine - assert service._session_factory.kw["expire_on_commit"] is False - repository_factory.create_api_workflow_run_repository.assert_called_once_with(service._session_factory) - with service._session_factory() as session: - assert session.scalar(text("SELECT 1")) == 1 + assert result is None + workflow_runs.get_pause_record.assert_called_once_with( + workspace_id="tenant-1", + workflow_run_id="run-1", + ) -def test_init_with_default_repository_dependencies(sqlite_session_factory: sessionmaker[Session]) -> None: - service = WorkflowRunService(sqlite_session_factory) +def test_get_pause_details_returns_empty_details_for_non_paused_run(workflow_runs: MagicMock) -> None: + workflow_runs.get_pause_record.return_value = WorkflowRunPauseRecord( + status=WorkflowExecutionStatus.SUCCEEDED, + paused_at=None, + reasons=(), + form_tokens={}, + ) - assert service._session_factory is sqlite_session_factory + result = _service(workflow_runs).get_pause_details(_request_context(), workflow_run_id="run-1") + + assert result == WorkflowRunPauseDetails(paused_at=None, paused_nodes=()) + + +def test_get_pause_details_maps_human_input_and_token(workflow_runs: MagicMock) -> None: + reason = HumanInputRequired( + form_id="form-1", + form_content="Approve?", + node_id="node-1", + node_title="Approval", + ) + paused_at = datetime(2026, 1, 2, 3, 4, 5) + workflow_runs.get_pause_record.return_value = WorkflowRunPauseRecord( + status=WorkflowExecutionStatus.PAUSED, + paused_at=paused_at, + reasons=(reason,), + form_tokens={"form-1": "form-token"}, + ) + + result = _service(workflow_runs).get_pause_details( + _request_context(workspace_id="tenant-context"), + workflow_run_id="run-1", + ) + + assert result == WorkflowRunPauseDetails( + paused_at=paused_at, + paused_nodes=( + WorkflowRunPausedNode( + node_id="node-1", + node_title="Approval", + form_id="form-1", + form_token="form-token", + ), + ), + ) + workflow_runs.get_pause_record.assert_called_once_with( + workspace_id="tenant-context", + workflow_run_id="run-1", + ) + + +def test_get_pause_details_rejects_unsupported_pause_reason(workflow_runs: MagicMock) -> None: + workflow_runs.get_pause_record.return_value = WorkflowRunPauseRecord( + status=WorkflowExecutionStatus.PAUSED, + paused_at=None, + reasons=(SchedulingPause(message="Waiting for external input"),), + form_tokens={}, + ) + + with pytest.raises(NotImplementedError, match="Pause details do not support SchedulingPause"): + _service(workflow_runs).get_pause_details(_request_context(), workflow_run_id="run-1") diff --git a/api/uv.lock b/api/uv.lock index 5575c5fdc2c..ffb54ba5bc8 100644 --- a/api/uv.lock +++ b/api/uv.lock @@ -1330,8 +1330,8 @@ provides-extras = ["server"] [package.metadata.requires-dev] dev = [ - { name = "basedpyright", specifier = ">=1.39.3" }, { name = "coverage", extras = ["toml"], specifier = ">=7.10.7" }, + { name = "pyrefly", specifier = ">=1.2.0" }, { name = "pytest", specifier = ">=9.0.3" }, { name = "pytest-examples", specifier = ">=0.0.18" }, { name = "pytest-mock", specifier = ">=3.14.0" }, diff --git a/dify-agent/Makefile b/dify-agent/Makefile index 0decb13ae3d..0755fd58ba5 100644 --- a/dify-agent/Makefile +++ b/dify-agent/Makefile @@ -10,7 +10,7 @@ help: @echo " make dev - Run the Dify Agent server with reload" @echo " make check - Run Ruff for dify-agent" @echo " make fix - Format and fix Ruff issues" - @echo " make typecheck - Run basedpyright for src, examples, and tests" + @echo " make typecheck - Run pyrefly for src and examples" @echo " make test - Run local tests and docs/example tests" @echo " make update-examples - Rewrite docs example outputs when needed" @echo " make docs - Build MkDocs documentation" @@ -30,7 +30,7 @@ fix: @uv --directory "$(PROJECT_DIR)" run --project . python -m ruff check --fix . typecheck: - @uv --directory "$(PROJECT_DIR)" run --project . basedpyright --level error src examples tests + @uv --directory "$(PROJECT_DIR)" run --project . --group dev --extra server pyrefly check test: @uv --directory "$(PROJECT_DIR)" run --project . --extra server python -m pytest tests diff --git a/dify-agent/pyproject.toml b/dify-agent/pyproject.toml index 58dcc648151..064dfe5dee6 100644 --- a/dify-agent/pyproject.toml +++ b/dify-agent/pyproject.toml @@ -38,12 +38,12 @@ include = ["agenton*", "agenton_collections*", "dify_agent*", "shellctl*"] [tool.setuptools.package-data] "dify_agent.layers" = ["_agent_cli_help.json"] -[tool.pyright] -include = ["src", "examples", "tests"] -venvPath = "." -venv = ".venv" -pythonVersion = "3.12" -extraPaths = ["src", "examples/agenton", "examples/dify_agent"] +[tool.pyrefly] +project-includes = ["src", "examples"] +project-excludes = [".venv", "tests/"] +search-path = ["src", "examples/agenton", "examples/dify_agent"] +python-platform = "linux" +python-version = "3.12.0" [tool.pytest.ini_options] # Several test modules share a basename across directories (test_layer.py, @@ -61,8 +61,8 @@ include = ["src/**/*.py", "examples/**/*.py", "tests/**/*.py", "docs/**/*.py"] [dependency-groups] dev = [ - "basedpyright>=1.39.3", "coverage[toml]>=7.10.7", + "pyrefly>=1.2.0", "pytest>=9.0.3", "pytest-examples>=0.0.18", "pytest-mock>=3.14.0", diff --git a/dify-agent/src/agenton/compositor/core.py b/dify-agent/src/agenton/compositor/core.py index b948ace52e0..eb3763b2f76 100644 --- a/dify-agent/src/agenton/compositor/core.py +++ b/dify-agent/src/agenton/compositor/core.py @@ -18,7 +18,7 @@ from collections import OrderedDict from collections.abc import AsyncGenerator, Mapping, Sequence from contextlib import asynccontextmanager from dataclasses import dataclass -from typing import Any, Generic, cast +from typing import Any, Generic, Self, cast from pydantic import JsonValue @@ -131,7 +131,7 @@ class Compositor(Generic[PromptT, ToolT, LayerPromptT, LayerToolT, UserPromptT, prompt_transformer: CompositorTransformer[LayerPromptT, PromptT] | None = None, user_prompt_transformer: CompositorTransformer[LayerUserPromptT, UserPromptT] | None = None, tool_transformer: CompositorTransformer[LayerToolT, ToolT] | None = None, - ) -> "Compositor[PromptT, ToolT, LayerPromptT, LayerToolT, UserPromptT, LayerUserPromptT]": + ) -> Self: """Create a reusable compositor plan from serializable graph config. ``providers`` resolve graph node ``type`` ids. ``node_providers`` are diff --git a/dify-agent/src/agenton/layers/base.py b/dify-agent/src/agenton/layers/base.py index 73f9f490143..d8f8035c1cd 100644 --- a/dify-agent/src/agenton/layers/base.py +++ b/dify-agent/src/agenton/layers/base.py @@ -226,7 +226,7 @@ class Layer( if deps_type is None and is_generic_template: return if deps_type is not None: - cls.deps_type = deps_type # pyright: ignore[reportAttributeAccessIssue] + cls.deps_type = cast(type[_DepsT], deps_type) if deps_type is None: raise TypeError(f"{cls.__name__} must define deps_type or inherit from Layer[DepsT].") if not isinstance(deps_type, type) or not issubclass(deps_type, LayerDeps): diff --git a/dify-agent/src/dify_agent/agent_stub/protocol/agent_stub.py b/dify-agent/src/dify_agent/agent_stub/protocol/agent_stub.py index 8a65c19d94d..ffd4aca3782 100644 --- a/dify-agent/src/dify_agent/agent_stub/protocol/agent_stub.py +++ b/dify-agent/src/dify_agent/agent_stub/protocol/agent_stub.py @@ -18,7 +18,7 @@ from urllib.parse import urlsplit, urlunsplit from pydantic import AliasChoices, BaseModel, ConfigDict, Field, JsonValue, model_validator -AGENT_STUB_PROTOCOL_VERSION: Final[int] = 1 +AGENT_STUB_PROTOCOL_VERSION: Final[Literal[1]] = 1 AGENT_STUB_API_BASE_URL_ENV_VAR: Final[str] = "DIFY_AGENT_STUB_API_BASE_URL" AGENT_STUB_AUTH_JWE_ENV_VAR: Final[str] = "DIFY_AGENT_STUB_AUTH_JWE" diff --git a/dify-agent/src/dify_agent/layers/ask_human/configs.py b/dify-agent/src/dify_agent/layers/ask_human/configs.py index 432d0aad56c..87fd1a4af0b 100644 --- a/dify-agent/src/dify_agent/layers/ask_human/configs.py +++ b/dify-agent/src/dify_agent/layers/ask_human/configs.py @@ -40,6 +40,10 @@ _HARD_MAX_ACTION_LABEL_CHARS = 120 _FILE_FIELD_TYPES: Final[frozenset[AskHumanFieldType]] = frozenset({"file", "file-list"}) +def _default_allowed_field_types() -> list[AskHumanFieldType]: + return ["paragraph", "select"] + + class DifyAskHumanLayerConfig(LayerConfig): """Public config for the optional ask-human deferred tool layer. @@ -58,7 +62,7 @@ class DifyAskHumanLayerConfig(LayerConfig): tool_description: str | None = None max_fields: int = Field(default=8, ge=0) max_actions: int = Field(default=4, ge=1) - allowed_field_types: list[AskHumanFieldType] = Field(default_factory=lambda: ["paragraph", "select"]) + allowed_field_types: list[AskHumanFieldType] = Field(default_factory=_default_allowed_field_types) allow_file_fields: bool = False max_markdown_chars: int = Field(default=8_000, ge=0) max_question_chars: int = Field(default=1_000, ge=1) diff --git a/dify-agent/src/dify_agent/layers/ask_human/layer.py b/dify-agent/src/dify_agent/layers/ask_human/layer.py index a94de9af72e..399f28ab063 100644 --- a/dify-agent/src/dify_agent/layers/ask_human/layer.py +++ b/dify-agent/src/dify_agent/layers/ask_human/layer.py @@ -20,7 +20,7 @@ from typing import Any, ClassVar, cast from pydantic import JsonValue, ValidationError from pydantic_ai import Tool from pydantic_ai.exceptions import ModelRetry -from pydantic_ai.tools import DeferredToolRequests, RunContext, ToolDefinition +from pydantic_ai.tools import ArgsValidatorFunc, DeferredToolRequests, RunContext, ToolDefinition from typing_extensions import Self, override from agenton.layers import EmptyRuntimeState, NoLayerDeps, PydanticAILayer, PydanticAIPrompt, PydanticAITool @@ -70,7 +70,7 @@ class DifyAskHumanLayer(PydanticAILayer[NoLayerDeps, object, DifyAskHumanLayerCo name=self.config.tool_name, description=self.config.effective_tool_description, prepare=self._prepare_tool_definition, - args_validator=self._validate_tool_args, + args_validator=cast(ArgsValidatorFunc[object, ...], self._validate_tool_args), sequential=True, ) ] diff --git a/dify-agent/src/dify_agent/layers/dify_core_tools/configs.py b/dify-agent/src/dify_agent/layers/dify_core_tools/configs.py index 47184d04b56..19a25898cf8 100644 --- a/dify-agent/src/dify_agent/layers/dify_core_tools/configs.py +++ b/dify-agent/src/dify_agent/layers/dify_core_tools/configs.py @@ -21,6 +21,10 @@ type DifyCoreToolProviderType = Literal["plugin", "builtin", "api", "workflow", DIFY_CORE_TOOLS_LAYER_TYPE_ID: Final[str] = "dify.core.tools" +def _default_parameters_json_schema() -> dict[str, JsonValue]: + return {"type": "object", "properties": {}, "required": []} + + class DifyCoreToolConfig(LayerConfig): """Prepared API-routed tool declaration exposed to the model.""" @@ -32,9 +36,7 @@ class DifyCoreToolConfig(LayerConfig): description: str | None = None runtime_parameters: dict[str, JsonValue] = Field(default_factory=dict) parameters: list[DifyPluginToolParameter] = Field(default_factory=list) - parameters_json_schema: dict[str, JsonValue] = Field( - default_factory=lambda: {"type": "object", "properties": {}, "required": []} - ) + parameters_json_schema: dict[str, JsonValue] = Field(default_factory=_default_parameters_json_schema) model_config: ClassVar[ConfigDict] = ConfigDict(extra="forbid", arbitrary_types_allowed=True) diff --git a/dify-agent/src/dify_agent/layers/dify_plugin/configs.py b/dify-agent/src/dify_agent/layers/dify_plugin/configs.py index 3d2ac615e5a..ad289c055d8 100644 --- a/dify-agent/src/dify_agent/layers/dify_plugin/configs.py +++ b/dify-agent/src/dify_agent/layers/dify_plugin/configs.py @@ -30,6 +30,10 @@ DIFY_PLUGIN_LLM_LAYER_TYPE_ID: Final[str] = "dify.plugin.llm" DIFY_PLUGIN_TOOLS_LAYER_TYPE_ID: Final[str] = "dify.plugin.tools" +def _default_parameters_json_schema() -> dict[str, JsonValue]: + return {"type": "object", "properties": {}, "required": []} + + class DifyPluginToolOption(BaseModel): """Selectable tool option value exposed to the model. @@ -151,9 +155,7 @@ class DifyPluginToolConfig(LayerConfig): credentials: dict[str, DifyPluginCredentialValue] = Field(default_factory=dict) runtime_parameters: dict[str, DifyPluginToolValue] = Field(default_factory=dict) parameters: list[DifyPluginToolParameter] = Field(default_factory=list) - parameters_json_schema: dict[str, JsonValue] = Field( - default_factory=lambda: {"type": "object", "properties": {}, "required": []} - ) + parameters_json_schema: dict[str, JsonValue] = Field(default_factory=_default_parameters_json_schema) model_config: ClassVar[ConfigDict] = ConfigDict(extra="forbid", arbitrary_types_allowed=True) diff --git a/dify-agent/src/dify_agent/runtime/compositor_factory.py b/dify-agent/src/dify_agent/runtime/compositor_factory.py index 5563d43a706..f56b37e3c17 100644 --- a/dify-agent/src/dify_agent/runtime/compositor_factory.py +++ b/dify-agent/src/dify_agent/runtime/compositor_factory.py @@ -29,7 +29,7 @@ snapshots. from __future__ import annotations from collections.abc import Mapping, Sequence -from typing import Any, cast +from typing import Any from pydantic_ai.messages import UserContent @@ -164,21 +164,18 @@ def build_pydantic_ai_compositor( selected provider set explicitly so provider defaulting stays at outer runtime boundaries rather than being duplicated here. """ - return cast( - Compositor[ - PydanticAIPrompt[object], - PydanticAITool[object], - AllPromptTypes, - AllToolTypes, - UserContent, - AllUserPromptTypes, - ], - Compositor.from_config( - config, - providers=providers, - node_providers=node_providers, - **PYDANTIC_AI_TRANSFORMERS, # pyright: ignore[reportArgumentType] - ), + return Compositor[ + PydanticAIPrompt[object], + PydanticAITool[object], + AllPromptTypes, + AllToolTypes, + UserContent, + AllUserPromptTypes, + ].from_config( + config, + providers=providers, + node_providers=node_providers, + **PYDANTIC_AI_TRANSFORMERS, ) diff --git a/dify-agent/src/dify_agent/server/app.py b/dify-agent/src/dify_agent/server/app.py index 4b1d4ce3782..5af355f376c 100644 --- a/dify-agent/src/dify_agent/server/app.py +++ b/dify-agent/src/dify_agent/server/app.py @@ -16,6 +16,7 @@ environment configuration provides a token. from collections.abc import AsyncGenerator from contextlib import asynccontextmanager +from typing_extensions import cast import httpx from fastapi import APIRouter, FastAPI @@ -132,10 +133,10 @@ def create_app(settings: ServerSettings | None = None) -> FastAPI: configure_server_observability(app) def get_store() -> RedisRunStore: - return state["store"] # pyright: ignore[reportReturnType] + return cast(RedisRunStore, state["store"]) def get_scheduler() -> RunScheduler: - return state["scheduler"] # pyright: ignore[reportReturnType] + return cast(RunScheduler, state["scheduler"]) control_plane_router = APIRouter( dependencies=[create_bearer_token_dependency(resolved_settings.api_token)], diff --git a/dify-agent/src/dify_agent/server/binding_files.py b/dify-agent/src/dify_agent/server/binding_files.py index a8d394bba3b..8fee4be39c1 100644 --- a/dify-agent/src/dify_agent/server/binding_files.py +++ b/dify-agent/src/dify_agent/server/binding_files.py @@ -109,9 +109,15 @@ data = data[:max_bytes] try: text = data.decode("utf-8") binary = False -except UnicodeDecodeError: - text = None - binary = True +except UnicodeDecodeError as exc: + # Truncation may split a trailing multi-byte code point. Keep the longest + # complete UTF-8 prefix; genuine invalid bytes still classify as binary. + if truncated and exc.reason == "unexpected end of data" and exc.end == len(data): + text = data[: exc.start].decode("utf-8") + binary = False + else: + text = None + binary = True payload = { "path": response_path, "size": size, diff --git a/dify-agent/tests/local/dify_agent/server/test_binding_files.py b/dify-agent/tests/local/dify_agent/server/test_binding_files.py index 68c7f38a3e0..1df092289d2 100644 --- a/dify-agent/tests/local/dify_agent/server/test_binding_files.py +++ b/dify-agent/tests/local/dify_agent/server/test_binding_files.py @@ -317,6 +317,66 @@ async def test_real_read_script_handles_boundary_truncation_and_binary(tmp_path: assert backend.releases == 3 +@pytest.mark.anyio +async def test_real_read_script_handles_utf8_truncation_at_multibyte_boundary(tmp_path: Path) -> None: + service, backend, commands, workspace, _ = _local_service(tmp_path) + + latin = "ñ".encode("utf-8") # 2-byte: c3 b1 + euro = "€".encode("utf-8") # 3-byte: e2 82 ac + smile = "😀".encode("utf-8") # 4-byte: f0 9f 98 80 + cjk = "中".encode("utf-8") # 3-byte: e4 b8 ad + + (workspace / "latin.txt").write_bytes(b"a" + latin) + (workspace / "euro.txt").write_bytes(b"a" + euro) + (workspace / "smile.txt").write_bytes(b"a" + smile) + (workspace / "cjk.txt").write_bytes(b"a" + cjk) + (workspace / "invalid.txt").write_bytes(b"a\xff" + euro) + + latin_result = await service.read_file( + BindingFileReadRequest(backend_binding_ref="binding-ref", path="latin.txt", max_bytes=2) + ) + euro_result = await service.read_file( + BindingFileReadRequest(backend_binding_ref="binding-ref", path="euro.txt", max_bytes=2) + ) + smile_result = await service.read_file( + BindingFileReadRequest(backend_binding_ref="binding-ref", path="smile.txt", max_bytes=3) + ) + cjk_result = await service.read_file( + BindingFileReadRequest(backend_binding_ref="binding-ref", path="cjk.txt", max_bytes=3) + ) + invalid_result = await service.read_file( + BindingFileReadRequest(backend_binding_ref="binding-ref", path="invalid.txt", max_bytes=4) + ) + + assert latin_result.size == 3 + assert latin_result.truncated is True + assert latin_result.binary is False + assert latin_result.text == "a" + + assert euro_result.size == 4 + assert euro_result.truncated is True + assert euro_result.binary is False + assert euro_result.text == "a" + + assert smile_result.size == 5 + assert smile_result.truncated is True + assert smile_result.binary is False + assert smile_result.text == "a" + + assert cjk_result.size == 4 + assert cjk_result.truncated is True + assert cjk_result.binary is False + assert cjk_result.text == "a" + + assert invalid_result.size == 5 + assert invalid_result.truncated is True + assert invalid_result.binary is True + assert invalid_result.text is None + + assert commands.deletes == ["local-job-1", "local-job-2", "local-job-3", "local-job-4", "local-job-5"] + assert backend.releases == 5 + + @pytest.mark.anyio async def test_real_browse_script_output_over_command_cap_normalizes_to_unavailable( tmp_path: Path, diff --git a/dify-agent/tests/local/test_packaging.py b/dify-agent/tests/local/test_packaging.py index d5f767dcdfd..c2b82a21d01 100644 --- a/dify-agent/tests/local/test_packaging.py +++ b/dify-agent/tests/local/test_packaging.py @@ -29,8 +29,8 @@ SERVER_RUNTIME_DEPENDENCIES = { } DEV_DEPENDENCIES = { - "basedpyright>=1.39.3", "coverage[toml]>=7.10.7", + "pyrefly>=1.2.0", "pytest>=9.0.3", "pytest-examples>=0.0.18", "pytest-mock>=3.14.0", diff --git a/dify-agent/uv.lock b/dify-agent/uv.lock index 282cdbd1917..88f8ccb3784 100644 --- a/dify-agent/uv.lock +++ b/dify-agent/uv.lock @@ -109,18 +109,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/3e/5c/fb93d3092640a24dfb7bd7727a24016d7c01774ca013e60efd3f683c8002/backrefs-7.0-py314-none-any.whl", hash = "sha256:a6448b28180e3ca01134c9cf09dcebafad8531072e09903c5451748a05f24bc9", size = 412349, upload-time = "2026-04-28T16:28:02.412Z" }, ] -[[package]] -name = "basedpyright" -version = "1.39.3" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "nodejs-wheel-binaries" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/04/19/5a5b9b9197973da732638957be3a65cf514d2f5a4964eeedbf33b6c65bbd/basedpyright-1.39.3.tar.gz", hash = "sha256:2f794e6b5f4260fb89f614ca6cd23c6f305373bb6b50c4ed7794ff2ae647fb14", size = 25503187, upload-time = "2026-04-20T22:14:47.424Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/54/5c/f950c1239ad26f3bb453e665428a2cf1893995de725a5eb0b64a2520b366/basedpyright-1.39.3-py3-none-any.whl", hash = "sha256:aba760dc83307727554f936d6b4381caa14482f30dbc2173167710e217c1f7ab", size = 12419181, upload-time = "2026-04-20T22:14:51.975Z" }, -] - [[package]] name = "beautifulsoup4" version = "4.14.3" @@ -627,8 +615,8 @@ server = [ [package.dev-dependencies] dev = [ - { name = "basedpyright" }, { name = "coverage" }, + { name = "pyrefly" }, { name = "pytest" }, { name = "pytest-examples" }, { name = "pytest-mock" }, @@ -664,8 +652,8 @@ provides-extras = ["server"] [package.metadata.requires-dev] dev = [ - { name = "basedpyright", specifier = ">=1.39.3" }, { name = "coverage", extras = ["toml"], specifier = ">=7.10.7" }, + { name = "pyrefly", specifier = ">=1.2.0" }, { name = "pytest", specifier = ">=9.0.3" }, { name = "pytest-examples", specifier = ">=0.0.18" }, { name = "pytest-mock", specifier = ">=3.14.0" }, @@ -707,7 +695,7 @@ wheels = [ [[package]] name = "e2b" -version = "2.38.0" +version = "2.46.4" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "attrs" }, @@ -723,9 +711,9 @@ dependencies = [ { name = "typing-extensions" }, { name = "wcmatch" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/e2/e2/1acb7bd6e16811a3affc89521304420410f97040a0a386497b101f898de3/e2b-2.38.0.tar.gz", hash = "sha256:b245976cc946d144baf77341b19fa537bfd7cbd9028dc8ef59bd6947f5e329c1", size = 205432, upload-time = "2026-08-10T17:57:07.643Z" } +sdist = { url = "https://files.pythonhosted.org/packages/7d/04/3e7d3ca8abaeffcfc41d36b9d6fe3aa055cecf3b56234e051991de848b2d/e2b-2.46.4.tar.gz", hash = "sha256:777d35beb00194b401a6c4ac2498aa836628839649f17aae86e85920382c8f6c", size = 228801, upload-time = "2026-09-02T20:48:36.725Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/03/fa/e6d50d307a9dcde63038589fce7c1cb32ce3efde3d70c20a6adb11eee3c6/e2b-2.38.0-py3-none-any.whl", hash = "sha256:9f43803aa23e9a33ed48c1c778d588f83c0390db21edf3098255137981541b67", size = 365646, upload-time = "2026-08-10T17:57:08.947Z" }, + { url = "https://files.pythonhosted.org/packages/a0/b5/585a30223d9ee95bc94f32f8d5358b147bf93832e37b809545a3ab7892a0/e2b-2.46.4-py3-none-any.whl", hash = "sha256:5d77e9fee8571a0d24a179f14244984127d18bd93a99a5e90226c876087035b5", size = 403067, upload-time = "2026-09-02T20:48:35.082Z" }, ] [[package]] @@ -922,15 +910,15 @@ wheels = [ [[package]] name = "h2" -version = "4.3.0" +version = "4.4.1" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "hpack" }, { name = "hyperframe" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/1d/17/afa56379f94ad0fe8defd37d6eb3f89a25404ffc71d4d848893d270325fc/h2-4.3.0.tar.gz", hash = "sha256:6c59efe4323fa18b47a632221a1888bd7fde6249819beda254aeca909f221bf1", size = 2152026, upload-time = "2025-08-23T18:12:19.778Z" } +sdist = { url = "https://files.pythonhosted.org/packages/e7/85/7c366e69d84c17bb778fe41419e1fbcce3033d5b7ce29bbffff0a98b859f/h2-4.4.1.tar.gz", hash = "sha256:4e866ffb1a869ae14dd9b5e6beb5c24a13da0495ad72b65925ded182521c1516", size = 2157281, upload-time = "2026-08-03T11:45:09.509Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/69/b2/119f6e6dcbd96f9069ce9a2665e0146588dc9f88f29549711853645e736a/h2-4.3.0-py3-none-any.whl", hash = "sha256:c438f029a25f7945c69e0ccf0fb951dc3f73a5f6412981daee861431b70e2bdd", size = 61779, upload-time = "2025-08-23T18:12:17.779Z" }, + { url = "https://files.pythonhosted.org/packages/7e/22/e85faf23bd72a92d1921e37d674ca56eb298a3c8be31fdecef0ff2b3aaac/h2-4.4.1-py3-none-any.whl", hash = "sha256:0e25f1462b23c9cb82d9eb02e28bc706dac2a68cb457c6a0d74d63c8a2a5d0e6", size = 62636, upload-time = "2026-08-03T11:44:59.164Z" }, ] [[package]] @@ -967,11 +955,11 @@ wheels = [ [[package]] name = "hpack" -version = "4.1.0" +version = "4.2.0" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/2c/48/71de9ed269fdae9c8057e5a4c0aa7402e8bb16f2c6e90b3aa53327b113f8/hpack-4.1.0.tar.gz", hash = "sha256:ec5eca154f7056aa06f196a557655c5b009b382873ac8d1e66e79e87535f1dca", size = 51276, upload-time = "2025-01-22T21:44:58.347Z" } +sdist = { url = "https://files.pythonhosted.org/packages/26/5b/fcabf6028144a8723726318b07a32c2f3314acdff6265743cf08a344b18e/hpack-4.2.0.tar.gz", hash = "sha256:0895cfa3b5531fc65fe439c05eb65144f123bf7a394fcaa56aa423548d8e45c0", size = 51300, upload-time = "2026-06-23T18:34:46.667Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/07/c6/80c95b1b2b94682a72cbdbfb85b81ae2daffa4291fbfa1b1464502ede10d/hpack-4.1.0-py3-none-any.whl", hash = "sha256:157ac792668d995c657d93111f46b4535ed114f0c9c8d672271bbec7eae1b496", size = 34357, upload-time = "2025-01-22T21:44:56.92Z" }, + { url = "https://files.pythonhosted.org/packages/71/b4/4a9fcfb2aef6ba44d9073ecd301443aa00b3dac95de5619f2a7de7ec8a91/hpack-4.2.0-py3-none-any.whl", hash = "sha256:858ac0b02280fa582b5080d68db0899c62a80375e0e5413a74970c5e518b6986", size = 34246, upload-time = "2026-06-23T18:34:45.472Z" }, ] [[package]] @@ -1593,7 +1581,7 @@ wheels = [ [[package]] name = "mkdocs-material" -version = "9.7.6" +version = "9.7.7" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "babel" }, @@ -1608,9 +1596,9 @@ dependencies = [ { name = "pymdown-extensions" }, { name = "requests" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/45/29/6d2bcf41ae40802c4beda2432396fff97b8456fb496371d1bc7aad6512ec/mkdocs_material-9.7.6.tar.gz", hash = "sha256:00bdde50574f776d328b1862fe65daeaf581ec309bd150f7bff345a098c64a69", size = 4097959, upload-time = "2026-03-19T15:41:58.161Z" } +sdist = { url = "https://files.pythonhosted.org/packages/f1/cd/c05d3a530ba7934f144fb45f7203cd236adc25c7bdcc34673d202f4b0278/mkdocs_material-9.7.7.tar.gz", hash = "sha256:c0649c065b1b0512d60aad8c10f947f8e455284475239b364b610f2deb4d0855", size = 4097923, upload-time = "2026-07-17T16:21:33.156Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/2c/01/bc663630c510822c95c47a66af9fa7a443c295b47d5f041e5e6ae62ef659/mkdocs_material-9.7.6-py3-none-any.whl", hash = "sha256:71b84353921b8ea1ba84fe11c50912cc512da8fe0881038fcc9a0761c0e635ba", size = 9305470, upload-time = "2026-03-19T15:41:55.217Z" }, + { url = "https://files.pythonhosted.org/packages/ad/21/17c1bc9e6f47c972ad66fb2ac2568f99f90f1207eeb6fc3b34d094dba7b5/mkdocs_material-9.7.7-py3-none-any.whl", hash = "sha256:8ea9bb1737a5b524a5f9dcf2e1b4ebda8274ae3008aa7845720a97083bef708f", size = 9305438, upload-time = "2026-07-17T16:21:30.017Z" }, ] [[package]] @@ -1710,22 +1698,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl", hash = "sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505", size = 4963, upload-time = "2025-04-22T14:54:22.983Z" }, ] -[[package]] -name = "nodejs-wheel-binaries" -version = "24.15.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/3f/70/a1e4f4d5986768ab90cc860b1cc3660fd2ded74ca175a900a5c29f839c7d/nodejs_wheel_binaries-24.15.0.tar.gz", hash = "sha256:b43f5c4f6e5768d8845b2ae4682eb703a19bf7aadc84187e2d903ed3a611c859", size = 8057, upload-time = "2026-04-19T15:48:16.899Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/85/66/54051d14853d6ab4fb85f8be9b042b530be653357fb9a19557498bc91ab7/nodejs_wheel_binaries-24.15.0-py2.py3-none-macosx_13_0_arm64.whl", hash = "sha256:a6232fa8b754220941f52388c8ead923f7c1c7fdf0ea0d98f657523bd9a81ef4", size = 55173485, upload-time = "2026-04-19T15:47:34.561Z" }, - { url = "https://files.pythonhosted.org/packages/ad/5f/66acada164da5ca10a0824db021aa7394ae18396c550cd9280e839a43126/nodejs_wheel_binaries-24.15.0-py2.py3-none-macosx_13_0_x86_64.whl", hash = "sha256:001a6b62c69d9109c1738163cca00608dd2722e8663af59300054ea02610972d", size = 55348100, upload-time = "2026-04-19T15:47:40.521Z" }, - { url = "https://files.pythonhosted.org/packages/0d/2d/0cbd5ff40c9bb030ca1735d8f8793bd74f08a4cbd49100a1d19313ea57ab/nodejs_wheel_binaries-24.15.0-py2.py3-none-manylinux_2_28_aarch64.whl", hash = "sha256:0fbc48765e60ed0ff30d43898dbf5cadbadf2e5f1e7f204afc2b01493b7ebce6", size = 59668206, upload-time = "2026-04-19T15:47:46.848Z" }, - { url = "https://files.pythonhosted.org/packages/da/d5/91ac63951ec75927a486b83b8cafe650e360fa70ac01dc94adfb32b93b97/nodejs_wheel_binaries-24.15.0-py2.py3-none-manylinux_2_28_x86_64.whl", hash = "sha256:20ee0536809795da8a4942fc1ab4cbdebbcaaf29383eab67ba8874268fb00008", size = 60206736, upload-time = "2026-04-19T15:47:52.668Z" }, - { url = "https://files.pythonhosted.org/packages/db/72/dc22776974d928869c0c30d23ee98ed7df254243c2df68f09f5963e8e8b8/nodejs_wheel_binaries-24.15.0-py2.py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:1fade6c214285e72472ca40a631e98ff36559671cd5eefc8bf009471d67f04b4", size = 61720456, upload-time = "2026-04-19T15:47:58.325Z" }, - { url = "https://files.pythonhosted.org/packages/01/0a/34461b9050cb45ee371dccdefc622aef6351506ea2691b08fc761ca67150/nodejs_wheel_binaries-24.15.0-py2.py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:3984cb8d87766567aee67a49743227ab40ede6f47734ec990ff90e50b74e7740", size = 62326172, upload-time = "2026-04-19T15:48:04.094Z" }, - { url = "https://files.pythonhosted.org/packages/c9/17/09252bf35672dba926649d59dfe51443a0f6955ad13784e91131d5ec82a2/nodejs_wheel_binaries-24.15.0-py2.py3-none-win_amd64.whl", hash = "sha256:a437601956b532dcb3082046e6978e622733f90edc0932cbb9adb3bb97a16501", size = 41543461, upload-time = "2026-04-19T15:48:09.332Z" }, - { url = "https://files.pythonhosted.org/packages/0a/7e/b649777d148e1e0c2ce349156603cdb12f7ed99921b95d93717393650193/nodejs_wheel_binaries-24.15.0-py2.py3-none-win_arm64.whl", hash = "sha256:bdf4a431e08321a32efc604111c6f23941f87055d796a537e8c4110daecad23f", size = 39233248, upload-time = "2026-04-19T15:48:13.326Z" }, -] - [[package]] name = "numba" version = "0.65.1" @@ -2687,48 +2659,67 @@ wheels = [ [[package]] name = "pyqwest" -version = "0.9.0" +version = "0.10.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "opentelemetry-api" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/8d/59/97531fd9d0a06d54e84f5493775739b0c1d0d13ec704974d04fa423a3332/pyqwest-0.9.0.tar.gz", hash = "sha256:514dd0b37d7a1bcb978b5d6423d15f89cf7dcf8058ac2a37aa42cd30090de89e", size = 477462, upload-time = "2026-08-10T01:55:36.219Z" } +sdist = { url = "https://files.pythonhosted.org/packages/07/ee/0ff9facfa9e7a4f6df2a770d4eaf1ad0f74165da7e8c28e888461f07604c/pyqwest-0.10.0.tar.gz", hash = "sha256:6c1a693be17d57d2c2eca4085e32c2809c53090c16719a907c90ebcf1f40dc01", size = 482248, upload-time = "2026-08-21T06:09:20.656Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/47/34/d556fa07d9bff21fc6c9c16738614c6336164b65d9ced1ce9f1c1044aaeb/pyqwest-0.9.0-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:517d2eb295d56ef1530cebe0cf290a66fa199c57955bd1c5f1dc592240fbb7d6", size = 5224594, upload-time = "2026-08-10T01:54:18.646Z" }, - { url = "https://files.pythonhosted.org/packages/c9/3f/1d7d1a08ebf8838c0c85e99ab5db5c1aae457f76ee03b60c8f4fc4948e01/pyqwest-0.9.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:408dc692835363b6824bee61ee9e4c9a5bba9c08d75b903f82f9a802e41d1b30", size = 5100753, upload-time = "2026-08-10T01:54:20.669Z" }, - { url = "https://files.pythonhosted.org/packages/66/fe/22c4056b526baa4c2c953f4e99925ce0bb14e7747d66e6ac3448a34f76e5/pyqwest-0.9.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e1abf1ab3a0c1b545651ab37a643acc7909c4e7644a63a2e4af36e3e230a0db5", size = 5619882, upload-time = "2026-08-10T01:54:22.468Z" }, - { url = "https://files.pythonhosted.org/packages/de/2c/8f1159a30bbc905b2fcbc386740a260e76ae64372feb263103438ef5201c/pyqwest-0.9.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9b02e5c9ce57e079eb3716c464d3818ad2573d7743483c39526ad77ee95af806", size = 5564744, upload-time = "2026-08-10T01:54:24.814Z" }, - { url = "https://files.pythonhosted.org/packages/ac/fd/ff15034d7874fd208d606bd80c924fb0b1091db8159c892672168b656de1/pyqwest-0.9.0-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:98bdaa30a3a8d8c71506a33c084e485a243fbf51e77130bb39e0f249dd116142", size = 5779373, upload-time = "2026-08-10T01:54:26.753Z" }, - { url = "https://files.pythonhosted.org/packages/ed/b5/eeafdeb65ad8fe9c39044aedcfc97610f4e5c7b5620ca9fe2d504f669108/pyqwest-0.9.0-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:10eba84f9d8512d23bd16dd60b1c77dab956470361559214981dda065df4ad1a", size = 5973306, upload-time = "2026-08-10T01:54:28.729Z" }, - { url = "https://files.pythonhosted.org/packages/34/db/a8979dab5c591529f858d1505d09b3c8b6dec337921f15bed4858c55716e/pyqwest-0.9.0-cp310-abi3-win_amd64.whl", hash = "sha256:af4e859800b02cd1fcdd898af26a881c4a8ce9520f71f6cb0a5403523d2cbbdf", size = 4834706, upload-time = "2026-08-10T01:54:30.335Z" }, - { url = "https://files.pythonhosted.org/packages/1d/9e/d070ca1ce3c202952770436172889a4b0556a564a490baf726f7bb895eba/pyqwest-0.9.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:abe322cc1b63947b493bc06028615e0d5de655928159227aab64f60a0eb0e27d", size = 5244745, upload-time = "2026-08-10T01:54:32.257Z" }, - { url = "https://files.pythonhosted.org/packages/41/97/ebded0bb35a79f51bb2d1519e34922d79c8825f9e71b696bb20f387a08c2/pyqwest-0.9.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:c6b2dc6a9c583859d031941e1041b8e711afe0ad4c130628207731cef2b1a486", size = 5101044, upload-time = "2026-08-10T01:54:34.088Z" }, - { url = "https://files.pythonhosted.org/packages/a9/2c/a7f117e793b4e27642807bf7230b9fa297ff1e1ac489fc260fc3d4350ae4/pyqwest-0.9.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:551e47714da72a72939958029eb37a754cecd974344471c4a2038583a66a5396", size = 5630841, upload-time = "2026-08-10T01:54:35.885Z" }, - { url = "https://files.pythonhosted.org/packages/b4/b4/be47fc141529941352a848cd77da581803137b733993a51bf464f2c48915/pyqwest-0.9.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3be3d38ccdab3077bf1cfe90c346ce927f9b49c56dcdf20e060f5f2502d01021", size = 5572909, upload-time = "2026-08-10T01:54:37.653Z" }, - { url = "https://files.pythonhosted.org/packages/cb/fd/703635e12710ce7ef5e961f5e1b22c609aecc8f07eedd79477b48d1edb74/pyqwest-0.9.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:45d9d424da878b22604799d1ce2a0a5df054924487ee5a851ad08025e2076e5c", size = 5794049, upload-time = "2026-08-10T01:54:39.363Z" }, - { url = "https://files.pythonhosted.org/packages/3c/15/941aad56b108c743df9d41e124be82ecb6be0eba1007e945b6a39f5dd1a0/pyqwest-0.9.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e4420adcd2b756da706b31a04f095a521f6e440dab284b7a1b46c76e048c1f78", size = 5974542, upload-time = "2026-08-10T01:54:41.202Z" }, - { url = "https://files.pythonhosted.org/packages/b7/86/8445fe05b47322047347464a909ae95dba023bf8409b9967f66bda2905ad/pyqwest-0.9.0-cp312-cp312-win_amd64.whl", hash = "sha256:25734bd9798bbfaa53c83d395649dd9ea9a791a64b9848734955cb78ad52c832", size = 4847034, upload-time = "2026-08-10T01:54:42.899Z" }, - { url = "https://files.pythonhosted.org/packages/a7/92/013fc3c94f0dc4eeca556431797da477d76ad43d69134837ba149468dcc3/pyqwest-0.9.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:6e4c2afe6c7293d9e42cfdb0489b0b5f64002159c2dc9c45d9322c1b4ffb42cd", size = 5244793, upload-time = "2026-08-10T01:54:44.891Z" }, - { url = "https://files.pythonhosted.org/packages/48/61/c685e0c595f3f1b7842a776c48aaf2b3f47270a61672967d9d38f7c5e7f6/pyqwest-0.9.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:67f153b94dfbb9aeabc38f20a1bada1acfd9361bec12db86268bd951be35a154", size = 5100715, upload-time = "2026-08-10T01:54:46.777Z" }, - { url = "https://files.pythonhosted.org/packages/f9/a1/a28b2cd704d7dd37043f1e6b398798576c12293f3efae84f00e7f6a5625c/pyqwest-0.9.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b9942ffa243c8c2243e3ce04e3e3ec9b4c119346c963fd0051cd989f0defecf6", size = 5628787, upload-time = "2026-08-10T01:54:48.488Z" }, - { url = "https://files.pythonhosted.org/packages/36/31/6255c7f17cd00e39f1c4df3003cea2d2d087e483fb4c49a6535307cd21e0/pyqwest-0.9.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7f8f9c880c19826fead1838e12b042865b15df3ec844928cc74188a8b740f761", size = 5571946, upload-time = "2026-08-10T01:54:50.181Z" }, - { url = "https://files.pythonhosted.org/packages/37/71/344738bdb38bddbc07cb29e5b1287c08db9e53d6b322581492571bdef748/pyqwest-0.9.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:56040761f6ca7abb036c7d288863bf143a9b166d824d98a604908ef20c6906e0", size = 5792872, upload-time = "2026-08-10T01:54:52.045Z" }, - { url = "https://files.pythonhosted.org/packages/fc/ee/c5c74e3afb6dd0a25bca23bb4757d1a7135fab09a0a054036e837d49dd11/pyqwest-0.9.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:02a51df744521e45eb5379bc325b91de4512ec64a9f4aba3248f9166ec4e5b88", size = 5974192, upload-time = "2026-08-10T01:54:53.818Z" }, - { url = "https://files.pythonhosted.org/packages/6e/08/302960836e3e43ca3051313ee3b767534d1610bd3b8033c7bbad41b06e1b/pyqwest-0.9.0-cp313-cp313-win_amd64.whl", hash = "sha256:e9a0e3123aee446d5f6e57c43cae87b4dab4b1af3fa970d114d74d5d4b9d075c", size = 4847050, upload-time = "2026-08-10T01:54:56.155Z" }, - { url = "https://files.pythonhosted.org/packages/fc/3e/a36940844325fbc59adf92116d3bcd1155cc2c72d1a390880886ca492bb4/pyqwest-0.9.0-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:fdab21bbedff4a21209135423f5f54f287a01dd6143cbd7aeab63d5d8c889654", size = 5246741, upload-time = "2026-08-10T01:54:57.697Z" }, - { url = "https://files.pythonhosted.org/packages/41/8d/efb9741449a81f2489f6929a7691a2145662c3650e06bcb139c64f543d0a/pyqwest-0.9.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:21a04ddf4497ef3c7fe36cdc473c094999b7710005d38bd8a0dfd4a3feace12f", size = 5102423, upload-time = "2026-08-10T01:54:59.253Z" }, - { url = "https://files.pythonhosted.org/packages/8f/a8/0b667cf9c07e62861cfe24c09ad30357a9c2a0fc3863cfc5610af5b66e3e/pyqwest-0.9.0-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c8c51c1a27ea529fffab90c127a045f0dfe1abc54d5b9ad5f4165e954439c9b2", size = 5627663, upload-time = "2026-08-10T01:55:01.036Z" }, - { url = "https://files.pythonhosted.org/packages/5f/7c/1f88be3d35afe92b79e361aa40faf3904e3a7852e8638a9d3d5c00891ab3/pyqwest-0.9.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:10f283ffc40a2774e0dc8fbb47e4a8f9f8a698a99feae3bf4ad75b340c98031b", size = 5572750, upload-time = "2026-08-10T01:55:03.163Z" }, - { url = "https://files.pythonhosted.org/packages/21/3e/98c1e151772e77dcf00cb8b1c070f2c8edcb15fe12fdee30c1022006ca8e/pyqwest-0.9.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:1f73d4a8e75b5860741fbd0623a0ce792a53336dbfec5f34736004acd5446843", size = 5790615, upload-time = "2026-08-10T01:55:04.758Z" }, - { url = "https://files.pythonhosted.org/packages/7b/f1/12bb4b119cffb7fe5301b2d997ec9ffc1719637f6f4faf2ff2315e13b201/pyqwest-0.9.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:e34d55a53492b44b82b690c2b11957fc35e99ebab4db2e9bcb2a1e2621d71b62", size = 5976516, upload-time = "2026-08-10T01:55:06.453Z" }, - { url = "https://files.pythonhosted.org/packages/53/55/046dbf68ccedd816770fd212c6c2612372a6f0d7ecf2d8273480db460ddf/pyqwest-0.9.0-cp314-cp314-win_amd64.whl", hash = "sha256:48c367150783f6866d0af43c209cc405f8287743d7b2472c7d757fda2478fd57", size = 4841470, upload-time = "2026-08-10T01:55:08.035Z" }, - { url = "https://files.pythonhosted.org/packages/55/cd/38ff77d145385afd71b2dbd6eb116251ce3ffddbed0edde16c64394fbec2/pyqwest-0.9.0-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:10fae206270355c993b49c49bf9778995167f5efe591807d50a8e682921d71ea", size = 5222689, upload-time = "2026-08-10T01:55:09.639Z" }, - { url = "https://files.pythonhosted.org/packages/82/95/d8c6804eeb72b7b89842b58f1f05d5cfda5be410c4a22442d2574d510d39/pyqwest-0.9.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:5f475129fb792593222a9ce0fefbbc10bda1544a7411f3946d771c9a61c50e61", size = 5085846, upload-time = "2026-08-10T01:55:11.569Z" }, - { url = "https://files.pythonhosted.org/packages/f2/e7/04fe8962ec416be0199093ebdd326653a5a7b8f53ddfdb29af81756f7c08/pyqwest-0.9.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:78f6f676d349535cb094e8e3bcf33863238a142fb7dcc7afb9e61749e2351047", size = 5613296, upload-time = "2026-08-10T01:55:13.315Z" }, - { url = "https://files.pythonhosted.org/packages/03/0c/7a11728155a34838d2e7c78fb24c937f2cd844ec791ef416030c52f4bb5e/pyqwest-0.9.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f460f5fcdae8cc46e6a679128836f6cd47fbf4bf73e5c9d6ea868e4a90b2e7ee", size = 5557492, upload-time = "2026-08-10T01:55:15.103Z" }, - { url = "https://files.pythonhosted.org/packages/4a/d5/8c6e68f0e978b5257b309809fec00d946c1ed20dacef97f25c23de5b91a9/pyqwest-0.9.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:cba6804151e973d27c99b9f02b3f0bcab7270c3f99725b508ada9c147365cabd", size = 5774204, upload-time = "2026-08-10T01:55:16.932Z" }, - { url = "https://files.pythonhosted.org/packages/17/20/fed89b62d50b4efad2ebe78c3867e0e402f824726652034728a5dd0a342e/pyqwest-0.9.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:bc9dfeca326918d70a5368114de0fc627d9f64fafa36563ab484ae58ffe42c0d", size = 5964506, upload-time = "2026-08-10T01:55:19.031Z" }, - { url = "https://files.pythonhosted.org/packages/11/de/30575019efebae84502269d097e706e89a30631e3719218b053e4e0dc179/pyqwest-0.9.0-cp314-cp314t-win_amd64.whl", hash = "sha256:384693adc0908f16324e5b508a3c25fb9a92b88f9bdf7ccb6820507db4a95888", size = 4809013, upload-time = "2026-08-10T01:55:21.042Z" }, + { url = "https://files.pythonhosted.org/packages/43/ee/b1a28f57c689606cfd065d8a553841150f7daaa91d20e58dcc2c5ea191f8/pyqwest-0.10.0-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:aa492d5777dd145a60795ed95d9d4707a3cd1091fdcdfc93a82ac7fdc43ebacd", size = 5261059, upload-time = "2026-08-21T06:08:04.999Z" }, + { url = "https://files.pythonhosted.org/packages/dc/13/9c5046cfd6ef705bde0b620ba8a794335bcabc0839342a2a647f2427b27e/pyqwest-0.10.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:59f3f16628e518c674102e7b5fcff2101bba6abb4f6737ec5fade9b9278e6a53", size = 5134207, upload-time = "2026-08-21T06:08:06.955Z" }, + { url = "https://files.pythonhosted.org/packages/93/7d/50021dd88d82d6966ab1c27593ceaee9d1ed62fbe597c40e8dc187cfa5fd/pyqwest-0.10.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d6e7db305a8318b1f3218053e87501f8f245ca8bd63e948e0282d04bf0883470", size = 5640730, upload-time = "2026-08-21T06:08:09.135Z" }, + { url = "https://files.pythonhosted.org/packages/ff/3f/5bf6c32e9e701837a8c47ce6e3ad38978cfec8eb7bc6596181e5f9e1eaeb/pyqwest-0.10.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a5c757cfac5f53c8671dcb4850d5fc4c4339ea3e90636331c9318f8e3ddabc06", size = 5561462, upload-time = "2026-08-21T06:08:10.836Z" }, + { url = "https://files.pythonhosted.org/packages/5f/61/ca9ba5721461b7ce5cfaac373ab3a1723ddcc434af7430f8bd628da6e623/pyqwest-0.10.0-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:234b3f71e3f314d997c203d8cf829b7117edd041153f9c277d0060ab90134148", size = 5801847, upload-time = "2026-08-21T06:08:12.502Z" }, + { url = "https://files.pythonhosted.org/packages/d2/44/95593919b996a417093f598d887822b9b899e8d025588c9bfaf8c60dd812/pyqwest-0.10.0-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:5637256a0dac0ef57e0eaa02b032014965e4a4c995e1deca1b1b97e6d1765f78", size = 5978692, upload-time = "2026-08-21T06:08:14.383Z" }, + { url = "https://files.pythonhosted.org/packages/a5/b9/b2821ce5188457168ebb25d5ff65b1ca1bf27bc6b4a33df4bcc2357e625c/pyqwest-0.10.0-cp310-abi3-win_amd64.whl", hash = "sha256:7ea761937acf3a00d1a7e70e982949d18946e5471d1419266ab3a78bbfa19759", size = 4876627, upload-time = "2026-08-21T06:08:16.084Z" }, + { url = "https://files.pythonhosted.org/packages/86/b4/16ccef1c203fa258ce46a86aefc1a79c13b5f0b8d49627347d90eef25efd/pyqwest-0.10.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:a21f1f15252a8303623b4f17b9c6de595ace11b3ade07f2adb6d07121e8191aa", size = 5274815, upload-time = "2026-08-21T06:08:17.777Z" }, + { url = "https://files.pythonhosted.org/packages/5f/61/6a87f84f571441ea43279587d4bfcad4543505918ae2b83a1ebdcfa98be5/pyqwest-0.10.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:eb472c6e5d6833ebfec79db310e426eb17b01ac64c0e2c251bd9192c0d2ee0c5", size = 5123656, upload-time = "2026-08-21T06:08:19.501Z" }, + { url = "https://files.pythonhosted.org/packages/8b/f2/ab69e581cf9b798b0e169f7b27fd3f8b6f9f1631bd4d3b6e22e5abaf8d8b/pyqwest-0.10.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:83578e24cccd5e0dc04d60a0af7bfb43325b5f22d03ff74ff79ed0ecf553b50d", size = 5641253, upload-time = "2026-08-21T06:08:21.627Z" }, + { url = "https://files.pythonhosted.org/packages/9f/dd/f1a62eebf8321ace506bd94551a01431f6a45b882225455eae3ea6e8c6d1/pyqwest-0.10.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6bb511c434f79c641efb5573e5795e56dc972252f4b96e52a9636d4ece5231a4", size = 5567341, upload-time = "2026-08-21T06:08:23.346Z" }, + { url = "https://files.pythonhosted.org/packages/4c/48/51d767691973e046887f5e6d96e32142fe296823b163ccba732233a6ef72/pyqwest-0.10.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:1aaccd8a9db9430b2aedb5bad8ead80742cbc056b85c229516c70dc80539f906", size = 5803874, upload-time = "2026-08-21T06:08:25.096Z" }, + { url = "https://files.pythonhosted.org/packages/58/0a/d2834ccc6e59ad110718895cc65ff2a68aa6e010f0ba8fbe42a57ea33c21/pyqwest-0.10.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:73d9eb438ab4a957a1ce0619d3af8c1c1126bfb9181033b123d792fcf4224531", size = 5981518, upload-time = "2026-08-21T06:08:27.594Z" }, + { url = "https://files.pythonhosted.org/packages/75/f3/274b4c268e9a55fbdb1b3637ac50b5bf42cd3a85d1cfbdc15c602a7b0d9c/pyqwest-0.10.0-cp312-cp312-win_amd64.whl", hash = "sha256:317a74d633abe3bc5bccabf479e069c515dab9e6a755274b0ccb1d8a5bbfede3", size = 4870638, upload-time = "2026-08-21T06:08:29.421Z" }, + { url = "https://files.pythonhosted.org/packages/8d/3d/8b1092f25159bf61a9470ebd35438c669b91ef553a7ee205bdec8006107b/pyqwest-0.10.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:3978e794b9cfd8eaa500fb5d7aee63bc6172c605efa0abc1f62d85485bc049e1", size = 5273599, upload-time = "2026-08-21T06:08:31.107Z" }, + { url = "https://files.pythonhosted.org/packages/75/10/54a9786123942b124c2afb9562b74e158afec7be40ef0caa0d37f615d379/pyqwest-0.10.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:715991fd4f04862cd7a9d7452daabcdbd74dff4dff55eb20c22d60382dc2a4ed", size = 5122920, upload-time = "2026-08-21T06:08:33.025Z" }, + { url = "https://files.pythonhosted.org/packages/a2/5b/6a6bd76f91e068b9a619f62aef9fe5ef201f859ddbb6b0a11ad3875ecdda/pyqwest-0.10.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c04798bed79c1dfa0e5b0e30fb137124311083490d44d6dfbe068d3dd254349e", size = 5639577, upload-time = "2026-08-21T06:08:34.896Z" }, + { url = "https://files.pythonhosted.org/packages/68/28/12277a24a8dd74b0a7f124c624d9ed58eccb41e2087ff1087a14d348c778/pyqwest-0.10.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:35b472877e73dd63fed089c2bc8fa198407f005c8c19e0a93f025ebefde01a81", size = 5565835, upload-time = "2026-08-21T06:08:36.567Z" }, + { url = "https://files.pythonhosted.org/packages/9e/b6/b7da4a3f1fe43600154ae75e91ba7969024d886d60077dd3a1ba8e66d170/pyqwest-0.10.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:564ec360b7848b35e009038ffbca00466305a9708ab21829477f64aa8cad4c64", size = 5803078, upload-time = "2026-08-21T06:08:38.362Z" }, + { url = "https://files.pythonhosted.org/packages/0c/6f/0c9ba210f49f232289afaa8f06369c5f135ac786e1ca0cb22243b7f1fe2c/pyqwest-0.10.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:b5c80e88a5967c1cadb3237c450f91a84a3683f8838c8dca96f09fee3612e762", size = 5980431, upload-time = "2026-08-21T06:08:39.967Z" }, + { url = "https://files.pythonhosted.org/packages/9b/f5/306eeed41a3cd3100247e6e442f4345277b70f1d24efe1641181b14839cd/pyqwest-0.10.0-cp313-cp313-win_amd64.whl", hash = "sha256:dc3d80b402fb59dbe015e25993ac8147456fb231a4c949f92a89f31315ad50f9", size = 4870198, upload-time = "2026-08-21T06:08:41.687Z" }, + { url = "https://files.pythonhosted.org/packages/64/18/0086a408e7cbf39dab18fa5b7e42c969a98382da5a4e6debe40f05acc6a1/pyqwest-0.10.0-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:23a28beb55fa6d975949bffae4adfb69378f3229bb5cbd71231e95bf66f5b26c", size = 5274542, upload-time = "2026-08-21T06:08:43.323Z" }, + { url = "https://files.pythonhosted.org/packages/0c/7b/e1b9aaaf7596e4faaa53cefc2efaca4e3cde721e308e6385e366361cfdcc/pyqwest-0.10.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:e4415ae40b8eedb1713dab14d7f9fecc3f79d26f3206c561087b88b99d5ce24b", size = 5127922, upload-time = "2026-08-21T06:08:45.116Z" }, + { url = "https://files.pythonhosted.org/packages/75/cd/e6d68bb1de5dd26100fcfc878cbd67c402a928774edf1e8ae304c5a84f5b/pyqwest-0.10.0-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:14b875d2273212d7fa8e4b755d8d736ffd226b1c707a9c0017dfdc8393a96eca", size = 5645856, upload-time = "2026-08-21T06:08:47.055Z" }, + { url = "https://files.pythonhosted.org/packages/f7/48/8c9f9f0467c41f6a563146d57a52f8f6d60c0cb09d0fd3ef88ad5a1c442f/pyqwest-0.10.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b5071491e416ea54e3b95bf9ffbed0bd065b093cb96e10a75c3d8f2cbe3c9823", size = 5569831, upload-time = "2026-08-21T06:08:48.78Z" }, + { url = "https://files.pythonhosted.org/packages/16/ac/c85ab70c6c72078d49a82da76b820e46aaf95a3f6fe271dac955ac195d21/pyqwest-0.10.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:b68b5e68d513a4c63a072f8f40e38015160cf90bfbf7e8ef7c3935ca87e9e022", size = 5806735, upload-time = "2026-08-21T06:08:50.652Z" }, + { url = "https://files.pythonhosted.org/packages/66/db/ad7375b22fb2d0807431dcc9bc2aaf840e374c298cd15071024d5f6dd6d1/pyqwest-0.10.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:c48910d27820b9c46fcd001b0fe514a3cf47d4784f59512dcdb8c91c395f82e4", size = 5984786, upload-time = "2026-08-21T06:08:52.377Z" }, + { url = "https://files.pythonhosted.org/packages/17/88/c449a772afe129683fd7acc657cbc7c69bec085dc75b6e8710a50fbb44e7/pyqwest-0.10.0-cp314-cp314-win_amd64.whl", hash = "sha256:d03ba2cd17948b623a6210981d342eb122546d8a8e910ec77511aff4b1acdd00", size = 4872288, upload-time = "2026-08-21T06:08:54.101Z" }, + { url = "https://files.pythonhosted.org/packages/3d/f8/439ffc0ee12cd7d9b57ac07ccea78ad3ba66b0d6817d429dd661d73308c4/pyqwest-0.10.0-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:07a0eb595f4096232c2d22549b6e4612c1ecada7934e46462c2c37ce14a89cfb", size = 5256823, upload-time = "2026-08-21T06:08:55.681Z" }, + { url = "https://files.pythonhosted.org/packages/df/2b/72ecd27d104d2b4284710194cce796d607674966c3ea66436768e812a66a/pyqwest-0.10.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:26401baf7dafc71c8d12d2e8389519d141e6f7c14094d0dd4cf9ec1d3b5555bd", size = 5112624, upload-time = "2026-08-21T06:08:57.366Z" }, + { url = "https://files.pythonhosted.org/packages/a4/24/0fb89c3f7d5a0410fcb7560588b4c741ef24b19195c307d4263f04e75c2b/pyqwest-0.10.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e5e3c436e041d8873ce5bb0fdcf9f9e86f5604e8f0ef9e03149efebd8cb474f6", size = 5631844, upload-time = "2026-08-21T06:08:59.165Z" }, + { url = "https://files.pythonhosted.org/packages/4c/4f/921d14754a186f0143ad62b50108dd808328e347e98e9dafab3897eeb405/pyqwest-0.10.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:09364115761579eabfc79d1e954cdb3ded508dac1903fac7285d4c6f058c683f", size = 5555869, upload-time = "2026-08-21T06:09:00.783Z" }, + { url = "https://files.pythonhosted.org/packages/b8/70/504780417319a626fe9549a7e6f9020a3d448eddf8a09617238a3426e90c/pyqwest-0.10.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:559674a98a8b1217e1830ecd41c9905bf2b60983c6b8017063dfac199f00727c", size = 5793738, upload-time = "2026-08-21T06:09:03.324Z" }, + { url = "https://files.pythonhosted.org/packages/45/f7/8d0a5b8a3289f4300dc9005ebde75d316f2371a2671617f942036337731a/pyqwest-0.10.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:f399a696392fff3db3eef0a18ef65b8a3b8396d129193487d966b8eb11006376", size = 5972889, upload-time = "2026-08-21T06:09:05.2Z" }, + { url = "https://files.pythonhosted.org/packages/89/c4/f4c781e475c451cb5f4762a2f814a1750ef375b8db4db09bb2dcea03c4e5/pyqwest-0.10.0-cp314-cp314t-win_amd64.whl", hash = "sha256:0f9163d6dd991bf1bf27308ba38ba021af660b15fffa47ebca98e41cf6f00309", size = 4858036, upload-time = "2026-08-21T06:09:06.926Z" }, +] + +[[package]] +name = "pyrefly" +version = "1.2.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/89/01/a86e9f24722b095c3f88e3616132b75a21b0df53804bdc6a45314dd4d93c/pyrefly-1.2.0.tar.gz", hash = "sha256:5485f960fc2481617068c918335c39ab1507ef90b6b5bd35bf57726e60e73185", size = 6243654, upload-time = "2026-08-01T02:56:27.592Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7d/9d/3c0ef1d4843987b22f996ed381ec9cf5a3b1273e29804db276252e4c95eb/pyrefly-1.2.0-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:7f46d983ac49ddd2b043694960a01dc6a19a5cfd8eec609d6bd9c42866f91b4e", size = 14026305, upload-time = "2026-08-01T02:56:02.611Z" }, + { url = "https://files.pythonhosted.org/packages/0a/06/03bbb78fbea54cdc65b626619f3597d5611aca4fdef11e72a4e8360e7e63/pyrefly-1.2.0-py3-none-macosx_11_0_arm64.whl", hash = "sha256:756f669b5555090f5c1a4fef30db1785fabe657764f7e4e6dc88994dfb8ca82d", size = 13463880, upload-time = "2026-08-01T02:56:04.93Z" }, + { url = "https://files.pythonhosted.org/packages/13/5a/7d8bc00a38e93bbc9c3e7bd14d305f7948717e667c9bcddeab9dd42fd255/pyrefly-1.2.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e3465812ce5ef4781fb592edbf2724547296f0a3124be115d73c7e8b2401862d", size = 13907329, upload-time = "2026-08-01T02:56:07.104Z" }, + { url = "https://files.pythonhosted.org/packages/be/94/9e08b4bf799d0b8f36b55a2783c7ba5f51730cf0632a85a67b5b5ed876cd/pyrefly-1.2.0-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:5de7b2ad2bba5c8055181681a84b74143eac2234a48ba5d1b7ed7e7a722b02bd", size = 15039020, upload-time = "2026-08-01T02:56:09.208Z" }, + { url = "https://files.pythonhosted.org/packages/5b/bd/bca5fd0c80f4daf8ee6903a29df9f3de1feb05ff0946b8f35ec8c5096b13/pyrefly-1.2.0-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:25822ea9505f589ea8a725e4268b475132fb89e038fbf092e446510443ac142a", size = 14986199, upload-time = "2026-08-01T02:56:11.924Z" }, + { url = "https://files.pythonhosted.org/packages/97/f7/f07087f3d185ad2eced0c56cef89ca5474dfb4ff25f146cd50a861c97553/pyrefly-1.2.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:90efe75e17491ef5d636e10469e9278d7d0256b3b4c5e1f4750069bf3ae0f5d1", size = 14393715, upload-time = "2026-08-01T02:56:14.143Z" }, + { url = "https://files.pythonhosted.org/packages/d3/70/0d142c320e284b9e3ce35e9b1e58b8ce2ee1f578f2a7234bc30e5022b94f/pyrefly-1.2.0-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:368aaf7eee4f511ddc0f8e564cf14e01ab2f10b0db9105c6d5b153bf498d07bf", size = 13933008, upload-time = "2026-08-01T02:56:16.525Z" }, + { url = "https://files.pythonhosted.org/packages/5d/e8/e84f11b6e1f63fd453ad3654213b9a0f6f4de8cef6b58038eef2d0d5955d/pyrefly-1.2.0-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:d52d5da7bc65fb7675fbaa80eda879d4f8787c494f04cac21603330d3abbdbbe", size = 14431827, upload-time = "2026-08-01T02:56:18.645Z" }, + { url = "https://files.pythonhosted.org/packages/0f/06/810d31380f66c75e1c0779a408d3b16117b1b368b57894f6aa66bef21686/pyrefly-1.2.0-py3-none-win32.whl", hash = "sha256:8c90751de8506d938e8f802659c74cf35bd7a0036510ee6c634a38eebb280bfa", size = 13229447, upload-time = "2026-08-01T02:56:20.921Z" }, + { url = "https://files.pythonhosted.org/packages/ed/98/4dafa3c7a1caed2dc8cc708dde09ba27963c7736508f55b626fff3024113/pyrefly-1.2.0-py3-none-win_amd64.whl", hash = "sha256:8a8964c224ccc4882730130955815de21ff443c1ac3f0b90685b19bf63848170", size = 14087387, upload-time = "2026-08-01T02:56:23.188Z" }, + { url = "https://files.pythonhosted.org/packages/1b/1c/df3cb0a2e5591660ded7a1836cd2f29dc48c91adb1c0a3a700a96f6d09e1/pyrefly-1.2.0-py3-none-win_arm64.whl", hash = "sha256:3a90bb8df39dfbac74b1f3b2e9d7c526b8f80568884c3944d955023a73ebf61e", size = 13430873, upload-time = "2026-08-01T02:56:25.425Z" }, ] [[package]] @@ -3728,7 +3719,7 @@ wheels = [ [[package]] name = "transformers" -version = "5.10.1" +version = "5.6.2" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "huggingface-hub" }, @@ -3741,9 +3732,9 @@ dependencies = [ { name = "tqdm" }, { name = "typer" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/65/d3/5b7c2f1a52ff0e57355efdc21554aab7e4602f6592ab4582a34c988ab956/transformers-5.10.1.tar.gz", hash = "sha256:31112d1dcdfcf9934242acbba891f44e2279ff74b9b8ba4595640e0e04195a3a", size = 8798372, upload-time = "2026-06-03T15:37:03.289Z" } +sdist = { url = "https://files.pythonhosted.org/packages/a4/e9/c6c80a07690142a7d05444271f47b9f3c8aac7dea01d52e1137ee480ad78/transformers-5.6.2.tar.gz", hash = "sha256:e657134c3e5a6bc00a3c35f4e2674bb51adfcd89898495b788a18552bac2b91a", size = 8311867, upload-time = "2026-04-23T18:33:29.332Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/e3/8c/3119596c7fcd9b8b8d924d5b5ba1bfdfafd6d8738bdd81ca09c60b4a38b3/transformers-5.10.1-py3-none-any.whl", hash = "sha256:ccb919ea1b77338b44d0d45d23f7472081906b1bb6ed8e5f5cf4d692d1da03d4", size = 11003770, upload-time = "2026-06-03T15:37:00.433Z" }, + { url = "https://files.pythonhosted.org/packages/5d/95/0b0218149b0d6f14df35f5b8f676fa83df4f19ed253c3cc447107ef86eca/transformers-5.6.2-py3-none-any.whl", hash = "sha256:f8d3a1bb96778fed9b8aabfd0dd6e19843e4b0f2bb6b59f32b8a92051b0f348f", size = 10364898, upload-time = "2026-04-23T18:33:26.081Z" }, ] [[package]] diff --git a/e2e/features/step-definitions/agent-v2/access-point-helpers.ts b/e2e/features/step-definitions/agent-v2/access-point-helpers.ts index 654455d1283..f0e5c01d776 100644 --- a/e2e/features/step-definitions/agent-v2/access-point-helpers.ts +++ b/e2e/features/step-definitions/agent-v2/access-point-helpers.ts @@ -28,7 +28,7 @@ export const getAccessRegion = (world: DifyWorld) => export type AccessSurfaceName = 'Web app' | 'Backend service API' export const getAccessSurfaceCard = (world: DifyWorld, surface: AccessSurfaceName) => - getAccessRegion(world).getByRole('article', { name: surface }).first() + getAccessRegion(world).getByRole('region', { name: surface }).first() export const getWebAppCard = (world: DifyWorld) => getAccessSurfaceCard(world, 'Web app') diff --git a/e2e/features/step-definitions/agent-v2/access-point-web-app.steps.ts b/e2e/features/step-definitions/agent-v2/access-point-web-app.steps.ts index 53eba93fdae..a71d04f295d 100644 --- a/e2e/features/step-definitions/agent-v2/access-point-web-app.steps.ts +++ b/e2e/features/step-definitions/agent-v2/access-point-web-app.steps.ts @@ -22,9 +22,9 @@ Then('I should see the Agent v2 Web app access URL', async function (this: DifyW const webAppCard = getWebAppCard(this) await expect(webAppCard.getByRole('heading', { name: 'Web app' })).toBeVisible() - await expect(webAppCard.getByText('Web App URL')).toBeVisible() + await expect(webAppCard.getByText('Access URL')).toBeVisible() await expect(webAppCard.getByLabel('Copy access URL')).toBeEnabled() - await expect(webAppCard.getByRole('link', { name: 'Launch' })).toBeVisible() + await expect(webAppCard.getByRole('link', { name: 'Open' })).toBeVisible() }) When('I copy the Agent v2 Web app access URL', async function (this: DifyWorld) { @@ -38,11 +38,11 @@ Then('the Agent v2 Web app access URL should show it was copied', async function When('I launch the Agent v2 Web app', async function (this: DifyWorld) { await recordComposerDraftSnapshot(this) - const launchLink = getWebAppCard(this).getByRole('link', { name: 'Launch' }) - const href = await launchLink.getAttribute('href') - if (!href) throw new Error('Agent v2 Web app Launch link does not expose an href.') + const openLink = getWebAppCard(this).getByRole('link', { name: 'Open' }) + const href = await openLink.getAttribute('href') + if (!href) throw new Error('Agent v2 Web app Open link does not expose an href.') - const [webAppPage] = await Promise.all([this.getPage().waitForEvent('popup'), launchLink.click()]) + const [webAppPage] = await Promise.all([this.getPage().waitForEvent('popup'), openLink.click()]) this.agentBuilder.accessPoint.webAppURL = href this.agentBuilder.accessPoint.webAppPage = webAppPage @@ -130,7 +130,7 @@ When('I close the Agent v2 Web app', async function (this: DifyWorld) { When('I open Agent v2 Embedded configuration', async function (this: DifyWorld) { await recordComposerDraftSnapshot(this) - await getWebAppCard(this).getByRole('button', { name: 'Embedded' }).click() + await getWebAppCard(this).getByRole('button', { name: 'Embed Into Site' }).click() }) Then('I should see the Agent v2 Embedded configuration dialog', async function (this: DifyWorld) { @@ -143,7 +143,7 @@ Then('I should see the Agent v2 Embedded configuration dialog', async function ( When('I open Agent v2 Web app customization', async function (this: DifyWorld) { await recordComposerDraftSnapshot(this) - await getWebAppCard(this).getByRole('button', { name: 'Custom Frontend' }).click() + await getWebAppCard(this).getByRole('button', { name: 'Custom frontend' }).click() }) Then('I should see the Agent v2 Web app customization dialog', async function (this: DifyWorld) { @@ -156,7 +156,7 @@ Then('I should see the Agent v2 Web app customization dialog', async function (t When('I open Agent v2 Web app settings', async function (this: DifyWorld) { await recordComposerDraftSnapshot(this) - await getWebAppCard(this).getByRole('button', { name: 'Branding' }).click() + await getWebAppCard(this).getByRole('button', { name: 'Settings' }).click() }) Then('I should see the Agent v2 Web app settings dialog', async function (this: DifyWorld) { diff --git a/e2e/features/step-definitions/agent-v2/access-point.steps.ts b/e2e/features/step-definitions/agent-v2/access-point.steps.ts index 38706e41a7d..956d4f0b776 100644 --- a/e2e/features/step-definitions/agent-v2/access-point.steps.ts +++ b/e2e/features/step-definitions/agent-v2/access-point.steps.ts @@ -81,7 +81,7 @@ Then( await expect(webAppCard.getByText('Out of service')).toBeVisible({ timeout: 30_000 }) await expect(webAppCard.getByLabel('Toggle Web app access')).toBeDisabled() - await expect(webAppCard.getByRole('button', { name: 'Launch' })).toBeDisabled() + await expect(webAppCard.getByRole('button', { name: 'Open' })).toBeDisabled() await expect(serviceApiCard.getByText('Out of service')).toBeVisible() await expect(serviceApiCard.getByLabel('Toggle Backend service API access')).toBeDisabled() await expect(serviceApiCard.getByRole('button', { name: /^API Key\b/ })).toBeDisabled() @@ -94,9 +94,9 @@ When( const accessSurfaceCard = getAccessSurfaceCard(this, surface) if (surface === 'Web app') { - const launchLink = accessSurfaceCard.getByRole('link', { name: 'Launch' }) - const href = await launchLink.getAttribute('href') - if (!href) throw new Error('Agent v2 Web app Launch link does not expose an href.') + const openLink = accessSurfaceCard.getByRole('link', { name: 'Open' }) + const href = await openLink.getAttribute('href') + if (!href) throw new Error('Agent v2 Web app Open link does not expose an href.') this.agentBuilder.accessPoint.webAppURL = href } @@ -122,7 +122,7 @@ Then( await expect(toggle).toBeEnabled() await expect(toggle).toHaveAttribute('aria-checked', 'false') if (surface === 'Web app') - await expect(accessSurfaceCard.getByRole('button', { name: 'Launch' })).toBeDisabled() + await expect(accessSurfaceCard.getByRole('button', { name: 'Open' })).toBeDisabled() }, ) @@ -136,6 +136,6 @@ Then( await expect(toggle).toBeEnabled() await expect(toggle).toHaveAttribute('aria-checked', 'true') if (surface === 'Web app') - await expect(accessSurfaceCard.getByRole('link', { name: 'Launch' })).toBeVisible() + await expect(accessSurfaceCard.getByRole('link', { name: 'Open' })).toBeVisible() }, ) diff --git a/e2e/features/step-definitions/apps/create-app.steps.ts b/e2e/features/step-definitions/apps/create-app.steps.ts index 88113afa3fb..f632f310a33 100644 --- a/e2e/features/step-definitions/apps/create-app.steps.ts +++ b/e2e/features/step-definitions/apps/create-app.steps.ts @@ -20,6 +20,10 @@ const getLatestCreatedAppId = (world: DifyWorld) => { return appId } +const expectAppEditorContent = async (world: DifyWorld) => { + await expect(world.getPage().getByRole('link', { name: 'Orchestrate' })).toBeVisible() +} + When('I start creating a blank app', async function (this: DifyWorld) { await openBlankAppCreation(this.getPage()) }) @@ -78,14 +82,17 @@ Then('I should land on the app editor', async function (this: DifyWorld) { await expect(this.getPage()).toHaveURL( new RegExp(`/app/${appId}/(workflow|configuration)(?:\\?.*)?$`), ) + await expectAppEditorContent(this) }) Then('I should land on the workflow editor', async function (this: DifyWorld) { const appId = getLatestCreatedAppId(this) await expect(this.getPage()).toHaveURL(new RegExp(`/app/${appId}/workflow(?:\\?.*)?$`)) + await expectAppEditorContent(this) }) Then('I should land on the app configuration page', async function (this: DifyWorld) { const appId = getLatestCreatedAppId(this) await expect(this.getPage()).toHaveURL(new RegExp(`/app/${appId}/configuration(?:\\?.*)?$`)) + await expectAppEditorContent(this) }) diff --git a/knip.config.ts b/knip.config.ts index 2914f398f8c..3931cfd7a2f 100644 --- a/knip.config.ts +++ b/knip.config.ts @@ -15,6 +15,10 @@ const config: KnipConfig = { 'tsslint.config.ts', 'dev-proxy.config.ts', 'plugins/eslint/index.js', + // Public surface consumed by the standalone Marketplace host. + // The `!` suffix keeps these entries in `knip --production`. + 'app/components/plugins/marketplace/standalone/server.ts!', + 'app/components/plugins/marketplace/standalone/client.ts!', ], project: [ '**/*.{js,mjs,cjs,jsx,ts,tsx,mts,cts,css,mdx}!', diff --git a/lint.config.ts b/lint.config.ts index 3ca800f1167..063c7dcaa7f 100644 --- a/lint.config.ts +++ b/lint.config.ts @@ -772,6 +772,7 @@ export const lintConfig = { '@tanstack/query/infinite-query-property-order': 'error', '@tanstack/query/no-void-query-fn': 'error', '@tanstack/query/mutation-property-order': 'error', + '@tanstack/query/prefer-query-options': 'error', 'react/exhaustive-deps': 'warn', 'react/no-array-index-key': 'warn', 'react/no-clone-element': 'warn', @@ -1300,15 +1301,6 @@ export const lintConfig = { ], }, }, - { - files: [ - 'packages/dify-ui/**/__tests__/**/*.{js,cjs,mjs,jsx,ts,cts,mts,tsx}', - 'packages/dify-ui/**/*.spec.{js,cjs,mjs,jsx,ts,cts,mts,tsx}', - ], - rules: { - 'eslint-react/purity': 'off', - }, - }, { files: ['cli/bin/**'], rules: { diff --git a/oxlint-suppressions.json b/oxlint-suppressions.json index 1e08448f984..fed2ac53743 100644 --- a/oxlint-suppressions.json +++ b/oxlint-suppressions.json @@ -191,12 +191,6 @@ "eslint-react/set-state-in-effect": { "count": 5 }, - "jsx-a11y/click-events-have-key-events": { - "count": 1 - }, - "jsx-a11y/no-static-element-interactions": { - "count": 1 - }, "typescript/no-explicit-any": { "count": 1 } @@ -295,12 +289,6 @@ "eslint-react/set-state-in-effect": { "count": 1 }, - "jsx-a11y/click-events-have-key-events": { - "count": 1 - }, - "jsx-a11y/no-static-element-interactions": { - "count": 1 - }, "typescript/no-explicit-any": { "count": 4 } @@ -368,14 +356,6 @@ "count": 1 } }, - "web/app/components/app/configuration/debug/chat-user-input.tsx": { - "jsx-a11y/no-autofocus": { - "count": 2 - }, - "no-restricted-imports": { - "count": 1 - } - }, "web/app/components/app/configuration/debug/debug-with-multiple-model/chat-item.tsx": { "typescript/no-explicit-any": { "count": 6 @@ -414,14 +394,6 @@ "count": 1 } }, - "web/app/components/app/configuration/prompt-value-panel/index.tsx": { - "jsx-a11y/no-autofocus": { - "count": 2 - }, - "no-restricted-imports": { - "count": 1 - } - }, "web/app/components/app/create-app-dialog/app-list/sidebar.tsx": { "erasable-syntax-only/enums": { "count": 1 @@ -430,6 +402,16 @@ "count": 1 } }, + "web/app/components/app/deploy/shared/deployment-configuration/use-deployment-configuration-queries.ts": { + "@tanstack/query/prefer-query-options": { + "count": 2 + } + }, + "web/app/components/app/in-site-message/__tests__/notification.spec.tsx": { + "@tanstack/query/prefer-query-options": { + "count": 1 + } + }, "web/app/components/app/log/filter.tsx": { "react/only-export-components": { "count": 1 @@ -464,11 +446,6 @@ "count": 1 } }, - "web/app/components/app/overview/workflow-hidden-input-fields.tsx": { - "no-restricted-imports": { - "count": 1 - } - }, "web/app/components/app/text-generate/item/index.tsx": { "typescript/no-explicit-any": { "count": 3 @@ -625,21 +602,10 @@ } }, "web/app/components/base/chat/chat-with-history/inputs-form/content.tsx": { - "no-restricted-imports": { - "count": 1 - }, "typescript/no-explicit-any": { "count": 3 } }, - "web/app/components/base/chat/chat-with-history/sidebar/item.tsx": { - "jsx-a11y/click-events-have-key-events": { - "count": 2 - }, - "jsx-a11y/no-static-element-interactions": { - "count": 2 - } - }, "web/app/components/base/chat/chat/__tests__/hooks.spec.tsx": { "no-restricted-imports": { "count": 1 @@ -715,9 +681,6 @@ } }, "web/app/components/base/chat/embedded-chatbot/inputs-form/content.tsx": { - "no-restricted-imports": { - "count": 1 - }, "typescript/no-explicit-any": { "count": 3 } @@ -732,14 +695,6 @@ "count": 10 } }, - "web/app/components/base/date-and-time-picker/date-picker/index.tsx": { - "jsx-a11y/click-events-have-key-events": { - "count": 1 - }, - "jsx-a11y/no-static-element-interactions": { - "count": 1 - } - }, "web/app/components/base/date-and-time-picker/hooks.ts": { "eslint-react/no-unnecessary-use-prefix": { "count": 2 @@ -1156,7 +1111,7 @@ }, "web/app/components/base/icons/src/vender/plugin/index.ts": { "no-barrel-files/no-barrel-files": { - "count": 3 + "count": 2 } }, "web/app/components/base/icons/src/vender/solid/FinanceAndECommerce/index.ts": { @@ -1234,14 +1189,6 @@ "count": 3 } }, - "web/app/components/base/image-gallery/index.tsx": { - "jsx-a11y/click-events-have-key-events": { - "count": 1 - }, - "jsx-a11y/no-noninteractive-element-interactions": { - "count": 1 - } - }, "web/app/components/base/image-uploader/__tests__/image-preview.spec.tsx": { "erasable-syntax-only/parameter-properties": { "count": 1 @@ -1265,14 +1212,6 @@ "count": 1 } }, - "web/app/components/base/image-uploader/image-list.tsx": { - "jsx-a11y/click-events-have-key-events": { - "count": 1 - }, - "jsx-a11y/no-noninteractive-element-interactions": { - "count": 1 - } - }, "web/app/components/base/image-uploader/image-preview.tsx": { "jsx-a11y/no-static-element-interactions": { "count": 1 @@ -1334,12 +1273,6 @@ } }, "web/app/components/base/markdown-blocks/link.tsx": { - "jsx-a11y/click-events-have-key-events": { - "count": 1 - }, - "jsx-a11y/no-noninteractive-element-interactions": { - "count": 1 - }, "typescript/no-explicit-any": { "count": 1 } @@ -1643,14 +1576,6 @@ "count": 1 } }, - "web/app/components/base/tab-slider-plain/index.tsx": { - "jsx-a11y/click-events-have-key-events": { - "count": 1 - }, - "jsx-a11y/no-static-element-interactions": { - "count": 1 - } - }, "web/app/components/base/tag-input/index.stories.tsx": { "jsx-a11y/label-has-associated-control": { "count": 12 @@ -1713,11 +1638,6 @@ "count": 3 } }, - "web/app/components/datasets/chunk.tsx": { - "jsx-a11y/label-has-associated-control": { - "count": 2 - } - }, "web/app/components/datasets/common/document-status-with-action/status-with-action.tsx": { "eslint-react/static-components": { "count": 2 @@ -1809,14 +1729,6 @@ "count": 2 } }, - "web/app/components/datasets/create/file-uploader/components/upload-dropzone.tsx": { - "jsx-a11y/click-events-have-key-events": { - "count": 1 - }, - "jsx-a11y/no-noninteractive-element-interactions": { - "count": 1 - } - }, "web/app/components/datasets/create/file-uploader/hooks/use-file-upload.ts": { "no-restricted-imports": { "count": 1 @@ -1978,14 +1890,6 @@ "count": 2 } }, - "web/app/components/datasets/documents/create-from-pipeline/data-source/local-file/components/upload-dropzone.tsx": { - "jsx-a11y/click-events-have-key-events": { - "count": 1 - }, - "jsx-a11y/no-noninteractive-element-interactions": { - "count": 1 - } - }, "web/app/components/datasets/documents/create-from-pipeline/data-source/online-documents/index.tsx": { "no-restricted-imports": { "count": 1 @@ -2152,6 +2056,11 @@ "count": 1 } }, + "web/app/components/datasets/documents/detail/embedding/hooks/use-embedding-status.ts": { + "@tanstack/query/prefer-query-options": { + "count": 1 + } + }, "web/app/components/datasets/documents/detail/metadata/index.tsx": { "no-barrel-files/no-barrel-files": { "count": 1 @@ -2170,12 +2079,6 @@ "web/app/components/datasets/external-knowledge-base/create/ExternalApiSelect.tsx": { "eslint-react/set-state-in-effect": { "count": 1 - }, - "jsx-a11y/click-events-have-key-events": { - "count": 3 - }, - "jsx-a11y/no-static-element-interactions": { - "count": 3 } }, "web/app/components/datasets/external-knowledge-base/create/ExternalApiSelection.tsx": { @@ -2371,6 +2274,11 @@ "count": 4 } }, + "web/app/components/header/account-setting/model-provider-page/hooks.ts": { + "@tanstack/query/prefer-query-options": { + "count": 2 + } + }, "web/app/components/header/account-setting/model-provider-page/model-auth/add-custom-model.tsx": { "jsx-a11y/click-events-have-key-events": { "count": 2 @@ -2473,6 +2381,11 @@ "count": 3 } }, + "web/app/components/header/account-setting/model-provider-page/provider-added-card/provider-card-actions.tsx": { + "@tanstack/query/prefer-query-options": { + "count": 2 + } + }, "web/app/components/header/account-setting/model-provider-page/utils.ts": { "no-barrel-files/no-barrel-files": { "count": 2 @@ -2486,6 +2399,11 @@ "count": 1 } }, + "web/app/components/header/github-star/index.tsx": { + "@tanstack/query/prefer-query-options": { + "count": 1 + } + }, "web/app/components/main-nav/components/workspace-switcher.tsx": { "jsx-a11y/no-autofocus": { "count": 1 @@ -2551,15 +2469,15 @@ } }, "web/app/components/plugins/marketplace/hooks.ts": { + "@tanstack/query/prefer-query-options": { + "count": 3 + }, "no-restricted-imports": { "count": 1 } }, - "web/app/components/plugins/marketplace/list/list-with-collection.tsx": { - "jsx-a11y/click-events-have-key-events": { - "count": 1 - }, - "jsx-a11y/no-static-element-interactions": { + "web/app/components/plugins/marketplace/query.ts": { + "@tanstack/query/prefer-query-options": { "count": 1 } }, @@ -2603,9 +2521,6 @@ } }, "web/app/components/plugins/plugin-detail-panel/app-selector/app-inputs-form.tsx": { - "no-restricted-imports": { - "count": 1 - }, "typescript/no-explicit-any": { "count": 8 } @@ -2984,9 +2899,6 @@ "eslint-react/set-state-in-effect": { "count": 1 }, - "no-restricted-imports": { - "count": 1 - }, "typescript/no-explicit-any": { "count": 3 } @@ -3004,6 +2916,11 @@ "count": 1 } }, + "web/app/components/snippet-list/__tests__/reflow.browser.spec.tsx": { + "@tanstack/query/prefer-query-options": { + "count": 2 + } + }, "web/app/components/snippets/components/snippet-run-panel.tsx": { "jsx-a11y/click-events-have-key-events": { "count": 5 @@ -3357,14 +3274,6 @@ "count": 2 } }, - "web/app/components/workflow/nodes/_base/components/form-input-boolean.tsx": { - "jsx-a11y/click-events-have-key-events": { - "count": 2 - }, - "jsx-a11y/no-static-element-interactions": { - "count": 2 - } - }, "web/app/components/workflow/nodes/_base/components/form-input-item.tsx": { "no-restricted-imports": { "count": 1 @@ -3604,6 +3513,11 @@ "count": 1 } }, + "web/app/components/workflow/nodes/agent-v2/__tests__/hooks.spec.tsx": { + "@tanstack/query/prefer-query-options": { + "count": 6 + } + }, "web/app/components/workflow/nodes/agent/default.ts": { "typescript/no-explicit-any": { "count": 3 @@ -3737,14 +3651,6 @@ "count": 1 } }, - "web/app/components/workflow/nodes/http/components/authorization/radio-group.tsx": { - "jsx-a11y/click-events-have-key-events": { - "count": 1 - }, - "jsx-a11y/no-static-element-interactions": { - "count": 1 - } - }, "web/app/components/workflow/nodes/http/components/key-value/key-value-edit/index.tsx": { "typescript/no-explicit-any": { "count": 2 @@ -4500,12 +4406,6 @@ "eslint-react/set-state-in-effect": { "count": 4 }, - "jsx-a11y/click-events-have-key-events": { - "count": 1 - }, - "jsx-a11y/no-static-element-interactions": { - "count": 1 - }, "no-restricted-imports": { "count": 1 } @@ -4877,6 +4777,11 @@ "count": 1 } }, + "web/app/signin/normal-form.tsx": { + "@tanstack/query/prefer-query-options": { + "count": 1 + } + }, "web/app/signup/layout.tsx": { "typescript/no-explicit-any": { "count": 1 @@ -4895,6 +4800,31 @@ "count": 1 } }, + "web/features/agent-v2/agent-detail/configure/components/orchestrate/files/__tests__/index.spec.tsx": { + "@tanstack/query/prefer-query-options": { + "count": 1 + } + }, + "web/features/agent-v2/agent-detail/configure/components/preview/working-directory-panel.tsx": { + "@tanstack/query/prefer-query-options": { + "count": 4 + } + }, + "web/features/agent-v2/agent-detail/configure/use-agent-configure-build-draft.ts": { + "@tanstack/query/prefer-query-options": { + "count": 1 + } + }, + "web/features/agent-v2/agent-detail/logs/__tests__/page.spec.tsx": { + "@tanstack/query/prefer-query-options": { + "count": 1 + } + }, + "web/features/skills/detail/file-editor.tsx": { + "@tanstack/query/prefer-query-options": { + "count": 2 + } + }, "web/features/tag-management/components/tag-item-editor.tsx": { "jsx-a11y/no-autofocus": { "count": 1 @@ -4996,21 +4926,33 @@ } }, "web/service/access-control/use-app-access-control.ts": { + "@tanstack/query/prefer-query-options": { + "count": 2 + }, "no-restricted-imports": { "count": 1 } }, "web/service/access-control/use-member-roles.ts": { + "@tanstack/query/prefer-query-options": { + "count": 2 + }, "no-restricted-imports": { "count": 1 } }, "web/service/access-control/use-workspace-access-rules.ts": { + "@tanstack/query/prefer-query-options": { + "count": 2 + }, "no-restricted-imports": { "count": 1 } }, "web/service/access-control/use-workspace-roles.ts": { + "@tanstack/query/prefer-query-options": { + "count": 6 + }, "no-restricted-imports": { "count": 1 } @@ -5044,6 +4986,11 @@ "count": 3 } }, + "web/service/common.spec.ts": { + "@tanstack/query/prefer-query-options": { + "count": 2 + } + }, "web/service/common.ts": { "no-restricted-imports": { "count": 1 @@ -5087,11 +5034,17 @@ } }, "web/service/knowledge/use-dataset.ts": { + "@tanstack/query/prefer-query-options": { + "count": 8 + }, "no-restricted-imports": { "count": 1 } }, "web/service/knowledge/use-document.ts": { + "@tanstack/query/prefer-query-options": { + "count": 4 + }, "no-restricted-imports": { "count": 1 } @@ -5102,16 +5055,25 @@ } }, "web/service/knowledge/use-import.ts": { + "@tanstack/query/prefer-query-options": { + "count": 2 + }, "no-restricted-imports": { "count": 1 } }, "web/service/knowledge/use-metadata.ts": { + "@tanstack/query/prefer-query-options": { + "count": 8 + }, "no-restricted-imports": { "count": 1 } }, "web/service/knowledge/use-segment.ts": { + "@tanstack/query/prefer-query-options": { + "count": 2 + }, "no-restricted-imports": { "count": 1 } @@ -5157,12 +5119,23 @@ "count": 1 } }, + "web/service/use-common.ts": { + "@tanstack/query/prefer-query-options": { + "count": 12 + } + }, "web/service/use-datasource.ts": { + "@tanstack/query/prefer-query-options": { + "count": 3 + }, "no-restricted-imports": { "count": 1 } }, "web/service/use-endpoints.ts": { + "@tanstack/query/prefer-query-options": { + "count": 2 + }, "no-restricted-imports": { "count": 1 }, @@ -5170,27 +5143,44 @@ "count": 7 } }, + "web/service/use-explore.ts": { + "@tanstack/query/prefer-query-options": { + "count": 2 + } + }, "web/service/use-flow.ts": { "eslint-react/no-unnecessary-use-prefix": { "count": 1 } }, "web/service/use-log.ts": { + "@tanstack/query/prefer-query-options": { + "count": 7 + }, "no-restricted-imports": { "count": 1 } }, "web/service/use-models.ts": { + "@tanstack/query/prefer-query-options": { + "count": 3 + }, "no-restricted-imports": { "count": 1 } }, "web/service/use-pipeline.ts": { + "@tanstack/query/prefer-query-options": { + "count": 9 + }, "no-restricted-imports": { "count": 1 } }, "web/service/use-plugins-auth.ts": { + "@tanstack/query/prefer-query-options": { + "count": 3 + }, "no-restricted-imports": { "count": 1 }, @@ -5198,12 +5188,53 @@ "count": 4 } }, + "web/service/use-plugins.ts": { + "@tanstack/query/prefer-query-options": { + "count": 13 + } + }, + "web/service/use-share.ts": { + "@tanstack/query/prefer-query-options": { + "count": 8 + } + }, + "web/service/use-snippet-workflows.ts": { + "@tanstack/query/prefer-query-options": { + "count": 2 + } + }, + "web/service/use-snippets.ts": { + "@tanstack/query/prefer-query-options": { + "count": 1 + } + }, + "web/service/use-strategy.ts": { + "@tanstack/query/prefer-query-options": { + "count": 2 + } + }, "web/service/use-tools.ts": { + "@tanstack/query/prefer-query-options": { + "count": 15 + }, "no-restricted-imports": { "count": 1 } }, + "web/service/use-triggers.ts": { + "@tanstack/query/prefer-query-options": { + "count": 6 + } + }, + "web/service/use-try-app.ts": { + "@tanstack/query/prefer-query-options": { + "count": 4 + } + }, "web/service/use-workflow.ts": { + "@tanstack/query/prefer-query-options": { + "count": 9 + }, "no-restricted-imports": { "count": 1 }, @@ -5212,6 +5243,9 @@ } }, "web/service/use-workspace.ts": { + "@tanstack/query/prefer-query-options": { + "count": 1 + }, "no-restricted-imports": { "count": 1 } diff --git a/packages/contracts/generated/api/console/explore/zod.gen.ts b/packages/contracts/generated/api/console/explore/zod.gen.ts index 272b2884146..995a5b98a58 100644 --- a/packages/contracts/generated/api/console/explore/zod.gen.ts +++ b/packages/contracts/generated/api/console/explore/zod.gen.ts @@ -137,7 +137,7 @@ export const zLearnDifyAppListResponseWritable = z.object({ }) export const zGetExploreAppsQuery = z.object({ - language: z.string().optional(), + language: z.string().optional().default('en-US'), }) /** @@ -146,7 +146,7 @@ export const zGetExploreAppsQuery = z.object({ export const zGetExploreAppsResponse = zRecommendedAppListResponse export const zGetExploreAppsLearnDifyQuery = z.object({ - language: z.string().optional(), + language: z.string().optional().default('en-US'), }) /** diff --git a/packages/contracts/generated/api/console/notification/orpc.gen.ts b/packages/contracts/generated/api/console/notification/orpc.gen.ts index 9e01e1a62e0..7024ed95c31 100644 --- a/packages/contracts/generated/api/console/notification/orpc.gen.ts +++ b/packages/contracts/generated/api/console/notification/orpc.gen.ts @@ -3,6 +3,7 @@ import { oc } from '@orpc/contract' import * as z from 'zod' import { + zGetNotificationQuery, zGetNotificationResponse, zPostNotificationDismissBody, zPostNotificationDismissResponse, @@ -28,18 +29,19 @@ export const dismiss = { } /** - * Return the active in-product notification for the current user in their interface language (falls back to English if unavailable). The notification is NOT marked as seen here; call POST /notification/dismiss when the user explicitly closes the modal. + * Return the active in-product notification for the current user in the requested language (defaults to English when omitted). Unavailable translations fall back to English, then the first available content. The notification is NOT marked as seen here; call POST /notification/dismiss when the user explicitly closes the modal. */ export const get = oc .route({ description: - 'Return the active in-product notification for the current user in their interface language (falls back to English if unavailable). The notification is NOT marked as seen here; call POST /notification/dismiss when the user explicitly closes the modal.', + 'Return the active in-product notification for the current user in the requested language (defaults to English when omitted). Unavailable translations fall back to English, then the first available content. The notification is NOT marked as seen here; call POST /notification/dismiss when the user explicitly closes the modal.', inputStructure: 'detailed', method: 'GET', operationId: 'getNotification', path: '/notification', tags: ['console'], }) + .input(z.object({ query: zGetNotificationQuery.optional() })) .output(zGetNotificationResponse) export const notification = { diff --git a/packages/contracts/generated/api/console/notification/types.gen.ts b/packages/contracts/generated/api/console/notification/types.gen.ts index 26469735f21..f89148d641a 100644 --- a/packages/contracts/generated/api/console/notification/types.gen.ts +++ b/packages/contracts/generated/api/console/notification/types.gen.ts @@ -30,7 +30,9 @@ export type NotificationItemResponse = { export type GetNotificationData = { body?: never path?: never - query?: never + query?: { + language?: string + } url: '/notification' } diff --git a/packages/contracts/generated/api/console/notification/zod.gen.ts b/packages/contracts/generated/api/console/notification/zod.gen.ts index fb39eada2d2..4fadd36755a 100644 --- a/packages/contracts/generated/api/console/notification/zod.gen.ts +++ b/packages/contracts/generated/api/console/notification/zod.gen.ts @@ -37,6 +37,10 @@ export const zNotificationResponse = z.object({ should_show: z.boolean(), }) +export const zGetNotificationQuery = z.object({ + language: z.string().optional().default('en-US'), +}) + /** * Success — inspect should_show to decide whether to render the modal */ diff --git a/packages/contracts/generated/enterprise-app-deploy/types.gen.ts b/packages/contracts/generated/enterprise-app-deploy/types.gen.ts index c64c433d4b6..f7c6aa4f4b9 100644 --- a/packages/contracts/generated/enterprise-app-deploy/types.gen.ts +++ b/packages/contracts/generated/enterprise-app-deploy/types.gen.ts @@ -4,6 +4,8 @@ export type ClientOptions = { baseUrl: `${string}://${string}` | (string & {}) } +export type GoogleProtobufValue = unknown + export const EnvironmentStatus = { ENVIRONMENT_STATUS_UNSPECIFIED: 'ENVIRONMENT_STATUS_UNSPECIFIED', ENVIRONMENT_STATUS_PENDING: 'ENVIRONMENT_STATUS_PENDING', @@ -21,11 +23,27 @@ export const ApplicationInteractionStatus = { APPLICATION_INTERACTION_STATUS_FAILED: 'APPLICATION_INTERACTION_STATUS_FAILED', APPLICATION_INTERACTION_STATUS_PARTIAL_SUCCEEDED: 'APPLICATION_INTERACTION_STATUS_PARTIAL_SUCCEEDED', + APPLICATION_INTERACTION_STATUS_STOPPED: 'APPLICATION_INTERACTION_STATUS_STOPPED', + APPLICATION_INTERACTION_STATUS_PAUSED: 'APPLICATION_INTERACTION_STATUS_PAUSED', } as const export type ApplicationInteractionStatus = (typeof ApplicationInteractionStatus)[keyof typeof ApplicationInteractionStatus] +export const ApplicationInteractionSource = { + APPLICATION_INTERACTION_SOURCE_UNSPECIFIED: 'APPLICATION_INTERACTION_SOURCE_UNSPECIFIED', + APPLICATION_INTERACTION_SOURCE_WEB_APP: 'APPLICATION_INTERACTION_SOURCE_WEB_APP', + APPLICATION_INTERACTION_SOURCE_SERVICE_API: 'APPLICATION_INTERACTION_SOURCE_SERVICE_API', + APPLICATION_INTERACTION_SOURCE_TRIGGER: 'APPLICATION_INTERACTION_SOURCE_TRIGGER', + APPLICATION_INTERACTION_SOURCE_EXPLORE: 'APPLICATION_INTERACTION_SOURCE_EXPLORE', + APPLICATION_INTERACTION_SOURCE_DEBUGGER: 'APPLICATION_INTERACTION_SOURCE_DEBUGGER', + APPLICATION_INTERACTION_SOURCE_VALIDATION: 'APPLICATION_INTERACTION_SOURCE_VALIDATION', + APPLICATION_INTERACTION_SOURCE_OPENAPI: 'APPLICATION_INTERACTION_SOURCE_OPENAPI', +} as const + +export type ApplicationInteractionSource = + (typeof ApplicationInteractionSource)[keyof typeof ApplicationInteractionSource] + export const EnvironmentMode = { ENVIRONMENT_MODE_UNSPECIFIED: 'ENVIRONMENT_MODE_UNSPECIFIED', ENVIRONMENT_MODE_SHARED: 'ENVIRONMENT_MODE_SHARED', @@ -124,36 +142,17 @@ export const EnvironmentBackend = { export type EnvironmentBackend = (typeof EnvironmentBackend)[keyof typeof EnvironmentBackend] -export const EnvironmentManagedBy = { - ENVIRONMENT_MANAGED_BY_UNSPECIFIED: 'ENVIRONMENT_MANAGED_BY_UNSPECIFIED', - ENVIRONMENT_MANAGED_BY_SYSTEM: 'ENVIRONMENT_MANAGED_BY_SYSTEM', - ENVIRONMENT_MANAGED_BY_USER: 'ENVIRONMENT_MANAGED_BY_USER', +export const RuntimeState = { + RUNTIME_STATE_UNSPECIFIED: 'RUNTIME_STATE_UNSPECIFIED', + RUNTIME_STATE_UNDEPLOYED: 'RUNTIME_STATE_UNDEPLOYED', + RUNTIME_STATE_RUNNING: 'RUNTIME_STATE_RUNNING', + RUNTIME_STATE_STARTING: 'RUNTIME_STATE_STARTING', + RUNTIME_STATE_STOPPING: 'RUNTIME_STATE_STOPPING', + RUNTIME_STATE_ERROR: 'RUNTIME_STATE_ERROR', + RUNTIME_STATE_UNKNOWN: 'RUNTIME_STATE_UNKNOWN', } as const -export type EnvironmentManagedBy = (typeof EnvironmentManagedBy)[keyof typeof EnvironmentManagedBy] - -export const EnvironmentDeployedAppStatus = { - ENVIRONMENT_DEPLOYED_APP_STATUS_UNSPECIFIED: 'ENVIRONMENT_DEPLOYED_APP_STATUS_UNSPECIFIED', - ENVIRONMENT_DEPLOYED_APP_STATUS_DEPLOYED: 'ENVIRONMENT_DEPLOYED_APP_STATUS_DEPLOYED', - ENVIRONMENT_DEPLOYED_APP_STATUS_DEPLOYING: 'ENVIRONMENT_DEPLOYED_APP_STATUS_DEPLOYING', - ENVIRONMENT_DEPLOYED_APP_STATUS_FAILED: 'ENVIRONMENT_DEPLOYED_APP_STATUS_FAILED', - ENVIRONMENT_DEPLOYED_APP_STATUS_UNDEPLOYED: 'ENVIRONMENT_DEPLOYED_APP_STATUS_UNDEPLOYED', -} as const - -export type EnvironmentDeployedAppStatus = - (typeof EnvironmentDeployedAppStatus)[keyof typeof EnvironmentDeployedAppStatus] - -export const DeploymentStatus = { - DEPLOYMENT_STATUS_UNSPECIFIED: 'DEPLOYMENT_STATUS_UNSPECIFIED', - DEPLOYMENT_STATUS_UNDEPLOYED: 'DEPLOYMENT_STATUS_UNDEPLOYED', - DEPLOYMENT_STATUS_DEPLOYING: 'DEPLOYMENT_STATUS_DEPLOYING', - DEPLOYMENT_STATUS_RUNNING: 'DEPLOYMENT_STATUS_RUNNING', - DEPLOYMENT_STATUS_UNDEPLOYING: 'DEPLOYMENT_STATUS_UNDEPLOYING', - DEPLOYMENT_STATUS_INVALID: 'DEPLOYMENT_STATUS_INVALID', - DEPLOYMENT_STATUS_FAILED: 'DEPLOYMENT_STATUS_FAILED', -} as const - -export type DeploymentStatus = (typeof DeploymentStatus)[keyof typeof DeploymentStatus] +export type RuntimeState = (typeof RuntimeState)[keyof typeof RuntimeState] export const EnvVarValueSource = { ENV_VAR_VALUE_SOURCE_UNSPECIFIED: 'ENV_VAR_VALUE_SOURCE_UNSPECIFIED', @@ -169,6 +168,7 @@ export const EnvVarValueType = { ENV_VAR_VALUE_TYPE_STRING: 'ENV_VAR_VALUE_TYPE_STRING', ENV_VAR_VALUE_TYPE_NUMBER: 'ENV_VAR_VALUE_TYPE_NUMBER', ENV_VAR_VALUE_TYPE_SECRET: 'ENV_VAR_VALUE_TYPE_SECRET', + ENV_VAR_VALUE_TYPE_LLM: 'ENV_VAR_VALUE_TYPE_LLM', } as const export type EnvVarValueType = (typeof EnvVarValueType)[keyof typeof EnvVarValueType] @@ -203,22 +203,17 @@ export type AppEnvironment = { export type ApplicationInteraction = { id: string timestamp: string - workflowRunId: string status: ApplicationInteractionStatus durationSeconds: number totalTokens: string workspace: NamedRef environment: NamedRef app: NamedRef - operator?: Operator - invokeFrom: string - traceId: string - difyTraceId: string - deploymentVersionId: string + operator?: NamedRef + source: ApplicationInteractionSource + version?: WorkflowVersion + traceId?: string error?: string - body?: string - attributesJson?: string - resourceAttributesJson?: string } export type BatchGetSourceVersionDeploymentsRequest = { @@ -262,7 +257,7 @@ export type CreateEnvironmentRequest = { displayName: string description?: string mode: EnvironmentMode - cpuPool: number + cpuPoolMillicores: number namespace?: string maxMemoryMib?: string } @@ -292,12 +287,7 @@ export type CredentialSlot = { last_deployed_credential_id?: string icon?: string icon_dark?: string -} - -export type DashboardApp = { - id: string - workspaceId: string - displayName: string + workflow_as_tool_dependency?: WorkflowAsToolDependency } export type DeleteEnvironmentApiKeyResponse = { @@ -308,6 +298,15 @@ export type DeleteEnvironmentResponse = { [key: string]: unknown } +export type DeleteServiceApiConversationRequest = { + conversationId: string + user: string +} + +export type DeleteServiceApiConversationResponse = { + [key: string]: unknown +} + export type DeployWorkflowResponse = { operation: DeploymentOperationReceipt } @@ -358,8 +357,7 @@ export type Environment = { statusMessage: string lastError?: Error namespace?: string - managedBy?: EnvironmentManagedBy - cpuPool: number + cpuPoolMillicores: number createdAt: string updatedAt: string memory?: RunnerMemory @@ -381,6 +379,14 @@ export type EnvironmentAccess = { enable_api: boolean } +export type EnvironmentActivity = { + environmentId: string + invocationCount: string + failedInvocationCount: string + failedDeploymentCount: string + deployedAppCount: string +} + export type EnvironmentApiKey = { id: string type: string @@ -393,13 +399,15 @@ export type EnvironmentDeployedApp = { deploymentId: string workspace: NamedRef app: NamedRef - status: EnvironmentDeployedAppStatus + runtimeState: RuntimeState currentVersion?: WorkflowVersion deployedAt?: string deployedBy?: Operator latestAttempt?: EnvironmentDeployedAppAttempt sizing?: RunnerSizing occupiesPool?: boolean + recentInvocationCount?: string + versionsBehind?: number } export type EnvironmentDeployedAppAttempt = { @@ -412,13 +420,6 @@ export type EnvironmentDeployedAppAttempt = { finalizedAt?: string } -export type EnvironmentDeployedAppSummary = { - total: number - deployed: number - deploying: number - failed: number -} - export type EnvironmentDeployment = { environment: DeploymentEnvironment deployment?: EnvironmentDeploymentState @@ -435,7 +436,7 @@ export type EnvironmentDeploymentOperation = { } export type EnvironmentDeploymentState = { - status: DeploymentStatus + runtimeState: RuntimeState current_version?: WorkflowVersion versions_behind?: number deployed_at?: number @@ -449,17 +450,17 @@ export type EnvironmentMcpServer = { export type EnvironmentPoolComposition = { topApps?: Array - otherCpu?: number + otherCpuMillicores?: number otherAppCount?: number } export type EnvironmentPoolShare = { app: NamedRef - isolatedCpu: number + isolatedCpuMillicores: number } export type EnvironmentPoolUsage = { - occupiedCpu: number + occupiedCpuMillicores: number appCount: number } @@ -478,10 +479,16 @@ export type EnvironmentTrigger = { [key: string]: unknown } +export type EnvironmentVariableGroup = { + from_app?: WorkflowReference + from_workflow_as_tool?: WorkflowAsToolSource + environment_variable_slots: Array +} + export type EnvironmentVariableInput = { key: string value_source: EnvVarValueSource - value?: string + value?: GoogleProtobufValue } export type EnvironmentVariableSlot = { @@ -490,8 +497,8 @@ export type EnvironmentVariableSlot = { description: string has_configured_value: boolean has_last_deployed_value: boolean - configured_value?: string - last_deployed_value?: string + configured_value?: GoogleProtobufValue + last_deployed_value?: GoogleProtobufValue } export type EnvironmentWebAppAccessModeUpdate = { @@ -540,7 +547,6 @@ export type Error = { | 'APPDEPLOY_APP_LOG_INVALID_TIME_RANGE' | 'APPDEPLOY_APP_LOG_INVALID_CURSOR' | 'APPDEPLOY_APP_LOG_CURSOR_FILTER_MISMATCH' - | 'APPDEPLOY_APP_LOG_ID_INVALID' | 'APPDEPLOY_UNSUPPORTED_NODE_TYPE' | 'APPDEPLOY_UNSUPPORTED_TOOL_PROVIDER_TYPE' | 'APPDEPLOY_TOOL_PROVIDER_TYPE_INVALID' @@ -552,15 +558,12 @@ export type Error = { | 'APPDEPLOY_INVALID_WORKFLOW_ID' | 'APPDEPLOY_INVALID_DEPLOYMENT_VERSION_ID' | 'APPDEPLOY_DEVELOPER_API_URL_NOT_CONFIGURED' - | 'APPDEPLOY_INVALID_DEPLOYMENT_OPERATION_ID' - | 'APPDEPLOY_APP_LOG_EXPORT_RANGE_TOO_WIDE' - | 'APPDEPLOY_APP_LOG_EXPORT_TOO_MANY_ROWS' - | 'APPDEPLOY_APP_LOG_EXPORT_TOO_LARGE' | 'APPDEPLOY_UNAUTHORIZED' | 'APPDEPLOY_FORBIDDEN' | 'APPDEPLOY_APP_RUNNER_AUTH_REQUIRED' | 'APPDEPLOY_APP_RUNNER_INVALID_JOIN_TOKEN' | 'APPDEPLOY_APP_RUNNER_INVALID_CONTROL_TOKEN' + | 'APPDEPLOY_WEB_APP_ACCESS_DENIED' | 'APPDEPLOY_ENVIRONMENT_NOT_FOUND' | 'APPDEPLOY_DEPLOYMENT_NOT_FOUND' | 'APPDEPLOY_REVISION_NOT_FOUND' @@ -571,15 +574,15 @@ export type Error = { | 'APPDEPLOY_ACCESS_SUBJECT_NOT_FOUND' | 'APPDEPLOY_API_KEY_NOT_FOUND' | 'APPDEPLOY_SOURCE_VERSION_NOT_FOUND' - | 'APPDEPLOY_APP_LOG_NOT_FOUND' | 'APPDEPLOY_WORKSPACE_NOT_FOUND' | 'APPDEPLOY_APP_RUNNER_NOT_FOUND' | 'APPDEPLOY_WORKFLOW_NOT_FOUND' - | 'APPDEPLOY_DEPLOYMENT_OPERATION_NOT_FOUND' | 'APPDEPLOY_RUN_FILE_NOT_FOUND' | 'APPDEPLOY_APPLICATION_UNAVAILABLE' | 'APPDEPLOY_TARGET_ENVIRONMENT_REMOVED' | 'APPDEPLOY_VERSION_UNAVAILABLE' + | 'APPDEPLOY_CONVERSATION_NOT_FOUND' + | 'APPDEPLOY_CHAT_MESSAGE_NOT_FOUND' | 'APPDEPLOY_CONFLICT' | 'APPDEPLOY_DEPLOYMENT_IN_PROGRESS' | 'APPDEPLOY_ALREADY_UNDEPLOYED' @@ -618,10 +621,13 @@ export type Error = { | 'APPDEPLOY_ENVIRONMENT_CPU_POOL_EXHAUSTED' | 'APPDEPLOY_RESOURCE_NOT_APPLICABLE_FOR_MODE' | 'APPDEPLOY_ENVIRONMENT_CPU_POOL_BELOW_ALLOCATED' + | 'APPDEPLOY_CHAT_CONTEXT_TOO_LARGE' + | 'APPDEPLOY_FILE_GRANT_UNAVAILABLE' | 'APPDEPLOY_APP_RUNNER_CONTROL_NOT_CONFIGURED' | 'APPDEPLOY_RUNTIME_ASSIGNMENT_FAILED' | 'APPDEPLOY_REVISION_TIMEOUT' | 'APPDEPLOY_INTERNAL_ERROR' + | 'APPDEPLOY_RECEIPT_RETRY' | 'APPDEPLOY_ENVIRONMENT_BOOTSTRAP_AUTH_REJECTED' | 'APPDEPLOY_ENVIRONMENT_BOOTSTRAP_NAMESPACE_MISSING' | 'APPDEPLOY_ENVIRONMENT_BOOTSTRAP_INSUFFICIENT_RBAC' @@ -640,12 +646,14 @@ export type Error = { detailCode?: string } -export type GetApplicationInteractionResponse = { - interaction: ApplicationInteraction +export type GetApplicationInteractionSummaryResponse = { + totalCount: string + failedCount: string + lookbackStart: string } -export type GetDeploymentOperationResponse = { - operation: DeploymentOperation +export type GetEnvironmentActivityResponse = { + data: Array } export type GetEnvironmentCapabilitiesResponse = { @@ -673,12 +681,17 @@ export type GetWebAppAccessModeResponse = { accessMode?: string } +export type GetWebAppLoginStatusResponse = { + logged_in?: boolean + app_logged_in?: boolean +} + export type GetWebAppPermissionResponse = { result?: boolean } export type GetWorkflowDeploymentOptionsResponse = { - environment_variable_slots: Array + environment_variable_groups: Array credential_slots: Array } @@ -686,14 +699,15 @@ export type ListAppEnvironmentsResponse = { data: Array } -export type ListApplicationInteractionsResponse = { - data: Array +export type ListApplicationInteractionAppsResponse = { + data: Array pagination: Pagination } -export type ListAppsResponse = { - data: Array - pagination: Pagination +export type ListApplicationInteractionsResponse = { + data: Array + nextPageToken?: string + previousPageToken?: string } export type ListDeploymentOperationsResponse = { @@ -707,7 +721,6 @@ export type ListEnvironmentApiKeysResponse = { export type ListEnvironmentDeployedAppsResponse = { data: Array - summary: EnvironmentDeployedAppSummary pagination: Pagination } @@ -724,11 +737,34 @@ export type ListEnvironmentsResponse = { pagination: Pagination } +export type ListOperationAppsResponse = { + data: Array + pagination: Pagination +} + +export type MintServiceApiFileGrantRequest = { + tenantId?: string + appId?: string + environmentId?: string + user?: string +} + +export type MintServiceApiFileGrantResponse = { + grant?: string + expiresAt?: string +} + export type NamedRef = { id: string displayName: string } +export type OperationApp = { + id?: string + workspaceId?: string + displayName?: string +} + export type Operator = { type: OperatorType id: string @@ -744,6 +780,20 @@ export type PrepareAppDeletionRequest = { appId?: string } +export type RenameServiceApiConversationRequest = { + conversationId: string + user: string + name?: string + autoGenerate?: boolean +} + +export type RenameWebAppConversationRequest = { + appCode: string + conversationId: string + name?: string + autoGenerate?: boolean +} + export type ResolveApiTokenRouteRequest = { token?: string } @@ -753,17 +803,14 @@ export type ResolveApiTokenRouteResponse = { namespace?: string serviceName?: string servicePort?: number - environmentStatus?: EnvironmentStatus appId?: string tenantId?: string deploymentId?: string servingRevisionId?: string - deploymentStatus?: DeploymentStatus - revoked?: boolean - unavailableReason?: string targetKind?: RouteTargetKind directUpstream?: string - deploymentGeneration?: string + assignmentGeneration?: string + decision?: string } export type ResolveWebAppRouteRequest = { @@ -777,18 +824,18 @@ export type ResolveWebAppRouteResponse = { namespace?: string serviceName?: string servicePort?: number - environmentStatus?: EnvironmentStatus appId?: string tenantId?: string deploymentId?: string servingRevisionId?: string - deploymentStatus?: DeploymentStatus - unavailableReason?: string targetKind?: RouteTargetKind directUpstream?: string - deploymentGeneration?: string - endUserId?: string - authType?: string + assignmentGeneration?: string + userId?: string + userFrom?: string + userAuthType?: string + fileGrant?: string + fileGrantExpiresAt?: string } export type RetryEnvironmentBootstrapRequest = { @@ -806,7 +853,7 @@ export type RunnerMemory = { } export type RunnerSizing = { - isolatedCpu: number + isolatedCpuMillicores: number memory: RunnerMemory } @@ -818,7 +865,12 @@ export type SimpleAccount = { export type SourceVersionDeployment = { sourceVersionId?: string - environments?: Array + environments?: Array +} + +export type SourceVersionDeploymentEnvironment = { + id?: string + name?: string } export type TestConnectionRequest = { @@ -843,6 +895,7 @@ export type UnsupportedNode = { type: string title: string provider?: UnsupportedNodeProvider + workflow_as_tool_dependency?: WorkflowAsToolDependency } export type UnsupportedNodeProvider = { @@ -855,15 +908,15 @@ export type UnsupportedNodeProvider = { export type UpdateEnvironmentDeployedAppResourcesRequest = { environmentId: string deploymentId: string - isolatedCpu: number + isolatedCpuMillicores: number maxMemoryMib?: string } export type UpdateEnvironmentDeployedAppResourcesResponse = { deploymentId: string - isolatedCpu: number - allocatedCpuCount: number - poolCpuCount: number + isolatedCpuMillicores: number + allocatedCpuMillicores: number + poolCpuMillicores: number memory: RunnerMemory } @@ -871,7 +924,7 @@ export type UpdateEnvironmentRequest = { environmentId?: string displayName?: string description?: string - cpuPool?: number + cpuPoolMillicores?: number maxMemoryMib?: string } @@ -879,16 +932,46 @@ export type UpdateEnvironmentResponse = { environment: Environment } -export type WorkflowDeploymentEnvironment = { - id?: string - name?: string +export type UpdateServiceApiConversationVariableRequest = { + conversationId: string + variableId: string + user: string + value: GoogleProtobufValue +} + +export type WorkflowAsToolDependency = { + paths: Array +} + +export type WorkflowAsToolSource = { + workflow: WorkflowReference + paths: Array } export type WorkflowDeploymentInput = { - environment_variables?: Array + environment_variable_groups: Array credentials?: Array } +export type WorkflowEnvironmentVariableInputGroup = { + workflow_id: string + environment_variables: Array +} + +export type WorkflowPath = { + workflows: Array +} + +export type WorkflowReference = { + app_id: string + workflow_id: string + name: string + icon: string + icon_background: string + icon_type: string + icon_url?: string +} + export type WorkflowVersion = { version?: string marked_name?: string @@ -898,7 +981,6 @@ export type WorkflowVersion = { created_at?: number created_by?: SimpleAccount dsl_hash?: string - deleted?: boolean } export type Pagination = { @@ -920,6 +1002,10 @@ export type DeleteEnvironmentResponseWritable = { [key: string]: unknown } +export type DeleteServiceApiConversationResponseWritable = { + [key: string]: unknown +} + export type EnvironmentWritable = { id: string displayName: string @@ -930,8 +1016,7 @@ export type EnvironmentWritable = { statusMessage: string lastError?: Error namespace?: string - managedBy?: EnvironmentManagedBy - cpuPool: number + cpuPoolMillicores: number createdAt: string updatedAt: string memory?: RunnerMemoryWritable @@ -942,13 +1027,15 @@ export type EnvironmentDeployedAppWritable = { deploymentId: string workspace: NamedRef app: NamedRef - status: EnvironmentDeployedAppStatus + runtimeState: RuntimeState currentVersion?: WorkflowVersion deployedAt?: string deployedBy?: Operator latestAttempt?: EnvironmentDeployedAppAttempt sizing?: RunnerSizingWritable occupiesPool?: boolean + recentInvocationCount?: string + versionsBehind?: number } export type EnvironmentMcpServerWritable = { @@ -966,7 +1053,6 @@ export type GetEnvironmentResponseWritable = { export type ListEnvironmentDeployedAppsResponseWritable = { data: Array - summary: EnvironmentDeployedAppSummary pagination: Pagination } @@ -984,15 +1070,15 @@ export type RunnerMemoryWritable = { } export type RunnerSizingWritable = { - isolatedCpu: number + isolatedCpuMillicores: number memory: RunnerMemoryWritable } export type UpdateEnvironmentDeployedAppResourcesResponseWritable = { deploymentId: string - isolatedCpu: number - allocatedCpuCount: number - poolCpuCount: number + isolatedCpuMillicores: number + allocatedCpuMillicores: number + poolCpuMillicores: number memory: RunnerMemoryWritable } diff --git a/packages/contracts/generated/enterprise-app-deploy/zod.gen.ts b/packages/contracts/generated/enterprise-app-deploy/zod.gen.ts index c823bce2015..ae2591ffbf2 100644 --- a/packages/contracts/generated/enterprise-app-deploy/zod.gen.ts +++ b/packages/contracts/generated/enterprise-app-deploy/zod.gen.ts @@ -2,6 +2,11 @@ import * as z from 'zod' +/** + * Represents a dynamically typed value which can be either null, a number, a string, a boolean, a recursive struct value, or a list of values. + */ +export const zGoogleProtobufValue = z.unknown() + export const zEnvironmentStatus = z.enum([ 'ENVIRONMENT_STATUS_UNSPECIFIED', 'ENVIRONMENT_STATUS_PENDING', @@ -16,6 +21,19 @@ export const zApplicationInteractionStatus = z.enum([ 'APPLICATION_INTERACTION_STATUS_SUCCEEDED', 'APPLICATION_INTERACTION_STATUS_FAILED', 'APPLICATION_INTERACTION_STATUS_PARTIAL_SUCCEEDED', + 'APPLICATION_INTERACTION_STATUS_STOPPED', + 'APPLICATION_INTERACTION_STATUS_PAUSED', +]) + +export const zApplicationInteractionSource = z.enum([ + 'APPLICATION_INTERACTION_SOURCE_UNSPECIFIED', + 'APPLICATION_INTERACTION_SOURCE_WEB_APP', + 'APPLICATION_INTERACTION_SOURCE_SERVICE_API', + 'APPLICATION_INTERACTION_SOURCE_TRIGGER', + 'APPLICATION_INTERACTION_SOURCE_EXPLORE', + 'APPLICATION_INTERACTION_SOURCE_DEBUGGER', + 'APPLICATION_INTERACTION_SOURCE_VALIDATION', + 'APPLICATION_INTERACTION_SOURCE_OPENAPI', ]) export const zEnvironmentMode = z.enum([ @@ -79,28 +97,14 @@ export const zEnvironmentBackend = z.enum([ 'ENVIRONMENT_BACKEND_EXTERNAL', ]) -export const zEnvironmentManagedBy = z.enum([ - 'ENVIRONMENT_MANAGED_BY_UNSPECIFIED', - 'ENVIRONMENT_MANAGED_BY_SYSTEM', - 'ENVIRONMENT_MANAGED_BY_USER', -]) - -export const zEnvironmentDeployedAppStatus = z.enum([ - 'ENVIRONMENT_DEPLOYED_APP_STATUS_UNSPECIFIED', - 'ENVIRONMENT_DEPLOYED_APP_STATUS_DEPLOYED', - 'ENVIRONMENT_DEPLOYED_APP_STATUS_DEPLOYING', - 'ENVIRONMENT_DEPLOYED_APP_STATUS_FAILED', - 'ENVIRONMENT_DEPLOYED_APP_STATUS_UNDEPLOYED', -]) - -export const zDeploymentStatus = z.enum([ - 'DEPLOYMENT_STATUS_UNSPECIFIED', - 'DEPLOYMENT_STATUS_UNDEPLOYED', - 'DEPLOYMENT_STATUS_DEPLOYING', - 'DEPLOYMENT_STATUS_RUNNING', - 'DEPLOYMENT_STATUS_UNDEPLOYING', - 'DEPLOYMENT_STATUS_INVALID', - 'DEPLOYMENT_STATUS_FAILED', +export const zRuntimeState = z.enum([ + 'RUNTIME_STATE_UNSPECIFIED', + 'RUNTIME_STATE_UNDEPLOYED', + 'RUNTIME_STATE_RUNNING', + 'RUNTIME_STATE_STARTING', + 'RUNTIME_STATE_STOPPING', + 'RUNTIME_STATE_ERROR', + 'RUNTIME_STATE_UNKNOWN', ]) export const zEnvVarValueSource = z.enum([ @@ -115,6 +119,7 @@ export const zEnvVarValueType = z.enum([ 'ENV_VAR_VALUE_TYPE_STRING', 'ENV_VAR_VALUE_TYPE_NUMBER', 'ENV_VAR_VALUE_TYPE_SECRET', + 'ENV_VAR_VALUE_TYPE_LLM', ]) export const zOperatorType = z.enum([ @@ -173,7 +178,10 @@ export const zCreateEnvironmentRequest = z.object({ displayName: z.string(), description: z.string().optional(), mode: zEnvironmentMode, - cpuPool: z.number(), + cpuPoolMillicores: z + .int() + .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) + .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }), namespace: z.string().optional(), maxMemoryMib: z.string().optional(), }) @@ -192,25 +200,17 @@ export const zCredentialSelectionInput = z.object({ credential_id: z.string(), }) -export const zCredentialSlot = z.object({ - provider_id: z.string(), - category: zPluginCategory, - candidates: z.array(zCredentialCandidate), - last_deployed_credential_id: z.string().optional(), - icon: z.string().optional(), - icon_dark: z.string().optional(), -}) - -export const zDashboardApp = z.object({ - id: z.string(), - workspaceId: z.string(), - displayName: z.string(), -}) - export const zDeleteEnvironmentApiKeyResponse = z.record(z.string(), z.unknown()) export const zDeleteEnvironmentResponse = z.record(z.string(), z.unknown()) +export const zDeleteServiceApiConversationRequest = z.object({ + conversationId: z.string(), + user: z.string(), +}) + +export const zDeleteServiceApiConversationResponse = z.record(z.string(), z.unknown()) + export const zDeploymentEnvironment = z.object({ id: z.string(), display_name: z.string(), @@ -257,6 +257,18 @@ export const zEnvironmentAccess = z.object({ enable_api: z.boolean(), }) +/** + * EnvironmentActivity reports a numerator and a denominator rather than a rate, + * so a caller can tell a healthy environment from one nothing has called. + */ +export const zEnvironmentActivity = z.object({ + environmentId: z.string(), + invocationCount: z.string(), + failedInvocationCount: z.string(), + failedDeploymentCount: z.string(), + deployedAppCount: z.string(), +}) + export const zEnvironmentApiKey = z.object({ id: z.string(), type: z.string(), @@ -275,29 +287,13 @@ export const zEnvironmentDeployedAppAttempt = z.object({ finalizedAt: z.iso.datetime().optional(), }) -export const zEnvironmentDeployedAppSummary = z.object({ - total: z - .int() - .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) - .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }), - deployed: z - .int() - .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) - .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }), - deploying: z - .int() - .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) - .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }), - failed: z - .int() - .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) - .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }), -}) - export const zEnvironmentMcpServer = z.record(z.string(), z.unknown()) export const zEnvironmentPoolUsage = z.object({ - occupiedCpu: z.number(), + occupiedCpuMillicores: z + .int() + .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) + .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }), appCount: z .int() .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) @@ -320,7 +316,7 @@ export const zEnvironmentTrigger = z.record(z.string(), z.unknown()) export const zEnvironmentVariableInput = z.object({ key: z.string(), value_source: zEnvVarValueSource, - value: z.string().optional(), + value: zGoogleProtobufValue.optional(), }) export const zEnvironmentVariableSlot = z.object({ @@ -329,8 +325,8 @@ export const zEnvironmentVariableSlot = z.object({ description: z.string(), has_configured_value: z.boolean(), has_last_deployed_value: z.boolean(), - configured_value: z.string().optional(), - last_deployed_value: z.string().optional(), + configured_value: zGoogleProtobufValue.optional(), + last_deployed_value: zGoogleProtobufValue.optional(), }) export const zEnvironmentWebAppSubjectAccountData = z.object({ @@ -384,7 +380,6 @@ export const zError = z.object({ 'APPDEPLOY_APP_LOG_INVALID_TIME_RANGE', 'APPDEPLOY_APP_LOG_INVALID_CURSOR', 'APPDEPLOY_APP_LOG_CURSOR_FILTER_MISMATCH', - 'APPDEPLOY_APP_LOG_ID_INVALID', 'APPDEPLOY_UNSUPPORTED_NODE_TYPE', 'APPDEPLOY_UNSUPPORTED_TOOL_PROVIDER_TYPE', 'APPDEPLOY_TOOL_PROVIDER_TYPE_INVALID', @@ -396,15 +391,12 @@ export const zError = z.object({ 'APPDEPLOY_INVALID_WORKFLOW_ID', 'APPDEPLOY_INVALID_DEPLOYMENT_VERSION_ID', 'APPDEPLOY_DEVELOPER_API_URL_NOT_CONFIGURED', - 'APPDEPLOY_INVALID_DEPLOYMENT_OPERATION_ID', - 'APPDEPLOY_APP_LOG_EXPORT_RANGE_TOO_WIDE', - 'APPDEPLOY_APP_LOG_EXPORT_TOO_MANY_ROWS', - 'APPDEPLOY_APP_LOG_EXPORT_TOO_LARGE', 'APPDEPLOY_UNAUTHORIZED', 'APPDEPLOY_FORBIDDEN', 'APPDEPLOY_APP_RUNNER_AUTH_REQUIRED', 'APPDEPLOY_APP_RUNNER_INVALID_JOIN_TOKEN', 'APPDEPLOY_APP_RUNNER_INVALID_CONTROL_TOKEN', + 'APPDEPLOY_WEB_APP_ACCESS_DENIED', 'APPDEPLOY_ENVIRONMENT_NOT_FOUND', 'APPDEPLOY_DEPLOYMENT_NOT_FOUND', 'APPDEPLOY_REVISION_NOT_FOUND', @@ -415,15 +407,15 @@ export const zError = z.object({ 'APPDEPLOY_ACCESS_SUBJECT_NOT_FOUND', 'APPDEPLOY_API_KEY_NOT_FOUND', 'APPDEPLOY_SOURCE_VERSION_NOT_FOUND', - 'APPDEPLOY_APP_LOG_NOT_FOUND', 'APPDEPLOY_WORKSPACE_NOT_FOUND', 'APPDEPLOY_APP_RUNNER_NOT_FOUND', 'APPDEPLOY_WORKFLOW_NOT_FOUND', - 'APPDEPLOY_DEPLOYMENT_OPERATION_NOT_FOUND', 'APPDEPLOY_RUN_FILE_NOT_FOUND', 'APPDEPLOY_APPLICATION_UNAVAILABLE', 'APPDEPLOY_TARGET_ENVIRONMENT_REMOVED', 'APPDEPLOY_VERSION_UNAVAILABLE', + 'APPDEPLOY_CONVERSATION_NOT_FOUND', + 'APPDEPLOY_CHAT_MESSAGE_NOT_FOUND', 'APPDEPLOY_CONFLICT', 'APPDEPLOY_DEPLOYMENT_IN_PROGRESS', 'APPDEPLOY_ALREADY_UNDEPLOYED', @@ -462,10 +454,13 @@ export const zError = z.object({ 'APPDEPLOY_ENVIRONMENT_CPU_POOL_EXHAUSTED', 'APPDEPLOY_RESOURCE_NOT_APPLICABLE_FOR_MODE', 'APPDEPLOY_ENVIRONMENT_CPU_POOL_BELOW_ALLOCATED', + 'APPDEPLOY_CHAT_CONTEXT_TOO_LARGE', + 'APPDEPLOY_FILE_GRANT_UNAVAILABLE', 'APPDEPLOY_APP_RUNNER_CONTROL_NOT_CONFIGURED', 'APPDEPLOY_RUNTIME_ASSIGNMENT_FAILED', 'APPDEPLOY_REVISION_TIMEOUT', 'APPDEPLOY_INTERNAL_ERROR', + 'APPDEPLOY_RECEIPT_RETRY', 'APPDEPLOY_ENVIRONMENT_BOOTSTRAP_AUTH_REJECTED', 'APPDEPLOY_ENVIRONMENT_BOOTSTRAP_NAMESPACE_MISSING', 'APPDEPLOY_ENVIRONMENT_BOOTSTRAP_INSUFFICIENT_RBAC', @@ -486,6 +481,16 @@ export const zError = z.object({ detailCode: z.string().optional(), }) +export const zGetApplicationInteractionSummaryResponse = z.object({ + totalCount: z.string(), + failedCount: z.string(), + lookbackStart: z.iso.datetime(), +}) + +export const zGetEnvironmentActivityResponse = z.object({ + data: z.array(zEnvironmentActivity), +}) + export const zGetEnvironmentCapabilitiesResponse = z.object({ backend: zEnvironmentBackend, supportedModes: z.array( @@ -506,13 +511,13 @@ export const zGetWebAppAccessModeResponse = z.object({ accessMode: z.string().optional(), }) -export const zGetWebAppPermissionResponse = z.object({ - result: z.boolean().optional(), +export const zGetWebAppLoginStatusResponse = z.object({ + logged_in: z.boolean().optional(), + app_logged_in: z.boolean().optional(), }) -export const zGetWorkflowDeploymentOptionsResponse = z.object({ - environment_variable_slots: z.array(zEnvironmentVariableSlot), - credential_slots: z.array(zCredentialSlot), +export const zGetWebAppPermissionResponse = z.object({ + result: z.boolean().optional(), }) export const zListAppEnvironmentsResponse = z.object({ @@ -527,6 +532,18 @@ export const zListEnvironmentTriggersResponse = z.object({ data: z.array(zEnvironmentTrigger), }) +export const zMintServiceApiFileGrantRequest = z.object({ + tenantId: z.string().optional(), + appId: z.string().optional(), + environmentId: z.string().optional(), + user: z.string().optional(), +}) + +export const zMintServiceApiFileGrantResponse = z.object({ + grant: z.string().optional(), + expiresAt: z.string().optional(), +}) + export const zNamedRef = z.object({ id: z.string(), displayName: z.string(), @@ -534,7 +551,10 @@ export const zNamedRef = z.object({ export const zEnvironmentPoolShare = z.object({ app: zNamedRef, - isolatedCpu: z.number(), + isolatedCpuMillicores: z + .int() + .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) + .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }), }) /** @@ -544,7 +564,11 @@ export const zEnvironmentPoolShare = z.object({ */ export const zEnvironmentPoolComposition = z.object({ topApps: z.array(zEnvironmentPoolShare).optional(), - otherCpu: z.number().optional(), + otherCpuMillicores: z + .int() + .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) + .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }) + .optional(), otherAppCount: z .int() .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) @@ -552,42 +576,37 @@ export const zEnvironmentPoolComposition = z.object({ .optional(), }) +export const zOperationApp = z.object({ + id: z.string().optional(), + workspaceId: z.string().optional(), + displayName: z.string().optional(), +}) + export const zOperator = z.object({ type: zOperatorType, id: z.string(), display_name: z.string(), }) -export const zApplicationInteraction = z.object({ - id: z.string(), - timestamp: z.iso.datetime(), - workflowRunId: z.string(), - status: zApplicationInteractionStatus, - durationSeconds: z.number(), - totalTokens: z.string(), - workspace: zNamedRef, - environment: zNamedRef, - app: zNamedRef, - operator: zOperator.optional(), - invokeFrom: z.string(), - traceId: z.string(), - difyTraceId: z.string(), - deploymentVersionId: z.string(), - error: z.string().optional(), - body: z.string().optional(), - attributesJson: z.string().optional(), - resourceAttributesJson: z.string().optional(), -}) - -export const zGetApplicationInteractionResponse = z.object({ - interaction: zApplicationInteraction, -}) - export const zPrepareAppDeletionRequest = z.object({ tenantId: z.string().optional(), appId: z.string().optional(), }) +export const zRenameServiceApiConversationRequest = z.object({ + conversationId: z.string(), + user: z.string(), + name: z.string().optional(), + autoGenerate: z.boolean().optional(), +}) + +export const zRenameWebAppConversationRequest = z.object({ + appCode: z.string(), + conversationId: z.string(), + name: z.string().optional(), + autoGenerate: z.boolean().optional(), +}) + export const zResolveApiTokenRouteRequest = z.object({ token: z.string().optional(), }) @@ -601,17 +620,14 @@ export const zResolveApiTokenRouteResponse = z.object({ .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }) .optional(), - environmentStatus: zEnvironmentStatus.optional(), appId: z.string().optional(), tenantId: z.string().optional(), deploymentId: z.string().optional(), servingRevisionId: z.string().optional(), - deploymentStatus: zDeploymentStatus.optional(), - revoked: z.boolean().optional(), - unavailableReason: z.string().optional(), targetKind: zRouteTargetKind.optional(), directUpstream: z.string().optional(), - deploymentGeneration: z.string().optional(), + assignmentGeneration: z.string().optional(), + decision: z.string().optional(), }) export const zResolveWebAppRouteRequest = z.object({ @@ -629,18 +645,18 @@ export const zResolveWebAppRouteResponse = z.object({ .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }) .optional(), - environmentStatus: zEnvironmentStatus.optional(), appId: z.string().optional(), tenantId: z.string().optional(), deploymentId: z.string().optional(), servingRevisionId: z.string().optional(), - deploymentStatus: zDeploymentStatus.optional(), - unavailableReason: z.string().optional(), targetKind: zRouteTargetKind.optional(), directUpstream: z.string().optional(), - deploymentGeneration: z.string().optional(), - endUserId: z.string().optional(), - authType: z.string().optional(), + assignmentGeneration: z.string().optional(), + userId: z.string().optional(), + userFrom: z.string().optional(), + userAuthType: z.string().optional(), + fileGrant: z.string().optional(), + fileGrantExpiresAt: z.string().optional(), }) export const zRetryEnvironmentBootstrapRequest = z.object({ @@ -667,8 +683,10 @@ export const zEnvironment = z.object({ statusMessage: z.string(), lastError: zError.optional(), namespace: z.string().optional(), - managedBy: zEnvironmentManagedBy.optional(), - cpuPool: z.number(), + cpuPoolMillicores: z + .int() + .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) + .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }), createdAt: z.iso.datetime(), updatedAt: z.iso.datetime(), memory: zRunnerMemory.optional(), @@ -693,7 +711,10 @@ export const zRetryEnvironmentBootstrapResponse = z.object({ * in an isolated environment; elsewhere the runner belongs to the environment. */ export const zRunnerSizing = z.object({ - isolatedCpu: z.number(), + isolatedCpuMillicores: z + .int() + .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) + .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }), memory: zRunnerMemory, }) @@ -707,6 +728,20 @@ export const zSimpleAccount = z.object({ email: z.string().optional(), }) +export const zSourceVersionDeploymentEnvironment = z.object({ + id: z.string().optional(), + name: z.string().optional(), +}) + +export const zSourceVersionDeployment = z.object({ + sourceVersionId: z.string().optional(), + environments: z.array(zSourceVersionDeploymentEnvironment).optional(), +}) + +export const zBatchGetSourceVersionDeploymentsResponse = z.object({ + items: z.array(zSourceVersionDeployment).optional(), +}) + export const zTestConnectionRequest = z.object({ environmentId: z.string().optional(), }) @@ -731,29 +766,30 @@ export const zUnsupportedNodeProvider = z.object({ provider_name: z.string(), }) -export const zUnsupportedNode = z.object({ - id: z.string(), - type: z.string(), - title: z.string(), - provider: zUnsupportedNodeProvider.optional(), -}) - -export const zPrecheckWorkflowDeploymentResponse = z.object({ - unsupported_nodes: z.array(zUnsupportedNode), -}) - export const zUpdateEnvironmentDeployedAppResourcesRequest = z.object({ environmentId: z.string(), deploymentId: z.string(), - isolatedCpu: z.number(), + isolatedCpuMillicores: z + .int() + .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) + .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }), maxMemoryMib: z.string().optional(), }) export const zUpdateEnvironmentDeployedAppResourcesResponse = z.object({ deploymentId: z.string(), - isolatedCpu: z.number(), - allocatedCpuCount: z.number(), - poolCpuCount: z.number(), + isolatedCpuMillicores: z + .int() + .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) + .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }), + allocatedCpuMillicores: z + .int() + .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) + .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }), + poolCpuMillicores: z + .int() + .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) + .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }), memory: zRunnerMemory, }) @@ -761,7 +797,11 @@ export const zUpdateEnvironmentRequest = z.object({ environmentId: z.string().optional(), displayName: z.string().optional(), description: z.string().optional(), - cpuPool: z.number().optional(), + cpuPoolMillicores: z + .int() + .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) + .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }) + .optional(), maxMemoryMib: z.string().optional(), }) @@ -769,25 +809,79 @@ export const zUpdateEnvironmentResponse = z.object({ environment: zEnvironment, }) -export const zWorkflowDeploymentEnvironment = z.object({ - id: z.string().optional(), - name: z.string().optional(), +export const zUpdateServiceApiConversationVariableRequest = z.object({ + conversationId: z.string(), + variableId: z.string(), + user: z.string(), + value: zGoogleProtobufValue, }) -export const zSourceVersionDeployment = z.object({ - sourceVersionId: z.string().optional(), - environments: z.array(zWorkflowDeploymentEnvironment).optional(), -}) - -export const zBatchGetSourceVersionDeploymentsResponse = z.object({ - items: z.array(zSourceVersionDeployment).optional(), +export const zWorkflowEnvironmentVariableInputGroup = z.object({ + workflow_id: z.string(), + environment_variables: z.array(zEnvironmentVariableInput), }) export const zWorkflowDeploymentInput = z.object({ - environment_variables: z.array(zEnvironmentVariableInput).optional(), + environment_variable_groups: z.array(zWorkflowEnvironmentVariableInputGroup), credentials: z.array(zCredentialSelectionInput).optional(), }) +export const zWorkflowReference = z.object({ + app_id: z.string(), + workflow_id: z.string(), + name: z.string(), + icon: z.string(), + icon_background: z.string(), + icon_type: z.string(), + icon_url: z.string().optional(), +}) + +export const zWorkflowPath = z.object({ + workflows: z.array(zWorkflowReference), +}) + +export const zWorkflowAsToolDependency = z.object({ + paths: z.array(zWorkflowPath), +}) + +export const zCredentialSlot = z.object({ + provider_id: z.string(), + category: zPluginCategory, + candidates: z.array(zCredentialCandidate), + last_deployed_credential_id: z.string().optional(), + icon: z.string().optional(), + icon_dark: z.string().optional(), + workflow_as_tool_dependency: zWorkflowAsToolDependency.optional(), +}) + +export const zUnsupportedNode = z.object({ + id: z.string(), + type: z.string(), + title: z.string(), + provider: zUnsupportedNodeProvider.optional(), + workflow_as_tool_dependency: zWorkflowAsToolDependency.optional(), +}) + +export const zPrecheckWorkflowDeploymentResponse = z.object({ + unsupported_nodes: z.array(zUnsupportedNode), +}) + +export const zWorkflowAsToolSource = z.object({ + workflow: zWorkflowReference, + paths: z.array(zWorkflowPath), +}) + +export const zEnvironmentVariableGroup = z.object({ + from_app: zWorkflowReference.optional(), + from_workflow_as_tool: zWorkflowAsToolSource.optional(), + environment_variable_slots: z.array(zEnvironmentVariableSlot), +}) + +export const zGetWorkflowDeploymentOptionsResponse = z.object({ + environment_variable_groups: z.array(zEnvironmentVariableGroup), + credential_slots: z.array(zCredentialSlot), +}) + export const zWorkflowVersion = z.object({ version: z.string().optional(), marked_name: z.string().optional(), @@ -801,7 +895,22 @@ export const zWorkflowVersion = z.object({ created_at: z.number().optional(), created_by: zSimpleAccount.optional(), dsl_hash: z.string().optional(), - deleted: z.boolean().optional(), +}) + +export const zApplicationInteraction = z.object({ + id: z.string(), + timestamp: z.iso.datetime(), + status: zApplicationInteractionStatus, + durationSeconds: z.number(), + totalTokens: z.string(), + workspace: zNamedRef, + environment: zNamedRef, + app: zNamedRef, + operator: zNamedRef.optional(), + source: zApplicationInteractionSource, + version: zWorkflowVersion.optional(), + traceId: z.string().optional(), + error: z.string().optional(), }) export const zDeploymentOperation = z.object({ @@ -824,13 +933,19 @@ export const zEnvironmentDeployedApp = z.object({ deploymentId: z.string(), workspace: zNamedRef, app: zNamedRef, - status: zEnvironmentDeployedAppStatus, + runtimeState: zRuntimeState, currentVersion: zWorkflowVersion.optional(), deployedAt: z.iso.datetime().optional(), deployedBy: zOperator.optional(), latestAttempt: zEnvironmentDeployedAppAttempt.optional(), sizing: zRunnerSizing.optional(), occupiesPool: z.boolean().optional(), + recentInvocationCount: z.string().optional(), + versionsBehind: z + .int() + .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) + .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }) + .optional(), }) export const zEnvironmentDeploymentOperation = z.object({ @@ -843,7 +958,7 @@ export const zEnvironmentDeploymentOperation = z.object({ }) export const zEnvironmentDeploymentState = z.object({ - status: zDeploymentStatus, + runtimeState: zRuntimeState, current_version: zWorkflowVersion.optional(), versions_behind: z .int() @@ -861,14 +976,16 @@ export const zEnvironmentDeployment = z.object({ access: zEnvironmentAccess, }) -export const zGetDeploymentOperationResponse = z.object({ - operation: zDeploymentOperation, -}) - export const zGetEnvironmentDeploymentResponse = z.object({ environment_deployment: zEnvironmentDeployment, }) +export const zListApplicationInteractionsResponse = z.object({ + data: z.array(zApplicationInteraction), + nextPageToken: z.string().optional(), + previousPageToken: z.string().optional(), +}) + export const zListEnvironmentDeploymentsResponse = z.object({ environment_deployments: z.array(zEnvironmentDeployment), }) @@ -899,13 +1016,8 @@ export const zPagination = z.object({ .optional(), }) -export const zListApplicationInteractionsResponse = z.object({ - data: z.array(zApplicationInteraction), - pagination: zPagination, -}) - -export const zListAppsResponse = z.object({ - data: z.array(zDashboardApp), +export const zListApplicationInteractionAppsResponse = z.object({ + data: z.array(zNamedRef), pagination: zPagination, }) @@ -916,7 +1028,6 @@ export const zListDeploymentOperationsResponse = z.object({ export const zListEnvironmentDeployedAppsResponse = z.object({ data: z.array(zEnvironmentDeployedApp), - summary: zEnvironmentDeployedAppSummary, pagination: zPagination, }) @@ -925,10 +1036,17 @@ export const zListEnvironmentsResponse = z.object({ pagination: zPagination, }) +export const zListOperationAppsResponse = z.object({ + data: z.array(zOperationApp), + pagination: zPagination, +}) + export const zDeleteEnvironmentApiKeyResponseWritable = z.record(z.string(), z.unknown()) export const zDeleteEnvironmentResponseWritable = z.record(z.string(), z.unknown()) +export const zDeleteServiceApiConversationResponseWritable = z.record(z.string(), z.unknown()) + export const zEnvironmentMcpServerWritable = z.record(z.string(), z.unknown()) export const zEnvironmentTriggerWritable = z.record(z.string(), z.unknown()) @@ -951,8 +1069,10 @@ export const zEnvironmentWritable = z.object({ statusMessage: z.string(), lastError: zError.optional(), namespace: z.string().optional(), - managedBy: zEnvironmentManagedBy.optional(), - cpuPool: z.number(), + cpuPoolMillicores: z + .int() + .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) + .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }), createdAt: z.iso.datetime(), updatedAt: z.iso.datetime(), memory: zRunnerMemoryWritable.optional(), @@ -982,7 +1102,10 @@ export const zRetryEnvironmentBootstrapResponseWritable = z.object({ * in an isolated environment; elsewhere the runner belongs to the environment. */ export const zRunnerSizingWritable = z.object({ - isolatedCpu: z.number(), + isolatedCpuMillicores: z + .int() + .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) + .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }), memory: zRunnerMemoryWritable, }) @@ -990,26 +1113,40 @@ export const zEnvironmentDeployedAppWritable = z.object({ deploymentId: z.string(), workspace: zNamedRef, app: zNamedRef, - status: zEnvironmentDeployedAppStatus, + runtimeState: zRuntimeState, currentVersion: zWorkflowVersion.optional(), deployedAt: z.iso.datetime().optional(), deployedBy: zOperator.optional(), latestAttempt: zEnvironmentDeployedAppAttempt.optional(), sizing: zRunnerSizingWritable.optional(), occupiesPool: z.boolean().optional(), + recentInvocationCount: z.string().optional(), + versionsBehind: z + .int() + .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) + .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }) + .optional(), }) export const zListEnvironmentDeployedAppsResponseWritable = z.object({ data: z.array(zEnvironmentDeployedAppWritable), - summary: zEnvironmentDeployedAppSummary, pagination: zPagination, }) export const zUpdateEnvironmentDeployedAppResourcesResponseWritable = z.object({ deploymentId: z.string(), - isolatedCpu: z.number(), - allocatedCpuCount: z.number(), - poolCpuCount: z.number(), + isolatedCpuMillicores: z + .int() + .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) + .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }), + allocatedCpuMillicores: z + .int() + .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) + .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }), + poolCpuMillicores: z + .int() + .min(-2147483648, { error: 'Invalid value: Expected int32 to be >= -2147483648' }) + .max(2147483647, { error: 'Invalid value: Expected int32 to be <= 2147483647' }), memory: zRunnerMemoryWritable, }) diff --git a/packages/contracts/marketplace.ts b/packages/contracts/marketplace.ts index 459f3d778e6..06353129a8f 100644 --- a/packages/contracts/marketplace.ts +++ b/packages/contracts/marketplace.ts @@ -22,6 +22,10 @@ export type MarketplaceCollection = { search_params?: SearchParamsFromCollection } +export type MarketplaceTimestamp = string | number +export type MarketplaceCreatorStatus = 'pending' | 'active' | 'inactive' | 'deleted' +export type MarketplaceOrganizationStatus = 'active' | 'inactive' | 'deleted' + export type PluginsSearchParams = { query: string page?: number @@ -44,6 +48,7 @@ export type CollectionsAndPluginsSearchParams = { condition?: string exclude?: string[] type?: 'plugin' | 'bundle' + limit?: number } export type MarketplaceTemplate = { @@ -53,9 +58,65 @@ export type MarketplaceTemplate = { icon: string icon_background: string icon_file_key: string - publisher_unique_handle: string + publisher_unique_handle?: string + publisher_handle?: string + publisher_type?: string + creator_email?: string usage_count: number categories: string[] + deps_plugins?: string[] + preferred_languages?: string[] + badges?: string[] + created_at?: MarketplaceTimestamp + updated_at?: MarketplaceTimestamp +} + +export type MarketplaceCreator = { + id?: string + email?: string + name?: string + display_name?: string + unique_handle: string + display_email?: string + description?: string + avatar?: string + background_image?: string + social_links?: string[] + badges?: string[] + verified?: boolean + status?: MarketplaceCreatorStatus + public?: boolean + plugin_count?: number + template_count?: number + created_at?: string + updated_at?: string +} + +export type MarketplaceOrganization = { + id?: string + email?: string + name?: string + display_name?: string + unique_handle?: string + display_email?: string + description?: string + avatar?: string + background_image?: string + social_links?: string[] + badges?: string[] + verified?: boolean + status?: MarketplaceOrganizationStatus + created_at?: string + updated_at?: string +} + +export type MarketplaceTemplateCollection = { + name: string + description: Record + label: Record + searchable?: boolean + search_params?: SearchParamsFromCollection + priority: number } export type MarketplacePluginCategory = @@ -109,6 +170,9 @@ export type MarketplacePlugin = { authorized_category: 'langgenius' | 'partner' | 'community' } from: MarketplacePluginDependencySource + created_at?: MarketplaceTimestamp + updated_at?: MarketplaceTimestamp + version_updated_at?: MarketplaceTimestamp | null } export type PluginInfoFromMarketPlace = { @@ -154,8 +218,151 @@ export type TemplateDetailResponse = { data: MarketplaceTemplate } +export type TemplateCollectionsResponse = { + data?: { + collections?: MarketplaceTemplateCollection[] + total?: number + } +} + +export type TemplateCollectionTemplatesResponse = { + data?: { + templates?: MarketplaceTemplate[] + total?: number + } +} + +export type TemplateSearchResponse = { + data?: { + templates?: MarketplaceTemplate[] + total?: number + } +} + export type DownloadPluginResponse = Blob +export type CreatorDetailResponse = { + code?: number + data?: { + creator?: MarketplaceCreator + } + msg?: string +} + +export type OrganizationDetailResponse = { + code?: number + data?: { + organization?: MarketplaceOrganization + } + msg?: string +} + +export type PublisherPluginsResponse = { + code?: number + data?: { + plugins?: MarketplacePlugin[] + total?: number + } + msg?: string +} + +export type PublisherTemplatesResponse = { + code?: number + data?: { + templates?: MarketplaceTemplate[] + total?: number + } + msg?: string +} + +// Banner payload shapes shared by the standalone marketplace and the embedded +// console. The banners endpoint output stays `unknown` in the contract because +// the delivery format is normalized and runtime-validated in +// `web/app/components/plugins/marketplace/home/banners.ts`. +export type BannerBase = { + id: string + title: string + sort: number + language: string +} + +export type BannerRecommendCard = { + item_type: 'plugin' | 'template' + item_id: string + display_name: string + icon_url?: string + icon?: string + icon_background?: string + creator?: string + badges?: Array<'partner' | 'verified'> + link: string + card_position: number + auto_batch_id?: string | null +} + +export type BannerRecommend = BannerBase & { + style_type: 'recommend' + content: { + theme_type: 'newest' | 'hottest' | 'partner' + heading?: string + subheadings?: string[] + description?: string + cards: BannerRecommendCard[] + } +} + +export type BannerBlog = BannerBase & { + style_type: 'blog' + content: { + blog_title: string + subtitle?: string + description?: string + link: string + link_target_type: 'blog' | 'github' + } +} + +export type BannerImageContent = { + images: { + desktop: string + tablet?: string + mobile?: string + } + link: string + alt_text?: string + activity_id?: string +} + +export type BannerEvent = BannerBase & { + style_type: 'event' + content: BannerImageContent +} + +export type BannerAd = BannerBase & { + style_type: 'ad' + content: BannerImageContent & { + partner_id?: string + campaign_id?: string + } +} + +export type PluginBanner = BannerRecommend | BannerBlog | BannerEvent | BannerAd + +const bannerListContract = base + .route({ + path: '/banners', + method: 'GET', + }) + .input( + type<{ + query: { + page: 'plugins' | 'templates' + language: string + } + }>(), + ) + .output(type()) + const collectionsContract = base .route({ path: '/collections', @@ -212,6 +419,58 @@ const templateDetailContract = base ) .output(type()) +const templateCollectionsContract = base + .route({ + path: '/template-collections', + method: 'GET', + }) + .input( + type<{ + query?: { + page?: number + page_size?: number + } + }>(), + ) + .output(type()) + +const templateCollectionTemplatesContract = base + .route({ + path: '/template-collections/{collectionName}/templates', + method: 'POST', + }) + .input( + type<{ + params: { + collectionName: string + } + body?: { + limit?: number + } + }>(), + ) + .output(type()) + +const templateSearchContract = base + .route({ + path: '/templates/search/advanced', + method: 'POST', + }) + .input( + type<{ + body: { + page: number + page_size: number + query: string + sort_by: string + sort_order: string + categories?: string[] + languages?: string[] + } + }>(), + ) + .output(type()) + const downloadPluginContract = base .route({ path: '/plugins/{organization}/{pluginName}/{version}/download', @@ -228,12 +487,90 @@ const downloadPluginContract = base ) .output(type()) +const creatorDetailContract = base + .route({ + path: '/creators/{uniqueHandle}', + method: 'GET', + }) + .input( + type<{ + params: { + uniqueHandle: string + } + }>(), + ) + .output(type()) + +const organizationDetailContract = base + .route({ + path: '/organizations/{id}', + method: 'GET', + }) + .input( + type<{ + params: { + id: string + } + }>(), + ) + .output(type()) + +const publisherPluginsContract = base + .route({ + path: '/plugins/publisher/{uniqueHandle}', + method: 'GET', + }) + .input( + type<{ + params: { + uniqueHandle: string + } + query: { + page: number + page_size: number + sort_by?: string + sort_order?: string + } + }>(), + ) + .output(type()) + +const publisherTemplatesContract = base + .route({ + path: '/templates/publisher/{uniqueHandle}', + method: 'GET', + }) + .input( + type<{ + params: { + uniqueHandle: string + } + query: { + page: number + page_size: number + sort_by?: string + sort_order?: string + } + }>(), + ) + .output(type()) + export const marketplaceRouterContract = { + banners: { + list: bannerListContract, + }, collections: collectionsContract, collectionPlugins: collectionPluginsContract, searchAdvanced: searchAdvancedContract, + templateCollections: templateCollectionsContract, + templateCollectionTemplates: templateCollectionTemplatesContract, templateDetail: templateDetailContract, + templateSearch: templateSearchContract, downloadPlugin: downloadPluginContract, + creatorDetail: creatorDetailContract, + organizationDetail: organizationDetailContract, + publisherPlugins: publisherPluginsContract, + publisherTemplates: publisherTemplatesContract, } export type MarketPlaceInputs = InferContractRouterInputs diff --git a/packages/dify-ui/AGENTS.md b/packages/dify-ui/AGENTS.md index 2cd05f418f9..5e17256c562 100644 --- a/packages/dify-ui/AGENTS.md +++ b/packages/dify-ui/AGENTS.md @@ -19,6 +19,7 @@ then read only the guide for the contract being changed. - Imports, exports, naming, public types, generics, and anatomy: [Public API authoring] - Button and icon-only action behavior: [Button contract] and [Icon Button contract] +- Cross-component accessible-name and description choices: [Accessible names and descriptions] - Compound input behavior: [Input Group contract] - Form structure, labels, and value ownership: [Forms] - Picker choice and typed values: [Selection] @@ -29,6 +30,7 @@ then read only the guide for the contract being changed. A component needs a local README only when it owns a substantial Dify-specific contract that its types, stories, and upstream documentation do not express. Do not create one for completeness. +[Accessible names and descriptions]: docs/accessible-names-and-descriptions.md [Button contract]: src/button/README.md [Forms]: docs/forms.md [Icon Button contract]: src/icon-button/README.md diff --git a/packages/dify-ui/README.md b/packages/dify-ui/README.md index 622b4f556b4..645665fb89e 100644 --- a/packages/dify-ui/README.md +++ b/packages/dify-ui/README.md @@ -68,20 +68,22 @@ Upstream behavior remains owned by the [Base UI documentation]. ### Cross-component guides -| Guide | Scope | -| ------------------------- | ------------------------------------------------------------------------------ | -| [Forms] | Native submit boundaries, value ownership, fields, labels, and errors. | -| [Selection] | Typed values and choosing among segmented controls, pickers, and radio groups. | -| [Overlays] | Portals, presence lifecycles, layering, trigger composition, and semantics. | -| [Styling] | Tailwind CSS integration and the Figma radius mapping. | -| [Public API authoring] | Subpath exports, naming, public types, generics, and private helpers. | -| [Testing and development] | Package commands, test ownership, accessibility, and animation setup. | +| Guide | Scope | +| ----------------------------------- | ------------------------------------------------------------------------------ | +| [Accessible names and descriptions] | Naming sources, descriptions, overrides, and safe label removal. | +| [Forms] | Native submit boundaries, value ownership, fields, labels, and errors. | +| [Selection] | Typed values and choosing among segmented controls, pickers, and radio groups. | +| [Overlays] | Portals, presence lifecycles, layering, trigger composition, and semantics. | +| [Styling] | Tailwind CSS integration and the Figma radius mapping. | +| [Public API authoring] | Subpath exports, naming, public types, generics, and private helpers. | +| [Testing and development] | Package commands, test ownership, accessibility, and animation setup. | ## Contributing Read [component authoring rules] before modifying the package, then open only the matching owner guide. This index intentionally does not duplicate those contracts. +[Accessible names and descriptions]: ./docs/accessible-names-and-descriptions.md [Base UI documentation]: https://base-ui.com/llms.txt [Base UI]: https://base-ui.com/react [Button]: ./src/button/README.md diff --git a/packages/dify-ui/docs/accessible-names-and-descriptions.md b/packages/dify-ui/docs/accessible-names-and-descriptions.md new file mode 100644 index 00000000000..d948c3290ff --- /dev/null +++ b/packages/dify-ui/docs/accessible-names-and-descriptions.md @@ -0,0 +1,175 @@ +# Accessible Names and Descriptions + +This cross-component contract is owned by Dify UI. It applies to Dify UI primitives and to +consumers composing those primitives. It depends only on Dify UI component contracts and upstream +web standards; application packages may add localization, testing, and product-specific rules +without redefining this contract. + +[Base UI accessibility] owns the primitive mechanics it implements, such as roles, relationships, +keyboard interaction, and focus management. Dify UI and its consumers still own the final element, +label content, composition, and product meaning. Use this guide to choose those naming and +description sources. Open a component guide only when the decision reaches that component. + +Use [ARIA in HTML][html-naming] for authoring conformance and the [name and description computation +specification][accname] to understand the current computation model. APG and MDN provide authoring +guidance; Base UI documents the behavior and usage guidance of the primitives Dify wraps. The Dify +conventions below choose among valid options without making a prohibited naming relationship valid. + +## Start Here + +An accessible name is the flat string that identifies a named element to assistive technology; not +every role permits one. An accessible description adds optional help, instructions, or consequences. +State such as checked, expanded, or disabled remains separate from the name, and changing status +remains with the feature's status or live-region owner. + +For each changed element: + +1. Inspect the final rendered element, role, text, and props forwarded by its primitive. +1. Prefer meaningful visible text or a native label relationship. +1. Use `aria-labelledby` when suitable visible text exists elsewhere in the DOM. +1. Use `aria-label` only when the role permits naming and no visible text can provide the name. +1. Add `aria-describedby` only for useful supplemental text; do not repeat the name. +1. Verify the computed name and description in every changed responsive and interaction state. + +These choices follow the [W3C APG naming techniques][apg] and [MDN `aria-label` guidance]. Nearby +text is not a label relationship by proximity alone. + +## Common Decisions + +| Surface | Contract | +| -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | +| Text button or link | Follow [Button]. Let meaningful child text name the action; do not repeat it in `aria-label`. | +| Form control | Follow [Forms]. Use its label primitive or an associated native `label`, preserving label activation. | +| Icon-only command | Follow [IconButton]. Its component-specific contract requires one accessible-name source and a decorative glyph. | +| Dialog or named region | Reuse the visible title through the primitive title API or `aria-labelledby`; use `aria-label` only when no suitable visible title exists. | +| Related form-control group | Follow [Forms]. Use `Fieldset` with `FieldsetLegend` and preserve each control's own label. Other composite widgets follow their owning primitive. | +| Table or figure | Prefer `caption` or `figcaption` when appropriate. See the [APG caption guidance][captions] for name and description differences. | +| Image | Supply meaningful `alt`, or `alt=""` for a decorative image. | +| Plain `div` or `span` | Keep readable content; do not add `aria-label` or `aria-labelledby` to the default `generic` role. | + +Naming permission comes from semantics, not the presence of an `aria-*` prop. Other roles also +prohibit naming. Do not invent a role merely to permit a label. A plain span may contain text +referenced by another element's `aria-labelledby`; that relationship names the referencing element, +not the span. Check [ARIA in HTML][html-naming] for restrictions on the final element. + +## Names, Descriptions, and State + +A description is optional when the name is sufficient. For a file action, the name might identify +the operation and file, while the description explains retention or recovery. Avoid repeating the +same sentence in both. See the [name and description computation specification][accname]. + +A name or description attribute is not an announcement mechanism. Keep progress and asynchronous +updates with their existing feature owner. Follow [Button] for loading behavior and [Forms] for +field error relationships. + +## Overrides and References + +Authoring preference differs from computation priority. An `aria-labelledby` value with at least +one valid ID reference is evaluated first. If its computed text is non-empty, it takes precedence +over `aria-label` and normal native or content naming. If its result is empty, name computation +continues to lower-priority sources; do not rely on that fallback to excuse a broken reference. A +non-empty `aria-label` also overrides normal native or content naming; these sources are not +concatenated. See the [computation steps][computation]. + +- With `aria-labelledby`, reference the intended text directly. Multiple IDs are read in attribute + order; do not build chains of elements that each use `aria-labelledby`. +- Overriding a button or link's content-derived name can suppress meaningful descendant content in + its accessible representation. Preserve the necessary visible wording in the resulting name. +- Inspect IDs generated by primitives before overriding them. Keep IDs unique across repeated rows + and simultaneous dialogs, and ensure referenced nodes exist in relevant open, closed, and + responsive states. Preserve existing description IDs when adding another relationship. +- Do not use `title`, `placeholder`, or Tooltip content as the only naming source. Native `title` + does not replace an intentional name or description relationship. + +## Write Useful Names + +Keep the visible label's wording in the accessible name, preferably at the beginning. Add target +context when identical visible actions would otherwise be ambiguous. Matching visible words also +lets speech-input users invoke what they see. See [WCAG Label in Name][label-in-name]. + +Use concise action or purpose wording. Avoid appending role words already announced by assistive +technology or duplicating state exposed by the control. Consumers own localization and pass the +complete localized text through public props or children; Dify UI primitives do not import +application i18n. + +The following fragment assumes localized strings and owner-scoped unique IDs. It combines the +visible action with the file it affects: + +```tsx +<> + {fileName} + + +``` + +## Associate Descriptions + +Use `aria-describedby` to associate concise help or consequences with a named control. The +referenced content becomes a plain string: headings, lists, and interactive links do not retain +their structure in the description. Keep rich instructions reachable as normal content or through +the [Overlay] contract. `aria-details` may supplement structured content where supported; it does +not replace that reachable content. See [MDN `aria-describedby` guidance][describedby]. + +In Dify fields, compose `FieldDescription` and `FieldError` with the appropriate label and control. +These primitives own their relationships, including invalid-state feedback. Do not overwrite them +with a second label or a competing error association: + +```tsx + + {fileNameLabel} + + {formatHint} + {requiredMessage} + +``` + +Use `DialogTitle` and, when useful, `DialogDescription` for a short dialog summary. Do not turn a +whole form or rich dialog body into one description. Per [Base UI Tooltip guidance], Tooltip is a +supplemental visual label, not the trigger's accessible-name source. Base UI specifically recommends +an `aria-label` that closely matches the Tooltip content; apply that to icon-only triggers. When +persistent visible trigger text already supplies the name, preserve the content-derived name per +W3C and MDN guidance instead of adding a redundant override merely because Tooltip is present. Use +[Overlay] choices for essential, structured, interactive, or touch-reachable information. + +Prefer descriptions associated with DOM text. When considering `aria-description`, verify target +browser and assistive-technology behavior. The [AccName 1.2 working draft] gives +`aria-describedby` precedence over `aria-description`, followed by applicable native description +sources and unused `title` fallback. It specifies using only the first applicable source, even when +that source computes to an empty description. Do not stack mechanisms to force repeated output. + +## Hidden Text and Safe Removal + +- `sr-only` hides text visually while retaining it for assistive technology. It can contribute to + a content-derived name or serve as a referenced label or description. A standalone span does not + name a sibling control, and `sr-only` is not an automatic replacement for `aria-label`. +- `hidden`, `display: none`, `visibility: hidden`, and `aria-hidden="true"` normally exclude content + during name calculation. Explicitly referenced hidden nodes can still contribute; inspect the + reference and its subtree instead of assuming all hidden text is ignored. See the [computation + steps][computation] and [description reference][describedby]. +- Before removing a label, inspect the resulting name and description in collapsed navigation, + responsive icon-only layouts, loading, and disabled states. CSS truncation alone does not remove + underlying text. Preserve primitive relationships and necessary status information. Add hidden + text only when information would otherwise be missing. +- Verify observable names, descriptions, and keyboard behavior at the changed boundary. Follow the + test policy owned by that package or consumer. Dify UI changes follow [Package testing]; complex + overrides may also require inspecting the rendered accessibility tree and relevant screen-reader + behavior. + +[APG]: https://www.w3.org/WAI/ARIA/apg/practices/names-and-descriptions#namingtechniques +[AccName 1.2 working draft]: https://www.w3.org/TR/accname-1.2#mapping_additional_nd_description +[Base UI Tooltip guidance]: https://base-ui.com/react/components/tooltip#usage-guidelines +[Base UI accessibility]: https://base-ui.com/react/overview/accessibility +[Button]: ../src/button/README.md +[Forms]: forms.md +[IconButton]: ../src/icon-button/README.md +[MDN `aria-label` guidance]: https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/Reference/Attributes/aria-label +[Overlay]: overlays.md +[Package testing]: testing.md +[accname]: https://www.w3.org/TR/accname-1.2#name_and_description +[captions]: https://www.w3.org/WAI/ARIA/apg/practices/names-and-descriptions +[computation]: https://www.w3.org/TR/accname-1.2#computation-steps +[describedby]: https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/Reference/Attributes/aria-describedby +[html-naming]: https://www.w3.org/TR/html-aria#requirements-for-use-of-aria-attributes-to-name-elements +[label-in-name]: https://www.w3.org/WAI/WCAG22/Understanding/label-in-name.html diff --git a/packages/dify-ui/docs/selection.md b/packages/dify-ui/docs/selection.md index 16a2a397a88..acdae46dcd6 100644 --- a/packages/dify-ui/docs/selection.md +++ b/packages/dify-ui/docs/selection.md @@ -17,6 +17,12 @@ domain value type through its public API. Multiple-selection comboboxes follow the Base UI chips composition: chips and the input share the input group, chips wrap, and the group grows vertically. +For chip-based multiple comboboxes, render `ComboboxValue` around `ComboboxChips` and label the +chips container only while it has the conditional `toolbar` role. Consumers must localize the +`ComboboxChip` Backspace/Delete description, the item-specific `ComboboxChipRemove` name, and the +input's selection count and Left Arrow hint. If `FieldDescription` already sets `aria-describedby`, +put the input hint there because it takes precedence over `aria-description`. + Autocomplete, Combobox, and Select popups use Base UI's `--anchor-width` and `--available-width` variables to follow their trigger while clamping to the viewport. Do not replace that sizing with a fixed width or an unclamped minimum width. @@ -74,6 +80,67 @@ single-or-multiple union. Prefer the Base UI `items` collection pattern so the root, value display, and item list share one runtime source of truth. Convert values to strings only at real serialization boundaries. +### Combobox source items and selected values + +Combobox has separate types for the selected business value and the source record rendered by the +list: + +```tsx +import { + Combobox, + ComboboxItem, + ComboboxList, + createComboboxItems, +} from '@langgenius/dify-ui/combobox' +import { useMemo } from 'react' + +const userItems = useMemo( + () => + createComboboxItems(users, { + getValue: user => user.id, + getLabel: user => user.name, + }), + [users], +) + + + multiple + items={userItems} + value={selectedUserIds} + onValueChange={setSelectedUserIds} +> + > + {user => value={user.id}>{user.name}} + + +``` + +The first generic is `Value`, the second is the literal multiple-selection mode, and the third is +the source `Item`. `ComboboxValue` and `ComboboxItem` use `Value`; `filter`, `ComboboxList`, +`ComboboxGroup`, `ComboboxCollection`, and `useComboboxFilteredItems` use source items. Grouped +roots use the leaf record as `Item`, while the list callback receives the group object. + +Use `createComboboxItems` when the business contract stores a stable primitive ID but list rows +need complete records. Its `getValue` result must be unique and stable, and `getLabel` owns default +filtering, typeahead, and selected-value display. Create static collections at module scope and +memoize collections derived from changing data. Treat the returned collection as opaque and pass +it directly to `items`. + +For server-side search, keep the complete set of records needed to resolve selected labels in the +collection passed to `items`, and pass the current result window to `filteredItems`. Filtered items +are source `Item` records, not derived IDs, and grouped results must retain the collection's group +shape. + +Keep object values when the selected record itself is the business state or the selection callback +immediately needs the full record. When async refreshes may replace object references, provide +`isItemEqualToValue` using the stable domain identity. + +`itemToStringValue` only serializes a selected `Value` for forms and autofill; it does not change +`onValueChange` into an ID callback. Do not add it, `itemToStringLabel`, or a comparator as a +mechanical trio. Primitive IDs normally use the default equality. For async or paged data, keep +selected records in the collection when their labels must remain available after they leave the +current result window, or provide an ID-only label fallback. + `CheckboxGroup` follows Base UI and uses `string[]`. Model stronger business ID distinctions at the domain boundary unless the upstream primitive contract changes. diff --git a/packages/dify-ui/src/button/README.md b/packages/dify-ui/src/button/README.md index 95c9468e0b0..8ece6c6bdc0 100644 --- a/packages/dify-ui/src/button/README.md +++ b/packages/dify-ui/src/button/README.md @@ -119,8 +119,10 @@ for the other variants. Use a `className` override only for a documented layout ## Related guides - Read [`IconButton`] for icon-only actions. +- Read [Accessible names and descriptions] when choosing or changing a naming source. - Read [Base UI Button] for the upstream interaction and composition contract. +[Accessible names and descriptions]: ../../docs/accessible-names-and-descriptions.md [Base UI Button]: https://base-ui.com/react/components/button [WAI-ARIA `aria-busy`]: https://www.w3.org/TR/wai-aria#aria-busy [`IconButton`]: ../icon-button/README.md diff --git a/packages/dify-ui/src/combobox/__tests__/index.spec.tsx b/packages/dify-ui/src/combobox/__tests__/index.spec.tsx index 0b3ab2673ee..893ff6c9001 100644 --- a/packages/dify-ui/src/combobox/__tests__/index.spec.tsx +++ b/packages/dify-ui/src/combobox/__tests__/index.spec.tsx @@ -7,6 +7,7 @@ import { ComboboxChipRemove, ComboboxChips, ComboboxClear, + ComboboxCollection, ComboboxEmpty, ComboboxGroup, ComboboxGroupLabel, @@ -25,8 +26,56 @@ import { ComboboxStatus, ComboboxTrigger, ComboboxValue, + createComboboxItems, } from '../index' +type ResourceOption = { + id: string + label: string +} + +const resourceOptions: ResourceOption[] = [ + { id: 'workflow', label: 'Workflow' }, + { id: 'dataset', label: 'Dataset' }, +] +const resourceItems = createComboboxItems(resourceOptions, { + getValue: (item) => item.id, + getLabel: (item) => item.label, +}) + +function ComboboxTypeExamples() { + return ( + + + multiple + items={resourceItems} + value={['workflow']} + filter={(item, query) => item.label.includes(query)} + onValueChange={(value) => { + const selectedIds: string[] = value + void selectedIds + }} + > + >{(value) => value?.join(', ') ?? ''} + > + {(item) => value={item.id}>{item.label}} + + items={resourceOptions}> + > + {(item) => value={item.id}>{item.label}} + + + {/* @ts-expect-error item anatomy accepts the derived string value, not the source object */} + value={resourceOptions[0]} /> + + {/* @ts-expect-error root value uses the derived string domain, not the source object */} + items={resourceItems} value={resourceOptions[0]} /> + + ) +} + +void ComboboxTypeExamples + const renderWithSafeViewport = (ui: React.ReactNode) => render(
{ui}
) @@ -113,6 +162,33 @@ describe('Combobox wrappers', () => { .element(screen.getByRole('combobox', { name: 'Resource type' })) .toBeInTheDocument() }) + + it('should expose readonly styling state while allowing options to be inspected', async () => { + const screen = await render( + + + + + + + + + Workflow + Dataset + + + + + , + ) + const trigger = screen.getByRole('combobox', { name: 'Resource type' }) + + await expect.element(trigger).toHaveAttribute('data-readonly') + await trigger.click() + await expect.element(screen.getByRole('option', { name: 'Dataset' })).toBeVisible() + await screen.getByRole('option', { name: 'Dataset' }).click() + await expect.element(trigger).toHaveTextContent('workflow') + }) }) describe('Input group and controls', () => { @@ -206,6 +282,35 @@ describe('Combobox wrappers', () => { }) describe('Popup anatomy and options', () => { + it('should render source objects while exposing primitive selected values', async () => { + const onValueChange = vi.fn() + const screen = await render( + + defaultOpen + items={resourceItems} + defaultValue="workflow" + filter={(item, query) => item.label.toLowerCase().includes(query.toLowerCase())} + onValueChange={(nextValue) => onValueChange(nextValue)} + > + + > + {(item) => ( + key={item.id} value={item.id}> + {item.label} + + )} + + , + ) + + await expect + .element(screen.getByRole('option', { name: 'Workflow' })) + .toHaveAttribute('aria-selected', 'true') + await userEvent.click(screen.getByRole('option', { name: 'Dataset' })) + + expect(onValueChange).toHaveBeenCalledWith('dataset') + }) + it('should use default overlay placement', async () => { const screen = await renderSelectLikeCombobox({ open: true }) diff --git a/packages/dify-ui/src/combobox/index.stories.tsx b/packages/dify-ui/src/combobox/index.stories.tsx index 7b93bff4fe8..406270cf695 100644 --- a/packages/dify-ui/src/combobox/index.stories.tsx +++ b/packages/dify-ui/src/combobox/index.stories.tsx @@ -28,6 +28,7 @@ import { ComboboxStatus, ComboboxTrigger, ComboboxValue, + createComboboxItems, useComboboxFilter, useComboboxFilteredItems, } from '.' @@ -191,6 +192,10 @@ const tagOptions: Option[] = [ { value: 'finance', label: 'Finance' }, { value: 'support', label: 'Support' }, ] +const tagItems = createComboboxItems(tagOptions, { + getValue: (option) => option.value, + getLabel: (option) => option.label, +}) const directoryOptions: Option[] = [ { @@ -832,24 +837,31 @@ const MultipleChipsDemo = () => { Reviewers - - > - {(selectedValue) => ( - - {selectedValue?.map((item) => ( - + > + {(selectedValue) => { + const selectedReviewers = selectedValue ?? [] + + return ( + 0 ? 'Selected reviewers' : undefined} + > + {selectedReviewers.map((item) => ( + {item.label} ))} 0 ? '' : 'Assign reviewers…'} className="min-w-24 px-1 py-0.5" /> - - )} - - + + ) + }} + @@ -861,6 +873,11 @@ const MultipleChipsDemo = () => { Selected reviewers wrap inside the input instead of scrolling horizontally. + {value.length > 0 && ( + + {` ${value.length} selected. From the start of the input, press Left Arrow to focus the selected items`} + + )} ) @@ -871,11 +888,25 @@ export const MultipleChips: Story = { play: async ({ canvas, userEvent }) => { await expect(canvas.getByText('Maya Chen')).toBeVisible() await expect(canvas.getByText('Liam Brooks')).toBeVisible() + await expect(canvas.getByRole('toolbar', { name: 'Selected reviewers' })).toBeVisible() - await userEvent.click(canvas.getByRole('button', { name: 'Remove Maya Chen' })) + await expect(canvas.getByText('Liam Brooks').parentElement!).toHaveAccessibleDescription( + 'Press Backspace or Delete to remove', + ) - await expect(canvas.queryByText('Maya Chen')).not.toBeInTheDocument() - await expect(canvas.getByText('Liam Brooks')).toBeVisible() + const input = canvas.getByRole('combobox', { name: 'Reviewers' }) + await expect(input).toHaveAccessibleDescription( + 'Selected reviewers wrap inside the input instead of scrolling horizontally. 2 selected. From the start of the input, press Left Arrow to focus the selected items', + ) + + input.focus() + await userEvent.keyboard('{ArrowLeft}{Delete}') + + await expect(canvas.queryByText('Liam Brooks')).not.toBeInTheDocument() + await expect(canvas.getByText('Maya Chen')).toBeVisible() + await expect(input).toHaveAccessibleDescription( + 'Selected reviewers wrap inside the input instead of scrolling horizontally. 1 selected. From the start of the input, press Left Arrow to focus the selected items', + ) }, } @@ -955,15 +986,31 @@ export const ReadOnly: Story = { ), + play: async ({ canvas, canvasElement, userEvent }) => { + const input = canvas.getByRole('combobox', { name: 'Read-only source' }) + const body = within(canvasElement.ownerDocument.body) + + await expect(input).toHaveValue('Website crawler') + await userEvent.click(input) + await waitFor(async () => { + await expect(body.getByRole('option', { name: /Notion/ })).toBeVisible() + }) + await userEvent.keyboard('{ArrowDown}') + await expect(body.getByRole('option', { name: /S3 bucket/ })).toHaveAttribute( + 'data-highlighted', + ) + await userEvent.keyboard('{Enter}') + await expect(input).toHaveValue('Website crawler') + }, } const ControlledDemo = () => { - const [value, setValue] = React.useState