Commit Graph

1684 Commits

Author SHA1 Message Date
Jyong
b79b8a879b feat(knowledge-fs): record and count every knowledge space query
Workflow KnowledgeFS retrieval nodes ran through `retrieval-tests`, which
never wrote an AnswerTrace or an overview event, so their queries were
missing from the retrieval history and from the space overview; the history
also only ever showed each member their own traces.

- AnswerTrace gains a `source` (retrieval_test | workflow | service_api |
  agent | mcp; migration 0051_answer_trace_source) derived from the
  Capability v2 caller kind. The retrieval-tests route records one trace per
  run (stages, evidence bundle, profile metadata), returns its id as
  `answerTraceId`, and the workflow node's failed-retrieval capture attaches
  to that trace instead of creating a second record.
- The quality trace list exposes and filters by `source`; traces from other
  caller kinds are visible to any current reader of the space, and counts and
  scores fall back to the evidence embedded in the trace when no bundle row
  exists.
- Overview accounting: retrieval-tests and Research tasks now emit
  `query.requested`, and the Research job state machine emits
  `query.completed` / `query.failed` on terminal stages (wired for both the
  in-process gateway and the durable runtime), so query volume, answer rate
  and outcomes include every caller. Activity details keep `source` and
  `taskKind`.
- Console and service trace routes accept a `source` filter; the retrieval
  test page shows a source badge and an all / retrieval tests / workflow
  filter, with translations for every locale.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015zw5G5SX3HmVfnZof6YWAc
2026-09-04 05:48:17 -04:00
FFXN
40e30d9fa6 feat(knowledge-fs): fix CI. 2026-09-04 16:55:47 +08:00
FFXN
ad059f55d4 feat(knowledge-fs): add namespace preview for initial website sources
- create website preview jobs before a knowledge space exists
- persist crawled page content in KnowledgeFS object storage
- import selected pages using preview job and page IDs
- validate preview ownership, expiry, and configuration fingerprint
- clean up preview content after cancellation, expiry, or consumption
- add namespace preview routes, capabilities, migrations, and contracts
- keep online document and drive flows unchanged
2026-09-04 15:52:36 +08:00
Jyong
8150a8fca7 feat(knowledge-fs): rate limit console writes and every service API route
The legacy dataset console rate limits every mutating, download, and
retrieval route and the dataset service API limits every request, while
KnowledgeFS only covered uploads and query admission. Apply the knowledge
rate limit to all space-scoped POST/PUT/PATCH/DELETE console routes, the
document downloads, and research/query entrypoints (workspace-level source
previews, staged-upload discards, the deprecated buffered query route, and
the internal stream transport stay exempt). Extract the service API check
into `check_knowledge_rate_limit` and call it from the KnowledgeFS profile
helper so all service routes share the legacy per-workspace limit.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015zw5G5SX3HmVfnZof6YWAc
2026-09-03 11:10:04 -04:00
Jyong
79116d8c4b feat(knowledge-fs): scope dataset API keys by knowledge base type
Legacy knowledge bases (`datasets`) and KnowledgeFS spaces
(`knowledge_fs_control_spaces`) live in different tables, so a dataset API
key binding could only ever name a legacy dataset and the KnowledgeFS service
API never consulted bindings at all: a key scoped to specific knowledge bases
still reached every KnowledgeFS space in the workspace.

- Bindings carry a `resource_type` (`dataset` | `knowledge_fs_space`) and a
  nullable `control_space_id` (migration 9a4e7d1c2b60, existing rows stay
  `dataset`), with a CHECK that exactly one id column matches the type.
- `dataset_api_key_service` exposes the key scope by kind, validates and
  binds KnowledgeFS spaces (tenant-owned, not deleting/deleted), and cleans up
  keys scoped only to a space when its deletion is requested. The orphan-key
  cleanup is now NULL-safe so a space binding keeps a key alive.
- Legacy dataset routes only honour `dataset` bindings; the KnowledgeFS
  authorization service only honours `knowledge_fs_space` bindings and raises
  a scope error that the service API maps to 403 (unknown keys stay 401).
- Console key creation accepts `knowledge_space_ids`; list/create responses
  return them next to `dataset_ids`.
- The web scope picker lists KnowledgeFS spaces alongside legacy datasets
  when KnowledgeFS is enabled and submits the ids by kind; the scope column
  counts both.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015zw5G5SX3HmVfnZof6YWAc
2026-09-03 05:59:39 -04:00
Jyong
180ac5ad4d refactor(knowledge-fs): remove deprecated buffered service API routes
Drop the deprecated `POST /knowledge-fs/spaces/<id>/queries` route and the
deprecated buffered `POST .../documents` upload from the service API. Both
only ever failed closed; queries go through admission + the query stream,
documents through the durable source import flow. Regenerate the service
contracts and markdown docs accordingly.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015zw5G5SX3HmVfnZof6YWAc
2026-09-03 05:59:39 -04:00
Stephen Zhou
3611a93a3a
Merge remote-tracking branch 'origin/deploy/konwledge' into deploy/konwledge
# Conflicts:
#	api/tests/unit_tests/services/test_knowledge_fs_initial_source_reliability.py
2026-09-03 16:57:14 +08:00
Stephen Zhou
42cb6519ea
Merge remote-tracking branch 'origin/main' into deploy/konwledge 2026-09-03 16:32:30 +08:00
Byron.wang
b015aa416f
refactor(api): extract account login service and consolidate adapters (#41184) 2026-09-03 06:20:21 +00:00
非法操作
ba1d5ba172
refactor(api): centralize remote file handling (#41627) 2026-09-03 06:00:13 +00:00
非法操作
d8c3f7c5ee
refactor(openapi): inject file upload service (#41606) 2026-09-03 04:42:31 +00:00
非法操作
69484a07e1
refactor(console): inject file service (#41603) 2026-09-03 04:42:30 +00:00
非法操作
5f4b1f867d
refactor(web): inject file upload service (#41586) 2026-09-03 04:42:30 +00:00
Eddy ZHANG
fc7193cf04
refactor(service_api): dep-inject the dataset create payload with @model_validate (#41651) 2026-09-03 03:52:41 +00:00
非法操作
ab0e92b893
refactor(service-api): inject file upload service (#41585) 2026-09-03 03:41:45 +00:00
Eddy ZHANG
95a6a14ed5
refactor(console,web): dep-inject query params with @model_validate (#41646) 2026-09-03 01:50:22 +00:00
Byron.wang
be7d11ca1d
refactor(api): extract forgot password application service (#41111)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-09-03 01:38:59 +00:00
Stephen Zhou
45bee40955
Merge remote-tracking branch 'origin/main' into deploy/konwledge
# Conflicts:
#	api/controllers/console/datasets/datasets.py
#	api/tests/unit_tests/controllers/console/datasets/test_datasets.py
2026-09-02 21:56:53 +08:00
YungLe
3cd1fc098c
feat: allow knowledge base API keys to be scoped to a single dataset (#37569)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-09-02 10:30:42 +00:00
非法操作
59ee8878f9
refactor(api): require active workspace in request context (#41280) 2026-09-02 06:54:47 +00:00
Jyong
222d09f474 feat(knowledge-fs): add profile-driven multimodal retrieval 2026-09-02 01:01:57 -04:00
FFXN
b55ec59d37 fix(knowledge-fs): handle source selection mismatch and improve error handling, make source selection replacement durable. 2026-09-02 12:56:37 +08:00
Eddy ZHANG
4d9bda7422
refactor(inner_api): dep-inject request payloads with @model_validate (#41575) 2026-09-02 03:40:46 +00:00
Eddy ZHANG
a5445f2424
refactor(service_api): dep-inject segment payloads with @model_validate (#41571) 2026-09-02 02:59:30 +00:00
Stephen Zhou
cd7e1be7cb
Merge remote-tracking branch 'origin/main' into deploy/konwledge
# Conflicts:
#	api/services/feature_service.py
#	api/tests/unit_tests/controllers/console/workspace/test_members.py
#	api/tests/unit_tests/core/workflow/nodes/agent_v2/test_validators.py
#	api/tests/unit_tests/services/test_system_feature_service_knowledge_fs.py
#	pnpm-lock.yaml
2026-09-02 10:31:10 +08:00
Xiyuan Chen
93dae7d8a0
fix(rbac): let agent.manage operate every agent page (#41583)
Co-authored-by: yunlu.wen <yunlu.wen@dify.ai>
2026-09-02 01:45:42 +00:00
Byron.wang
c4ea406586
refactor(api): decouple system features and web adapters (#40772)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-09-01 11:50:52 +00:00
Keith
566b8857dd
refactor(models): remove legacy db.session property wrappers on Dataset (#41569) 2026-09-01 06:28:24 +00:00
Eddy ZHANG
6411d6235c
refactor(service_api): dep-inject query params with @model_validate (#41562) 2026-09-01 05:55:50 +00:00
Eddy ZHANG
1977355dd1
refactor(console): dep-inject model provider payloads with @model_validate (#41539) 2026-09-01 04:24:45 +00:00
Joel
5cbc42cd38
chore: Revert "feat: use app.acl.access_point_manage to control access point page access" (#41563) 2026-09-01 03:46:03 +00:00
Stephen Zhou
ecd41aed3f
Merge remote-tracking branch 'origin/main' into deploy/konwledge 2026-09-01 09:45:49 +08:00
Jyong
4f3a2952d6 perf(knowledge-fs): harden cross-format document ingestion 2026-08-31 07:48:35 -04:00
zyssyz123
762019a160
fix(workflow): block manual runs for trigger workflows (#41519)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-08-31 09:30:52 +00:00
Byron.wang
d25e2b51fc
refactor(api): decouple onboarding and notification services (#40759)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: hjlarry <hjlarry@163.com>
2026-08-31 04:37:07 +00:00
Byron.wang
9e07c2dac6
refactor(api): extract email registration application service (#41108)
Co-authored-by: hjlarry <hjlarry@163.com>
2026-08-31 03:22:01 +00:00
wangxiaolei
6a00c23e49
feat: use app.acl.access_point_manage to control access point page access (#41462)
Co-authored-by: twwu <twwu@dify.ai>
2026-08-31 03:05:19 +00:00
zyssyz123
ad86351cc1
fix(agent): reject incompatible session snapshots (#41447) 2026-08-31 02:26:07 +00:00
Eddy ZHANG
6130f081e0
refactor(console): dep-inject account payloads with @model_validate (#41500) 2026-08-31 00:52:10 +00:00
Eddy ZHANG
8894b09c3b
refactor(web): dep-inject payloads with @model_validate (#41501) 2026-08-31 00:50:12 +00:00
FFXN
949549b9b1 feat(knowledge-fs): enhance source update payload with selection and sync policy validation 2026-08-30 23:58:27 +08:00
Eddy ZHANG
884ece570e
refactor(service_api): dep-inject dataset payloads with @model_validate (#41490) 2026-08-30 15:06:36 +00:00
FFXN
52df787444 feat(knowledge-fs): support source config updates and deferred sync
- allow updating mutable source configuration fields
- enqueue sync after active business configuration changes
- pause scheduled sync while a source is disabled
- preserve and rebind sync policies across source version updates
- support website, online document, and online drive sources
2026-08-30 22:10:12 +08:00
Eddy ZHANG
8b3c2ee274
refactor(service_api): dep-inject app payloads with @model_validate (#41491) 2026-08-30 12:51:50 +00:00
Weike Zhang
02f9b3ca7c
refactor: pass db.session explicitly in instruction_modify_workflow (#37403) (#40827) 2026-08-30 00:48:00 +00:00
jjj-n
09054169c4
refactor(models): pass session into Workflow accessors (#41464) 2026-08-29 14:28:15 +00:00
lin
1264d29a4f
chore: remove unnecessary type conversions (#41454) 2026-08-29 07:04:18 +00:00
Terminator666666
f8dd276e2c
refactor(controllers/console): dep-inject workflow payloads with @model_validate (#41244)
Co-authored-by: Terminator666666 <Terminator666666@users.noreply.github.com>
2026-08-28 14:49:46 +00:00
Eddy ZHANG
37988de678
refactor(service_api): dep-inject payloads with @model_validate (#41374) 2026-08-28 14:47:18 +00:00
yyh
78c85ae7c1
fix(agent): add roster publication status contract (#41435)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-08-28 08:50:30 +00:00