Commit Graph

974 Commits

Author SHA1 Message Date
Yansong Zhang
77c182f738 feat(api): propagate all app features in transparent upgrade
VirtualWorkflowSynthesizer._build_features() now extracts ALL legacy
app features from AppModelConfig into the synthesized workflow.features:

- opening_statement + suggested_questions
- sensitive_word_avoidance (keywords/API moderation)
- more_like_this
- speech_to_text / text_to_speech
- retriever_resource

Previously workflow.features was hardcoded to "{}", losing all these
features during transparent upgrade. Now AdvancedChatAppRunner's
moderation, opening text, and other feature layers work correctly
for transparently upgraded old apps.

Made-with: Cursor
2026-04-10 18:47:18 +08:00
Yansong Zhang
e50c36526e fix(api): fix transparent upgrade SSE channel mismatch and chat mode routing
- workflow_execute_task: add AppMode.CHAT/AGENT_CHAT/COMPLETION to the
  AdvancedChatAppGenerator routing branch so transparently upgraded old
  apps can execute through the workflow engine.
- app_generate_service: use app_model.mode (not hardcoded AppMode.AGENT)
  for SSE event subscription channel, ensuring the subscriber and
  Celery publisher use the same Redis channel key.

Made-with: Cursor
2026-04-09 17:27:41 +08:00
Yansong Zhang
e2e16772a1 fix(api): fix DSL import, memory loading, and remaining test coverage
1. DSL Import fix: change self._session.commit() to self._session.flush()
   in app_dsl_service.py _create_or_update_app() to avoid "closed transaction"
   error. DSL import now works: export agent app -> import -> new app created.

2. Memory loading attempt: added _load_memory_messages() to AgentV2Node
   that loads TokenBufferMemory from conversation history. However, chatflow
   engine manages conversations differently from easy-UI (conversation may
   not be in DB at query time, or uses ConversationVariablePersistenceLayer
   instead of Message table). Memory needs further investigation.

Test results:
- Multi-turn memory: Turn 1 OK, Turn 2 LLM doesn't see history (needs deeper fix)
- Service API with API Key: PASSED (answer="Sixteen" for 8+8)
- DSL Import: PASSED (status=completed, new app created)
- Token aggregation: PASSED (node=49, workflow=49)

Known: memory in multi-turn chatflow needs to use graphon's built-in
memory mechanism (MemoryConfig on node + ConversationVariablePersistenceLayer)
rather than direct DB query.

Made-with: Cursor
2026-04-09 14:47:55 +08:00
Yansong Zhang
482a004efe fix(api): fix duplicate answer and completion app upgrade issues
1. Remove StreamChunkEvent from AgentV2Node._run_without_tools():
   The agent-v2 node was yielding StreamChunkEvent during LLM streaming,
   AND the downstream answer node was outputting the same text via
   {{#agent.text#}} variable reference, causing "FourFour" duplication.
   Now text only flows through outputs.text -> answer node (single path).

2. Map inputs to query for completion app transparent upgrade:
   Completion apps send {inputs: {query: "..."}} not {query: "..."}.
   VirtualWorkflowSynthesizer route now extracts query from inputs
   when the top-level query is missing.

Verified:
- Old chat app: "What is 2+2?" -> "Four" (was "FourFour")
- Old completion app: {inputs: {query: "What is 3+3?"}} -> "3 + 3 = 6" (was failing)
- Old agent-chat app: still works

Made-with: Cursor
2026-04-09 12:02:43 +08:00
Yansong Zhang
7052257c8d fix(api): use lazy workflow persistence for transparent upgrade of old apps
VirtualWorkflowSynthesizer.ensure_workflow() creates a real draft
workflow on first call for a legacy app, persisting it to the database.
On subsequent calls, returns the existing draft.

This is needed because AdvancedChatAppGenerator's worker thread looks
up workflows from the database by ID. Instead of hacking the generator
to skip DB lookups, we treat this as a lazy one-time upgrade: the old
app gets a real workflow that can also be edited in the workflow editor.

Verified: old chat app created on main branch ("What is 2+2?" -> "Four")
and old agent-chat app ("Say hello" -> "Hello!") both successfully
execute through the Agent V2 engine with AGENT_V2_TRANSPARENT_UPGRADE=true.

Made-with: Cursor
2026-04-09 11:28:16 +08:00
Yansong Zhang
edfcab6455 fix(api): add AGENT mode to app list filtering
Add AppMode.AGENT branch in get_paginate_apps() so that
filtering apps by mode=agent works correctly.
Discovered during comprehensive E2E testing.

14/14 E2E tests pass covering:
- A: New Agent app full lifecycle (create, draft, configs, publish, run)
- B: Old app creation compat (chat, completion, agent-chat, advanced-chat, workflow)
- C: App listing and filtering (all modes, agent filter)
- D: Workflow editor compat (block configs)
- E: DSL export

Made-with: Cursor
2026-04-09 10:54:05 +08:00
Yansong Zhang
66212e3575 feat(api): implement zero-migration transparent upgrade (Phase 8)
Add two feature-flag-controlled upgrade paths that allow existing apps
and LLM nodes to transparently run through the Agent V2 engine without
any database migration:

1. AGENT_V2_TRANSPARENT_UPGRADE (default: off):
   When enabled, old apps (chat/completion/agent-chat) bypass legacy
   Easy-UI runners. VirtualWorkflowSynthesizer converts AppModelConfig
   to an in-memory Workflow (start -> agent-v2 -> answer) at runtime,
   then executes via AdvancedChatAppGenerator. Falls back to legacy
   path on any synthesis error.

   VirtualWorkflowSynthesizer maps:
   - model JSON -> ModelConfig
   - pre_prompt/chat_prompt_config -> prompt_template
   - agent_mode.tools -> ToolMetadata[]
   - agent_mode.strategy -> agent_strategy
   - dataset_configs -> context
   - file_upload -> vision

2. AGENT_V2_REPLACES_LLM (default: off):
   When enabled, DifyNodeFactory.create_node() transparently remaps
   nodes with type="llm" to type="agent-v2" before class resolution.
   Since AgentV2NodeData is a strict superset of LLMNodeData, the
   mapping is lossless. With tools=[], Agent V2 behaves identically
   to LLM Node.

Both flags default to False for safety. Turn off = instant rollback.
46 existing tests pass. Flask starts successfully.

Made-with: Cursor
2026-04-09 10:30:52 +08:00
Yansong Zhang
44491e427c feat(api): enable all sandbox/skill controller routes and resolve dependencies (P0)
Resolve the full dependency chain to enable all previously disabled controllers:

Enabled routes:
- sandbox_files: sandbox file browser API
- sandbox_providers: sandbox provider management API
- app_asset: app asset management API
- skills: skill extraction API
- CLI API blueprint: DifyCli callback endpoints (/cli/api/*)

Dependencies extracted (64 files, ~8000 lines):
- models/sandbox.py, models/app_asset.py: DB models
- core/zip_sandbox/: zip-based sandbox execution
- core/session/: CLI API session management
- core/memory/: base memory + node token buffer
- core/helper/creators.py: helper utilities
- core/llm_generator/: context models, output models, utils
- core/workflow/nodes/command/: command node type
- core/workflow/nodes/file_upload/: file upload node type
- core/app/entities/: app_asset_entities, app_bundle_entities, llm_generation_entities
- services/: asset_content, skill, workflow_collaboration, workflow_comment
- controllers/console/app/error.py: AppAsset error classes
- core/tools/utils/system_encryption.py

Import fixes:
- dify_graph.enums -> graphon.enums in skill_service.py
- get_signed_file_url_for_plugin -> get_signed_file_url in cli_api.py

All 5 controllers verified: import OK, Flask starts successfully.
46 existing tests still pass.

Made-with: Cursor
2026-04-09 09:36:16 +08:00
Yansong Zhang
0c7e7e0c4e feat(api): port Sandbox + VirtualEnvironment + Skill system from feat/support-agent-sandbox (Phase 5-6)
Port the complete infrastructure for agent sandbox execution and skill system:

Sandbox & Virtual Environment (core/sandbox/, core/virtual_environment/):
- Sandbox entity with lifecycle management (ready/failed/cancelled states)
- SandboxBuilder with fluent API for configuring providers
- 5 VM providers: Local, SSH, Docker, E2B, AWS CodeInterpreter
- VirtualEnvironment base with command execution, file transfer, transport layers
- Channel transport: pipe, queue, socket implementations
- Bash session management and DifyCli binary integration
- Storage: archive storage, file storage, noop storage, presign storage
- Initializers: DifyCli, AppAssets, DraftAppAssets, Skills
- Inspector: file browser, archive/runtime source, script utils
- Security: encryption utils, debug helpers

Skill & App Assets (core/skill/, core/app_assets/, core/app_bundle/):
- Skill entity and manager
- App asset accessor, builder pipeline (file, skill builders)
- App bundle source zip extractor
- Storage and converter utilities

API Endpoints:
- CLI API blueprint (controllers/cli_api/) for sandbox callback
- Sandbox provider management (workspace/sandbox_providers)
- Sandbox file browser (console/sandbox_files)
- App asset management (console/app/app_asset)
- Skill management (console/app/skills)
- Storage file endpoints (controllers/files/storage_files)

Services:
- Sandbox service, provider service, file service
- App asset service, app bundle service

Config:
- CliApiConfig, CreatorsPlatformConfig, CollaborationConfig
- FILES_API_URL for sandbox file access

Note: Controller route registration temporarily commented out (marked TODO)
pending resolution of deep dependency chains (socketio, workflow_comment,
command node, etc.). Core sandbox modules are fully ported and syntax-validated.
110 files changed, 10,549 insertions.

Made-with: Cursor
2026-04-08 17:39:02 +08:00
Yansong Zhang
d9d1e9b63a fix(api): resolve Agent V2 node E2E runtime issues
Fixes discovered during end-to-end testing of Agent workflow execution:

1. ModelManager instantiation: use ModelManager.for_tenant() instead of
   ModelManager() which requires a ProviderManager argument
2. Variable template resolution: use VariableTemplateParser(template).format()
   instead of non-existent resolve_template() static method
3. invoke_llm() signature: remove unsupported 'user' keyword argument
4. Event dispatch: remove ModelInvokeCompletedEvent from _run() yield
   (graphon base Node._dispatch doesn't support it via singledispatch)
5. NodeRunResult metadata: use WorkflowNodeExecutionMetadataKey enum keys
   (TOTAL_TOKENS, TOTAL_PRICE, CURRENCY) instead of arbitrary string keys
6. SSE topic mismatch: use AppMode.AGENT (not ADVANCED_CHAT) in
   retrieve_events() so publisher and subscriber share the same channel
7. Celery task routing: add AppMode.AGENT to workflow_execute_task._run_app()
   alongside ADVANCED_CHAT

All issues verified fixed: Agent V2 node successfully invokes LLM and
returns "Hello there!" through the full SSE streaming pipeline.

Made-with: Cursor
2026-04-08 16:21:12 +08:00
Yansong Zhang
1835a1dc5d fix(api): allow AGENT mode in workflow features validation
Add AppMode.AGENT to validate_features_structure() match case
alongside ADVANCED_CHAT, fixing 'Invalid app mode: agent' error
when creating Agent apps (which auto-generate a workflow draft).

Discovered during E2E testing of the full create -> draft -> publish flow.

Made-with: Cursor
2026-04-08 13:19:59 +08:00
Yansong Zhang
96641a93f6 feat(api): add Agent V2 node and new Agent app type (Phase 1-3)
Introduce a new unified Agent V2 workflow node that combines LLM capabilities
with agent tool-calling loops, along with a new AppMode.AGENT for standalone
agent apps backed by single-node workflows.

Phase 1 — Agent Patterns:
- Add core/agent/patterns/ module (AgentPattern, FunctionCallStrategy,
  ReActStrategy, StrategyFactory) ported from feat/support-agent-sandbox
- Add ExecutionContext, AgentLog, AgentResult entities
- Add Tool.to_prompt_message_tool() for LLM-consumable tool conversion

Phase 2 — Agent V2 Workflow Node:
- Add core/workflow/nodes/agent_v2/ (AgentV2Node, AgentV2NodeData,
  AgentV2ToolManager, AgentV2EventAdapter)
- Register agent-v2 node type in DifyNodeFactory
- No-tools path: single LLM call (LLM Node equivalent)
- Tools path: FC/ReAct loop via StrategyFactory

Phase 3 — Agent App Type:
- Add AppMode.AGENT to model enum
- Add WorkflowGraphFactory for auto-generating start->agent_v2->answer graphs
- AppService.create_app() creates workflow draft for AGENT mode
- AppGenerateService.generate() routes AGENT to AdvancedChatAppGenerator
- Console API and DSL import/export support AGENT mode
- Default app template for AGENT mode

Old agent/agent-chat/LLM node paths are fully preserved.
38 unit tests all passing.

Made-with: Cursor
2026-04-08 12:31:23 +08:00
corevibe555
c825d5dcf6
refactor(api): tighten types for Tenant.custom_config_dict and MCPToolProvider.headers (#34698) 2026-04-08 01:36:42 +00:00
corevibe555
624db69f12
refactor(api): remove duplicated RAG entities from services layer (#34689) 2026-04-07 23:36:59 +00:00
corevibe555
80a7843f45
refactor(api): migrate consumers to shared RAG domain entities from core/rag/entities/ (#34692) 2026-04-07 23:22:56 +00:00
corevibe555
f8f7b0ec1a
refactor(api): deduplicate shared auth request payloads into auth_entities.py (#34694) 2026-04-07 22:51:11 +00:00
Dream
89ce61cfea
refactor(api): replace json.loads with Pydantic validation in security and tools layers (#34380) 2026-04-07 12:11:51 +00:00
Statxc
63db9a7a2f
refactor(api): type load balancing config dicts with TypedDict (#34639) 2026-04-07 05:58:10 +00:00
YBoy
1261e5e5e8
refactor(api): type webhook validation result and workflow inputs with TypedDict (#34645) 2026-04-07 05:57:02 +00:00
Pulakesh
bceb0eee9b
refactor(api): migrate dict returns to TypedDicts in billing service (#34649) 2026-04-07 05:56:02 +00:00
Renzo
1194957fde
refactor: migrate session.query to select API in end_user_service and small tasks (#34620) 2026-04-07 04:25:55 +00:00
Renzo
68bd29eda2
refactor: migrate session.query to select API in sync task and services (#34619) 2026-04-07 04:23:14 +00:00
YBoy
f67a811f7f
refactor: replace dict params with BaseModel payloads in TagService (#34422)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-07 04:20:02 +00:00
Statxc
a7b6307d32
refactor(api): type dataset service dicts with TypedDict (#34625) 2026-04-07 02:10:52 +00:00
Statxc
2883ad6764
refactor(api): type plugin migration results with TypedDict (#34627) 2026-04-07 02:10:23 +00:00
Pulakesh
0feff5b048
refactor(api): enforce strict typing on retrieval_model to resolve FIXME (#34614) 2026-04-07 01:10:53 +00:00
YBoy
89e23456f0
refactor(api): type invitation detail with InvitationDetailDict TypedDict (#34613)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-07 01:03:31 +00:00
YBoy
922f9242e4
refactor(api): type crawl status dicts with CrawlStatusDict TypedDict (#34611) 2026-04-07 01:01:04 +00:00
YBoy
7fc0a791a2
refactor(api): type document summary status detail with TypedDict (#34610) 2026-04-07 01:00:39 +00:00
YBoy
5ad906ea6a
refactor(api): type workflow run related counts with RelatedCountsDict TypedDict (#34530) 2026-04-06 13:17:01 +00:00
YBoy
1e5cd69205
refactor(api): type archive manifest with ArchiveManifestDict TypedDict (#34594) 2026-04-06 11:35:11 +00:00
YBoy
b71b9f80b9
refactor(api): type workflow run delete/count results with RunsWithRelatedCountsDict TypedDict (#34531)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-05 16:11:41 +00:00
agenthaulk
ee87289917
refactor: convert AppMode if/elif to match/case in app_generate_service (#30001) (#34563)
Co-authored-by: agenthaulk <agenthaulk@users.noreply.github.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-05 10:23:51 +00:00
agenthaulk
5ad8c3e249
refactor: convert AppMode if/elif to match/case in service files (#30001) (#34562)
Co-authored-by: agenthaulk <agenthaulk@users.noreply.github.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-05 10:22:11 +00:00
Renzo
eca0cdc7a9
refactor: select in dataset_service (SegmentService and remaining cla… (#34547)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-05 00:13:06 +00:00
Renzo
779e6b8e0b
refactor: select in datasource_provider_service (#34548)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-05 00:12:15 +00:00
Renzo
c2428361c4
refactor: select in dataset_service (DocumentService class) (#34528)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-03 22:52:01 +00:00
Renzo
68e4d13f36
refactor: select in annotation_service (#34503)
Co-authored-by: Asuka Minato <i@asukaminato.eu.org>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-03 22:47:22 +00:00
YBoy
8a398f3105
refactor(api): type messages cleanup stats with MessagesCleanStatsDict TypedDict (#34527) 2026-04-03 12:29:41 +00:00
Renzo
e85d9a0d72
refactor: select in dataset_service (DatasetService class) (#34525)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-03 12:01:31 +00:00
Renzo
06dde4f503
refactor: select in account_service (TenantService class) (#34499)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Asuka Minato <i@asukaminato.eu.org>
2026-04-03 11:03:45 +00:00
YBoy
64ddec0d67
refactor(api): type annotation service dicts with TypedDict (#34482)
Co-authored-by: Asuka Minato <i@asukaminato.eu.org>
2026-04-03 06:25:52 +00:00
Renzo
da3b0caf5e
refactor: select in account_service (RegisterService class) (#34500)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-03 06:21:26 +00:00
Renzo
608958de1c
refactor: select in external_knowledge_service (#34493)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-03 03:42:16 +00:00
Renzo
7eb632eb34
refactor: select in rag_pipeline (#34495)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-03 03:42:01 +00:00
Renzo
33d4fd357c
refactor: select in account_service (AccountService class) (#34496) 2026-04-03 03:41:46 +00:00
YBoy
71d299d0d3
refactor(api): type hit testing retrieve responses with TypedDict (#34484) 2026-04-03 02:25:30 +00:00
YBoy
9a6222f245
refactor(api): type webhook data extraction with RawWebhookDataDict TypedDict (#34486) 2026-04-03 02:24:17 +00:00
YBoy
affe5ed30b
refactor(api): type get_knowledge_rate_limit with KnowledgeRateLimitD… (#34483) 2026-04-03 02:23:32 +00:00
Renzo
dbfb474eab
refactor: select in workflow_tools_manage_service (#34477) 2026-04-02 12:35:04 +00:00