Commit Graph

3371 Commits

Author SHA1 Message Date
Yansong Zhang
e9ee897973 fix: resolve remaining CI failures for style checks and unit tests
- Add model_features property and build_execution_context method to
  AgentAppRunner to fix mypy attr-defined errors
- Export WorkflowComment, WorkflowCommentReply, WorkflowCommentMention
  from models/__init__.py to fix import errors
- Add NestedNodeGraphRequest, NestedNodeGraphResponse,
  NestedNodeParameterSchema to services/workflow/entities.py
- Update test_agent_chat_app_runner: tests for invalid LLM mode and
  invalid strategy now reflect unified AgentAppRunner behavior
  (no longer raises ValueError for these cases)

Made-with: Cursor
2026-04-13 16:07:38 +08:00
Yansong Zhang
971828615e fix: resolve CI failures for Python style, DB migration, and unit tests
- Fix type errors in dify_graph/nodes/agent/agent_node.py:
  - Add missing user_id param to get_agent_tool_runtime call
  - Use create_plugin_provider_manager instead of bare ProviderManager()
  - Pass provider_manager to ModelManager constructor
  - Add access_controller param to file_factory.build_from_mapping
  - Fix return type annotation for _fetch_memory
- Fix DB migration chain: update workflow_comments migration to point
  to correct parent after sandbox migration removal
- Fix test_app_generate_service: set AGENT_V2_TRANSPARENT_UPGRADE=False
  in mock config to prevent transparent upgrade intercepting test flow
- Fix test_app_generator: add scalar method to mock db.session
- Fix test_app_models: add AppMode.AGENT to expected modes set
- Remove unnecessary db.session.close() from agent_chat app_runner

Made-with: Cursor
2026-04-13 15:07:16 +08:00
zyssyz123
3e0578a1c6
Merge branch 'main' into feat/new-agent-node 2026-04-13 13:43:47 +08:00
sxxtony
7dd507af04
refactor: migrate SegmentAttachmentBinding to TypeBase (#34810) 2026-04-13 05:22:43 +00:00
Yansong Zhang
5f87239abc fix: resolve CI failures — unused imports, type errors, test updates
- Remove 12 unused imports across node.py, tool_manager.py,
  event_adapter.py, legacy_response_adapter.py
- Fix Sequence[str] → list[str] type annotation in node.py
- Update test_agent_chat_app_runner.py: CotChatAgentRunner →
  AgentAppRunner (old runner classes replaced by unified runner)

Made-with: Cursor
2026-04-13 13:10:08 +08:00
Yansong Zhang
c03b25a940 merge: resolve conflicts with origin/main
Conflicts resolved:
- workflow_app_runner.py: adopt main's DifyGraphInitContext pattern
- token_buffer_memory.py: adopt main's match/case, add AppMode.AGENT
- app_dsl_service.py: adopt main's match/case, add AppMode.AGENT

Made-with: Cursor
2026-04-13 12:52:56 +08:00
XHamzaX
596559efc9
fix(rag): include is_summary and original_chunk_id in default vector projection (#34950)
Co-authored-by: VFootball Dev <vfootball@example.com>
2026-04-13 03:11:08 +00:00
Yansong Zhang
90cce7693f revert: remove all sandbox and skill related code
Remove ~12,900 lines of sandbox/skill code that was ported from
feat/support-agent-sandbox. This reverts to direct tool execution
(the original behavior before sandbox integration).

Removed:
- core/sandbox/ (SandboxBuilder, bash tools, providers, initializers)
- core/skill/ (SkillManager, assembler, entities)
- core/virtual_environment/ (5 provider implementations)
- core/zip_sandbox/ (archive operations)
- core/app_assets/ (asset management)
- core/app_bundle/ (bundle management)
- controllers/cli_api/ (DifyCli callback endpoints)
- services/sandbox/ (provider service)
- services/skill_service, app_asset_service, app_bundle_service
- models/sandbox.py, app_asset.py
- bin/dify-cli-* (3 platform binaries)
- web sandbox-provider-page and service
- SandboxLayer, _resolve_sandbox_context, _invoke_tool_in_sandbox
- CliApiConfig, DIFY_SANDBOX_CONTEXT_KEY
- sandbox-related migrations

Preserved: All Agent V2 core functionality (agent-v2 node, strategy
engine, transparent upgrade, LLM remapping, memory, context, tools
via direct execution).

Made-with: Cursor
2026-04-13 10:42:36 +08:00
aviu16
d06bc2f2e1
refactor(api): type _build_log_dict return with LogDict TypedDict (#34983)
Co-authored-by: Asuka Minato <i@asukaminato.eu.org>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-13 01:06:36 +00:00
wdeveloper16
534fea7104
refactor: replace bare dict with typed annotations in external_data_tool (#34996) 2026-04-13 01:02:22 +00:00
wdeveloper16
602753b68a
refactor: replace bare dict with ActionDict TypedDict in cot_agent_runner (#34997) 2026-04-13 00:53:20 +00:00
wdeveloper16
0841b4c663
refactor(api): migrate tools, account, workflow and plugin services to SQLAlchemy 2.0 (#34966)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-12 01:45:27 +00:00
wdeveloper16
12814b55d2
refactor(api): migrate core RAG layer to SQLAlchemy 2.0 select() API (#34965) 2026-04-11 16:32:20 +00:00
wangxiaolei
65d66768c1
fix: fix tool output duplicate (#34962) 2026-04-11 15:07:31 +00:00
YBoy
e0d69204cd
refactor(api): type DatasourceInvokeMeta.to_dict with TypedDict (#34940) 2026-04-11 00:39:06 +00:00
YBoy
f2d6275da4
refactor(api): type get_prompt_template with TypedDict (#34943) 2026-04-11 00:38:16 +00:00
YBoy
992ac38d0d
refactor(api): type ToolInvokeMeta.to_dict with TypedDict (#34942) 2026-04-11 00:37:10 +00:00
Ygor Leal
8633b2f1f7
refactor(tools): replace redundant dict[str, str] with EmojiIconDict (#34786)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-10 18:53:05 +08:00
Yansong Zhang
e04f00d29b feat(api): add context injection and Jinja2 support to Agent V2 node
Agent V2 now fully covers all LLM node capabilities:
- Context injection: {{#context#}} placeholder replaced with upstream
  knowledge retrieval results via _build_context_string()
- Jinja2 template rendering via _render_jinja2() with variable pool
- Multi-variable references across upstream nodes

Compatibility verified (7/7):
- T1: Context injection ({{#context#}})
- T2: Variable template resolution ({{#start.var#}})
- T3: Multi-upstream variable refs
- T4: Old Chat app with opening_statement
- T5: Old app sensitive_word_avoidance
- T6: Old app more_like_this
- T7: Old Completion app with variable substitution

Made-with: Cursor
2026-04-10 17:05:48 +08:00
corevibe555
488fcd4f83
refactor(services): replace Union with | syntax in service layer (batch 2) (#34906)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-10 07:05:31 +00:00
Yansong Zhang
59b9221501 fix(api): fix AWS CodeInterpreter stdout capture failure
Root cause: _WORKDIR was hardcoded to "/home/user" which doesn't exist
in AWS AgentCore Code Interpreter environment (actual pwd is
/opt/amazon/genesis1p-tools/var). Every command was prefixed with
"cd /home/user && ..." which failed silently, producing empty stdout.

Fix:
- Default _WORKDIR to "/tmp" (universally available)
- Auto-detect actual working directory via "pwd" during
  _construct_environment and override _WORKDIR dynamically

Verified: echo, python3, uname all return correct stdout.
Made-with: Cursor
2026-04-10 14:21:06 +08:00
corevibe555
7f4bf19186
refactor(mcp): remove unused AnyFunction alias, tighten callback type (#34890) 2026-04-10 05:48:01 +00:00
corevibe555
07c573e52f
refactor(api): replace Optional/Union with | syntax, remove dead AnyFunction (#34894) 2026-04-10 05:46:05 +00:00
corevibe555
5fafac0ca4
refactor(api): modernize type annotations — replace Optional/Union with | syntax (#34888)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-10 05:04:10 +00:00
volcano303
c41b62f47e
refactor(api): type format_preview returns with TypedDicts in index processors (#34893) 2026-04-10 05:01:01 +00:00
Yansong Zhang
218c10ba4f feat(api): add SSH private key auth support and verify SSH/E2B providers
- SSH Provider: add automatic private key detection in ssh_password
  field (RSA/Ed25519/ECDSA) alongside existing password auth.
- SSH Provider verified end-to-end on EC2: connection, command exec,
  CLI binary upload via SFTP, dify init, tool symlink creation.
- E2B Provider verified: cloud sandbox creation, CLI binary upload,
  dify init with tool symlinks.
- Add linux/amd64 CLI binary for E2B (x86_64 cloud sandboxes).

Made-with: Cursor
2026-04-10 12:57:40 +08:00
Yansong Zhang
698af54c4f feat(api): complete end-to-end Docker sandbox auto tool execution
Full pipeline working: Agent V2 node → Docker container creation →
CLI binary upload (linux/arm64) → dify init (fetch tools from API) →
dify execute (tool callback via CLI API) → result returned.

Fixes:
- Use sandbox.id (not vm.metadata.id) for CLI paths
- Upload CLI binary to container during sandbox creation
- Resolve linux binary separately for Docker containers on macOS
- Save Docker provider config via SandboxProviderService (proper
  encryption) instead of raw DB insert
- Add verbose logging for sandbox tool execution path
- Fix NameError: binary not defined

Made-with: Cursor
2026-04-10 11:28:02 +08:00
Pulakesh
d19f47b458
fix(api): replace assert isinstance with proper runtime type checks in message transformers (#34865) 2026-04-10 03:19:52 +00:00
carlos4s
86fd94767c
refactor(api): use sessionmaker in relyt & tidb_vector VDB services (#34848) 2026-04-10 03:16:25 +00:00
corevibe555
d826ac7099
refactor(models): replace Any with precise types in Tenant and MCPToo… (#34880)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-10 03:12:38 +00:00
Yansong Zhang
73fd439541 fix(api): resolve sandbox deadlock under gevent and refine integration
- Skip Local sandbox provider under gevent worker (subprocess pipes
  cause cooperative threading deadlock with Celery's gevent pool).
- Add non-blocking sandbox readiness check before tool execution.
- Add gevent timeout wrapper for sandbox bash session.
- Fix CLI binary resolution: add SANDBOX_DIFY_CLI_ROOT config field.
- Fix ExecutionContext.node_id propagation.
- Fix SkillInitializer to gracefully handle missing skill bundles.
- Update _invoke_tool_in_sandbox to use correct `dify execute` CLI
  subcommand format (not `invoke-tool`).

The full sandbox-in-agent pipeline works end-to-end for network-based
providers (Docker, E2B, SSH). Local provider is skipped under gevent
but works in non-gevent contexts.

Made-with: Cursor
2026-04-10 10:51:40 +08:00
Yansong Zhang
5cdae671d5 feat(api): integrate Sandbox Provider into Agent V2 execution pipeline
Close 3 integration gaps between the ported Sandbox system and Agent V2:

1. Fix _invoke_tool_in_sandbox to use SandboxBashSession context manager
   API correctly (keyword args, bash_tool, ToolReference), with graceful
   fallback to direct invocation when DifyCli binary is unavailable.

2. Inject sandbox into run_context via _resolve_sandbox_context() in
   WorkflowBasedAppRunner — automatically creates a sandbox when a
   tenant has an active sandbox provider configured.

3. Register SandboxLayer in both advanced_chat and workflow app runners
   for proper sandbox lifecycle cleanup on graph end.

Also: make SkillInitializer non-fatal when no skill bundle exists,
add node_id to ExecutionContext for sandbox session scoping.

Made-with: Cursor
2026-04-10 10:14:42 +08:00
volcano303
40e23ce8dc
refactor(api): type DatasourceProviderApiEntity.to_dict with TypedDict (#34879) 2026-04-10 01:47:59 +00:00
Jean Ibarz
d50f096b14
fix(mcp): catch JSONDecodeError in OAuth discovery functions 🤖🤖🤖 (#34868) 2026-04-10 01:28:57 +00:00
dataCenter430
1117b6e72d
refactor: convert appmode misc if/elif to match/case (#30001) (#34869) 2026-04-10 00:35:12 +00:00
dataCenter430
c5c5c71d15
refactor(api): type OpenSearch/Lindorm/Huawei VDB config params dicts with TypedDicts (#34870) 2026-04-10 00:34:34 +00:00
Yansong Zhang
2de2a8fd3a fix(api): resolve multi-turn memory failure in Agent apps
- Auto-resolve parent_message_id when not provided by client,
  querying the latest message in the conversation to maintain
  the thread chain that extract_thread_messages() relies on.
- Add AppMode.AGENT to TokenBufferMemory mode checks so file
  attachments in memory are handled via the workflow branch.
- Add debug logging for memory injection in node_factory and node.

Made-with: Cursor
2026-04-09 16:27:38 +08:00
wangxiaolei
d042cbc62e
fix: fix remove_leading_symbols remove [ (#34832)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-09 08:22:09 +00:00
-LAN-
d1e33ba9ea
refactor(api): reduce Dify GraphInitParams usage (#34825) 2026-04-09 07:59:15 +00:00
Yansong Zhang
e2e16772a1 fix(api): fix DSL import, memory loading, and remaining test coverage
1. DSL Import fix: change self._session.commit() to self._session.flush()
   in app_dsl_service.py _create_or_update_app() to avoid "closed transaction"
   error. DSL import now works: export agent app -> import -> new app created.

2. Memory loading attempt: added _load_memory_messages() to AgentV2Node
   that loads TokenBufferMemory from conversation history. However, chatflow
   engine manages conversations differently from easy-UI (conversation may
   not be in DB at query time, or uses ConversationVariablePersistenceLayer
   instead of Message table). Memory needs further investigation.

Test results:
- Multi-turn memory: Turn 1 OK, Turn 2 LLM doesn't see history (needs deeper fix)
- Service API with API Key: PASSED (answer="Sixteen" for 8+8)
- DSL Import: PASSED (status=completed, new app created)
- Token aggregation: PASSED (node=49, workflow=49)

Known: memory in multi-turn chatflow needs to use graphon's built-in
memory mechanism (MemoryConfig on node + ConversationVariablePersistenceLayer)
rather than direct DB query.

Made-with: Cursor
2026-04-09 14:47:55 +08:00
carlos4s
d360929af1
refactor(api): use sessionmaker in pgvecto_rs VDB service (#34818) 2026-04-09 05:49:03 +00:00
dataCenter430
5f53748d07
refactor: convert ToolProviderType if/elif to match/case (#30001) (#34794) 2026-04-09 05:48:40 +00:00
aliworksx08
b5acc8e392
refactor: migrate session.query to select API in core tools (#34814)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-09 05:44:49 +00:00
Yansong Zhang
b21a443d56 fix(api): resolve all remaining known issues
1. Fix workflow-level total_tokens=0:
   Call graph_runtime_state.add_tokens(usage.total_tokens) in both
   _run_without_tools and _run_with_tools paths after node execution.
   Previously only graphon's internal ModelInvokeCompletedEvent handler
   called add_tokens, which agent-v2 doesn't emit.

2. Fix Turn 2 SSE empty response:
   Set PUBSUB_REDIS_CHANNEL_TYPE=streams in .env. Redis Streams
   provides durable event delivery (consumers can replay past events),
   solving the pub/sub at-most-once timing issue.

3. Skill -> Agent runtime integration:
   SandboxBuilder.build() now auto-includes SkillInitializer if not
   already present. This ensures sandbox.attrs has the skill bundle
   loaded for downstream consumers (tool execution in sandbox).

4. LegacyResponseAdapter:
   New module at core/app/apps/common/legacy_response_adapter.py.
   Filters workflow-specific SSE events (workflow_started, node_started,
   node_finished, workflow_finished) from the stream, passing through
   only message/message_end/agent_log/error/ping events that old
   clients expect.

46 unit tests pass.

Made-with: Cursor
2026-04-09 12:53:11 +08:00
Yansong Zhang
4f010cd4f5 fix(api): stop emitting StreamChunkEvent from tool path to prevent answer duplication
The EventAdapter was converting every LLMResultChunk from the agent
strategy into StreamChunkEvent. Combined with the answer node's
{{#agent.text#}} variable output, this caused the final answer to
appear twice (e.g., "It is 2026-04-09 04:27:45.It is 2026-04-09 04:27:45.").

Now LLMResultChunk from strategy output is silently consumed (text still
accumulates in AgentResult.text via the strategy). Only AgentLogEvent
(thought/tool_call/round) is forwarded to the pipeline.

Known remaining issues:
- workflow/message level total_tokens=0 (node level is correct at 33)
  because pipeline aggregation doesn't include agent-v2 node tokens
- Turn 2 SSE delivery timing with Redis pubsub (celery executes OK)

Made-with: Cursor
2026-04-09 12:31:49 +08:00
Yansong Zhang
3d4be88d97 fix(api): remove unsupported 'user' param from FC/ReAct invoke_llm calls
FunctionCallStrategy and ReActStrategy were passing user=self.context.user_id
to ModelInstance.invoke_llm() which doesn't accept that parameter.
This caused tool-using agent runs to fail with:
  "ModelInstance.invoke_llm() got an unexpected keyword argument 'user'"

Verified: Agent V2 with current_time tool now works end-to-end:
  ROUND 1: LLM thought -> CALL current_time -> got time
  ROUND 2: LLM generates answer with time info
Made-with: Cursor
2026-04-09 12:18:07 +08:00
Jake Armstrong
c19a822e1b
refactor: deduplicate DefaultRetrievalModelDict TypedDict into retrieval_service.py (#34758) 2026-04-09 04:13:04 +00:00
Yansong Zhang
482a004efe fix(api): fix duplicate answer and completion app upgrade issues
1. Remove StreamChunkEvent from AgentV2Node._run_without_tools():
   The agent-v2 node was yielding StreamChunkEvent during LLM streaming,
   AND the downstream answer node was outputting the same text via
   {{#agent.text#}} variable reference, causing "FourFour" duplication.
   Now text only flows through outputs.text -> answer node (single path).

2. Map inputs to query for completion app transparent upgrade:
   Completion apps send {inputs: {query: "..."}} not {query: "..."}.
   VirtualWorkflowSynthesizer route now extracts query from inputs
   when the top-level query is missing.

Verified:
- Old chat app: "What is 2+2?" -> "Four" (was "FourFour")
- Old completion app: {inputs: {query: "What is 3+3?"}} -> "3 + 3 = 6" (was failing)
- Old agent-chat app: still works

Made-with: Cursor
2026-04-09 12:02:43 +08:00
Yansong Zhang
66212e3575 feat(api): implement zero-migration transparent upgrade (Phase 8)
Add two feature-flag-controlled upgrade paths that allow existing apps
and LLM nodes to transparently run through the Agent V2 engine without
any database migration:

1. AGENT_V2_TRANSPARENT_UPGRADE (default: off):
   When enabled, old apps (chat/completion/agent-chat) bypass legacy
   Easy-UI runners. VirtualWorkflowSynthesizer converts AppModelConfig
   to an in-memory Workflow (start -> agent-v2 -> answer) at runtime,
   then executes via AdvancedChatAppGenerator. Falls back to legacy
   path on any synthesis error.

   VirtualWorkflowSynthesizer maps:
   - model JSON -> ModelConfig
   - pre_prompt/chat_prompt_config -> prompt_template
   - agent_mode.tools -> ToolMetadata[]
   - agent_mode.strategy -> agent_strategy
   - dataset_configs -> context
   - file_upload -> vision

2. AGENT_V2_REPLACES_LLM (default: off):
   When enabled, DifyNodeFactory.create_node() transparently remaps
   nodes with type="llm" to type="agent-v2" before class resolution.
   Since AgentV2NodeData is a strict superset of LLMNodeData, the
   mapping is lossless. With tools=[], Agent V2 behaves identically
   to LLM Node.

Both flags default to False for safety. Turn off = instant rollback.
46 existing tests pass. Flask starts successfully.

Made-with: Cursor
2026-04-09 10:30:52 +08:00
dataCenter430
7ca5b726a2
refactor: convert ProviderQuota if/elif to match/case (#30001) (#34791) 2026-04-09 02:28:19 +00:00