04919195cc
Merge remote-tracking branch 'upstream/feat/queue-based-graph-engine' into feat/rag-2
2025-09-17 18:00:48 +08:00
73a7756350
feat(graph_engine): allow to dumps and loads RSC
2025-09-17 12:45:51 +08:00
02d15ebd5a
feat(graph_engine): support dumps and loads in GraphExecution
2025-09-16 19:38:10 +08:00
dd34002db2
Merge remote-tracking branch 'upstream/main' into feat/rag-2
2025-09-16 14:59:35 +08:00
b5684f1992
refactor(graph_engine): remove unused parameters from Engine
2025-09-16 14:11:42 +08:00
5f263147f9
fix: make mypy happy
2025-09-16 12:51:11 +08:00
da87fce751
feat(graph_engine): dump and load ready queue
2025-09-16 04:19:46 +08:00
bab4975809
chore: add ast-grep rule to convert Optional[T] to T | None ( #25560 )
...
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2025-09-15 13:06:33 +08:00
b4ef1de30f
feat(graph_engine): add ready_queue state persistence to GraphRuntimeState
...
- Add ReadyQueueState TypedDict for type-safe queue serialization
- Add ready_queue attribute to GraphRuntimeState for initializing with pre-existing queue state
- Update GraphEngine to load ready_queue from GraphRuntimeState on initialization
- Implement proper type hints using ReadyQueueState for better type safety
- Add comprehensive tests for ready_queue loading functionality
The ready_queue is read-only after initialization and allows resuming workflow
execution with a pre-populated queue of nodes ready to execute.
2025-09-15 03:05:10 +08:00
4cdc19fd05
feat(graph_engine): add abstract layer and dump / load methods for ready queue.
2025-09-14 04:19:24 +08:00
a13d7987e0
chore: adopt StrEnum and auto() for some string-typed enums ( #25129 )
...
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
2025-09-12 21:14:26 +08:00
7e69403dda
refactor(graph_engine): use singledispatchmethod in event_handler
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-10 03:12:33 +08:00
836ed1f380
refactor(graph_engine): Move ErrorHandler into a single file package
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-10 02:35:05 +08:00
f17c71e08a
refactor(graph_engine): Move GraphStateManager to single file package.
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-10 01:55:30 +08:00
d52621fce3
refactor(graph_engine): Merge error strategies into error_handler.py
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-10 01:49:46 +08:00
a23c8fcb1a
refactor: move execution limits from engine core to layer
...
Remove max_execution_time and max_execution_steps from ExecutionContext and GraphEngine since these limits are now handled by ExecutionLimitsLayer. This follows the separation of concerns principle by keeping execution limits as a cross-cutting concern handled by layers rather than embedded in core engine components.
Changes:
- Remove max_execution_time and max_execution_steps from ExecutionContext
- Remove these parameters from GraphEngine.__init__()
- Remove max_execution_time from Dispatcher
- Update workflow_entry.py to no longer pass these parameters
- Update all tests to remove these parameters
2025-09-10 01:32:45 +08:00
23cd615489
Merge branch 'feat/queue-based-graph-engine' into feat/rag-2
2025-09-08 14:30:43 +08:00
a78339a040
remove bare list, dict, Sequence, None, Any ( #25058 )
...
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: -LAN- <laipz8200@outlook.com>
2025-09-06 03:32:23 +08:00
103a9a4e67
fix(graph_engine): add type hint for workers_to_remove
2025-09-05 01:59:11 +08:00
15b3443e9e
fix(debug_logging_layer): remove access for variable pool
2025-09-05 01:52:19 +08:00
b9394d542c
Merge branch 'feat/queue-based-graph-engine' into feat/rag-2
...
# Conflicts:
# api/core/app/apps/advanced_chat/generate_task_pipeline.py
# api/pyproject.toml
# api/uv.lock
# docker/docker-compose-template.yaml
# docker/docker-compose.yaml
# web/package.json
2025-09-04 20:30:08 +08:00
e229510e73
perf: eliminate lock contention in worker pool by removing callbacks
...
Remove worker idle/active callbacks that caused severe lock contention.
Instead, use sampling-based monitoring where worker states are queried
on-demand during scaling decisions. This eliminates the performance
bottleneck caused by workers acquiring locks 10+ times per second.
Changes:
- Remove callback parameters from Worker class
- Add properties to expose worker idle state directly
- Update WorkerPool to query worker states without callbacks
- Maintain scaling functionality with better performance
2025-09-04 19:37:31 +08:00
36048d1526
feat(graph_engine): allow to scale down without lock
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-04 19:32:07 +08:00
ad9eed2551
fix: disable scale for perfermance
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-04 19:11:22 +08:00
4d63bd2083
refactor(graph_engine): rename SimpleWorkerPool to WorkerPool
2025-09-04 17:47:13 +08:00
226f14a20f
feat(graph_engine): implement scale down worker
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-04 15:35:20 +08:00
ac057a2d40
Chore: remove dead code in class Graph ( #22791 )
...
Co-authored-by: Yongtao Huang <99629139+hyongtao-db@users.noreply.github.com>
2025-09-04 10:30:04 +08:00
04bbf540d9
chore: code format
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-04 02:33:53 +08:00
657c27ec75
feat(graph_engine): make runtime state read-only in layer
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-04 02:30:40 +08:00
8332472944
refactor(graph_engine): rename Layer to GraphEngineLayer
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-04 02:11:31 +08:00
fe3f03e50a
feat: add property-based access control to GraphRuntimeState
...
- Replace direct field access with private attributes and property decorators
- Implement deep copy protection for mutable objects (dict, LLMUsage)
- Add helper methods: set_output(), get_output(), update_outputs()
- Add increment_node_run_steps() and add_tokens() convenience methods
- Update loop_node and event_handlers to use new accessor methods
- Add comprehensive unit tests for immutability and validation
- Ensure backward compatibility with existing property access patterns
2025-09-04 02:08:58 +08:00
d4aed3df5c
Merge branch 'feat/queue-based-graph-engine' into feat/rag-2
...
# Conflicts:
# api/core/memory/token_buffer_memory.py
# api/core/rag/extractor/notion_extractor.py
# api/core/repositories/sqlalchemy_workflow_node_execution_repository.py
# api/core/variables/variables.py
# api/core/workflow/graph/graph.py
# api/core/workflow/graph_engine/entities/event.py
# api/services/dataset_service.py
# web/app/components/app-sidebar/index.tsx
# web/app/components/base/tag-management/selector.tsx
# web/app/components/base/toast/index.tsx
# web/app/components/datasets/create/website/index.tsx
# web/app/components/datasets/create/website/jina-reader/base/options-wrap.tsx
# web/app/components/workflow/header/version-history-button.tsx
# web/app/components/workflow/hooks/use-inspect-vars-crud-common.ts
# web/app/components/workflow/hooks/use-workflow-interactions.ts
# web/app/components/workflow/panel/version-history-panel/index.tsx
# web/service/base.ts
2025-09-03 15:01:06 +08:00
7ace7e65e1
fix preview and recommend plugins
2025-09-02 16:44:18 +08:00
1770b93e5b
chore(graph_engine): Add a TODO commment in `_update_response_outputs` in event_handlers
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-02 15:20:03 +08:00
d8ff4aa9ba
feat(graph_engine): Handle NodeRunAgentLogEvent
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-02 15:02:07 +08:00
9f8f21bf87
chore: remove backup files
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-02 15:01:58 +08:00
0b0dc63f29
Merge remote-tracking branch 'origin/main' into feat/queue-based-graph-engine
2025-09-02 11:52:25 +08:00
ffba341258
[CHORE]: remove redundant-cast ( #24807 )
2025-09-01 14:05:32 +08:00
8433cf4437
refactor(graph_engine): Merge event_collector and event_emitter into event_manager
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-01 13:15:58 +08:00
bb5d52539c
refactor(graph_engine): Merge branch_handler into edge_processor
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-01 12:53:06 +08:00
88622f70fb
refactor(graph_engine): Move setup methods into `__init__`
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-01 12:08:03 +08:00
0fdb1b2bc9
refactor(graph_engine): Correct private attributes and private methods naming
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-01 04:37:23 +08:00
a5cb9d2b73
refactor(graph_engine): inline output_registry into response_coordinator
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-01 03:59:53 +08:00
64c1234724
refactor(graph_engine): Merge worker management into one WorkerPool
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-01 03:23:47 +08:00
202fdfcb81
refactor(graph_engine): Remove backward compatibility code
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-01 02:41:16 +08:00
e2f4c9ba8d
refactor(graph_engine): Merge state managers into unified_state_manager
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-09-01 02:08:08 +08:00
82193580de
chore: improve typing
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-08-30 16:35:57 +08:00
d0dd728e6c
Merge branch 'main' into feat/rag-2
...
# Conflicts:
# api/core/app/entities/queue_entities.py
# api/core/workflow/graph_engine/entities/event.py
2025-08-29 11:29:51 +08:00
1db04aa729
Merge branch 'feat/queue-based-graph-engine' into feat/rag-2
2025-08-28 18:12:49 +08:00
c396788128
chore(graph_engine): add final mark to classes
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-08-28 15:38:35 +08:00
QuantumGhost