opensource
/
dify
mirror of https://github.com/langgenius/dify.git
2
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
dify/api/services
History
Frederick2313072 626e71cb3b feat: implement content-based deduplication for document segments 
- Add database index on (dataset_id, index_node_hash) for efficient deduplication queries
- Add deduplication check in SegmentService.create_segment and multi_create_segment methods
- Add deduplication check in DatasetDocumentStore.add_documents method to prevent duplicate embedding processing
- Skip creating segments with identical content hashes across the entire dataset

This prevents duplicate content from being re-processed and re-embedded when uploading documents with repeated content, improving efficiency and reducing unnecessary compute costs.
 		2025-09-20 06:28:14 +08:00
..
auth update sql in batch (#24801) 2025-09-10 13:00:17 +08:00
enterprise feat: knowledge pipeline (#25360) 2025-09-18 12:49:10 +08:00
entities feat: knowledge pipeline (#25360) 2025-09-18 12:49:10 +08:00
errors chore: add ast-grep rule to convert Optional[T] to T | None (#25560) 2025-09-15 13:06:33 +08:00
plugin refactor: replace print statements with proper logging (#25773) 2025-09-18 20:35:47 +08:00
rag_pipeline Chore: correct inconsistent logging and typo (#25945) 2025-09-19 10:36:16 +08:00
recommend_app chore: add ast-grep rule to convert Optional[T] to T | None (#25560) 2025-09-15 13:06:33 +08:00
tools fix: handle None description in MCP tool transformation (#25872) 2025-09-18 13:11:38 +08:00
workflow feat: knowledge pipeline (#25360) 2025-09-18 12:49:10 +08:00
__init__.py chore(api/services): apply ruff reformatting (#7599) 2024-08-26 13:43:57 +08:00
account_service.py fix: remove billing cache when add or delete app or member (#25885) 2025-09-18 12:18:07 +08:00
advanced_prompt_template_service.py chore: adopt StrEnum and auto() for some string-typed enums (#25129) 2025-09-12 21:14:26 +08:00
agent_service.py chore: add ast-grep rule to convert Optional[T] to T | None (#25560) 2025-09-15 13:06:33 +08:00
annotation_service.py chore: add ast-grep rule to convert Optional[T] to T | None (#25560) 2025-09-15 13:06:33 +08:00
api_based_extension_service.py remove bare list, dict, Sequence, None, Any (#25058) 2025-09-06 03:32:23 +08:00
app_dsl_service.py feat: knowledge pipeline (#25360) 2025-09-18 12:49:10 +08:00
app_generate_service.py feat: knowledge pipeline (#25360) 2025-09-18 12:49:10 +08:00
app_model_config_service.py remove bare list, dict, Sequence, None, Any (#25058) 2025-09-06 03:32:23 +08:00
app_service.py fix: remove billing cache when add or delete app or member (#25885) 2025-09-18 12:18:07 +08:00
audio_service.py chore: add ast-grep rule to convert Optional[T] to T | None (#25560) 2025-09-15 13:06:33 +08:00
billing_service.py fix: remove billing cache when add or delete app or member (#25885) 2025-09-18 12:18:07 +08:00
clear_free_plan_tenant_expired_logs.py update sql in batch (#24801) 2025-09-10 13:00:17 +08:00
code_based_extension_service.py remove bare list, dict, Sequence, None, Any (#25058) 2025-09-06 03:32:23 +08:00
conversation_service.py chore: add ast-grep rule to convert Optional[T] to T | None (#25560) 2025-09-15 13:06:33 +08:00
dataset_service.py feat: implement content-based deduplication for document segments 2025-09-20 06:28:14 +08:00
datasource_provider_service.py feat: knowledge pipeline (#25360) 2025-09-18 12:49:10 +08:00
external_knowledge_service.py chore: add ast-grep rule to convert Optional[T] to T | None (#25560) 2025-09-15 13:06:33 +08:00
feature_service.py feat: knowledge pipeline (#25360) 2025-09-18 12:49:10 +08:00
file_service.py feat: knowledge pipeline (#25360) 2025-09-18 12:49:10 +08:00
hit_testing_service.py remove bare list, dict, Sequence, None, Any (#25058) 2025-09-06 03:32:23 +08:00
knowledge_service.py feat: mypy for all type check (#10921) 2024-12-24 18:38:51 +08:00
message_service.py fix: Message => str (#25876) 2025-09-18 17:57:57 +08:00
metadata_service.py chore: add ast-grep rule to convert Optional[T] to T | None (#25560) 2025-09-15 13:06:33 +08:00
model_load_balancing_service.py chore: add ast-grep rule to convert Optional[T] to T | None (#25560) 2025-09-15 13:06:33 +08:00
model_provider_service.py chore: add ast-grep rule to convert Optional[T] to T | None (#25560) 2025-09-15 13:06:33 +08:00
oauth_server.py feat: oauth provider (#24206) 2025-08-29 14:10:51 +08:00
operation_service.py chore(api/services): apply ruff reformatting (#7599) 2024-08-26 13:43:57 +08:00
ops_service.py chore: add ast-grep rule to convert Optional[T] to T | None (#25560) 2025-09-15 13:06:33 +08:00
recommended_app_service.py chore: add ast-grep rule to convert Optional[T] to T | None (#25560) 2025-09-15 13:06:33 +08:00
saved_message_service.py chore: add ast-grep rule to convert Optional[T] to T | None (#25560) 2025-09-15 13:06:33 +08:00
tag_service.py chore: add ast-grep rule to convert Optional[T] to T | None (#25560) 2025-09-15 13:06:33 +08:00
variable_truncator.py feat: knowledge pipeline (#25360) 2025-09-18 12:49:10 +08:00
vector_service.py chore: add ast-grep rule to convert Optional[T] to T | None (#25560) 2025-09-15 13:06:33 +08:00
web_conversation_service.py chore: add ast-grep rule to convert Optional[T] to T | None (#25560) 2025-09-15 13:06:33 +08:00
webapp_auth_service.py chore: add ast-grep rule to convert Optional[T] to T | None (#25560) 2025-09-15 13:06:33 +08:00
website_service.py feat: knowledge pipeline (#25360) 2025-09-18 12:49:10 +08:00
workflow_app_service.py feat: knowledge pipeline (#25360) 2025-09-18 12:49:10 +08:00
workflow_draft_variable_service.py feat: knowledge pipeline (#25360) 2025-09-18 12:49:10 +08:00
workflow_run_service.py chore: add ast-grep rule to convert Optional[T] to T | None (#25560) 2025-09-15 13:06:33 +08:00
workflow_service.py Refactor WorkflowService to handle missing default credentials gracef… (#25960) 2025-09-19 00:45:35 -07:00
workspace_service.py Fix basedpyright type errors (#25435) 2025-09-10 01:54:26 +08:00