opensource
/
dify
mirror of https://github.com/langgenius/dify.git
2
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,451 Commits 291 Branches 160 Tags 518 MiB
Commit Graph

517 Commits

Author SHA1 Message Date
Guangdong Liu d299e75e1b
refactor: use dynamic max characters for chunking in extractors (#26782) 2025-10-13 10:22:59 +08:00
carribean cbf2ba6cec
Feature integrate alibabacloud mysql vector (#25994) 
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
 2025-10-11 10:47:28 +08:00
Asuka Minato 1bd621f819
remove .value (#26633) 
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-10-11 09:08:29 +08:00
Asuka Minato bb6a331490
change all to httpx (#26119) 
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
 2025-10-10 23:41:16 +08:00
Asuka Minato ab2eacb6c1
use model_validate (#26182) 
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
 2025-10-10 17:30:13 +09:00
znn 2b6882bd97
fix chunks 2 (#26623) 
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
 2025-10-10 16:01:33 +08:00
kenwoodjw 8d803a26eb
fix: duplicate chunks (#26360) 
Signed-off-by: kenwoodjw <blackxin55+@gmail.com>
 2025-09-30 10:53:55 +08:00
AkisAya 66196459d5
fix db connection error in embed_documents() (#26196) 2025-09-28 13:44:51 +08:00
longbingljw 24b4289d6c
fix:add some explanation for oceanbase parser selection (#26071) 
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
 2025-09-23 17:06:06 +08:00
Shili Cao 345ac8333c
Add Full-Text & Hybrid Search Support to Baidu Vector DB and Update SDK, Closes #25982 (#25983) 
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
 2025-09-22 10:17:35 +08:00
longbingljw 208fe3d7de
feat:support selecting different ftparser for OceanBase. (#25970) 
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
 2025-09-22 09:56:33 +08:00
-LAN- 4ba1292455
refactor: replace print statements with proper logging (#25773) 2025-09-18 20:35:47 +08:00
-LAN- 85cda47c70
feat: knowledge pipeline (#25360) 
Signed-off-by: -LAN- <laipz8200@outlook.com>
Co-authored-by: twwu <twwu@dify.ai>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
Co-authored-by: jyong <718720800@qq.com>
Co-authored-by: Wu Tianwei <30284043+WTW0313@users.noreply.github.com>
Co-authored-by: QuantumGhost <obelisk.reg+git@gmail.com>
Co-authored-by: lyzno1 <yuanyouhuilyz@gmail.com>
Co-authored-by: quicksand <quicksandzn@gmail.com>
Co-authored-by: Jyong <76649700+JohnJyong@users.noreply.github.com>
Co-authored-by: lyzno1 <92089059+lyzno1@users.noreply.github.com>
Co-authored-by: zxhlyh <jasonapring2015@outlook.com>
Co-authored-by: Yongtao Huang <yongtaoh2022@gmail.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Joel <iamjoel007@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: nite-knite <nkCoding@gmail.com>
Co-authored-by: Hanqing Zhao <sherry9277@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Harry <xh001x@hotmail.com>
 2025-09-18 12:49:10 +08:00
Jiang b283b10d3e
Fix/lindorm vdb optimize (#25748) 
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
 2025-09-16 16:54:18 +08:00
Asuka Minato bdd85b36a4
ruff check preview (#25653) 
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
 2025-09-16 12:58:12 +08:00
湛露先生 0bbf4fb66a
correct typos . (#25717) 
Signed-off-by: zhanluxianshen <zhanluxianshen@163.com>
 2025-09-15 21:22:40 +08:00
-LAN- bab4975809
chore: add ast-grep rule to convert Optional[T] to T | None (#25560) 
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
 2025-09-15 13:06:33 +08:00
Ritoban Dutta 67a686cf98
[Chore/Refactor] use __all__ to specify export member. (#25681) 2025-09-15 09:45:35 +08:00
ChasePassion a3f2c05632
optimize _merge_splits function by using enumerate instead of manual index tracking (#25680) 
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-09-15 09:41:16 +08:00
Krito. a13d7987e0
chore: adopt StrEnum and auto() for some string-typed enums (#25129) 
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
 2025-09-12 21:14:26 +08:00
kenwoodjw c91253d05d
fix segment deletion race condition (#24408) 
Signed-off-by: kenwoodjw <blackxin55+@gmail.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
 2025-09-12 15:29:57 +08:00
Asuka Minato cbc0e639e4
update sql in batch (#24801) 
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: -LAN- <laipz8200@outlook.com>
 2025-09-10 13:00:17 +08:00
-LAN- 08dd3f7b50
Fix basedpyright type errors (#25435) 
Signed-off-by: -LAN- <laipz8200@outlook.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
 2025-09-10 01:54:26 +08:00
Yongtao Huang 2ac7a9c8fc
Chore: thanks to bump-pydantic (#25437) 2025-09-09 20:07:17 +08:00
Asuka Minato 38057b1b0e
add typing to all wraps (#25405) 
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
 2025-09-09 16:48:33 +08:00
Asuka Minato f6059ef389
add more typing (#24949) 
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
 2025-09-08 10:40:00 +08:00
-LAN- 9b8a03b53b
[Chore/Refactor] Improve type annotations in models module (#25281) 
Signed-off-by: -LAN- <laipz8200@outlook.com>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
 2025-09-08 09:42:27 +08:00
NeatGuyCoding afa7228076
fix: a failed index to be marked as created (#25290) 
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
 2025-09-06 22:53:26 +08:00
Asuka Minato a78339a040
remove bare list, dict, Sequence, None, Any (#25058) 
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: -LAN- <laipz8200@outlook.com>
 2025-09-06 03:32:23 +08:00
Yongtao Huang 432f89cf33
Chore: clean some # type: ignore (#25157) 2025-09-05 11:30:04 +08:00
Yoshio Sugiyama 4966e4e1fb
fix: Remove invalid key from firecrawl request payload. (#25190) 
Signed-off-by: SUGIYAMA Yoshio <nenegi.01mo@gmail.com>
 2025-09-05 10:10:56 +08:00
Davide Delbianco cdf9b674dc
chore: Bump weaviate-client to latest v3 version (#25096) 2025-09-04 11:15:36 +08:00
-LAN- 53c4a8787f
[Chore/Refactor] Improve type safety and resolve type checking issues (#25104) 2025-09-04 09:35:32 +08:00
GuanMu ff7a0e3170
fix: improve error logging for vector search operation in MyScale (#25087) 2025-09-03 22:24:45 +08:00
-LAN- 9d5956cef8
[Chore/Refactor] Switch from MyPy to Basedpyright for type checking (#25047) 
Signed-off-by: -LAN- <laipz8200@outlook.com>
 2025-09-03 11:52:26 +08:00
湛露先生 1fff4620e6
clean console apis and rag cleans. (#25042) 
Signed-off-by: zhanluxianshen <zhanluxianshen@163.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
 2025-09-03 11:25:18 +08:00
Yongtao Huang bc9efa7ea8
Refactor: use DatasourceType.XX.value instead of hardcoded (#25015) 
Signed-off-by: Yongtao Huang <yongtaoh2022@gmail.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
 2025-09-03 08:56:48 +08:00
Yongtao Huang 8fcc864fb7
Post fix of #23224 (#25007) 2025-09-02 20:59:08 +08:00
Bowen Liang 7b379e2a61
chore: apply ty checks on api code with script and ci action (#24653) 2025-09-02 16:05:13 +08:00
湛露先生 deea07e905
make clean() function in index_processor_base abstractmethod (#24959) 
Signed-off-by: zhanluxianshen <zhanluxianshen@163.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
 2025-09-02 14:48:45 +08:00
Novice ca96350707
chore: optimize SQL queries that perform partial full table scans (#24786) 2025-09-02 11:46:11 +08:00
Yongtao Huang be3af1e234
Migrate SQLAlchemy from 1.x to 2.0 with automated and manual adjustments (#23224) 
Co-authored-by: Yongtao Huang <99629139+hyongtao-db@users.noreply.github.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
 2025-09-02 10:30:19 +08:00
Frederick2313072 2042353526
fix:score threshold (#24897) 2025-09-02 08:58:14 +08:00
wlleiiwang 9486715929
FEAT: Tencent Vector optimize BM25 initialization to reduce loading time (#24915) 
Co-authored-by: wlleiiwang <wlleiiwang@tencent.com>
 2025-09-01 21:08:41 +08:00
Frederick2313072 5b3cc560d5
fix:hard-coded top-k fallback issue. (#24879) 2025-09-01 15:46:37 +08:00
willzhao ffba341258
[CHORE]: remove redundant-cast (#24807) 2025-09-01 14:05:32 +08:00
Bowen Liang 39064197da
chore: cleanup unnecessary mypy suppressions on imports (#24712) 2025-08-28 23:17:25 +08:00
Asuka Minato 4adf85d7d4
example for rm extra cast (#24646) 2025-08-28 09:37:39 +08:00
Asuka Minato d2f234757b
example try rm ignore (#24649) 2025-08-28 09:36:16 +08:00
Yongtao Huang 826f19e968
Chore : rm dead code detected by pylance (#24588) 2025-08-27 13:19:40 +08:00
Petrus Han d9e26eba65
fix: rag/milvus clarify full-text search warning with actionable guidance (#24570) 2025-08-26 23:32:26 +08:00
Yongtao Huang fa753239ad
Refactor: use logger = logging.getLogger(__name__) in logging (#24515) 
Co-authored-by: Yongtao Huang <99629139+hyongtao-db@users.noreply.github.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
 2025-08-26 18:10:31 +08:00
GuanMu 47f480c0dc
fix: unify log format, use placeholders instead of string concatenation (#24544) 2025-08-26 15:45:16 +08:00
huangzhuo1949 98473e9d4f
fix:external dataset weight rerank bug (#24533) 
Co-authored-by: huangzhuo <huangzhuo1@xiaomi.com>
 2025-08-26 14:54:40 +08:00
-LAN- 04954918a5
Merge commit from fork 
* fix(oraclevector): SQL Injection

Signed-off-by: -LAN- <laipz8200@outlook.com>

* fix(oraclevector): Remove bind variables from FETCH FIRST clause

Oracle doesn't support bind variables in the FETCH FIRST clause.
Fixed by using validated integers directly in the SQL string while
maintaining proper input validation to prevent SQL injection.

- Updated search_by_vector method to use validated top_k directly
- Updated search_by_full_text method to use validated top_k directly
- Adjusted parameter numbering for document_ids_filter placeholders

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Signed-off-by: -LAN- <laipz8200@outlook.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-08-26 13:51:23 +08:00
huayaoyue6 23dcb2dc1b
fix(vector): use semantic version comparison for version check (#24409) 2025-08-24 21:04:33 +08:00
-LAN- da9af7b547
[Chore/Refactor] Use centralized naive_utc_now for UTC datetime operations (#24352) 
Signed-off-by: -LAN- <laipz8200@outlook.com>
 2025-08-22 23:53:05 +08:00
Asuka Minato 51cc2bf429
example of next(, None) (#24345) 2025-08-22 18:32:22 +08:00
willzhao 5ab6bc283c
[CHORE]: x: T = None to x: Optional[T] = None (#24217) 2025-08-21 21:58:39 +08:00
Yongtao Huang 106ab7f2a8
Fix: safe defaults for BaseModel dict fields (#24098) 
Co-authored-by: Yongtao Huang <99629139+hyongtao-db@users.noreply.github.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
 2025-08-21 21:38:55 +08:00
Guangdong Liu 1abf1240b2
refactor: replace try-except blocks with contextlib.suppress for cleaner exception handling (#24284) 2025-08-21 18:18:49 +08:00
8bitpd a183b2affb
fix: rollback when AnalyticDB create zhparser failed (#24260) 
Co-authored-by: xiaozeyu <xiaozeyu.xzy@alibaba-inc.com>
 2025-08-21 15:00:26 +08:00
Amy 738aaee101
fix(api):Fix the issue of empty and not empty operations failing in k… (#24276) 
Co-authored-by: liumin <min.liu@tongdun.net>
 2025-08-21 14:43:08 +08:00
8bitpd 6b1606f4f4
fix: keep idempotent when init AnalyticdbVectorBySql (#24239) 
Co-authored-by: xiaozeyu <xiaozeyu.xzy@alibaba-inc.com>
 2025-08-20 23:22:27 +08:00
yihong 4c1ad40f8e
docs: format all md files (#24195) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-08-20 13:49:11 +08:00
He Wang 670d479e32
Bump pyobvector to 0.2.15 (#24120) 2025-08-18 17:36:27 +08:00
crazywoola 8288b1dcab
Revert "fix pg_vector extension requires SUPERUSER, but not availabl… (#24108) 2025-08-18 16:46:15 +08:00
Elvis_LEE 16d1289a0a
fix pg_vector extension requires SUPERUSER, but not available on Huawei Cloud RDS (#24093) 
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
 2025-08-18 16:29:36 +08:00
Bo Wu 790a6ec203
fix: return empty list instead of raising exception for qdrant search when score_threshold is 1 (#24032) 2025-08-18 12:44:05 +08:00
-LAN- e340fccafb
feat: integrate flask-orjson for improved JSON serialization performance (#23935) 2025-08-14 19:50:59 +08:00
engchina 7566d90dfe
fix issue #23758 (#23764) 
Co-authored-by: root <root@thinkpad-pc.localdomain>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
 2025-08-12 10:26:13 +08:00
yunqiqiliang 14e1c16cf2
Fix ClickZetta stability and reduce logging noise (#23632) 
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
 2025-08-08 22:57:47 +08:00
湛露先生 fd536a943a
word extractor cleans. (#20926) 
Signed-off-by: zhanluxianshen <zhanluxianshen@163.com>
 2025-08-08 09:37:51 +08:00
yunqiqiliang 62772e8871
fix: ensure vector database cleanup on dataset deletion regardless of document presence (affects all 33 vector databases) (#23574) 
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
 2025-08-08 09:18:43 +08:00
Qiang Lee e9045a8838
Fix: Apply Metadata Filters Correctly in Full-Text Search Mode for Tencent Cloud Vector Database (#23564) 2025-08-07 05:36:06 -07:00
yunqiqiliang e01510e2a6
feat: Add Clickzetta Lakehouse vector database integration (#22551) 
Co-authored-by: Claude <noreply@anthropic.com>
 2025-08-07 14:21:46 +08:00
Yongtao Huang 6b8b31ff64
Remove unnecessary issubclass check (#23455) 
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
 2025-08-06 13:43:55 +08:00
Yongtao Huang 406c1952b8
Fix version comparison with imported_version (#23326) 
Signed-off-by: Yongtao Huang <yongtaoh2022@gmail.com>
 2025-08-04 10:40:49 +08:00
wanttobeamaster da5c003f97
chore: tablestore full text search support score normalization (#23255) 
Co-authored-by: xiaozhiqing.xzq <xiaozhiqing.xzq@alibaba-inc.com>
 2025-08-01 14:14:11 +08:00
Aurelius Huang ffddabde43
feat(notion): Notion Database extracts Rows content `in row order` and appends `Row Page URL` (#22646) 
Co-authored-by: Aurelius Huang <cm.huang@aftership.com>
 2025-07-30 21:35:20 +08:00
kenwoodjw 28478cdc41
feat: support metadata condition filter string array (#23111) 
Signed-off-by: kenwoodjw <blackxin55+@gmail.com>
 2025-07-30 16:13:45 +08:00
rhochman eee576355b
Fix: Support for Elasticsearch Cloud Connector (#23017) 
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-07-30 11:12:16 +08:00
Yongtao Huang 1c05491f1c
Chore: remove duplicate TYPE_CHECKING import (#23013) 
Signed-off-by: Yongtao Huang <yongtaoh2022@gmail.com>
 2025-07-28 10:04:45 +08:00
Asuka Minato a189d293f8
make logging not use f-str, change others to f-str (#22882) 2025-07-25 10:32:48 +08:00
Asuka Minato ef51678c73
orm filter -> where (#22801) 
Signed-off-by: -LAN- <laipz8200@outlook.com>
Co-authored-by: -LAN- <laipz8200@outlook.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-07-24 00:57:45 +08:00
wanttobeamaster 8278b39f85
fix tablestore full text search bug (#22853) 2025-07-23 19:31:47 +08:00
wanttobeamaster 1c3c40db69
fix: tablestore TypeError when vector is missing (#22843) 
Co-authored-by: xiaozhiqing.xzq <xiaozhiqing.xzq@alibaba-inc.com>
 2025-07-23 18:59:16 +08:00
wlleiiwang b4e152f775
FEAT: Tencent Vector search supports backward compatibility with the previous score calculation approach. (#22820) 
Co-authored-by: wlleiiwang <wlleiiwang@tencent.com>
 2025-07-23 15:38:31 +08:00
Asuka Minato 6d3e198c3c
Mapped column (#22644) 
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-07-23 00:39:59 +08:00
wanttobeamaster a2048fd0f4
fix: tablestore vdb support metadata filter (#22774) 
Co-authored-by: xiaozhiqing.xzq <xiaozhiqing.xzq@alibaba-inc.com>
 2025-07-22 16:48:59 +08:00
issac2e 58d92970a9
Optimize tencent_vector knowledge base deletion error handling with batch processing support (#22726) 
Co-authored-by: liuchen15 <liuchen15@gaotu.cn>
Co-authored-by: crazywoola <427733928@qq.com>
 2025-07-22 08:21:41 +08:00
uply23333 ab012fe1a2
fix: improve document filtering in full text search(elasticsearch) (#22683) 2025-07-21 15:59:37 +08:00
8bitpd 9251a66a10
fix: update analyticdb vector to do filter by metadata (#22698) 
Co-authored-by: xiaozeyu <xiaozeyu.xzy@alibaba-inc.com>
 2025-07-21 15:03:37 +08:00
znn ed263aed9f
fix text splitter (#22596) 2025-07-18 13:51:58 +08:00
-LAN- 460a825ef1
refactor: decouple Node and NodeData (#22581) 
Signed-off-by: -LAN- <laipz8200@outlook.com>
Co-authored-by: QuantumGhost <obelisk.reg+git@gmail.com>
 2025-07-18 10:08:51 +08:00
helojo e7d80bf7bf
Fix: the pict type picture was not processed in the docx (#19305) 
Co-authored-by: zqgame <zqgame@zqgame.local>
 2025-07-17 22:53:35 +08:00
yihong d2933c2bfe
fix: drop dead code phase2 unused class (#22042) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-07-17 09:33:07 +08:00
wanttobeamaster bf7b2c339b
tablestore vector support more method (#22225) 
Co-authored-by: xiaozhiqing.xzq <xiaozhiqing.xzq@alibaba-inc.com>
 2025-07-15 09:58:48 +08:00
Jacky Wu 3e96c0c468
fix: close session before doing long latency operation (#22306) 2025-07-14 15:16:10 +08:00
luckylhb90 a371390d6c
optimize: batch embedding and qdrant write_consistency_factor parameter (#21776) 
Co-authored-by: hobo.l <hobo.l@binance.com>
 2025-07-10 10:16:59 +08:00