opensource
/
dify
mirror of https://github.com/langgenius/dify.git
2
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,400 Commits 288 Branches 160 Tags 494 MiB
Commit Graph

478 Commits

Author SHA1 Message Date
chenhuan0728 770c461a8f
feat： add openGauss PQ acceleration feature (#16432) 
Co-authored-by: chenhuan <huan.chen0728@foxmail>
 2025-03-24 15:16:40 +08:00
Jyong d135677c25
add vdb document id index (#16244) 
Co-authored-by: crazywoola <427733928@qq.com>
 2025-03-20 01:38:15 +08:00
Jyong a8879057c0
fix tidb metadata filter (#16237) 2025-03-19 19:44:56 +08:00
Jyong 81325df368
fix weaviate metadata filter (#16230) 2025-03-19 18:26:53 +08:00
Jyong b8ef3149ef
metadata expect value check error (#16210) 2025-03-19 17:48:01 +08:00
Jyong c3c957bb80
change recreate_collection function to create_collection (#16212) 2025-03-19 17:13:08 +08:00
Jyong abeaea4f79
Support knowledge metadata filter (#15982) 2025-03-18 16:42:19 +08:00
Jyong 33ba7e659b
fix vector db sql injection (#16096) 2025-03-18 15:07:29 +08:00
LittleFish-15 223ab5a38f
feat: support openGauss vector database (#15865) 2025-03-17 19:42:54 +08:00
huangzhuo1949 695a7400a9
fix:delete empty table bug (#15517) 
Co-authored-by: huangzhuo <huangzhuo1@xiaomi.com>
 2025-03-17 10:53:26 +08:00
Jyong 84a866028a
fix document could be None (#15818) 2025-03-14 16:40:01 +08:00
Yuichiro Utsumi 5f9d236d22
Feat: Add pg_bigm for keyword search in pgvector (#13876) 
Signed-off-by: Yuichiro Utsumi <utsumi.yuichiro@fujitsu.com>
 2025-03-13 16:32:34 +08:00
Jyong a8e8c37fdd
improve text split (#15719) 2025-03-13 15:29:33 +08:00
kenwoodjw 087bb60b31
fix: preserve Unicode characters in keyword search queries (#15522) 
Signed-off-by: kenwoodjw <blackxin55+@gmail.com>
 2025-03-12 18:34:42 +08:00
Jyong f77f7e1437
fix text split (#15426) 2025-03-11 00:24:27 +08:00
Jyong 435564f0f2
fix parent-child retrival count (#15119) 2025-03-06 22:32:38 +08:00
engchina 9c1db7dca7
modify oracle lexer name Fixes #15106 (#15108) 
Co-authored-by: engchina <atjapan2015@gmail.com>
 2025-03-06 18:58:51 +08:00
llinvokerl d04f40c274
Fix empty results issue in full-text search with Milvus vector database (#14885) 
Co-authored-by: liusurong.lsr <liusurong.lsr@alibaba-inc.com>
 2025-03-05 12:27:01 +08:00
engchina c8de30f3d9
feat: support oracle oci autonomouse database. Fixes #14792 and Fixes #14628. (#14804) 
Co-authored-by: engchina <atjapan2015@gmail.com>
 2025-03-04 09:22:04 +08:00
yuhaowin 1e3197a1ea
Fixes 14217: database retrieve api and chat-messages api response doc_metadata (#14219) 2025-02-27 14:56:46 +08:00
Rhys 548f6ef2b6
fix: incorrect score in the chroma vector (#14273) 2025-02-25 09:40:22 +08:00
Bowen Liang dfdd6dfa20
fix: change the config name and fix typo in description of the number of retrieval executors (#13856) 2025-02-19 09:13:36 +08:00
Jyong aa19bb3f30
fix session close issue (#13946) 2025-02-18 19:29:57 +08:00
Charlie.Wei abe5aca3e2
Retrieval service optimization (#13849) 2025-02-17 18:22:36 +08:00
Yeuoly 403e2d58b9
Introduce Plugins (#13836) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Signed-off-by: -LAN- <laipz8200@outlook.com>
Signed-off-by: xhe <xw897002528@gmail.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: takatost <takatost@gmail.com>
Co-authored-by: kurokobo <kuro664@gmail.com>
Co-authored-by: Novice Lee <novicelee@NoviPro.local>
Co-authored-by: zxhlyh <jasonapring2015@outlook.com>
Co-authored-by: AkaraChen <akarachen@outlook.com>
Co-authored-by: Yi <yxiaoisme@gmail.com>
Co-authored-by: Joel <iamjoel007@gmail.com>
Co-authored-by: JzoNg <jzongcode@gmail.com>
Co-authored-by: twwu <twwu@dify.ai>
Co-authored-by: Hiroshi Fujita <fujita-h@users.noreply.github.com>
Co-authored-by: AkaraChen <85140972+AkaraChen@users.noreply.github.com>
Co-authored-by: NFish <douxc512@gmail.com>
Co-authored-by: Wu Tianwei <30284043+WTW0313@users.noreply.github.com>
Co-authored-by: 非法操作 <hjlarry@163.com>
Co-authored-by: Novice <857526207@qq.com>
Co-authored-by: Hiroki Nagai <82458324+nagaihiroki-git@users.noreply.github.com>
Co-authored-by: Gen Sato <52241300+halogen22@users.noreply.github.com>
Co-authored-by: eux <euxuuu@gmail.com>
Co-authored-by: huangzhuo1949 <167434202+huangzhuo1949@users.noreply.github.com>
Co-authored-by: huangzhuo <huangzhuo1@xiaomi.com>
Co-authored-by: lotsik <lotsik@mail.ru>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
Co-authored-by: nite-knite <nkCoding@gmail.com>
Co-authored-by: Jyong <76649700+JohnJyong@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: gakkiyomi <gakkiyomi@aliyun.com>
Co-authored-by: CN-P5 <heibai2006@gmail.com>
Co-authored-by: CN-P5 <heibai2006@qq.com>
Co-authored-by: Chuehnone <1897025+chuehnone@users.noreply.github.com>
Co-authored-by: yihong <zouzou0208@gmail.com>
Co-authored-by: Kevin9703 <51311316+Kevin9703@users.noreply.github.com>
Co-authored-by: -LAN- <laipz8200@outlook.com>
Co-authored-by: Boris Feld <lothiraldan@gmail.com>
Co-authored-by: mbo <himabo@gmail.com>
Co-authored-by: mabo <mabo@aeyes.ai>
Co-authored-by: Warren Chen <warren.chen830@gmail.com>
Co-authored-by: JzoNgKVO <27049666+JzoNgKVO@users.noreply.github.com>
Co-authored-by: jiandanfeng <chenjh3@wangsu.com>
Co-authored-by: zhu-an <70234959+xhdd123321@users.noreply.github.com>
Co-authored-by: zhaoqingyu.1075 <zhaoqingyu.1075@bytedance.com>
Co-authored-by: 海狸大師 <86974027+yenslife@users.noreply.github.com>
Co-authored-by: Xu Song <xusong.vip@gmail.com>
Co-authored-by: rayshaw001 <396301947@163.com>
Co-authored-by: Ding Jiatong <dingjiatong@gmail.com>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: JasonVV <jasonwangiii@outlook.com>
Co-authored-by: le0zh <newlight@qq.com>
Co-authored-by: zhuxinliang <zhuxinliang@didiglobal.com>
Co-authored-by: k-zaku <zaku99@outlook.jp>
Co-authored-by: luckylhb90 <luckylhb90@gmail.com>
Co-authored-by: hobo.l <hobo.l@binance.com>
Co-authored-by: jiangbo721 <365065261@qq.com>
Co-authored-by: 刘江波 <jiangbo721@163.com>
Co-authored-by: Shun Miyazawa <34241526+miya@users.noreply.github.com>
Co-authored-by: EricPan <30651140+Egfly@users.noreply.github.com>
Co-authored-by: crazywoola <427733928@qq.com>
Co-authored-by: sino <sino2322@gmail.com>
Co-authored-by: Jhvcc <37662342+Jhvcc@users.noreply.github.com>
Co-authored-by: lowell <lowell.hu@zkteco.in>
Co-authored-by: Boris Polonsky <BorisPolonsky@users.noreply.github.com>
Co-authored-by: Ademílson Tonato <ademilsonft@outlook.com>
Co-authored-by: Ademílson Tonato <ademilson.tonato@refurbed.com>
Co-authored-by: IWAI, Masaharu <iwaim.sub@gmail.com>
Co-authored-by: Yueh-Po Peng (Yabi) <94939112+y10ab1@users.noreply.github.com>
Co-authored-by: Jason <ggbbddjm@gmail.com>
Co-authored-by: Xin Zhang <sjhpzx@gmail.com>
Co-authored-by: yjc980121 <3898524+yjc980121@users.noreply.github.com>
Co-authored-by: heyszt <36215648+hieheihei@users.noreply.github.com>
Co-authored-by: Abdullah AlOsaimi <osaimiacc@gmail.com>
Co-authored-by: Abdullah AlOsaimi <189027247+osaimi@users.noreply.github.com>
Co-authored-by: Yingchun Lai <laiyingchun@apache.org>
Co-authored-by: Hash Brown <hi@xzd.me>
Co-authored-by: zuodongxu <192560071+zuodongxu@users.noreply.github.com>
Co-authored-by: Masashi Tomooka <tmokmss@users.noreply.github.com>
Co-authored-by: aplio <ryo.091219@gmail.com>
Co-authored-by: Obada Khalili <54270856+obadakhalili@users.noreply.github.com>
Co-authored-by: Nam Vu <zuzoovn@gmail.com>
Co-authored-by: Kei YAMAZAKI <1715090+kei-yamazaki@users.noreply.github.com>
Co-authored-by: TechnoHouse <13776377+deephbz@users.noreply.github.com>
Co-authored-by: Riddhimaan-Senapati <114703025+Riddhimaan-Senapati@users.noreply.github.com>
Co-authored-by: MaFee921 <31881301+2284730142@users.noreply.github.com>
Co-authored-by: te-chan <t-nakanome@sakura-is.co.jp>
Co-authored-by: HQidea <HQidea@users.noreply.github.com>
Co-authored-by: Joshbly <36315710+Joshbly@users.noreply.github.com>
Co-authored-by: xhe <xw897002528@gmail.com>
Co-authored-by: weiwenyan-dev <154779315+weiwenyan-dev@users.noreply.github.com>
Co-authored-by: ex_wenyan.wei <ex_wenyan.wei@tcl.com>
Co-authored-by: engchina <12236799+engchina@users.noreply.github.com>
Co-authored-by: engchina <atjapan2015@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: 呆萌闷油瓶 <253605712@qq.com>
Co-authored-by: Kemal <kemalmeler@outlook.com>
Co-authored-by: Lazy_Frog <4590648+lazyFrogLOL@users.noreply.github.com>
Co-authored-by: Yi Xiao <54782454+YIXIAO0@users.noreply.github.com>
Co-authored-by: Steven sun <98230804+Tuyohai@users.noreply.github.com>
Co-authored-by: steven <sunzwj@digitalchina.com>
Co-authored-by: Kalo Chin <91766386+fdb02983rhy@users.noreply.github.com>
Co-authored-by: Katy Tao <34019945+KatyTao@users.noreply.github.com>
Co-authored-by: depy <42985524+h4ckdepy@users.noreply.github.com>
Co-authored-by: 胡春东 <gycm520@gmail.com>
Co-authored-by: Junjie.M <118170653@qq.com>
Co-authored-by: MuYu <mr.muzea@gmail.com>
Co-authored-by: Naoki Takashima <39912547+takatea@users.noreply.github.com>
Co-authored-by: Summer-Gu <37869445+gubinjie@users.noreply.github.com>
Co-authored-by: Fei He <droxer.he@gmail.com>
Co-authored-by: ybalbert001 <120714773+ybalbert001@users.noreply.github.com>
Co-authored-by: Yuanbo Li <ybalbert@amazon.com>
Co-authored-by: douxc <7553076+douxc@users.noreply.github.com>
Co-authored-by: liuzhenghua <1090179900@qq.com>
Co-authored-by: Wu Jiayang <62842862+Wu-Jiayang@users.noreply.github.com>
Co-authored-by: Your Name <you@example.com>
Co-authored-by: kimjion <45935338+kimjion@users.noreply.github.com>
Co-authored-by: AugNSo <song.tiankai@icloud.com>
Co-authored-by: llinvokerl <38915183+llinvokerl@users.noreply.github.com>
Co-authored-by: liusurong.lsr <liusurong.lsr@alibaba-inc.com>
Co-authored-by: Vasu Negi <vasu-negi@users.noreply.github.com>
Co-authored-by: Hundredwz <1808096180@qq.com>
Co-authored-by: Xiyuan Chen <52963600+GareArc@users.noreply.github.com>
 2025-02-17 17:05:13 +08:00
Charlie.Wei 222df44d21
Retrieval Service efficiency optimization (#13543) 2025-02-17 14:09:57 +08:00
Bowen Liang 0751ad1eeb
feat(vdb): add HNSW vector index for TiDB vector store with TiFlash (#12043) 2025-02-12 13:53:51 +08:00
liuzhenghua 47a64610ca
Fix the issue of repeated escaping of quotes in hit test (#13477) 2025-02-11 09:58:31 +08:00
Ademílson Tonato d0a21086bd
refactor: Update Firecrawl API parameters and default settings (#13082) 2025-01-29 11:21:05 +08:00
Ademílson Tonato 6024d8a42d
refactor: Update Firecrawl to use v1 API (#12574) 
Co-authored-by: Ademílson Tonato <ademilson.tonato@refurbed.com>
 2025-01-23 11:14:48 +08:00
huangzhuo1949 4c3076f2a4
feat: add pg vector index (#12338) 
Co-authored-by: huangzhuo <huangzhuo1@xiaomi.com>
 2025-01-22 17:07:18 +08:00
Bowen Liang 166221d784
chore(lint): fix quotes for f-string formatting by bumping ruff to 0.9.x (#12702) 2025-01-21 10:12:29 +08:00
yihong 4e101604c3
fix: ruff check for True if ... else (#12576) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-01-13 09:38:48 +08:00
CN-P5 cd257b91c5
Fix pandas indexing method for knowledge base imports (#12637) (#12638) 
Co-authored-by: CN-P5 <heibai2006@qq.com>
 2025-01-13 09:06:59 +08:00
YoungLH 040a3b782c
FEAT: support milvus to full text search (#11430) 
Signed-off-by: YoungLH <974840768@qq.com>
 2025-01-08 17:39:53 +08:00
Yingchun Lai 53bb37b749
fix: fix the incorrect plaintext file key when saving (#10429) 2025-01-08 12:52:45 +08:00
Hiroshi Fujita d2586278d6
Feat elasticsearch japanese (#12194) 2025-01-08 12:35:41 +08:00
Jyong 05bda6f38d
add tidb on qdrant redis lock (#12462) 2025-01-08 08:55:44 +08:00
huangzhuo1949 70698024f5
fix: empty delete bug (#12339) 
Co-authored-by: huangzhuo <huangzhuo1@xiaomi.com>
 2025-01-03 20:46:39 +08:00
Jyong b873e6349c
add child chunk preview number limit (#12309) 2025-01-03 16:14:27 +08:00
-LAN- 8d15c8cfbf
fix: improve error handling in NotionExtractor data fetching (#12182) 
Signed-off-by: -LAN- <laipz8200@outlook.com>
 2024-12-29 11:53:09 +08:00
-LAN- dae1b5a619
fix: import jieba.analyse (#12133) 
Signed-off-by: -LAN- <laipz8200@outlook.com>
 2024-12-27 11:37:55 +08:00
Jyong 811e4bd0cf
fix unstructured setting (#12116) 2024-12-26 12:08:36 +08:00
Jyong 84ac004772
py lint (#12102) 
Signed-off-by: -LAN- <laipz8200@outlook.com>
Co-authored-by: -LAN- <laipz8200@outlook.com>
 2024-12-26 00:16:35 +08:00
Jyong 9231fdbf4c
Feat/support parent child chunk (#12092) 2024-12-25 19:49:07 +08:00
yihong 56e15d09a9
feat: mypy for all type check (#10921) 2024-12-24 18:38:51 +08:00
-LAN- 599d410d99
fix: validate reranking model attributes before processing (#11930) 
Signed-off-by: -LAN- <laipz8200@outlook.com>
 2024-12-21 21:23:12 +08:00
-LAN- 8c559d6231
fix(retrieval_service): avoid to use exception (#11925) 
Signed-off-by: -LAN- <laipz8200@outlook.com>
 2024-12-21 21:19:46 +08:00
yihong 7b03a0316d
fix: better memory usage from 800+ to 500+ (#11796) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2024-12-20 14:51:43 +08:00
yihong 463fbe2680
fix: better gard nan value from numpy for issue #11827 (#11864) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2024-12-20 09:28:32 +08:00
yihong 5a8a901560
fix: float values are not json for nan value close #11827 (#11840) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2024-12-19 20:50:20 +08:00
Jiang ad17ff9a92
Lindorm vdb bug-fix (#11790) 
Co-authored-by: jiangzhijie <jiangzhijie.jzj@alibaba-inc.com>
 2024-12-18 15:19:20 +08:00
Bowen Liang 924b4fe742
test: run vdb tests on TiDB Vector with docker in CI tests (#11645) 2024-12-15 17:16:40 +08:00
yihong 22258fb0bf
fix: filter bug for keywork cause code can not reach (#11666) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2024-12-15 17:12:06 +08:00
yihong 36cb25b341
fix: support mdx files close #11557 (#11565) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2024-12-12 13:37:56 +08:00
Jiang 0d04cdc323
Lindorm vdb (#11574) 
Co-authored-by: jiangzhijie <jiangzhijie.jzj@alibaba-inc.com>
 2024-12-12 09:43:27 +08:00
Jyong 9b7adcd4d9
update tidb batch get endpoint to basic mode (#11426) 2024-12-06 17:06:46 +08:00
Jyong d7c1f43b49
fix tidb full-text-search vector missed (#11337) 2024-12-04 16:13:23 +08:00
Jyong c58d2fce89
roll back rerank topn setting (#11297) 2024-12-03 17:34:56 +08:00
yihong e686f12317
fix: better handle error (#11265) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2024-12-03 09:15:38 +08:00
-LAN- 9601102885
fix(word_extractor): Fix type error and remove stream in ssrf_proxy (#11241) 
Signed-off-by: -LAN- <laipz8200@outlook.com>
 2024-12-02 10:24:03 +08:00
Cling_o3 f9c2aa7689
feat: add retireval_top_n to config in env (#11132) 2024-11-30 11:14:45 +08:00
kazuya-awano 2d6865d421
Ensure consistent float type for cached embedding return values (#10185) 2024-11-29 09:18:41 +08:00
yihong d7160ee563
fix: typo in upstashVector if id is always true, also fix some type hint (#11183) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2024-11-28 14:05:25 +08:00
-LAN- 9789905a1f
chore(*): Removes debugging print statements (#11145) 
Signed-off-by: -LAN- <laipz8200@outlook.com>
 2024-11-26 22:03:19 +08:00
Bowen Liang 6c8e208ef3
chore: bump minimum supported Python version to 3.11 (#10386) 2024-11-24 13:28:46 +08:00
yihong ed55de888a
fix: rules should not be None for in (#10977) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2024-11-22 23:04:20 +08:00
AkisAya cb0c55daa7
fix weight rerank of knowledge retrieval (#10931) 2024-11-21 17:53:20 +08:00
yihong 58a9d9eb9a
fix: better WeightRerankRunner run logic use O(1) and delete unused code (#10849) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2024-11-19 20:12:13 +08:00
Zane 14f3d44c37
refactor: improve handling of leading punctuation removal (#10761) 2024-11-18 21:32:33 +08:00
8bitpd 873e9720e9
feat: AnalyticDB vector store supports invocation via SQL. (#10802) 
Co-authored-by: 璟义 <yangshangpo.ysp@alibaba-inc.com>
 2024-11-18 19:29:54 +08:00
Bowen Liang 51db59622c
chore(lint): cleanup repeated cause exception in logging.exception replaced by helpful message (#10425) 2024-11-15 15:41:40 +08:00
Jyong 0b2d51d859
add the index field for elasticsearch (#10592) 2024-11-12 21:43:16 +08:00
-LAN- a1543b7da0
fix(extractor): temporary file (#10543) 2024-11-11 17:31:27 +08:00
Leo.Wang c9f785e00f
Feat/tools/gitlab (#10407) 2024-11-08 09:53:03 +08:00
Bowen Liang 574c4a264f
chore(lint): Use logging.exception instead of logging.error (#10415) 2024-11-07 21:13:02 +08:00
Jyong 1024fc623e
fix the ssrf of docx file extractor external images (#10237) 2024-11-04 15:22:07 +08:00
Jiang 0c9e79cd67
Add Lindorm as a VDB choice (#10202) 
Co-authored-by: jiangzhijie <jiangzhijie.jzj@alibaba-inc.com>
 2024-11-04 09:10:26 +08:00
Shili Cao b61baa87ec
fix: avoid unexpected error when create knowledge base with baidu vector database and wenxin embedding model (#10130) 2024-10-31 21:34:23 +08:00
Jyong dad041c49f
fix issue: query is none when doing retrieval (#10129) 2024-10-31 21:25:00 +08:00
omr 11ca1bec0b
fix: optimize unique document filtering with set (#10082) 2024-10-31 16:32:58 +08:00
zhuhao 7433095240
chore: use dify_config.TIDB_SPEND_LIMIT instead of constant value (#10038) 2024-10-30 15:43:07 +08:00
Jyong 9ebd453b87
add rerank check when doing mutil-retrieval (#9998) 2024-10-30 11:17:39 +08:00
powerfool 878d13ef42
Added OceanBase as an option for the vector store in Dify (#10010) 2024-10-29 21:10:18 +08:00
Jyong 5580bcf870
add tidb spend limit config (#9999) 2024-10-29 17:51:13 +08:00
roadgoat19 c8ef9223e5
feat: couchbase integration (#6165) 
Co-authored-by: crazywoola <427733928@qq.com>
Co-authored-by: Elliot Scribner <elliot.scribner@couchbase.com>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
 2024-10-29 15:00:23 +08:00
Jyong f47177ecb4
add top_k for es full text search (#9963) 2024-10-28 23:04:54 +08:00
virgosoy 17cacf258e
fix: wrong element object (#9868) 2024-10-25 22:32:41 +08:00
Jyong 18106a4fc6
add tidb on qdrant type (#9831) 
Co-authored-by: Zhaofeng Miao <522856232@qq.com>
 2024-10-25 13:57:03 +08:00
Zixuan Cheng 88dec6ef2b
Added description for .ppt, specify the reason for unstructured.io (#9452) 
Co-authored-by: crazywoola <427733928@qq.com>
 2024-10-24 22:13:06 +08:00
Jyong 5f11fe521d
remove unstructured pdf extract (#9794) 2024-10-24 18:13:05 +08:00
Jyong 3e9d271b52
nltk security issue and upgrade unstructured (#9558) 2024-10-23 16:23:55 +08:00
ice yao ceb2c4f3ef
chore: reuse existing test functions with upstash vdb (#9679) 2024-10-23 10:42:11 +08:00
Zven 8e7a752b2a
feat: add upstash as a new vector database provider (#9644) 2024-10-23 09:16:35 +08:00
-LAN- 5f12c17355
fix(core): use CreatedByRole enum for role consistency (#9607) 2024-10-22 13:03:50 +08:00
Bowen Liang 4d9160ca9f
refactor: use dify_config to replace legacy usage of flask app's config (#9089) 2024-10-22 11:01:32 +08:00
-LAN- e61752bd3a
feat/enhance the multi-modal support (#8818) 2024-10-21 10:43:49 +08:00
ice yao 2155bba5b0
fix: update mismatch vector type (#9462) 2024-10-18 08:21:41 +08:00
zhuhao b90ad587c2
refactor: move the embedding to the rag module and abstract the rerank runner for extension (#9423) 2024-10-17 19:12:42 +08:00
zhuhao 86594851cb
refactor: update the default values of top-k parameter in vdb to be consistent (#9367) 2024-10-16 16:00:21 +08:00
Jyong 50635e9c15
Fix/economical knowledge retrieval (#9396) 2024-10-16 15:13:45 +08:00
zhuhao cd7ab6231f
refactor: Add an enumeration type and use the factory pattern to obtain the corresponding class (#9356) 2024-10-15 12:51:13 +08:00
ice yao d15ba3939d
Add Volcengine VikingDB as new vector provider (#9287) 2024-10-13 21:26:05 +08:00
zhuhao d97d3ff5fc
chore: add abstract decorator and output log when query embedding fails (#9264) 2024-10-12 23:58:41 +08:00
Shili Cao 2ec6ffe478
feat:support baidu vector db (#9185) 2024-10-12 23:24:17 +08:00
Jyong 42b02b3a5f
Fix/agent external knowledge retrieval (#9241) 2024-10-11 19:21:03 +08:00
Jyong 80b62d50f5
Fix/add es num_candidates (#9225) 2024-10-11 16:04:23 +08:00
Jyong cabdb4ef17
fix retrieval resource positon missed (#9155) 
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
 2024-10-11 10:32:42 +08:00
Bowen Liang 240b66d737
chore: avoid implicit optional in type annotations of method (#8727) 2024-10-09 14:36:43 +08:00
Aurelius Huang 4585cffce1
fix: Compatible with special characters in pg full-text search. (#8921) 
Co-authored-by: Aurelius Huang <cm.huang@aftership.com>
 2024-09-30 16:32:23 +08:00
Jyong 9d221a5e19
external knowledge api (#8913) 
Co-authored-by: Yi <yxiaoisme@gmail.com>
 2024-09-30 15:38:43 +08:00
Zhaofeng Miao 369e1e6f58
feat(website-crawl): add jina reader as additional alternative for website crawling (#8761) 2024-09-30 09:57:19 +08:00
Bowen Liang 74f58f29f9
chore: bump ruff to 0.6.8 for fixing violation in SIM910 (#8869) 2024-09-29 00:29:59 +08:00
ice yao 27e33fb15c
chore: fix wrong VectorType match case (#8857) 2024-09-28 10:54:04 +08:00
zhuhao 55e6123db9
feat: add min-connection and max-connection for pgvector (#8841) 2024-09-27 18:16:20 +08:00
8bitpd 4c1063e1c5
fix: AnalyticdbVector retrieval scores (#8803) 2024-09-27 12:05:21 +08:00
zhuhao 008e0efeb0
refactor: update delete method as an abstract method (#8794) 2024-09-26 16:36:21 +08:00
crazywoola bf64ff215b
fix: . is missing in file_extension (#8736) 2024-09-25 10:09:20 +08:00
omr 8fd297f8b4
fix: redundant check for available_document_count (#8491) 2024-09-22 13:39:41 +08:00
Jyong 7e611ffbf3
multi-retrival use dataset's top-k (#8416) 2024-09-14 21:48:44 +08:00
Bowen Liang a1104ab97e
chore: refurish python code by applying Pylint linter rules (#8322) 2024-09-13 22:42:08 +08:00
Bowen Liang 6613b8f2e0
chore: fix unnecessary string concatation in single line (#8311) 2024-09-13 14:24:49 +08:00
-LAN- 08c486452f
fix: score_threshold handling in vector search methods (#8356) 2024-09-13 14:24:35 +08:00
Jyong 49cee773c5
fixed score threshold is none (#8342) 2024-09-13 10:21:58 +08:00
Bowen Liang 40fb4d16ef
chore: refurbish Python code by applying refurb linter rules (#8296) 2024-09-12 15:50:49 +08:00
Bowen Liang c69f5b07ba
chore: apply ruff E501 line-too-long linter rule (#8275) 
Co-authored-by: -LAN- <laipz8200@outlook.com>
 2024-09-12 14:00:36 +08:00
takatost 56c90e212a
fix(workflow): missing content in the answer node stream output during iterations (#8292) 
Co-authored-by: -LAN- <laipz8200@outlook.com>
 2024-09-12 13:59:48 +08:00
Bowen Liang 0f14873255
chore: cleanup ruff flake8-simplify linter rules (#8286) 
Co-authored-by: -LAN- <laipz8200@outlook.com>
 2024-09-12 12:55:45 +08:00
Bowen Liang 292220c596
chore: apply pep8-naming rules for naming convention (#8261) 2024-09-11 16:40:52 +08:00
Jyong bb3002b173
revert page column (#8217) 2024-09-10 18:21:22 +08:00
Bowen Liang 2cf1187b32
chore(api/core): apply ruff reformatting (#7624) 2024-09-10 17:00:20 +08:00
Jyong af92f19291
filter excel empty sheet (#8194) 2024-09-10 14:55:08 +08:00
Nam Vu 2d7954c7da
Fix variable typo (#8084) 2024-09-08 13:14:11 +08:00
ChengZi 2060db8e11
fix: change milvus init args from (host, port) to (url, token) (#8019) 
Signed-off-by: ChengZi <chen.zhang@zilliz.com>
 2024-09-06 17:32:48 +08:00
Byeongjin Kang d489b8b3e0
feat: return page number of pdf documents upon retrieval (#7749) 2024-09-05 16:43:26 +08:00
Jyong 0e71f6db84
fix spliter length missed (#7987) 2024-09-04 21:47:12 +08:00
KVOJJJin 14af87527f
Feat:remove estimation of embedding cost (#7950) 
Co-authored-by: jyong <718720800@qq.com>
 2024-09-04 14:41:47 +08:00
Sumkor 571415d1a4
fix: split text keep separator (#7930) 2024-09-04 12:59:10 +08:00
Jyong d8b6c053a2
fix rerank model value is empty string (#7937) 2024-09-03 21:25:21 +08:00
Jyong 01581dd35f
improve the notion table extract (#7925) 2024-09-03 17:52:07 +08:00
Jyong 6f33351eb3
ignore linked images when image id is none (#7890) 2024-09-02 19:37:05 +08:00
Jyong 60001a62c4
fixed chunk_overlap is None (#7703) 2024-08-27 16:38:06 +08:00
Kenn 122ce41020
feat: rewrite Elasticsearch index and search code to achieve Elasticsearch vector and full-text search (#7641) 
Co-authored-by: haokai <haokai@shuwen.com>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Co-authored-by: wellCh4n <wellCh4n@foxmail.com>
 2024-08-27 11:43:44 +08:00
Qin Liu 162faee4f2
fix: set score_threshold to zero if it is None for MyScale vectordb (#7640) 
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
 2024-08-27 09:47:16 +08:00
Jyong 7ae728a9a3
fix nltk averaged_perceptron_tagger download and fix score limit is none (#7582) 
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
 2024-08-26 15:14:05 +08:00
张皮皮 f29685f8a1
fix score_threshold is none, return all top K documents (#7581) 2024-08-23 16:59:34 +08:00
Byeongjin Kang 0223fc6fd5
feat: add pgvector full_text_search (#7396) 2024-08-20 11:01:13 +08:00
mochi ba79088ffc
Fix SQL parser Error in MyScale vdb. (#7255) 2024-08-14 16:41:18 +08:00
miendinh f104b930cf
feat: support elasticsearch vector database (#3558) 
Co-authored-by: miendinh <miendinh@users.noreply.github.com>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
Co-authored-by: crazywoola <427733928@qq.com>
 2024-08-13 17:36:20 +08:00
Bowen Liang ccb6ddd840
chore: bump Ruff to 0.5.7 (#7174) 2024-08-12 10:24:48 +08:00
Joe 425174e82f
feat: update ops trace (#7102) 2024-08-09 15:22:16 +08:00
Jyong 12095f8cd6
extract docx filter comment element (#7092) 2024-08-08 16:53:29 +08:00
Jyong 169cde6c3c
add nltk punkt resource (#7063) 2024-08-08 14:23:22 +08:00
8bitpd 40c6f3c724
fix: add redis lock to AnalyticdbVector init (#6859) 
Co-authored-by: xiaozeyu <xiaozeyu.xzy@alibaba-inc.com>
 2024-08-07 17:32:06 +08:00
-LAN- df8f8c9a2c
feat(api/core/rag/datasource/vdb/analyticdb/analyticdb_vector.py): Checking config before init analyticdb (#7050) 2024-08-07 17:31:36 +08:00
Jyong 80c94f02e9
add vector field for other vectordb (#7051) 2024-08-07 17:14:03 +08:00
chenxu9741 72c75b75cf
feat: Add hyperlink parsing to the DOCX document. (#7017) 2024-08-07 16:01:14 +08:00
Bryan eece50acec
fix: tran list issue (#7009) 
Co-authored-by: libing <libing@healink.cn>
 2024-08-06 21:01:38 +08:00
Jyong 28d4e5b045
Fix/reranking mode is null (#7012) 2024-08-06 19:12:04 +08:00
yanghx c53875ce8c
fix #6902 .docx handles images within tables and handles cross-column tables (#6951) 2024-08-06 17:14:24 +08:00
Joe f97a51ce24
fix: reranking disable timer error (#6910) 2024-08-02 16:34:50 +08:00
Bowen Liang 6aa02f8c63
dep: bump pgvecto-rs client from 0.1.x to 0.2.x (#6891) 2024-08-02 15:51:23 +08:00
Jyong 44801df8f8
fix score threshold limit be None (#6900) 2024-08-02 12:10:51 +08:00
Jyong 8157fccf6d
delete weight_type (#6865) 2024-08-01 13:02:33 +08:00
-LAN- a98284b1ef
refactor(api): Switch to `dify_config` (#6750) 
Signed-off-by: -LAN- <laipz8200@outlook.com>
 2024-07-30 11:15:26 +08:00
Bowen Liang b414ea41d6
dep: initial support for Milvus 2.4.x (#6084) 2024-07-29 19:56:45 +08:00
Jyong cf258b7a67
add xlsx support hyperlink extract (#6722) 2024-07-26 19:26:52 +08:00
Yeuoly 79cb23e8ac
security/SSRF vulns (#6682) 2024-07-25 20:50:26 +08:00
灰灰 5e4ac11df3
fix: code block segmentation problem of markdown document (#6465) 2024-07-25 17:24:37 +08:00
tmuife 16b4f560cd
fix bugs(when using Oracle23ai as Vector DB) (#6658) 2024-07-25 17:07:14 +08:00
Jyong e4bb943fe5
Feat/delete single dataset retrival (#6570) 2024-07-24 12:50:11 +08:00
崔亮 0569c547ee
fix the issue of MILVUS_DATABASE has no effect. (#6424) 2024-07-23 15:03:55 +08:00
tmuife 06fc1bce9e
Add search by full text when using Oracle23ai as vector DB (#6559) 2024-07-23 15:03:21 +08:00
Sangmin Ahn 093b8ca475
fix: escape double quotation marks in the vector DB search query (#6506) 2024-07-23 15:02:25 +08:00
-LAN- cd7fa8027a
fix(api/core/model_manager.py): Avoid mutation during iteration. (#6536) 2024-07-22 22:58:22 +08:00
-LAN- 5e6fc58db3
Feat/environment variables in workflow (#6515) 
Co-authored-by: JzoNg <jzongcode@gmail.com>
 2024-07-22 15:29:39 +08:00
Jyong f38034e455
clean vector collection redis cache (#6494) 2024-07-21 15:09:09 +08:00
Waffle 2ba05b041f
refactor(myscale):Set the default value of the myscale vector db in DifyConfig. (#6441) 2024-07-19 10:57:45 +08:00
leoterry dc847ba145
Fix the vector retrieval sorting issue (#6431) 
Co-authored-by: weifj <“weifj@tuyuansu.com.cn”>
 2024-07-18 19:25:41 +08:00
Poorandy c8f5dfcf17
refactor(rag): switch to dify_config. (#6410) 
Co-authored-by: -LAN- <laipz8200@outlook.com>
 2024-07-18 18:40:36 +08:00
Jyong 0de224b153
fix wrong using of RetrievalMethod Enum (#6345) 2024-07-16 19:09:04 +08:00
tangyoha 0cbbaf3f68
fix: markdown proc will remove image (#5855) 2024-07-12 20:07:22 +08:00
Matri a9ee52f2d7
Fix/firecrawl parameters issue (#6213) 2024-07-12 12:59:50 +08:00
Waffle 7b225a5ab0
refactor(services/tasks): Swtich to dify_config witch Pydantic (#6203) 2024-07-12 12:25:38 +08:00
Zhuo Qiu 63e34e5227
feat: support MyScale vector database (#6092) 2024-07-11 15:21:59 +08:00
8bitpd 7c70eb87bc
feat: support AnalyticDB vector store (#5586) 
Co-authored-by: xiaozeyu <xiaozeyu.xzy@alibaba-inc.com>
 2024-07-09 13:32:04 +08:00
Aurelius Huang f546db5437
fix: document truncation and loss in notion document sync (#5631) 
Co-authored-by: Aurelius Huang <cm.huang@aftership.com>
 2024-07-05 11:48:17 +08:00
Joe e8b8f6c6dd
Feat/fix ops trace (#5672) 
Co-authored-by: takatost <takatost@gmail.com>
 2024-06-28 00:24:37 +08:00
Bowen Liang dcb72e0067
chore: apply flake8-comprehensions Ruff rules to improve collection comprehensions (#5652) 
Co-authored-by: -LAN- <laipz8200@outlook.com>
 2024-06-27 11:21:31 +08:00
Joe 4e2de638af
feat: add ops trace (#5483) 
Co-authored-by: takatost <takatost@gmail.com>
 2024-06-26 17:33:29 +08:00
Jyong 43335b5c87
delete the deprecated method (#5612) 2024-06-26 12:51:50 +08:00
Xiaoguang Sun 13fcd7a901
feat: Add program_name attribute to TiDB connection (#5499) 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@gmail.com>
 2024-06-24 14:41:07 +08:00
tmuife 6a09409ec9
Add Oracle23ai as a vector datasource (#5342) 
Co-authored-by: walter from vm <walter.jin@oracle.com>
 2024-06-22 01:48:07 +08:00
Bowen Liang 39c14ec7c1
improve: unify Excel files parsing in either xls or xlsx file format by Pandas (#4965) 2024-06-20 16:14:49 +08:00
Bowen Liang aed56b1a8f
fix: Revert "feat: initial support for Milvus 2.4.x (#3795)" downgrading to 2.3.x for Linux arm64 installation failure (#5414) 2024-06-20 11:18:05 +08:00
quicksand 0105129fa8
fix bug: tencent vdb #5378 (#5408) 2024-06-20 10:37:39 +08:00
Bowen Liang c923684edd
chore: extract retrival method literal values into enum (#5060) 2024-06-19 16:05:27 +08:00
Bowen Liang bb33ffc332
feat: initial support for Milvus 2.4.x (#3795) 2024-06-19 13:55:44 +08:00
baojingyu d160d1ed02
feat: support opensearch approximate k-NN (#5322) 2024-06-19 12:44:33 +08:00
sino be3512aa57
fix: unable to reindex documents (#5276) 2024-06-17 17:19:43 +08:00
takatost 12c815c597
fix: ExtractSetting optional value missing None as default val (#5238) 2024-06-15 02:58:47 +08:00
Jyong ba5f8afaa8
Feat/firecrawl data source (#5232) 
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
Co-authored-by: chenhe <guchenhe@gmail.com>
Co-authored-by: takatost <takatost@gmail.com>
 2024-06-15 02:46:02 +08:00
quicksand 4080f7b8ad
feat: support tencent vector db (#3568) 2024-06-14 19:25:17 +08:00
saga.rey edf2047f04
fix: milvus_vector default dataset index_struct type from weaviate to milvus (#5098) 2024-06-14 02:36:01 +08:00
Bowen Liang f976740b57
improve: mordernizing validation by migrating pydantic from 1.x to 2.x (#4592) 2024-06-14 01:05:37 +08:00
Bowen Liang cdc08a434f
feat: support Chroma vector store (#5015) 2024-06-13 18:02:18 +08:00
Ikko Eltociear Ashimine a40f68cf94
chore: update qdrant_vector.py (#5128) 2024-06-13 15:35:14 +08:00
Bowen Liang bdad993901
improve: generalize vector factory classes and vector type (#5033) 2024-06-08 22:29:24 +08:00
Jyong 3b60c28b3a
deal the external image when extract docx image (#5024) 2024-06-07 20:00:39 +08:00
Jyong 237b8fe3d9
add meta.doc_id index for tidb (#4963) 2024-06-05 20:45:43 +08:00
Jyong 02e4de5166
fix some tidb bugs (#4960) 2024-06-05 19:14:18 +08:00
Weaxs 0797f9bc05
feat: support tidb vector (#4588) 2024-06-05 18:19:53 +08:00
YC 9f8ca75a81
fixing a bug of handling header row when parsing xls file, and tune xls/xlsx parsing result to be more structured (#3600) 2024-06-05 15:28:43 +08:00
Bowen Liang f32b440c4a
chore: fix indention violations by applying E111 to E117 ruff rules (#4925) 2024-06-05 14:05:15 +08:00
JasonVV 7749b71fff
Optimize knowledge retrieval performance by batching dataset quries. (#4917) 2024-06-05 13:30:32 +08:00
takatost d1dbbc1e33
feat: backend model load balancing support (#4927) 2024-06-05 00:13:04 +08:00
Jyong b6631cd878
modify rerank and splitter code directory (#4924) 2024-06-04 20:07:25 +08:00
Bowen Liang 58db719a2c
dep: bump pandas from 1.x to 2.x (#4820) 2024-06-04 13:24:28 +08:00
Oliver Lee 176d91937d
fix 'NoneType' and new ContentType supported. (#4818) 2024-05-31 14:19:33 +08:00
Jyong 1b2d862973
add error msg for hit test (#4704) 2024-05-28 14:54:53 +08:00
yalei 026175c8f7
feat: update notion extractor (#3898) 
Co-authored-by: duyalei <>
 2024-05-24 20:30:48 +08:00
Jyong 24624491cd
add qdrant metadata.doc_id index when create qdrant collection (#4570) 2024-05-23 18:11:01 +08:00
Jyong 233c4150d1
support images and tables extract from docx (#4619) 2024-05-23 18:05:23 +08:00
Rain Chen c255a20d7c
allow to config max segmentation tokens length for RAG document using environment variable (#4375) 2024-05-20 13:20:27 +08:00
majian b5204111da
Add UNSTRUCTURED_API_KEY env support (#4369) 2024-05-20 13:14:17 +08:00
Bowen Liang 04ad46dd31
chore: skip unnecessary key checks prior to accessing a dictionary (#4497) 2024-05-19 18:30:45 +08:00
Charlie.Wei 97b65f9b4b
Optimize webscraper (#4392) 
Co-authored-by: luowei <glpat-EjySCyNjWiLqAED-YmwM>
Co-authored-by: crazywoola <427733928@qq.com>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
 2024-05-15 15:23:16 +08:00
kerlion 98140ae5d9
fix the issue of MILVUS_DATABASE has no effect. (#4353) 2024-05-14 19:54:31 +08:00
LiuVaayne 875249eb00
Feat/vector db pgvector (#3879) 2024-05-10 17:20:30 +08:00
Bowen Liang 142814d451
chore: skip deprecated field_schema param in creating payload index on Qdrant (#3903) 2024-04-30 16:16:10 +08:00
Jyong 3e9dbe3e0a
add pgvecto_rs support and upgrade SQLAlchemy (#3833) 2024-04-29 11:58:17 +08:00
呆萌闷油瓶 0940f01634
enhancement:support Qdrant gRPC mode (#3929) 2024-04-28 15:33:32 +08:00
Bowen Liang 045827043d
test: improve vector store tests (#3855) 2024-04-26 19:18:42 +08:00
Bowen Liang 7919596a21
fix: UP031 style rule violation (#3866) 2024-04-26 11:24:08 +08:00
Bowen Liang 45dd1683fd
test: add tests covering all methods of vector store (#3849) 2024-04-25 22:27:30 +08:00
Jingpan Xiong 1be222af2e
fix: using api can not execute relyt vector database (#3766) 
Co-authored-by: jingsi <jingsi@leadincloud.com>
 2024-04-25 19:46:20 +08:00
呆萌闷油瓶 78988ed60e
fix:still enable SSL verification when using qdrant based on HTTP protocol (#3805) 2024-04-25 13:04:31 +08:00
Bowen Liang 9cec8c1750
test: add unit tests for vector stores of Milvus, Qdrant and Weaviate (#3688) 2024-04-24 21:52:42 +08:00
Jyong f257f2c396
Knowledge optimization (#3755) 
Co-authored-by: crazywoola <427733928@qq.com>
Co-authored-by: JzoNg <jzongcode@gmail.com>
 2024-04-24 15:02:29 +08:00
Bowen Liang 2867d29021
fix: milvus usage with create_collection (#3683) 2024-04-23 17:37:40 +08:00
Jingpan Xiong 33397836a5
feat: support relyt vector database (#3367) 
Co-authored-by: jingsi <jingsi@leadincloud.com>
 2024-04-15 11:52:34 +08:00
Jyong 0737e930cb
chore: remove Langchain tools import (#3407) 2024-04-12 16:26:09 +08:00
chenxu9741 ad65c891e7
add xls file suport (#3321) 2024-04-12 14:53:44 +08:00
LiuVaayne b00466f025
feat:api Add support for extracting EPUB files in ExtractProcessor (#3254) 
Co-authored-by: crazywoola <427733928@qq.com>
 2024-04-12 11:25:02 +08:00
Jyong 1f302990c6
add segment with keyword issue (#3351) 
Co-authored-by: StyleZhang <jasonapring2015@outlook.com>
 2024-04-11 16:57:02 +08:00
Jyong 6164604462
fix dataset retrival in dataset mode (#3334) 2024-04-11 02:11:21 +08:00
Jyong b6de97ad53
Remove langchain dataset retrival agent logic (#3311) 2024-04-10 20:37:22 +08:00
Jyong 8fcf459285
fix milvus database name parameter missed (#3229) 2024-04-09 15:54:13 +08:00
Leo Q 9c01bcb3e5
feat: support setting database used in Milvus (#3003) 2024-04-09 15:39:36 +08:00
Jyong 283979fc46
fix keyword index error when storage source is S3 (#3182) 2024-04-09 01:42:58 +08:00