Commit Graph

316 Commits

Author SHA1 Message Date
twwu
386614951f Merge branch 'main' into feat/rag-2 2025-08-18 11:16:18 +08:00
-LAN-
e340fccafb
feat: integrate flask-orjson for improved JSON serialization performance (#23935) 2025-08-14 19:50:59 +08:00
jyong
22b3933cc3 Merge branch 'main' into feat/rag-2
# Conflicts:
#	api/core/workflow/entities/variable_pool.py
2025-08-12 11:13:04 +08:00
engchina
7566d90dfe
fix issue #23758 (#23764)
Co-authored-by: root <root@thinkpad-pc.localdomain>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2025-08-12 10:26:13 +08:00
twwu
fc779d00df Merge branch 'main' into feat/rag-2 2025-08-11 11:15:58 +08:00
yunqiqiliang
14e1c16cf2
Fix ClickZetta stability and reduce logging noise (#23632)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2025-08-08 22:57:47 +08:00
yunqiqiliang
62772e8871
fix: ensure vector database cleanup on dataset deletion regardless of document presence (affects all 33 vector databases) (#23574)
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2025-08-08 09:18:43 +08:00
Qiang Lee
e9045a8838
Fix: Apply Metadata Filters Correctly in Full-Text Search Mode for Tencent Cloud Vector Database (#23564) 2025-08-07 05:36:06 -07:00
twwu
1b3860d012 Merge branch 'main' into feat/rag-2 2025-08-07 16:27:20 +08:00
yunqiqiliang
e01510e2a6
feat: Add Clickzetta Lakehouse vector database integration (#22551)
Co-authored-by: Claude <noreply@anthropic.com>
2025-08-07 14:21:46 +08:00
zxhlyh
201e4cd64d merge main 2025-08-05 10:30:53 +08:00
Yongtao Huang
406c1952b8
Fix version comparison with imported_version (#23326)
Signed-off-by: Yongtao Huang <yongtaoh2022@gmail.com>
2025-08-04 10:40:49 +08:00
wanttobeamaster
da5c003f97
chore: tablestore full text search support score normalization (#23255)
Co-authored-by: xiaozhiqing.xzq <xiaozhiqing.xzq@alibaba-inc.com>
2025-08-01 14:14:11 +08:00
jyong
a7a4c8228e Merge branch 'main' into feat/rag-2
# Conflicts:
#	web/app/components/workflow/hooks/use-workflow.ts
2025-07-31 10:30:28 +08:00
rhochman
eee576355b
Fix: Support for Elasticsearch Cloud Connector (#23017)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-30 11:12:16 +08:00
zxhlyh
a95cf6f8b0 merge main 2025-07-28 16:00:38 +08:00
Asuka Minato
a189d293f8
make logging not use f-str, change others to f-str (#22882) 2025-07-25 10:32:48 +08:00
twwu
bae2af0c85 Merge branch main into feat/rag-2 2025-07-24 17:40:04 +08:00
Asuka Minato
ef51678c73
orm filter -> where (#22801)
Signed-off-by: -LAN- <laipz8200@outlook.com>
Co-authored-by: -LAN- <laipz8200@outlook.com>
Co-authored-by: Claude <noreply@anthropic.com>
2025-07-24 00:57:45 +08:00
wanttobeamaster
8278b39f85
fix tablestore full text search bug (#22853) 2025-07-23 19:31:47 +08:00
wanttobeamaster
1c3c40db69
fix: tablestore TypeError when vector is missing (#22843)
Co-authored-by: xiaozhiqing.xzq <xiaozhiqing.xzq@alibaba-inc.com>
2025-07-23 18:59:16 +08:00
wlleiiwang
b4e152f775
FEAT: Tencent Vector search supports backward compatibility with the previous score calculation approach. (#22820)
Co-authored-by: wlleiiwang <wlleiiwang@tencent.com>
2025-07-23 15:38:31 +08:00
Asuka Minato
6d3e198c3c
Mapped column (#22644)
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-23 00:39:59 +08:00
wanttobeamaster
a2048fd0f4
fix: tablestore vdb support metadata filter (#22774)
Co-authored-by: xiaozhiqing.xzq <xiaozhiqing.xzq@alibaba-inc.com>
2025-07-22 16:48:59 +08:00
zxhlyh
3b8d96f45c merge main 2025-07-22 13:52:24 +08:00
issac2e
58d92970a9
Optimize tencent_vector knowledge base deletion error handling with batch processing support (#22726)
Co-authored-by: liuchen15 <liuchen15@gaotu.cn>
Co-authored-by: crazywoola <427733928@qq.com>
2025-07-22 08:21:41 +08:00
zxhlyh
4d36e784b7 merge main 2025-07-21 17:45:26 +08:00
uply23333
ab012fe1a2
fix: improve document filtering in full text search(elasticsearch) (#22683) 2025-07-21 15:59:37 +08:00
8bitpd
9251a66a10
fix: update analyticdb vector to do filter by metadata (#22698)
Co-authored-by: xiaozeyu <xiaozeyu.xzy@alibaba-inc.com>
2025-07-21 15:03:37 +08:00
zxhlyh
01566035e3 merge main 2025-07-17 16:48:43 +08:00
yihong
d2933c2bfe
fix: drop dead code phase2 unused class (#22042)
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2025-07-17 09:33:07 +08:00
wanttobeamaster
bf7b2c339b
tablestore vector support more method (#22225)
Co-authored-by: xiaozhiqing.xzq <xiaozhiqing.xzq@alibaba-inc.com>
2025-07-15 09:58:48 +08:00
Jacky Wu
3e96c0c468
fix: close session before doing long latency operation (#22306) 2025-07-14 15:16:10 +08:00
twwu
682b65034c Merge branch 'main' into feat/rag-2 2025-07-14 11:17:42 +08:00
luckylhb90
a371390d6c
optimize: batch embedding and qdrant write_consistency_factor parameter (#21776)
Co-authored-by: hobo.l <hobo.l@binance.com>
2025-07-10 10:16:59 +08:00
twwu
e2a141b3bb Merge branch 'main' into feat/rag-2 2025-07-10 10:14:12 +08:00
wlleiiwang
89b52471fb
Optimize the memory usage of Tencent Vector Database (#22079)
Co-authored-by: wlleiiwang <wlleiiwang@tencent.com>
2025-07-09 15:53:06 +08:00
jyong
f7fbded8b9 Merge branch 'main' into feat/r2 2025-06-19 13:32:07 +08:00
LiuBo
17fe62cf91
feat: add support for Matrixone database (#20714) 2025-06-19 10:20:12 +08:00
NeatGuyCoding
9835730278
Translation fix (#21194) 2025-06-19 09:36:56 +08:00
jyong
f7a4e5d1a6 Merge branch 'main' into feat/r2 2025-06-18 10:57:44 +08:00
kurokobo
4689e8953e
fix: shorten connection timeout to pypi.org for deprecation check for weaviate client (#21131) 2025-06-18 09:25:52 +08:00
jyong
b277acc298 Merge branch 'main' into feat/r2 2025-06-16 14:08:02 +08:00
Bowen Liang
366ddb05ae
test: run vdb test of oceanbase with docker compose in CI tests (#20945) 2025-06-16 11:05:19 +08:00
jyong
1d8b390584 Merge branch 'main' into feat/r2
# Conflicts:
#	docker/docker-compose.middleware.yaml
2025-06-11 17:16:27 +08:00
yihong
65c7c01d90
fix: clean up two unreachable code (#20773)
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2025-06-07 23:06:46 +08:00
jefferyvvv
37c3283450
fix: opensearch vector search falls back to keyword search (#20723)
Co-authored-by: wenjun.gu <wenjun.gu@envision-energy.com>
2025-06-06 16:29:15 +08:00
jefferyvvv
4271602cfc
fix: opensearch metadata filtering returns empty (#20701)
Co-authored-by: wenjun.gu <wenjun.gu@envision-energy.com>
Co-authored-by: crazywoola <427733928@qq.com>
2025-06-06 09:10:01 +08:00
jefferyvvv
138ad6e8b3
fix: opensearch fulltext search with metadata filtering dsl error (#20702)
Co-authored-by: wenjun.gu <wenjun.gu@envision-energy.com>
2025-06-05 23:09:00 +08:00
kenwoodjw
01d500db14
fix: autocorrect everything in web (#20605)
Signed-off-by: kenwoodjw <blackxin55+@gmail.com>
2025-06-04 14:12:24 +08:00
jyong
9cdd2cbb27 r2 2025-06-03 19:02:57 +08:00
jyong
309fffd1e4 Merge branch 'main' into feat/r2
# Conflicts:
#	api/core/repositories/sqlalchemy_workflow_node_execution_repository.py
#	api/core/workflow/entities/node_entities.py
#	api/core/workflow/enums.py
2025-06-03 18:56:49 +08:00
jyong
4130c50643 r2 2025-06-03 18:32:39 +08:00
zhaobingshuang
3f7aa38d77
fix: #20560 When elasticsearch is used as the vector database, the Retrieval Test fails to filter the data after setting the Score Threshold, and the score of the recalled results is empty (#20561) 2025-06-03 13:24:26 +08:00
Cheney Zhang
b4b59148dc
check zilliz cloud of full-text search (#20519) 2025-06-02 18:04:13 +08:00
yihong
5a991295e0
fix: drop some type fixme (#20344) 2025-05-30 14:10:09 +08:00
rouxiaomin
4c4887c5fc
feat(qdrant):add replication_factor when create collection in qdrant (#20133)
Co-authored-by: 刘敏 <min.liu@tongdun.net>
2025-05-27 14:46:04 +08:00
He Huang
6f48af2610
Refactor OpenSearch config to separate use_ssl and verify_certs flags (#20075)
Co-authored-by: he.huang <he.huang1@outlook.com>
Co-authored-by: crazywoola <427733928@qq.com>
2025-05-22 10:14:38 +08:00
wlleiiwang
7d230acf40
tencent vectordb compatible with version 1.1.3 and below (#20056)
Co-authored-by: wlleiiwang <wlleiiwang@tencent.com>
2025-05-21 20:24:05 +08:00
-LAN-
3196dc2d61
refactor: Use typed SQLAlchemy base model and fix type errors (#19980)
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-05-21 15:38:03 +08:00
k-kayashima
b292990075
Fix: Ensure unique index names for pgvector knowledge tables (#19672)
Co-authored-by: crazywoola <427733928@qq.com>
2025-05-15 11:43:44 +08:00
Will
bfa652f2d0
fix: metadata filtering condition variable unassigned; fix External K… (#19208) 2025-05-07 14:52:09 +08:00
QuantumGhost
bd1bbfee4b
Enhance Code Consistency Across Repository with .editorconfig (#19023) 2025-04-29 18:04:33 +08:00
Ahmad Zidan
8266815cda
feat: add AWS Managed IAM auth for OpenSearch vector DB (#18963) 2025-04-29 15:10:08 +08:00
Ethan
8b4ea01810
feat: support access milvus with token (#19034) 2025-04-29 14:52:13 +08:00
Jiang
37e2f73909
[Lindorm VDB] Add the QUERY_TIMEOUT parameter to force the search query to fail. (#18613)
Co-authored-by: jiangzhijie <jiangzhijie.jzj@alibaba-inc.com>
2025-04-25 09:42:58 +08:00
王晓阳
0babdffe3e
feat: support vastbase vector database (#16308) 2025-04-24 18:04:57 +08:00
lauding
eb1ce3dd6b
feat: support huawei cloud vector database (#16141) 2025-04-22 13:03:35 +08:00
tmuife
7b6523e54d
Update Oracle db connection library and change connection pool to single connection (#18466) 2025-04-21 17:56:57 +08:00
Rain Wang
83f1aeec1d
Fix ORDER BY (score, id) error in api/core/rag/datasource/vdb/analyticdb/analyticdb_vector_sql.py line 249 (#18252) 2025-04-17 14:15:05 +08:00
Rain Wang
e8d98e3d89
Add analyzer_params config for milvus vectordb (#18180) 2025-04-17 10:38:56 +08:00
YANG
d119c7d629
ignore errors when creating duplicate indexes (#18069)
Co-authored-by: 璟义 <yangshangpo.ysp@alibaba-inc.com>
2025-04-15 15:48:16 +08:00
Jasonfish
1f722cde22
fix(api): Some params were ignored when creating empty Datasets through API (#17932) 2025-04-14 10:24:01 +08:00
Yongtao Huang
5d72003ebb
Remove dead code (#17899) 2025-04-11 20:33:52 +08:00
yihong
f04d52c044
fix: autocorrect everything in api (#17859)
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2025-04-11 15:24:39 +08:00
wlleiiwang
9d20561af4
create db if not exists (#17796)
Co-authored-by: wlleiiwang <wlleiiwang@tencent.com>
2025-04-10 18:03:22 +08:00
Jyong
8b3be4224d
revert batch query (#17707) 2025-04-09 20:25:36 +08:00
wlleiiwang
f148f1efa2
fix: Check collection exists before drop it. (#17692)
Co-authored-by: wlleiiwang <wlleiiwang@tencent.com>
2025-04-09 19:14:32 +08:00
Jyong
abfcd9d3b6
fix segment query index not effect (#17704) 2025-04-09 19:09:08 +08:00
Han
f1e4d5ed6c
Fix Performance Issues: (#17083)
Co-authored-by: Wang Han <wanghan@zhejianglab.org>
2025-04-09 11:22:53 +08:00
wlleiiwang
42a42a7962
FEAT: support Tencent vectordb to full text search (#16865)
Co-authored-by: wlleiiwang <wlleiiwang@tencent.com>
2025-04-07 09:50:03 +08:00
Jiang
fd1e40d22e
Lindorm VDB bugfix (#17357)
Co-authored-by: jiangzhijie <jiangzhijie.jzj@alibaba-inc.com>
2025-04-02 21:31:59 +08:00
Jiang
ff388fe3e6
optimize lindorm vdb add_texts (#17212)
Co-authored-by: jiangzhijie <jiangzhijie.jzj@alibaba-inc.com>
2025-04-01 11:06:35 +08:00
非法操作
44f911a0a8
chore: docstring not match the function parameter (#17162) 2025-03-31 13:19:15 +08:00
wanttobeamaster
7f70cadacb
feat: support Tablestore vector database (#16601)
Co-authored-by: xiaozhiqing.xzq <xiaozhiqing.xzq@alibaba-inc.com>
2025-03-27 15:53:33 +08:00
wlleiiwang
a743d5dc71
feat: tencent vectordb: use grpc client and set upsert batch size (#16016)
Co-authored-by: wlleiiwang <wlleiiwang@tencent.com>
2025-03-27 12:20:16 +08:00
yourchanges
59a86dabee
fix: fix missing oceanbase config enable_hybrid_search init (#16852)
Co-authored-by: 李远军 <4842@9ji.com>
2025-03-26 21:15:54 +08:00
Jyong
2174225259
fix milvus filter search (#16725) 2025-03-25 16:22:43 +08:00
hsiong
6157f57872
feat: Add OceanBase hybrid search features (#16652)
Co-authored-by: 李远军 <4842@9ji.com>
Co-authored-by: yourchanges <yourchanges@gmail.com>
2025-03-25 14:32:00 +08:00
kenwoodjw
a113356695
fix: pgvector metadata filter (#16688)
Signed-off-by: kenwoodjw <blackxin55+@gmail.com>
2025-03-25 11:34:33 +08:00
Jiang
fc8c765215
Fix/vdb lindorm (#16660)
Co-authored-by: jiangzhijie <jiangzhijie.jzj@alibaba-inc.com>
2025-03-25 09:19:06 +08:00
Jyong
1be0d26c1f
fix metadata filter not affect in keyword-search and fulltext-search (#16644) 2025-03-24 18:35:16 +08:00
chenhuan0728
770c461a8f
feat: add openGauss PQ acceleration feature (#16432)
Co-authored-by: chenhuan <huan.chen0728@foxmail>
2025-03-24 15:16:40 +08:00
Jyong
d135677c25
add vdb document id index (#16244)
Co-authored-by: crazywoola <427733928@qq.com>
2025-03-20 01:38:15 +08:00
Jyong
a8879057c0
fix tidb metadata filter (#16237) 2025-03-19 19:44:56 +08:00
Jyong
81325df368
fix weaviate metadata filter (#16230) 2025-03-19 18:26:53 +08:00
Jyong
c3c957bb80
change recreate_collection function to create_collection (#16212) 2025-03-19 17:13:08 +08:00
Jyong
abeaea4f79
Support knowledge metadata filter (#15982) 2025-03-18 16:42:19 +08:00
Jyong
33ba7e659b
fix vector db sql injection (#16096) 2025-03-18 15:07:29 +08:00
LittleFish-15
223ab5a38f
feat: support openGauss vector database (#15865) 2025-03-17 19:42:54 +08:00