559ab46ee1
fix: Removes redundant token calculations and updates dependencies
...
Eliminates unnecessary pre-calculation of token limits and recalculation of max tokens
across multiple app runners, simplifying the logic for prompt handling.
Updates tiktoken library from version 0.8.0 to 0.9.0 for improved tokenization performance.
Increases default token limit in TokenBufferMemory to accommodate larger prompt messages.
These changes streamline the token management process and leverage the latest
improvements in the tiktoken library.
Fixes potential token overflow issues and prepares the system for handling larger
inputs more efficiently.
Relates to internal optimization tasks.
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-04-28 15:39:12 +08:00
d3895bcd6b
revert
2025-02-19 14:32:28 +08:00
eeb390650b
fix: build failed
2025-02-19 14:32:28 +08:00
87763fc234
feat(model_providers): Support deepseek for Azure AI Foundry ( #13267 )
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-02-06 12:45:48 +08:00
7203991032
feat: add parameter "reasoning_effort" and Openai o3-mini ( #13243 )
2025-02-06 09:29:48 +08:00
03243cb422
Modify params for bedrock retrieve generate ( #13182 )
2025-02-05 12:17:42 +08:00
d4c916b496
chore(pyproject): Add type stubs into `pyproject.toml` ( #13145 )
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-02-04 12:01:28 +08:00
b2bbc28580
support bedrock kb: retrieve and generate ( #13027 )
2025-01-25 17:28:06 +08:00
c62b7cc679
chore(build): bump poetry from 1.x to 2.x ( #12369 )
2025-01-22 13:38:24 +08:00
166221d784
chore(lint): fix quotes for f-string formatting by bumping ruff to 0.9.x ( #12702 )
2025-01-21 10:12:29 +08:00
69d58fbb50
Add new integration with Opik Tracking tool ( #11501 )
2025-01-13 17:41:44 +08:00
040a3b782c
FEAT: support milvus to full text search ( #11430 )
...
Signed-off-by: YoungLH <974840768@qq.com>
2025-01-08 17:39:53 +08:00
baeddd4d15
feat:Add support for stop parameter in hunyuan model #12313 ( #12315 )
...
Co-authored-by: xander-art <xander-art@gmail.com>
2025-01-03 09:15:04 +08:00
e7dffcd0f6
chore(deps): bump aiohttp from 3.10.5 to 3.10.11 in /api ( #12303 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-01-02 16:42:47 +08:00
15f43dd326
chore(deps): update yarl version from 1.9.4 to 1.18.3 ( #12302 )
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2025-01-02 16:26:47 +08:00
68757950ce
chore(deps): bump jinja2 from 3.1.4 to 3.1.5 in /api ( #12300 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-01-02 16:04:10 +08:00
63a0b8ba79
feat: integrate psycogreen for gevent compatibility in PostgreSQL ( #12253 )
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2024-12-31 14:45:59 +08:00
db2aa83a7c
Revert "Feat/parent child retrieval" ( #12095 )
2024-12-25 20:55:44 +08:00
2b2263a349
Feat/parent child retrieval ( #12086 )
...
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Signed-off-by: -LAN- <laipz8200@outlook.com>
Co-authored-by: AkaraChen <akarachen@outlook.com>
Co-authored-by: nite-knite <nkCoding@gmail.com>
Co-authored-by: Joel <iamjoel007@gmail.com>
Co-authored-by: Warren Chen <warren.chen830@gmail.com>
Co-authored-by: crazywoola <427733928@qq.com>
Co-authored-by: Yi Xiao <54782454+YIXIAO0@users.noreply.github.com>
Co-authored-by: yihong <zouzou0208@gmail.com>
Co-authored-by: -LAN- <laipz8200@outlook.com>
Co-authored-by: KVOJJJin <jzongcode@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: JzoNgKVO <27049666+JzoNgKVO@users.noreply.github.com>
Co-authored-by: Charlie.Wei <luowei@cvte.com>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
Co-authored-by: huayaoyue6 <huayaoyue@163.com>
Co-authored-by: kurokobo <kuro664@gmail.com>
Co-authored-by: Matsuda <yiyth.fcb6@gmail.com>
Co-authored-by: shirochan <s.yusuke0711@gmail.com>
Co-authored-by: Jyong <76649700+JohnJyong@users.noreply.github.com>
Co-authored-by: Huỳnh Gia Bôi <boihuynh147@gmail.com>
Co-authored-by: Julian Huynh <julian.huynh@immersio.io>
Co-authored-by: Hash Brown <hi@xzd.me>
Co-authored-by: 非法操作 <hjlarry@163.com>
Co-authored-by: Kazuki Takamatsu <kazuki.takamatsu@chowagiken.co.jp>
Co-authored-by: Trey Dong <1346650911@qq.com>
Co-authored-by: VoidIsVoid <343750470@qq.com>
Co-authored-by: Gimling <huangjl@ruyi.ai>
Co-authored-by: xiandan-erizo <xiandan.erizo@gmail.com>
Co-authored-by: Muneyuki Noguchi <nogu.dev@gmail.com>
Co-authored-by: zhaobingshuang <1475195565@qq.com>
Co-authored-by: zhaobs <zhaobs@cailian.net>
Co-authored-by: suzuki.sh <s2terminal@users.noreply.github.com>
Co-authored-by: Yingchun Lai <laiyingchun@apache.org>
Co-authored-by: huanshare <huanshare@live.com>
Co-authored-by: huanshare <liuhuan101@longfor.com>
Co-authored-by: orangeclk <orangeclk@users.noreply.github.com>
Co-authored-by: 문정현 <120004247+JungHyunMoon@users.noreply.github.com>
Co-authored-by: barabicu <kztk533@gmail.com>
Co-authored-by: Wei Mingzhi <whistler_wmz@users.sf.net>
Co-authored-by: Paul van Oorschot <20116814+pvoo@users.noreply.github.com>
Co-authored-by: zkyTech <zhangkunyuan@hotmail.com>
Co-authored-by: zhangkunyuan <zhangkunyuan@cmhi.chinamobile.com>
Co-authored-by: Tommy <34446820+Asterovim@users.noreply.github.com>
Co-authored-by: zxhlyh <jasonapring2015@outlook.com>
Co-authored-by: Novice <857526207@qq.com>
Co-authored-by: Novice Lee <novicelee@NovicedeMacBook-Pro.local>
Co-authored-by: Novice Lee <novicelee@NoviPro.local>
Co-authored-by: zxhlyh <16177003+zxhlyh@users.noreply.github.com>
Co-authored-by: liuzhenghua <1090179900@qq.com>
Co-authored-by: Jiang <65766008+AlwaysBluer@users.noreply.github.com>
Co-authored-by: jiangzhijie <jiangzhijie.jzj@alibaba-inc.com>
Co-authored-by: Joe <79627742+ZhouhaoJiang@users.noreply.github.com>
Co-authored-by: Alok Shrivastwa <alok.shrivastwa@gmail.com>
Co-authored-by: Alok Shrivastwa <Alok.Shrivastwa@microland.com>
Co-authored-by: JasonVV <jasonwangiii@outlook.com>
Co-authored-by: Hiroshi Fujita <fujita-h@users.noreply.github.com>
Co-authored-by: Kevin9703 <51311316+Kevin9703@users.noreply.github.com>
Co-authored-by: NFish <douxc512@gmail.com>
Co-authored-by: Junyan Qin <1010553892@qq.com>
Co-authored-by: IWAI, Masaharu <iwaim.sub@gmail.com>
Co-authored-by: IWAI, Masaharu <iwai_masaharu@funkit.co.jp>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: luckylhb90 <luckylhb90@gmail.com>
Co-authored-by: hobo.l <hobo.l@binance.com>
Co-authored-by: douxc <7553076+douxc@users.noreply.github.com>
2024-12-25 18:17:15 +08:00
56e15d09a9
feat: mypy for all type check ( #10921 )
2024-12-24 18:38:51 +08:00
efa8eb379f
fix: memory leak by pypdfium2 close(maybe) #11510 ( #11700 )
...
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2024-12-17 00:42:01 +08:00
430ca3322b
chore(dependency): bump gunicorn to 23.0 ( #11560 )
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2024-12-13 16:16:58 +08:00
8d4bb9b40d
feat: integrate opendal storage ( #11508 )
...
Signed-off-by: -LAN- <laipz8200@outlook.com>
2024-12-11 14:50:54 +08:00
376726cf90
[feat] Add AWS Bedrock rerank ( #11349 )
...
Co-authored-by: crazywoola <427733928@qq.com>
2024-12-05 11:31:43 +08:00
a67b0d4771
chore(lint): extract ruff configs into .ruff.toml file keeping pyproject.toml clean ( #11222 )
2024-12-01 12:51:28 +08:00
2feb44e2c5
chore(dep): bump flask from 3.0.1 to 3.1.0 and flask-compress to 1.17 ( #11195 )
2024-11-29 09:28:53 +08:00
b791a80b75
chore: update chromadb version to 0.5.20 ( #11038 )
...
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2024-11-25 11:14:04 +08:00
6c8e208ef3
chore: bump minimum supported Python version to 3.11 ( #10386 )
2024-11-24 13:28:46 +08:00
d3051eed48
chore (dep): bump gevent from v23 to v24 for better support for Python 3.11 and 3.12 ( #10387 )
2024-11-23 00:07:07 +08:00
c5f7d650b5
feat: Allow using file variables directly in the LLM node and support more file types. ( #10679 )
...
Co-authored-by: Joel <iamjoel007@gmail.com>
2024-11-22 16:30:22 +08:00
90d6ebc879
Add youtube-transcript-api as tool ( #10772 )
...
Co-authored-by: crazywoola <427733928@qq.com>
2024-11-18 10:58:16 +08:00
365cb4b368
chore(lint): bump ruff from 0.6.9 to 0.7.3 ( #10714 )
2024-11-15 09:19:41 +08:00
ad16180b1a
feat(tool): fal ai wizper ASR built-in tool ( #10716 )
2024-11-15 09:01:07 +08:00
7c2a9b0744
celery worker log format following LOG_FORMAT env#9404 ( #10016 )
...
Signed-off-by: root <root@localhost.localdomain>
Co-authored-by: root <root@localhost.localdomain>
2024-11-08 17:12:09 +08:00
d45d90e8ae
chore: lazy import sagemaker ( #10342 )
2024-11-06 12:45:22 +08:00
acb22f0fde
Updates: Add mplfonts library for customizing matplotlib fonts and Va… ( #9903 )
2024-11-05 10:34:28 +08:00
b61baa87ec
fix: avoid unexpected error when create knowledge base with baidu vector database and wenxin embedding model ( #10130 )
2024-10-31 21:34:23 +08:00
878d13ef42
Added OceanBase as an option for the vector store in Dify ( #10010 )
2024-10-29 21:10:18 +08:00
c8ef9223e5
feat: couchbase integration ( #6165 )
...
Co-authored-by: crazywoola <427733928@qq.com>
Co-authored-by: Elliot Scribner <elliot.scribner@couchbase.com>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
2024-10-29 15:00:23 +08:00
caf7bc8569
upgrade nltk, unstructured and starlette ( #9860 )
2024-10-25 17:15:44 +08:00
121bb99cc2
downgrade unstructured nltk version ( #9726 )
2024-10-23 19:02:27 +08:00
cac1ef7ade
remove ppt import ( #9721 )
2024-10-23 18:22:30 +08:00
3e9d271b52
nltk security issue and upgrade unstructured ( #9558 )
2024-10-23 16:23:55 +08:00
8e7a752b2a
feat: add upstash as a new vector database provider ( #9644 )
2024-10-23 09:16:35 +08:00
495cf58014
dep: bump pydantic to 2.9 ( #9077 )
2024-10-21 23:32:09 +08:00
e61752bd3a
feat/enhance the multi-modal support ( #8818 )
2024-10-21 10:43:49 +08:00
b597a0d31c
fix: Azure OpenAI o1 max_completion_token and get_num_token_from_messages error ( #9326 )
...
Co-authored-by: wwwc <wwwc@outlook.com>
2024-10-15 16:26:44 +08:00
7838f9f3a3
fix: Add new Milvus Lite wheel for manylinux2014_aarch64 ( #9316 )
2024-10-14 16:27:26 +08:00
ffc3f33670
chore: remove the copied zhipu_ai sdk ( #9270 )
2024-10-14 10:53:45 +08:00
d15ba3939d
Add Volcengine VikingDB as new vector provider ( #9287 )
2024-10-13 21:26:05 +08:00
-LAN-