+ alt="follow on X(Twitter)">
@@ -196,10 +196,14 @@ If you'd like to configure a highly-available setup, there are community-contrib
#### Using Terraform for Deployment
+Deploy Dify to Cloud Platform with a single click using [terraform](https://www.terraform.io/)
+
##### Azure Global
-Deploy Dify to Azure with a single click using [terraform](https://www.terraform.io/).
- [Azure Terraform by @nikawang](https://github.com/nikawang/dify-azure-terraform)
+##### Google Cloud
+- [Google Cloud Terraform by @sotazum](https://github.com/DeNA/dify-google-cloud-terraform)
+
## Contributing
For those who'd like to contribute code, see our [Contribution Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md).
@@ -219,7 +223,7 @@ At the same time, please consider supporting Dify by sharing it on social media
* [Github Discussion](https://github.com/langgenius/dify/discussions). Best for: sharing feedback and asking questions.
* [GitHub Issues](https://github.com/langgenius/dify/issues). Best for: bugs you encounter using Dify.AI, and feature proposals. See our [Contribution Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md).
* [Discord](https://discord.gg/FngNHpbcY7). Best for: sharing your applications and hanging out with the community.
-* [Twitter](https://twitter.com/dify_ai). Best for: sharing your applications and hanging out with the community.
+* [X(Twitter)](https://twitter.com/dify_ai). Best for: sharing your applications and hanging out with the community.
## Star history
diff --git a/README_AR.md b/README_AR.md
index 10d572cc49..e46ba73738 100644
--- a/README_AR.md
+++ b/README_AR.md
@@ -17,7 +17,7 @@
alt="chat on Discord">
+ alt="follow on X(Twitter)">
@@ -179,10 +179,13 @@ docker compose up -d
#### استخدام Terraform للتوزيع
+انشر Dify إلى منصة السحابة بنقرة واحدة باستخدام [terraform](https://www.terraform.io/)
+
##### Azure Global
-استخدم [terraform](https://www.terraform.io/) لنشر Dify على Azure بنقرة واحدة.
- [Azure Terraform بواسطة @nikawang](https://github.com/nikawang/dify-azure-terraform)
+##### Google Cloud
+- [Google Cloud Terraform بواسطة @sotazum](https://github.com/DeNA/dify-google-cloud-terraform)
## المساهمة
diff --git a/README_CN.md b/README_CN.md
index 32551fcc31..4553524ce5 100644
--- a/README_CN.md
+++ b/README_CN.md
@@ -17,7 +17,7 @@
alt="chat on Discord">
+ alt="follow on X(Twitter)">
@@ -202,10 +202,14 @@ docker compose up -d
#### 使用 Terraform 部署
+使用 [terraform](https://www.terraform.io/) 一键将 Dify 部署到云平台
+
##### Azure Global
-使用 [terraform](https://www.terraform.io/) 一键部署 Dify 到 Azure。
- [Azure Terraform by @nikawang](https://github.com/nikawang/dify-azure-terraform)
+##### Google Cloud
+- [Google Cloud Terraform by @sotazum](https://github.com/DeNA/dify-google-cloud-terraform)
+
## Star History
[](https://star-history.com/#langgenius/dify&Date)
@@ -232,7 +236,7 @@ docker compose up -d
- [GitHub Issues](https://github.com/langgenius/dify/issues)。👉：使用 Dify.AI 时遇到的错误和问题，请参阅[贡献指南](CONTRIBUTING.md)。
- [电子邮件支持](mailto:hello@dify.ai?subject=[GitHub]Questions%20About%20Dify)。👉：关于使用 Dify.AI 的问题。
- [Discord](https://discord.gg/FngNHpbcY7)。👉：分享您的应用程序并与社区交流。
-- [Twitter](https://twitter.com/dify_ai)。👉：分享您的应用程序并与社区交流。
+- [X(Twitter)](https://twitter.com/dify_ai)。👉：分享您的应用程序并与社区交流。
- [商业许可](mailto:business@dify.ai?subject=[GitHub]Business%20License%20Inquiry)。👉：有关商业用途许可 Dify.AI 的商业咨询。
- [微信]() 👉：扫描下方二维码，添加微信好友，备注 Dify，我们将邀请您加入 Dify 社区。
diff --git a/README_ES.md b/README_ES.md
index 2ae044b328..7da5ac7b61 100644
--- a/README_ES.md
+++ b/README_ES.md
@@ -17,7 +17,7 @@
alt="chat en Discord">
+ alt="seguir en X(Twitter)">
@@ -204,10 +204,13 @@ Si desea configurar una configuración de alta disponibilidad, la comunidad prop
#### Uso de Terraform para el despliegue
+Despliega Dify en una plataforma en la nube con un solo clic utilizando [terraform](https://www.terraform.io/)
+
##### Azure Global
-Utiliza [terraform](https://www.terraform.io/) para desplegar Dify en Azure con un solo clic.
- [Azure Terraform por @nikawang](https://github.com/nikawang/dify-azure-terraform)
+##### Google Cloud
+- [Google Cloud Terraform por @sotazum](https://github.com/DeNA/dify-google-cloud-terraform)
## Contribuir
@@ -228,7 +231,7 @@ Al mismo tiempo, considera apoyar a Dify compartiéndolo en redes sociales y en
* [Discusión en GitHub](https://github.com/langgenius/dify/discussions). Lo mejor para: compartir comentarios y hacer preguntas.
* [Reporte de problemas en GitHub](https://github.com/langgenius/dify/issues). Lo mejor para: errores que encuentres usando Dify.AI y propuestas de características. Consulta nuestra [Guía de contribución](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md).
* [Discord](https://discord.gg/FngNHpbcY7). Lo mejor para: compartir tus aplicaciones y pasar el rato con la comunidad.
-* [Twitter](https://twitter.com/dify_ai). Lo mejor para: compartir tus aplicaciones y pasar el rato con la comunidad.
+* [X(Twitter)](https://twitter.com/dify_ai). Lo mejor para: compartir tus aplicaciones y pasar el rato con la comunidad.
## Historial de Estrellas
diff --git a/README_FR.md b/README_FR.md
index 681d596749..15f6f2650f 100644
--- a/README_FR.md
+++ b/README_FR.md
@@ -17,7 +17,7 @@
alt="chat sur Discord">
+ alt="suivre sur X(Twitter)">
@@ -202,10 +202,13 @@ Si vous souhaitez configurer une configuration haute disponibilité, la communau
#### Utilisation de Terraform pour le déploiement
+Déployez Dify sur une plateforme cloud en un clic en utilisant [terraform](https://www.terraform.io/)
+
##### Azure Global
-Utilisez [terraform](https://www.terraform.io/) pour déployer Dify sur Azure en un clic.
- [Azure Terraform par @nikawang](https://github.com/nikawang/dify-azure-terraform)
+##### Google Cloud
+- [Google Cloud Terraform par @sotazum](https://github.com/DeNA/dify-google-cloud-terraform)
## Contribuer
@@ -226,7 +229,7 @@ Dans le même temps, veuillez envisager de soutenir Dify en le partageant sur le
* [Discussion GitHub](https://github.com/langgenius/dify/discussions). Meilleur pour: partager des commentaires et poser des questions.
* [Problèmes GitHub](https://github.com/langgenius/dify/issues). Meilleur pour: les bogues que vous rencontrez en utilisant Dify.AI et les propositions de fonctionnalités. Consultez notre [Guide de contribution](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md).
* [Discord](https://discord.gg/FngNHpbcY7). Meilleur pour: partager vos applications et passer du temps avec la communauté.
-* [Twitter](https://twitter.com/dify_ai). Meilleur pour: partager vos applications et passer du temps avec la communauté.
+* [X(Twitter)](https://twitter.com/dify_ai). Meilleur pour: partager vos applications et passer du temps avec la communauté.
## Historique des étoiles
diff --git a/README_JA.md b/README_JA.md
index e6a8621e7b..a2e6b173f5 100644
--- a/README_JA.md
+++ b/README_JA.md
@@ -17,7 +17,7 @@
alt="Discordでチャット">
+ alt="X(Twitter)でフォロー">
@@ -68,7 +68,7 @@ DifyはオープンソースのLLMアプリケーション開発プラットフ
プロンプトの作成、モデルパフォーマンスの比較が行え、チャットベースのアプリに音声合成などの機能も追加できます。
**4. RAGパイプライン**:
- ドキュメントの取り込みから検索までをカバーする広範なRAG機能ができます。ほかにもPDF、PPT、その他の一般的なドキュメントフォーマットからのテキスト抽出のサーポイントも提供します。
+ ドキュメントの取り込みから検索までをカバーする広範なRAG機能ができます。ほかにもPDF、PPT、その他の一般的なドキュメントフォーマットからのテキスト抽出のサポートも提供します。
**5. エージェント機能**:
LLM Function CallingやReActに基づくエージェントの定義が可能で、AIエージェント用のプリビルトまたはカスタムツールを追加できます。Difyには、Google検索、DALL·E、Stable Diffusion、WolframAlphaなどのAIエージェント用の50以上の組み込みツールが提供します。
@@ -201,10 +201,13 @@ docker compose up -d
#### Terraformを使用したデプロイ
-##### Azure Global
-[terraform](https://www.terraform.io/) を使用して、AzureにDifyをワンクリックでデプロイします。
-- [nikawangのAzure Terraform](https://github.com/nikawang/dify-azure-terraform)
+[terraform](https://www.terraform.io/) を使用して、ワンクリックでDifyをクラウドプラットフォームにデプロイします
+##### Azure Global
+- [@nikawangによるAzure Terraform](https://github.com/nikawang/dify-azure-terraform)
+
+##### Google Cloud
+- [@sotazumによるGoogle Cloud Terraform](https://github.com/DeNA/dify-google-cloud-terraform)
## 貢献
@@ -225,7 +228,7 @@ docker compose up -d
* [Github Discussion](https://github.com/langgenius/dify/discussions). 主に: フィードバックの共有や質問。
* [GitHub Issues](https://github.com/langgenius/dify/issues). 主に: Dify.AIを使用する際に発生するエラーや問題については、[貢献ガイド](CONTRIBUTING_JA.md)を参照してください
* [Discord](https://discord.gg/FngNHpbcY7). 主に: アプリケーションの共有やコミュニティとの交流。
-* [Twitter](https://twitter.com/dify_ai). 主に: アプリケーションの共有やコミュニティとの交流。
+* [X(Twitter)](https://twitter.com/dify_ai). 主に: アプリケーションの共有やコミュニティとの交流。
diff --git a/README_KL.md b/README_KL.md
index 04620d42bb..8f2affdce5 100644
--- a/README_KL.md
+++ b/README_KL.md
@@ -17,7 +17,7 @@
alt="chat on Discord">
+ alt="follow on X(Twitter)">
@@ -202,10 +202,13 @@ If you'd like to configure a highly-available setup, there are community-contrib
#### Terraform atorlugu pilersitsineq
-##### Azure Global
-Atoruk [terraform](https://www.terraform.io/) Dify-mik Azure-mut ataatsikkut ikkussuilluarlugu.
-- [Azure Terraform atorlugu @nikawang](https://github.com/nikawang/dify-azure-terraform)
+wa'logh nIqHom neH ghun deployment toy'wI' [terraform](https://www.terraform.io/) lo'laH.
+##### Azure Global
+- [Azure Terraform mung @nikawang](https://github.com/nikawang/dify-azure-terraform)
+
+##### Google Cloud
+- [Google Cloud Terraform qachlot @sotazum](https://github.com/DeNA/dify-google-cloud-terraform)
## Contributing
@@ -228,7 +231,7 @@ At the same time, please consider supporting Dify by sharing it on social media
). Best for: sharing feedback and asking questions.
* [GitHub Issues](https://github.com/langgenius/dify/issues). Best for: bugs you encounter using Dify.AI, and feature proposals. See our [Contribution Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md).
* [Discord](https://discord.gg/FngNHpbcY7). Best for: sharing your applications and hanging out with the community.
-* [Twitter](https://twitter.com/dify_ai). Best for: sharing your applications and hanging out with the community.
+* [X(Twitter)](https://twitter.com/dify_ai). Best for: sharing your applications and hanging out with the community.
## Star History
diff --git a/README_KR.md b/README_KR.md
index a5f3bc68d0..6c3a9ed7f6 100644
--- a/README_KR.md
+++ b/README_KR.md
@@ -17,7 +17,7 @@
alt="chat on Discord">
+ alt="follow on X(Twitter)">
@@ -39,7 +39,6 @@
-
@@ -195,10 +194,14 @@ Dify를 Kubernetes에 배포하고 프리미엄 스케일링 설정을 구성했
#### Terraform을 사용한 배포
+[terraform](https://www.terraform.io/)을 사용하여 단 한 번의 클릭으로 Dify를 클라우드 플랫폼에 배포하십시오
+
##### Azure Global
-[terraform](https://www.terraform.io/)을 사용하여 Azure에 Dify를 원클릭으로 배포하세요.
- [nikawang의 Azure Terraform](https://github.com/nikawang/dify-azure-terraform)
+##### Google Cloud
+- [sotazum의 Google Cloud Terraform](https://github.com/DeNA/dify-google-cloud-terraform)
+
## 기여
코드에 기여하고 싶은 분들은 [기여 가이드](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md)를 참조하세요.
diff --git a/README_TR.md b/README_TR.md
index 54b6db3f82..a75889e576 100644
--- a/README_TR.md
+++ b/README_TR.md
@@ -17,7 +17,7 @@
alt="Discord'da sohbet et">
+ alt="X(Twitter)'da takip et">
@@ -200,9 +200,13 @@ Yüksek kullanılabilirliğe sahip bir kurulum yapılandırmak isterseniz, Dify'
#### Dağıtım için Terraform Kullanımı
+Dify'ı bulut platformuna tek tıklamayla dağıtın [terraform](https://www.terraform.io/) kullanarak
+
##### Azure Global
-[Terraform](https://www.terraform.io/) kullanarak Dify'ı Azure'a tek tıklamayla dağıtın.
-- [@nikawang tarafından Azure Terraform](https://github.com/nikawang/dify-azure-terraform)
+- [Azure Terraform tarafından @nikawang](https://github.com/nikawang/dify-azure-terraform)
+
+##### Google Cloud
+- [Google Cloud Terraform tarafından @sotazum](https://github.com/DeNA/dify-google-cloud-terraform)
## Katkıda Bulunma
@@ -222,7 +226,7 @@ Aynı zamanda, lütfen Dify'ı sosyal medyada, etkinliklerde ve konferanslarda p
* [Github Tartışmaları](https://github.com/langgenius/dify/discussions). En uygun: geri bildirim paylaşmak ve soru sormak için.
* [GitHub Sorunları](https://github.com/langgenius/dify/issues). En uygun: Dify.AI kullanırken karşılaştığınız hatalar ve özellik önerileri için. [Katkı Kılavuzumuza](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) bakın.
* [Discord](https://discord.gg/FngNHpbcY7). En uygun: uygulamalarınızı paylaşmak ve toplulukla vakit geçirmek için.
-* [Twitter](https://twitter.com/dify_ai). En uygun: uygulamalarınızı paylaşmak ve toplulukla vakit geçirmek için.
+* [X(Twitter)](https://twitter.com/dify_ai). En uygun: uygulamalarınızı paylaşmak ve toplulukla vakit geçirmek için.
## Star history
diff --git a/README_VI.md b/README_VI.md
index 6d4035eceb..8d49e49766 100644
--- a/README_VI.md
+++ b/README_VI.md
@@ -17,7 +17,7 @@
alt="chat trên Discord">
+ alt="theo dõi trên X(Twitter)">
@@ -196,10 +196,14 @@ Nếu bạn muốn cấu hình một cài đặt có độ sẵn sàng cao, có
#### Sử dụng Terraform để Triển khai
+Triển khai Dify lên nền tảng đám mây với một cú nhấp chuột bằng cách sử dụng [terraform](https://www.terraform.io/)
+
##### Azure Global
-Triển khai Dify lên Azure chỉ với một cú nhấp chuột bằng cách sử dụng [terraform](https://www.terraform.io/).
- [Azure Terraform bởi @nikawang](https://github.com/nikawang/dify-azure-terraform)
+##### Google Cloud
+- [Google Cloud Terraform bởi @sotazum](https://github.com/DeNA/dify-google-cloud-terraform)
+
## Đóng góp
Đối với những người muốn đóng góp mã, xem [Hướng dẫn Đóng góp](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) của chúng tôi.
@@ -219,7 +223,7 @@ Triển khai Dify lên Azure chỉ với một cú nhấp chuột bằng cách s
* [Thảo luận GitHub](https://github.com/langgenius/dify/discussions). Tốt nhất cho: chia sẻ phản hồi và đặt câu hỏi.
* [Vấn đề GitHub](https://github.com/langgenius/dify/issues). Tốt nhất cho: lỗi bạn gặp phải khi sử dụng Dify.AI và đề xuất tính năng. Xem [Hướng dẫn Đóng góp](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) của chúng tôi.
* [Discord](https://discord.gg/FngNHpbcY7). Tốt nhất cho: chia sẻ ứng dụng của bạn và giao lưu với cộng đồng.
-* [Twitter](https://twitter.com/dify_ai). Tốt nhất cho: chia sẻ ứng dụng của bạn và giao lưu với cộng đồng.
+* [X(Twitter)](https://twitter.com/dify_ai). Tốt nhất cho: chia sẻ ứng dụng của bạn và giao lưu với cộng đồng.
## Lịch sử Yêu thích
diff --git a/api/.env.example b/api/.env.example
index a3db406aea..aa155003ab 100644
--- a/api/.env.example
+++ b/api/.env.example
@@ -20,6 +20,9 @@ FILES_URL=http://127.0.0.1:5001
# The time in seconds after the signature is rejected
FILES_ACCESS_TIMEOUT=300
+# Access token expiration time in minutes
+ACCESS_TOKEN_EXPIRE_MINUTES=60
+
# celery configuration
CELERY_BROKER_URL=redis://:difyai123456@localhost:6379/1
@@ -39,7 +42,7 @@ DB_DATABASE=dify
# Storage configuration
# use for store upload files, private keys...
-# storage type: local, s3, azure-blob, google-storage, tencent-cos, huawei-obs, volcengine-tos
+# storage type: local, s3, azure-blob, google-storage, tencent-cos, huawei-obs, volcengine-tos, baidu-obs, supabase
STORAGE_TYPE=local
STORAGE_LOCAL_PATH=storage
S3_USE_AWS_MANAGED_IAM=false
@@ -79,6 +82,12 @@ HUAWEI_OBS_SECRET_KEY=your-secret-key
HUAWEI_OBS_ACCESS_KEY=your-access-key
HUAWEI_OBS_SERVER=your-server-url
+# Baidu OBS Storage Configuration
+BAIDU_OBS_BUCKET_NAME=your-bucket-name
+BAIDU_OBS_SECRET_KEY=your-secret-key
+BAIDU_OBS_ACCESS_KEY=your-access-key
+BAIDU_OBS_ENDPOINT=your-server-url
+
# OCI Storage configuration
OCI_ENDPOINT=your-endpoint
OCI_BUCKET_NAME=your-bucket-name
@@ -93,11 +102,16 @@ VOLCENGINE_TOS_ACCESS_KEY=your-access-key
VOLCENGINE_TOS_SECRET_KEY=your-secret-key
VOLCENGINE_TOS_REGION=your-region
+# Supabase Storage Configuration
+SUPABASE_BUCKET_NAME=your-bucket-name
+SUPABASE_API_KEY=your-access-key
+SUPABASE_URL=your-server-url
+
# CORS configuration
WEB_API_CORS_ALLOW_ORIGINS=http://127.0.0.1:3000,*
CONSOLE_CORS_ALLOW_ORIGINS=http://127.0.0.1:3000,*
-# Vector database configuration, support: weaviate, qdrant, milvus, myscale, relyt, pgvecto_rs, pgvector, pgvector, chroma, opensearch, tidb_vector
+# Vector database configuration, support: weaviate, qdrant, milvus, myscale, relyt, pgvecto_rs, pgvector, pgvector, chroma, opensearch, tidb_vector, vikingdb
VECTOR_STORE=weaviate
# Weaviate configuration
@@ -162,6 +176,8 @@ PGVECTOR_PORT=5433
PGVECTOR_USER=postgres
PGVECTOR_PASSWORD=postgres
PGVECTOR_DATABASE=postgres
+PGVECTOR_MIN_CONNECTION=1
+PGVECTOR_MAX_CONNECTION=5
# Tidb Vector configuration
TIDB_VECTOR_HOST=xxx.eu-central-1.xxx.aws.tidbcloud.com
@@ -195,6 +211,24 @@ OPENSEARCH_USER=admin
OPENSEARCH_PASSWORD=admin
OPENSEARCH_SECURE=true
+# Baidu configuration
+BAIDU_VECTOR_DB_ENDPOINT=http://127.0.0.1:5287
+BAIDU_VECTOR_DB_CONNECTION_TIMEOUT_MS=30000
+BAIDU_VECTOR_DB_ACCOUNT=root
+BAIDU_VECTOR_DB_API_KEY=dify
+BAIDU_VECTOR_DB_DATABASE=dify
+BAIDU_VECTOR_DB_SHARD=1
+BAIDU_VECTOR_DB_REPLICAS=3
+
+# ViKingDB configuration
+VIKINGDB_ACCESS_KEY=your-ak
+VIKINGDB_SECRET_KEY=your-sk
+VIKINGDB_REGION=cn-shanghai
+VIKINGDB_HOST=api-vikingdb.xxx.volces.com
+VIKINGDB_SCHEMA=http
+VIKINGDB_CONNECTION_TIMEOUT=30
+VIKINGDB_SOCKET_TIMEOUT=30
+
# Upload configuration
UPLOAD_FILE_SIZE_LIMIT=15
UPLOAD_FILE_BATCH_LIMIT=5
@@ -263,6 +297,9 @@ HTTP_REQUEST_MAX_WRITE_TIMEOUT=600
HTTP_REQUEST_NODE_MAX_BINARY_SIZE=10485760
HTTP_REQUEST_NODE_MAX_TEXT_SIZE=1048576
+# Respect X-* headers to redirect clients
+RESPECT_XFORWARD_HEADERS_ENABLED=false
+
# Log file path
LOG_FILE=
diff --git a/api/app.py b/api/app.py
index 1b58beee15..52dd492225 100644
--- a/api/app.py
+++ b/api/app.py
@@ -26,7 +26,7 @@ from commands import register_commands
from configs import dify_config
# DO NOT REMOVE BELOW
-from events import event_handlers
+from events import event_handlers # noqa: F401
from extensions import (
ext_celery,
ext_code_based_extension,
@@ -36,6 +36,7 @@ from extensions import (
ext_login,
ext_mail,
ext_migrate,
+ ext_proxy_fix,
ext_redis,
ext_sentry,
ext_storage,
@@ -45,7 +46,7 @@ from extensions.ext_login import login_manager
from libs.passport import PassportService
# TODO: Find a way to avoid importing models here
-from models import account, dataset, model, source, task, tool, tools, web
+from models import account, dataset, model, source, task, tool, tools, web # noqa: F401
from services.account_service import AccountService
# DO NOT REMOVE ABOVE
@@ -156,6 +157,7 @@ def initialize_extensions(app):
ext_mail.init_app(app)
ext_hosting_provider.init_app(app)
ext_sentry.init_app(app)
+ ext_proxy_fix.init_app(app)
# Flask-Login configuration
@@ -181,10 +183,10 @@ def load_user_from_request(request_from_flask_login):
decoded = PassportService().verify(auth_token)
user_id = decoded.get("user_id")
- account = AccountService.load_logged_in_account(account_id=user_id, token=auth_token)
- if account:
- contexts.tenant_id.set(account.current_tenant_id)
- return account
+ logged_in_account = AccountService.load_logged_in_account(account_id=user_id)
+ if logged_in_account:
+ contexts.tenant_id.set(logged_in_account.current_tenant_id)
+ return logged_in_account
@login_manager.unauthorized_handler
diff --git a/api/commands.py b/api/commands.py
index 7ef4aed7f7..dbcd8a744d 100644
--- a/api/commands.py
+++ b/api/commands.py
@@ -347,6 +347,14 @@ def migrate_knowledge_vector_database():
index_name = Dataset.gen_collection_name_by_id(dataset_id)
index_struct_dict = {"type": "elasticsearch", "vector_store": {"class_prefix": index_name}}
dataset.index_struct = json.dumps(index_struct_dict)
+ elif vector_type == VectorType.BAIDU:
+ dataset_id = dataset.id
+ collection_name = Dataset.gen_collection_name_by_id(dataset_id)
+ index_struct_dict = {
+ "type": VectorType.BAIDU,
+ "vector_store": {"class_prefix": collection_name},
+ }
+ dataset.index_struct = json.dumps(index_struct_dict)
else:
raise ValueError(f"Vector store {vector_type} is not supported.")
diff --git a/api/configs/feature/__init__.py b/api/configs/feature/__init__.py
index 9218d529cc..a3334d1634 100644
--- a/api/configs/feature/__init__.py
+++ b/api/configs/feature/__init__.py
@@ -247,6 +247,12 @@ class HttpConfig(BaseSettings):
default=None,
)
+ RESPECT_XFORWARD_HEADERS_ENABLED: bool = Field(
+ description="Enable or disable the X-Forwarded-For Proxy Fix middleware from Werkzeug"
+ " to respect X-* headers to redirect clients",
+ default=False,
+ )
+
class InnerAPIConfig(BaseSettings):
"""
@@ -354,9 +360,9 @@ class WorkflowConfig(BaseSettings):
)
-class OAuthConfig(BaseSettings):
+class AuthConfig(BaseSettings):
"""
- Configuration for OAuth authentication
+ Configuration for authentication and OAuth
"""
OAUTH_REDIRECT_PATH: str = Field(
@@ -365,7 +371,7 @@ class OAuthConfig(BaseSettings):
)
GITHUB_CLIENT_ID: Optional[str] = Field(
- description="GitHub OAuth client secret",
+ description="GitHub OAuth client ID",
default=None,
)
@@ -384,6 +390,11 @@ class OAuthConfig(BaseSettings):
default=None,
)
+ ACCESS_TOKEN_EXPIRE_MINUTES: PositiveInt = Field(
+ description="Expiration time for access tokens in minutes",
+ default=60,
+ )
+
class ModerationConfig(BaseSettings):
"""
@@ -601,6 +612,7 @@ class PositionConfig(BaseSettings):
class FeatureConfig(
# place the configs in alphabet order
AppExecutionConfig,
+ AuthConfig, # Changed from OAuthConfig to AuthConfig
BillingConfig,
CodeExecutionSandboxConfig,
DataSetConfig,
@@ -615,14 +627,13 @@ class FeatureConfig(
MailConfig,
ModelLoadBalanceConfig,
ModerationConfig,
- OAuthConfig,
+ PositionConfig,
RagEtlConfig,
SecurityConfig,
ToolConfig,
UpdateConfig,
WorkflowConfig,
WorkspaceConfig,
- PositionConfig,
# hosted services config
HostedServiceConfig,
CeleryBeatConfig,
diff --git a/api/configs/middleware/__init__.py b/api/configs/middleware/__init__.py
index 6ad216c191..fa7f41d630 100644
--- a/api/configs/middleware/__init__.py
+++ b/api/configs/middleware/__init__.py
@@ -8,9 +8,11 @@ from configs.middleware.cache.redis_config import RedisConfig
from configs.middleware.storage.aliyun_oss_storage_config import AliyunOSSStorageConfig
from configs.middleware.storage.amazon_s3_storage_config import S3StorageConfig
from configs.middleware.storage.azure_blob_storage_config import AzureBlobStorageConfig
+from configs.middleware.storage.baidu_obs_storage_config import BaiduOBSStorageConfig
from configs.middleware.storage.google_cloud_storage_config import GoogleCloudStorageConfig
from configs.middleware.storage.huawei_obs_storage_config import HuaweiCloudOBSStorageConfig
from configs.middleware.storage.oci_storage_config import OCIStorageConfig
+from configs.middleware.storage.supabase_storage_config import SupabaseStorageConfig
from configs.middleware.storage.tencent_cos_storage_config import TencentCloudCOSStorageConfig
from configs.middleware.storage.volcengine_tos_storage_config import VolcengineTOSStorageConfig
from configs.middleware.vdb.analyticdb_config import AnalyticdbConfig
@@ -26,6 +28,7 @@ from configs.middleware.vdb.qdrant_config import QdrantConfig
from configs.middleware.vdb.relyt_config import RelytConfig
from configs.middleware.vdb.tencent_vector_config import TencentVectorDBConfig
from configs.middleware.vdb.tidb_vector_config import TiDBVectorConfig
+from configs.middleware.vdb.vikingdb_config import VikingDBConfig
from configs.middleware.vdb.weaviate_config import WeaviateConfig
@@ -190,6 +193,22 @@ class CeleryConfig(DatabaseConfig):
return self.CELERY_BROKER_URL.startswith("rediss://") if self.CELERY_BROKER_URL else False
+class InternalTestConfig(BaseSettings):
+ """
+ Configuration settings for Internal Test
+ """
+
+ AWS_SECRET_ACCESS_KEY: Optional[str] = Field(
+ description="Internal test AWS secret access key",
+ default=None,
+ )
+
+ AWS_ACCESS_KEY_ID: Optional[str] = Field(
+ description="Internal test AWS access key ID",
+ default=None,
+ )
+
+
class MiddlewareConfig(
# place the configs in alphabet order
CeleryConfig,
@@ -200,12 +219,14 @@ class MiddlewareConfig(
StorageConfig,
AliyunOSSStorageConfig,
AzureBlobStorageConfig,
+ BaiduOBSStorageConfig,
GoogleCloudStorageConfig,
- TencentCloudCOSStorageConfig,
HuaweiCloudOBSStorageConfig,
- VolcengineTOSStorageConfig,
- S3StorageConfig,
OCIStorageConfig,
+ S3StorageConfig,
+ SupabaseStorageConfig,
+ TencentCloudCOSStorageConfig,
+ VolcengineTOSStorageConfig,
# configs of vdb and vdb providers
VectorStoreConfig,
AnalyticdbConfig,
@@ -222,5 +243,7 @@ class MiddlewareConfig(
TiDBVectorConfig,
WeaviateConfig,
ElasticsearchConfig,
+ InternalTestConfig,
+ VikingDBConfig,
):
pass
diff --git a/api/configs/middleware/storage/baidu_obs_storage_config.py b/api/configs/middleware/storage/baidu_obs_storage_config.py
new file mode 100644
index 0000000000..c511628a15
--- /dev/null
+++ b/api/configs/middleware/storage/baidu_obs_storage_config.py
@@ -0,0 +1,29 @@
+from typing import Optional
+
+from pydantic import BaseModel, Field
+
+
+class BaiduOBSStorageConfig(BaseModel):
+ """
+ Configuration settings for Baidu Object Storage Service (OBS)
+ """
+
+ BAIDU_OBS_BUCKET_NAME: Optional[str] = Field(
+ description="Name of the Baidu OBS bucket to store and retrieve objects (e.g., 'my-obs-bucket')",
+ default=None,
+ )
+
+ BAIDU_OBS_ACCESS_KEY: Optional[str] = Field(
+ description="Access Key ID for authenticating with Baidu OBS",
+ default=None,
+ )
+
+ BAIDU_OBS_SECRET_KEY: Optional[str] = Field(
+ description="Secret Access Key for authenticating with Baidu OBS",
+ default=None,
+ )
+
+ BAIDU_OBS_ENDPOINT: Optional[str] = Field(
+ description="URL of the Baidu OSS endpoint for your chosen region (e.g., 'https://.bj.bcebos.com')",
+ default=None,
+ )
diff --git a/api/configs/middleware/storage/supabase_storage_config.py b/api/configs/middleware/storage/supabase_storage_config.py
new file mode 100644
index 0000000000..a3e905b21c
--- /dev/null
+++ b/api/configs/middleware/storage/supabase_storage_config.py
@@ -0,0 +1,24 @@
+from typing import Optional
+
+from pydantic import BaseModel, Field
+
+
+class SupabaseStorageConfig(BaseModel):
+ """
+ Configuration settings for Supabase Object Storage Service
+ """
+
+ SUPABASE_BUCKET_NAME: Optional[str] = Field(
+ description="Name of the Supabase bucket to store and retrieve objects (e.g., 'dify-bucket')",
+ default=None,
+ )
+
+ SUPABASE_API_KEY: Optional[str] = Field(
+ description="API KEY for authenticating with Supabase",
+ default=None,
+ )
+
+ SUPABASE_URL: Optional[str] = Field(
+ description="URL of the Supabase",
+ default=None,
+ )
diff --git a/api/configs/middleware/vdb/baidu_vector_config.py b/api/configs/middleware/vdb/baidu_vector_config.py
new file mode 100644
index 0000000000..44742c2e2f
--- /dev/null
+++ b/api/configs/middleware/vdb/baidu_vector_config.py
@@ -0,0 +1,45 @@
+from typing import Optional
+
+from pydantic import Field, NonNegativeInt, PositiveInt
+from pydantic_settings import BaseSettings
+
+
+class BaiduVectorDBConfig(BaseSettings):
+ """
+ Configuration settings for Baidu Vector Database
+ """
+
+ BAIDU_VECTOR_DB_ENDPOINT: Optional[str] = Field(
+ description="URL of the Baidu Vector Database service (e.g., 'http://vdb.bj.baidubce.com')",
+ default=None,
+ )
+
+ BAIDU_VECTOR_DB_CONNECTION_TIMEOUT_MS: PositiveInt = Field(
+ description="Timeout in milliseconds for Baidu Vector Database operations (default is 30000 milliseconds)",
+ default=30000,
+ )
+
+ BAIDU_VECTOR_DB_ACCOUNT: Optional[str] = Field(
+ description="Account for authenticating with the Baidu Vector Database",
+ default=None,
+ )
+
+ BAIDU_VECTOR_DB_API_KEY: Optional[str] = Field(
+ description="API key for authenticating with the Baidu Vector Database service",
+ default=None,
+ )
+
+ BAIDU_VECTOR_DB_DATABASE: Optional[str] = Field(
+ description="Name of the specific Baidu Vector Database to connect to",
+ default=None,
+ )
+
+ BAIDU_VECTOR_DB_SHARD: PositiveInt = Field(
+ description="Number of shards for the Baidu Vector Database (default is 1)",
+ default=1,
+ )
+
+ BAIDU_VECTOR_DB_REPLICAS: NonNegativeInt = Field(
+ description="Number of replicas for the Baidu Vector Database (default is 3)",
+ default=3,
+ )
diff --git a/api/configs/middleware/vdb/pgvector_config.py b/api/configs/middleware/vdb/pgvector_config.py
index 395dcaa420..85f5dca7e2 100644
--- a/api/configs/middleware/vdb/pgvector_config.py
+++ b/api/configs/middleware/vdb/pgvector_config.py
@@ -33,3 +33,13 @@ class PGVectorConfig(BaseSettings):
description="Name of the PostgreSQL database to connect to",
default=None,
)
+
+ PGVECTOR_MIN_CONNECTION: PositiveInt = Field(
+ description="Min connection of the PostgreSQL database",
+ default=1,
+ )
+
+ PGVECTOR_MAX_CONNECTION: PositiveInt = Field(
+ description="Max connection of the PostgreSQL database",
+ default=5,
+ )
diff --git a/api/configs/middleware/vdb/vikingdb_config.py b/api/configs/middleware/vdb/vikingdb_config.py
new file mode 100644
index 0000000000..5ad98d898a
--- /dev/null
+++ b/api/configs/middleware/vdb/vikingdb_config.py
@@ -0,0 +1,37 @@
+from typing import Optional
+
+from pydantic import BaseModel, Field
+
+
+class VikingDBConfig(BaseModel):
+ """
+ Configuration for connecting to Volcengine VikingDB.
+ Refer to the following documentation for details on obtaining credentials:
+ https://www.volcengine.com/docs/6291/65568
+ """
+
+ VIKINGDB_ACCESS_KEY: Optional[str] = Field(
+ default=None, description="The Access Key provided by Volcengine VikingDB for API authentication."
+ )
+ VIKINGDB_SECRET_KEY: Optional[str] = Field(
+ default=None, description="The Secret Key provided by Volcengine VikingDB for API authentication."
+ )
+ VIKINGDB_REGION: Optional[str] = Field(
+ default="cn-shanghai",
+ description="The region of the Volcengine VikingDB service.(e.g., 'cn-shanghai', 'cn-beijing').",
+ )
+ VIKINGDB_HOST: Optional[str] = Field(
+ default="api-vikingdb.mlp.cn-shanghai.volces.com",
+ description="The host of the Volcengine VikingDB service.(e.g., 'api-vikingdb.volces.com', \
+ 'api-vikingdb.mlp.cn-shanghai.volces.com')",
+ )
+ VIKINGDB_SCHEME: Optional[str] = Field(
+ default="http",
+ description="The scheme of the Volcengine VikingDB service.(e.g., 'http', 'https').",
+ )
+ VIKINGDB_CONNECTION_TIMEOUT: Optional[int] = Field(
+ default=30, description="The connection timeout of the Volcengine VikingDB service."
+ )
+ VIKINGDB_SOCKET_TIMEOUT: Optional[int] = Field(
+ default=30, description="The socket timeout of the Volcengine VikingDB service."
+ )
diff --git a/api/configs/packaging/__init__.py b/api/configs/packaging/__init__.py
index c752660122..c311a989b4 100644
--- a/api/configs/packaging/__init__.py
+++ b/api/configs/packaging/__init__.py
@@ -9,7 +9,7 @@ class PackagingInfo(BaseSettings):
CURRENT_VERSION: str = Field(
description="Dify version",
- default="0.8.3",
+ default="0.9.1",
)
COMMIT_SHA: str = Field(
diff --git a/api/controllers/console/__init__.py b/api/controllers/console/__init__.py
index eb7c1464d3..c7282fcf14 100644
--- a/api/controllers/console/__init__.py
+++ b/api/controllers/console/__init__.py
@@ -37,7 +37,16 @@ from .auth import activate, data_source_bearer_auth, data_source_oauth, forgot_p
from .billing import billing
# Import datasets controllers
-from .datasets import data_source, datasets, datasets_document, datasets_segments, file, hit_testing, website
+from .datasets import (
+ data_source,
+ datasets,
+ datasets_document,
+ datasets_segments,
+ external,
+ file,
+ hit_testing,
+ website,
+)
# Import explore controllers
from .explore import (
diff --git a/api/controllers/console/app/conversation.py b/api/controllers/console/app/conversation.py
index df7bd352af..c1e16b3b9b 100644
--- a/api/controllers/console/app/conversation.py
+++ b/api/controllers/console/app/conversation.py
@@ -188,6 +188,7 @@ class ChatConversationApi(Resource):
subquery.c.from_end_user_session_id.ilike(keyword_filter),
),
)
+ .group_by(Conversation.id)
)
account = current_user
diff --git a/api/controllers/console/auth/login.py b/api/controllers/console/auth/login.py
index 62837af2b9..18a7b23166 100644
--- a/api/controllers/console/auth/login.py
+++ b/api/controllers/console/auth/login.py
@@ -7,7 +7,7 @@ from flask_restful import Resource, reqparse
import services
from controllers.console import api
from controllers.console.setup import setup_required
-from libs.helper import email, get_remote_ip
+from libs.helper import email, extract_remote_ip
from libs.password import valid_password
from models.account import Account
from services.account_service import AccountService, TenantService
@@ -40,17 +40,16 @@ class LoginApi(Resource):
"data": "workspace not found, please contact system admin to invite you to join in a workspace",
}
- token = AccountService.login(account, ip_address=get_remote_ip(request))
+ token_pair = AccountService.login(account=account, ip_address=extract_remote_ip(request))
- return {"result": "success", "data": token}
+ return {"result": "success", "data": token_pair.model_dump()}
class LogoutApi(Resource):
@setup_required
def get(self):
account = cast(Account, flask_login.current_user)
- token = request.headers.get("Authorization", "").split(" ")[1]
- AccountService.logout(account=account, token=token)
+ AccountService.logout(account=account)
flask_login.logout_user()
return {"result": "success"}
@@ -106,5 +105,19 @@ class ResetPasswordApi(Resource):
return {"result": "success"}
+class RefreshTokenApi(Resource):
+ def post(self):
+ parser = reqparse.RequestParser()
+ parser.add_argument("refresh_token", type=str, required=True, location="json")
+ args = parser.parse_args()
+
+ try:
+ new_token_pair = AccountService.refresh_token(args["refresh_token"])
+ return {"result": "success", "data": new_token_pair.model_dump()}
+ except Exception as e:
+ return {"result": "fail", "data": str(e)}, 401
+
+
api.add_resource(LoginApi, "/login")
api.add_resource(LogoutApi, "/logout")
+api.add_resource(RefreshTokenApi, "/refresh-token")
diff --git a/api/controllers/console/auth/oauth.py b/api/controllers/console/auth/oauth.py
index ad0c0580ae..c5909b8c10 100644
--- a/api/controllers/console/auth/oauth.py
+++ b/api/controllers/console/auth/oauth.py
@@ -9,7 +9,7 @@ from flask_restful import Resource
from configs import dify_config
from constants.languages import languages
from extensions.ext_database import db
-from libs.helper import get_remote_ip
+from libs.helper import extract_remote_ip
from libs.oauth import GitHubOAuth, GoogleOAuth, OAuthUserInfo
from models.account import Account, AccountStatus
from services.account_service import AccountService, RegisterService, TenantService
@@ -81,9 +81,14 @@ class OAuthCallback(Resource):
TenantService.create_owner_tenant_if_not_exist(account)
- token = AccountService.login(account, ip_address=get_remote_ip(request))
+ token_pair = AccountService.login(
+ account=account,
+ ip_address=extract_remote_ip(request),
+ )
- return redirect(f"{dify_config.CONSOLE_WEB_URL}?console_token={token}")
+ return redirect(
+ f"{dify_config.CONSOLE_WEB_URL}?access_token={token_pair.access_token}&refresh_token={token_pair.refresh_token}"
+ )
def _get_account_by_openid_or_email(provider: str, user_info: OAuthUserInfo) -> Optional[Account]:
diff --git a/api/controllers/console/datasets/datasets.py b/api/controllers/console/datasets/datasets.py
index 2c4e5ac607..6583356d23 100644
--- a/api/controllers/console/datasets/datasets.py
+++ b/api/controllers/console/datasets/datasets.py
@@ -49,7 +49,7 @@ class DatasetListApi(Resource):
page = request.args.get("page", default=1, type=int)
limit = request.args.get("limit", default=20, type=int)
ids = request.args.getlist("ids")
- provider = request.args.get("provider", default="vendor")
+ # provider = request.args.get("provider", default="vendor")
search = request.args.get("keyword", default=None, type=str)
tag_ids = request.args.getlist("tag_ids")
@@ -57,7 +57,7 @@ class DatasetListApi(Resource):
datasets, total = DatasetService.get_datasets_by_ids(ids, current_user.current_tenant_id)
else:
datasets, total = DatasetService.get_datasets(
- page, limit, provider, current_user.current_tenant_id, current_user, search, tag_ids
+ page, limit, current_user.current_tenant_id, current_user, search, tag_ids
)
# check embedding setting
@@ -110,6 +110,26 @@ class DatasetListApi(Resource):
nullable=True,
help="Invalid indexing technique.",
)
+ parser.add_argument(
+ "external_knowledge_api_id",
+ type=str,
+ nullable=True,
+ required=False,
+ )
+ parser.add_argument(
+ "provider",
+ type=str,
+ nullable=True,
+ choices=Dataset.PROVIDER_LIST,
+ required=False,
+ default="vendor",
+ )
+ parser.add_argument(
+ "external_knowledge_id",
+ type=str,
+ nullable=True,
+ required=False,
+ )
args = parser.parse_args()
# The role of the current user in the ta table must be admin, owner, or editor, or dataset_operator
@@ -123,6 +143,9 @@ class DatasetListApi(Resource):
indexing_technique=args["indexing_technique"],
account=current_user,
permission=DatasetPermissionEnum.ONLY_ME,
+ provider=args["provider"],
+ external_knowledge_api_id=args["external_knowledge_api_id"],
+ external_knowledge_id=args["external_knowledge_id"],
)
except services.errors.dataset.DatasetNameDuplicateError:
raise DatasetNameDuplicateError()
@@ -211,6 +234,33 @@ class DatasetApi(Resource):
)
parser.add_argument("retrieval_model", type=dict, location="json", help="Invalid retrieval model.")
parser.add_argument("partial_member_list", type=list, location="json", help="Invalid parent user list.")
+
+ parser.add_argument(
+ "external_retrieval_model",
+ type=dict,
+ required=False,
+ nullable=True,
+ location="json",
+ help="Invalid external retrieval model.",
+ )
+
+ parser.add_argument(
+ "external_knowledge_id",
+ type=str,
+ required=False,
+ nullable=True,
+ location="json",
+ help="Invalid external knowledge id.",
+ )
+
+ parser.add_argument(
+ "external_knowledge_api_id",
+ type=str,
+ required=False,
+ nullable=True,
+ location="json",
+ help="Invalid external knowledge api id.",
+ )
args = parser.parse_args()
data = request.get_json()
@@ -563,10 +613,12 @@ class DatasetRetrievalSettingApi(Resource):
case (
VectorType.MILVUS
| VectorType.RELYT
- | VectorType.PGVECTOR
| VectorType.TIDB_VECTOR
| VectorType.CHROMA
| VectorType.TENCENT
+ | VectorType.PGVECTO_RS
+ | VectorType.BAIDU
+ | VectorType.VIKINGDB
):
return {"retrieval_method": [RetrievalMethod.SEMANTIC_SEARCH.value]}
case (
@@ -577,6 +629,7 @@ class DatasetRetrievalSettingApi(Resource):
| VectorType.MYSCALE
| VectorType.ORACLE
| VectorType.ELASTICSEARCH
+ | VectorType.PGVECTOR
):
return {
"retrieval_method": [
@@ -602,6 +655,8 @@ class DatasetRetrievalSettingMockApi(Resource):
| VectorType.CHROMA
| VectorType.TENCENT
| VectorType.PGVECTO_RS
+ | VectorType.BAIDU
+ | VectorType.VIKINGDB
):
return {"retrieval_method": [RetrievalMethod.SEMANTIC_SEARCH.value]}
case (
diff --git a/api/controllers/console/datasets/external.py b/api/controllers/console/datasets/external.py
new file mode 100644
index 0000000000..2dc054cfbd
--- /dev/null
+++ b/api/controllers/console/datasets/external.py
@@ -0,0 +1,263 @@
+from flask import request
+from flask_login import current_user
+from flask_restful import Resource, marshal, reqparse
+from werkzeug.exceptions import Forbidden, InternalServerError, NotFound
+
+import services
+from controllers.console import api
+from controllers.console.datasets.error import DatasetNameDuplicateError
+from controllers.console.setup import setup_required
+from controllers.console.wraps import account_initialization_required
+from fields.dataset_fields import dataset_detail_fields
+from libs.login import login_required
+from services.dataset_service import DatasetService
+from services.external_knowledge_service import ExternalDatasetService
+from services.hit_testing_service import HitTestingService
+from services.knowledge_service import ExternalDatasetTestService
+
+
+def _validate_name(name):
+ if not name or len(name) < 1 or len(name) > 100:
+ raise ValueError("Name must be between 1 to 100 characters.")
+ return name
+
+
+def _validate_description_length(description):
+ if description and len(description) > 400:
+ raise ValueError("Description cannot exceed 400 characters.")
+ return description
+
+
+class ExternalApiTemplateListApi(Resource):
+ @setup_required
+ @login_required
+ @account_initialization_required
+ def get(self):
+ page = request.args.get("page", default=1, type=int)
+ limit = request.args.get("limit", default=20, type=int)
+ search = request.args.get("keyword", default=None, type=str)
+
+ external_knowledge_apis, total = ExternalDatasetService.get_external_knowledge_apis(
+ page, limit, current_user.current_tenant_id, search
+ )
+ response = {
+ "data": [item.to_dict() for item in external_knowledge_apis],
+ "has_more": len(external_knowledge_apis) == limit,
+ "limit": limit,
+ "total": total,
+ "page": page,
+ }
+ return response, 200
+
+ @setup_required
+ @login_required
+ @account_initialization_required
+ def post(self):
+ parser = reqparse.RequestParser()
+ parser.add_argument(
+ "name",
+ nullable=False,
+ required=True,
+ help="Name is required. Name must be between 1 to 100 characters.",
+ type=_validate_name,
+ )
+ parser.add_argument(
+ "settings",
+ type=dict,
+ location="json",
+ nullable=False,
+ required=True,
+ )
+ args = parser.parse_args()
+
+ ExternalDatasetService.validate_api_list(args["settings"])
+
+ # The role of the current user in the ta table must be admin, owner, or editor, or dataset_operator
+ if not current_user.is_dataset_editor:
+ raise Forbidden()
+
+ try:
+ external_knowledge_api = ExternalDatasetService.create_external_knowledge_api(
+ tenant_id=current_user.current_tenant_id, user_id=current_user.id, args=args
+ )
+ except services.errors.dataset.DatasetNameDuplicateError:
+ raise DatasetNameDuplicateError()
+
+ return external_knowledge_api.to_dict(), 201
+
+
+class ExternalApiTemplateApi(Resource):
+ @setup_required
+ @login_required
+ @account_initialization_required
+ def get(self, external_knowledge_api_id):
+ external_knowledge_api_id = str(external_knowledge_api_id)
+ external_knowledge_api = ExternalDatasetService.get_external_knowledge_api(external_knowledge_api_id)
+ if external_knowledge_api is None:
+ raise NotFound("API template not found.")
+
+ return external_knowledge_api.to_dict(), 200
+
+ @setup_required
+ @login_required
+ @account_initialization_required
+ def patch(self, external_knowledge_api_id):
+ external_knowledge_api_id = str(external_knowledge_api_id)
+
+ parser = reqparse.RequestParser()
+ parser.add_argument(
+ "name",
+ nullable=False,
+ required=True,
+ help="type is required. Name must be between 1 to 100 characters.",
+ type=_validate_name,
+ )
+ parser.add_argument(
+ "settings",
+ type=dict,
+ location="json",
+ nullable=False,
+ required=True,
+ )
+ args = parser.parse_args()
+ ExternalDatasetService.validate_api_list(args["settings"])
+
+ external_knowledge_api = ExternalDatasetService.update_external_knowledge_api(
+ tenant_id=current_user.current_tenant_id,
+ user_id=current_user.id,
+ external_knowledge_api_id=external_knowledge_api_id,
+ args=args,
+ )
+
+ return external_knowledge_api.to_dict(), 200
+
+ @setup_required
+ @login_required
+ @account_initialization_required
+ def delete(self, external_knowledge_api_id):
+ external_knowledge_api_id = str(external_knowledge_api_id)
+
+ # The role of the current user in the ta table must be admin, owner, or editor
+ if not current_user.is_editor or current_user.is_dataset_operator:
+ raise Forbidden()
+
+ ExternalDatasetService.delete_external_knowledge_api(current_user.current_tenant_id, external_knowledge_api_id)
+ return {"result": "success"}, 200
+
+
+class ExternalApiUseCheckApi(Resource):
+ @setup_required
+ @login_required
+ @account_initialization_required
+ def get(self, external_knowledge_api_id):
+ external_knowledge_api_id = str(external_knowledge_api_id)
+
+ external_knowledge_api_is_using, count = ExternalDatasetService.external_knowledge_api_use_check(
+ external_knowledge_api_id
+ )
+ return {"is_using": external_knowledge_api_is_using, "count": count}, 200
+
+
+class ExternalDatasetCreateApi(Resource):
+ @setup_required
+ @login_required
+ @account_initialization_required
+ def post(self):
+ # The role of the current user in the ta table must be admin, owner, or editor
+ if not current_user.is_editor:
+ raise Forbidden()
+
+ parser = reqparse.RequestParser()
+ parser.add_argument("external_knowledge_api_id", type=str, required=True, nullable=False, location="json")
+ parser.add_argument("external_knowledge_id", type=str, required=True, nullable=False, location="json")
+ parser.add_argument(
+ "name",
+ nullable=False,
+ required=True,
+ help="name is required. Name must be between 1 to 100 characters.",
+ type=_validate_name,
+ )
+ parser.add_argument("description", type=str, required=False, nullable=True, location="json")
+ parser.add_argument("external_retrieval_model", type=dict, required=False, location="json")
+
+ args = parser.parse_args()
+
+ # The role of the current user in the ta table must be admin, owner, or editor, or dataset_operator
+ if not current_user.is_dataset_editor:
+ raise Forbidden()
+
+ try:
+ dataset = ExternalDatasetService.create_external_dataset(
+ tenant_id=current_user.current_tenant_id,
+ user_id=current_user.id,
+ args=args,
+ )
+ except services.errors.dataset.DatasetNameDuplicateError:
+ raise DatasetNameDuplicateError()
+
+ return marshal(dataset, dataset_detail_fields), 201
+
+
+class ExternalKnowledgeHitTestingApi(Resource):
+ @setup_required
+ @login_required
+ @account_initialization_required
+ def post(self, dataset_id):
+ dataset_id_str = str(dataset_id)
+ dataset = DatasetService.get_dataset(dataset_id_str)
+ if dataset is None:
+ raise NotFound("Dataset not found.")
+
+ try:
+ DatasetService.check_dataset_permission(dataset, current_user)
+ except services.errors.account.NoPermissionError as e:
+ raise Forbidden(str(e))
+
+ parser = reqparse.RequestParser()
+ parser.add_argument("query", type=str, location="json")
+ parser.add_argument("external_retrieval_model", type=dict, required=False, location="json")
+ args = parser.parse_args()
+
+ HitTestingService.hit_testing_args_check(args)
+
+ try:
+ response = HitTestingService.external_retrieve(
+ dataset=dataset,
+ query=args["query"],
+ account=current_user,
+ external_retrieval_model=args["external_retrieval_model"],
+ )
+
+ return response
+ except Exception as e:
+ raise InternalServerError(str(e))
+
+
+class BedrockRetrievalApi(Resource):
+ # this api is only for internal testing
+ def post(self):
+ parser = reqparse.RequestParser()
+ parser.add_argument("retrieval_setting", nullable=False, required=True, type=dict, location="json")
+ parser.add_argument(
+ "query",
+ nullable=False,
+ required=True,
+ type=str,
+ )
+ parser.add_argument("knowledge_id", nullable=False, required=True, type=str)
+ args = parser.parse_args()
+
+ # Call the knowledge retrieval service
+ result = ExternalDatasetTestService.knowledge_retrieval(
+ args["retrieval_setting"], args["query"], args["knowledge_id"]
+ )
+ return result, 200
+
+
+api.add_resource(ExternalKnowledgeHitTestingApi, "/datasets/{userProfile.name}
+{userProfile.email}
+{t('common.account.account')}
+