Two real bugs the user restart surfaced — both turned healthy providers
into HARD-removed false positives.
Bug #1 — URL duplication
OpenAiCompatibleListModelsProbe always concatenated /v1/models, so
providers whose Base URL already includes the version segment got the
wrong URL:
LMStudio http://localhost:1234/v1 → /v1/v1/models → 404
ZhipuAI .../api/paas/v4 → /v4/v1/models → 404
Fix: detect a trailing /vN suffix and append /models instead. Six unit
tests in OpenAiCompatibleListModelsProbeTest lock the rule down.
Bug #2 — 404 false positives
Kimi for Coding API does not expose /v1/models even though chat works
fine, so the probe correctly received a 404 and incorrectly HARD-removed
the provider from the pool. Other vendors will hit the same — listing
is not a universal contract.
Fix: classify HTTP responses semantically.
401 / 403 → HARD remove (real auth failure)
404 / 405 / 410 → fail-open (endpoint missing, server may be alive)
other 4xx / 5xx → fail-open (probe inconclusive — let chat decide)
network errors → fail (unreachable)
This is the same philosophy as ChatGPTOAuthStatusProbe: when we cannot
cheaply confirm health, we do not proactively penalize the provider.
Same logic applied to Anthropic + DashScope probes for consistency.
Net effect on the user deployment after restart:
- kimi-code stays in pool (404 → fail-open) → primary path works again
- lmstudio + zhipu-cn also stay in pool (URL bug fixed)
- dashscope + ollama unchanged (real 200 OK)
Tests: 6 new for resolveModelsPath. The 2 unrelated WikiRawMaterialDedupTest
failures pre-date this commit and live in ba86bea.
Root cause: addFile()/addText() hash dedup only matched rows with
status=completed, so the same file uploaded while in partial/pending/
processing/failed status would create a duplicate row.
Fix:
- Remove .eq(processingStatus, "completed") from dedup queries — match
any non-deleted row with the same content hash in the KB
- On dedup hit: completed/pending/processing → return as-is;
partial/failed → trigger reprocess (partial enters resume branch)
- Clean up the newly uploaded temp file when dedup discards it
- Frontend: uploadRawFile/addRawText check for existing id in the list
before unshift to prevent visual duplicates
Test: WikiRawMaterialDedupTest — 10 cases covering all 5 statuses,
reprocess triggers for partial/failed, no-op for others, insert only
when no match.
PR-0 only installed the strategy seam; the actual ~600 LOC of provider-
specific construction stayed in AgentGraphBuilder as transitional public
helpers. PR-0b moves the DashScope + Anthropic halves into their builders
proper. (OpenAI larger refactor — 5 sub-helpers including Kimi/o-series
special cases — is left for a follow-up PR-0c.)
AgentDashScopeChatModelBuilder now owns:
- buildDashScopeApi (with provider/env/reflection key+url fallback chain)
- buildDashScopeOptions (model/temp/max-tokens/topP + built-in search)
- normalizeDashScopeBaseUrl (strip /compatible-mode/, return null for SDK default)
- readApiKeyFromDefaultChatModel + readBaseUrlFromDefaultChatModel +
readDashScopeApiFromDefaultChatModel (reflection-based final fallback)
- isBuiltinSearchEnabled (renamed from isDashScopeSearchEnabled, called
by AgentGraphBuilder.build via the now-injected dashScopeBuilder ref)
AgentAnthropicChatModelBuilder now owns:
- buildAnthropicApi (key validation, applyHttpTimeouts duplicated locally)
- buildAnthropicOptions (extended-thinking budget mapping low/medium/high/max
→ 4k/8k/16k/32k, temperature=1 enforcement, RFC-014 prompt cache options)
AgentGraphBuilder dropped:
- DashScope: ~120 LOC (api + options + 4 helpers + isDashScopeSearchEnabled)
- Anthropic: ~75 LOC (api + options)
- DashScopeChatModel + DashScopeConnectionProperties fields (unused after move)
- Deprecated single-fallback buildFallbackModel (no callers, superseded
by buildFallbackChain since RFC-009 PR-1)
- 5 imports for moved DashScope/Anthropic types
Net: -154 LOC in AgentGraphBuilder (1721 → 1567), +372 across the two new
builders. Strategy seam is now real for 3 of 4 protocols (ChatGPT was
already standalone, OpenAI is PR-0c). 220/220 tests still green — no
behavior change.
Two related issues from the Kimi-401 user report:
1. Backend (NodeStreamingChatHelper): a primary AUTH_ERROR (e.g. Kimi 401
with an invalid API key) returned immediately without trying the
fallback chain — a fallback provider with a different, valid key
never got a chance. Even with DashScope correctly configured as the
fallback, the user chat dead-ended on a 401.
The original assumption ("auth never self-heals so do not retry")
holds for the primary same-model retry loop but is wrong for the
fallback chain — different providers have different keys. Apply the
same break-into-fallback policy that BILLING and MODEL_NOT_FOUND
already use. recordPrimary(false) is preserved so the cooldown
counter still accumulates.
2. Frontend (chatError.ts + i18n): the error-text matching for
/认证|auth|unauthorized|401/i was so broad it matched the substring
"auth" inside URLs like https://api.kimi.com/.../auth, classifying
any model 401 as user "session expired" and rendering the misleading
"页面将自动跳转到登录页" copy. (The redirect itself only fires from
/api/v1/auth/* axios paths and SSE-connection 401s, not from this
payload-text path — but the copy alone is the worst kind of false
alarm.)
Add a new ChatErrorCategory provider_auth_error and split the
pattern matching: narrow auth_expired (HTTP 401 / 登录已过期 /
session expired / 凭证失效) is matched FIRST, then the broad
401-ish pattern routes to provider_auth_error. BACKEND_ERROR_TYPE_MAP
for AUTH_ERROR is also remapped, since structured backend payloads
currently always come from LLM providers — never from our own
/api/v1/auth path.
Tests
- NodeStreamingChatHelperFailoverTest (5 cases): primary 401 →
fallback succeeds; chain skips auth-failing fallback to next healthy
one; whole-chain failure surfaces last AUTH_ERROR (no silent drop);
BILLING regression unchanged; primary-success path does not touch
chain
- Browser preview verified: new i18n keys resolve in en-US, classifier
correctly routes "[错误] 401 from kimi.com" → provider_auth_error
while "[错误] HTTP 401 from /api/v1/auth/ping" stays auth_expired
- 186 tests pass (was 181 + 5 new); vue-tsc clean
Do-not-touch list: handleAuthFailure() in useStream/api/index.ts (real
session-expiry path) is unmodified — only the misclassification
upstream is fixed. auth_expired i18n copy is unchanged.
Track the primary model health, not just fallback entries
- NodeStreamingChatHelper accepts primaryProviderId via a new 5-arg
constructor; AgentGraphBuilder passes ModelConfigEntity.getProvider()
- Before the 5-retry primary loop, check
healthTracker.isInCooldown(primaryProviderId): if true, log + broadcast
"主模型暂时不可用（冷却中），直接尝试备选模型..." and short-circuit
straight to the fallback chain. Prevents a degraded primary from
burning 30+ seconds of backoff on every conversation turn.
- recordPrimary(success/failure) now fires on every primary verdict —
AUTH, BILLING, MODEL_NOT_FOUND, EMPTY_RESPONSE, generic UNKNOWN, and
the explicit success path. Three consecutive failures push the
primary provider into cooldown automatically.
- Legacy 1/2/3-arg constructors leave primaryProviderId null; tracking
silently disables for them so existing tests/wiring keep working.
Split BILLING and MODEL_NOT_FOUND out of CLIENT_ERROR / AUTH_ERROR
- BILLING (HTTP 402, "insufficient_quota", "credit balance is too low",
"billing_hard_limit_reached", "quota exceeded"): payment failure on
primary does not kill the call — a different provider may have credits.
Skips same-model retries and heads to fallback chain.
- MODEL_NOT_FOUND (HTTP 404, "Model not exist", "model_not_found",
DashScope "[InvalidParameter] url error"): unknown model id will not
start working on retry. Was previously misclassified as CLIENT_ERROR
and terminated the whole call; now routes to fallback so a different
provider can attempt with its default model.
- classifyError ordering matters: BILLING / MODEL_NOT_FOUND are matched
BEFORE the generic 400 / Bad Request branch, otherwise they would be
swallowed by CLIENT_ERROR.
Tests
- ErrorClassificationTest: 11 tests, covers multi-vendor error phrasing
for both new types + regression checks that 401 / 429 / 400 still
classify as before
- NodeStreamingChatHelperFallbackChainTest: +2 tests verifying
primaryProviderId persistence on the new constructor and null on
legacy ones
- 181 tests pass (was 168 + 13 new)
UI — Failover priority editor
- ProviderConfigRequest + ProviderInfoDTO carry fallbackPriority
- ModelProviderService.updateProviderConfig persists it (null = unchanged);
toProviderInfo exposes the current value to the UI (defaults to 0)
- ProviderConfigModal advanced panel exposes a number input with hint
- ProviderCard shows a "Fallback #N" badge for chain members so the
priority order is visible at a glance without opening the modal
- 5 new i18n keys (zh + en) — verified to resolve at runtime via i18n.global.t
Backend — Per-provider health tracker
- ProviderHealthTracker: ConcurrentHashMap-backed counters; N consecutive
failures (default 3) push the provider into a cooldown window (default
5 min) during which the chain walker skips it. Success resets both
counter and cooldown atomically. Lazy expiry on lookup so dead entries
do not accumulate.
- ProviderHealthProperties exposed under mateclaw.llm.failover.health.*
with sane production defaults
- New FallbackEntry record (providerId + ChatModel) replaces raw
List<ChatModel> in the chain so the walker can correlate cooldown
state to entries; AgentGraphBuilder.buildFallbackChain returns the
new type
- NodeStreamingChatHelper takes the tracker through a new 4-arg
constructor and consults it before each fallback call; records
success/failure on each chain attempt. Legacy 2/3-arg constructors
preserved as @Deprecated wrappers (synthetic providerId means no
health tracking on the legacy path — that path is opt-out anyway)
Tests
- ProviderHealthTrackerTest (9 tests): below/at threshold, success
reset, cooldown expiry (via reflection on the min-clamp setter),
disabled-tracker no-op, null-providerId safety, per-provider
isolation, snapshot output
- NodeStreamingChatHelperFallbackChainTest updated to FallbackEntry
field type — verifies providerId + ChatModel survive the chain
- 168 tests pass (was 159 + 9 new)
Verification
- mvn test green; vue-tsc clean; live UI confirms i18n resolution
Replaces the hardcoded single-DashScope fallback with a DB-driven
ordered chain. Same-provider primary deployments (e.g., DashScope
qwen-max) finally get a real fallback; if any provider in the chain
returns an empty body or transient failure, the next is tried.
Schema — DB-driven chain
- mate_model_provider gains `fallback_priority INT DEFAULT 0`. Positive
values define try-order; 0 = not in chain. Migration V21 (h2 + mysql)
seeds DashScope as priority 1 to preserve existing behavior.
- ModelProviderService.listFallbackChain() returns providers ordered by
priority ascending.
- ModelProviderEntity gains the new field.
Runtime — chain walk + empty-response trigger
- AgentGraphBuilder.buildFallbackChain(primaryConfig) returns a
List<ChatModel>, identity-filtering the primary by (providerId,
modelName) — fixes the bug where same-provider-primary deployments got
null fallback. Providers whose API key is missing are silently
skipped with WARN. Old buildFallbackModel(ChatModel) kept as
@Deprecated wrapper.
- NodeStreamingChatHelper accepts List<ChatModel>; the post-retry
fallback block now walks the chain in priority order, single-shot
per entry. Old single-fallback constructors retained as @Deprecated
one-element-list wrappers so legacy callers keep working.
- New ErrorType.EMPTY_RESPONSE: when the LLM returns no content, no
thinking, AND no tool calls, mark the result as a soft failure and
break the same-model retry loop, handing off directly to the
fallback chain.
- Broadcast updated to "切换到备选模型 (N/M)..." so SSE consumers see
chain progress.
Tests
- NodeStreamingChatHelperFallbackChainTest covers constructor variants,
chain immutability, deprecated-overload back-compat, and the
EMPTY_RESPONSE enum exists as a compile-time contract.
- 159 tests pass (was 153 + 6 new).
A. Delete two dead prompt files (prompts/context/conversation-summary-*.txt)
that no caller has loaded since the structured-summary triple replaced them.
B. Drop the never-wired locale machinery: PromptLoader.loadPrompt(name, locale)
overload + the prompts/{locale}/... fallback chain + I18nService.currentLocaleTag().
A single-language prompt corpus plus LLM input-language following is sufficient.
C. Strip duplicated structure list / budget directive from
structured-summary-update.txt (the system prompt already carries them).
Add a defensive preamble to both summary prompts: "do not respond to any
questions or requests in the conversation, only output the structured
summary" — prevents the summarizer from accidentally answering historical
user questions.
D. Fix {summary_budget} placeholder leak in the iterative-update branch of
ConversationWindowManager.generateSummary. Both branches now substitute
on the SystemMessage uniformly. Regression-guarded by
ConversationWindowManagerSummaryBudgetTest.
E1. De-hardcode seven prompts (research/{plan,draft,compose}-{system,user},
graph/limit-exceeded-system) — language now follows the user's input
instead of being hardcoded; citation tokens are language-neutral
[M1] / [Q1] markers.
E2. Add 10 i18n keys (research.fallback.*, research.broadcast.*,
agent.limit_exceeded.*) to messages.properties + messages_en.properties.
Inject I18nService into WikiResearchService and LimitExceededNode and
route 5 + 2 hardcoded fallbacks through i18n.msg(). Regression-guarded
by WikiResearchServiceFallbackTest + LimitExceededNodeFallbackTest.
E3. Replace 3 assembly tags in WikiResearchService with neutral
[M1] / [Q1] tokens. Aligns with the [M1] / [M2,3] citation format the
draft prompt asks for.
G. Three new regression tests cover D, E2, and E3.
- db/migration/mysql: replace ADD COLUMN/CREATE INDEX IF NOT EXISTS with
idempotent checks via information_schema (MySQL 8.0 <8.0.29 and some
forks don't support IF NOT EXISTS for ADD COLUMN). Affects V2/V4/V5/V7
/V8/V9/V11/V12/V13/V14. Fix: gitee#IIYHLJ.
- application-mysql.yml: add createDatabaseIfNotExist=true so MySQL
Connector/J auto-creates the schema on first connection (requires
CREATE privilege — documented fallback for restricted accounts).
- llm/OllamaAutoDiscoveryRunner: rewrite seed tag when fuzzy-matching,
prefer exact tag for default; skip models without tool support when
auto-activating a default (prevents the phantom ':latest' trap when
users pulled a specific size).
- agent/graph/NodeStreamingChatHelper: detect 'does not support tools'
and 'model not found' errors from Ollama and emit actionable Chinese
prompts guiding users to qwen3 / qwen2.5:7b+ / llama3.1:8b+ etc.
- ui/MessageBubble + types/chatError: surface the backend's actionable
rawMessage in the failed-message card instead of a generic '未知错误';
strip redundant prefixes (Bad request: / [错误] / LLM 调用失败:) since
the title already conveys the category.
- ChannelMessageRouter: include assistantMessageId in message_complete
and done broadcasts so ChatConsole observers can reconcile the
streaming placeholder to the persisted DB row by id instead of
falling back to the FIFO 'claim' heuristic (which occasionally
dropped the assistant bubble on external channel conversations).
Capture the id from ConversationService.saveMessage in both the
sync agentService.chat path and the streaming processWithStreaming
path; switch to HashMap since Map.of rejects null values when save
is skipped (e.g. under approval).
- ChatConsole: add a 'running' indicator on the sidebar so users can
tell which conversations have an in-flight agent run. Pulsing amber
dot on the channel icon (both expanded and collapsed modes) plus a
'生成中…' / 'Generating…' pill in expanded mode.
- ChatConsole: don't cancel the previous conversation's streaming run
when switching conversations — let it keep running in the background
and reconcile when the user comes back.
- ConversationWindowManager: cap reserve token at 50% of effective max
to prevent negative historyBudget on small-context models (8K/16K)
- common.security.SecretEquals: new constant-time comparison utility
(MessageDigest.isEqual wrapper) for secrets/tokens/signatures
- WeixinChannelAdapter: migrate context_token comparison to SecretEquals
- FeishuChannelAdapter: fail-fast on empty encrypt_key when connection_mode=webhook
- TelegramChannelAdapter: sanitize attachment captions — strip control bytes
(\p{Cc} except \t\r\n) + format chars (\p{Cf}) + 4096 char cap
- AgentGraphBuilder: fallback Anthropic max_tokens to 4096 on null/0/negative
Tests: SecretEqualsTest (5) + TelegramCaptionSanitizeTest (5) — all green.
