Two real bugs the user restart surfaced — both turned healthy providers
into HARD-removed false positives.
Bug #1 — URL duplication
OpenAiCompatibleListModelsProbe always concatenated /v1/models, so
providers whose Base URL already includes the version segment got the
wrong URL:
LMStudio http://localhost:1234/v1 → /v1/v1/models → 404
ZhipuAI .../api/paas/v4 → /v4/v1/models → 404
Fix: detect a trailing /vN suffix and append /models instead. Six unit
tests in OpenAiCompatibleListModelsProbeTest lock the rule down.
Bug #2 — 404 false positives
Kimi for Coding API does not expose /v1/models even though chat works
fine, so the probe correctly received a 404 and incorrectly HARD-removed
the provider from the pool. Other vendors will hit the same — listing
is not a universal contract.
Fix: classify HTTP responses semantically.
401 / 403 → HARD remove (real auth failure)
404 / 405 / 410 → fail-open (endpoint missing, server may be alive)
other 4xx / 5xx → fail-open (probe inconclusive — let chat decide)
network errors → fail (unreachable)
This is the same philosophy as ChatGPTOAuthStatusProbe: when we cannot
cheaply confirm health, we do not proactively penalize the provider.
Same logic applied to Anthropic + DashScope probes for consistency.
Net effect on the user deployment after restart:
- kimi-code stays in pool (404 → fail-open) → primary path works again
- lmstudio + zhipu-cn also stay in pool (URL bug fixed)
- dashscope + ollama unchanged (real 200 OK)
Tests: 6 new for resolveModelsPath. The 2 unrelated WikiRawMaterialDedupTest
failures pre-date this commit and live in ba86bea.
Two related issues from the Kimi-401 user report:
1. Backend (NodeStreamingChatHelper): a primary AUTH_ERROR (e.g. Kimi 401
with an invalid API key) returned immediately without trying the
fallback chain — a fallback provider with a different, valid key
never got a chance. Even with DashScope correctly configured as the
fallback, the user chat dead-ended on a 401.
The original assumption ("auth never self-heals so do not retry")
holds for the primary same-model retry loop but is wrong for the
fallback chain — different providers have different keys. Apply the
same break-into-fallback policy that BILLING and MODEL_NOT_FOUND
already use. recordPrimary(false) is preserved so the cooldown
counter still accumulates.
2. Frontend (chatError.ts + i18n): the error-text matching for
/认证|auth|unauthorized|401/i was so broad it matched the substring
"auth" inside URLs like https://api.kimi.com/.../auth, classifying
any model 401 as user "session expired" and rendering the misleading
"页面将自动跳转到登录页" copy. (The redirect itself only fires from
/api/v1/auth/* axios paths and SSE-connection 401s, not from this
payload-text path — but the copy alone is the worst kind of false
alarm.)
Add a new ChatErrorCategory provider_auth_error and split the
pattern matching: narrow auth_expired (HTTP 401 / 登录已过期 /
session expired / 凭证失效) is matched FIRST, then the broad
401-ish pattern routes to provider_auth_error. BACKEND_ERROR_TYPE_MAP
for AUTH_ERROR is also remapped, since structured backend payloads
currently always come from LLM providers — never from our own
/api/v1/auth path.
Tests
- NodeStreamingChatHelperFailoverTest (5 cases): primary 401 →
fallback succeeds; chain skips auth-failing fallback to next healthy
one; whole-chain failure surfaces last AUTH_ERROR (no silent drop);
BILLING regression unchanged; primary-success path does not touch
chain
- Browser preview verified: new i18n keys resolve in en-US, classifier
correctly routes "[错误] 401 from kimi.com" → provider_auth_error
while "[错误] HTTP 401 from /api/v1/auth/ping" stays auth_expired
- 186 tests pass (was 181 + 5 new); vue-tsc clean
Do-not-touch list: handleAuthFailure() in useStream/api/index.ts (real
session-expiry path) is unmodified — only the misclassification
upstream is fixed. auth_expired i18n copy is unchanged.
UI — Failover priority editor
- ProviderConfigRequest + ProviderInfoDTO carry fallbackPriority
- ModelProviderService.updateProviderConfig persists it (null = unchanged);
toProviderInfo exposes the current value to the UI (defaults to 0)
- ProviderConfigModal advanced panel exposes a number input with hint
- ProviderCard shows a "Fallback #N" badge for chain members so the
priority order is visible at a glance without opening the modal
- 5 new i18n keys (zh + en) — verified to resolve at runtime via i18n.global.t
Backend — Per-provider health tracker
- ProviderHealthTracker: ConcurrentHashMap-backed counters; N consecutive
failures (default 3) push the provider into a cooldown window (default
5 min) during which the chain walker skips it. Success resets both
counter and cooldown atomically. Lazy expiry on lookup so dead entries
do not accumulate.
- ProviderHealthProperties exposed under mateclaw.llm.failover.health.*
with sane production defaults
- New FallbackEntry record (providerId + ChatModel) replaces raw
List<ChatModel> in the chain so the walker can correlate cooldown
state to entries; AgentGraphBuilder.buildFallbackChain returns the
new type
- NodeStreamingChatHelper takes the tracker through a new 4-arg
constructor and consults it before each fallback call; records
success/failure on each chain attempt. Legacy 2/3-arg constructors
preserved as @Deprecated wrappers (synthetic providerId means no
health tracking on the legacy path — that path is opt-out anyway)
Tests
- ProviderHealthTrackerTest (9 tests): below/at threshold, success
reset, cooldown expiry (via reflection on the min-clamp setter),
disabled-tracker no-op, null-providerId safety, per-provider
isolation, snapshot output
- NodeStreamingChatHelperFallbackChainTest updated to FallbackEntry
field type — verifies providerId + ChatModel survive the chain
- 168 tests pass (was 159 + 9 new)
Verification
- mvn test green; vue-tsc clean; live UI confirms i18n resolution
Replaces the hardcoded single-DashScope fallback with a DB-driven
ordered chain. Same-provider primary deployments (e.g., DashScope
qwen-max) finally get a real fallback; if any provider in the chain
returns an empty body or transient failure, the next is tried.
Schema — DB-driven chain
- mate_model_provider gains `fallback_priority INT DEFAULT 0`. Positive
values define try-order; 0 = not in chain. Migration V21 (h2 + mysql)
seeds DashScope as priority 1 to preserve existing behavior.
- ModelProviderService.listFallbackChain() returns providers ordered by
priority ascending.
- ModelProviderEntity gains the new field.
Runtime — chain walk + empty-response trigger
- AgentGraphBuilder.buildFallbackChain(primaryConfig) returns a
List<ChatModel>, identity-filtering the primary by (providerId,
modelName) — fixes the bug where same-provider-primary deployments got
null fallback. Providers whose API key is missing are silently
skipped with WARN. Old buildFallbackModel(ChatModel) kept as
@Deprecated wrapper.
- NodeStreamingChatHelper accepts List<ChatModel>; the post-retry
fallback block now walks the chain in priority order, single-shot
per entry. Old single-fallback constructors retained as @Deprecated
one-element-list wrappers so legacy callers keep working.
- New ErrorType.EMPTY_RESPONSE: when the LLM returns no content, no
thinking, AND no tool calls, mark the result as a soft failure and
break the same-model retry loop, handing off directly to the
fallback chain.
- Broadcast updated to "切换到备选模型 (N/M)..." so SSE consumers see
chain progress.
Tests
- NodeStreamingChatHelperFallbackChainTest covers constructor variants,
chain immutability, deprecated-overload back-compat, and the
EMPTY_RESPONSE enum exists as a compile-time contract.
- 159 tests pass (was 153 + 6 new).
- db/migration/mysql: replace ADD COLUMN/CREATE INDEX IF NOT EXISTS with
idempotent checks via information_schema (MySQL 8.0 <8.0.29 and some
forks don't support IF NOT EXISTS for ADD COLUMN). Affects V2/V4/V5/V7
/V8/V9/V11/V12/V13/V14. Fix: gitee#IIYHLJ.
- application-mysql.yml: add createDatabaseIfNotExist=true so MySQL
Connector/J auto-creates the schema on first connection (requires
CREATE privilege — documented fallback for restricted accounts).
- llm/OllamaAutoDiscoveryRunner: rewrite seed tag when fuzzy-matching,
prefer exact tag for default; skip models without tool support when
auto-activating a default (prevents the phantom ':latest' trap when
users pulled a specific size).
- agent/graph/NodeStreamingChatHelper: detect 'does not support tools'
and 'model not found' errors from Ollama and emit actionable Chinese
prompts guiding users to qwen3 / qwen2.5:7b+ / llama3.1:8b+ etc.
- ui/MessageBubble + types/chatError: surface the backend's actionable
rawMessage in the failed-message card instead of a generic '未知错误';
strip redundant prefixes (Bad request: / [错误] / LLM 调用失败:) since
the title already conveys the category.
Full-stack AI assistant built on Spring AI Alibaba.
Features: ReAct Agent, Plan-and-Execute, MCP Protocol, Multi-Model, Multi-Channel.
Apache-2.0 License
