Apply a Reactor inter-frame idle timeout at the streaming chat chokepoint so half-open provider body streams surface through the normal retry/failover path. Also keep the HTTP timeout documentation accurate and cover the behavior with focused tests.
Strict OpenAI-compatible providers reject the /chat/completions request
with HTTP 400 when an assistant message in history carries a tool call
whose function.arguments is not parseable JSON. Normalize blank or
non-JSON arguments to "{}" at the send chokepoint so streaming,
history-replay, and older-persisted tool calls all stay well-formed.
- goal: continue (not skip) on max-iterations and evidence-insufficient turns.
A max-iterations turn grants a fresh iteration budget ("hard continuation"),
bounded per run and sized into the graph recursion ceiling, so a task too big
for one budget keeps going instead of stalling until the next user message.
- plan-execute: re-plan the remaining work on a step exception, and on a
signature-based stall (repeated failures / identical results / no usable
result) instead of advancing dependent steps with junk; bounded by a per-run
re-plan cap, with a graduated change-strategy nudge before the hard stop.
- plan-execute: auto-derive a goal from a genuine multi-step plan, seeding the
acceptance criteria from the plan steps, so the goal subsystem engages without
the model calling setGoal; broadcast goal_created so the UI hydrates.
- react: refund the iteration for setup-only rounds (load_skill / enable_tool)
so a tight budget is not eaten by the load-then-use two-step.
- ui: re-fetch the active goal when a turn finishes so a goal created or mutated
mid-conversation surfaces without depending on an SSE event.
- streaming: make retry backoff / total-time budget instance fields with a
test-only seam; clarify that the wall-clock budget (not max-retries) bounds a
sustained SERVER_ERROR loop to ~8 attempts, fixing the slow/flaky retry test.
Add ErrorClassificationTest regression cases for the AI-gateway retry
hardening: 5xx-before-4xx ordering (a proxy 502 whose body says
"bad request" stays retryable), Chinese / numeric provider billing
patterns, and DNS / TLS infrastructure-fatal detection.
Fix the cert-trust pattern while adding its test: Java's ValidatorException
emits "PKIX path building failed" with an uppercase PKIX and the error
chain is not lower-cased, so the previous lowercase pattern never matched
— an untrusted/expired cert chain fell through to the retryable
SERVER_ERROR bucket and was retried in vain instead of failing over.
Some OpenAI-compatible providers (LM Studio's built-in server, certain
strict-mode vLLM / SGLang deployments) reject 400 "System message must
be at the beginning" when SystemMessages appear after user / assistant
/ tool messages. The reasoning loop currently emits four SystemMessage
segments — main prompt at index 0, skill catalog inserted at index 1,
progress-ledger snapshot and stale-reminder appended at the end of
nonHistoryPrefix after the runtime-context UserMessage. The latter two
violate the strict shape, so conversations on LM Studio 400 on the
first turn (reported in #218).
Add MessageNormalizer: collects every SystemMessage in the outbound
prompt regardless of position, joins their text with a blank-line
separator, and emits a single SystemMessage at index 0. Non-system
messages keep their relative order, so AssistantMessage(tool_calls) ↔
ToolResponseMessage adjacency is preserved verbatim (required by strict
pair validators).
Wire it into doStreamCall as the first pre-egress step so every node
(reasoning, step-execution, summarizing, plan-generation, limit-exceeded)
inherits the fix without per-node changes, and any future node that
emits multiple SystemMessages stays compliant.
The transformation is semantically equivalent on permissive providers
(OpenAI, DashScope, Ollama, DeepSeek, Kimi, Doubao, GLM) — the merged
token sequence matches what they would have seen across N SystemMessages
— and safe on non-OpenAI protocols (Anthropic, Vertex / Gemini), whose
adapters already extract SystemMessages into a top-level system field
and receive an identical payload.
Kill switch: -Dmateclaw.llm.message-normalizer.enabled=false reverts to
the prior behavior for emergency rollback.
Tests: 11 unit tests on MessageNormalizer cover empty / no-system /
canonical / mid-list / tail / blanks / tool-pair preservation / Prompt
option-reference preservation / kill switch. 1 wiring test pins the
call site in doStreamCall. Full vip.mate.agent.** suite (504 tests)
stays green.
Closes#218.
Some providers (notably SiliconFlow) return "network connection error" in the response body when their backend is overloaded or the upstream model connection is disrupted. classifyError() had no pattern for this string, so it fell through to UNKNOWN (non-retryable), surfacing the raw error to the user on the first failure instead of running the exponential-backoff recovery. Adds the pattern to the SERVER_ERROR classifier and a friendly message mapping in extractUserFriendlyError(); bumps MAX_RETRIES from 5 to 10 so sustained wiki batch load can ride out provider flaps without surfacing an error to the channel user.
Closes#178
- Generalize the OpenAI-compatible chat/models path resolver so any
baseUrl ending in /v{N} (Ark /v3, Zhipu /v4, ...) drops the duplicate
/v1 prefix. Volcano Engine test-connection and chat were posting to
/api/v3/v1/chat/completions and getting 404.
- Replace the six pre-seeded Doubao alias rows (doubao-1.5-*) with five
valid Ark direct-call ids (doubao-seed-1-8-251228 etc.) and flip
support_model_discovery=TRUE so users can refresh their account's
actual catalog. Aliases were marketing names, not API names, so every
call hit InvalidEndpointOrModel.NotFound.
- Translate Ark business errors into actionable Chinese hints: include
the response body in the error chain, match ModelNotOpen and
InvalidEndpointOrModel codes, extract the offending model id, and
classify them as MODEL_NOT_FOUND so failover skips retries.
A bundle of stability fixes that all surfaced together while running
the same long-form generation task across multiple turns. Each one
addresses a distinct way the previous behavior silently dropped
content the user had already seen on screen.
1. Mid-turn narrative persistence (StateGraphReActAgent +
SummarizingNode). Intermediate ReasoningNode rounds and
SummarizingNode broadcast their content_delta directly to the
SSE channel for live display, but the StreamAccumulator only
received the final answer. After refresh the assistant message
showed only tool_call cards with no body text.
StateGraphReActAgent now also forwards STREAMED_CONTENT (already
set per round) as a persistOnly StreamDelta whenever it changes,
so every narrative chunk lands in the accumulator's content
buffer and gets written to mate_message. SummarizingNode now
writes its summary into the same key so summarize narratives
persist too.
2. Follow-up message queue, not dispose (ChatController#interruptStream).
Sending a new message while a turn was running called
requestInterrupt, which dispose()d the active Reactor chain mid
LLM call. That cancelled the in-flight generation, lost partial
tokens, and left the user staring at a half-finished bubble.
The endpoint now uses enqueueMessage in all paths, matching
the "wait for current turn, then run" behavior. The old
requestInterrupt API is kept for any future force-replace UI
but no caller routes to it.
3. Queued user message ordering (ChatStreamTracker.QueuedInput +
ChatController.startQueuedMessage). interruptStream used to save
the queued user message immediately, before the in-flight
assistant message finalized in doOnError. listMessages orders
by create_time ASC, so the queued user message ended up above
the assistant reply it was supposed to follow. QueuedInput now
carries contentParts; persistence is delayed to startQueuedMessage,
which runs only after Asst-N is on disk.
4. JVM shutdown flush (ChatStreamTracker @PreDestroy +
emergencySaveAccumulator). A mvn spring-boot:run restart used to
wipe in-flight turns: SSE emitter timed out, ShutdownHook fired,
HikariPool closed before doOnError could save. ChatStreamTracker
now exposes an emergency-save callback per RunState; ChatController
registers one per stream that snapshots the accumulator and
writes status="interrupted_shutdown". @PreDestroy walks active
runs, invokes the callback, then disposes. Spring's reverse-order
bean teardown keeps ConversationService and Hikari alive long
enough for the save to complete.
5. Observation thresholds for summarize (GraphObservationProperties +
application.yml). The previous total-chars threshold of 12 KB
triggered summarize after one or two RFC reads, costing a 40 to
80 second compaction LLM call per loop. Tuned to: total 200 KB,
single 16 KB, large-result 32 KB, rounds safety net 25. Java
field defaults reverted to the conservative original values so
application.yml stays the source of truth.
6. Frontend thinking segmentation (useChat.ts thinking_delta +
phase). Multi-round ReAct turns merged every reasoning + summarize
round's thinking into one segment, accumulating to 9 KB+ in a
single bubble. thinking_delta now uses findLast(running) so a
tool_call_started or phase transition closes the previous segment
and the next delta opens a fresh one. phase event also closes
running thinking/content segments.
7. Other small things bundled: removed a debug metadata-keys log
that flooded the log file with one line per stream chunk; fixed
three stale tests that didn't compile after earlier constructor
changes (WikiLogServiceTest, WikiOverviewSpliceTest,
WikiProcessingServiceLazyTest); added rfc-066 documenting the
unified message queue + priority refactor as the next logical
step on top of these stabilizations.
Verified end-to-end with multiple full sessions: a four-minute
generation that produced the expected docx and a follow-up enqueue
that ran cleanly after the previous turn naturally completed,
without the old "Disposable unavailable" interrupt path.
A series of cross-cutting stability fixes that surfaced together
during a long debugging session.
reasoning_content / Claude prefill self-replicating 400:
- ChatController persists typed errors (content starts with '[错误] ')
with status='error', so the failure text stops being re-sent as
multi-turn context — DeepSeek thinking 400 ('reasoning_content
must be passed back') and Claude 400 ('does not support assistant
message prefill') used to recursively re-create themselves every
retry by polluting history.
- BaseAgent.sanitizeForLlm filters status='error' / '[错误] ' prefix
assistant messages from history before LLM dispatch.
- BaseAgent.fetchHistoryMessages defensively drops trailing
AssistantMessages — Claude rejects assistant-tail prompts.
- NodeStreamingChatHelper.dropTrailingAssistant runs the same
defense at every doStreamCall pre-egress, so the in-turn
summarizing→reasoning transition (which leaves an assistant
scaffold at the tail) doesn't trip Claude either.
- AgentGraphBuilder.FallbackPolicy.DEEPSEEK switched (null,true,true)
→ (' ',false,true), aligning with KIMI/OPENAI's tolerant ' '
fallback. The previous 'force explicit 400' design was the
self-replicating loop's prime mover.
narration + tool args truncation:
- ReasoningNode.DEFAULT_MAX_OUTPUT_TOKENS 4096 → 16384. The 4k cap
was decapitating renderDocx tool_call args mid-stream when the
model emitted a long content field on top of thinking content;
the resulting 'invalid JSON' aborted execution silently.
- ReasoningNode appends a hermes-style TOOL_USE_ENFORCEMENT clause
to every system prompt: 'when you say you will perform an action,
call the tool now in the same response — narration is a protocol
violation'. Treats 'now I will generate the docx' (and never
actually calling renderDocx) as a forbidden pattern.
- ToolExecutionExecutor.normalizeToolExecutionError reframes the
JSON-truncated error as actionable instructions: 're-call the
same tool now with shorter content or split into multiple
sequential calls; do NOT describe the result as text'.
side fixes from the same evening:
- ChatController doOnComplete skips completionPublisher.publish
when isError=true, keeping memory extraction off the garbage path.
- ChatController doOnComplete queued-message guard simplified to
'cr.queuedInput() != null', matching the other 4 sites in the
controller. The previous 'isInterruptFollowup || !wasStopped'
guard silently dropped queued messages when the user did
Stop-then-Enqueue (wasStopped=true && interruptType=null), losing
the freshly-typed follow-up message.
- prompts/graph/summarize-system.txt now distinguishes 'single
task' (default; output one cohesive summary) from 'multiple
independent sub-tasks' (use the子任务 N format). Stops the
summarizer from inventing '子任务 1: PRO-027' decomposition for
unitary requests like 'write me a project proposal'.
Two related issues from the Kimi-401 user report:
1. Backend (NodeStreamingChatHelper): a primary AUTH_ERROR (e.g. Kimi 401
with an invalid API key) returned immediately without trying the
fallback chain — a fallback provider with a different, valid key
never got a chance. Even with DashScope correctly configured as the
fallback, the user chat dead-ended on a 401.
The original assumption ("auth never self-heals so do not retry")
holds for the primary same-model retry loop but is wrong for the
fallback chain — different providers have different keys. Apply the
same break-into-fallback policy that BILLING and MODEL_NOT_FOUND
already use. recordPrimary(false) is preserved so the cooldown
counter still accumulates.
2. Frontend (chatError.ts + i18n): the error-text matching for
/认证|auth|unauthorized|401/i was so broad it matched the substring
"auth" inside URLs like https://api.kimi.com/.../auth, classifying
any model 401 as user "session expired" and rendering the misleading
"页面将自动跳转到登录页" copy. (The redirect itself only fires from
/api/v1/auth/* axios paths and SSE-connection 401s, not from this
payload-text path — but the copy alone is the worst kind of false
alarm.)
Add a new ChatErrorCategory provider_auth_error and split the
pattern matching: narrow auth_expired (HTTP 401 / 登录已过期 /
session expired / 凭证失效) is matched FIRST, then the broad
401-ish pattern routes to provider_auth_error. BACKEND_ERROR_TYPE_MAP
for AUTH_ERROR is also remapped, since structured backend payloads
currently always come from LLM providers — never from our own
/api/v1/auth path.
Tests
- NodeStreamingChatHelperFailoverTest (5 cases): primary 401 →
fallback succeeds; chain skips auth-failing fallback to next healthy
one; whole-chain failure surfaces last AUTH_ERROR (no silent drop);
BILLING regression unchanged; primary-success path does not touch
chain
- Browser preview verified: new i18n keys resolve in en-US, classifier
correctly routes "[错误] 401 from kimi.com" → provider_auth_error
while "[错误] HTTP 401 from /api/v1/auth/ping" stays auth_expired
- 186 tests pass (was 181 + 5 new); vue-tsc clean
Do-not-touch list: handleAuthFailure() in useStream/api/index.ts (real
session-expiry path) is unmodified — only the misclassification
upstream is fixed. auth_expired i18n copy is unchanged.
Track the primary model health, not just fallback entries
- NodeStreamingChatHelper accepts primaryProviderId via a new 5-arg
constructor; AgentGraphBuilder passes ModelConfigEntity.getProvider()
- Before the 5-retry primary loop, check
healthTracker.isInCooldown(primaryProviderId): if true, log + broadcast
"主模型暂时不可用（冷却中），直接尝试备选模型..." and short-circuit
straight to the fallback chain. Prevents a degraded primary from
burning 30+ seconds of backoff on every conversation turn.
- recordPrimary(success/failure) now fires on every primary verdict —
AUTH, BILLING, MODEL_NOT_FOUND, EMPTY_RESPONSE, generic UNKNOWN, and
the explicit success path. Three consecutive failures push the
primary provider into cooldown automatically.
- Legacy 1/2/3-arg constructors leave primaryProviderId null; tracking
silently disables for them so existing tests/wiring keep working.
Split BILLING and MODEL_NOT_FOUND out of CLIENT_ERROR / AUTH_ERROR
- BILLING (HTTP 402, "insufficient_quota", "credit balance is too low",
"billing_hard_limit_reached", "quota exceeded"): payment failure on
primary does not kill the call — a different provider may have credits.
Skips same-model retries and heads to fallback chain.
- MODEL_NOT_FOUND (HTTP 404, "Model not exist", "model_not_found",
DashScope "[InvalidParameter] url error"): unknown model id will not
start working on retry. Was previously misclassified as CLIENT_ERROR
and terminated the whole call; now routes to fallback so a different
provider can attempt with its default model.
- classifyError ordering matters: BILLING / MODEL_NOT_FOUND are matched
BEFORE the generic 400 / Bad Request branch, otherwise they would be
swallowed by CLIENT_ERROR.
Tests
- ErrorClassificationTest: 11 tests, covers multi-vendor error phrasing
for both new types + regression checks that 401 / 429 / 400 still
classify as before
- NodeStreamingChatHelperFallbackChainTest: +2 tests verifying
primaryProviderId persistence on the new constructor and null on
legacy ones
- 181 tests pass (was 168 + 13 new)
UI — Failover priority editor
- ProviderConfigRequest + ProviderInfoDTO carry fallbackPriority
- ModelProviderService.updateProviderConfig persists it (null = unchanged);
toProviderInfo exposes the current value to the UI (defaults to 0)
- ProviderConfigModal advanced panel exposes a number input with hint
- ProviderCard shows a "Fallback #N" badge for chain members so the
priority order is visible at a glance without opening the modal
- 5 new i18n keys (zh + en) — verified to resolve at runtime via i18n.global.t
Backend — Per-provider health tracker
- ProviderHealthTracker: ConcurrentHashMap-backed counters; N consecutive
failures (default 3) push the provider into a cooldown window (default
5 min) during which the chain walker skips it. Success resets both
counter and cooldown atomically. Lazy expiry on lookup so dead entries
do not accumulate.
- ProviderHealthProperties exposed under mateclaw.llm.failover.health.*
with sane production defaults
- New FallbackEntry record (providerId + ChatModel) replaces raw
List<ChatModel> in the chain so the walker can correlate cooldown
state to entries; AgentGraphBuilder.buildFallbackChain returns the
new type
- NodeStreamingChatHelper takes the tracker through a new 4-arg
constructor and consults it before each fallback call; records
success/failure on each chain attempt. Legacy 2/3-arg constructors
preserved as @Deprecated wrappers (synthetic providerId means no
health tracking on the legacy path — that path is opt-out anyway)
Tests
- ProviderHealthTrackerTest (9 tests): below/at threshold, success
reset, cooldown expiry (via reflection on the min-clamp setter),
disabled-tracker no-op, null-providerId safety, per-provider
isolation, snapshot output
- NodeStreamingChatHelperFallbackChainTest updated to FallbackEntry
field type — verifies providerId + ChatModel survive the chain
- 168 tests pass (was 159 + 9 new)
Verification
- mvn test green; vue-tsc clean; live UI confirms i18n resolution
Replaces the hardcoded single-DashScope fallback with a DB-driven
ordered chain. Same-provider primary deployments (e.g., DashScope
qwen-max) finally get a real fallback; if any provider in the chain
returns an empty body or transient failure, the next is tried.
Schema — DB-driven chain
- mate_model_provider gains `fallback_priority INT DEFAULT 0`. Positive
values define try-order; 0 = not in chain. Migration V21 (h2 + mysql)
seeds DashScope as priority 1 to preserve existing behavior.
- ModelProviderService.listFallbackChain() returns providers ordered by
priority ascending.
- ModelProviderEntity gains the new field.
Runtime — chain walk + empty-response trigger
- AgentGraphBuilder.buildFallbackChain(primaryConfig) returns a
List<ChatModel>, identity-filtering the primary by (providerId,
modelName) — fixes the bug where same-provider-primary deployments got
null fallback. Providers whose API key is missing are silently
skipped with WARN. Old buildFallbackModel(ChatModel) kept as
@Deprecated wrapper.
- NodeStreamingChatHelper accepts List<ChatModel>; the post-retry
fallback block now walks the chain in priority order, single-shot
per entry. Old single-fallback constructors retained as @Deprecated
one-element-list wrappers so legacy callers keep working.
- New ErrorType.EMPTY_RESPONSE: when the LLM returns no content, no
thinking, AND no tool calls, mark the result as a soft failure and
break the same-model retry loop, handing off directly to the
fallback chain.
- Broadcast updated to "切换到备选模型 (N/M)..." so SSE consumers see
chain progress.
Tests
- NodeStreamingChatHelperFallbackChainTest covers constructor variants,
chain immutability, deprecated-overload back-compat, and the
EMPTY_RESPONSE enum exists as a compile-time contract.
- 159 tests pass (was 153 + 6 new).
- db/migration/mysql: replace ADD COLUMN/CREATE INDEX IF NOT EXISTS with
idempotent checks via information_schema (MySQL 8.0 <8.0.29 and some
forks don't support IF NOT EXISTS for ADD COLUMN). Affects V2/V4/V5/V7
/V8/V9/V11/V12/V13/V14. Fix: gitee#IIYHLJ.
- application-mysql.yml: add createDatabaseIfNotExist=true so MySQL
Connector/J auto-creates the schema on first connection (requires
CREATE privilege — documented fallback for restricted accounts).
- llm/OllamaAutoDiscoveryRunner: rewrite seed tag when fuzzy-matching,
prefer exact tag for default; skip models without tool support when
auto-activating a default (prevents the phantom ':latest' trap when
users pulled a specific size).
- agent/graph/NodeStreamingChatHelper: detect 'does not support tools'
and 'model not found' errors from Ollama and emit actionable Chinese
prompts guiding users to qwen3 / qwen2.5:7b+ / llama3.1:8b+ etc.
- ui/MessageBubble + types/chatError: surface the backend's actionable
rawMessage in the failed-message card instead of a generic '未知错误';
strip redundant prefixes (Bad request: / [错误] / LLM 调用失败:) since
the title already conveys the category.
Full-stack AI assistant built on Spring AI Alibaba.
Features: ReAct Agent, Plan-and-Execute, MCP Protocol, Multi-Model, Multi-Channel.
Apache-2.0 License
