15 — Reliability: QoS, crash, recovery, start/stop/resume

Status: implemented in 2.0.0

Tool and MCP error recovery

Tool and MCP failures are append-only, model-visible values rather than terminal exceptions. When a failure is plausibly correctable, Vak appends a recovery hint to that result so the next LLM turn can inspect the failure and retry with corrected arguments or choose an alternative capability. This is a bounded repair loop, not blind replay: the model must produce the new call, identical arguments must not be repeated, and every new call is authorized and executed through the normal broker.

Recovery hints are deliberately omitted for user cancellation, permission or approval denial, revocation, authentication, and rate-limit failures. Those conditions require an external state change or human decision. Existing turn limits, doom-loop detection, provider dispatch ceilings, and append-only session recording remain the upper bounds for the repair loop.

Per-run repair budget (escalating, system-authored)

A recovery hint alone is a nudge to the model; it is not a backstop. The loop tracks consecutive turns that end with an unresolved classifiable (vak_tools::ToolErrorKind::is_correctable()) tool failure and escalates:

1. First correctable failure this run — the per-call [recovery] hint is appended to the result (as above; the model still produces the next call).
2. Second consecutive turn failing on a correctable fault — the loop injects an authoritative, schema-resurfacing directive, a tagged ControlKind::RepairDirective message ([repair-directive] …) that no client shows as the person's words: it re-surfaces the exact admitted Tool::schema() for each rejected tool (or, for an unknown name, the full admitted inventory) so the repair is no longer a guess. This is issued by the runtime, not the model.
3. Third consecutive turn still failing — the budget is exhausted: the loop stops re-dispatching and returns a degraded, system-authored TurnOutcome::Completed that states the failure instead of inventing an answer. A Diagnostic activity is appended to the ledger (tool-repair-exhausted).

The budget is per Agent run (RepairState, reset on every run) and is general across all tool-error classes — it does not special-case the mcp broker; ToolErrorKind::classify is the single classification authority.

Outcome linkage (no false Produced)

The outcome gate (vak_core → vak_intent::evaluate_response_with_failures) inspects the latest turn: when it ends with classifiable correctable failures and zero successful receipts, the run is downgraded to Unknown rather than Produced — a non-empty fallback answer is not evidence. Combined with the budget above, a run that cannot repair its tool calls fails honest instead of signing off a fabricated result.

The standard failure matrix and where each case is handled.

Transient failures (QoS)

• Retry with backoff: every model step (connect + stream + collect) is retried up to max_retries (default 3) when the error is retryable (429 / 529 / network). Delay = retry_base_backoff_ms * 2^(n-1) with ±50% jitter, capped at 30s; server-advised Retry-After overrides. User aborts and partial-output aborts are never retried.
• Watchdog: each step runs under request_timeout (default 600s, request_timeout_secs = 0 disables). A hung stream becomes a retryable deadline error instead of hanging forever.
• Surfaced as AgentEvent::RetryScheduled{attempt, delay_ms, reason} so all UIs show the wait; config keys max_retries, retry_base_backoff_ms, request_timeout_secs.
• Run-level endurance (run_retry_attempts, default 6; run_retry_base_backoff_ms, default 2s): a fault window can outlast one step's retry budget — a sustained 429 window or slow/hung upstream used to kill the whole run on first step exhaustion (found in the live chaos campaign). Now, when a step exhausts its retries with a transient error (429/529/network/truncated stream), the loop backs off cancel-aware and re-attempts the same turn: nothing was committed to the ledger, so the re-attempt is exact and turn budgets are not consumed by infrastructure pain. An OPEN circuit breaker still fails fast — fresh evidence of a dead provider must not cost the user minutes of waiting. Permanent errors (auth/bad request/api) are never endured.
• Planner calls retry too (found in live testing): plan's model calls bypass the agent loop, so they carry their own bounded retry (3 attempts, exponential backoff, Retry-After honored, cancel-aware). A transient failure mid-planning no longer fails the whole run closed.
• Truncated SSE streams on OpenAI-compatible proxies: some free-tier endpoints end the body after the last content chunk without [DONE]/finish_reason. The openai-completions adapter treats a clean close with content as de facto completion (EndTurn); a close with no content still fails closed as Parse.

Crash & recovery

• Session ledgers are append-only JSONL — a crash mid-write loses at most the trailing partial line, which SessionLog::open skips; nothing else is rewritten. Model context is always derived from the log, so a crashed run resumes with full history.
• Checkpoint ledgers write via tmp-file + rename (atomic).
• Flow state ledgers persist after every node completion; flow run --resume replays only non-completed nodes.
• Terminal restore: TUI raw mode is restored by a Drop guard even on panic.

Start / stop / resume

Circuit breaker (cross-run QoS)

Per-step retries protect one run; the circuit breaker protects every run from a dead provider. Shared via Core across all runs of a process, with one state machine per provider endpoint and credential fingerprint:

• Only blind failures count: network loss, watchdog deadlines, truncated or malformed streams. Informed transience — 429 with Retry-After, explicit 503/529 overload — is the server saying "try again later"; it feeds endurance and must not open the circuit mid-window (found in the live chaos campaign: an opened breaker killed runs the window would have released seconds later).
• circuit_breaker_threshold consecutive failures (default 5) open the circuit; while open, steps fail fast with the remaining cooldown instead of burning their retry budget.
• After circuit_breaker_cooldown_secs (default 60) the circuit half-closes: one probe gets through, and any success resets the counter.
• A frozen ladder checks each leg's circuit independently, so an open primary fails fast while a healthy fallback can still serve. Run-level endurance uses normal cancel-aware backoff; each leg check prevents no-op calls until that leg's cooldown expires.

Config keys: circuit_breaker_threshold, circuit_breaker_cooldown_secs (0 cooldown disables opening).

Frozen route ladder (landed)

The frozen ladder is live: at session admission an ordered candidate ladder is computed (primary + warm-discovery fallbacks only — no invented ids, no network) and frozen INTO the contract header. Dispatch walks legs top-down on typed failure domains; the first dispatch of each next leg is receipted route-fallback and surfaced as a RouteFallback event. Ceiling, receipts, and endurance budget are shared across all legs, so walking the ladder is contract execution, never mid-contract switching (invariant 7 above carries the new wording).

Phase R (vakrouter adoption) upgrades the ordering machinery:

• Attribution is real: every attempt records the (provider, model) leg that actually served or failed; evidence rows land keyed correctly in routing-evidence.jsonl with true p50 latency.
• Demand-scored objectives (order_ladder_v2): request difficulty picks utility/balanced/quality-critical ordering; [route].objective overrides; [route].quality_hints replaces hardcoded model-name bands (invariant 9). v1 stays only for replaying old contracts. The difficulty facts themselves come from the intent kernel (docs/design/47-commitment-kernel.md): reasoning_required, evidence_required and structured_output are derived from the turn's reading. Until that landed, plan_route_ladder passed all three as false with estimated_input_tokens: 0, so every session scored identical demand and this ordering function never actually varied. A turn may also restrict the frozen ladder to a prefix — never reordering, never extending — which keeps dispatch inside the committed contract and leaves replay exact.
• Cross-model fallbacks are opt-in: [route].fallback_models allowlist ∩ warm discovery; the user's primary never loses the head position.
• Diversity caps + annotations: ⌈max_total/3⌉ seats per provider; thin-chain/dominant-domain/unreachable warnings frozen into the header, visible in TUI introspection.
• Beliefs: domain-weighted doubt demotes flaky legs below trusted peers until one success clears them; governance failures are not evidence.

Evidence rows land in routing-evidence.jsonl; unknown settlements shrink confidence without punishing direction. FinOps attribution follows the serving leg per dispatch (CostRow.provider).

Invariants

• Retry never changes the frozen contract: same model, same request body.
• Retries are unbounded by wall-clock but bounded by count and cancel token.
• A denied/failed tool result is data, not an exception — the loop continues; only provider-step exhaustion or required-node failure ends a run.

Network events (docs/design/31-network-resilience.md)

§Turn-Level Routing

Status: shipped.

Problem with session-frozen ladders

The original Phase R design froze provider, model, and route_ladder into the FrozenContract at session creation (start_session_with_route_for). Every subsequent turn in that session read the frozen values unconditionally, which caused:

• A user changing provider in Settings had zero effect on open sessions.
• The auto-routing algorithm (plan_route_ladder: evidence, beliefs, v2 ordering, demand-scoring) ran exactly once at admission, then was inert for the session's life.
• If a provider failed mid-session the frozen ladder was always tried first, even if the user had switched away from it.
• Workers inherited the parent session's frozen route rather than the current effective route at spawn time.

Design change

Route planning is now per-turn, not per-session.

run_turn_inner (vak-core/src/lib.rs) now:

1. Resolves (provider, model) from Core::effective_route() — always the live operator selection, never the session header's initial snapshot.
2. Calls Core::plan_route_ladder(turn_primary_leg, Some(engagement.posture.demand)) after intent resolution to assemble a fresh ladder using:
   • Current evidence ledger (routing-evidence.jsonl, 30-day TTL)
   • Session belief state (domain-weighted doubt, clears on success)
   • Warm discovery cache (TTL 5 min)
   • Demand facts from this turn's intent reading (not hardcoded constants)
3. Uses the fresh turn_plan.ladder to populate cfg.ladder (fallback legs), not session_contract.route_ladder.

What the FrozenContract still governs (immutable)

What changed to initial snapshot only

Audit preservation

WorkReceipt records the exact provider, model, and attempt outcome for every dispatch, per turn. Per-turn audit is strictly richer than the previous session-level frozen record — you can now reconstruct which provider handled turn 3 even if the user switched providers between turn 2 and turn 3.

Affected call sites

• vak-core/src/lib.rs — run_turn_inner: provider/model resolution + ladder assembly
• vak-agent/src/lib.rs — run_turn, run_judge, write_handoff, managed-work authoring: all read self.config.model (set per-turn by run_turn_inner)
• vak-server/src/lib.rs — configuration_mismatch field: always false (concept retired)
• vak-server/src/admin.rs — stale_reasons: provider_changed/model_changed removed
• vak-server/src/gateway.rs — session_matches_route: no longer checks initial contract
• vak-server/src/agent_chats.rs — candidate sessions no longer filtered by initial route
• vak-session/src/types.rs — FrozenContract doc comments updated
• AGENTS.md — invariant 7 updated