1 Critical Finding 2 Cascading Effects ARK Diagnostic Report

OpenClaw #114255 — Restart Mid-Run Leaves `status: "running"` Forever, and the Spool Retries a Poison Message Until the Lane Dies

Generated by ARK (Agent Reliability Kit) · 2026-07-28 · Target: OpenClaw 2026.7.2-beta.5, Docker/Linux, Telegram channel · Original issue

1 · Symptom Summary

ID	Severity	Symptom
F-1	Critical	A clean gateway restart (`SIGTERM`, `outcome=clean_stop`) during an in-flight run persists the session as `status: "running"` with live `restartRecovery*` claim fields. No recovery path ever demotes it — every later message on that session throws `restart recovery claim changed before agent adoption`, deterministically.
F-2	Cascading	The throw is classified retryable. The Telegram ingress spool retries on exponential backoff and never dead-letters, so the poison message is retried forever (10 attempts and counting in the report).
F-3	Cascading	The spool is FIFO per lane — the poison entry head-of-line blocks every later message on that chat. The user sees silence, not errors. Only hand-editing `session_nodes.entry_json` in SQLite recovers the lane; there is no CLI path.

2 · Root Cause Analysis

F-1: "running" is a lie the process left behind

status: "running" is only meaningful while the process that owns the run is alive. After a restart, the value persists but its owner is gone — it describes a run that no longer exists. The recovery machinery does not account for this:

isRecoverableTerminalSessionStatus() covers failed | timeout | killed — not running
The gateway restart-recovery path gates on status === "failed" || status === "killed"
The channel-delivery claim guard treats status === "running" as fatal and throws

So a restart-orphaned running session matches no recovery path and trips every guard. This is an orphaned-state bug: state that encodes "a process is doing something" without encoding which incarnation of the process — the same identity blind spot as the PID-reuse lock in #114234, one layer up the stack.

The reporter's code-level follow-up sharpens it further: the transcript-only claim branch requires running to adopt, while the channel-delivery branch treats the same value as fatal. Same field, opposite meanings, selected by claim shape — a semantic fork that guarantees one branch is wrong after a restart.

F-2/F-3: A permanent error retried as if transient

The error message says "claim changed" — implying a lost race that a retry could win. But the CAS at line 189 is never reached; the state is permanently unacceptable. Retrying a deterministic failure converts one broken message into a dead chat lane. The missing primitive is error classification: retryable (transient) vs. dead-letter (permanent). Related history: #112893, #97798 — same spool, same missing classification.

3 · Fix Plan

Canonical candidate: the reporter's own PR

PR #114272 (by the issue reporter) implements the central clean-restart recovery path, with no config or migration burden — ClawSweeper's review flags it as the canonical repair. The report below adds the defense-in-depth layers that PR intentionally leaves as follow-ups.

Layer 1 — Demote on shutdown (covers clean stops)

// gateway shutdown hook
for (const s of sessions.where({ status: "running" })) {
  s.status = "killed";          // now matches isRecoverableTerminalSessionStatus()
  s.clearRestartRecoveryClaims();
}

Layer 2 — Sweep on startup (covers SIGKILL / crash / OOM)

// gateway boot: no run from a previous incarnation can still be live
for (const s of sessions.where({ status: "running" })) {
  if (!liveRuns.has(s.restartRecoveryDeliveryRunId)) {
    s.status = "killed";
    s.clearRestartRecoveryClaims();
  }
}

Layer 2 is the load-bearing one: it makes the invariant hold regardless of how the previous process died. Shutdown hooks are a courtesy; boot sweeps are a guarantee.

Layer 3 — Classify the error as non-retryable

// spool error classification
if (err.message.includes("restart recovery claim changed")) {
  return deadLetter(event);     // permanent state error — retry cannot help
}

4 · Stopgap Checklist (until patched)

☐ Before restarting the gateway to apply config, wait for in-flight runs to finish (watch for active runs going to 0)
☐ If a chat goes silent after a restart, check for the signature: identical restart recovery claim changed before agent adoption errors on exponential backoff in gateway logs
☐ Recovery (mirrors recoverTerminalSessionEntryForVisibleTurn()): in session_nodes.entry_json, set status to done and delete all restartRecovery* fields — the reporter confirmed the stuck message delivers within ~40s
☐ Monitor spool depth per lane: attempts > 5 on a pending entry with zero dead-letters is the poison-message signature
☐ Audit for the state proactively: SELECT session_key FROM session_nodes WHERE entry_json LIKE '%"status":"running"%' AND entry_json LIKE '%restartRecoveryDeliveryRunId%' after any restart

5 · Prevention — the ARK pattern

This is the third OpenClaw case we've analyzed in the same family — state that outlives the process that owned it:

Case	Orphaned state	Failure class	ARK guard
#113434	Retired generation, reused session ID	Stale handle after mutation	OutputValidator
#114234	Lock owned by a dead incarnation at a reused PID	Ownership proven by address, not identity	IdempotencyGuard
#114255	`running` status owned by a dead process	Liveness claim without an incarnation token	IdempotencyGuard + CircuitBreaker (dead-letter permanent errors instead of retrying)

The shared invariant all three violate: any persisted claim of "I am alive / I am running / I hold this" must be verifiable against the current process incarnation, and every consumer must have a recovery path for when it isn't.

pip install ark-trust        # Python
npm install @feilunxitong/arkit   # TypeScript

Does your agent have orphaned-state crash risks?

Idempotency guards · retry policy · logging — free 30-second check.

Run the free ARK diagnostic →

OpenClaw #114255 — Restart Mid-Run Leaves status: "running" Forever, and the Spool Retries a Poison Message Until the Lane Dies