{
  "summary": "## Executive finding\n\n**Strongest v0.33.0 → v0.34.0 regression candidate: the REDISPATCH-TRUTH retry/reconstruction changes can repeatedly install and replay a stream subscriber after a 15-second broker-connection write failure, converting v0.33's one-shot/permanent abandonment into v0.34's recurrent 15–30+ second transient stalls.** The field trace materially supports this candidate: 4,361 `stream-sub-attach` records and 33 `CONN_WRITE_POISONED` records, every poison at approximately 15,000 ms; minimum poison interarrival 29.7 s; `DISPATCH` failures 15.1–62 s after write start. Those timings match one or two subscriber-write deadline windows plus v0.34's retry scheduling, not any new 20- or 30-second timer.\n\nThis is **not yet proof that every poisoned stream caused a PTY stall**: `stream-subscriber` is also used by registry/pump/other network streams. The causal tie is direct only when the poisoned stream's pinned opener classifies as `Attach` and maps to the affected endpoint/controller. For such a stream, the PTY child and broker `OutputLog` remain alive and may keep advancing, while the target-side `serve_attach` worker cannot drain/forward output to the operator, so remote `spt rc` appears frozen. A poisoned registry/pump stream is not a PTY stall.\n\n### Exact history boundary\n\n- v0.33.0 annotated tag object: `21a46d5560819e505c83f0908b7b562d65b7d1ac`; release commit: `42d06a87401a62d2a8a6a81ccff2532d43ca6ede` (`release: v0.33.0 (counter 60) — PROJECT-INDEX milestone`).\n- REDISPATCH triage: `3d9f2ed4a38bdc6dbe4e7157bd2b35d049a79d9f`.\n- Runtime change cluster:\n  1. `ad7fde8acb97b5c839316121d341375b077092df` — `broker stream-table lifecycle truth — retire, durable opener, retryable claims`.\n  2. `2a110eed6f97d3dab96ed301d1b6d1df9a2cc002` — finished `Attach` rows terminal/unserved; request/reply FIN remains servable.\n  3. `84253572cd5e0cef57201ba671a8c201c6dbbdd8` — retirement changed to eligibility visibility only, never transport teardown.\n  4. `571d6a55e9949ce1b96aef1ba00f5e007a0a9521` — ADR round-2 documentation.\n- v0.34.0 release commit: `eb9fd8643921930ce21cc7ef0751a27360fc14bc`; annotated tag object: `b7b074c2509c1015fe341185662b4622d661f274` (`release: v0.34.0 (counter 61) — REDISPATCH-TRUTH W1`).\n\nThe reflog records these commits consecutively at `.git/logs/HEAD:1455-1465`; the release branch reflogs independently identify both release commits.\n\n## Relevant behavior changes, ranked\n\n### 1. HIGH — retryable claims repeatedly recreate the failing subscriber/replay path\n\n**v0.33 behavior:** the dispatcher used a process-local one-shot `HashSet`; once a worker claimed a stream, any setup failure burned the claim permanently.\n\n**v0.34 behavior:** `dispatch.rs:179-230, 299-365` introduces `ClaimState::{InFlight, Terminal, Retry}`, `MAX_DISPATCH_ATTEMPTS = 3`, and `DISPATCH_RETRY_BASE = 500 ms`. A failed worker is re-eligible after 500 ms, then 1 s; `Served` and `Unknown` are terminal. The 100 ms dispatcher poll discovers the retry promptly.\n\nEach `Attach` attempt reaches `serve_attach` (`dispatch.rs:539-622`), which immediately calls `brain.net_stream_subscribe(...)` (`attach.rs:298-304`). On the broker, `StreamLog::attach` (`nethost.rs:320-345`) installs the new physical broker connection as subscriber and synchronously replays every retained ring frame. Every replay write uses `BrokerConn::write`, whose absolute production bound is 15 s (`broker.rs:172-187`; `conn.rs:422-610`). The replay is performed while the per-stream `StreamLog` mutex is held.\n\nThe critical regression relation is therefore:\n\n- v0.33: a bad subscriber attempt could abandon the stream permanently, but it did not repeatedly reconstruct it in the same dispatcher generation.\n- v0.34: the same transport failure returns `DispatchOutcome::Failed`, then the claim requeues and creates another subscriber/replay attempt. A first 15 s poison followed by a retry and another 15 s poison predicts approximately 30–32 s. A single poison followed by a successful retry predicts approximately 15.5–20 s depending on scheduling/backlog. This fits the reported 20–30 s transient stalls and the field minimum poison interarrival of 29.7 s.\n\n**Why the current field trace strengthens this rank:** 4,361 subscriber attaches are far beyond five stable PTYs; the 33 exact-15-second poison events demonstrate the source's deadline mechanism is firing. The large attach count is the expected observable of repeated reconstruction/claim churn.\n\n**Affected scope:** intrinsically **one network stream and one physical broker IPC connection per failed worker**. Stream logs and subscriber connections are per-stream/per-worker; there is no new global PTY lock in this change. Several/all PTYs can stall together when a cold brain generation enumerates all active attach streams and launches their workers concurrently, or shared Windows/IPC/CPU pressure makes all of those independent workers stop draining. A single poisoned stream cannot, by source alone, freeze unrelated ConPTY children.\n\n### 2. HIGH-MEDIUM — `first_line` falls back to subscription/replay after *any* opener RPC error\n\n`dispatch.rs:390-418` first opens a fresh broker client and calls the new v0.34 `net_stream_opener` RPC. The intended compatibility rule is to fall back to the old ring-peek path for an N-1 broker that does not know the verb. The implementation, however, uses a broad `Err(_) => {}` and falls back for **all** errors, including transport failure/timeout/connection retirement.\n\nThe fallback `peek_first_line` (`dispatch.rs:420-451`) opens another `Brain` connection and calls `net_stream_subscribe`, entering the same synchronous replay path. Thus a 15-second failure in the opener RPC can be followed by a second 15-second failure in the fallback subscriber, yielding a diff-local approximately 30-second mechanism. On a healthy fully-v0.34 broker, pinned opener lookup should normally avoid this fallback; repeated fallback subscribers imply either an opener RPC error, unsupported/mixed-version broker, a row-retirement race, or that the subscriber is the later `serve_attach` subscription rather than the peek. The log must distinguish these stages by stream id and connection attribution.\n\n**Affected scope:** one dispatcher worker/stream; concurrent rediscovery can make it multi-stream. It is generation/startup/retry-coupled, not an autonomous 30-second periodic task.\n\n### 3. MEDIUM — post-poison worker completion is not bounded by the 15-second connection poison\n\nA `CONN_WRITE_POISONED` timestamp is **not** the same as a completed `DISPATCH` worker. `BrokerConn` poisons and aborts one physical IPC connection at 15 s, but there is no dispatcher-level deadline that forces `serve_attach` to return at that instant.\n\nSource reasons the observed `DISPATCH` completion can lag poison:\n\n1. `StreamLog::attach`, `append`, and `finish` discard every `sub.write(...)` result (`let _ = ...`; `nethost.rs:283-288, 307-316, 320-345`). `attach` continues walking the retained replay ring after the first failed write. Subsequent calls on the already-poisoned connection should fail quickly, but a large ring still must be traversed and the per-stream lock/handler cleanup cannot complete until that traversal returns.\n2. `Brain::net_stream_subscribe` is fire-and-forget (`brain.rs:1589+`): it sends the request but receives no subscribe-success/failure acknowledgement. `serve_attach` can continue setup while the broker-side replay is still blocked.\n3. `serve_attach` deliberately uses a second `wire` broker connection (`attach.rs:304-337`). Already-buffered output is forwarded through `wire.net_stream_send`. Those unacknowledged client calls return after local IPC write, but the broker handler executes `NetHost::send_stream` synchronously; QUIC stream sends have their own 10-second bound. If the separate wire handler is behind blocked 10-second sends, the worker's client-side pipe can back up and the worker may remain blocked after its subscriber connection was poisoned.\n4. The retry transition occurs only after the worker thread returns and writes its `ClaimState`; poisoning alone does not update the claim.\n\nItems 1–4 explain why the field sees `DISPATCH` failures at 17.6, 25.4, 29.1, 30.1, 31.1, 36.6, 59.4, and 61.6 seconds after write start. The exact contributor should be falsified with worker-stage stamps; the source does not provide a single timer equal to those delays. The 584 `PUMP_PEER_FAIL` 10-second events are evidence of substantial concurrent network failure/load, but the pump is isolated and those events are not by themselves proof of a PTY stall. They can increase scheduling/IPC pressure or share the remote network failure that makes the attach wire slow.\n\n### 4. LOW-MEDIUM — retirement adds broker IPC and short shared-lock work\n\nv0.34 adds `net-stream-retire` message types/client/broker handler and `NetHost::retire_stream` (`msg.rs:148-156, 808-811`; `brain.rs:1446-1474`; `broker.rs:4315-4338`; `nethost.rs:1551-1580`). Every `Served` worker opens a fresh broker connection and waits for a retirement reply. Retirement marks an `AtomicBool`, filters the row out of `stream_infos`, and clears its ring under that stream's mutex. The connection-close watcher also retains/removes all rows for the dead connection (`nethost.rs:698-715`).\n\nThis increases connection/thread/IPC churn and can contend briefly on the global stream table, especially when clearing a historical backlog. It has **no intrinsic 20–30 s deadline** and occurs after the stream is served/terminal, so it is weaker than retry/replay as a live PTY-stall explanation. A global stream-table lock delay would affect network stream enumeration/opener/subscription operations, not child ConPTY output or unrelated broker session output directly.\n\n### 5. LOW — durable opener capture adds bounded work under the per-stream lock\n\nv0.34 adds `OpenerCapture::{Buffering,Pinned,Oversize}` and captures at most 64 KiB while `StreamLog::append` already holds the per-stream mutex (`nethost.rs:93-113, 200-222, 251-282`). A normal attach opener is one short NDJSON line, so capture terminates on the first chunk. It is per-stream, bounded, and contains no wait/deadline. It cannot plausibly create a 20–30 s stall alone.\n\n### 6. EXONERATED AS DIFF-LOCAL CAUSE — refresh/supervisor cadence and ConPTY session transport\n\nThe v0.34 release cluster is REDISPATCH-TRUTH: dispatcher claims, broker network stream lifecycle/opener/retirement, message surface, fixture/tests/docs. The source-grounded regression story does **not** require a change to ConPTY, PTY `OutputLog`, brain heartbeat, or restart signaling. Relevant pre-existing timings remain:\n\n- dispatcher poll: 100 ms (`dispatch.rs:86-87`);\n- retry backoffs introduced in v0.34: 500 ms, then 1 s (`dispatch.rs:190-230`);\n- broker physical connection write deadline: 15 s (`broker.rs:172-187`; `conn.rs`);\n- controller queue-full deadline: 5 s (`broker.rs:144-154`);\n- brain heartbeat: 500 ms (`brainproc.rs:61-63`);\n- broker-side QUIC operation bound: 10 s (existing network deadline path).\n\nThe 20- and 30-second values appearing in `redispatch.rs` are test observation deadlines, not production timers.\n\n## Mixed-version caveat\n\nThe new brain treats retirement/opener verbs as best-effort N-1 additions. A v0.34 brain running against an older broker cannot retire historical rows; retirement waits for a broker cycle, and opener lookup falls back to subscription/replay. Therefore a brain-only update can leave the old broker behavior active and continue redispatching history on later refreshes. The stated current deployment has both broker and brain at v0.34, so this caveat does not explain the current cold-generation trace by itself, but it is a real release-contract gap for the v0.33→v0.34 rollout path.\n\n## Falsifiable causal checks\n\nFor each poison suspected of causing an RC stall, correlate the same stream id through:\n\n1. pinned opener / `DISPATCH:<id>` classification = `Attach`;\n2. endpoint/session/controller identity;\n3. `stream-sub-attach` connection id and from-seq;\n4. exact 15-second `CONN_WRITE_POISONED` on that connection;\n5. target PTY/session `OutputLog` high-water continues advancing while the operator render cursor stops;\n6. operator viewport resumes only after retry installs a new subscriber;\n7. other broker IPC probes and unrelated PTY rings continue, distinguishing per-stream transport failure from broker-wide IPC or child ConPTY failure.\n\nThis candidate is falsified for a particular stall if the poisoned stream is not `Attach`, if the target PTY ring itself stops advancing, or if unrelated broker IPC connections simultaneously stop answering without their own poisoned writes.\n\n## Focused fix design\n\n1. **Stop replay on the first subscriber write failure.** Make `StreamLog::attach` return `io::Result<()>`; do not discard `sub.write` errors. Clear/detach the just-installed subscriber and let the broker handler close promptly. Apply the same explicit failed-subscriber removal discipline to `append` and `finish` rather than retaining a poisoned subscriber.\n2. **Acknowledge or otherwise surface subscription setup failure.** `Brain::net_stream_subscribe` currently cannot distinguish “request queued” from “replay installed and writable.” A reply or a deterministic connection-close result should prevent `serve_attach` from entering its main loop on a subscriber already poisoned during replay.\n3. **Narrow compatibility fallback.** In `first_line`, fall back to `peek_first_line` only for an explicit unsupported/unknown-opener-verb response from an N-1 broker. Propagate timeout, EOF, poisoned connection, and other transport errors into the retry state; do not immediately create a second subscriber after a failed opener RPC.\n4. **Keep retryability, but make one attempt fully retire before the next.** The fix should preserve recovery from a genuinely transient worker-start error while preventing a new subscriber from being installed until the prior subscriber connection/handler has completed cleanup. Do not solve this by removing retries wholesale; that restores v0.33's permanent abandonment.\n5. Add stage telemetry keyed by dispatcher generation, stream id, attempt, and conn id: opener start/end, fallback selected/reason, subscribe request/replay completion, attach-as completion, first read, each wire send start/end, subscriber EOF, worker return, and claim transition. This is the minimum evidence needed to explain the 15→62 s tail precisely.\n\n## Deterministic regression-test seam\n\nExtend the production-path `dispatch_fixture`/`redispatch.rs` seam rather than manually invoking `serve_attach`:\n\n- Start a real broker and five real active attach streams.\n- Set the existing test-only `SPT_BRAIN_WRITE_DEADLINE_MS` to 100–400 ms.\n- Use a deterministic broker connection gate/black-hole seam to stop one target dispatcher subscriber from draining during synchronous ring replay; keep the other four draining.\n- Restart the real dispatcher generation so production discovery → opener → worker → `serve_attach` runs.\n- Assert: PTY/ring high-water continues; only the selected remote viewport pauses; unrelated attach streams and an independent broker status connection continue; the failed replay stops on its first write error; the old handler is gone before retry; the claim retries at most the declared budget; the selected viewport resumes from its cursor exactly once; poison-to-worker-failure is bounded tightly around one configured deadline, not two or four.\n- Add a second deterministic compatibility test with injected opener outcomes: `UnsupportedVerb` must invoke ring-peek fallback exactly once; `TimedOut`, EOF, or poisoned transport must **not** invoke fallback. This directly reds the current broad `Err(_)` branch.\n- Add a post-poison worker-stage assertion so the worker must return within a small bound after the injected write deadline; if it does not, the test reports the exact stuck stage (`StreamLog` replay, subscriber EOF, or separate wire send).\n\nNo files, processes, daemon state, endpoints, or live sessions were modified. Per assignment constraints, no tests or state-changing commands were run; findings are from source, refs/reflogs, current regression tests, architecture docs, and the supplied read-only field trace.",
  "files": [
    {
      "path": ".git/logs/HEAD",
      "description": "Exact v0.33→v0.34 commit sequence and release boundaries, including REDISPATCH W1 commits ad7fde8, 2a110ee, 8425357, 571d6a5 and release commit eb9fd864."
    },
    {
      "path": ".git/refs/tags/v0.33.0",
      "description": "Annotated v0.33.0 tag object id 21a46d5560819e505c83f0908b7b562d65b7d1ac."
    },
    {
      "path": ".git/refs/tags/v0.34.0",
      "description": "Annotated v0.34.0 tag object id b7b074c2509c1015fe341185662b4622d661f274."
    },
    {
      "path": "docs/REDISPATCH-TRUTH-TRIAGE.md",
      "description": "Field RCA and intended v0.34 behavior: finished attach retirement, durable opener, retryable claims, and production-path tests."
    },
    {
      "path": "docs/adr/0038-broker-stream-lifecycle-truth.md",
      "description": "Accepted lifecycle design and round-2 ruling that retirement changes eligibility visibility but must not tear down transport."
    },
    {
      "path": "crates/spt-daemon/src/dispatch.rs",
      "description": "Primary regression surface: ClaimState retries (500 ms/1 s, three attempts), 100 ms polling, broad opener-error fallback to subscriber peek, worker completion/retirement, and seeded attach reconstruction."
    },
    {
      "path": "crates/spt-daemon/src/nethost.rs",
      "description": "StreamLog synchronous subscriber replay under per-stream mutex; ignored write errors; 64 KiB opener capture; retired visibility; stream-table connection-close sweep."
    },
    {
      "path": "crates/spt-daemon/src/attach.rs",
      "description": "Target attach worker: initial net-stream subscription, durable opener seed, two-broker-connection split, session output forwarding, and worker return conditions."
    },
    {
      "path": "crates/spt-daemon/src/brain.rs",
      "description": "New opener/retire IPC clients and fire-and-forget net-stream subscribe/send behavior."
    },
    {
      "path": "crates/spt-daemon/src/broker.rs",
      "description": "New opener/retire handlers; existing 15-second connection bound plumbing and network stream subscription dispatch."
    },
    {
      "path": "crates/spt-daemon/src/conn.rs",
      "description": "BrokerConn absolute write deadline, watchdog, Windows CancelIoEx/DisconnectNamedPipe retirement, and poison semantics explaining exact 15,000 ms events."
    },
    {
      "path": "crates/spt-daemon/src/msg.rs",
      "description": "v0.34 additive net-stream opener and retirement protocol messages."
    },
    {
      "path": "crates/spt-daemon/tests/redispatch.rs",
      "description": "Production-dispatch regressions D1/D1b/D2/D3; shows intended retry and reconstruction behavior and provides the correct deterministic test harness seam."
    },
    {
      "path": "crates/spt-daemon/tests/fixtures/dispatch_fixture.rs",
      "description": "Killable production dispatcher generation used to reproduce brain restart/rediscovery behavior without manual re-serving."
    },
    {
      "path": "CHANGELOG.md",
      "description": "v0.34 public contract: refresh/update should no longer replay finished output or freeze remote terminals."
    }
  ],
  "architecture": "PTY custody and remote rendering are separate layers. The broker owns each ConPTY child and per-session PTY `OutputLog`; a target-side brain dispatcher reconstructs each peer network stream after a brain cycle. For an Attach stream, the dispatcher classifies from the broker-pinned opener, starts one `serve_attach` worker, subscribes one broker IPC connection to the peer `StreamLog`, then subscribes that same worker to the local session output. A second broker IPC connection forwards session output back over the network stream. Physical broker IPC writes are independently serialized and bounded per connection at 15 seconds. Consequently a poisoned network-stream subscriber can freeze the remote viewport while the ConPTY and PTY ring remain healthy. v0.34 did not introduce a global PTY lock or 20–30-second timer; it introduced lifecycle/opener/retry behavior that repeatedly re-enters this 15-second subscriber replay seam. The normal blast radius is one Attach stream, but brain-generation rediscovery starts all active workers together, allowing synchronized multi-PTY symptoms under common Windows/IPC pressure. Broker-wide IPC failure would require independent evidence that unrelated connections also stop; child ConPTY failure would require the PTY ring high-water itself to stop."
}