{
  "summary": "HFENDULEAM runtime/log RCA (read-only, no session mutation)\n\nBottom line\n- The field evidence does NOT show a 20–30 s broker-wide IPC freeze. It shows a recurring per-physical-connection failure of brain `stream-subscriber` connections: the broker write blocks, the v0.34 `BrokerConn` watchdog retires that one connection at almost exactly 15 s, and the corresponding dispatcher worker reports failure/retries later. Unrelated broker connections continue to complete during the same interval, the peer-pump heartbeat keeps advancing, and the brain PID/generation remains stable.\n- The strongest v0.33→v0.34 regression relation is the v0.34 redispatch fix itself: v0.34 added the pinned-opener RPC, an `Err`-on-any-kind fallback to `peek_first_line` (which opens a fresh subscribing `Brain` connection), and retryable claims (500 ms, then 1 s; max 3 attempts). It did not add a 20/30 s timer, but it can turn the pre-existing Windows 15 s broker-write expiry into a 20–62 s dispatch lifecycle through fallback subscriber creation, variable client-side completion, and retry. Commits identified by the regression investigator: `ad7fde8`, `2a110ee`, `8425357`.\n- The remaining unexplained portion is now sharply localized: the server logs `transport-close poisoned=true timed_out=true` at the 15 s bound, but the ordinary dispatcher-side `Brain` can remain inside an unbounded `read_event()` or family-specific work for another ~0.1–47 s. Only the pump carrier has the split reader plus `recv_timeout`; ordinary dispatcher clients do not. The present logs lack worker-stage/client-receive timestamps, so buffered named-pipe data, Windows disconnect/drop propagation, and time spent outside the read call cannot yet be separated. This is a falsifiable mechanism, not a claim that the exact sub-cause has already been proved.\n\nTimestamped evidence (UTC; epoch wall-ms retained for exact correlation)\n1. Cold runtime start: `2026-07-16T21:01:35.891Z`, wall_ms `1784235695891`: `BRAIN_UP`, generation 0, brain PID 47668, broker socket `spt-daemon-broker-78bf18a9fd96fdc6.sock`.\n2. Exact 20.354 s specimen, stream 3907:\n   - `2026-07-16T23:21:22.619Z` / `1784244082619`: conn 10179 `stream-sub-attach stream=3907 from_seq=0`, then `write-start`.\n   - `2026-07-16T23:21:37.627Z` / `1784244097627`: `CONN_WRITE_POISONED` and timed-out close; elapsed 15.008 s.\n   - Near `2026-07-16T23:21:42.368Z` / neighboring wall_ms `1784244102368`: `DISPATCH:3907:Failed(\"failed to fill whole buffer\")`; elapsed ~19.749 s from attach and ~4.741 s after poison.\n   - `2026-07-16T23:21:42.973Z` / `1784244102973`: fresh stream-3907 subscriber conn 10195; elapsed 20.354 s from the original attach.\n3. Longer specimen, stream 3711:\n   - `2026-07-16T23:13:45.858Z` / `1784243625858`: conn 9612 subscriber/write start.\n   - `2026-07-16T23:14:01.012Z` / `1784243641012`: poison; 15.154 s.\n   - Around `2026-07-16T23:14:48.480Z` / `1784243688480`: `DISPATCH:3711:Failed`; ~62.622 s from start and ~47.468 s after server poison.\n4. 32.790 s specimen, stream 3721:\n   - `2026-07-16T23:14:15.664Z` / `1784243655664`: conn 9641 subscriber/write start.\n   - `2026-07-16T23:14:30.679Z` / `1784243670679`: poison; 15.015 s.\n   - Around `2026-07-16T23:14:48.454Z` / `1784243688454`: `DISPATCH:3721:Failed`; ~32.790 s from start.\n5. Aggregate trace supplied by Main over this same cold v0.34 generation: 33 `CONN_WRITE_POISONED` incidents, all `stream-subscriber`; every poison at ~15,000 ms; dispatch completion 15.1–62 s from write start, with recent examples 17.6, 25.4, 29.1, 30.1, 31.1, 36.6, 59.4, and 61.6 s; median poison interarrival 95.5 s, minimum 29.7 s. The same trace counted 4,361 `stream-sub-attach` records and 584 per-peer 10 s `PUMP_PEER_FAIL` events.\n6. Broker-wide counterexample during stream 3907's blocked 15 s window: conn 10180 completed normally at wall_ms `1784244083264`, conn 10181 at `1784244088327`, and conn 10182 at `1784244093410`; fresh conns began at `1784244097732`, only 105 ms after the poison. The broker accept/dispatch surface was therefore making unrelated progress while conn 10179 was blocked.\n7. Pump continuity: repeated reads of `identity/pump-heartbeat.json` advanced `1784247371915` → `1784247430856` → `1784247477529` (`2026-07-17T00:17:57.529Z`), an advance of 105.614 s across the investigation. This contradicts a contemporaneous peer-pump wedge. The current log showed the original two `PUMP_IPC_READER: spawned` lines and no observed current-generation reader exit/reconnect sequence.\n8. Process facts remained stable across reads: broker/daemon PID breadcrumb 82868; `brain.ready` = `{pid:47668,generation:0,exe_hash:\"c4284a14db2ae23f48e1322514f6c929fd135e76799e5680cf8921edf742d3c3\"}`. No current-log brain generation change accompanied the sampled incidents.\n9. Current hosted endpoint records were live/online and preserved: deployah PID 16844, doyle 13800, lia 69996, webbie 81556, todlando 42588, emphasys 31352. Deployah/lia/webbie `last_active_ms` advanced near `1784247434xxx`; Doyle's recorded last-active was older (`1784247059062`) but its process/status remained online. These are durable/status facts, not proof of each PTY's output progress.\n10. Main's webbie correlation is not sufficient to make stream 4834 the webbie RC connection: webbie's session controller attach occurred at wall_ms `1784245849362`; stream 4834 began 37.354 s later and poisoned 52.366 s after attach. The stream poison label contains `stream-subscriber stream=4834` but no controller/session/endpoint attribution. Temporal proximity supports a shared-load hypothesis, not identity or causality.\n\nBroker-wide versus endpoint-local conclusion\n- Strongly disfavored: broker-wide IPC stall. Prediction contradicted by unrelated connection completions throughout blocked-write windows and by the advancing pump heartbeat.\n- Not observed: brain restart/reconnect/subscriber replay caused by a brain lifecycle transition during the sampled cold generation. Rotated logs do contain earlier generation-2/3 update cycles and their expected `BRAIN_RESUMED`/dispatch failures, but the current incidents occur with generation 0 stable.\n- Directly observed: isolated `stream-subscriber` broker connections stop draining and hit the exact per-connection 15 s watchdog. This is a brain/dispatcher net-stream path. It is not itself a child ConPTY-output marker and is not attributed to a particular PTY.\n- Still possible as a user-visible consequence: shared brain-process or Windows scheduling/resource pressure from very high subscriber/connection churn may delay several remote views at once. No CPU, memory, handle, thread-count, or per-process I/O telemetry is present in these logs or the daemon-status DTO, so resource pressure is currently unsupported rather than ruled out.\n\nRanked hypotheses and live-probe predictions\n1. HIGH — v0.34 opener fallback/retry plus unbounded dispatcher completion is producing the 20–62 s lifecycle. Prediction: a worker-stage trace will show `net_stream_opener` error or a serve subscription, then one stream-specific conn blocked exactly 15 s, client read/worker return later, followed by the 500 ms/1 s claim retry. Unrelated broker status probes remain prompt.\n2. MEDIUM — an RC/view process or remote subscriber stops draining; the server-side stream subscriber fills while its child PTY continues. Prediction: affected child PID `Read/WriteOperationCount` or byte counters and broker session output sequence continue to advance, while only the affected RC/Windows Terminal output counters or delivery timestamps pause; the poisoned conn can be joined to a controller/session only after adding attribution/stage IDs.\n3. MEDIUM-LOW — aggregate resource pressure from 4,361 subscriber attaches, 33 watchdog cancellations, 584 dead-peer 10 s dials, and coincident 30 s psyche subprocess timeouts increases dispatcher scheduling/teardown latency. Prediction: CPU, handle/thread count, context switches, or I/O latency rise across the brain/daemon/RC processes around clusters; local `daemon status --json` may show jitter but should not necessarily block for the full stall. Reducing only fallback subscriber churn should reduce stalls if causal.\n4. LOW/UNRESOLVED — child/ConPTY pauses. Prediction: only the victim child's output/I/O counters and broker `OutputLog::append` sequence stop, while daemon status, other sessions, and RC transport remain prompt. Current logs have no child-output timestamps, so this cannot be confirmed from stderr.\n5. LOW — pump/peer-dial or brain-lifecycle wedge. Prediction would be a stale 5 s heartbeat, `PUMP_IPC_READER` exit/spawn or `BRAIN_UP` generation change, and correlated delays across independent broker calls. Current evidence contradicts those predictions. `PUMP_PEER_FAIL` is source-designed as concurrent, per-peer isolation; the 10 s peer bound is not a serial broker-wide stop.\n\nFocused fix and validation design\n- Fix the source of avoidable fallback subscribers: `first_line` must fall back to `peek_first_line` only for an explicit N-1/unsupported-opener reply. A transient opener error, row race, or local IPC failure should release the v0.34 retryable claim rather than create a replay subscriber. Do not treat every `Err` as protocol unsupported.\n- Bound dispatcher worker reads independently of broker writes. Generalize the pump's split-reader/channel-deadline approach, or add an owned/cancelable dispatcher carrier, so a worker exits within a small deterministic bound after its server connection is poisoned. Do not use Windows `PIPE_NOWAIT`; the source documents its mid-frame corruption risk.\n- Add correlation telemetry at the missing boundary: worker ID + stream ID + family + stage (`opener-query`, `fallback-subscribe`, `serve-subscribe`, `read-enter`, `read-error`, `return`, `retry`) and client-side conn identity. This will determine whether the poisoned conn is the temporary peek subscriber or the family serve subscriber and exactly where post-poison time is spent.\n- Regression test A: on a fully v0.34 broker, inject a transient opener-RPC error with a large retained stream. Assert no fallback `stream-sub-attach` occurs; the claim re-enters at the documented 500 ms backoff and succeeds through the pinned opener.\n- Regression test B (Windows, production path, shortened `SPT_BRAIN_WRITE_DEADLINE_MS=400`): create a non-draining replay subscriber; assert server poison near 400 ms, dispatcher worker failure within a tight post-poison bound, exactly bounded retries, and no parked worker/connection leak.\n- Regression test C: during B, keep an unrelated session viewer draining and assert its sequence/frame count advances across the entire incident. The existing `conn_blackhole_lifecycle` test provides the right unrelated-viewer invariant but should be extended to the redispatch/fallback seam.\n- Live read-only probe: sample `spt daemon status --json` every 250 ms (documented read-only/ungated), record existing passive viewer/output sequence timestamps, and snapshot child PID, affected `spt rc` PID, and Windows Terminal PID CPU plus read/write operation/byte counters without sending input or resize. Prompt status + progressing unrelated viewer + one frozen view means isolated RC/transport; prompt status + victim child counters flat means child/ConPTY; all control replies and viewers delayed together means shared broker/IPC; child output advancing while RC/WT stalls means local render/pipe transport. Preserve all current sessions and use naturally occurring output; deterministic injected markers belong in a disposable integration harness, not these live PTYs.",
  "files": [
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/logs/daemon.stderr.log",
      "description": "Current cold-generation broker/brain stderr. Contains exact wall_ms/mono_ms connection lifecycle, 15 s `CONN_WRITE_POISONED` events, stream subscriber attachments, dispatch failures, unrelated connection progress, peer failures, and endpoint/psyche activity."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/logs/daemon.stderr.log.1",
      "description": "Rotated prior-generation evidence: generation 2/3 brain update starts, `BRAIN_RESUMED`, initial dispatch pipe failures, and peer-pump failures. Useful contrast to the stable current generation."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/daemon.pid",
      "description": "Current daemon/broker PID breadcrumb: 82868. Source explicitly says PID files are breadcrumbs, while socket ping is authoritative liveness."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/brain.ready",
      "description": "Current brain readiness record: PID 47668, generation 0, executable hash c4284a14db2ae23f48e1322514f6c929fd135e76799e5680cf8921edf742d3c3."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/identity/pump-heartbeat.json",
      "description": "Pump heartbeat advanced repeatedly during investigation, disproving a contemporaneous pump-loop stall."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/identity/pump-ops.json",
      "description": "Current pump operation owner agrees with brain PID/generation (PID 47668, generation 0)."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/owlery/*/info.json",
      "description": "Read-only per-endpoint runtime records used for online state, process IDs, adapters, controller state, and last-active stamps."
    },
    {
      "path": "crates/spt-daemon/src/conn.rs",
      "description": "Windows bounded framed-write mechanism. `BrokerConn` uses a 15 s absolute write/gate deadline, `CancelIoEx`, `DisconnectNamedPipe`, poison/retire semantics, watchdog thread, and lifecycle stamps. Server-side cancellation does not itself impose a client dispatcher read deadline."
    },
    {
      "path": "crates/spt-daemon/src/broker.rs",
      "description": "Defines `BRAIN_WRITE_DEADLINE=15s`, per-connection subscriber/controller writers, controller-only stall-evict accounting, and role attribution. Explains why stream-subscriber poison events need not appear in `stall_evict_count`."
    },
    {
      "path": "crates/spt-daemon/src/brain.rs",
      "description": "Ordinary `Brain` clients use blocking unbounded reads; only the peer pump uses a split reader and channel `recv_timeout`. This is the key source-grounded post-poison completion asymmetry."
    },
    {
      "path": "crates/spt-daemon/src/dispatch.rs",
      "description": "v0.34 dispatcher: opener RPC, catch-all `Err` fallback to `peek_first_line` subscriber, worker family subscriptions, claim retry state, 500 ms exponential backoff, max three attempts, 100 ms poll."
    },
    {
      "path": "crates/spt-daemon/src/attach.rs",
      "description": "Remote attach serve loop and ordinary blocking `Brain::read_event` use; establishes the RC/view path and the observability needed to distinguish transport from child output."
    },
    {
      "path": "crates/spt-daemon/src/pump/mod.rs",
      "description": "Five-second heartbeat, per-peer isolation, concurrent dial handling, 10 s peer failures, and the separate 30 s pump IPC deadline. Current heartbeat progress and lack of restarts disfavor this path."
    },
    {
      "path": "crates/spt-daemon/src/brainproc.rs",
      "description": "500 ms brain heartbeat, broker liveness probe, sessions reap, and spawning of dispatcher/pump consumers. No current generation transition accompanied incidents."
    },
    {
      "path": "crates/spt/src/cli.rs",
      "description": "`spt daemon status --json` is the non-mutating status surface. Fields include running/PID, net status, pump heartbeat, broker image/staleness, controller stall-evict tally, peer health when available, and local endpoint rows."
    },
    {
      "path": "CHANGELOG.md",
      "description": "v0.34.0 release contract: redispatch/finished-session replay fix; v0.33.0 comparison."
    },
    {
      "path": "docs/KNOWN-HAZARDS.md",
      "description": "Source-grounded history for Windows blocked writes (7.36/7.39) and v0.34 terminal-stream redispatch changes (7.41)."
    },
    {
      "path": "crates/spt-daemon/tests/redispatch.rs",
      "description": "Existing production-path redispatch tests; correct seam for opener fallback/retry regression coverage."
    },
    {
      "path": "crates/spt-daemon/tests/conn_blackhole_lifecycle.rs",
      "description": "Existing Windows-shaped blackhole test with an unrelated draining viewer; correct base invariant for proving per-connection rather than broker-wide impact."
    }
  ],
  "architecture": "Observed runtime flow and failure boundary:\n\n```mermaid\nflowchart LR\n  PTY[Hosted child / ConPTY] --> OL[Broker OutputLog]\n  OL --> C[Controller/viewer broker conn]\n  Peer[Remote peer QUIC stream] --> NS[Broker Net StreamLog]\n  NS -->|replay/live frames| BC[BrokerConn: per physical named pipe]\n  BC -->|15 s bounded write| DB[Dispatcher Brain client]\n  DB --> FL[first_line: opener RPC]\n  FL -->|any Err in v0.34| PF[peek_first_line fallback subscriber]\n  FL -->|classified| FW[family worker / serve_attach etc.]\n  PF --> UR[ordinary unbounded read_event]\n  FW --> UR\n  BC -->|watchdog at 15 s| PC[CancelIoEx + DisconnectNamedPipe + poison]\n  PC --> TC[server transport-close immediately]\n  UR -->|variable later error/return| DF[DISPATCH Failed]\n  DF -->|500 ms then 1 s, max 3| DB\n  Pump[Peer pump split Brain] --> RT[reader thread + recv_timeout]\n```\n\nThe important isolation boundary is `BrokerConn`: every observed poison is keyed to one conn and one `stream-subscriber stream=N`. The broker remains able to accept and complete other IPC connections while that conn is blocked. The variable 20–62 s user-visible-shaped lifecycle is not one timer; it is the fixed 15 s server write bound plus unbounded/variable dispatcher client completion and v0.34 retry/fallback. The PTY path is adjacent but not joined by current log attribution, so a stream timeout cannot honestly be assigned to webbie, Doyle, or another PTY without the proposed worker/endpoint correlation telemetry."
}