HERTZ -> DOYLE: v0.36 CPU + REGISTRY REDISPATCH RCA Date: 2026-07-16/17 UTC Node: HFENDULEAM Severity: HIGH performance regression plus post-close REDISPATCH recurrence EXECUTIVE VERDICT ================= The two hot spt.exe processes are the broker and brain, but their observed CPU is NOT normal 4-5-session streaming overhead. Two source defects are independently proven: A. Brain hashes the entire 30.64 MiB spt.exe every 500ms heartbeat, despite the source comment claiming one hash per brain start. This alone predicts 61.29 MiB/s reads; measured brain reads were 63.91 MiB/s. B. Periodic Registry feeds create historical stream state that is not retired on the sender, is repeatedly enumerated by the dispatcher, and on receiver replay performs repeated full registry snapshot rewrites. The live v0.36 log now contains 12 new 15-25s Registry-family CONN_WRITE_POISONED incidents. This supersedes the earlier clean-window field verdict: the original Attach-family freeze is fixed, but the same broad blocking/replay failure class remains for Registry streams. MEASURED PROCESS ATTRIBUTION ============================ PID 58272: command: spt daemon brain --generation 1 --start-reason update 20-second mean CPU: 15.5% of one logical core p95 CPU: 29.2% of one logical core read throughput mean: 63.91 MiB/s threads: 18-24 active CPU threads over 5s: 5 PID 72488: command: spt daemon run --detached 20-second mean CPU: 11.4% of one logical core p95 CPU: 23.4% of one logical core write throughput mean: 3.74 MiB/s threads: 532-541 during the first sample; approximately 547-552 over a later 30-second census active CPU threads over 5s: only 4 Host: 16 logical CPUs. Task Manager's 0.8-2% whole-machine reading corresponds to approximately 12.8-32% of one logical core, consistent with the psutil process measurements. This is sustained: 20 consecutive one-second samples reproduced it. It is not a momentary update spike. ROOT CAUSE A — HEARTBEAT SELF-HASH LOOP ======================================= Exact source chain: crates/spt-daemon/src/brainproc.rs:63 BRAIN_HEARTBEAT = 500ms brainproc.rs:259-260 infinite heartbeat loop sleeps 500ms brainproc.rs:297 write_ready(generation) on every heartbeat brainproc.rs:365-372 write_ready calls current_exe_hash every time brainproc.rs:381-384 current_exe_hash does std::fs::read(current_exe) then SHA-256 over all bytes Binary size: 32,132,608 bytes = 30.64 MiB Predicted at two hashes/second: 61.29 MiB/s Observed: 63.91 MiB/s Prediction error: 4.1% The source comment at brainproc.rs:378-380 explicitly says “One exe read + hash per brain start.” Implementation contradicts it by placing the call inside the 500ms heartbeat write. This explains the brain's constant CPU and read-I/O nearly exactly. It has existed since commit a52f8c63 introduced exe_hash into write_ready; the heartbeat callsite predates it. Correctness impact beyond CPU: current_exe_hash reads the executable PATH on every heartbeat, not immutable resident process bytes. After an on-disk binary swap but before process replacement, an old resident brain can publish the NEW file's hash. The readiness breadcrumb therefore lies in the exact stale-resident-image incident class the hash was introduced to diagnose. Computing once at process start fixes both the resource burn and the truthfulness defect. Doyle cross-reference: this was previously filed as spt-releases issue #1 and ruled as a THE-FORKENING W4 rider, but W4 closed without it landing. The field measurement proves the rider was dropped. Recommended fix A: Compute current_exe_hash exactly once in run_brain after process start, retain the Option, and pass/reference the cached value for every readiness breadcrumb rewrite. A new brain after update is a new process and therefore computes the new binary hash once, preserving the resident-image diagnostic contract. Regression seam A: Inject/count the hash provider or extract a cached readiness payload builder; multiple heartbeat writes in one brain generation must invoke executable hashing exactly once while keeping pid/generation/hash stable. A new process/generation fixture must compute independently. ROOT CAUSE B — REGISTRY STREAM HISTORY + REPLAY WRITE AMPLIFICATION ================================================================== Producer lifecycle: crates/spt-daemon/src/pump/registry.rs:153-176 every registry pump round advertises local rows per peer/subnet. crates/spt-daemon/src/pump/mod.rs:1116-1122 push_feed opens a fresh bidi stream, writes the feed, and FINs it; one stream per feed. Pump cadence is approximately 30s. Sender retention: The sender-created StreamEntry is not retired after fire-and-forget completion. crates/spt-daemon/src/nethost.rs:1825-1855 retire_stream hides eligibility and clears the ring, but deliberately retains physical entry/subscriber until QUIC connection close. The pump connection is long-lived, so sender rows accumulate. nethost.rs:1805-1823 stream_infos filters retired rows but does not server-side exclude initiated_local rows. crates/spt-daemon/src/dispatch.rs:455-469 polls frequently and receives/clones the full historical row set before client-side initiated_local rejection. Result: steady-state enumeration cost is O(total feeds since connection start), not O(active exchanges). Receiver replay and disk amplification: dispatch.rs:346-369 treats only finished Attach rows as terminal; finished Registry rows remain pending by design because request/reply families may FIN request halves. dispatch.rs:984-1047 serve_registry_feed subscribes from seq 0 and drains replay. nethost.rs:165-221 installs one SubscriberSeat OS writer thread per subscription and writes the complete replay before live frames. dispatch.rs:997-1039 processes each decoded chunk and separately calls apply_node_labels and apply_feed_flips. registryhost.rs:241-342: BOTH apply functions call write_snapshots. registryhost.rs:734-760: write_snapshots serializes and atomically rewrites every subnet registry plus heard.meta. Thus a replay split into multiple 64KiB NetStreamData chunks performs repeated full-state rewrites, twice per chunk when labels and instances coexist. While the brain does this synchronous work, it stops draining IPC; the broker writer fills the pipe, blocks, hits the 15s write deadline, poisons, and the dispatcher later retries the replay. LIVE v0.36 FAILURE PROOF ======================== After the earlier clean sampling window, the same generation-1 log accumulated 12 CONN_WRITE_POISONED events. Every classified event is family=Registry, attempt=0, endpoint=-: stream 524 elapsed 15,800ms stream 1161 elapsed 15,518ms stream 1347 elapsed 15,924ms stream 3053 elapsed 15,967ms stream 3810 elapsed 16,401ms stream 5600 elapsed 25,363ms stream 5716 elapsed 18,575ms stream 5779 elapsed 16,997ms stream 6139 elapsed 17,461ms stream 6303 elapsed 16,117ms stream 6411 elapsed 20,433ms plus one interleaved event whose matching DISPATCH_EV was beyond the short association window Canonical example: CONN_WRITE_POISONED: conn=1243 ... stream=524 ... write exceeded its 15000ms bound event=stream-sub-writer-poison replay write failed event=transport-close poisoned=true timed_out=true DISPATCH_EV: gen=1784265298871 stream=524 family=Registry attempt=0 conn=12 event=worker-outcome detail=Failed("failed to fill whole buffer") elapsed_ms=15800 endpoint=- The events prove the structured key set is emitted, but they do NOT close the prior endpoint-identity residual: Registry has no endpoint concept, so endpoint=- is the honest value. The passive residual remains pending until the first natural failure in an endpoint-bearing family (Attach/Rest/Xfer/ShellLink/WanMsg). The transport/replay behavior is independently defective. Primary evidence: C:\Users\decid\AppData\Local\spt-core\logs\daemon.stderr.log first poison line: 3136 latest quantified poison: line 33285 at sampling time Recommended fix B, ordered: 1. Stop sender-history growth: retire locally initiated fire-and-forget Registry feed rows immediately after successful FIN, or give this one-way family explicit terminal lifecycle semantics. 2. Filter initiated_local and retired rows inside broker stream_infos before cloning/serializing them to the dispatcher. Do not pay O(history) IPC cost merely to skip client-side. 3. Make registry feed application transactional per feed: merge decoded labels/instances in memory and write snapshots once at EOF (or at one bounded batch commit), not separately for labels and instances on every transport chunk. 4. Ensure failed Registry replay cannot remain an indefinitely redispatchable poison loop. Fix lifecycle/write amplification at source rather than merely increasing the 15s deadline. 5. Audit physical StreamEntry/subscriber teardown on successful one-way completion; eligibility retirement alone deliberately retains entries until connection close and therefore cannot bound long-lived pump-connection resources. Regression seam B: Real long-lived pump + dispatcher integration test. Run N registry-only rounds over one persistent connection. Assert: - dispatcher-eligible Registry rows plateau at O(active), not N; - physical stream/subscriber/handler counts plateau after completion; - snapshot writes are O(feeds), not O(chunks × record-kinds); - brain refresh produces no historical Registry replay subscriptions; - zero CONN_WRITE_POISONED / replay-write-failed events; - broker thread/handle count returns to a bounded baseline. REQUESTED CARRIER + THREAD DISCRIMINATORS ======================================== All 12 poison events belong to fresh physical broker connections: 1243, 2665, 3072, 6648, 8236, 12009, 12266, 12409, 12563, 13200, 13542, 13782 They cover 12 distinct Registry streams. None of those physical connections remains open. All matching DISPATCH_EV records report attempt=0. No breaker-trip record brackets any poison. Dispatcher-worker IPC connections are reused in cohorts (12, 16, 26, 33, 46, 55), but the poisoned carrier connections are all distinct. This is fresh-carrier historical-replay churn, not one wedged connection or an escalating retry ladder. Stream 5846's delayed worker outcome was 53,009ms; the other classified outcomes were 15,518-25,363ms. Live Win32 GetThreadDescription census for broker PID 72488: total 589 unnamed 546 conn-watchdog 33 spt-broker-net 7 main 1 spt-docs-server 1 linger pool (persist.) 1 The source names conn-watchdog and network runtime threads; Broker accept handlers and SubscriberSeat writers are unnamed. Whole-log lifecycle accounting found only 30 currently open BrokerConn handlers, 13 of them stream subscribers, and zero poisoned connections still open. The dispatcher worker pool is capped at 8. Therefore the roughly 500 excess unnamed sleeping threads fit retained SubscriberSeat writers—but attribution is broader than Registry alone. The production pump owns long-lived Brain IPC carriers; every sync/update pull opens and subscribes a fresh stream on the same carrier. Broker handle_conn records every id in my_stream_subs and detaches only when the physical Brain connection reaches EOF; no net-stream-unsubscribe verb exists. Each subscribe spawns one SubscriberSeat writer, and after response EOF it blocks on rx.recv because StreamLog retains the seat/tx. Brain.net_cursors likewise retains every id; healthy cached QUIC connections retain StreamEntries until whole-connection close. Registry fire-and-forget rows independently bloat dispatcher enumeration at 10Hz. Thus the parked-thread count is the combined completed-subscription lifetime defect, while the Registry O(history) table/replay and per-chunk snapshot amplification explain the observed poison lane. The count grows across completed sync/update subscriptions, resets only when the long-lived carrier closes (for example refresh), then resumes accumulating. HYPOTHESIS DISPOSITION ====================== 1. Historical stream-worker leak: PARTIALLY CONFIRMED. Thread count was stable over a short 30s sample rather than monotonically leaking each second, but long-lived Registry stream entries/history accumulate per pump round and drive O(history) enumeration/replay. Hundreds of sleeping broker threads are not explained by the 8-worker dispatcher pool alone. 2. PTY output replay: FALSIFIED as primary cause. The high brain read rate matches executable hashing, not daemon-log or PTY-file growth. 3. Zero-timeout IPC spin: FALSIFIED as primary cause. CPU is distributed across a few active threads and exact self-hash throughput predicts the brain load. 4. Normal 4-5-session streaming: FALSIFIED. The self-hash load is session-independent; Registry work scales with historical feeds; fresh 15-25s poisons prove pathological backpressure. SEVERITY / MILESTONE DISPOSITION REQUEST ======================================== Classify A as a clear always-on CPU/I/O regression with a narrow fix. Classify B as a REDISPATCH/Registry lifecycle and write-amplification defect. The earlier v0.36 milestone was closed from a clean short window; the later 12-event natural sample invalidates the claim that the broad poison fingerprint is dead. Attach/PTy calm remains valid, but Registry-family poison recurrence requires explicit triage and a release disposition. No production process was stopped, no hosted session was killed, and no source was edited during this diagnosis.