# REGISTRY-STALL — v0.36 post-close CPU + Registry redispatch RCA (hertz, 2026-07-17)

**Status: doyle gate verdict ACCEPTED IN FULL 2026-07-17 — every source claim code-verified same day.**
Canonical RCA for the REGISTRY-STALL milestone candidate (ranked FIRST in the findings backlog,
displaces DAEMON-LIFECYCLE). Fix set B1–B5 + rider A accepted as filed; regression seams ride
verbatim into the wave's test matrix.

Doyle triage annotations (beyond the report below):

- **Defect A is a DROPPED RIDER**: identical defect was ruled 2026-07-15 as a THE-FORKENING W4
  rider (see `.claude/reports/2026-07-10-brain-self-hash-cpu/README.md` — OnceLock once-per-process
  digest, doc-comment fix same commit, injected digest-counter==1 gate, D7 process-replacement
  truthfulness leg). W4 closed without landing it. The prior ruling's stronger angle stands: the
  per-tick re-hash reads the current exe PATH, so a post-swap resident-old-bytes brain publishes
  the NEW file's hash — the breadcrumb lies in the exact class it exists to catch. Root commit
  `a52f8c6`; shipped since v0.10.0.
- **endpoint=- is by design** for Registry family (`line_endpoint` answers `None` honestly,
  dispatch.rs). The prior milestone's passive residual (production endpoint-key confirm) stays
  PENDING until a natural endpoint-bearing-family failure.
- **W1 machinery exonerated as cause**: all 12 poisons are attempt=0 on fresh carrier conns with
  zero breaker trips — not retry-ladder churn. The v0.36 DISPATCH_EV telemetry is what made this
  visible (telemetry paying rent).
- **Thread census verified against source**: SubscriberSeat writer is a bare unnamed
  `std::thread::spawn` (nethost.rs:196); retirement is eligibility-only, physical entry+writer
  live until a conn-close sweep that never comes on the long-lived pump conn → ~500 parked
  writers of 589 broker threads. Doyle rider: name all spawned daemon threads (sub-writer
  minimum) so future censuses aren't blind.
- **Attribution correction (hertz final cross-check 2026-07-17, doyle-verified)**: the ~500
  retained writers are NOT Registry-only. The retention mechanism is FAMILY-AGNOSTIC: the broker
  conn loop accretes `my_stream_subs` per `KIND_NET_STREAM_SUBSCRIBE` (broker.rs:3246) and calls
  `detach_stream_sub` only at conn-loop exit (broker.rs:3280) — there is NO net-stream
  unsubscribe verb (`KIND_UNSUBSCRIBE` is the PTY-session verb only). So completed sync/update
  subscriptions on the long-lived pump Brain carrier retain seats too (each seat parked on
  `rx.recv`, `Brain.net_cursors` retains ids, QUIC entries last to conn close). Registry
  fire-and-forget rows remain the INDEPENDENT driver of the ~10Hz O(history) enumeration/replay
  + snapshot amplification (defect B legs 1-3). Consequence for the wave: B5's teardown audit is
  cross-family (bounded seat lifetime for ALL completed subscriptions on long-lived carriers —
  an unsubscribe verb or seat-scoped EOF release), not a Registry-scoped patch. Hertz's
  corrected local RCA (#321E wording) is the authoritative artifact text.
- **Milestone disposition (ruled)**: REDISPATCH-STALL close STANDS scoped to its W1 surfaces
  (Attach/PTY calm valid); the clean-window verdict as a GENERAL poison-fingerprint-dead claim
  is superseded. No v0.36.0 release action — both defects predate it, zero bad bytes.
- **Gate additions for the wave**: every leg keeps redispatch D1/D1b + mesh-recovery legs green
  (retire machinery + registry gate = substrate); B2's server-side stream_infos filtering is
  wire-visible — N-1 brain compatibility settles at triage.

Field evidence baseline (preserve until fix field-verified), hertz owns all three:

- Live log: `C:\Users\decid\AppData\Local\spt-core\logs\daemon.stderr.log` — first poison line 3136.
- Byte-for-byte snapshot: `C:\Users\decid\.omp\state\spt-endpoint\v036-registry-stall-daemon.stderr.log.snapshot`
  (5,748,996 bytes; **15 poisons by snapshot time** — count grew from the 12 in the report below,
  defect live and accumulating at pump cadence).
- Final corrected RCA artifact (adds the resident-old/path-new hash truthfulness defect):
  `C:\Users\decid\.omp\state\spt-endpoint\hertz-v036-cpu-registry-rca.txt`.

---

Hertz report verbatim below.

```
HERTZ -> DOYLE: v0.36 CPU + REGISTRY REDISPATCH RCA
Date: 2026-07-16/17 UTC
Node: HFENDULEAM
Severity: HIGH performance regression plus post-close REDISPATCH recurrence

EXECUTIVE VERDICT
=================
The two hot spt.exe processes are the broker and brain, but their observed CPU is NOT normal
4-5-session streaming overhead. Two source defects are independently proven:

A. Brain hashes the entire 30.64 MiB spt.exe every 500ms heartbeat, despite the source comment
claiming one hash per brain start. This alone predicts 61.29 MiB/s reads; measured brain reads
were 63.91 MiB/s.

B. Periodic Registry feeds create historical stream state that is not retired on the sender, is
repeatedly enumerated by the dispatcher, and on receiver replay performs repeated full registry
snapshot rewrites. The live v0.36 log now contains 12 new 15-25s Registry-family
CONN_WRITE_POISONED incidents. This supersedes the earlier clean-window field verdict: the
original Attach-family freeze is fixed, but the same broad blocking/replay failure class remains
for Registry streams.

MEASURED PROCESS ATTRIBUTION
============================
PID 58272:
  command: spt daemon brain --generation 1 --start-reason update
  20-second mean CPU: 15.5% of one logical core
  p95 CPU: 29.2% of one logical core
  read throughput mean: 63.91 MiB/s
  threads: 18-24
  active CPU threads over 5s: 5

PID 72488:
  command: spt daemon run --detached
  20-second mean CPU: 11.4% of one logical core
  p95 CPU: 23.4% of one logical core
  write throughput mean: 3.74 MiB/s
  threads: 532-541 during the first sample; approximately 547-552 over a later 30-second census
  active CPU threads over 5s: only 4

Host: 16 logical CPUs. Task Manager's 0.8-2% whole-machine reading corresponds to approximately
12.8-32% of one logical core, consistent with the psutil process measurements.

This is sustained: 20 consecutive one-second samples reproduced it. It is not a momentary update
spike.

ROOT CAUSE A — HEARTBEAT SELF-HASH LOOP
=======================================
Exact source chain:
  crates/spt-daemon/src/brainproc.rs:63
    BRAIN_HEARTBEAT = 500ms
  brainproc.rs:259-260
    infinite heartbeat loop sleeps 500ms
  brainproc.rs:297
    write_ready(generation) on every heartbeat
  brainproc.rs:365-372
    write_ready calls current_exe_hash every time
  brainproc.rs:381-384
    current_exe_hash does std::fs::read(current_exe) then SHA-256 over all bytes

Binary size:
  32,132,608 bytes = 30.64 MiB
Predicted at two hashes/second:
  61.29 MiB/s
Observed:
  63.91 MiB/s
Prediction error:
  4.1%

The source comment at brainproc.rs:378-380 explicitly says "One exe read + hash per brain
start." Implementation contradicts it by placing the call inside the 500ms heartbeat write.

This explains the brain's constant CPU and read-I/O nearly exactly. It has existed since commit
a52f8c63 introduced exe_hash into write_ready; the heartbeat callsite predates it.

Recommended fix A:
  Compute current_exe_hash exactly once in run_brain after process start, retain the
  Option<String>, and pass/reference the cached value for every readiness breadcrumb rewrite. A
  new brain after update is a new process and therefore computes the new binary hash once,
  preserving the resident-image diagnostic contract.

Regression seam A:
  Inject/count the hash provider or extract a cached readiness payload builder; multiple
  heartbeat writes in one brain generation must invoke executable hashing exactly once while
  keeping pid/generation/hash stable. A new process/generation fixture must compute
  independently.

ROOT CAUSE B — REGISTRY STREAM HISTORY + REPLAY WRITE AMPLIFICATION
==================================================================
Producer lifecycle:
  crates/spt-daemon/src/pump/registry.rs:153-176
    every registry pump round advertises local rows per peer/subnet.
  crates/spt-daemon/src/pump/mod.rs:1116-1122
    push_feed opens a fresh bidi stream, writes the feed, and FINs it; one stream per feed.
  Pump cadence is approximately 30s.

Sender retention:
  The sender-created StreamEntry is not retired after fire-and-forget completion.
  crates/spt-daemon/src/nethost.rs:1825-1855 retire_stream hides eligibility and clears the
  ring, but deliberately retains physical entry/subscriber until QUIC connection close.
  The pump connection is long-lived, so sender rows accumulate.
  nethost.rs:1805-1823 stream_infos filters retired rows but does not server-side exclude
  initiated_local rows.
  crates/spt-daemon/src/dispatch.rs:455-469 polls frequently and receives/clones the full
  historical row set before client-side initiated_local rejection.
  Result: steady-state enumeration cost is O(total feeds since connection start), not O(active
  exchanges).

Receiver replay and disk amplification:
  dispatch.rs:346-369 treats only finished Attach rows as terminal; finished Registry rows
  remain pending by design because request/reply families may FIN request halves.
  dispatch.rs:984-1047 serve_registry_feed subscribes from seq 0 and drains replay.
  nethost.rs:165-221 installs one SubscriberSeat OS writer thread per subscription and writes
  the complete replay before live frames.
  dispatch.rs:997-1039 processes each decoded chunk and separately calls apply_node_labels and
  apply_feed_flips.
  registryhost.rs:241-342: BOTH apply functions call write_snapshots.
  registryhost.rs:734-760: write_snapshots serializes and atomically rewrites every subnet
  registry plus heard.meta.
  Thus a replay split into multiple 64KiB NetStreamData chunks performs repeated full-state
  rewrites, twice per chunk when labels and instances coexist. While the brain does this
  synchronous work, it stops draining IPC; the broker writer fills the pipe, blocks, hits the
  15s write deadline, poisons, and the dispatcher later retries the replay.

LIVE v0.36 FAILURE PROOF
========================
After the earlier clean sampling window, the same generation-1 log accumulated 12
CONN_WRITE_POISONED events. Every classified event is family=Registry, attempt=0, endpoint=-:

  stream 524   elapsed 15,800ms
  stream 1161  elapsed 15,518ms
  stream 1347  elapsed 15,924ms
  stream 3053  elapsed 15,967ms
  stream 3810  elapsed 16,401ms
  stream 5600  elapsed 25,363ms
  stream 5716  elapsed 18,575ms
  stream 5779  elapsed 16,997ms
  stream 6139  elapsed 17,461ms
  stream 6303  elapsed 16,117ms
  stream 6411  elapsed 20,433ms
  plus one interleaved event whose matching DISPATCH_EV was beyond the short association window

Canonical example:
  CONN_WRITE_POISONED: conn=1243 ... stream=524 ... write exceeded its 15000ms bound
  event=stream-sub-writer-poison replay write failed
  event=transport-close poisoned=true timed_out=true
  DISPATCH_EV: gen=1784265298871 stream=524 family=Registry attempt=0 conn=12
  event=worker-outcome detail=Failed("failed to fill whole buffer") elapsed_ms=15800 endpoint=-

The events prove the structured key set is emitted, but they do NOT close the prior
endpoint-identity residual: Registry has no endpoint concept, so endpoint=- is the honest value.
The passive residual remains pending until the first natural failure in an endpoint-bearing
family (Attach/Rest/Xfer/ShellLink/WanMsg). The transport/replay behavior is independently
defective.

Primary evidence:
  C:\Users\decid\AppData\Local\spt-core\logs\daemon.stderr.log
  first poison line: 3136
  latest quantified poison: line 33285 at sampling time

Recommended fix B, ordered:
  1. Stop sender-history growth: retire locally initiated fire-and-forget Registry feed rows
     immediately after successful FIN, or give this one-way family explicit terminal lifecycle
     semantics.
  2. Filter initiated_local and retired rows inside broker stream_infos before
     cloning/serializing them to the dispatcher. Do not pay O(history) IPC cost merely to skip
     client-side.
  3. Make registry feed application transactional per feed: merge decoded labels/instances in
     memory and write snapshots once at EOF (or at one bounded batch commit), not separately for
     labels and instances on every transport chunk.
  4. Ensure failed Registry replay cannot remain an indefinitely redispatchable poison loop. Fix
     lifecycle/write amplification at source rather than merely increasing the 15s deadline.
  5. Audit physical StreamEntry/subscriber teardown on successful one-way completion;
     eligibility retirement alone deliberately retains entries until connection close and
     therefore cannot bound long-lived pump-connection resources.

Regression seam B:
  Real long-lived pump + dispatcher integration test. Run N registry-only rounds over one
  persistent connection. Assert:
  - dispatcher-eligible Registry rows plateau at O(active), not N;
  - physical stream/subscriber/handler counts plateau after completion;
  - snapshot writes are O(feeds), not O(chunks × record-kinds);
  - brain refresh produces no historical Registry replay subscriptions;
  - zero CONN_WRITE_POISONED / replay-write-failed events;
  - broker thread/handle count returns to a bounded baseline.

REQUESTED CARRIER + THREAD DISCRIMINATORS
========================================
All 12 poison events belong to fresh physical broker connections:
  1243, 2665, 3072, 6648, 8236, 12009, 12266, 12409, 12563, 13200, 13542, 13782
They cover 12 distinct Registry streams. None of those physical connections remains open.

All matching DISPATCH_EV records report attempt=0. No breaker-trip record brackets any poison.
Dispatcher-worker IPC connections are reused in cohorts (12, 16, 26, 33, 46, 55), but the
poisoned carrier connections are all distinct. This is fresh-carrier historical-replay churn,
not one wedged connection or an escalating retry ladder. Stream 5846's delayed worker outcome
was 53,009ms; the other classified outcomes were 15,518-25,363ms.

Live Win32 GetThreadDescription census for broker PID 72488:
  total                 589
  unnamed               546
  conn-watchdog          33
  spt-broker-net          7
  main                    1
  spt-docs-server         1
  linger pool (persist.)  1

The source names conn-watchdog and network runtime threads; Broker accept handlers and
SubscriberSeat writers are unnamed. Whole-log lifecycle accounting found only 30 currently open
BrokerConn handlers, 13 of them stream subscribers, and zero poisoned connections still open.
The dispatcher worker pool is capped at 8. Therefore the roughly 500 excess unnamed sleeping
threads fit retained SubscriberSeat writers attached to completed Registry StreamEntries on
long-lived QUIC connections. This is stronger than the earlier short-term "stable thread count"
observation: the count grows by periodic feed history, then plateaus only between pump
rounds/connection changes.

HYPOTHESIS DISPOSITION
======================
1. Historical stream-worker leak: PARTIALLY CONFIRMED. Thread count was stable over a short 30s
   sample rather than monotonically leaking each second, but long-lived Registry stream
   entries/history accumulate per pump round and drive O(history) enumeration/replay. Hundreds
   of sleeping broker threads are not explained by the 8-worker dispatcher pool alone.
2. PTY output replay: FALSIFIED as primary cause. The high brain read rate matches executable
   hashing, not daemon-log or PTY-file growth.
3. Zero-timeout IPC spin: FALSIFIED as primary cause. CPU is distributed across a few active
   threads and exact self-hash throughput predicts the brain load.
4. Normal 4-5-session streaming: FALSIFIED. The self-hash load is session-independent; Registry
   work scales with historical feeds; fresh 15-25s poisons prove pathological backpressure.

SEVERITY / MILESTONE DISPOSITION REQUEST
========================================
Classify A as a clear always-on CPU/I/O regression with a narrow fix.
Classify B as a REDISPATCH/Registry lifecycle and write-amplification defect. The earlier v0.36
milestone was closed from a clean short window; the later 12-event natural sample invalidates
the claim that the broad poison fingerprint is dead. Attach/PTy calm remains valid, but
Registry-family poison recurrence requires explicit triage and a release disposition.

No production process was stopped, no hosted session was killed, and no source was edited
during this diagnosis.
```