# ER bring-up residual — design round 2 (todlando → doyle, 2026-08-19) **NOTHING BUILT.** This is the design. Build starts on your ruling. All source read at **`bd942f6`** — the sha your x20 ran. Stated because I nearly designed against the wrong tree: `main@80ab77a` does NOT contain this work (`git diff bd942f6 80ab77a` = −349 lines across `rc.rs`/`broker.rs`/the cell), and `.worktrees/keystone-w1@ba2d998` differs again (`broker.rs` +125, the cell +81). Line numbers below are `bd942f6` blobs, extracted with `git show`. --- ## 0. What the logs said that my brief did not have I read all 20 of `/rca-v2-runs/run-N.log` directly. **All four failures panic at the SAME line**, `engine_room_bringup_e2e.rs:1213:5`: ``` assert!(offline.spawned, "THE INVARIANT: ...") ``` Three consequences, and they are measured, not inferred: 1. **Only the OFFLINE arm ever fails.** `assert!(cleared.spawned)` at :1194 is evaluated first and passed in 4/4. The control arm brought the room up every single time. 2. **The broker precondition passed in 4/4.** `truth_before == None` (:1183–1192) is asserted for BOTH arms and sits above :1213. So at the instant of the measured rc, the broker reported **no live session** — and rc refused anyway. Broker-side clean, disk-side dirty. That is the asymmetry the JIT predicted; it is now a measurement. 3. **The residue is intra-arm.** The offline arm is the only arm that runs a *prior controller* (the warm `rc_once` → kill → `endpoint stop`). The control arm never has one and never fails. So the residue is left by **this arm's own warm controller**, not by a neighbour cell, a stale global, or the box. Combined with your no-preceding-cell x20, cross-cell contamination is refuted from both ends. The refusal text is `'engine-room' is controlled by **another window on this machine**.` — `driver_phrase` (rc.rs:1270) emits that branch only when `is_own_node_hex(driver, own_hex)` is true. So the latched `driven_by` carries **our own node hex**, written by a local controller. --- ## 1. Open question 4 (the 60–65s bimodality) — ANSWERED, and it is the rig, not the product Arithmetic off the rig, no theory: - The measured rc is a **spawned child** (`rc.child`, killed later at :607) — the test does not block on it. - On a non-spawn, `wait_for_broker_session(id, 45s)` (:601) burns its **full 45s**, then `wait_for_session_rotation(&perch, &sid_before, 10s)` (:604) burns its **full 10s**. `55s` of pure timeout tail, in the offline arm only. - The control arm then runs normally. `55 + [5.8..10.0] = [60.8..65.0]`. Measured failures: **60.9, 62.8, 65.5, 65.5**. Fits the pass band to within 0.5s. **So the bimodality is the rig's own timeout tail. It carries zero product signal — and, importantly, it DISCRIMINATES NOTHING between the two faces**, because rc's internal 30s erhost bound runs *concurrently* with the test's 45s poll. Both faces land in the same band by construction. I am retiring open question 4 as answered; it should not be treated as a clue again. --- ## 2. `set_driven_by` census — cfg(test)-aware, at `bd942f6` `#[cfg(test)] mod tests` is a **col-0 item with no col-0 item after it** in all three files, so each test module runs to EOF. Regions computed, not grepped: | file | `#[cfg(test)]` at | EOF | ⇒ test region | |---|---|---|---| | `spt-daemon/src/broker.rs` | 7603 | 12131 | 7603–EOF | | `spt-daemon/src/livehost.rs` | 1309 | 2742 | 1309–EOF | | `spt-store/src/info.rs` | 853 | 1780 | 853–EOF | | `spt/src/api/reporting.rs` | 1200 | — | 1200–EOF | **PRODUCTION call sites — five, and only TWO of them stamp:** | site | call | what it is | |---|---|---| | `broker.rs:3058` | `set_driven_by(&perch, self.controller_by())` | `OutputLog::stamp_driven_by` — **the stamper**. Attach-time. | | `broker.rs:1033` | `set_driven_by(&perch, want_driven_by)` | `converge_perch_stamps` — divergence-gated converge off the `KIND_SESSIONS` poll. Stamps *or* clears. | | `broker.rs:3104` | `set_driven_by(&perch, None)` | `OutputLog::stamp_reaped` — clear on session reap. | | `livehost.rs:822` | `set_driven_by(&perch, None)` | reconcile tick, `CONTROL_REAP_NO_SESSION` split cleanup. | | `livehost.rs:988` | `set_driven_by(&perch, None)` | reconcile tick, `DRIVEN_BY_SELFHEAL_OFFLINE` belt (REQ-HAZARD-DRIVEN-BY-SELFHEAL). | Everything else (`broker.rs:7783,9202`, `livehost.rs:1489,1605,1705`, `info.rs:1064,1404,1410,1419`, `reporting.rs:1737`, and all of `spt-daemon/tests/*`) is test code. **Correction to my own JIT, stated plainly.** The JIT cited `info.rs:141-147` as "a local-only controller must NOT latch `driven_by`". That reading is **stale**. `broker.rs:3038-3050` rules the opposite, by name: > `driven_by` = the CONTROLLING NODE, own node INCLUDED … the own-hex latch is TRUTHFUL — a local > controller IS a controller (doyle ruling v3, 2026-07-19). An earlier remote-only claim here was a > legacy-local-path (`by = None`) artifact, not the model. The stale-latch hazard (KH 7.15 / > REQ-HAZARD-DRIVEN-BY-SELFHEAL) is about clearing this stamp **when the SESSION is gone** — it does not > make `driven_by` remote-only. So the stamp our warm controller wrote was **correct**. The defect is entirely on the clearing side, and KH 7.15 is exactly on point in its *real* form. --- ## 3. Open question 2 (missing / racing / late) — the leading mechanism, derived from source **Every clear of `driven_by` lives in the DAEMON. The write that satisfies the rig's precondition happens in the CLI.** `spt endpoint stop` → `cli.rs:11422`: ```rust let _ = spt_store::info::terminal_normalize(&perch_path, None); ``` `terminal_normalize` (`info.rs:483`) writes **exactly three fields**: `status=offline`, `rest_state="suspended"`, `dormant_since_ms=None`. It **does not touch `driven_by`, `controlled`, or `viewer_count`.** It runs synchronously, in the `spt endpoint stop` process, unconditionally. The rig's gate is `wait_for_offline_row` — it polls **`status`** and returns the instant it reads `offline`. That row is the CLI's own write. So: > **The offline arm's precondition is satisfied by a process that structurally cannot clear > `driven_by`, and it then races three daemon-side clears that are all on a different schedule > (`stamp_reaped` on the reap path, `converge_perch_stamps` on the sessions poll, the reconcile tick).** And the reader has no defence: `current_driver` (rc.rs:1312) is ```rust spt_store::info::read_info(&perch).and_then(|i| i.driven_by) ``` — a raw disk read with **no liveness check at all**, feeding `pre_broker_busy_guidance` (rc.rs:1698), which prints and `return Ok(())` **before any broker traffic**. The broker's own truth (`no session`, which the rig proved) is never consulted. That is a race whose outcome depends on daemon tick phase — which is what a 4/20 looks like. **Leading reading: the clear is LATE, not missing.** I am not writing that as established; §5 is the instrument that decides it. ### The second face is NOT this mechanism Run 4's rc printed `the code was accepted, but … erhost did not come up within 30s`. "The code was accepted" is downstream of the busy gate — so on run 4 the gate **passed**, i.e. `current_driver()` returned `None`. Run 4 therefore cannot be the disk-latch mechanism, and it is not "the same mechanism one step earlier" either. It is **1/20**. I am not designing a fix against it. The instrument in §5 captures it; if the capture shows `driven_by = Some` at the refusal on an erhost-not-up failure, my reading here is wrong and the faces merge. I would rather that be a recorded falsification than a guess. --- ## 4. The fix — recommendation, with the alternative stated **Primary (recommended): make the READER refuse only on a live marker.** `current_driver()` returns `None` when the record it read is not live — i.e. when `status` reads `offline`. One reader-side predicate, in `rc.rs`. Why this one: - **It adds no writer.** `driven_by` stays single-writer (the broker). Every alternative that clears from the CLI puts a second writer on the field the broker owns, which is the invariant this codebase has defended repeatedly. - **It does not invent a rule — it echoes one already ratified.** `livehost.rs:983-991` already asserts, in the product, that an offlined endpoint's `driven_by` is stale by definition ("its broker session (the single writer) is gone"). The daemon believes this on a tick. The reader would simply stop having to wait for that tick to agree. - **It is the narrowest change that makes the failing state unreachable**, and it fixes the class, not the rig: any operator hitting a cleanly-stopped endpoint inside the tick window gets the same false refusal today, with no test involved. - `--view`/`--take`/qualified targets are unaffected (they already bypass the gate, `rc.rs:2982-2996`). **Belt (yours to rule, I do not need it): also clear the control triple in the stop path.** `cli.rs` already calls the stop a "DEFINITIVE death observation", and `terminal_normalize`'s own doc names `endpoint stop` as one of the three authoritative death observers. A definitive death observer that normalizes the *rest* triple and leaves the *control* triple standing is incomplete by its own charter. The no-live-session race-free argument used at `livehost.rs:812-816` applies verbatim here. **But it is a second writer on a broker-owned field, so I will not take it without your ruling.** **Explicitly rejected: gating the rig on `driven_by == None`.** That would make the cell green by teaching it to wait for the defect to clear. The false refusal is real and operator-visible. --- ## 5. Instrument BEFORE fix — and the confirming vector, pre-registered Pure capture. No new assertion, no change to any existing one, no product change. Carried out on `BringupAttempt` (nothing may panic between `spawn_broker` and `reap`), printed only through the failure messages that already exist. Capture the ER perch's **full** `info.json` record (`driven_by`, `controlled`, `viewer_count`, `status`, `rest_state`, `session_id`) plus an elapsed-ms stamp, at three points in `attempt_bringup_with_status_row`: - **(i)** the instant `wait_for_offline_row` returns — i.e. the precondition the arm believes it has - **(ii)** the instant the measured rc child exits — the refusal instant - **(iii)** after `reap` — did the clear ever land Passing runs print nothing under default nextest capture, so the arms are run with `--success-output immediate` to get the pass vectors too. That is a run flag, not a code change. **Pre-registered, before the run:** | capture | verdict | |---|---| | seat-contention fail, (i) `driven_by=Some()` **and** `status=offline`, (ii) still `Some()` | **CONFIRMS** the latch mechanism | | …and (iii) `driven_by=None` with no other intervention | **CONFIRMS "LATE"** — the clear does land, just after the reader | | …and (iii) still `Some` | **"MISSING", not late** — §4's primary still fixes the reader, but a clear path is also broken and gets its own defect | | seat-contention fail with (i) `driven_by=None` | **REFUTES the whole hypothesis.** The driver came from somewhere other than the disk record and §3–§4 are void | | erhost-not-up fail with (i) `driven_by=Some` | **REFUTES §3's split** — the two faces are one mechanism and my "the gate passed" reading is wrong | | passing runs show `driven_by=None` at (i) | consistent with the race reading; a passing run with `Some` at (i) would refute it | The capture is **run on the same rig, same box, same scrub**. It is not a fix and does not pre-suppose one. --- ## 6. The e2e join for the fix (after the instrument, after your ruling) The existing cell already expresses the invariant and needs **no change** — the fix must make :1213 green as it stands. What it does not express is the *reader* claim, so the fix carries a unit in `rc.rs`: `current_driver` returns `None` for a record carrying `driven_by=Some` beside `status=offline`, and returns the driver verbatim for an online one. That is the falsifier in both directions, and it does not depend on tick phase, so it is not a flake. REQ: the reader half joins `REQ-HAZARD-DRIVEN-BY-SELFHEAL` (KH 7.15) — it is the same hazard, observed from the client side. I will confirm whether that needs a new `REQ-*` or a stage on the existing one before tagging. --- ## 7. What I am asking you to rule 1. **Instrument first, as specified in §5?** (My default: yes, and I do not build a fix before it runs.) 2. **Primary fix = the reader-side liveness check in `current_driver` (§4)?** 3. **Belt = the stop path clearing the control triple — in, or out?** (Second writer on a broker-owned field; your call, not mine.) 4. **The erhost-not-up face at 1/20 — file it and move on, or is it in scope for this dispatch?** Box is mine; hertz's clippy corrective runs in parallel and I have not touched it. No pool claimed for this yet — I will claim from the rig's own cwd when you rule the build.