# Mesh Isolation RCA: Peer-Address Bootstrap Deletion Trap

**Owner:** hertz  
**Date:** 2026-07-10  
**Status:** Self-perpetuating isolation mechanism pinned; initial transient and exact ID-only failure stage not yet pinned

## Problem

HFENDULEAM and ENLYZEAM became entirely sequestered from every remote node on their member subnets, including each other, while `spt daemon status` reported `running: true`, `net_up: true`, fresh pump heartbeat, and normal durable subnet counts.

## Reproduction

`spt subnet status --nodes --json` produced symmetric results:

- HFENDULEAM reported only itself online and every remote node offline.
- ENLYZEAM reported only itself online and every remote node offline.
- Daemon logs repeatedly reported `PUMP_PEER_FAIL:<peer>:submit-dial exceeded the 10s bound` for all targets.

Basic transport prerequisites were present:

- Tailscale HFENDULEAM↔ENLYZEAM direct ping: approximately 12 ms both directions.
- Both reached a healthy reference machine.
- Both daemon PIDs owned UDP listeners.
- HFENDULEAM's inbound SPT UDP firewall rule was enabled for all profiles.
- SPT_DEV and BIGNET seeds/epochs matched.
- `detached` was empty on both machines.
- Clock offsets were approximately +0.003 s and −0.135 s, far below the 30-second rendezvous step.
- Both ran SPT/broker image 0.31.0.

## Pinned steady-state root

The peer pump creates a recovery trap:

1. Targets come from durable roster IDs (`crates/spt-daemon/src/pump/mod.rs:920-936`).
2. Address resolution checks only the exact `peer-addrs[peer_hex]` entry, then falls back to an ID-only Iroh address (`pump/mod.rs:940-958`). It does not consult the valid `RosterEntry.address`.
3. Every failed presence dial unconditionally deletes the exact cached address (`pump/mod.rs:869-880`).
4. The next attempt is therefore ID-only.
5. Correct roster addresses are copied into the peer cache only after a future successful seed-proof connection (`seedproofx.rs:923-936`).
6. If ID-only discovery/connect is unavailable or stalled, isolation prevents its own repair. Per-peer backoff reduces retry frequency but restores no route.

Live state satisfies the predicted steady state:

- HFENDULEAM's peer-address cache lacked ENLYZEAM's exact key.
- ENLYZEAM's cache lacked HFENDULEAM's exact key.
- Both `roster.json` files contained correct, current direct addresses and matching key/address identities.
- The pump ignored those roster addresses and continued all-peer ten-second timeouts.

The original transient that caused the first exact-address deletion cannot be proven without an older cache snapshot. The durable deletion/recovery trap is code- and state-grounded.

## Independent cache-invariant defect

Both caches also contained a persisted invalid row under SCELTOUIN's outer key `5ff50e75…`:

- HFENDULEAM: outer `5ff…` → nested ENLYZEAM ID `ecb39…`.
- ENLYZEAM: outer `5ff…` → nested HFENDULEAM ID `14ef…`.

Current rosters are correct. `gapfill_peeraddrs` can persist `RosterEntry.pubkey_hex -> RosterEntry.address`, and `PeerAddrStore::put` validates neither the outer key nor `outer key == address.id` (`seedproofx.rs:862-879`; `peeraddrs.rs:80-93`). Fill-only behavior preserves historical corruption after the roster is corrected.

This defect poisons reachability to the real `5ff…` member and proves the cache accepts invalid mappings. It amplifies isolation but did not cause deletion of the current HFENDULEAM/ENLYZEAM keys; its historical producer remains unproved.

## Why status remains green

The surfaces report different state planes:

- `net_up` means only that the broker owns a local `NetHost` (`broker.rs:3818-3840`); it does not require any live peer.
- The pump heartbeat is written before fanout and stays fresh when all dials fail (`pump/mod.rs:560-579`).
- Subnet counts union durable local/roster/registry membership, not reachable nodes (`cli.rs:6713-6763`).
- JSON intentionally emits `nodes: []` unless `--nodes` is supplied (`cli.rs:7253-7341`).
- Remote endpoint rows are evicted after five minutes without admitted registry traffic (`registryhost.rs:608-650`), explaining local-only endpoint lists.

## Agreed production plan

This is one milestone contract because the documented REQ-CONV-1 premise—“a stale addr never strands a peer”—is falsified by the RCA. The plan must update the design invariant and its registry/CONTEXT evidence, not only code.

### Wave 1: close the bootstrap trap

1. Resolve through exact peer cache, then validated `RosterEntry.address`, then ID-only discovery.
2. Do not destructively forget the only bootstrap route after one failed dial; retain a reconstructible last-known route.
3. Reconcile validated roster addresses at startup and after roster merge; recovery must not require an already-successful connection.

Alternative-rotation machinery is explicitly out of scope unless a post-fix incident demonstrates a need. Retention plus roster fallback closes the pinned trap without speculative policy.

### Wave 2: enforce and migrate the cache invariant

1. Enforce `outer peer key == address.id` on load and write; repair or reject invalid rows.
2. Rebuild validated cache rows from the current roster. Do not merely delete `peer-addrs.json`, because current cold recovery depends on failing ID-only discovery.

### Wave 3: prove transport stage and user-meaningful health

1. Split the combined ten-second failure into address-resolution, QUIC connect, ALPN, seed-proof send, seed-proof receive/verify, and roster-exchange stages.
2. Report live peer count, last successful peer dial, last admitted registry update, and duration of all-peer failure.
3. Keep truthful health in this same contract: it is the acceptance surface for Waves 1–2, not a detached observability follow-up.

## Regression test

At the pump resolver/failure lifecycle seam:

1. Seed a valid roster and matching cache address.
2. Force one transient dial failure.
3. Disable ID-only discovery.
4. Assert the next attempt still has the roster-derived route.
5. Fail every peer once, restore listeners, and assert automatic convergence without state surgery.
6. Seed an outer-key/nested-ID mismatch and assert repair or rejection.
7. Verify health becomes degraded during all-peer failure and healthy only after real peer progress.

## Remaining discriminator

The existing ten-second token does not reveal whether current ID-only attempts stall during QUIC connection or after transport during seed proof. A packet capture or stage-specific instrumentation is required. That boundary does not change the pinned bootstrap deletion trap or its fix shape.

Build-time cross-check: Todlando preserved the 2026-07-14 HFENDULEAM pump-stall evidence at `C:/spt-diag/pump-stall-2026-07-14/`. Its restart-cured behavior may fit this class and should be compared against the planned lifecycle assertions.