{
  "summary": "Root-cause investigation outcome: `net_up`, subnet node counts, empty/non-empty node arrays, endpoint visibility, and actual peer reachability come from different state planes. `net_up=true` proves only that the broker owns a locally bound `NetHost`; it does not prove a relay session, any live connection, an accepted seed-proof, a successful dial, or registry convergence. The reported SPT_DEV=4 and BIGNET=3 counts on HFENDULEAM exactly equal durable non-tombstoned roster membership, not reachable peers. `spt subnet status --json` intentionally emits `nodes: []` unless `--nodes` is supplied. Remote endpoints come from persisted registry snapshots; after five minutes without an admitted registry feed, remote rows are evicted, so `endpoint list` naturally collapses to local endpoints.\n\nThe strongest causal chain is an address-bootstrap deletion trap. The pump enumerates peer IDs from `roster.json`, but resolves each target only from the exact `peer-addrs.json` key and otherwise constructs an id-only Iroh address. It never consults the correct `RosterEntry.address` already stored beside that peer. Every failed dial unconditionally deletes that peer's exact cached address. Rebuilding the cache from the roster happens only after a future successful seed-proof connection. Therefore transient failures can erase every direct bootstrap route, and if id-only n0/mDNS discovery is unavailable or broken, isolation becomes self-perpetuating even while the heartbeat remains fresh. The live state fits the steady-state prediction on both HFENDULEAM and ENLYZEAM: exact keys for each other are absent, their current roster entries are correct and carry addresses, and both repeatedly time out dialing each other and all other peers. Historical deletion of those exact keys is strongly implied but cannot be proven without an older `peer-addrs.json` snapshot.\n\nA separate cache-invariant defect is also proven: on both machines the outer key `5ff50e75…` contains an address whose nested `id` is the opposite machine, HF has `5ff… -> ecb39…` and EN has `5ff… -> 14ef…`. The pump never scans nested IDs, so these rows cannot help HF reach EN or EN reach HF. Current code permits this because roster gap-fill writes `RosterEntry.pubkey_hex -> RosterEntry.address` and `PeerAddrStore::put` validates neither shape nor `outer key == nested id`; fill-only semantics then preserve historical corruption after `roster.json` is corrected. This defect amplifies isolation and poisons reachability to the real `5ff…` member, but it did not itself delete the exact HF/EN keys: `drop_seed(ecb…)` and `drop_seed(14ef…)` cannot delete outer `5ff…`.\n\nThe remaining unresolved boundary is what initially and currently makes id-only attempts exceed ten seconds. The same timeout wraps Iroh transport connect plus the mutual seed-proof/roster control stream, so the existing `PUMP_PEER_FAIL:…submit-dial exceeded the 10s bound` token cannot distinguish a QUIC listener/firewall/relay-discovery black hole from a post-QUIC seed-proof/protocol stall. Matching seeds and epochs, empty detach state, healthy bidirectional Tailscale IP reachability, near-zero clock offsets, and both nodes running 0.31.0 lower the probability of membership, clock, detach, and version mismatch. A packet capture or equivalent QUIC-stage observation is the clean discriminator.",
  "files": [
    {
      "path": "spt-core/crates/spt/src/cli.rs",
      "description": "Daemon status at 3967-4244; `net_up` is read from broker `enabled` and defaults true on broker/query failure at 4121-4135 and 4223-4233. Subnet membership/count dataflow at 6713-6763. Node-liveness and probe construction at 6914-7465. JSON intentionally emits empty node arrays without `--nodes` at 7253-7341. Endpoint-list local/remote assembly at 2816-3088."
    },
    {
      "path": "spt-core/crates/spt-daemon/src/broker.rs",
      "description": "`dispatch_net_status` at 3818-3840 sets `enabled=true` solely when `self.net.get()` returns a local `NetHost`; it also computes `conn_count`, but CLI daemon status discards that field."
    },
    {
      "path": "spt-core/crates/spt-daemon/src/msg.rs",
      "description": "`NetStatusReply` at 525-538 defines the actual meanings: `enabled` means broker hosts a network endpoint; `addr` is the local endpoint address; `conn_count` is accepted+dialed live broker connections."
    },
    {
      "path": "spt-core/crates/spt-daemon/src/brainproc.rs",
      "description": "At 234-267, a brain polls local broker net status and starts dispatcher+pump once `enabled`; at 309-322 the pump starts only when `peer_pump` is enabled. `BRAIN_NET_CONSUMERS_UP` therefore proves local consumer startup, not peer reachability."
    },
    {
      "path": "spt-core/crates/spt-daemon/src/pump/mod.rs",
      "description": "Core causal path. Heartbeat writes before peer work and during healthy idle operation at 560-579. Per-round roster fanout and nonblocking dials at 603-690. Failed peers are isolated while heartbeat continues. `PRESENCE_DIAL_FAILED` logs and unconditionally drops the exact cache key at 857-881. Fan targets are non-tombstoned roster IDs at 902-936. Address resolution checks only exact `peer-addrs[peer_hex]`, then id-only discovery, at 940-958. No roster-address fallback exists. Supervisor behavior at 1041-1110. Backoff reaches five minutes but does not restore addresses."
    },
    {
      "path": "spt-core/crates/spt-daemon/src/nethost.rs",
      "description": "At 537-558, `DialPlan::connect` performs Iroh endpoint connect, then mutual seed-proof, then registers the connection. At 1089-1150, the same ten-second timeout wraps the whole operation, making the current timeout token stage-ambiguous. At 559-633, `register_conn` and `PRESENCE_CONNECTED` occur only after proof succeeds."
    },
    {
      "path": "spt-core/crates/spt-daemon/src/seedproofx.rs",
      "description": "Membership proof contract at 192-258 and 396-526. Exact seed/epoch mismatch or protocol failure drops the connection. The address-cache invariant defect is at 862-879: gap-fill blindly writes `e.pubkey_hex -> e.address`. Production sink at 923-936 runs gap-fill only after a successful member connection, explaining why isolation cannot self-repair from the already-correct roster. No key-vs-nested-ID validation exists."
    },
    {
      "path": "spt-core/crates/spt-store/src/peeraddrs.rs",
      "description": "At 1-17 the store is documented as an optional bootstrap cache. `get` is exact-key lookup at 79-82. `put` blindly stores arbitrary JSON under the supplied key at 84-94. `drop_seed` removes only the named exact key at 95-121. Missing/corrupt files degrade to empty/id-only discovery, but nothing here rebuilds from the roster."
    },
    {
      "path": "spt-core/crates/spt-store/src/roster.rs",
      "description": "Durable membership model. `is_member` and `members_in` at 164-201 exclude tombstoned rows. The pump and subnet counts consume this roster even when transport is unreachable."
    },
    {
      "path": "spt-core/crates/spt-daemon/src/registryhost.rs",
      "description": "Inbound registry feeds are admitted only for local subnet + non-tombstoned roster origin at 58-88 and 240-334. Admitted feeds stamp `heard.meta`. Silent foreign rows are evicted and snapshots rewritten after `registry_evict_after` at 608-650; snapshot/heard persistence at 706-745. This explains local-only endpoint visibility as a consequence of transport/feed silence."
    },
    {
      "path": "spt-core/crates/spt-daemon/src/pump/registry.rs",
      "description": "Registry worker advertises own rows once per due round and pushes them only after the pump has a peer connection; silent-peer eviction runs before advertisement at 124-157; per-peer feed push at 161-177."
    },
    {
      "path": "spt-core/crates/spt/src/wansend.rs",
      "description": "Snapshot loader at 689-712 silently ignores missing/unparseable snapshots. `probe_node_serving` at 584-610 uses id-only addressing too, so `subnet status --nodes` is not an independent direct-address check."
    },
    {
      "path": "spt-core/crates/spt-net/src/net/endpoint.rs",
      "description": "Network bind and family gating at 140-430; production uses N0 preset plus mDNS. `addr_for_node_hex` at 529-541 constructs an ID-only `EndpointAddr`; it contains no roster/direct addresses. `conn_remote_addr` at 543-559 captures paths only after a real connection."
    },
    {
      "path": "spt-core/crates/spt-daemon/src/daemon.rs",
      "description": "Production network config at 374-404: relay from daemon config, mDNS enabled, all-interface bind, live seed-proof source. `net_up` is `net.is_some()` at 185. A successful local bind is enough for healthy status."
    },
    {
      "path": "spt-core/crates/spt-store/src/attachment.rs",
      "description": "At 1-17 and 74-103, detached state is distinct from held membership and causes the pump/responder to skip a subnet while the daemon/net endpoint may remain healthy."
    },
    {
      "path": "spt-core/crates/spt-store/src/visibility.rs",
      "description": "At 145-153, endpoint visibility can suppress advertisement/routing of selected endpoints. It does not stop node-to-node connections or erase durable member counts. HF live visibility has empty hide maps, excluding this as the observed cause."
    },
    {
      "path": "spt-core/installer/install.ps1",
      "description": "Windows UDP firewall rule creation at 157-180. The rule is program-path scoped. CLI checks only whether a named rule exists, not whether it is enabled, applies to the active profile, or points to the running binary."
    },
    {
      "path": "spt-core/CHANGELOG.md",
      "description": "0.30.1 at 162-171 fixed one silent peer stalling the whole pump; 0.30.2 at 148-158 made dials concurrent and peer-isolated. These fixes allow the heartbeat to stay healthy while every peer independently fails. Full-mesh membership shipped in 0.3.0 at 882-883; address seeding and restart convergence are documented at 933-935."
    },
    {
      "path": "spt-core/.git/logs/HEAD",
      "description": "Line 348 anchors introduction of roster-to-peer-address reconciliation: commit `0c42869024b7158cca6fc6b204ac8fbb30840922`, epoch `1780953369 -0700`, message `feat(mesh): on-connect roster propagation + peeraddrs reconcile (REQ-MESH-2) — Mesh-D4`."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/daemon.json",
      "description": "HF live config evidence: `relay=n0`, `peer_pump=true`, `registry_pump_period_ms=30000`, `registry_evict_after_ms=300000`, `detached_subnets=[]`."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/identity/subnet.json",
      "description": "HF held memberships: SPT_DEV and BIGNET, both epoch 1. Main-agent comparison established EN has the same seeds and epochs."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/identity/attachment.json",
      "description": "HF live detach set is empty; main-agent comparison established EN is also empty."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/identity/roster.json",
      "description": "HF durable roster explains status exactly: SPT_DEV has four non-tombstoned members; BIGNET has three. Current EN key `ecb39e40…` is present; old EN key `39c018a4…` is tombstoned in both subnets. Main-agent comparison established both current rosters have correct key/address-ID pairs and direct addresses."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/identity/peer-addrs.json",
      "description": "HF cache evidence: exact current EN key `ecb39e40…` is absent. Outer SCELTOUIN key `5ff50e75…` incorrectly contains nested EN ID `ecb39e40…`, relay-only. Main-agent EN mirror found outer `5ff50e75…` incorrectly contains nested HF ID `14efb80c…`, relay-only."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/identity/registry/SPT_DEV.json",
      "description": "HF snapshot contains only HF endpoint instances, although node labels retain EN. This is the direct data source behind local-only endpoint listing."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/identity/registry/BIGNET.json",
      "description": "HF snapshot likewise contains only HF instances; remote endpoint rows have decayed."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/identity/registry/heard.meta",
      "description": "HF last admitted EN feed is `ecb39e40…: 1784008794517` epoch-ms, many hours behind the observed fresh pump heartbeat. This establishes control-plane activity without remote feed freshness."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/identity/pump-heartbeat.json",
      "description": "Observed fresh HF heartbeat `1784069117875` epoch-ms. Its freshness proves loop progress only, not any successful peer operation."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/logs/daemon.stderr.log",
      "description": "Lines 1-6 show brain/pump startup; lines 7-10 show independent ten-second timeouts for several peers including current EN `ecb39e40…` and old EN `39c018a4…`, while the process remains up."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/logs/daemon.stderr.log.1",
      "description": "Lines 1-12 show an older run where EN timed out and another peer caused a pump read-deadline failure followed by supervised restart. Lines 20-22 and later `PAIR_MEET_UP` records show rotating pre-trust meet endpoints binding; these are separate identities/ALPNs and do not prove the stable SPT endpoint accepts member traffic."
    }
  ],
  "architecture": "CAUSAL DATAFLOW\n\n1. Daemon process and local network status\n\n`Daemon::run` loads the durable node key, calls `try_start_net`, and sets `net_up = net.is_some()` (`daemon.rs:168-185`). `NetHost::start` binds a local Iroh endpoint. The broker stores it in `self.net`; `dispatch_net_status` reports `enabled=true` whenever that local object exists (`broker.rs:3818-3834`). The CLI reports that bit as `net_up`, ignoring `addr` and `conn_count`, and even defaults it to true if broker IPC/status query fails (`cli.rs:4121-4135`). Thus:\n\n`net_up=true` => local broker owns a bound network endpoint\n\nIt does NOT imply:\n- relay registration is usable;\n- mDNS/n0 can resolve another node;\n- Windows permits inbound UDP to the running executable;\n- a QUIC handshake completed;\n- seed-proof completed;\n- `conn_count > 0`;\n- any registry feed was sent or admitted.\n\n2. Peer-pump liveness\n\nThe brain starts the dispatcher and pump once local `enabled=true` (`brainproc.rs:254-267`). The pump writes `pump-heartbeat.json` before loading/fanning out peer work, including idle ticks (`pump/mod.rs:560-579`). Current 0.31 behavior submits peers concurrently and turns each transport/protocol failure into a peer-local backoff (`pump/mod.rs:609-690, 722-790, 857-881`). Therefore a loop that wakes, tries every peer, times every peer out, and backs each off remains heartbeat-healthy. This is intentional after the 0.30.1/0.30.2 fixes, not proof of connectivity.\n\n3. Durable membership and status counts\n\n`subnet_status_rows` iterates held subnet records, inserts self, inserts every non-tombstoned `roster.members_in(subnet)`, then unions any nodes found in registry snapshots (`cli.rs:6731-6762`). It reports the set size, not a live-connection count. HF's live roster yields exactly the observed values:\n- SPT_DEV: HF `14ef…`, current EN `ecb39…`, KITSUBITO `cfd9…`, SCELTOUIN `5ff5…` = 4.\n- BIGNET: HF `14ef…`, current EN `ecb39…`, GRAVITY-RUNNER `c893…` = 3.\n- Old EN `39c0…` is tombstoned and excluded.\n\nThe JSON DTO always has a `nodes` field, but the non-`--nodes` branch explicitly assigns `Vec::new()` (`cli.rs:7319-7341`). Therefore `spt subnet status --json` with `node_count > 0` and `nodes: []` is correct API behavior, not evidence that the runtime discovered zero nodes. The command needed for per-node liveness is `spt subnet status --nodes --json`; note that it actively performs bounded network probes for stale rows, so it is not a purely filesystem-only observation.\n\n4. Endpoint visibility\n\nLocal endpoint rows come from local perch/roster truth. Remote endpoint rows come only from `identity/registry/<subnet>.json`, passed through visibility and routability projection (`cli.rs:3015-3078`; `registry.rs:566-606`). Inbound admitted registry feeds stamp `heard.meta`. Once a peer has not been heard for `registry_evict_after` (HF config: 300,000 ms), `evict_silent_peers` deletes that peer's registry instances and rewrites snapshots (`registryhost.rs:608-650`). Node labels can remain while endpoint rows disappear. Consequently local-only `endpoint list` is the expected downstream symptom of no admitted feeds for over five minutes; it does not establish that durable membership vanished.\n\n5. Peer dialing and the self-perpetuating bootstrap gap\n\nFor each non-detached subnet, the pump gets target IDs from `roster.members_in`, excluding self and tombstones (`pump/mod.rs:902-936`). For each target `peer_hex`, it resolves exactly one address:\n\n`peer-addrs.json.addrs[peer_hex]` if present\notherwise `EndpointAddr::new(peer_id)` for id-only n0/mDNS discovery\n\nThis is `resolve_submit_addr` at `pump/mod.rs:940-958`. It never reads `RosterEntry.address`, even though the roster carries a correct durable address for the same target.\n\nOn every `PRESENCE_DIAL_FAILED`, the pump:\n- logs `PUMP_PEER_FAIL`;\n- schedules peer-local exponential backoff;\n- calls `PeerAddrStore::drop_seed(peer_hex)` unconditionally (`pump/mod.rs:869-880`).\n\nThe next attempt is necessarily id-only. The only steady-state path that copies correct roster addresses back into `peer-addrs.json` is `production_roster_exchange`'s sink, which runs after a successful QUIC + seed-proof connection (`seedproofx.rs:923-936`). That yields a closed recovery loop:\n\ncached direct address fails -> exact key deleted -> id-only discovery fails -> no connection -> roster gap-fill cannot run -> exact key stays absent -> repeat\n\nThis explains how a transient fleet-wide event can become durable global isolation while all local health fields remain green. Current missing exact HF/EN cache keys plus correct current roster addresses are the precise steady-state signature. The all-peer ten-second timeouts on both nodes show the common id-only path is not recovering. What is not directly evidenced is the historical moment at which each exact key was deleted.\n\n6. Independent malformed-cache invariant\n\nOn both nodes, the one malformed cache row uses outer key `5ff50e75…` but contains the opposite machine's nested endpoint ID. The pump performs no nested-ID scan, so that row is useless for the opposite target. It also means the actual `5ff…` target has a cache hit whose TLS/QUIC endpoint identity is not `5ff…`, guaranteeing failure or refusal.\n\nCurrent code admits this state:\n- roster gap-fill writes `pa.put(e.pubkey_hex, e.address.clone())` without checking the serialized address ID (`seedproofx.rs:870-879`);\n- `PeerAddrStore::put` accepts opaque JSON under any key (`peeraddrs.rs:84-94`);\n- gap-fill is fill-only, so a malformed existing `5ff…` row is never replaced by a later correct roster row;\n- a successful direct `5ff…` connection would overwrite it, but the malformed row prevents that bootstrap.\n\nBoth current rosters are now correct, so the exact historical bad input is gone. The likely historical writer is the roster gap-fill path during a transient bad incoming `5ff…` roster entry; a later higher-lease roster merge corrected `roster.json` while fill-only peer cache state remained. This is an inference, not a proven historical event. Pairing and presence writers derive the key and address from the same live connection and therefore do not naturally create this cross-key mapping. Concurrent cache writes can lose an update but cannot transpose a key and nested ID.\n\nThe capability was introduced by commit `0c42869024b7158cca6fc6b204ac8fbb30840922` (`.git/logs/HEAD:348`), the Mesh-D4 roster/peer-address reconciliation change, and shipped with the 0.3.0 full-mesh model (`CHANGELOG.md:882-883`).\n\nRANKED HYPOTHESES\n\n1. HIGH: Direct-address deletion cascade plus unusable id-only discovery.\n\nEvidence:\n- Exact HF and EN cache keys are absent on the opposite machines.\n- Both correct rosters still carry the addresses, but code never consults them while dialing.\n- Both logs show repeated all-peer ten-second failures.\n- Failure unconditionally deletes each target's exact seed; no roster recovery can run without success.\n- Fresh heartbeat and `net_up` are expected under this failure mode.\n\nPredictions:\n- For every repeatedly failing peer, either the exact cache key is absent or was removed after the first failure.\n- Correct direct addresses may still exist in `roster.json` and remain unused.\n- Clearing `peer-addrs.json` alone will not repair the node; it forces all peers to id-only and likely leaves behavior unchanged or worse.\n- A controlled cache rebuilt from verified-current roster/live endpoint addresses would move attempts past address discovery. If those ports are stale, the first failure will delete them and return to id-only.\n- If id-only n0/mDNS starts working, a success will immediately repopulate the exact target key from `PRESENCE_CONNECTED` and convergence will resume within a pump cadence.\n\n2. HIGH-MEDIUM: Stable SPT QUIC transport/listener path is black-holed by firewall, current-port discovery, relay registration, or ALPN reachability.\n\nEvidence:\n- Both nodes can bind network and rotating meet endpoints, and Tailscale IP transport is healthy, but every stable peer dial times out.\n- The Windows installer itself documents detached daemons being silently unreachable without a program-scoped inbound UDP rule (`install.ps1:157-180`).\n- `net_up` does not test incoming UDP, relay registration, or a remote handshake.\n- `PAIR_MEET_UP` uses temporary token-derived identities and the pre-trust meet ALPN, not the stable node ID/member ALPN.\n- `NET_FAMILY_GATE` selected IPv4-only on HF, making current IPv4 UDP listener/rule/path evidence especially relevant.\n\nPredictions:\n- Packet capture shows outbound QUIC Initial packets but no peer Handshake response if transport/listener/firewall is the boundary.\n- The named firewall rule may exist yet be disabled, profile-inapplicable, or point to a different executable. CLI's existence-only check will miss this, and JSON status never renders the human warning.\n- Stable broker UDP ports may differ from cached/roster ports after restart. Meet-listener ports rotating every 30 seconds do not identify the stable SPT port.\n- A direct current stable `EndpointAddr` will fail before any seed-proof stream traffic if this hypothesis is correct.\n\n3. MEDIUM-LOW: QUIC connects, but mutual seed-proof or roster-control protocol stalls inside the same ten-second envelope.\n\nEvidence:\n- The timeout covers both `endpoint.connect` and `prove_membership`, so logs cannot exclude it.\n- Connection registration and `conn_count` occur only after proof, so zero live connections does not distinguish transport from proof.\n\nAgainst:\n- HF and EN seeds and epochs match exactly for both shared subnets.\n- Both run 0.31.0.\n- Clock offsets are irrelevant to steady-state seed-proof and are near zero anyway.\n- A clean proof denial should normally produce `seed-proof failed: peer is not a subnet member`, not only the outer timeout, if the exchange completes.\n\nPredictions:\n- Packet capture shows a completed QUIC handshake, followed by a bidi control stream and then silence/stall.\n- Any exact seed/epoch mismatch, tombstone of the current peer key, or asymmetric protocol branch would reproduce on every direct address.\n- Instrumentation would need stage-specific timing to separate `connect`, proof Hello, proof sets, roster exchange, and seed-transfer. Current production logs do not provide it.\n\n4. LOW / state-falsified on HF and EN: Detachment, tombstoning current identities, disabled pump, or endpoint visibility policy.\n\nCapability:\n- Detachment skips the entire subnet fanout while local net stays bound (`pump/mod.rs:907-914`).\n- Tombstones remove targets from `members_in` and inbound authorization.\n- `peer_pump=false` prevents pump spawn.\n- Visibility can hide all endpoints while nodes remain members.\n\nObserved contradiction:\n- Both `attachment.json` and `daemon.json.detached_subnets` are empty.\n- Both have `peer_pump=true` and relay n0.\n- Current HF/EN keys are non-tombstoned; only old EN `39c…` is tombstoned.\n- HF visibility hide maps are empty.\n- Logs show actual dial attempts, proving peers are fan targets.\n\n5. LOW as current primary cause: Whole-pump wedge or stale broker image.\n\nCapability/history:\n- Rotated HF log lines 7-12 show an older peer reply-read failure causing `PEER_PUMP_FAIL` and a supervised restart.\n- Changelog 0.30.1 and 0.30.2 explicitly fixed whole-round poisoning and serial head-of-line blocking.\n\nObserved contradiction:\n- Current 0.31.0 logs show independent `PUMP_PEER_FAIL` rows while the heartbeat stays fresh.\n- Broker image matches 0.31.0.\n- A current whole-pump wedge would stale the heartbeat beyond three 30-second registry cadences; observed heartbeat is fresh.\n\nREAD-ONLY DISCRIMINATING PROBES\n\nA. Validate every cache key/ID invariant on both hosts, without touching SPT:\n\nPowerShell concept:\n`$j = Get-Content \"$env:LOCALAPPDATA\\spt-core\\identity\\peer-addrs.json\" -Raw | ConvertFrom-Json; $j.addrs.psobject.Properties | ForEach-Object { [pscustomobject]@{ outer=$_.Name; inner=$_.Value.id; match=($_.Name -eq $_.Value.id) } }`\n\nResult discriminator:\n- `match=false` proves malformed cache rows.\n- Exact remote key absent proves pump will use id-only, regardless of a matching nested ID elsewhere.\n\nB. Compare roster recoverability versus actual dial input:\n- Read `identity/roster.json` and locate the exact current remote pubkey in each shared subnet.\n- Compare its `address.id` and addresses with `peer-addrs.json.addrs.<exact-key>`.\n- If roster is correct and cache exact-key is absent, the recovery-gap hypothesis is directly satisfied because code does not consult roster addresses.\n- Record file `LastWriteTimeUtc` and hash for forensic correlation only; do not rewrite it.\n\nC. Verify firewall semantics, not merely rule existence:\n- `netsh advfirewall firewall show rule name=\"spt-core inbound UDP\" verbose`\n- Compare Enabled, Direction, Profiles, Protocol, and Program against the actual running broker executable path on HF and EN.\n- This is more discriminating than the CLI check at `cli.rs:6888-6904`, which asks only whether `netsh show rule` exits successfully.\n\nD. Correlate current stable UDP listeners with cached ports:\n- Read broker PID from daemon status/PID file.\n- Use `Get-NetUDPEndpoint` filtered by that owning PID.\n- Distinguish the stable broker listener from the separately rotating `PAIR_MEET_UP` sockets.\n- Compare the stable port to roster/cache direct addresses. A stale cached port predicts timeout and then exact-key deletion.\n\nE. Observe QUIC stage without changing SPT state:\n- Capture UDP traffic between HF and EN during an already-occurring pump attempt, preferably on Tailscale and physical adapters.\n- Outbound Initial with no valid response => address/listener/firewall/relay layer.\n- Completed QUIC handshake followed by stream traffic and a ten-second stall => seed-proof/roster protocol layer.\n- No packets toward any current direct address, only discovery/relay traffic => exact-key miss and id-only fallback.\n\nF. Compare credential/gate state without publishing secrets:\n- Compare subnet names, seed epochs, and cryptographic hashes of seed values between hosts.\n- Confirm current peer keys are not tombstoned and held subnets are not detached.\n- This has already been done for HF/EN: seeds/epochs match and detached sets are empty, substantially lowering hypotheses 3 and 4.\n\nG. Log signatures:\n- `PUMP_PEER_FAIL:<id>:submit-dial exceeded the 10s bound` is stage-ambiguous.\n- `PUMP_PEER_FAIL:<id>:connect failed: No addressing information available` points to address discovery before membership success.\n- `seed-proof failed: peer is not a subnet member` points to completed transport plus failed/empty proof.\n- `REGISTRY_EVICT:<subnet>: ...` confirms endpoint disappearance is silent-feed cleanup, not membership removal.\n- `PEER_PUMP_FAIL` / `PEER_PUMP_RESTART` indicates a whole-pump carrier failure; fresh heartbeat with only `PUMP_PEER_FAIL` indicates per-peer failure isolation.\n\nH. Do not use cache deletion as the first diagnostic:\n- The store is correctness-optional, so clearing it is structurally safe for durable membership/secrets.\n- Operationally it removes all remaining direct bootstrap routes.\n- The current code does not reconstruct it from `roster.json` on startup or next tick; reconstruction requires a successful member connection.\n- Prediction: clearing alone leaves the nodes id-only and therefore does not distinguish or repair the observed state. A controlled, reversible experiment with verified-current direct addresses would be sharper, but it mutates live state and was not performed under this assignment.\n\nConclusion: local health reporting is behaving according to its current definitions, but those definitions do not cover transport success. The highest-value engineering target is the recovery gap between durable roster addresses and the dial cache: validate key/address identity on write and load, consult a correct roster address before id-only fallback, and avoid permanently erasing the only direct bootstrap route without a usable replacement. No remediation was performed."
}