{
  "summary": "HFENDULEAM RCA — pinned steady-state mechanism vs unproven trigger\n\nPINNED STEADY STATE\n1. The live broker/brain is locally alive but cannot complete an outbound peer dial. `logs/daemon.stderr.log:1-11` shows generation 1 starting normally (`BRAIN_UP`, `BRAIN_NET_CONSUMERS_UP`) and then immediate 10-second whole-operation timeouts for AEA `aea099…`, ENLYZEAM current key `ecb39e…`, ENLYZEAM old key `39c018…`, and KITSUBITO `cfd9fc…`. The same failures remain present near the live tail: `logs/daemon.stderr.log:643399-643520`; at wall_ms 1784069260078–1784069441492 (2026-07-14 22:47:40.078Z–22:50:41.492Z), ENLYZEAM repeatedly times out and cfd9fc/c8939d also time out. EN retries recur roughly every 15–30 seconds because it is a target in both attached subnets and the pump cadence is 15 seconds.\n2. The error is generated around the *entire* `DialPlan::connect`, not a narrow auth check: `crates/spt-daemon/src/nethost.rs:1125-1151` acquires a dial permit then wraps `plan.connect(addr)` in `tokio::time::timeout`; expiration becomes exactly `submit-dial exceeded the 10s bound (peer unresponsive)`. Therefore the current log proves no connect+proof+register completion, but cannot distinguish address resolution/relay, QUIC handshake, ALPN, or seed-proof without stage instrumentation.\n3. Failure is made durable at the registry/control-plane projection. `identity/registry/heard.meta:1-7` records the last admitted ENLYZEAM feed at 1784008794517 = 2026-07-14 05:59:54.517Z and GRAVITY-RUNNER at 1784008286121 = 05:51:26.121Z. `daemon.json:7-10` sets registry eviction after 300000 ms. `crates/spt-daemon/src/registryhost.rs:607-649` defines silence as no admitted inbound feed for >TTL and then evicts every registry row authored by that peer, logging `REGISTRY_EVICT`. Thus, absent another admitted feed, EN’s foreign endpoint rows become eligible just after 06:04:54.517Z and gravity’s just after 05:56:26.121Z. This directly explains `endpoint list` collapsing to HF-only and subnet views losing remote endpoint rows while durable membership can remain.\n4. `daemon status` green is explicitly only local control-plane/process liveness, not peer reachability. `crates/spt/src/cli.rs:3960-4043` renders `net_up=true` as the net endpoint being bound and declares the pump live solely from a fresh heartbeat. `crates/spt/src/cli.rs:4110-4198` obtains `net_up` from broker `net_status`, reads the heartbeat file, and constructs subnet counts from `SubnetStore` + registry + durable roster; it performs no successful remote handshake requirement. Consequently a bound socket, live scheduler, fresh heartbeat, and persisted member counts can all remain healthy while every transport route fails.\n5. Local meet activity is not evidence of cross-node reachability. At `logs/daemon.stderr.log:643412-643415`, `643427-643433`, `643449-643454`, `643468-643473`, `643486-643490`, and `643514-643519`, fresh SPT_DEV/BIGNET rendezvous endpoints continue binding on HF’s local 100.68.35.65/192.168.1.81 addresses with rotating ports. `crates/spt-net/src/net/endpoint.rs:314-404` shows that a successful bind is local endpoint construction; external reachability still depends on relay/DNS/direct paths and remote QUIC completion.\n\nAFFECTED KEYS/SUBNETS\n- HFENDULEAM/self: `14efb80c…c85f2d`, SPT_DEV + BIGNET (`identity/roster.json:2-25`, `:82-116`).\n- ENLYZEAM/current: `ecb39e40…e55657`, SPT_DEV + BIGNET (`identity/roster.json:28-66`, `:151-190`); last admitted feed 2026-07-14 05:59:54.517Z (`identity/registry/heard.meta:5`).\n- ENLYZEAM/retired: `39c018a4…eabeab1`, still represented in roster but tombstoned in both subnets (`identity/roster.json:191-276`); its dial timeout is therefore not evidence that current EN identity is wrong, only that old durable state was also targeted at generation start.\n- Additional independently failing peers: `aea099…`, `cfd9fc…`, `c8939d…`, proving the outage is not confined to the EN key or one subnet.\n\nSTRONGEST ROOT CANDIDATE, WITH CONFIDENCE BOUNDARY\nThe strongest supported *trigger class* is a common transport reachability regression/blackhole at or below the v0.31 broker endpoint (listener/firewall, n0 relay/DNS path, or QUIC connect path), not registry, subnet, or endpoint rendering. Evidence: every unrelated peer route expires at the same whole-dial 10-second boundary; both HF/EN reportedly run broker 0.31.0; local endpoint binds and schedulers continue; no typed auth/key/subnet denial appears. This is a candidate, not a proved initial trigger: the existing timeout collapses all connect stages, and the provided HF logs do not expose relay registration, packet arrival, QUIC establishment, ALPN selection, or seed-proof stage.\n\nA concrete secondary defect exists but cannot explain fleet-wide isolation: `identity/peer-addrs.json:29-38` stores map key `5ff50e75…` (SCELTOUIN per `identity/roster.json:119-150`) whose value has ENLYZEAM id `ecb39e40…`; there is no correctly keyed `ecb39e40…` entry. `crates/spt-daemon/src/pump/mod.rs:942-958` looks up the map by requested peer key first, then falls back to id-only n0 discovery. `crates/spt-daemon/src/pump/mod.rs:852-881` records `remote_id_hex -> remote_addr` on CONNECTED and drops that key after DIAL_FAILED; `crates/spt-daemon/src/seedproofx.rs:870-936` gap-fill keys entries by their own roster pubkey. A normal roster gap-fill cannot create this key/value mismatch. Prediction: an invariant checker over `peer-addrs.json` will flag exactly this record; however removing it would not restore the other failing peers and is not recommended as a live-state experiment.\n\nALTERNATIVES FALSIFIED / DOWNRANKED\n- Pump deadlock/starvation: falsified as the steady-state cause. Fresh heartbeat/status plus ongoing 15–30s failure scheduling and rotating meet binds show active loops.\n- Missing local bind: falsified at the coarse level by `net_up=true`, `BRAIN_NET_CONSUMERS_UP`, and repeated successful rendezvous binds; this does *not* falsify firewall filtering or wrong listener/path externally.\n- Single bad EN key, stale address, or subnet mismatch: falsified as global cause by simultaneous timeouts for aea099, cfd9fc, c8939d, current EN, and old EN across both subnets. The peer-address mismatch is real but only exacerbates EN resolution.\n- Seed/auth rejection as primary: downranked. No `AUTH`, `SEED`, `PROOF`, `MEMBERSHIP`, `UNTRUSTED`, or denial is emitted in the relevant failures; the generic wrapper timeout means a proof-stage hang remains possible but an explicit rejection is unsupported.\n- IPv6-only failure: falsified as sufficient cause. The log repeatedly says `NET_FAMILY_GATE: binding IPv4-only`, while advertised IPv4/Tailscale/LAN and n0 relay paths exist.\n- Clock/TOTP skew: downranked/falsified for steady state by the supplied read-only `w32tm` evidence (HF +0.003s, EN -0.135s) and successful 30-second rendezvous rotation. Normal member traffic is stable-key QUIC rather than TOTP rendezvous anyway.\n- Registry eviction as initial cause: falsified directionally. Code evicts only *after* admitted feeds stop; it explains disappearance from views, not why transport first stopped.\n- `database is locked` dispatcher errors: not a complete explanation. They affect local notification joins (`logs/daemon.stderr.log:12-36`) while unrelated broker peer dials independently hit the network-wide 10-second bound.\n\nNON-MUTATING DISCRIMINATORS / REMEDIATION-TEST SHAPE\n- Add diagnostic stage timestamps (future code/test, not live state): `address selected -> endpoint.connect start -> QUIC connected -> ALPN accepted -> seed-proof sent/received -> roster/register complete`, retaining peer id and route class (direct/relay). The first missing edge will separate relay/listener/firewall from protocol proof.\n- Add a reachability field distinct from health: last successful peer connect/feed per subnet, connected peer count, last dial failure + stage. Keep `net_up` and pump heartbeat explicitly labeled local liveness.\n- Add an invariant at peer-address load/record: map key must equal serialized `EndpointAddr.id`; reject/quarantine mismatches and log both keys.\n- Reproduce hermetically with two v0.31 Windows nodes across separate NATs and an external packet/relay observer. Matrix: id-only n0 resolution, explicit current EndpointAddr, direct Tailscale/LAN address, and relay-only path. Rotate one node’s UDP port and prove rediscovery. Cross-version 0.30.x↔0.31.0 and 0.31.0↔0.31.0 distinguishes wire incompatibility from environment.\n- Falsifiable predictions: (a) listener/firewall blackhole => outbound attempts and/or relay registration exist, but no inbound QUIC completion; packet capture/firewall log shows dropped UDP or no return traffic; (b) n0 DNS/relay failure => id-only fails while an explicit current direct EndpointAddr succeeds; (c) seed-proof regression => QUIC+ALPN succeeds and timeout occurs only after proof start; (d) protocol-version regression => failure tracks version pairing in the matrix; (e) stale-seed-only => explicit current address succeeds and other correctly seeded peers do not all fail (contradicted by present multi-peer evidence). No daemon lifecycle or state-changing operation is needed to collect these observations.",
  "files": [
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/logs/daemon.stderr.log",
      "description": "Generation startup and live-tail evidence: locally healthy broker/brain and meet binds coexist with repeated whole-dial 10-second timeouts for multiple unrelated peer keys."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/identity/registry/heard.meta",
      "description": "Last admitted inbound-feed watermarks; ENLYZEAM current key stops at 2026-07-14 05:59:54.517Z and GRAVITY-RUNNER at 05:51:26.121Z."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/daemon.json",
      "description": "Live configuration: n0 relay enabled, peer pump enabled, 30s registry pump, 300s silent-peer eviction, 15s notification pump, 60s sync pull."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/identity/roster.json",
      "description": "Durable cross-subnet membership and addresses for HFENDULEAM, ENLYZEAM current/old keys, KITSUBITO, SCELTOUIN, and GRAVITY-RUNNER; old EN key tombstoned in both subnets."
    },
    {
      "path": "C:/Users/decid/AppData/Local/spt-core/identity/peer-addrs.json",
      "description": "Dial seed cache; contains a concrete key/id integrity violation: SCELTOUIN map key 5ff50e75… holds ENLYZEAM EndpointAddr id ecb39e40…, while the current EN key has no own entry."
    },
    {
      "path": "C:/Users/decid/Documents/projects/spt-core/crates/spt-daemon/src/nethost.rs",
      "description": "`submit_dial` wraps the entire connect/prove/register plan in one 10-second timeout, explaining why current logs cannot localize the failed transport stage."
    },
    {
      "path": "C:/Users/decid/Documents/projects/spt-core/crates/spt-daemon/src/pump/mod.rs",
      "description": "Fan-out, seed-first resolution, id-only fallback, failure logging/backoff, successful-address writeback, and stale-seed deletion paths."
    },
    {
      "path": "C:/Users/decid/Documents/projects/spt-core/crates/spt-daemon/src/registryhost.rs",
      "description": "Silent-peer eviction implementation: foreign rows are removed after no admitted feed exceeds configured TTL."
    },
    {
      "path": "C:/Users/decid/Documents/projects/spt-core/crates/spt/src/cli.rs",
      "description": "Daemon status semantics: process/broker bind and pump-heartbeat liveness, plus roster/registry-derived counts, with no remote-connectivity criterion."
    },
    {
      "path": "C:/Users/decid/Documents/projects/spt-core/crates/spt-net/src/net/endpoint.rs",
      "description": "Production endpoint binds with N0 preset, IPv4 family gate, n0 relay/DNS and optional mDNS; successful bind is not proof of external route reachability."
    },
    {
      "path": "C:/Users/decid/Documents/projects/spt-core/crates/spt-daemon/src/seedproofx.rs",
      "description": "Roster gap-fill keys peer-address seeds by roster pubkey, showing the observed 5ff→ecb cache mismatch is not a valid normal gap-fill output."
    }
  ],
  "architecture": "Observed failure chain:\n\n`broker endpoint locally bound (net_up)` + `brain peer-pump heartbeat fresh`\n  -> fan-out selects every durable roster peer in SPT_DEV/BIGNET\n  -> seed cache or id-only n0 lookup supplies EndpointAddr\n  -> broker `submit_dial` enters one 10s `DialPlan::connect` envelope\n  -> no connect+proof+register completion; generic `PUMP_PEER_FAIL`\n  -> no admitted remote registry feeds\n  -> `heard.meta` watermark freezes\n  -> after daemon-configured 300s, `RegistryHost::evict_silent_peers` removes foreign endpoint rows\n  -> endpoint/subnet projections become local-only or empty\n\nMeanwhile `daemon status` remains green because it independently observes only daemon PID, broker `net_status.enabled`, pump heartbeat freshness, and durable roster/registry counts. This is a textbook separation between local control-plane liveness/fresh cached membership and data-plane transport reachability. The steady-state feedback loop is proved; the initial event that made all dials stop is not stage-localized by existing telemetry."
}