diff --git a/CHANGELOG.md b/CHANGELOG.md index eafe18d6..39bf8144 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,19 @@ breaks something, or changes the observable behavior of existing surfaces broadly; **patch** for fixes, and for additive opt-in capability — a new key, flag, or page that no existing user can encounter without opting into it. +## [Unreleased] + +### Fixed + +- Every persistent shell comes back with its owner after a daemon-only restart, + not just after a machine reboot. Shells launched during the current boot or + without a saved launch stamp are no longer left down, and recovery notices + name every healed instance for each owner. A persistent shell must be torn + down or made nonpersistent to stay down; killing its process does not opt out + of restoration. +- A successful `spt shell relink` reports `binding` while awaiting its handshake, + rather than making the successful launch read as `offline`. + ## [0.68.0] - 2026-09-08 Web serving. Files, directories, adapter documentation and the changelog are diff --git a/CONTEXT.md b/CONTEXT.md index cdb8a8e1..6698a590 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -388,11 +388,11 @@ The manifest value is the **floor**: a node/endpoint setting may **tighten** (de A shell automatically goes **offline when its owner endpoint goes offline**. Two complementary manifest options bring it (and its owner) back: -- **`persistent`** (bool): the shell is automatically brought **online whenever its owner endpoint is online**. Covers "owner already up." **Node-qualified once instances span nodes** (ruled 2026-07-24, #70 grill): when an owner comes online at a given node, only the **latest** instances qualified to that *endpoint@node* pairing come online with it — a stale instance of the same shell left on another node does not auto-wake alongside. **A node restart is covered by a boot sweep, and an owner that comes online after that sweep by an owner-online edge** (BAROMETER W2 releases#78; CONDUIT W2 releases#228): a machine death breaks no link, so the instance's record survives the reboot still saying `online` and the wake cascade — which reads the *recorded* field — skips it, stranding every `persistent` instance on the node at once. The daemon therefore (i) heals a recorded `online` its derived status contradicts, every reconcile cycle, writing only when the value actually changes, and (ii) runs a **once-per-daemon-generation boot sweep** that relaunches an instance when **all four** conditions hold: the adapter declares `persistent`; the **owner endpoint is online** (an offline owner is owed nothing); the instance is down **in fact**; and its **recorded launch predates the boot instant**. A launch parks a **birth stamp** beside the pid to make that last question answerable — **no stamp ⇒ not restored** (the sweep cannot prove the corpse predates boot, and leaving a shell down beats relaunching a binary an operator may have killed deliberately), and a platform with no boot oracle restores nothing. A shell force-killed *during* this boot is launched after it, can never satisfy the predicate, and so is never spontaneously relaunched — the deploy/quarantine ruling of KNOWN-HAZARDS 2.6 preserved by construction. - **Daemons boot before endpoints do, so the sweep is not the only trigger** (releases#228): an owner that is not yet online when the sweep runs is skipped by that very conjunct, and — the sweep being once per daemon generation, and bringup from offline emitting no rest edge — nothing revisited it, so its shells stayed down until a human relinked. The reconcile loop therefore runs the **same four conditions** for every owner that has gone **offline→online since the last pass**. The trigger is that **edge, never the level “the owner is online”**: a successful restore restamps itself out of eligibility, but a *failing* launch under a level trigger would be retried every reconcile tick for as long as the owner stayed up, where the edge bounds it to one attempt per owner-online event. Re-evaluating later in a generation can only refuse **more**, never less — every launch path restamps the birth stamp, a kill never does, and the boot instant is fixed for the machine boot — so the force-kill ruling above is preserved identically on the second trigger. -- **`wake_command`** (template): a long-running **wake-watcher** process spt-core runs *while the shell is offline* **and eligible** (the eligibility rule is below — an offline instance sitting on a same-boot corpse arms no watcher). Its sole job is to fire a wake. It runs on the **shell's node** (where the platform wake-event originates — e.g. interacting with a "shut-down" avatar). Covers "owner is down, wake it from outside." +- **`persistent`** (bool): the shell is automatically brought **online whenever its owner endpoint is online**. Covers "owner already up." **Node-qualified once instances span nodes** (ruled 2026-07-24, #70 grill): when an owner comes online at a given node, only the **latest** instances qualified to that *endpoint@node* pairing come online with it — a stale instance of the same shell left on another node does not auto-wake alongside. A **once-per-daemon-generation boot sweep** restores every down persistent instance whose owner is online, after a machine reboot **or a daemon-only restart**. Birth-safe liveness protects both an online binary and a live launch awaiting bind. Machine boot time, recorded launch age, and missing launch stamps do **not** gate restoration (releases#287 operator ruling). The daemon also heals stale `online` records to `offline` each reconcile cycle, only on an actual change. + **Daemons boot before endpoints do, so the sweep is not the only trigger:** the reconcile loop runs that **same per-owner restore decision** on each owner **offline→online edge**. The edge is seeded before the sweep; a live launch awaiting bind prevents duplicate launch if both triggers reach it. Failed launches are attempted once per owner-online event, not every tick while the owner stays online. A force-kill does not revoke persistence: use `teardown` or a nonpersistent manifest to keep the instance down. +- **`wake_command`** (template): a long-running **wake-watcher** process spt-core runs while the shell is offline and eligible. Its sole job is to fire a wake. It runs on the **shell's node**, where the platform wake-event originates (e.g. interacting with a shut-down avatar). Covers "owner is down, wake it from outside." -**Online/offline are mutually-exclusive processes:** online ⇒ the shell binary runs (no watcher); offline ⇒ the wake-watcher runs (no shell binary). spt-core flips between them. **Offline does not by itself arm the watcher — a third outcome exists** (BAROMETER W2, releases#78): an instance that is offline over a **corpse launched during this boot** (a force-kill, a crash) is deliberately held out of watcher eligibility, so *neither* process runs until recovery is demanded — `relink`, or a `shell cmd` that wakes. Eligibility is recorded-`offline` **AND** (no corpse, **or** the corpse's launch predates the boot instant); an instance whose corpse cannot be dated — no birth stamp, or no boot oracle on the platform — is treated as ineligible rather than guessed at. Only a cleanly-closed instance and a **restart casualty** arm a watcher. The reason is the deploy case: on Windows an operator kills a shell precisely to free its exe for overwrite, and a watcher armed there would re-lock the file under them. +**Online/offline are mutually-exclusive processes:** a live shell binary (including a launch awaiting bind) excludes its wake-watcher. A recorded-offline instance is watcher-eligible when it is persistent, carries no corpse, or its corpse's launch predates machine boot. **Only nonpersistent instances retain the same-boot force-kill freeze:** their same-boot or undatable corpses arm no watcher, preserving the deploy/overwrite case. A persistent corpse bypasses that freeze even without a launch stamp or boot oracle (releases#287). Missing birth evidence retains the existing fail-toward-alive process probe; it is not itself a restore refusal. **Exit-opcode supervision:** the wake-watcher exiting with the **wake opcode** → spt-core runs the wake resolution (below) and brings the shell online. Any *other* exit (crash) → respawn the watcher with **exponential backoff + eventual give-up** (until next shell activity) — crash-bug safety. One watcher per offline shell instance. diff --git a/docs-site/src/shells/overview.md b/docs-site/src/shells/overview.md index 0e372ced..16cbaeda 100644 --- a/docs-site/src/shells/overview.md +++ b/docs-site/src/shells/overview.md @@ -93,30 +93,22 @@ Lifecycle extras: `persistent` shells auto-online with their owner; with `can_shutdown = true` may suspend its own owner (`api owner-shutdown`) — fail-closed otherwise. -Two lifecycle facts worth knowing before you rely on either: - -- **A node restart brings `persistent` instances back by a boot sweep, and an - owner that comes online after that sweep by an owner-online edge.** A machine - death breaks no link, so the instance's record survives the reboot claiming it - is still up; the daemon corrects that record and then, once per daemon - generation, relaunches an instance whose adapter is `persistent`, whose **owner - endpoint is online**, that is down in fact, and whose recorded launch - **predates the boot instant**. An instance last launched before this behavior - shipped carries no launch stamp, so the sweep cannot date its corpse and leaves - it down — it is covered from its next launch onward. Daemons start before - endpoints do, so those same four conditions are re-run for **any owner that - goes offline→online** between reconcile passes: without that, a shell whose - owner was still down when the sweep ran would stay down until someone relinked - it by hand. It is that transition — not the standing state "the owner is - online" — that triggers the retry, which is what keeps a launch that keeps - failing to one attempt per owner-online event rather than one every reconcile - tick. -- **Offline alone does not arm the watcher.** A shell force-killed (or crashed) - *during the current boot* is offline with no watcher running, and comes back - only on demand — `spt shell relink `, or a `shell cmd` that wakes it. This - is deliberate: killing a shell to overwrite its exe is a routine deploy step, - and a watcher armed underneath would re-lock the file. A cleanly-closed - instance and a restart casualty both arm a watcher normally. +Lifecycle facts worth knowing before you rely on either: + +- **Persistent instances return after a daemon-only restart as well as a machine + reboot.** The boot sweep and each owner offline→online edge restore every + persistent instance that is down in fact while its owner is online. Launch + age and missing launch stamps do not exclude it. The birth-safe process probe + still protects a live binary, including a launch awaiting bind. Failed + launches are retried on an owner-online edge, not every reconcile tick. +- **Only nonpersistent instances retain the force-kill freeze.** Their + same-boot or undatable corpses arm no watcher; cleanly closed instances and + dated pre-boot casualties remain watcher-eligible. Persistent instances + bypass this freeze. To keep a persistent shell down, use `teardown` or a + nonpersistent manifest rather than killing its process. +- The daemon's `SHELL_RECORD_HEALED:/` diagnostics enumerate + every instance whose stale online record was corrected, not a representative + shell per owner. Already-correct records produce no repeated notice. - **`relink` refuses an instance that is still running; `--force` is the @@ -129,6 +121,11 @@ Two lifecycle facts worth knowing before you rely on either: canonical id, the perch and its persisted state, and the alias all survive; teardown + spawn is what churns the id and frees the mint slot. + + A successful local relink reports `status=binding (online at bind)`: the + binary has launched and the bind handshake is still the online transition. + It does not describe that successful launch as `offline`. + Two things `--force` deliberately will **not** do: - **It refuses on an `ephemeral` instance.** An ephemeral shell's close *is* diff --git a/docs/KNOWN-HAZARDS.md b/docs/KNOWN-HAZARDS.md index 8ab726b7..f6898b86 100644 --- a/docs/KNOWN-HAZARDS.md +++ b/docs/KNOWN-HAZARDS.md @@ -75,20 +75,23 @@ Hard-won edge cases harvested from the sister project (`claude_skill_owl`, ~80 c ### 2.6 A shell instance's `online` is a recorded field with no death edge - **Failure:** a shell perch's `status` has exactly two writers — the bind handshake (→`online`) and `close_shell` (→`offline`) — and `close_shell` runs on the **link-break** path alone. A binary that dies **abruptly** (force-kill, crash, OOM) breaks no link, so nothing ever writes the flip: the record says `online` forever while the pid it names is gone. Every consumer of the recorded field then inherits the lie — `shell list` reports online indefinitely; `relink` (the one command that recovers the instance) refuses `SHELL_ALREADY_ONLINE`, gated on the very state that is wrong; and `shell cmd`'s wake-if-offline arm reads "not offline", skips the wake, and spools the command to a corpse — accepted happily, drained by nobody, **indistinguishable from a busy shell**. The only exit was `teardown` + `spawn`, which destroys the perch (losing the instance's persisted state) and frees the mint slot, so the instance returns under a **different canonical id** and every reference to the old name — docs, other agents' notes, tag addressing — silently points at nothing. The rename is the *visible* cost; the expensive one is **silent**: a consumer's perch-persisted cursor re-baselines on the fresh perch, so work arriving between the death and the re-bind is never scanned — not failed-and-retried, just never seen (alchemy's tag scanner, 2026-07-25). Recovery must therefore preserve the perch, not merely restore a running binary. -- **Invariant:** ONLINE-ness is **derived**, never the recorded field alone: `status == online` **AND** the recorded `shell.pid` is not *provably* dead. Provably dead is narrow — pid parked, non-zero, `!is_process_alive`; pid absent, unparseable, or `0` (a broker-hosted spawn whose backend exposed no pid) reads **alive**, the same fail-toward-alive parity 2.5 holds, so a backend the resolver cannot see is never declared dead. **The recycled-pid caveat is narrowed since BAROMETER W2 and is now conditional on the stamp:** a launch parks a **birth stamp** beside the pid (`shell.launch.json`), and the probe runs the PAIR test — same pid, *different* start time ⇒ **gone**, so a pid the OS handed to an unrelated process no longer masks a dead instance. That case matters most across a restart, where the pid namespace resets and a corpse pid is routinely reused; without the pair test the corpse reads alive, the derived status reads `online`, and `relink` refuses `SHELL_ALREADY_ONLINE` on the one instance needing recovery. Where there is **no stamp to compare** (an instance last launched by a pre-W2 build, or a backend exposing no start time) the original stance is unchanged and unchanged in *direction*: fail toward alive, heal *missed* rather than *mis-fired*. Derivation NEVER promotes: a recorded-`offline` instance stays offline whatever its pid file says. Site class is part of the invariant: **derive** at the gates and renders (relink, the cmd wake arm, drive's drop-if-offline, `shelldisc::discover` = the single source of both `shell list` surfaces, the activity fan-out); keep the **recorded** field at the two writers and at the owner-suspend cascade, whose `close_shell` **is** the cleanup that would be skipped; and keep it at the wake reconciler's eligibility read, which since BAROMETER W2 additionally asks the **corpse-boot discriminant** (releases#78, doyle's amendment on comment 5156810267): the read stays on the recorded field, but "no spontaneous relaunch" is now a stated predicate rather than a side effect of the record being wrong — see the bullet below. -- **No spontaneous relaunch of a SAME-BOOT death (operator-ratified, flynn 2026-07-25; re-based BAROMETER W2):** the reconciler does NOT adopt an instance whose binary died during this boot. Every reason an operator stops a shell is a reason not to want it back a tick later: mid-deploy (a shared install dir makes "kill the process" routine on Windows, since a running exe cannot be overwritten) an auto-relaunch would run the **old** binary out of the file being replaced — worse than a failed install — and it turns quarantining a misbehaving shell into a restart loop. For those, recovery stays demand-driven: `relink`, or a `shell cmd` that wakes. `relink` probes **locally** rather than trusting a daemon sweep, so recovery holds with the daemon down. -- **What holds that open is no longer the stale record (BAROMETER W2):** until W2 the guarantee rested on an accident: a force-killed instance was held out of the eligible set because its record still said `online`, and the eligibility read took that field raw. 2.7 heals that record to the truth, which removes the accidental protection — so the guard had to move to a rule that *states* it. Eligibility is now recorded-`offline` **AND** (no corpse, **or** the corpse's recorded launch **predates the boot instant**). A steady-state force-kill leaves a corpse launched *after* boot, which the predicate can never accept, so the ruling is preserved **by construction** rather than by a second rule. A restart casualty is the one case the predicate does accept, and it is exactly the scope 2.7 was greenlit for. Where the discriminant **cannot be evaluated** — no launch stamp (an instance last launched by a pre-W2 build), or no boot oracle on the platform — a corpse-bearing instance is **not** eligible: leaving a shell down beats arming a relaunch of a binary an operator may have killed on purpose. +- **Invariant:** ONLINE-ness is **derived**, never the recorded field alone: `status == online` AND the recorded process identity is not *provably* gone. Missing/unparseable/zero PID or unavailable birth evidence fails toward alive; a positively mismatching native birth identifies a recycled PID as gone. Derivation never promotes a recorded-offline instance. Derive at relink, drive/cmd-wake, discovery and activity renders; preserve the recorded field at writers, owner-suspend cleanup and watcher eligibility. The latter adds the nonpersistent corpse-boot rule below rather than relying on stale status to suppress relaunch. + +- **One process generation per read (releases#287):** modern `shell.launch.json` atomically stores `pid`, native `pid_started_at`, and wall-clock `launched_ms`. A modern `pid: 0` means explicitly pidless/cleared and never falls back to the numeric `shell.pid` mirror. Legacy records without `pid` retain their old numeric-PID plus optional-birth read. Publish the complete JSON identity before updating the compatibility mirror; failed publication preserves the previous identity. Never delete the previous birth before a fallible spawn: failure would leave a recycled PID looking unambiguously alive and could authorize its termination. Both launch arbitration and kill authorization consume the same coherent PID/birth snapshot; clearing retains launch age while retiring the process identity. +- **No spontaneous relaunch of a NONPERSISTENT same-boot death:** the releases#78 force-kill freeze still protects nonpersistent instances, so killing one to free its exe does not arm a watcher underneath an overwrite. Recovery remains demand-driven. **Persistent instances are exempt** (operator ruling, releases#287): a kill is not a request to keep a declared-persistent shell down; use `teardown` or a nonpersistent manifest instead. +- **Watcher eligibility:** recorded `offline`, no live binary awaiting bind, and either `persistent`, no corpse, or a corpse whose launch predates the machine boot instant. A nonpersistent corpse without a launch stamp or boot oracle remains frozen; persistent instances require neither to become eligible. Birth-safe probes prevent a recycled pid from masquerading as a live shell. - **spt-core mapping:** `spt_store::shellinfo::{shell_pid_provably_dead, effective_status, is_shell_online}` (the resolver — the shell-side twin of 2.5's `liveness.rs`, which gave *agent* perches exactly this and which shells never got); consumers in `linkhost::{relink_shell, prepare_drive, run_action}`, `shelldisc::discover`, `activity::observe_links`; the class-(c) eligibility read is `spt_daemon::shellwake::watcher_eligible` (+ `launch_predates_boot`, `BOOT_RESTORE_SLACK_MS`), and the birth stamp it rests on is `shellinfo::{ShellLaunch, record_shell_launch, read_shell_launch}`. - **Source:** spt-core, flynn's spt-alchemy field report (2026-07-25) — a deterministic recipe, not a race. Class-(c) re-based BAROMETER W2 (releases#78, doyle's ruling on comment 5156810267). -### 2.7 A node restart permanently strands every `persistent` shell — and the existing cascade test cannot see that it does `[REQ-HAZARD-RESTART-STRANDS-PERSISTENT-SHELLS]` -- **Failure:** the contract's own sentence says a `persistent` shell is **online whenever its owner endpoint is online**, and a node restart is the one case where that promise was never kept. A machine death breaks no link, so `close_shell` never runs and 2.6's stale record survives the reboot; when the owner comes back online, the wake cascade reads the **recorded** field, sees `online`, and skips the relaunch of a binary that has not existed since the previous boot. Nothing else corrects it: the instance is down forever, with no error anywhere, until a human happens to run `relink`. Every `persistent` instance on the node is stranded by one restart, together. +### 2.7 A daemon or machine restart strands `persistent` shells — and a graceful cascade fixture suppresses the failing arm `[REQ-HAZARD-RESTART-STRANDS-PERSISTENT-SHELLS]` +- **Failure:** a restart can leave an `online` record over a corpse without a link-break or suspend edge. The original boot sweep fixed only machine reboots: it rejected corpses launched after machine boot, so a daemon-only restart still stranded every persistent shell. The measured releases#287 specimen launched at machine boot +46s and stayed down for 27 hours after a daemon restart despite its owner being online. - **The owner-facing surfaces HIDE the fault**, which is why it survived unreported for days: every display path (`shell list`, its `--json` twin, the shell-context render) routes through the ONE discovery seam, which **derives** status (2.6) and so correctly reports the binary as offline — while the only consumer whose decision matters, the wake cascade, reads the RECORDED field that no owner-facing surface shows. Absence of a visible symptom is therefore **not** evidence of absence of this fault, and a clean-looking context render must never be taken as proof the class did not occur. The correct falsifier is the **on-disk record**. - **The existing cascade test cannot fail on it — its SETUP SUPPRESSES THE FAILING ARM.** `rest_edges_cascade_shells_with_divergence` establishes its fixture by driving the suspend path, which itself writes the `offline` the code under test is supposed to encounter — so the restart shape (a record still saying `online` over a corpse, reached with **no** suspend edge) is unreachable from that fixture BY CONSTRUCTION. The test is not weak; it gives a true answer to the wrong question, so a reader asking "is the cascade covered?" is told yes. This is the third suppressed-arm instance found in one week, and the pattern is the same each time: **a fixture that establishes the precondition the code under test is supposed to establish.** -- **Invariant:** a node restart must not strand a `persistent` instance. Two legs, and each is worthless alone. **(a) The record stops lying:** the daemon-side reconciler heals a recorded `online` that 2.6's derivation contradicts, writing `offline` — every cycle, **guarded on an actual change** (a heal that rewrote each tick would be a stream of identical writes and would destroy the record's mtime as a signal). **(b) The restart shape reaches a restoration path:** a once-per-daemon-generation boot sweep — and, for an owner that comes online after that sweep has run, an **owner offline→online edge** in the reconcile loop running the identical body (releases#228) — relaunches an instance when **all four** hold — the adapter section declares `persistent`; the **owner endpoint is online** (an offline owner is owed nothing — that is the line between healing a record and resurrecting a shell nobody is there to drive); the instance is down **in fact** (the derived read, made trustworthy across a pid reuse by 2.6's birth stamp); and its recorded launch **predates the boot instant**. No stamp ⇒ **not restored** (the sweep cannot prove it predates boot); no boot oracle on the platform ⇒ nothing restored at all. The boot instant is derived (Linux `btime`; Windows now − `GetTickCount64`), so a named slack constant absorbs derivation jitter and keeps a process launched moments after boot from reading as a corpse from the previous one. **The once-ness of the sweep is scheduling, not safety** (ruled releases#228): the predates-boot discriminant is what forbids relaunching a force-killed shell, and it is monotone within a machine boot — every launch restamps, a kill never does — so a second trigger evaluating the same conjuncts can only refuse more. The second trigger is an **edge**, not the level “owner is online”, which is what bounds a failing launch to one attempt per owner-online event instead of one per reconcile tick. -- **Regression shape (the row the suppressed fixture cannot express):** construct the restart shape **without a suspend edge** — an instance whose record says `online` over a corpse that predates boot, with an online owner — and assert it is restored. A fixture that suspends first re-creates the suppressed arm and proves nothing about this class. -- **spt-core mapping:** `spt_daemon::shellwake::{heal_stale_online_records, restore_persistent_shells_at_boot, restore_persistent_shells_on_owner_online, OwnerOnlineEdge, launch_predates_boot, BOOT_RESTORE_SLACK_MS}`; `spt_store::proc::boot_instant_ms`; `spt_store::shellinfo::{ShellLaunch, record_shell_launch, read_shell_launch}` (the birth stamp, parked at the two production pid-write sites the launch-site census identified). -- **Source:** spt-core, BAROMETER W2 (releases#78) — field specimen `liam/alchemy-0`, a genuinely stranded record: `online` over a dead pid, no launch stamp, days old. +- **Invariant:** a daemon-only or machine restart must not strand a persistent instance. **(a) Heal:** every reconcile cycle writes `offline` only where the derived birth-safe read contradicts recorded `online`; already-correct records are untouched. **(b) Restore:** the once-per-daemon-generation boot sweep and every owner offline→online edge run the same per-owner body. Relaunch every instance whose adapter declares `persistent`, whose owner is online, and whose binary is down in fact. There is **no machine-boot, launch-age, or stamp-presence eligibility gate**. A live binary awaiting bind is not down and must not be duplicated; missing birth evidence keeps the existing fail-toward-alive liveness behavior. The owner-online trigger remains an edge, bounding failed launches to one attempt per event. +- **Regression shape:** construct recorded `online` over a corpse **without a suspend edge**, including a launch **after machine boot**, then bring its owner online after a spent boot sweep. Every stranded persistent sibling must be restored; offline owners and nonpersistent instances are not restored. Separately defend live/binding exclusion and the nonpersistent watcher freeze. Suspending in setup suppresses the failing arm and proves nothing about this class. +- **Per-owner enumeration:** the existing `SHELL_RECORD_HEALED:/` daemon diagnostic names **every** successfully healed instance, not one representative. Healthy and already-healed records produce no event. The restore diagnostics likewise identify each attempted instance and distinguish boot from owner-online triggers. +- **spt-core mapping:** `spt_daemon::shellwake::{heal_stale_online_records, restore_persistent_shells_at_boot, restore_persistent_shells_on_owner_online, OwnerOnlineEdge}`; `shellhost::live_launch_winner`; `spt_store::shellinfo::{ShellLaunch, shell_pid_provably_dead}`. `launch_predates_boot` and `BOOT_RESTORE_SLACK_MS` now govern only nonpersistent watcher eligibility. +- **Source:** BAROMETER W2 (releases#78); releases#287 operator ruling (2026-09-08), superseding the persistent force-kill freeze and boot-instant conjunct. --- @@ -1015,9 +1018,9 @@ The kill-path rule above generalizes: `daemon.pid` is not authority for *"which | 2.1/5.1 | Stable PID/broker-handle over ephemeral PID | liveness detection | | 2.3 | Handoff argv/IPC version-tolerant (newer brain ↔ older broker) | broker↔brain IPC, self-update | | 2.4 | gen_start = now() on cold-start + handoff | per-instance generation | -| 2.6 | A shell's ONLINE-ness is DERIVED (recorded status AND a not-provably-dead `shell.pid`) — an abruptly-killed binary breaks no link, so `close_shell`'s offline flip never runs and the record lies forever; derive at the gates/renders, keep the recorded field at the writers, the suspend-cascade close, and the wake reconciler — whose "no spontaneous relaunch" (deploys and quarantine own that decision) is since W2 the stated corpse-predates-boot predicate, not the stale record holding it open, and whose recycled-pid blindness is closed by the launch birth stamp wherever one is parked. Not only a crash edge: the GRACEFUL daemon-stop path kills bound shells at stop-begin with `close_shell` unrun (field 2026-07-25 — shells died abruptly, `info.json` still `online`, while the broker drained on 2m10s), so every daemon restart manufactures these stale records routinely; the derivation heals them at the gates | `spt_store::shellinfo::{shell_pid_provably_dead,effective_status,is_shell_online}`, `linkhost` relink/drive/cmd-wake, `shelldisc::discover`, `activity::observe_links` | +| 2.6 | A shell's ONLINE-ness is DERIVED from recorded status and a not-provably-gone PID/birth pair. Abrupt death and daemon-stop leave stale `online` records; derive truth at gates/renders and heal each changed record once. The same-boot no-spontaneous-relaunch rule applies only to nonpersistent shells. Persistent instances follow 2.7 regardless of launch age. Modern PID and birth are published and read as one atomic identity; failed replacement preserves the old evidence, and a cleared identity cannot resurrect its numeric mirror | `spt_store::shellinfo::{read_shell_process,record_shell_launch,clear_shell_pid,shell_pid_provably_dead,effective_status,is_shell_online}`, `linkhost`, `shelldisc`, `activity` | -| 2.7 | A node restart must not permanently strand every `persistent` shell: (a) the daemon heals a recorded `online` the derivation contradicts, write-guarded on an actual change, and (b) a once-per-generation boot sweep PLUS an owner offline->online edge in the reconcile loop (releases#228 — the sweep alone strands the shells of any owner not yet online when it ran) relaunch an instance only when ALL of — adapter `persistent`, owner endpoint online, down in fact, recorded launch predates the boot instant (slack-absorbed) — hold; no stamp or no boot oracle ⇒ not restored. The owner-facing surfaces derive and look clean while the cascade reads the recorded field, so a clean render is NOT evidence the class did not occur — the falsifier is the on-disk record. The pre-existing cascade test's setup suppresses the failing arm (it suspends first), so the restart shape must be asserted on a fixture that never calls the suspend path | `shellwake::{heal_stale_online_records,restore_persistent_shells_at_boot,restore_persistent_shells_on_owner_online,OwnerOnlineEdge,launch_predates_boot}`, `proc::boot_instant_ms`, `shellinfo::{record_shell_launch,read_shell_launch}` | +| 2.7 | A once-per-generation boot sweep and each owner offline->online edge restore every instance whose adapter is `persistent`, owner is online, and binary is down in fact. Machine boot time, launch age, and absent launch stamps do not block persistent restoration; already-online and live binding launches are excluded. Heal every stale instance per owner with no repeat writes. Prove the restart shape without first suspending the fixture; a clean derived display alone is not evidence the record was healed | `shellwake::{heal_stale_online_records,restore_persistent_shells_at_boot,restore_persistent_shells_on_owner_online,OwnerOnlineEdge}`, `shellhost::live_launch_winner` | | 3.1 | Ephemeral perch cleanup on all exit paths | `ring` (RAII guard) | | 3.4 | A ring never adopts (so never deletes) a perch it did not create — probe the DIR, not the ready marker; record/spool/unreadable = occupied, empty = refused too; deliver + loud `RING_PERCH_EXISTS`/`RING_STALE_DIR` instead of block-waiting | `spt_msg::ring` probe + `create_dir` leaf | | 3.5 | A perch GC classifies on record PRESENCE only and never asks a liveness resolver — `is_perch_alive` is INVERTED here (recordless residue reads ALIVE, an offline endpoint reads DEAD: 24 vs 6 measured on HFENDULEAM 2026-08-04), so `!is_perch_alive` authorizes exactly the inverse set; the registry is never asked either (offline endpoints are legitimately absent from it) | `spt_store::perchgc::sweep`, `spt endpoint gc` | diff --git a/docs/MANIFEST.md b/docs/MANIFEST.md index b468481b..781d270c 100644 --- a/docs/MANIFEST.md +++ b/docs/MANIFEST.md @@ -788,7 +788,13 @@ types = ["image", "sound", "event"] # "event" = arbitrary descriptive payloads The capability + sensory vocabularies live **here**, not on the perch — spt-core resolves them by `adapter_name`. Command-delivery (`command_receipt`) reuses the agent inject-input modes. The shell binary binds via `api bind` (type=Shell, owner from the link) and pushes sensory via `api emit --type `. -**Sleep/wake:** when offline **and eligible** (+ `wake_command` set), spt-core runs the wake-watcher (mutually exclusive with the shell binary); `exit(wake-opcode)` → wake resolution; crash-exit → respawn with exponential backoff + give-up (until the next relink/launch). **The wake opcode is exit code 86.** The wake-watcher template fills `{id}`, `{adapter_name}`, `{link_token}`, `{adapter_dir}` (NOT `{perch_dir}` — that key is spawn-only; the waker fill site carries no perch context), and its `{link_token}` is minted *unparked* — an offline link has no live credential (the close retired it); a waker wakes by exit code, never by driving the link. **Eligibility (BAROMETER W2, releases#78):** a watcher is armed for an instance recorded `offline` that either carries no corpse (cleanly closed, or never launched) or whose corpse's recorded launch **predates the current boot** — a restart casualty. An instance whose binary died **during this boot** (force-kill, crash) arms no watcher and returns only on demand (`relink`, or a waking `shell cmd`), because killing a shell to overwrite its exe is a routine deploy step and a watcher armed underneath would re-lock the file; an instance whose corpse cannot be dated (no launch stamp, or no boot oracle on the platform) is treated the same way rather than guessed at. `persistent` auto-onlines the shell with its owner, and across a **node restart** — where nothing breaks the link, so the instance's record survives the reboot still claiming it is up — the daemon heals that record and a **once-per-daemon-generation boot sweep** relaunches the instance if its adapter is `persistent`, its owner endpoint is online, it is down in fact, and its recorded launch predates the boot instant. Because daemons boot before endpoints do, the same four conditions are re-run for any owner that goes **offline→online since the last reconcile pass** (releases#228) — without that edge, an owner that was still down when the sweep ran keeps its shells down until someone relinks by hand. It is an edge and not the level “the owner is online” so that a failing launch is attempted once per owner-online event rather than once per tick. `can_shutdown` authorizes `api owner-shutdown`. Cross-node *fresh-spawn* to wake an owner requires the owner's `shell_wake_spawn_anywhere` settings flag (pre-consent). +**Sleep/wake:** when offline **and eligible** (+ `wake_command` set), spt-core runs the wake-watcher, mutually exclusive with a live shell binary including one awaiting bind. Exit code **86** is the wake opcode; other exits respawn with exponential backoff and eventual give-up until the next shell launch. Templates fill `{id}`, `{adapter_name}`, `{link_token}`, `{adapter_dir}` (not `{perch_dir}`, which is spawn-only). The watcher's link token is minted *unparked*: it wakes by exit code, never by driving an offline link. + +**Persistent restoration:** on the once-per-daemon-generation boot sweep and each owner offline→online edge, spt-core restores **every** down persistent instance whose owner is online. This covers machine reboots and daemon-only restarts. The decision uses birth-safe liveness, not machine boot time or recorded launch age; a missing launch stamp does not disqualify a down instance. A live launch awaiting bind is not duplicated. The edge bounds failed launches to one attempt per owner-online event rather than every reconcile tick. Stale online records are healed only on an actual change. + +**Watcher eligibility:** recorded-offline instances with no live binary are eligible if persistent, cleanly closed/never launched (no corpse), or a dated pre-boot corpse. The same-boot force-kill freeze applies **only to nonpersistent** instances, whose undatable corpses also remain frozen. Persistent instances bypass that freeze (releases#287 operator ruling); keeping one down requires `teardown` or a nonpersistent manifest, not a kill. + +`can_shutdown` authorizes `api owner-shutdown`. Cross-node *fresh-spawn* to wake an owner requires the owner's `shell_wake_spawn_anywhere` settings flag (pre-consent). **Instantiation governance:** `require_approval` gates `shell spawn` (reusing the consent grant store — `remembered` persists a grant via allow-always, `always` prompts every spawn); `max_instances_per_owner` + `over_cap` cap how many instances one owner may hold (online + offline both count toward the cap). An instance's **alias** is **not** a manifest field — it is per-instance runtime state on the shell perch's `info.json` (set at `shell spawn --alias` / `shell rename`), since the manifest describes the *adapter*, not individual instances. diff --git a/docs/STORAGE.md b/docs/STORAGE.md index 17297f14..5f80d3cd 100644 --- a/docs/STORAGE.md +++ b/docs/STORAGE.md @@ -33,6 +33,8 @@ $SPT_HOME/ nested// Worker / Psyche perches (own info.json, spool) shells/-/ Shell perches owned by this endpoint (GameRobot-0, GameRobot-1…) info.json type=Shell, owner, adapter_name, status (online|offline), alias? + shell.launch.json atomic PID + native birth + launch wall-clock; pid 0 clears custody + shell.pid numeric compatibility mirror for older core readers spool.db command + durable text/file (sensory is REST-only, unspooled) history/ node-local logs (mirrors perch structure) / @@ -48,6 +50,14 @@ $SPT_HOME/ serve/registry authoritative serving entries + persistent name ownership ``` + +Shell launch identity is published as one `shell.launch.json` record before the +best-effort numeric mirror. Readers prefer its `pid` and `pid_started_at` together; +old records without `pid` retain the legacy two-file read. Clearing publishes +`pid: 0` and no birth while preserving `launched_ms`, so a stale numeric mirror +cannot resurrect a closed process. Failed replacement leaves the previous +identity intact rather than stripping its protection against recycled PIDs. + ### The contact ledger — `identity/trust/contacts.json` (DOORBELL W3) diff --git a/traceable-reqs.toml b/traceable-reqs.toml index 21917ea9..a3039948 100644 --- a/traceable-reqs.toml +++ b/traceable-reqs.toml @@ -3183,6 +3183,12 @@ required_stages = ["doc", "impl", "unit", "int"] # W1 ACTIVATED at mint (doyle # satisfying. --- [[requirements]] id = "REQ-HAZARD-SHELL-STALE-ONLINE" +# AMENDED releases#287, operator ruling 2026-09-08: the historical title below's +# same-boot/undatable-corpse freeze and no-spontaneous-relaunch clauses now apply +# ONLY to NONPERSISTENT instances. Persistent restoration and watcher eligibility +# follow REQ-SHELL-PERSISTENT-BOOT-RESTORE / REQ-SHELL-OWNER-ONLINE-RESTORE, without +# launch-age or stamp-presence gates. Birth-safe live/binding exclusion remains. +# The liveness, identity/state preservation and durable-channel clauses stand. title = "A shell instance's ONLINE-ness is DERIVED (recorded status AND its recorded `shell.pid` not provably dead), never the recorded `status` field alone — an abruptly-dead binary (force-kill, crash, OOM: no link-break, so `close_shell`'s offline flip never runs) must not read online forever. The shell-side twin of REQ-HAZARD-DAEMON-HOSTED-LIVENESS, which gave AGENT perches exactly this resolver and which shells never got. PROVABLY DEAD is the narrow discriminant: `shell.pid` present AND parses non-zero AND `!is_process_alive` — pid absent, unparseable, or 0 (a broker-hosted spawn whose backend exposed no pid records 0) reads ALIVE, the same interim-parity/fail-toward-alive stance `liveness.rs` already holds, so a pid-less backend is NEVER falsely declared dead. Recycled-pid caveat, accepted at mint: a reused pid reads alive, so the heal is missed, never mis-fired — the failure direction is 'stays stale', never 'kills a live instance'. SITE CLASSIFICATION IS PART OF THE REQUIREMENT (authoritative cfg(test)-excluded census at mint = 8 status reads, 3 classes — do NOT blanket-swap the predicate): (a) DERIVED — relink's already-online refusal (the gate that made recovery impossible), the `shell cmd` wake-if-offline arm (which silently spooled to a corpse), the drive drop-if-offline branch, `shelldisc::discover` (the single source of BOTH `shell list` renders, text + --json), and the activity fan-out's online filter; (b) RAW status, deliberately — `bind_shell_by_token`/`close_shell` (the WRITERS) and `cascade_owner_edge`'s suspend-close arm, where routing through the resolver would SKIP the close that is itself the cleanup, removing a heal path; (c) RAW recorded status AND the corpse-boot discriminant — the `shellwake::reconcile_once` watcher-eligibility read. AMENDED BAROMETER W2 (doyle, releases#78 comment 5156810267): this class's original text — 'RAW status, pending an operator ruling', with no spontaneous relaunch of anything — IS STRUCK, because it rested on a fact leg (a) deliberately ends. What actually held a force-killed instance out of the eligible set was the STALE `online` record itself; healing that record to the truth removes the accidental protection, so the guard had to move to a rule that STATES it. Eligibility is now recorded `offline` AND (no corpse, or the corpse's launch PREDATES THE BOOT INSTANT), reusing leg (b)'s `launch_predates_boot` and its slack constant verbatim. NO SPONTANEOUS RELAUNCH OF SAME-BOOT DEATHS: flynn's mid-deploy forcing case (an operator kills the process precisely to free the exe for overwrite — shared install dir ⇒ 'kill the process' is a routine deploy step) is preserved BY CONSTRUCTION, since a force-kill during steady state leaves a corpse launched AFTER boot, which the predicate can never accept; for those, recovery stays demand-driven (an unblocked `relink`, or a `shell cmd` that wakes). Boot-restoration of PRE-boot casualties is enabled, per #78 leg (b)'s operator-greenlit scope and this ruling (#78 thread). CORPSE IS DEFINED ON THE PARKED STAMPS, never on a bare pid re-probe: a RECYCLED pid — alive, but its native start stamp mismatching the one leg (a) parked — is corpse-equivalent because our process is dead, and the parked ABSOLUTE launch stamp is what decides pre/post-boot. Without that, the heal-then-recycle race reads 'alive ⇒ no corpse' on the next tick and walks back into the forbidden shape through a side door. Deriving is still WRONG at this site and this is NOT a swap to class (a): the census stays 8 reads / 3 classes — the site reclassifies, it does not multiply, the do-NOT-blanket-swap sentence above stands, and `shell list` still tells the truth by derivation (class (a), untouched). LOCALITY SYMMETRY (field-caught by flynn's leg-2 run, 2026-07-25): 'a shell cmd that wakes' holds for BOTH the local CLI cmd and the cross-node serve — at mint the wake arm lived in the SHELL_LINK_CMD serve handler only (module docs scoped wake-if-offline to remote cmd), so a LOCAL cmd against a corpse spooled silently and nothing woke, the exact accepted-happily/drained-by-nobody shape this req exists to kill; the arm is now the shared `linkhost::wake_if_offline_persistent`, called by both, and the derived status READ inside it is the same single census site as before (the census of 8 reads/3 classes is unchanged — the read moved, it did not multiply). ROTATION CARRIES THE DURABLE CHANNEL (flynn's frame-loss field catch, 2026-07-25, both trials): a spooled command frame is MAC-stamped under the link token current at SPOOL time, and the drain is a raw destructive passthrough — the shell verifies against its CURRENT key — so a relink's token mint orphaned every pending frame: drained to the woken binary, failed verify, correctly discarded BY THE ADAPTER, lost permanently; the wake-triggering command itself was the frame the wake lost (#23 armed the wake and never answered; #24/#19 spooled after rotation and drained in order). The fix is three-layered (todlando's A1/A2 race+crash hazards addressed at mint): (1) the token mint re-stamps all pending rows old-key→new-key (idempotent, crash-partial converges — an old-stamped remainder is converted by the next rotation or drain); (2) the rotated-out token is STASHED (`link.token.retired`, never a live credential — bind resolves only the parked file) so (3) the drain paths give any race straggler a second-chance re-stamp at delivery (`restamp_for_drain`), while frames verifying under neither key pass through untouched for the adapter to refuse exactly as before. Rows are selected by MAC verification against the threaded new key, never by token-snapshot equality (the D-2 class). MOCK CONFORMANCE IS A PROPERTY OF THE RIG, not a patch (todlando A4): mock-shell MUST verify inbound frame MACs exactly as the public contract demands of a real adapter and drop failures loudly — a mock that accepts what the field rejects is a broken rig, and that exact divergence (credulous mock) is how the frame loss passed the e2e while failing in the field. The adapter-side discard behavior is CORRECT and must never be softened to paper over the spool side. Relink additionally probes LOCALLY rather than trusting a daemon sweep, so recovery holds with the daemon down. RECOVERY PRESERVES CONSUMER STATE (the property consumers actually depend on, flynn 2026-07-25): 'same canonical id, same perch' exists so that state a consumer PERSISTED IN THE PERCH survives the recovery — an adapter's repo binding, a scanner's cursor. The teardown+spawn workaround destroyed exactly that, and its worst cost was SILENT, not the rename: alchemy's tag cursor re-baselines at the digest tip, so tags written between the kill and the re-bind are never scanned — not failed-and-retried, just never seen. A loud failure gets retried; a silent one does not. Gate: int — force-kill a bound persistent instance's process, then prove (1) `shell list` reads offline, (2) `relink` succeeds instead of SHELL_ALREADY_ONLINE and the SAME canonical id + perch survive, carrying perch-persisted consumer state with them (no teardown+spawn, no id churn, no re-baselined cursor), (3) no spontaneous relaunch occurs while the instance sits dead and undriven. FIELD VERDICT — PASS END-TO-END (flynn, alchemy-0, delivered 2026-07-26; v0.43.0, counter 77): every gate leg held in the field, on a record the pre-fix code had already poisoned. (1) `shell list` read OFFLINE while the on-disk info.json still said status=online — the daemon DERIVED offline from the corpse pid (29036 absent from the process table); the record was never corrected and did not need to be. (2) relink ADMITTED, no SHELL_ALREADY_ONLINE — verbatim SHELL_RELINKED:alchemy-0 owner=flynn pid=38644 status=offline, list online thereafter. (3) identity/state integrity: same canonical id, same owner, same perch; repo binding byte-identical (token_provenance=gh-cli); armed=true preserved; no spontaneous relaunch across the ~6.5h dead window. CURSOR SCOPING, ruled at closure (doyle 2026-07-26) — the one non-byte-identical field: the tag cursor moved 206158430541→210453397553 (gen:seq 48:333→49:49) because the recovery crossed a DAEMON restart that slid the retained digest window past the armed cursor. NOT a gap and NOT this req's property failing: spt-core raised after_predates_window and the consumer took its specced armed-cursor branch (alchemy REQ-TAG-SCANNER missed-rows leg — warn the owner, never silently re-baseline an armed cursor; alchemy src/tags.rs, two unit tests) — the FIRST FIELD FIRING of a path previously exercised only by unit tests. The 'no re-baselined cursor' gate property is hereby scoped: byte-identity holds across SHELL death within one daemon generation (the prior LEG 5 measurement, one daemon instance, uncontradicted — different scenario, generation never moved); across a daemon restart that slides the window the contract is DETECT-AND-ANNOUNCE, not preserve — corroborated independently by a second record's cursor decomposing under the same gen:seq scheme (4294967486 = 1:190). Practical exposure in this event: nil (every agent stood down through the gap; no tag traffic to miss). SECOND SPECIMEN HELD, not spent (doyle ruling 2026-07-26): liam's record — precondition verified by flynn (pid 35464 a CLEAN corpse, not recycled, so the accepted-at-mint recycled-pid caveat does not apply; same poisoned status=online shape; binding intact; cursor armed at 4294967486) — is deliberately NOT relinked while its owner is absent: the admit gate is already field-proven, a relink measurably re-baselines the cursor across a daemon-restart boundary, and liam's is the only untouched specimen with the longer (~8.5h) gap — spending it would destroy the cursor datum liam's own return produces, to buy a redundant second sample of a gate already proven to admit. It waits for liam. SPECIMEN HARVESTED (liam's own return, 2026-07-26 — operator-authorized endpoint start executed by doyle; pre-state snapshotted by doyle AND independently re-verified on disk by liam before touching anything): (1) relink ADMITTED at the ~8.5h gap — verbatim SHELL_RELINKED:alchemy-0 owner=liam pid=47608 status=offline (online at bind); the poisoned record did not refuse; info.json online is now TRUE. (2) integrity FULL PASS: id unchanged (no new mint slot), perch onlined at bind, repo binding byte-identical (token_provenance gh-cli), and the ROTATION-CARRIES-THE-DURABLE-CHANNEL machinery observed live in the field for the first time — link.token.retired present beside the fresh mint, and a post-relink round trip proven end-to-end (shell cmd spooled -> engine executed -> reply home on the owner's perch), not merely an outbound spool. (3) cursor, the sharpest datum of the harvest: BYTE-IDENTICAL through the relink itself (4294967486 armed at pid-change instant, doyle's file watch) — the relink preserves; the SCANNER moves. First scan re-baselined 1:190 -> 2:0 silently, and liam PROVED no-fire-by-precondition rather than assuming: the daemon restart bumped the generation but EVICTED NOTHING (window floor 4294967464 < cursor 4294967486, every gen-1 row past the cursor still retained), so after_predates_window is correctly false at every link of the chain (filter_after floor test -> emit-only-when-true -> unwrap_or(false)) and the missed-rows announce had no precondition to fire on. Flynn's earlier gen-bump reading is refined by this: a restart alone does not slide the window; flynn's announce fired because HIS window had genuinely slid. Scope note carried: a valid negative — the fix's announce chain verified link-by-link to contract on a specimen where it correctly stayed quiet. RESIDUE from the harvest (liam's find, alchemy lane, no core REQ): permanently-unsealable turns from a dark session are jumped by scanner cursor advance with no possible missed-rows notice (a generation bump is not a slide); ruled scanner-side announce for input-bearing null-seq jumps, pseudo-turns jump silently by design; core's per-turn input/input_seq/partial already suffice to discriminate." required_stages = ["doc", "impl", "unit", "int"] # ACTIVATED at mint (doyle 2026-07-25) — field-blocking; flynn eats id churn every alchemy release until it lands. LANDED v0.43.0 (counter 77, PR #76/#77); field-verified 2026-07-26 — see FIELD VERDICT in title. @@ -4151,13 +4157,23 @@ required_stages = ["doc", "impl", "unit"] # ACTIVATED BAROMETER W1 at impl+unit [[requirements]] id = "REQ-SHELL-PERSISTENT-BOOT-RESTORE" -title = "A `persistent` SHELL COMES BACK AFTER A NODE RESTART, WHICH IS THE ONE CASE THE CONTRACT'S OWN SENTENCE PROMISES AND THE IMPLEMENTATION NEVER HANDLED. CONTEXT.md states it plainly -- a persistent shell is automatically brought online whenever its owner endpoint is online -- and the shipped code honors that across a graceful suspend->wake rest edge ONLY. A machine restart produces no such edge and there is no boot-time restoration at all, so after a reboot every persistent shell on the node stays down until a human runs `relink` by hand, with no surface saying so. Field, HFENDULEAM 2026-08-01: the fleet came up at 01:11 and NOT ONE persistent shell came back; the two instances that did were relaunched by a human at 01:27, sixteen minutes later. TWO INDEPENDENT GATES, both of which must fall or the promise stays broken. GATE 1, no restoration path: the five launch sites all require either an explicit command, a rest edge, or a `wake_command` the adapter may deliberately not declare, and daemon boot starts only the servicehost sweep and the wake reconciler. GATE 2, the stale record blocks the cascade even when an edge DOES arrive: the to_active arm gates on the RECORDED status being offline, and a machine death breaks no link, so the close that would write offline never runs and the record says online over a corpse forever. LEG (a) HEALS THE RECORD TO THE TRUTH, with a BIRTH-STAMPED probe rather than bare pid liveness -- a pid namespace resets at boot, so a corpse pid can be REUSED by an unrelated process, and a reused pid reads alive, makes the derived status read online, and makes `relink` refuse SHELL_ALREADY_ONLINE: the one command that recovers the instance refusing on the very state that is wrong. TWO STAMPS ARE PARKED, NOT ONE, AND THE REASON IS A PLATFORM ASYMMETRY THAT WOULD OTHERWISE SHIP SILENTLY BROKEN ON LINUX: `process_started_at` returns a Windows FILETIME (absolute, comparable across boots) but on Linux returns field 22 of /proc//stat, which is JIFFIES SINCE BOOT and resets every boot, so a value recorded before a restart is indistinguishable from a fresh one after it. The NATIVE stamp is therefore used for one question only -- the same-pid-different-process PAIR test within a single boot, mirroring `liveness::relay_liveness` -- and an ABSOLUTE epoch-ms launch stamp is parked beside it for the cross-boot question. LEG (b) IS A ONCE-PER-DAEMON-GENERATION BOOT SWEEP relaunching persistent instances whose owners are online, and its discriminant is THE LAUNCH STAMP PREDATES THE BOOT INSTANT, which preserves the class-c force-kill ruling BY CONSTRUCTION rather than by a second rule: a force-kill during steady-state operation leaves a corpse launched AFTER boot, so it is never spontaneously relaunched and the mid-deploy exe-overwrite hazard that ruling protects is untouched. A NAMED SLACK CONSTANT absorbs boot-instant jitter so a process launched moments after boot is never misclassified as a corpse. Gate: impl -- the two stamps parked at the two production pid-write sites (the census is authoritative: five LAUNCH sites funnel through exactly TWO pid writes), the birth-stamped death probe, the record heal in the daemon-side reconciler guarded on an actual change of the recorded value, the boot-instant helper on both platforms, and the boot sweep with its predates-boot discriminant plus slack; unit -- a reused pid (same pid, different birth stamp) reads GONE where bare-pid liveness reads alive (the row the shipped probe cannot pass), a same-pid-same-birth process reads HELD, the heal writes offline over a stale online record and writes NOTHING when the record already matches the truth (the guard asserted by counting writes, since a heal that rewrites every cycle passes any state-only assertion), a corpse launched BEFORE the boot instant is eligible for restore while one launched AFTER it is NOT (the class-c pair, asserted together since either alone passes an implementation that restores everything or nothing), and an instance whose OWNER is offline is not restored even though its corpse predates boot." -required_stages = ["doc", "impl", "unit"] # doc ACTIVATED BAROMETER W2 doc leg (doyle-ruled, this milestone): the CONTEXT.md `persistent` amendment states the boot sweep and its four conjunct gates incl. no-stamp => not restored, which IS this requirement's doc evidence; leaving the stage inactive would have made it evidence the registry cannot see. +title = "Persistent shells are restored on each daemon-generation boot sweep whenever their owner is online and they are down in fact, regardless of machine boot time or recorded launch age (releases#287 operator ruling). Missing launch stamps do not disqualify a down instance. Preserve birth-safe liveness, do not duplicate a live launch awaiting bind, and heal stale online records only on an actual change. The force-kill freeze applies only to nonpersistent instances. Regression: restore same-boot and unstamped corpses without a suspend edge; refuse offline owners, nonpersistent instances and live binaries." +required_stages = ["doc", "impl", "unit"] [[requirements]] id = "REQ-SHELL-OWNER-ONLINE-RESTORE" -title = "A `persistent` SHELL WHOSE OWNER ENDPOINT WAS NOT YET ONLINE WHEN THE BOOT SWEEP RAN IS RESTORED WHEN THAT OWNER COMES ONLINE, RATHER THAN STAYING DOWN UNTIL A HUMAN RELINKS IT. ROOT (releases#228): daemons boot before endpoints do, so the boot sweep of REQ-SHELL-PERSISTENT-BOOT-RESTORE correctly SKIPS an owner that is not up yet -- an offline owner is owed nothing -- and then nothing ever revisits it: the sweep is once per daemon generation, and bringup from offline produces no rest edge, so the ADR-0048 cascade never fires either. The shell stays down until a `shell cmd` wakes it or an operator relinks by hand, and whether a given node restart strands its shells is an ordering accident. THE ONCE-NESS IS SCHEDULING, NOT SAFETY (doyle, releases#228, re-measured at head): the `launch_predates_boot` corpse-boot discriminant is what preserves the force-kill ruling, and it is MONOTONE WITHIN A MACHINE BOOT -- every launch path restamps `launched_ms`, a kill never restamps, `boot_ms` is the machine boot instant -- so re-evaluating the conjuncts later in a generation can only refuse MORE, never less. FIX: the reconcile loop carries the SAME per-owner restore body the boot sweep runs, triggered on owner offline->online transitions since the last pass. THE TRIGGER IS AN EDGE, NOT A LEVEL, and the distinction is the whole bound: success self-limits because the launch restamps the instance out of eligibility, but a FAILING launch under a level trigger -- owner is online -- retries every five seconds for as long as the owner stays up, while an edge bounds it to one attempt per owner-online event. THE REFUSAL ARMS ARE CARRIED BY CONSTRUCTION, not by restatement: both triggers call one shared per-owner body, so persistent-only, owner online, down in fact via the derived read with the pair test, stamp present, launch predates boot, and no boot oracle means restore nothing can never drift apart between the two paths. THE EDGE STATE IS SEEDED BEFORE THE SWEEP, deliberately: seeded after, an owner that comes online in the window between the sweep and the first tick reads as already-online and never fires -- the exact miss this requirement exists to close -- while seeding before can at worst re-attempt an instance the sweep already restored, which the RESTAMPED BIRTH STAMP refuses -- and NOT the down-in-fact arm, which is the arm that reads like the answer and is not: a launch is not the online switch, so a just-restored instance keeps an `offline` record until its bind-shell handshake and still reads as down in fact; what refuses the second attempt is that the launch moved `launched_ms` past the boot instant. Each trigger names ITSELF in its event line, because the field evidence that diagnosed this ticket was a count of those events and a shared name would leave the next investigation unable to say which trigger fired. Kin REQ-SHELL-PERSISTENT-BOOT-RESTORE (the boot trigger and the shared arms), REQ-HAZARD-RESTART-STRANDS-PERSISTENT-SHELLS (the invariant this closes the remaining seam of), REQ-SHELL-2 (the online/offline mutual exclusivity the edge path holds by stopping an armed watcher before it launches -- mid-generation a watcher CAN be armed for the casualty, unlike at boot)." -required_stages = ["doc", "impl", "unit"] # ACTIVATED CONDUIT W2 in the commit landing the evidence (doyle ruled shape (c) with conditions on releases#228; (a) not built, (b) rejected). doc = the CONTEXT.md `persistent` amendment plus KNOWN-HAZARDS 2.7 and MANIFEST.md Sleep/wake, all of which stated the boot sweep as the ONLY trigger and went false the moment a second trigger existed. MINTED CONDUIT W2 (todlando, 2026-08-28), releases#228. +title = "Every down persistent shell of an owner is restored on its offline-to-online edge, including corpses launched after machine boot and instances with no launch stamp (releases#287 operator ruling supersedes the releases#228 boot-age gates). Both triggers share the same persistent-only, owner-online, birth-safe down-in-fact decision; a live launch awaiting bind is not duplicated. The edge is seeded before the boot sweep and bounds failed attempts to one per owner-online event. Stop an armed watcher before launching, and name the triggering event and every restored owner-qualified instance." +required_stages = ["doc", "impl", "unit"] + +[[requirements]] +id = "REQ-SHELL-HEAL-ENUMERATION" +title = "The daemon's existing SHELL_RECORD_HEALED diagnostic enumerates every successfully healed instance per owner by owner-qualified identity, not only the first or a representative shell; healthy and already-healed instances produce no heal event (releases#287 rider)." +required_stages = ["doc", "impl", "unit"] + +[[requirements]] +id = "REQ-SHELL-RELINK-BINDING-RESULT" +title = "A successful local shell relink reports status=binding while awaiting the bind handshake, rather than describing the successful launch as offline; bind remains the transition to online (releases#287 rider)." +required_stages = ["doc", "impl", "int"] [[requirements]] id = "REQ-SHELL-KILL-AUTHENTICATED" @@ -4166,7 +4182,7 @@ required_stages = ["doc", "impl", "unit", "int"] # ACTIVATED BAROMETER W2 reope [[requirements]] id = "REQ-HAZARD-RESTART-STRANDS-PERSISTENT-SHELLS" -title = "A NODE RESTART MUST NOT PERMANENTLY STRAND EVERY `persistent` SHELL, AND THE EXISTING CASCADE TEST CANNOT SEE THAT IT DOES. Distinct from REQ-HAZARD-SHELL-STALE-ONLINE, which governs a force-kill DURING operation: this is a different trigger with a different safe remedy, and conflating them is how the restart class stays invisible. THE COVERAGE GAP IS THE HAZARD'S OWN SUBJECT, and it must be named here so no future reader mistakes the cascade test for coverage of this class: `rest_edges_cascade_shells_with_divergence` (spt-daemon resting.rs) is the cascade's only coverage and it walks the graceful path ONLY -- it SUSPENDS FIRST, and the suspend edge WRITES the offline record, so by the time the wake edge fires the instance is already in exactly the state the to_active arm requires. The restart shape -- record left online over a corpse, no rest edge ever -- is UNREACHABLE FROM THAT TEST BY CONSTRUCTION. The test is not weak; its SETUP SUPPRESSES THE FAILING ARM, so a reader asking whether the cascade is covered gets a true answer to the wrong question. This is the third suppressed-arm instance found in one week, and the pattern is the same each time: a fixture that establishes the precondition the code under test is supposed to establish. THE OWNER-FACING SURFACES HIDE THE FAULT, which is why it survived unreported for days: every display path (`shell list`, its --json twin, the shell-context render) routes through the ONE discovery seam, which DERIVES status and so correctly reports the binary as offline, while the only consumer whose decision matters -- the wake cascade -- reads the RECORDED field that no owner-facing surface shows. Absence of a visible symptom is therefore not evidence of absence of this fault, and a clean-looking context render must never be taken as proof the class did not occur; the correct falsifier is the on-disk record. Gate: impl -- the restart shape reaches a restoration path at all (leg b) and the record stops lying (leg a); unit -- the RESTART shape specifically, constructed WITHOUT a suspend edge: an instance whose record says online over a corpse that predates boot, with an online owner, is restored -- the row `rest_edges_cascade_shells_with_divergence` cannot express, asserted on a fixture that never calls the suspend path, since a fixture that suspends first re-creates the suppressed arm and proves nothing about this class." +title = "A daemon-only or machine restart must not strand persistent shells. Heal stale online records and restore every down persistent instance when its owner is online, without boot-age or launch-stamp eligibility gates (releases#287 operator ruling). Birth-safe liveness still excludes a live binary, including one awaiting bind. The regression fixture must construct online records over corpses WITHOUT a suspend edge, including a corpse launched AFTER machine boot; the existing rest_edges_cascade_shells_with_divergence fixture suspends first and therefore suppresses the failing arm rather than covering it." required_stages = ["doc", "impl", "unit"] # doc ACTIVATED BAROMETER W2 doc leg (doyle-ruled, this milestone): the KNOWN-HAZARDS 2.7 entry the mint note said was owed. AGENTS.md rule 4 makes every KNOWN-HAZARDS entry a REQ-HAZARD-* requirement and rule 1 makes the tag ride the same change as the evidence, so an untagged entry would be the untagged-evidence shape rule 1 bans. -- ACTIVATED BAROMETER W2 at impl+unit once BOTH legs landed -- deliberately not at leg (a), since the gate clause covers the boot sweep too and activating early would have let the requirement read satisfied while half its impl was unbuilt. MINTED BAROMETER W2 (todlando, 2026-08-02), releases#78 -- inactive-first. KNOWN-HAZARDS entry owed with the doc leg. The suppressed-arm sentence is REQUIRED text per the filing and doyle's dispatch: the hazard exists partly BECAUSE the existing test reads as coverage, so a requirement that did not name the gap would leave the same trap for the next reader. @@ -4386,6 +4402,7 @@ requirements = [ [[groups]] name = "cli-output-surface" requirements = [ + "REQ-SHELL-RELINK-BINDING-RESULT", "REQ-BIND-HONEST-SELF-STAMP", "REQ-CLI-1", "REQ-CLI-2", @@ -4550,6 +4567,7 @@ requirements = [ [[groups]] name = "endpoint-presence-truth" requirements = [ + "REQ-SHELL-HEAL-ENUMERATION", "REQ-CONN-POISON-ATTRIBUTION", "REQ-DAEMON-5", "REQ-DAEMON-6",