---
phase: 05-deploy
plan: 11
subsystem: soak / observability
tags: [soak, multi-client, ws-stability, ping-interval, dep-08, dep-06, dispatched]
dependency_graph:
  requires: [05-05, 05-08, 05-09, 05-10]
  provides: [soak-harness, soak-ci-workflow]
  affects: [05-13, 05-14]
tech_stack:
  added: []
  patterns: [2-client-soak, @colyseus/sdk joinOrCreate, OpenObserve REST best-effort]
key_files:
  created:
    - scripts/soak-staging.mjs
    - .github/workflows/soak-staging.yml
  modified:
    - package.json
decisions:
  - "p95 latency thresholds: warn at 25ms, hard-fail at 100ms — calibrated for CI runner jitter"
  - "OpenObserve ingest check is best-effort; flycast unreachable from public CI runners by design"
  - "Soak failure opens GH Issue (soak-fail label) but does NOT auto-rollback (per 05-CONTEXT.md D-11)"
metrics:
  duration: "2m"
  completed: "2026-05-08T11:45:40Z"
  tasks_completed: 2
  tasks_total: 2
---

# Phase 05 Plan 11: Soak Harness + Nightly CI Workflow Summary

30-min 2-client soak harness against wss://rebno-staging.fly.dev using @colyseus/sdk joinOrCreate with staging-invite token, asserting zero spurious WS disconnects (DEP-08), zero baseline RATE_LIMITED events, and p95 tick latency < 100ms — dispatched from deploy-staging.yml and run nightly.

## Tasks Completed

| Task | Name | Commit | Files |
|------|------|--------|-------|
| 1 | scripts/soak-staging.mjs — 30-min 2-client harness | bd460ba | scripts/soak-staging.mjs, package.json |
| 2 | .github/workflows/soak-staging.yml — workflow_dispatch + nightly cron | 3542b30 | .github/workflows/soak-staging.yml |

## What Was Built

### Task 1: scripts/soak-staging.mjs

A standalone Node.js ESM harness that:

- **Two clients** (`alice`, `bob`) join the Colyseus `rebno` room via `@colyseus/sdk` `joinOrCreate`, supplying `protocol_version` (read from `packages/protocol/dist/index.js` at runtime, fallback to `1`) and `invite` (from `STAGING_INVITE_TOKEN` env — Plan 10 staging-invite middleware gate).
- **Drive cadence** (mirrors `authority.integ.test.ts:135-156`):
  - Move input every 100ms (10 Hz) via `room.send('input', {...})`
  - Chat send every 10s (every 100 ticks) via `room.send('chat_send', {...})`
  - Colyseus heartbeat is automatic (pingInterval=3000ms set in Phase 4 `apps/server/src/index.ts`)
- **Collects**:
  - `stats.disconnects`: count of `onLeave` events with code !== 1000 (spurious)
  - `stats.rateLimited`: count of `rate_limited` / `RATE_LIMITED` messages
  - `stats.latencies`: round-trip samples from input-send → first `onStateChange`
- **Post-soak assertions**:
  - `disconnects === 0` — FAIL with DEP-08 violation message
  - `rateLimited === 0` — FAIL with server drift message
  - `p95 > 100ms` — FAIL (hard cap)
  - `p95 > 25ms` — WARN only (soft cap; CI runner jitter tolerance)
- **OpenObserve ingest check** (DEP-06, best-effort): GETs `${OBS_URL}/api/default/traces/_search?size=1` with basic auth; skips gracefully if `OBS_USER`/`OBS_PASS` unset or flycast unreachable.
- **isMain guard**: allows the file to be imported in tests without executing.

### Task 2: .github/workflows/soak-staging.yml

CI workflow:
- **Triggers**: `workflow_dispatch` (optional `sha` input) + `schedule: cron '0 6 * * *'` (nightly)
- **Concurrency**: `group: soak-staging, cancel-in-progress: true` — a new staging deploy cancels the previous soak run
- **Job timeout**: 35 minutes (30-min soak + 5-min slop)
- **Steps**: checkout → pnpm/action-setup → setup-node → `pnpm install --frozen-lockfile` → `pnpm -r --filter @rebno/protocol build` → `pnpm soak:staging`
- **Env**: `STAGING_INVITE_TOKEN` from secret, `STAGING_WSS_URL` from variable (default `wss://rebno-staging.fly.dev`), `SOAK_DURATION_MINUTES: '30'`
- **Failure step**: `gh issue create --label soak-fail` with run URL — no auto-rollback (per D-11)

## Soak Cadence and Thresholds

| Parameter | Value | Source |
|-----------|-------|--------|
| Move input frequency | 10 Hz (every 100ms) | authority.integ.test.ts:135-156 |
| Chat frequency | 0.1 Hz (every 10s) | 05-PATTERNS.md §soak-staging.mjs |
| Colyseus pingInterval | 3000ms (server-side) | Phase 4 apps/server/src/index.ts |
| Fly idle timeout | 60s | 05-RESEARCH.md DEP-08 analysis |
| p95 latency soft warn | 25ms | 05-PATTERNS.md calibration |
| p95 latency hard fail | 100ms | CI runner jitter tolerance |
| Soak duration | 30 min (default) | SOAK_DURATION_MINUTES env |

The Fly idle timeout (60s) is well above the Colyseus pingInterval (3s) + soak move cadence (100ms). Continuous 10Hz traffic keeps connections alive empirically, validating DEP-08.

## OpenObserve Ingest Verification (DEP-06)

The OBS query targets `${OBS_URL}/api/default/traces/_search?size=1`. In default GitHub Actions CI runners, the `rebno-obs.flycast:5080` endpoint is unreachable (private Fly.io 6PN / wireguard). The check is **best-effort only**:

- If `OBS_USER`/`OBS_PASS` are unset → logs `SKIPPED` and continues.
- If reachable → asserts `hits.total >= 1`; warns if 0 (does not fail the soak).
- If unreachable → logs `unreachable (expected if CI lacks wireguard/flycast)` and continues.

DEP-06 [int] live verification is recorded in `05-HUMAN-UAT.md` after first operator-run soak with wireguard access to flycast.

## Operator Notes for Plan 13 RESTORE.md

To enable soak in CI the following must be configured in the GitHub repository:

1. **Secret `STAGING_INVITE_TOKEN`** — the staging-invite token set in the `STAGING_MODE_INVITE_TOKEN` environment variable on the Fly.io staging app (Plan 10). Set via: `gh secret set STAGING_INVITE_TOKEN`
2. **Variable `STAGING_WSS_URL`** (optional) — defaults to `wss://rebno-staging.fly.dev`. Override if staging app URL changes. Set via: `gh variable set STAGING_WSS_URL`
3. **Label `soak-fail`** — must exist in the GitHub repo for the failure step to tag issues. Create via: `gh label create soak-fail --color cc0000`

## Plan 14 Note (verify-phase-5.mjs)

The soak workflow is **out-of-band** per 05-RESEARCH.md Pattern 7. It is:
- Dispatched by `deploy-staging.yml` after a successful staging deploy (not part of the composite gate)
- NOT included in `verify-phase-5.mjs` assertions
- Recorded in `05-HUMAN-UAT.md` after first green run by operator

The first green soak result (zero disconnects, zero rate-limited, p95 < 25ms on staging infrastructure) is what closes DEP-08 [int]. Until that first run, `05-VALIDATION.md` row `5-11-01` remains `⬜ pending`.

## Deviations from Plan

None — plan executed exactly as written.

The soak script provides slightly more detail in the isMain guard (uses both `process.argv[1]` equality check and `.endsWith` pattern) for robustness across different invocation styles. This is a minor enhancement, not a deviation.

## Threat Model Compliance

| Threat ID | Disposition | Verification |
|-----------|-------------|--------------|
| T-DEP-Soak-Token-Leak | mitigated | STAGING_INVITE_TOKEN only read from env; never echoed to stdout/stderr; GH Actions auto-masks `secrets.*` in logs |
| T-DEP-Soak-Block-Deploy | mitigated | `cancel-in-progress: true` on concurrency group; failure does not auto-rollback |
| T-DEP-Soak-Drift | accepted-with-doc | CI ubuntu-latest is source of truth; p95 warn/fail thresholds calibrated for runner jitter |
| T-DEP-08 | mitigated | Soak IS the empirical verification; 10Hz continuous traffic far below Fly 60s idle timeout |

## Self-Check: PASSED

- `scripts/soak-staging.mjs` — FOUND
- `.github/workflows/soak-staging.yml` — FOUND
- `package.json` `soak:staging` script entry — FOUND
- Commit bd460ba (Task 1) — FOUND
- Commit 3542b30 (Task 2) — FOUND