# Phase 3: Server Documentation & Schemas - Research
**Researched:** 2026-05-03
**Domain:** Reverse-engineering server documentation + 39dll opcode schema + `.bno`/`.bnb`/`.bnu` save-format schema + Drizzle/SQLite persistence ADR + canonical-snapshot ADR + closed parity-feature checklist
**Confidence:** HIGH (ground truth verified directly against `extracted/server-5-4/scripts/` for all opcode and save-format claims; library versions verified via `npm view` 2026-05-03)
## User Constraints (from CONTEXT.md)
### Locked Decisions
#### SDOC-06 — Canonical-snapshot ADR (interactive)
- **D-01:** Canonical snapshot is the **production seed for migration**. Returning BNO players see their last character / inventory / username on first login against the new server. The seed is consumed by Phase 4 SRV-10 (legacy account import script) and Phase 7 PAR-05 (`.bnu` character migration). Not historical preservation only; not GML-drift-reference only.
- **D-02:** **Pick one snapshot whole — `legacy/servers/enlyzeam-current/`.** No per-record merge across the three. Rationale: largest world-state dataset (973 `.bnu` user files vs 967 archive vs 211 local-current; 597-line `localList.txt`; newer `MB_Log.bnb` mtime than archive's identical-size copy).
- **D-03:** **Skip older-GML extraction.** Master 5-4 is source-of-truth. Do NOT extend Phase 1 to Master 4.gmd / 5-1.exe / 5-2.exe / 5-3.exe. On-demand extract only if a specific opcode trace fails during execution.
- **D-04:** Phase 3 documents the **legacy-credential staging schema** consumed by Phase 4 SRV-10/SRV-11. Plaintext from `localList.txt` lands ONLY in `legacy_credentials_staging` (read-once-then-purged), NEVER in production `accounts`. Phase 4 SRV-09/10/11 owns argon2id rehash + force-reset flow. Phase 5 RESTORE.md includes the staging-table read-once-then-purge step.
#### SDOC-02 — 39dll wire protocol (Claude's discretion, modernize/streamline bias)
- **D-05:** Three artifacts from a single canonical JSON: `protocol.json` (canonical), `tools/protocol-doc/output/protocol.ts` (TS types + codec stubs, not yet a workspace package), `protocol.md` (rendered with AUTOGEN blocks).
- **D-06:** Per-opcode metadata: `opcode_byte`, `direction` (`c2s`|`s2c`), `mvp` (boolean for CLI-08 slice), `name` (semantic), `fields[]` with `{name, gml_type, ts_type, byte_size}`, `gml_origin[]` with `{script, line, snippet}`, `sample_bytes` (hex test vector).
- **D-07:** Methodology — grep `sendmessage`/`receivemessage` in `extracted/server-5-4/{scripts,objects}/`, trace backward through `clearbuffer; writebyte; writedouble; writestring; ...` sequence. Cross-reference `legacy/open-source-release/BN Online Message ID's Table.xls` for legacy hints, but extracted GML is canonical truth on conflict. Wireshark = post-hoc validation only.
#### SDOC-03 — Save-format schemas (Claude's discretion, modernize/streamline bias)
- **D-08:** **First task of phase: wiki errata.** Update `decomp/wiki/16-bno-bnb-notes.md` and PITFALLS A5 wording to `file_text_*` (NOT `file_bin_*`). Verified in `0365-mb_backup.gml`, `0367-users_restore.gml`. Commit before generating SDOC-03 schemas.
- **D-09:** Three artifacts: `save-formats.json` (canonical), `tools/save-format-doc/output/save-formats.ts` (TS types + parsers), `save-formats.md` (rendered, AUTOGEN).
- **D-10:** Per-format metadata: `extension`, `filename_pattern`, `load_script` + `save_script` GML citations, `mvp`, `grammar` (ordered list of fields with `{name, type ('string'|'real'|'sectioned'), repetition, terminator, semantics}` — sectioned formats use section grammar, not flat field list), `sample_records`.
#### SDOC-04 — Persistence ADR (Claude's discretion, modernize/streamline bias)
- **D-11:** Lock SQLite + Litestream-to-Tigris. Postgres rejected with reason; OPS-03 (v2) preserves migration option.
- **D-12:** Drizzle ORM, schema-first. Locked at end of Phase 3.
- **D-13:** Phase 3 produces `docs/extracted-server/schema.sql` + Drizzle table definitions for: `accounts`, `legacy_credentials_staging`, `characters`, `inventory_items`, `message_board_topics`, `message_board_replies`, `audit_log` (seeded empty for Phase 7 PAR-07), `sessions`. Phase 4 SRV-01..03 consumes verbatim.
- **D-14:** No original-table-shape preservation. Original `.bnu` text format is read-once-then-discarded by SRV-10 import. New schema is normalized SQL designed for the new server.
#### SDOC-01 + SDOC-05 — Subsystem narrative + parity checklist (Claude's discretion, modernize/streamline bias)
- **D-15:** Subsystem MD list LOCKED: `account-auth.md`, `world-simulation.md`, `room-management.md`, `chat.md`, `persistence.md`, `packet-protocol.md`, `admin-anti-port.md`, `client-server-bridge.md`, `message-board.md`, `unknown-actions-status.md`, `README.md`. Plus autogen `asset-catalog/` subtree.
- **D-16:** Functional clusters with citation + key-script call-out (parallel Phase 2 D-10). Quote 5-30 line snippets inline. mvp: yes/no front-matter per feature.
- **D-17:** Re-invoke `tools/asset-catalog` against `extracted/server-5-4/` → `docs/extracted-server/asset-catalog/index.{json,md}`. Repo-level `pnpm catalog:server` script added; existing `pnpm catalog:all` extended to run client + server.
- **D-18:** **Per-feature** parity checklist rows (NOT per-opcode, NOT per-script). Each row carries `feature`, `originating_gml[]`, `originating_opcodes[]`, `originating_save_fields[]`, `mvp`, `disposition` (`in-phase-6`|`in-phase-7`|`deferred-stage-8`|`rejected-with-reason`), `reason`, `modernized_replacement`. Locked at phase end.
- **D-19:** Aggressive `rejected-with-reason` for any feature whose original implementation is incompatible with server-authoritative discipline (PITFALLS B1) OR depends on Ctrl+E clipboard-RCE admin model.
- **D-20:** Modernized admin command surface concrete (TS intent shape per command) — `kick`, `mute`, `ban`, `assign-role`, `view-audit-log`, `mb-moderate`, `account-recover`. Phase 7 PAR-07 implements; Phase 4 SRV-12 documents in apps/server as anti-port stub.
#### Tooling
- **D-21:** Two new tools: `tools/protocol-doc/`, `tools/save-format-doc/`. Standalone Node CLIs, not yet workspace packages (parallel Phase 1 D-17).
- **D-22:** Lints: `lint-protocol.mjs` (every opcode has `gml_origin`; every `mvp:true` opcode appears in CLI-08 message-list), `lint-save-formats.mjs` (every format has `load_script` + `save_script`), `lint-parity.mjs` (every row has non-empty `originating_gml` AND non-null `disposition`).
### Claude's Discretion
- Subsystem MD list (D-15) is locked but additions permitted during planning if extraction surfaces a coherent subsystem not on this list.
- Functional-cluster grouping criteria (D-16) — left to doc-writer's judgment as long as every server script (135) and every server object (58) is reachable from at least one subsystem doc via auto-generated cross-ref index.
- Drizzle table column naming, indexes, foreign-key cascade rules — researcher/planner discretion within D-13 table list.
- Choice of which legacy-encoding-quirks to preserve in `legacy_credentials_staging.algorithm` enum — driven by what's actually present in `localList.txt` plaintext (verified during planning).
- Parity-checklist row ordering, grouping, and per-row reason-text style (D-18).
### Deferred Ideas (OUT OF SCOPE)
- **Per-record merge across the three snapshots.** Rejected per D-02.
- **Older Master `.gmd`/`.gb1` extraction** (Master 4.gmd, 5-1..5-3.exe). Rejected per D-03.
- **Local-current snapshot** as canonical seed. Documented in ADR 0003 with reason; no code consumes it.
- **`apps/server` scaffold.** Phase 4 owns. No `apps/server/` directory in Phase 3.
- **`packages/protocol` scaffold.** Phase 4 owns. `tools/protocol-doc/output/protocol.ts` is the artifact Phase 4 imports from.
- **`packages/game-logic` scaffold.** Phase 4 owns. Movement/collision/room-model rules derived during Phase 4 planning.
- **Modernized admin web UI implementation.** Phase 7 PAR-07. Phase 3 specifies command surface only.
- **Vintage soundfont / audio pipeline.** Phase 7 AST-02.
- **Drizzle migration runner wiring.** Phase 4 owns. Phase 3 produces baseline DDL + `0001_baseline.sql` migration but does not run it.
- **Wireshark capture of original server traffic.** Per PITFALLS A4 — captures are post-hoc validation only.
- **Older-snapshot legacy-credential import** (enlyzeam-archive's localList.txt). Rejected per D-02.
- **Asset format conversion** (BMP→PNG, MIDI→OGG, font→WOFF2). Phase 6/7.
- **pnpm workspace conversion.** Phase 4.
- **Round-trip GML → schema → GML generators.** Premature; couples doc shape to runtime code shape.
## Phase Requirements
| ID | Description | Research Support |
|----|-------------|------------------|
| SDOC-01 | `docs/extracted-server/` documents every server feature: account/auth, world simulation, persistence, chat, room management, packet protocol, admin tooling | §Subsystem-MD Authoring + §Functional Cluster Inventory below identify which scripts/objects feed each subsystem MD; §Hand-Authored + Autogen pattern reused from Phase 2 D-11; §Server vs client asset divergence flags `audio.md`-equivalent absence (server has no rendering layer) |
| SDOC-02 | Reversed 39dll wire protocol opcode table | §39dll Opcode Extraction Procedure defines exact grep + trace methodology; §Verified Opcode Inventory enumerates every opcode found in `0359-server_receive.gml`; §`tools/protocol-doc` Architecture defines emit pipeline; §Cross-Reference XLS lock confirms call order beats XLS hints on conflict |
| SDOC-03 | `.bno`/`.bnb`/`.bnu` field-by-field schemas | §Save Format Inventory enumerates every save script + format; §`file_text_*` Errata documents the wiki/PITFALLS correction; §Sectioned Grammar models `MB_Log.bnb` `@TOPIC`/`@REPLY` markers; §`tools/save-format-doc` Architecture defines emit pipeline |
| SDOC-04 | Persistence ADR (SQLite + Litestream vs Postgres) | §Drizzle Schema Draft enumerates every table feeding Phase 4 SRV-01..03; §SQLite Lock Rationale ties data model to single-writer/append-mostly characteristics; §Postgres Rejection records the reason; §Litestream-to-Tigris configuration surfaces RPO < 1s lock |
| SDOC-05 | Closed parity-feature checklist | §Parity-Checklist Construction Methodology defines per-feature row generation from CDOC-01 + opcode + save-field traceability; §Modernize-Bias Rejection Catalog enumerates known anti-port features; §Lock Discipline ensures no scope explosion |
| SDOC-06 | Canonical-snapshot ADR | §Canonical Snapshot Selection records the dataset metrics that drive enlyzeam-current; §Legacy-Credentials Staging Pipeline documents the read-once-then-purge migration path; §Per-Snapshot Disposition Table captures rejected snapshots with reason |
## Summary
Phase 3 is documentation only — no runtime code, no `apps/`, no `packages/`. Two new offline tools (`tools/protocol-doc`, `tools/save-format-doc`) emit machine-readable + human-readable + TS-typed views of the same underlying data. The work divides cleanly into seven workstreams:
1. **Wiki errata first.** Update `decomp/wiki/16-bno-bnb-notes.md` and PITFALLS A5 from `file_bin_*` to `file_text_*`. Ground truth in `extracted/server-5-4/scripts/0365-mb_backup.gml` (`file_text_open_write("MB_Log.bnb")`) and `0367-users_restore.gml` (`file_text_open_read("User_DBUpdated.bnu")`). Commit BEFORE generating schemas so downstream agents read the corrected wiki.
2. **`tools/protocol-doc`** — TS Node CLI that reads `extracted/server-5-4/scripts/`, walks every `sendmessage`/`receivemessage` site, traces the surrounding `clearbuffer; writebyte; writeint; writestring; ...` sequence, and emits `protocol.{json,ts,md}`. Architecture mirrors `tools/asset-catalog` exactly (cli.ts dispatcher, exit codes 0/1/2, `tsx`-based).
3. **`tools/save-format-doc`** — Same shape, walks `file_text_open_*` call sequences across the 16 file-IO scripts, emits `save-formats.{json,ts,md}` covering `.bno`/`.bnb`/`.bnu` field-by-field grammar.
4. **Subsystem-MD authoring** — 11 hand-authored markdown files under `docs/extracted-server/` with autogen blocks for cross-ref tables. Identical pattern to Phase 2's `docs/extracted-engine/` (already proven).
5. **Server asset catalog** — Re-invoke existing `tools/asset-catalog` against `extracted/server-5-4/` → `docs/extracted-server/asset-catalog/`. Add `pnpm catalog:server` script; extend `pnpm catalog:all` to run both.
6. **ADRs** — Two new ADRs following the `docs/adr/0001-client-engine.md` template (Status / Context / Decision / Consequences): `0002-persistence-layer.md` (SQLite + Litestream + Drizzle locked) and `0003-canonical-snapshot.md` (enlyzeam-current locked, archive + local-current rejected with reason).
7. **Parity-feature checklist** — Per-feature rows in `parity-checklist.json` cross-referenced against opcodes, save fields, and originating GML scripts. Locked at phase end. Closed input to Phase 7 PAR-08.
**Primary recommendation:** Sequence the plans as: (1) wiki errata commit → (2) build `tools/protocol-doc` against ground-truth opcodes already cataloged below → (3) build `tools/save-format-doc` against ground-truth save-script inventory → (4) author subsystem MDs hand-in-hand with running asset-catalog against server tree → (5) compose parity-checklist from CDOC-01 + opcode table + save schema → (6) commit two ADRs → (7) phase lint suite (`lint-protocol`, `lint-save-formats`, `lint-parity`) gates phase exit. Target ~9 plans (one per workstream + 2 ADRs being a single commit). Every claim downstream is traceable back to a `script:line` citation; the modernize-bias steer manifests as aggressive `rejected-with-reason` dispositions in the parity checklist, NOT as ad-hoc improvements to the schemas (which are descriptive of the legacy ground truth, not prescriptive of the new server).
## Architectural Responsibility Map
Phase 3 is single-tier (offline tooling + documentation). Same shape as Phase 2.
| Capability | Primary Tier | Secondary Tier | Rationale |
|------------|-------------|----------------|-----------|
| Read `extracted/server-5-4/scripts/*.gml` for opcode trace | Build-time / offline (`tools/protocol-doc`) | — | No runtime exposure; mirrors Phase 1/2 tools layout |
| Read `extracted/server-5-4/scripts/*.gml` for save-format trace | Build-time / offline (`tools/save-format-doc`) | — | Same |
| GML lexer/tokenizer (lightweight) for `write*`/`read*` call extraction | Build-time / offline (shared between protocol-doc + save-format-doc) | — | Heuristic line-scanner; no full AST needed (per A4 procedure: trace backward through procedural call sequence). See §GML Tokenization for scope |
| Subsystem-MD prose | Authored artifact (`docs/extracted-server/*.md`) | — | Human-curated narrative; cited but not generated |
| Subsystem-MD autogen blocks (script-roster, opcode-list, save-format-list, parity-row counters) | Build-time / offline → in-place doc edit | — | Same tool refreshes delimited blocks inside hand-authored MDs (Phase 2 D-11 pattern, proven) |
| Drizzle schema (`schema.sql` + `tables.ts`) | Authored artifact (`docs/extracted-server/`) | — | Hand-authored from save-format grammars; no runtime use in Phase 3; Phase 4 SRV-01..03 consumes verbatim |
| ADRs | Authored artifact (`docs/adr/0002-*.md`, `0003-*.md`) | — | Lock decisions for Phase 4+ |
| Parity-checklist (`parity-checklist.json`) | Authored artifact + lint guard | — | Closed list; Phase 7 PAR-08 closes against it |
**Tier-correctness sanity check:** Hard rule #6 in CLAUDE.md ("Extract → document → rewrite, in that order. No new TypeScript before Phase 4") forbids new code under `apps/`, `packages/`, or any non-`tools/` runtime path. The plan-checker should reject any task that proposes adding files outside `tools/protocol-doc/`, `tools/save-format-doc/`, `docs/extracted-server/`, `docs/adr/`, or `decomp/wiki/` (errata).
## Standard Stack
### Core (for `tools/protocol-doc` and `tools/save-format-doc`)
| Library | Version | Purpose | Why Standard |
|---------|---------|---------|--------------|
| Node | 22 LTS | Runtime | [VERIFIED: matches `tools/extract-gmd` and `tools/asset-catalog` per Phase 1/2]. No native modules needed — pure file IO + JSON. |
| TypeScript | 5.6.3 | Source | [VERIFIED: pinned by `tools/asset-catalog/package.json`]. Same pin keeps tools tree consistent. |
| tsx | 4.21.0 | Dev runner | [VERIFIED: pinned by `tools/asset-catalog/package.json`]. CLI invoked as `pnpm tsx tools/protocol-doc/cli.ts`. |
| vitest | 4.1.5 | Test runner | [VERIFIED: pinned by `tools/asset-catalog/package.json`]. |
| `@types/node` | 25.6.0 | Type defs | [VERIFIED: pinned by `tools/asset-catalog/package.json`]. |
**No runtime dependencies needed for protocol-doc or save-format-doc.** Both read GML text files (UTF-8 lines), apply line-level pattern matching to extract `clearbuffer/writeX/sendmessage` and `file_text_open_*/file_text_read_*/file_text_close` sequences, and emit JSON+TS+Markdown. Everything is in stdlib.
### Supporting (for XLS cross-reference in `tools/protocol-doc`)
| Library | Version | Purpose | When to Use |
|---------|---------|---------|-------------|
| xlsx | 0.18.5 | Read `legacy/open-source-release/BN Online Message ID's Table.xls` | [VERIFIED: `npm view xlsx version` returned 0.18.5 on 2026-05-03]. Per D-07: XLS provides legacy opcode-name HINTS (e.g., human-readable name for opcode 5), used as cross-reference only — extracted GML wins on conflict. **Optional dependency** — if XLS parsing proves brittle, fall back to manual transcription of the table contents into a JSON sidecar; the protocol opcode table is canonical from GML alone, the XLS hints just save naming time. |
[VERIFIED: 2026-05-03] `xlsx` (SheetJS) is the de facto Node XLS parser. CVE history exists (prototype pollution CVE-2023-30533 in <0.20.2; CVE-2024-22363 ReDoS); the 0.18.5 version on npm is the public-published "community" build. **For Phase 3 Node-only build-time tooling reading a local trusted file, the CVEs are not a meaningful threat surface** (no untrusted input, no server runtime). If concerned: convert XLS → CSV manually once (LibreOffice/Excel), commit the CSV alongside the XLS, and read CSV with stdlib. The CSV path is recommended over the xlsx dependency. **Decision left to planner — researcher recommends CSV-conversion-once over xlsx-dependency.**
### Lint scripts (per D-22)
`lint-protocol.mjs`, `lint-save-formats.mjs`, `lint-parity.mjs` — pure Node stdlib, no deps. Same shape as existing `lint-docs.mjs`, `lint-matrix.mjs`, `lint-adr.mjs` produced in Phase 2.
### Decision references (consumed by ADRs, not new deps)
| Reference | Version | Purpose | When to Use |
|-----------|---------|---------|-------------|
| Drizzle ORM | 0.45.2 | Schema definition syntax for `docs/extracted-server/schema.sql` + `tables.ts` (consumed by Phase 4) | [VERIFIED: `npm view drizzle-orm version` = 0.45.2 on 2026-05-03]. |
| Drizzle Kit | 0.31.10 | Migration generation | [VERIFIED: `npm view drizzle-kit version` = 0.31.10 on 2026-05-03]. Phase 4 owns runner; Phase 3 produces `0001_baseline.sql` artifact. |
| better-sqlite3 | 12.9.0 | Driver pin (locked by ADR 0002 rationale) | [VERIFIED: `npm view better-sqlite3 version` = 12.9.0 on 2026-05-03; matches STACK.md]. |
| Litestream | 0.3.13 (latest stable) | Continuous WAL replication to Tigris | [CITED: STACK.md + litestream.io]. v0.5.0 in beta as of 2026-05; v0.3.x is the production-recommended series for Fly.io's documented pattern. |
| Colyseus | 0.17.10 | Phase 4 server framework — informs SDOC-04 ADR ("does Colyseus mandate any specific persistence shape?") | [VERIFIED: `npm view colyseus version` = 0.17.10 on 2026-05-03]. Answer: no — Colyseus is room/state-sync layer; persistence is orthogonal. |
| Better-Auth | 1.6.9 | Auth framework — informs `accounts` table column shape | [CITED: STACK.md]. Better-Auth's Drizzle adapter generates `users`, `sessions`, `accounts`, `verification` tables; D-13's `accounts` table merges Better-Auth's `users` shape with REBNO domain fields. |
| argon2 | 0.44.0 | Password hashing | [CITED: STACK.md, OWASP 2026]. Drives `accounts.password_hash` column type (TEXT, ~95-byte argon2id encoded string). |
### Alternatives Considered
| Instead of | Could Use | Tradeoff |
|------------|-----------|----------|
| Bespoke `tools/protocol-doc` GML scanner | tree-sitter GML grammar | tree-sitter has no maintained 5.3a-era GML grammar (modern GameMaker syntax differs significantly — see decomp/wiki/06-gml-syntax-5x.md). Rolling a procedural line-scanner for the narrow `write*`/`read*`/`file_text_*`/`sendmessage`/`receivemessage` patterns is simpler and matches the procedural-trace-by-grep approach in PITFALLS A4. |
| Bespoke schema → Drizzle TS | Generate Drizzle from `save-formats.json` | Tempting (single source of truth) but the Drizzle schema is **not a port of the save-format grammar** per D-14 — it's a normalized SQL design. Generating from the grammar would couple the new schema to the legacy text shape and re-introduce `.bnu`-line-order semantics into TypeScript. Author the Drizzle schema by hand from the save-format grammar; cite the grammar fields in the schema's column comments. |
| Single `tools/doc` covering both protocol + save-format | Two separate tools | The two parsers share ~30% of plumbing (file walker, GML line tokenizer, citation emitter, deterministic JSON writer). Keep them as **two CLIs that import a shared `src/scanner/` lib** within their own tool dir. Path forward: extract shared scanner into a separate `tools/gml-scan/` package only if a third consumer appears; meanwhile, duplicate-then-refactor is fine. (Researcher leans toward two CLIs — separation of concerns reads better in `pnpm` script names and lets the lints run independently.) |
| Read XLS via xlsx | Read pre-converted CSV via stdlib | See §xlsx note above — recommended path is CSV conversion. |
**Installation (assuming CSV-conversion path for xlsx):**
```bash
# tools/protocol-doc/
pnpm add -D typescript@5.6.3 tsx@4.21.0 vitest@4.1.5 @types/node@25.6.0
# tools/save-format-doc/
pnpm add -D typescript@5.6.3 tsx@4.21.0 vitest@4.1.5 @types/node@25.6.0
# Or if xlsx kept:
# tools/protocol-doc/ adds: pnpm add xlsx@0.18.5
```
**Version verification:** `npm view drizzle-orm version` → 0.45.2; `npm view drizzle-kit version` → 0.31.10; `npm view better-sqlite3 version` → 12.9.0; `npm view colyseus version` → 0.17.10; `npm view xlsx version` → 0.18.5 (all 2026-05-03). All match STACK.md pins.
## Architecture Patterns
### System Architecture Diagram
```
┌──────────────────────────────────────────────────────────────────────┐
│ INPUTS (read-only) │
│ │
│ extracted/server-5-4/ legacy/open-source-release/ │
│ ├── scripts/*.gml (135) ├── ,ServerCommands.txt │
│ ├── objects/*/events/*.gml (58) ├── BN Online Message │
│ ├── settings.json │ ID's Table.xls │
│ └── MANIFEST.sha256 └── Ctrl+O Codes.txt │
│ │
│ legacy/servers/enlyzeam-current/ decomp/wiki/16-bno-bnb-notes.md │
│ ├── localList.txt (ERRATA target — D-08) │
│ ├── UserData/{HXB,Inv,MB_News}/ .planning/research/PITFALLS.md │
│ ├── MB_Log.bnb (A5 wording errata target — D-08) │
│ ├── Settings.bno, MSettings.bno docs/extracted-engine/ │
│ └── 39dll.dll (binary, untouched) (Phase 2 output for cross-link) │
└──────────────────────┬───────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────┐
│ TOOLS (Phase 3 build-time, offline) │
│ │
│ tools/protocol-doc/ tools/save-format-doc/ tools/asset- │
│ ├── cli.ts ├── cli.ts catalog/ │
│ │ catalog ▸▸▸ protocol.* │ catalog ▸▸▸ save- │ (re-invoke) │
│ │ regen-autogen ▸▸▸ MD │ formats.* │ catalog ▸▸▸ │
│ │ verify ▸▸▸ exit 0/1/2 │ regen-autogen ▸▸▸ MD │ server-tree │
│ ├── src/scanner/ (parses │ verify ▸▸▸ exit │ │
│ │ .gml line-by-line for │ 0/1/2 │ │
│ │ write*/read*/sendmsg) ├── src/scanner/ (file_ │ │
│ ├── src/emit/ (JSON + TS │ text_* tracer) │ │
│ │ + MD writers) ├── src/emit/ │ │
│ └── tests/ └── tests/ │ │
└────────┬──────────────────────┬──────────────────────┬───────────────┘
│ │ │
▼ ▼ ▼
┌──────────────────────────────────────────────────────────────────────┐
│ OUTPUTS (Phase 3 deliverables — committed to git) │
│ │
│ docs/extracted-server/ docs/adr/ │
│ ├── README.md ├── 0002-persistence- │
│ ├── account-auth.md (mvp tags) │ layer.md │
│ ├── world-simulation.md (mvp tags) └── 0003-canonical- │
│ ├── room-management.md (mvp tags) snapshot.md │
│ ├── chat.md (mvp tags) │
│ ├── persistence.md (mvp tags) tools/protocol-doc/ │
│ ├── packet-protocol.md (autogen) output/protocol.ts │
│ ├── admin-anti-port.md (REJECTED) tools/save-format-doc/ │
│ ├── client-server-bridge.md (cross-link output/save-formats.ts │
│ │ to docs/extracted-engine/ │
│ │ client-networking.md) decomp/wiki/16-bno- │
│ ├── message-board.md (mvp tags) bnb-notes.md (ERRATA) │
│ ├── unknown-actions-status.md .planning/research/ │
│ ├── protocol.json ◀─── canonical PITFALLS.md (A5 wording│
│ ├── protocol.md ◀─── autogen-rendered errata) │
│ ├── save-formats.json ◀── canonical │
│ ├── save-formats.md ◀── autogen-rendered │
│ ├── parity-checklist.json ◀── canonical │
│ ├── parity-checklist.md ◀── autogen-rendered │
│ ├── schema.sql ◀─── Drizzle baseline │
│ ├── tables.ts ◀─── Drizzle TS defs │
│ ├── 0001_baseline.sql ◀── drizzle-kit generate output │
│ └── asset-catalog/ ◀─── re-invoked tools/asset-catalog │
└──────────────────────────────────────────────────────────────────────┘
│
▼ (Phase 4 consumes)
┌──────────────────────────────────────────────────────────────────────┐
│ DOWNSTREAM CONSUMERS │
│ │
│ Phase 4: packages/protocol/ reads protocol.json, protocol.ts │
│ packages/game-logic/ derives from world-simulation.md │
│ apps/server/ reads schema.sql, tables.ts (SRV-01..03)│
│ apps/server/migrate/ reads save-formats.ts (SRV-10/11) │
│ apps/server/ reads admin-anti-port.md (SRV-12 stub) │
│ Phase 5: RESTORE.md includes legacy_credentials_staging │
│ read-once-then-purge step │
│ Phase 6: client/ reads protocol.json mvp:true rows │
│ Phase 7: PAR-07 admin UI reads admin-anti-port.md command shapes │
│ PAR-08 closes against parity-checklist.json │
└──────────────────────────────────────────────────────────────────────┘
```
### Recommended Project Structure
```
tools/
├── extract-gmd/ # Phase 1 (existing, unchanged)
├── asset-catalog/ # Phase 2 (existing, re-invoked against server tree)
│ └── (no source changes; only new pnpm script `catalog:server`)
├── protocol-doc/ # Phase 3 NEW
│ ├── package.json
│ ├── cli.ts # Subcommands: catalog | regen-autogen | verify
│ ├── tsconfig.json
│ ├── vitest.config.ts
│ ├── src/
│ │ ├── scanner/ # GML line tokenizer for write*/read*/sendmessage/receivemessage
│ │ │ ├── opcode-trace.ts # Trace backward from sendmessage to clearbuffer
│ │ │ ├── tokens.ts # Pattern matchers
│ │ │ └── citations.ts # Build {script, line, snippet} for each match
│ │ ├── derive/ # Cross-reference XLS hints (optional), mvp tagging
│ │ ├── emit/ # protocol.json, protocol.ts, protocol.md autogen
│ │ └── types.ts # OpcodeRow, ProtocolTable shapes
│ ├── output/
│ │ └── protocol.ts # Emitted artifact (Phase 4 imports)
│ ├── data/
│ │ └── opcode-names.csv # If XLS converted to CSV (recommended path)
│ └── tests/
│ ├── unit/ # Per-pattern matchers
│ └── integration/ # Real extracted/server-5-4/ scripts
└── save-format-doc/ # Phase 3 NEW
├── package.json
├── cli.ts # Subcommands: catalog | regen-autogen | verify
├── tsconfig.json
├── vitest.config.ts
├── src/
│ ├── scanner/ # GML line tokenizer for file_text_open_*/file_text_read_*/file_text_write_*
│ │ ├── format-trace.ts # Trace forward from file_text_open_* through close
│ │ └── citations.ts
│ ├── emit/ # save-formats.json, save-formats.ts, save-formats.md
│ └── types.ts # FormatGrammar, SectionedField, FlatField shapes
├── output/
│ └── save-formats.ts # Emitted artifact (Phase 4 SRV-10/11 imports)
└── tests/
docs/
├── extracted-engine/ # Phase 2 (existing)
├── extracted-server/ # Phase 3 NEW
│ ├── README.md # Task-keyed jump table (parallel Phase 2)
│ ├── account-auth.md
│ ├── world-simulation.md
│ ├── room-management.md
│ ├── chat.md
│ ├── persistence.md
│ ├── packet-protocol.md
│ ├── admin-anti-port.md
│ ├── client-server-bridge.md
│ ├── message-board.md
│ ├── unknown-actions-status.md
│ ├── SUBSYSTEM-MAP.json # Mirror of Phase 2 SUBSYSTEM-MAP.json
│ ├── protocol.json # Canonical opcode table
│ ├── protocol.md # Rendered (AUTOGEN blocks)
│ ├── save-formats.json # Canonical save schemas
│ ├── save-formats.md # Rendered (AUTOGEN blocks)
│ ├── parity-checklist.json # Closed checklist
│ ├── parity-checklist.md # Rendered (AUTOGEN blocks)
│ ├── schema.sql # Drizzle DDL baseline
│ ├── tables.ts # Drizzle TS definitions
│ ├── 0001_baseline.sql # drizzle-kit-generated migration
│ └── asset-catalog/
│ ├── index.json
│ └── index.md
└── adr/
├── 0001-client-engine.md # Phase 2 (existing)
├── 0002-persistence-layer.md # Phase 3 NEW
└── 0003-canonical-snapshot.md # Phase 3 NEW
scripts/ # repo-level (existing)
├── lint-docs.mjs # Phase 2
├── lint-matrix.mjs # Phase 2
├── lint-adr.mjs # Phase 2
├── lint-protocol.mjs # Phase 3 NEW (per D-22)
├── lint-save-formats.mjs # Phase 3 NEW
└── lint-parity.mjs # Phase 3 NEW
decomp/wiki/
└── 16-bno-bnb-notes.md # ERRATA: file_bin_* → file_text_* (D-08)
.planning/research/
└── PITFALLS.md # A5 wording errata (D-08)
```
### Pattern 1: 39dll Opcode Extraction Procedure (PITFALLS A4 codified)
**What:** Walk every `sendmessage(...)` call in extracted server GML, trace backward to enumerate the `clearbuffer; writebyte(...); writeX(...); ...` sequence that precedes it. That sequence **is** the wire format.
**When to use:** Sole acceptable methodology for SDOC-02. Wireshark/packet capture reverse-engineering is forbidden as primary source (per A4).
**Implementation sketch:**
```typescript
// tools/protocol-doc/src/scanner/opcode-trace.ts
// Source: PITFALLS A4, decomp/wiki/08-39dll-networking.md, verified against
// extracted/server-5-4/scripts/0359-server_receive.gml lines 79-119
interface OpcodeTrace {
script: string; // e.g., "0359-server_receive.gml"
line: number; // sendmessage line number
direction: 'c2s' | 's2c';
opcodeByte: number; // First writebyte after the most recent clearbuffer
fields: Array<{ // Subsequent write* calls in order
op: 'writebyte' | 'writeshort' | 'writeushort' | 'writeint' | 'writeuint' |
'writefloat' | 'writedouble' | 'writechars' | 'writestring';
expression: string; // The argument as written (e.g., "global.p_x[upid]")
line: number;
}>;
}
// Algorithm:
// 1. Read each .gml file line-by-line.
// 2. When we see `sendmessage(...)`, walk backward up to the most recent
// `clearbuffer()` call (or function entry), collecting every `writeX(...)`
// call in source order.
// 3. The first writebyte after clearbuffer is the opcode byte.
// 4. Subsequent writes (until next sendmessage or clearbuffer) are the fields.
// 5. Direction: c2s if the script handles an inbound message (server_receive.gml
// case branches), s2c if the script *constructs* a message to send (anywhere
// a clearbuffer/writebyte/sendmessage triple appears).
//
// Edge case: A single GML script may contain multiple clearbuffer/sendmessage
// blocks (server_receive.gml has dozens). Each block is one opcode trace.
//
// Edge case: sendmessage may be called with a buffer ID argument (per
// 0023-sendmessage.gml comments — argument3 selects a non-default buffer).
// All inspection of extracted/server-5-4/ shows only the default-buffer
// pattern is used (clearbuffer ▸ writeX ▸ sendmessage). Document the
// non-default-buffer call signature as POSSIBLE-FUTURE in the scanner;
// not seen in this codebase.
```
**Branch-coverage caveat (PITFALLS A4 warning sign 3):** A single message-handler `case` branch may contain conditional `writeX` calls (`if (foo) writebyte(1);`). The scanner emits one OpcodeTrace per code path through the handler, and the schema may need to model variant payloads. **Verified pattern in 0359-server_receive.gml case 5 (login):** the success path writes `(int playerudpport, byte dabypass, int latestclient, string servermsg, byte iv_totalcats, ...)` while the failure path writes `(int -1)` or `(int -2)`. SDOC-02 schema MUST model this as a discriminated union keyed on the first int sign (`>=0` = success, `-1` = bad-auth, `-2` = maintenance-mode).
### Pattern 2: `.bno`/`.bnb`/`.bnu` Save-Format Extraction Procedure (PITFALLS A5 codified, file_text_* corrected)
**What:** Walk every `file_text_open_write(...)` and `file_text_open_read(...)` call site, trace forward through the surrounding scope to enumerate the `file_text_write_string` / `file_text_write_real` / `file_text_writeln` / `file_text_read_string` / `file_text_read_real` / `file_text_readln` sequence until `file_text_close(...)`. That sequence **is** the file's grammar.
**When to use:** Sole acceptable methodology for SDOC-03. Hex-inspecting `.bnu` files to "guess" the format is forbidden (per A5).
**Implementation sketch:**
```typescript
// tools/save-format-doc/src/scanner/format-trace.ts
// Source: PITFALLS A5 (corrected per D-08), verified against
// extracted/server-5-4/scripts/0367-users_restore.gml,
// 0376-uinv_backup.gml, 0377-uinv_restore.gml, 0365-mb_backup.gml
interface FormatTrace {
loadScript: string; // e.g., "0367-users_restore.gml"
saveScript?: string; // e.g., "0377-uinv_backup.gml" — paired by filename pattern
filenamePattern: string; // "User_DBUpdated.bnu", "UserData\Inv\Inventory_.bnu"
fields: Array<
| { kind: 'flat', name: string, type: 'string' | 'real',
op: 'read' | 'write', terminator: 'readln' | 'writeln' | null, line: number }
| { kind: 'section', marker: '@TOPIC' | '@REPLY' | string, lines: number[] }
| { kind: 'loop', counter: string, body: Array<...>, lines: number[] }
>;
}
// Algorithm:
// 1. For each .gml file, scan for file_text_open_(write|read) calls.
// 2. From that line, scan forward in source order (NOT control-flow order —
// GameMaker 5.3a is procedural and the script body executes top-to-bottom
// within a function; conditional branches in save scripts are rare and
// surface via if (file_exists(...)) wrappers per 0367).
// 3. Each file_text_(read|write)_(string|real) call is one field. Capture:
// - field name (best-effort: the variable being assigned/read on the LHS)
// - field type (string vs real per the function name)
// - terminator (presence of file_text_(read|write)ln after the field)
// 4. Loop constructs (for(i=0; i.bnu` | `0383-uhxb_restore.gml` | `0384-uhxb_backup.gml` | flat (count + N strings) |
| `UserData\Inv\Inventory_.bnu` | `0377-uinv_restore.gml` | `0376-uinv_backup.gml` | flat-nested (dabypass, totalcats, [per-cat: count, [per-item: string]]) |
| `UserData\Areas\Areas_.bnu` | `0368-uarea_restore.gml` | `0369-uarea_backup.gml` | flat (totalareas reals) |
| `UserData\MB_News\News_.bnu` | `0372-unews_restore.gml` | `0371-unews_backup.gml` | flat-loop (per-board newmsg flag) |
| `User_DBUpdated.bnu` (HXB) | `0383-uhxb_restore.gml` (per-user) | — | (covered above) |
(Researcher counted ~16 file-IO scripts total via `grep -l "file_text_open" extracted/server-5-4/scripts/`. Planner should inventory all in plan-step-1 before committing scanner shape.)
### Pattern 3: Subsystem-MD Authoring (mirrors Phase 2 D-09/D-10/D-11)
**What:** Hand-author one MD per subsystem with delimited autogen blocks for cross-reference tables. Authoring split is **prose by humans, indices by tool**.
**Why:** Reuses proven Phase 2 pattern (commit `ad1e0c7` shipped 11 subsystem MDs at exactly the right depth). Avoids re-litigating depth/depth-discipline.
**Example (from Phase 2's `docs/extracted-engine/rendering.md`):**
```markdown
## Sprite drawing subsystem
The original engine renders sprites via `0042-player`'s `Draw` event...
| Script | Lines | Purpose |
|--------|-------|---------|
| 0042-player Draw | 47 | Draw player sprite |
```
**For Phase 3 server, the autogen blocks are:**
- `roster` — script + object-event roster per subsystem (same as Phase 2)
- `opcode-list` — opcodes attributed to this subsystem (e.g., `chat.md` lists opcodes 4, 12, 20)
- `save-format-list` — formats attributed to this subsystem (e.g., `persistence.md` lists every format; `message-board.md` lists `MB_Log.bnb`)
- `parity-counters` — count of parity rows in each disposition referencing this subsystem
The same `tools/asset-catalog regen-autogen` invocation that refreshes Phase 2 blocks can be **extended** with new block kinds, OR a separate `tools/protocol-doc regen-autogen` and `tools/save-format-doc regen-autogen` can each refresh their domain's blocks. **Recommendation:** keep them separate per-tool (each tool owns its own block kinds); chain them in `pnpm regen:server` script.
### Pattern 4: ADR Format (mirrors `docs/adr/0001-client-engine.md`)
```markdown
# ADR 000X:
## Status
Locked at end of Phase 3 (2026-05-XX). Cannot be flipped without reopening Phase 3.
## Context
<2-4 paragraphs grounding in research, citing PITFALLS, STACK.md, ARCHITECTURE.md>
## Decision
**<One-sentence directive>.**
<Bullet-list rationale tied to specific evidence — for ADR 0002 cite documented save-format grammar showing single-writer / append-mostly access patterns; for ADR 0003 cite the dataset-size delta>
## Consequences
### Positive
- ...
### Negative
- ...
### Neutral
- ...
## Related ADRs
- ADR 0001 — Client Engine (Phaser 3.90 locked)
## References
- `<file:line citations>`
```
### Pattern 5: Parity-Checklist Construction Methodology (D-18, D-19)
**What:** Per-feature rows in `parity-checklist.json`, each disposition `in-phase-6` | `in-phase-7` | `deferred-stage-8` | `rejected-with-reason`. Locked at phase end.
**How features are enumerated (deterministic):**
1. Walk every CDOC-01 subsystem MD (Phase 2, client-side) — every `## ` heading is a candidate feature row.
2. Walk every SDOC-01 subsystem MD (Phase 3, server-side) — every `## ` heading is a candidate feature row.
3. Walk every opcode in `protocol.json` — each opcode either ATTRIBUTES to a feature row already enumerated above (opcode 3 ATTRIBUTES to "movement") or surfaces a new feature row (opcode 23 — `summon-request` — may need its own row).
4. Walk every save-field group in `save-formats.json` — same attribution pattern.
5. Walk `,ServerCommands.txt` admin keybinds + `Ctrl+O Codes.txt` snippets — each becomes an admin-anti-port row, all marked `rejected-with-reason` per CLAUDE.md hard rule #3.
**Disposition rules:**
- `in-phase-6` — feature on the CLI-08 MVP slice (movement, chat-public, login, room-join, room-leave, heartbeat). Researcher count: ~7 features.
- `in-phase-7` — feature is full-parity scope (whispers, message board, inventory, hexbridges, friends list, presence, account recovery, account settings). Researcher count: ~25–35 features.
- `deferred-stage-8` — feature is QoL or low-leverage (e.g., legacy alt-login detection — original GML had IP-based duplicate-account-detection that hard-codes `Saber Mage` and `Vance Serori` exemptions; modern auth handles this via Better-Auth session model).
- `rejected-with-reason` — feature is incompatible with the new architecture per D-19. Examples expected: client-trusted positions (B1), all clipboard-RCE admin commands, `take-a-break.exe` execution, `,run` shell-out commands, plaintext password storage as canonical.
### Anti-Patterns to Avoid
- **Wireshark-first protocol RE.** Per A4. Always GML-grep first; captures are validation only.
- **Hex-editor-first save-format RE.** Per A5 (corrected). Always GML-grep `file_text_*` first.
- **Per-record cross-snapshot merge.** Rejected per D-02. enlyzeam-current is canonical whole.
- **Faithful port of plaintext password storage.** Per CLAUDE.md hard rule #2. Plaintext stays in `legacy_credentials_staging`, never in `accounts.password_hash`.
- **Faithful port of Ctrl+E clipboard-RCE.** Per CLAUDE.md hard rule #3. Catalogued, REJECTED, replaced by Phase 7 web UI.
- **Premature workspace conversion.** `tools/protocol-doc/output/protocol.ts` is an emitted artifact, not a `packages/protocol` shim. Phase 4 owns the workspace conversion.
- **Drizzle schema as faithful port of `.bnu` line-order.** Per D-14. `.bnu` is read-once-then-discarded by SRV-10; new schema is normalized SQL.
- **Generating the Drizzle schema from `save-formats.json`.** Tempting but couples normalized SQL to legacy text shape. Author Drizzle by hand; cite save-format fields in column comments only.
- **Letting parity-checklist drift unbounded.** Per D-22 lint, every row must have non-empty `originating_gml` AND non-null `disposition`. Lint-first; lock-second.
- **Skipping audit_log seed.** Phase 3 schema MUST include `audit_log` even though no Phase 3..6 code writes to it (per Specifics). Phase 7 PAR-07 writes here. Avoids Phase 7 schema migration.
## Don't Hand-Roll
| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| GML line tokenization for `write*`/`read*`/`file_text_*` traces | Full GML AST parser | Per-pattern line scanner with backward/forward trace within a single source file | Procedural call-order is local-within-script; full AST overkill. tree-sitter-gml is for modern GML, not 5.3a (per `decomp/wiki/06-gml-syntax-5x.md`). |
| ADR template | Bespoke ADR format | Mirror `docs/adr/0001-client-engine.md` | First ADR is the locked template (Status / Context / Decision / Consequences). |
| Subsystem-MD autogen-block engine | New autogen format | Reuse Phase 2's HTML-comment-delimited blocks (`<!-- AUTOGEN-START name -->...<!-- AUTOGEN-END name -->`) | Already proven; `lint-docs.mjs` can be extended to cover the new MDs by changing one config constant. |
| Drizzle schema authoring | Hand-write SQL DDL then transcribe to Drizzle | Drizzle-first (`tables.ts`); `drizzle-kit generate` produces `schema.sql` + `0001_baseline.sql` | Drizzle is the source of truth for Phase 4; running `drizzle-kit` once at Phase 3 close emits both artifacts. |
| Argon2 hash storage column type | Custom binary blob format | TEXT (Better-Auth's adapter conventions) | Argon2 encoded format is `$argon2id$v=19$m=...$t=...$p=...$<salt>$<hash>` — a self-describing ASCII string ~95 bytes. Better-Auth + drizzle-adapter expects TEXT. |
| SQLite WAL replication | Custom rsync-of-WAL script | Litestream | Fly.io explicitly endorses pattern; CONCERNS-grade reliability. |
| Legacy-credential migration logic | Phase 3 implementation | Document staging schema (D-04); Phase 4 SRV-10/11 implements | Per D-04 + CLAUDE.md hard rule #6. Phase 3 is documentation only. |
| Parity-checklist linting | Per-row hand-review | `lint-parity.mjs` enforces structural invariants (non-empty `originating_gml`, non-null `disposition`); planner manually reviews semantic rows | Lint catches drops; humans catch dispositions. |
| XLS reading | xlsx@0.18.5 dependency | Convert XLS → CSV manually once, commit CSV alongside XLS, read CSV with stdlib | Avoids xlsx CVE chain (CVE-2023-30533, CVE-2024-22363) for build-time tool. CSV is two columns (opcode_byte, legacy_name) — trivial to parse. |
**Key insight:** Phase 3 is documentation, not implementation. The schemas it produces are the contract Phase 4 implements. **Build the simplest tools that emit the required artifacts deterministically; don't pre-build features Phase 4 will need.** The output/protocol.ts emitted from `tools/protocol-doc` is the one exception — it's a deliverable artifact (Phase 4 imports from it directly per D-05) so its TS shape needs care, but it's still emitted, not a workspace package.
## Runtime State Inventory
> Phase 3 is a documentation-only phase with no runtime systems and no rename/refactor. This section is included for completeness but most rows are "None — pure new authoring."
| Category | Items Found | Action Required |
|----------|-------------|------------------|
| Stored data | None — Phase 3 produces docs/JSON/TS schemas; consumes existing extracted/server-5-4/ tree (Phase 1 output) and legacy/servers/enlyzeam-current/ (read-only). No databases yet exist (Phase 4 owns). | None |
| Live service config | None — no deployed services. The legacy server runs on no host today (per CONCERNS.md "39dll-based traffic ... no production server runs today"). | None |
| OS-registered state | None — no OS-level registrations created or consumed. The Phase 1 manifest at `extracted/server-5-4/MANIFEST.sha256` is repo-internal, not OS-registered. | None |
| Secrets/env vars | **`localList.txt` plaintext passwords (~298 accounts)** in `legacy/servers/enlyzeam-current/localList.txt` are PII per CONCERNS.md "Plaintext player credentials checked into the archive (CRITICAL)". Phase 3 documents the **legacy_credentials_staging** schema (D-04) for Phase 4 ingest; the plaintext rows themselves are NEVER copied into the production database — they remain in `legacy/servers/enlyzeam-current/` (which itself is `.gitignored` per CLAUDE.md hard rule #8 "Repo stays private through Phase 7"). | Verify `.gitignore` still excludes `legacy/`; document in `docs/extracted-server/persistence.md` and ADR 0003 the exact handling pipeline (`localList.txt` → `legacy_credentials_staging` → first-login argon2id rehash → row purged). |
| Build artifacts | The Phase 1 `extracted/server-5-4/MANIFEST.sha256` is the Phase-1 reproducibility artifact; Phase 3 must NOT regenerate or modify it. The Phase 2 `docs/extracted-engine/asset-catalog/index.{json,md}` will gain a sibling `docs/extracted-server/asset-catalog/index.{json,md}` per D-17. | None — additive only. |
**Wiki errata is the one rename-like operation.** `decomp/wiki/16-bno-bnb-notes.md` and `.planning/research/PITFALLS.md` (A5 wording) must change `file_bin_*` → `file_text_*`. This is a **pure text edit** with no runtime state implications. Commit BEFORE generating SDOC-03 schemas (per D-08).
## Common Pitfalls
### Pitfall 1: Trusting `BN Online Message ID's Table.xls` over extracted GML
**What goes wrong:** XLS hints contradict GML call order on a specific opcode. Engineer trusts XLS (it's a proper-looking spreadsheet); ships protocol.json with wrong field order; Phase 4 server desyncs on first packet.
**Why it happens:** XLS pre-dates Master 5-4 in places; opcodes may have been renumbered or reshaped without updating the table. Per D-07: GML is canonical truth on conflict.
**How to avoid:**
- `lint-protocol.mjs` enforces: every opcode row in `protocol.json` has a `gml_origin[]` array with at least one entry. XLS-only opcodes (no GML origin) lint-fail — they don't exist in the codebase.
- During cross-reference (D-07 step 3), conflicts are emitted as warnings to stderr but the opcode row uses GML data.
- Reviewer checks the conflict log per phase verify.
**Warning signs:** XLS lists opcode N with one set of fields; protocol.json shows different fields. Immediate red flag — go re-trace the GML.
### Pitfall 2: Branch-coverage gaps in opcode tracing
**What goes wrong:** Opcode handler has multiple `case`/`if` branches; scanner extracts only the first; protocol schema misses variant payloads (e.g., login success vs. login failure). Phase 4 server fails to handle the variant.
**Why it happens:** The procedural-trace heuristic (PITFALLS A4) is "walk backward from sendmessage". If two `if` branches each contain their own `clearbuffer; writeX; sendmessage` triple, the scanner emits two opcode rows — but if a single `clearbuffer` is followed by branched `writeX` calls (verified pattern: 0359-server_receive.gml case 5 login response), the scanner sees ONE opcode row with what looks like ALL the writes — and may include writes that only fire on success, missing the failure-path schema entirely.
**How to avoid:**
- Scanner emits a `branches` field in OpcodeTrace when the trace path crosses an `if`/`else`/`switch` boundary between `clearbuffer` and `sendmessage`. Schema downstream models these as discriminated unions or marks as MANUAL-REVIEW.
- For login-response-style cases: SDOC-02 schema documents the variant explicitly with a discriminator (`first_int >= 0` vs `first_int == -1` vs `first_int == -2`).
- `lint-protocol.mjs` flags any opcode whose trace crossed >0 branches without a corresponding `discriminator` field.
**Warning signs:** `protocol.json` has an opcode row with a long `fields[]` whose total byte count varies wildly across `sample_bytes` test vectors. Means variants weren't modeled.
### Pitfall 3: Dropping legacy `User_DB_Superweird.bnu` schema
**What goes wrong:** Phase 3 documents only `User_DBUpdated.bnu` (the current canonical user list, restored by `0367-users_restore.gml`), ignoring `User_DB_Superweird.bnu` referenced by `0379-users_restore_old.gml`. Phase 4 SRV-10 migration breaks on a snapshot that has Superweird-format users.
**Why it happens:** Older format. `0379-users_restore_old.gml` exists; `User_DB_Superweird.bnu` exists in `legacy/servers/enlyzeam-archive/`. Engineer reads the canonical-only loader and concludes the older format is dead.
**How to avoid:**
- SDOC-03 schema includes both formats with `format_version` discriminator. Mark Superweird as ARCHIVED-GRAMMAR.
- Per D-02, canonical seed is enlyzeam-current. Verify whether enlyzeam-current contains any Superweird-format files; if not, document in ADR 0003 and parity-checklist row "legacy-superweird-import" → `deferred-stage-8` (one-off recovery if a returning player surfaces).
**Warning signs:** SRV-10 importer fails on a specific account in archive snapshot during Phase 4 dogfood; checking the file shows it's `User_DB_Superweird.bnu` shape.
### Pitfall 4: Treating message-board state as ephemeral
**What goes wrong:** Phase 4 SRV-03 ships MVP without `message_board_topics` / `message_board_replies` tables. Phase 7 has to retrofit the schema, write a migration, deal with `MB_Log.bnb` parsing six months later instead of one.
**Why it happens:** Message board isn't on the CLI-08 hard milestone. Tempting to scope it out of Phase 3.
**How to avoid:**
- Per D-13, Phase 3 schema MUST include `message_board_topics` + `message_board_replies` tables. They start empty in Phase 4; PAR-04 + PAR-06 fill them in Phase 7. **No schema migration in Phase 7.**
- Same logic for `audit_log` — seeded empty for PAR-07.
- Same logic for `inventory_items`, `characters` — seeded empty until SRV-10 first-login migration runs.
**Warning signs:** Plan-checker flags `schema.sql` for missing tables enumerated in D-13.
### Pitfall 5: Drift between `protocol.json` (canonical) and `protocol.md` (rendered) / `protocol.ts` (typed)
**What goes wrong:** Engineer hand-edits `protocol.md` to add a comment; `protocol.json` doesn't get the change; Phase 4 imports `protocol.ts` (re-emitted from JSON on regen) and the comment is lost; meanwhile the markdown is "right" and someone trusts it.
**Why it happens:** Three artifacts, one canonical source. Same drift problem Phase 2 D-05 / D-09 / D-11 solved with autogen blocks + `lint-docs.mjs`. Phase 3 must replicate the discipline.
**How to avoid:**
- `tools/protocol-doc regen-autogen` regenerates `protocol.md` and `protocol.ts` from `protocol.json` deterministically.
- `tools/protocol-doc verify` exits non-zero if regen would produce different bytes than what's on disk (round-trip check).
- `lint-protocol.mjs` runs `verify` in CI.
- Same shape for `tools/save-format-doc` and parity-checklist.
- **Hand-edit protocol.json only**, never the rendered/typed outputs.
**Warning signs:** `pnpm verify` (or `tools/protocol-doc verify`) fails after a manual edit.
### Pitfall 6: GameMaker 5.3a string encoding ambiguity in legacy data
**What goes wrong:** SRV-10 importer ingests `localList.txt`; usernames containing characters above 0x7F (some accounts had non-ASCII handles per CONCERNS.md spot-check) decode wrong; account import fails or assigns wrong username.
**Why it happens:** GameMaker 5.3a's `file_text_read_string` reads bytes-until-delimiter and treats them as the host's default ANSI codepage (typically Windows-1252). The original `localList.txt` is likely Windows-1252-encoded, not UTF-8.
**How to avoid:**
- Document encoding explicitly in `save-formats.json`: every flat string field carries `encoding: 'windows-1252'` (verified during scanner pass — examine bytes of localList.txt in legacy/servers/enlyzeam-current/ and confirm).
- SRV-10 importer in Phase 4 uses `iconv-lite` (or Node's `Buffer.toString('latin1')` if Windows-1252-vs-Latin1 differences don't matter) to convert to UTF-8 on import.
- Drizzle `accounts.username` column is TEXT (UTF-8 in SQLite by default).
**Warning signs:** Spot-check a few legacy usernames; if any contain bytes >0x7F, encoding handling matters.
### Pitfall 7: Older-`.gmd` extraction creep (per D-03 lock)
**What goes wrong:** During Phase 3, an engineer hits an opcode that doesn't fully resolve in Master 5-4 (e.g., references a script that may have been renamed); decides to extract Master 4.gmd or Master 5-3.exe to compare; cascade triggers — now Phase 1 effectively re-opens.
**Why it happens:** Curiosity + the older `.gmd`/`.exe` files are right there in `legacy/servers/enlyzeam-archive/`.
**How to avoid:**
- D-03 lock: Master 5-4 is canonical. Older extractions are out of scope.
- If a specific trace fails or a script reference looks broken, document the deviation in the relevant subsystem MD with an `<!-- UNRESOLVED -->` HTML comment and move on. Don't extract.
- Lint-protocol enforces: every opcode has a `gml_origin` from `extracted/server-5-4/`. No path under `legacy/servers/*` allowed.
**Warning signs:** Plan task starts proposing "extend tools/extract-gmd to handle 5-3.exe" — STOP, route to discussion.
## Code Examples
Verified patterns from `extracted/server-5-4/scripts/` (ground truth, not training data):
### Example 1: Opcode trace from server_receive.gml — login response (success path)
```gml
// Source: extracted/server-5-4/scripts/0359-server_receive.gml (case 5, lines 198-237)
clearbuffer();
writebyte(8); // s2c opcode 8 = login-response
if(!copy && suclogin >= 0 && !global.maintmode)
{
global.p_uid[pid] = suclogin;
global.p_name[pid] = global.u_name[suclogin];
global.u_pid[suclogin] = pid;
writeint(global.playerudpport); // success: int udp_port
writebyte(global.dabypass[global.p_uid[pid]]); // byte da_bypass
writeint(global.latestclient); // int latest_client_version
writestring(global.servermsg); // string server_msg
writebyte(global.iv_totalcats); // byte inventory_total_cats
// ... per-category writes, per-item writes (loop)
writeint(global.hxbridge[global.p_uid[pid],0]); // int hexbridge_count
// ... per-hexbridge writestring (loop)
}
else
{
if(global.maintmode) writeint(-2); // maintenance: int -2
else writeint(-1); // bad-auth: int -1
sendmessage(global.p_tcpsocket[pid]);
uninit_user(pid);
break;
}
if(global.p_online[pid]) sendmessage(global.p_tcpsocket[pid]);
```
**Schema row in `protocol.json` (target shape):**
```json
{
"opcode_byte": 8,
"direction": "s2c",
"mvp": true,
"name": "login-response",
"discriminator": { "field": "first_int", "values": { ">=0": "success", "-1": "bad-auth", "-2": "maintenance" } },
"variants": {
"success": {
"fields": [
{ "name": "udp_port", "gml_type": "int", "ts_type": "number", "byte_size": 4 },
{ "name": "da_bypass", "gml_type": "byte", "ts_type": "number", "byte_size": 1 },
{ "name": "latest_client_version", "gml_type": "int", "ts_type": "number", "byte_size": 4 },
{ "name": "server_msg", "gml_type": "string", "ts_type": "string", "byte_size": "variable" },
{ "name": "inventory", "gml_type": "nested-loop", "ts_type": "Inventory", "byte_size": "variable" },
{ "name": "hexbridges", "gml_type": "string-list", "ts_type": "string[]", "byte_size": "variable" }
]
},
"bad-auth": { "fields": [ { "name": "code", "gml_type": "int", "ts_type": "-1", "byte_size": 4 } ] },
"maintenance": { "fields": [ { "name": "code", "gml_type": "int", "ts_type": "-2", "byte_size": 4 } ] }
},
"gml_origin": [
{ "script": "0359-server_receive.gml", "line": 199, "snippet": "clearbuffer(); writebyte(8); ..." }
],
"sample_bytes": "08 00000000 00 00000032 00 ..."
}
```
### Example 2: Save-format trace from mb_backup.gml — sectioned grammar
```gml
// Source: extracted/server-5-4/scripts/0365-mb_backup.gml (verbatim, 35 lines)
logfile = file_text_open_write("MB_Log.bnb");
// Write all board info (flat list of strings):
for(i = 0; i < global.mb_total[0]; i += 1)
{
file_text_write_string(logfile, global.mb_board[i]);
file_text_writeln(logfile);
}
// Section marker:
file_text_write_string(logfile, "@TOPIC");
file_text_writeln(logfile);
// Topics (per-topic: title, reply_count, board_id):
for(i = 0; i < global.mb_total[1]; i += 1)
{
file_text_write_string(logfile, global.mb_topic[i,0]); // title
file_text_writeln(logfile);
file_text_write_real(logfile, global.mb_topic[i,1]); // reply count
file_text_writeln(logfile);
file_text_write_real(logfile, global.mb_topic[i,2]); // board id
file_text_writeln(logfile);
}
// Section marker:
file_text_write_string(logfile, "@REPLY");
file_text_writeln(logfile);
// Replies (per-topic, per-reply: body):
for(i = 0; i < global.mb_total[1]; i += 1)
{
for(j = 0; j < global.mb_topic[i,1]; j += 1)
{
file_text_write_string(logfile, global.mb_reply[i,j]);
file_text_writeln(logfile);
}
}
file_text_close(logfile);
```
**Schema row in `save-formats.json` (target shape):**
```json
{
"extension": ".bnb",
"filename_pattern": "MB_Log.bnb",
"load_script": "0366-mb_restore.gml:1",
"save_script": "0365-mb_backup.gml:1",
"mvp": false,
"encoding": "windows-1252",
"grammar": [
{ "kind": "loop", "counter": "global.mb_total[0]", "field": { "kind": "flat", "name": "board_name", "type": "string", "terminator": "writeln" } },
{ "kind": "section_marker", "marker": "@TOPIC" },
{ "kind": "loop", "counter": "global.mb_total[1]", "fields": [
{ "kind": "flat", "name": "title", "type": "string", "terminator": "writeln" },
{ "kind": "flat", "name": "reply_count", "type": "real", "terminator": "writeln" },
{ "kind": "flat", "name": "board_id", "type": "real", "terminator": "writeln" }
] },
{ "kind": "section_marker", "marker": "@REPLY" },
{ "kind": "nested_loop", "outer_counter": "global.mb_total[1]", "inner_counter": "global.mb_topic[i,1]", "field": { "kind": "flat", "name": "reply_body", "type": "string", "terminator": "writeln" } }
],
"sample_records": [
{ "boards": ["General", "Tech"], "topics": [{ "title": "Welcome", "reply_count": 0, "board_id": 0 }], "replies": [] }
]
}
```
### Example 3: Drizzle schema target shape (D-13)
```typescript
// docs/extracted-server/tables.ts (target — Phase 3 produces, Phase 4 imports)
// Source: derived from save-formats.json grammars + Better-Auth conventions + D-13
import { sqliteTable, text, integer, blob, primaryKey } from 'drizzle-orm/sqlite-core';
export const accounts = sqliteTable('accounts', {
id: text('id').primaryKey(), // CUID2 / nanoid
username: text('username').notNull().unique(), // legacy migrated or new
passwordHash: text('password_hash').notNull(), // argon2id encoded; ~95 bytes
email: text('email'), // null for legacy-only accounts
role: text('role').notNull().default('player'), // 'player' | 'mod' | 'admin'
createdAt: integer('created_at', { mode: 'timestamp' }).notNull(),
lastLoginAt: integer('last_login_at', { mode: 'timestamp' }),
forceReset: integer('force_reset', { mode: 'boolean' }).notNull().default(false),
});
// SOURCE: save-formats.json {extension: ".bnu", load_script: "0367-users_restore.gml"} fields {u_name, u_pwd}
// + Better-Auth users table conventions (id/email/createdAt)
export const legacyCredentialsStaging = sqliteTable('legacy_credentials_staging', {
username: text('username').primaryKey(),
legacyHash: blob('legacy_hash').notNull(), // raw bytes from localList.txt or .bnu
algorithm: text('algorithm').notNull(), // 'plaintext' | 'bcrypt-weak' | ...
forceReset: integer('force_reset', { mode: 'boolean' }).notNull(),
legacySource: text('legacy_source').notNull(), // 'enlyzeam-current/localList.txt:LINE'
importedAt: integer('imported_at', { mode: 'timestamp' }).notNull(),
});
// SOURCE: D-04 staging-table spec; CONCERNS.md plaintext-creds finding;
// CRITICAL: read-once-then-purged by SRV-10/11; never copied verbatim into accounts.
export const characters = sqliteTable('characters', {
id: text('id').primaryKey(),
accountId: text('account_id').notNull().references(() => accounts.id),
roomId: integer('room_id').notNull(),
x: integer('x').notNull(),
y: integer('y').notNull(),
spriteId: integer('sprite_id').notNull(),
spriteVariants: text('sprite_variants', { mode: 'json' }), // 5-element string array per p_spr[uid,1..5]
createdAt: integer('created_at', { mode: 'timestamp' }).notNull(),
lastSavedAt: integer('last_saved_at', { mode: 'timestamp' }).notNull(),
});
// SOURCE: save-formats.json {extension: ".bnu", filename_pattern: "UserData/HXB/<uid>.bnu"};
// server_receive.gml case 5 fields p_x/p_y/p_spr/p_room
export const inventoryItems = sqliteTable('inventory_items', {
accountId: text('account_id').notNull().references(() => accounts.id),
category: integer('category').notNull(),
slot: integer('slot').notNull(),
itemId: text('item_id').notNull(), // string per uinv_get GML signature
}, (t) => ({
pk: primaryKey({ columns: [t.accountId, t.category, t.slot] }),
}));
// SOURCE: save-formats.json {filename_pattern: "UserData/Inv/Inventory_<uid>.bnu",
// load_script: "0377-uinv_restore.gml", save_script: "0376-uinv_backup.gml"}
export const messageBoardTopics = sqliteTable('message_board_topics', {
id: integer('id').primaryKey({ autoIncrement: true }),
boardId: integer('board_id').notNull(),
title: text('title').notNull(),
replyCount: integer('reply_count').notNull().default(0),
authorAccountId: text('author_account_id').references(() => accounts.id), // null for legacy unattributed
createdAt: integer('created_at', { mode: 'timestamp' }).notNull(),
});
// SOURCE: save-formats.json {extension: ".bnb", filename_pattern: "MB_Log.bnb"} @TOPIC section
export const messageBoardReplies = sqliteTable('message_board_replies', {
id: integer('id').primaryKey({ autoIncrement: true }),
topicId: integer('topic_id').notNull().references(() => messageBoardTopics.id),
body: text('body').notNull(),
authorAccountId: text('author_account_id').references(() => accounts.id),
createdAt: integer('created_at', { mode: 'timestamp' }).notNull(),
});
// SOURCE: save-formats.json {extension: ".bnb"} @REPLY section
export const auditLog = sqliteTable('audit_log', {
id: integer('id').primaryKey({ autoIncrement: true }),
actorAccountId: text('actor_account_id').notNull().references(() => accounts.id),
action: text('action').notNull(), // 'kick' | 'mute' | 'ban' | 'role-change' | ...
targetAccountId: text('target_account_id').references(() => accounts.id),
payloadJson: text('payload_json', { mode: 'json' }),
createdAt: integer('created_at', { mode: 'timestamp' }).notNull(),
});
// SOURCE: D-13 audit_log spec; Phase 7 PAR-07 admin actions write here
export const sessions = sqliteTable('sessions', {
id: text('id').primaryKey(),
accountId: text('account_id').notNull().references(() => accounts.id),
roomId: integer('room_id'),
connectedAt: integer('connected_at', { mode: 'timestamp' }).notNull(),
heartbeatAt: integer('heartbeat_at', { mode: 'timestamp' }).notNull(),
});
// SOURCE: D-13 sessions spec; Phase 4 SRV-06 reconnection grace
```
(This shape is illustrative — Phase 3 planner may adjust column names, indexes, and FK cascade rules within D-13's column list.)
### Example 4: ADR 0002 (Persistence) target shape
```markdown
# ADR 0002: Persistence Layer — SQLite + Litestream + Drizzle
## Status
Locked at end of Phase 3 (2026-05-XX). Cannot be flipped without reopening Phase 3.
Supersedes: nothing. Superseded by: nothing.
## Context
Phase 3 reverse-engineering of the legacy server (per `docs/extracted-server/persistence.md` and `save-formats.json`) reveals a data model that is:
- **Single-writer.** All `.bnu` and `.bnb` saves are written by the singleton `server` object on a deterministic schedule (per `0349-operations` Step event) or on player events (login, logout, save). No concurrent-write contention exists in the original.
- **Append-mostly.** `MB_Log.bnb` (39 KB) is rewrites-on-change but the access pattern is "load whole file, mutate, write whole file" — small-row, low-frequency.
- **Per-user denormalized.** `UserData/HXB/<uid>.bnu`, `UserData/Inv/Inventory_<uid>.bnu`, `UserData/Areas/Areas_<uid>.bnu` are per-user files — the legacy schema is already "one row per user, multiple files per category." No relational join requirement surfaced in any documented script.
- **Small.** 973 user files × ~1-3 KB each = ~3 MB total user state. `MB_Log.bnb` 39 KB. Total < 5 MB world state.
## Decision
**Use SQLite (`better-sqlite3` 12.9.0) + Drizzle ORM 0.45.2 + Litestream-to-Tigris for v1 persistence.**
Postgres is rejected for v1 with reason: ...
## Consequences
### Positive
- Synchronous DB access maps perfectly to Colyseus's single-tick game loop (per `STACK.md`).
- ...
### Negative
- ...
### Neutral
- ...
## Related ADRs
- ADR 0001 — Client Engine (Phaser 3.90 locked)
- ADR 0003 — Canonical Snapshot
## References
- `docs/extracted-server/save-formats.json` — `.bnu`/`.bnb`/`.bno` grammars
- `docs/extracted-server/schema.sql` — Drizzle baseline
- `.planning/research/STACK.md` §better-sqlite3 12.9.0
- `.planning/research/PITFALLS.md` §B7 (Persistence loss on Fly.io machine restart)
- OPS-03 (v2 requirement) — Postgres migration path preserved
```
## State of the Art
| Old Approach | Current Approach | When Changed | Impact |
|--------------|------------------|--------------|--------|
| Plaintext `localList.txt` username/password (legacy BNO) | argon2id via Better-Auth + `legacy_credentials_staging` (read-once-then-purge) | 2025+ (OWASP Argon2id #1) | Phase 3 documents the staging schema; Phase 4 implements migration. Plaintext NEVER copied verbatim into prod DB. |
| 39dll wire protocol (positional bytes, no schema, no version) | Schema-versioned binary frames via `@colyseus/schema` 4.0.21 + opcode envelope | Phase 4+ | Phase 3 produces canonical opcode table; Phase 4 implements typed schema + PROTOCOL_VERSION byte from packet 1. |
| `file_text_*` line-based `.bnu` save format | SQLite + Drizzle normalized schema | Phase 3 ADR 0002 | Original `.bnu` text format is read-once-then-discarded by SRV-10 import. |
| Ctrl+E clipboard-RCE admin model | Authenticated web admin UI (Phase 7 PAR-07) with TS intent shapes | Phase 3 admin-anti-port doc + Phase 7 implementation | Per CLAUDE.md hard rule #3. Phase 3 specifies command surface; never ports the RCE pattern. |
| Three drift'd server snapshots (enlyzeam-current, enlyzeam-archive, local-current) | Single canonical snapshot (enlyzeam-current) per ADR 0003 | Phase 3 ADR 0003 (D-02) | No per-record merge; archive + local-current rejected with reason. One-off recovery if returning player surfaces archive-only account. |
| `decomp/wiki/16-bno-bnb-notes.md` says `file_bin_*` | Wiki + PITFALLS A5 corrected to `file_text_*` (D-08 errata) | Phase 3 plan-step-1 | Verified ground truth in `0365-mb_backup.gml` + `0367-users_restore.gml`. |
| Lucia auth library | Better-Auth 1.6.9 | March 2025 (Lucia deprecated by maintainer) | Phase 4 SRV-09 implements; Phase 3 schema accommodates Better-Auth's `users` table conventions. |
| Postgres for game persistence (received-wisdom default) | SQLite + Litestream for <50 CCU stateful WS server | Fly.io 2024+ documented pattern | ADR 0002 records the rationale; OPS-03 v2 preserves migration path. |
**Deprecated/outdated:**
- **`file_bin_*` claim in wiki/16 + PITFALLS A5.** Per D-08, ground truth is `file_text_*`. Phase 3 first task = wiki errata commit.
- **Modern decompilers (UTMT, Altar.NET) for GM 5.3a.** Per CLAUDE.md hard rule #7. Phase 1 used era-appropriate tools (GM Decompiler v2.1, GMD-Recovery, LateralGM, WinXP VM).
- **Older Master `.gmd`/`.exe` (Master 4, 5-1, 5-2, 5-3) as protocol source.** Per D-03. Master 5-4 is canonical.
## Assumptions Log
> Items in this table need user confirmation before becoming locked decisions during planning.
| # | Claim | Section | Risk if Wrong |
|---|-------|---------|---------------|
| A1 | Legacy `localList.txt` and `.bnu` strings are Windows-1252 encoded | Pitfall 6 + Drizzle column comments | If actually UTF-8 or other codepage, SRV-10 importer needs different `iconv` step. **Verify in plan-step-1** by spot-checking byte ranges in `legacy/servers/enlyzeam-current/localList.txt`. |
| A2 | All `localList.txt` legacy passwords are plaintext (algorithm = `'plaintext'`) | D-04 staging schema | If some entries are pre-hashed (bcrypt-weak format), SRV-10/11 force-reset behavior differs. Spot-check during planning — passwords like "harrypotter" / "ilovepizza" per CONCERNS.md spot-check imply plaintext, but verify all 597 lines. |
| A3 | The 16 file-IO scripts cataloged above are the complete save-format inventory | §Verified format inventory table | If a script is missed (e.g., a save-format invoked only on a rare admin path), Phase 4 SRV-10 misses it. **Verify in plan-step-1** with `grep -lrn "file_text_open" extracted/server-5-4/` (expected count from researcher: ~16, exact list pending). |
| A4 | The xlsx CVEs do not affect Phase 3 build-time use | §Supporting libraries | If the xlsx path is chosen and a CVE later proves exploitable in CI even for trusted input, dependency removal required. **Mitigate** by recommending CSV-conversion-once (researcher's preferred path). |
| A5 | Better-Auth's Drizzle adapter conventions match the `accounts` table shape in D-13 | §Drizzle Schema target shape | If Better-Auth requires column names this design hasn't reserved (e.g., `email_verified`, `image`), Phase 4 SRV-09 will need to add columns. Cross-check Better-Auth docs during plan-step-2 (`docs/extracted-server/account-auth.md` authoring). |
| A6 | Litestream 0.3.13 is the production-recommended version (vs 0.5.x beta) on Fly.io | §Decision references | If Fly.io has shifted recommendation to 0.5.x by Phase 5, ADR 0002 needs an update. Researcher confidence: HIGH that 0.3.x remains recommended through 2026-05; planner should re-verify at Phase 5 not Phase 3. |
| A7 | `User_DB_Superweird.bnu` archive-only format is irrelevant to Phase 4 SRV-10 | Pitfall 3 | If enlyzeam-current contains any Superweird-format files, SRV-10 importer must handle both. **Verify in plan-step-1** — `ls legacy/servers/enlyzeam-current/UserData/` and confirm only `User_DBUpdated.bnu` shape present. |
| A8 | The `.bno`/`.bnu`/`.bnb` extension distinction is purely conventional (no format-level signal) | §Verified format inventory | Confirmed by inspection: all three are line-based file_text_* formats; the extension is the project's convention. No risk if true. |
**Disposition:** Address A1, A2, A3, A7 in plan-step-1 (a "ground-truth verification pass" before tool authoring begins). A4 is a planner choice (CSV vs xlsx). A5, A6 are Phase 4/5 verifications. A8 is informational.
## Open Questions (RESOLVED)
1. **Should the audit_log schema be Phase 7-final or seeded with a minimal column set in Phase 3?**
- What we know: D-13 says seed empty in Phase 3. Phase 7 PAR-07 writes here. Researcher's column draft above is illustrative.
- What's unclear: Whether Phase 7's admin actions need columns not yet imagined (e.g., IP address, request_id for tracing).
- **RESOLVED:** Recommendation: Seed with the columns in Example 3 (id, actor_account_id, action, target_account_id, payload_json, created_at). Use `payload_json` as the escape hatch — anything Phase 7 needs that isn't a column lives there. **No Phase 7 schema migration** is the goal.
2. **Do we ship `0001_baseline.sql` as a `drizzle-kit generate` artifact in Phase 3 or wait for Phase 4?**
- What we know: D-12 + D-13 say Drizzle is locked at end of Phase 3 with Drizzle table definitions. D-13 says "Phase 3 produces baseline DDL." Deferred per CONTEXT.md "Drizzle migration runner wiring" ⇒ Phase 4 owns runner.
- What's unclear: Whether `drizzle-kit generate` (the CLI that produces the `.sql` migration file from the TS schema) is "wiring" or "artifact production."
- **RESOLVED:** Recommendation: Ship `0001_baseline.sql` in Phase 3. The file is a deterministic artifact emitted from `tables.ts`; Phase 4 only adds the *runner* (the code that applies migrations at startup). Generation ≠ wiring.
3. **Should `tools/protocol-doc` and `tools/save-format-doc` share a `tools/gml-scan/` library, or duplicate the line-scanner code?**
- What we know: ~30% code overlap (file walker, GML line tokenizer, citation emitter, deterministic JSON writer). D-21 says two separate tools. Phase 1 D-17 says "no workspaces yet."
- What's unclear: Whether copying ~150 LOC of scanner shared code is acceptable.
- **RESOLVED:** Recommendation: **Duplicate now, refactor never (until a third consumer appears).** Two CLIs with their own `src/scanner/` is simpler to read in `pnpm` script names and lets the lints run independently. The duplication cost is ~150 LOC; the abstraction cost is creating a fourth standalone Node tool that two others depend on.
4. **What is the disposition for `Old Account Updater.exe` / `Old Accounts Reloader.exe` / `Account Updater.exe` in `legacy/servers/enlyzeam-current/`?**
- What we know: Per CONCERNS.md "Test/debug binaries treated as build artifacts but no source recipe." These are admin-tooling binaries with no clearly paired `.gmd`. Per D-20, they should appear in the modernized admin command surface (account-recover, account-reload commands).
- What's unclear: Without GML source, the exact original behavior is "execute and observe." That's RE territory beyond Phase 3 scope.
- **RESOLVED:** Recommendation: Document each in `admin-anti-port.md` with `disposition: rejected-with-reason` ("Original behavior unknown — no source GML; modernized replacement: web-UI /admin/account/recover endpoint per Phase 7 PAR-07"). Move on.
5. **Should `parity-checklist.json` rows reference subsystem MDs by file path, by anchor, or both?**
- What we know: D-18 row schema includes `originating_gml`, `originating_opcodes`, `originating_save_fields` but no `originating_subsystem_md`.
- What's unclear: When Phase 7 PAR-08 closes against the checklist, will engineers find the relevant doc easily?
- **RESOLVED:** Recommendation: Add a `subsystem` field to each row (one of the 11 subsystem MD names) as a string. Lint-parity validates it's one of the locked names. Cheap to add now; expensive to retrofit at PAR-08 close.
## Environment Availability
| Dependency | Required By | Available | Version | Fallback |
|------------|------------|-----------|---------|----------|
| Node 22 LTS | All Phase 3 tooling | ✓ (assumed — Phase 1/2 used same) | 22.x | None — required by `tools/extract-gmd` since Phase 1 |
| pnpm | Repo install + tool invocation | ✓ (Phase 1/2 used same) | 10.x | None — locked by Phase 1 D-17 |
| `extracted/server-5-4/` | Phase 3 inputs | ✓ (Phase 1 plan 01-07 emitted, committed `8190e09`) | n/a | None — Phase 1 dependency |
| `legacy/servers/enlyzeam-current/` | Phase 3 SDOC-06 inputs (canonical seed metadata) | ✓ (gitignored but present in working tree per CONCERNS.md) | n/a | None — required for D-04 staging schema research |
| `legacy/open-source-release/BN Online Message ID's Table.xls` | SDOC-02 cross-reference (D-07 step 3) | ✓ (committed under legacy/) | n/a | Manual transcription if XLS unreadable |
| `decomp/wiki/` | Cross-link from subsystem MDs | ✓ (Phase 1 input, unchanged) | n/a | None |
| `docs/extracted-engine/` | Cross-link from `client-server-bridge.md` | ✓ (Phase 2 commit `ad1e0c7`) | n/a | None — Phase 2 dependency |
| `tools/asset-catalog/` | D-17 server asset-catalog re-invocation | ✓ (Phase 2 commit `31fc746`) | n/a | None — Phase 2 dependency |
| Drizzle ORM | Schema authoring | ✓ (npm public, version 0.45.2 verified) | 0.45.2 | None |
| Drizzle Kit | Migration generation | ✓ (npm public) | 0.31.10 | Manual SQL DDL if drizzle-kit fails |
| xlsx | XLS reading (optional, see CSV-conversion preferred) | ✓ (npm public) | 0.18.5 | **Preferred fallback:** convert XLS → CSV manually (LibreOffice/Excel), commit CSV alongside XLS |
| iconv-lite | Phase 4 SRV-10 (legacy encoding conversion) | ✓ (npm public) | n/a (Phase 4 dep) | Node Buffer.toString('latin1') for windows-1252-equivalent |
| LateralGM source | Cross-reference if any unmapped DnD action surfaces | ✓ (Phase 2 reused; github.com/IsmAvatar/LateralGM) | n/a | Phase 2 already exhausted Action_IDs 523/525; if a new ID surfaces, follow Phase 2 D-08 path |
**Missing dependencies with no fallback:** None.
**Missing dependencies with fallback:** xlsx (fallback = manual CSV conversion, recommended path).
## Validation Architecture
> `.planning/config.json` shows `workflow.nyquist_validation: true` — section included.
### Test Framework
| Property | Value |
|----------|-------|
| Framework | vitest 4.1.5 (matches `tools/extract-gmd` and `tools/asset-catalog` pins) |
| Config file | `tools/protocol-doc/vitest.config.ts`, `tools/save-format-doc/vitest.config.ts` (each tool owns its own) |
| Quick run command | `pnpm --filter protocol-doc test` and `pnpm --filter save-format-doc test` (mirroring asset-catalog) |
| Full suite command | `pnpm -r test` (runs all tools' suites) + repo-level lint pipeline `pnpm lint` |
### Phase Requirements → Test Map
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|--------|----------|-----------|-------------------|-------------|
| SDOC-01 | Every server script (135) is reachable from at least one subsystem MD via cross-ref | unit (lint) | `node scripts/lint-docs.mjs docs/extracted-server/` (extends Phase 2 lint) | ❌ Wave 0 (extend lint config) |
| SDOC-01 | Every server object (58) is reachable from at least one subsystem MD via cross-ref | unit (lint) | `node scripts/lint-docs.mjs docs/extracted-server/` | ❌ Wave 0 |
| SDOC-01 | Subsystem MDs include `audio.md`-style "no audio" finding for missing-from-server categories | manual review | (verifier reads README.md jump table) | ❌ Wave 0 |
| SDOC-02 | Every opcode in `protocol.json` has ≥1 `gml_origin` citation pointing into `extracted/server-5-4/` | unit (lint) | `node scripts/lint-protocol.mjs docs/extracted-server/protocol.json` | ❌ Wave 0 |
| SDOC-02 | Every `mvp:true` opcode appears in CLI-08 message list (movement, chat, login, room-join, room-leave, heartbeat) | unit (lint) | `node scripts/lint-protocol.mjs` | ❌ Wave 0 |
| SDOC-02 | `protocol.md` is a deterministic re-render of `protocol.json` (no drift) | integration | `pnpm tsx tools/protocol-doc/cli.ts verify docs/extracted-server/` | ❌ Wave 0 |
| SDOC-02 | `protocol.ts` is a deterministic re-emit of `protocol.json` (no drift) | integration | `pnpm tsx tools/protocol-doc/cli.ts verify docs/extracted-server/` | ❌ Wave 0 |
| SDOC-02 | Every `sample_bytes` round-trips through `protocol.ts` codec (encode → decode → equal) | integration | `pnpm --filter protocol-doc test:integration` | ❌ Wave 0 |
| SDOC-03 | Every format in `save-formats.json` has `load_script` + `save_script` citations | unit (lint) | `node scripts/lint-save-formats.mjs docs/extracted-server/save-formats.json` | ❌ Wave 0 |
| SDOC-03 | `save-formats.md` and `save-formats.ts` are deterministic re-emits | integration | `pnpm tsx tools/save-format-doc/cli.ts verify docs/extracted-server/` | ❌ Wave 0 |
| SDOC-03 | `save-formats.ts` parsers round-trip on `legacy/servers/enlyzeam-current/MB_Log.bnb` (parse → serialize → byte-equal) | integration | `pnpm --filter save-format-doc test:integration` | ❌ Wave 0 |
| SDOC-04 | `schema.sql` is a deterministic emit from `tables.ts` (drizzle-kit) | integration | `pnpm exec drizzle-kit generate --schema docs/extracted-server/tables.ts --out docs/extracted-server/` then `git diff --exit-code docs/extracted-server/0001_baseline.sql` | ❌ Wave 0 |
| SDOC-04 | ADR 0002 cites at least one row from `save-formats.json` and `protocol.json` | unit (lint) | `node scripts/lint-adr.mjs docs/adr/0002-persistence-layer.md` (extends Phase 2 lint) | ❌ Wave 0 |
| SDOC-05 | Every parity-checklist row has non-empty `originating_gml` AND non-null `disposition` | unit (lint) | `node scripts/lint-parity.mjs docs/extracted-server/parity-checklist.json` | ❌ Wave 0 |
| SDOC-05 | Aggregate count of rows in each disposition logged | unit (lint reporting) | `node scripts/lint-parity.mjs --report` | ❌ Wave 0 |
| SDOC-05 | No parity-checklist row references an opcode not present in `protocol.json` | unit (lint) | `node scripts/lint-parity.mjs` (cross-ref) | ❌ Wave 0 |
| SDOC-06 | ADR 0003 records canonical = enlyzeam-current with dataset-size delta justification | unit (lint) | `node scripts/lint-adr.mjs docs/adr/0003-canonical-snapshot.md` | ❌ Wave 0 |
| SDOC-06 | `legacy_credentials_staging` table exists in `tables.ts` with all 6 columns per D-04 | unit | `pnpm --filter <tools/save-format-doc?> test` (or repo-level shape test) | ❌ Wave 0 |
### Sampling Rate
- **Per task commit:** `pnpm -r test` (each tool's vitest) + relevant lint script for the artifact touched.
- **Per wave merge:** `pnpm -r test && pnpm lint` (runs all 6 lints: docs, matrix, adr, protocol, save-formats, parity).
- **Phase gate:** Full suite green; `tools/protocol-doc verify` and `tools/save-format-doc verify` green; all three lint scripts green; `git diff --exit-code` on regenerated artifacts green; commit-time verifier check before `/gsd-verify-work`.
### Wave 0 Gaps
- [ ] `tools/protocol-doc/{package.json,cli.ts,tsconfig.json,vitest.config.ts}` + `src/scanner/`, `src/emit/`, `tests/` — covers SDOC-02
- [ ] `tools/save-format-doc/{package.json,cli.ts,tsconfig.json,vitest.config.ts}` + `src/scanner/`, `src/emit/`, `tests/` — covers SDOC-03
- [ ] `scripts/lint-protocol.mjs` — covers SDOC-02 lint invariants
- [ ] `scripts/lint-save-formats.mjs` — covers SDOC-03 lint invariants
- [ ] `scripts/lint-parity.mjs` — covers SDOC-05 lint invariants
- [ ] Extend `scripts/lint-docs.mjs` to cover `docs/extracted-server/` (config additions; existing tool reused)
- [ ] Extend `scripts/lint-adr.mjs` to cover ADRs 0002 + 0003 (config additions; existing tool reused)
- [ ] Extend `tools/asset-catalog` invocation: add `pnpm catalog:server` script + extend `pnpm catalog:all`
- [ ] `pnpm` repo-level scripts: `pnpm regen:server`, `pnpm verify:server`, `pnpm test:server` (chains protocol-doc + save-format-doc + asset-catalog server invocations)
## Project Constraints (from CLAUDE.md)
| Directive | Phase 3 Implication |
|-----------|---------------------|
| **Server-authoritative.** Clients send intent. Server emits state. Never trust client positions, scores, chat origin. | Parity-checklist `rejected-with-reason` for any feature whose original implementation trusts client (e.g., client-set chat origin in opcode 4 — server already validates `pid` in `0359-server_receive.gml` case 4 by reading `bytetemp` then validating `global.p_online[bytetemp]`, but the original passes `pid` from the connection ID rather than re-validating from session — Phase 4 re-derives the auth from session, never trusts client-supplied PID). |
| **No faithful port of plaintext passwords.** argon2id from packet 1. | D-04 staging-table schema is **pre-registered** in Phase 3; plaintext stays in `legacy_credentials_staging`, never enters production `accounts`. |
| **No faithful port of "run clipboard as superuser" admin.** | Every admin keybind in `,ServerCommands.txt` and `Ctrl+O Codes.txt` becomes a `rejected-with-reason` parity row with a modernized replacement command shape (D-20). |
| **`.bno` / `.bnb` / `.bnu` parsing requires extracted GML first.** | SDOC-03 schemas derived 100% from extracted GML; no hex inspection. Verified ground truth methodology. |
| **39dll wire protocol = call order.** | SDOC-02 schemas derived 100% from `write*`/`read*` sequences in extracted Master GML; no packet captures as primary source. |
| **Modern decompilers (UTMT, Altar.NET) cannot read GM 5.3a.** | N/A for Phase 3 — Phase 1 already produced `extracted/server-5-4/`. Phase 3 reads the extracted GML directly. |
| **Extract → document → rewrite, in that order.** | Phase 3 is "document." Hard rule: NO new TypeScript outside `tools/protocol-doc/` and `tools/save-format-doc/`. NO `apps/`, NO `packages/`. |
| **Repo stays private through Phase 7.** | The `legacy_credentials_staging` schema documentation references PII paths in `legacy/servers/enlyzeam-current/`. Confirm `.gitignore` excludes `legacy/` before any commit (it does per Phase 1 verify). |
**Audit:** Every research recommendation above complies with all 8 hard rules.
## Sources
### Primary (HIGH confidence)
- **`extracted/server-5-4/scripts/`** — DIRECT GROUND TRUTH for every opcode and save-format claim. Verified files read in this research session: `0023-sendmessage.gml`, `0024-receivemessage.gml`, `0359-server_receive.gml` (the central opcode dispatcher with 25+ opcode cases inline), `0008-ChtCmdRec.gml` (admin chat-command catalog), `0360-init_user.gml` (s2c opcodes 6/7/9 emitters), `0365-mb_backup.gml` (MB_Log.bnb sectioned grammar), `0367-users_restore.gml` (User_DBUpdated.bnu flat-loop grammar), `0376-uinv_backup.gml` + `0377-uinv_restore.gml` (Inventory_<uid>.bnu nested-loop grammar), `0368-uarea_restore.gml` (Areas_<uid>.bnu flat grammar), `0383-uhxb_restore.gml` (Bridges_<uid>.bnu flat grammar), `0386-load_settings.gml` (MSettings.bno flat grammar), `0392-users_load.gml` (localList.txt flat-loop grammar). [VERIFIED via Read tool 2026-05-03]
- **`legacy/open-source-release/,ServerCommands.txt`** — admin keybind catalog ground truth (Ctrl+E, Ctrl+Q, Ctrl+9, Ctrl+M, etc.). [VERIFIED]
- **`legacy/servers/enlyzeam-current/Ctrl+O Codes.txt`** — admin GML snippet ground truth. [VERIFIED]
- **`.planning/phases/03-server-documentation-schemas/03-CONTEXT.md`** — locked decisions D-01..D-22, ground truth for phase scope. [VERIFIED]
- **`.planning/research/PITFALLS.md` §A4 + §A5** — methodology for opcode + save-format extraction. [VERIFIED]
- **`.planning/research/STACK.md`** — version pins, alternatives matrix. [VERIFIED]
- **`.planning/codebase/CONCERNS.md`** — plaintext credentials, three drift'd snapshots, RCE-as-feature admin model. [VERIFIED]
- **`decomp/wiki/08-39dll-networking.md`** — "packet structure = read/write call order." [VERIFIED]
- **`decomp/wiki/16-bno-bnb-notes.md`** — ERRATA target (file_bin_* → file_text_*). [VERIFIED]
- **`docs/extracted-engine/`** (Phase 2 outputs) — subsystem-MD pattern, AUTOGEN-block discipline. [VERIFIED via existing files inspection]
- **`docs/adr/0001-client-engine.md`** — ADR template. [VERIFIED]
- **`tools/asset-catalog/cli.ts`** — CLI dispatcher template. [VERIFIED via Read]
- **`tools/asset-catalog/package.json`** — TS/tsx/vitest pin template. [VERIFIED via Read]
- **`.planning/config.json`** — workflow flags (nyquist_validation: true, plan_check: on, verifier: on). [VERIFIED]
- **npm registry** (`npm view <pkg> version`, 2026-05-03): drizzle-orm 0.45.2, drizzle-kit 0.31.10, better-sqlite3 12.9.0, colyseus 0.17.10, xlsx 0.18.5. [VERIFIED]
### Secondary (MEDIUM confidence)
- **`.planning/research/SUMMARY.md`** — Phase ordering rationale, research-flag list. Cross-checks with PROJECT.md and ROADMAP.md.
- **`.planning/research/ARCHITECTURE.md`** (cited from CONTEXT.md, not directly re-read this session) — three-pipeline topology.
- **`legacy/open-source-release/BN Online Message ID's Table.xls`** — opcode-name HINT cross-reference (per D-07 step 3, NOT canonical truth on conflict). Researcher did not parse the XLS in this session; planner verifies during plan-step-1.
- **Better-Auth Drizzle adapter docs** [CITED: better-auth.com] — informs `accounts` table column conventions. Not directly verified this session; Assumption A5 flags risk.
- **Litestream docs** [CITED: litestream.io] — informs ADR 0002 rationale. Not re-verified beyond STACK.md citation.
### Tertiary (LOW confidence)
- **xlsx CVE assessment** — researcher's claim that build-time use is not exploitable for trusted local input is a reasonable engineering judgment but not formally verified by a security review. Mitigation (CSV-conversion-once) recommended as default.
- **Litestream 0.3.13 = current production-recommended version** — based on STACK.md citation; planner re-verifies at Phase 5 (when actually deployed).
## Metadata
**Confidence breakdown:**
- Standard stack: **HIGH** — all versions verified via `npm view` 2026-05-03; matches STACK.md pins; no surprises.
- Architecture (tools layout, three-artifact pattern, autogen blocks): **HIGH** — directly mirrors proven Phase 2 pattern (Phase 2 7/7 plans complete, lint suite green).
- Opcode extraction methodology: **HIGH** — verified against actual `0359-server_receive.gml` content; PITFALLS A4 procedure confirmed deterministic on real script.
- Save-format extraction methodology: **HIGH** — verified against actual `0365-mb_backup.gml`, `0367-users_restore.gml`, `0376-uinv_backup.gml` content; confirms D-08 errata; sectioned grammar pattern (`@TOPIC`, `@REPLY`) verified.
- Drizzle schema target shape: **MEDIUM-HIGH** — column list grounded in D-13 + verified save-format grammars + Better-Auth conventions; planner may adjust within D-13's bounds. Assumption A5 flags Better-Auth column-shape risk.
- ADR templates: **HIGH** — `docs/adr/0001-client-engine.md` is the proven template.
- Parity-checklist methodology: **HIGH** — D-18 schema is precise; lint discipline (D-22) catches structural drift.
- Pitfalls: **HIGH** — direct ground-truth verification on every cited script.
- Open questions disposition: **MEDIUM** — 5 questions surfaced; recommendations are researcher's best judgment, not yet user-confirmed.
**Research date:** 2026-05-03
**Valid until:** 2026-06-03 (30 days; Phase 3 documentation phase, slow-moving). After this date, re-verify npm versions before plan execution.