# Phase 1: Extraction - Context

**Gathered:** 2026-05-02
**Status:** Ready for planning
**Mode:** Claude's discretion (user delegated all gray-area decisions)

<domain>
## Phase Boundary

Both source `.gmd` files (`legacy/open-source-release/BN Online Client 5-8.gmd` and `BN Online Master 5-4.gmd`) are reproducibly exported into a structured, diffable per-resource tree, committed to git, with the era-appropriate decompiler stack documented in `decomp/TOOLS.md`.

In scope:
- Extraction tool (`tools/extract-gmd`) — TS Node CLI walking the `.gmd` binary block format per `decomp/wiki/03-gmd-format.md`.
- Per-resource VCS tree under `extracted/<source>/...` covering scripts, objects, sprites, backgrounds, rooms, sounds, fonts, paths, timelines, datafiles, settings.
- DnD action graphs serialized as JSON descriptors plus best-effort GML transcompilation.
- Reproducibility verification (SHA256 manifest re-run check).
- `decomp/TOOLS.md` documenting the era-appropriate stack and Rank-3 WinXP fallback.

Out of scope (belongs in later phases):
- Any new TypeScript outside `tools/` (Phases 4+).
- Documenting engine/server features from extracted GML (Phases 2/3).
- Asset format conversion beyond canonical decode — MIDI→OGG, BMP→atlas, font→WOFF2 are Phase 6/7 (`tools/asset-pipeline`).
- Reverse-engineering the 39dll wire protocol or `.bno`/`.bnu`/`.bnb` schemas (Phase 3).

</domain>

<decisions>
## Implementation Decisions

### Extraction Tool & Method

- **D-01:** Build `tools/extract-gmd` as a TS Node CLI (Node 22, strict TS). Walks `.gmd` binary sequentially per wiki/03-gmd-format. Headless, CI-runnable, no JVM or Windows runtime dependency on the daily-dev critical path.
  - Reason: TS-everywhere convention (PROJECT.md, research/STACK.md). Avoids Java/JRE for routine extraction. LateralGM's `LibReader.java` stays the canonical reference parser — port its block walker semantics, do not depend on its runtime.
- **D-02:** Ship LateralGM only as a verification crosscheck. `tools/verify-extract` runs LateralGM headless export on a small set of resources and diffs against the TS-parser output. Run on CI for both source files. Detects parser drift early.
  - Reason: Independent oracle. Wiki/11 names LateralGM as canonical parser — use it to validate, not as the runtime.
- **D-03:** Pull the DnD `Action_ID` lookup table verbatim from LateralGM source into `tools/extract-gmd/data/action-ids.json`. Commit with attribution comment. Required because raw IDs are opaque (wiki/04).
- **D-04:** WinXP VM + 5.3a IDE is documented as Rank-3 fallback only (per wiki/15 ranking), NOT a Phase 1 deliverable. We have `.gmd` directly — Rank 1 (native source) already satisfied. VM only needed if dynamic recovery from `.exe` is ever required for an unknown future snapshot. Skip building/sourcing the VM image now; document the procedure.
  - Reason: User explicitly said legacy/extras (cracked GM 5.3a) is not a concern, and the LateralGM path makes daily extraction VM-free. Don't burn Phase 1 time on a deliverable nothing else consumes.

### Output Tree Layout

- **D-05:** Canonical committed path: `extracted/<source>/...` at repo root. NOT under `decomp/` (that's the RE knowledge base) and NOT under `docs/` (those are written documentation, Phases 2/3 deliverables).
- **D-06:** Two top-level subtrees, one per source `.gmd`:
  - `extracted/client-5-8/`
  - `extracted/server-5-4/`
- **D-07:** GmkSplitter-style hierarchy (per wiki/12) inside each subtree:
  ```
  extracted/<source>/
    settings.json
    scripts/<name>.gml
    objects/<name>/
      meta.json              # parent, sprite, mask, depth, persistent, solid, visible
      events/<event-name>.gml      # transcompiled GML (best-effort, see D-09)
      events/<event-name>.dnd.json # canonical DnD descriptor (lossless)
    sprites/<name>/
      meta.json              # bbox, origin, mask, subimage count
      frames/img_NN.png      # one PNG per subimage, deterministically encoded
    backgrounds/<name>/
      meta.json              # tile dims, transparency
      image.png
    rooms/<name>/
      meta.json              # size, speed, persistent, view config
      instances.json         # object instance positions
      tiles.json             # tile placements
      backgrounds.json       # background-layer config
      creation-code.gml      # if present
    sounds/<name>/
      meta.json              # type, sample rate, length, embedded vs path
      audio.<wav|mid|mp3>    # only when embedded byte buffer present
    fonts/<name>/
      meta.json              # glyph set, render bounds
    paths/<name>.json        # waypoints + speed nodes (whole resource one file)
    timelines/<name>/
      meta.json
      moments/<step>.gml
      moments/<step>.dnd.json
    datafiles/<name>.<ext>   # raw embedded data files (5.x feature, wiki/03 §8)
  ```
- **D-08:** `.extracted-cache/` (gitignored) holds intermediate working state during a run — partial dumps, hash sidecars, parser logs. The `extracted/` tree is the committed, golden output.

### DnD Action Handling

- **D-09:** Emit BOTH for every Object event and Timeline moment that contains DnD nodes:
  1. `<name>.dnd.json` — canonical lossless descriptor (per-node Action_ID, Applies_To, Is_Relative, Argument_Types, Argument_Values).
  2. `<name>.gml` — best-effort GML transcompile using known Action_ID → GML mappings from LateralGM's source.
- **D-10:** Diffs are read primarily on `.gml`; descriptor JSON is the round-trip fallback. Mark transcompiled events with a header comment `// auto-transcompiled from event.dnd.json — see descriptor for canonical truth`. Wiki/04 warns the round-trip is lossy; descriptor preserves truth.
- **D-11:** When an Action_ID is unknown to the lookup table, emit only `.dnd.json` and a stub `.gml` containing `// UNKNOWN ACTION_ID=<id>; see .dnd.json` plus log to a `extracted/<source>/UNKNOWN-ACTIONS.md` index for follow-up.

### Asset Binary Handling

- **D-12:** ~~Sprites and backgrounds: GMD stores ZLIB-compressed raw pixel buffers (wiki/03), NOT BMP. Decode to PNG using a deterministic encoder (sharp with fixed `compressionLevel: 9`, `palette: false`, `effort: 10`, `progressive: false`).~~  **REVISED 2026-05-02 plan 01-07 (Option A pivot):** Per LateralGM `GmStreamDecoder.readZlibImage` and validated against real BNO `.gmd` files, the ZLIB-deflated payload INFLATES to a **complete BMP file** (magic `0x42 0x4D = "BM"`), NOT raw RGBA. Phase 1 carries `imageBytes: Buffer` opaquely on `SpriteFrame` / `BackgroundImage` / `Font.glyphs`; emit writes `<frame>.bmp` (or `<frame>.bin` if format unrecognized). **BMP→PNG decode deferred to Phase 6/7 asset-pipeline (AST-01)** per CLAUDE.md hard rule #6 "Extract → document → rewrite, in that order. Phase 1 = extract." Rationale: removes BMP-decoder risk from Phase 1 (sharp/libvips BMP coverage on v530-era variants is unverified), preserves MANIFEST.sha256 byte-identity trivially, and yields a smaller cleaner Phase 1 close. Wiki/03 errata section documents this. EXT-04 acceptance relaxed accordingly (REQUIREMENTS.md).
- **D-13:** Sounds: when GMD stores embedded byte buffer, write the raw bytes as `audio.<original-extension>` (WAV/MIDI/MP3 detectable via magic bytes). When GMD stores a relative file path only, record the path in `meta.json` and skip writing audio bytes — the original `.wav`/`.mid` lives in `legacy/audio/` and is referenced.
- **D-14:** Fonts in 5.3a are typically rasterized glyph metrics + render bounds, not TTF blobs. Emit `meta.json` with the metrics + a `glyphs.png` strip when raster glyph data is present. WOFF2 conversion is Phase 7 asset pipeline (AST-03).

### Reproducibility Strategy

- **D-15:** EXT-07 (byte-identical re-runs) enforced by:
  - All JSON: sorted keys, 2-space indent, LF line endings, no trailing newline drift, no timestamps, no absolute paths.
  - All GML output: LF line endings, preserve original indentation/whitespace from the .gmd plaintext blocks.
  - PNG encoder pinned (sharp version + options listed above).
  - No file mtimes embedded in any output.
  - Resource enumeration order = the source `.gmd`'s native sequential order (per wiki/03 — strict sequential serialization). Never sort by name.
- **D-16:** `tools/extract-gmd verify <source>` recomputes a SHA256 manifest of the entire output tree and diffs against `extracted/<source>/MANIFEST.sha256` (committed). CI runs this on every PR. Any drift fails the build.

### Tooling & Repo Structure

- **D-17:** Tool lives at `tools/extract-gmd/` per research/ARCHITECTURE.md. Standalone Node CLI, not yet a workspace package (pnpm workspaces are introduced Phase 4).
- **D-18:** Single CLI handles both source files. Invocation: `pnpm tsx tools/extract-gmd/cli.ts <input.gmd> <out-dir>`. Repo-level npm script `pnpm extract:all` runs both extractions.
- **D-19:** `decomp/TOOLS.md` (EXT-08) is the documentation deliverable: era-appropriate stack (GM Decompiler v2.1, GMD-Recovery, LateralGM), Rank 1-4 procedure (already in wiki/15 — TOOLS.md links + adds installation/provenance notes), WinXP VM as Rank-3 fallback procedure, `tools/extract-gmd` as the project's primary daily extractor.

### Claude's Discretion

User delegated all gray areas. The above decisions reflect best judgment grounded in PROJECT.md constraints, research/STACK.md, and the decomp/wiki. Specific points where Claude exercised discretion (override these in planning if any look wrong):

- LateralGM as verification oracle, not runtime (D-02).
- WinXP VM deferred to documentation-only (D-04).
- Both DnD descriptor and transcompiled GML emitted (D-09).
- ZLIB pixel buffers decoded to deterministic PNG, not BMP (D-12).
- `extracted/` at repo root, not under `decomp/` (D-05).

### Folded Todos

None — todo cross-reference returned no matches for Phase 1 scope.

</decisions>

<canonical_refs>
## Canonical References

**Downstream agents MUST read these before planning or implementing.**

### Project planning
- `.planning/PROJECT.md` — vision, constraints, key decisions, anti-decisions
- `.planning/REQUIREMENTS.md` §Extraction — EXT-01..08 acceptance criteria
- `.planning/ROADMAP.md` §"Phase 1: Extraction" — goal + 5 success criteria
- `.planning/research/SUMMARY.md` §"Phase 1: Stage 1 — Extraction (Foundations)" — research-flagged items
- `.planning/research/STACK.md` — TS/Node tooling versions
- `.planning/research/ARCHITECTURE.md` — `tools/extract-gmd` topology
- `.planning/research/PITFALLS.md` §A1, §A2, §A6, §A7, §D6 — pitfalls Phase 1 must avoid
- `.planning/codebase/CONCERNS.md` §Legal/IP, §Security, §Data Integrity — what NOT to commit, what NOT to publish

### Reverse-engineering wiki (decomp/wiki/)
- `decomp/wiki/00-overview.md` — entry point
- `decomp/wiki/03-gmd-format.md` — top-level block layout, sequential parsing rules, ZLIB blocks, ID conventions
- `decomp/wiki/04-dnd-serialization.md` — per-node binary layout, Action_ID lookup, GML transcompile guidance
- `decomp/wiki/05-gml-vm.md` — GML script semantics
- `decomp/wiki/06-gml-syntax-5x.md` — 5.x-specific syntax (relevant for transcompile validity)
- `decomp/wiki/07-gml-core-functions.md` — core functions (relevant for DnD → GML mappings)
- `decomp/wiki/11-tool-lateralgm.md` — `LibReader.java` is the canonical reference parser
- `decomp/wiki/12-tool-gmksplitter.md` — VCS-tree pattern (target shape for D-07)
- `decomp/wiki/13-modern-tool-incompat.md` — UTMT/Altar.NET will fail; locked OUT
- `decomp/wiki/14-gb1-backups.md` — `.gb1` is byte-identical to `.gmd` (relevant if older snapshots ever extracted)
- `decomp/wiki/15-extraction-pipeline.md` — Rank 1-4 procedure (informs `decomp/TOOLS.md`)
- `decomp/wiki/quick-reference.md` + `glossary.md` — terminology

### Source files (read-only inputs)
- `legacy/open-source-release/BN Online Client 5-8.gmd` — client source `.gmd` (input to extraction)
- `legacy/open-source-release/BN Online Master 5-4.gmd` — server source `.gmd` (input to extraction)

### External implementation references
- LateralGM `LibReader.java` upstream (`github.com/IsmAvatar/LateralGM`) — block walker semantics + Action_ID table to port
- GmkSplitter pattern (`github.com/Medo42/Gmk-Splitter`) — hierarchical-tree shape (per D-07)

</canonical_refs>

<code_context>
## Existing Code Insights

### Reusable Assets

- **None in repo yet** — `rebno/` is brownfield-archive only. `tools/extract-gmd` is the first new code to land.
- `decomp/wiki/` (17 docs) is the load-bearing knowledge asset. Parser implementation transcribes wiki/03 + wiki/04 directly.
- LateralGM `LibReader.java` is the external reusable asset (port semantics, do not vendor source — license check needed if vendored).

### Established Patterns

- Repo convention: `decomp/` = RE knowledge, `legacy/` = original artifacts (gitignored per `.gitignore`), `.planning/` = GSD workflow state. New extracted source belongs in a NEW top-level path that is neither — `extracted/` (D-05).
- TS-everywhere mandate from PROJECT.md applies even to tooling.
- `.gitignore` currently excludes only `legacy/` — must extend to add `.extracted-cache/` and any working-state dirs Phase 1 introduces.

### Integration Points

- `tools/extract-gmd` output → `extracted/{client-5-8,server-5-4}/` → consumed by Phase 2 (`docs/extracted-engine/` written from client tree) and Phase 3 (`docs/extracted-server/` + 39dll opcode reverse + save-format schemas written from server tree).
- `decomp/TOOLS.md` cross-links to existing `decomp/wiki/15-extraction-pipeline.md`.
- CI: GitHub Actions (introduced Phase 5) will eventually run `pnpm extract:verify`. Phase 1 ships the script; Phase 5 wires it into CI.

</code_context>

<specifics>
## Specific Ideas

- Pin `sharp` to a single version with deterministic PNG options (D-12). Surface `sharp` version in `extracted/<source>/MANIFEST.sha256` header so any future encoder change is detectable.
- `extracted/<source>/UNKNOWN-ACTIONS.md` (D-11) becomes a forcing-function list for Phase 2 — any unmapped Action_ID is a Phase 2 followup before the engine doc can be considered complete.
- Treat `decomp/TOOLS.md` (EXT-08) as a thin wrapper that links into `decomp/wiki/15-extraction-pipeline.md` rather than duplicating content. Single source of truth = the wiki.

</specifics>

<deferred>
## Deferred Ideas

- **Extracting older `.gmd` / `.gb1` snapshots** in `legacy/source-archive/` and `legacy/servers/*/`. Source-of-truth files are locked to the two latest revisions (PROJECT.md). Older snapshots become useful only if (a) Phase 3 canonical-snapshot ADR (SDOC-06) needs cross-referencing, or (b) Phase 7 historical-comparison work surfaces a need. Deferred to Phase 3+ as needed.
- **Round-trip `.gmd` writer** (extracted-tree → `.gmd`). Not needed for the rebuild — we read once and rewrite in TS. If ever wanted for legacy IDE editing, defer to a future tools task.
- **Asset format conversion** (BMP atlas, MIDI→OGG, font→WOFF2). Belongs in `tools/asset-pipeline` (Phase 6 AST-01 + Phase 7 AST-02..04), not extraction.
- **WinXP VM image build + GM 5.3a IDE installation procedure**. Documented-only in Phase 1 (D-04). Build the VM if a future phase actually needs Rank-3 dynamic recovery against an .exe.
- **Vendoring LateralGM source**. License check + selective vendor of `LibReader.java` + Action_ID table only if the upstream becomes unavailable. For now, port semantics from upstream and link.

### Reviewed Todos (not folded)

None — none surfaced.

</deferred>

---

*Phase: 01-extraction*
*Context gathered: 2026-05-02*
</content>
</invoke>