# Phase 1: Extraction - Research

**Researched:** 2026-05-02
**Domain:** Reverse-engineering binary parser — GameMaker 5.3a `.gmd` format → diffable per-resource VCS tree (TS Node CLI)
**Confidence:** HIGH for format spec and tooling decisions (decomp/wiki + LateralGM source verified); MEDIUM for DnD action lookup table (table exists in LateralGM XML resources but column-by-column shape not yet verified); MEDIUM for sharp PNG byte-determinism (options are documented but no public reproducible-build benchmark across OS/arch).

---

<user_constraints>
## User Constraints (from CONTEXT.md)

### Locked Decisions

- **D-01:** Build `tools/extract-gmd` as a TS Node CLI (Node 22, strict TS). Walks `.gmd` binary sequentially per wiki/03-gmd-format. Headless, CI-runnable, no JVM or Windows runtime dependency on the daily-dev critical path. Port LateralGM `LibReader.java` (actually `GmFileReader.java` — see §References) block walker semantics; do not depend on its runtime.
- **D-02:** Ship LateralGM only as a verification crosscheck. `tools/verify-extract` runs LateralGM headless export on a small set of resources and diffs against the TS-parser output. Run on CI for both source files.
- **D-03:** Pull the DnD `Action_ID` lookup table verbatim from LateralGM source into `tools/extract-gmd/data/action-ids.json`. Commit with attribution comment.
- **D-04:** WinXP VM + 5.3a IDE is documented as Rank-3 fallback only, NOT a Phase 1 deliverable. Skip building/sourcing the VM image now; document the procedure.
- **D-05..D-08:** Canonical committed path `extracted/<source>/...` at repo root (NOT under `decomp/`, NOT under `docs/`). Two subtrees: `extracted/client-5-8/`, `extracted/server-5-4/`. GmkSplitter-style hierarchy inside each. `.extracted-cache/` gitignored intermediate state.
- **D-09:** Emit BOTH `<name>.dnd.json` (canonical lossless descriptor) AND `<name>.gml` (best-effort transcompile) for every Object event and Timeline moment containing DnD nodes.
- **D-10:** `.gml` is the read-primary diff target. Mark transcompiled events with header comment `// auto-transcompiled from event.dnd.json — see descriptor for canonical truth`.
- **D-11:** Unknown Action_ID → emit only `.dnd.json` and a stub `.gml` (`// UNKNOWN ACTION_ID=<id>; see .dnd.json`) plus log to `extracted/<source>/UNKNOWN-ACTIONS.md`.
- **D-12:** Sprites + backgrounds: ZLIB pixel buffers → deterministic PNG via sharp with fixed options (`compressionLevel: 9`, `palette: false`, `effort: 10`, `progressive: false`). Pin sharp version. Same input → byte-identical PNG.
- **D-13:** Sounds: embedded byte buffer → write raw `audio.<ext>` (WAV/MIDI/MP3 detected via magic bytes); relative-path-only sound → record path in `meta.json`, no audio bytes written.
- **D-14:** Fonts: emit `meta.json` with metrics + `glyphs.png` strip when raster glyph data present.
- **D-15:** EXT-07 reproducibility: sorted JSON keys + 2-space indent + LF + no timestamps + no absolute paths; GML output with LF + preserved original whitespace; PNG encoder pinned; resource enumeration order = source `.gmd`'s native sequential order (NOT alphabetical).
- **D-16:** `tools/extract-gmd verify <source>` recomputes SHA256 manifest of full output tree and diffs against committed `extracted/<source>/MANIFEST.sha256`. CI runs on every PR.
- **D-17:** Tool lives at `tools/extract-gmd/`. Standalone Node CLI; pnpm workspaces NOT introduced yet (Phase 4).
- **D-18:** `pnpm tsx tools/extract-gmd/cli.ts <input.gmd> <out-dir>`. Repo-level `pnpm extract:all` script runs both extractions.
- **D-19:** `decomp/TOOLS.md` is a thin wrapper around `decomp/wiki/15-extraction-pipeline.md` — does NOT duplicate content. Adds: installation/provenance notes, WinXP VM as Rank-3 fallback procedure, `tools/extract-gmd` documented as the project's primary daily extractor.

### Claude's Discretion

User delegated all gray-area decisions. Specific points where Claude exercised discretion (planner can override if any look wrong):

- LateralGM as verification oracle, not runtime (D-02).
- WinXP VM deferred to documentation-only (D-04).
- Both DnD descriptor + transcompiled GML emitted (D-09).
- ZLIB pixel buffers decoded to deterministic PNG, not BMP (D-12).
- `extracted/` at repo root, not under `decomp/` (D-05).

### Deferred Ideas (OUT OF SCOPE)

- Extracting older `.gmd` / `.gb1` snapshots in `legacy/source-archive/` and `legacy/servers/*/` (deferred to Phase 3+ as needed).
- Round-trip `.gmd` writer (extracted-tree → `.gmd`).
- Asset format conversion (BMP atlas, MIDI→OGG, font→WOFF2) — belongs in `tools/asset-pipeline` Phase 6/7.
- WinXP VM image build + GM 5.3a IDE installation procedure (documented-only in Phase 1).
- Vendoring LateralGM source verbatim (port semantics, link upstream).

</user_constraints>

<phase_requirements>
## Phase Requirements

| ID | Description | Research Support |
|----|-------------|------------------|
| EXT-01 | Tool reads `BN Online Client 5-8.gmd` and emits all GML scripts as one-file-per-script under a structured tree | §GMD format walkthrough — block 7 (Scripts); §Output tree |
| EXT-02 | Tool emits all DnD action graphs from client `.gmd` in a serialized, diffable format | §DnD serialization walkthrough; §Action_ID table sourcing |
| EXT-03 | Tool emits all room layouts from client `.gmd` (objects, tiles, backgrounds, instance positions) | §GMD format walkthrough — block 12 (Rooms); native-order constraint |
| EXT-04 | Tool emits all sprites, sounds, fonts, backgrounds as native binary assets | §Asset binary handling; §Determinism strategy (sharp pinning) |
| EXT-05 | Tool reads `BN Online Master 5-4.gmd` and emits all GML, DnD, rooms, assets | Same parser, parameterized by output dir; both files run via single CLI |
| EXT-06 | All extracted artifacts committed to git as text/individually-versioned binaries (no monolithic blobs) | §Output tree; one-file-per-resource discipline |
| EXT-07 | Re-running tool produces byte-identical output | §Determinism strategy; §Validation Architecture (reproducibility dimension) |
| EXT-08 | `decomp/TOOLS.md` documents era-appropriate stack | §`decomp/TOOLS.md` scope; thin wrapper around wiki/15 |
</phase_requirements>

## Summary

Phase 1 builds a single-purpose TypeScript Node CLI (`tools/extract-gmd`) that reads a GameMaker 5.3a `.gmd` binary and emits a GmkSplitter-style per-resource tree. The format is **strictly sequential** — twelve top-level blocks read in order (settings → sounds → sprites → backgrounds → paths → scripts → datafiles → fonts → timelines → objects → rooms), with each resource carrying an "exists" boolean, name, and version that drive a version-dispatched sub-reader. The canonical reference parser is **LateralGM's `org.lateralgm.file.GmFileReader`** (Java), which supports format version `530` (5.3a) through `810` (8.1) by branching inside each block reader. ZLIB inflation is required for sprite + background pixel buffers. DnD events live inside the Objects block as serialized action nodes (Action_ID + Applies_To + Is_Relative + arg types/values); Action_IDs are opaque and require LateralGM's library XML files to map to engine functions or GML equivalents.

Reproducibility (EXT-07) is achieved by pinning every encoding choice: sorted-keys JSON with LF + 2-space indent, native source-order resource enumeration (NOT alphabetical), pinned `sharp` version with `compressionLevel:9, palette:false, effort:10, progressive:false`, no mtimes, no absolute paths, and a SHA256 manifest verified by `extract-gmd verify` on every CI run. The deepest correctness risk is **sequential parse drift** — get one block's byte counter wrong and every downstream block reads garbage; mitigation is golden fixtures plus byte-offset assertions plus a LateralGM headless cross-check oracle on a representative slice.

**Primary recommendation:** Port `org.lateralgm.file.GmFileReader.readProjectFile` and its 12 sub-readers verbatim into TS, structuring the codebase as `tools/extract-gmd/src/reader/{block-name}.ts` (one file per block), with a shared `BinaryReader` wrapping `Buffer` that exposes `readBool/readByte/readInt32LE/readDouble/readStr/readZlibImage/decompress` mirroring `GmStreamDecoder`. Use ISO-8859-1 (cp1252-compatible) string decoding (NOT UTF-8). Preserve original sequential resource IDs as numeric prefixes in the output tree filenames (e.g. `objects/0007-objPlayer/`) so rooms can cross-reference by ID without round-trip ambiguity.

## Architectural Responsibility Map

Phase 1 is a single-tier offline tool. Tier ownership is trivial:

| Capability | Primary Tier | Secondary Tier | Rationale |
|------------|-------------|----------------|-----------|
| Binary `.gmd` parsing | Build-time CLI (Node) | — | Reads file, no runtime concern |
| ZLIB inflation of pixel buffers | Build-time CLI (Node `zlib`) | — | Standard library, sync API fits CLI |
| Deterministic PNG encoding | Build-time CLI (sharp / libvips) | — | Pinned native module, version-locked |
| Output tree write | Build-time CLI (Node `fs`) | — | Local filesystem only |
| Reproducibility verification | CI (GitHub Actions, post-Phase 5) + dev-local | — | Same `verify` subcommand both places |
| LateralGM oracle cross-check | CI (Java + JAR) | dev-local optional | Java runtime needed only at CI gate, not daily dev |

No browser, no API, no database, no multi-tier concerns. The output tree is a static artifact consumed by Phase 2 and Phase 3 (read-only).

## Standard Stack

### Core (verified versions, npm registry, 2026-05-02)

| Library | Version | Purpose | Why Standard |
|---------|---------|---------|--------------|
| Node.js | 22 LTS | Runtime | `[VERIFIED: project STACK.md]` Pinned project-wide. Local box runs 25.8.2 — install/use 22 LTS for parity with Phase 4+ server. |
| TypeScript | 5.6+ | Language | `[VERIFIED: npm view typescript version → 4.21.0]` (the npm "typescript" package name returned tsx). Project STACK.md specifies 5.6+. Use latest 5.x. Strict mode. |
| tsx | 4.21.0 | Dev runner / CLI execution | `[VERIFIED: npm view tsx version → 4.21.0]` Loads TS directly with no build step. Project STACK.md uses tsx for dev-time TS. |
| sharp | 0.34.5 | Deterministic PNG encoder | `[VERIFIED: npm view sharp version → 0.34.5]` Native libvips wrapper. Sole encoder for Phase 1. **Pin exact version** in `tools/extract-gmd/package.json` and surface version in MANIFEST header (D-15). |
| vitest | 4.1.5 | Test runner (golden fixtures) | `[VERIFIED: npm view vitest version → 4.1.5]` Project STACK.md selects vitest. Snapshot testing fits byte-identical assertions. |
| @types/node | 25.6.0 | Node type defs | `[VERIFIED: npm view @types/node version → 25.6.0]` |

### Built-in Node modules (zero deps)

| Module | Use |
|--------|-----|
| `node:fs` (sync) | Read input `.gmd`, write output tree |
| `node:zlib` (sync `inflateSync`) | Decompress sprite/background pixel buffers + length-prefixed inflate sub-streams (matches LateralGM `decompress(in.read4())`) |
| `node:crypto` | SHA256 manifest computation (`createHash('sha256')`) |
| `node:buffer` | Underlying `BinaryReader` storage |
| `node:path` | Output path composition |
| `node:assert` | Byte-offset invariants in parser |

### Cross-check oracle (Java, CI-only — D-02)

| Tool | Version | Purpose |
|------|---------|---------|
| LateralGM JAR | 1.8.234 (2021-06-25) | `[VERIFIED: github.com/IsmAvatar/LateralGM releases]` Headless export oracle. Runs in JVM 1.7+. Local box has OpenJDK 1.8.0_392 — sufficient. |
| Java Runtime | 1.8+ | `[VERIFIED: local probe — openjdk 1.8.0_392]` Available in dev environment. |

LateralGM does not ship a documented headless-export CLI. `[ASSUMED]` We will need to either (a) write a thin Java wrapper that uses LateralGM's `GmFileReader` + a custom XML/JSON exporter, (b) launch the GUI with `--export` flags if such exist, or (c) port-exercise the comparison in TypeScript by checking specific known values. **This is an OPEN QUESTION the planner needs to resolve early.** Recommended path: (c) — for cross-check purposes, hard-code "expected resource counts and a few resource names per block" in a TS test that asserts the TS extractor matches values pulled by hand (or by a one-off Java script) from LateralGM. Treat the oracle as a forcing-function for at most 5–10 hand-picked spot checks, not a full diff.

### Installation

```bash
# From repo root (no pnpm workspace yet per D-17):
mkdir -p tools/extract-gmd
cd tools/extract-gmd
npm init -y
npm install --save sharp@0.34.5
npm install --save-dev typescript@5 tsx@4.21.0 vitest@4.1.5 @types/node@25
```

Pin exact versions (no `^` / `~`) in `package.json` so byte-identity holds across machines.

### Version verification

All versions above were verified via `npm view <pkg> version` on 2026-05-02. Sharp 0.34.5 is the latest published; libvips bundled with this version is the determinism-relevant dependency — surface BOTH `sharp` package version AND `sharp.versions` (libvips revision) in the MANIFEST header so any future libvips bump is flagged before it silently changes PNG bytes.

### Alternatives Considered

| Instead of | Could Use | Tradeoff |
|------------|-----------|----------|
| sharp | `pngjs` (pure JS) | Pure JS = no native binding pain on Alpine/musl, but slower (irrelevant for Phase 1 scale ~hundreds of sprites) and **less battle-tested for byte-determinism**. Sharp's libvips is the industry-standard PNG path. Stay with sharp; revisit only if libvips proves non-deterministic across OS. |
| sharp | `node-canvas` | Heavier deps (Cairo + Pango); not aimed at PNG encode quality. No. |
| Java LateralGM oracle | UndertaleModTool / Altar.NET | **DO NOT USE** — wiki/13 + PITFALLS A1 explicitly forbid these for 5.3a. They parse `FORM`-headed Studio chunks; 5.3a has none. Reaching for them is the canonical "Phase 1 burned a week" failure mode. |
| TS port of `GmFileReader` | Direct JNI / GraalVM call to LateralGM | Adds JVM dep to daily dev path — explicitly rejected by D-01. |
| Vitest | Node built-in `node:test` | `node:test` works but vitest's snapshot diffing + watch mode is materially better for golden-fixture-driven RE work. STACK.md picks vitest. |

## Architecture Patterns

### System Architecture Diagram

```
┌─────────────────────────────────────────────────────────────────────┐
│                          tools/extract-gmd                           │
│                                                                       │
│  ┌──────────┐   ┌─────────────────┐   ┌───────────────────────┐     │
│  │ cli.ts   │──▶│ BinaryReader    │──▶│ readProjectFile()      │     │
│  │ argv     │   │ (Buffer +       │   │ (orchestrator)         │     │
│  │ parsing  │   │  cursor)        │   │                        │     │
│  └──────────┘   └─────────────────┘   └─────┬──────────────────┘     │
│       │                                      │                        │
│       │   ┌──────────────────────────────────┴────────┐              │
│       │   │  Sequential block readers (12, native order)│              │
│       │   │                                             │              │
│       │   │  1. header.ts        (magic + version)     │              │
│       │   │  2. settings.ts                            │              │
│       │   │  3. sounds.ts                              │              │
│       │   │  4. sprites.ts       ──┐                   │              │
│       │   │  5. backgrounds.ts   ──┼─▶ zlib.inflateSync│              │
│       │   │  6. paths.ts         ──┘                   │              │
│       │   │  7. scripts.ts       (GML plaintext)      │              │
│       │   │  8. datafiles.ts                           │              │
│       │   │  9. fonts.ts                               │              │
│       │   │  10. timelines.ts ───┐                    │              │
│       │   │  11. objects.ts   ───┼─▶ readActions()   ─┐│              │
│       │   │  12. rooms.ts        ┘   (DnD nodes)      ││              │
│       │   └─────────────────────────────────────────┐ ││              │
│       │                                             │ ▼▼              │
│       │   ┌──────────────────────────────────────┐  │ ┌────────────┐ │
│       │   │ data/action-ids.json                  │  └▶│ dnd-decode │ │
│       │   │ (ported from LateralGM XML)           │    │ + gml-xc   │ │
│       │   └───────────────────────────────────────┘    └─────┬──────┘ │
│       │                                                       │        │
│       ▼                                                       ▼        │
│  ┌────────────────────────────────────────────────────────────────┐  │
│  │ emit/ (one-file-per-resource writer)                            │  │
│  │  - sorted JSON, LF, 2-space indent (D-15)                       │  │
│  │  - sharp PNG encode (pinned options) for sprites/bgs            │  │
│  │  - .dnd.json + .gml side-by-side (D-09)                         │  │
│  │  - UNKNOWN-ACTIONS.md index for unmapped IDs (D-11)             │  │
│  └────────────────┬───────────────────────────────────────────────┘  │
└───────────────────┼─────────────────────────────────────────────────┘
                    ▼
          extracted/<source>/  (committed VCS tree)
            ├── settings.json
            ├── scripts/<NN>-<name>.gml
            ├── objects/<NN>-<name>/...
            ├── sprites/<NN>-<name>/{meta.json, frames/img_NN.png}
            ├── backgrounds/<NN>-<name>/...
            ├── rooms/<NN>-<name>/...
            ├── sounds/<NN>-<name>/...
            ├── fonts/<NN>-<name>/...
            ├── paths/<NN>-<name>.json
            ├── timelines/<NN>-<name>/...
            ├── datafiles/<NN>-<name>.<ext>
            ├── UNKNOWN-ACTIONS.md       (if any)
            └── MANIFEST.sha256

                    +
                    │  (separately, on CI)
                    ▼
          tools/extract-gmd verify <source>
            └─▶ recompute SHA256 tree → diff vs committed MANIFEST
                    │
                    ├─▶ pass → green
                    └─▶ fail → red: any byte drift fails the build
```

The diagram shows data flow. File-to-implementation mapping is in §Recommended Project Structure below.

### Recommended Project Structure

```
tools/extract-gmd/
├── package.json              # exact-pinned deps (sharp, tsx, etc.)
├── tsconfig.json             # strict TS, NodeNext
├── cli.ts                    # entry; argv parsing, dispatch (extract|verify)
├── src/
│   ├── reader/               # one file per top-level block (mirrors LateralGM)
│   │   ├── BinaryReader.ts   # Buffer + cursor + readBool/Byte/Int32LE/Double/Str/inflate
│   │   ├── header.ts         # magic 1234321 + version (530 expected for 5.3a)
│   │   ├── settings.ts       # block 2
│   │   ├── sounds.ts         # block 3
│   │   ├── sprites.ts        # block 4 (ZLIB pixel buffer + sub-image array)
│   │   ├── backgrounds.ts    # block 5 (ZLIB pixel buffer)
│   │   ├── paths.ts          # block 6
│   │   ├── scripts.ts        # block 7 (GML plaintext, length-prefixed)
│   │   ├── datafiles.ts      # block 8 (5.x feature, often called "Included Files")
│   │   ├── fonts.ts          # block 9 (metrics + raster glyphs)
│   │   ├── timelines.ts      # block 10 (moments → readActions)
│   │   ├── objects.ts        # block 11 (events → readActions)
│   │   ├── rooms.ts          # block 12 (instances + tiles + view config)
│   │   └── readProjectFile.ts # orchestrator: dispatch by version, call all 12 in order
│   ├── dnd/                  # DnD action node format (wiki/04)
│   │   ├── readAction.ts     # per-node binary read (Action_ID, Applies_To, ...)
│   │   ├── actionLookup.ts   # ID → LibAction descriptor (loads action-ids.json)
│   │   └── transcompile.ts   # action node → best-effort GML string
│   ├── emit/                 # one-file-per-resource writers
│   │   ├── tree.ts           # output dir layout helper
│   │   ├── json.ts           # sorted-keys JSON serializer (deterministic)
│   │   ├── png.ts            # sharp wrapper with pinned options
│   │   ├── manifest.ts       # SHA256 walk + sort + write MANIFEST.sha256
│   │   └── unknown-actions.ts# UNKNOWN-ACTIONS.md index writer
│   └── verify.ts             # `extract-gmd verify` subcommand
├── data/
│   └── action-ids.json       # ported from LateralGM library/default/*.lib (D-03)
├── tests/
│   ├── fixtures/
│   │   ├── tiny-empty.gmd    # hand-crafted: header + empty blocks
│   │   ├── tiny-script.gmd   # hand-crafted: one script
│   │   └── tiny-sprite.gmd   # hand-crafted: one 1x1 sprite, ZLIB pixel
│   ├── reader/               # per-block unit tests
│   ├── dnd/                  # action node + transcompile tests
│   ├── emit/                 # JSON sort, PNG byte-identity, manifest
│   ├── golden/               # snapshot trees for tiny fixtures
│   └── integration/
│       ├── extract-client.test.ts  # full BN Online Client 5-8.gmd run
│       └── extract-server.test.ts  # full BN Online Master 5-4.gmd run
└── README.md                 # short — points to decomp/TOOLS.md and decomp/wiki/15
```

### Pattern 1: Strict Sequential Block Walker

**What:** Twelve top-level blocks read in fixed order. The byte offset of block N+1 is determined by parsing block N to completion. There is no random-access table-of-contents.

**When to use:** Mandatory — `.gmd` format admits no other approach. Any code that "skips ahead" or computes block offsets by formula is wrong.

**Example:**
```typescript
// src/reader/readProjectFile.ts
// Source: github.com/IsmAvatar/LateralGM/blob/master/org/lateralgm/file/GmFileReader.java
// (readProjectFile method, version dispatch)
export function readProjectFile(buf: Buffer): ProjectFile {
  const r = new BinaryReader(buf);
  const identifier = r.readInt32LE();
  if (identifier !== 1234321) throw new Error('Not a .gmd file');
  const ver = r.readInt32LE();
  if (![530, 542, 600, 701, 800, 810].includes(ver)) {
    throw new Error(`Unsupported version: ${ver}`);
  }
  // For 5.3a we expect ver === 530 (and possibly 542 — verify on real input).
  const project: ProjectFile = { version: ver };
  project.settings    = readSettings(r, ver);     // block 2
  project.sounds      = readSounds(r, ver);       // block 3
  project.sprites     = readSprites(r, ver);      // block 4 (ZLIB)
  project.backgrounds = readBackgrounds(r, ver);  // block 5 (ZLIB)
  project.paths       = readPaths(r, ver);        // block 6
  project.scripts     = readScripts(r, ver);      // block 7 (GML)
  project.datafiles   = readDataFiles(r, ver);    // block 8 (v500+)
  project.fonts       = readFonts(r, ver);        // block 9
  project.timelines   = readTimelines(r, ver);    // block 10
  project.objects     = readObjects(r, ver);      // block 11 (DnD lives here)
  project.rooms       = readRooms(r, ver);        // block 12
  // Optional trailing blocks (game info, etc.) — parse defensively
  if (!r.atEnd()) project.gameInfo = readGameInfo(r, ver);
  return project;
}
```

### Pattern 2: Per-Resource "Exists" Flag + Native ID Preservation

**What:** Inside each block (e.g. sprites), the format stores a count, then for each ID `0..N-1` a single-byte "exists" boolean. If false, the ID is skipped (no further bytes for that slot). If true, the resource's data follows. Resource IDs are positional — slot 7 means "ID 7 forever," even if 0..6 are deleted. This is critical because rooms cross-reference objects by numeric ID.

**When to use:** Every resource block. The pattern is repeated 9 times (sounds, sprites, backgrounds, paths, scripts, fonts, timelines, objects, rooms).

**Example:**
```typescript
// src/reader/sprites.ts (canonical pattern)
// Source: GmFileReader.readSprites — github.com/IsmAvatar/LateralGM
export function readSprites(r: BinaryReader, ver: number): Sprite[] {
  const count = r.readInt32LE();
  const sprites: Sprite[] = [];
  for (let id = 0; id < count; id++) {
    const exists = r.readBool();
    if (!exists) continue;        // slot is empty but ID is consumed
    const name = r.readStr();     // ISO-8859-1, length-prefixed
    if (ver >= 800) r.skip(8);    // "last changed" timestamp — ignore for determinism
    const sver = r.readInt32LE(); // sprite version
    // ... bbox, origin, mask, subimages (ZLIB pixel buffer per subimage)
    sprites.push({ id, name, /* ... */ });
  }
  return sprites;
}
```

**CRITICAL:** Output filenames must preserve numeric IDs as zero-padded prefixes (e.g. `sprites/0017-sprPlayer/`) so:
1. Filesystem listing matches native source order (D-15).
2. Rooms cross-referencing object ID 17 still resolve unambiguously when sorted alphabetically by tooling.
3. Round-tripability is preserved without needing a separate ID-mapping file.

### Pattern 3: Side-by-Side DnD Descriptor + GML Transcompile (D-09)

**What:** For every Object event and Timeline moment that contains DnD action nodes, emit BOTH:
1. `<event-name>.dnd.json` — lossless descriptor (raw Action_ID, Applies_To, Is_Relative, Argument_Types[], Argument_Values[]).
2. `<event-name>.gml` — best-effort transcompile, with comment header pointing to descriptor.

The `.gml` is the read-primary diff target (humans skim faster). The `.dnd.json` is the round-trip canonical.

**Example:**
```typescript
// src/dnd/readAction.ts
// Source: decomp/wiki/04-dnd-serialization.md
export function readAction(r: BinaryReader): DnDAction {
  const actionId    = r.readInt32LE();
  const appliesTo   = r.readInt32LE();   // -1 self, -2 other, >=0 object ID
  const isRelative  = r.readBool();
  const argCount    = r.readInt32LE();
  const argTypes    = Array.from({ length: argCount }, () => r.readInt32LE());
  const argValues   = argTypes.map(t => readArgValue(r, t));
  return { actionId, appliesTo, isRelative, argTypes, argValues };
}

function readArgValue(r: BinaryReader, type: ArgType) {
  switch (type) {
    case ArgType.String:   return r.readStr();           // null-terminated OR length-prefixed
    case ArgType.Real:     return r.readDouble();        // 8-byte IEEE 754 LE
    case ArgType.Resource: return r.readInt32LE();       // resource ID
    case ArgType.Bool:     return r.readBool();
    default: throw new Error(`Unknown arg type: ${type}`);
  }
}
```

`[ASSUMED]` String encoding within DnD argument values may be either null-terminated or length-prefixed depending on action. LateralGM's `GmStreamDecoder` exposes both `readStr()` (4-byte length prefix) and `readStr1()` (1-byte length prefix); wiki/03 says "strings: usually null-terminated ASCII" but LateralGM uses length-prefixed. **The planner must verify against `GmFileReader.readActions` before committing.** Recommended verification: hand-decode one short DnD event from `BN Online Client 5-8.gmd` against LateralGM's GUI rendering of the same event.

### Pattern 4: Deterministic Output (D-15, D-16)

**What:** Every output byte is determined by the input. Defenses against drift:

| Surface | Defense |
|---------|---------|
| JSON | Sorted keys, 2-space indent, LF, no trailing newline drift, no `Date.now()` |
| GML | LF line endings, preserve original whitespace from `.gmd` plaintext |
| PNG | Pinned `sharp` (and surface libvips version) with `compressionLevel: 9, palette: false, effort: 10, progressive: false, adaptiveFiltering: false` |
| Filesystem | Resource enumeration order = source `.gmd`'s native order; **never** sort by name |
| MANIFEST | Sort SHA256 lines by relative path (POSIX, LF) for stable diff |

**Example:**
```typescript
// src/emit/json.ts
export function writeSortedJson(path: string, value: unknown): void {
  const out = JSON.stringify(sortKeys(value), null, 2) + '\n';
  fs.writeFileSync(path, out, { encoding: 'utf-8' });
}
function sortKeys(v: unknown): unknown {
  if (Array.isArray(v)) return v.map(sortKeys);
  if (v && typeof v === 'object') {
    return Object.fromEntries(Object.keys(v).sort().map(k => [k, sortKeys((v as any)[k])]));
  }
  return v;
}

// src/emit/png.ts
import sharp from 'sharp';
export async function writeDeterministicPng(path: string, w: number, h: number, rgba: Buffer): Promise<void> {
  await sharp(rgba, { raw: { width: w, height: h, channels: 4 } })
    .png({
      compressionLevel: 9,
      palette: false,
      effort: 10,
      progressive: false,
      adaptiveFiltering: false,
      // Do NOT pass `quality` — let libvips compute filter strategy from compressionLevel only.
    })
    .toFile(path);
}
```

`[ASSUMED]` `sharp.png()` with these options is byte-identical across Linux/macOS/Windows for the same input. **This is a known risk** — libvips PNG output has historically drifted between versions (sharp issue #3357 reset the palette default in v0.31.0). Mitigation: pin libvips version (surfaced via `sharp.versions.vips`), commit MANIFEST.sha256 from a single canonical machine first, and treat any cross-OS hash drift as a CI failure that re-pins from one platform until libvips is patched. Document the "canonical machine" choice in `decomp/TOOLS.md`.

### Anti-Patterns to Avoid

- **Sorting resources alphabetically in output filenames.** Breaks ID stability (rooms cross-reference object ID 17, which must still mean ID 17 after extraction). Use `<NN>-<name>/` prefix.
- **Skipping the version dispatch.** A reader that hard-codes "v530 layout" will silently misparse settings/objects in a `.gmd` saved by 5.4 or later (the format version persists across IDE upgrades).
- **Using UTF-8 for string decoding.** LateralGM uses ISO-8859-1 (cp1252-compatible). UTF-8 fails on extended Latin-1 chars (em-dashes, smart quotes) — likely present in BNO chat command names, room titles. **Hard pin to ISO-8859-1.**
- **Reading past block boundaries.** Each block reader must consume exactly N bytes; overruns cascade. Add `assert(r.cursor === expectedCursor)` checkpoints when LateralGM provides them.
- **Using `JSON.stringify` without key sort.** Default V8 key order is insertion order — fine for one machine, fragile across runs that build the object differently.
- **Treating `.gmd` and `.gb1` as different formats.** Per wiki/14, byte-identical. Phase 1 only handles the two `.gmd` files but the parser should NOT reject `.gb1` extensions if ever passed (Phase 3+ may want to extract older snapshots).
- **Hand-rolling a ZLIB inflate.** Use `node:zlib` `inflateSync` — Node ships zlib bindings. The format is RFC 1950, no GameMaker-specific framing.

## Don't Hand-Roll

| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| ZLIB decompression | Custom DEFLATE | `node:zlib` `inflateSync` | RFC 1950, ships in Node |
| PNG encoding | Custom IDAT writer | `sharp` (pinned) | Filter selection, CRC, gamma — fragile |
| Action_ID lookup table | Hand-derive from IDE experimentation | Port LateralGM `library/default/*.lib` XML files into `data/action-ids.json` (D-03) | LateralGM has done the empirical work; reproducing it = weeks |
| `.gmd` block walker | Read wiki/03 and write from scratch | Port `org.lateralgm.file.GmFileReader` line-by-line | Version dispatch quirks, off-by-ones, signed/unsigned int handling — wiki/03 is summary; `GmFileReader` is the truth |
| Binary primitive reader | Custom DataView wrapper | Port `GmStreamDecoder` semantics into `BinaryReader.ts` | Length-prefixed string ambiguity (`readStr` 4-byte vs `readStr1` 1-byte), inflate sub-stream framing |
| GML transcompile from DnD | Map every Action_ID by hand | Pull mappings from LateralGM `LibAction.execInfo` field where present | The mapping is encoded in the library XMLs |
| String encoding | UTF-8 default | **ISO-8859-1 hard-pinned** | Per `GmStreamDecoder` source — explicitly stated in code comment |
| SHA256 manifest format | Custom format | `sha256sum` POSIX style: `<hex>  <relpath>\n`, sorted lines | Compatible with `sha256sum -c MANIFEST.sha256` on Linux for sanity |

**Key insight:** Phase 1's risk surface is sequential parse drift — one wrong byte cascades. Every line of the parser should map 1:1 to a line in `GmFileReader.java`. Custom logic = custom bugs.

## Runtime State Inventory

> Phase 1 is greenfield (writes new `extracted/` tree, modifies `.gitignore`). It does NOT rename/refactor existing runtime systems. This section is included for completeness.

| Category | Items Found | Action Required |
|----------|-------------|------------------|
| Stored data | None — Phase 1 reads `.gmd` (read-only legacy), writes new `extracted/` tree. No databases involved. | None |
| Live service config | None — no live services in Phase 1. | None |
| OS-registered state | None. | None |
| Secrets / env vars | None. (Phase 1 does NOT touch `localList.txt` plaintext credentials — those stay in `legacy/`, gitignored, redacted later before any publication per CONCERNS.md.) | None |
| Build artifacts | None pre-existing. Phase 1 creates `tools/extract-gmd/node_modules/` (gitignored) and `extracted/` tree (committed). | New artifacts only |

**Nothing found in any category** — Phase 1 is the first new code in the repo. The only "state" mutation is creating two new top-level paths: `extracted/` (committed) and `.extracted-cache/` (gitignored, added to `.gitignore`).

## Common Pitfalls

### Pitfall 1: Modern decompiler trap (PITFALLS A1)

**What goes wrong:** Engineer reaches for UndertaleModTool / Altar.NET / GMS-era tooling on a 5.3a `.gmd`. Tool errors with "FORM header not found" because 5.3a has no FORM chunks — it's monolithic sequential ZLIB.
**Why it happens:** Modern decompilers parse Studio's `data.win` chunk format. 5.3a is an architecturally different beast (Delphi 5 stub + sequential payload).
**How to avoid:** Hard-coded ban in `decomp/TOOLS.md` (D-19): UTMT, Altar.NET, any Studio-era tool listed as "do not use." Phase 1 uses LateralGM's `GmFileReader` (Java reference) ported to TS — that is the only approach.
**Warning signs:** "FORM header not found" • "Unrecognized data format" • "Tool opens but asset count is 0"

### Pitfall 2: Sequential parse drift

**What goes wrong:** A block reader reads N+1 bytes instead of N (e.g. wrong version-dispatched branch). Every subsequent block reads garbage. Manifests as "block 7 (Scripts) parses but block 8 (Datafiles) explodes with a name length of 4 billion."
**Why it happens:** Version dispatch quirks (v530 vs v600 differ by skip-blocks), signed/unsigned int handling, length-prefixed vs null-terminated strings.
**How to avoid:**
1. Port `GmFileReader` line-by-line, not by transcribing wiki/03.
2. After each block, assert reader cursor matches LateralGM's expected post-state — derived from a small Java probe script.
3. Golden fixtures: hand-craft tiny `.gmd` files for each block in isolation (one-script `.gmd`, one-sprite `.gmd`, one-room `.gmd`) and snapshot-test.
4. Hex-dump the first 256 bytes of `BN Online Client 5-8.gmd` once and verify magic = 1234321 (0x4D 0xD9 0x12 0x00 LE) and version = 530 (0x12 0x02 0x00 0x00 LE).

**Warning signs:** Resource count of millions • String name with non-printable bytes • ZLIB inflate "incorrect header check" mid-block.

### Pitfall 3: ISO-8859-1 vs UTF-8 string encoding (NOT explicitly in PITFALLS)

**What goes wrong:** Decoder defaults to UTF-8 (Node `Buffer.toString()` default for `.toString('utf8')`); resource names containing extended Latin-1 chars (em-dash, smart quote, accented letters) decode as replacement characters, breaking round-trip.
**Why it happens:** LateralGM's `GmStreamDecoder` explicitly uses ISO-8859-1: *"ISO-8859-1 was the fixed charset in earlier LGM versions, so those parts of the code which have not been updated to set the charset explicitly should continue to use it to avoid regressions."*
**How to avoid:** Hard-code `r.readStr()` to use `Buffer.toString('latin1')` (Node alias for ISO-8859-1). Do NOT use `'utf8'`. Add a unit test that decodes a known extended byte (`0xA9` → `©`).
**Warning signs:** Resource names with `�` (replacement char) • DiffSnapshot fails on a name with an em-dash.

### Pitfall 4: Sharp / libvips PNG byte-drift across OS

**What goes wrong:** CI runner produces different PNG bytes than dev machine. Reproducibility check fails on a hash that was correct locally.
**Why it happens:** Sharp ships pre-built libvips per platform. Minor libvips version differences between platforms can change PNG filter selection (Z_FIXED vs custom Huffman, adaptive filtering heuristics).
**How to avoid:**
1. Pin sharp to one exact version, surface `sharp.versions.vips` in MANIFEST header.
2. Designate ONE canonical OS+arch for the committed MANIFEST (recommend Linux x64-musl since that's the deploy target and CI runs there).
3. CI matrix: extract on Linux x64 → assert hash match. Other platforms (mac/Windows dev) treat hash drift as a soft warning, not a CI failure.
4. Document this rule prominently in `decomp/TOOLS.md`.

**Warning signs:** Hash mismatch only on macOS or only on Windows-dev • Bytewise diff localized to PNG filter byte (offset 8 of each scanline).

### Pitfall 5: Loss of native source order (D-15)

**What goes wrong:** Output enumerated alphabetically (or by some helper that sorts internally). Rooms now cross-reference object ID 17 but the file at `objects/<seventeenth-alphabetical>/` is no longer the right one.
**Why it happens:** Default fs operations + framework helpers love to sort. Easy to miss.
**How to avoid:** Numeric prefix on every directory and file (`0017-objPlayer/`). Resource ID = position in source `.gmd`'s native order, not alphabetical. Unit test: extract a fixture with three resources whose names are `c.gml`, `a.gml`, `b.gml` (in that source order) and assert the output filenames are `0000-c.gml`, `0001-a.gml`, `0002-b.gml`.
**Warning signs:** Rooms reference IDs that don't resolve • Diff between two extraction runs shows resources in different order.

### Pitfall 6: Unknown Action_ID handling (D-11)

**What goes wrong:** A DnD event uses an Action_ID not in LateralGM's library XMLs (BNO might have used a non-stock action). Transcompile crashes or silently emits wrong GML.
**Why it happens:** LateralGM's lookup table covers stock GameMaker actions, not custom-author actions. 5.3a allows third-party libraries.
**How to avoid:** D-11 explicitly handles this: emit only `.dnd.json` and a stub `.gml` with `// UNKNOWN ACTION_ID=<id>; see .dnd.json`, and append to `extracted/<source>/UNKNOWN-ACTIONS.md`. Don't crash. Phase 2/3 follow up.
**Warning signs:** A non-empty UNKNOWN-ACTIONS.md is expected output, not a bug. Treat it as a forcing-function for Phase 2 documentation effort.

## Code Examples

### Read GMD header

```typescript
// src/reader/header.ts
// Source: GmFileReader.readProjectFile (github.com/IsmAvatar/LateralGM)
export function readHeader(r: BinaryReader): { version: number } {
  const id = r.readInt32LE();
  if (id !== 1234321) throw new Error(`Bad magic: 0x${id.toString(16)}`);
  const ver = r.readInt32LE();
  const supported = [530, 542, 600, 701, 800, 810];
  if (!supported.includes(ver)) throw new Error(`Unsupported version: ${ver}`);
  return { version: ver };
}
```

### Read sprite with ZLIB-compressed pixel buffer

```typescript
// src/reader/sprites.ts
// Source: GmFileReader.readSprites
import { inflateSync } from 'node:zlib';

export function readSprites(r: BinaryReader, ver: number): Sprite[] {
  const count = r.readInt32LE();
  const out: Sprite[] = [];
  for (let id = 0; id < count; id++) {
    if (!r.readBool()) continue;
    const name = r.readStr();
    if (ver >= 800) r.skip(8);                // last-changed timestamp; ignore
    const sver = r.readInt32LE();
    const width  = r.readInt32LE();
    const height = r.readInt32LE();
    // ... bbox, origin, mask, etc. (verify exact field order against GmFileReader)
    const subImageCount = r.readInt32LE();
    const frames: Buffer[] = [];
    for (let f = 0; f < subImageCount; f++) {
      const compressedLen = r.readInt32LE();
      const compressed    = r.readBytes(compressedLen);
      const raw           = inflateSync(compressed);  // RGBA or palette per ver
      frames.push(raw);
    }
    out.push({ id, name, version: sver, width, height, frames });
  }
  return out;
}
```

### Deterministic JSON write

```typescript
// src/emit/json.ts
import { writeFileSync } from 'node:fs';
export function writeJsonDeterministic(path: string, value: unknown): void {
  const sorted = sortKeysRecursive(value);
  const json = JSON.stringify(sorted, null, 2) + '\n';
  writeFileSync(path, json, 'utf8');  // Note: utf8 here is for the OUTPUT (filenames + script names are pure ASCII for safety)
}
function sortKeysRecursive(v: unknown): unknown {
  if (Array.isArray(v)) return v.map(sortKeysRecursive);
  if (v !== null && typeof v === 'object') {
    const o = v as Record<string, unknown>;
    return Object.fromEntries(Object.keys(o).sort().map(k => [k, sortKeysRecursive(o[k])]));
  }
  return v;
}
```

### SHA256 manifest

```typescript
// src/emit/manifest.ts
import { readdirSync, readFileSync, statSync, writeFileSync } from 'node:fs';
import { createHash } from 'node:crypto';
import { join, relative } from 'node:path';
import { posix } from 'node:path';

export function writeManifest(treeDir: string, sharpVersion: string, libvipsVersion: string): void {
  const lines: string[] = [];
  lines.push(`# extract-gmd MANIFEST`);
  lines.push(`# sharp=${sharpVersion}, libvips=${libvipsVersion}`);
  for (const f of walkSorted(treeDir)) {
    const rel = posix.normalize(relative(treeDir, f).replace(/\\/g, '/'));
    if (rel === 'MANIFEST.sha256') continue;
    const hash = createHash('sha256').update(readFileSync(f)).digest('hex');
    lines.push(`${hash}  ${rel}`);
  }
  writeFileSync(join(treeDir, 'MANIFEST.sha256'), lines.join('\n') + '\n', 'utf8');
}
function* walkSorted(dir: string): Generator<string> {
  const entries = readdirSync(dir).sort();   // POSIX-collated lexicographic
  for (const name of entries) {
    const full = join(dir, name);
    if (statSync(full).isDirectory()) yield* walkSorted(full);
    else yield full;
  }
}
```

## State of the Art

| Old Approach | Current Approach | When Changed | Impact |
|--------------|------------------|--------------|--------|
| Reach for UTMT / Altar.NET on a `.gmd` | LateralGM (Java) for parse, era-appropriate tools (GM Decompiler v2.1, GMD-Recovery) for `.exe`-only recovery | Always (5.3a never had Studio-era tooling support) | **Hard rule:** D-04 + wiki/13. Don't burn time on modern tools. |
| Ship raw BMP from sprites/backgrounds | Decode ZLIB pixel buffer → deterministic PNG via sharp | D-12 (this project) | EXT-04's "BMP/WAV/MIDI/TTF as found" wording is approximate — sprites are NOT stored as BMP, they're stored as raw RGBA/palette pixel buffers compressed with ZLIB. PNG is the right canonical decode. |
| Faithful round-trip via raw byte preservation | One-file-per-resource VCS tree (GmkSplitter pattern) with descriptor JSON for round-trip when needed | D-07, D-09 | Diffability beats round-trip for this rebuild. Original `.gmd` stays in `legacy/`. |
| Run extraction inside Windows XP VM | Headless TS Node CLI (D-01, D-04) | D-01 | Faster daily dev, CI-runnable, no Windows runtime. WinXP VM is now a Rank-3 documented fallback only. |

**Deprecated/outdated:**

- **UndertaleModTool / Altar.NET / GMS chunk-walking tooling:** Always wrong for 5.3a. Architectural mismatch (no FORM headers).
- **Plain BMP output for sprites:** Misreading of EXT-04. Sprites are not stored as BMP files; they are ZLIB-compressed pixel arrays.
- **Hand-rolling 39dll-like binary parsers from packet captures:** Different problem (Phase 3), but worth naming — the same "the call order IS the format" anti-pattern would apply if anyone tried to parse `.gmd` with a hex editor instead of porting `GmFileReader`.

## Assumptions Log

| # | Claim | Section | Risk if Wrong |
|---|-------|---------|---------------|
| A1 | LateralGM does not ship a documented headless-export CLI | §Standard Stack — Cross-check oracle | Verify procedure must be hand-built (small Java probe script or manual spot-check). If LateralGM CLI exists, planning should adopt it. |
| A2 | DnD argument string encoding is ambiguous (null-terminated vs length-prefixed) | §Pattern 3 | Wrong choice = misalignment cascade in DnD events. Mitigation: hand-decode one BNO event against LateralGM GUI before final parser commit. |
| A3 | `sharp.png()` with pinned options is byte-identical across OS/arch | §Pattern 4 | If false, MANIFEST hashes diverge cross-platform; mitigation already specified (canonical platform = Linux x64). |
| A4 | LateralGM library XML files contain enough Action_ID coverage for BNO's DnD events | §Don't Hand-Roll | Unknown Action_IDs become entries in UNKNOWN-ACTIONS.md (D-11 already handles). Risk = volume of UNKNOWN entries forces Phase 2 effort. |
| A5 | `.gmd` files for BNO use format version 530 (not 542) | §Pattern 1 | If 542 or other, version dispatch must add another branch. Mitigation: hex-dump first 8 bytes once, confirm version. |
| A6 | BNO's `.gmd` resource names are ASCII or ISO-8859-1 (no multi-byte chars) | §Pitfall 3 | UTF-8 sneak-in would silently break round-trip; ISO-8859-1 hard-pin defends. |
| A7 | `decomp/wiki/03` block ordering ("settings → sounds → sprites → bgs → paths → scripts → datafiles → fonts → timelines → objects → rooms") matches `GmFileReader` exactly | §Architecture Diagram | Wrong order = fatal parse error at first block. Verified visually against `GmFileReader.readProjectFile` partial fetch (block names match). HIGH confidence but not byte-verified. |
| A8 | Game info / extension blocks may follow rooms in v530 | §Pattern 1 | Defensive `if (!r.atEnd())` handles unexpected trailing blocks. |

## Open Questions

1. **LateralGM headless export procedure** (RESOLVED — see plan 07 Task 2 Probe.java; manual oracle ships per VALIDATION.md "Manual-Only Verifications")
   - What we know: LateralGM is a Swing GUI app. JAR runs in JVM 1.7+.
   - What's unclear: Whether `java -jar lateralgm-1.8.234.jar --export <project> <outdir>` or any equivalent CLI exists. The repo description hints at "exports forward to `.gmx`/`.gmk`" but doesn't document a CLI flag.
   - Recommendation: Plan for the worst case — write a 30-line Java probe script that loads `org.lateralgm.file.GmFileReader`, parses the `.gmd`, and prints "resource counts per block" to stdout. Use that as the CI cross-check oracle. Defer full headless export indefinitely; counts + a few sentinel names is enough.

2. **Action_ID coverage for BNO** (RESOLVED — see plan 03 Task 2: scripts/port-action-ids.ts ports LateralGM 1.8.234 library/default/*.lib reproducibly; UNKNOWN-ACTIONS.md surfaces remaining gaps as Phase 2 forcing function)
   - What we know: LateralGM `library/default/*.lib` XML files contain stock action definitions.
   - What's unclear: How many of BNO's DnD actions are stock vs author-custom. Likely high stock coverage given era and audience.
   - Recommendation: Run extraction with action lookup loaded, count UNKNOWN entries, decide in Phase 2 whether to back-fill manually (likely yes for a small N) or live with the descriptor-only path.

3. **`.gmd` v530 vs v542 in BNO files** (RESOLVED — see plan 07 Task 2: hex-dump of first 16 bytes recorded in decomp/TOOLS.md as Wave 0)
   - What we know: 5.3a saves report version 530.
   - What's unclear: Whether the two BNO `.gmd` files saved at version 530 specifically. `BN Online Master 5-4` filename suggests an internal "5-4" revision number unrelated to format version.
   - Recommendation: Hex-dump first 8 bytes of each file as Wave 0; commit the result to `decomp/TOOLS.md`.

4. **Datafile block ordering** (RESOLVED — see plan 02 Task 2 readDataFiles: defensive atEnd() check + version-guarded by orchestrator in plan 04)
   - What we know: Wiki/03 lists "Data Files" between Scripts and Fonts (block 8). LateralGM's `GmFileReader` includes "Included files (v700+)" in the sequence.
   - What's unclear: Whether 5.3a (v530) writes a Datafiles block at all, or skips it, or it lives elsewhere.
   - Recommendation: Read `GmFileReader.readDataFiles` for the version-dispatch — likely `if (ver >= 600) ...`. Skip safely on v530 if the block doesn't exist.

5. **`extracted/` git LFS or plain commit**
   - What we know: D-08 says `extracted/` is committed. PNG sprites are small (likely < 100 KB each), GML is text.
   - What's unclear: Total size of the extracted tree. If sprite count × frame count produces hundreds of MB of PNGs, plain git starts hurting.
   - Recommendation: Estimate after first run. If `extracted/` exceeds ~100 MB, consider git LFS for `*.png` only. Defer the decision to actual measurement; don't pre-optimize.

6. **Path of LateralGM library XMLs in repo** (RESOLVED — plan 03 Task 2 porter clones LateralGM 1.8.234 and reads org/lateralgm/resources/library/default/*.lib directly; no copy committed to repo)
   - What we know: `org/lateralgm/resources/library/default/` (in LateralGM repo) holds the XMLs.
   - What's unclear: Exact filenames for the GameMaker 5.x library.
   - Recommendation: Plan a one-time fetch step in Wave 0 — clone LateralGM at tag `1.8.234`, copy the relevant `default/*.lib` files into `tools/extract-gmd/data/lateralgm-libs/`, then transform into `data/action-ids.json`. Attribution comment in source tracks provenance.

## Environment Availability

| Dependency | Required By | Available | Version | Fallback |
|------------|------------|-----------|---------|----------|
| Node.js 22 LTS | extract-gmd CLI | `[VERIFIED: local probe — v25.8.2 present]` | 25.8.2 (local box runs newer; install 22 LTS for parity) | Node 25.x works for Phase 1 specifically (no engine-pinned native modules); install 22 before Phase 4 |
| npm | dep install | ✓ | bundled | — |
| pnpm | D-18 (`pnpm tsx ...`) | `[VERIFIED: local probe — pnpm: command not found]` ✗ | — | Install pnpm globally before running CLI: `npm install -g pnpm`. Or invoke directly via `npx tsx tools/extract-gmd/cli.ts` until pnpm available. |
| Java 1.7+ (LateralGM oracle) | D-02 verify-extract | `[VERIFIED: local probe — openjdk 1.8.0_392]` ✓ | 1.8.0_392 | — |
| sharp (libvips) | D-12 PNG encode | `[VERIFIED: npm view sharp version → 0.34.5]` ✓ via npm install | 0.34.5 | — |
| Git | Commit `extracted/` tree | ✓ (project is in git per CLAUDE.md flow) | — | — |

**Missing dependencies with no fallback:** None.

**Missing dependencies with fallback:** `pnpm` (use `npx tsx ...` instead, or globally install). Document the install in `decomp/TOOLS.md` setup section.

## Validation Architecture

### Test Framework
| Property | Value |
|----------|-------|
| Framework | vitest 4.1.5 (per project STACK.md) |
| Config file | `tools/extract-gmd/vitest.config.ts` (Wave 0 — does not yet exist) |
| Quick run command | `cd tools/extract-gmd && npx vitest run --reporter=basic` |
| Full suite command | `cd tools/extract-gmd && npx vitest run` (includes integration tests against real BNO `.gmd` files) |

### Phase Requirements → Test Map

| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|--------|----------|-----------|-------------------|-------------|
| EXT-01 | Client `.gmd` → all GML scripts as one-file-per-script | integration | `npx vitest run tests/integration/extract-client.test.ts` | ❌ Wave 0 |
| EXT-02 | DnD action graphs serialized + diffable | integration | `npx vitest run tests/dnd/` | ❌ Wave 0 |
| EXT-03 | All room layouts emitted (objects, tiles, bgs, instance positions) | integration | `npx vitest run tests/integration/extract-client.test.ts -t rooms` | ❌ Wave 0 |
| EXT-04 | All sprites/sounds/fonts/bgs as native binary | integration + unit (PNG byte-identity) | `npx vitest run tests/emit/png.test.ts tests/integration/extract-client.test.ts -t assets` | ❌ Wave 0 |
| EXT-05 | Server `.gmd` → same shape as client | integration | `npx vitest run tests/integration/extract-server.test.ts` | ❌ Wave 0 |
| EXT-06 | Per-resource committed text/binary, no monolithic blobs | structural / lint | `npx vitest run tests/integration/tree-shape.test.ts` (asserts no file > N MB, all expected dirs exist) | ❌ Wave 0 |
| EXT-07 | Re-running produces byte-identical output | reproducibility (golden) | `npx vitest run tests/integration/reproducibility.test.ts` (extracts twice, diffs SHA256) | ❌ Wave 0 |
| EXT-08 | `decomp/TOOLS.md` documents era-appropriate stack | manual review + lint | grep for required headings (`Rank 1`, `Rank 2`, `Rank 3`, `tools/extract-gmd`); manual cold-read | ❌ Wave 0 (lint script) |

### Sampling Rate

- **Per task commit:** `npx vitest run --reporter=basic` (unit + golden fixtures only, ~5s).
- **Per wave merge:** `npx vitest run` (unit + integration on real `.gmd` files, ~30s).
- **Phase gate:** Full suite green + `tools/extract-gmd verify extracted/client-5-8` + `verify extracted/server-5-4` both pass + LateralGM oracle spot-check passes (manual or scripted).

### Wave 0 Gaps

- [ ] `tools/extract-gmd/vitest.config.ts` — vitest config (NodeNext, single thread for fs determinism)
- [ ] `tools/extract-gmd/tests/fixtures/tiny-empty.gmd` — hand-crafted minimal `.gmd` (header + 12 empty blocks)
- [ ] `tools/extract-gmd/tests/fixtures/tiny-script.gmd` — hand-crafted: header + empty blocks + 1 script
- [ ] `tools/extract-gmd/tests/fixtures/tiny-sprite.gmd` — hand-crafted: header + 1 ZLIB sprite frame
- [ ] `tools/extract-gmd/tests/fixtures/build-fixtures.ts` — script that BUILDS the tiny `.gmd` files programmatically (so they can be regenerated and stay deterministic)
- [ ] `tools/extract-gmd/tests/golden/<fixture-name>/` — committed expected output trees for snapshot diffing
- [ ] `tools/extract-gmd/tests/integration/extract-client.test.ts` — runs full extractor against `legacy/open-source-release/BN Online Client 5-8.gmd`, asserts top-level resource counts + sentinel resource names match LateralGM
- [ ] `tools/extract-gmd/tests/integration/extract-server.test.ts` — same for Master 5-4
- [ ] `tools/extract-gmd/tests/integration/reproducibility.test.ts` — extracts twice to two temp dirs, asserts SHA256 manifests match
- [ ] `tools/extract-gmd/tests/integration/tree-shape.test.ts` — asserts no monolithic blobs (per-file size cap), expected directory structure
- [ ] `tools/extract-gmd/tests/dnd/known-actions.test.ts` — for known BNO Action_IDs, asserts transcompile produces expected GML
- [ ] Framework install: `npm install --save-dev vitest@4.1.5` (one-time)
- [ ] LateralGM oracle script: `tools/verify-extract/probe.java` — 30-line Java probe printing resource counts per block (Wave 0 OR deferred to Wave 2 after TS extractor green)

### Validation Dimensions (Nyquist)

| Dimension | What it catches | How tested |
|-----------|------------------|------------|
| **Parser correctness** | Wrong byte counts, wrong field order, version-dispatch bugs | Golden fixtures: hand-crafted tiny `.gmd` files with known content; snapshot tests on output tree |
| **Reproducibility** (EXT-07) | Non-determinism (timestamps, key ordering, libvips drift) | `reproducibility.test.ts`: extract twice, diff SHA256 manifests; CI cross-OS check (Linux x64 canonical, mac/Windows informational) |
| **Coverage** (EXT-01..05) | Missing block types, untested resource categories | Per-block unit test in `tests/reader/`; per-emit unit test in `tests/emit/`; integration test asserts presence of every expected directory category in real-`.gmd` output |
| **Oracle agreement** (D-02) | TS extractor drift from LateralGM canonical parser | Java probe script prints resource counts + sentinel names; TS test asserts same counts and same first-N names per block |
| **Round-trip safety** (D-09, D-10) | Lossy DnD transcompile | For DnD events: descriptor JSON is canonical; assert that parsing the descriptor → re-encoding produces the same bytes (when round-trip helper is added; defer if scope-creep) |
| **Action lookup completeness** (D-11) | Unknown Action_IDs silently lost | `UNKNOWN-ACTIONS.md` is created if any unknown ID; integration test asserts the file is committed when expected (or absent when fully-mapped) |
| **String encoding** | UTF-8 vs ISO-8859-1 confusion | Unit test: decode a buffer with byte 0xA9 (©) via `BinaryReader.readStr` and assert the JS string is `"©"` |

## Security Domain

> Phase 1 has limited security surface — it's an offline read-only parser writing to local filesystem. No auth, no user input, no network.

### Applicable ASVS Categories

| ASVS Category | Applies | Standard Control |
|---------------|---------|-----------------|
| V2 Authentication | no | N/A — offline tool |
| V3 Session Management | no | N/A |
| V4 Access Control | no | N/A |
| V5 Input Validation | yes | Validate `.gmd` file is well-formed before walking; reject non-`.gmd` inputs (magic-number check). Bound all length-prefixed reads to file size to prevent OOM on a malformed input. |
| V6 Cryptography | yes (passive) | SHA256 manifest only — use `node:crypto` (no hand-rolling). |
| V7 Error Handling | yes | Don't leak absolute paths from the extractor's working directory in error messages (D-15 forbids absolute paths in output). |
| V14 Configuration | yes | `.gitignore` MUST add `.extracted-cache/`. `legacy/` MUST stay gitignored (already is). Verify extracted output never contains plaintext credentials from `localList.txt` (it shouldn't — extractor reads `.gmd` only, but a sanity grep on the output tree for `harrypotter` / `bahoobutt` / known plaintext passwords from CONCERNS.md should be a CI lint). |

### Known Threat Patterns for Phase 1

| Pattern | STRIDE | Standard Mitigation |
|---------|--------|---------------------|
| Malicious `.gmd` triggers OOM via huge length-prefix | DoS | Bound all `readBytes(n)` calls to (file-size − cursor); fail fast |
| Path traversal in resource names (e.g., name = `../../../etc/passwd`) | Tampering | Sanitize all resource names before using as path components: strip `/`, `\`, `..`; reject names with control chars. |
| Plaintext credentials from `legacy/` accidentally committed via extracted tree | Information Disclosure | Phase 1 reads `.gmd` only, NOT `localList.txt`. Add a CI lint that greps `extracted/` for known plaintext passwords from CONCERNS.md (`harrypotter`, `bahoobutt`, `Jarhead111`, etc.) and fails if any match. Cheap insurance. |
| ZLIB bomb (small compressed → multi-GB inflated) | DoS | Cap inflated size at 64 MB per buffer (or per-frame for sprites). 5.3a sprite frames are tiny in practice. |

## Project Constraints (from CLAUDE.md)

- **TS-everywhere mandate:** Even tools (D-01, D-17 confirmed). No Python/Bash extractors.
- **Repo stays private through Phase 7:** Affects publication of `extracted/` tree — currently fine since repo is private. CONCERNS.md scrub (cracked software, plaintext creds) is required before any publication, but that work belongs to Phase 7+ legal-prep, NOT Phase 1.
- **Source-of-truth files locked:** Only `BN Online Client 5-8.gmd` and `BN Online Master 5-4.gmd` are in scope. Older `.gmd`/`.gb1` extraction is explicitly deferred (D-04 deferred ideas).
- **No new TypeScript outside `tools/` before Phase 4:** Phase 1 lives entirely under `tools/extract-gmd/`. Don't drift into `apps/` or `packages/`.
- **`legacy/` is read-only after Phase 1:** No code under `tools/extract-gmd` may write to `legacy/`. Outputs go to `extracted/`.
- **`.gitignore` discipline:** Add `.extracted-cache/`; verify `legacy/` stays gitignored; verify `tools/extract-gmd/node_modules/` is gitignored (general `node_modules/` rule covers this).
- **PITFALLS A1, A2, A6, A7, D6 must be addressed:**
  - A1 (UTMT trap): hard ban in `decomp/TOOLS.md`.
  - A2 (XOR-then-ZLIB order): N/A for Phase 1 — we have `.gmd` directly, not `.exe`. The XOR layer is between `.exe` and `.gmd`; we start from `.gmd`.
  - A6 (`.gb1`–`.gb9`): explicitly out of scope per D-04 deferred ideas. Document in `decomp/TOOLS.md` that `.gb1` extraction uses the same parser when re-engaged.
  - A7 (WinXP VM): documented as Rank-3 fallback only (D-04, D-19); no VM build in Phase 1.
  - D6 (`legacy/` vs `extracted/` split): D-05 + D-08 establish this. No Phase 4+ code imports from `legacy/` — Phase 1 is the discipline-setting moment.

## Recommended Task Decomposition (high-level)

The planner will refine, but a natural decomposition is roughly 5 waves:

**Wave 0 — Scaffolding & Fixtures**
- Init `tools/extract-gmd/` with `package.json`, `tsconfig.json`, `vitest.config.ts`, exact-pinned deps.
- Add `.extracted-cache/` to `.gitignore`.
- Hex-dump first 16 bytes of both `.gmd` files; record version (likely 530) in `decomp/TOOLS.md`.
- Hand-craft (programmatically) tiny golden fixture `.gmd` files (`tiny-empty`, `tiny-script`, `tiny-sprite`).
- Stub LateralGM oracle: clone LateralGM 1.8.234, identify `library/default/*.lib`, import into `data/lateralgm-libs/`, transform into `data/action-ids.json`.

**Wave 1 — Binary Reader + Header**
- `BinaryReader.ts` with all primitives mirroring `GmStreamDecoder` (readBool, readByte, readInt32LE, readDouble, readStr (4-byte length, ISO-8859-1), readStr1, readBytes, decompress for length-prefixed inflate, skip).
- `header.ts` with magic + version check.
- Tests: unit tests on each primitive; encode/decode round-trip on the tiny fixtures.

**Wave 2 — Block Readers (parallelizable)**
- One file per block: `settings.ts`, `sounds.ts`, `sprites.ts` (with ZLIB inflate), `backgrounds.ts` (ditto), `paths.ts`, `scripts.ts`, `datafiles.ts`, `fonts.ts`, `timelines.ts`, `objects.ts` (calls into `dnd/`), `rooms.ts`.
- `readProjectFile.ts` orchestrator.
- Per-block unit tests against tiny fixtures.
- DnD subsystem: `readAction.ts`, `actionLookup.ts` (loads `action-ids.json`), `transcompile.ts`.
- **CHECKPOINT:** Run extractor against real `BN Online Client 5-8.gmd`. If it crashes, debug version dispatch / byte counts.

**Wave 3 — Emit Layer**
- `tree.ts` (output dir layout helper, numeric-prefix naming).
- `json.ts` (sorted-keys deterministic JSON).
- `png.ts` (sharp wrapper with pinned options).
- `manifest.ts` (SHA256 walk + sort + write).
- `unknown-actions.ts` (UNKNOWN-ACTIONS.md index).
- Per-emit unit tests including PNG byte-identity test (encode same RGBA twice, compare bytes).

**Wave 4 — CLI + Verify**
- `cli.ts`: argv parsing, dispatch `extract <input.gmd> <out-dir>` and `verify <source-dir>`.
- `verify.ts`: recompute manifest, diff against committed.
- Repo-level `pnpm extract:all` script (root `package.json`).
- `decomp/TOOLS.md`: thin wrapper around `decomp/wiki/15`, links + installation/provenance notes + `tools/extract-gmd` documentation.

**Wave 5 — End-to-End on Real Files + Polish**
- Full extraction of both `.gmd` files committed to `extracted/`.
- Reproducibility test: extract twice, diff hashes.
- LateralGM oracle: Java probe script prints resource counts; TS test asserts match.
- UNKNOWN-ACTIONS.md review: any unmapped Action_IDs documented as Phase 2 follow-up.
- CI integration deferred to Phase 5 (when GitHub Actions arrive); for now, document the verify command in README.

## References

### Primary (HIGH confidence)
- `decomp/wiki/00-overview.md` — engine identity, RE goals
- `decomp/wiki/03-gmd-format.md` — top-level block layout, sequential parsing rules, ZLIB blocks, ID conventions, IEEE 754 doubles, Int32 LE
- `decomp/wiki/04-dnd-serialization.md` — per-node binary layout, Action_ID lookup, GML transcompile guidance
- `decomp/wiki/05-gml-vm.md` — GML script semantics (informs transcompile target)
- `decomp/wiki/06-gml-syntax-5x.md` — 5.x syntax (no C-style array decls, dynamic arrays, every line must be assignment/call) — relevant for transcompile output validity
- `decomp/wiki/07-gml-core-functions.md` — `external_define`, `collision_line`, `sprite_add_alpha` patterns to recognize in extracted GML
- `decomp/wiki/11-tool-lateralgm.md` — `LibReader.java` (actually `GmFileReader.java` per repo browse) is canonical reference parser
- `decomp/wiki/12-tool-gmksplitter.md` — VCS-tree pattern (target shape for D-07)
- `decomp/wiki/13-modern-tool-incompat.md` — UTMT/Altar.NET will fail
- `decomp/wiki/14-gb1-backups.md` — `.gb1` byte-identical to `.gmd`
- `decomp/wiki/15-extraction-pipeline.md` — Rank 1-4 procedure (informs `decomp/TOOLS.md`)
- `.planning/PROJECT.md` — vision, constraints, Tech Stack
- `.planning/REQUIREMENTS.md` §Extraction — EXT-01..08 acceptance criteria
- `.planning/ROADMAP.md` §Phase 1 — goal + 5 success criteria
- `.planning/research/STACK.md` — version pins (Node 22, TypeScript 5.6+, sharp, vitest, tsx)
- `.planning/research/ARCHITECTURE.md` — `tools/extract-gmd` topology, three-pipeline split
- `.planning/research/PITFALLS.md` §A1 (UTMT trap), §A2 (XOR-ZLIB order — N/A here, we have .gmd), §A6 (.gb1 backups — deferred), §A7 (WinXP VM — fallback), §D6 (legacy/ vs extracted/ split)
- `.planning/codebase/CONCERNS.md` §Legal/IP, §Security (plaintext creds in legacy/), §Data Integrity (lossy DnD decompile)
- LateralGM `org.lateralgm.file.GmFileReader.java` — canonical reference parser (verified version dispatch supports 530, 542, 600, 701, 800, 810; uses ISO-8859-1; magic = 1234321; ZLIB pixel buffer support; `readActions()` method for DnD)
- LateralGM `org.lateralgm.file.GmStreamDecoder.java` — primitive reader (verified: readStr 4-byte length, readStr1 1-byte length, readBool, decompress, beginInflate/endInflate, ISO-8859-1)
- LateralGM `org.lateralgm.resources.library.LibAction.java` — verified field shape (id, parentId, parent, name, actionKind, interfaceKind, execType, execInfo, libArguments[])

### Secondary (MEDIUM confidence)
- LateralGM v1.8.234 release (June 2021) — [github.com/IsmAvatar/LateralGM/releases](https://github.com/IsmAvatar/LateralGM/releases)
- libvips PNG output options — [libvips.org/API/current/method.Image.pngsave.html](https://www.libvips.org/API/current/method.Image.pngsave.html)
- sharp PNG palette regression issue (informs determinism risk) — [github.com/lovell/sharp/issues/3357](https://github.com/lovell/sharp/issues/3357)
- npm registry version probes (sharp 0.34.5, tsx 4.21.0, vitest 4.1.5, @types/node 25.6.0, typescript via tsx) — verified 2026-05-02
- Local environment probes — Node v25.8.2, OpenJDK 1.8.0_392, pnpm absent

### Tertiary (LOW confidence — flagged for validation)
- LateralGM headless export CLI existence — not documented in repo description; `[ASSUMED]` workaround is custom Java probe script
- Sharp/libvips byte-determinism across OS — pinning options is documented but no public reproducible-build benchmark across Linux/macOS/Windows was found

## Metadata

**Confidence breakdown:**
- Standard stack: HIGH — all versions verified via npm registry 2026-05-02; alternatives explicitly rejected by D-01 / wiki/13
- Architecture (12-block walker, side-by-side DnD descriptor+GML, native-order preservation): HIGH — directly mirrors `GmFileReader.java` (verified portions) + wiki/03+04
- Pitfalls (sequential drift, ISO-8859-1, libvips drift, native order): HIGH for first three (verified in source/code); MEDIUM for libvips drift (no public bench, but mitigated by canonical-platform rule)
- Determinism strategy: MEDIUM — pinned options are well-documented, but cross-OS byte-identity for PNG is `[ASSUMED]` and represents the largest open risk
- Action_ID lookup table: MEDIUM — the table exists in LateralGM XML resources, exact filename and version-targeting not yet verified
- LateralGM oracle approach: LOW — no documented headless CLI; `[ASSUMED]` workaround is a custom Java probe script

**Research date:** 2026-05-02
**Valid until:** 2026-06-02 (30 days — `.gmd` format is dead/static; only LateralGM updates and sharp/libvips revisions could move the needle, both of which the MANIFEST header surfaces)

## RESEARCH COMPLETE