# Phase 3: Diagnostic Features - Research

**Researched:** 2026-04-12
**Domain:** Image processing (pixel comparison, overlay compositing, GIF encoding)
**Confidence:** HIGH

## Summary

Phase 3 adds three diagnostic capabilities to captured frame sequences: delta highlighting (DIAG-01), idle frame compression (DIAG-02), and animated GIF export (DIAG-03). All three operate on the existing `CaptureFrame[]` array between capture completion and grid compilation.

The core technical challenge is pixel-level frame comparison using sharp's raw pixel buffer extraction. Both delta highlighting and idle compression share the same underlying operation: comparing consecutive frames pixel-by-pixel. The GIF export uses `@skyra/gifenc` to encode raw RGBA pixel data into an animated GIF with per-frame delay timing.

**Primary recommendation:** Build a shared pixel comparison utility that both delta-highlighter and idle-compressor consume, then wire three new processing modules into the existing pipeline as pre-compilation transforms.

<user_constraints>
## User Constraints (from CONTEXT.md)

### Locked Decisions
- D-01: Compare consecutive frame pixel buffers with semi-transparent red overlay on changed regions above threshold
- D-02: Per-pixel RGB distance calculation for comparison
- D-03: Delta overlay composited onto each frame BEFORE grid compilation
- D-04: First frame has no delta overlay
- D-05: Threshold fixed at sensible default (~10% pixel difference), not user-configurable in v1
- D-06: `delta_highlight: boolean` parameter on `start_capture` (default false)
- D-07: Compare consecutive frames for near-identity (all pixels below threshold)
- D-08: Identical frames collapsed into single grid cell with "unchanged for N.Ns" label
- D-09: Collapsed cell uses first frame of identical sequence
- D-10: Compression happens BEFORE grid compilation (reduce frame array, then lay out)
- D-11: `compress_idle: boolean` parameter on `start_capture` (default false)
- D-12: Both features can be enabled simultaneously; compression runs first, then delta highlighting
- D-13: Use @skyra/gifenc for GIF encoding
- D-14: `gif_export: boolean` parameter on `start_capture` (default false)
- D-15: Completed session produces both grid JPEG AND animated GIF when gif_export is true
- D-16: GIF exposed as MCP resource at `capture://{sessionId}/gif` with mime type `image/gif`
- D-17: GIF frame delay calculated from actual elapsedMs between frames
- D-18: GIF frames resized to max 800px wide
- D-19: GIF generation happens after grid compilation
- D-20: All three features are optional boolean flags on start_capture
- D-21: Flags stored in CaptureConfig and passed through pipeline
- D-22: get_capture_status returns `gifUri` field when gif_export enabled and complete
- D-23: Grid resource and GIF resource are independent

### Claude's Discretion
- Exact pixel comparison algorithm (simple RGB distance vs perceptual difference)
- Delta highlight color (red suggested, any high-contrast overlay works)
- GIF color quantization strategy (256-color palette handling)
- Whether to add gif_quality parameter or hardcode sensible defaults
- Internal buffer handling for delta computation (in-memory vs streaming)

### Deferred Ideas (OUT OF SCOPE)
None
</user_constraints>

<phase_requirements>
## Phase Requirements

| ID | Description | Research Support |
|----|-------------|------------------|
| DIAG-01 | User can enable delta highlighting to show regions that changed between consecutive frames | Sharp raw pixel extraction + per-pixel RGB distance + semi-transparent overlay compositing |
| DIAG-02 | Idle frame compression collapses sequences of identical frames into one with "unchanged for Ns" label | Same pixel comparison as DIAG-01 with aggregate check + timestamp overlay pattern from existing code |
| DIAG-03 | User can optionally export the capture sequence as an animated GIF | @skyra/gifenc GifEncoder with raw RGBA pixel data from sharp, per-frame delay from elapsedMs |
</phase_requirements>

## Standard Stack

### Core (already installed)
| Library | Version | Purpose | Why Standard |
|---------|---------|---------|--------------|
| sharp | 0.34.5 | Raw pixel extraction, overlay compositing, resize | Already in project; `.raw().toBuffer()` gives pixel-level access; `.composite()` for overlays [VERIFIED: npm registry] |

### New Dependency
| Library | Version | Purpose | Why Standard |
|---------|---------|---------|--------------|
| @skyra/gifenc | 1.0.1 | Animated GIF encoding | Per CLAUDE.md tech stack decision; accepts raw Uint8ClampedArray pixel data; per-frame delay support [VERIFIED: npm registry] |

### Alternatives Considered
| Instead of | Could Use | Tradeoff |
|------------|-----------|----------|
| @skyra/gifenc | gifencoder | gifencoder unmaintained 7+ years; @skyra/gifenc is the locked decision |
| Per-pixel RGB distance | Perceptual color distance (CIEDE2000) | Overkill for "did this region change" detection; RGB Euclidean distance sufficient |

**Installation:**
```bash
npm install @skyra/gifenc@^1.0.1
```

## Architecture Patterns

### Recommended Project Structure
```
src/
├── processing/
│   ├── pixel-compare.ts      # Shared: raw pixel extraction + comparison
│   ├── delta-highlighter.ts   # DIAG-01: overlay changed regions
│   ├── idle-compressor.ts     # DIAG-02: collapse identical frames
│   ├── gif-exporter.ts        # DIAG-03: animated GIF encoding
│   ├── grid-compiler.ts       # Existing: grid assembly
│   └── timestamp-overlay.ts   # Existing: timestamp text overlays
├── types.ts                   # Extended: new config flags
└── server.ts                  # Extended: new tool params + GIF resource
```

### Pattern 1: Shared Pixel Comparison Utility
**What:** Extract raw RGBA pixel data from PNG frame buffers using sharp, then compare per-pixel with RGB Euclidean distance.
**When to use:** Both delta highlighting and idle compression need frame-to-frame comparison.
**Example:**
```typescript
// Source: sharp docs (https://sharp.pixelplumbing.com/api-output)
import sharp from "sharp";

interface RawFrame {
  data: Buffer;
  width: number;
  height: number;
  channels: number;
}

async function extractRawPixels(pngBuffer: Buffer): Promise<RawFrame> {
  const { data, info } = await sharp(pngBuffer)
    .ensureAlpha()       // Guarantee 4 channels (RGBA)
    .raw()
    .toBuffer({ resolveWithObject: true });
  return { data, width: info.width, height: info.height, channels: info.channels };
}

/**
 * Compare two frames pixel-by-pixel.
 * Returns a boolean mask (one byte per pixel: 0=same, 255=changed).
 * Also returns the fraction of pixels that changed (0.0-1.0).
 */
function compareFrames(
  a: RawFrame,
  b: RawFrame,
  threshold: number = 25  // ~10% of 255
): { mask: Uint8Array; changedFraction: number } {
  const pixelCount = a.width * a.height;
  const mask = new Uint8Array(pixelCount);
  let changed = 0;

  for (let i = 0; i < pixelCount; i++) {
    const offset = i * 4; // RGBA
    const dr = a.data[offset] - b.data[offset];
    const dg = a.data[offset + 1] - b.data[offset + 1];
    const db = a.data[offset + 2] - b.data[offset + 2];
    const dist = Math.sqrt(dr * dr + dg * dg + db * db);
    if (dist > threshold) {
      mask[i] = 255;
      changed++;
    }
  }

  return { mask, changedFraction: changed / pixelCount };
}
```
[VERIFIED: sharp `.raw().toBuffer({ resolveWithObject: true })` API from sharp official docs]

### Pattern 2: Delta Highlight Overlay
**What:** Convert the boolean change mask into a semi-transparent red RGBA overlay, then composite onto the frame.
**When to use:** DIAG-01 delta highlighting.
**Example:**
```typescript
// Source: sharp composite docs (https://sharp.pixelplumbing.com/api-composite)
function createHighlightOverlay(
  mask: Uint8Array,
  width: number,
  height: number
): Buffer {
  // Create RGBA buffer: red channel overlay where mask=255
  const rgba = Buffer.alloc(width * height * 4);
  for (let i = 0; i < mask.length; i++) {
    if (mask[i] === 255) {
      const offset = i * 4;
      rgba[offset] = 255;     // R
      rgba[offset + 1] = 0;   // G
      rgba[offset + 2] = 0;   // B
      rgba[offset + 3] = 100; // A (~40% opacity)
    }
    // Unchanged pixels remain transparent (0,0,0,0)
  }
  return rgba;
}

async function applyHighlight(
  framePng: Buffer,
  overlayRgba: Buffer,
  width: number,
  height: number
): Promise<Buffer> {
  return sharp(framePng)
    .composite([{
      input: overlayRgba,
      raw: { width, height, channels: 4 },
      blend: "over",
    }])
    .png()
    .toBuffer();
}
```
[VERIFIED: sharp `.composite()` with `raw` input from sharp composite docs]

### Pattern 3: GIF Encoding from Raw Pixels
**What:** Use @skyra/gifenc GifEncoder with raw RGBA pixel data extracted via sharp.
**When to use:** DIAG-03 GIF export.
**Example:**
```typescript
// Source: @skyra/gifenc docs (https://skyra-project.github.io/gifenc/classes/GifEncoder.html)
import { GifEncoder } from "@skyra/gifenc";
import { buffer as streamToBuffer } from "node:stream/consumers";

async function createGif(
  frames: { rgba: Uint8ClampedArray; delayMs: number }[],
  width: number,
  height: number
): Promise<Buffer> {
  const encoder = new GifEncoder(width, height);
  const stream = encoder.createReadStream();
  const bufferPromise = streamToBuffer(stream);

  encoder.setRepeat(0);  // Loop forever
  encoder.setQuality(10); // Default quality balance

  encoder.start();
  for (const frame of frames) {
    encoder.setDelay(Math.max(10, frame.delayMs)); // min 10ms per GIF spec
    encoder.addFrame(frame.rgba);
  }
  encoder.finish();

  return Buffer.from(await bufferPromise);
}
```
[CITED: https://skyra-project.github.io/gifenc/classes/GifEncoder.html]

### Pattern 4: Processing Pipeline Order
**What:** The processing pipeline transforms the raw frame array before grid compilation.
**When to use:** Always -- this is the integration architecture.
```
Raw CaptureFrame[] 
  -> (if compress_idle) idle-compressor: reduce array, add "unchanged" labels
  -> (if delta_highlight) delta-highlighter: overlay change regions on each frame
  -> grid-compiler: compile to JPEG grid (existing)
  -> (if gif_export) gif-exporter: encode frames to animated GIF
```
Per D-12: compression runs first so delta highlighting operates on the compressed (meaningful) sequence.
Per D-19: GIF generation uses the same processed frame array but runs after/alongside grid compilation.

### Anti-Patterns to Avoid
- **Comparing compressed JPEG/PNG buffers directly:** Compression artifacts make byte-level comparison meaningless. Always decode to raw pixels first.
- **Processing at full capture resolution:** Frames may be large (1920x1080+). For comparison, resize to cell dimensions first to save memory and CPU. GIF export also resizes to max 800px.
- **Holding all raw pixel buffers in memory simultaneously:** Process frame pairs sequentially (current vs previous), not all at once. For a 50-frame capture at 1080p RGBA, that would be ~400MB.

## Don't Hand-Roll

| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| GIF color quantization | Custom 256-color palette algorithm | @skyra/gifenc's built-in quantizer | NeuQuant algorithm handles 24-bit to 8-bit reduction; quality parameter controls speed vs fidelity |
| Image resizing | Manual pixel sampling | sharp `.resize()` | Proper interpolation, sRGB color space handling |
| PNG decoding to raw pixels | Manual PNG chunk parsing | sharp `.raw().toBuffer()` | Handles all PNG variants, color spaces, bit depths |
| Text overlay rendering | Manual bitmap font | sharp Pango text (existing pattern) | Already working in timestamp-overlay.ts |

## Common Pitfalls

### Pitfall 1: Frame Dimension Mismatch
**What goes wrong:** Raw pixel comparison assumes identical dimensions. If frames have different sizes (window resize during capture), buffer lengths differ and comparison crashes.
**Why it happens:** Window can be resized during a capture session.
**How to avoid:** Resize both frames to a common dimension before comparison. The grid compiler already resizes to cell dimensions -- do comparison at the same target size.
**Warning signs:** Buffer length assertion failure, "offset out of range" errors.

### Pitfall 2: RGBA vs RGB Channel Count
**What goes wrong:** sharp `.raw()` outputs RGB (3 channels) for JPEG input or RGBA (4 channels) for PNG with alpha. Channel mismatch corrupts pixel offset calculations.
**Why it happens:** Frame buffers are PNG but may lack alpha channel.
**How to avoid:** Always call `.ensureAlpha()` before `.raw()` to guarantee 4-channel RGBA output. @skyra/gifenc's `addFrame()` expects RGBA (Uint8ClampedArray with 4 bytes per pixel).
**Warning signs:** Color shift in output, striped/skewed overlay images.

### Pitfall 3: GIF Frame Delay Minimum
**What goes wrong:** GIF spec minimum delay is 10ms. Many viewers treat delays below 20ms as 100ms. Fast capture intervals (100ms) work but very fast ones may display incorrectly.
**Why it happens:** GIF format limitation from the 1989 spec.
**How to avoid:** Clamp delay to `Math.max(20, delayMs)` for reliable playback. The `setDelay()` method accepts range 10-655360. [CITED: https://skyra-project.github.io/gifenc/classes/GifEncoder.html]
**Warning signs:** GIF plays back at wrong speed.

### Pitfall 4: Memory Pressure from Raw Pixel Buffers
**What goes wrong:** Converting many frames to raw RGBA simultaneously exhausts memory. A 1920x1080 RGBA frame is ~8MB raw; 50 frames = 400MB.
**Why it happens:** Temptation to map all frames to raw in parallel.
**How to avoid:** Process pairs sequentially for comparison. For GIF, resize to 800px max BEFORE extracting raw pixels (reduces to ~1.4MB per frame).
**Warning signs:** Node.js heap out of memory errors, extreme GC pauses.

### Pitfall 5: Sharp Concurrency on Windows
**What goes wrong:** Parallel sharp operations can deadlock on Windows due to fontconfig mutex issues.
**Why it happens:** Already encountered in Phase 1 -- sharp.concurrency(1) is set in grid-compiler.ts.
**How to avoid:** Ensure new processing modules import sharp after concurrency is set, or set it in each module. Better: set it once in a shared init module.
**Warning signs:** Process hangs during image processing.

## Code Examples

### Idle Frame Label Overlay
```typescript
// Reuse existing timestamp-overlay.ts pattern for "unchanged for N.Ns" labels
import { createTimestampOverlay } from "./timestamp-overlay.js";

async function createIdleLabel(
  durationMs: number,
  cellWidth: number,
  cellHeight: number
): Promise<Buffer> {
  const text = `unchanged ${(durationMs / 1000).toFixed(1)}s`;
  return createTimestampOverlay(text, cellWidth, cellHeight);
}
```
[VERIFIED: existing timestamp-overlay.ts pattern in codebase]

### Extending CaptureConfig
```typescript
// Add to existing CaptureConfig interface in types.ts
export interface CaptureConfig {
  // ... existing fields ...
  deltaHighlight?: boolean;
  compressIdle?: boolean;
  gifExport?: boolean;
}
```

### GIF Resource Registration
```typescript
// Follow existing capture-grid resource pattern in server.ts
server.registerResource(
  "capture-gif",
  new ResourceTemplate("capture://{sessionId}/gif", {
    list: async () => ({
      resources: sessionManager.listCompleted()
        .filter(s => s.config.gifExport)
        .map(s => ({
          uri: `capture://${s.id}/gif`,
          name: `Capture ${s.id} GIF`,
          mimeType: "image/gif",
        })),
    }),
  }),
  {
    description: "Animated GIF for a completed capture session",
    mimeType: "image/gif",
  },
  async (uri, { sessionId }) => {
    // ... encode GIF on demand or retrieve cached buffer ...
  },
);
```
[VERIFIED: existing resource registration pattern in server.ts]

## State of the Art

| Old Approach | Current Approach | When Changed | Impact |
|--------------|------------------|--------------|--------|
| gifencoder (unmaintained) | @skyra/gifenc | ~2022 | Modern maintained alternative with same API shape |
| sharp.raw() returns RGB | sharp.ensureAlpha().raw() for RGBA | Available since sharp 0.30+ | Consistent 4-channel output for pixel math |
| Manual NeuQuant for GIF | Built into @skyra/gifenc | N/A | No separate quantization step needed |

## Assumptions Log

| # | Claim | Section | Risk if Wrong |
|---|-------|---------|---------------|
| A1 | RGB Euclidean distance with threshold ~25 (10% of 255) is sufficient for detecting meaningful visual changes | Architecture Patterns | Threshold too sensitive or too lenient; easy to tune post-implementation |
| A2 | 40% opacity red overlay is visible enough for LLM vision models to detect | Architecture Patterns | May need higher opacity; trivial CSS-style adjustment |
| A3 | GIF quality=10 (default) produces acceptable file sizes for LLM consumption | Architecture Patterns | May need tuning; @skyra/gifenc quality range is 1-30 |
| A4 | 20ms minimum GIF delay is sufficient for reliable cross-viewer playback | Common Pitfalls | Some viewers may still interpret differently; low risk |

## Open Questions

1. **Sharp concurrency singleton**
   - What we know: `sharp.concurrency(1)` is set in grid-compiler.ts. New modules also use sharp.
   - What's unclear: Whether the concurrency setting is process-global (likely yes for libvips) or per-import.
   - Recommendation: Verify it's global. If not, extract to a shared `sharp-init.ts` module imported first.

2. **GIF buffer storage**
   - What we know: Grid JPEG is compiled on-demand when the resource is read. D-19 says GIF generation happens after grid compilation.
   - What's unclear: Should GIF buffer be cached on the session (eager) or generated on-demand (lazy)?
   - Recommendation: Generate eagerly after capture completes (alongside grid), cache on session. GIF encoding is slower than grid compilation and re-encoding on every resource read would be wasteful.

## Sources

### Primary (HIGH confidence)
- [sharp output API](https://sharp.pixelplumbing.com/api-output/) - `.raw()` pixel extraction, channel handling
- [sharp composite API](https://sharp.pixelplumbing.com/api-composite/) - overlay compositing with raw input buffers
- [@skyra/gifenc GifEncoder class](https://skyra-project.github.io/gifenc/classes/GifEncoder.html) - constructor, addFrame(Uint8ClampedArray), setDelay, setRepeat, setQuality
- [@skyra/gifenc GitHub](https://github.com/skyra-project/gifenc) - usage examples, stream-to-buffer pattern
- Existing codebase: `src/processing/grid-compiler.ts`, `src/processing/timestamp-overlay.ts`, `src/server.ts`, `src/types.ts`

### Secondary (MEDIUM confidence)
- npm registry: sharp@0.34.5, @skyra/gifenc@1.0.1 version verification

## Metadata

**Confidence breakdown:**
- Standard stack: HIGH - sharp already in use, @skyra/gifenc version verified on npm
- Architecture: HIGH - patterns verified against sharp and gifenc official docs, integration points clear from existing codebase
- Pitfalls: HIGH - dimension mismatch and memory pressure are well-known image processing issues; sharp concurrency issue already encountered in Phase 1

**Research date:** 2026-04-12
**Valid until:** 2026-05-12 (stable libraries, no breaking changes expected)