open-wc/docs/modules/renderer-closeout-verification.md

# Renderer Closeout Verification

## Metadata

| Field | Value |
|---|---|
| Status | Implemented |
| Target/work package | M03 / `M03-QAR-INTEGRATOR-CLOSEOUT-001` |
| Owner | Renderer structural, cache and performance acceptance gates |
| Last verified | Worktree `work/sindo-main-codex-m03-integrator/m03-closeout`, 2026-08-02 |
| Profile | `Blizzlike335`; quality preset `High` for checkpoint evidence |

## Purpose

Provide reproducible closeout checks for the M03 renderer decomposition. The
checks prove that worker boundaries remain CPU-only, main-thread frame steps are
budgeted, cache versions remain explicit, and paired M00/M03 reports stay within
the agreed 10% performance budgets.

## Non-goals

- Claim pixel-level parity with the original build-12340 client.
- Replace subsystem unit verifiers or long-traversal release tests.
- Generate, mutate or migrate production cache payloads.
- Hide incompatible environments or cache inventories by normalizing reports.

## Context and boundaries

```mermaid
flowchart LR
    M00[M00 commit capture] --> Reports[Render checkpoint reports]
    M03[M03 commit capture] --> Reports
    Reports --> Comparator[compare_render_performance.ps1]
    Sources[Renderer source and cache versions] --> Contracts[verify_renderer_closeout_contracts.gd]
    Comparator --> Stability[repeatability gate across short and long windows]
    Stability --> Evidence[JSON comparison and exit code]
    Contracts --> Evidence
    Evidence --> Target[M03 Evidence / DONE decision]
```

The capture command owns SceneTree execution and PNG/report writes. The
PowerShell comparator is read-only except for its requested JSON output. The
GDScript contract verifier reads source files and creates no renderer resources.

## Public API

| Symbol | Kind | Purpose | Preconditions | Failure |
|---|---|---|---|---|
| `compare_render_performance.ps1 -BaselineReport <paths> -CandidateReport <paths> [-OutputReport <path>]` | CLI | Compare one report or median of repeated reports | Same schema, profile, environment, cache contract/inventory and result keys | Exit 1 and enumerate incompatible fields or budget regressions |
| `verify_render_performance_stability.ps1 -RepeatedSampleComparison <path> -LongWindowComparison <path>` | CLI | Reject only a metric regression reproduced by both independent protocols | Both comparator reports contain the same 84 metrics | Exit 1 with every repeatable result/metric key |
| `verify_renderer_closeout_contracts.gd` | Godot CLI | Check worker/main-thread/cache/converter source contracts | Project parses and referenced sources exist | Exit 1 with named contract failure |

## Inputs and outputs

| Direction | Contract/data | Producer | Consumer | Ownership | Lifetime |
|---|---|---|---|---|---|
| Input | M00/M03 `report.json` paths | `capture_render_checkpoints.gd` | Comparator | Filesystem-owned immutable evidence | One comparison |
| Input | Renderer/tool GDScript sources | Repository | Contract verifier | Read-only | One verifier run |
| Output | 84 metric comparisons and failures | Comparator | Integrator/CI | Optional JSON plus process output | Evidence retention |
| Output | Repeatable and protocol-local regression inventories | Stability gate | Integrator/CI | Optional JSON plus process output | Evidence retention |
| Output | Structural pass/fail summary | Contract verifier | Integrator/CI | Process output | One run |

## Data flow

```mermaid
flowchart TD
    Read[Read every supplied report] --> Compatible{Metadata and inventory match?}
    Compatible -->|no| Fail[Exit 1 with exact mismatch]
    Compatible -->|yes| Median[Median each metric per checkpoint/pass]
    Median --> Pair[Pair 14 result keys]
    Pair --> Budget[Compare load, p95, p99, hitch and memory]
    Budget --> Json[Optional comparison JSON]
    Budget --> Cohorts[Repeated-sample and long-window comparisons]
    Cohorts --> Repeated{Same metric fails both?}
    Repeated -->|no| Pass[Exit 0 with protocol-local diagnostics]
    Repeated -->|yes| Fail
```

## Lifecycle and sequence

```mermaid
sequenceDiagram
    participant I as Integrator
    participant B as M00 worktree
    participant C as M03 worktree
    participant R as Checkpoint capture
    participant G as Performance comparator
    I->>B: capture repeated baseline samples
    B->>R: same viewport, driver and cache inventory
    I->>C: capture repeated candidate samples
    C->>R: same viewport, driver and cache inventory
    I->>G: baseline paths plus candidate paths
    G->>G: compatibility checks and per-metric medians
    G-->>I: short-window and long-window JSON evidence
    I->>G: verify stability across both protocols
    G-->>I: repeatable-regression pass/fail
```

There is no persistent state machine. Each invocation is read, validate,
aggregate, compare and terminate.

## Ownership, threading and resources

- Reports and source files are borrowed read-only for one process.
- Median aggregation deep-copies the first report and never rewrites inputs.
- Rendering remains owned by the GUI capture process on Godot's main thread.
- The contract verifier is headless and does not instantiate the streaming world.
- The optional comparison report is wholly owned by the caller-selected path.

## Errors, cancellation and recovery

| Failure | Behavior | Recovery |
|---|---|---|
| Missing/malformed report | Terminate with path/shape error | Regenerate that capture |
| Environment/cache mismatch | Fail before accepting metrics | Recapture both commits on the same machine and cache |
| Missing/duplicate checkpoint | Fail result-key validation | Repair manifest/capture completeness |
| Metric over budget | Record baseline, candidate, limit and percentage | Diagnose named checkpoint/lane; rerun only after a code or evidence correction |
| Interrupted GUI capture | No complete report is accepted | Remove/ignore partial output and rerun |

## Configuration and capabilities

The comparator reads thresholds from the baseline report. M03 uses 10% maximum
regression for load time, frame p95, frame p99, maximum hitch, static memory and
video memory. Repeated input paths are optional; when supplied, each side is
reduced independently to the median for every metric.

Closeout uses two independent protocols: repeated three-second captures and a
ten-second measurement window. A regression is accepted as real only when the
same checkpoint/pass/metric exceeds its unchanged 10% budget in both protocols.
Protocol-local failures remain in the JSON as noise diagnostics rather than
being discarded.

## Persistence, cache and migrations

The comparison JSON uses schema version 1 and contains source paths, sample
counts, revisions, all metric pairs and failures. It is evidence, not a runtime
cache. Renderer cache versions are read from the manifest/source contracts; this
module performs no migration or invalidation.

## Diagnostics and observability

- Success reports result pairs, comparison count and budget percentage.
- Failure output names every checkpoint/pass, metric, values and limit.
- The structural verifier reports worker count, frame-step count and cache-version count.
- Capture reports retain PNG hashes, queue snapshots, environment and cache inventory.

## Verification, fidelity and performance

- `verify_renderer_closeout_contracts.gd` covers four worker boundaries, fifteen
  frame steps, seven cache versions and the nested M2 GLB output contract.
- `compare_render_performance.ps1` compares 14 cold/warm result pairs and 84 metrics.
- `verify_render_performance_stability.ps1` requires metric-key agreement and
  rejects any budget regression reproduced by both sampling protocols.
- M00 and M03 must be captured from their exact commits against the same cache
  inventory; old reports with a different inventory are rejected.
- PNG hashes and asset-backed coverage prove that terrain, ADT boundaries, dense
  M2, large WMO, liquid, animated M2 and sky were rendered. They do not prove
  original-client pixel parity without human/reference-image approval.

## Extension points

CI may retain reports and comparison JSON as artifacts. A future release gate may
add driver-version metadata, long-traversal samples or approved visual-diff
thresholds without changing runtime renderer contracts.

## Known gaps and risks

- Godot reports the rendering API but not the installed NVIDIA driver version.
- A 0.5-second historical M00 measurement window requires repeated median samples.
- Original-client screenshots are not part of the repository evidence set.
- Long-traversal descriptor pressure remains a later quality/release gate.

## Source map

| Path | Responsibility |
|---|---|
| `tools/compare_render_performance.ps1` | Compatibility, median aggregation and metric budgets |
| `tools/verify_render_performance_stability.ps1` | Cross-protocol repeatability acceptance |
| `src/tools/verify_renderer_closeout_contracts.gd` | Structural source/cache/converter contracts |
| `src/tools/capture_render_checkpoints.gd` | Asset-backed GUI capture and report generation |
| `src/tools/render_baseline_manifest.json` | Coverage, viewport, cache contract and budgets |
| `targets/00-render-baseline.md` | Accepted M00 measurement protocol |
| `targets/03-renderer-facade.md` | M03 acceptance and Evidence |