test(M00): record final paired evidence #13
@@ -0,0 +1,78 @@
|
|||||||
|
# M00-QAR-FINAL-PAIRED-EVIDENCE-001 — Final paired evidence audit
|
||||||
|
|
||||||
|
<!-- OPENWC_CLAIM:M00-QAR-FINAL-PAIRED-EVIDENCE-001:sindo-main-codex:2026-07-14 -->
|
||||||
|
|
||||||
|
## Ownership
|
||||||
|
|
||||||
|
- Target: M00
|
||||||
|
- Program: QAR
|
||||||
|
- Owner/Agent ID: sindo-main-codex
|
||||||
|
- Branch: `work/sindo-main-codex/m00-final-paired-evidence`
|
||||||
|
- Lease expires UTC: 2026-07-14
|
||||||
|
- Integrator: milestone integrator
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
Produce one final WMO-ready paired report for all available build 12340 references and reconcile the remaining M00 acceptance gaps.
|
||||||
|
|
||||||
|
## Non-goals
|
||||||
|
|
||||||
|
- Achieving renderer parity inside M00.
|
||||||
|
- Calibrating unknown manual camera parameters.
|
||||||
|
- Marking the target DONE; only the milestone integrator may do so.
|
||||||
|
|
||||||
|
## Paths
|
||||||
|
|
||||||
|
- Exclusive: paired-run exit handling and final evidence record
|
||||||
|
- Shared/hotspots: renderer baseline/module findings
|
||||||
|
- Generated/ignored: final captures and reports under `user://`
|
||||||
|
|
||||||
|
## Contracts and data
|
||||||
|
|
||||||
|
- Public API/events: visual comparison exit 0/1 both count as completed report generation
|
||||||
|
- Schema/format version: unchanged
|
||||||
|
- Migration/compatibility: comparator infrastructure errors remain fatal
|
||||||
|
- Consumers: M00 integrator
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
- Requires: merged region diagnostic and five private reference images
|
||||||
|
- Blocks: integrator decision on M00 completion
|
||||||
|
- External state: private images remain outside Git
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
- Commands: complete paired baseline run, report audit, repository gates
|
||||||
|
- Fixtures: five build 12340 references
|
||||||
|
- Fidelity evidence: cold/warm comparison metrics and human gap classification
|
||||||
|
- Performance budget: existing M00 capture budgets
|
||||||
|
|
||||||
|
## Documentation deliverables
|
||||||
|
|
||||||
|
- Inline public API docs: PowerShell comparison step semantics
|
||||||
|
- Module specification: final evidence status
|
||||||
|
- Data-flow diagram: unchanged
|
||||||
|
- Sequence/state/dependency diagrams: unchanged
|
||||||
|
- Source map/status updates: baseline acceptance summary
|
||||||
|
|
||||||
|
## Simplicity and naming
|
||||||
|
|
||||||
|
- Important names introduced: `Invoke-GodotComparisonStep`
|
||||||
|
- Simplest considered solution: allow only comparator exit codes 0 and 1
|
||||||
|
- Rejected complexity/abstractions: suppressing all process failures
|
||||||
|
- Unavoidable complexity and justification: threshold exceedance is evidence, not infrastructure failure
|
||||||
|
- Measured optimization evidence: not applicable
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
- State: ready-for-review
|
||||||
|
- Done: paired-run exit contract fixed; full baseline and visual capture completed; ten pairs audited; final acceptance document written
|
||||||
|
- Next: integrator decides animated/sky reference policy and GUI RID-risk disposition
|
||||||
|
- Blocked by:
|
||||||
|
|
||||||
|
## Handoff
|
||||||
|
|
||||||
|
- Commit: this work-package commit
|
||||||
|
- Results: compared=10, failed=10, missing=0; mean error 0.066892..0.176952; cold/warm gaps stable; full workflow returned success with threshold evidence preserved
|
||||||
|
- Remaining risks: no build 12340 animated-M2 or sky-transition reference; GUI renderer RID/resources still reported at exit
|
||||||
|
- Documentation updated: `docs/M00_FINAL_PAIRED_EVIDENCE.md`, `docs/modules/world-renderer.md`
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
# M00 Final Paired Evidence — 2026-07-12
|
||||||
|
|
||||||
|
## Scope and result
|
||||||
|
|
||||||
|
The final local M00 audit used Godot 4.6.1, the existing cache state, the manifest-standard 8-second readiness wait, and `2560x1440` visual candidates. Five private WoW 3.3.5a build 12340 references produced ten cold/warm comparisons with no missing pair. Private images and generated reports remain outside Git.
|
||||||
|
|
||||||
|
The command completed even though every pair exceeded the strict regression tolerance. Comparator exit code `1` means a valid report with visual gaps; only codes outside `0/1` are infrastructure failures.
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
.\tools\run_render_baseline.ps1 `
|
||||||
|
-Output user://render_baseline_m00_final_2026-07-12 `
|
||||||
|
-CacheState existing `
|
||||||
|
-WaitSeconds 8 `
|
||||||
|
-MeasureSeconds 0.5 `
|
||||||
|
-ReferenceCheckpointDirectory .\sources\OpenWCReferenceCheckpoints `
|
||||||
|
-VisualComparisonReport user://render_baseline_m00_final_2026-07-12/visual_comparison.json `
|
||||||
|
-VisualComparisonWidth 2560 `
|
||||||
|
-VisualComparisonHeight 1440
|
||||||
|
```
|
||||||
|
|
||||||
|
## Paired metrics
|
||||||
|
|
||||||
|
| Checkpoint | Pass | Mean perceptual error | Changed-pixel ratio | Classification |
|
||||||
|
|---|---|---:|---:|---|
|
||||||
|
| `elwynn_adt_boundary` | cold | 0.087347 | 0.634984 | Composition plus terrain/vegetation/material gaps |
|
||||||
|
| `elwynn_adt_boundary` | warm | 0.087255 | 0.634598 | Stable against revisit |
|
||||||
|
| `elwynn_terrain_overview` | cold | 0.174136 | 0.851806 | Largest composition/framing gap |
|
||||||
|
| `elwynn_terrain_overview` | warm | 0.176952 | 0.861275 | Stable gap, not a cold-load artifact |
|
||||||
|
| `elwynn_waterfall_liquid` | cold | 0.088526 | 0.706897 | Framing, terrain, liquid and lighting gaps |
|
||||||
|
| `elwynn_waterfall_liquid` | warm | 0.087490 | 0.695281 | Stable against revisit |
|
||||||
|
| `goldshire_dense_m2` | cold | 0.066892 | 0.575205 | Smallest available paired gap; vegetation/material differences remain |
|
||||||
|
| `goldshire_dense_m2` | warm | 0.066927 | 0.575547 | Stable against revisit |
|
||||||
|
| `goldshire_inn_large_wmo` | cold | 0.101230 | 0.701836 | WMO is ready; manual framing and presentation gaps remain |
|
||||||
|
| `goldshire_inn_large_wmo` | warm | 0.101203 | 0.701855 | Stable against revisit |
|
||||||
|
|
||||||
|
Cold and warm values are close for all five references. The paired gaps are therefore not explained by first-visit cache readiness. Goldshire timing evidence separately demonstrated that the 8-second wait is necessary for WMO publication.
|
||||||
|
|
||||||
|
## Fidelity interpretation
|
||||||
|
|
||||||
|
M00 now has reproducible quantitative evidence for terrain, ADT boundary, dense M2, large WMO and liquid checkpoints. The reports demonstrate current differences; they do not claim renderer parity. Camera calibration experiments also show that unknown manual yaw/pitch/FOV cannot be recovered reliably from full-frame color error alone. Region scoring reduces terrain bias but still requires semantic landmark approval.
|
||||||
|
|
||||||
|
Two manifest checkpoints still lack original-client paired references:
|
||||||
|
|
||||||
|
- `gryphon_roost_native_animation` is a synthetic native-animation probe and has no matching build 12340 fixture.
|
||||||
|
- `elwynn_sky_dusk` has no approved build 12340 sky-transition fixture at the same time/weather state.
|
||||||
|
|
||||||
|
These must either receive private reference evidence or be explicitly accepted by the milestone integrator as unpaired diagnostic coverage. Until that decision, the target remains `ACTIVE`.
|
||||||
|
|
||||||
|
## Operational risks
|
||||||
|
|
||||||
|
The full GUI capture completed and wrote both reports, but Godot still emitted renderer RID/resource leak diagnostics at exit. The earlier anonymous headless `RefCounted` leak is fixed; GUI renderer allocations are a distinct known risk. They do not invalidate the written images or metrics, but must remain recorded rather than hidden.
|
||||||
|
|
||||||
|
## Integrator decision checklist
|
||||||
|
|
||||||
|
- Five available reference checkpoints: paired cold/warm report complete, missing pairs zero.
|
||||||
|
- Animated M2 fixture: missing or explicitly accepted as synthetic-only.
|
||||||
|
- Sky-transition fixture: missing or explicitly accepted as unpaired.
|
||||||
|
- Renderer visual differences: registered as gaps, not parity claims.
|
||||||
|
- GUI renderer RID/resource diagnostics: accepted as a tracked risk or assigned a follow-up blocker.
|
||||||
|
- Target status and `OPENWC_TARGET_DONE` marker: integrator-only action.
|
||||||
@@ -234,6 +234,7 @@ Exact exported settings and cache versions remain documented in [`../../RENDER.m
|
|||||||
- A ready-scene Goldshire 3x3 grid improved mean error from `0.101402` at zero offsets to `0.087952` at yaw `10`/pitch `-10`; pitch/FOV refinement remains required before manifest calibration.
|
- A ready-scene Goldshire 3x3 grid improved mean error from `0.101402` at zero offsets to `0.087952` at yaw `10`/pitch `-10`; pitch/FOV refinement remains required before manifest calibration.
|
||||||
- Joint Goldshire pitch/FOV refinement had no human-approved optimum: full-frame error preferred grass-heavy FOV 62/pitch -25, while FOV 38/pitch -10 better matched building scale. Landmark/region scoring is required before calibration.
|
- Joint Goldshire pitch/FOV refinement had no human-approved optimum: full-frame error preferred grass-heavy FOV 62/pitch -25, while FOV 38/pitch -10 better matched building scale. Landmark/region scoring is required before calibration.
|
||||||
- Common-region scoring `[300,100,1400,650]` moved Goldshire's numeric optimum to FOV 38/yaw 10/pitch -15, but human inspection still found an oversized/cropped facade; semantic landmarks or masks remain required.
|
- Common-region scoring `[300,100,1400,650]` moved Goldshire's numeric optimum to FOV 38/yaw 10/pitch -15, but human inspection still found an oversized/cropped facade; semantic landmarks or masks remain required.
|
||||||
|
- Final paired audit compared ten cold/warm images for five private build 12340 references with zero missing pairs; mean error ranges `0.066892..0.176952`. Animated-M2 and sky-transition references remain an integrator acceptance decision.
|
||||||
- D3D12 descriptor issues remain; the capture-path anonymous `RefCounted` shutdown leak is regression-covered by a clean verbose dry-run, while other RID/resource diagnostics still require independent evidence.
|
- D3D12 descriptor issues remain; the capture-path anonymous `RefCounted` shutdown leak is regression-covered by a clean verbose dry-run, while other RID/resource diagnostics still require independent evidence.
|
||||||
- M2/WMO/material/particle/ribbon/portal parity incomplete.
|
- M2/WMO/material/particle/ribbon/portal parity incomplete.
|
||||||
- Public API is mostly exported configuration rather than stable contracts.
|
- Public API is mostly exported configuration rather than stable contracts.
|
||||||
@@ -263,5 +264,6 @@ Exact exported settings and cache versions remain documented in [`../../RENDER.m
|
|||||||
- [`../../RENDER.md`](../../RENDER.md)
|
- [`../../RENDER.md`](../../RENDER.md)
|
||||||
- [`../RENDER_BASELINE.md`](../RENDER_BASELINE.md)
|
- [`../RENDER_BASELINE.md`](../RENDER_BASELINE.md)
|
||||||
- [`../CAMERA_POSE_SWEEP.md`](../CAMERA_POSE_SWEEP.md)
|
- [`../CAMERA_POSE_SWEEP.md`](../CAMERA_POSE_SWEEP.md)
|
||||||
|
- [`../M00_FINAL_PAIRED_EVIDENCE.md`](../M00_FINAL_PAIRED_EVIDENCE.md)
|
||||||
- [`../../targets/roadmap/02-rendering-and-graphics.md`](../../targets/roadmap/02-rendering-and-graphics.md)
|
- [`../../targets/roadmap/02-rendering-and-graphics.md`](../../targets/roadmap/02-rendering-and-graphics.md)
|
||||||
- [`../GODOT_BEST_PRACTICES.md`](../GODOT_BEST_PRACTICES.md)
|
- [`../GODOT_BEST_PRACTICES.md`](../GODOT_BEST_PRACTICES.md)
|
||||||
|
|||||||
@@ -40,6 +40,15 @@ function Invoke-GodotStep {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function Invoke-GodotComparisonStep {
|
||||||
|
param([string]$Name, [string[]]$Arguments)
|
||||||
|
Write-Host "[$Name] $GodotPath $($Arguments -join ' ')"
|
||||||
|
$process = Start-Process -FilePath $GodotPath -ArgumentList $Arguments -NoNewWindow -Wait -PassThru
|
||||||
|
if ($process.ExitCode -notin @(0, 1)) {
|
||||||
|
throw "$Name failed with exit code $($process.ExitCode)"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
Push-Location $repoRoot
|
Push-Location $repoRoot
|
||||||
try {
|
try {
|
||||||
Invoke-GodotStep "project-load" @("--headless", "--path", ".", "--quit")
|
Invoke-GodotStep "project-load" @("--headless", "--path", ".", "--quit")
|
||||||
@@ -80,7 +89,7 @@ try {
|
|||||||
"--viewport-height", $VisualComparisonHeight.ToString()
|
"--viewport-height", $VisualComparisonHeight.ToString()
|
||||||
)
|
)
|
||||||
Invoke-GodotStep "visual-capture" $visualCaptureArgs
|
Invoke-GodotStep "visual-capture" $visualCaptureArgs
|
||||||
Invoke-GodotStep "visual-comparison" @(
|
Invoke-GodotComparisonStep "visual-comparison" @(
|
||||||
"--headless", "--path", ".",
|
"--headless", "--path", ".",
|
||||||
"--script", "res://src/tools/compare_render_checkpoints.gd",
|
"--script", "res://src/tools/compare_render_checkpoints.gd",
|
||||||
"--",
|
"--",
|
||||||
|
|||||||
Reference in New Issue
Block a user