test(M00): add landmark-region visual diff #12
@@ -0,0 +1,78 @@
|
|||||||
|
# M00-QAR-LANDMARK-REGION-DIFF-001 — Landmark-region visual diff
|
||||||
|
|
||||||
|
<!-- OPENWC_CLAIM:M00-QAR-LANDMARK-REGION-DIFF-001:sindo-main-codex:2026-07-14 -->
|
||||||
|
|
||||||
|
## Ownership
|
||||||
|
|
||||||
|
- Target: M00
|
||||||
|
- Program: QAR
|
||||||
|
- Owner/Agent ID: sindo-main-codex
|
||||||
|
- Branch: `work/sindo-main-codex/m00-landmark-region-diff`
|
||||||
|
- Lease expires UTC: 2026-07-14
|
||||||
|
- Integrator: milestone integrator
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
Rank camera candidates inside an approved common screen region so unrelated grass/sky coverage cannot dominate landmark framing.
|
||||||
|
|
||||||
|
## Non-goals
|
||||||
|
|
||||||
|
- Automatic feature detection or image warping.
|
||||||
|
- Different crops for reference and candidate.
|
||||||
|
- Changing manifest camera values without human approval.
|
||||||
|
|
||||||
|
## Paths
|
||||||
|
|
||||||
|
- Exclusive: comparator common-region option and sweep pass-through
|
||||||
|
- Shared/hotspots: camera-pose documentation and renderer fidelity findings
|
||||||
|
- Generated/ignored: local ROI sweep images/reports
|
||||||
|
|
||||||
|
## Contracts and data
|
||||||
|
|
||||||
|
- Public API/events: additive comparator `--region x,y,width,height` and sweep `-ComparisonRegion`
|
||||||
|
- Schema/format version: report/ranking schema remains version 1 with additive region metadata
|
||||||
|
- Migration/compatibility: omitted region preserves full-frame behavior
|
||||||
|
- Consumers: M00 paired camera calibration
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
- Requires: ready-scene joint pitch/FOV sweep
|
||||||
|
- Blocks: landmark-weighted Goldshire framing decision
|
||||||
|
- External state: private build 12340 reference
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
- Commands: comparator synthetic ROI regression, invalid-bound regression, sweep plan, bounded real ROI sweep, repository gates
|
||||||
|
- Fixtures: synthetic 2x2 images and private Goldshire reference
|
||||||
|
- Fidelity evidence: facade-region ranking plus human inspection
|
||||||
|
- Performance budget: region comparison visits only selected pixels
|
||||||
|
|
||||||
|
## Documentation deliverables
|
||||||
|
|
||||||
|
- Inline public API docs: comparator usage and PowerShell parameter
|
||||||
|
- Module specification: ROI finding
|
||||||
|
- Data-flow diagram: approved region into comparator
|
||||||
|
- Sequence/state/dependency diagrams: unchanged
|
||||||
|
- Source map/status updates: camera-pose guide
|
||||||
|
|
||||||
|
## Simplicity and naming
|
||||||
|
|
||||||
|
- Important names introduced: `ComparisonRegion`, `comparison_region`
|
||||||
|
- Simplest considered solution: one common integer rectangle
|
||||||
|
- Rejected complexity/abstractions: masks, feature detectors, separate movable crops
|
||||||
|
- Unavoidable complexity and justification: full-frame color error is biased by large terrain regions
|
||||||
|
- Measured optimization evidence: bounded pixel loop
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
- State: ready-for-review
|
||||||
|
- Done: common ROI parser/validation/reporting, synthetic inclusion/exclusion regression, sweep pass-through, 15-candidate Goldshire rescoring and human inspection
|
||||||
|
- Next: define explicit approved landmark coordinates or mask; do not calibrate manifest from the common rectangle alone
|
||||||
|
- Blocked by:
|
||||||
|
|
||||||
|
## Handoff
|
||||||
|
|
||||||
|
- Commit: this work-package commit
|
||||||
|
- Results: ROI [300,100,1400,650] ranked FOV 38/yaw 10/pitch -15 first at mean 0.081777 and ratio 0.697185; visually the facade remains oversized and cropped
|
||||||
|
- Remaining risks: rectangle includes non-semantic road/terrain pixels and cannot prove landmark alignment
|
||||||
|
- Documentation updated: `docs/CAMERA_POSE_SWEEP.md`, `docs/modules/world-renderer.md`
|
||||||
@@ -54,4 +54,10 @@ A WMO-ready joint grid fixed yaw at 10 degrees and evaluated pitch `[-25, -20, -
|
|||||||
|
|
||||||
The grid therefore has no approved joint optimum. Full-frame color error continues to trade landmark framing against large terrain regions. The manifest remains unchanged. The next registration method must score an approved inn/road/tree landmark region or explicit landmark coordinates rather than the entire image.
|
The grid therefore has no approved joint optimum. Full-frame color error continues to trade landmark framing against large terrain regions. The manifest remains unchanged. The next registration method must score an approved inn/road/tree landmark region or explicit landmark coordinates rather than the entire image.
|
||||||
|
|
||||||
|
## Common landmark region — 2026-07-12
|
||||||
|
|
||||||
|
The comparator accepts an optional common screen rectangle through `--region x,y,width,height`; the sweep runner exposes it as `-ComparisonRegion x,y,width,height`. Coordinates are integer pixels in the shared reference/candidate image space. Both images must have identical dimensions and the rectangle must be fully inside them. Omitting the option preserves full-frame behavior.
|
||||||
|
|
||||||
|
The existing 15 WMO-ready Goldshire candidates were rescored inside `[300,100,1400,650]`, covering the reference facade/chimney and nearby trees. ROI error ranked FOV 38/yaw 10/pitch -15 first at `0.081777` with changed-pixel ratio `0.697185`. This is an interior pitch candidate and improves on the full-frame grass preference, but human inspection still rejects manifest calibration: the facade is too large, its upper part is cropped, and the rectangle retains substantial road/terrain pixels. A common rectangle is useful diagnostic weighting, not semantic landmark registration. Explicit approved landmark coordinates or a mask remain necessary.
|
||||||
|
|
||||||
During this run the sweep orchestration was hardened in two ways: expected comparator exit code `1` is now collected through an explicit child process, and viewport dimensions are explicit. A reference/candidate `size_mismatch` now fails before ranking instead of producing an empty metric.
|
During this run the sweep orchestration was hardened in two ways: expected comparator exit code `1` is now collected through an explicit child process, and viewport dimensions are explicit. A reference/candidate `size_mismatch` now fails before ranking instead of producing an empty metric.
|
||||||
|
|||||||
@@ -233,6 +233,7 @@ Exact exported settings and cache versions remain documented in [`../../RENDER.m
|
|||||||
- The first Goldshire pose grid was invalidated by WMO readiness: the inn is absent after 2 seconds but visible after the manifest-standard 8-second wait. Pose comparisons must stabilize asynchronous checkpoint content first.
|
- The first Goldshire pose grid was invalidated by WMO readiness: the inn is absent after 2 seconds but visible after the manifest-standard 8-second wait. Pose comparisons must stabilize asynchronous checkpoint content first.
|
||||||
- A ready-scene Goldshire 3x3 grid improved mean error from `0.101402` at zero offsets to `0.087952` at yaw `10`/pitch `-10`; pitch/FOV refinement remains required before manifest calibration.
|
- A ready-scene Goldshire 3x3 grid improved mean error from `0.101402` at zero offsets to `0.087952` at yaw `10`/pitch `-10`; pitch/FOV refinement remains required before manifest calibration.
|
||||||
- Joint Goldshire pitch/FOV refinement had no human-approved optimum: full-frame error preferred grass-heavy FOV 62/pitch -25, while FOV 38/pitch -10 better matched building scale. Landmark/region scoring is required before calibration.
|
- Joint Goldshire pitch/FOV refinement had no human-approved optimum: full-frame error preferred grass-heavy FOV 62/pitch -25, while FOV 38/pitch -10 better matched building scale. Landmark/region scoring is required before calibration.
|
||||||
|
- Common-region scoring `[300,100,1400,650]` moved Goldshire's numeric optimum to FOV 38/yaw 10/pitch -15, but human inspection still found an oversized/cropped facade; semantic landmarks or masks remain required.
|
||||||
- D3D12 descriptor issues remain; the capture-path anonymous `RefCounted` shutdown leak is regression-covered by a clean verbose dry-run, while other RID/resource diagnostics still require independent evidence.
|
- D3D12 descriptor issues remain; the capture-path anonymous `RefCounted` shutdown leak is regression-covered by a clean verbose dry-run, while other RID/resource diagnostics still require independent evidence.
|
||||||
- M2/WMO/material/particle/ribbon/portal parity incomplete.
|
- M2/WMO/material/particle/ribbon/portal parity incomplete.
|
||||||
- Public API is mostly exported configuration rather than stable contracts.
|
- Public API is mostly exported configuration rather than stable contracts.
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ extends SceneTree
|
|||||||
## Compares original-client JPG/PNG references with OpenWC checkpoint PNGs.
|
## Compares original-client JPG/PNG references with OpenWC checkpoint PNGs.
|
||||||
## Usage: godot --headless --path . --script res://src/tools/compare_render_checkpoints.gd --
|
## Usage: godot --headless --path . --script res://src/tools/compare_render_checkpoints.gd --
|
||||||
## --reference <directory> --candidate <directory> [--output <report.json>]
|
## --reference <directory> --candidate <directory> [--output <report.json>]
|
||||||
|
## [--region x,y,width,height]
|
||||||
## [--pixel-threshold 0.04] [--mean-threshold 0.015] [--changed-ratio-threshold 0.01]
|
## [--pixel-threshold 0.04] [--mean-threshold 0.015] [--changed-ratio-threshold 0.01]
|
||||||
## Use --self-test for a synthetic regression check that writes only under user://.
|
## Use --self-test for a synthetic regression check that writes only under user://.
|
||||||
|
|
||||||
@@ -56,13 +57,18 @@ func _parse_arguments(raw_arguments: PackedStringArray) -> Dictionary:
|
|||||||
parsed.self_test = true
|
parsed.self_test = true
|
||||||
index += 1
|
index += 1
|
||||||
continue
|
continue
|
||||||
if argument not in ["--reference", "--candidate", "--output", "--only", "--pixel-threshold", "--mean-threshold", "--changed-ratio-threshold"]:
|
if argument not in ["--reference", "--candidate", "--output", "--only", "--region", "--pixel-threshold", "--mean-threshold", "--changed-ratio-threshold"]:
|
||||||
return {"error": "unknown argument: %s" % argument}
|
return {"error": "unknown argument: %s" % argument}
|
||||||
if index + 1 >= raw_arguments.size():
|
if index + 1 >= raw_arguments.size():
|
||||||
return {"error": "missing value for %s" % argument}
|
return {"error": "missing value for %s" % argument}
|
||||||
var key := argument.trim_prefix("--").replace("-", "_")
|
var key := argument.trim_prefix("--").replace("-", "_")
|
||||||
var value := raw_arguments[index + 1]
|
var value := raw_arguments[index + 1]
|
||||||
if key.ends_with("threshold"):
|
if key == "region":
|
||||||
|
var region_parse_result := _parse_comparison_region(value)
|
||||||
|
if region_parse_result.has("error"):
|
||||||
|
return region_parse_result
|
||||||
|
parsed[key] = region_parse_result.region
|
||||||
|
elif key.ends_with("threshold"):
|
||||||
if not value.is_valid_float():
|
if not value.is_valid_float():
|
||||||
return {"error": "%s requires a number" % argument}
|
return {"error": "%s requires a number" % argument}
|
||||||
parsed[key] = float(value)
|
parsed[key] = float(value)
|
||||||
@@ -72,6 +78,24 @@ func _parse_arguments(raw_arguments: PackedStringArray) -> Dictionary:
|
|||||||
return parsed
|
return parsed
|
||||||
|
|
||||||
|
|
||||||
|
func _parse_comparison_region(value: String) -> Dictionary:
|
||||||
|
var components := value.split(",", false)
|
||||||
|
if components.size() != 4:
|
||||||
|
return {"error": "--region requires x,y,width,height"}
|
||||||
|
for component in components:
|
||||||
|
if not component.strip_edges().is_valid_int():
|
||||||
|
return {"error": "--region components must be integers"}
|
||||||
|
var region := Rect2i(
|
||||||
|
int(components[0]),
|
||||||
|
int(components[1]),
|
||||||
|
int(components[2]),
|
||||||
|
int(components[3])
|
||||||
|
)
|
||||||
|
if region.position.x < 0 or region.position.y < 0 or region.size.x <= 0 or region.size.y <= 0:
|
||||||
|
return {"error": "--region requires non-negative x/y and positive width/height"}
|
||||||
|
return {"region": region}
|
||||||
|
|
||||||
|
|
||||||
func _compare_directories(reference_directory: String, candidate_directory: String, options: Dictionary) -> Dictionary:
|
func _compare_directories(reference_directory: String, candidate_directory: String, options: Dictionary) -> Dictionary:
|
||||||
reference_directory = ProjectSettings.globalize_path(reference_directory)
|
reference_directory = ProjectSettings.globalize_path(reference_directory)
|
||||||
candidate_directory = ProjectSettings.globalize_path(candidate_directory)
|
candidate_directory = ProjectSettings.globalize_path(candidate_directory)
|
||||||
@@ -108,6 +132,7 @@ func _compare_directories(reference_directory: String, candidate_directory: Stri
|
|||||||
"pixel_threshold": options.pixel_threshold,
|
"pixel_threshold": options.pixel_threshold,
|
||||||
"mean_threshold": options.mean_threshold,
|
"mean_threshold": options.mean_threshold,
|
||||||
"changed_ratio_threshold": options.changed_ratio_threshold,
|
"changed_ratio_threshold": options.changed_ratio_threshold,
|
||||||
|
"comparison_region": _region_array(options.get("region", Rect2i())),
|
||||||
"compared_count": results.size() - missing_count,
|
"compared_count": results.size() - missing_count,
|
||||||
"failed_count": failed_count,
|
"failed_count": failed_count,
|
||||||
"missing_count": missing_count,
|
"missing_count": missing_count,
|
||||||
@@ -127,12 +152,22 @@ func _compare_images(reference_path: String, candidate_path: String, options: Di
|
|||||||
"reference_size": [reference_image.get_width(), reference_image.get_height()],
|
"reference_size": [reference_image.get_width(), reference_image.get_height()],
|
||||||
"candidate_size": [candidate_image.get_width(), candidate_image.get_height()],
|
"candidate_size": [candidate_image.get_width(), candidate_image.get_height()],
|
||||||
}
|
}
|
||||||
|
var comparison_region: Rect2i = options.get("region", Rect2i(Vector2i.ZERO, reference_image.get_size()))
|
||||||
|
if comparison_region == Rect2i():
|
||||||
|
comparison_region = Rect2i(Vector2i.ZERO, reference_image.get_size())
|
||||||
|
var image_region := Rect2i(Vector2i.ZERO, reference_image.get_size())
|
||||||
|
if not image_region.encloses(comparison_region):
|
||||||
|
return {
|
||||||
|
"status": "region_out_of_bounds",
|
||||||
|
"region": _region_array(comparison_region),
|
||||||
|
"image_size": [reference_image.get_width(), reference_image.get_height()],
|
||||||
|
}
|
||||||
|
|
||||||
var error_sum := 0.0
|
var error_sum := 0.0
|
||||||
var changed_pixel_count := 0
|
var changed_pixel_count := 0
|
||||||
var pixel_count := reference_image.get_width() * reference_image.get_height()
|
var pixel_count := comparison_region.size.x * comparison_region.size.y
|
||||||
for y in reference_image.get_height():
|
for y in range(comparison_region.position.y, comparison_region.end.y):
|
||||||
for x in reference_image.get_width():
|
for x in range(comparison_region.position.x, comparison_region.end.x):
|
||||||
var perceptual_error := _perceptual_color_error(reference_image.get_pixel(x, y), candidate_image.get_pixel(x, y))
|
var perceptual_error := _perceptual_color_error(reference_image.get_pixel(x, y), candidate_image.get_pixel(x, y))
|
||||||
error_sum += perceptual_error
|
error_sum += perceptual_error
|
||||||
if perceptual_error > float(options.pixel_threshold):
|
if perceptual_error > float(options.pixel_threshold):
|
||||||
@@ -144,11 +179,18 @@ func _compare_images(reference_path: String, candidate_path: String, options: Di
|
|||||||
"status": "passed" if passed else "threshold_exceeded",
|
"status": "passed" if passed else "threshold_exceeded",
|
||||||
"width": reference_image.get_width(),
|
"width": reference_image.get_width(),
|
||||||
"height": reference_image.get_height(),
|
"height": reference_image.get_height(),
|
||||||
|
"region": _region_array(comparison_region),
|
||||||
"mean_perceptual_error": mean_perceptual_error,
|
"mean_perceptual_error": mean_perceptual_error,
|
||||||
"changed_pixel_ratio": changed_pixel_ratio,
|
"changed_pixel_ratio": changed_pixel_ratio,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
func _region_array(region: Rect2i) -> Array[int]:
|
||||||
|
if region == Rect2i():
|
||||||
|
return []
|
||||||
|
return [region.position.x, region.position.y, region.size.x, region.size.y]
|
||||||
|
|
||||||
|
|
||||||
func _perceptual_color_error(reference_color: Color, candidate_color: Color) -> float:
|
func _perceptual_color_error(reference_color: Color, candidate_color: Color) -> float:
|
||||||
var reference_linear := reference_color.srgb_to_linear()
|
var reference_linear := reference_color.srgb_to_linear()
|
||||||
var candidate_linear := candidate_color.srgb_to_linear()
|
var candidate_linear := candidate_color.srgb_to_linear()
|
||||||
@@ -229,6 +271,22 @@ func _run_self_test() -> int:
|
|||||||
if not identical_report.passed or changed_report.passed:
|
if not identical_report.passed or changed_report.passed:
|
||||||
push_error("RENDER_CHECKPOINT_DIFF SELF_TEST: expected identical pass and changed failure")
|
push_error("RENDER_CHECKPOINT_DIFF SELF_TEST: expected identical pass and changed failure")
|
||||||
return 1
|
return 1
|
||||||
|
var region_options := options.duplicate()
|
||||||
|
region_options["region"] = Rect2i(1, 1, 1, 1)
|
||||||
|
var excluded_change_report := _compare_directories(reference_directory, changed_directory, region_options)
|
||||||
|
if not excluded_change_report.passed or excluded_change_report.comparison_region != [1, 1, 1, 1]:
|
||||||
|
push_error("RENDER_CHECKPOINT_DIFF SELF_TEST: region must exclude pixels outside the rectangle")
|
||||||
|
return 1
|
||||||
|
var out_of_bounds_options := options.duplicate()
|
||||||
|
out_of_bounds_options["region"] = Rect2i(1, 1, 2, 2)
|
||||||
|
var out_of_bounds_result := _compare_images(
|
||||||
|
reference_directory.path_join("synthetic.jpg"),
|
||||||
|
changed_directory.path_join("synthetic__cold_process.png"),
|
||||||
|
out_of_bounds_options
|
||||||
|
)
|
||||||
|
if out_of_bounds_result.status != "region_out_of_bounds":
|
||||||
|
push_error("RENDER_CHECKPOINT_DIFF SELF_TEST: out-of-bounds region must fail")
|
||||||
|
return 1
|
||||||
if reference_image.save_jpg(reference_directory.path_join("ignored.jpg"), 1.0) != OK:
|
if reference_image.save_jpg(reference_directory.path_join("ignored.jpg"), 1.0) != OK:
|
||||||
return 1
|
return 1
|
||||||
var filtered_options := options.duplicate()
|
var filtered_options := options.duplicate()
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ param(
|
|||||||
[double[]]$CameraFovValues = @(),
|
[double[]]$CameraFovValues = @(),
|
||||||
[int]$ViewportWidth = 1280,
|
[int]$ViewportWidth = 1280,
|
||||||
[int]$ViewportHeight = 900,
|
[int]$ViewportHeight = 900,
|
||||||
|
[int[]]$ComparisonRegion = @(),
|
||||||
[double]$WaitSeconds = 8.0,
|
[double]$WaitSeconds = 8.0,
|
||||||
[double]$MeasureSeconds = 0.1,
|
[double]$MeasureSeconds = 0.1,
|
||||||
[switch]$PlanOnly
|
[switch]$PlanOnly
|
||||||
@@ -33,6 +34,9 @@ if (-not (Test-Path -LiteralPath $GodotPath)) {
|
|||||||
if (-not (Test-Path -LiteralPath $ReferenceDirectory)) {
|
if (-not (Test-Path -LiteralPath $ReferenceDirectory)) {
|
||||||
throw "Reference directory not found: $ReferenceDirectory"
|
throw "Reference directory not found: $ReferenceDirectory"
|
||||||
}
|
}
|
||||||
|
if ($ComparisonRegion.Count -notin @(0, 4)) {
|
||||||
|
throw "ComparisonRegion requires x,y,width,height"
|
||||||
|
}
|
||||||
|
|
||||||
$invariantCulture = [Globalization.CultureInfo]::InvariantCulture
|
$invariantCulture = [Globalization.CultureInfo]::InvariantCulture
|
||||||
$isJointFovSweep = $CameraFovValues.Count -gt 0
|
$isJointFovSweep = $CameraFovValues.Count -gt 0
|
||||||
@@ -89,6 +93,9 @@ try {
|
|||||||
"--reference", $ReferenceDirectory, "--candidate", $candidateOutput,
|
"--reference", $ReferenceDirectory, "--candidate", $candidateOutput,
|
||||||
"--only", $Checkpoint, "--output", $comparisonReport
|
"--only", $Checkpoint, "--output", $comparisonReport
|
||||||
)
|
)
|
||||||
|
if ($ComparisonRegion.Count -eq 4) {
|
||||||
|
$comparisonArguments += @("--region", ($ComparisonRegion -join ","))
|
||||||
|
}
|
||||||
$comparisonProcess = Start-Process -FilePath $GodotPath -ArgumentList $comparisonArguments -Wait -PassThru -NoNewWindow
|
$comparisonProcess = Start-Process -FilePath $GodotPath -ArgumentList $comparisonArguments -Wait -PassThru -NoNewWindow
|
||||||
if ($comparisonProcess.ExitCode -notin @(0, 1)) {
|
if ($comparisonProcess.ExitCode -notin @(0, 1)) {
|
||||||
throw "Comparison failed for yaw=$yawText pitch=$pitchText"
|
throw "Comparison failed for yaw=$yawText pitch=$pitchText"
|
||||||
@@ -131,6 +138,7 @@ try {
|
|||||||
viewport = @($ViewportWidth, $ViewportHeight)
|
viewport = @($ViewportWidth, $ViewportHeight)
|
||||||
wait_seconds = $WaitSeconds
|
wait_seconds = $WaitSeconds
|
||||||
measure_seconds = $MeasureSeconds
|
measure_seconds = $MeasureSeconds
|
||||||
|
comparison_region = @($ComparisonRegion)
|
||||||
candidates = $rankedCandidates
|
candidates = $rankedCandidates
|
||||||
}
|
}
|
||||||
New-Item -ItemType Directory -Force -Path $absoluteOutput | Out-Null
|
New-Item -ItemType Directory -Force -Path $absoluteOutput | Out-Null
|
||||||
|
|||||||
Reference in New Issue
Block a user