test(M00): add checkpoint perceptual comparison #1
@@ -4,6 +4,8 @@
|
||||
|
||||
Парные checkpoint PNG можно автоматически проверить через `src/tools/compare_render_checkpoints.gd`; reference-снимки оригинального клиента остаются вне Git, а автоматический tolerance не заменяет human fidelity approval.
|
||||
|
||||
Paired run 2026-07-11 подтвердил крупный coordinate/placement gap: некоторые server-derived camera positions оказываются под terrain или внутри WMO/rocks OpenWC. До исправления этого расхождения perceptual metrics измеряют также несовпадение композиции, а не только материалы и свет.
|
||||
|
||||
Цель renderer-работы в этом проекте: добиться ощущения производительности оригинального клиента WoW 3.3.5a в Godot, без видимых фризов при переходе ADT -> ADT и без постоянного отката видимых участков к низкому качеству.
|
||||
|
||||
Этот документ фиксирует текущее состояние рендера, сделанные оптимизации и практические правила дальнейшей работы.
|
||||
|
||||
@@ -66,13 +66,13 @@ Provide a deterministic, headless perceptual comparison for paired renderer chec
|
||||
## Status
|
||||
|
||||
- State: ready
|
||||
- Done: comparator, runner integration, manifest tolerance contract, synthetic regression, module/baseline documentation, repository gates
|
||||
- Next: integrator review and paired run when approved build 12340 screenshots become available
|
||||
- Done: comparator, runner integration, manifest tolerance contract, synthetic regression, five local build 12340 references, three viewpoint calibrations, corrected visual capture and first paired report
|
||||
- Next: integrator review; coordinate/placement gap should be resolved before tolerance calibration
|
||||
- Blocked by: real paired screenshots only for final human fidelity approval
|
||||
|
||||
## Handoff
|
||||
|
||||
- Commit: branch HEAD
|
||||
- Results: synthetic identical image passed, changed image failed; full M00 dry-run and repository gates passed
|
||||
- Remaining risks: thresholds need calibration against approved paired captures; animation phase and camera alignment still require human review
|
||||
- Results: ten pairs compared with no missing candidates; mean error 0.0707..0.1746 and changed ratio 0.5504..0.8187; human review identified under-terrain/inside-geometry camera mismatches
|
||||
- Remaining risks: coordinate/placement mismatch blocks meaningful tolerance calibration; synthetic animation, dusk and full human approval remain incomplete
|
||||
- Documentation updated: `docs/RENDER_BASELINE.md`, `docs/modules/world-renderer.md`, `RENDER.md`
|
||||
|
||||
@@ -33,7 +33,7 @@
|
||||
-ReferenceCheckpointDirectory 'D:\private-fixtures\wow-3.3.5a-checkpoints'
|
||||
```
|
||||
|
||||
Результат записывается в `user://render_baseline/visual_comparison.json`. Другой путь задаётся через `-VisualComparisonReport`. `-DryRun` несовместим с визуальным сравнением, потому что не создаёт PNG.
|
||||
Результат записывается в `user://render_baseline/visual_comparison.json`. Другой путь задаётся через `-VisualComparisonReport`. `-DryRun` несовместим с визуальным сравнением, потому что не создаёт PNG. Runner сохраняет обычный performance capture в нормативном `1280×900`, затем отдельно снимает visual candidates в `2560×1440`; разрешение можно переопределить через `-VisualComparisonWidth` и `-VisualComparisonHeight`, не меняя performance baseline.
|
||||
|
||||
Отдельный запуск сравнения:
|
||||
|
||||
@@ -45,6 +45,8 @@ godot --headless --path . `
|
||||
--output user://render_baseline/visual_comparison.json
|
||||
```
|
||||
|
||||
Reference-каталог принимает JPG/JPEG/PNG оригинального клиента с именем checkpoint, например `goldshire_dense_m2.jpg`. Диагностические кадры с префиксом `diagnostic_` игнорируются. Один reference автоматически сопоставляется с `<checkpoint>__cold_process.png` и `<checkpoint>__warm_revisit.png` OpenWC.
|
||||
|
||||
Сравнение переводит sRGB в linear color space, вычисляет взвешенную ошибку яркости/цвета для каждого пикселя, среднюю ошибку кадра и долю пикселей выше локального порога. Нормативные defaults находятся в `comparison_budgets` manifest: mean error `0.015`, changed-pixel ratio `0.01`, pixel error threshold `0.04`. Любое превышение, несовпадение размеров, отсутствие пары или пустой reference-каталог возвращает ненулевой exit code.
|
||||
|
||||
Автоматический pass не является доказательством parity: animation phase, weather, camera alignment и intentional gaps требуют human approval. Для проверки алгоритма без proprietary данных используется:
|
||||
@@ -112,3 +114,9 @@ Baseline пока не имеет approved парных кадров ориги
|
||||
- возможная смена D3D12 на Vulkan при ошибке descriptor heap — фактический backend всегда берётся из `report.json`.
|
||||
|
||||
Парное сравнение с клиентом 3.3.5a должно использовать те же map position, local time и weather. Каждый gap регистрируется как terrain/material, placement, animation, lighting, liquid или culling; baseline сам по себе gap не закрывает.
|
||||
|
||||
Локальная сессия build 12340 от 2026-07-11 откалибровала три непригодные исходные позиции: dense-M2 camera была полностью закрыта кронами, large-WMO camera находилась у дымохода кузни, liquid camera попадала под terrain оригинального клиента. Manifest теперь содержит проверенные replacement camera positions. Пять approved локальных reference JPG хранятся вне Git; synthetic animated probe и dusk checkpoint всё ещё требуют отдельной процедуры.
|
||||
|
||||
Первый paired run после калибровки создал десять сравнений (пять reference × cold/warm), без missing pairs. Все пары ожидаемо превысили строгий tolerance: mean perceptual error `0.0707..0.1746`, changed-pixel ratio `0.5504..0.8187`. Human inspection показал, что это пока не чистая material/lighting ошибка: при тех же WoW-derived координатах OpenWC terrain-overview camera находится под terrain/placements, WMO camera — внутри таверны, liquid camera — внутри скал; ADT и dense-M2 композиции также существенно смещены. До исправления coordinate/placement mismatch эти значения являются gap evidence, а не основанием расширять tolerance.
|
||||
|
||||
Capture tool строит camera basis явно из target и world-up. Это исключает неоднозначный roll `look_at` при автоматической съёмке. `ViewportTexture.get_image()` сохраняется без дополнительного vertical flip для Godot 4.6.1.
|
||||
|
||||
@@ -194,7 +194,7 @@ Exact exported settings and cache versions remain documented in [`../../RENDER.m
|
||||
|
||||
- Unit/contract tests: material mapping, unique-ID dedupe, placement probes, baseline manifest, synthetic perceptual checkpoint diff.
|
||||
- Integration/E2E: Eastern Kingdoms/Kalimdor streaming scenes and seven cold/warm checkpoints.
|
||||
- Fidelity evidence: current fixed-position captures and automated paired-PNG metrics exist; approved original-client paired screenshots and human approval are still missing.
|
||||
- Fidelity evidence: пять локальных build 12340 reference JPG откалибровали terrain/ADT/M2/WMO/liquid viewpoints; automated paired-image metrics exist, но synthetic animation/dusk и полный human approval ещё не закрыты.
|
||||
- Performance budgets: M00 report records cold/warm p95 and max hitch; no final acceptance threshold yet.
|
||||
- Manual diagnostics: [`../RENDER_BASELINE.md`](../RENDER_BASELINE.md) and [`../../RENDER.md`](../../RENDER.md).
|
||||
|
||||
@@ -222,6 +222,7 @@ Exact exported settings and cache versions remain documented in [`../../RENDER.m
|
||||
- Monolithic streamer mixes planning, jobs, caches and presentation.
|
||||
- Direct camera path remains until M01/M03.
|
||||
- Original-client paired fidelity evidence incomplete.
|
||||
- Первый paired run выявил coordinate/placement mismatch: несколько совпадающих server-derived camera positions оказываются под terrain или внутри WMO/rocks OpenWC.
|
||||
- D3D12 descriptor and shutdown RID/resource issues remain.
|
||||
- M2/WMO/material/particle/ribbon/portal parity incomplete.
|
||||
- Public API is mostly exported configuration rather than stable contracts.
|
||||
@@ -239,7 +240,8 @@ Exact exported settings and cache versions remain documented in [`../../RENDER.m
|
||||
| `src/native/src/*_loader.cpp` | Native binary parsing |
|
||||
| `src/tools/build_*cache.gd`, `src/tools/bake_*cache.gd` | Offline cache generation |
|
||||
| `tools/run_render_baseline.ps1` | Unified M00 baseline runner |
|
||||
| `src/tools/compare_render_checkpoints.gd` | Offline paired-PNG perceptual metrics and JSON pass/fail report |
|
||||
| `src/tools/compare_render_checkpoints.gd` | Offline JPG/PNG paired-image perceptual metrics and JSON pass/fail report |
|
||||
| `src/tools/capture_render_checkpoints.gd` | Deterministic no-roll checkpoint camera, performance and visual capture |
|
||||
|
||||
## Related decisions and references
|
||||
|
||||
|
||||
@@ -25,7 +25,9 @@ func _capture_async() -> void:
|
||||
var headless := DisplayServer.get_name().to_lower() == "headless"
|
||||
var dry_run := args.has("--dry-run") or headless
|
||||
var viewport: Array = manifest.get("viewport", [1280, 900])
|
||||
get_root().size = Vector2i(maxi(16, int(viewport[0])), maxi(16, int(viewport[1])))
|
||||
var viewport_width := int(_arg(args, "--viewport-width", str(viewport[0])))
|
||||
var viewport_height := int(_arg(args, "--viewport-height", str(viewport[1])))
|
||||
get_root().size = Vector2i(maxi(16, viewport_width), maxi(16, viewport_height))
|
||||
|
||||
var abs_output_dir := ProjectSettings.globalize_path(output_dir)
|
||||
DirAccess.make_dir_recursive_absolute(abs_output_dir)
|
||||
@@ -107,7 +109,7 @@ func _capture_async() -> void:
|
||||
probe.scale = Vector3.ONE * float(checkpoint.get("probe_scale", 1.0))
|
||||
|
||||
camera.global_position = _vector3(checkpoint.get("camera", [0.0, 0.0, 0.0]))
|
||||
camera.look_at(_vector3(checkpoint.get("target", [0.0, 0.0, 0.0])), Vector3.UP)
|
||||
_orient_camera_without_roll(camera, _vector3(checkpoint.get("target", [0.0, 0.0, 0.0])))
|
||||
if player != null:
|
||||
player.global_position = _vector3(checkpoint.get("player", checkpoint.get("target", [0.0, 0.0, 0.0])))
|
||||
_set_sky_time(world, float(checkpoint.get("time_hours", 13.0)))
|
||||
@@ -135,7 +137,6 @@ func _capture_async() -> void:
|
||||
await RenderingServer.frame_post_draw
|
||||
|
||||
var image := get_root().get_texture().get_image()
|
||||
image.flip_y()
|
||||
var file_name := "%s__%s.png" % [checkpoint_name, pass_name]
|
||||
var abs_path := abs_output_dir.path_join(file_name)
|
||||
var err := image.save_png(abs_path)
|
||||
@@ -196,6 +197,15 @@ func _measure_frames(seconds: float, world: Node) -> Dictionary:
|
||||
}
|
||||
|
||||
|
||||
func _orient_camera_without_roll(camera: Camera3D, target_position: Vector3) -> void:
|
||||
var forward := (target_position - camera.global_position).normalized()
|
||||
var right := forward.cross(Vector3.UP).normalized()
|
||||
if right.is_zero_approx():
|
||||
right = Vector3.RIGHT
|
||||
var corrected_up := right.cross(forward).normalized()
|
||||
camera.global_basis = Basis(right, corrected_up, -forward)
|
||||
|
||||
|
||||
func _result_record(checkpoint: Dictionary, pass_name: String, load_time_ms: float, metrics: Dictionary, sha256: String) -> Dictionary:
|
||||
return {
|
||||
"name": checkpoint.get("name", "checkpoint"),
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
extends SceneTree
|
||||
|
||||
## Compares paired PNG checkpoint directories without storing proprietary reference images.
|
||||
## Compares original-client JPG/PNG references with OpenWC checkpoint PNGs.
|
||||
## Usage: godot --headless --path . --script res://src/tools/compare_render_checkpoints.gd --
|
||||
## --reference <directory> --candidate <directory> [--output <report.json>]
|
||||
## [--pixel-threshold 0.04] [--mean-threshold 0.015] [--changed-ratio-threshold 0.01]
|
||||
@@ -73,19 +73,26 @@ func _parse_arguments(raw_arguments: PackedStringArray) -> Dictionary:
|
||||
|
||||
|
||||
func _compare_directories(reference_directory: String, candidate_directory: String, options: Dictionary) -> Dictionary:
|
||||
var reference_files := _png_file_names(reference_directory)
|
||||
reference_directory = ProjectSettings.globalize_path(reference_directory)
|
||||
candidate_directory = ProjectSettings.globalize_path(candidate_directory)
|
||||
var reference_files := _reference_image_file_names(reference_directory)
|
||||
var results: Array[Dictionary] = []
|
||||
var failed_count := 0
|
||||
var missing_count := 0
|
||||
for file_name in reference_files:
|
||||
var reference_path := reference_directory.path_join(file_name)
|
||||
var candidate_path := candidate_directory.path_join(file_name)
|
||||
if not FileAccess.file_exists(candidate_path):
|
||||
results.append({"name": file_name, "status": "missing_candidate"})
|
||||
var checkpoint_name := file_name.get_basename()
|
||||
var candidate_file_names := _candidate_file_names(candidate_directory, checkpoint_name)
|
||||
if candidate_file_names.is_empty():
|
||||
results.append({"checkpoint": checkpoint_name, "reference": file_name, "status": "missing_candidate"})
|
||||
missing_count += 1
|
||||
continue
|
||||
for candidate_file_name in candidate_file_names:
|
||||
var candidate_path := candidate_directory.path_join(candidate_file_name)
|
||||
var comparison := _compare_images(reference_path, candidate_path, options)
|
||||
comparison.name = file_name
|
||||
comparison.checkpoint = checkpoint_name
|
||||
comparison.reference = file_name
|
||||
comparison.candidate = candidate_file_name
|
||||
results.append(comparison)
|
||||
if comparison.status != "passed":
|
||||
failed_count += 1
|
||||
@@ -149,18 +156,32 @@ func _perceptual_color_error(reference_color: Color, candidate_color: Color) ->
|
||||
return 0.2126 * red_delta + 0.7152 * green_delta + 0.0722 * blue_delta + 0.25 * alpha_delta
|
||||
|
||||
|
||||
func _png_file_names(directory_path: String) -> Array[String]:
|
||||
func _reference_image_file_names(directory_path: String) -> Array[String]:
|
||||
var directory := DirAccess.open(directory_path)
|
||||
if directory == null:
|
||||
return []
|
||||
var file_names: Array[String] = []
|
||||
for file_name in directory.get_files():
|
||||
if file_name.to_lower().ends_with(".png"):
|
||||
var lower_name := file_name.to_lower()
|
||||
if not lower_name.begins_with("diagnostic_") and (lower_name.ends_with(".png") or lower_name.ends_with(".jpg") or lower_name.ends_with(".jpeg")):
|
||||
file_names.append(file_name)
|
||||
file_names.sort()
|
||||
return file_names
|
||||
|
||||
|
||||
func _candidate_file_names(candidate_directory: String, checkpoint_name: String) -> Array[String]:
|
||||
var candidates: Array[String] = []
|
||||
for pass_name in ["cold_process", "warm_revisit"]:
|
||||
var file_name := "%s__%s.png" % [checkpoint_name, pass_name]
|
||||
if FileAccess.file_exists(candidate_directory.path_join(file_name)):
|
||||
candidates.append(file_name)
|
||||
if candidates.is_empty():
|
||||
var exact_file_name := "%s.png" % checkpoint_name
|
||||
if FileAccess.file_exists(candidate_directory.path_join(exact_file_name)):
|
||||
candidates.append(exact_file_name)
|
||||
return candidates
|
||||
|
||||
|
||||
func _write_report(output_path: String, report: Dictionary) -> bool:
|
||||
var absolute_path := ProjectSettings.globalize_path(output_path)
|
||||
var error := DirAccess.make_dir_recursive_absolute(absolute_path.get_base_dir())
|
||||
@@ -186,14 +207,13 @@ func _run_self_test() -> int:
|
||||
reference_image.fill(Color(0.25, 0.5, 0.75, 1.0))
|
||||
var changed_image := reference_image.duplicate()
|
||||
changed_image.set_pixel(0, 0, Color.WHITE)
|
||||
for pair in [
|
||||
[reference_directory, reference_image],
|
||||
[identical_directory, reference_image],
|
||||
[changed_directory, changed_image],
|
||||
]:
|
||||
var save_error: Error = pair[1].save_png(String(pair[0]).path_join("synthetic.png"))
|
||||
if save_error != OK:
|
||||
push_error("RENDER_CHECKPOINT_DIFF SELF_TEST: cannot save fixture")
|
||||
if reference_image.save_jpg(reference_directory.path_join("synthetic.jpg"), 1.0) != OK:
|
||||
push_error("RENDER_CHECKPOINT_DIFF SELF_TEST: cannot save JPG reference")
|
||||
return 1
|
||||
for pass_name in ["cold_process", "warm_revisit"]:
|
||||
if reference_image.save_png(identical_directory.path_join("synthetic__%s.png" % pass_name)) != OK:
|
||||
return 1
|
||||
if changed_image.save_png(changed_directory.path_join("synthetic__%s.png" % pass_name)) != OK:
|
||||
return 1
|
||||
var options := {
|
||||
"pixel_threshold": DEFAULT_PIXEL_THRESHOLD,
|
||||
|
||||
@@ -5,6 +5,7 @@
|
||||
"map": "Azeroth",
|
||||
"quality_preset": "High",
|
||||
"viewport": [1280, 900],
|
||||
"fidelity_comparison_viewport": [2560, 1440],
|
||||
"camera_fov": 62.0,
|
||||
"default_wait_seconds": 8.0,
|
||||
"default_measure_seconds": 3.0,
|
||||
@@ -57,7 +58,7 @@
|
||||
{
|
||||
"name": "goldshire_dense_m2",
|
||||
"coverage": ["dense_m2"],
|
||||
"camera": [16835.0, 135.0, 26435.0],
|
||||
"camera": [16956.666, 150.0, 26466.666],
|
||||
"target": [17015.0, 62.0, 26525.0],
|
||||
"player": [17015.0, 58.0, 26525.0],
|
||||
"time_hours": 13.0
|
||||
@@ -65,7 +66,7 @@
|
||||
{
|
||||
"name": "goldshire_inn_large_wmo",
|
||||
"coverage": ["large_wmo"],
|
||||
"camera": [16942.0, 82.0, 26503.0],
|
||||
"camera": [17013.666, 72.0, 26451.666],
|
||||
"target": [17042.27, 66.0, 26530.91],
|
||||
"player": [17042.27, 58.0, 26530.91],
|
||||
"time_hours": 13.0
|
||||
@@ -73,7 +74,7 @@
|
||||
{
|
||||
"name": "elwynn_waterfall_liquid",
|
||||
"coverage": ["liquid"],
|
||||
"camera": [16445.0, 125.0, 26295.0],
|
||||
"camera": [16481.666, 190.0, 26366.666],
|
||||
"target": [16518.84, 68.0, 26427.27],
|
||||
"player": [16518.84, 55.0, 26427.27],
|
||||
"time_hours": 13.0
|
||||
|
||||
@@ -20,6 +20,8 @@ func _initialize() -> void:
|
||||
_expect(int(manifest.get("schema_version", 0)) == 1, "manifest schema must be version 1", failures)
|
||||
_expect(String(manifest.get("profile", "")) == "Blizzlike335", "manifest profile must be Blizzlike335", failures)
|
||||
_expect(ResourceLoader.exists(String(manifest.get("scene", ""))), "streaming scene must exist", failures)
|
||||
var fidelity_viewport: Array = manifest.get("fidelity_comparison_viewport", [])
|
||||
_expect(fidelity_viewport.size() == 2 and int(fidelity_viewport[0]) > 0 and int(fidelity_viewport[1]) > 0, "fidelity comparison viewport must contain positive width and height", failures)
|
||||
|
||||
var covered := {}
|
||||
var names := {}
|
||||
|
||||
@@ -7,6 +7,8 @@ param(
|
||||
[double]$MeasureSeconds = 3.0,
|
||||
[string]$ReferenceCheckpointDirectory,
|
||||
[string]$VisualComparisonReport = "user://render_baseline/visual_comparison.json",
|
||||
[int]$VisualComparisonWidth = 2560,
|
||||
[int]$VisualComparisonHeight = 1440,
|
||||
[switch]$DryRun
|
||||
)
|
||||
|
||||
@@ -63,12 +65,26 @@ try {
|
||||
if ($DryRun) {
|
||||
throw "Reference checkpoint comparison requires PNG capture; remove -DryRun"
|
||||
}
|
||||
$visualCandidateOutput = "$Output/visual_comparison_candidates"
|
||||
$visualCaptureArgs = @(
|
||||
"--path", ".",
|
||||
"--script", "res://src/tools/capture_render_checkpoints.gd",
|
||||
"--",
|
||||
"--output", $visualCandidateOutput,
|
||||
"--cache-state", $CacheState,
|
||||
"--revision", $revision,
|
||||
"--wait", $WaitSeconds.ToString([Globalization.CultureInfo]::InvariantCulture),
|
||||
"--measure", $MeasureSeconds.ToString([Globalization.CultureInfo]::InvariantCulture),
|
||||
"--viewport-width", $VisualComparisonWidth.ToString(),
|
||||
"--viewport-height", $VisualComparisonHeight.ToString()
|
||||
)
|
||||
Invoke-GodotStep "visual-capture" $visualCaptureArgs
|
||||
Invoke-GodotStep "visual-comparison" @(
|
||||
"--headless", "--path", ".",
|
||||
"--script", "res://src/tools/compare_render_checkpoints.gd",
|
||||
"--",
|
||||
"--reference", $ReferenceCheckpointDirectory,
|
||||
"--candidate", $Output,
|
||||
"--candidate", $visualCandidateOutput,
|
||||
"--output", $VisualComparisonReport
|
||||
)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user