From 4a8338f2f0503c2ba31c495e00a696d359518c06 Mon Sep 17 00:00:00 2001 From: sindoring Date: Sat, 18 Jul 2026 02:23:40 +0400 Subject: [PATCH] render: extract M2 build queue --- RENDER.md | 6 +- docs/modules/README.md | 1 + docs/modules/m2-build-batch-planner.md | 17 +- docs/modules/m2-build-queue.md | 258 ++++++++++++++++ docs/modules/world-renderer.md | 19 +- src/render/m2/m2_build_job.gd | 78 +++++ src/render/m2/m2_build_job.gd.uid | 1 + src/render/m2/m2_build_queue.gd | 169 +++++++++++ src/render/m2/m2_build_queue.gd.uid | 1 + .../streaming/streaming_world_loader.gd | 103 +++---- src/tools/verify_m2_build_queue.gd | 279 ++++++++++++++++++ src/tools/verify_m2_build_queue.gd.uid | 1 + targets/03-renderer-facade.md | 17 +- 13 files changed, 882 insertions(+), 68 deletions(-) create mode 100644 docs/modules/m2-build-queue.md create mode 100644 src/render/m2/m2_build_job.gd create mode 100644 src/render/m2/m2_build_job.gd.uid create mode 100644 src/render/m2/m2_build_queue.gd create mode 100644 src/render/m2/m2_build_queue.gd.uid create mode 100644 src/tools/verify_m2_build_queue.gd create mode 100644 src/tools/verify_m2_build_queue.gd.uid diff --git a/RENDER.md b/RENDER.md index eaa405e..4f74da3 100644 --- a/RENDER.md +++ b/RENDER.md @@ -40,6 +40,7 @@ Paired run 2026-07-11 подтвердил крупный coordinate/placement g - `src/render/m2/m2_animation_playback_controller.gd` - deterministic per-instance phase, default animation selection and native/imported playback mutation. - `src/render/m2/m2_animated_instance_materializer.gd` - main-thread animated instance duplication, render settings, playback startup and non-empty batch attachment. - `src/render/m2/m2_static_batch_materializer.gd` - main-thread static M2 MultiMesh construction, render settings and attachment. +- `src/render/m2/m2_build_queue.gd` / `m2_build_job.gd` - typed pending M2 jobs, FIFO/stale tile keys, grouped-transform references and progress cursors without engine destruction. - `src/render/m2/m2_animation_load_pipeline_state.gd` - animated M2 threaded-load request records and completion-order finalize FIFO without I/O or Node ownership. - `src/render/m2/m2_mesh_load_pipeline_state.gd` - static M2 threaded-load request records, terminal statuses and completion-order finalize FIFO without I/O or Mesh ownership. - `src/render/m2/m2_mesh_resource_cache_state.gd` - normalized-path prepared static M2 Mesh references with final-shutdown lifetime. @@ -1051,8 +1052,9 @@ $exe = Join-Path $env:TEMP 'godot-4.6.1-openwc\Godot_v4.6.1-stable_win64.exe' remaining-count capping and next-offset/group-completion calculation. - Existing animated/static limits still clamp to at least one; resource readiness, queue rotation, serial progression and `M2_BUILD` permit consumption are unchanged. -- The loader retains build jobs/queues, tile cancellation, animation/mesh caches, - retries and all Node/MultiMesh/Mesh/material/RID finalization. +- `M2BuildQueue` now retains typed pending jobs, FIFO/stale keys and cursors. + The loader retains tile cancellation, resource readiness/retries and all + Node/MultiMesh/Mesh/material/RID finalization. - Cache formats, quality profiles, batching output and visible rules are unchanged. Asset-backed p95/p99 and spatial-cell batching evidence remain pending. diff --git a/docs/modules/README.md b/docs/modules/README.md index 760aee7..f44ba4e 100644 --- a/docs/modules/README.md +++ b/docs/modules/README.md @@ -21,6 +21,7 @@ | M2 placement transform resolver | Implemented | [`m2-placement-transform-resolver.md`](m2-placement-transform-resolver.md) | | M2 placement grouper | Implemented extraction | [`m2-placement-grouper.md`](m2-placement-grouper.md) | | M2 build batch planner | Implemented extraction | [`m2-build-batch-planner.md`](m2-build-batch-planner.md) | +| M2 build queue | Implemented extraction | [`m2-build-queue.md`](m2-build-queue.md) | | M2 static batch materializer | Implemented extraction | [`m2-static-batch-materializer.md`](m2-static-batch-materializer.md) | | M2 runtime mesh rebuild classifier | Implemented extraction | [`m2-runtime-mesh-rebuild-classifier.md`](m2-runtime-mesh-rebuild-classifier.md) | | M2 animated scene finalizer | Implemented extraction | [`m2-animated-scene-finalizer.md`](m2-animated-scene-finalizer.md) | diff --git a/docs/modules/m2-build-batch-planner.md b/docs/modules/m2-build-batch-planner.md index 2f39c79..efb36af 100644 --- a/docs/modules/m2-build-batch-planner.md +++ b/docs/modules/m2-build-batch-planner.md @@ -102,8 +102,8 @@ sequenceDiagram ## Ownership, threading and resources - The planner owns only call-local scalar values and the returned Dictionary. -- The loader owns build-job Dictionaries, queue ordering, tile checks, resource - readiness/retry, serial numbers and group-index mutation. +- `M2BuildQueue` owns typed jobs, FIFO ordering, serial numbers and group/offset + cursors. The loader owns tile checks, resource readiness/retry and adoption calls. - Materializers own main-thread Node/MultiMesh construction under loader roots. - The scheduler owns the frame-local `M2_BUILD` counter. - Pure planning is thread-safe, though the current adapter calls it on main thread. @@ -149,8 +149,8 @@ queue depth, build activity and hitch observability. ## Extension points -- A later package may define typed build-job state once queue/resource transitions - are extracted together with explicit cancellation and recovery. +- A later package may extract resource readiness/dispatch while retaining the + typed build-job and FIFO contracts defined by `M2BuildQueue`. - Spatial-cell batching must use measured culling/performance evidence and must not silently change this model-path batch cursor. @@ -159,12 +159,14 @@ queue depth, build activity and hitch observability. | Capability | Status | Evidence | Gap/next step | |---|---|---|---| | Static/animated batch cursor planning | Implemented extraction | Contract/source/timing verifier | Asset-backed p95/p99 pending | -| Build queue/resource state machine | Remains in loader | Existing lifecycle regressions | Stateful extraction pending | +| Typed build queue/cursor state | Implemented extraction | M2 build queue lifecycle verifier | Asset-backed traversal pending | +| Resource readiness/dispatch | Remains in loader | Existing lifecycle regressions | Stateful extraction pending | | Spatial-cell batching | Planned | Renderer roadmap | Culling evidence/design pending | ## Known gaps and risks -- The result remains a Dictionary while loader build jobs are untyped Dictionaries. +- The planner result remains a Dictionary; `M2BuildQueue` adopts its scalar + cursor values through one explicit progress call. - Negative offsets are preserved rather than rejected to avoid a hidden behavior change. - Planning timing does not measure resource loading or materialization. - Private asset-backed p95/p99 and visual comparison remain unavailable. @@ -174,7 +176,8 @@ queue depth, build activity and hitch observability. | Path | Responsibility | |---|---| | `src/render/m2/m2_build_batch_planner.gd` | Pure limit/count/cursor planning | -| `src/scenes/streaming/streaming_world_loader.gd` | Build job, queue, resource readiness, materializer adapters and budgets | +| `src/render/m2/m2_build_queue.gd` | Typed pending jobs, FIFO order and cursor ownership | +| `src/scenes/streaming/streaming_world_loader.gd` | Tile checks, resource readiness, progress adoption, materializer adapters and budgets | | `src/render/m2/m2_static_batch_materializer.gd` | Planned static-slice MultiMesh construction and attachment | | `src/render/m2/m2_animation_playback_controller.gd` | Animated-instance phase/selection/native/imported playback | | `src/render/m2/m2_animated_instance_materializer.gd` | Planned animated-slice duplication, playback startup and attachment | diff --git a/docs/modules/m2-build-queue.md b/docs/modules/m2-build-queue.md new file mode 100644 index 0000000..c867c33 --- /dev/null +++ b/docs/modules/m2-build-queue.md @@ -0,0 +1,258 @@ +# M2 Build Queue + +## Metadata + +| Field | Value | +|---|---| +| Status | Implemented | +| Target/work package | M03 / `M03-RND-M2-BUILD-QUEUE-001` | +| Owners | Pending M2 build-job records, FIFO tile keys and build cursors | +| Last verified | Worktree `work/sindo-main-codex/m03-m2-build-queue`, 2026-07-18 | +| Profiles/capabilities | Existing static MultiMesh and animated-instance build paths | + +## Purpose + +Own typed pending M2 build state outside `StreamingWorldLoader`: a keyed job +retains its M2 root, grouped transforms, insertion-order group keys and three +progress cursors, while a separate FIFO retains scheduling order and stale keys. + +## Non-goals + +- Decide tile eligibility, detail radius, resource readiness or retries. +- Consume scheduler permits or plan batch sizes. +- Create, attach or free Nodes, Meshes, MultiMeshes or animated instances. +- Load/cache resources, group placements or mutate tile state. +- Change model order, transform order, render settings or visible output. + +## Context and boundaries + +```mermaid +flowchart LR + Groups[M2PlacementGrouper result] --> Loader[StreamingWorldLoader adapter] + Loader --> Queue[M2BuildQueue] + Queue --> Job[M2BuildJob] + Queue --> Loader + Loader --> Planner[M2BuildBatchPlanner] + Loader --> Static[M2StaticBatchMaterializer] + Loader --> Animated[M2AnimatedInstanceMaterializer] + Loader --> Scheduler[RenderBudgetScheduler] +``` + +Allowed dependencies are String, Dictionary, Array, RefCounted and a strong +Node3D reference. SceneTree destruction, materializers, Mesh/MultiMesh, +RenderingServer/RIDs, ResourceLoader/files, workers/mutexes, scheduler policy, +gameplay, network and Editor APIs are forbidden. + +## Public API + +### `M2BuildJob` + +| Symbol | Kind | Purpose | Thread/lifetime | Errors | +|---|---|---|---|---| +| `tile_key()` | Query | Immutable keyed identity | Main thread; job lifetime | None | +| `root()` | Query | Borrow strongly retained M2 root | Main thread; no ownership transfer | May be externally invalidated | +| `groups()` | Query | Borrow exact grouped-transform Dictionary | Main thread; job lifetime | None | +| `group_keys()` | Query | Borrow enqueue-time insertion-order key snapshot | Main thread; job lifetime | None | +| `group_index()` | Query | Current model-path cursor | Main thread | None | +| `transform_offset()` | Query | Current transform cursor within group | Main thread | None | +| `batch_serial()` | Query | Current batch naming serial | Main thread | None | +| `adopt_progress(group_index, offset, serial)` | Command | Atomically replace all progress values | Main thread | Raw integer semantics retained | +| `diagnostic_snapshot()` | Query | Detached scalar/count state | Any serialized caller | Omits engine references | + +### `M2BuildQueue` + +| Symbol | Kind | Purpose | Thread/lifetime | Errors | +|---|---|---|---|---| +| `enqueue(tile_key, root, groups)` | Command | Replace keyed job and append FIFO key | Main thread; until erase/clear | Invalid/empty input returns false | +| `has_pending/front_key/pop_front/rotate_front` | Commands/queries | Inspect, drain or rotate FIFO independently of job validity | Main thread | Empty returns false/empty String | +| `has_job/job_for` | Queries | Inspect keyed record, including stale-key distinction | Main thread | Missing returns false/null | +| `root_for/groups_for/group_keys_for` | Queries | Borrow current job inputs | Main thread | Stale returns null/empty collection | +| `group_index_for/transform_offset_for/batch_serial_for` | Queries | Read current progress | Main thread | Stale returns zero | +| `adopt_progress(tile_key, group_index, offset, serial)` | Command | Atomically update current job progress | Main thread | Stale returns false | +| `erase_job(tile_key)` | Command | Remove keyed job but preserve FIFO keys | Main thread | Unknown returns false | +| `job_keys()` | Query | Detached active-key snapshot for loader cleanup | Main thread | Empty returns empty Array | +| `clear()` | Command | Release all job and FIFO references | Main thread | Idempotent; never frees Nodes | +| `pending_count/active_job_count` | Queries | FIFO and keyed-job metrics | Main thread | None | +| `diagnostic_snapshot()` | Query | Detached FIFO and sorted job diagnostics | Main thread | None | + +## Inputs and outputs + +| Direction | Contract/data | Producer | Consumer | Ownership | Thread/lifetime | +|---|---|---|---|---|---| +| Input | Tile key and M2 root | Loader group-result finalizer | Queue/job | Copied String; strong borrowed Node reference | Until erase/clear | +| Input | Grouped transforms Dictionary | M2PlacementGrouper through loader mailbox | Job | Exact strong reference | Job lifetime | +| Input | `groups.keys()` order | Job constructor | Loader group selection | Job-owned Array snapshot | Job lifetime | +| Input | Next group/offset/serial | Planner/materializer adapter | Job progress | Copied integers | Until next adoption | +| Output | Front/rotated/popped tile key | Queue | Loader scheduling loop | Copied String | One drain step | +| Output | Borrowed root/groups/group keys | Job/queue | Loader readiness/materialization | No ownership transfer | One drain step | +| Output | Detached diagnostics/key snapshots | Queue/job | Tests/metrics/cleanup adapter | Caller-owned collections | Call result | + +Side effects are keyed/FIFO collection mutation, strong-reference retention and +cursor mutation. No engine object is created, destroyed or attached. + +## Data flow + +```mermaid +flowchart TD + Enqueue[Tile key, root and grouped transforms] --> Job[Create or replace M2BuildJob] + Enqueue --> FIFO[Append tile key to FIFO] + FIFO --> Front[Borrow front key] + Front --> Current{Current keyed job exists?} + Current -->|no| Pop[Pop stale key] + Current -->|yes| Loader[Loader eligibility/readiness/materialization] + Loader -->|resource pending| Rotate[Move front key to tail] + Loader -->|operation complete| Progress[Adopt group, offset and serial] + Loader -->|finish/cancel| Erase[Erase job only] + Erase --> Pop +``` + +## Lifecycle/state + +```mermaid +stateDiagram-v2 + [*] --> Absent + Absent --> Queued: enqueue valid job and FIFO key + Queued --> Rotated: resource pending + Rotated --> Queued: returns to FIFO front + Queued --> Progressed: adopt progress + Progressed --> Queued: group remains + Queued --> StaleQueued: erase job before FIFO pop + Progressed --> StaleQueued: finish or cancel + StaleQueued --> Absent: pop stale key + Queued --> Absent: clear + StaleQueued --> Absent: clear +``` + +Duplicate enqueue replaces the current keyed job and adds another FIFO entry; +later erase can therefore leave multiple stale entries, matching historical raw state. + +## Main sequence + +```mermaid +sequenceDiagram + participant G as M2 group-result drain + participant Q as M2BuildQueue + participant L as StreamingWorldLoader + participant P as M2BuildBatchPlanner + participant M as M2 materializer + G->>Q: enqueue(tile key, root, groups) + L->>Q: front_key and has_job + Q-->>L: borrowed root/groups/keys and cursors + L->>P: plan selected transform slice + alt resource pending + L->>Q: rotate_front + else materializable + L->>M: materialize selected slice + L->>Q: adopt_progress(next index, offset, serial) + else cancelled or complete + L->>L: free/queue-free root if required + L->>Q: erase_job + L->>Q: pop_front + end +``` + +## Dependency diagram + +```mermaid +flowchart TB + Loader[StreamingWorldLoader] --> Queue[M2BuildQueue] + Queue --> Job[M2BuildJob] + Job --> EngineBase[Node3D strong reference] + Loader --> Planner[M2BuildBatchPlanner] + Loader --> Materializers[Static and animated materializers] + Loader --> Scheduler[RenderBudgetScheduler] + Queue -. no dependency .-> SceneMutation[SceneTree destruction/attachment] + Queue -. no dependency .-> Resources[Mesh / MultiMesh / ResourceLoader] +``` + +## Ownership, threading and resources + +- Queue owns keyed job records and FIFO String entries. +- Job retains the exact groups Dictionary, a fresh group-key snapshot and root. +- Root ownership remains with the loader/tile SceneTree; queue release never frees it. +- Loader validates `is_instance_valid`, frees empty/aborted roots and mutates tile state. +- All current operations run on the renderer main thread; no mutex is required. +- Group worker results cross their existing mutex mailbox before enqueue. + +## Errors, cancellation and recovery + +| Failure/state | Detection | Behavior | Diagnostic | Recovery | +|---|---|---|---|---| +| Invalid enqueue input | Entry guard | Reject without mutation | Contract fixture | Correct caller composition | +| Duplicate tile key | Keyed assignment | Replace job and append FIFO key | Duplicate fixture | Stale entries drain normally | +| Stale FIFO key | `has_job == false` | Loader pops and continues without permit | Stale fixture | No action required | +| Resource pending | Loader cache state | Rotate one front key and consume existing permit | Rotation fixture/build diagnostics | Retry when key returns front | +| Root externally invalid | Loader validity check | Erase job and pop key | Loader source contract | Tile may regroup/requeue later | +| Tile cancelled | Loader policy | Queue-free root, erase job; FIFO key drains/pops | Shutdown/lifetime regression | Eligible tile may requeue | +| World clear | Loader job-key snapshot | Cancel roots, then clear queue | Shutdown regression | New map starts empty | + +## Configuration and capabilities + +The queue introduces no settings. Existing static/animated batch limits, +visibility/shadow settings, resource caches and `M2_BUILD` permits remain loader, +planner, materializer and scheduler contracts. + +## Persistence, cache and migration + +No state is serialized. No ADT/M2 cache format or version changes; no rebake or +migration is required. + +## Diagnostics and observability + +- `pending_count` preserves the existing FIFO-backed `m2_build` queue metric. +- `active_job_count` distinguishes keyed jobs from stale/duplicate FIFO entries. +- Diagnostic snapshots retain FIFO order and sort job records by tile key. +- Job diagnostics expose cursors/counts but never root/groups references. +- The module emits no logs. + +## Verification + +- `verify_m2_build_queue.gd` covers invalid input, exact groups/root retention, + FIFO, duplicate replacement, rotation, stale entries, atomic progress, erase/ + clear engine lifetime, detached sorted diagnostics and loader boundaries. +- Planner, static/animated materializer, Mesh pipeline/cache/prototype, shutdown, + facade, internal-access and checkpoint regressions protect adjacent behavior. +- Fidelity evidence is exact state-transition extraction; visual rules are not changed. +- Performance budget: 100 cycles of 256 enqueue/rotate/erase operations under one second. +- No original-client or proprietary asset comparison is claimed for bookkeeping state. + +## Extension points + +Remaining resource readiness and dispatch logic may later move behind explicit +commands once its cancellation semantics are independently fixed. A generic +queue base, signals and callbacks are intentionally excluded. + +## Capability status + +| Capability | Status | Evidence | Gap/next step | +|---|---|---|---| +| Typed keyed M2 jobs and FIFO | Implemented extraction | Lifecycle/order/source/timing verifier | Asset-backed traversal pending | +| Cursor/serial ownership | Implemented extraction | Atomic progress fixtures | Typed batch-plan result remains Dictionary | +| Root destruction | Existing loader-owned | Lifetime/source/shutdown regressions | Keep outside state service | +| Resource readiness/dispatch | Existing loader-owned | Adjacent cache/build tests | Safe extraction remains | + +## Known gaps and risks + +- Grouped transforms remain untyped Dictionary/Array data. +- Queue permits duplicate and stale FIFO keys intentionally for compatibility. +- Job accessors return borrowed mutable groups/key collections; current loader is the sole consumer. +- Private asset traversal, long-flight p95/p99 and leak evidence remain unavailable. + +## Source map + +| Path | Responsibility | +|---|---| +| `src/render/m2/m2_build_job.gd` | Strong root/groups references, key snapshot and progress cursors | +| `src/render/m2/m2_build_queue.gd` | Keyed job ownership, FIFO/stale/rotation lifecycle and diagnostics | +| `src/scenes/streaming/streaming_world_loader.gd` | Eligibility, readiness, permits, materialization, tile state and root cleanup | +| `src/render/m2/m2_build_batch_planner.gd` | Batch count and cursor-plan calculation | +| `src/tools/verify_m2_build_queue.gd` | Lifecycle/order/lifetime/boundary/timing regression | + +## Related decisions and references + +- [`m2-build-batch-planner.md`](m2-build-batch-planner.md) +- [`m2-static-batch-materializer.md`](m2-static-batch-materializer.md) +- [`m2-animated-instance-materializer.md`](m2-animated-instance-materializer.md) +- [`world-renderer.md`](world-renderer.md) +- [`../../RENDER.md`](../../RENDER.md) +- [`../../targets/roadmap/02-rendering-and-graphics.md`](../../targets/roadmap/02-rendering-and-graphics.md) diff --git a/docs/modules/world-renderer.md b/docs/modules/world-renderer.md index 24d4bae..037487e 100644 --- a/docs/modules/world-renderer.md +++ b/docs/modules/world-renderer.md @@ -54,6 +54,8 @@ flowchart LR M2Transform --> Loader Loader --> M2Batch[M2BuildBatchPlanner] M2Batch --> Loader + Loader --> M2Queue[M2BuildQueue] + M2Queue --> Loader Loader --> M2Static[M2StaticBatchMaterializer] M2Static --> Scene Loader --> WmoPlacement[WmoPlacementResolver] @@ -134,6 +136,7 @@ from externally reading/writing loader-private queue, task, cache and tile-state | `M2PlacementTransformResolver.resolve_basis/resolve_origin_offset` | Internal pure M2 service | Resolves regular and calibrated model-specific ADT placement transforms | Worker/main thread; stateless | Unknown paths use regular basis and zero offset | | `M2PlacementGrouper.group_placements` | Internal pure M2 service | Validates and groups ordered tile-local placement transforms by normalized path | Worker/main thread; stateless | Invalid variants/name IDs/empty paths are skipped | | `M2BuildBatchPlanner.plan_batch` | Internal pure M2 service | Selects static/animated batch count and next group cursor | Main/any thread; stateless | Non-positive selected limit clamps to one; empty range completes | +| `M2BuildQueue` / `M2BuildJob` | Internal M2 pending-state service | Own typed root/groups/cursor jobs and FIFO/stale tile keys | Renderer main thread; map session | Invalid enqueue rejected; stale keys drain independently of jobs | | `M2StaticBatchMaterializer.materialize_batch` | Internal M2 scene-materialization service | Builds and attaches one prepared-Mesh MultiMesh transform slice | Renderer main thread; stateless after each call | Invalid/empty input returns null; bounds are caller precondition | | `M2RuntimeMeshRebuildClassifier` | Internal memoized M2 service | Detects billboard/UV-rotation metadata requiring stale cached-mesh rebuild | Renderer main thread; cached until reset | Invalid variants/indices skipped; first path decision wins | | `M2AnimationLoadPipelineState` | Internal M2 async-state service | Owns animated scene pending request records and terminal finalize FIFO | Renderer main thread; map/session | Empty/duplicate/unknown transitions rejected | @@ -182,6 +185,7 @@ loader configuration remains transitional composition data, not a caller API. | Internal transform | Rotation/path/scale | Loader or grouper / `M2PlacementTransformResolver` | Group/placeholder/instance transforms | Value-only Basis/Vector3 | One placement | | Internal grouping | Tile origin, M2 names and placements | Loader / `M2PlacementGrouper` | Loader worker result/build job | Fresh Dictionary/Transform3D arrays | One grouping task | | Internal batch plan | Transform count/offset, path kind and limits | Loader / `M2BuildBatchPlanner` | Loader materialization/cursor adapter | Fresh scalar Dictionary | One build operation | +| Internal M2 pending build | Tile key, M2 root, grouped transforms and cursors | Loader / `M2BuildQueue` | Loader readiness, planner and materializer adapters | Queue-owned job/keys and strong references | Until finish/cancel/clear/replacement | | Internal static M2 materialization | Parent, prepared Mesh, ordered transform slice and render settings | Loader / `M2StaticBatchMaterializer` | Attached MultiMeshInstance3D | Parent owns node/MultiMesh; exact Mesh reference retained | One main-thread build batch | | Internal WMO placement | Path, MODF placement, tile/index | Loader / `WmoPlacementResolver` | WMO caches, registry and three instance adapters | Value-only String/Transform3D | Lookup/placement lifetime | | Internal WMO ownership | Resolved placement key and tile/global reference key | Loader / `WmoPlacementRegistry` | Loader create/retain/final-free decisions | Registry-owned String sets; detached diagnostics | Map session or final release | @@ -243,6 +247,8 @@ flowchart TD M2Registry --> M2Grouper[M2PlacementGrouper] M2Transform[M2PlacementTransformResolver] --> M2Grouper M2Grouper --> M2Batch[M2BuildBatchPlanner] + M2Grouper --> M2Queue[M2BuildQueue] + M2Queue --> M2Batch M2Batch --> M2Static[M2StaticBatchMaterializer] M2Static --> M2 R --> WmoPlacement[WmoPlacementResolver] @@ -368,11 +374,14 @@ sequenceDiagram grouper owns worker-path final transforms; direct placeholder/instance transforms and every build/render side effect remain loader-owned. - `M2PlacementGrouper` is stateless and owns only call-local grouped transforms. - The loader retains tasks, mutex/result queues, stale checks and build state. + The loader retains tasks, mutex/result queues and stale-result checks; accepted + groups enter `M2BuildQueue` as typed pending jobs. - `M2BuildBatchPlanner` is stateless and owns only call-local scalar plans. + `M2BuildQueue` owns typed pending jobs, FIFO/stale keys, grouped-transform + references and group/offset/serial cursors without freeing engine objects. `M2StaticBatchMaterializer` owns static MultiMesh construction/attachment; - the loader retains queue/resource transitions, cursor adoption, animated/static - dispatch, budgets and Editor ownership. + the loader retains resource transitions, cursor-adoption decisions, + animated/static dispatch, root cleanup, budgets and Editor ownership. - `WmoPlacementResolver` is stateless and owns only call-local cache-key, identity and transform values. `WmoPlacementRegistry` owns only placement-key reference sets. `WmoRenderBuildStepPlanner` owns only a call-local operation @@ -554,6 +563,7 @@ Exact exported settings and cache versions remain documented in [`../../RENDER.m | M2 placement transform resolver | Implemented extraction | Scene-free formula/source/timing contract across three consumers | Asset-backed visual recheck and general placement parity pending | | M2 placement grouper | Implemented extraction | Scene-free validation/order/transform/source/timing contract | Worker/build state, spatial cells and asset-backed p95/p99 remain pending | | M2 build batch planner | Implemented extraction | Scene-free limit/count/cursor/source/timing contract | Queue/resource state, spatial cells and asset-backed p95/p99 remain pending | +| M2 build queue | Implemented extraction | Typed lifecycle/FIFO/progress/lifetime/source/timing contract | Resource dispatch and asset-backed traversal remain pending | | M2 animation load pipeline state | Implemented extraction | Synthetic lifecycle/FIFO/source/timing contract | Asset-backed traversal/leak/animation-fidelity/p95/p99 pending | | M2 animated scene finalizer | Implemented extraction | Synthetic type/lifetime/material/player/source/timing contract | Asset-backed GLB traversal/material/animation comparison pending | | M2 animation playback controller | Implemented extraction | Synthetic phase/selection/loop/native-copy/source/timing contract | Asset-backed animation timing/name/native comparison pending | @@ -624,6 +634,8 @@ Exact exported settings and cache versions remain documented in [`../../RENDER.m | `src/render/m2/m2_animation_playback_controller.gd` | Per-instance phase, selection, native copy/start and imported playback | | `src/render/m2/m2_animated_instance_materializer.gd` | Animated duplicate/render/playback startup and non-empty batch attachment | | `src/render/m2/m2_static_batch_materializer.gd` | Static MultiMesh construction, render setup and attachment | +| `src/render/m2/m2_build_job.gd` | M2 root/groups references and group/offset/serial progress | +| `src/render/m2/m2_build_queue.gd` | Keyed pending jobs and FIFO/stale tile-key lifecycle | | `src/render/m2/m2_mesh_load_pipeline_state.gd` | Static M2 pending Resource paths, terminal statuses and finalize FIFO | | `src/render/m2/m2_mesh_resource_cache_state.gd` | Prepared static M2 Mesh references and final-shutdown release | | `src/render/m2/m2_mesh_resource_extractor.gd` | First-Mesh selection and temporary PackedScene instance lifetime | @@ -650,6 +662,7 @@ Exact exported settings and cache versions remain documented in [`../../RENDER.m | `src/tools/verify_m2_animation_playback_controller.gd` | Animated M2 phase/selection/playback/native/boundary/timing regression | | `src/tools/verify_m2_animated_instance_materializer.gd` | Animated M2 order/render/playback/attachment/boundary/timing regression | | `src/tools/verify_m2_static_batch_materializer.gd` | Static M2 node/Mesh/render/attachment/boundary/timing regression | +| `src/tools/verify_m2_build_queue.gd` | M2 job/FIFO/progress/lifetime/boundary/timing regression | | `src/tools/verify_m2_mesh_load_pipeline_state.gd` | Static M2 request/terminal/FIFO/boundary/timing regression | | `src/tools/verify_m2_mesh_resource_cache_state.gd` | Static M2 Mesh cache ownership/lifetime/boundary/timing regression | | `src/tools/verify_m2_mesh_resource_extractor.gd` | M2 direct/PackedScene/subtree order/lifetime/boundary/timing regression | diff --git a/src/render/m2/m2_build_job.gd b/src/render/m2/m2_build_job.gd new file mode 100644 index 0000000..44cb96f --- /dev/null +++ b/src/render/m2/m2_build_job.gd @@ -0,0 +1,78 @@ +class_name M2BuildJob +extends RefCounted + +## Holds one M2 build cursor, grouped transforms and a strong root reference. +## Releasing this record never frees the referenced Node. + +var _tile_key: String +var _root: Node3D +var _groups: Dictionary +var _group_keys: Array +var _group_index: int = 0 +var _transform_offset: int = 0 +var _batch_serial: int = 0 + + +func _init(tile_key: String, root: Node3D, groups: Dictionary) -> void: + _tile_key = tile_key + _root = root + _groups = groups + _group_keys = groups.keys() + + +## Returns the immutable tile identity used by the queue. +func tile_key() -> String: + return _tile_key + + +## Returns the strongly referenced M2 scene root without transferring ownership. +func root() -> Node3D: + return _root + + +## Returns the exact grouped-transform Dictionary supplied at enqueue time. +func groups() -> Dictionary: + return _groups + + +## Returns the insertion-order group-key snapshot created at enqueue time. +func group_keys() -> Array: + return _group_keys + + +## Returns the current model-path group cursor. +func group_index() -> int: + return _group_index + + +## Returns the current transform offset within the selected model-path group. +func transform_offset() -> int: + return _transform_offset + + +## Returns the next static/animated batch serial used for node naming. +func batch_serial() -> int: + return _batch_serial + + +## Atomically adopts all progress values after one planned build operation. +func adopt_progress( + next_group_index: int, + next_transform_offset: int, + next_batch_serial: int +) -> void: + _group_index = next_group_index + _transform_offset = next_transform_offset + _batch_serial = next_batch_serial + + +## Returns detached scalar/count diagnostics without engine-object references. +func diagnostic_snapshot() -> Dictionary: + return { + "tile_key": _tile_key, + "group_index": _group_index, + "transform_offset": _transform_offset, + "batch_serial": _batch_serial, + "group_count": _group_keys.size(), + "has_root": _root != null, + } diff --git a/src/render/m2/m2_build_job.gd.uid b/src/render/m2/m2_build_job.gd.uid new file mode 100644 index 0000000..38b744e --- /dev/null +++ b/src/render/m2/m2_build_job.gd.uid @@ -0,0 +1 @@ +uid://drawyd5qmel7j diff --git a/src/render/m2/m2_build_queue.gd b/src/render/m2/m2_build_queue.gd new file mode 100644 index 0000000..81242b9 --- /dev/null +++ b/src/render/m2/m2_build_queue.gd @@ -0,0 +1,169 @@ +class_name M2BuildQueue +extends RefCounted + +## Owns pending M2 build-job records and FIFO tile keys. Stale FIFO keys are +## intentional: erasing a job never removes its queued key. + +const JOB_SCRIPT := preload("res://src/render/m2/m2_build_job.gd") + +var _jobs_by_tile_key: Dictionary = {} +var _queued_tile_keys: Array[String] = [] + + +## Enqueues a typed job. Duplicate tile keys replace the keyed job and append +## another FIFO entry, matching raw Dictionary assignment plus Array append. +func enqueue(tile_key: String, root: Node3D, groups: Dictionary) -> bool: + if tile_key.is_empty() or root == null or groups.is_empty(): + return false + _jobs_by_tile_key[tile_key] = JOB_SCRIPT.new(tile_key, root, groups) + _queued_tile_keys.append(tile_key) + return true + + +## Returns whether at least one FIFO key, including a stale key, is pending. +func has_pending() -> bool: + return not _queued_tile_keys.is_empty() + + +## Returns the front FIFO tile key, or an empty String when the queue is empty. +func front_key() -> String: + if _queued_tile_keys.is_empty(): + return "" + return _queued_tile_keys.front() + + +## Pops and returns the front FIFO key, or an empty String when already empty. +func pop_front() -> String: + if _queued_tile_keys.is_empty(): + return "" + return _queued_tile_keys.pop_front() + + +## Moves the front FIFO key to the tail and reports whether a key existed. +func rotate_front() -> bool: + if _queued_tile_keys.is_empty(): + return false + _queued_tile_keys.append(_queued_tile_keys.pop_front()) + return true + + +## Returns whether a tile key still has a current job record. +func has_job(tile_key: String) -> bool: + return _jobs_by_tile_key.has(tile_key) + + +## Returns the current job as its engine base type, or null for a stale key. +func job_for(tile_key: String) -> RefCounted: + return _jobs_by_tile_key.get(tile_key, null) as RefCounted + + +## Returns the borrowed root for a current job, or null for a stale key. +func root_for(tile_key: String) -> Node3D: + var job := job_for(tile_key) + if job == null: + return null + return job.call("root") as Node3D + + +## Returns the exact grouped-transform Dictionary, or an empty Dictionary. +func groups_for(tile_key: String) -> Dictionary: + var job := job_for(tile_key) + if job == null: + return {} + return job.call("groups") as Dictionary + + +## Returns the borrowed group-key snapshot, or an empty Array for a stale key. +func group_keys_for(tile_key: String) -> Array: + var job := job_for(tile_key) + if job == null: + return [] + return job.call("group_keys") as Array + + +## Returns the current model-path group cursor, or zero for a stale key. +func group_index_for(tile_key: String) -> int: + var job := job_for(tile_key) + if job == null: + return 0 + return int(job.call("group_index")) + + +## Returns the current in-group transform offset, or zero for a stale key. +func transform_offset_for(tile_key: String) -> int: + var job := job_for(tile_key) + if job == null: + return 0 + return int(job.call("transform_offset")) + + +## Returns the current batch serial, or zero for a stale key. +func batch_serial_for(tile_key: String) -> int: + var job := job_for(tile_key) + if job == null: + return 0 + return int(job.call("batch_serial")) + + +## Atomically adopts all progress values for a current job. +func adopt_progress( + tile_key: String, + next_group_index: int, + next_transform_offset: int, + next_batch_serial: int +) -> bool: + var job := job_for(tile_key) + if job == null: + return false + job.call( + "adopt_progress", + next_group_index, + next_transform_offset, + next_batch_serial + ) + return true + + +## Erases only the keyed job and intentionally leaves every FIFO entry stale. +## Returns whether an active record existed. Engine objects are never freed. +func erase_job(tile_key: String) -> bool: + var had_job := _jobs_by_tile_key.has(tile_key) + _jobs_by_tile_key.erase(tile_key) + return had_job + + +## Returns a detached snapshot of active job keys for loader-owned cleanup. +func job_keys() -> Array: + return _jobs_by_tile_key.keys() + + +## Releases all job/key references without freeing referenced engine objects. +func clear() -> void: + _jobs_by_tile_key.clear() + _queued_tile_keys.clear() + + +## Returns FIFO entry count, including duplicate and stale keys. +func pending_count() -> int: + return _queued_tile_keys.size() + + +## Returns active keyed job count, excluding stale FIFO keys. +func active_job_count() -> int: + return _jobs_by_tile_key.size() + + +## Returns detached FIFO order and tile-key-sorted scalar job diagnostics. +func diagnostic_snapshot() -> Dictionary: + var tile_keys: Array = _jobs_by_tile_key.keys() + tile_keys.sort() + var jobs: Array[Dictionary] = [] + for tile_key_variant in tile_keys: + var tile_key := String(tile_key_variant) + var job := job_for(tile_key) + if job != null: + jobs.append(job.call("diagnostic_snapshot") as Dictionary) + return { + "queued_tile_keys": _queued_tile_keys.duplicate(), + "jobs": jobs, + } diff --git a/src/render/m2/m2_build_queue.gd.uid b/src/render/m2/m2_build_queue.gd.uid new file mode 100644 index 0000000..90e5b3a --- /dev/null +++ b/src/render/m2/m2_build_queue.gd.uid @@ -0,0 +1 @@ +uid://57qrpiqqgpr4 diff --git a/src/scenes/streaming/streaming_world_loader.gd b/src/scenes/streaming/streaming_world_loader.gd index 57ddede..cf82e2f 100644 --- a/src/scenes/streaming/streaming_world_loader.gd +++ b/src/scenes/streaming/streaming_world_loader.gd @@ -60,6 +60,7 @@ const M2_PLACEMENT_GROUPER_SCRIPT := preload( const M2_BUILD_BATCH_PLANNER_SCRIPT := preload( "res://src/render/m2/m2_build_batch_planner.gd" ) +const M2_BUILD_QUEUE_SCRIPT := preload("res://src/render/m2/m2_build_queue.gd") const M2_STATIC_BATCH_MATERIALIZER_SCRIPT := preload( "res://src/render/m2/m2_static_batch_materializer.gd" ) @@ -268,8 +269,7 @@ var _detail_asset_queued: Dictionary = {} var _m2_group_tasks: Dictionary = {} var _m2_group_result_mutex := Mutex.new() var _m2_group_result_queue: Array = [] -var _m2_build_jobs: Dictionary = {} -var _m2_build_queue: Array = [] +var _m2_build_queue_state := M2_BUILD_QUEUE_SCRIPT.new() var _m2_unique_placement_registry := ( M2_UNIQUE_PLACEMENT_REGISTRY_SCRIPT.new() ) @@ -1033,7 +1033,7 @@ func _log_hitch_profile(start_usec: int, timings: Array[String], did_refresh: bo _m2_group_tasks.size(), _m2_animation_load_pipeline_state.total_work_count(), _m2_mesh_load_pipeline_state.total_work_count(), - _m2_build_queue.size(), + _m2_build_queue_state.pending_count(), ( _wmo_build_queue.size() + _wmo_scene_resource_cache_state.pending_request_count() @@ -1064,7 +1064,7 @@ func render_baseline_snapshot() -> Dictionary: "m2_task": _m2_group_tasks.size(), "m2_animation": _m2_animation_load_pipeline_state.total_work_count(), "m2_mesh": _m2_mesh_load_pipeline_state.total_work_count(), - "m2_build": _m2_build_queue.size(), + "m2_build": _m2_build_queue_state.pending_count(), "wmo_build": ( _wmo_build_queue.size() + _wmo_scene_resource_cache_state.pending_request_count() @@ -1350,7 +1350,7 @@ func _tick_runtime_stats(delta: float) -> void: _m2_group_tasks.size(), _m2_animation_load_pipeline_state.total_work_count(), _m2_mesh_load_pipeline_state.total_work_count(), - _m2_build_queue.size(), + _m2_build_queue_state.pending_count(), ( _wmo_build_queue.size() + _wmo_scene_resource_cache_state.pending_request_count() @@ -3011,9 +3011,9 @@ func _clear_streamed_world() -> void: _adt_water_load_pipeline_state.clear() _detail_asset_queue.clear() _detail_asset_queued.clear() - for key in _m2_build_jobs.keys(): + for key in _m2_build_queue_state.job_keys(): _cancel_m2_build_job(String(key)) - _m2_build_queue.clear() + _m2_build_queue_state.clear() _m2_unique_placement_registry.clear() _m2_mesh_load_pipeline_state.clear() _m2_runtime_mesh_finalizer.clear() @@ -3440,7 +3440,7 @@ func _sync_detail_assets(state: Dictionary, focus_pos: Vector3) -> Dictionary: not bool(state.get("m2_built", false)) and not m2_placements.is_empty() and not _m2_group_tasks.has(key) - and not _m2_build_jobs.has(key) + and not _m2_build_queue_state.has_job(key) ): _enqueue_detail_asset_sync(key) else: @@ -4090,7 +4090,7 @@ func _request_tile_m2_assets( if existing_m2_root != null: result["built"] = not existing_m2_root.is_queued_for_deletion() return result - if _m2_group_tasks.has(key) or _m2_build_jobs.has(key): + if _m2_group_tasks.has(key) or _m2_build_queue_state.has_job(key): return result var filtered: Dictionary = _m2_unique_placement_registry.reserve( @@ -4193,7 +4193,7 @@ func _drain_m2_group_results() -> void: continue if not _tile_within_detail_radius(state, _last_focus_pos, m2_tile_radius): continue - if bool(state.get("m2_built", false)) or _m2_build_jobs.has(key): + if bool(state.get("m2_built", false)) or _m2_build_queue_state.has_job(key): continue var groups: Dictionary = result.get("groups", {}) @@ -4210,15 +4210,7 @@ func _drain_m2_group_results() -> void: (tile_root as Node3D).add_child(m2_root) _set_editor_owner_recursive(m2_root) - _m2_build_jobs[key] = { - "root": m2_root, - "groups": groups, - "keys": groups.keys(), - "index": 0, - "offset": 0, - "serial": 0, - } - _m2_build_queue.append(key) + _m2_build_queue_state.enqueue(key, m2_root, groups) func _drain_m2_animation_loads() -> void: @@ -4308,14 +4300,14 @@ func _drain_m2_mesh_loads() -> void: func _process_m2_build_jobs() -> void: while _render_budget_scheduler.has_remaining_permit( - RENDER_BUDGET_SCHEDULER_SCRIPT.M2_BUILD) and not _m2_build_queue.is_empty(): - var key: String = String(_m2_build_queue.front()) - if not _m2_build_jobs.has(key): - _m2_build_queue.pop_front() + RENDER_BUDGET_SCHEDULER_SCRIPT.M2_BUILD) and _m2_build_queue_state.has_pending(): + var key := _m2_build_queue_state.front_key() + if not _m2_build_queue_state.has_job(key): + _m2_build_queue_state.pop_front() continue if not _tile_states.has(key): _cancel_m2_build_job(key) - _m2_build_queue.pop_front() + _m2_build_queue_state.pop_front() continue var state: Dictionary = _tile_states[key] @@ -4324,36 +4316,34 @@ func _process_m2_build_jobs() -> void: _release_tile_m2_unique_keys(state) state["m2_built"] = false _tile_states[key] = state - _m2_build_queue.pop_front() + _m2_build_queue_state.pop_front() continue - var job: Dictionary = _m2_build_jobs[key] - var root: Node = job.get("root", null) + var root := _m2_build_queue_state.root_for(key) if root == null or not is_instance_valid(root): - _m2_build_jobs.erase(key) - _m2_build_queue.pop_front() + _m2_build_queue_state.erase_job(key) + _m2_build_queue_state.pop_front() continue - var group_keys: Array = job.get("keys", []) - var index: int = int(job.get("index", 0)) + var group_keys := _m2_build_queue_state.group_keys_for(key) + var index := _m2_build_queue_state.group_index_for(key) if index >= group_keys.size(): _finish_m2_build_job(key) - _m2_build_queue.pop_front() + _m2_build_queue_state.pop_front() continue var rel_path: String = String(group_keys[index]) var normalized_rel := _normalize_m2_rel_path(rel_path) - var groups: Dictionary = job.get("groups", {}) + var groups := _m2_build_queue_state.groups_for(key) var transforms: Array = groups.get(rel_path, []) - var offset: int = int(job.get("offset", 0)) + var offset := _m2_build_queue_state.transform_offset_for(key) var animated_prototype: Node3D = null if enable_m2_animated_instances: animated_prototype = _get_or_load_m2_native_animated_prototype(rel_path) if animated_prototype == null: animated_prototype = _get_or_load_m2_animated_prototype(rel_path) if enable_m2_animated_instances and _m2_animation_load_pipeline_state.has_request(normalized_rel): - _m2_build_queue.pop_front() - _m2_build_queue.append(key) + _m2_build_queue_state.rotate_front() _render_budget_scheduler.try_consume_permit(RENDER_BUDGET_SCHEDULER_SCRIPT.M2_BUILD) continue var batch_plan: Dictionary = _m2_build_batch_planner.plan_batch( @@ -4364,7 +4354,8 @@ func _process_m2_build_jobs() -> void: m2_multimesh_batch_size ) var batch_count: int = int(batch_plan["batch_count"]) - var serial: int = int(job.get("serial", 0)) + var serial := _m2_build_queue_state.batch_serial_for(key) + var next_serial := serial if batch_count > 0: if animated_prototype != null: _materialize_m2_animated_batch(root as Node3D, rel_path, animated_prototype, transforms, offset, batch_count, serial) @@ -4372,24 +4363,30 @@ func _process_m2_build_jobs() -> void: var mesh := _get_m2_mesh_or_request(rel_path) if mesh == null: if not _m2_prototype_cache_state.is_model_missing(normalized_rel): - _m2_build_queue.pop_front() - _m2_build_queue.append(key) + _m2_build_queue_state.rotate_front() _render_budget_scheduler.try_consume_permit(RENDER_BUDGET_SCHEDULER_SCRIPT.M2_BUILD) continue else: _materialize_m2_group_batch(root as Node3D, rel_path, mesh, transforms, offset, batch_count, serial) - job["serial"] = serial + 1 + next_serial = serial + 1 + var next_group_index := index + var next_transform_offset := offset if bool(batch_plan["group_complete"]): - job["index"] = index + 1 - job["offset"] = 0 + next_group_index = index + 1 + next_transform_offset = 0 else: - job["offset"] = int(batch_plan["next_offset"]) - _m2_build_jobs[key] = job + next_transform_offset = int(batch_plan["next_offset"]) + _m2_build_queue_state.adopt_progress( + key, + next_group_index, + next_transform_offset, + next_serial + ) _render_budget_scheduler.try_consume_permit(RENDER_BUDGET_SCHEDULER_SCRIPT.M2_BUILD) - if int(job["index"]) >= group_keys.size(): + if next_group_index >= group_keys.size(): _finish_m2_build_job(key) - _m2_build_queue.pop_front() + _m2_build_queue_state.pop_front() func _materialize_m2_group_batch( @@ -4458,10 +4455,9 @@ func _materialize_m2_animated_batch( func _finish_m2_build_job(key: String) -> void: - if not _m2_build_jobs.has(key): + if not _m2_build_queue_state.has_job(key): return - var job: Dictionary = _m2_build_jobs[key] - var root: Node = job.get("root", null) + var root := _m2_build_queue_state.root_for(key) var built := false if root != null and is_instance_valid(root): built = root.get_child_count() > 0 @@ -4471,17 +4467,16 @@ func _finish_m2_build_job(key: String) -> void: var state: Dictionary = _tile_states[key] state["m2_built"] = true _tile_states[key] = state - _m2_build_jobs.erase(key) + _m2_build_queue_state.erase_job(key) func _cancel_m2_build_job(key: String) -> void: - if not _m2_build_jobs.has(key): + if not _m2_build_queue_state.has_job(key): return - var job: Dictionary = _m2_build_jobs[key] - var root: Node = job.get("root", null) + var root := _m2_build_queue_state.root_for(key) if root != null and is_instance_valid(root): _queue_free_streamed_node(root) - _m2_build_jobs.erase(key) + _m2_build_queue_state.erase_job(key) func _remove_tile_m2_assets(state: Dictionary) -> void: diff --git a/src/tools/verify_m2_build_queue.gd b/src/tools/verify_m2_build_queue.gd new file mode 100644 index 0000000..b2e7fc5 --- /dev/null +++ b/src/tools/verify_m2_build_queue.gd @@ -0,0 +1,279 @@ +extends SceneTree + +## Asset-free M2 build-job lifecycle, FIFO, progress, engine-lifetime, source +## boundary and bounded-timing regression. + +const QUEUE_SCRIPT := preload("res://src/render/m2/m2_build_queue.gd") +const QUEUE_PATH := "res://src/render/m2/m2_build_queue.gd" +const JOB_PATH := "res://src/render/m2/m2_build_job.gd" +const LOADER_PATH := "res://src/scenes/streaming/streaming_world_loader.gd" + + +func _initialize() -> void: + var failures: Array[String] = [] + _verify_invalid_and_fifo_jobs(failures) + _verify_progress_adoption(failures) + _verify_rotation_duplicate_and_stale_keys(failures) + _verify_erase_clear_and_engine_lifetime(failures) + _verify_detached_sorted_diagnostics(failures) + _verify_loader_and_engine_lifetime_boundaries(failures) + var elapsed_milliseconds := _verify_bounded_timing(failures) + if not failures.is_empty(): + for failure in failures: + push_error("M2_BUILD_QUEUE: %s" % failure) + quit(1) + return + print( + "M2_BUILD_QUEUE PASS cases=13 iterations=100 elapsed_ms=%.3f" + % elapsed_milliseconds + ) + quit(0) + + +func _verify_invalid_and_fifo_jobs(failures: Array[String]) -> void: + var queue: RefCounted = QUEUE_SCRIPT.new() + var root := Node3D.new() + var groups := _make_groups("world/a.m2", 2) + _expect_false(queue.call("enqueue", "", root, groups), "empty key rejected", failures) + _expect_false(queue.call("enqueue", "1_1", null, groups), "null root rejected", failures) + _expect_false(queue.call("enqueue", "1_1", root, {}), "empty groups rejected", failures) + _expect_true(queue.call("enqueue", "2_2", root, groups), "first enqueue", failures) + _expect_true( + queue.call("enqueue", "3_3", root, _make_groups("world/b.m2", 1)), + "second enqueue", + failures + ) + _expect_string_equal(String(queue.call("front_key")), "2_2", "FIFO front", failures) + _expect_equal(int(queue.call("pending_count")), 2, "FIFO pending count", failures) + _expect_equal(int(queue.call("active_job_count")), 2, "active job count", failures) + var first_job: RefCounted = queue.call("job_for", "2_2") + _expect_string_equal(String(first_job.call("tile_key")), "2_2", "typed job key", failures) + _expect_same(first_job.call("root"), root, "job retains root", failures) + _expect_same(first_job.call("groups"), groups, "job retains exact groups", failures) + _expect_string_array(first_job.call("group_keys"), ["world/a.m2"], "group-key snapshot", failures) + queue.call("clear") + root.free() + + +func _verify_progress_adoption(failures: Array[String]) -> void: + var queue: RefCounted = QUEUE_SCRIPT.new() + var root := Node3D.new() + queue.call("enqueue", "4_4", root, _make_groups("world/a.m2", 2)) + _expect_equal(int(queue.call("group_index_for", "4_4")), 0, "initial group index", failures) + _expect_equal(int(queue.call("transform_offset_for", "4_4")), 0, "initial offset", failures) + _expect_equal(int(queue.call("batch_serial_for", "4_4")), 0, "initial serial", failures) + _expect_true(queue.call("adopt_progress", "4_4", 7, 3, 11), "progress adopted", failures) + _expect_equal(int(queue.call("group_index_for", "4_4")), 7, "adopted group index", failures) + _expect_equal(int(queue.call("transform_offset_for", "4_4")), 3, "adopted offset", failures) + _expect_equal(int(queue.call("batch_serial_for", "4_4")), 11, "adopted serial", failures) + _expect_false( + queue.call("adopt_progress", "missing", 1, 1, 1), + "stale progress rejected", + failures + ) + queue.call("clear") + root.free() + + +func _verify_rotation_duplicate_and_stale_keys(failures: Array[String]) -> void: + var queue: RefCounted = QUEUE_SCRIPT.new() + var first_root := Node3D.new() + var second_root := Node3D.new() + var groups := _make_groups("world/a.m2", 1) + queue.call("enqueue", "5_5", first_root, groups) + queue.call("enqueue", "6_6", first_root, groups) + _expect_true(queue.call("rotate_front"), "front rotates", failures) + _expect_string_equal(String(queue.call("front_key")), "6_6", "rotated FIFO front", failures) + queue.call("enqueue", "6_6", second_root, groups) + _expect_equal(int(queue.call("pending_count")), 3, "duplicate appends FIFO key", failures) + _expect_equal(int(queue.call("active_job_count")), 2, "duplicate replaces keyed job", failures) + _expect_same(queue.call("root_for", "6_6"), second_root, "duplicate latest root wins", failures) + _expect_true(queue.call("erase_job", "6_6"), "job erased", failures) + _expect_equal(int(queue.call("pending_count")), 3, "erase preserves FIFO keys", failures) + _expect_false(queue.call("has_job", "6_6"), "front key becomes stale", failures) + _expect_string_equal(String(queue.call("pop_front")), "6_6", "stale front popped", failures) + queue.call("clear") + first_root.free() + second_root.free() + + +func _verify_erase_clear_and_engine_lifetime(failures: Array[String]) -> void: + var queue: RefCounted = QUEUE_SCRIPT.new() + var root := Node3D.new() + queue.call("enqueue", "7_7", root, _make_groups("world/a.m2", 1)) + _expect_false(queue.call("erase_job", "missing"), "unknown erase rejected", failures) + _expect_true(queue.call("erase_job", "7_7"), "known job erased", failures) + _expect_true(is_instance_valid(root), "erase keeps root valid", failures) + _expect_equal(int(queue.call("pending_count")), 1, "erase leaves stale key", failures) + queue.call("clear") + queue.call("clear") + _expect_equal(int(queue.call("pending_count")), 0, "clear idempotent", failures) + _expect_true(is_instance_valid(root), "clear keeps root valid", failures) + root.free() + + +func _verify_detached_sorted_diagnostics(failures: Array[String]) -> void: + var queue: RefCounted = QUEUE_SCRIPT.new() + var root := Node3D.new() + queue.call("enqueue", "20_20", root, _make_groups("world/a.m2", 1)) + queue.call("enqueue", "10_10", root, _make_groups("world/b.m2", 2)) + queue.call("adopt_progress", "10_10", 1, 2, 3) + var snapshot: Dictionary = queue.call("diagnostic_snapshot") + _expect_string_array( + snapshot["queued_tile_keys"], + ["20_20", "10_10"], + "diagnostic FIFO order", + failures + ) + var jobs: Array = snapshot["jobs"] + _expect_string_equal(String(jobs[0]["tile_key"]), "10_10", "diagnostic job sort", failures) + _expect_equal(int(jobs[0]["batch_serial"]), 3, "diagnostic progress", failures) + (snapshot["queued_tile_keys"] as Array).append("mutated") + jobs[0]["batch_serial"] = 99 + var fresh_snapshot: Dictionary = queue.call("diagnostic_snapshot") + _expect_string_array( + fresh_snapshot["queued_tile_keys"], + ["20_20", "10_10"], + "diagnostic FIFO detached", + failures + ) + _expect_equal( + int((fresh_snapshot["jobs"] as Array)[0]["batch_serial"]), + 3, + "job diagnostic detached", + failures + ) + queue.call("clear") + root.free() + + +func _verify_loader_and_engine_lifetime_boundaries(failures: Array[String]) -> void: + var loader_source := FileAccess.get_file_as_string(LOADER_PATH) + var queue_source := FileAccess.get_file_as_string(QUEUE_PATH) + var job_source := FileAccess.get_file_as_string(JOB_PATH) + _expect_true( + loader_source.contains("M2_BUILD_QUEUE_SCRIPT.new()"), + "loader composes queue", + failures + ) + for delegated_call in [ + "_m2_build_queue_state.enqueue(", + "_m2_build_queue_state.has_job(", + "_m2_build_queue_state.root_for(", + "_m2_build_queue_state.rotate_front()", + "_m2_build_queue_state.adopt_progress(", + "_m2_build_queue_state.erase_job(", + "_m2_build_queue_state.clear()", + ]: + _expect_true( + loader_source.contains(delegated_call), + "loader delegates %s" % delegated_call, + failures + ) + _expect_false( + loader_source.contains("var _m2_build_jobs"), + "legacy job Dictionary removed", + failures + ) + _expect_false( + loader_source.contains("var _m2_build_queue: Array"), + "legacy FIFO Array removed", + failures + ) + _expect_true( + loader_source.contains("_queue_free_streamed_node(root)") + and loader_source.contains("_m2_static_batch_materializer.materialize_batch(") + and loader_source.contains("_m2_animated_instance_materializer.materialize_batch(") + and loader_source.contains("try_consume_permit("), + "loader retains destruction, materialization and permits", + failures + ) + for forbidden_token in [ + "queue_free", + "RenderingServer", + "RID", + "MeshInstance3D", + "MultiMesh", + "ResourceLoader", + "FileAccess", + "WorkerThreadPool", + "Thread", + "Mutex", + ]: + _expect_false( + queue_source.contains(forbidden_token) or job_source.contains(forbidden_token), + "queue/job omit %s ownership" % forbidden_token, + failures + ) + + +func _verify_bounded_timing(failures: Array[String]) -> float: + var queue: RefCounted = QUEUE_SCRIPT.new() + var root := Node3D.new() + var groups := _make_groups("world/timing.m2", 4) + var started_microseconds := Time.get_ticks_usec() + for _iteration in range(100): + for job_index in range(256): + queue.call("enqueue", "%d_%d" % [job_index, job_index], root, groups) + for _job_index in range(256): + queue.call("rotate_front") + for job_index in range(256): + queue.call("erase_job", "%d_%d" % [job_index, job_index]) + queue.call("clear") + var elapsed_milliseconds := float(Time.get_ticks_usec() - started_microseconds) / 1000.0 + _expect_true(elapsed_milliseconds < 1000.0, "100 queue cycles remain bounded", failures) + root.free() + return elapsed_milliseconds + + +func _make_groups(relative_path: String, transform_count: int) -> Dictionary: + var transforms: Array = [] + for transform_index in transform_count: + transforms.append( + Transform3D(Basis.IDENTITY, Vector3(float(transform_index), 0.0, 0.0)) + ) + return {relative_path: transforms} + + +func _expect_equal(actual: int, expected: int, label: String, failures: Array[String]) -> void: + if actual != expected: + failures.append("%s expected=%d actual=%d" % [label, expected, actual]) + + +func _expect_string_equal( + actual: String, + expected: String, + label: String, + failures: Array[String] +) -> void: + if actual != expected: + failures.append("%s expected=%s actual=%s" % [label, expected, actual]) + + +func _expect_string_array( + actual: Array, + expected: Array, + label: String, + failures: Array[String] +) -> void: + if actual != expected: + failures.append("%s expected=%s actual=%s" % [label, expected, actual]) + + +func _expect_same( + actual: Variant, + expected: Variant, + label: String, + failures: Array[String] +) -> void: + if not is_same(actual, expected): + failures.append(label) + + +func _expect_true(condition: bool, label: String, failures: Array[String]) -> void: + if not condition: + failures.append(label) + + +func _expect_false(condition: bool, label: String, failures: Array[String]) -> void: + _expect_true(not condition, label, failures) diff --git a/src/tools/verify_m2_build_queue.gd.uid b/src/tools/verify_m2_build_queue.gd.uid new file mode 100644 index 0000000..5feffd5 --- /dev/null +++ b/src/tools/verify_m2_build_queue.gd.uid @@ -0,0 +1 @@ +uid://v7s73cxuwli6 diff --git a/targets/03-renderer-facade.md b/targets/03-renderer-facade.md index e8694fa..6af5d4d 100644 --- a/targets/03-renderer-facade.md +++ b/targets/03-renderer-facade.md @@ -55,7 +55,8 @@ Runtime и Editor используют facade; planner/scheduler тестиру `M03-RND-M2-ANIMATED-SCENE-FINALIZER-001`, `M03-RND-M2-ANIMATION-PLAYBACK-001`, `M03-RND-M2-ANIMATED-INSTANCE-MATERIALIZER-001`, - `M03-RND-M2-STATIC-BATCH-MATERIALIZER-001` + `M03-RND-M2-STATIC-BATCH-MATERIALIZER-001`, + `M03-RND-M2-BUILD-QUEUE-001` - Commands: dedicated scheduler/planner/facade/focus/coordinate/manifest/shutdown headless verifiers; Godot cold editor parse; coordination, documentation and diff gates. - Results: scheduler `cases=6 iterations=20000 elapsed_ms=10.216`; planner @@ -240,6 +241,13 @@ Runtime и Editor используют facade; planner/scheduler тестиру Post-merge static materializer (`1.327ms`), build planner, Mesh cache, animated materializer, shutdown, facade, internal-access `30`, checkpoint dry-run `7/7`, documentation and coordination gates remained green. + M2 build queue passed `cases=13 iterations=100 elapsed_ms=66.247` with invalid + enqueue guards, exact root/group references, insertion-order key snapshots, + FIFO rotation, duplicate replacement, stale-key cleanup, atomic cursor adoption, + detached diagnostics and engine-lifetime/source boundaries. All 52 autonomous + headless regressions and checkpoint dry-run `7/7` passed; the proprietary ADT + probe remained unavailable. Internal-access inventory remains `30`; resource + readiness, permits, materializer dispatch and root destruction remain loader-owned. M02 terrain-query regression remained green (13 pre-existing expired M00 claim warnings). - Fidelity comparison: all 16 historical operation limits and drain sites were @@ -268,6 +276,9 @@ Runtime и Editor используют facade; planner/scheduler тестиру origin subtraction and the `0.0001` scale clamp. M2 build batching retains animated/static limit selection, minimum-one clamp, remaining-transform cap and the exact offset/group-completion cursor rules. + Pending M2 builds retain keyed latest-job replacement, duplicate/stale FIFO + entries, one-key rotation, strong root/group references and exact group/offset/ + serial progression; releasing queue records still never frees scene nodes. WMO cache-key normalization, positive MODF identity with tile/index fallback and world position/Euler/unclamped-scale transforms are unchanged across lightweight render, cached-scene and live-prototype paths. @@ -382,6 +393,7 @@ Runtime и Editor используют facade; planner/scheduler тестиру M2 animation playback controller and loader native/imported playback adapters, M2 animated instance materializer and loader log/Editor-owner adapter, M2 static batch materializer and loader readiness/cursor/Editor-owner adapter, + typed M2 build job/queue and loader enqueue/drain/progress/cancel/clear adapters, expanded facade/internal-access/coordinate verifiers; work-package claims and this evidence. - Remaining risks: worker concurrency and stale-result validation remain streamer-owned; cancellation stops new permits but does not interrupt @@ -404,7 +416,8 @@ Runtime и Editor используют facade; planner/scheduler тестиру main-thread work; asset-backed traversal/visual/leak/p95/p99 evidence is pending; static MultiMesh materialization is also separated but remains synchronous; headless timing does not measure transform upload GPU cost and spatial-cell - batching still needs culling/performance evidence; + batching still needs culling/performance evidence; typed M2 pending state is + separated, while resource readiness/dispatch and root cleanup remain in loader; WMO ResourceLoader/FileAccess I/O, live fallback and group materialization remain in the loader; asset-backed WMO placement/portal/material/leak/p95/p99 evidence remains pending;