open-wc/docs/modules/m2-build-queue.md

M2 Build Queue

Metadata

Field	Value
Status	Implemented
Target/work package	M03 / M03-RND-M2-BUILD-QUEUE-001
Owners	Pending M2 build-job records, FIFO tile keys and build cursors
Last verified	Worktree work/sindo-main-codex/m03-m2-build-queue, 2026-07-18
Profiles/capabilities	Existing static MultiMesh and animated-instance build paths

Purpose

Own typed pending M2 build state outside StreamingWorldLoader: a keyed job retains its M2 root, grouped transforms, insertion-order group keys and three progress cursors, while a separate FIFO retains scheduling order and stale keys.

Non-goals

• Decide tile eligibility, detail radius, resource readiness or retries.
• Consume scheduler permits or plan batch sizes.
• Create, attach or free Nodes, Meshes, MultiMeshes or animated instances.
• Load/cache resources, group placements or mutate tile state.
• Change model order, transform order, render settings or visible output.

Context and boundaries

flowchart LR
    Groups[M2PlacementGrouper result] --> Loader[StreamingWorldLoader adapter]
    Loader --> Queue[M2BuildQueue]
    Queue --> Job[M2BuildJob]
    Queue --> Loader
    Loader --> Planner[M2BuildBatchPlanner]
    Loader --> Dispatch[M2BuildDispatchPlanner]
    Loader --> Static[M2StaticBatchMaterializer]
    Loader --> Animated[M2AnimatedInstanceMaterializer]
    Loader --> Scheduler[RenderBudgetScheduler]

Allowed dependencies are String, Dictionary, Array, RefCounted and a strong Node3D reference. SceneTree destruction, materializers, Mesh/MultiMesh, RenderingServer/RIDs, ResourceLoader/files, workers/mutexes, scheduler policy, gameplay, network and Editor APIs are forbidden.

Public API

M2BuildJob

Symbol	Kind	Purpose	Thread/lifetime	Errors
tile_key()	Query	Immutable keyed identity	Main thread; job lifetime	None
root()	Query	Borrow strongly retained M2 root	Main thread; no ownership transfer	May be externally invalidated
groups()	Query	Borrow exact grouped-transform Dictionary	Main thread; job lifetime	None
group_keys()	Query	Borrow enqueue-time insertion-order key snapshot	Main thread; job lifetime	None
group_index()	Query	Current model-path cursor	Main thread	None
transform_offset()	Query	Current transform cursor within group	Main thread	None
batch_serial()	Query	Current batch naming serial	Main thread	None
adopt_progress(group_index, offset, serial)	Command	Atomically replace all progress values	Main thread	Raw integer semantics retained
diagnostic_snapshot()	Query	Detached scalar/count state	Any serialized caller	Omits engine references

M2BuildQueue

Symbol	Kind	Purpose	Thread/lifetime	Errors
enqueue(tile_key, root, groups)	Command	Replace keyed job and append FIFO key	Main thread; until erase/clear	Invalid/empty input returns false
has_pending/front_key/pop_front/rotate_front	Commands/queries	Inspect, drain or rotate FIFO independently of job validity	Main thread	Empty returns false/empty String
has_job/job_for	Queries	Inspect keyed record, including stale-key distinction	Main thread	Missing returns false/null
root_for/groups_for/group_keys_for	Queries	Borrow current job inputs	Main thread	Stale returns null/empty collection
group_index_for/transform_offset_for/batch_serial_for	Queries	Read current progress	Main thread	Stale returns zero
adopt_progress(tile_key, group_index, offset, serial)	Command	Atomically update current job progress	Main thread	Stale returns false
erase_job(tile_key)	Command	Remove keyed job but preserve FIFO keys	Main thread	Unknown returns false
job_keys()	Query	Detached active-key snapshot for loader cleanup	Main thread	Empty returns empty Array
clear()	Command	Release all job and FIFO references	Main thread	Idempotent; never frees Nodes
pending_count/active_job_count	Queries	FIFO and keyed-job metrics	Main thread	None
diagnostic_snapshot()	Query	Detached FIFO and sorted job diagnostics	Main thread	None

Inputs and outputs

Direction	Contract/data	Producer	Consumer	Ownership	Thread/lifetime
Input	Tile key and M2 root	Loader group-result finalizer	Queue/job	Copied String; strong borrowed Node reference	Until erase/clear
Input	Grouped transforms Dictionary	M2PlacementGrouper through loader mailbox	Job	Exact strong reference	Job lifetime
Input	groups.keys() order	Job constructor	Loader group selection	Job-owned Array snapshot	Job lifetime
Input	Next group/offset/serial	Planner/materializer adapter	Job progress	Copied integers	Until next adoption
Output	Front/rotated/popped tile key	Queue	Loader scheduling loop	Copied String	One drain step
Output	Borrowed root/groups/group keys	Job/queue	Loader readiness/materialization	No ownership transfer	One drain step
Output	Detached diagnostics/key snapshots	Queue/job	Tests/metrics/cleanup adapter	Caller-owned collections	Call result

Side effects are keyed/FIFO collection mutation, strong-reference retention and cursor mutation. No engine object is created, destroyed or attached.

Data flow

flowchart TD
    Enqueue[Tile key, root and grouped transforms] --> Job[Create or replace M2BuildJob]
    Enqueue --> FIFO[Append tile key to FIFO]
    FIFO --> Front[Borrow front key]
    Front --> Current{Current keyed job exists?}
    Current -->|no| Pop[Pop stale key]
    Current -->|yes| Loader[Loader eligibility/readiness/materialization]
    Loader -->|resource pending| Rotate[Move front key to tail]
    Loader -->|operation complete| Progress[Adopt group, offset and serial]
    Loader -->|finish/cancel| Erase[Erase job only]
    Erase --> Pop

Lifecycle/state

stateDiagram-v2
    [*] --> Absent
    Absent --> Queued: enqueue valid job and FIFO key
    Queued --> Rotated: resource pending
    Rotated --> Queued: returns to FIFO front
    Queued --> Progressed: adopt progress
    Progressed --> Queued: group remains
    Queued --> StaleQueued: erase job before FIFO pop
    Progressed --> StaleQueued: finish or cancel
    StaleQueued --> Absent: pop stale key
    Queued --> Absent: clear
    StaleQueued --> Absent: clear

Duplicate enqueue replaces the current keyed job and adds another FIFO entry; later erase can therefore leave multiple stale entries, matching historical raw state.

Main sequence

sequenceDiagram
    participant G as M2 group-result drain
    participant Q as M2BuildQueue
    participant L as StreamingWorldLoader
    participant P as M2BuildBatchPlanner
    participant M as M2 materializer
    G->>Q: enqueue(tile key, root, groups)
    L->>Q: front_key and has_job
    Q-->>L: borrowed root/groups/keys and cursors
    L->>P: plan selected transform slice
    alt resource pending
        L->>Q: rotate_front
    else materializable
        L->>M: materialize selected slice
        L->>Q: adopt_progress(next index, offset, serial)
    else cancelled or complete
        L->>L: free/queue-free root if required
        L->>Q: erase_job
        L->>Q: pop_front
    end

Dependency diagram

flowchart TB
    Loader[StreamingWorldLoader] --> Queue[M2BuildQueue]
    Queue --> Job[M2BuildJob]
    Job --> EngineBase[Node3D strong reference]
    Loader --> Planner[M2BuildBatchPlanner]
    Loader --> Materializers[Static and animated materializers]
    Loader --> Scheduler[RenderBudgetScheduler]
    Queue -. no dependency .-> SceneMutation[SceneTree destruction/attachment]
    Queue -. no dependency .-> Resources[Mesh / MultiMesh / ResourceLoader]

Ownership, threading and resources

• Queue owns keyed job records and FIFO String entries.
• Job retains the exact groups Dictionary, a fresh group-key snapshot and root.
• Root ownership remains with the loader/tile SceneTree; queue release never frees it.
• Loader validates is_instance_valid, frees empty/aborted roots and mutates tile state.
• M2BuildDispatchPlanner selects wait/materializer/advance actions without borrowing queue-owned engine references.
• All current operations run on the renderer main thread; no mutex is required.
• Group worker results cross their existing mutex mailbox before enqueue.

Errors, cancellation and recovery

Failure/state	Detection	Behavior	Diagnostic	Recovery
Invalid enqueue input	Entry guard	Reject without mutation	Contract fixture	Correct caller composition
Duplicate tile key	Keyed assignment	Replace job and append FIFO key	Duplicate fixture	Stale entries drain normally
Stale FIFO key	has_job == false	Loader pops and continues without permit	Stale fixture	No action required
Resource pending	Loader cache state	Rotate one front key and consume existing permit	Rotation fixture/build diagnostics	Retry when key returns front
Root externally invalid	Loader validity check	Erase job and pop key	Loader source contract	Tile may regroup/requeue later
Tile cancelled	Loader policy	Queue-free root, erase job; FIFO key drains/pops	Shutdown/lifetime regression	Eligible tile may requeue
World clear	Loader job-key snapshot	Cancel roots, then clear queue	Shutdown regression	New map starts empty

Configuration and capabilities

The queue introduces no settings. Existing static/animated batch limits, visibility/shadow settings, resource caches and M2_BUILD permits remain loader, planner, materializer and scheduler contracts.

Persistence, cache and migration

No state is serialized. No ADT/M2 cache format or version changes; no rebake or migration is required.

Diagnostics and observability

• pending_count preserves the existing FIFO-backed m2_build queue metric.
• active_job_count distinguishes keyed jobs from stale/duplicate FIFO entries.
• Diagnostic snapshots retain FIFO order and sort job records by tile key.
• Job diagnostics expose cursors/counts but never root/groups references.
• The module emits no logs.

Verification

• verify_m2_build_queue.gd covers invalid input, exact groups/root retention, FIFO, duplicate replacement, rotation, stale entries, atomic progress, erase/ clear engine lifetime, detached sorted diagnostics and loader boundaries.
• Planner, static/animated materializer, Mesh pipeline/cache/prototype, shutdown, facade, internal-access and checkpoint regressions protect adjacent behavior.
• Fidelity evidence is exact state-transition extraction; visual rules are not changed.
• Performance budget: 100 cycles of 256 enqueue/rotate/erase operations under one second.
• No original-client or proprietary asset comparison is claimed for bookkeeping state.

Extension points

Remaining resource readiness and dispatch logic may later move behind explicit commands once its cancellation semantics are independently fixed. A generic queue base, signals and callbacks are intentionally excluded.

Capability status

Capability	Status	Evidence	Gap/next step
Typed keyed M2 jobs and FIFO	Implemented extraction	Lifecycle/order/source/timing verifier	Asset-backed traversal pending
Cursor/serial ownership	Implemented extraction	Atomic progress fixtures	Typed batch-plan result remains Dictionary
Root destruction	Existing loader-owned	Lifetime/source/shutdown regressions	Keep outside state service
Resource dispatch decision	Implemented separately	Dispatch planner verifier	Asset-backed traversal pending
Resource observation/requests	Existing loader-owned	Adjacent cache/build tests	Safe extraction remains

Known gaps and risks

• Grouped transforms remain untyped Dictionary/Array data.
• Queue permits duplicate and stale FIFO keys intentionally for compatibility.
• Job accessors return borrowed mutable groups/key collections; current loader is the sole consumer.
• Private asset traversal, long-flight p95/p99 and leak evidence remain unavailable.

Source map

Path	Responsibility
src/render/m2/m2_build_job.gd	Strong root/groups references, key snapshot and progress cursors
src/render/m2/m2_build_queue.gd	Keyed job ownership, FIFO/stale/rotation lifecycle and diagnostics
src/scenes/streaming/streaming_world_loader.gd	Eligibility, readiness, permits, materialization, tile state and root cleanup
src/render/m2/m2_build_batch_planner.gd	Batch count and cursor-plan calculation
src/render/m2/m2_build_dispatch_planner.gd	Resource-state action and transition planning
src/tools/verify_m2_build_queue.gd	Lifecycle/order/lifetime/boundary/timing regression

Related decisions and references

• m2-build-batch-planner.md
• m2-build-dispatch-planner.md
• m2-build-resource-snapshot.md
• m2-static-batch-materializer.md
• m2-animated-instance-materializer.md
• world-renderer.md
• ../../RENDER.md
• ../../targets/roadmap/02-rendering-and-graphics.md