open-wc/docs/modules/m2-build-queue.md

# M2 Build Queue

## Metadata

| Field | Value |
|---|---|
| Status | Implemented |
| Target/work package | M03 / `M03-RND-M2-BUILD-QUEUE-001` |
| Owners | Pending M2 build-job records, FIFO tile keys and build cursors |
| Last verified | Worktree `work/sindo-main-codex/m03-m2-build-queue`, 2026-07-18 |
| Profiles/capabilities | Existing static MultiMesh and animated-instance build paths |

## Purpose

Own typed pending M2 build state outside `StreamingWorldLoader`: a keyed job
retains its M2 root, grouped transforms, insertion-order group keys and three
progress cursors, while a separate FIFO retains scheduling order and stale keys.

## Non-goals

- Decide tile eligibility, detail radius, resource readiness or retries.
- Consume scheduler permits or plan batch sizes.
- Create, attach or free Nodes, Meshes, MultiMeshes or animated instances.
- Load/cache resources, group placements or mutate tile state.
- Change model order, transform order, render settings or visible output.

## Context and boundaries

```mermaid
flowchart LR
    Groups[M2PlacementGrouper result] --> Loader[StreamingWorldLoader adapter]
    Loader --> Queue[M2BuildQueue]
    Queue --> Job[M2BuildJob]
    Queue --> Loader
    Loader --> Planner[M2BuildBatchPlanner]
    Loader --> Dispatch[M2BuildDispatchPlanner]
    Loader --> Static[M2StaticBatchMaterializer]
    Loader --> Animated[M2AnimatedInstanceMaterializer]
    Loader --> Scheduler[RenderBudgetScheduler]
```

Allowed dependencies are String, Dictionary, Array, RefCounted and a strong
Node3D reference. SceneTree destruction, materializers, Mesh/MultiMesh,
RenderingServer/RIDs, ResourceLoader/files, workers/mutexes, scheduler policy,
gameplay, network and Editor APIs are forbidden.

## Public API

### `M2BuildJob`

| Symbol | Kind | Purpose | Thread/lifetime | Errors |
|---|---|---|---|---|
| `tile_key()` | Query | Immutable keyed identity | Main thread; job lifetime | None |
| `root()` | Query | Borrow strongly retained M2 root | Main thread; no ownership transfer | May be externally invalidated |
| `groups()` | Query | Borrow exact grouped-transform Dictionary | Main thread; job lifetime | None |
| `group_keys()` | Query | Borrow enqueue-time insertion-order key snapshot | Main thread; job lifetime | None |
| `group_index()` | Query | Current model-path cursor | Main thread | None |
| `transform_offset()` | Query | Current transform cursor within group | Main thread | None |
| `batch_serial()` | Query | Current batch naming serial | Main thread | None |
| `adopt_progress(group_index, offset, serial)` | Command | Atomically replace all progress values | Main thread | Raw integer semantics retained |
| `diagnostic_snapshot()` | Query | Detached scalar/count state | Any serialized caller | Omits engine references |

### `M2BuildQueue`

| Symbol | Kind | Purpose | Thread/lifetime | Errors |
|---|---|---|---|---|
| `enqueue(tile_key, root, groups)` | Command | Replace keyed job and append FIFO key | Main thread; until erase/clear | Invalid/empty input returns false |
| `has_pending/front_key/pop_front/rotate_front` | Commands/queries | Inspect, drain or rotate FIFO independently of job validity | Main thread | Empty returns false/empty String |
| `has_job/job_for` | Queries | Inspect keyed record, including stale-key distinction | Main thread | Missing returns false/null |
| `root_for/groups_for/group_keys_for` | Queries | Borrow current job inputs | Main thread | Stale returns null/empty collection |
| `group_index_for/transform_offset_for/batch_serial_for` | Queries | Read current progress | Main thread | Stale returns zero |
| `adopt_progress(tile_key, group_index, offset, serial)` | Command | Atomically update current job progress | Main thread | Stale returns false |
| `erase_job(tile_key)` | Command | Remove keyed job but preserve FIFO keys | Main thread | Unknown returns false |
| `job_keys()` | Query | Detached active-key snapshot for loader cleanup | Main thread | Empty returns empty Array |
| `clear()` | Command | Release all job and FIFO references | Main thread | Idempotent; never frees Nodes |
| `pending_count/active_job_count` | Queries | FIFO and keyed-job metrics | Main thread | None |
| `diagnostic_snapshot()` | Query | Detached FIFO and sorted job diagnostics | Main thread | None |

## Inputs and outputs

| Direction | Contract/data | Producer | Consumer | Ownership | Thread/lifetime |
|---|---|---|---|---|---|
| Input | Tile key and M2 root | Loader group-result finalizer | Queue/job | Copied String; strong borrowed Node reference | Until erase/clear |
| Input | Grouped transforms Dictionary | M2PlacementGrouper through loader mailbox | Job | Exact strong reference | Job lifetime |
| Input | `groups.keys()` order | Job constructor | Loader group selection | Job-owned Array snapshot | Job lifetime |
| Input | Next group/offset/serial | Planner/materializer adapter | Job progress | Copied integers | Until next adoption |
| Output | Front/rotated/popped tile key | Queue | Loader scheduling loop | Copied String | One drain step |
| Output | Borrowed root/groups/group keys | Job/queue | Loader readiness/materialization | No ownership transfer | One drain step |
| Output | Detached diagnostics/key snapshots | Queue/job | Tests/metrics/cleanup adapter | Caller-owned collections | Call result |

Side effects are keyed/FIFO collection mutation, strong-reference retention and
cursor mutation. No engine object is created, destroyed or attached.

## Data flow

```mermaid
flowchart TD
    Enqueue[Tile key, root and grouped transforms] --> Job[Create or replace M2BuildJob]
    Enqueue --> FIFO[Append tile key to FIFO]
    FIFO --> Front[Borrow front key]
    Front --> Current{Current keyed job exists?}
    Current -->|no| Pop[Pop stale key]
    Current -->|yes| Loader[Loader eligibility/readiness/materialization]
    Loader -->|resource pending| Rotate[Move front key to tail]
    Loader -->|operation complete| Progress[Adopt group, offset and serial]
    Loader -->|finish/cancel| Erase[Erase job only]
    Erase --> Pop
```

## Lifecycle/state

```mermaid
stateDiagram-v2
    [*] --> Absent
    Absent --> Queued: enqueue valid job and FIFO key
    Queued --> Rotated: resource pending
    Rotated --> Queued: returns to FIFO front
    Queued --> Progressed: adopt progress
    Progressed --> Queued: group remains
    Queued --> StaleQueued: erase job before FIFO pop
    Progressed --> StaleQueued: finish or cancel
    StaleQueued --> Absent: pop stale key
    Queued --> Absent: clear
    StaleQueued --> Absent: clear
```

Duplicate enqueue replaces the current keyed job and adds another FIFO entry;
later erase can therefore leave multiple stale entries, matching historical raw state.

## Main sequence

```mermaid
sequenceDiagram
    participant G as M2 group-result drain
    participant Q as M2BuildQueue
    participant L as StreamingWorldLoader
    participant P as M2BuildBatchPlanner
    participant M as M2 materializer
    G->>Q: enqueue(tile key, root, groups)
    L->>Q: front_key and has_job
    Q-->>L: borrowed root/groups/keys and cursors
    L->>P: plan selected transform slice
    alt resource pending
        L->>Q: rotate_front
    else materializable
        L->>M: materialize selected slice
        L->>Q: adopt_progress(next index, offset, serial)
    else cancelled or complete
        L->>L: free/queue-free root if required
        L->>Q: erase_job
        L->>Q: pop_front
    end
```

## Dependency diagram

```mermaid
flowchart TB
    Loader[StreamingWorldLoader] --> Queue[M2BuildQueue]
    Queue --> Job[M2BuildJob]
    Job --> EngineBase[Node3D strong reference]
    Loader --> Planner[M2BuildBatchPlanner]
    Loader --> Materializers[Static and animated materializers]
    Loader --> Scheduler[RenderBudgetScheduler]
    Queue -. no dependency .-> SceneMutation[SceneTree destruction/attachment]
    Queue -. no dependency .-> Resources[Mesh / MultiMesh / ResourceLoader]
```

## Ownership, threading and resources

- Queue owns keyed job records and FIFO String entries.
- Job retains the exact groups Dictionary, a fresh group-key snapshot and root.
- Root ownership remains with the loader/tile SceneTree; queue release never frees it.
- Loader validates `is_instance_valid`, frees empty/aborted roots and mutates tile state.
- `M2BuildDispatchPlanner` selects wait/materializer/advance actions without
  borrowing queue-owned engine references.
- All current operations run on the renderer main thread; no mutex is required.
- Group worker results cross their existing mutex mailbox before enqueue.

## Errors, cancellation and recovery

| Failure/state | Detection | Behavior | Diagnostic | Recovery |
|---|---|---|---|---|
| Invalid enqueue input | Entry guard | Reject without mutation | Contract fixture | Correct caller composition |
| Duplicate tile key | Keyed assignment | Replace job and append FIFO key | Duplicate fixture | Stale entries drain normally |
| Stale FIFO key | `has_job == false` | Loader pops and continues without permit | Stale fixture | No action required |
| Resource pending | Loader cache state | Rotate one front key and consume existing permit | Rotation fixture/build diagnostics | Retry when key returns front |
| Root externally invalid | Loader validity check | Erase job and pop key | Loader source contract | Tile may regroup/requeue later |
| Tile cancelled | Loader policy | Queue-free root, erase job; FIFO key drains/pops | Shutdown/lifetime regression | Eligible tile may requeue |
| World clear | Loader job-key snapshot | Cancel roots, then clear queue | Shutdown regression | New map starts empty |

## Configuration and capabilities

The queue introduces no settings. Existing static/animated batch limits,
visibility/shadow settings, resource caches and `M2_BUILD` permits remain loader,
planner, materializer and scheduler contracts.

## Persistence, cache and migration

No state is serialized. No ADT/M2 cache format or version changes; no rebake or
migration is required.

## Diagnostics and observability

- `pending_count` preserves the existing FIFO-backed `m2_build` queue metric.
- `active_job_count` distinguishes keyed jobs from stale/duplicate FIFO entries.
- Diagnostic snapshots retain FIFO order and sort job records by tile key.
- Job diagnostics expose cursors/counts but never root/groups references.
- The module emits no logs.

## Verification

- `verify_m2_build_queue.gd` covers invalid input, exact groups/root retention,
  FIFO, duplicate replacement, rotation, stale entries, atomic progress, erase/
  clear engine lifetime, detached sorted diagnostics and loader boundaries.
- Planner, static/animated materializer, Mesh pipeline/cache/prototype, shutdown,
  facade, internal-access and checkpoint regressions protect adjacent behavior.
- Fidelity evidence is exact state-transition extraction; visual rules are not changed.
- Performance budget: 100 cycles of 256 enqueue/rotate/erase operations under one second.
- No original-client or proprietary asset comparison is claimed for bookkeeping state.

## Extension points

Remaining resource readiness and dispatch logic may later move behind explicit
commands once its cancellation semantics are independently fixed. A generic
queue base, signals and callbacks are intentionally excluded.

## Capability status

| Capability | Status | Evidence | Gap/next step |
|---|---|---|---|
| Typed keyed M2 jobs and FIFO | Implemented extraction | Lifecycle/order/source/timing verifier | Asset-backed traversal pending |
| Cursor/serial ownership | Implemented extraction | Atomic progress fixtures | Typed batch-plan result remains Dictionary |
| Root destruction | Existing loader-owned | Lifetime/source/shutdown regressions | Keep outside state service |
| Resource dispatch decision | Implemented separately | Dispatch planner verifier | Asset-backed traversal pending |
| Resource observation/requests | Existing loader-owned | Adjacent cache/build tests | Safe extraction remains |

## Known gaps and risks

- Grouped transforms remain untyped Dictionary/Array data.
- Queue permits duplicate and stale FIFO keys intentionally for compatibility.
- Job accessors return borrowed mutable groups/key collections; current loader is the sole consumer.
- Private asset traversal, long-flight p95/p99 and leak evidence remain unavailable.

## Source map

| Path | Responsibility |
|---|---|
| `src/render/m2/m2_build_job.gd` | Strong root/groups references, key snapshot and progress cursors |
| `src/render/m2/m2_build_queue.gd` | Keyed job ownership, FIFO/stale/rotation lifecycle and diagnostics |
| `src/scenes/streaming/streaming_world_loader.gd` | Eligibility, readiness, permits, materialization, tile state and root cleanup |
| `src/render/m2/m2_build_batch_planner.gd` | Batch count and cursor-plan calculation |
| `src/render/m2/m2_build_dispatch_planner.gd` | Resource-state action and transition planning |
| `src/tools/verify_m2_build_queue.gd` | Lifecycle/order/lifetime/boundary/timing regression |

## Related decisions and references

- [`m2-build-batch-planner.md`](m2-build-batch-planner.md)
- [`m2-build-dispatch-planner.md`](m2-build-dispatch-planner.md)
- [`m2-build-resource-snapshot.md`](m2-build-resource-snapshot.md)
- [`m2-static-batch-materializer.md`](m2-static-batch-materializer.md)
- [`m2-animated-instance-materializer.md`](m2-animated-instance-materializer.md)
- [`world-renderer.md`](world-renderer.md)
- [`../../RENDER.md`](../../RENDER.md)
- [`../../targets/roadmap/02-rendering-and-graphics.md`](../../targets/roadmap/02-rendering-and-graphics.md)