sindoring/open-wc
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

render: extract M2 build queue

Browse Source
This commit is contained in:
sindoring
2026-07-18 02:23:40 +04:00
parent 3ee9e77422
commit 4a8338f2f0
13 changed files with 882 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/modules/m2-build-queue.md
+258
View File
@@ -0,0 +1,258 @@
# M2 Build Queue
## Metadata
| Field | Value |
|---|---|
| Status | Implemented |
| Target/work package | M03 / `M03-RND-M2-BUILD-QUEUE-001` |
| Owners | Pending M2 build-job records, FIFO tile keys and build cursors |
| Last verified | Worktree `work/sindo-main-codex/m03-m2-build-queue`, 2026-07-18 |
| Profiles/capabilities | Existing static MultiMesh and animated-instance build paths |
## Purpose
Own typed pending M2 build state outside `StreamingWorldLoader`: a keyed job
retains its M2 root, grouped transforms, insertion-order group keys and three
progress cursors, while a separate FIFO retains scheduling order and stale keys.
## Non-goals
- Decide tile eligibility, detail radius, resource readiness or retries.
- Consume scheduler permits or plan batch sizes.
- Create, attach or free Nodes, Meshes, MultiMeshes or animated instances.
- Load/cache resources, group placements or mutate tile state.
- Change model order, transform order, render settings or visible output.
## Context and boundaries
```mermaid
flowchart LR
Groups[M2PlacementGrouper result] --> Loader[StreamingWorldLoader adapter]
Loader --> Queue[M2BuildQueue]
Queue --> Job[M2BuildJob]
Queue --> Loader
Loader --> Planner[M2BuildBatchPlanner]
Loader --> Static[M2StaticBatchMaterializer]
Loader --> Animated[M2AnimatedInstanceMaterializer]
Loader --> Scheduler[RenderBudgetScheduler]
```
Allowed dependencies are String, Dictionary, Array, RefCounted and a strong
Node3D reference. SceneTree destruction, materializers, Mesh/MultiMesh,
RenderingServer/RIDs, ResourceLoader/files, workers/mutexes, scheduler policy,
gameplay, network and Editor APIs are forbidden.
## Public API
### `M2BuildJob`
| Symbol | Kind | Purpose | Thread/lifetime | Errors |
|---|---|---|---|---|
| `tile_key()` | Query | Immutable keyed identity | Main thread; job lifetime | None |
| `root()` | Query | Borrow strongly retained M2 root | Main thread; no ownership transfer | May be externally invalidated |
| `groups()` | Query | Borrow exact grouped-transform Dictionary | Main thread; job lifetime | None |
| `group_keys()` | Query | Borrow enqueue-time insertion-order key snapshot | Main thread; job lifetime | None |
| `group_index()` | Query | Current model-path cursor | Main thread | None |
| `transform_offset()` | Query | Current transform cursor within group | Main thread | None |
| `batch_serial()` | Query | Current batch naming serial | Main thread | None |
| `adopt_progress(group_index, offset, serial)` | Command | Atomically replace all progress values | Main thread | Raw integer semantics retained |
| `diagnostic_snapshot()` | Query | Detached scalar/count state | Any serialized caller | Omits engine references |
### `M2BuildQueue`
| Symbol | Kind | Purpose | Thread/lifetime | Errors |
|---|---|---|---|---|
| `enqueue(tile_key, root, groups)` | Command | Replace keyed job and append FIFO key | Main thread; until erase/clear | Invalid/empty input returns false |
| `has_pending/front_key/pop_front/rotate_front` | Commands/queries | Inspect, drain or rotate FIFO independently of job validity | Main thread | Empty returns false/empty String |
| `has_job/job_for` | Queries | Inspect keyed record, including stale-key distinction | Main thread | Missing returns false/null |
| `root_for/groups_for/group_keys_for` | Queries | Borrow current job inputs | Main thread | Stale returns null/empty collection |
| `group_index_for/transform_offset_for/batch_serial_for` | Queries | Read current progress | Main thread | Stale returns zero |
| `adopt_progress(tile_key, group_index, offset, serial)` | Command | Atomically update current job progress | Main thread | Stale returns false |
| `erase_job(tile_key)` | Command | Remove keyed job but preserve FIFO keys | Main thread | Unknown returns false |
| `job_keys()` | Query | Detached active-key snapshot for loader cleanup | Main thread | Empty returns empty Array |
| `clear()` | Command | Release all job and FIFO references | Main thread | Idempotent; never frees Nodes |
| `pending_count/active_job_count` | Queries | FIFO and keyed-job metrics | Main thread | None |
| `diagnostic_snapshot()` | Query | Detached FIFO and sorted job diagnostics | Main thread | None |
## Inputs and outputs
| Direction | Contract/data | Producer | Consumer | Ownership | Thread/lifetime |
|---|---|---|---|---|---|
| Input | Tile key and M2 root | Loader group-result finalizer | Queue/job | Copied String; strong borrowed Node reference | Until erase/clear |
| Input | Grouped transforms Dictionary | M2PlacementGrouper through loader mailbox | Job | Exact strong reference | Job lifetime |
| Input | `groups.keys()` order | Job constructor | Loader group selection | Job-owned Array snapshot | Job lifetime |
| Input | Next group/offset/serial | Planner/materializer adapter | Job progress | Copied integers | Until next adoption |
| Output | Front/rotated/popped tile key | Queue | Loader scheduling loop | Copied String | One drain step |
| Output | Borrowed root/groups/group keys | Job/queue | Loader readiness/materialization | No ownership transfer | One drain step |
| Output | Detached diagnostics/key snapshots | Queue/job | Tests/metrics/cleanup adapter | Caller-owned collections | Call result |
Side effects are keyed/FIFO collection mutation, strong-reference retention and
cursor mutation. No engine object is created, destroyed or attached.
## Data flow
```mermaid
flowchart TD
Enqueue[Tile key, root and grouped transforms] --> Job[Create or replace M2BuildJob]
Enqueue --> FIFO[Append tile key to FIFO]
FIFO --> Front[Borrow front key]
Front --> Current{Current keyed job exists?}
Current -->|no| Pop[Pop stale key]
Current -->|yes| Loader[Loader eligibility/readiness/materialization]
Loader -->|resource pending| Rotate[Move front key to tail]
Loader -->|operation complete| Progress[Adopt group, offset and serial]
Loader -->|finish/cancel| Erase[Erase job only]
Erase --> Pop
```
## Lifecycle/state
```mermaid
stateDiagram-v2
[*] --> Absent
Absent --> Queued: enqueue valid job and FIFO key
Queued --> Rotated: resource pending
Rotated --> Queued: returns to FIFO front
Queued --> Progressed: adopt progress
Progressed --> Queued: group remains
Queued --> StaleQueued: erase job before FIFO pop
Progressed --> StaleQueued: finish or cancel
StaleQueued --> Absent: pop stale key
Queued --> Absent: clear
StaleQueued --> Absent: clear
```
Duplicate enqueue replaces the current keyed job and adds another FIFO entry;
later erase can therefore leave multiple stale entries, matching historical raw state.
## Main sequence
```mermaid
sequenceDiagram
participant G as M2 group-result drain
participant Q as M2BuildQueue
participant L as StreamingWorldLoader
participant P as M2BuildBatchPlanner
participant M as M2 materializer
G->>Q: enqueue(tile key, root, groups)
L->>Q: front_key and has_job
Q-->>L: borrowed root/groups/keys and cursors
L->>P: plan selected transform slice
alt resource pending
L->>Q: rotate_front
else materializable
L->>M: materialize selected slice
L->>Q: adopt_progress(next index, offset, serial)
else cancelled or complete
L->>L: free/queue-free root if required
L->>Q: erase_job
L->>Q: pop_front
end
```
## Dependency diagram
```mermaid
flowchart TB
Loader[StreamingWorldLoader] --> Queue[M2BuildQueue]
Queue --> Job[M2BuildJob]
Job --> EngineBase[Node3D strong reference]
Loader --> Planner[M2BuildBatchPlanner]
Loader --> Materializers[Static and animated materializers]
Loader --> Scheduler[RenderBudgetScheduler]
Queue -. no dependency .-> SceneMutation[SceneTree destruction/attachment]
Queue -. no dependency .-> Resources[Mesh / MultiMesh / ResourceLoader]
```
## Ownership, threading and resources
- Queue owns keyed job records and FIFO String entries.
- Job retains the exact groups Dictionary, a fresh group-key snapshot and root.
- Root ownership remains with the loader/tile SceneTree; queue release never frees it.
- Loader validates `is_instance_valid`, frees empty/aborted roots and mutates tile state.
- All current operations run on the renderer main thread; no mutex is required.
- Group worker results cross their existing mutex mailbox before enqueue.
## Errors, cancellation and recovery
| Failure/state | Detection | Behavior | Diagnostic | Recovery |
|---|---|---|---|---|
| Invalid enqueue input | Entry guard | Reject without mutation | Contract fixture | Correct caller composition |
| Duplicate tile key | Keyed assignment | Replace job and append FIFO key | Duplicate fixture | Stale entries drain normally |
| Stale FIFO key | `has_job == false` | Loader pops and continues without permit | Stale fixture | No action required |
| Resource pending | Loader cache state | Rotate one front key and consume existing permit | Rotation fixture/build diagnostics | Retry when key returns front |
| Root externally invalid | Loader validity check | Erase job and pop key | Loader source contract | Tile may regroup/requeue later |
| Tile cancelled | Loader policy | Queue-free root, erase job; FIFO key drains/pops | Shutdown/lifetime regression | Eligible tile may requeue |
| World clear | Loader job-key snapshot | Cancel roots, then clear queue | Shutdown regression | New map starts empty |
## Configuration and capabilities
The queue introduces no settings. Existing static/animated batch limits,
visibility/shadow settings, resource caches and `M2_BUILD` permits remain loader,
planner, materializer and scheduler contracts.
## Persistence, cache and migration
No state is serialized. No ADT/M2 cache format or version changes; no rebake or
migration is required.
## Diagnostics and observability
- `pending_count` preserves the existing FIFO-backed `m2_build` queue metric.
- `active_job_count` distinguishes keyed jobs from stale/duplicate FIFO entries.
- Diagnostic snapshots retain FIFO order and sort job records by tile key.
- Job diagnostics expose cursors/counts but never root/groups references.
- The module emits no logs.
## Verification
- `verify_m2_build_queue.gd` covers invalid input, exact groups/root retention,
FIFO, duplicate replacement, rotation, stale entries, atomic progress, erase/
clear engine lifetime, detached sorted diagnostics and loader boundaries.
- Planner, static/animated materializer, Mesh pipeline/cache/prototype, shutdown,
facade, internal-access and checkpoint regressions protect adjacent behavior.
- Fidelity evidence is exact state-transition extraction; visual rules are not changed.
- Performance budget: 100 cycles of 256 enqueue/rotate/erase operations under one second.
- No original-client or proprietary asset comparison is claimed for bookkeeping state.
## Extension points
Remaining resource readiness and dispatch logic may later move behind explicit
commands once its cancellation semantics are independently fixed. A generic
queue base, signals and callbacks are intentionally excluded.
## Capability status
| Capability | Status | Evidence | Gap/next step |
|---|---|---|---|
| Typed keyed M2 jobs and FIFO | Implemented extraction | Lifecycle/order/source/timing verifier | Asset-backed traversal pending |
| Cursor/serial ownership | Implemented extraction | Atomic progress fixtures | Typed batch-plan result remains Dictionary |
| Root destruction | Existing loader-owned | Lifetime/source/shutdown regressions | Keep outside state service |
| Resource readiness/dispatch | Existing loader-owned | Adjacent cache/build tests | Safe extraction remains |
## Known gaps and risks
- Grouped transforms remain untyped Dictionary/Array data.
- Queue permits duplicate and stale FIFO keys intentionally for compatibility.
- Job accessors return borrowed mutable groups/key collections; current loader is the sole consumer.
- Private asset traversal, long-flight p95/p99 and leak evidence remain unavailable.
## Source map
| Path | Responsibility |
|---|---|
| `src/render/m2/m2_build_job.gd` | Strong root/groups references, key snapshot and progress cursors |
| `src/render/m2/m2_build_queue.gd` | Keyed job ownership, FIFO/stale/rotation lifecycle and diagnostics |
| `src/scenes/streaming/streaming_world_loader.gd` | Eligibility, readiness, permits, materialization, tile state and root cleanup |
| `src/render/m2/m2_build_batch_planner.gd` | Batch count and cursor-plan calculation |
| `src/tools/verify_m2_build_queue.gd` | Lifecycle/order/lifetime/boundary/timing regression |
## Related decisions and references
- [`m2-build-batch-planner.md`](m2-build-batch-planner.md)
- [`m2-static-batch-materializer.md`](m2-static-batch-materializer.md)
- [`m2-animated-instance-materializer.md`](m2-animated-instance-materializer.md)
- [`world-renderer.md`](world-renderer.md)
- [`../../RENDER.md`](../../RENDER.md)
- [`../../targets/roadmap/02-rendering-and-graphics.md`](../../targets/roadmap/02-rendering-and-graphics.md)