248 lines
12 KiB
Markdown
248 lines
12 KiB
Markdown
# ADT Water Load Pipeline State
|
|
|
|
## Metadata
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Status | Implemented |
|
|
| Target/work package | M03 / `M03-RND-ADT-WATER-LOAD-PIPELINE-001` |
|
|
| Owners | ADT water pending FIFO/dedupe, active task IDs and worker-safe result mailbox |
|
|
| Last verified | Worktree `work/sindo-main-codex/m03-adt-water-load-pipeline`, 2026-07-17 |
|
|
| Profiles/capabilities | ADT MH2O/MCLQ water loading; profile-independent state |
|
|
|
|
## Purpose
|
|
|
|
Own the cross-frame and cross-thread bookkeeping for asynchronous ADT water
|
|
loads outside `StreamingWorldLoader`: FIFO requests, queued-key deduplication,
|
|
opaque active task IDs and parsed-result delivery. The loader retains task
|
|
execution, parsing, budgets, stale-result decisions and scene finalization.
|
|
|
|
## Non-goals
|
|
|
|
- Start, cancel or wait WorkerThreadPool tasks.
|
|
- Instantiate `ADTLoader` or interpret MH2O/MCLQ data.
|
|
- Select concurrency limits or consume render permits.
|
|
- Create, attach or free Nodes, Meshes, materials or RIDs.
|
|
- Change liquid visuals, coordinates, tile acceptance or loaded-state timing.
|
|
|
|
## Context and boundaries
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
Tile[Loaded tile needing water] --> Loader[StreamingWorldLoader]
|
|
Loader --> State[AdtWaterLoadPipelineState]
|
|
State --> Request[Oldest tile/path request]
|
|
Request --> Loader
|
|
Loader --> Worker[WorkerThreadPool ADTLoader task]
|
|
Worker --> State
|
|
State --> Result[Parsed ADT result]
|
|
Result --> Loader
|
|
Loader --> Budget[WATER_FINALIZE permit]
|
|
Loader --> Builder[ADTBuilder water scene]
|
|
Builder --> Scene[Main-thread tile Node attach]
|
|
```
|
|
|
|
Allowed dependencies are RefCounted, Dictionary/Array/String and one Godot
|
|
`Mutex` around results. WorkerThreadPool, ADTLoader, scheduler, tile state,
|
|
scene-tree/render resources and gameplay/editor/network are forbidden.
|
|
|
|
## Public API
|
|
|
|
| Symbol | Kind | Purpose | Thread/lifetime | Errors |
|
|
|---|---|---|---|---|
|
|
| `enqueue_request(tile_key, adt_path)` | Command/query | Append deduplicated FIFO request | Renderer main thread; until dequeue/cancel/clear | Empty/queued/active returns false |
|
|
| `take_next_request()` | Command/query | Pop oldest request and release queued marker | Renderer main thread | Empty returns `{}` |
|
|
| `cancel_pending_requests(tile_key)` | Command/query | Remove all matching pending entries | Tile release/main thread | Empty/unknown removes zero; active unaffected |
|
|
| `remember_active_task(tile_key, task_id)` | Command/query | Retain loader-created opaque task ID | Main thread; until completion/clear | Empty/duplicate returns false |
|
|
| `has_active_task` / `active_task_id_for` | Query | Test/borrow active task identity | Main thread | Unknown ID is `-1` |
|
|
| `complete_active_task(tile_key)` | Command/query | Remove and return completed task ID | Main thread after wait | Unknown returns `-1` |
|
|
| `active_task_ids_snapshot()` | Query | Copy IDs for orderly shutdown waiting | Main thread | Detached Array |
|
|
| `publish_result(tile_key, path, data)` | Command/query | Append parsed result under mutex | Worker or main thread | Empty key/path returns false |
|
|
| `has_result()` / `pop_result()` | Query/command | Observe/pop oldest result under mutex | Main-thread drain | Empty pop returns `{}` |
|
|
| `total_load_count()` | Query | Pending plus active metric | Main thread | Results intentionally excluded |
|
|
| `pending_request_count()` / `active_task_count()` | Query | Support loader concurrency loop | Main thread | None |
|
|
| `clear()` | Command | Release all bookkeeping and results | Main thread after wait or map reset | Does not interrupt workers |
|
|
| `diagnostic_snapshot()` | Query | Return detached scalar requests/tasks/results | Main thread; result mailbox locked | Water data omitted |
|
|
|
|
## Inputs and outputs
|
|
|
|
| Direction | Contract/data | Producer | Consumer | Ownership | Thread/lifetime |
|
|
|---|---|---|---|---|---|
|
|
| Input | Tile key and ADT path | Loader tile state | Pending FIFO | Copied Strings | Until dequeue/cancel/clear |
|
|
| Input | Opaque task ID | Loader WorkerThreadPool adapter | Active task map | Integer value | Until wait/completion/clear |
|
|
| Input | Tile/path/parsed ADT Dictionary | Worker parser adapter | Result mailbox | Dictionary reference retained | Until pop/clear |
|
|
| Output | Oldest request Dictionary | Pipeline state | Loader task-start adapter | Fresh record from FIFO | One start attempt |
|
|
| Output | Active task ID snapshot | Pipeline state | Loader shutdown wait | Detached Array | Shutdown loop |
|
|
| Output | Oldest result Dictionary | Pipeline state | Loader finalize adapter | Mailbox record transferred | One finalize attempt |
|
|
| Output | Scalar diagnostics | Pipeline state | Verifier/future metrics | Detached records | Snapshot lifetime |
|
|
|
|
Side effects are limited to collection mutation, reference retention and result
|
|
mutex locking. Parsing, permits and scene mutation remain outside the module.
|
|
|
|
## Data flow
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
Enqueue[enqueue tile/path] --> Duplicate{Queued or active?}
|
|
Duplicate -->|yes| Reject[false; unchanged]
|
|
Duplicate -->|no| FIFO[Append FIFO and queued key]
|
|
FIFO --> Take[take next; release queued marker]
|
|
Take --> Validate[Loader checks tile/path]
|
|
Validate --> Start[Loader starts worker and remembers task ID]
|
|
Start --> Parse[Worker parses ADT]
|
|
Parse --> Publish[Mutex-protected publish result]
|
|
Publish --> Permit[Loader obtains WATER_FINALIZE permit]
|
|
Permit --> Pop[Pop oldest result]
|
|
Pop --> Wait[Loader waits task and completes active ID]
|
|
Wait --> Stale{Tile/path/root current?}
|
|
Stale -->|no| Discard[Discard result]
|
|
Stale -->|yes| Attach[Build and attach water scene]
|
|
```
|
|
|
|
## Lifecycle/state
|
|
|
|
A tile request progresses through `Absent`, `Queued`, `Active` and
|
|
`ResultReady`; the loader wait/completion returns bookkeeping to `Absent` before
|
|
validating/attaching the result. Queued tile release cancels only `Queued`.
|
|
Active work is not interrupted; a later stale result is drained and discarded.
|
|
|
|
```mermaid
|
|
stateDiagram-v2
|
|
[*] --> Absent
|
|
Absent --> Queued: accepted enqueue
|
|
Queued --> Absent: pending cancellation or invalid dequeue
|
|
Queued --> Active: loader starts task
|
|
Active --> ResultReady: worker publishes result
|
|
ResultReady --> Absent: loader waits and completes task
|
|
Queued --> Absent: clear
|
|
Active --> Absent: clear bookkeeping only
|
|
ResultReady --> Absent: clear
|
|
```
|
|
|
|
## Main sequence
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant Loader as StreamingWorldLoader main thread
|
|
participant State as AdtWaterLoadPipelineState
|
|
participant Worker as WorkerThreadPool task
|
|
participant Budget as RenderBudgetScheduler
|
|
participant Builder as ADTBuilder
|
|
Loader->>State: enqueue_request(tile key, ADT path)
|
|
Loader->>State: take_next_request()
|
|
Loader->>Worker: add_task(parse binding)
|
|
Loader->>State: remember_active_task(key, task ID)
|
|
Worker->>Worker: ADTLoader.load_adt(path)
|
|
Worker->>State: publish_result(key, path, data)
|
|
Note over State: result mutex protects publication/pop
|
|
Loader->>Budget: try_consume_permit(WATER_FINALIZE)
|
|
Loader->>State: pop_result()
|
|
Loader->>Worker: wait_for_task_completion(task ID)
|
|
Loader->>State: complete_active_task(key)
|
|
Loader->>Loader: reject stale tile/path/root
|
|
Loader->>Builder: build_tile_water_scene(data, origin)
|
|
Loader->>Loader: attach Node and mark water_loaded
|
|
```
|
|
|
|
## Dependency diagram
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
Loader[StreamingWorldLoader] --> State[AdtWaterLoadPipelineState]
|
|
Loader --> Scheduler[RenderBudgetScheduler]
|
|
Loader --> Worker[WorkerThreadPool / ADTLoader]
|
|
Loader --> Builder[ADTBuilder / SceneTree]
|
|
Worker -->|publish only| State
|
|
State -. no dependency .-> Scheduler
|
|
State -. no dependency .-> Builder
|
|
```
|
|
|
|
## Ownership, threading and resources
|
|
|
|
- Main thread owns pending and active-task collection mutation.
|
|
- Worker threads may call only `publish_result`; result observe/pop/clear also lock.
|
|
- Parsed water Dictionary identity is retained until main-thread pop.
|
|
- Loader owns WorkerThreadPool task lifetime and must wait before orderly shutdown clear.
|
|
- Loader owns tile states, ADTBuilder, water Nodes and every render resource.
|
|
- `clear` cannot cancel a worker; post-reset publication becomes a stale result
|
|
and is discarded because no active task/tile ownership remains.
|
|
|
|
## Errors, cancellation and recovery
|
|
|
|
| Failure | Detection | Behavior | Diagnostic | Recovery |
|
|
|---|---|---|---|---|
|
|
| Empty/duplicate request | Enqueue guards | Reject unchanged | Contract verifier | Correct caller state |
|
|
| Tile removed while queued | Loader release | Remove matching pending requests | Source/lifecycle verifier | Reload can enqueue later |
|
|
| Tile removed while active | Loader stale check after result | Wait task, remove active ID, discard result | Existing metrics | Reload can enqueue later |
|
|
| Parser unavailable/empty | Worker publishes empty Dictionary | Loader marks `water_loaded` without Node | Existing behavior | Future tile reload |
|
|
| Path changed | Loader compares result path | Discard after task completion | Source regression | Current tile can request again |
|
|
| Shutdown | Loader snapshots/waits active IDs | Clear all bookkeeping/results | Shutdown/source verifier | New loader starts empty |
|
|
|
|
## Configuration and capabilities
|
|
|
|
| Setting/capability | Default | Profile | Runtime mutable | Effect |
|
|
|---|---|---|---|---|
|
|
| `enable_water` | Loader setting | Quality/custom | Yes | Gates enqueue/process, unchanged |
|
|
| `max_concurrent_water_tasks` | Loader setting | Quality/custom | Yes | Bounds active tasks outside state |
|
|
| `water_finalize_ops_per_tick` | Loader setting | Quality/custom | Yes | Bounds result pops via scheduler |
|
|
|
|
## Persistence, cache and migration
|
|
|
|
No state is serialized or cached. ADT source/cache formats and liquid material
|
|
versions are unchanged; no migration or rebuild is required.
|
|
|
|
## Diagnostics and observability
|
|
|
|
- `total_load_count` preserves the prior pending-plus-active `water` metric.
|
|
- Snapshot exposes FIFO tile/path, sorted active keys and result tile/path only.
|
|
- Parsed liquid payloads, Nodes and task internals are not exposed.
|
|
- No logs or correlation IDs are introduced.
|
|
|
|
## Verification
|
|
|
|
- `verify_adt_water_load_pipeline_state.gd`: invalid/dedupe/FIFO behavior,
|
|
active IDs, payload identity, real Thread publication, pending cancellation,
|
|
clear, detached diagnostics, loader boundaries and bounded timing.
|
|
- Existing material/shutdown/facade/streaming regressions cover adjacent behavior.
|
|
- Fidelity evidence is preservation of current scheduling/finalize transitions;
|
|
no MH2O/MCLQ visual or original-client parity claim is made.
|
|
- Performance budget: 100 cycles of 256 request/task/result transitions under 1 second.
|
|
|
|
## Extension points
|
|
|
|
- ADT parse execution can later move behind a parser/job adapter without changing state.
|
|
- Water scene materialization can be extracted separately because it owns Nodes
|
|
and main-thread effects rather than async bookkeeping.
|
|
|
|
## Capability status
|
|
|
|
| Capability | Status | Evidence | Gap/next step |
|
|
|---|---|---|---|
|
|
| ADT water load pipeline state | Implemented extraction | FIFO/task/thread/source/timing verifier | Asset-backed traversal/leak evidence pending |
|
|
| ADT liquid parsing | Existing/loader-owned adapter | Native ADTLoader and dry-run paths | MH2O/MCLQ fidelity fixtures pending |
|
|
| Water scene finalization | Existing/loader-owned | Scheduler/material/shutdown regressions | Separate materialization extraction possible |
|
|
|
|
## Known gaps and risks
|
|
|
|
- `clear` intentionally does not interrupt active WorkerThreadPool work.
|
|
- Result payload Dictionaries are retained by reference and must not be mutated after publication.
|
|
- Result availability and pop use separate mutex acquisitions; one serialized
|
|
main-thread consumer preserves ordering, but the API is not a multi-consumer queue.
|
|
- No proprietary liquid corpus, traversal/leak run, p95/p99 or paired capture is included.
|
|
|
|
## Source map
|
|
|
|
| Path | Responsibility |
|
|
|---|---|
|
|
| `src/render/liquid/adt_water_load_pipeline_state.gd` | Request/task/result bookkeeping and result mutex |
|
|
| `src/scenes/streaming/streaming_world_loader.gd` | Parse task execution, permits, stale checks and Node finalization |
|
|
| `addons/mpq_extractor/loaders/adt_builder.gd` | ADT liquid geometry/material construction |
|
|
| `src/tools/verify_adt_water_load_pipeline_state.gd` | Lifecycle/thread/boundary/timing regression |
|
|
|
|
## Related decisions and references
|
|
|
|
- [`world-renderer.md`](world-renderer.md)
|
|
- [`render-budget-scheduler.md`](render-budget-scheduler.md)
|
|
- [`../../RENDER.md`](../../RENDER.md)
|
|
- [`../../targets/roadmap/02-rendering-and-graphics.md`](../../targets/roadmap/02-rendering-and-graphics.md)
|