refactor(M03): extract ADT water load pipeline state

2026-07-17 09:38:24 +04:00
parent 7973945582
commit b7c036f14d
9 changed files with 824 additions and 53 deletions
@@ -0,0 +1,247 @@
+# ADT Water Load Pipeline State
+
+## Metadata
+
+| Field | Value |
+|---|---|
+| Status | Implemented |
+| Target/work package | M03 / `M03-RND-ADT-WATER-LOAD-PIPELINE-001` |
+| Owners | ADT water pending FIFO/dedupe, active task IDs and worker-safe result mailbox |
+| Last verified | Worktree `work/sindo-main-codex/m03-adt-water-load-pipeline`, 2026-07-17 |
+| Profiles/capabilities | ADT MH2O/MCLQ water loading; profile-independent state |
+
+## Purpose
+
+Own the cross-frame and cross-thread bookkeeping for asynchronous ADT water
+loads outside `StreamingWorldLoader`: FIFO requests, queued-key deduplication,
+opaque active task IDs and parsed-result delivery. The loader retains task
+execution, parsing, budgets, stale-result decisions and scene finalization.
+
+## Non-goals
+
+- Start, cancel or wait WorkerThreadPool tasks.
+- Instantiate `ADTLoader` or interpret MH2O/MCLQ data.
+- Select concurrency limits or consume render permits.
+- Create, attach or free Nodes, Meshes, materials or RIDs.
+- Change liquid visuals, coordinates, tile acceptance or loaded-state timing.
+
+## Context and boundaries
+
+```mermaid
+flowchart LR
+    Tile[Loaded tile needing water] --> Loader[StreamingWorldLoader]
+    Loader --> State[AdtWaterLoadPipelineState]
+    State --> Request[Oldest tile/path request]
+    Request --> Loader
+    Loader --> Worker[WorkerThreadPool ADTLoader task]
+    Worker --> State
+    State --> Result[Parsed ADT result]
+    Result --> Loader
+    Loader --> Budget[WATER_FINALIZE permit]
+    Loader --> Builder[ADTBuilder water scene]
+    Builder --> Scene[Main-thread tile Node attach]
+```
+
+Allowed dependencies are RefCounted, Dictionary/Array/String and one Godot
+`Mutex` around results. WorkerThreadPool, ADTLoader, scheduler, tile state,
+scene-tree/render resources and gameplay/editor/network are forbidden.
+
+## Public API
+
+| Symbol | Kind | Purpose | Thread/lifetime | Errors |
+|---|---|---|---|---|
+| `enqueue_request(tile_key, adt_path)` | Command/query | Append deduplicated FIFO request | Renderer main thread; until dequeue/cancel/clear | Empty/queued/active returns false |
+| `take_next_request()` | Command/query | Pop oldest request and release queued marker | Renderer main thread | Empty returns `{}` |
+| `cancel_pending_requests(tile_key)` | Command/query | Remove all matching pending entries | Tile release/main thread | Empty/unknown removes zero; active unaffected |
+| `remember_active_task(tile_key, task_id)` | Command/query | Retain loader-created opaque task ID | Main thread; until completion/clear | Empty/duplicate returns false |
+| `has_active_task` / `active_task_id_for` | Query | Test/borrow active task identity | Main thread | Unknown ID is `-1` |
+| `complete_active_task(tile_key)` | Command/query | Remove and return completed task ID | Main thread after wait | Unknown returns `-1` |
+| `active_task_ids_snapshot()` | Query | Copy IDs for orderly shutdown waiting | Main thread | Detached Array |
+| `publish_result(tile_key, path, data)` | Command/query | Append parsed result under mutex | Worker or main thread | Empty key/path returns false |
+| `has_result()` / `pop_result()` | Query/command | Observe/pop oldest result under mutex | Main-thread drain | Empty pop returns `{}` |
+| `total_load_count()` | Query | Pending plus active metric | Main thread | Results intentionally excluded |
+| `pending_request_count()` / `active_task_count()` | Query | Support loader concurrency loop | Main thread | None |
+| `clear()` | Command | Release all bookkeeping and results | Main thread after wait or map reset | Does not interrupt workers |
+| `diagnostic_snapshot()` | Query | Return detached scalar requests/tasks/results | Main thread; result mailbox locked | Water data omitted |
+
+## Inputs and outputs
+
+| Direction | Contract/data | Producer | Consumer | Ownership | Thread/lifetime |
+|---|---|---|---|---|---|
+| Input | Tile key and ADT path | Loader tile state | Pending FIFO | Copied Strings | Until dequeue/cancel/clear |
+| Input | Opaque task ID | Loader WorkerThreadPool adapter | Active task map | Integer value | Until wait/completion/clear |
+| Input | Tile/path/parsed ADT Dictionary | Worker parser adapter | Result mailbox | Dictionary reference retained | Until pop/clear |
+| Output | Oldest request Dictionary | Pipeline state | Loader task-start adapter | Fresh record from FIFO | One start attempt |
+| Output | Active task ID snapshot | Pipeline state | Loader shutdown wait | Detached Array | Shutdown loop |
+| Output | Oldest result Dictionary | Pipeline state | Loader finalize adapter | Mailbox record transferred | One finalize attempt |
+| Output | Scalar diagnostics | Pipeline state | Verifier/future metrics | Detached records | Snapshot lifetime |
+
+Side effects are limited to collection mutation, reference retention and result
+mutex locking. Parsing, permits and scene mutation remain outside the module.
+
+## Data flow
+
+```mermaid
+flowchart TD
+    Enqueue[enqueue tile/path] --> Duplicate{Queued or active?}
+    Duplicate -->|yes| Reject[false; unchanged]
+    Duplicate -->|no| FIFO[Append FIFO and queued key]
+    FIFO --> Take[take next; release queued marker]
+    Take --> Validate[Loader checks tile/path]
+    Validate --> Start[Loader starts worker and remembers task ID]
+    Start --> Parse[Worker parses ADT]
+    Parse --> Publish[Mutex-protected publish result]
+    Publish --> Permit[Loader obtains WATER_FINALIZE permit]
+    Permit --> Pop[Pop oldest result]
+    Pop --> Wait[Loader waits task and completes active ID]
+    Wait --> Stale{Tile/path/root current?}
+    Stale -->|no| Discard[Discard result]
+    Stale -->|yes| Attach[Build and attach water scene]
+```
+
+## Lifecycle/state
+
+A tile request progresses through `Absent`, `Queued`, `Active` and
+`ResultReady`; the loader wait/completion returns bookkeeping to `Absent` before
+validating/attaching the result. Queued tile release cancels only `Queued`.
+Active work is not interrupted; a later stale result is drained and discarded.
+
+```mermaid
+stateDiagram-v2
+    [*] --> Absent
+    Absent --> Queued: accepted enqueue
+    Queued --> Absent: pending cancellation or invalid dequeue
+    Queued --> Active: loader starts task
+    Active --> ResultReady: worker publishes result
+    ResultReady --> Absent: loader waits and completes task
+    Queued --> Absent: clear
+    Active --> Absent: clear bookkeeping only
+    ResultReady --> Absent: clear
+```
+
+## Main sequence
+
+```mermaid
+sequenceDiagram
+    participant Loader as StreamingWorldLoader main thread
+    participant State as AdtWaterLoadPipelineState
+    participant Worker as WorkerThreadPool task
+    participant Budget as RenderBudgetScheduler
+    participant Builder as ADTBuilder
+    Loader->>State: enqueue_request(tile key, ADT path)
+    Loader->>State: take_next_request()
+    Loader->>Worker: add_task(parse binding)
+    Loader->>State: remember_active_task(key, task ID)
+    Worker->>Worker: ADTLoader.load_adt(path)
+    Worker->>State: publish_result(key, path, data)
+    Note over State: result mutex protects publication/pop
+    Loader->>Budget: try_consume_permit(WATER_FINALIZE)
+    Loader->>State: pop_result()
+    Loader->>Worker: wait_for_task_completion(task ID)
+    Loader->>State: complete_active_task(key)
+    Loader->>Loader: reject stale tile/path/root
+    Loader->>Builder: build_tile_water_scene(data, origin)
+    Loader->>Loader: attach Node and mark water_loaded
+```
+
+## Dependency diagram
+
+```mermaid
+flowchart TB
+    Loader[StreamingWorldLoader] --> State[AdtWaterLoadPipelineState]
+    Loader --> Scheduler[RenderBudgetScheduler]
+    Loader --> Worker[WorkerThreadPool / ADTLoader]
+    Loader --> Builder[ADTBuilder / SceneTree]
+    Worker -->|publish only| State
+    State -. no dependency .-> Scheduler
+    State -. no dependency .-> Builder
+```
+
+## Ownership, threading and resources
+
+- Main thread owns pending and active-task collection mutation.
+- Worker threads may call only `publish_result`; result observe/pop/clear also lock.
+- Parsed water Dictionary identity is retained until main-thread pop.
+- Loader owns WorkerThreadPool task lifetime and must wait before orderly shutdown clear.
+- Loader owns tile states, ADTBuilder, water Nodes and every render resource.
+- `clear` cannot cancel a worker; post-reset publication becomes a stale result
+  and is discarded because no active task/tile ownership remains.
+
+## Errors, cancellation and recovery
+
+| Failure | Detection | Behavior | Diagnostic | Recovery |
+|---|---|---|---|---|
+| Empty/duplicate request | Enqueue guards | Reject unchanged | Contract verifier | Correct caller state |
+| Tile removed while queued | Loader release | Remove matching pending requests | Source/lifecycle verifier | Reload can enqueue later |
+| Tile removed while active | Loader stale check after result | Wait task, remove active ID, discard result | Existing metrics | Reload can enqueue later |
+| Parser unavailable/empty | Worker publishes empty Dictionary | Loader marks `water_loaded` without Node | Existing behavior | Future tile reload |
+| Path changed | Loader compares result path | Discard after task completion | Source regression | Current tile can request again |
+| Shutdown | Loader snapshots/waits active IDs | Clear all bookkeeping/results | Shutdown/source verifier | New loader starts empty |
+
+## Configuration and capabilities
+
+| Setting/capability | Default | Profile | Runtime mutable | Effect |
+|---|---|---|---|---|
+| `enable_water` | Loader setting | Quality/custom | Yes | Gates enqueue/process, unchanged |
+| `max_concurrent_water_tasks` | Loader setting | Quality/custom | Yes | Bounds active tasks outside state |
+| `water_finalize_ops_per_tick` | Loader setting | Quality/custom | Yes | Bounds result pops via scheduler |
+
+## Persistence, cache and migration
+
+No state is serialized or cached. ADT source/cache formats and liquid material
+versions are unchanged; no migration or rebuild is required.
+
+## Diagnostics and observability
+
+- `total_load_count` preserves the prior pending-plus-active `water` metric.
+- Snapshot exposes FIFO tile/path, sorted active keys and result tile/path only.
+- Parsed liquid payloads, Nodes and task internals are not exposed.
+- No logs or correlation IDs are introduced.
+
+## Verification
+
+- `verify_adt_water_load_pipeline_state.gd`: invalid/dedupe/FIFO behavior,
+  active IDs, payload identity, real Thread publication, pending cancellation,
+  clear, detached diagnostics, loader boundaries and bounded timing.
+- Existing material/shutdown/facade/streaming regressions cover adjacent behavior.
+- Fidelity evidence is preservation of current scheduling/finalize transitions;
+  no MH2O/MCLQ visual or original-client parity claim is made.
+- Performance budget: 100 cycles of 256 request/task/result transitions under 1 second.
+
+## Extension points
+
+- ADT parse execution can later move behind a parser/job adapter without changing state.
+- Water scene materialization can be extracted separately because it owns Nodes
+  and main-thread effects rather than async bookkeeping.
+
+## Capability status
+
+| Capability | Status | Evidence | Gap/next step |
+|---|---|---|---|
+| ADT water load pipeline state | Implemented extraction | FIFO/task/thread/source/timing verifier | Asset-backed traversal/leak evidence pending |
+| ADT liquid parsing | Existing/loader-owned adapter | Native ADTLoader and dry-run paths | MH2O/MCLQ fidelity fixtures pending |
+| Water scene finalization | Existing/loader-owned | Scheduler/material/shutdown regressions | Separate materialization extraction possible |
+
+## Known gaps and risks
+
+- `clear` intentionally does not interrupt active WorkerThreadPool work.
+- Result payload Dictionaries are retained by reference and must not be mutated after publication.
+- Result availability and pop use separate mutex acquisitions; one serialized
+  main-thread consumer preserves ordering, but the API is not a multi-consumer queue.
+- No proprietary liquid corpus, traversal/leak run, p95/p99 or paired capture is included.
+
+## Source map
+
+| Path | Responsibility |
+|---|---|
+| `src/render/liquid/adt_water_load_pipeline_state.gd` | Request/task/result bookkeeping and result mutex |
+| `src/scenes/streaming/streaming_world_loader.gd` | Parse task execution, permits, stale checks and Node finalization |
+| `addons/mpq_extractor/loaders/adt_builder.gd` | ADT liquid geometry/material construction |
+| `src/tools/verify_adt_water_load_pipeline_state.gd` | Lifecycle/thread/boundary/timing regression |
+
+## Related decisions and references
+
+- [`world-renderer.md`](world-renderer.md)
+- [`render-budget-scheduler.md`](render-budget-scheduler.md)
+- [`../../RENDER.md`](../../RENDER.md)
+- [`../../targets/roadmap/02-rendering-and-graphics.md`](../../targets/roadmap/02-rendering-and-graphics.md)