sindoring/open-wc
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

refactor(M03): extract ADT water load pipeline state

Browse Source
This commit is contained in:
sindoring
2026-07-17 09:38:24 +04:00
parent 7973945582
commit b7c036f14d
9 changed files with 824 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/modules/README.md
+1
View File
@@ -15,6 +15,7 @@
| Terrain quality Mesh cache | Implemented extraction | [`terrain-quality-mesh-cache.md`](terrain-quality-mesh-cache.md) |
| Terrain chunk LOD planner | Implemented extraction | [`terrain-chunk-lod-planner.md`](terrain-chunk-lod-planner.md) |
| Terrain chunk geometry queue planner | Implemented extraction | [`terrain-chunk-geometry-queue-planner.md`](terrain-chunk-geometry-queue-planner.md) |
| ADT water load pipeline state | Implemented extraction | [`adt-water-load-pipeline-state.md`](adt-water-load-pipeline-state.md) |
| M2 unique placement registry | Implemented extraction | [`m2-unique-placement-registry.md`](m2-unique-placement-registry.md) |
| M2 placement transform resolver | Implemented | [`m2-placement-transform-resolver.md`](m2-placement-transform-resolver.md) |
| M2 placement grouper | Implemented extraction | [`m2-placement-grouper.md`](m2-placement-grouper.md) |
docs/modules/adt-water-load-pipeline-state.md
+247
View File
@@ -0,0 +1,247 @@
# ADT Water Load Pipeline State
## Metadata
| Field | Value |
|---|---|
| Status | Implemented |
| Target/work package | M03 / `M03-RND-ADT-WATER-LOAD-PIPELINE-001` |
| Owners | ADT water pending FIFO/dedupe, active task IDs and worker-safe result mailbox |
| Last verified | Worktree `work/sindo-main-codex/m03-adt-water-load-pipeline`, 2026-07-17 |
| Profiles/capabilities | ADT MH2O/MCLQ water loading; profile-independent state |
## Purpose
Own the cross-frame and cross-thread bookkeeping for asynchronous ADT water
loads outside `StreamingWorldLoader`: FIFO requests, queued-key deduplication,
opaque active task IDs and parsed-result delivery. The loader retains task
execution, parsing, budgets, stale-result decisions and scene finalization.
## Non-goals
- Start, cancel or wait WorkerThreadPool tasks.
- Instantiate `ADTLoader` or interpret MH2O/MCLQ data.
- Select concurrency limits or consume render permits.
- Create, attach or free Nodes, Meshes, materials or RIDs.
- Change liquid visuals, coordinates, tile acceptance or loaded-state timing.
## Context and boundaries
```mermaid
flowchart LR
Tile[Loaded tile needing water] --> Loader[StreamingWorldLoader]
Loader --> State[AdtWaterLoadPipelineState]
State --> Request[Oldest tile/path request]
Request --> Loader
Loader --> Worker[WorkerThreadPool ADTLoader task]
Worker --> State
State --> Result[Parsed ADT result]
Result --> Loader
Loader --> Budget[WATER_FINALIZE permit]
Loader --> Builder[ADTBuilder water scene]
Builder --> Scene[Main-thread tile Node attach]
```
Allowed dependencies are RefCounted, Dictionary/Array/String and one Godot
`Mutex` around results. WorkerThreadPool, ADTLoader, scheduler, tile state,
scene-tree/render resources and gameplay/editor/network are forbidden.
## Public API
| Symbol | Kind | Purpose | Thread/lifetime | Errors |
|---|---|---|---|---|
| `enqueue_request(tile_key, adt_path)` | Command/query | Append deduplicated FIFO request | Renderer main thread; until dequeue/cancel/clear | Empty/queued/active returns false |
| `take_next_request()` | Command/query | Pop oldest request and release queued marker | Renderer main thread | Empty returns `{}` |
| `cancel_pending_requests(tile_key)` | Command/query | Remove all matching pending entries | Tile release/main thread | Empty/unknown removes zero; active unaffected |
| `remember_active_task(tile_key, task_id)` | Command/query | Retain loader-created opaque task ID | Main thread; until completion/clear | Empty/duplicate returns false |
| `has_active_task` / `active_task_id_for` | Query | Test/borrow active task identity | Main thread | Unknown ID is `-1` |
| `complete_active_task(tile_key)` | Command/query | Remove and return completed task ID | Main thread after wait | Unknown returns `-1` |
| `active_task_ids_snapshot()` | Query | Copy IDs for orderly shutdown waiting | Main thread | Detached Array |
| `publish_result(tile_key, path, data)` | Command/query | Append parsed result under mutex | Worker or main thread | Empty key/path returns false |
| `has_result()` / `pop_result()` | Query/command | Observe/pop oldest result under mutex | Main-thread drain | Empty pop returns `{}` |
| `total_load_count()` | Query | Pending plus active metric | Main thread | Results intentionally excluded |
| `pending_request_count()` / `active_task_count()` | Query | Support loader concurrency loop | Main thread | None |
| `clear()` | Command | Release all bookkeeping and results | Main thread after wait or map reset | Does not interrupt workers |
| `diagnostic_snapshot()` | Query | Return detached scalar requests/tasks/results | Main thread; result mailbox locked | Water data omitted |
## Inputs and outputs
| Direction | Contract/data | Producer | Consumer | Ownership | Thread/lifetime |
|---|---|---|---|---|---|
| Input | Tile key and ADT path | Loader tile state | Pending FIFO | Copied Strings | Until dequeue/cancel/clear |
| Input | Opaque task ID | Loader WorkerThreadPool adapter | Active task map | Integer value | Until wait/completion/clear |
| Input | Tile/path/parsed ADT Dictionary | Worker parser adapter | Result mailbox | Dictionary reference retained | Until pop/clear |
| Output | Oldest request Dictionary | Pipeline state | Loader task-start adapter | Fresh record from FIFO | One start attempt |
| Output | Active task ID snapshot | Pipeline state | Loader shutdown wait | Detached Array | Shutdown loop |
| Output | Oldest result Dictionary | Pipeline state | Loader finalize adapter | Mailbox record transferred | One finalize attempt |
| Output | Scalar diagnostics | Pipeline state | Verifier/future metrics | Detached records | Snapshot lifetime |
Side effects are limited to collection mutation, reference retention and result
mutex locking. Parsing, permits and scene mutation remain outside the module.
## Data flow
```mermaid
flowchart TD
Enqueue[enqueue tile/path] --> Duplicate{Queued or active?}
Duplicate -->|yes| Reject[false; unchanged]
Duplicate -->|no| FIFO[Append FIFO and queued key]
FIFO --> Take[take next; release queued marker]
Take --> Validate[Loader checks tile/path]
Validate --> Start[Loader starts worker and remembers task ID]
Start --> Parse[Worker parses ADT]
Parse --> Publish[Mutex-protected publish result]
Publish --> Permit[Loader obtains WATER_FINALIZE permit]
Permit --> Pop[Pop oldest result]
Pop --> Wait[Loader waits task and completes active ID]
Wait --> Stale{Tile/path/root current?}
Stale -->|no| Discard[Discard result]
Stale -->|yes| Attach[Build and attach water scene]
```
## Lifecycle/state
A tile request progresses through `Absent`, `Queued`, `Active` and
`ResultReady`; the loader wait/completion returns bookkeeping to `Absent` before
validating/attaching the result. Queued tile release cancels only `Queued`.
Active work is not interrupted; a later stale result is drained and discarded.
```mermaid
stateDiagram-v2
[*] --> Absent
Absent --> Queued: accepted enqueue
Queued --> Absent: pending cancellation or invalid dequeue
Queued --> Active: loader starts task
Active --> ResultReady: worker publishes result
ResultReady --> Absent: loader waits and completes task
Queued --> Absent: clear
Active --> Absent: clear bookkeeping only
ResultReady --> Absent: clear
```
## Main sequence
```mermaid
sequenceDiagram
participant Loader as StreamingWorldLoader main thread
participant State as AdtWaterLoadPipelineState
participant Worker as WorkerThreadPool task
participant Budget as RenderBudgetScheduler
participant Builder as ADTBuilder
Loader->>State: enqueue_request(tile key, ADT path)
Loader->>State: take_next_request()
Loader->>Worker: add_task(parse binding)
Loader->>State: remember_active_task(key, task ID)
Worker->>Worker: ADTLoader.load_adt(path)
Worker->>State: publish_result(key, path, data)
Note over State: result mutex protects publication/pop
Loader->>Budget: try_consume_permit(WATER_FINALIZE)
Loader->>State: pop_result()
Loader->>Worker: wait_for_task_completion(task ID)
Loader->>State: complete_active_task(key)
Loader->>Loader: reject stale tile/path/root
Loader->>Builder: build_tile_water_scene(data, origin)
Loader->>Loader: attach Node and mark water_loaded
```
## Dependency diagram
```mermaid
flowchart TB
Loader[StreamingWorldLoader] --> State[AdtWaterLoadPipelineState]
Loader --> Scheduler[RenderBudgetScheduler]
Loader --> Worker[WorkerThreadPool / ADTLoader]
Loader --> Builder[ADTBuilder / SceneTree]
Worker -->|publish only| State
State -. no dependency .-> Scheduler
State -. no dependency .-> Builder
```
## Ownership, threading and resources
- Main thread owns pending and active-task collection mutation.
- Worker threads may call only `publish_result`; result observe/pop/clear also lock.
- Parsed water Dictionary identity is retained until main-thread pop.
- Loader owns WorkerThreadPool task lifetime and must wait before orderly shutdown clear.
- Loader owns tile states, ADTBuilder, water Nodes and every render resource.
- `clear` cannot cancel a worker; post-reset publication becomes a stale result
and is discarded because no active task/tile ownership remains.
## Errors, cancellation and recovery
| Failure | Detection | Behavior | Diagnostic | Recovery |
|---|---|---|---|---|
| Empty/duplicate request | Enqueue guards | Reject unchanged | Contract verifier | Correct caller state |
| Tile removed while queued | Loader release | Remove matching pending requests | Source/lifecycle verifier | Reload can enqueue later |
| Tile removed while active | Loader stale check after result | Wait task, remove active ID, discard result | Existing metrics | Reload can enqueue later |
| Parser unavailable/empty | Worker publishes empty Dictionary | Loader marks `water_loaded` without Node | Existing behavior | Future tile reload |
| Path changed | Loader compares result path | Discard after task completion | Source regression | Current tile can request again |
| Shutdown | Loader snapshots/waits active IDs | Clear all bookkeeping/results | Shutdown/source verifier | New loader starts empty |
## Configuration and capabilities
| Setting/capability | Default | Profile | Runtime mutable | Effect |
|---|---|---|---|---|
| `enable_water` | Loader setting | Quality/custom | Yes | Gates enqueue/process, unchanged |
| `max_concurrent_water_tasks` | Loader setting | Quality/custom | Yes | Bounds active tasks outside state |
| `water_finalize_ops_per_tick` | Loader setting | Quality/custom | Yes | Bounds result pops via scheduler |
## Persistence, cache and migration
No state is serialized or cached. ADT source/cache formats and liquid material
versions are unchanged; no migration or rebuild is required.
## Diagnostics and observability
- `total_load_count` preserves the prior pending-plus-active `water` metric.
- Snapshot exposes FIFO tile/path, sorted active keys and result tile/path only.
- Parsed liquid payloads, Nodes and task internals are not exposed.
- No logs or correlation IDs are introduced.
## Verification
- `verify_adt_water_load_pipeline_state.gd`: invalid/dedupe/FIFO behavior,
active IDs, payload identity, real Thread publication, pending cancellation,
clear, detached diagnostics, loader boundaries and bounded timing.
- Existing material/shutdown/facade/streaming regressions cover adjacent behavior.
- Fidelity evidence is preservation of current scheduling/finalize transitions;
no MH2O/MCLQ visual or original-client parity claim is made.
- Performance budget: 100 cycles of 256 request/task/result transitions under 1 second.
## Extension points
- ADT parse execution can later move behind a parser/job adapter without changing state.
- Water scene materialization can be extracted separately because it owns Nodes
and main-thread effects rather than async bookkeeping.
## Capability status
| Capability | Status | Evidence | Gap/next step |
|---|---|---|---|
| ADT water load pipeline state | Implemented extraction | FIFO/task/thread/source/timing verifier | Asset-backed traversal/leak evidence pending |
| ADT liquid parsing | Existing/loader-owned adapter | Native ADTLoader and dry-run paths | MH2O/MCLQ fidelity fixtures pending |
| Water scene finalization | Existing/loader-owned | Scheduler/material/shutdown regressions | Separate materialization extraction possible |
## Known gaps and risks
- `clear` intentionally does not interrupt active WorkerThreadPool work.
- Result payload Dictionaries are retained by reference and must not be mutated after publication.
- Result availability and pop use separate mutex acquisitions; one serialized
main-thread consumer preserves ordering, but the API is not a multi-consumer queue.
- No proprietary liquid corpus, traversal/leak run, p95/p99 or paired capture is included.
## Source map
| Path | Responsibility |
|---|---|
| `src/render/liquid/adt_water_load_pipeline_state.gd` | Request/task/result bookkeeping and result mutex |
| `src/scenes/streaming/streaming_world_loader.gd` | Parse task execution, permits, stale checks and Node finalization |
| `addons/mpq_extractor/loaders/adt_builder.gd` | ADT liquid geometry/material construction |
| `src/tools/verify_adt_water_load_pipeline_state.gd` | Lifecycle/thread/boundary/timing regression |
## Related decisions and references
- [`world-renderer.md`](world-renderer.md)
- [`render-budget-scheduler.md`](render-budget-scheduler.md)
- [`../../RENDER.md`](../../RENDER.md)
- [`../../targets/roadmap/02-rendering-and-graphics.md`](../../targets/roadmap/02-rendering-and-graphics.md)
docs/modules/world-renderer.md
+13 -1
View File
@@ -7,7 +7,7 @@
| Status | Partial |
| Target/work package | M00 baseline; `M01-RND-STREAMING-FOCUS-001`; `M01-QAR-SERVER-SPAWN-RENDERER-001`; M03 facade/planner/scheduler/internal-access/ground/environment/entity packages; M03 terrain packages; M03 M2 packages; M03 WMO placement package |
| Owners | Renderer workstream / milestone integrator |
| Last verified | Worktree `work/sindo-main-codex/m03-wmo-scene-resource-cache`, 2026-07-17 |
| Last verified | Worktree `work/sindo-main-codex/m03-adt-water-load-pipeline`, 2026-07-17 |
| Profiles/capabilities | `Performance`, `Balanced`, `High`, `Custom`; Blizzlike fidelity incomplete |
## Purpose
@@ -138,6 +138,7 @@ from externally reading/writing loader-private queue, task, cache and tile-state
| `WmoRenderBuildQueue` / `WmoRenderBuildJob` | Internal WMO pending-state service | Owns typed root/resource/cursor jobs and FIFO placement keys | Renderer main thread; map session | Invalid enqueue rejected; duplicate/stale behavior preserved |
| `WmoRenderResourceCacheState` | Internal WMO cache-state service | Owns validated Resources, negative entries and pending cache paths | Renderer main thread; map/cache session | Invalid/occupied request and unknown completion are rejected |
| `WmoSceneResourceCacheState` | Internal WMO cache-state service | Owns validated PackedScenes, negative entries and pending `.tscn` paths | Renderer main thread; map/cache session | Direct missing and terminal request transitions remain distinct |
| `AdtWaterLoadPipelineState` | Internal liquid async-state service | Owns ADT water FIFO/dedupe, active task IDs and mutex result mailbox | Main-thread state; worker result publication | Invalid/duplicate requests rejected; clear does not interrupt workers |
Публичным contract не считаются `StreamingWorldLoader` methods/properties,
внутренние dictionaries, queues, job records и generated node names. Scene-owned
@@ -172,6 +173,7 @@ loader configuration remains transitional composition data, not a caller API.
| Internal WMO pending build | Placement key, Node3D root, WMO Resource and cursors | Loader / `WmoRenderBuildQueue` | Loader drain and step planner adapter | Queue-owned job and strong references | Until cancel/clear/replacement |
| Internal WMO render cache | Normalized path, cache path and validated Resource | Loader / `WmoRenderResourceCacheState` | Loader lookup, ResourceLoader poll and build queue | State-owned Resource/path references; detached request snapshots | Until transient/full clear |
| Internal WMO scene cache | Normalized path, `.tscn` path and validated PackedScene | Loader / `WmoSceneResourceCacheState` | Loader lookup, request poll and scene instantiation | State-owned PackedScene/path references; detached request snapshots | Until transient/full clear |
| Internal ADT water load | Tile key, ADT path, task ID and parsed Dictionary | Loader/worker / `AdtWaterLoadPipelineState` | Loader task start, budgeted drain and finalization | State-owned records; mutex result mailbox | Request through result completion/reset |
| Output | Desired tile/detail operations | Streamer plan application | Finalize queues | Loader-owned | Cross-frame |
| Output | Terrain/M2/WMO/liquid instances | Loader/builders | Godot world/renderer | Loader/world owner | Main-thread attach |
| Output | Detached renderer metrics | `WorldRenderFacade` | Capture/baseline tools | Caller-owned deep copy | On demand |
@@ -356,6 +358,10 @@ sequenceDiagram
PackedScenes, negative entries and pending `.tscn` paths. The loader retains
ResourceLoader/FileAccess I/O, size and cache-version validation, live fallback,
materialization, permits, validity reactions and every Node lifecycle action.
- `AdtWaterLoadPipelineState` owns pending request order/deduplication, opaque
active task IDs and the worker-safe parsed-result mailbox. The loader retains
WorkerThreadPool start/wait, ADTLoader parsing, concurrency/finalize permits,
stale tile/path/root checks and ADTBuilder/Node finalization.
- Rendered-ground query results and diagnostic snapshots are caller-owned values;
the facade never returns Mesh, Node, tile-state or queue references.
- Loaded-mesh ground sampling is renderer diagnostics, not authoritative gameplay
@@ -461,6 +467,9 @@ Exact exported settings and cache versions remain documented in [`../../RENDER.m
- WMO scene Resource cache contract: direct missing sources, pending terminal
transitions, transient/full reset, detached diagnostics, loader-owned size/
metadata validation and bounded timing.
- ADT water load pipeline contract: FIFO/dedupe, active task identity, real
worker-thread publication, pending-only cancellation, result payload/order,
clear/source boundaries and bounded timing.
- Budget scheduler contract: exact lane exhaustion, shared chunk removal/create
priority, independent lanes, frame reset, invalid limits, terminal cancellation,
dependency boundaries and bounded permit timing without loading a world scene.
@@ -501,6 +510,7 @@ Exact exported settings and cache versions remain documented in [`../../RENDER.m
| WMO render Resource cache state | Implemented extraction | Scene-free lifecycle/exclusivity/source/timing plus shutdown contract | ResourceLoader I/O and asset-backed traversal/leak evidence pending |
| WMO scene Resource cache state | Implemented extraction | Scene-free lifecycle/direct-missing/source/timing plus shutdown contract | ResourceLoader/live-fallback extraction and asset-backed traversal/leak evidence pending |
| WMO rendering | Partial | Cached group rendering | Portals/rooms/material parity |
| ADT water load pipeline state | Implemented extraction | Scene-free FIFO/task/thread/source/timing contract | Parse/finalization and asset-backed traversal/leak evidence pending |
| Liquids | Partial | MH2O/MLIQ paths | LiquidType/depth/shore fidelity |
| Sky/light | Partial | DBC controller/logs | Zone transition paired validation |
| Character presentation | Prototype | Character creator/player experiment | Full customization/equipment states |
@@ -550,6 +560,7 @@ Exact exported settings and cache versions remain documented in [`../../RENDER.m
| `src/render/entities/world_entity_presenter.gd` | Main-thread entity visual subtree ownership and lifecycle |
| `src/render/terrain/rendered_ground_sample.gd` | Immutable renderer-owned available/unavailable ground query result |
| `src/render/terrain/terrain_quality_mesh_cache.gd` | Full-quality terrain revisit Mesh retention and LRU ownership |
| `src/render/liquid/adt_water_load_pipeline_state.gd` | ADT water request/task/result bookkeeping and worker-safe mailbox |
| `src/render/streaming/streaming_target_planner.gd` | Scene-free wanted/retained ADT target calculation |
| `src/render/streaming/streaming_target_policy.gd` | Immutable renderer radius/prefetch policy |
| `src/render/streaming/streaming_target_plan.gd` | Immutable planner result with read-only tile-key sets |
@@ -562,6 +573,7 @@ Exact exported settings and cache versions remain documented in [`../../RENDER.m
| `src/tools/verify_world_entity_presentation.gd` | Entity snapshot, create/update/replace/remove and ownership regression |
| `src/tools/verify_rendered_ground_sample.gd` | Cold-start renderer ground result invariant regression |
| `src/tools/verify_terrain_quality_mesh_cache.gd` | Terrain cache admission, LRU, ownership and loader-boundary regression |
| `src/tools/verify_adt_water_load_pipeline_state.gd` | ADT water FIFO/task/thread/boundary/timing regression |
| `src/tools/verify_streaming_target_planner.gd` | Planner behavior, dependency and bounded timing regression |
| `src/tools/verify_render_budget_scheduler.gd` | Scheduler bounds, shared-lane priority, cancellation and timing regression |
| `src/tools/verify_renderer_internal_access.gd` | Gameplay/EditorPlugin/registered renderer-tool boundary gate derived from private streamer fields |