TRANSIT/quic_ecs_dt
1
0
Fork 0
Code Issues Pull Requests Activity

Flip T3 to substrate-initiated actuator commands

Browse Source
This commit is contained in:
Valère Plantevin
2026-05-13 15:03:23 -04:00
parent 272d3b3c59
commit baa075fe0f
22 changed files with 1003 additions and 749 deletions
Download Patch File Download Diff File Expand all files Collapse all files

51
CLAUDE.md
View File

@@ -11,17 +11,19 @@ Source repo for **"QUIC + ECS as Complementary Transport and Runtime Substrates
## Architecture
Three-tier QUIC ↔ ECS bridge, headless Bevy runtime:
Three-tier QUIC ↔ ECS bridge, headless Bevy runtime. **T1/T2 are inbound (device → substrate); T3 is outbound (substrate → device, actuator commands):**
| Tier | QUIC primitive | Use case | Channel cap | Tx newtype |
|------|----------------|----------|-------------|------------|
| T1 | Unreliable datagrams (RFC 9221) | High-freq ephemeral telemetry; drops OK | 1024 | `T1Sender::send_lossy` (try_send, drop on full) |
| T2 | Unidirectional streams | Ordered threshold events; reliable | 512 | `T2Sender::send` (await, backpressure) |
| T3 | Bidirectional streams | Actuator commands w/ ACK; per-command oneshot reply | 256 | `T3Sender::send` of `T3Inbound { command, reply }` |
| Tier | QUIC primitive | Direction | Use case | Channel cap | Sender |
|------|----------------|-----------|----------|-------------|--------|
| T1 | Unreliable datagrams (RFC 9221) | device → substrate | High-freq ephemeral telemetry; drops OK | 1024 | `T1Sender::send_lossy` (try_send, drop on full) |
| T2 | Unidirectional streams | device → substrate | Ordered threshold events; reliable | 512 | `T2Sender::send` (await, backpressure) |
| T3 | Bidirectional streams | **substrate → device** | Actuator commands w/ ACK | 256 | `T3OutboundSender::try_send` of `OutboundT3 { target_device, sensor_id, raw_value, sensor_type }` |
QUIC server runs on a dedicated OS thread with a Tokio multi-thread runtime; pushes decoded `QuicMessage` (UUID + sensor_id + f64 + ts + seq, 38 B fixed LE) into `tokio::sync::mpsc` per tier via the `T1Sender / T2Sender / T3Sender` newtypes (in [substrate/src/transport/mod.rs](substrate/src/transport/mod.rs)) so misuse is a type error. Bevy `ingest_system` drains in `PreUpdate`, gated by `run_if(in_state(ServerState::Started))`. Pattern is in [substrate/src/transport/ecs.rs](substrate/src/transport/ecs.rs).
QUIC server runs on a dedicated OS thread with a Tokio multi-thread runtime. T1/T2 decoded `QuicMessage`s (39 B fixed LE: UUID + sensor_id + f64 + ts + seq + sensor_type) flow into per-tier `tokio::sync::mpsc` channels and are drained by Bevy's `ingest_system` in `PreUpdate`, gated by `run_if(in_state(ServerState::Started))`. T3 flows the other way: `automation_system` constructs `OutboundT3` items and the tokio-side `drain_outbound_t3` task opens bi-streams to the target device. The per-tier sender newtypes (in [substrate/src/transport/mod.rs](substrate/src/transport/mod.rs)) make tier mixups a type error. Pattern is in [substrate/src/transport/ecs.rs](substrate/src/transport/ecs.rs).
**T3 ack protocol.** A device opens a bi-stream and writes one `QuicMessage` (the command). The demux task reads it, builds a `T3Inbound { command, reply: oneshot::Sender<QuicMessage> }`, and sends it on the T3 mpsc. The ECS handler writes the ack into `reply`; the demux task awaits `reply_rx` and writes the resulting `QuicMessage` back on the bi-stream. Dropping the oneshot signals "no handler" and propagates as a stream close — used by the placeholder ingest until M4 installs real handlers.
**T3 actuator-command protocol.** The substrate's `automation_system` decides to actuate (e.g. Presence < 1.0 ⇒ Relay = stop) and pushes an `OutboundT3` onto the outbound channel. The tokio drain task pops it, looks up the target device's `quinn::Connection` in a `ConnectionRegistry` (populated by `read_datagrams` / `read_one_uni_stream` on first sight of each device UUID), then **spawns one task per command** to do `conn.open_bi() → write 39 B → finish → read 39 B ack`. Per-task spawning means a single stuck `read_exact` can't stall the pipeline. Latency from `open_bi()` to ack-receipt is recorded as `substrate_latency_us{tier="t3"}` and a successful ack increments `substrate_received_total{tier="t3"}`. Misses (`substrate_t3_outbound_no_route_total`), drops (`substrate_t3_outbound_dropped_total`), and bi-stream errors (`substrate_t3_outbound_errors_total`) each have their own counter.
**Connection registry.** `Arc<std::sync::RwLock<HashMap<Uuid, quinn::Connection>>>`. `quinn::Connection` is internally `Arc`; one simulator process commonly hosts 7 device UUIDs sharing one connection. Registry insert is idempotent (`ensure_registered`). On `conn.closed().await` returning, `handle_incoming` purges every key whose `Connection::stable_id()` matches the closed connection.
**Target hardware:** CM5 (BCM2712, Cortex-A76, 4 GB) as DT runtime; M4 Max as traffic generator; 1 Gbps direct Ethernet. Both rigs are in hand.
@@ -48,24 +50,26 @@ quic_ecs_dt/
| Area | State |
|------|-------|
| `AppConfig` figment loader (defaults → TOML → env) | Done — [substrate/src/config.rs:42](substrate/src/config.rs#L42) |
| 3-tier MPSC bridge scaffolding (Tokio thread + Bevy plugin) | Done — [substrate/src/transport/ecs.rs](substrate/src/transport/ecs.rs) |
| `QuicMessage` struct (no codec yet) | Defined — [substrate/src/transport/mod.rs:4](substrate/src/transport/mod.rs#L4) |
| Quinn server lifecycle | Listener up — `ServerState{Starting,Started}` in [substrate/src/transport/state.rs](substrate/src/transport/state.rs); `OnEnter(Starting)` → bind + accept loop in [substrate/src/transport/ecs.rs](substrate/src/transport/ecs.rs). Explicit `TransportConfig` w/ tuned datagram recv buffer (256 KiB) in [substrate/src/transport/server.rs](substrate/src/transport/server.rs). Per-tier sender newtypes (`T1Sender::send_lossy`, `T2Sender::send`, `T3Sender::send`) in [substrate/src/transport/mod.rs](substrate/src/transport/mod.rs) |
| T1 demux (datagrams → ECS) | Done — `handle_incoming` orchestrator + `read_datagrams` reader in [substrate/src/transport/server.rs](substrate/src/transport/server.rs); decode errors logged but non-fatal; channel-full drops silent at trace; received/dropped/decode_errors counters in the end-of-stream debug line |
| T2 demux (uni streams → ECS) | Done — `read_uni_streams` accepts streams in [substrate/src/transport/server.rs](substrate/src/transport/server.rs), spawns one task per stream that reads 38 B chunks until EOF; decode failure resets the stream via `recv.stop(0)` (one bad stream doesn't kill the connection); `t2.send().await` honours backpressure |
| T3 demux (bi streams ↔ ECS) | Done — `accept_bi_streams` + `read_one_bi_stream` in [substrate/src/transport/server.rs](substrate/src/transport/server.rs); reads 38 B command, ships `T3Inbound { command, reply: oneshot::Sender }` to the ECS, awaits the reply, writes 38 B ack and finishes. If the ECS drops the oneshot (no handler installed yet — the M4 placeholder) `send.reset(0)` gives the client a clean signal instead of a half-open stream. `handle_incoming` joins all three readers on close |
| `AppConfig` figment loader (defaults → TOML → env, `__` split) | Done — [substrate/src/config.rs](substrate/src/config.rs) |
| Inbound bridge scaffolding (Tokio thread + Bevy plugin) | Done — [substrate/src/transport/ecs.rs](substrate/src/transport/ecs.rs) |
| `QuicMessage` struct + 39 B LE codec | Done — [substrate/src/transport/mod.rs](substrate/src/transport/mod.rs); 5 unit tests passing |
| Quinn server lifecycle | Listener up — `ServerState{Starting,Started}` in [substrate/src/transport/state.rs](substrate/src/transport/state.rs); `OnEnter(Starting)` → bind + accept loop in [substrate/src/transport/ecs.rs](substrate/src/transport/ecs.rs). Explicit `TransportConfig` w/ tuned datagram recv buffer (256 KiB) in [substrate/src/transport/server.rs](substrate/src/transport/server.rs). Per-tier sender newtypes (`T1Sender::send_lossy`, `T2Sender::send`, `T3OutboundSender::try_send`) in [substrate/src/transport/mod.rs](substrate/src/transport/mod.rs) |
| T1 demux (datagrams → ECS) | Done — `handle_incoming` orchestrator + `read_datagrams` reader in [substrate/src/transport/server.rs](substrate/src/transport/server.rs); decode errors logged but non-fatal; channel-full drops silent at trace; received/dropped/decode_errors counters in the end-of-stream debug line. Calls `ensure_registered` on first decode so outbound T3 can route to this device |
| T2 demux (uni streams → ECS) | Done — `read_uni_streams` accepts streams in [substrate/src/transport/server.rs](substrate/src/transport/server.rs), spawns one task per stream that reads 39 B chunks until EOF; decode failure resets the stream via `recv.stop(0)` (one bad stream doesn't kill the connection); `t2.send().await` honours backpressure; first decode also calls `ensure_registered` |
| T3 outbound (ECS → device, substrate-initiated) | Done — `drain_outbound_t3` task in [substrate/src/transport/server.rs](substrate/src/transport/server.rs) pops `OutboundT3` items, looks up the target device's `Connection` in `ConnectionRegistry`, **spawns one task per command** to do `open_bi → write 39 B → finish → read ack`. Per-task spawning ensures one stuck ack can't stall the pipeline. Records `substrate_latency_us{tier="t3"}` on success; counts no-route, dropped, and error cases separately. The old simulator-initiated T3 inbound path (`T3Sender` / `T3Inbound` / `accept_bi_streams`) is **gone** as of this refactor |
| Connection registry (Uuid → Connection) | Done — `Arc<RwLock<HashMap<Uuid, quinn::Connection>>>` populated by readers; purged in `handle_incoming` after `conn.closed().await` using `Connection::stable_id()`. Constructor `new_connection_registry`; idempotent insert via `ensure_registered` |
| TLS / self-signed cert | Done (M1) — `certs/server.{crt,key}` via `make certs`, gitignored. PEM loader in [substrate/src/transport/server.rs:15](substrate/src/transport/server.rs#L15); rustls `aws-lc-rs` default provider installed in [substrate/src/main.rs](substrate/src/main.rs) |
| Wire codec for `QuicMessage` (39 B fixed LE, incl. `sensor_type: u8`) | Done — [substrate/src/transport/mod.rs](substrate/src/transport/mod.rs); 5 unit tests passing. `SensorType` enum: `Generic / Temperature / Humidity / Pressure / Voltage / Current` |
| `tracing-subscriber` init w/ `RUST_LOG` | Done (M1) — [substrate/src/main.rs:8-12](substrate/src/main.rs#L8-L12) |
| ECS components (`RawSensorData`, `SmoothedValue`) + 4 systems (Ingest/Sim/Export/Diagnostics) | Done — entities = `(DeviceId, SensorId, SensorTypeTag, RawSensorData, SmoothedValue, Asset)` per (device, sensor); `SensorRegistry` upserts via `HashMap<(Uuid, u16), Entity>` in [substrate/src/world.rs](substrate/src/world.rs). `IngestSystem` drains all three tiers; T3 ack preserves command's `sensor_type` and returns the device's most recent `raw_value`. `SimulationSystem` maintains a 16-sample rolling mean per entity and emits `substrate_threshold_crossings_total{type, direction}` when the smoothed mean crosses a per-type threshold (`Changed<RawSensorData>` query so cost scales with ingress, not fleet size). `ExportSystem` samples `substrate_{entities,channel_depth,channel_capacity,rss_bytes}` + `sensor_aggregate{type, stat}` once per second. `Diagnostics` logs `tick_hz` once per second |
| Schedule rate-gating | Done (M4) — `MinimalPlugins.set(ScheduleRunnerPlugin::run_loop(1/tick_rate_hz))` in [substrate/src/main.rs](substrate/src/main.rs); replaces the default busy-loop with the configured period |
| Prometheus exporter + Grafana dashboards | Done (M5) — `ObservabilityPlugin` in [substrate/src/observability.rs](substrate/src/observability.rs) installs `metrics-exporter-prometheus` on the existing tokio runtime. **Runtime surface** (paper §Evaluation): counters `substrate_received_total{tier}`, `dropped_total{tier=t1}`, `decode_errors_total{tier}`, `t3_no_handler_total`; latency histograms `substrate_latency_us{tier}`; gauges `substrate_tick_hz`, `substrate_entities`, `substrate_channel_depth{tier}`, `substrate_channel_capacity{tier}`, `substrate_rss_bytes`. **Sensor data surface** (operator dashboard): per-type aggregates `sensor_aggregate{type, stat=count|mean|min|max}` computed once per second over the live world, cardinality bounded by `\|SensorType\| × 4` so it scales to thousands of sensors. Two dashboards: [dashboards/runtime.json](dashboards/runtime.json) and [dashboards/sensors.json](dashboards/sensors.json) (thermometer/gauge/stat panels per type) |
| Simulator (Quinn client + sensor generators) | `SimulatorClient` lib in [simulator/src/client.rs](simulator/src/client.rs) — connects, trusts the substrate's PEM cert via custom `ServerCertVerifier` (sidesteps `CaUsedAsEndEntity`); `send_datagram(QuicMessage)` for T1, `send_uni_stream(&[QuicMessage])` for T2, `request(&QuicMessage) -> QuicMessage` for T3. CLI driver in [simulator/src/main.rs](simulator/src/main.rs) with clap flags (`--addr`, `--rate-hz`, `--t2-rate-hz`, `--t3-rate-hz`, `--t3-timeout-ms`, `--count`, `--devices`, `--sensor-id`, `--sensor-type`, `--profile`, `--cert`, `--server-name`); parallel T1+T2+T3 emitters, per-(device,sensor) sequence counters, type-appropriate waveform generators (sin/cos curves centred on realistic sensor ranges), 1-Hz combined progress logs, Ctrl-C drain. `--profile industrial` fans out to 5 sensors per device (Temperature/Humidity/Pressure/Voltage/Current). Bevy-driven sensor generator still pending |
| End-to-end test harness | Six integration tests across [simulator/tests/end_to_end_t1.rs](simulator/tests/end_to_end_t1.rs), [simulator/tests/end_to_end_t2.rs](simulator/tests/end_to_end_t2.rs), [simulator/tests/end_to_end_t3.rs](simulator/tests/end_to_end_t3.rs): T1 single-datagram round-trip + 32-msg burst order; T2 single-stream order-preservation + 4-stream concurrent per-device ordering; T3 round-trip with fake-ECS handler + no-handler stream-reset. Each test calls `bind_endpoint` + `accept_loop` in-process with channels owned by the test |
| `config.toml` at repo root | Done (M1) — [config.toml](config.toml); loaded by [substrate/src/main.rs:9](substrate/src/main.rs#L9) |
| Benchmark harness (sweep + CSV writer) | Missing |
| CM5 cross-compile / deploy | Wired in [Makefile:30](Makefile#L30); not exercised |
| Simulator (Quinn client + sensor generators) | `SimulatorClient` lib in [simulator/src/client.rs](simulator/src/client.rs) — connects, trusts the substrate's PEM cert via custom `ServerCertVerifier` (sidesteps `CaUsedAsEndEntity`); `send_datagram(QuicMessage)` for T1, `send_uni_stream(&[QuicMessage])` for T2. `SimulatorClient::request` exists for ad-hoc tests but the binary no longer initiates T3. CLI driver in [simulator/src/main.rs](simulator/src/main.rs) with clap flags (`--addr`, `--rate-hz`, `--t2-rate-hz`, `--count`, `--devices`, `--sensor-id`, `--sensor-type`, `--profile`, `--cert`, `--server-name`). `--profile industrial` fans out to **7 sensors per device** (Temperature/Humidity/Pressure/Voltage/Current/Presence/Relay). T1/T2 emitters check `engine_running` per-tick — Voltage stays at ~230 V regardless; Current drops to ~0 when stopped. HTTP trigger on `:9002` (`POST /trigger`) pushes a Presence=0 reading via T2 for Grafana-driven demos |
| Simulator command receiver (substrate → device T3) | Done — `run_command_receiver` in [simulator/src/commands.rs](simulator/src/commands.rs) loops on `conn.accept_bi()`, decodes 39 B, sets `engine_running` from `raw_value` when `sensor_type == Relay`, writes 39 B ack. Spawned by `main.rs` post-connect. `new_engine_state()` constructor exported for integration tests |
| End-to-end test harness | 18 tests across [simulator/tests/end_to_end_t1.rs](simulator/tests/end_to_end_t1.rs), [simulator/tests/end_to_end_t2.rs](simulator/tests/end_to_end_t2.rs), [simulator/tests/end_to_end_full_loop.rs](simulator/tests/end_to_end_full_loop.rs): T1 single-datagram + 32-msg burst order; T2 single-stream + 4-stream concurrent ordering; **full closed loop** (Presence < 1.0 → substrate T3 → simulator `engine_running` flips, then Presence > 1.0 → flips back). Plus codec + world unit tests including `automation_dispatches_relay_stop_when_presence_drops` |
| `config.toml` at repo root | Done — [config.toml](config.toml); loaded by [substrate/src/main.rs](substrate/src/main.rs); env override via `APP_*` with `__` split (`Env::prefixed("APP_").split("__")`) actually works now |
| Benchmark harness (sweep + CSV writer) | Done — [scripts/bench-loss.sh](scripts/bench-loss.sh) for entity×loss → `data/two_machine/final_table.csv`; [scripts/bench-scaling.sh](scripts/bench-scaling.sh) for T1 rate sweep with optional substrate-side synthetic T3 (`T3_RATE_HZ=100 ./scripts/bench-scaling.sh` enables `APP_NETWORK__SYNTHETIC_T3_RATE_HZ`) → `data/local/cross_tier.csv`. The synthetic driver lives in `accept_loop` and pushes through the same outbound channel `automation_system` uses |
| CM5 cross-compile / deploy | Wired in [Makefile:30](Makefile#L30); first trial run completed (commit `272d3b3`); [scripts/setup-cm5.sh](scripts/setup-cm5.sh) provisions the Pi |
`cargo run -p substrate` boots, prints the loaded config, and idles on the (still-empty) Quinn server. `MinimalPlugins` busy-loops the ECS schedule by default — expected, will gate to `tick_rate_hz` in M4.
@@ -101,11 +105,10 @@ Each milestone has one verification gate. Update Status here as we go.
## Known deferrals
- **Channel ownership is per-host, not per-connection.** All connections share the same three mpsc channels. Fairness under N-device load relies on tokio scheduling. Acceptable for the "one ECS world per host" model the paper describes; revisit if many-device benchmarks show starvation.
- **No graceful shutdown.** The `quic-runtime` thread is parked on `pending()`; spawned tasks (accept loop, per-conn demux) are orphaned at process exit. Fine for research runs; we'll need an `OnExit(Started)` (or a `Stopping` state) when M5 observability needs clean drain or M8 wants finalised CSV writes.
- **Channel ownership is per-host, not per-connection.** All connections share the same inbound mpsc channels and the same outbound T3 channel. Fairness under N-device load relies on tokio scheduling. Acceptable for the "one ECS world per host" model the paper describes; revisit if many-device benchmarks show starvation.
- **No graceful shutdown.** The `quic-runtime` thread is parked on `pending()`; spawned tasks (accept loop, per-conn demux, outbound drain, per-command T3 spawns) are orphaned at process exit. Fine for research runs.
- **Bind failure is fatal.** `OnEnter(Starting)` panics if `bind_endpoint` fails. A `ServerState::Failed` variant joins when we wire proper error surfacing.
- **T3 ack semantics are minimal.** The current handler echoes the device's most recent `raw_value` with a server timestamp — adequate for "read sensor" commands, not for actuator-write semantics. A future iteration may introduce an `ActuatorState` component and a setpoint-apply path; for now T3 is best framed as "reliable read/query RPC" in the paper.
- **T3 outbound concurrency is unbounded.** `drain_outbound_t3` spawns one task per command (so a stuck `read_exact` can't stall the pipeline). Under sustained T1 ingest beyond ~10k msg/s the per-command tasks queue behind the tokio scheduler and T3 P99 latency climbs into the hundreds of ms while throughput holds. If we need true latency isolation under load, add a `tokio::Semaphore` cap or a dedicated runtime/thread for T3.
- **Schedule rate-gating is approximate.** `ScheduleRunnerPlugin::run_loop(period)` honours `period` as a minimum; observed `tick_hz` runs ~85% of target on macOS dev (target 60 → ~50). Should be tighter on the CM5; revisit if M6 sweeps depend on a steady tick.
## Run / verify