# LedGrab TODO ## HTTP polling automation trigger Goal: a new automation trigger that periodically polls an HTTP endpoint and activates a scene when the response matches a condition. Split into three single-responsibility entities so the endpoint can be reused beyond automations (e.g. as a value-source driving brightness/color): - `HTTPEndpoint` (storage/http_endpoint.py) — connection definition: URL + auth + headers + timeout. NO polling cadence; NO extraction. - `HTTPValueSource` (storage/value_source.py, source_type='http') — references an endpoint + owns json_path + interval + min/max + EMA smoothing. Backed by `HTTPValueStream` (core/processing/value_stream.py) which lives under the existing `ValueStreamManager` (ref-counted, one poll task per unique value source). - `HTTPPollRule` (storage/automation.py) — thin: `{value_source_id, operator, value}`. Reads `stream.get_raw_value()` from the value source and compares with `_apply_operator`. Pivoted from a 2-entity shape mid-build (was: HTTPSource+rule with interval+json_path mushed). The 3-entity shape mirrors HA's pattern (HomeAssistantSource → HAEntityValueSource → rule). ### Phase 1 — endpoint + value source + thin rule (backend) ✅ - [x] `storage/http_endpoint.py` — `HTTPEndpoint` dataclass with secret_box auth_token encryption + `__post_init__` plaintext invariant. NO `default_interval_s` (moved to value source). - [x] `storage/http_endpoint_store.py` — `HTTPEndpointStore` with `_migrate_plaintext_tokens()`. ID prefix `htep_`. - [x] `storage/database.py` — `"http_endpoints"` in `_ENTITY_TABLES` (replaces the old `"http_sources"`). - [x] `storage/value_source.py` — added `HTTPValueSource` alongside `HAEntityValueSource` (endpoint_id, json_path, interval_s, min/max, smoothing). Registered in `_VALUE_SOURCE_MAP`. - [x] `storage/value_source_store.py` — CRUD branch for `source_type = "http"` + new kwargs on create/update. - [x] `core/processing/value_stream.py` — `HTTPValueStream` with poll task + `get_value()` (normalized 0-1) + `get_raw_value()` (raw extracted value). Dispatched in `ValueStreamManager._create_stream`. Manager now takes `http_endpoint_store` so the stream can resolve endpoints at fetch time. ### Phase 2 — rule + engine wiring ✅ - [x] `storage/automation.py` — `HTTPPollRule` is now thin: just `{value_source_id, operator, value}` (no http_source_id, no json_path on the rule). Legacy keys silently dropped on load. - [x] `core/automations/automation_engine.py` — drops the standalone http_poll_manager; takes `value_stream_manager`. Engine `_sync_value_stream_refs` acquires/releases value streams for every enabled HTTPPollRule, mirroring the HA/MQTT sync pattern. `_evaluate_http_poll` reads `stream.get_raw_value()` and applies the operator. `_apply_operator` kept at module top. - [x] `api/schemas/automations.py` — RuleSchema fields are now `value_source_id + operator + value` (dropped http_source_id + json_path). - [x] `api/routes/automations.py` — `http_poll` factory updated. ### Phase 3 — CRUD endpoints + wiring ✅ - [x] `api/schemas/http_endpoints.py` — Create/Update/Response/List/Test (no interval field; that's on the value source). - [x] `api/routes/http_endpoints.py` — full CRUD + `/test` + plaintext-http-token warning. - [x] `api/schemas/value_sources.py` — `HTTPValueSource{Create,Update,Response}` added to the discriminated unions. - [x] `api/routes/value_sources.py` — `_RESPONSE_MAP` entry for `HTTPValueSource`. - [x] `api/__init__.py` — `http_endpoints_router` registered. - [x] `api/dependencies.py` — `get_http_endpoint_store` (dropped the http_poll_manager getter). - [x] `main.py` — instantiate `HTTPEndpointStore`, pass it through `ProcessorDependencies`, wire `value_stream_manager` + `value_source_store` into `AutomationEngine`. - [x] `core/processing/processor_manager.py` — `ProcessorDependencies` gains `http_endpoint_store`; threaded into `ValueStreamManager`. ### Phase 4 — tests ✅ - [x] `tests/storage/test_http_endpoint_store.py` — 14 tests (CRUD + auth_token encryption + headers + case-insensitive Authorization). - [x] `tests/core/test_automation_engine.py` — `TestApplyOperator` + `TestHTTPPollRuleEvaluation` (new shape: mock ValueStreamManager with `_streams` dict) + `TestSyncValueStreamRefs` (acquire / release / disabled-ignored) + `TestHTTPValueStreamExtraction` (`_extract_simple_path` now lives in value_stream.py). - [x] `tests/api/routes/test_http_endpoints_routes.py` — CRUD shape, no auth_token leak in responses, schema-layer method allowlist, CRLF / invalid header rejection, `/test` endpoint, LAN policy. - [x] Removed: `tests/core/test_http_poll_manager.py` (manager deleted — polling now lives inside `HTTPValueStream`). - [x] Full suite: 1426 passed, ruff clean. ### Phase 5 — frontend ✅ - [x] `static/js/features/http-endpoints.ts` (new, ~540 LOC) — endpoint CRUD, modal subclass with dirty-check, headers row editor, test result rendering, card builder, event delegation. Mirrors `home-assistant-sources.ts`. - [x] `templates/modals/http-endpoint-editor.html` (new) — sectioned rack-panel modal (Identity / Request / Headers / Notes) with IconSelect method picker, password-toggle on auth token, inline Test button + result block. - [x] `static/js/features/value-sources.ts` — added `http` branch with EntitySelect over `httpEndpointsCache`, edit-data/defaults, `onValueSourceTypeChange` section toggle, save-payload assembly + required-field validation. - [x] `templates/modals/value-source-editor.html` — new `#value-source-http-section` with endpoint picker + json_path + interval + min/max + smoothing. - [x] `static/js/features/automations.ts` — `http_poll` rule type with operator IconSelect + value-source EntitySelect; hides Value field when operator is `exists`. - [x] `static/js/features/integrations.ts` — `csHTTPEndpoints` section, tree/tab entry, render + reconcile + delegation paths. - [x] `static/js/types.ts` — `HTTPEndpoint`, `HTTPMethod`, `HTTPEndpointListResponse`, `HTTPTestRequest/Response`, `HTTPValueSource`, `HTTPPollOperator`; extended `RuleType` + `AutomationRule`. - [x] `static/js/core/state.ts` — `httpEndpointsCache` (`/http/endpoints`). - [x] `static/js/core/icons.ts` — `http: P.globe` in `_valueSourceTypeIcons`. - [x] `templates/index.html` — includes `modals/http-endpoint-editor.html`. - [x] Locales: 77 new keys per file in `en.json` / `ru.json` / `zh.json` (parity confirmed). - [x] Verification: `npx tsc --noEmit` clean; `npm run build` clean (app.bundle.css 366.6kb, app.bundle.js 2.7mb). ### Follow-ups (out of scope for initial PR) - [ ] **Global concurrency cap / minimum interval.** Each `HTTPValueStream` runs its own task at `interval_s` (min 1s); no project-wide cap. Reviewer flagged: pick a min (e.g. 5s) + max active runtimes (e.g. 32) + shared `httpx.AsyncClient` with `limits=httpx.Limits(max_connections=N)`. - [ ] **DNS-rebinding hardening.** `safe_request_bounded` validates the URL hostname's resolved IPs once; httpx independently re-resolves. The window is short but not zero. True fix: pin to the validated IP + set Host header (and SNI for HTTPS). This affects every outbound caller (`safe_fetch`, weather, image sources) — handle as a project-wide hardening, not local to this feature. - [ ] **`delete_http_endpoint` orphan refs.** When an admin deletes an endpoint referenced by N value sources, the value-stream task keeps polling until its source is also deleted. Same shape as the MQTT defect — fix both together (refuse-with-409 when in use, or cascade value-source deletion). - [ ] **Per-endpoint `connected` / last-poll status on the response** (frontend agent flagged). `HTTPEndpointResponse` has no live status, unlike HA/MQTT sources. Card LEDs default to "on". Could aggregate `last_status_code` / `last_error` from all `HTTPValueStream` instances referencing the endpoint and surface on `GET /http/endpoints/{id}`. - [x] **Per-endpoint live `/test` after save** — added `POST /http/endpoints/{id}/test` (runs stored config server-side so the auth token never round-trips) and wired a flask-icon test action on the endpoint card (toasts the result). Custom-headers section and inline test-result UI in the editor modal also restyled to match the `.group-child-row` and result-card vocabulary. - [ ] **Dedicated icon for HTTP value source / endpoint** (frontend agent flagged). Both use `P.globe` — visually fine in practice but adding a `cable`/`webhook` glyph in `icon-paths.ts` would improve differentiation. ## Multi-broker MQTT refactor Goal: drop the global `MQTTService` / `MQTTConfig`. Every MQTT consumer references an `MQTTSource.id`; `MQTTManager` is the only entry point. `MQTTManager` + `MQTTRuntime` already exist — the job is to migrate every caller off the legacy path, then delete it. ### Phase 1 — `mqtt_source_id` on Z2M target - [x] Field on `Z2MLightOutputTarget` storage dataclass (+ to/from_dict) - [x] Field on Z2M create/update/response schemas - [x] Validate referenced `MQTTSource` exists at create/update - [x] Thread through `output_target_store.create_z2m_light_target` + update - [x] Thread through `ProcessorManager.add_z2m_light_target` - [x] Thread through `Z2MLightTargetProcessor` constructor ### Phase 2 — Z2M processor uses `MQTTManager` - [x] Replace `_mqtt_service` with `_mqtt_runtime` acquired from manager - [x] `start()` acquire / `stop()` release - [x] `_publish_payload` → `self._mqtt_runtime.publish(...)` - [x] `turn_off_lights` borrow-pattern via manager (mirror HA-light) - [x] Add `mqtt_manager` to `ProcessorDependencies` / `TargetContext` ### Phase 3 — Z2M editor UI - [x] Add MQTT broker `EntitySelect` in Routing - [x] Reuse `mqttSourcesCache` - [x] Wire `mqtt_source_id` into edit-load + save payload + validation ### Phase 4 — DIY MQTT device (`MQTTLEDClient`) - [x] `mqtt_source_id` field on `Device` storage - [x] Field on `device_config.MQTTConfig` - [x] `MQTTLEDClient` acquires runtime in `connect()`, releases in `close()` - [x] Provider threads `mqtt_manager` via `ProviderDeps` - [x] Device editor: MQTT source picker shown for `device_type=mqtt`. Turned out the API layer was *also* missing it (the TODO's "backend accepts the field" was wrong — `mqtt_source_id` lived in `device_store` + `device_config.MQTTConfig` but was dropped by `DeviceCreate/Update/Response` and the routes). Added: schema fields + route threading + referenced-source validation (`_validate_mqtt_source_exists`, mirrors output_targets) + `except HTTPException: raise` guard in `update_device` (it was masking its own 4xx as 500). Frontend: broker `EntitySelect` (reusing `mqttSourcesCache`) in both the add-device (`device-discovery.ts`) and settings (`devices.ts`) modals — shown for `device_type=mqtt`, wired into load/save/validate/dirty-check/clone. Empty = "first available broker". 4 regression tests in `test_devices_routes.py::TestMqttSourceId`; full suite 1567 passing; en/ru/zh keys added. ### Phase 5 — `AutomationEngine` - [x] Drop `mqtt_service` ctor parameter - [x] Drop legacy fallback in `_evaluate_mqtt` (rule must reference a source) ### Phase 6 — `api/routes/system.py` - [x] Replace integration status with `mqtt_manager.get_all_sources_status()` - [x] Update frontend dashboard payload (MQTT widget now expects a list of sources instead of a single `enabled`/`connected` pair — surface in UI). Done: `dashboard.ts` `_renderMQTTIntegrationCard` renders one card per `mqttStatus.connections` entry; `_updateIntegrationsInPlace` iterates the list. ### Phase 7 — Startup migration - [x] Seed a "Default Broker" `MQTTSource` if legacy YAML / env had a broker configured and the store is empty (`core.mqtt.legacy_migration`) - [x] Deprecation warning logged on migration; YAML/env no longer read after ### Phase 8 — Remove legacy - [x] Delete `core/mqtt/mqtt_service.py` - [x] Delete `set_mqtt_service` / `get_mqtt_service` (mqtt_client.py) - [x] Remove `MQTTService` from `main.py` - [x] Remove `MQTTConfig` + `resolve_mqtt_password` from `config.py` - [x] Remove `mqtt: MQTTConfig` from `Config` (with `extra="ignore"` so legacy YAML still loads) ### Phase 9 — Verification - [x] `pytest tests/ --no-cov -q` clean (973 passing; removed obsolete `test_default_mqtt_disabled`) - [x] `ruff check src/` clean - [x] `tsc --noEmit` + `npm run build` - [ ] Smoke test: Z2M target on a configured MQTT Source publishes to broker (manual) ## Refactor: typed output-target factories + auto-registry Replaced `target_type` string elif chains in `OutputTargetStore` and `OutputTarget.from_dict` with: (1) `__init_subclass__` registry for deserialization, (2) per-type typed `create_*_target` / `update_*_target` methods called directly from the route layer's `match data:` dispatch. API contract unchanged, no DB migration. ### Phase 1 — Registry on `OutputTarget` - [x] Added `_registry` + `_type_key` ClassVars + `__init_subclass__(*, type_key)` - [x] Rewrote `OutputTarget.from_dict` to dispatch via registry - [x] Declared `type_key="led"` / `"ha_light"` / `"z2m_light"` on the three subclasses ### Phase 2 — Typed `create_*_target` methods - [x] Extracted `_resolve_brightness`, `_resolve_transition`, `_check_unique_name`, `_new_id_and_now`, `_finalize` helpers on the store - [x] Added `create_wled_target` / `create_ha_light_target` / `create_z2m_light_target` with per-type defaults (transition 0.5/0.3, update_rate 2.0/5.0) baked into their signatures ### Phase 3 — Typed `update_*_target` methods - [x] Added `update_wled_target` / `update_ha_light_target` / `update_z2m_light_target` with `_begin_update` / `_commit_update` helpers - [x] Each typed update method validates the target's class before mutating ### Phase 4 — Route migration - [x] `create_target` route uses `match data:` to call typed store methods — no more `getattr(data, "x", default)` pyramid - [x] `update_target` route uses `match data:` and computes `settings_changed` / `css_changed` / `brightness_changed` per-arm from typed fields - [x] Helpers `_build_ha_mappings`, `_build_z2m_mappings`, `_validate_device_exists`, `_resolve_effective_color_vs_id` extracted ### Phase 5 — Decision: keep both shims After grepping for callers, `src/ledgrab/core/scenes/scene_activator.py:90` calls `target_store.update_target(target_id, **changed)` with a dynamically built dict — it legitimately doesn't know the target's type at the call site. The shims are now ~30-line dispatchers that route to typed methods (no more inline construction elif chains), so the original anti-pattern is gone while the generic API remains available for "don't-know-the-type" callers like the scene activator. Tests continue to use the shorthand `create_target("A", "led")` form without churn. ### Phase 6 — Verify - [x] `ruff check` clean on all modified files - [x] `py -3.13 -m pytest tests/ --no-cov -q` — 974 passed (was 974 before) - [ ] Manual smoke test in UI: create/edit/delete each of the three target types ## Custom card icons — extend to all card types Migrate the existing icon-plate work (devices, LED targets, HA-light targets) to all remaining card types. ~17 entity types. Branch: `feat/icons-everywhere`. ### Foundation - [x] Refactor `icon-picker.ts` — replace hardcoded 2-entry `_adapters` record with a `Map` and expose `registerIconEntityType()` for feature modules to register their own. Added `makeSimpleIconAdapter()` helper that reduces a registration to ~6 lines. - [x] Generalised `bodyExtras` for discriminated routes (output-targets `target_type` etc.) — now keyed off id, adapter does its own lookup. - [x] `_onDocumentClick` accepts any registered type instead of hardcoded device/target check. - [x] Locale entity-type labels added to en/ru/zh for 18 new types (picture_source, audio_source, weather_source, value_source, mqtt_source, ha_source, automation, scene_preset, sync_clock, game_integration, audio_processing_template, pattern_template, capture_template, pp_template, cspt, audio_template, gradient, color_strip_source, asset). ### Backend (storage + schemas + routes per entity) Recipe: add `icon: str = ""` + `icon_color: str = ""` to dataclass, emit-when-truthy in `to_dict`, default `""` in `from_dict`; add 3 `Optional[str]` Field defs to Create/Response/Update schemas; thread `getattr(entity, "icon", "") or ""` into the response builder. SQLite JSON-blob storage means **no migration required**. - [x] Integrations (6): weather_sources, value_sources, mqtt_source, home_assistant_source, sync_clocks, game_integration - [x] Streams (10): picture_source, audio_source, audio_template, audio_processing_template, pattern_template, postprocessing_template, color_strip_processing_template, color_strip_source, gradient, capture_template (`storage/template.py` — was missed by initial pass) - [x] Other (3): automation, scene_preset, asset ### Frontend (per feature module) For each card render call: - Use the new `core/card-icon.ts` helper: `...makeCardIconFields('', entity.id, entity)` spread into the mod-card head — computes `iconHtml`/`iconColor`/`iconAttrs` in one go. - Register the entity type in the feature module via `registerIconEntityType('', makeSimpleIconAdapter({ … }))`. Modules wired: - [x] streams.ts (7 cards: picture, capture, pp, cspt, audio source, audio template, gradient — built-in gradients skip the plate) - [x] automations.ts - [x] scene-presets.ts - [x] sync-clocks.ts - [x] weather-sources.ts - [x] value-sources.ts (bodyExtras propagates `source_type`) - [x] mqtt-sources.ts - [x] home-assistant-sources.ts - [x] game-integration.ts - [x] audio-processing-templates.ts - [x] assets.ts - [x] color-strips/cards.ts (bodyExtras propagates `source_type`) - [WONTDO] pattern-templates.ts — uses legacy `wrapCard({content, actions})` string API, not the mod-card system. Migration would be a separate effort and the cards are tiny (name + rect count) so the value is low. ### Discriminated routes Adapters provide `bodyExtras` to inject the discriminator field on PUT so the Pydantic discriminated-union route validators don't reject the icon-only update: - output-targets → `target_type` (already wired before) - color-strip-sources → `source_type` - audio-sources → `source_type` - value-sources → `source_type` - picture-sources → `stream_type` ### Verification - [x] `cd server && ruff check src/ tests/` clean - [x] `cd server && npx tsc --noEmit` clean - [x] `cd server && npm run build` produces 2.6 MB bundle - [x] `cd server && py -3.13 -m pytest tests/ --no-cov -q` — 949 passed - [ ] Manual: open picker on each card type, confirm save persists, confirm channel-color preview matches the live card ## Device Event Notifications Notify the user when LED devices come online/go offline (configured targets), and when new WLED/serial devices are discovered or disappear from the LAN/USB. Each event class has a configurable channel: `none` | `snack` | `os` | `both`. OS channel uses Web Notifications (works in any browser tab and in the PWA shell — no platform-specific Python). Branch: `feat/device-event-notifications`. Default ON. ### Backend - [x] `core/devices/discovery_watcher.py` — long-running mDNS browser (`AsyncServiceBrowser` kept alive for the process lifetime) + 10 s serial-port poller. Fires `device_discovered`/`device_lost` via `processor_manager.fire_event`, suppresses events for URLs already in `device_store`. Seeded ports do NOT generate startup-time toasts. - [x] Wired into `lifespan` (`main.py`). Gated by `notification_preferences. background_discovery_enabled`. Default True. Stops before health monitor stop. - [x] `api/schemas/preferences.py` — `NotificationPreferences` Pydantic v2 model with the 4-event channel matrix, `background_discovery_enabled`, `startup_grace_sec` (0..300), `flap_debounce_sec` (0..60). - [x] `api/routes/preferences.py` — `GET/PUT /api/v1/preferences/notifications`, persisted under `db.set_setting("notification_preferences", …)`. Corrupt stored values fall back to defaults instead of 500. - [x] Reuses existing `device_health_changed` event from `device_health.py` (already fires online/offline transitions on the same event bus). - [x] Tests: 7 in `tests/test_preferences_notifications_api.py`, 6 in `tests/test_discovery_watcher.py`. Full pytest suite still 899 passing. ### Frontend - [x] `js/features/notifications-watcher.ts` — listens to the three `server:*` DOM events. Applies user prefs. Pipeline: startup grace → flap debounce → bulk coalesce (≥3 events / 800 ms collapse to one summary). - [x] Web Notification permission requested from the Settings → Notifications panel via a user-gesture button. State chip reflects granted/denied/default. - [x] Settings panel — new "Notifications" subtab between Backup and Appearance. 4 IconSelects (`none`/`snack`/`os`/`both`) + background-discovery toggle + permission row + Test-notification button. - [x] i18n: `settings.notifications.*` and `notifications.*` keys in en/ru/zh. ### Verification (notifications) - [x] `npx tsc --noEmit` clean, `npm run build` produces 2.5 MB bundle. - [x] `ruff check src/ tests/` clean. 899/899 pytest pass. - [x] App import smoke-test (`from ledgrab.main import app`) loads 233 routes without errors. - [ ] Real-hardware test pending — verify on user's network: (1) plug a fresh WLED in → snack toast appears, (2) configure it → next offline transition fires both snack + OS toast, (3) Background-discovery toggle off → no more discovered/lost events. ### Out of scope for v1 - Per-device-type granularity (we ship one matrix per event-type, no device-type split) - Per-device mute list (deferred — user can globally toggle off if noisy) - Native OS toast via Windows winrt API (Web Notifications cover the use case; also avoids the `os_notification_listener` feedback loop) - Notification history panel — could land later as the reserved `alerts` dashboard cell ## Server shutdown action Let user choose what happens to LED targets on server shutdown. - [x] Backend storage: `shutdown_action` in `db.settings` (`"stop_targets"` default | `"nothing"`) - [x] Backend route: `GET/PUT /api/v1/system/shutdown-action` in `system_settings.py` - [x] Backend schema: `ShutdownActionResponse/Request` in `schemas/system.py` - [x] Backend wiring: lifespan shutdown in `main.py` reads action, passes `restore_devices` flag to `processor_manager.stop_all()` - [x] `processor_manager.stop_all(restore_devices: bool = True)` — when False, calls public `proc.cancel_task()` (defined on `TargetProcessor`) which awaits cancellation without restoring device state; skips `_restore_device_idle_state` loop. No reach into private `_task` attribute. - [x] Frontend: hidden `` audit deferred (project already enforces via CLAUDE.md rule + IconSelect/EntitySelect wrappers). ### Phase 6 — Mobile dedicated shell - [x] `mobile.css` (existing file, not forked) — fixed-bottom `.tab-bar` promoted to full Lumenworks treatment: gradient background + hairline divider at top + channel-accent rule matching the transport-bar bottom. Active tab gets an LED pip above the icon and a channel-tint background. Tab labels + badges use mono uppercase to match the rest of the app. Phone (≤600 px): modal corner-bracket hidden (fullscreen modals), modal-header stripe slimmed to 18 px. - [x] Phase 2's layout.css already strips the transport-center on phones and collapses the sidebar via `display: contents`, so the mobile shell automatically routes the tab-bar to the bottom without a separate JS hook. - [WONTDO] Fork into `mobile-shell.css` — keeping changes in `mobile.css` since the cascade was already organized by viewport. A rename adds churn without improving maintainability. ### Phase 7 — Microcopy + retire legacy - [x] Locale rename: `targets.title` + `dashboard.section.targets` → "Channels" (en) / "Каналы" (ru) / "通道" (zh); `streams.title` → "Inputs" / "Входы" / "输入". Automations kept as-is (Automations + Scenes is a meaningful distinction; "Patches" would conflate them). Internal tab keys (`dashboard` / `automations` / `targets` / `streams` / `integrations` / `graph`) unchanged so no JS or localStorage migration needed. - [x] Ambient WebGL background — default is already `off`; kept the toggle button and localStorage preference so users who want the shader can turn it on. No entry-point change needed: `data-bg-anim` is initialized from localStorage with `off` fallback. - [DEFERRED] Delete DM Sans + legacy color tokens — would cascade through every file that reads `--primary-color` / `--text-color` etc. Safer as a separate cleanup PR after the new design has soaked. - [WONTDO] Delete `mobile.css` — Phase 6 kept the filename. ## Dashboard Customization Per-account dashboard layout — slide-in Customize panel lets users toggle section / perf-cell visibility, reorder via drag, change density, pick presets, and import/export the layout as JSON. Server-synced via `db.get_setting('dashboard_layout')` so settings follow the user. - [x] `js/features/dashboard-layout.ts` — schema (open registry of section / perf-cell keys so v1.1 cards slot in with no migration), defaults, 5 built-in presets (Studio/Operator/Showrunner/Diagnostics/TV), localStorage cache + server sync, legacy-key migration from `dashboard_collapsed`, `perfMetricsMode`, `perfChartColor_*`. - [x] `api/routes/preferences.py` — `GET/PUT/DELETE /api/v1/preferences/dashboard-layout`. Treats payload as opaque (frontend owns the schema); validates only that body is an object with a numeric `version`. 6 pytest tests in `tests/test_preferences_api.py` cover round-trip, default-empty, validation, delete, and unknown-field passthrough. - [x] `js/features/dashboard.ts` — sections rendered into a fragment map, then assembled in layout-driven order; perf section stays pinned top (chart-persistence reasons) but its visibility is layout- driven. Layout-change subscription invalidates the in-place-update optimization so density / order / visibility changes always rebuild section HTML. - [x] `js/features/perf-charts.ts` — `renderPerfSection()` iterates `getOrderedPerfCells()`; existing legacy `setPerfMode` writes through to the layout so the global toggle and the customize panel stay in sync. - [x] `js/features/dashboard-customize.ts` + `css/dashboard-customize.css` — slide-in panel, hand-rolled HTML5 drag-and-drop reorder, ↑/↓ buttons for keyboard / TV remote, debounced (300 ms) autosave, live preview while open. Reset / export / import actions. - [x] i18n keys for `dashboard.customize.*` in en/ru/zh. - [ ] (v1.1) Audio meters section — peak / RMS / BPM bars per audio source. Schema key `audio-meters` already reserved. - [ ] (v1.1) Alerts section — quiet by default, loud on issues. Reserved key `alerts`. - [ ] (v1.1) Live LED preview strip per running device. Reserved key `led-preview`. - [ ] (v1.1) Source thumbnails grid (1 fps multiviewer). Reserved key `source-thumbs`. - [ ] (v1.2) Pinned section (user-curated mix of targets / scenes / devices). Reserved key `pinned`. - [ ] (v1.2) Patch/flow map — read-only mini graph of routing. Reserved key `flow`. ## BLE LED Controller Support (SP110E / Triones / Zengge / Govee) Add support for Bluetooth Low Energy LED controllers driven by mobile apps like "LED Hue", HappyLighting, iLightsIn. Whole-strip ambient-color output only — these protocols don't support per-pixel streaming. - [x] Add `bleak>=0.22` as optional extra `[ble]` in `server/pyproject.toml` (desktop-only, NOT in android `build.gradle.kts`) - [x] `core/devices/ble_transport.py` — bleak wrapper: scan, connect, write-with/without-response - [x] `core/devices/ble_protocols/` package - [x] `__init__.py` — `BLEProtocol` dataclass + registry (family → encoder) - [x] `sp110e.py` — SP110E / SP108E (service FFE0, char FFE1, `RR GG BB 00 1E` static-color frame) - [x] `triones.py` — Triones / HappyLighting / LEDnet (service FFE5, char FFE9, `7E 07 05 03 RR GG BB 10 EF`) - [x] `zengge.py` — Zengge / iLightsIn (service FFE0, framing `56 RR GG BB 00 F0 AA`) - [x] `govee.py` — Govee unencrypted framed protocol (AES keyed variants — marked experimental) - [x] `core/devices/ble_client.py` — unified `BLEClient(LEDClient)` — picks protocol by `ble_family`, averages strip → one color, drops duplicate frames, rate-limits to BLE connection interval - [x] `core/devices/ble_provider.py` — `BLEDeviceProvider` + discovery via `BleakScanner` - [x] Register in `core/devices/led_client.py::_register_builtin_providers` (guarded `try/except ImportError`) - [x] Storage: `ble_family`, `ble_govee_key` fields threaded through `Device.__init__`/`to_dict`/`from_dict`/`_UPDATABLE_FIELDS`/`create_device` - [x] Schemas: BLE fields on `DeviceCreate`, `DeviceUpdate`, `DeviceResponse` - [x] Routes: BLE fields propagated through create/update in `api/routes/devices.py` + `_device_to_response` - [x] ProcessorManager: `ble_family`/`ble_govee_key` added to `_DEVICE_FIELD_DEFAULTS` and `DeviceInfo`; passed through `wled_target_processor.py` and `group_client.py` to `create_led_client` - [x] Tests: 21 protocol encoder unit tests + 16 BLEClient fake-transport tests — all passing, 814 total tests still green - [x] Frontend: BLE option in the device type picker with a bluetooth Lucide icon; add-device modal shows a 4-option `IconSelect` for protocol family (SP110E / Triones / Zengge / Govee) with a Govee-only AES key field that auto-hides for the other three families; URL label/placeholder/hint adapt to `ble://
` pattern; submit payload carries `ble_family` (+ optional `ble_govee_key`); clone flow pre-fills family and key; modal dirty-check snapshots the new fields; network scan button now also discovers BLE peripherals via the existing `/api/v1/devices/discover?device_type=ble` endpoint - [x] Frontend: `isBleDevice` helper in `core/api.ts`; `ICON_BLUETOOTH` + `ICON_LIGHTBULB` constants in `core/icons.ts`; `bluetooth` path in `core/icon-paths.ts`; i18n keys in `en.json` / `ru.json` / `zh.json`; TypeScript compiles; esbuild bundle rebuilt - [x] Android BLE via Kotlin bridge — `BleBridge.kt` singleton (scan/connect/write/disconnect); `android_ble_transport.py` Python wrapper; `make_transport()` factory in `ble_transport.py` auto-selects backend; `BleBridge.init()` called from `LedGrabApp.onCreate`; BLE permissions in `AndroidManifest.xml` - [x] Govee per-model AES key — `_encrypt_govee_frame()` in `ble_client.py` uses AES-128-ECB from `cryptography`; key validated on `BLEClient` construction; applied to both `send_pixels` and `set_power`; 8 new AES unit tests ## Android — Restore Multi-ABI Wheels During emulator testing, we switched the build to **x86 only** (see `android/app/build.gradle.kts` `abiFilters`) to avoid having to keep the arm64-v8a / x86_64 pydantic-core wheels current. Before shipping, restore all three ABIs: - [x] Rebuild `pydantic-core` wheels for all three ABIs with the current SOABI + libpython linking settings (`android/build-scripts/build-pydantic-core.sh` — now supports `arm64`, `x86_64`, `x86` args; defaults to all three). - [x] Verify wheels: all three now list `libpython3.11.so` in `NEEDED` (`llvm-readelf -d`), automated in the build script. - [x] Restored `abiFilters += listOf("arm64-v8a", "x86_64", "x86")` in `build.gradle.kts`. Multi-ABI debug APK builds cleanly (~99 MB). - [ ] Re-test on real ARM64 Android TV hardware (still pending — only emulator-verified build). Build cache + scripts live in `android/build-scripts/` and `android/.build-cache/` (junction host + sysconfigdata for each ABI). ## Android CI Pipeline Build the Android APK automatically on push/tag. - [x] Generate Gradle wrapper (`gradlew`) and commit it - [x] Create CI workflow (`.gitea/workflows/build-android.yml`) - JDK 17 + Android SDK + NDK setup - Python 3.11 for Chaquopy build - Recreate the directory junction via `ln -s` on Linux CI - `./gradlew assembleDebug` on master push, `assembleRelease` on `v*` tags (if signing secrets set) - Uploads APK as CI artifact; attaches to Gitea release on tag push - [x] Commit pre-built pydantic-core wheels to `android/wheels/` (arm64, x86, x86_64) - [x] APK signing for release builds — conditional signing config reads keystore from env vars (`ANDROID_KEYSTORE_PATH/_PASSWORD/_ALIAS/_KEY_PASSWORD`), falls back to debug signing locally - [ ] Provision a real keystore and add the four CI secrets: - `ANDROID_KEYSTORE_BASE64` (base64-encoded .jks) - `ANDROID_KEYSTORE_PASSWORD` - `ANDROID_KEY_ALIAS` - `ANDROID_KEY_PASSWORD` - [ ] Add `LedGrab-{tag}-android-release.apk` row to the release description table in `.gitea/workflows/release.yml` → `create-release` job - [ ] Verify the CI workflow passes end-to-end with the now-restored multi-ABI build (larger APK, longer Android build step) ## Android Root Capture (No Permission Dialog, No System Indicator) MediaProjection shows a mandatory system overlay/indicator while capturing — unavoidable on stock Android. Many cheap Android TV boxes ship pre-rooted, so an alternative root-only path gives much better UX. - [x] Root detection — `Root.kt` checks common `su` binary paths and, on demand, runs `su -c id` to actually prove UID 0. First call triggers Magisk's grant dialog; grant is cached per session. Exposed to Python via Chaquopy. - [x] `RootScreenrecord.kt` — spawns `su -c screenrecord --output-format=h264 --size=WxH -`, feeds the H.264 stdout through a MediaCodec decoder whose output Surface is wired into an ImageReader (RGBA_8888, row-stride-aware). Decoded frames reach the Python pipeline via `PythonBridge.pushRootFrame`. - [x] Python-side `RootScreenrecordEngine` (`core/capture_engines/root_screenrecord_engine.py`) mirrors `MediaProjectionEngine` with `ENGINE_PRIORITY=110` (> MediaProjection's 100) so the factory picks it automatically when available. - [x] `MainActivity` tries `Root.requestGrant()` before launching the MediaProjection consent flow — on rooted devices the consent dialog is skipped entirely. `CaptureService` has a `createRootIntent()` entry point that bypasses the MediaProjection path. - [x] Fallback: if `Root.requestGrant()` returns false (no root, user denied, or `su` timeout) the existing MediaProjection flow runs unchanged. - [ ] Real-hardware test pending — need to verify on the user's Magisk'd TV box that: (1) grant dialog appears once, (2) frames actually flow through MediaCodec without the Android 14 capture indicator showing, (3) stop/start cycle terminates the `su` process cleanly. - [WONTDO] `SurfaceControl.screenshot()` via reflection — renamed/moved across API 28/29/30/33, hidden-API blocklist varies by release, even rooted apps hit it; days of maintenance for a marginal latency win over the screenrecord path. Not worth it. - [WONTDO] `adb screencap` fallback — full-PNG-per-frame pipeline is slower than MediaProjection, no value as a last resort. Known projects using the screenrecord approach for reference: scrcpy (over ADB), scrcpy-hidden-api, shizuku. ## Android Autostart on Boot Boot-time, zero-interaction startup so LedGrab always has display capture and control on rooted TV boxes. - [x] Manifest: declare `RECEIVE_BOOT_COMPLETED`, `REQUEST_IGNORE_BATTERY_OPTIMIZATIONS`, `WAKE_LOCK`; register `.BootReceiver` for `BOOT_COMPLETED` / `LOCKED_BOOT_COMPLETED` / `MY_PACKAGE_REPLACED`. - [x] `BootReceiver.kt` — gated by `AutostartPrefs` + `Root.looksRooted()`; dispatches `CaptureService.createRootIntent()` via `ContextCompat.startForegroundService`. Unrooted devices are a no-op because MediaProjection consent cannot be bypassed silently. - [x] `AutostartPrefs.kt` — thin SharedPreferences wrapper, defaults to enabled. Shown as a CheckBox on the stopped panel; greyed out on unrooted devices. - [x] `CaptureService` returns `START_REDELIVER_INTENT` for root-mode intents so the OS can cleanly restart the service after being killed (token-free path). MediaProjection-mode keeps `START_NOT_STICKY` — restart is pointless with a dead consent token. - [x] `isRunning` race: moved assignment to after `startForeground` succeeds, resets on exception; `onStartCommand` wraps `startForeground` in try/catch and stops the service cleanly if the FG transition fails. - [x] Root-capture watchdog: coroutine on `serviceScope` checks `RootScreenrecord.framesDelivered` every 5s after a 5s grace. Respawns the pipeline (reusing the existing Python bridge) on stall, caps at 3 consecutive restarts before giving up. - [x] `RootScreenrecord.framesDelivered` exposed as a property backed by `AtomicInteger` (was `@Volatile var framesDelivered = 0` with non-atomic `+= 1`). - [x] `ScreenCapture` accepts `onProjectionStopped` lambda — `MediaProjection.Callback.onStop` now tears the whole service down instead of leaving a stale FG notification. - [x] `MainActivity` wires the autostart toggle to `AutostartPrefs`; enabling it prompts `REQUEST_IGNORE_BATTERY_OPTIMIZATIONS` so Doze doesn't kill the FG service on phones. - [x] `versionCode` derived from `git rev-list --count HEAD` (or `ANDROID_VERSION_CODE` env var in CI). Was stuck at 1 — sideload updates were silently refusing to install. - [ ] Real-hardware test pending — need to verify on the user's Magisk'd TV box that: (1) boot-time autostart dispatches the service without UI, (2) capture indicator still absent under root mode post-reboot, (3) watchdog respawns the pipeline when `screenrecord` is externally killed, (4) sideload upgrade installs cleanly after the versionCode bump. - [ ] Optional follow-up: "kiosk" mode — add `` to `MainActivity` so power users can set LedGrab as the default TV launcher for truly always-running behavior. ## Android USB Serial Support Drive USB LED controllers (APA102, WS2812) connected directly to the Android TV box via USB-to-serial adapters. - [x] Added `com.github.mik3y:usb-serial-for-android:3.8.1` (via JitPack) to `android/app/build.gradle.kts`. - [x] Kotlin `UsbSerialBridge` singleton (`android/app/src/main/java/com/ledgrab/android/UsbSerialBridge.kt`) — exposes `listDevices()`, `open(vid, pid, serial, baud)`, `write(handle, ByteArray)`, `close(handle)`. Permission request fires automatically from `open()` when the user hasn't granted access yet. Handles are opaque integers, port map is synchronized, so Python threads can share one bridge. - [x] Python `AndroidSerialTransport` in `server/src/ledgrab/core/devices/android_serial_transport.py` drives the bridge through Chaquopy. `SerialTransport` Protocol + `PySerialTransport` + `list_serial_ports()` factory live in `serial_transport.py`; `AdalightClient` and `SerialDeviceProvider` now go through the abstraction instead of importing `pyserial` directly. - [x] URL scheme extended: `usb:VID:PID[:serial][@baud]` on Android alongside the existing `COM3[:baud]` / `/dev/ttyUSB0[:baud]` desktop paths. - [x] App initializes the bridge on startup (`LedGrabApp.onCreate` → `UsbSerialBridge.init(this)`); manifest declares `uses-feature android.hardware.usb.host`. - [ ] Real-device test pending — no USB-serial hardware on dev machine. Need to verify on a TV box with CH340, CP2102, or FTDI adapter. - [ ] Document supported USB LED controllers in README (once real-device test passes). - [ ] Optional: auto-launch the app when a known USB-serial adapter is plugged in (intent-filter on `USB_DEVICE_ATTACHED` + `res/xml/device_filter.xml`). Skipped in v1 — users can just open LedGrab and hit "Discover". - [x] ESP-NOW client (`espnow_client.py` / `espnow_provider.py`) now routes through `SerialTransport` — `open_transport()` for the gateway serial link, `list_serial_ports()` + `port_exists()` for discovery/validation. Works transparently with `usb:VID:PID` URLs on Android. (Gateway protocol is write-only, so no `read()` extension was needed after all.) ## Performance Metrics Abstraction - [x] `MetricsProvider` protocol + dataclass DTOs (`MemorySnapshot`, `ProcessSnapshot`) live in `server/src/ledgrab/utils/metrics/types.py`. Each provider has its own module: `psutil_provider.py`, `null_provider.py`, `android_provider.py`. - [x] Factory `get_metrics_provider()` in `utils/metrics/__init__.py` selects Android → psutil → Null. `psutil` import is now confined to one place. - [x] `api/routes/system.py` and `core/processing/metrics_history.py` use the provider; no more `if psutil is not None` guards in the hot paths. - [x] Android `/proc`-backed provider implemented (`/proc/stat`, `/proc/meminfo`, `/proc/self/stat`, `/proc/self/status`). Carries previous-sample state for delta-based CPU%; degrades to zeros if any `/proc` file is locked down. 12 unit tests cover both desktop and Android paths. ## Android Performance Metrics — Future Enhancements Beyond the `/proc`-based AndroidMetricsProvider that's now in place: - [x] Device battery + thermal-zone readings (`/sys/class/power_supply/battery/{capacity,temp}`, `/sys/class/thermal/thermal_zone*/temp` filtered by zone type). Surfaced through `MetricsProvider.thermals()`, `PerformanceResponse.{cpu_temp_c,battery_percent,battery_temp_c}`, the metrics-history snapshot, and a new dashboard temperature chart that hides itself when the backend reports null. GPU card now hides (no "unavailable" placeholder) when no GPU is present. - [WONTDO] Optional: app-specific memory via `Debug.getMemoryInfo()` through a Kotlin → Python Chaquopy bridge (more accurate than `VmRSS` for split-app-process accounting) - [WONTDO] Optional: GPU usage via `/sys/class/kgsl/kgsl-3d0/gpubusy` on Adreno, Mali-specific paths for Mali GPUs ## Refactor: Per-Provider Device Configs Replace flat `DeviceInfo` + `**kwargs` provider contract with a discriminated union of typed per-provider config dataclasses. - [x] Phase 1 — `DeviceConfig` hierarchy + `Device.to_config()` (non-breaking, additive only) - [x] Phases 2+3 — narrow `LEDDeviceProvider.create_client` to typed configs; migrate 3 call sites; delete `DeviceInfo` + `_get_device_info` + `_DEVICE_FIELD_DEFAULTS` (single PR) - [x] Phase 4 — migrate `tests/test_group_device.py` to `GroupConfig`/`ProviderDeps`; remove legacy `GroupLEDClient` init path; 47-test config suite with 100% coverage on `device_config.py` - [ ] Phase 5 (separate PR, optional) — Pydantic v2 discriminated union in `api/schemas/devices.py`; scope frontend POST/PATCH payloads by `device_type` ## Expand device support (Phase 1: open protocols) Branch: `feat/expand-device-support`. Goal: maximize the universe of LED controllers LedGrab can drive by adding aggregator + open-protocol providers in roughly-this order. Each driver follows the established `LEDDeviceProvider` + `*Config` + tests pattern. ### Phase 1.1 — Standalone DDP target ✅ shipped (commit `8f1140a`) DDP packet layer (previously WLED-internal) promoted to a first-class device type. Pixelblaze, ESPixelStick, xLights/Falcon endpoints, and generic DDP receivers are now drivable directly without WLED in the path. ### Phase 1.2 — Yeelight LAN Xiaomi/Yeelight bulbs, port 55443 TCP JSON. Direct protocol (no `python-yeelight` dependency — implementation is ~200 lines). - [x] `YeelightConfig` dataclass with `yeelight_min_interval_ms` rate limit - [x] `YeelightClient` in `core/devices/yeelight_client.py` — TCP JSON-RPC, averaging single-pixel adapter, client-side rate gate - [x] SSDP-style discovery (Yeelight's variant on `239.255.255.250:1982`) - [x] `YeelightDeviceProvider` with validate/health/discover - [x] Storage + API schemas + route handler wiring - [x] 34 unit tests (URL parsing, RGB packing, averaging, rate limit, SSDP parsing, provider validate/discover, Device.to_config round-trip) - [ ] Frontend: Yeelight in device-type picker + edit form (spawned to a `frontend-design` subagent) - [ ] Locale strings (en/ru/zh) - [ ] Music mode (~60 Hz updates via reverse-TCP) — follow-up, current MVP caps at ~2 Hz via the client-side rate gate ### Phase 1.3 — WiZ Connected Philips' UDP-local budget tier. Port 38899 JSON UDP. - [x] `WiZConfig` + `WiZClient` + `WiZDeviceProvider` - [x] UDP broadcast discovery on 255.255.255.255:38899 with the standard `registration` envelope; replies parsed for IP+MAC. - [x] Sync `send_pixels_fast` for the hot loop (UDP is fire-and-forget, no async needed). 50 ms default min interval → ~20 Hz cap. - [x] Health check sends `getPilot` and waits for any reply. - [x] Storage + API schemas + route handler wiring - [x] 36 unit tests - [ ] Frontend: WiZ in device-type picker + edit form - [ ] Locale strings (en/ru/zh) ### Phase 2 — Unified discovery + pairing UX layer After phase 1 the codebase will have 3 fresh examples of "ping the LAN, listen for replies, present a list". Factor that out into a generic discovery scaffold + a "first-run pairing" UX component before adding Tuya/Govee/etc., which each need a one-time pairing dance. - [WONTDO] Generic `NetworkDiscoveryService` — the existing `/api/v1/devices/discover` route already runs all providers in parallel via `asyncio.gather(return_exceptions=True)`. Extracting it would not unlock anything; revisit only if discovery cadence/dedup becomes a real complaint. - [WONTDO] Unified scan UI — already exists; one "Scan network" button triggers the cross-provider fan-out. - [x] **Reusable pair-device scaffold** (the actually-needed piece). Backend: `LEDDeviceProvider.pair_device(url)` abstract method with `PairingNotReady` sentinel; `POST /api/v1/devices/pair` endpoint with status-code mapping (200/400/409/422/502); 8 route tests covering every outcome. Frontend: `templates/modals/pair-device.html` five-state modal (idle / pairing / not-ready / success / failed) with a 30-second SVG progress ring; reusable `static/js/features/pairing-flow.ts` exposing `runPairingFlow({deviceType, url}) → Promise<{fields}>` with `PairingCancelled` sentinel; locale strings in en/ru/zh. No driver uses it yet — Nanoleaf will be the first concrete consumer. ### Phase 3 — Big aggregator unlocks - [ ] ESPHome native API (`aioesphomeapi`) - [ ] Tuya Local (`tinytuya`) — biggest single market unlock; needs the pairing UX from Phase 2 - [ ] Matter over IP (forward-looking) - [ ] Hyperion JSON downstream ### Phase 4 — Major consumer brands - [x] **LIFX LAN** — UDP binary protocol on port 56700; RGB→HSBK 16-bit conversion; broadcast discovery via GetService/StateService probe; 47 unit tests. Single-pixel adapter shape, identical to WiZ structurally. Frontend wired via subagent. - [x] **Govee LAN API** — UDP JSON on port 4003 (control) + 4002 (responses) + 4001 (multicast discovery on 239.255.255.250). Single-pixel `colorwc` command with `colorTemInKelvin=0` for RGB mode. **Per-device "LAN Control" toggle required in Govee Home app.** 40 unit tests. Frontend wired via subagent. - [x] **Nanoleaf OpenAPI** — Light Panels / Canvas / Shapes / Lines / Elements via HTTP REST on port 16021. **First concrete user of the pairing-UX scaffold from Phase 2.** mDNS discovery via `_nanoleafapi._tcp`. Single-pixel adapter (averaged strip → HSB `PUT /state`). Auth token encrypted at rest via `_enc`/`_dec`. 42 unit tests covering URL parsing, RGB→HSB conversion, pairing handshake (200/403/500/missing-token/transport-error), state mutations, brightness clamping, Device.to_config round-trip including encrypted-token roundtrip. - [ ] Twinkly — multi-pixel + login flow; deferred - [WONTDO] Mi-Light / MiBoxer UDP gateway — the recommended path for modern Mi-Light deployments is `esp8266_milight_hub` firmware → MQTT, which LedGrab already supports through the existing MQTT device target (commit `530316c`). Native V6 driver would be ~400 lines + finicky session protocol + custom 1-byte hue table; the marginal benefit over the MQTT path is small. Revisit if a user complaint surfaces. ### Phase 5 — Open pixel protocols (cheap completionism) - [x] **OPC (Open Pixel Control)** — TCP, port 7890, 4-byte header `[channel][cmd][len_hi][len_lo]` + RGB body. Channel 0 broadcasts. Single-pixel-strip protocol, no discovery, no pairing. 36 unit tests. Fadecandy + xLights + hobbyist receivers reachable. - [ ] TPM2.net ### Phase 6 — PC gaming RGB completion - [ ] Corsair iCUE SDK - [ ] Logitech LIGHTSYNC - [ ] ASUS Aura SDK ### Phase 7 — Proprietary USB HID ambient kits - [ ] Generic HID-ambient framework + VID/PID registry - [ ] First reverse-engineered target (probably Govee Immersion / DreamView) ### Cleanup + verification - [x] **`_average_color` extraction** (commit `cc87fba`). Six identical copies (Yeelight / WiZ / LIFX / Govee / Nanoleaf / BLE) collapsed into `core/devices/pixel_reduce.average_color`. Net -76 lines. Hue is out by design — its Entertainment API addresses up to seven lights individually. - [x] **Pre-merge verification pass.** 1358 pytest tests pass; ruff clean across all device modules and tests; black clean against the pre-commit-pinned 24.10.0; `npx tsc --noEmit` clean; bundle compiles. - [x] **Pre-merge code review (subagent)** — surfaced 2 CRITICAL + 4 HIGH + 3 MEDIUM + 3 LOW findings. - [x] **All review findings fixed** (commits `7736bc6` + `0e3ae78`): - CRITICAL #1: missing `url_scheme.py` / `net_classify.py` committed (4 files / 557 lines). - CRITICAL #2: `update_device` no longer re-encrypts secrets in memory via the `to_dict()` round-trip (uses `vars()` directly). - HIGH #3: `nanoleaf_token` / `hue_username` / `hue_client_key` stripped from `DeviceResponse`; replaced with paired-flag booleans. Frontend updated. - HIGH #4: `validate_lan_host()` rejects literal public IPs at each driver's `validate_device` + `pair_device`. - HIGH #5: `_dec()` failures clear the field and log, not crash the row. - HIGH #6: update route now rstrip's URL for all device types. - MEDIUM #7: Govee discovery serialized via `asyncio.Lock`. - MEDIUM #8: Nanoleaf mDNS browser cleanup moved to `finally`. - MEDIUM #9: pair endpoint sanitizes URL userinfo in logs. - LOW: Nanoleaf `.port` property added; pair-then-create E2E test added. - Tests: 1379 pass (+21 regression tests). ## Graph editor — "full control of wiring via graph" (in progress) Goal: make the visual graph a first-class wiring control surface, not just a viewer. Driven by the ULTRA-DEEP review (findings A1–A5, B1–B6, C1–C6, D1–D6). ### Done (NOT yet committed — awaiting review/commit) - [x] **A1** Undo/redo wired to connect/detach/move (was dead code); inverse ops throw on failure so the stack can't silently desync. - [x] **A2** Manual node layout persists to `localStorage` (`graph_node_positions`), cleared on relayout. - [x] **A3** Scene-preset disambiguation — deactivation scene now reachable via a field picker (was always picking the first match). - [x] **B6** Edge field labels (revealed on zoom ≥ 0.9). - [x] **C3** Health overlay — broken refs (referrer exists, target missing), dependency cycles, orphans; node warning badges + an issues toolbar button. - [x] **D1** `GET /api/v1/graph/schema` — authoritative connectable-field registry (`api/graph_schema.py`, pure + unit-tested). - [x] **D2** `GET /api/v1/graph` (nodes+edges+validation) and `GET /api/v1/graph/dependents/{kind}/{id}`. - [x] **D4** `POST /api/v1/graph/validate-connection` — existence + source-kind + cycle pre-flight; frontend validates before every write (fails open if the endpoint is unreachable). List/double-nested fields rejected. - [x] **B2** Drop-on-node connect — empty top-level slots are now wireable (drop a source onto any compatible node body, not just an existing port). - [x] **C4** Overwrite-occupied-slot confirm + delete-with-dependents warning (single delete only; bulk keeps the batch confirm). - [x] **D5** Create-and-connect — drag a port onto empty canvas → pick a compatible new entity kind → it's created and auto-wired (kind-scoped watcher). - [x] **D6 (read-only half)** "Export graph (JSON)" toolbar action. - [x] Custom per-entity `icon` + `icon_color` now render on graph nodes (parity with custom node colours; fallback to kind/subtype glyph). - [x] **B1** Edit single-level **BindableFloat** value slots from the graph (`brightness`, `smoothing`, `intensity`, `scale`, `speed`, … on color_strip_source; `brightness`/`transition` on output_target). Subtype-safe (only offers slots the target entity actually has). Writes the partial `{ : { source_id } }` payload → backend `Bindable*.apply_update` merges, preserving the static value. Verified data-safe (no `from_raw`/value-reset path). - [x] Render the two functional value-source references `buildGraph` was missing — `value_source.value_source_id` (gradient_map → inner value source) and `value_source.color_strip_source_id` (css_extract → strip). Both are runtime- resolved and already drag-editable; now visible/detachable in the graph. - [x] **B4 foundation:** backend schema now authoritative about graph-editability (`is_editable()` + `editable` flag in `/graph/schema`); `validate-connection` hardened to reject non-editable fields (colour/list/double-nested), not just lists. - [x] **B4 drift guard + gap fixes:** `checkSchemaDrift()` (graph-connections.ts) warns once if the frontend `CONNECTION_MAP` editable set diverges from `/graph/schema` (the automated "10-step checklist"). Surfacing it found 3 real gaps; fixed 2: `color_strip_source.input_source_id` + `processing_template_id` are now drag-editable (processed-strip wiring; `apply_update` is partial-safe). The 3rd — `device.default_css_processing_template_id` — is intentionally NOT drag-editable (the device PUT route isn't partial-safe; a one-field PUT could null the URL) and is in the drift-check exclude set. Also broadened `_availableMatches` to hide any slot the target entity doesn't expose (subtype-accurate; refs are always-emitted so empty slots stay wireable). Review also caught a **dead `output_target.picture_source_id` slot** (no output target stores it — not a field/schema, never emitted) — removed from both registries + `buildGraph`. - [x] **Comprehensive review pass (4 subagents: backend/frontend-core/orchestrator/security).** Findings fixed: - **CRITICAL (security):** `GET /api/v1/graph` leaked plaintext **webhook tokens** (`asdict` recursed `Automation.rules[].token`, an auth-equivalent secret). Fixed with **field-projection** — `serialize_entity_for_graph()` / `graph_field_roots()` project each entity to only `{id, name, subtype, reference-roots}`; secrets can't survive. Added a structural regression test asserting no projection root is secret-bearing for any kind (drift-proof boundary) + a token-drop test. - MEDIUM: added missing `value_source.clock_id` (AnimatedColorValueSource → sync_clock) to the backend registry for topology/dependents completeness (drift-excluded on the frontend — value-source PUT needs a `source_type` discriminator, so it's editor-only). - MEDIUM/LOW: `CSS.escape` on the markIssues id selector; grouped/clarified `_DRIFT_EXCLUDE`; fixed the stale `_availableMatches` JSDoc; documented the `checkSchemaDrift` forward-reference. Orchestrator + frontend-core + security: APPROVE. - Verification: `npm --prefix server run typecheck` + `run build` clean; ruff clean; graph backend tests 35 pass; full backend suite green. ~8 code-review passes, all CRITICAL/HIGH findings fixed. ### Left to do (deferred) - [x] **BindableColor slots** — CHECKED, decision: keep read-only (won't fix). Value sources are scalar-only (`ValueStream.get_value() -> float`) and every colour consumer (`color_strip/single.py`, `effect_stream.py`) reads the static RGB via `bcolor()`, ignoring `source_id`. So a value_source cannot drive a colour — wiring `color`/`color_peak`/… would be a dead binding. Documented in `api/graph_schema.py` next to the BindableColor entries. (Would only become viable if a colour-producing value-source type is added.) - [~] **B4 — delete the frontend `CONNECTION_MAP` duplication.** - [x] **Foundation done:** the backend schema now carries an authoritative `editable` flag per field (`is_editable()` in `api/graph_schema.py`, mirroring the frontend `_isEditable`: top-level refs + single-level BindableFloat slots; NOT colour/list/double-nested). `validate-connection` is hardened to reject any non-editable field (was list-only). `editable` is surfaced in `/graph/schema`. - [ ] **Remaining (the refactor):** frontend fetches `/graph/schema` on load and derives connection metadata + edges from it (port the `extract_refs` dot-path/list grammar to TS), keeping only a tiny `kind → {endpoint, cache}` write-routing table; then delete the field-level `CONNECTION_MAP` + the `buildGraph` edge loops (graph-connections.ts / graph-layout.ts). Removes the 10-step sync checklist in `contexts/graph-editor.md`. **A backend apply-write endpoint is NOT required** — keep the proven per-entity PUT. Risk: regressing drag-connect/bindable; keep a dev drift-check (frontend editable set vs `/graph/schema`) during the transition. Note: frontend `CONNECTION_MAP` also has inert `ha_source_id`/`gradient_id` entries (no graph node kind) — drop them, the backend schema already omits them. - [ ] **D6 — blueprint import/instantiate.** Export exists; the apply half (serialize a selected subgraph's topology + entities, re-import with id remapping, conflict handling) is large and data-integrity-sensitive (see Data Migration Policy in CLAUDE.md). Scope as its own feature. - [ ] **List-slot editing** (composite `layers[]`, mapped `zones[]`, scene preset `targets[]`) — needs an element index in the write + validate paths (`validate_connection` currently rejects list fields). Edit via entity modal for now. ### Notes / decisions - The backend `CONNECTION_SCHEMA` (`api/graph_schema.py`) is the authoritative superset; it already declares the bindable + list + value_source-chain edges. The frontend `CONNECTION_MAP` still owns write-routing (endpoint/cache) — that's the only reason it survives (see B4). - Bindable edges render dashed (`.graph-edge-nested`) but ARE editable — the dashed style intentionally distinguishes value bindings from structural edges. - `validate-connection` and `dependents` fail **open/safe** on the frontend so the graph keeps working against an older server without these endpoints.