alexei.dolgolyov
25c613c5cb
- entity CRUD via fire_entity_event choke point (name resolved/sanitized; deletes pass name explicitly) - auth: failures + WS session establishment (no tokens logged); per-IP audit-record throttle - device: online/offline (health), discovered/lost (zeroconf), ADB connect/disconnect - capture/system: target start-stop, scenes, playlists, automations, backup/restore, update, restart, calibration, settings - security hardening: sanitize_display strips control/NUL/ANSI/newlines from untrusted strings; malformed-IPv6 origin guard - 129 instrumentation tests (incl. secret-leak, log-injection, throttle, best-effort) + autouse throttle-reset fixture
71 lines
6.7 KiB
Markdown
71 lines
6.7 KiB
Markdown
# CONTEXT — Activity / Audit Log
|
||
|
||
Living scratchpad for the feature. The orchestrator updates this between phases. Tier-2
|
||
context (survives across phases; graduates to CLAUDE.md only if it's a lasting project truth).
|
||
|
||
## Config (from approval)
|
||
|
||
- **Mode:** Automated · **Execution:** Orchestrator · **Strategy:** Incremental
|
||
- **Base/merge target:** `master` · **Branch:** `feature/activity-log` · **Branch point:** `17dd2e0`
|
||
- Final merge ALWAYS requires user approval (even in Automated mode).
|
||
|
||
## Product decisions (locked)
|
||
|
||
- Placement: BOTH a top-level **Activity** tab AND a Dashboard **Recent Activity** widget,
|
||
plus a Settings retention panel.
|
||
- Scope: all four categories — entity CRUD, auth, device connect/disconnect, capture & system.
|
||
- Detail: **action metadata only** (no before/after diffs).
|
||
- Durability: **export on demand (CSV/JSON)** + existing whole-DB backup. **No** separate
|
||
backup subsystem.
|
||
- WebUI work uses the `frontend-design` skill (Phases 5–6).
|
||
|
||
## Key codebase facts (verified during planning)
|
||
|
||
- `fire_entity_event(entity_type, action, entity_id)` @ `api/dependencies.py:202` — central
|
||
hook, called by every entity route synchronously in-request; has `_deps` store access.
|
||
- `ProcessorManager.fire_event(dict)` / `subscribe_events()` @ `core/processing/processor_manager.py`
|
||
back `/api/v1/events/ws` (`api/routes/output_targets_control.py:206`). `fire_event` does
|
||
`put_nowait` (no `call_soon_threadsafe`) — fine for the existing consumer; recorder marshals.
|
||
- Frontend realtime: `core/events-ws.ts` re-dispatches `server:<type>`; `_ALLOWED_SERVER_EVENT_TYPES`
|
||
(line 39) is parity-checked by `tests/test_events_ws_parity.py`. Must add `activity_logged`.
|
||
- Actor: `request.state.auth_label` set in `api/auth.py` (129 authenticated, 83 anonymous).
|
||
- Storage: `Database` singleton (`storage/database.py`, single conn, RLock, WAL,
|
||
`synchronous=FULL`, `get_setting/set_setting`). `BaseSqliteStore` loads ALL rows to memory →
|
||
AVOID for the log. Migrations: `storage/data_migrations.py` (`ALL_MIGRATIONS`, idempotent).
|
||
- Backup/restore is **whole-DB** (`Database.backup_to`/`restore_from`, no STORE_MAP allowlist)
|
||
→ the new `activity_log` table is auto-covered. No `system.py` STORE_MAP edit needed.
|
||
- Background-engine pattern: `core/backup/auto_backup.py` (start/stop loop, `_prune`, settings).
|
||
- Thread-marshal precedent: `utils/log_broadcaster.py` (`ensure_loop` + `call_soon_threadsafe`).
|
||
- Device seams: `device_health_changed` (`core/processing/device_health.py`, on transition) +
|
||
`device_discovered`/`device_lost` (`core/devices/discovery_watcher.py`, **zeroconf thread**).
|
||
- **No** API-key create/rotate/revoke routes exist (only `GET /system/api-keys`) → those auth
|
||
events are DESCOPED.
|
||
- Existing **Log Viewer** (`utils/log_broadcaster.py`) = ephemeral debug-log tail; the audit
|
||
log is a different, persistent, structured feature. Differentiate; do not duplicate.
|
||
|
||
## Frozen contracts (fill as phases complete)
|
||
|
||
- ActivityLogEntry fields / dict shape: **frozen** — see phase-1-storage.md Handoff section. 11 fields: `id`, `ts`, `category`, `action`, `severity`, `actor`, `message`, `entity_type`, `entity_id`, `entity_name`, `metadata`. `seq` is DB-only (not on dataclass).
|
||
- ActivityLogFilters shape: **frozen** — 8 optional fields: `categories`, `severities`, `actor`, `entity_type`, `entity_id`, `since`, `until`, `message_like`. See phase-1-storage.md Handoff.
|
||
- recorder.record(...) signature + actor ContextVar import path: **frozen** — see phase-2-recorder-retention.md Handoff section. Signature: `record(category, action, *, severity="info", actor=None, entity_type=None, entity_id=None, entity_name=None, message, metadata=None, _bypass_enabled=False)`. ContextVar: `from ledgrab.core.activity_log.context import current_actor`. Module accessor: `from ledgrab.core.activity_log.recorder import get_module_recorder`. Event payload: `{"type": "activity_logged", "entry": {11-field dict with ts as ISO string, metadata as dict}}`. DI getters: `get_activity_recorder()`, `get_activity_log_repo()`, `get_activity_log_retention_engine()`.
|
||
- API endpoints + query params + page envelope + settings bounds: _(Phase 4 handoff)_
|
||
|
||
## Failed approaches / rejected designs
|
||
|
||
- Buffered async-writer subsystem (asyncio.Queue): REJECTED — unsafe from the zeroconf thread
|
||
and adds a shutdown-flush ordering hazard. Using direct synchronous-on-loop writes with
|
||
`call_soon_threadsafe` marshaling instead (simpler + correct).
|
||
- Using `BaseSqliteStore` for the log: REJECTED — loads all rows into memory.
|
||
- Separate activity-log backup subsystem: REJECTED — whole-DB backup already covers the table;
|
||
export-on-demand is the portability story.
|
||
|
||
## Deferred / open
|
||
|
||
- Setup-scaffold first-run noise suppression (batch/suppress flag) — deferred (nice-to-have).
|
||
|
||
## Phase progress notes
|
||
|
||
Phase 1 landed (2026-06-09): `activity_log.py` (dataclass + enums + filters + codec), `AddActivityLogTableMigration` (`002_add_activity_log`) appended to `ALL_MIGRATIONS`, `ActivityLogRepository` (record/query/count/prune/clear/iter_export), 41 new tests — all green. Full suite 2226 passed, 0 failed. Schema and method signatures frozen in phase-1-storage.md Handoff. Gotcha: `Database.execute` takes a positional tuple — use `?` placeholders (not `:name`), otherwise Python 3.14 will raise `ProgrammingError`.
|
||
Phase 2 landed (2026-06-09): `core/activity_log/` package (`context.py`, `recorder.py`, `retention.py`, `__init__.py`); actor ContextVar set in `api/auth.py` (both branches); `ActivityLogRetentionEngine` mirroring AutoBackupEngine; full wiring in `main.py` (repo at module level, recorder+engine in lifespan, `server.shutting_down` first shutdown action, engine stop before db.close); DI getters in `api/dependencies.py`; `activity_logged` added to `_ALLOWED_SERVER_EVENT_TYPES` in `events-ws.ts`; `set_module_recorder` exposes recorder to non-DI sites; 24 new tests — all green. Full suite 2309 passed, 2 skipped, 0 failed. Ruff clean.
|
||
Phase 3 landed (2026-06-09): instrumented all four categories — entity CRUD via `fire_entity_event` choke-point (`dependencies.py`), auth failures + WS session in `auth.py`, device online/offline in `device_health.py`, device discovered/lost in `discovery_watcher.py`, ADB connect/disconnect in `system_settings.py`, capture start/stop (individual + bulk) in `output_targets_control.py`, scene/playlist/automation activate in their respective route/engine files, backup/restore/delete + restart/shutdown/update/calibration/settings in `backup.py`/`update.py`/`calibration.py`; all 11 entity delete handlers pass `entity_name` to `fire_entity_event`; 22 new tests (security: token never in any field, explicitly asserted) — all green. Full suite 2369 passed, 2 skipped, 0 failed. Ruff clean. Complete (category, action) inventory in phase-3-instrumentation.md Handoff section.
|