From 1afe7d6fcc7524f7252b38f149deac6b71e96c98 Mon Sep 17 00:00:00 2001 From: "alexei.dolgolyov" Date: Tue, 9 Jun 2026 17:14:50 +0300 Subject: [PATCH] chore(activity-log): scaffold feature plan and phase subplans --- plans/activity-log/CONTEXT.md | 68 ++++++++++ plans/activity-log/PLAN.md | 116 +++++++++++++++++ plans/activity-log/phase-1-storage.md | 102 +++++++++++++++ .../phase-2-recorder-retention.md | 118 ++++++++++++++++++ plans/activity-log/phase-3-instrumentation.md | 108 ++++++++++++++++ plans/activity-log/phase-4-api.md | 79 ++++++++++++ plans/activity-log/phase-5-frontend-tab.md | 91 ++++++++++++++ .../phase-6-dashboard-settings.md | 74 +++++++++++ 8 files changed, 756 insertions(+) create mode 100644 plans/activity-log/CONTEXT.md create mode 100644 plans/activity-log/PLAN.md create mode 100644 plans/activity-log/phase-1-storage.md create mode 100644 plans/activity-log/phase-2-recorder-retention.md create mode 100644 plans/activity-log/phase-3-instrumentation.md create mode 100644 plans/activity-log/phase-4-api.md create mode 100644 plans/activity-log/phase-5-frontend-tab.md create mode 100644 plans/activity-log/phase-6-dashboard-settings.md diff --git a/plans/activity-log/CONTEXT.md b/plans/activity-log/CONTEXT.md new file mode 100644 index 0000000..d3b5ba6 --- /dev/null +++ b/plans/activity-log/CONTEXT.md @@ -0,0 +1,68 @@ +# CONTEXT — Activity / Audit Log + +Living scratchpad for the feature. The orchestrator updates this between phases. Tier-2 +context (survives across phases; graduates to CLAUDE.md only if it's a lasting project truth). + +## Config (from approval) + +- **Mode:** Automated · **Execution:** Orchestrator · **Strategy:** Incremental +- **Base/merge target:** `master` · **Branch:** `feature/activity-log` · **Branch point:** `17dd2e0` +- Final merge ALWAYS requires user approval (even in Automated mode). + +## Product decisions (locked) + +- Placement: BOTH a top-level **Activity** tab AND a Dashboard **Recent Activity** widget, + plus a Settings retention panel. +- Scope: all four categories — entity CRUD, auth, device connect/disconnect, capture & system. +- Detail: **action metadata only** (no before/after diffs). +- Durability: **export on demand (CSV/JSON)** + existing whole-DB backup. **No** separate + backup subsystem. +- WebUI work uses the `frontend-design` skill (Phases 5–6). + +## Key codebase facts (verified during planning) + +- `fire_entity_event(entity_type, action, entity_id)` @ `api/dependencies.py:202` — central + hook, called by every entity route synchronously in-request; has `_deps` store access. +- `ProcessorManager.fire_event(dict)` / `subscribe_events()` @ `core/processing/processor_manager.py` + back `/api/v1/events/ws` (`api/routes/output_targets_control.py:206`). `fire_event` does + `put_nowait` (no `call_soon_threadsafe`) — fine for the existing consumer; recorder marshals. +- Frontend realtime: `core/events-ws.ts` re-dispatches `server:`; `_ALLOWED_SERVER_EVENT_TYPES` + (line 39) is parity-checked by `tests/test_events_ws_parity.py`. Must add `activity_logged`. +- Actor: `request.state.auth_label` set in `api/auth.py` (129 authenticated, 83 anonymous). +- Storage: `Database` singleton (`storage/database.py`, single conn, RLock, WAL, + `synchronous=FULL`, `get_setting/set_setting`). `BaseSqliteStore` loads ALL rows to memory → + AVOID for the log. Migrations: `storage/data_migrations.py` (`ALL_MIGRATIONS`, idempotent). +- Backup/restore is **whole-DB** (`Database.backup_to`/`restore_from`, no STORE_MAP allowlist) + → the new `activity_log` table is auto-covered. No `system.py` STORE_MAP edit needed. +- Background-engine pattern: `core/backup/auto_backup.py` (start/stop loop, `_prune`, settings). +- Thread-marshal precedent: `utils/log_broadcaster.py` (`ensure_loop` + `call_soon_threadsafe`). +- Device seams: `device_health_changed` (`core/processing/device_health.py`, on transition) + + `device_discovered`/`device_lost` (`core/devices/discovery_watcher.py`, **zeroconf thread**). +- **No** API-key create/rotate/revoke routes exist (only `GET /system/api-keys`) → those auth + events are DESCOPED. +- Existing **Log Viewer** (`utils/log_broadcaster.py`) = ephemeral debug-log tail; the audit + log is a different, persistent, structured feature. Differentiate; do not duplicate. + +## Frozen contracts (fill as phases complete) + +- ActivityLogEntry fields / dict shape: _(Phase 1/2 handoff)_ +- ActivityLogFilters shape: _(Phase 1 handoff)_ +- recorder.record(...) signature + actor ContextVar import path: _(Phase 2 handoff)_ +- API endpoints + query params + page envelope + settings bounds: _(Phase 4 handoff)_ + +## Failed approaches / rejected designs + +- Buffered async-writer subsystem (asyncio.Queue): REJECTED — unsafe from the zeroconf thread + and adds a shutdown-flush ordering hazard. Using direct synchronous-on-loop writes with + `call_soon_threadsafe` marshaling instead (simpler + correct). +- Using `BaseSqliteStore` for the log: REJECTED — loads all rows into memory. +- Separate activity-log backup subsystem: REJECTED — whole-DB backup already covers the table; + export-on-demand is the portability story. + +## Deferred / open + +- Setup-scaffold first-run noise suppression (batch/suppress flag) — deferred (nice-to-have). + +## Phase progress notes + +_(Orchestrator appends a short note per phase: what landed, commit sha, any warnings.)_ diff --git a/plans/activity-log/PLAN.md b/plans/activity-log/PLAN.md new file mode 100644 index 0000000..67819cd --- /dev/null +++ b/plans/activity-log/PLAN.md @@ -0,0 +1,116 @@ +# Feature: Activity / Audit Log + +**Branch:** `feature/activity-log` +**Base branch:** `master` (merge target) +**Branch point:** `17dd2e02bab4d00a93479eb6af1a8c6ddc0c7224` (use for clean review diffs) +**Created:** 2026-06-09 +**Status:** 🟡 In Progress +**Strategy:** Incremental +**Mode:** Automated +**Execution:** Orchestrator +**Remote:** origin → https://git.dolgolyov-family.by/alexei.dolgolyov/ledgrab.git + +## Summary + +A persistent, queryable audit log of meaningful LedGrab actions, surfaced in the WebUI. +Captures four categories — entity CRUD, authentication, device connect/disconnect, and +capture & system events — as **action-metadata-only** records (who/what/when + entity +type/name/id + a human-readable message + small structured metadata; **no before/after +diffs**). Surfaced as a dedicated top-level **Activity** tab with smart filtering + live +updates, a compact **Recent Activity** widget on the Dashboard, and a **Settings** panel +for retention. Durability rides on the existing whole-DB `ledgrab.db` backup; portability +is an on-demand CSV/JSON **export** (no separate backup subsystem). + +## Design pillars (the load-bearing decisions) + +1. **Dedicated `activity_log` table + repository — NOT `BaseSqliteStore`.** That base loads + every row into an in-memory cache and uses a generic `id/name/data` blob — wrong for an + append-heavy, unbounded log. We use a purpose-built indexed table with query-on-demand + keyset pagination. +2. **Central choke-point instrumentation.** `fire_entity_event()` (`api/dependencies.py:202`) + is already called by every entity route on create/update/delete and has `_deps` access to + resolve names. The recorder hooks there for all entity CRUD. Non-entity events get explicit + `recorder.record(...)` calls. +3. **Actor via `ContextVar`.** Set inside `verify_api_key` (next to `request.state.auth_label`), + default `"system"`, reset per-request. The recorder reads it without threading actor + through every call. +4. **Direct synchronous write on the event-loop thread** (no separate buffered-writer + subsystem — simpler, and the request already did a `synchronous=FULL` entity write). + Cross-thread callers (zeroconf discovery thread) marshal via `loop.call_soon_threadsafe`, + mirroring `utils/log_broadcaster.py`. The "server shutting down" event is recorded as the + FIRST action in the lifespan shutdown block, before any teardown. +5. **Reuse the existing realtime bus.** A new `activity_logged` event over `/api/v1/events/ws` + (one `events-ws.ts` allowlist entry + `test_events_ws_parity.py` update). No new socket. +6. **Never log secrets.** API-key *tokens* are never stored — only labels/ids. +7. **Differentiate from the existing Log Viewer.** `utils/log_broadcaster.py` is an ephemeral + 500-line debug-log tail. The audit log is persistent, structured, semantic. Cross-link in + the Settings panel; never duplicate. + +## Build & Test Commands + +- **Build (frontend):** `cd server && npm run build` +- **Type-check (TS):** `cd server && npx tsc --noEmit` +- **Test:** `cd server && py -3.13 -m pytest tests/ --no-cov -q` +- **Lint (Python):** `cd server && ruff check src/ tests/ --fix` +- **Events parity test (load-bearing for P2):** included in pytest (`tests/test_events_ws_parity.py`) + +> Scope checks to the files actually edited: backend phases run ruff + pytest; frontend-only +> phases run `tsc --noEmit` + `npm run build` (no pytest/ruff). Phases touching both run both. + +## Phases + +- [ ] Phase 1: Storage — model, migration, repository [domain: data] → [subplan](./phase-1-storage.md) +- [ ] Phase 2: Recorder, actor context, retention, lifecycle [domain: backend] → [subplan](./phase-2-recorder-retention.md) +- [ ] Phase 3: Event instrumentation (4 categories) [domain: backend] → [subplan](./phase-3-instrumentation.md) +- [ ] Phase 4: REST API — query/filter/export/settings/clear [domain: backend] → [subplan](./phase-4-api.md) +- [ ] Phase 5: Frontend — Activity tab + smart filtering + live updates [domain: frontend] → [subplan](./phase-5-frontend-tab.md) +- [ ] Phase 6: Dashboard widget + Settings panel + docs [domain: frontend] → [subplan](./phase-6-dashboard-settings.md) + +## Parallelizable Phase Groups (Orchestrator mode only) + +- Phases 3 and 4 are **parallelizable only after the schema is frozen at end of Phase 2**. + Both depend on the P2 recorder/schema; P4 registers the router in `api/__init__.py`, P3 + edits `dependencies.py`/`auth.py` (different files, shared schema contract). To keep the + Automated run simple and low-risk, they run **sequentially** (3 → 4) unless time pressure + warrants worktree-isolated parallelism. Phases 5 → 6 are sequential (6 reuses P5 formatters + and the feature module). + +## Phase Progress Log + +| Phase | Domain | Status | Review | Build | Committed | +|-------|--------|--------|--------|-------|-----------| +| Phase 1: Storage | data | ⬜ Not Started | ⬜ | ⬜ | ⬜ | +| Phase 2: Recorder/Retention | backend | ⬜ Not Started | ⬜ | ⬜ | ⬜ | +| Phase 3: Instrumentation | backend | ⬜ Not Started | ⬜ | ⬜ | ⬜ | +| Phase 4: REST API | backend | ⬜ Not Started | ⬜ | ⬜ | ⬜ | +| Phase 5: Frontend tab | frontend | ⬜ Not Started | ⬜ | ⬜ | ⬜ | +| Phase 6: Dashboard/Settings | frontend | ⬜ Not Started | ⬜ | ⬜ | ⬜ | + +## Outstanding Warnings + +| Phase | Warning | Severity | Status (open / resolved / accepted) | +|-------|---------|----------|-------------------------------------| +| | | | | + +## Final Review + +- [ ] Comprehensive code review +- [ ] Security review (auth/PII-in-logs/secrets/log-injection — triggered) +- [ ] All Outstanding Warnings resolved or consciously accepted +- [ ] Full build passes (`npm run build` + `tsc --noEmit`) +- [ ] Full test suite passes (`pytest`) +- [ ] Merged to `master` + +## Amendment Log + +_(Filled in if the plan is amended mid-implementation.)_ + +- 2026-06-09: Plan reviewer (pre-implementation) → ⚠️ with 3 Critical Gaps, all resolved + before Phase 1: (G1) descoped non-existent API-key mutation events; (G2) dropped the + buffered-writer subsystem for a direct synchronous-on-loop write with `call_soon_threadsafe` + marshaling for thread-origin events; (G3) record "server shutting down" first in the shutdown + block (no buffer to flush). Concerns folded in: device events via `device_health_changed` + + `device_discovered/_lost`; actor ContextVar in `verify_api_key`; name-on-delete passed + explicitly; settings-audit scoped + self-key excluded; parity allowlist before emit site. + Adopted suggestions: rowid keyset tiebreaker; log the disable action. Deferred: setup-scaffold + noise suppression. diff --git a/plans/activity-log/phase-1-storage.md b/plans/activity-log/phase-1-storage.md new file mode 100644 index 0000000..5b13255 --- /dev/null +++ b/plans/activity-log/phase-1-storage.md @@ -0,0 +1,102 @@ +# Phase 1: Storage — model, migration, repository + +**Status:** ⬜ Not Started +**Parent plan:** [PLAN.md](./PLAN.md) +**Domain:** data + +## Objective + +Create the persistent foundation for the audit log: an `ActivityLogEntry` dataclass, an +additive idempotent SQLite migration that creates a dedicated indexed `activity_log` table, +and a purpose-built `ActivityLogRepository` (NOT `BaseSqliteStore`) supporting append, +keyset-paginated filtered query, count, time/count-based prune, and streaming export. + +## Tasks + +- [ ] Create `server/src/ledgrab/storage/activity_log.py`: + - `ActivityCategory` and `ActivitySeverity` string enums (or `Literal` unions used as + constants). Categories: `auth`, `device`, `entity`, `capture`, `system`. Severities: + `info`, `warning`, `error`. + - `@dataclass ActivityLogEntry` with fields: `id: str` (e.g. `al_`), `ts: datetime` + (UTC, server-assigned), `category: str`, `action: str`, `severity: str`, `actor: str`, + `entity_type: str | None`, `entity_id: str | None`, `entity_name: str | None`, + `message: str`, `metadata: dict` (small JSON; default empty). Provide `to_row()` / + `from_row()` (column tuple/dict ↔ dataclass; `metadata` JSON-encoded; `ts` isoformat). +- [ ] Add migration to `server/src/ledgrab/storage/data_migrations.py`: + - New `DataMigration` subclass `AddActivityLogTableMigration` with unique `name` + (next sequential id, e.g. `"NNN_add_activity_log"` — match existing naming) and + `apply(conn)` creating `activity_log` with an INTEGER PRIMARY KEY AUTOINCREMENT `seq` + (monotonic keyset tiebreaker) plus columns: `id TEXT UNIQUE NOT NULL`, `ts TEXT NOT NULL`, + `category TEXT NOT NULL`, `action TEXT NOT NULL`, `severity TEXT NOT NULL`, + `actor TEXT NOT NULL`, `entity_type TEXT`, `entity_id TEXT`, `entity_name TEXT`, + `message TEXT NOT NULL`, `metadata TEXT NOT NULL DEFAULT '{}'`. + - Indexes: `(ts DESC, seq DESC)` (primary keyset/sort), `category`, `severity`, `actor`, + `(entity_type, entity_id)`. Use `CREATE TABLE/INDEX IF NOT EXISTS` for idempotency. + - Append the instance to `ALL_MIGRATIONS` (never reorder existing entries). +- [ ] Create `server/src/ledgrab/storage/activity_log_repository.py`: + - `class ActivityLogRepository` taking `db: Database` (NOT subclassing `BaseSqliteStore`). + - `record(entry: ActivityLogEntry) -> None`: single parameterized INSERT via + `db.execute(...)` (auto-commit). The `seq` is DB-assigned. **Caller guarantees this runs + on the event-loop thread** (see Phase 2 — cross-thread marshaling lives in the recorder). + - `query(filters: ActivityLogFilters, *, before_seq: int | None, limit: int) -> list[ActivityLogEntry]`: + keyset pagination `WHERE seq < ? ORDER BY seq DESC LIMIT ?` plus optional filters — + `category IN (...)`, `severity IN (...)`, `actor = ?`, `entity_type = ?`, `entity_id = ?`, + `ts >= ?` / `ts <= ?`, `message LIKE ?` (free-text, `%q%`, escaped). All parameterized. + - `count(filters) -> int`. + - `prune(*, before_ts: datetime | None, max_entries: int | None) -> int`: delete rows older + than `before_ts`, and/or trim to the newest `max_entries` by `seq`. Returns rows deleted. + - `clear() -> int`: delete all rows (used by the API clear endpoint; the clear action is + itself audited by the recorder, not here). Returns rows deleted. + - `iter_export(filters) -> Iterator[ActivityLogEntry]`: cursor-based streaming for export + (does not load all rows into memory). + - Define a small `ActivityLogFilters` dataclass (all-optional fields) in the repository or + `activity_log.py` and reuse it across query/count/prune/export. +- [ ] Unit tests in `server/tests/storage/test_activity_log_repository.py`: + - insert + read back round-trip (incl. metadata JSON, UTC ts); + - filter by each dimension (category/severity/actor/entity/date/free-text); + - keyset pagination stability across two pages with same-`ts` rows (seq tiebreaker); + - prune by age and by max_entries; + - clear; count; export iterator yields all matching rows; + - migration idempotency (constructing the repo twice / running migrations twice is safe). + +## Files to Modify/Create + +- `server/src/ledgrab/storage/activity_log.py` — new: dataclass + enums + filters + row codec +- `server/src/ledgrab/storage/data_migrations.py` — modify: add migration + append to `ALL_MIGRATIONS` +- `server/src/ledgrab/storage/activity_log_repository.py` — new: repository +- `server/tests/storage/test_activity_log_repository.py` — new: unit tests + +## Acceptance Criteria + +- `activity_log` table + indexes created idempotently on startup (running migrations twice is a no-op). +- Query is keyset-paginated and index-backed; a 10k-row table never loads fully into memory. +- Pagination is stable when many rows share the same millisecond `ts` (uses `seq` tiebreaker). +- `prune` removes by age AND by max-entry cap; `clear` empties the table; `export` streams. +- All filters use parameterized SQL (no string interpolation of user input). +- New unit tests pass; `ruff check` clean; existing tests still green. + +## Notes + +- Reference patterns: `storage/database.py` (`execute`, `transaction`, `get_setting`), + `storage/data_migrations.py` (`DataMigration`, `MigrationRunner`, `ALL_MIGRATIONS`), + `storage/sync_clock.py` (dataclass `to_dict`/`from_dict` style). +- 🔒 **Migration-safety addendum (data domain):** this migration is purely additive (new + table) — no rename, no field/key/file move, no data movement → no data-loss risk. Still + idempotent (`IF NOT EXISTS`). Rollback = drop the table; no user data is transformed. +- Do NOT wire the repository into `main.py` or `dependencies.py` here — that is Phase 2. +- `Database`'s connection is created with the existing threading model; the repository must + not assume it can be called from arbitrary threads. Thread marshaling is Phase 2's job. + +## Review Checklist + +- [ ] All tasks completed +- [ ] Code follows project conventions (dataclass codec style, migration naming) +- [ ] No unintended side effects (no startup wiring yet) +- [ ] Build passes (ruff + pytest) +- [ ] Tests pass (new + existing) + +## Handoff to Next Phase + + diff --git a/plans/activity-log/phase-2-recorder-retention.md b/plans/activity-log/phase-2-recorder-retention.md new file mode 100644 index 0000000..cc9fea0 --- /dev/null +++ b/plans/activity-log/phase-2-recorder-retention.md @@ -0,0 +1,118 @@ +# Phase 2: Recorder, actor context, retention, lifecycle + +**Status:** ⬜ Not Started +**Parent plan:** [PLAN.md](./PLAN.md) +**Domain:** backend + +## Objective + +Build the runtime layer over the Phase 1 repository: a thread-safe `ActivityRecorder` facade +that persists an entry AND pushes a live `activity_logged` event; an actor `ContextVar` +populated by the auth layer; a background `ActivityLogRetentionEngine` mirroring +`AutoBackupEngine`; and the `main.py`/`dependencies.py` wiring (init, DI getter, retention +start/stop, shutdown ordering). After this phase the audit log records nothing yet (no call +sites) — that is Phase 3 — but the full machinery is live and unit-tested. + +## Tasks + +- [ ] Create `server/src/ledgrab/core/activity_log/__init__.py` and + `server/src/ledgrab/core/activity_log/recorder.py`: + - `ActivityRecorder(repo: ActivityLogRepository, processor_manager, *, loop=None)`. + - `record(category, action, *, severity="info", actor=None, entity_type=None, + entity_id=None, entity_name=None, message, metadata=None) -> None`: + - resolve `actor` from the actor `ContextVar` when not supplied, default `"system"`; + - build an `ActivityLogEntry` (id `al_`, `ts=datetime.now(timezone.utc)`); + - **thread-safe write:** if called on the event loop thread, write inline via + `repo.record(entry)` then fire the live event; if called from another thread (zeroconf + discovery), marshal the whole write+emit onto the loop via + `loop.call_soon_threadsafe(...)`. Capture the loop lazily (mirror + `utils/log_broadcaster.py:ensure_loop`/`call_soon_threadsafe`). Never raise into the + caller — audit recording is best-effort and must not break the audited action; log + failures at `warning`. + - live push: `processor_manager.fire_event({"type": "activity_logged", "entry": entry_as_dict})`. + - Provide a tiny helper to serialize an entry to the same dict shape the API returns + (reuse in Phase 4 / frontend). + - `enabled` flag honored: when retention settings say `enabled=false`, `record()` is a + no-op — EXCEPT the "audit log disabled" event itself, which must be recorded before the + flag takes effect (see retention engine). +- [ ] Actor `ContextVar`: + - Add `current_actor: ContextVar[str]` (module-level, e.g. in `core/activity_log/context.py` + or `api/auth.py`). In `verify_api_key` (`api/auth.py`), set it next to the existing + `request.state.auth_label = ...` (both the authenticated label and the `"anonymous"` + branch). Default `"system"` when unset. Ensure no cross-request leakage (set on every + auth evaluation). +- [ ] Create `server/src/ledgrab/core/activity_log/retention.py`: + - `ActivityLogRetentionEngine(repo, db, recorder)` mirroring `core/backup/auto_backup.py`: + `_load_settings()`/`_save_settings()` via `db.get_setting("activity_log")` / + `db.set_setting("activity_log", {...})`, `DEFAULT_SETTINGS = {"enabled": True, + "max_days": 90, "max_entries": 20000}`. + `async start()` → spawn `_retention_loop()` (`asyncio.create_task`); loop sleeps a sane + interval (e.g. hourly) then calls `repo.prune(before_ts=now-max_days, max_entries=...)`. + `async stop()` → cancel + await task. `get_settings()` / `async update_settings(...)` + that persist and apply (changing `enabled` is logged via the recorder BEFORE disabling). +- [ ] Wiring: + - `main.py`: instantiate `activity_log_repo = ActivityLogRepository(db)` (module level near + other stores); in `lifespan` startup build `activity_recorder` + `activity_log_retention_engine`, + pass to `init_dependencies(...)`, and `await activity_log_retention_engine.start()`. + - In `lifespan` **shutdown**: record a `system` / `server_shutting_down` event via the + recorder as the **first** shutdown action (before engines/db close), then + `await _bounded("activity_log_retention.stop", activity_log_retention_engine.stop(), timeout=0.5)`. + - `api/dependencies.py`: add `activity_recorder` + `activity_log_repo` + + `activity_log_retention_engine` to `_deps`, parameters to `init_dependencies`, and + getters `get_activity_recorder()`, `get_activity_log_repo()`, + `get_activity_log_retention_engine()`. +- [ ] Realtime allowlist (order matters — do allowlist FIRST so the parity test stays green): + - Add `'activity_logged'` to `_ALLOWED_SERVER_EVENT_TYPES` in + `server/src/ledgrab/static/js/core/events-ws.ts` (+ a one-line comment naming the source). + - Confirm `tests/test_events_ws_parity.py` passes with the new emit type. +- [ ] Unit tests `server/tests/core/test_activity_recorder.py` + + `test_activity_log_retention.py`: + - recorder persists an entry AND calls `fire_event` with `type=="activity_logged"`; + - actor resolves from ContextVar; defaults to `"system"`; failure in repo doesn't raise; + - cross-thread `record()` (call from a `threading.Thread`) routes through the loop and persists; + - retention prunes per settings; settings round-trip via db; disabling logs the disable event. + +## Files to Modify/Create + +- `server/src/ledgrab/core/activity_log/__init__.py` — new +- `server/src/ledgrab/core/activity_log/recorder.py` — new +- `server/src/ledgrab/core/activity_log/context.py` — new (actor ContextVar) *(or place in auth.py)* +- `server/src/ledgrab/core/activity_log/retention.py` — new +- `server/src/ledgrab/api/auth.py` — modify: set actor ContextVar in `verify_api_key` +- `server/src/ledgrab/main.py` — modify: instantiate, wire lifespan start/shutdown +- `server/src/ledgrab/api/dependencies.py` — modify: `_deps`, `init_dependencies`, getters +- `server/src/ledgrab/static/js/core/events-ws.ts` — modify: allowlist `activity_logged` +- `server/tests/core/test_activity_recorder.py` — new +- `server/tests/core/test_activity_log_retention.py` — new + +## Acceptance Criteria + +- Recorder persists + fires `activity_logged`; never raises into callers; thread-safe from + non-loop threads. +- Actor ContextVar populated by auth; default `"system"`; no cross-request leakage. +- Retention engine starts/stops cleanly in lifespan; prunes by age + count; settings persist. +- `server_shutting_down` is recorded before teardown; no lost-on-graceful-shutdown entries. +- `test_events_ws_parity.py` green (allowlist updated). Existing tests still green; `ruff` clean. + +## Notes + +- Reference: `core/backup/auto_backup.py` (engine shape, settings persistence, `_bounded` + shutdown in `main.py`), `utils/log_broadcaster.py` (`ensure_loop`, `call_soon_threadsafe` + thread marshaling), `core/processing/processor_manager.py:247` (`fire_event`). +- **Do not add any instrumentation call sites in this phase** — only the machinery. Phase 3 + adds the `record(...)` calls. (Intermediate commit emits nothing; that is fine and green.) +- Freeze the `ActivityLogEntry` dict shape here — Phase 4 (API response) and Phase 5 + (frontend `entry`) consume it. + +## Review Checklist + +- [ ] All tasks completed +- [ ] Code follows project conventions (engine/DI patterns) +- [ ] No unintended side effects (no call sites yet; lifespan order correct) +- [ ] Build passes (ruff + pytest, incl. parity test) +- [ ] Tests pass (new + existing) + +## Handoff to Next Phase + + diff --git a/plans/activity-log/phase-3-instrumentation.md b/plans/activity-log/phase-3-instrumentation.md new file mode 100644 index 0000000..9b24d4e --- /dev/null +++ b/plans/activity-log/phase-3-instrumentation.md @@ -0,0 +1,108 @@ +# Phase 3: Event instrumentation (4 categories) + +**Status:** ⬜ Not Started +**Parent plan:** [PLAN.md](./PLAN.md) +**Domain:** backend · 🔒 security-sensitive (security reviewer triggers) + +## Objective + +Emit audit records at the real call sites for all four categories, using the Phase 2 recorder. +Maximize coverage via the central `fire_entity_event` choke point; add explicit +`recorder.record(...)` calls for non-entity events. Never log secrets. + +## Tasks + +### Entity CRUD (via the choke point) +- [ ] In `api/dependencies.py`, extend `fire_entity_event` to ALSO record an audit entry: + - Signature gains an optional `entity_name: str | None = None`. + - For `created`/`updated`: if `entity_name` not supplied, best-effort resolve from the + matching store in `_deps` keyed by `entity_type` (entity still present). For `deleted`: + **do not** resolve post-hoc — rely on the explicit `entity_name` passed by the handler + (deletes are the most important; a name-less delete entry is unacceptable). + - Map `action` → severity (`info`), category `entity`. Build a human message + (e.g. `"Target 'Desk' updated"`). Read actor from the ContextVar. + - Recording is best-effort (never break the entity operation). +- [ ] Update entity **delete** handlers to pass `entity_name` into `fire_entity_event` + (the entity object is already loaded for the 404 check). Cover the representative/most-used + entities at minimum: output targets, sync clocks, devices, picture/audio/color-strip + sources, automations, scene presets/playlists, templates, gradients. (Create/update can rely + on hook resolution but pass the name where trivially available.) + +### Authentication (DESCOPED: no key create/rotate/revoke — those routes don't exist) +- [ ] In `api/auth.py`, record: + - auth **failures**: missing/invalid Bearer token (HTTP), rejected LAN-without-keys, rejected + WS origin (4403), WS auth handshake failure (4401). Category `auth`, severity `warning`. + Include the caller IP/label and the reason in `metadata` — **never** the attempted token. + - WS **session establishment** (successful `accept_and_authenticate_ws`): category `auth`, + severity `info`, actor = authenticated label. + - (Do NOT record per-request HTTP auth *success* — too frequent.) + +### Device connect/disconnect (use existing discrete seams) +- [ ] Hook `device_health_changed` (`core/processing/device_health.py`, fired only on + `online != prev_online`) → record online/offline transition. Category `device`, + severity `info` (online) / `warning` (offline). +- [ ] Hook `device_discovered` / `device_lost` (`core/devices/discovery_watcher.py`, **runs on + the zeroconf thread** → recorder must marshal to the loop, which Phase 2 handles). Category + `device`. +- [ ] ADB connect/disconnect (`api/routes/system_settings.py:adb_connect/adb_disconnect`). + +### Capture & system events (explicit record calls) +- [ ] Target processing start/stop + bulk (`api/routes/output_targets_control.py`). +- [ ] Scene activation (`scene_presets.py:activate_scene_preset`), playlist start/stop + (`scene_playlists.py`), automation activate/deactivate (`automation_engine.py`). +- [ ] System: backup create/restore/delete (`backup.py`), update apply/dismiss (`update.py`), + restart/shutdown (`backup.py`), calibration start/stop/cancel (`calibration.py`). +- [ ] Settings changes: scope to high-value settings only (auto-backup, update, shutdown + action). **Exclude the activity-log's own `"activity_log"` settings key** to avoid + self-referential churn. + +### Tests +- [ ] `server/tests/test_activity_instrumentation.py` (or per-area): + - representative entity create/update/delete produces a record with correct category/actor/ + name (incl. a delete carrying its name); + - an auth failure produces a `warning` record and the token never appears in any field; + - a device health transition and a discovery event produce records; + - a capture start and a backup/restore produce records. + +## Files to Modify/Create + +- `server/src/ledgrab/api/dependencies.py` — modify: `fire_entity_event` records + `entity_name` +- entity **delete** route handlers under `api/routes/` — modify: pass `entity_name` +- `server/src/ledgrab/api/auth.py` — modify: auth-failure + WS-session records +- `server/src/ledgrab/core/processing/device_health.py` — modify: online/offline record +- `server/src/ledgrab/core/devices/discovery_watcher.py` — modify: discovered/lost record +- `server/src/ledgrab/api/routes/system_settings.py` — modify: ADB + settings records +- `server/src/ledgrab/api/routes/output_targets_control.py` — modify: start/stop records +- `server/src/ledgrab/api/routes/{scene_presets,scene_playlists,backup,update,calibration}.py` — modify +- `server/src/ledgrab/core/automations/automation_engine.py` — modify: activate/deactivate records +- `server/tests/test_activity_instrumentation.py` — new + +## Acceptance Criteria + +- All four categories emit records at the named sites; entity deletes carry the entity name. +- API-key tokens / secrets never appear in any audit field (test-enforced). +- Recording never breaks the audited action (best-effort; failures swallowed + logged). +- Actor is the authenticated label for request-originated events, `"system"` for engine/thread + events. New + existing tests green; `ruff` clean. + +## Notes + +- Get the recorder via the Phase 2 DI getter; for engine/thread sites that lack DI, use the + module singleton/accessor Phase 2 exposes. +- Keep messages human-readable and localized-agnostic (English source strings; the frontend + renders structured fields, not server message translation — message is a fallback/summary). +- This is the security-sensitive phase — the security reviewer runs here AND at final review. + +## Review Checklist + +- [ ] All tasks completed +- [ ] Code follows project conventions +- [ ] No unintended side effects (audited actions still succeed on recorder failure) +- [ ] No secrets logged (token never recorded) — explicitly verified +- [ ] Build passes (ruff + pytest) +- [ ] Tests pass (new + existing) + +## Handoff to Next Phase + + diff --git a/plans/activity-log/phase-4-api.md b/plans/activity-log/phase-4-api.md new file mode 100644 index 0000000..1cd15e5 --- /dev/null +++ b/plans/activity-log/phase-4-api.md @@ -0,0 +1,79 @@ +# Phase 4: REST API — query / filter / export / settings / clear + +**Status:** ⬜ Not Started +**Parent plan:** [PLAN.md](./PLAN.md) +**Domain:** backend + +## Objective + +Expose the audit log over the REST API: a filtered, keyset-paginated list endpoint; a +streaming CSV/JSON export honoring the same filters; retention settings get/update; and a +destructive clear. Apply the project's auth posture (stricter auth on export + clear). + +## Tasks + +- [ ] `server/src/ledgrab/api/schemas/activity_log.py` (Pydantic): + - `ActivityLogEntryResponse` (matches the frozen Phase 2 entry dict shape). + - `ActivityLogPageResponse` { `entries: list[...]`, `next_before_seq: int | None`, + `total: int` (optional/over filters), `has_more: bool` }. + - `ActivityLogSettingsResponse` / `UpdateActivityLogSettingsRequest` + (`enabled`, `max_days`, `max_entries`) with validation bounds. +- [ ] `server/src/ledgrab/api/routes/activity_log.py` — `APIRouter(prefix="/api/v1/activity-log")`: + - `GET ""` — list. Query params: `categories`, `severities`, `actor`, `entity_type`, + `entity_id`, `since`/`until` (ISO), `q` (free-text), `before_seq` (cursor), `limit` + (default 50, capped e.g. 200). `AuthRequired`. Maps params → `ActivityLogFilters`, + calls `repo.query(...)`, returns page envelope. + - `GET "/export"` — streaming export. Same filters; `format=csv|json`. Uses + `StreamingResponse` over `repo.iter_export(...)`. **`require_authenticated()`** (may + contain IPs/labels). Sets `Content-Disposition` with a timestamped filename. + - `GET "/settings"` / `PUT "/settings"` — retention settings via the retention engine. + `AuthRequired`; updates apply immediately. + - `DELETE ""` — clear all entries. **`require_authenticated()`**. The clear is itself + audited (recorder records a `system`/`activity_log_cleared` entry AFTER the wipe, so the + log shows who cleared it and when). + - Register the router in `server/src/ledgrab/api/__init__.py` (aggregator). +- [ ] API tests `server/tests/api/routes/test_activity_log_api.py`: + - list returns entries; each filter narrows results; `before_seq` cursor paginates without + overlap/gaps; `limit` cap enforced; + - export CSV and JSON both stream and honor filters; export requires authentication + (401 for loopback-anonymous when keys configured); + - settings get/update round-trip + validation rejects out-of-range; + - clear empties the log, requires auth, and leaves exactly one post-clear audit entry. + +## Files to Modify/Create + +- `server/src/ledgrab/api/schemas/activity_log.py` — new +- `server/src/ledgrab/api/routes/activity_log.py` — new +- `server/src/ledgrab/api/__init__.py` — modify: register router +- `server/tests/api/routes/test_activity_log_api.py` — new + +## Acceptance Criteria + +- List is filterable on every dimension and keyset-paginated (stable, no dupes/gaps). +- Export streams CSV + JSON, honors filters, and requires authentication. +- Settings get/update works and validates bounds; changes take effect immediately. +- Clear requires authentication and is itself audited. +- New + existing tests green; `ruff` clean. + +## Notes + +- Auth helpers: `AuthRequired` dependency for normal endpoints; `require_authenticated()` for + export + clear (pattern: backup download / secret reveal). See `api/auth.py` + `server/CLAUDE.md`. +- Follow the existing route/schema conventions (one schema file per entity, router registered + in `api/__init__.py`). Reference `api/routes/backup.py` for settings-style GET/PUT + a + `StreamingResponse`/download pattern. +- Reuse the entry→dict serializer from Phase 2 to keep the response shape single-sourced. +- Backup/restore: no `STORE_MAP` change needed — backup is whole-DB; the table is auto-covered. + +## Review Checklist + +- [ ] All tasks completed +- [ ] Code follows project conventions (router registration, schema-per-entity, auth posture) +- [ ] No unintended side effects +- [ ] Build passes (ruff + pytest) +- [ ] Tests pass (new + existing) + +## Handoff to Next Phase + + diff --git a/plans/activity-log/phase-5-frontend-tab.md b/plans/activity-log/phase-5-frontend-tab.md new file mode 100644 index 0000000..0e2a509 --- /dev/null +++ b/plans/activity-log/phase-5-frontend-tab.md @@ -0,0 +1,91 @@ +# Phase 5: Frontend — Activity tab + smart filtering + live updates + +**Status:** ⬜ Not Started +**Parent plan:** [PLAN.md](./PLAN.md) +**Domain:** frontend · uses the `frontend-design` skill + +## Objective + +Build the dedicated top-level **Activity** tab: a read-only, smart-filterable, +keyset-paginated log viewer with an entry detail view, live-append of new events, and export. +This is a viewer (Dashboard-style), NOT a CRUD card section. + +## Tasks + +- [ ] `core/ui.ts`: add `formatTimestamp(isoOrMs)` (Today/Yesterday/Date · HH:MM, i18n-aware) + and `formatRelativeTime(isoOrMs)` ("2m ago"), with `tabular-nums` styling guidance for the + list. Reuse existing `time.*` i18n key conventions. +- [ ] `features/activity-log.ts`: + - `export async function loadActivityLog()` — fetch first page from + `GET /activity-log` (via `fetchWithAuth`), render the toolbar + list into the panel. + - **Smart filter toolbar:** category (multi, IconSelect/chips), severity (chips), actor + (EntitySelect/text), entity type, date range, free-text search (debounced). Quick presets: + Today / Errors / Auth / Devices. Filters drive server-side query params (no client-side + filtering of a partial page). Re-query on change; reset cursor. + - **List:** one row per entry — severity icon, category badge, relative time (title=absolute), + actor, message, entity crosslink (use `navigateToCard(...)` when the referenced entity is + resolvable). Keyset "load more" (or infinite scroll) using `next_before_seq`. + - **Detail:** expandable row / drawer showing full metadata JSON, exact timestamp, ids. + - **Live append:** `document.addEventListener('server:activity_logged', e => …)` — prepend + the new entry if it passes the active filters; show a subtle "new" affordance. (Depends on + the Phase 2 allowlist entry — already shipped.) + - **Export button:** triggers `GET /activity-log/export?format=…` with current filters via an + authed blob download (use `fetchWithAuth` → blob URL, per frontend.md auth rules). + - Empty / loading / error states; re-render on `languageChanged`. +- [ ] Tab wiring: + - `core/tab-registry.ts`: add `activity_log: { loadFnName: 'loadActivityLog', autoRefresh: false }`. + - `templates/index.html`: sidebar tab button (`data-tab`, `switchTab('activity_log')`, + history/clock SVG icon, `data-i18n`) + `
`. + - `app.ts`: import + `Object.assign(window, { loadActivityLog, … })`; `global.d.ts` decls. +- [ ] Icons: add a history/audit icon to `core/icon-paths.ts` + `core/icons.ts`; severity icons + (info/warning/error) reuse existing constants where possible. +- [ ] i18n: add `activity_log.*` keys to `static/locales/{en,ru,zh}.json` (title, filter labels, + category/severity names, column labels, presets, empty/error, export, "N entries"). +- [ ] Tutorials: add an Activity-tab step to the getting-started tour in + `features/tutorials.ts` + `tour.*` keys in all 3 locales. + +## Files to Modify/Create + +- `server/src/ledgrab/static/js/core/ui.ts` — modify: timestamp/relative-time formatters +- `server/src/ledgrab/static/js/features/activity-log.ts` — new: the viewer +- `server/src/ledgrab/static/js/core/tab-registry.ts` — modify: register tab +- `server/src/ledgrab/templates/index.html` — modify: tab button + panel +- `server/src/ledgrab/static/js/app.ts` — modify: import + window globals +- `server/src/ledgrab/static/js/global.d.ts` — modify: window decls +- `server/src/ledgrab/static/js/core/icon-paths.ts` / `core/icons.ts` — modify: icons +- `server/src/ledgrab/static/locales/{en,ru,zh}.json` — modify: i18n keys +- `server/src/ledgrab/static/js/features/tutorials.ts` — modify: tour step +- `server/src/ledgrab/static/css/*` — modify/new: list + toolbar styling (follow base.css vars) + +## Acceptance Criteria + +- New **Activity** tab loads, lists entries, and paginates via keyset "load more". +- Filters hit server-side query params; quick presets work; free-text is debounced. +- New events append live via `server:activity_logged` and respect active filters. +- Export downloads CSV/JSON with auth, honoring current filters. +- Fully localized (en/ru/zh); empty/loading/error states; re-renders on language change. +- No plain ``; SVG icons only. + +## Notes + +- **Use the `frontend-design` skill** for the widget + settings panel layout. +- Reuse Phase 5's render helper and i18n namespace — no duplicated row markup or keys. +- Settings UI model: `features/settings.ts switchSettingsTab` + `modals/settings.html` rail. +- Frontend-only phase → run `tsc --noEmit` + `npm run build`; do NOT run pytest/ruff (docs + + TS only). +- Backup/restore cross-ref: no `STORE_MAP` edit needed (whole-DB backup covers the table) — + confirm nothing else (graph editor) needs syncing for this viewer (verified: not needed). + +## Review Checklist + +- [ ] All tasks completed +- [ ] Code follows frontend conventions (i18n ×3, icons, selectors, CSS vars, settings pattern) +- [ ] No unintended side effects +- [ ] Build passes (`tsc --noEmit` + `npm run build`) +- [ ] Manual smoke: widget live-updates + links; settings save/clear/export; docs updated + +## Handoff to Next Phase + +