diff --git a/plans/activity-log/CONTEXT.md b/plans/activity-log/CONTEXT.md deleted file mode 100644 index efdcb2d..0000000 --- a/plans/activity-log/CONTEXT.md +++ /dev/null @@ -1,73 +0,0 @@ -# CONTEXT — Activity / Audit Log - -Living scratchpad for the feature. The orchestrator updates this between phases. Tier-2 -context (survives across phases; graduates to CLAUDE.md only if it's a lasting project truth). - -## Config (from approval) - -- **Mode:** Automated · **Execution:** Orchestrator · **Strategy:** Incremental -- **Base/merge target:** `master` · **Branch:** `feature/activity-log` · **Branch point:** `17dd2e0` -- Final merge ALWAYS requires user approval (even in Automated mode). - -## Product decisions (locked) - -- Placement: BOTH a top-level **Activity** tab AND a Dashboard **Recent Activity** widget, - plus a Settings retention panel. -- Scope: all four categories — entity CRUD, auth, device connect/disconnect, capture & system. -- Detail: **action metadata only** (no before/after diffs). -- Durability: **export on demand (CSV/JSON)** + existing whole-DB backup. **No** separate - backup subsystem. -- WebUI work uses the `frontend-design` skill (Phases 5–6). - -## Key codebase facts (verified during planning) - -- `fire_entity_event(entity_type, action, entity_id)` @ `api/dependencies.py:202` — central - hook, called by every entity route synchronously in-request; has `_deps` store access. -- `ProcessorManager.fire_event(dict)` / `subscribe_events()` @ `core/processing/processor_manager.py` - back `/api/v1/events/ws` (`api/routes/output_targets_control.py:206`). `fire_event` does - `put_nowait` (no `call_soon_threadsafe`) — fine for the existing consumer; recorder marshals. -- Frontend realtime: `core/events-ws.ts` re-dispatches `server:`; `_ALLOWED_SERVER_EVENT_TYPES` - (line 39) is parity-checked by `tests/test_events_ws_parity.py`. Must add `activity_logged`. -- Actor: `request.state.auth_label` set in `api/auth.py` (129 authenticated, 83 anonymous). -- Storage: `Database` singleton (`storage/database.py`, single conn, RLock, WAL, - `synchronous=FULL`, `get_setting/set_setting`). `BaseSqliteStore` loads ALL rows to memory → - AVOID for the log. Migrations: `storage/data_migrations.py` (`ALL_MIGRATIONS`, idempotent). -- Backup/restore is **whole-DB** (`Database.backup_to`/`restore_from`, no STORE_MAP allowlist) - → the new `activity_log` table is auto-covered. No `system.py` STORE_MAP edit needed. -- Background-engine pattern: `core/backup/auto_backup.py` (start/stop loop, `_prune`, settings). -- Thread-marshal precedent: `utils/log_broadcaster.py` (`ensure_loop` + `call_soon_threadsafe`). -- Device seams: `device_health_changed` (`core/processing/device_health.py`, on transition) + - `device_discovered`/`device_lost` (`core/devices/discovery_watcher.py`, **zeroconf thread**). -- **No** API-key create/rotate/revoke routes exist (only `GET /system/api-keys`) → those auth - events are DESCOPED. -- Existing **Log Viewer** (`utils/log_broadcaster.py`) = ephemeral debug-log tail; the audit - log is a different, persistent, structured feature. Differentiate; do not duplicate. - -## Frozen contracts (fill as phases complete) - -- ActivityLogEntry fields / dict shape: **frozen** — see phase-1-storage.md Handoff section. 11 fields: `id`, `ts`, `category`, `action`, `severity`, `actor`, `message`, `entity_type`, `entity_id`, `entity_name`, `metadata`. `seq` is DB-only (not on dataclass). -- ActivityLogFilters shape: **frozen** — 8 optional fields: `categories`, `severities`, `actor`, `entity_type`, `entity_id`, `since`, `until`, `message_like`. See phase-1-storage.md Handoff. -- recorder.record(...) signature + actor ContextVar import path: **frozen** — see phase-2-recorder-retention.md Handoff section. Signature: `record(category, action, *, severity="info", actor=None, entity_type=None, entity_id=None, entity_name=None, message, metadata=None, _bypass_enabled=False)`. ContextVar: `from ledgrab.core.activity_log.context import current_actor`. Module accessor: `from ledgrab.core.activity_log.recorder import get_module_recorder`. Event payload: `{"type": "activity_logged", "entry": {11-field dict with ts as ISO string, metadata as dict}}`. DI getters: `get_activity_recorder()`, `get_activity_log_repo()`, `get_activity_log_retention_engine()`. -- API endpoints + query params + page envelope + settings bounds: **frozen** — see phase-4-api.md Handoff section. Endpoints: `GET /api/v1/activity-log` (list, AuthRequired), `GET /api/v1/activity-log/export` (stream CSV/JSON, require_authenticated), `GET|PUT /api/v1/activity-log/settings` (AuthRequired), `DELETE /api/v1/activity-log` (clear, require_authenticated). Page envelope: `entries`, `next_before_seq`, `has_more`, `total`. Settings fields: `enabled` (bool), `max_days` (0–3650), `max_entries` (0–10_000_000). Export: `?format=csv|json`. - -## Failed approaches / rejected designs - -- Buffered async-writer subsystem (asyncio.Queue): REJECTED — unsafe from the zeroconf thread - and adds a shutdown-flush ordering hazard. Using direct synchronous-on-loop writes with - `call_soon_threadsafe` marshaling instead (simpler + correct). -- Using `BaseSqliteStore` for the log: REJECTED — loads all rows into memory. -- Separate activity-log backup subsystem: REJECTED — whole-DB backup already covers the table; - export-on-demand is the portability story. - -## Deferred / open - -- Setup-scaffold first-run noise suppression (batch/suppress flag) — deferred (nice-to-have). - -## Phase progress notes - -Phase 1 landed (2026-06-09): `activity_log.py` (dataclass + enums + filters + codec), `AddActivityLogTableMigration` (`002_add_activity_log`) appended to `ALL_MIGRATIONS`, `ActivityLogRepository` (record/query/count/prune/clear/iter_export), 41 new tests — all green. Full suite 2226 passed, 0 failed. Schema and method signatures frozen in phase-1-storage.md Handoff. Gotcha: `Database.execute` takes a positional tuple — use `?` placeholders (not `:name`), otherwise Python 3.14 will raise `ProgrammingError`. -Phase 2 landed (2026-06-09): `core/activity_log/` package (`context.py`, `recorder.py`, `retention.py`, `__init__.py`); actor ContextVar set in `api/auth.py` (both branches); `ActivityLogRetentionEngine` mirroring AutoBackupEngine; full wiring in `main.py` (repo at module level, recorder+engine in lifespan, `server.shutting_down` first shutdown action, engine stop before db.close); DI getters in `api/dependencies.py`; `activity_logged` added to `_ALLOWED_SERVER_EVENT_TYPES` in `events-ws.ts`; `set_module_recorder` exposes recorder to non-DI sites; 24 new tests — all green. Full suite 2309 passed, 2 skipped, 0 failed. Ruff clean. -Phase 4 landed (2026-06-09): schemas (`api/schemas/activity_log.py`), routes (`api/routes/activity_log.py`: list/export/settings/clear), router registration in `api/__init__.py`, `get_seq_for_id` helper on `ActivityLogRepository`. 49 new tests — all green. Full suite 2486 passed, 2 skipped, 0 failed. Ruff clean. Pagination bug found and fixed (limit+1 probe must drop oldest row when has_more, not tail). -Phase 3 landed (2026-06-09): instrumented all four categories — entity CRUD via `fire_entity_event` choke-point (`dependencies.py`), auth failures + WS session in `auth.py`, device online/offline in `device_health.py`, device discovered/lost in `discovery_watcher.py`, ADB connect/disconnect in `system_settings.py`, capture start/stop (individual + bulk) in `output_targets_control.py`, scene/playlist/automation activate in their respective route/engine files, backup/restore/delete + restart/shutdown/update/calibration/settings in `backup.py`/`update.py`/`calibration.py`; all 11 entity delete handlers pass `entity_name` to `fire_entity_event`; 22 new tests (security: token never in any field, explicitly asserted) — all green. Full suite 2369 passed, 2 skipped, 0 failed. Ruff clean. Complete (category, action) inventory in phase-3-instrumentation.md Handoff section. -Phase 5 landed (2026-06-09): Activity tab frontend — `features/activity-log.ts` (loadActivityLog, filter toolbar with category/severity chips + presets + debounced search + date range + actor/entity-type text fields, keyset-paginated list, expandable detail drawer, live-prepend via `server:activity_logged`, authed CSV/JSON blob export); `core/ui.ts` additions (`formatTimestamp`, `formatRelativeTime`); `core/icon-paths.ts` + `core/icons.ts` additions (scrollText/audit, circleAlert, info, filter, xCircle); `core/tab-registry.ts` registered; `templates/index.html` tab button + panel; `app.ts` + `global.d.ts` wired; `static/css/activity-log.css` (precision-ledger design, category color rail, live-dot pulse, detail drawer, responsive breakpoints); all three locales updated (activity_log.* + time.* relative-time keys + tour.activity_log); `features/tutorials.ts` getting-started tour extended. `tsc --noEmit` clean, `npm run build` passes. Reusable helpers for Phase 6 documented in phase-5-frontend-tab.md Handoff section. -Phase 6 landed (2026-06-09): Dashboard "Recent Activity" widget (live SSE, View-all link, `.dal-*` CSS, loading/empty states) + Settings "Activity Log" panel (enabled toggle, max_days/max_entries, Save toast, authed CSV/JSON export, confirmed Clear, audit-vs-debug cross-links) + 32 i18n keys per locale + README "Activity Log" section. `tsc --noEmit` clean, `npm run build` passes. All six phases complete — feature ready for final review. diff --git a/plans/activity-log/PLAN.md b/plans/activity-log/PLAN.md deleted file mode 100644 index fe17718..0000000 --- a/plans/activity-log/PLAN.md +++ /dev/null @@ -1,141 +0,0 @@ -# Feature: Activity / Audit Log - -**Branch:** `feature/activity-log` -**Base branch:** `master` (merge target) -**Branch point:** `17dd2e02bab4d00a93479eb6af1a8c6ddc0c7224` (use for clean review diffs) -**Created:** 2026-06-09 -**Status:** 🟢 All phases complete — awaiting final review + merge -**Strategy:** Incremental -**Mode:** Automated -**Execution:** Orchestrator -**Remote:** origin → https://git.dolgolyov-family.by/alexei.dolgolyov/ledgrab.git - -## Summary - -A persistent, queryable audit log of meaningful LedGrab actions, surfaced in the WebUI. -Captures four categories — entity CRUD, authentication, device connect/disconnect, and -capture & system events — as **action-metadata-only** records (who/what/when + entity -type/name/id + a human-readable message + small structured metadata; **no before/after -diffs**). Surfaced as a dedicated top-level **Activity** tab with smart filtering + live -updates, a compact **Recent Activity** widget on the Dashboard, and a **Settings** panel -for retention. Durability rides on the existing whole-DB `ledgrab.db` backup; portability -is an on-demand CSV/JSON **export** (no separate backup subsystem). - -## Design pillars (the load-bearing decisions) - -1. **Dedicated `activity_log` table + repository — NOT `BaseSqliteStore`.** That base loads - every row into an in-memory cache and uses a generic `id/name/data` blob — wrong for an - append-heavy, unbounded log. We use a purpose-built indexed table with query-on-demand - keyset pagination. -2. **Central choke-point instrumentation.** `fire_entity_event()` (`api/dependencies.py:202`) - is already called by every entity route on create/update/delete and has `_deps` access to - resolve names. The recorder hooks there for all entity CRUD. Non-entity events get explicit - `recorder.record(...)` calls. -3. **Actor via `ContextVar`.** Set inside `verify_api_key` (next to `request.state.auth_label`), - default `"system"`, reset per-request. The recorder reads it without threading actor - through every call. -4. **Direct synchronous write on the event-loop thread** (no separate buffered-writer - subsystem — simpler, and the request already did a `synchronous=FULL` entity write). - Cross-thread callers (zeroconf discovery thread) marshal via `loop.call_soon_threadsafe`, - mirroring `utils/log_broadcaster.py`. The "server shutting down" event is recorded as the - FIRST action in the lifespan shutdown block, before any teardown. -5. **Reuse the existing realtime bus.** A new `activity_logged` event over `/api/v1/events/ws` - (one `events-ws.ts` allowlist entry + `test_events_ws_parity.py` update). No new socket. -6. **Never log secrets.** API-key *tokens* are never stored — only labels/ids. -7. **Differentiate from the existing Log Viewer.** `utils/log_broadcaster.py` is an ephemeral - 500-line debug-log tail. The audit log is persistent, structured, semantic. Cross-link in - the Settings panel; never duplicate. - -## Build & Test Commands - -- **Build (frontend):** `cd server && npm run build` -- **Type-check (TS):** `cd server && npx tsc --noEmit` -- **Test:** `cd server && py -3.13 -m pytest tests/ --no-cov -q` -- **Lint (Python):** `cd server && ruff check src/ tests/ --fix` -- **Events parity test (load-bearing for P2):** included in pytest (`tests/test_events_ws_parity.py`) - -> Scope checks to the files actually edited: backend phases run ruff + pytest; frontend-only -> phases run `tsc --noEmit` + `npm run build` (no pytest/ruff). Phases touching both run both. - -## Phases - -- [x] Phase 1: Storage — model, migration, repository [domain: data] → [subplan](./phase-1-storage.md) -- [x] Phase 2: Recorder, actor context, retention, lifecycle [domain: backend] → [subplan](./phase-2-recorder-retention.md) -- [x] Phase 3: Event instrumentation (4 categories) [domain: backend] → [subplan](./phase-3-instrumentation.md) -- [x] Phase 4: REST API — query/filter/export/settings/clear [domain: backend] → [subplan](./phase-4-api.md) -- [x] Phase 5: Frontend — Activity tab + smart filtering + live updates [domain: frontend] → [subplan](./phase-5-frontend-tab.md) -- [x] Phase 6: Dashboard widget + Settings panel + docs [domain: frontend] → [subplan](./phase-6-dashboard-settings.md) - -## Parallelizable Phase Groups (Orchestrator mode only) - -- Phases 3 and 4 are **parallelizable only after the schema is frozen at end of Phase 2**. - Both depend on the P2 recorder/schema; P4 registers the router in `api/__init__.py`, P3 - edits `dependencies.py`/`auth.py` (different files, shared schema contract). To keep the - Automated run simple and low-risk, they run **sequentially** (3 → 4) unless time pressure - warrants worktree-isolated parallelism. Phases 5 → 6 are sequential (6 reuses P5 formatters - and the feature module). - -## Phase Progress Log - -| Phase | Domain | Status | Review | Build | Committed | -|-------|--------|--------|--------|-------|-----------| -| Phase 1: Storage | data | ✅ Done | ✅ Passed | ✅ Passed | ✅ | -| Phase 2: Recorder/Retention | backend | ✅ Done | ✅ Passed | ✅ Passed | ✅ | -| Phase 3: Instrumentation | backend | ✅ Done | ✅ Passed | ✅ Passed | ✅ | -| Phase 4: REST API | backend | ✅ Done | ✅ Passed | ✅ Passed | ✅ | -| Phase 5: Frontend tab | frontend | ✅ Done | ✅ Passed | ✅ Passed (tsc+build) | ✅ | -| Phase 6: Dashboard/Settings | frontend | ✅ Done | ✅ Passed | ✅ Passed (tsc+build) | ✅ | - -## Outstanding Warnings - -| Phase | Warning | Severity | Status (open / resolved / accepted) | -|-------|---------|----------|-------------------------------------| -| 3 | Log injection via unauth mDNS device name/url into audit message | 🟠 High (security) | resolved — `sanitize_display` helper applied | -| 3 | Origin sanitizer missed spaces/NUL/ANSI | 🟠 High (security) | resolved — `sanitize_display` over netloc | -| 3 | Unauth auth-failure audit-write flood (no write-rate bound) | 🟠 High (security) | resolved — per-IP audit-record throttle (10s, capped) | -| 3 | Malformed-IPv6 Origin → urlparse ValueError into WS handler | 🟡 Warning | resolved — try/except guard | -| 3 | Throttle module-global state caused flaky test contamination | 🟡 Warning | resolved — autouse conftest reset fixture | -| 4 | Export held global DB write-lock across the stream (slow-client DoS) | 🟠 High (security) | resolved — chunked keyset export releases lock per batch | -| 4 | PUT /settings only AuthRequired → anon could disable auditing/prune trail | 🟠 High (security) | resolved — `require_authenticated` on settings PUT | -| 4 | CSV formula-injection missed leading TAB/CR | 🟡 Medium (security) | resolved — added `\t`/`\r` to guard | -| 4 | `total` count full-scans on every list request | 🔵 Low (perf) | accepted — bounded by retention; read-only; optional opt-in deferred | -| 5 | Inverted list ordering broke pagination + live-append | 🔴 Blocker | resolved — pages reversed to newest-first; re-review PASS | -| 5 | Attribute-context XSS (entity_name title + JSON.stringify onclick) | 🟡 Warning (security) | resolved — `_escapeAttr` + data-attr event delegation | -| 5 | Filter toolbar value= attrs not quote-escaped (new code) | 🟡 Warning (security) | resolved — `_escapeAttr` on q/actor/entity_type/since/until | -| 5 | Manual browser smoke test (tab loads, filters, live, export) | 🔵 Note | open — recommend at final review (server restart needed) | -| 6 | clearActivityLog() 401 path unreachable → silent failure | 🟡 Warning | resolved — `handle401:false` surfaces auth-required toast | -| 6 | Recent Activity widget dropped first live event when empty | 🔵 Note | resolved — empty→list transition on first live event | -| 6 | Widget outside dashboard layout-toggle/ordering system | 🔵 Note | accepted — deliberate (always-visible), collapse still works | -| Final | Entity-crosslink map keys mismatched backend entity_type (device/color_strip_source/audio_source) | 🟡 Warning | resolved — `_ENTITY_NAV` keys corrected + scene_playlist added | -| Final | Owner-authored names interpolated raw at some record sites | 🔵 Note (defense-in-depth) | resolved — `sanitize_display` applied uniformly | -| Manual test | Entry descriptions rendered server English (not localized) | 🟡 Warning (acceptance criterion) | resolved — client-side `localizeMessage` from structured fields + `activity_log.msg.*`/`entity_type.*` templates ×3 | -| Manual test | Redundant "Activity" header banner at top of tab | 🔵 Note | resolved — header block removed | -| Manual test | Recent Activity widget missing from Customize Dashboard | 🟡 Warning | resolved — registered as a first-class dashboard section (show/hide/reorder; pre-existing layouts preserved) | -| Manual test | Activity widget live event rebuilt the whole dashboard | 🟡 Warning (perf) | resolved — surgical list update; single listener with teardown | -| Manual test | Relative-time labels static (never tick) | 🟡 Warning | resolved — shared `ensureRelativeTimeTicker` (single 30s interval, visibility-aware) | -| Manual test | Dashboard fully rebuilt/jumped on entity edits (pre-existing `forceFullRender` wholesale innerHTML) | 🟡 Warning (UX) | resolved — section-level reconciliation (`_reconcileDynamicSections`): only changed sections replaced; widget live DOM + perf strip preserved | -| Manual test | Activity list columns slightly misaligned | 🔵 Note | resolved — CSS grid with fixed badge/actor columns | -| Manual test | Reconciler left orphan "no targets" node on empty→populated | 🟡 Warning (regression, caught in review) | resolved — sweep non-section top-level children | - -## Final Review - -- [ ] Comprehensive code review -- [ ] Security review (auth/PII-in-logs/secrets/log-injection — triggered) -- [ ] All Outstanding Warnings resolved or consciously accepted -- [ ] Full build passes (`npm run build` + `tsc --noEmit`) -- [ ] Full test suite passes (`pytest`) -- [ ] Merged to `master` - -## Amendment Log - -_(Filled in if the plan is amended mid-implementation.)_ - -- 2026-06-09: Plan reviewer (pre-implementation) → ⚠️ with 3 Critical Gaps, all resolved - before Phase 1: (G1) descoped non-existent API-key mutation events; (G2) dropped the - buffered-writer subsystem for a direct synchronous-on-loop write with `call_soon_threadsafe` - marshaling for thread-origin events; (G3) record "server shutting down" first in the shutdown - block (no buffer to flush). Concerns folded in: device events via `device_health_changed` + - `device_discovered/_lost`; actor ContextVar in `verify_api_key`; name-on-delete passed - explicitly; settings-audit scoped + self-key excluded; parity allowlist before emit site. - Adopted suggestions: rowid keyset tiebreaker; log the disable action. Deferred: setup-scaffold - noise suppression. diff --git a/plans/activity-log/phase-1-storage.md b/plans/activity-log/phase-1-storage.md deleted file mode 100644 index 0cb7518..0000000 --- a/plans/activity-log/phase-1-storage.md +++ /dev/null @@ -1,177 +0,0 @@ -# Phase 1: Storage — model, migration, repository - -**Status:** ✅ Done -**Parent plan:** [PLAN.md](./PLAN.md) -**Domain:** data - -## Objective - -Create the persistent foundation for the audit log: an `ActivityLogEntry` dataclass, an -additive idempotent SQLite migration that creates a dedicated indexed `activity_log` table, -and a purpose-built `ActivityLogRepository` (NOT `BaseSqliteStore`) supporting append, -keyset-paginated filtered query, count, time/count-based prune, and streaming export. - -## Tasks - -- [x] Create `server/src/ledgrab/storage/activity_log.py`: - - `ActivityCategory` and `ActivitySeverity` string enums (or `Literal` unions used as - constants). Categories: `auth`, `device`, `entity`, `capture`, `system`. Severities: - `info`, `warning`, `error`. - - `@dataclass ActivityLogEntry` with fields: `id: str` (e.g. `al_`), `ts: datetime` - (UTC, server-assigned), `category: str`, `action: str`, `severity: str`, `actor: str`, - `entity_type: str | None`, `entity_id: str | None`, `entity_name: str | None`, - `message: str`, `metadata: dict` (small JSON; default empty). Provide `to_row()` / - `from_row()` (column tuple/dict ↔ dataclass; `metadata` JSON-encoded; `ts` isoformat). -- [x] Add migration to `server/src/ledgrab/storage/data_migrations.py`: - - New `DataMigration` subclass `AddActivityLogTableMigration` with unique `name` - (next sequential id, e.g. `"NNN_add_activity_log"` — match existing naming) and - `apply(conn)` creating `activity_log` with an INTEGER PRIMARY KEY AUTOINCREMENT `seq` - (monotonic keyset tiebreaker) plus columns: `id TEXT UNIQUE NOT NULL`, `ts TEXT NOT NULL`, - `category TEXT NOT NULL`, `action TEXT NOT NULL`, `severity TEXT NOT NULL`, - `actor TEXT NOT NULL`, `entity_type TEXT`, `entity_id TEXT`, `entity_name TEXT`, - `message TEXT NOT NULL`, `metadata TEXT NOT NULL DEFAULT '{}'`. - - Indexes: `(ts DESC, seq DESC)` (primary keyset/sort), `category`, `severity`, `actor`, - `(entity_type, entity_id)`. Use `CREATE TABLE/INDEX IF NOT EXISTS` for idempotency. - - Append the instance to `ALL_MIGRATIONS` (never reorder existing entries). -- [x] Create `server/src/ledgrab/storage/activity_log_repository.py`: - - `class ActivityLogRepository` taking `db: Database` (NOT subclassing `BaseSqliteStore`). - - `record(entry: ActivityLogEntry) -> None`: single parameterized INSERT via - `db.execute(...)` (auto-commit). The `seq` is DB-assigned. **Caller guarantees this runs - on the event-loop thread** (see Phase 2 — cross-thread marshaling lives in the recorder). - - `query(filters: ActivityLogFilters, *, before_seq: int | None, limit: int) -> list[ActivityLogEntry]`: - keyset pagination `WHERE seq < ? ORDER BY seq DESC LIMIT ?` plus optional filters — - `category IN (...)`, `severity IN (...)`, `actor = ?`, `entity_type = ?`, `entity_id = ?`, - `ts >= ?` / `ts <= ?`, `message LIKE ?` (free-text, `%q%`, escaped). All parameterized. - - `count(filters) -> int`. - - `prune(*, before_ts: datetime | None, max_entries: int | None) -> int`: delete rows older - than `before_ts`, and/or trim to the newest `max_entries` by `seq`. Returns rows deleted. - - `clear() -> int`: delete all rows (used by the API clear endpoint; the clear action is - itself audited by the recorder, not here). Returns rows deleted. - - `iter_export(filters) -> Iterator[ActivityLogEntry]`: cursor-based streaming for export - (does not load all rows into memory). - - Define a small `ActivityLogFilters` dataclass (all-optional fields) in the repository or - `activity_log.py` and reuse it across query/count/prune/export. -- [x] Unit tests in `server/tests/storage/test_activity_log_repository.py`: - - insert + read back round-trip (incl. metadata JSON, UTC ts); - - filter by each dimension (category/severity/actor/entity/date/free-text); - - keyset pagination stability across two pages with same-`ts` rows (seq tiebreaker); - - prune by age and by max_entries; - - clear; count; export iterator yields all matching rows; - - migration idempotency (constructing the repo twice / running migrations twice is safe). - -## Files to Modify/Create - -- `server/src/ledgrab/storage/activity_log.py` — new: dataclass + enums + filters + row codec -- `server/src/ledgrab/storage/data_migrations.py` — modify: add migration + append to `ALL_MIGRATIONS` -- `server/src/ledgrab/storage/activity_log_repository.py` — new: repository -- `server/tests/storage/test_activity_log_repository.py` — new: unit tests - -## Acceptance Criteria - -- `activity_log` table + indexes created idempotently on startup (running migrations twice is a no-op). -- Query is keyset-paginated and index-backed; a 10k-row table never loads fully into memory. -- Pagination is stable when many rows share the same millisecond `ts` (uses `seq` tiebreaker). -- `prune` removes by age AND by max-entry cap; `clear` empties the table; `export` streams. -- All filters use parameterized SQL (no string interpolation of user input). -- New unit tests pass; `ruff check` clean; existing tests still green. - -## Notes - -- Reference patterns: `storage/database.py` (`execute`, `transaction`, `get_setting`), - `storage/data_migrations.py` (`DataMigration`, `MigrationRunner`, `ALL_MIGRATIONS`), - `storage/sync_clock.py` (dataclass `to_dict`/`from_dict` style). -- 🔒 **Migration-safety addendum (data domain):** this migration is purely additive (new - table) — no rename, no field/key/file move, no data movement → no data-loss risk. Still - idempotent (`IF NOT EXISTS`). Rollback = drop the table; no user data is transformed. -- Do NOT wire the repository into `main.py` or `dependencies.py` here — that is Phase 2. -- `Database`'s connection is created with the existing threading model; the repository must - not assume it can be called from arbitrary threads. Thread marshaling is Phase 2's job. - -## Review Checklist - -- [x] All tasks completed -- [x] Code follows project conventions (dataclass codec style, migration naming) -- [x] No unintended side effects (no startup wiring yet) -- [x] Build passes (ruff + pytest) -- [x] Tests pass (new + existing) - -## Handoff to Next Phase - -### ActivityLogEntry — final field list and dict shape - -```python -@dataclass -class ActivityLogEntry: - id: str # "al_" — caller-assigned - ts: datetime # UTC-aware; stored as ISO-8601 string in DB - category: str # ActivityCategory constant - action: str # verb-object label, e.g. "entity.created" - severity: str # ActivitySeverity constant - actor: str # API-key label or "system" - message: str # human-readable description - entity_type: str | None # e.g. "output_target" - entity_id: str | None # stable entity id - entity_name: str | None # name at time of event - metadata: dict # JSON-serialisable; default {} -``` - -`to_row()` returns a flat dict with 11 keys (same names); `metadata` is JSON string, `ts` is isoformat string. `seq` is NOT in `to_row()` — it is DB-assigned. - -### ActivityLogFilters — shape (all fields optional, default None) - -```python -@dataclass -class ActivityLogFilters: - categories: Sequence[str] | None # category IN (...) - severities: Sequence[str] | None # severity IN (...) - actor: str | None # exact match - entity_type: str | None # exact match - entity_id: str | None # exact match - since: datetime | None # ts >= since - until: datetime | None # ts <= until - message_like: str | None # LIKE %value% (escaped) -``` - -### Migration name used - -`"002_add_activity_log"` — appended as position [1] in `ALL_MIGRATIONS`. - -### ActivityLogRepository — exact method signatures - -```python -class ActivityLogRepository: - def __init__(self, db: Database) -> None - def record(self, entry: ActivityLogEntry) -> None - def query( - self, - filters: ActivityLogFilters, - *, - before_seq: int | None = None, - limit: int = 50, - ) -> list[ActivityLogEntry] - def count(self, filters: ActivityLogFilters | None = None) -> int - def prune( - self, - *, - before_ts: datetime | None = None, - max_entries: int | None = None, - ) -> int - def clear(self) -> int - def iter_export( - self, filters: ActivityLogFilters | None = None - ) -> Iterator[ActivityLogEntry] -``` - -### Key behavioural notes for Phase 2/3/4 - -- `record()` expects to be called from the event-loop thread (or with `Database` RLock already held). Phase 2 is responsible for thread marshaling via `loop.call_soon_threadsafe`. -- `query()` returns entries in **ascending chronological order within the page** (reversed internally from DESC fetch for display convenience). The smallest `seq` on a page is `page[0]`'s seq — pass that as `before_seq` for the next page. -- `count(None)` == `count(ActivityLogFilters())` — both count all rows. -- `prune(before_ts=X, max_entries=N)` applies both predicates independently (age prune first, then count cap). -- `iter_export` holds `db._lock` for the entire iteration. Phase 4 should stream the response and consume promptly. -- `ActivityLogCategory` and `ActivityLogSeverity` are plain classes with string class-attributes and an `ALL` tuple — NOT `enum.Enum`. -- Imports for Phase 2/3/4: - ```python - from ledgrab.storage.activity_log import ActivityLogEntry, ActivityLogFilters, ActivityCategory, ActivitySeverity - from ledgrab.storage.activity_log_repository import ActivityLogRepository - ``` diff --git a/plans/activity-log/phase-2-recorder-retention.md b/plans/activity-log/phase-2-recorder-retention.md deleted file mode 100644 index fd33c95..0000000 --- a/plans/activity-log/phase-2-recorder-retention.md +++ /dev/null @@ -1,196 +0,0 @@ -# Phase 2: Recorder, actor context, retention, lifecycle - -**Status:** ✅ Done -**Parent plan:** [PLAN.md](./PLAN.md) -**Domain:** backend - -## Objective - -Build the runtime layer over the Phase 1 repository: a thread-safe `ActivityRecorder` facade -that persists an entry AND pushes a live `activity_logged` event; an actor `ContextVar` -populated by the auth layer; a background `ActivityLogRetentionEngine` mirroring -`AutoBackupEngine`; and the `main.py`/`dependencies.py` wiring (init, DI getter, retention -start/stop, shutdown ordering). After this phase the audit log records nothing yet (no call -sites) — that is Phase 3 — but the full machinery is live and unit-tested. - -## Tasks - -- [x] Create `server/src/ledgrab/core/activity_log/__init__.py` and - `server/src/ledgrab/core/activity_log/recorder.py`: - - `ActivityRecorder(repo: ActivityLogRepository, processor_manager, *, loop=None)`. - - `record(category, action, *, severity="info", actor=None, entity_type=None, - entity_id=None, entity_name=None, message, metadata=None) -> None`: - - resolve `actor` from the actor `ContextVar` when not supplied, default `"system"`; - - build an `ActivityLogEntry` (id `al_`, `ts=datetime.now(timezone.utc)`); - - **thread-safe write:** if called on the event loop thread, write inline via - `repo.record(entry)` then fire the live event; if called from another thread (zeroconf - discovery), marshal the whole write+emit onto the loop via - `loop.call_soon_threadsafe(...)`. Capture the loop lazily (mirror - `utils/log_broadcaster.py:ensure_loop`/`call_soon_threadsafe`). Never raise into the - caller — audit recording is best-effort and must not break the audited action; log - failures at `warning`. - - live push: `processor_manager.fire_event({"type": "activity_logged", "entry": entry_as_dict})`. - - Provide a tiny helper to serialize an entry to the same dict shape the API returns - (reuse in Phase 4 / frontend). - - `enabled` flag honored: when retention settings say `enabled=false`, `record()` is a - no-op — EXCEPT the "audit log disabled" event itself, which must be recorded before the - flag takes effect (see retention engine). -- [x] Actor `ContextVar`: - - Add `current_actor: ContextVar[str]` (module-level, e.g. in `core/activity_log/context.py` - or `api/auth.py`). In `verify_api_key` (`api/auth.py`), set it next to the existing - `request.state.auth_label = ...` (both the authenticated label and the `"anonymous"` - branch). Default `"system"` when unset. Ensure no cross-request leakage (set on every - auth evaluation). -- [x] Create `server/src/ledgrab/core/activity_log/retention.py`: - - `ActivityLogRetentionEngine(repo, db, recorder)` mirroring `core/backup/auto_backup.py`: - `_load_settings()`/`_save_settings()` via `db.get_setting("activity_log")` / - `db.set_setting("activity_log", {...})`, `DEFAULT_SETTINGS = {"enabled": True, - "max_days": 90, "max_entries": 20000}`. - `async start()` → spawn `_retention_loop()` (`asyncio.create_task`); loop sleeps a sane - interval (e.g. hourly) then calls `repo.prune(before_ts=now-max_days, max_entries=...)`. - `async stop()` → cancel + await task. `get_settings()` / `async update_settings(...)` - that persist and apply (changing `enabled` is logged via the recorder BEFORE disabling). -- [x] Wiring: - - `main.py`: instantiate `activity_log_repo = ActivityLogRepository(db)` (module level near - other stores); in `lifespan` startup build `activity_recorder` + `activity_log_retention_engine`, - pass to `init_dependencies(...)`, and `await activity_log_retention_engine.start()`. - - In `lifespan` **shutdown**: record a `system` / `server_shutting_down` event via the - recorder as the **first** shutdown action (before engines/db close), then - `await _bounded("activity_log_retention.stop", activity_log_retention_engine.stop(), timeout=0.5)`. - - `api/dependencies.py`: add `activity_recorder` + `activity_log_repo` + - `activity_log_retention_engine` to `_deps`, parameters to `init_dependencies`, and - getters `get_activity_recorder()`, `get_activity_log_repo()`, - `get_activity_log_retention_engine()`. -- [x] Realtime allowlist (order matters — do allowlist FIRST so the parity test stays green): - - Add `'activity_logged'` to `_ALLOWED_SERVER_EVENT_TYPES` in - `server/src/ledgrab/static/js/core/events-ws.ts` (+ a one-line comment naming the source). - - Confirm `tests/test_events_ws_parity.py` passes with the new emit type. -- [x] Unit tests `server/tests/core/test_activity_recorder.py` + - `test_activity_log_retention.py`: - - recorder persists an entry AND calls `fire_event` with `type=="activity_logged"`; - - actor resolves from ContextVar; defaults to `"system"`; failure in repo doesn't raise; - - cross-thread `record()` (call from a `threading.Thread`) routes through the loop and persists; - - retention prunes per settings; settings round-trip via db; disabling logs the disable event. - -## Files to Modify/Create - -- `server/src/ledgrab/core/activity_log/__init__.py` — new -- `server/src/ledgrab/core/activity_log/recorder.py` — new -- `server/src/ledgrab/core/activity_log/context.py` — new (actor ContextVar) *(or place in auth.py)* -- `server/src/ledgrab/core/activity_log/retention.py` — new -- `server/src/ledgrab/api/auth.py` — modify: set actor ContextVar in `verify_api_key` -- `server/src/ledgrab/main.py` — modify: instantiate, wire lifespan start/shutdown -- `server/src/ledgrab/api/dependencies.py` — modify: `_deps`, `init_dependencies`, getters -- `server/src/ledgrab/static/js/core/events-ws.ts` — modify: allowlist `activity_logged` -- `server/tests/core/test_activity_recorder.py` — new -- `server/tests/core/test_activity_log_retention.py` — new - -## Acceptance Criteria - -- Recorder persists + fires `activity_logged`; never raises into callers; thread-safe from - non-loop threads. -- Actor ContextVar populated by auth; default `"system"`; no cross-request leakage. -- Retention engine starts/stops cleanly in lifespan; prunes by age + count; settings persist. -- `server_shutting_down` is recorded before teardown; no lost-on-graceful-shutdown entries. -- `test_events_ws_parity.py` green (allowlist updated). Existing tests still green; `ruff` clean. - -## Notes - -- Reference: `core/backup/auto_backup.py` (engine shape, settings persistence, `_bounded` - shutdown in `main.py`), `utils/log_broadcaster.py` (`ensure_loop`, `call_soon_threadsafe` - thread marshaling), `core/processing/processor_manager.py:247` (`fire_event`). -- **Do not add any instrumentation call sites in this phase** — only the machinery. Phase 3 - adds the `record(...)` calls. (Intermediate commit emits nothing; that is fine and green.) -- Freeze the `ActivityLogEntry` dict shape here — Phase 4 (API response) and Phase 5 - (frontend `entry`) consume it. - -## Review Checklist - -- [x] All tasks completed -- [x] Code follows project conventions (engine/DI patterns) -- [x] No unintended side effects (no call sites yet; lifespan order correct) -- [x] Build passes (ruff + pytest, incl. parity test) -- [x] Tests pass (new + existing) - -## Handoff to Next Phase - -### recorder.record(...) — final signature - -```python -recorder.record( - category: str, # ActivityCategory constant - action: str, # verb-object label - *, - severity: str = "info", # ActivitySeverity constant - actor: str | None = None, # resolved from current_actor ContextVar when None - entity_type: str | None = None, - entity_id: str | None = None, - entity_name: str | None = None, - message: str, - metadata: dict | None = None, - _bypass_enabled: bool = False, # internal: used by retention engine only -) -> None -``` - -### Actor ContextVar import path - -```python -from ledgrab.core.activity_log.context import current_actor -``` - -### Module accessor (for non-DI sites) - -```python -from ledgrab.core.activity_log.recorder import get_module_recorder, set_module_recorder -recorder = get_module_recorder() # returns ActivityRecorder | None -``` - -### entry_to_dict helper (for API response serialisation) - -```python -from ledgrab.core.activity_log.recorder import entry_to_dict -d = entry_to_dict(entry) # returns dict with 11 keys -``` - -### Frozen `activity_logged` event payload shape - -```python -{ - "type": "activity_logged", - "entry": { - "id": str, # "al_<8-hex>" - "ts": str, # ISO-8601 UTC string - "category": str, - "action": str, - "severity": str, - "actor": str, - "entity_type": str | None, - "entity_id": str | None, - "entity_name": str | None, - "message": str, - "metadata": dict, # real dict, not JSON string - } -} -``` - -### DI getter names (in `api/dependencies.py`) - -```python -from ledgrab.api.dependencies import ( - get_activity_recorder, - get_activity_log_repo, - get_activity_log_retention_engine, -) -``` - -### Notes for Phase 3 - -- Phase 3 instruments `fire_entity_event` in `api/dependencies.py` by calling - `get_module_recorder()` there (not via FastAPI DI — it's a plain function). -- The actor ContextVar is already set by `verify_api_key` before any route - handler runs, so entity events carry the correct actor automatically. -- `recorder.record(...)` never raises; Phase 3 call sites need no try/except. - -Phase 2 landed (2026-06-09): ActivityRecorder, actor ContextVar, ActivityLogRetentionEngine, -all wiring in main.py/dependencies.py/auth.py, activity_logged allowlist in events-ws.ts, -24 new tests — all green. Full suite 2309 passed, 2 skipped, 0 failed. Ruff clean. diff --git a/plans/activity-log/phase-3-instrumentation.md b/plans/activity-log/phase-3-instrumentation.md deleted file mode 100644 index efc2195..0000000 --- a/plans/activity-log/phase-3-instrumentation.md +++ /dev/null @@ -1,172 +0,0 @@ -# Phase 3: Event instrumentation (4 categories) - -**Status:** ✅ Done -**Parent plan:** [PLAN.md](./PLAN.md) -**Domain:** backend · 🔒 security-sensitive (security reviewer triggers) - -## Objective - -Emit audit records at the real call sites for all four categories, using the Phase 2 recorder. -Maximize coverage via the central `fire_entity_event` choke point; add explicit -`recorder.record(...)` calls for non-entity events. Never log secrets. - -## Tasks - -### Entity CRUD (via the choke point) -- [x] In `api/dependencies.py`, extend `fire_entity_event` to ALSO record an audit entry: - - Signature gains an optional `entity_name: str | None = None`. - - For `created`/`updated`: if `entity_name` not supplied, best-effort resolve from the - matching store in `_deps` keyed by `entity_type` (entity still present). For `deleted`: - **do not** resolve post-hoc — rely on the explicit `entity_name` passed by the handler - (deletes are the most important; a name-less delete entry is unacceptable). - - Map `action` → severity (`info`), category `entity`. Build a human message - (e.g. `"Target 'Desk' updated"`). Read actor from the ContextVar. - - Recording is best-effort (never break the entity operation). -- [x] Update entity **delete** handlers to pass `entity_name` into `fire_entity_event` - (the entity object is already loaded for the 404 check). Cover the representative/most-used - entities at minimum: output targets, sync clocks, devices, picture/audio/color-strip - sources, automations, scene presets/playlists, templates, gradients. (Create/update can rely - on hook resolution but pass the name where trivially available.) - -### Authentication (DESCOPED: no key create/rotate/revoke — those routes don't exist) -- [x] In `api/auth.py`, record: - - auth **failures**: missing/invalid Bearer token (HTTP), rejected LAN-without-keys, rejected - WS origin (4403), WS auth handshake failure (4401). Category `auth`, severity `warning`. - Include the caller IP/label and the reason in `metadata` — **never** the attempted token. - - WS **session establishment** (successful `accept_and_authenticate_ws`): category `auth`, - severity `info`, actor = authenticated label. - - (Do NOT record per-request HTTP auth *success* — too frequent.) - -### Device connect/disconnect (use existing discrete seams) -- [x] Hook `device_health_changed` (`core/processing/device_health.py`, fired only on - `online != prev_online`) → record online/offline transition. Category `device`, - severity `info` (online) / `warning` (offline). -- [x] Hook `device_discovered` / `device_lost` (`core/devices/discovery_watcher.py`, **runs on - the zeroconf thread** → recorder must marshal to the loop, which Phase 2 handles). Category - `device`. -- [x] ADB connect/disconnect (`api/routes/system_settings.py:adb_connect/adb_disconnect`). - -### Capture & system events (explicit record calls) -- [x] Target processing start/stop + bulk (`api/routes/output_targets_control.py`). -- [x] Scene activation (`scene_presets.py:activate_scene_preset`), playlist start/stop - (`scene_playlists.py`), automation activate/deactivate (`automation_engine.py`). -- [x] System: backup create/restore/delete (`backup.py`), update apply/dismiss (`update.py`), - restart/shutdown (`backup.py`), calibration start/stop/cancel (`calibration.py`). -- [x] Settings changes: scope to high-value settings only (auto-backup, update, shutdown - action). **Exclude the activity-log's own `"activity_log"` settings key** to avoid - self-referential churn. - -### Tests -- [x] `server/tests/test_activity_instrumentation.py` (or per-area): - - representative entity create/update/delete produces a record with correct category/actor/ - name (incl. a delete carrying its name); - - an auth failure produces a `warning` record and the token never appears in any field; - - a device health transition and a discovery event produce records; - - a capture start and a backup/restore produce records. - -## Files to Modify/Create - -- `server/src/ledgrab/api/dependencies.py` — modify: `fire_entity_event` records + `entity_name` -- entity **delete** route handlers under `api/routes/` — modify: pass `entity_name` -- `server/src/ledgrab/api/auth.py` — modify: auth-failure + WS-session records -- `server/src/ledgrab/core/processing/device_health.py` — modify: online/offline record -- `server/src/ledgrab/core/devices/discovery_watcher.py` — modify: discovered/lost record -- `server/src/ledgrab/api/routes/system_settings.py` — modify: ADB + settings records -- `server/src/ledgrab/api/routes/output_targets_control.py` — modify: start/stop records -- `server/src/ledgrab/api/routes/{scene_presets,scene_playlists,backup,update,calibration}.py` — modify -- `server/src/ledgrab/core/automations/automation_engine.py` — modify: activate/deactivate records -- `server/tests/test_activity_instrumentation.py` — new - -## Acceptance Criteria - -- All four categories emit records at the named sites; entity deletes carry the entity name. -- API-key tokens / secrets never appear in any audit field (test-enforced). -- Recording never breaks the audited action (best-effort; failures swallowed + logged). -- Actor is the authenticated label for request-originated events, `"system"` for engine/thread - events. New + existing tests green; `ruff` clean. - -## Notes - -- Get the recorder via the Phase 2 DI getter; for engine/thread sites that lack DI, use the - module singleton/accessor Phase 2 exposes. -- Keep messages human-readable and localized-agnostic (English source strings; the frontend - renders structured fields, not server message translation — message is a fallback/summary). -- This is the security-sensitive phase — the security reviewer runs here AND at final review. - -## Review Checklist - -- [x] All tasks completed -- [x] Code follows project conventions -- [x] No unintended side effects (audited actions still succeed on recorder failure) -- [x] No secrets logged (token never recorded) — explicitly verified -- [x] Build passes (ruff + pytest) -- [x] Tests pass (new + existing) - -## Handoff to Next Phase - -Phase 3 is complete. The following (category, action) pairs are now emitted, along with their -metadata keys, for Phase 4 to expose via query/filter and for Phase 5 quick-filter presets. - -### `entity` category - -| Action | Severity | Metadata keys | Notes | -|--------|----------|---------------|-------| -| `entity.created` | info | — | All entity types via `fire_entity_event` choke-point | -| `entity.updated` | info | — | All entity types; name resolved from store when not passed | -| `entity.deleted` | info | — | Name passed explicitly by delete handler before deletion | - -### `auth` category - -| Action | Severity | Metadata keys | Notes | -|--------|----------|---------------|-------| -| `auth.rejected` | warning | `reason` (str), `client` (str/IP) | Missing Bearer, invalid Bearer, LAN-no-keys, WS origin, WS auth timeout, invalid WS token | -| `auth.ws_connected` | info | `client` (str/IP) | Successful WS session established | - -### `device` category - -| Action | Severity | Metadata keys | Notes | -|--------|----------|---------------|-------| -| `device.online` | info | `latency_ms` (float) | Health monitor, transition only | -| `device.offline` | warning | `latency_ms` (float) | Health monitor, transition only | -| `device.discovered` | info | `url` (str), `device_type` (str) | Zeroconf discovery thread; recorder marshals to loop | -| `device.lost` | warning | `url` (str), `device_type` (str) | Zeroconf discovery thread | -| `device.adb_connected` | info | `address` (str) | ADB route success | -| `device.adb_disconnected` | info | `address` (str) | ADB route success | - -### `capture` category - -| Action | Severity | Metadata keys | Notes | -|--------|----------|---------------|-------| -| `capture.started` | info | — | Per target (individual + bulk) | -| `capture.stopped` | info | — | Per target (individual + bulk) | -| `scene.activated` | info | — | `scene_presets.py:activate_scene_preset` | -| `playlist.started` | info | — | `scene_playlists.py:start_scene_playlist` | -| `playlist.stopped` | info | — | `scene_playlists.py:stop_scene_playlist` | -| `automation.activated` | info | — | `automation_engine.py:_activate_automation`; actor="system" | -| `automation.deactivated` | info | — | `automation_engine.py:_deactivate_automation`; actor="system" | - -### `system` category - -| Action | Severity | Metadata keys | Notes | -|--------|----------|---------------|-------| -| `backup.created` | info | `filename` (str) | `backup.py:backup_config` | -| `backup.restored` | info | — | `backup.py:restore_config` | -| `backup.deleted` | info | `filename` (str) | `backup.py:delete_saved_backup` | -| `server.restarting` | info | — | `backup.py:restart_server` | -| `server.shutdown_requested` | info | — | `backup.py:shutdown_server` | -| `update.dismissed` | info | `version` (str) | `update.py:dismiss_update` | -| `update.applied` | info | `version` (str) | `update.py:apply_update` | -| `settings.changed` | info | `setting_key` (str) + setting-specific keys | `setting_key` values: `"auto_backup"`, `"update"`, `"shutdown_action"`. Activity-log own key excluded. | -| `calibration.started` | info | — | `calibration.py`; entity_type="device", entity_id=device_id | -| `calibration.stopped` | info | — | `calibration.py` | -| `calibration.cancelled` | info | — | `calibration.py` | - -### Implementation notes for Phase 4 - -- The `metadata` field is a JSON `TEXT` column. All keys above are scalars (str, float). -- Phase 4 filter `metadata_key` / `metadata_value` lookup, if added, can target `setting_key` - for settings-change filtering. -- `entity_type` is populated for entity CRUD and `calibration.started`. For auth/system/capture - events `entity_type` may be None. -- `entity_name` is always populated for `entity.deleted`; populated for CRUD create/update - when resolved; populated for most capture/system events where a name is meaningful. diff --git a/plans/activity-log/phase-4-api.md b/plans/activity-log/phase-4-api.md deleted file mode 100644 index d0d53eb..0000000 --- a/plans/activity-log/phase-4-api.md +++ /dev/null @@ -1,163 +0,0 @@ -# Phase 4: REST API — query / filter / export / settings / clear - -**Status:** ✅ Done -**Parent plan:** [PLAN.md](./PLAN.md) -**Domain:** backend - -## Objective - -Expose the audit log over the REST API: a filtered, keyset-paginated list endpoint; a -streaming CSV/JSON export honoring the same filters; retention settings get/update; and a -destructive clear. Apply the project's auth posture (stricter auth on export + clear). - -## Tasks - -- [x] `server/src/ledgrab/api/schemas/activity_log.py` (Pydantic): - - `ActivityLogEntryResponse` (matches the frozen Phase 2 entry dict shape). - - `ActivityLogPageResponse` { `entries: list[...]`, `next_before_seq: int | None`, - `total: int` (optional/over filters), `has_more: bool` }. - - `ActivityLogSettingsResponse` / `UpdateActivityLogSettingsRequest` - (`enabled`, `max_days`, `max_entries`) with validation bounds. -- [x] `server/src/ledgrab/api/routes/activity_log.py` — `APIRouter(prefix="/api/v1/activity-log")`: - - `GET ""` — list. Query params: `categories`, `severities`, `actor`, `entity_type`, - `entity_id`, `since`/`until` (ISO), `q` (free-text), `before_seq` (cursor), `limit` - (default 50, capped e.g. 200). `AuthRequired`. Maps params → `ActivityLogFilters`, - calls `repo.query(...)`, returns page envelope. - - `GET "/export"` — streaming export. Same filters; `format=csv|json`. Uses - `StreamingResponse` over `repo.iter_export(...)`. **`require_authenticated()`** (may - contain IPs/labels). Sets `Content-Disposition` with a timestamped filename. - - `GET "/settings"` / `PUT "/settings"` — retention settings via the retention engine. - `AuthRequired`; updates apply immediately. - - `DELETE ""` — clear all entries. **`require_authenticated()`**. The clear is itself - audited (recorder records a `system`/`activity_log_cleared` entry AFTER the wipe, so the - log shows who cleared it and when). - - Register the router in `server/src/ledgrab/api/__init__.py` (aggregator). -- [x] API tests `server/tests/api/routes/test_activity_log_api.py`: - - list returns entries; each filter narrows results; `before_seq` cursor paginates without - overlap/gaps; `limit` cap enforced; - - export CSV and JSON both stream and honor filters; export requires authentication - (401 for loopback-anonymous when keys configured); - - settings get/update round-trip + validation rejects out-of-range; - - clear empties the log, requires auth, and leaves exactly one post-clear audit entry. - -## Files to Modify/Create - -- `server/src/ledgrab/api/schemas/activity_log.py` — new -- `server/src/ledgrab/api/routes/activity_log.py` — new -- `server/src/ledgrab/api/__init__.py` — modify: register router -- `server/tests/api/routes/test_activity_log_api.py` — new - -## Acceptance Criteria - -- List is filterable on every dimension and keyset-paginated (stable, no dupes/gaps). -- Export streams CSV + JSON, honors filters, and requires authentication. -- Settings get/update works and validates bounds; changes take effect immediately. -- Clear requires authentication and is itself audited. -- New + existing tests green; `ruff` clean. - -## Notes - -- Auth helpers: `AuthRequired` dependency for normal endpoints; `require_authenticated()` for - export + clear (pattern: backup download / secret reveal). See `api/auth.py` + `server/CLAUDE.md`. -- Follow the existing route/schema conventions (one schema file per entity, router registered - in `api/__init__.py`). Reference `api/routes/backup.py` for settings-style GET/PUT + a - `StreamingResponse`/download pattern. -- Reuse the entry→dict serializer from Phase 2 to keep the response shape single-sourced. -- Backup/restore: no `STORE_MAP` change needed — backup is whole-DB; the table is auto-covered. - -## Review Checklist - -- [x] All tasks completed -- [x] Code follows project conventions (router registration, schema-per-entity, auth posture) -- [x] No unintended side effects -- [x] Build passes (ruff + pytest) -- [x] Tests pass (new + existing) - -## Handoff to Next Phase - -### Endpoint paths - -| Method | Path | Auth | -|--------|------|------| -| GET | `/api/v1/activity-log` | AuthRequired (anonymous allowed) | -| GET | `/api/v1/activity-log/export` | require_authenticated (no anonymous) | -| GET | `/api/v1/activity-log/settings` | AuthRequired | -| PUT | `/api/v1/activity-log/settings` | AuthRequired | -| DELETE | `/api/v1/activity-log` | require_authenticated (no anonymous) | - -### List query parameters (GET /api/v1/activity-log) - -| Param | Type | Default | Notes | -|-------|------|---------|-------| -| `categories` | `list[str]` | — | Repeatable. Values: auth, device, entity, capture, system | -| `severities` | `list[str]` | — | Repeatable. Values: info, warning, error | -| `actor` | `str` | — | Exact match | -| `entity_type` | `str` | — | Exact match | -| `entity_id` | `str` | — | Exact match | -| `since` | `datetime` (ISO-8601) | — | Inclusive lower bound on ts | -| `until` | `datetime` (ISO-8601) | — | Inclusive upper bound on ts | -| `q` | `str` | — | Substring match on message (LIKE %q%) | -| `before_seq` | `int` | — | Keyset cursor from previous page's `next_before_seq` | -| `limit` | `int` | 50 | Max entries per page. ge=1, le=200 | - -Export endpoint (`GET /api/v1/activity-log/export`) accepts the same filter params plus `format=csv|json`. - -### Page envelope fields (ActivityLogPageResponse) - -```json -{ - "entries": [...], // list[ActivityLogEntryResponse] - "next_before_seq": 42, // int | null — pass as before_seq for next page - "has_more": true, // bool - "total": 1337 // int — total matching all filters (all pages) -} -``` - -### Entry dict shape (ActivityLogEntryResponse) - -11 fields — identical to `entry_to_dict()` output: -```json -{ - "id": "al_abcd1234", - "ts": "2026-06-09T12:34:56.789+00:00", - "category": "entity", - "action": "entity.created", - "severity": "info", - "actor": "my-api-key", - "entity_type": "output_target", - "entity_id": "pt_abc", - "entity_name": "Desk", - "message": "Output target 'Desk' created", - "metadata": {} -} -``` - -### Settings field bounds (UpdateActivityLogSettingsRequest) - -| Field | Type | ge | le | Notes | -|-------|------|----|----|-------| -| `enabled` | `bool` | — | — | Enable/disable recording | -| `max_days` | `int` | 0 | 3650 | 0 = no age-based pruning | -| `max_entries` | `int` | 0 | 10_000_000 | 0 = no count-based pruning | - -### Export format param - -`?format=csv` (default) → `text/csv; charset=utf-8` -`?format=json` → `application/json` (streamed JSON array) - -### Pagination algorithm - -Keyset cursor (`before_seq`) works as follows: -- Omit `before_seq` (or pass `null`) to get the FIRST (newest) page. -- Each page response includes `next_before_seq` (the seq of the oldest entry on the page). -- Pass `next_before_seq` as `before_seq` in the next request to get the following (older) page. -- `has_more=false` means there are no more pages; `next_before_seq` is `null`. -- `total` is constant across pages for the same filter set. - -### New method added to ActivityLogRepository (additive, not breaking) - -`get_seq_for_id(entry_id: str) -> int | None` — indexed point-lookup of seq by entry id. -Used internally by the list endpoint to build the keyset cursor. - -Phase 4 landed (2026-06-09): schemas, route (list/export/settings/clear), router registration, -49 new tests — all green. Full suite 2486 passed, 2 skipped, 0 failed. Ruff clean. diff --git a/plans/activity-log/phase-5-frontend-tab.md b/plans/activity-log/phase-5-frontend-tab.md deleted file mode 100644 index 2d88c42..0000000 --- a/plans/activity-log/phase-5-frontend-tab.md +++ /dev/null @@ -1,137 +0,0 @@ -# Phase 5: Frontend — Activity tab + smart filtering + live updates - -**Status:** ✅ Done -**Parent plan:** [PLAN.md](./PLAN.md) -**Domain:** frontend · uses the `frontend-design` skill - -## Objective - -Build the dedicated top-level **Activity** tab: a read-only, smart-filterable, -keyset-paginated log viewer with an entry detail view, live-append of new events, and export. -This is a viewer (Dashboard-style), NOT a CRUD card section. - -## Tasks - -- [x] `core/ui.ts`: add `formatTimestamp(isoOrMs)` (Today/Yesterday/Date · HH:MM, i18n-aware) - and `formatRelativeTime(isoOrMs)` ("2m ago"), with `tabular-nums` styling guidance for the - list. Reuse existing `time.*` i18n key conventions. -- [x] `features/activity-log.ts`: - - `export async function loadActivityLog()` — fetch first page from - `GET /activity-log` (via `fetchWithAuth`), render the toolbar + list into the panel. - - **Smart filter toolbar:** category (multi, chips), severity (chips), actor - (text input), entity type, date range, free-text search (debounced). Quick presets: - Today / Errors / Auth / Devices. Filters drive server-side query params (no client-side - filtering of a partial page). Re-query on change; reset cursor. - - **List:** one row per entry — severity icon, category badge, relative time (title=absolute), - actor, message, entity crosslink (use `navigateToCard(...)` when the referenced entity is - resolvable). Keyset "load more" (or infinite scroll) using `next_before_seq`. - - **Detail:** expandable row / drawer showing full metadata JSON, exact timestamp, ids. - - **Live append:** `document.addEventListener('server:activity_logged', e => …)` — prepend - the new entry if it passes the active filters; show a subtle "new" affordance. (Depends on - the Phase 2 allowlist entry — already shipped.) - - **Export button:** triggers `GET /activity-log/export?format=…` with current filters via an - authed blob download (use `fetchWithAuth` → blob URL, per frontend.md auth rules). - - Empty / loading / error states; re-render on `languageChanged`. -- [x] Tab wiring: - - `core/tab-registry.ts`: add `activity_log: { loadFnName: 'loadActivityLog', autoRefresh: false }`. - - `templates/index.html`: sidebar tab button (`data-tab`, `switchTab('activity_log')`, - history/clock SVG icon, `data-i18n`) + `
`. - - `app.ts`: import + `Object.assign(window, { loadActivityLog, … })`; `global.d.ts` decls. -- [x] Icons: add a history/audit icon to `core/icon-paths.ts` + `core/icons.ts`; severity icons - (info/warning/error) reuse existing constants where possible. -- [x] i18n: add `activity_log.*` keys to `static/locales/{en,ru,zh}.json` (title, filter labels, - category/severity names, column labels, presets, empty/error, export, "N entries"). -- [x] Tutorials: add an Activity-tab step to the getting-started tour in - `features/tutorials.ts` + `tour.*` keys in all 3 locales. - -## Files to Modify/Create - -- `server/src/ledgrab/static/js/core/ui.ts` — modified: timestamp/relative-time formatters -- `server/src/ledgrab/static/js/features/activity-log.ts` — NEW: the viewer -- `server/src/ledgrab/static/js/core/tab-registry.ts` — modified: register tab -- `server/src/ledgrab/templates/index.html` — modified: tab button + panel -- `server/src/ledgrab/static/js/app.ts` — modified: import + window globals -- `server/src/ledgrab/static/js/global.d.ts` — modified: window decls -- `server/src/ledgrab/static/js/core/icon-paths.ts` — modified: icons (scrollText, circleAlert, info, filter, xCircle) -- `server/src/ledgrab/static/js/core/icons.ts` — modified: ICON_ACTIVITY_LOG, ICON_SEVERITY_*, ICON_FILTER, ICON_X_CIRCLE -- `server/src/ledgrab/static/locales/{en,ru,zh}.json` — modified: activity_log.* + time.* + tour.activity_log -- `server/src/ledgrab/static/js/features/tutorials.ts` — modified: gettingStartedSteps -- `server/src/ledgrab/static/css/activity-log.css` — NEW: list + toolbar styling -- `server/src/ledgrab/static/css/all.css` — modified: import activity-log.css - -## Acceptance Criteria - -- [x] New **Activity** tab loads, lists entries, and paginates via keyset "load more". -- [x] Filters hit server-side query params; quick presets work; free-text is debounced. -- [x] New events append live via `server:activity_logged` and respect active filters. -- [x] Export downloads CSV/JSON with auth, honoring current filters. -- [x] Fully localized (en/ru/zh); empty/loading/error states; re-renders on language change. -- [x] No plain ``; SVG icons only. - -## Notes - -- **Use the `frontend-design` skill** for the widget + settings panel layout. -- Reuse Phase 5's render helper and i18n namespace — no duplicated row markup or keys. -- Settings UI model: `features/settings.ts switchSettingsTab` + `modals/settings.html` rail. -- Frontend-only phase → run `tsc --noEmit` + `npm run build`; do NOT run pytest/ruff (docs + - TS only). -- Backup/restore cross-ref: no `STORE_MAP` edit needed (whole-DB backup covers the table) — - confirm nothing else (graph editor) needs syncing for this viewer (verified: not needed). - -## Review Checklist - -- [x] All tasks completed -- [x] Code follows frontend conventions (i18n ×3, icons, selectors, CSS vars, settings pattern) -- [x] No unintended side effects -- [x] Build passes (`tsc --noEmit` + `npm run build`) -- [ ] Manual smoke: widget live-updates + links; settings save/clear/export; docs updated (recommend at final review — requires server restart) - -## Handoff to Next Phase - -This is the **final implementation phase**. All six phases are complete. Notes for the final reviewer: - -**What was implemented in Phase 6:** - -- `features/dashboard.ts`: `_loadRecentActivityWidget()`, `_renderRecentActivityList()`, `_startRecentActivityLive()` — SSE live-update listener (`server:activity_logged`) with cap-to-5 prepend logic. Widget appended after `getOrderedSections()` loop (outside the layout toggle system — always visible). Non-blocking: `.catch()` on the async load call. -- `features/settings.ts`: `loadActivityLogSettings()`, `saveActivityLogSettings()`, `activityLogSettingsExport(format)`, `clearActivityLog()` — all exported and exposed on `window` via `app.ts` + `global.d.ts`. -- `templates/modals/settings.html`: Activity Log rail entry (System group, cyan channel) + full panel (`id="settings-panel-activity_log"`) with enabled toggle, `max_days`/`max_entries` inputs, Save, CSV/JSON export, Clear (danger zone). Audit-vs-debug distinction note with cross-links in both directions (`closeSettingsModal(); openLogOverlay()` and `closeSettingsModal(); switchTab('activity_log')`). -- `static/css/activity-log.css`: `.dal-*` dashboard widget styles + `.ds-info-note` / `.ds-inline-link` settings panel utilities appended (no new CSS file). -- `static/locales/{en,ru,zh}.json`: 32 new keys each under `dashboard.section.recent_activity`, `dashboard.recent_activity.*`, `settings.tab.activity_log`, `settings.activity_log.*`. -- `README.md`: "### Activity Log" section documenting tab, retention settings, export, and audit-vs-debug distinction. - -**Reused Phase 5 helpers (no duplication):** - -- `fetchRecentEntries(limit)` and `renderCompactEntry(entry)` — new public exports added to `activity-log.ts` in Phase 6 (not duplicated in dashboard.ts). -- All `.al-*` CSS classes from Phase 5 are reused in the compact rows inside the widget. - -**Build verification:** `tsc --noEmit` clean, `npm run build` passed (2.8 MB bundle, 258 ms) at time of implementation. - -**Remaining manual smoke test (requires server restart):** - -- Dashboard widget loads recent entries, prepends live on new activity, "View all →" switches to Activity tab. -- Settings panel reads/writes retention, Save shows toast, Clear prompts confirm then deletes, Export downloads authed blob. -- Cross-links: note in Settings opens Log Viewer overlay; note in Log Viewer links back to Activity tab. - -**Outstanding open note from PLAN.md:** The manual browser smoke test across the whole feature (P5 note) remains open — it requires a server restart to exercise the live API endpoints. Recommend as first step in the final review session. diff --git a/server/CLAUDE.md b/server/CLAUDE.md index 746c59d..67d5d17 100644 --- a/server/CLAUDE.md +++ b/server/CLAUDE.md @@ -28,6 +28,18 @@ API key authentication via Bearer token in the `Authorization` header (`Authoriz - When `api_keys` is **set**: a valid Bearer token is required from every client (loopback included). - `require_authenticated()` rejects even loopback-anonymous callers on sensitive endpoints (e.g. backup download, secret reveal). +## Activity / Audit Log + +Persistent, queryable audit log of meaningful actions (auth, device, entity CRUD, capture, system), surfaced in the WebUI (Activity tab + Dashboard widget + Settings retention panel). + +- **Storage is NOT a `BaseSqliteStore`.** `storage/activity_log_repository.py` is a purpose-built repository over a dedicated indexed `activity_log` table (migration `002_add_activity_log`) — query-on-demand with **keyset pagination** (`seq` cursor), never load-all-into-memory. Don't route it through the entity-store pattern. +- **Recording.** `core/activity_log/recorder.py` (`ActivityRecorder`) is best-effort (never raises into the audited action) and **thread-safe** (inline on the event loop; `loop.call_soon_threadsafe` from non-loop threads, e.g. zeroconf discovery). It persists the entry **and** fires an `activity_logged` realtime event. Actor comes from the `current_actor` `ContextVar` (set in `verify_api_key`), default `"system"`. +- **Entity CRUD is auto-audited** via the `fire_entity_event()` choke point in `api/dependencies.py` — every create/update/delete already calls it. **Delete handlers must pass `entity_name`** (the entity is gone by record time). Non-entity events use explicit `recorder.record(...)` (get it via `get_activity_recorder()` DI or `get_module_recorder()` for engine/thread sites). +- **Never log secrets.** API-key tokens are never recorded. Wrap any untrusted/attacker-controllable string (mDNS names, headers, user-authored names) with `sanitize_display()` (`core/activity_log/sanitize.py`) before it enters a `message`/`metadata` field. Per-IP throttle bounds auth-failure audit writes. +- **Adding a new audited event:** pick a dotted `action` (e.g. `"thing.created"`), call the recorder; for it to render localized in the UI, add `activity_log.msg.` to all three `static/locales/*.json` (the frontend `localizeMessage()` maps action→template; falls back to the server `message`). Entity-type labels live under `activity_log.entity_type.`. +- **Adding a new realtime event type** (`pm.fire_event({"type": ...})`): add it to `_ALLOWED_SERVER_EVENT_TYPES` in `static/js/core/events-ws.ts` AND keep `tests/test_events_ws_parity.py` green. +- **Retention + API.** `core/activity_log/retention.py` prunes by `max_days` + `max_entries` (settings persisted via `db.set_setting("activity_log")`); the recorder's `enabled` flag is rehydrated from those settings on startup. REST in `api/routes/activity_log.py`: `GET /activity-log` (list, `AuthRequired`), `GET /export` (CSV/JSON stream — `require_authenticated`; chunked keyset so it never holds the DB lock across the stream; CSV formula-injection guarded), `GET|PUT /settings` (PUT is `require_authenticated`), `DELETE` (clear — `require_authenticated`, self-audited). The table is covered by the existing whole-DB backup (no `STORE_MAP` change needed). + ## Common Tasks ### Adding a new API endpoint