chore(activity-log): post-merge cleanup + graduate context to CLAUDE.md
- remove plans/activity-log/ (feature merged; learnings in CLAUDE.md + git history) - server/CLAUDE.md: Activity/Audit Log architecture + extension points (recorder, fire_entity_event hook, sanitize_display, events allowlist, retention, API auth posture)
This commit is contained in:
@@ -1,73 +0,0 @@
|
||||
# CONTEXT — Activity / Audit Log
|
||||
|
||||
Living scratchpad for the feature. The orchestrator updates this between phases. Tier-2
|
||||
context (survives across phases; graduates to CLAUDE.md only if it's a lasting project truth).
|
||||
|
||||
## Config (from approval)
|
||||
|
||||
- **Mode:** Automated · **Execution:** Orchestrator · **Strategy:** Incremental
|
||||
- **Base/merge target:** `master` · **Branch:** `feature/activity-log` · **Branch point:** `17dd2e0`
|
||||
- Final merge ALWAYS requires user approval (even in Automated mode).
|
||||
|
||||
## Product decisions (locked)
|
||||
|
||||
- Placement: BOTH a top-level **Activity** tab AND a Dashboard **Recent Activity** widget,
|
||||
plus a Settings retention panel.
|
||||
- Scope: all four categories — entity CRUD, auth, device connect/disconnect, capture & system.
|
||||
- Detail: **action metadata only** (no before/after diffs).
|
||||
- Durability: **export on demand (CSV/JSON)** + existing whole-DB backup. **No** separate
|
||||
backup subsystem.
|
||||
- WebUI work uses the `frontend-design` skill (Phases 5–6).
|
||||
|
||||
## Key codebase facts (verified during planning)
|
||||
|
||||
- `fire_entity_event(entity_type, action, entity_id)` @ `api/dependencies.py:202` — central
|
||||
hook, called by every entity route synchronously in-request; has `_deps` store access.
|
||||
- `ProcessorManager.fire_event(dict)` / `subscribe_events()` @ `core/processing/processor_manager.py`
|
||||
back `/api/v1/events/ws` (`api/routes/output_targets_control.py:206`). `fire_event` does
|
||||
`put_nowait` (no `call_soon_threadsafe`) — fine for the existing consumer; recorder marshals.
|
||||
- Frontend realtime: `core/events-ws.ts` re-dispatches `server:<type>`; `_ALLOWED_SERVER_EVENT_TYPES`
|
||||
(line 39) is parity-checked by `tests/test_events_ws_parity.py`. Must add `activity_logged`.
|
||||
- Actor: `request.state.auth_label` set in `api/auth.py` (129 authenticated, 83 anonymous).
|
||||
- Storage: `Database` singleton (`storage/database.py`, single conn, RLock, WAL,
|
||||
`synchronous=FULL`, `get_setting/set_setting`). `BaseSqliteStore` loads ALL rows to memory →
|
||||
AVOID for the log. Migrations: `storage/data_migrations.py` (`ALL_MIGRATIONS`, idempotent).
|
||||
- Backup/restore is **whole-DB** (`Database.backup_to`/`restore_from`, no STORE_MAP allowlist)
|
||||
→ the new `activity_log` table is auto-covered. No `system.py` STORE_MAP edit needed.
|
||||
- Background-engine pattern: `core/backup/auto_backup.py` (start/stop loop, `_prune`, settings).
|
||||
- Thread-marshal precedent: `utils/log_broadcaster.py` (`ensure_loop` + `call_soon_threadsafe`).
|
||||
- Device seams: `device_health_changed` (`core/processing/device_health.py`, on transition) +
|
||||
`device_discovered`/`device_lost` (`core/devices/discovery_watcher.py`, **zeroconf thread**).
|
||||
- **No** API-key create/rotate/revoke routes exist (only `GET /system/api-keys`) → those auth
|
||||
events are DESCOPED.
|
||||
- Existing **Log Viewer** (`utils/log_broadcaster.py`) = ephemeral debug-log tail; the audit
|
||||
log is a different, persistent, structured feature. Differentiate; do not duplicate.
|
||||
|
||||
## Frozen contracts (fill as phases complete)
|
||||
|
||||
- ActivityLogEntry fields / dict shape: **frozen** — see phase-1-storage.md Handoff section. 11 fields: `id`, `ts`, `category`, `action`, `severity`, `actor`, `message`, `entity_type`, `entity_id`, `entity_name`, `metadata`. `seq` is DB-only (not on dataclass).
|
||||
- ActivityLogFilters shape: **frozen** — 8 optional fields: `categories`, `severities`, `actor`, `entity_type`, `entity_id`, `since`, `until`, `message_like`. See phase-1-storage.md Handoff.
|
||||
- recorder.record(...) signature + actor ContextVar import path: **frozen** — see phase-2-recorder-retention.md Handoff section. Signature: `record(category, action, *, severity="info", actor=None, entity_type=None, entity_id=None, entity_name=None, message, metadata=None, _bypass_enabled=False)`. ContextVar: `from ledgrab.core.activity_log.context import current_actor`. Module accessor: `from ledgrab.core.activity_log.recorder import get_module_recorder`. Event payload: `{"type": "activity_logged", "entry": {11-field dict with ts as ISO string, metadata as dict}}`. DI getters: `get_activity_recorder()`, `get_activity_log_repo()`, `get_activity_log_retention_engine()`.
|
||||
- API endpoints + query params + page envelope + settings bounds: **frozen** — see phase-4-api.md Handoff section. Endpoints: `GET /api/v1/activity-log` (list, AuthRequired), `GET /api/v1/activity-log/export` (stream CSV/JSON, require_authenticated), `GET|PUT /api/v1/activity-log/settings` (AuthRequired), `DELETE /api/v1/activity-log` (clear, require_authenticated). Page envelope: `entries`, `next_before_seq`, `has_more`, `total`. Settings fields: `enabled` (bool), `max_days` (0–3650), `max_entries` (0–10_000_000). Export: `?format=csv|json`.
|
||||
|
||||
## Failed approaches / rejected designs
|
||||
|
||||
- Buffered async-writer subsystem (asyncio.Queue): REJECTED — unsafe from the zeroconf thread
|
||||
and adds a shutdown-flush ordering hazard. Using direct synchronous-on-loop writes with
|
||||
`call_soon_threadsafe` marshaling instead (simpler + correct).
|
||||
- Using `BaseSqliteStore` for the log: REJECTED — loads all rows into memory.
|
||||
- Separate activity-log backup subsystem: REJECTED — whole-DB backup already covers the table;
|
||||
export-on-demand is the portability story.
|
||||
|
||||
## Deferred / open
|
||||
|
||||
- Setup-scaffold first-run noise suppression (batch/suppress flag) — deferred (nice-to-have).
|
||||
|
||||
## Phase progress notes
|
||||
|
||||
Phase 1 landed (2026-06-09): `activity_log.py` (dataclass + enums + filters + codec), `AddActivityLogTableMigration` (`002_add_activity_log`) appended to `ALL_MIGRATIONS`, `ActivityLogRepository` (record/query/count/prune/clear/iter_export), 41 new tests — all green. Full suite 2226 passed, 0 failed. Schema and method signatures frozen in phase-1-storage.md Handoff. Gotcha: `Database.execute` takes a positional tuple — use `?` placeholders (not `:name`), otherwise Python 3.14 will raise `ProgrammingError`.
|
||||
Phase 2 landed (2026-06-09): `core/activity_log/` package (`context.py`, `recorder.py`, `retention.py`, `__init__.py`); actor ContextVar set in `api/auth.py` (both branches); `ActivityLogRetentionEngine` mirroring AutoBackupEngine; full wiring in `main.py` (repo at module level, recorder+engine in lifespan, `server.shutting_down` first shutdown action, engine stop before db.close); DI getters in `api/dependencies.py`; `activity_logged` added to `_ALLOWED_SERVER_EVENT_TYPES` in `events-ws.ts`; `set_module_recorder` exposes recorder to non-DI sites; 24 new tests — all green. Full suite 2309 passed, 2 skipped, 0 failed. Ruff clean.
|
||||
Phase 4 landed (2026-06-09): schemas (`api/schemas/activity_log.py`), routes (`api/routes/activity_log.py`: list/export/settings/clear), router registration in `api/__init__.py`, `get_seq_for_id` helper on `ActivityLogRepository`. 49 new tests — all green. Full suite 2486 passed, 2 skipped, 0 failed. Ruff clean. Pagination bug found and fixed (limit+1 probe must drop oldest row when has_more, not tail).
|
||||
Phase 3 landed (2026-06-09): instrumented all four categories — entity CRUD via `fire_entity_event` choke-point (`dependencies.py`), auth failures + WS session in `auth.py`, device online/offline in `device_health.py`, device discovered/lost in `discovery_watcher.py`, ADB connect/disconnect in `system_settings.py`, capture start/stop (individual + bulk) in `output_targets_control.py`, scene/playlist/automation activate in their respective route/engine files, backup/restore/delete + restart/shutdown/update/calibration/settings in `backup.py`/`update.py`/`calibration.py`; all 11 entity delete handlers pass `entity_name` to `fire_entity_event`; 22 new tests (security: token never in any field, explicitly asserted) — all green. Full suite 2369 passed, 2 skipped, 0 failed. Ruff clean. Complete (category, action) inventory in phase-3-instrumentation.md Handoff section.
|
||||
Phase 5 landed (2026-06-09): Activity tab frontend — `features/activity-log.ts` (loadActivityLog, filter toolbar with category/severity chips + presets + debounced search + date range + actor/entity-type text fields, keyset-paginated list, expandable detail drawer, live-prepend via `server:activity_logged`, authed CSV/JSON blob export); `core/ui.ts` additions (`formatTimestamp`, `formatRelativeTime`); `core/icon-paths.ts` + `core/icons.ts` additions (scrollText/audit, circleAlert, info, filter, xCircle); `core/tab-registry.ts` registered; `templates/index.html` tab button + panel; `app.ts` + `global.d.ts` wired; `static/css/activity-log.css` (precision-ledger design, category color rail, live-dot pulse, detail drawer, responsive breakpoints); all three locales updated (activity_log.* + time.* relative-time keys + tour.activity_log); `features/tutorials.ts` getting-started tour extended. `tsc --noEmit` clean, `npm run build` passes. Reusable helpers for Phase 6 documented in phase-5-frontend-tab.md Handoff section.
|
||||
Phase 6 landed (2026-06-09): Dashboard "Recent Activity" widget (live SSE, View-all link, `.dal-*` CSS, loading/empty states) + Settings "Activity Log" panel (enabled toggle, max_days/max_entries, Save toast, authed CSV/JSON export, confirmed Clear, audit-vs-debug cross-links) + 32 i18n keys per locale + README "Activity Log" section. `tsc --noEmit` clean, `npm run build` passes. All six phases complete — feature ready for final review.
|
||||
@@ -1,141 +0,0 @@
|
||||
# Feature: Activity / Audit Log
|
||||
|
||||
**Branch:** `feature/activity-log`
|
||||
**Base branch:** `master` (merge target)
|
||||
**Branch point:** `17dd2e02bab4d00a93479eb6af1a8c6ddc0c7224` (use for clean review diffs)
|
||||
**Created:** 2026-06-09
|
||||
**Status:** 🟢 All phases complete — awaiting final review + merge
|
||||
**Strategy:** Incremental
|
||||
**Mode:** Automated
|
||||
**Execution:** Orchestrator
|
||||
**Remote:** origin → https://git.dolgolyov-family.by/alexei.dolgolyov/ledgrab.git
|
||||
|
||||
## Summary
|
||||
|
||||
A persistent, queryable audit log of meaningful LedGrab actions, surfaced in the WebUI.
|
||||
Captures four categories — entity CRUD, authentication, device connect/disconnect, and
|
||||
capture & system events — as **action-metadata-only** records (who/what/when + entity
|
||||
type/name/id + a human-readable message + small structured metadata; **no before/after
|
||||
diffs**). Surfaced as a dedicated top-level **Activity** tab with smart filtering + live
|
||||
updates, a compact **Recent Activity** widget on the Dashboard, and a **Settings** panel
|
||||
for retention. Durability rides on the existing whole-DB `ledgrab.db` backup; portability
|
||||
is an on-demand CSV/JSON **export** (no separate backup subsystem).
|
||||
|
||||
## Design pillars (the load-bearing decisions)
|
||||
|
||||
1. **Dedicated `activity_log` table + repository — NOT `BaseSqliteStore`.** That base loads
|
||||
every row into an in-memory cache and uses a generic `id/name/data` blob — wrong for an
|
||||
append-heavy, unbounded log. We use a purpose-built indexed table with query-on-demand
|
||||
keyset pagination.
|
||||
2. **Central choke-point instrumentation.** `fire_entity_event()` (`api/dependencies.py:202`)
|
||||
is already called by every entity route on create/update/delete and has `_deps` access to
|
||||
resolve names. The recorder hooks there for all entity CRUD. Non-entity events get explicit
|
||||
`recorder.record(...)` calls.
|
||||
3. **Actor via `ContextVar`.** Set inside `verify_api_key` (next to `request.state.auth_label`),
|
||||
default `"system"`, reset per-request. The recorder reads it without threading actor
|
||||
through every call.
|
||||
4. **Direct synchronous write on the event-loop thread** (no separate buffered-writer
|
||||
subsystem — simpler, and the request already did a `synchronous=FULL` entity write).
|
||||
Cross-thread callers (zeroconf discovery thread) marshal via `loop.call_soon_threadsafe`,
|
||||
mirroring `utils/log_broadcaster.py`. The "server shutting down" event is recorded as the
|
||||
FIRST action in the lifespan shutdown block, before any teardown.
|
||||
5. **Reuse the existing realtime bus.** A new `activity_logged` event over `/api/v1/events/ws`
|
||||
(one `events-ws.ts` allowlist entry + `test_events_ws_parity.py` update). No new socket.
|
||||
6. **Never log secrets.** API-key *tokens* are never stored — only labels/ids.
|
||||
7. **Differentiate from the existing Log Viewer.** `utils/log_broadcaster.py` is an ephemeral
|
||||
500-line debug-log tail. The audit log is persistent, structured, semantic. Cross-link in
|
||||
the Settings panel; never duplicate.
|
||||
|
||||
## Build & Test Commands
|
||||
|
||||
- **Build (frontend):** `cd server && npm run build`
|
||||
- **Type-check (TS):** `cd server && npx tsc --noEmit`
|
||||
- **Test:** `cd server && py -3.13 -m pytest tests/ --no-cov -q`
|
||||
- **Lint (Python):** `cd server && ruff check src/ tests/ --fix`
|
||||
- **Events parity test (load-bearing for P2):** included in pytest (`tests/test_events_ws_parity.py`)
|
||||
|
||||
> Scope checks to the files actually edited: backend phases run ruff + pytest; frontend-only
|
||||
> phases run `tsc --noEmit` + `npm run build` (no pytest/ruff). Phases touching both run both.
|
||||
|
||||
## Phases
|
||||
|
||||
- [x] Phase 1: Storage — model, migration, repository [domain: data] → [subplan](./phase-1-storage.md)
|
||||
- [x] Phase 2: Recorder, actor context, retention, lifecycle [domain: backend] → [subplan](./phase-2-recorder-retention.md)
|
||||
- [x] Phase 3: Event instrumentation (4 categories) [domain: backend] → [subplan](./phase-3-instrumentation.md)
|
||||
- [x] Phase 4: REST API — query/filter/export/settings/clear [domain: backend] → [subplan](./phase-4-api.md)
|
||||
- [x] Phase 5: Frontend — Activity tab + smart filtering + live updates [domain: frontend] → [subplan](./phase-5-frontend-tab.md)
|
||||
- [x] Phase 6: Dashboard widget + Settings panel + docs [domain: frontend] → [subplan](./phase-6-dashboard-settings.md)
|
||||
|
||||
## Parallelizable Phase Groups (Orchestrator mode only)
|
||||
|
||||
- Phases 3 and 4 are **parallelizable only after the schema is frozen at end of Phase 2**.
|
||||
Both depend on the P2 recorder/schema; P4 registers the router in `api/__init__.py`, P3
|
||||
edits `dependencies.py`/`auth.py` (different files, shared schema contract). To keep the
|
||||
Automated run simple and low-risk, they run **sequentially** (3 → 4) unless time pressure
|
||||
warrants worktree-isolated parallelism. Phases 5 → 6 are sequential (6 reuses P5 formatters
|
||||
and the feature module).
|
||||
|
||||
## Phase Progress Log
|
||||
|
||||
| Phase | Domain | Status | Review | Build | Committed |
|
||||
|-------|--------|--------|--------|-------|-----------|
|
||||
| Phase 1: Storage | data | ✅ Done | ✅ Passed | ✅ Passed | ✅ |
|
||||
| Phase 2: Recorder/Retention | backend | ✅ Done | ✅ Passed | ✅ Passed | ✅ |
|
||||
| Phase 3: Instrumentation | backend | ✅ Done | ✅ Passed | ✅ Passed | ✅ |
|
||||
| Phase 4: REST API | backend | ✅ Done | ✅ Passed | ✅ Passed | ✅ |
|
||||
| Phase 5: Frontend tab | frontend | ✅ Done | ✅ Passed | ✅ Passed (tsc+build) | ✅ |
|
||||
| Phase 6: Dashboard/Settings | frontend | ✅ Done | ✅ Passed | ✅ Passed (tsc+build) | ✅ |
|
||||
|
||||
## Outstanding Warnings
|
||||
|
||||
| Phase | Warning | Severity | Status (open / resolved / accepted) |
|
||||
|-------|---------|----------|-------------------------------------|
|
||||
| 3 | Log injection via unauth mDNS device name/url into audit message | 🟠 High (security) | resolved — `sanitize_display` helper applied |
|
||||
| 3 | Origin sanitizer missed spaces/NUL/ANSI | 🟠 High (security) | resolved — `sanitize_display` over netloc |
|
||||
| 3 | Unauth auth-failure audit-write flood (no write-rate bound) | 🟠 High (security) | resolved — per-IP audit-record throttle (10s, capped) |
|
||||
| 3 | Malformed-IPv6 Origin → urlparse ValueError into WS handler | 🟡 Warning | resolved — try/except guard |
|
||||
| 3 | Throttle module-global state caused flaky test contamination | 🟡 Warning | resolved — autouse conftest reset fixture |
|
||||
| 4 | Export held global DB write-lock across the stream (slow-client DoS) | 🟠 High (security) | resolved — chunked keyset export releases lock per batch |
|
||||
| 4 | PUT /settings only AuthRequired → anon could disable auditing/prune trail | 🟠 High (security) | resolved — `require_authenticated` on settings PUT |
|
||||
| 4 | CSV formula-injection missed leading TAB/CR | 🟡 Medium (security) | resolved — added `\t`/`\r` to guard |
|
||||
| 4 | `total` count full-scans on every list request | 🔵 Low (perf) | accepted — bounded by retention; read-only; optional opt-in deferred |
|
||||
| 5 | Inverted list ordering broke pagination + live-append | 🔴 Blocker | resolved — pages reversed to newest-first; re-review PASS |
|
||||
| 5 | Attribute-context XSS (entity_name title + JSON.stringify onclick) | 🟡 Warning (security) | resolved — `_escapeAttr` + data-attr event delegation |
|
||||
| 5 | Filter toolbar value= attrs not quote-escaped (new code) | 🟡 Warning (security) | resolved — `_escapeAttr` on q/actor/entity_type/since/until |
|
||||
| 5 | Manual browser smoke test (tab loads, filters, live, export) | 🔵 Note | open — recommend at final review (server restart needed) |
|
||||
| 6 | clearActivityLog() 401 path unreachable → silent failure | 🟡 Warning | resolved — `handle401:false` surfaces auth-required toast |
|
||||
| 6 | Recent Activity widget dropped first live event when empty | 🔵 Note | resolved — empty→list transition on first live event |
|
||||
| 6 | Widget outside dashboard layout-toggle/ordering system | 🔵 Note | accepted — deliberate (always-visible), collapse still works |
|
||||
| Final | Entity-crosslink map keys mismatched backend entity_type (device/color_strip_source/audio_source) | 🟡 Warning | resolved — `_ENTITY_NAV` keys corrected + scene_playlist added |
|
||||
| Final | Owner-authored names interpolated raw at some record sites | 🔵 Note (defense-in-depth) | resolved — `sanitize_display` applied uniformly |
|
||||
| Manual test | Entry descriptions rendered server English (not localized) | 🟡 Warning (acceptance criterion) | resolved — client-side `localizeMessage` from structured fields + `activity_log.msg.*`/`entity_type.*` templates ×3 |
|
||||
| Manual test | Redundant "Activity" header banner at top of tab | 🔵 Note | resolved — header block removed |
|
||||
| Manual test | Recent Activity widget missing from Customize Dashboard | 🟡 Warning | resolved — registered as a first-class dashboard section (show/hide/reorder; pre-existing layouts preserved) |
|
||||
| Manual test | Activity widget live event rebuilt the whole dashboard | 🟡 Warning (perf) | resolved — surgical list update; single listener with teardown |
|
||||
| Manual test | Relative-time labels static (never tick) | 🟡 Warning | resolved — shared `ensureRelativeTimeTicker` (single 30s interval, visibility-aware) |
|
||||
| Manual test | Dashboard fully rebuilt/jumped on entity edits (pre-existing `forceFullRender` wholesale innerHTML) | 🟡 Warning (UX) | resolved — section-level reconciliation (`_reconcileDynamicSections`): only changed sections replaced; widget live DOM + perf strip preserved |
|
||||
| Manual test | Activity list columns slightly misaligned | 🔵 Note | resolved — CSS grid with fixed badge/actor columns |
|
||||
| Manual test | Reconciler left orphan "no targets" node on empty→populated | 🟡 Warning (regression, caught in review) | resolved — sweep non-section top-level children |
|
||||
|
||||
## Final Review
|
||||
|
||||
- [ ] Comprehensive code review
|
||||
- [ ] Security review (auth/PII-in-logs/secrets/log-injection — triggered)
|
||||
- [ ] All Outstanding Warnings resolved or consciously accepted
|
||||
- [ ] Full build passes (`npm run build` + `tsc --noEmit`)
|
||||
- [ ] Full test suite passes (`pytest`)
|
||||
- [ ] Merged to `master`
|
||||
|
||||
## Amendment Log
|
||||
|
||||
_(Filled in if the plan is amended mid-implementation.)_
|
||||
|
||||
- 2026-06-09: Plan reviewer (pre-implementation) → ⚠️ with 3 Critical Gaps, all resolved
|
||||
before Phase 1: (G1) descoped non-existent API-key mutation events; (G2) dropped the
|
||||
buffered-writer subsystem for a direct synchronous-on-loop write with `call_soon_threadsafe`
|
||||
marshaling for thread-origin events; (G3) record "server shutting down" first in the shutdown
|
||||
block (no buffer to flush). Concerns folded in: device events via `device_health_changed` +
|
||||
`device_discovered/_lost`; actor ContextVar in `verify_api_key`; name-on-delete passed
|
||||
explicitly; settings-audit scoped + self-key excluded; parity allowlist before emit site.
|
||||
Adopted suggestions: rowid keyset tiebreaker; log the disable action. Deferred: setup-scaffold
|
||||
noise suppression.
|
||||
@@ -1,177 +0,0 @@
|
||||
# Phase 1: Storage — model, migration, repository
|
||||
|
||||
**Status:** ✅ Done
|
||||
**Parent plan:** [PLAN.md](./PLAN.md)
|
||||
**Domain:** data
|
||||
|
||||
## Objective
|
||||
|
||||
Create the persistent foundation for the audit log: an `ActivityLogEntry` dataclass, an
|
||||
additive idempotent SQLite migration that creates a dedicated indexed `activity_log` table,
|
||||
and a purpose-built `ActivityLogRepository` (NOT `BaseSqliteStore`) supporting append,
|
||||
keyset-paginated filtered query, count, time/count-based prune, and streaming export.
|
||||
|
||||
## Tasks
|
||||
|
||||
- [x] Create `server/src/ledgrab/storage/activity_log.py`:
|
||||
- `ActivityCategory` and `ActivitySeverity` string enums (or `Literal` unions used as
|
||||
constants). Categories: `auth`, `device`, `entity`, `capture`, `system`. Severities:
|
||||
`info`, `warning`, `error`.
|
||||
- `@dataclass ActivityLogEntry` with fields: `id: str` (e.g. `al_<uuid8>`), `ts: datetime`
|
||||
(UTC, server-assigned), `category: str`, `action: str`, `severity: str`, `actor: str`,
|
||||
`entity_type: str | None`, `entity_id: str | None`, `entity_name: str | None`,
|
||||
`message: str`, `metadata: dict` (small JSON; default empty). Provide `to_row()` /
|
||||
`from_row()` (column tuple/dict ↔ dataclass; `metadata` JSON-encoded; `ts` isoformat).
|
||||
- [x] Add migration to `server/src/ledgrab/storage/data_migrations.py`:
|
||||
- New `DataMigration` subclass `AddActivityLogTableMigration` with unique `name`
|
||||
(next sequential id, e.g. `"NNN_add_activity_log"` — match existing naming) and
|
||||
`apply(conn)` creating `activity_log` with an INTEGER PRIMARY KEY AUTOINCREMENT `seq`
|
||||
(monotonic keyset tiebreaker) plus columns: `id TEXT UNIQUE NOT NULL`, `ts TEXT NOT NULL`,
|
||||
`category TEXT NOT NULL`, `action TEXT NOT NULL`, `severity TEXT NOT NULL`,
|
||||
`actor TEXT NOT NULL`, `entity_type TEXT`, `entity_id TEXT`, `entity_name TEXT`,
|
||||
`message TEXT NOT NULL`, `metadata TEXT NOT NULL DEFAULT '{}'`.
|
||||
- Indexes: `(ts DESC, seq DESC)` (primary keyset/sort), `category`, `severity`, `actor`,
|
||||
`(entity_type, entity_id)`. Use `CREATE TABLE/INDEX IF NOT EXISTS` for idempotency.
|
||||
- Append the instance to `ALL_MIGRATIONS` (never reorder existing entries).
|
||||
- [x] Create `server/src/ledgrab/storage/activity_log_repository.py`:
|
||||
- `class ActivityLogRepository` taking `db: Database` (NOT subclassing `BaseSqliteStore`).
|
||||
- `record(entry: ActivityLogEntry) -> None`: single parameterized INSERT via
|
||||
`db.execute(...)` (auto-commit). The `seq` is DB-assigned. **Caller guarantees this runs
|
||||
on the event-loop thread** (see Phase 2 — cross-thread marshaling lives in the recorder).
|
||||
- `query(filters: ActivityLogFilters, *, before_seq: int | None, limit: int) -> list[ActivityLogEntry]`:
|
||||
keyset pagination `WHERE seq < ? ORDER BY seq DESC LIMIT ?` plus optional filters —
|
||||
`category IN (...)`, `severity IN (...)`, `actor = ?`, `entity_type = ?`, `entity_id = ?`,
|
||||
`ts >= ?` / `ts <= ?`, `message LIKE ?` (free-text, `%q%`, escaped). All parameterized.
|
||||
- `count(filters) -> int`.
|
||||
- `prune(*, before_ts: datetime | None, max_entries: int | None) -> int`: delete rows older
|
||||
than `before_ts`, and/or trim to the newest `max_entries` by `seq`. Returns rows deleted.
|
||||
- `clear() -> int`: delete all rows (used by the API clear endpoint; the clear action is
|
||||
itself audited by the recorder, not here). Returns rows deleted.
|
||||
- `iter_export(filters) -> Iterator[ActivityLogEntry]`: cursor-based streaming for export
|
||||
(does not load all rows into memory).
|
||||
- Define a small `ActivityLogFilters` dataclass (all-optional fields) in the repository or
|
||||
`activity_log.py` and reuse it across query/count/prune/export.
|
||||
- [x] Unit tests in `server/tests/storage/test_activity_log_repository.py`:
|
||||
- insert + read back round-trip (incl. metadata JSON, UTC ts);
|
||||
- filter by each dimension (category/severity/actor/entity/date/free-text);
|
||||
- keyset pagination stability across two pages with same-`ts` rows (seq tiebreaker);
|
||||
- prune by age and by max_entries;
|
||||
- clear; count; export iterator yields all matching rows;
|
||||
- migration idempotency (constructing the repo twice / running migrations twice is safe).
|
||||
|
||||
## Files to Modify/Create
|
||||
|
||||
- `server/src/ledgrab/storage/activity_log.py` — new: dataclass + enums + filters + row codec
|
||||
- `server/src/ledgrab/storage/data_migrations.py` — modify: add migration + append to `ALL_MIGRATIONS`
|
||||
- `server/src/ledgrab/storage/activity_log_repository.py` — new: repository
|
||||
- `server/tests/storage/test_activity_log_repository.py` — new: unit tests
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- `activity_log` table + indexes created idempotently on startup (running migrations twice is a no-op).
|
||||
- Query is keyset-paginated and index-backed; a 10k-row table never loads fully into memory.
|
||||
- Pagination is stable when many rows share the same millisecond `ts` (uses `seq` tiebreaker).
|
||||
- `prune` removes by age AND by max-entry cap; `clear` empties the table; `export` streams.
|
||||
- All filters use parameterized SQL (no string interpolation of user input).
|
||||
- New unit tests pass; `ruff check` clean; existing tests still green.
|
||||
|
||||
## Notes
|
||||
|
||||
- Reference patterns: `storage/database.py` (`execute`, `transaction`, `get_setting`),
|
||||
`storage/data_migrations.py` (`DataMigration`, `MigrationRunner`, `ALL_MIGRATIONS`),
|
||||
`storage/sync_clock.py` (dataclass `to_dict`/`from_dict` style).
|
||||
- 🔒 **Migration-safety addendum (data domain):** this migration is purely additive (new
|
||||
table) — no rename, no field/key/file move, no data movement → no data-loss risk. Still
|
||||
idempotent (`IF NOT EXISTS`). Rollback = drop the table; no user data is transformed.
|
||||
- Do NOT wire the repository into `main.py` or `dependencies.py` here — that is Phase 2.
|
||||
- `Database`'s connection is created with the existing threading model; the repository must
|
||||
not assume it can be called from arbitrary threads. Thread marshaling is Phase 2's job.
|
||||
|
||||
## Review Checklist
|
||||
|
||||
- [x] All tasks completed
|
||||
- [x] Code follows project conventions (dataclass codec style, migration naming)
|
||||
- [x] No unintended side effects (no startup wiring yet)
|
||||
- [x] Build passes (ruff + pytest)
|
||||
- [x] Tests pass (new + existing)
|
||||
|
||||
## Handoff to Next Phase
|
||||
|
||||
### ActivityLogEntry — final field list and dict shape
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class ActivityLogEntry:
|
||||
id: str # "al_<uuid8>" — caller-assigned
|
||||
ts: datetime # UTC-aware; stored as ISO-8601 string in DB
|
||||
category: str # ActivityCategory constant
|
||||
action: str # verb-object label, e.g. "entity.created"
|
||||
severity: str # ActivitySeverity constant
|
||||
actor: str # API-key label or "system"
|
||||
message: str # human-readable description
|
||||
entity_type: str | None # e.g. "output_target"
|
||||
entity_id: str | None # stable entity id
|
||||
entity_name: str | None # name at time of event
|
||||
metadata: dict # JSON-serialisable; default {}
|
||||
```
|
||||
|
||||
`to_row()` returns a flat dict with 11 keys (same names); `metadata` is JSON string, `ts` is isoformat string. `seq` is NOT in `to_row()` — it is DB-assigned.
|
||||
|
||||
### ActivityLogFilters — shape (all fields optional, default None)
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class ActivityLogFilters:
|
||||
categories: Sequence[str] | None # category IN (...)
|
||||
severities: Sequence[str] | None # severity IN (...)
|
||||
actor: str | None # exact match
|
||||
entity_type: str | None # exact match
|
||||
entity_id: str | None # exact match
|
||||
since: datetime | None # ts >= since
|
||||
until: datetime | None # ts <= until
|
||||
message_like: str | None # LIKE %value% (escaped)
|
||||
```
|
||||
|
||||
### Migration name used
|
||||
|
||||
`"002_add_activity_log"` — appended as position [1] in `ALL_MIGRATIONS`.
|
||||
|
||||
### ActivityLogRepository — exact method signatures
|
||||
|
||||
```python
|
||||
class ActivityLogRepository:
|
||||
def __init__(self, db: Database) -> None
|
||||
def record(self, entry: ActivityLogEntry) -> None
|
||||
def query(
|
||||
self,
|
||||
filters: ActivityLogFilters,
|
||||
*,
|
||||
before_seq: int | None = None,
|
||||
limit: int = 50,
|
||||
) -> list[ActivityLogEntry]
|
||||
def count(self, filters: ActivityLogFilters | None = None) -> int
|
||||
def prune(
|
||||
self,
|
||||
*,
|
||||
before_ts: datetime | None = None,
|
||||
max_entries: int | None = None,
|
||||
) -> int
|
||||
def clear(self) -> int
|
||||
def iter_export(
|
||||
self, filters: ActivityLogFilters | None = None
|
||||
) -> Iterator[ActivityLogEntry]
|
||||
```
|
||||
|
||||
### Key behavioural notes for Phase 2/3/4
|
||||
|
||||
- `record()` expects to be called from the event-loop thread (or with `Database` RLock already held). Phase 2 is responsible for thread marshaling via `loop.call_soon_threadsafe`.
|
||||
- `query()` returns entries in **ascending chronological order within the page** (reversed internally from DESC fetch for display convenience). The smallest `seq` on a page is `page[0]`'s seq — pass that as `before_seq` for the next page.
|
||||
- `count(None)` == `count(ActivityLogFilters())` — both count all rows.
|
||||
- `prune(before_ts=X, max_entries=N)` applies both predicates independently (age prune first, then count cap).
|
||||
- `iter_export` holds `db._lock` for the entire iteration. Phase 4 should stream the response and consume promptly.
|
||||
- `ActivityLogCategory` and `ActivityLogSeverity` are plain classes with string class-attributes and an `ALL` tuple — NOT `enum.Enum`.
|
||||
- Imports for Phase 2/3/4:
|
||||
```python
|
||||
from ledgrab.storage.activity_log import ActivityLogEntry, ActivityLogFilters, ActivityCategory, ActivitySeverity
|
||||
from ledgrab.storage.activity_log_repository import ActivityLogRepository
|
||||
```
|
||||
@@ -1,196 +0,0 @@
|
||||
# Phase 2: Recorder, actor context, retention, lifecycle
|
||||
|
||||
**Status:** ✅ Done
|
||||
**Parent plan:** [PLAN.md](./PLAN.md)
|
||||
**Domain:** backend
|
||||
|
||||
## Objective
|
||||
|
||||
Build the runtime layer over the Phase 1 repository: a thread-safe `ActivityRecorder` facade
|
||||
that persists an entry AND pushes a live `activity_logged` event; an actor `ContextVar`
|
||||
populated by the auth layer; a background `ActivityLogRetentionEngine` mirroring
|
||||
`AutoBackupEngine`; and the `main.py`/`dependencies.py` wiring (init, DI getter, retention
|
||||
start/stop, shutdown ordering). After this phase the audit log records nothing yet (no call
|
||||
sites) — that is Phase 3 — but the full machinery is live and unit-tested.
|
||||
|
||||
## Tasks
|
||||
|
||||
- [x] Create `server/src/ledgrab/core/activity_log/__init__.py` and
|
||||
`server/src/ledgrab/core/activity_log/recorder.py`:
|
||||
- `ActivityRecorder(repo: ActivityLogRepository, processor_manager, *, loop=None)`.
|
||||
- `record(category, action, *, severity="info", actor=None, entity_type=None,
|
||||
entity_id=None, entity_name=None, message, metadata=None) -> None`:
|
||||
- resolve `actor` from the actor `ContextVar` when not supplied, default `"system"`;
|
||||
- build an `ActivityLogEntry` (id `al_<uuid8>`, `ts=datetime.now(timezone.utc)`);
|
||||
- **thread-safe write:** if called on the event loop thread, write inline via
|
||||
`repo.record(entry)` then fire the live event; if called from another thread (zeroconf
|
||||
discovery), marshal the whole write+emit onto the loop via
|
||||
`loop.call_soon_threadsafe(...)`. Capture the loop lazily (mirror
|
||||
`utils/log_broadcaster.py:ensure_loop`/`call_soon_threadsafe`). Never raise into the
|
||||
caller — audit recording is best-effort and must not break the audited action; log
|
||||
failures at `warning`.
|
||||
- live push: `processor_manager.fire_event({"type": "activity_logged", "entry": entry_as_dict})`.
|
||||
- Provide a tiny helper to serialize an entry to the same dict shape the API returns
|
||||
(reuse in Phase 4 / frontend).
|
||||
- `enabled` flag honored: when retention settings say `enabled=false`, `record()` is a
|
||||
no-op — EXCEPT the "audit log disabled" event itself, which must be recorded before the
|
||||
flag takes effect (see retention engine).
|
||||
- [x] Actor `ContextVar`:
|
||||
- Add `current_actor: ContextVar[str]` (module-level, e.g. in `core/activity_log/context.py`
|
||||
or `api/auth.py`). In `verify_api_key` (`api/auth.py`), set it next to the existing
|
||||
`request.state.auth_label = ...` (both the authenticated label and the `"anonymous"`
|
||||
branch). Default `"system"` when unset. Ensure no cross-request leakage (set on every
|
||||
auth evaluation).
|
||||
- [x] Create `server/src/ledgrab/core/activity_log/retention.py`:
|
||||
- `ActivityLogRetentionEngine(repo, db, recorder)` mirroring `core/backup/auto_backup.py`:
|
||||
`_load_settings()`/`_save_settings()` via `db.get_setting("activity_log")` /
|
||||
`db.set_setting("activity_log", {...})`, `DEFAULT_SETTINGS = {"enabled": True,
|
||||
"max_days": 90, "max_entries": 20000}`.
|
||||
`async start()` → spawn `_retention_loop()` (`asyncio.create_task`); loop sleeps a sane
|
||||
interval (e.g. hourly) then calls `repo.prune(before_ts=now-max_days, max_entries=...)`.
|
||||
`async stop()` → cancel + await task. `get_settings()` / `async update_settings(...)`
|
||||
that persist and apply (changing `enabled` is logged via the recorder BEFORE disabling).
|
||||
- [x] Wiring:
|
||||
- `main.py`: instantiate `activity_log_repo = ActivityLogRepository(db)` (module level near
|
||||
other stores); in `lifespan` startup build `activity_recorder` + `activity_log_retention_engine`,
|
||||
pass to `init_dependencies(...)`, and `await activity_log_retention_engine.start()`.
|
||||
- In `lifespan` **shutdown**: record a `system` / `server_shutting_down` event via the
|
||||
recorder as the **first** shutdown action (before engines/db close), then
|
||||
`await _bounded("activity_log_retention.stop", activity_log_retention_engine.stop(), timeout=0.5)`.
|
||||
- `api/dependencies.py`: add `activity_recorder` + `activity_log_repo` +
|
||||
`activity_log_retention_engine` to `_deps`, parameters to `init_dependencies`, and
|
||||
getters `get_activity_recorder()`, `get_activity_log_repo()`,
|
||||
`get_activity_log_retention_engine()`.
|
||||
- [x] Realtime allowlist (order matters — do allowlist FIRST so the parity test stays green):
|
||||
- Add `'activity_logged'` to `_ALLOWED_SERVER_EVENT_TYPES` in
|
||||
`server/src/ledgrab/static/js/core/events-ws.ts` (+ a one-line comment naming the source).
|
||||
- Confirm `tests/test_events_ws_parity.py` passes with the new emit type.
|
||||
- [x] Unit tests `server/tests/core/test_activity_recorder.py` +
|
||||
`test_activity_log_retention.py`:
|
||||
- recorder persists an entry AND calls `fire_event` with `type=="activity_logged"`;
|
||||
- actor resolves from ContextVar; defaults to `"system"`; failure in repo doesn't raise;
|
||||
- cross-thread `record()` (call from a `threading.Thread`) routes through the loop and persists;
|
||||
- retention prunes per settings; settings round-trip via db; disabling logs the disable event.
|
||||
|
||||
## Files to Modify/Create
|
||||
|
||||
- `server/src/ledgrab/core/activity_log/__init__.py` — new
|
||||
- `server/src/ledgrab/core/activity_log/recorder.py` — new
|
||||
- `server/src/ledgrab/core/activity_log/context.py` — new (actor ContextVar) *(or place in auth.py)*
|
||||
- `server/src/ledgrab/core/activity_log/retention.py` — new
|
||||
- `server/src/ledgrab/api/auth.py` — modify: set actor ContextVar in `verify_api_key`
|
||||
- `server/src/ledgrab/main.py` — modify: instantiate, wire lifespan start/shutdown
|
||||
- `server/src/ledgrab/api/dependencies.py` — modify: `_deps`, `init_dependencies`, getters
|
||||
- `server/src/ledgrab/static/js/core/events-ws.ts` — modify: allowlist `activity_logged`
|
||||
- `server/tests/core/test_activity_recorder.py` — new
|
||||
- `server/tests/core/test_activity_log_retention.py` — new
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Recorder persists + fires `activity_logged`; never raises into callers; thread-safe from
|
||||
non-loop threads.
|
||||
- Actor ContextVar populated by auth; default `"system"`; no cross-request leakage.
|
||||
- Retention engine starts/stops cleanly in lifespan; prunes by age + count; settings persist.
|
||||
- `server_shutting_down` is recorded before teardown; no lost-on-graceful-shutdown entries.
|
||||
- `test_events_ws_parity.py` green (allowlist updated). Existing tests still green; `ruff` clean.
|
||||
|
||||
## Notes
|
||||
|
||||
- Reference: `core/backup/auto_backup.py` (engine shape, settings persistence, `_bounded`
|
||||
shutdown in `main.py`), `utils/log_broadcaster.py` (`ensure_loop`, `call_soon_threadsafe`
|
||||
thread marshaling), `core/processing/processor_manager.py:247` (`fire_event`).
|
||||
- **Do not add any instrumentation call sites in this phase** — only the machinery. Phase 3
|
||||
adds the `record(...)` calls. (Intermediate commit emits nothing; that is fine and green.)
|
||||
- Freeze the `ActivityLogEntry` dict shape here — Phase 4 (API response) and Phase 5
|
||||
(frontend `entry`) consume it.
|
||||
|
||||
## Review Checklist
|
||||
|
||||
- [x] All tasks completed
|
||||
- [x] Code follows project conventions (engine/DI patterns)
|
||||
- [x] No unintended side effects (no call sites yet; lifespan order correct)
|
||||
- [x] Build passes (ruff + pytest, incl. parity test)
|
||||
- [x] Tests pass (new + existing)
|
||||
|
||||
## Handoff to Next Phase
|
||||
|
||||
### recorder.record(...) — final signature
|
||||
|
||||
```python
|
||||
recorder.record(
|
||||
category: str, # ActivityCategory constant
|
||||
action: str, # verb-object label
|
||||
*,
|
||||
severity: str = "info", # ActivitySeverity constant
|
||||
actor: str | None = None, # resolved from current_actor ContextVar when None
|
||||
entity_type: str | None = None,
|
||||
entity_id: str | None = None,
|
||||
entity_name: str | None = None,
|
||||
message: str,
|
||||
metadata: dict | None = None,
|
||||
_bypass_enabled: bool = False, # internal: used by retention engine only
|
||||
) -> None
|
||||
```
|
||||
|
||||
### Actor ContextVar import path
|
||||
|
||||
```python
|
||||
from ledgrab.core.activity_log.context import current_actor
|
||||
```
|
||||
|
||||
### Module accessor (for non-DI sites)
|
||||
|
||||
```python
|
||||
from ledgrab.core.activity_log.recorder import get_module_recorder, set_module_recorder
|
||||
recorder = get_module_recorder() # returns ActivityRecorder | None
|
||||
```
|
||||
|
||||
### entry_to_dict helper (for API response serialisation)
|
||||
|
||||
```python
|
||||
from ledgrab.core.activity_log.recorder import entry_to_dict
|
||||
d = entry_to_dict(entry) # returns dict with 11 keys
|
||||
```
|
||||
|
||||
### Frozen `activity_logged` event payload shape
|
||||
|
||||
```python
|
||||
{
|
||||
"type": "activity_logged",
|
||||
"entry": {
|
||||
"id": str, # "al_<8-hex>"
|
||||
"ts": str, # ISO-8601 UTC string
|
||||
"category": str,
|
||||
"action": str,
|
||||
"severity": str,
|
||||
"actor": str,
|
||||
"entity_type": str | None,
|
||||
"entity_id": str | None,
|
||||
"entity_name": str | None,
|
||||
"message": str,
|
||||
"metadata": dict, # real dict, not JSON string
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### DI getter names (in `api/dependencies.py`)
|
||||
|
||||
```python
|
||||
from ledgrab.api.dependencies import (
|
||||
get_activity_recorder,
|
||||
get_activity_log_repo,
|
||||
get_activity_log_retention_engine,
|
||||
)
|
||||
```
|
||||
|
||||
### Notes for Phase 3
|
||||
|
||||
- Phase 3 instruments `fire_entity_event` in `api/dependencies.py` by calling
|
||||
`get_module_recorder()` there (not via FastAPI DI — it's a plain function).
|
||||
- The actor ContextVar is already set by `verify_api_key` before any route
|
||||
handler runs, so entity events carry the correct actor automatically.
|
||||
- `recorder.record(...)` never raises; Phase 3 call sites need no try/except.
|
||||
|
||||
Phase 2 landed (2026-06-09): ActivityRecorder, actor ContextVar, ActivityLogRetentionEngine,
|
||||
all wiring in main.py/dependencies.py/auth.py, activity_logged allowlist in events-ws.ts,
|
||||
24 new tests — all green. Full suite 2309 passed, 2 skipped, 0 failed. Ruff clean.
|
||||
@@ -1,172 +0,0 @@
|
||||
# Phase 3: Event instrumentation (4 categories)
|
||||
|
||||
**Status:** ✅ Done
|
||||
**Parent plan:** [PLAN.md](./PLAN.md)
|
||||
**Domain:** backend · 🔒 security-sensitive (security reviewer triggers)
|
||||
|
||||
## Objective
|
||||
|
||||
Emit audit records at the real call sites for all four categories, using the Phase 2 recorder.
|
||||
Maximize coverage via the central `fire_entity_event` choke point; add explicit
|
||||
`recorder.record(...)` calls for non-entity events. Never log secrets.
|
||||
|
||||
## Tasks
|
||||
|
||||
### Entity CRUD (via the choke point)
|
||||
- [x] In `api/dependencies.py`, extend `fire_entity_event` to ALSO record an audit entry:
|
||||
- Signature gains an optional `entity_name: str | None = None`.
|
||||
- For `created`/`updated`: if `entity_name` not supplied, best-effort resolve from the
|
||||
matching store in `_deps` keyed by `entity_type` (entity still present). For `deleted`:
|
||||
**do not** resolve post-hoc — rely on the explicit `entity_name` passed by the handler
|
||||
(deletes are the most important; a name-less delete entry is unacceptable).
|
||||
- Map `action` → severity (`info`), category `entity`. Build a human message
|
||||
(e.g. `"Target 'Desk' updated"`). Read actor from the ContextVar.
|
||||
- Recording is best-effort (never break the entity operation).
|
||||
- [x] Update entity **delete** handlers to pass `entity_name` into `fire_entity_event`
|
||||
(the entity object is already loaded for the 404 check). Cover the representative/most-used
|
||||
entities at minimum: output targets, sync clocks, devices, picture/audio/color-strip
|
||||
sources, automations, scene presets/playlists, templates, gradients. (Create/update can rely
|
||||
on hook resolution but pass the name where trivially available.)
|
||||
|
||||
### Authentication (DESCOPED: no key create/rotate/revoke — those routes don't exist)
|
||||
- [x] In `api/auth.py`, record:
|
||||
- auth **failures**: missing/invalid Bearer token (HTTP), rejected LAN-without-keys, rejected
|
||||
WS origin (4403), WS auth handshake failure (4401). Category `auth`, severity `warning`.
|
||||
Include the caller IP/label and the reason in `metadata` — **never** the attempted token.
|
||||
- WS **session establishment** (successful `accept_and_authenticate_ws`): category `auth`,
|
||||
severity `info`, actor = authenticated label.
|
||||
- (Do NOT record per-request HTTP auth *success* — too frequent.)
|
||||
|
||||
### Device connect/disconnect (use existing discrete seams)
|
||||
- [x] Hook `device_health_changed` (`core/processing/device_health.py`, fired only on
|
||||
`online != prev_online`) → record online/offline transition. Category `device`,
|
||||
severity `info` (online) / `warning` (offline).
|
||||
- [x] Hook `device_discovered` / `device_lost` (`core/devices/discovery_watcher.py`, **runs on
|
||||
the zeroconf thread** → recorder must marshal to the loop, which Phase 2 handles). Category
|
||||
`device`.
|
||||
- [x] ADB connect/disconnect (`api/routes/system_settings.py:adb_connect/adb_disconnect`).
|
||||
|
||||
### Capture & system events (explicit record calls)
|
||||
- [x] Target processing start/stop + bulk (`api/routes/output_targets_control.py`).
|
||||
- [x] Scene activation (`scene_presets.py:activate_scene_preset`), playlist start/stop
|
||||
(`scene_playlists.py`), automation activate/deactivate (`automation_engine.py`).
|
||||
- [x] System: backup create/restore/delete (`backup.py`), update apply/dismiss (`update.py`),
|
||||
restart/shutdown (`backup.py`), calibration start/stop/cancel (`calibration.py`).
|
||||
- [x] Settings changes: scope to high-value settings only (auto-backup, update, shutdown
|
||||
action). **Exclude the activity-log's own `"activity_log"` settings key** to avoid
|
||||
self-referential churn.
|
||||
|
||||
### Tests
|
||||
- [x] `server/tests/test_activity_instrumentation.py` (or per-area):
|
||||
- representative entity create/update/delete produces a record with correct category/actor/
|
||||
name (incl. a delete carrying its name);
|
||||
- an auth failure produces a `warning` record and the token never appears in any field;
|
||||
- a device health transition and a discovery event produce records;
|
||||
- a capture start and a backup/restore produce records.
|
||||
|
||||
## Files to Modify/Create
|
||||
|
||||
- `server/src/ledgrab/api/dependencies.py` — modify: `fire_entity_event` records + `entity_name`
|
||||
- entity **delete** route handlers under `api/routes/` — modify: pass `entity_name`
|
||||
- `server/src/ledgrab/api/auth.py` — modify: auth-failure + WS-session records
|
||||
- `server/src/ledgrab/core/processing/device_health.py` — modify: online/offline record
|
||||
- `server/src/ledgrab/core/devices/discovery_watcher.py` — modify: discovered/lost record
|
||||
- `server/src/ledgrab/api/routes/system_settings.py` — modify: ADB + settings records
|
||||
- `server/src/ledgrab/api/routes/output_targets_control.py` — modify: start/stop records
|
||||
- `server/src/ledgrab/api/routes/{scene_presets,scene_playlists,backup,update,calibration}.py` — modify
|
||||
- `server/src/ledgrab/core/automations/automation_engine.py` — modify: activate/deactivate records
|
||||
- `server/tests/test_activity_instrumentation.py` — new
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- All four categories emit records at the named sites; entity deletes carry the entity name.
|
||||
- API-key tokens / secrets never appear in any audit field (test-enforced).
|
||||
- Recording never breaks the audited action (best-effort; failures swallowed + logged).
|
||||
- Actor is the authenticated label for request-originated events, `"system"` for engine/thread
|
||||
events. New + existing tests green; `ruff` clean.
|
||||
|
||||
## Notes
|
||||
|
||||
- Get the recorder via the Phase 2 DI getter; for engine/thread sites that lack DI, use the
|
||||
module singleton/accessor Phase 2 exposes.
|
||||
- Keep messages human-readable and localized-agnostic (English source strings; the frontend
|
||||
renders structured fields, not server message translation — message is a fallback/summary).
|
||||
- This is the security-sensitive phase — the security reviewer runs here AND at final review.
|
||||
|
||||
## Review Checklist
|
||||
|
||||
- [x] All tasks completed
|
||||
- [x] Code follows project conventions
|
||||
- [x] No unintended side effects (audited actions still succeed on recorder failure)
|
||||
- [x] No secrets logged (token never recorded) — explicitly verified
|
||||
- [x] Build passes (ruff + pytest)
|
||||
- [x] Tests pass (new + existing)
|
||||
|
||||
## Handoff to Next Phase
|
||||
|
||||
Phase 3 is complete. The following (category, action) pairs are now emitted, along with their
|
||||
metadata keys, for Phase 4 to expose via query/filter and for Phase 5 quick-filter presets.
|
||||
|
||||
### `entity` category
|
||||
|
||||
| Action | Severity | Metadata keys | Notes |
|
||||
|--------|----------|---------------|-------|
|
||||
| `entity.created` | info | — | All entity types via `fire_entity_event` choke-point |
|
||||
| `entity.updated` | info | — | All entity types; name resolved from store when not passed |
|
||||
| `entity.deleted` | info | — | Name passed explicitly by delete handler before deletion |
|
||||
|
||||
### `auth` category
|
||||
|
||||
| Action | Severity | Metadata keys | Notes |
|
||||
|--------|----------|---------------|-------|
|
||||
| `auth.rejected` | warning | `reason` (str), `client` (str/IP) | Missing Bearer, invalid Bearer, LAN-no-keys, WS origin, WS auth timeout, invalid WS token |
|
||||
| `auth.ws_connected` | info | `client` (str/IP) | Successful WS session established |
|
||||
|
||||
### `device` category
|
||||
|
||||
| Action | Severity | Metadata keys | Notes |
|
||||
|--------|----------|---------------|-------|
|
||||
| `device.online` | info | `latency_ms` (float) | Health monitor, transition only |
|
||||
| `device.offline` | warning | `latency_ms` (float) | Health monitor, transition only |
|
||||
| `device.discovered` | info | `url` (str), `device_type` (str) | Zeroconf discovery thread; recorder marshals to loop |
|
||||
| `device.lost` | warning | `url` (str), `device_type` (str) | Zeroconf discovery thread |
|
||||
| `device.adb_connected` | info | `address` (str) | ADB route success |
|
||||
| `device.adb_disconnected` | info | `address` (str) | ADB route success |
|
||||
|
||||
### `capture` category
|
||||
|
||||
| Action | Severity | Metadata keys | Notes |
|
||||
|--------|----------|---------------|-------|
|
||||
| `capture.started` | info | — | Per target (individual + bulk) |
|
||||
| `capture.stopped` | info | — | Per target (individual + bulk) |
|
||||
| `scene.activated` | info | — | `scene_presets.py:activate_scene_preset` |
|
||||
| `playlist.started` | info | — | `scene_playlists.py:start_scene_playlist` |
|
||||
| `playlist.stopped` | info | — | `scene_playlists.py:stop_scene_playlist` |
|
||||
| `automation.activated` | info | — | `automation_engine.py:_activate_automation`; actor="system" |
|
||||
| `automation.deactivated` | info | — | `automation_engine.py:_deactivate_automation`; actor="system" |
|
||||
|
||||
### `system` category
|
||||
|
||||
| Action | Severity | Metadata keys | Notes |
|
||||
|--------|----------|---------------|-------|
|
||||
| `backup.created` | info | `filename` (str) | `backup.py:backup_config` |
|
||||
| `backup.restored` | info | — | `backup.py:restore_config` |
|
||||
| `backup.deleted` | info | `filename` (str) | `backup.py:delete_saved_backup` |
|
||||
| `server.restarting` | info | — | `backup.py:restart_server` |
|
||||
| `server.shutdown_requested` | info | — | `backup.py:shutdown_server` |
|
||||
| `update.dismissed` | info | `version` (str) | `update.py:dismiss_update` |
|
||||
| `update.applied` | info | `version` (str) | `update.py:apply_update` |
|
||||
| `settings.changed` | info | `setting_key` (str) + setting-specific keys | `setting_key` values: `"auto_backup"`, `"update"`, `"shutdown_action"`. Activity-log own key excluded. |
|
||||
| `calibration.started` | info | — | `calibration.py`; entity_type="device", entity_id=device_id |
|
||||
| `calibration.stopped` | info | — | `calibration.py` |
|
||||
| `calibration.cancelled` | info | — | `calibration.py` |
|
||||
|
||||
### Implementation notes for Phase 4
|
||||
|
||||
- The `metadata` field is a JSON `TEXT` column. All keys above are scalars (str, float).
|
||||
- Phase 4 filter `metadata_key` / `metadata_value` lookup, if added, can target `setting_key`
|
||||
for settings-change filtering.
|
||||
- `entity_type` is populated for entity CRUD and `calibration.started`. For auth/system/capture
|
||||
events `entity_type` may be None.
|
||||
- `entity_name` is always populated for `entity.deleted`; populated for CRUD create/update
|
||||
when resolved; populated for most capture/system events where a name is meaningful.
|
||||
@@ -1,163 +0,0 @@
|
||||
# Phase 4: REST API — query / filter / export / settings / clear
|
||||
|
||||
**Status:** ✅ Done
|
||||
**Parent plan:** [PLAN.md](./PLAN.md)
|
||||
**Domain:** backend
|
||||
|
||||
## Objective
|
||||
|
||||
Expose the audit log over the REST API: a filtered, keyset-paginated list endpoint; a
|
||||
streaming CSV/JSON export honoring the same filters; retention settings get/update; and a
|
||||
destructive clear. Apply the project's auth posture (stricter auth on export + clear).
|
||||
|
||||
## Tasks
|
||||
|
||||
- [x] `server/src/ledgrab/api/schemas/activity_log.py` (Pydantic):
|
||||
- `ActivityLogEntryResponse` (matches the frozen Phase 2 entry dict shape).
|
||||
- `ActivityLogPageResponse` { `entries: list[...]`, `next_before_seq: int | None`,
|
||||
`total: int` (optional/over filters), `has_more: bool` }.
|
||||
- `ActivityLogSettingsResponse` / `UpdateActivityLogSettingsRequest`
|
||||
(`enabled`, `max_days`, `max_entries`) with validation bounds.
|
||||
- [x] `server/src/ledgrab/api/routes/activity_log.py` — `APIRouter(prefix="/api/v1/activity-log")`:
|
||||
- `GET ""` — list. Query params: `categories`, `severities`, `actor`, `entity_type`,
|
||||
`entity_id`, `since`/`until` (ISO), `q` (free-text), `before_seq` (cursor), `limit`
|
||||
(default 50, capped e.g. 200). `AuthRequired`. Maps params → `ActivityLogFilters`,
|
||||
calls `repo.query(...)`, returns page envelope.
|
||||
- `GET "/export"` — streaming export. Same filters; `format=csv|json`. Uses
|
||||
`StreamingResponse` over `repo.iter_export(...)`. **`require_authenticated()`** (may
|
||||
contain IPs/labels). Sets `Content-Disposition` with a timestamped filename.
|
||||
- `GET "/settings"` / `PUT "/settings"` — retention settings via the retention engine.
|
||||
`AuthRequired`; updates apply immediately.
|
||||
- `DELETE ""` — clear all entries. **`require_authenticated()`**. The clear is itself
|
||||
audited (recorder records a `system`/`activity_log_cleared` entry AFTER the wipe, so the
|
||||
log shows who cleared it and when).
|
||||
- Register the router in `server/src/ledgrab/api/__init__.py` (aggregator).
|
||||
- [x] API tests `server/tests/api/routes/test_activity_log_api.py`:
|
||||
- list returns entries; each filter narrows results; `before_seq` cursor paginates without
|
||||
overlap/gaps; `limit` cap enforced;
|
||||
- export CSV and JSON both stream and honor filters; export requires authentication
|
||||
(401 for loopback-anonymous when keys configured);
|
||||
- settings get/update round-trip + validation rejects out-of-range;
|
||||
- clear empties the log, requires auth, and leaves exactly one post-clear audit entry.
|
||||
|
||||
## Files to Modify/Create
|
||||
|
||||
- `server/src/ledgrab/api/schemas/activity_log.py` — new
|
||||
- `server/src/ledgrab/api/routes/activity_log.py` — new
|
||||
- `server/src/ledgrab/api/__init__.py` — modify: register router
|
||||
- `server/tests/api/routes/test_activity_log_api.py` — new
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- List is filterable on every dimension and keyset-paginated (stable, no dupes/gaps).
|
||||
- Export streams CSV + JSON, honors filters, and requires authentication.
|
||||
- Settings get/update works and validates bounds; changes take effect immediately.
|
||||
- Clear requires authentication and is itself audited.
|
||||
- New + existing tests green; `ruff` clean.
|
||||
|
||||
## Notes
|
||||
|
||||
- Auth helpers: `AuthRequired` dependency for normal endpoints; `require_authenticated()` for
|
||||
export + clear (pattern: backup download / secret reveal). See `api/auth.py` + `server/CLAUDE.md`.
|
||||
- Follow the existing route/schema conventions (one schema file per entity, router registered
|
||||
in `api/__init__.py`). Reference `api/routes/backup.py` for settings-style GET/PUT + a
|
||||
`StreamingResponse`/download pattern.
|
||||
- Reuse the entry→dict serializer from Phase 2 to keep the response shape single-sourced.
|
||||
- Backup/restore: no `STORE_MAP` change needed — backup is whole-DB; the table is auto-covered.
|
||||
|
||||
## Review Checklist
|
||||
|
||||
- [x] All tasks completed
|
||||
- [x] Code follows project conventions (router registration, schema-per-entity, auth posture)
|
||||
- [x] No unintended side effects
|
||||
- [x] Build passes (ruff + pytest)
|
||||
- [x] Tests pass (new + existing)
|
||||
|
||||
## Handoff to Next Phase
|
||||
|
||||
### Endpoint paths
|
||||
|
||||
| Method | Path | Auth |
|
||||
|--------|------|------|
|
||||
| GET | `/api/v1/activity-log` | AuthRequired (anonymous allowed) |
|
||||
| GET | `/api/v1/activity-log/export` | require_authenticated (no anonymous) |
|
||||
| GET | `/api/v1/activity-log/settings` | AuthRequired |
|
||||
| PUT | `/api/v1/activity-log/settings` | AuthRequired |
|
||||
| DELETE | `/api/v1/activity-log` | require_authenticated (no anonymous) |
|
||||
|
||||
### List query parameters (GET /api/v1/activity-log)
|
||||
|
||||
| Param | Type | Default | Notes |
|
||||
|-------|------|---------|-------|
|
||||
| `categories` | `list[str]` | — | Repeatable. Values: auth, device, entity, capture, system |
|
||||
| `severities` | `list[str]` | — | Repeatable. Values: info, warning, error |
|
||||
| `actor` | `str` | — | Exact match |
|
||||
| `entity_type` | `str` | — | Exact match |
|
||||
| `entity_id` | `str` | — | Exact match |
|
||||
| `since` | `datetime` (ISO-8601) | — | Inclusive lower bound on ts |
|
||||
| `until` | `datetime` (ISO-8601) | — | Inclusive upper bound on ts |
|
||||
| `q` | `str` | — | Substring match on message (LIKE %q%) |
|
||||
| `before_seq` | `int` | — | Keyset cursor from previous page's `next_before_seq` |
|
||||
| `limit` | `int` | 50 | Max entries per page. ge=1, le=200 |
|
||||
|
||||
Export endpoint (`GET /api/v1/activity-log/export`) accepts the same filter params plus `format=csv|json`.
|
||||
|
||||
### Page envelope fields (ActivityLogPageResponse)
|
||||
|
||||
```json
|
||||
{
|
||||
"entries": [...], // list[ActivityLogEntryResponse]
|
||||
"next_before_seq": 42, // int | null — pass as before_seq for next page
|
||||
"has_more": true, // bool
|
||||
"total": 1337 // int — total matching all filters (all pages)
|
||||
}
|
||||
```
|
||||
|
||||
### Entry dict shape (ActivityLogEntryResponse)
|
||||
|
||||
11 fields — identical to `entry_to_dict()` output:
|
||||
```json
|
||||
{
|
||||
"id": "al_abcd1234",
|
||||
"ts": "2026-06-09T12:34:56.789+00:00",
|
||||
"category": "entity",
|
||||
"action": "entity.created",
|
||||
"severity": "info",
|
||||
"actor": "my-api-key",
|
||||
"entity_type": "output_target",
|
||||
"entity_id": "pt_abc",
|
||||
"entity_name": "Desk",
|
||||
"message": "Output target 'Desk' created",
|
||||
"metadata": {}
|
||||
}
|
||||
```
|
||||
|
||||
### Settings field bounds (UpdateActivityLogSettingsRequest)
|
||||
|
||||
| Field | Type | ge | le | Notes |
|
||||
|-------|------|----|----|-------|
|
||||
| `enabled` | `bool` | — | — | Enable/disable recording |
|
||||
| `max_days` | `int` | 0 | 3650 | 0 = no age-based pruning |
|
||||
| `max_entries` | `int` | 0 | 10_000_000 | 0 = no count-based pruning |
|
||||
|
||||
### Export format param
|
||||
|
||||
`?format=csv` (default) → `text/csv; charset=utf-8`
|
||||
`?format=json` → `application/json` (streamed JSON array)
|
||||
|
||||
### Pagination algorithm
|
||||
|
||||
Keyset cursor (`before_seq`) works as follows:
|
||||
- Omit `before_seq` (or pass `null`) to get the FIRST (newest) page.
|
||||
- Each page response includes `next_before_seq` (the seq of the oldest entry on the page).
|
||||
- Pass `next_before_seq` as `before_seq` in the next request to get the following (older) page.
|
||||
- `has_more=false` means there are no more pages; `next_before_seq` is `null`.
|
||||
- `total` is constant across pages for the same filter set.
|
||||
|
||||
### New method added to ActivityLogRepository (additive, not breaking)
|
||||
|
||||
`get_seq_for_id(entry_id: str) -> int | None` — indexed point-lookup of seq by entry id.
|
||||
Used internally by the list endpoint to build the keyset cursor.
|
||||
|
||||
Phase 4 landed (2026-06-09): schemas, route (list/export/settings/clear), router registration,
|
||||
49 new tests — all green. Full suite 2486 passed, 2 skipped, 0 failed. Ruff clean.
|
||||
@@ -1,137 +0,0 @@
|
||||
# Phase 5: Frontend — Activity tab + smart filtering + live updates
|
||||
|
||||
**Status:** ✅ Done
|
||||
**Parent plan:** [PLAN.md](./PLAN.md)
|
||||
**Domain:** frontend · uses the `frontend-design` skill
|
||||
|
||||
## Objective
|
||||
|
||||
Build the dedicated top-level **Activity** tab: a read-only, smart-filterable,
|
||||
keyset-paginated log viewer with an entry detail view, live-append of new events, and export.
|
||||
This is a viewer (Dashboard-style), NOT a CRUD card section.
|
||||
|
||||
## Tasks
|
||||
|
||||
- [x] `core/ui.ts`: add `formatTimestamp(isoOrMs)` (Today/Yesterday/Date · HH:MM, i18n-aware)
|
||||
and `formatRelativeTime(isoOrMs)` ("2m ago"), with `tabular-nums` styling guidance for the
|
||||
list. Reuse existing `time.*` i18n key conventions.
|
||||
- [x] `features/activity-log.ts`:
|
||||
- `export async function loadActivityLog()` — fetch first page from
|
||||
`GET /activity-log` (via `fetchWithAuth`), render the toolbar + list into the panel.
|
||||
- **Smart filter toolbar:** category (multi, chips), severity (chips), actor
|
||||
(text input), entity type, date range, free-text search (debounced). Quick presets:
|
||||
Today / Errors / Auth / Devices. Filters drive server-side query params (no client-side
|
||||
filtering of a partial page). Re-query on change; reset cursor.
|
||||
- **List:** one row per entry — severity icon, category badge, relative time (title=absolute),
|
||||
actor, message, entity crosslink (use `navigateToCard(...)` when the referenced entity is
|
||||
resolvable). Keyset "load more" (or infinite scroll) using `next_before_seq`.
|
||||
- **Detail:** expandable row / drawer showing full metadata JSON, exact timestamp, ids.
|
||||
- **Live append:** `document.addEventListener('server:activity_logged', e => …)` — prepend
|
||||
the new entry if it passes the active filters; show a subtle "new" affordance. (Depends on
|
||||
the Phase 2 allowlist entry — already shipped.)
|
||||
- **Export button:** triggers `GET /activity-log/export?format=…` with current filters via an
|
||||
authed blob download (use `fetchWithAuth` → blob URL, per frontend.md auth rules).
|
||||
- Empty / loading / error states; re-render on `languageChanged`.
|
||||
- [x] Tab wiring:
|
||||
- `core/tab-registry.ts`: add `activity_log: { loadFnName: 'loadActivityLog', autoRefresh: false }`.
|
||||
- `templates/index.html`: sidebar tab button (`data-tab`, `switchTab('activity_log')`,
|
||||
history/clock SVG icon, `data-i18n`) + `<div class="tab-panel" id="tab-activity_log">`.
|
||||
- `app.ts`: import + `Object.assign(window, { loadActivityLog, … })`; `global.d.ts` decls.
|
||||
- [x] Icons: add a history/audit icon to `core/icon-paths.ts` + `core/icons.ts`; severity icons
|
||||
(info/warning/error) reuse existing constants where possible.
|
||||
- [x] i18n: add `activity_log.*` keys to `static/locales/{en,ru,zh}.json` (title, filter labels,
|
||||
category/severity names, column labels, presets, empty/error, export, "N entries").
|
||||
- [x] Tutorials: add an Activity-tab step to the getting-started tour in
|
||||
`features/tutorials.ts` + `tour.*` keys in all 3 locales.
|
||||
|
||||
## Files to Modify/Create
|
||||
|
||||
- `server/src/ledgrab/static/js/core/ui.ts` — modified: timestamp/relative-time formatters
|
||||
- `server/src/ledgrab/static/js/features/activity-log.ts` — NEW: the viewer
|
||||
- `server/src/ledgrab/static/js/core/tab-registry.ts` — modified: register tab
|
||||
- `server/src/ledgrab/templates/index.html` — modified: tab button + panel
|
||||
- `server/src/ledgrab/static/js/app.ts` — modified: import + window globals
|
||||
- `server/src/ledgrab/static/js/global.d.ts` — modified: window decls
|
||||
- `server/src/ledgrab/static/js/core/icon-paths.ts` — modified: icons (scrollText, circleAlert, info, filter, xCircle)
|
||||
- `server/src/ledgrab/static/js/core/icons.ts` — modified: ICON_ACTIVITY_LOG, ICON_SEVERITY_*, ICON_FILTER, ICON_X_CIRCLE
|
||||
- `server/src/ledgrab/static/locales/{en,ru,zh}.json` — modified: activity_log.* + time.* + tour.activity_log
|
||||
- `server/src/ledgrab/static/js/features/tutorials.ts` — modified: gettingStartedSteps
|
||||
- `server/src/ledgrab/static/css/activity-log.css` — NEW: list + toolbar styling
|
||||
- `server/src/ledgrab/static/css/all.css` — modified: import activity-log.css
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] New **Activity** tab loads, lists entries, and paginates via keyset "load more".
|
||||
- [x] Filters hit server-side query params; quick presets work; free-text is debounced.
|
||||
- [x] New events append live via `server:activity_logged` and respect active filters.
|
||||
- [x] Export downloads CSV/JSON with auth, honoring current filters.
|
||||
- [x] Fully localized (en/ru/zh); empty/loading/error states; re-renders on language change.
|
||||
- [x] No plain `<select>` (use chips); SVG icons only (no emoji).
|
||||
- [x] `npx tsc --noEmit` clean; `npm run build` succeeds.
|
||||
|
||||
## Notes
|
||||
|
||||
- **Used the `frontend-design` skill** for the viewer layout, filter toolbar, and detail design.
|
||||
- Models mirrored: `features/dashboard.ts` (live viewer pattern), `core/events-ws.ts` (WS dispatch),
|
||||
`core/api.ts` (fetchWithAuth), `core/navigation.ts` (navigateToCard).
|
||||
- Frontend-only phase — ran `tsc --noEmit` + `npm run build`; did NOT run pytest/ruff.
|
||||
|
||||
## Review Checklist
|
||||
|
||||
- [x] All tasks completed
|
||||
- [x] Code follows frontend conventions (i18n ×3, icons, auth fetch, CSS vars)
|
||||
- [x] No unintended side effects
|
||||
- [x] Build passes (`tsc --noEmit` + `npm run build`)
|
||||
- [ ] Manual smoke: tab loads, filters query server, live append, export
|
||||
|
||||
## Handoff to Next Phase
|
||||
|
||||
### Reusable helpers for Phase 6 (Dashboard widget + Settings panel)
|
||||
|
||||
**From `features/activity-log.ts`:**
|
||||
|
||||
| Export | Purpose | How Phase 6 uses it |
|
||||
|--------|---------|---------------------|
|
||||
| `loadActivityLog()` | Loads the full tab (already registered as the tab loader) | Phase 6 doesn't call this, but it shares state |
|
||||
| `activityLogToggleDetail(id)` | Expand/collapse an entry row | Dashboard widget can reuse for the Recent Activity list |
|
||||
| `activityLogExport(format)` | Authed blob download with current filters | Could surface a direct link in the Settings panel |
|
||||
| `_renderEntryRow(entry, isNew)` | Renders a single entry as an HTML string | Dashboard widget should import this to render its 5-10 most-recent entries |
|
||||
| `_fetchPage(beforeSeq, append)` | Keyset-paged fetch from `/activity-log` | Dashboard widget calls with `limit=5&categories=…` for its mini-list |
|
||||
|
||||
**Note:** `_renderEntryRow` and `_fetchPage` are module-private (underscore prefix). Phase 6 should
|
||||
either (a) re-export them with public names, or (b) duplicate the minimal render logic for the
|
||||
compact widget format.
|
||||
|
||||
**Recommended approach for Phase 6:** Export two new public helpers:
|
||||
|
||||
```typescript
|
||||
// Add to activity-log.ts
|
||||
export async function fetchRecentEntries(limit = 5): Promise<ActivityEntry[]>
|
||||
export function renderCompactEntry(entry: ActivityEntry): string
|
||||
```
|
||||
|
||||
### i18n namespace
|
||||
|
||||
All keys are under `activity_log.*`. Time-relative formatters use `time.*` keys.
|
||||
|
||||
### CSS classes and tokens introduced
|
||||
|
||||
| Class | Purpose |
|
||||
|-------|---------|
|
||||
| `.al-panel` | Tab root wrapper |
|
||||
| `.al-toolbar` | Filter toolbar container |
|
||||
| `.al-chip` / `.al-chip.active` | Category/severity toggle chips |
|
||||
| `.al-preset-btn` | Quick-preset buttons |
|
||||
| `.al-entry` / `.al-entry-row` | Log entry row |
|
||||
| `.al-detail` / `.al-detail-grid` | Expandable entry detail |
|
||||
| `.al-cat-*` | Per-category badge colors (auth/device/entity/capture/system) |
|
||||
| `.al-sev-info/warning/error` | Severity icon color classes |
|
||||
| `.al-live-dot` | Pulsing green live-update dot |
|
||||
| `.al-meta-pre` | Scrollable metadata JSON block |
|
||||
| `.tabular-nums` | `font-variant-numeric: tabular-nums` utility |
|
||||
|
||||
### Settings endpoint shape used
|
||||
|
||||
Phase 6 Settings panel will call:
|
||||
- `GET /activity-log/settings` → `{ enabled: bool, max_days: int, max_entries: int }`
|
||||
- `PUT /activity-log/settings` → same shape (bounds: enabled=bool, max_days=0–3650, max_entries=0–10_000_000)
|
||||
@@ -1,98 +0,0 @@
|
||||
# Phase 6: Dashboard widget + Settings retention panel + docs
|
||||
|
||||
**Status:** ✅ Done
|
||||
**Parent plan:** [PLAN.md](./PLAN.md)
|
||||
**Domain:** frontend · uses the `frontend-design` skill
|
||||
|
||||
## Objective
|
||||
|
||||
Add the two secondary surfaces: a compact **Recent Activity** widget on the Dashboard linking
|
||||
to the full tab, and a **Settings** panel for retention configuration (+ clear + export entry)
|
||||
positioned beside the existing Log Viewer with a clear "what's the difference" note. Update
|
||||
docs/tutorials.
|
||||
|
||||
## Tasks
|
||||
|
||||
- [x] Dashboard **Recent Activity** widget (`features/dashboard.ts` + dashboard CSS):
|
||||
- Compact card showing the latest ~5 entries (severity icon, relative time, message).
|
||||
- Reuse the Phase 5 render helper (don't duplicate row markup).
|
||||
- Live update via `server:activity_logged` (prepend, cap to N).
|
||||
- **View all →** navigates to the Activity tab (`switchTab('activity_log')`).
|
||||
- Respect the existing dashboard card layout/toggle system; localized; empty state.
|
||||
- [x] **Settings** retention panel (`features/settings.ts` + `templates/modals/settings.html`):
|
||||
- New rail entry `Activity Log` (beside the existing **Log Viewer**).
|
||||
- Controls: `enabled` toggle, `max_days` number, `max_entries` number → `GET/PUT
|
||||
/activity-log/settings`. Save → toast; validation feedback.
|
||||
- **Clear log** button (confirm dialog) → `DELETE /activity-log`; **Export** button →
|
||||
`/activity-log/export`.
|
||||
- One-line note distinguishing this persistent audit log from the ephemeral debug Log Viewer
|
||||
(cross-link both ways).
|
||||
- i18n for all controls/labels/hints.
|
||||
- [x] Docs: update user-facing docs/README/feature list for the new Activity tab + retention
|
||||
settings + export (and the audit-vs-debug-log distinction). Keep it brief.
|
||||
- [x] Tutorials/cross-links: ensure the Settings tutorial (if any) and tab tour mention the
|
||||
panel; `tour.*`/`settings.*` i18n keys in all 3 locales.
|
||||
|
||||
## Files to Modify/Create
|
||||
|
||||
- `server/src/ledgrab/static/js/features/dashboard.ts` — modify: Recent Activity widget
|
||||
- `server/src/ledgrab/static/css/dashboard.css` (or relevant sheet) — modify: widget styles
|
||||
- `server/src/ledgrab/static/js/features/settings.ts` — modify: retention panel + handlers
|
||||
- `server/src/ledgrab/templates/modals/settings.html` — modify: rail entry + panel HTML
|
||||
- `server/src/ledgrab/static/js/app.ts` / `global.d.ts` — modify: new window handlers
|
||||
- `server/src/ledgrab/static/locales/{en,ru,zh}.json` — modify: i18n keys
|
||||
- docs (README / feature list / relevant context doc) — modify: brief feature documentation
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Dashboard shows a live Recent Activity widget; **View all →** opens the Activity tab.
|
||||
- Settings panel reads/writes retention settings, clears (with confirm + auth), and exports.
|
||||
- Audit-log vs debug-Log-Viewer distinction is explicit and cross-linked.
|
||||
- Fully localized (en/ru/zh); empty/loading states; consistent with app design.
|
||||
- `npx tsc --noEmit` clean; `npm run build` succeeds. No plain `<select>`; SVG icons only.
|
||||
|
||||
## Notes
|
||||
|
||||
- **Use the `frontend-design` skill** for the widget + settings panel layout.
|
||||
- Reuse Phase 5's render helper and i18n namespace — no duplicated row markup or keys.
|
||||
- Settings UI model: `features/settings.ts switchSettingsTab` + `modals/settings.html` rail.
|
||||
- Frontend-only phase → run `tsc --noEmit` + `npm run build`; do NOT run pytest/ruff (docs +
|
||||
TS only).
|
||||
- Backup/restore cross-ref: no `STORE_MAP` edit needed (whole-DB backup covers the table) —
|
||||
confirm nothing else (graph editor) needs syncing for this viewer (verified: not needed).
|
||||
|
||||
## Review Checklist
|
||||
|
||||
- [x] All tasks completed
|
||||
- [x] Code follows frontend conventions (i18n ×3, icons, selectors, CSS vars, settings pattern)
|
||||
- [x] No unintended side effects
|
||||
- [x] Build passes (`tsc --noEmit` + `npm run build`)
|
||||
- [ ] Manual smoke: widget live-updates + links; settings save/clear/export; docs updated (recommend at final review — requires server restart)
|
||||
|
||||
## Handoff to Next Phase
|
||||
|
||||
This is the **final implementation phase**. All six phases are complete. Notes for the final reviewer:
|
||||
|
||||
**What was implemented in Phase 6:**
|
||||
|
||||
- `features/dashboard.ts`: `_loadRecentActivityWidget()`, `_renderRecentActivityList()`, `_startRecentActivityLive()` — SSE live-update listener (`server:activity_logged`) with cap-to-5 prepend logic. Widget appended after `getOrderedSections()` loop (outside the layout toggle system — always visible). Non-blocking: `.catch()` on the async load call.
|
||||
- `features/settings.ts`: `loadActivityLogSettings()`, `saveActivityLogSettings()`, `activityLogSettingsExport(format)`, `clearActivityLog()` — all exported and exposed on `window` via `app.ts` + `global.d.ts`.
|
||||
- `templates/modals/settings.html`: Activity Log rail entry (System group, cyan channel) + full panel (`id="settings-panel-activity_log"`) with enabled toggle, `max_days`/`max_entries` inputs, Save, CSV/JSON export, Clear (danger zone). Audit-vs-debug distinction note with cross-links in both directions (`closeSettingsModal(); openLogOverlay()` and `closeSettingsModal(); switchTab('activity_log')`).
|
||||
- `static/css/activity-log.css`: `.dal-*` dashboard widget styles + `.ds-info-note` / `.ds-inline-link` settings panel utilities appended (no new CSS file).
|
||||
- `static/locales/{en,ru,zh}.json`: 32 new keys each under `dashboard.section.recent_activity`, `dashboard.recent_activity.*`, `settings.tab.activity_log`, `settings.activity_log.*`.
|
||||
- `README.md`: "### Activity Log" section documenting tab, retention settings, export, and audit-vs-debug distinction.
|
||||
|
||||
**Reused Phase 5 helpers (no duplication):**
|
||||
|
||||
- `fetchRecentEntries(limit)` and `renderCompactEntry(entry)` — new public exports added to `activity-log.ts` in Phase 6 (not duplicated in dashboard.ts).
|
||||
- All `.al-*` CSS classes from Phase 5 are reused in the compact rows inside the widget.
|
||||
|
||||
**Build verification:** `tsc --noEmit` clean, `npm run build` passed (2.8 MB bundle, 258 ms) at time of implementation.
|
||||
|
||||
**Remaining manual smoke test (requires server restart):**
|
||||
|
||||
- Dashboard widget loads recent entries, prepends live on new activity, "View all →" switches to Activity tab.
|
||||
- Settings panel reads/writes retention, Save shows toast, Clear prompts confirm then deletes, Export downloads authed blob.
|
||||
- Cross-links: note in Settings opens Log Viewer overlay; note in Log Viewer links back to Activity tab.
|
||||
|
||||
**Outstanding open note from PLAN.md:** The manual browser smoke test across the whole feature (P5 note) remains open — it requires a server restart to exercise the live API endpoints. Recommend as first step in the final review session.
|
||||
@@ -28,6 +28,18 @@ API key authentication via Bearer token in the `Authorization` header (`Authoriz
|
||||
- When `api_keys` is **set**: a valid Bearer token is required from every client (loopback included).
|
||||
- `require_authenticated()` rejects even loopback-anonymous callers on sensitive endpoints (e.g. backup download, secret reveal).
|
||||
|
||||
## Activity / Audit Log
|
||||
|
||||
Persistent, queryable audit log of meaningful actions (auth, device, entity CRUD, capture, system), surfaced in the WebUI (Activity tab + Dashboard widget + Settings retention panel).
|
||||
|
||||
- **Storage is NOT a `BaseSqliteStore`.** `storage/activity_log_repository.py` is a purpose-built repository over a dedicated indexed `activity_log` table (migration `002_add_activity_log`) — query-on-demand with **keyset pagination** (`seq` cursor), never load-all-into-memory. Don't route it through the entity-store pattern.
|
||||
- **Recording.** `core/activity_log/recorder.py` (`ActivityRecorder`) is best-effort (never raises into the audited action) and **thread-safe** (inline on the event loop; `loop.call_soon_threadsafe` from non-loop threads, e.g. zeroconf discovery). It persists the entry **and** fires an `activity_logged` realtime event. Actor comes from the `current_actor` `ContextVar` (set in `verify_api_key`), default `"system"`.
|
||||
- **Entity CRUD is auto-audited** via the `fire_entity_event()` choke point in `api/dependencies.py` — every create/update/delete already calls it. **Delete handlers must pass `entity_name`** (the entity is gone by record time). Non-entity events use explicit `recorder.record(...)` (get it via `get_activity_recorder()` DI or `get_module_recorder()` for engine/thread sites).
|
||||
- **Never log secrets.** API-key tokens are never recorded. Wrap any untrusted/attacker-controllable string (mDNS names, headers, user-authored names) with `sanitize_display()` (`core/activity_log/sanitize.py`) before it enters a `message`/`metadata` field. Per-IP throttle bounds auth-failure audit writes.
|
||||
- **Adding a new audited event:** pick a dotted `action` (e.g. `"thing.created"`), call the recorder; for it to render localized in the UI, add `activity_log.msg.<action>` to all three `static/locales/*.json` (the frontend `localizeMessage()` maps action→template; falls back to the server `message`). Entity-type labels live under `activity_log.entity_type.<type>`.
|
||||
- **Adding a new realtime event type** (`pm.fire_event({"type": ...})`): add it to `_ALLOWED_SERVER_EVENT_TYPES` in `static/js/core/events-ws.ts` AND keep `tests/test_events_ws_parity.py` green.
|
||||
- **Retention + API.** `core/activity_log/retention.py` prunes by `max_days` + `max_entries` (settings persisted via `db.set_setting("activity_log")`); the recorder's `enabled` flag is rehydrated from those settings on startup. REST in `api/routes/activity_log.py`: `GET /activity-log` (list, `AuthRequired`), `GET /export` (CSV/JSON stream — `require_authenticated`; chunked keyset so it never holds the DB lock across the stream; CSV formula-injection guarded), `GET|PUT /settings` (PUT is `require_authenticated`), `DELETE` (clear — `require_authenticated`, self-audited). The table is covered by the existing whole-DB backup (no `STORE_MAP` change needed).
|
||||
|
||||
## Common Tasks
|
||||
|
||||
### Adding a new API endpoint
|
||||
|
||||
Reference in New Issue
Block a user