ledgrab/plans/activity-log/CONTEXT.md

CONTEXT — Activity / Audit Log

Living scratchpad for the feature. The orchestrator updates this between phases. Tier-2 context (survives across phases; graduates to CLAUDE.md only if it's a lasting project truth).

Config (from approval)

• Mode: Automated · Execution: Orchestrator · Strategy: Incremental
• Base/merge target: master · Branch: feature/activity-log · Branch point: 17dd2e0
• Final merge ALWAYS requires user approval (even in Automated mode).

Product decisions (locked)

• Placement: BOTH a top-level Activity tab AND a Dashboard Recent Activity widget, plus a Settings retention panel.
• Scope: all four categories — entity CRUD, auth, device connect/disconnect, capture & system.
• Detail: action metadata only (no before/after diffs).
• Durability: export on demand (CSV/JSON) + existing whole-DB backup. No separate backup subsystem.
• WebUI work uses the frontend-design skill (Phases 5–6).

Key codebase facts (verified during planning)

• fire_entity_event(entity_type, action, entity_id) @ api/dependencies.py:202 — central hook, called by every entity route synchronously in-request; has _deps store access.
• ProcessorManager.fire_event(dict) / subscribe_events() @ core/processing/processor_manager.py back /api/v1/events/ws (api/routes/output_targets_control.py:206). fire_event does put_nowait (no call_soon_threadsafe) — fine for the existing consumer; recorder marshals.
• Frontend realtime: core/events-ws.ts re-dispatches server:<type>; _ALLOWED_SERVER_EVENT_TYPES (line 39) is parity-checked by tests/test_events_ws_parity.py. Must add activity_logged.
• Actor: request.state.auth_label set in api/auth.py (129 authenticated, 83 anonymous).
• Storage: Database singleton (storage/database.py, single conn, RLock, WAL, synchronous=FULL, get_setting/set_setting). BaseSqliteStore loads ALL rows to memory → AVOID for the log. Migrations: storage/data_migrations.py (ALL_MIGRATIONS, idempotent).
• Backup/restore is whole-DB (Database.backup_to/restore_from, no STORE_MAP allowlist) → the new activity_log table is auto-covered. No system.py STORE_MAP edit needed.
• Background-engine pattern: core/backup/auto_backup.py (start/stop loop, _prune, settings).
• Thread-marshal precedent: utils/log_broadcaster.py (ensure_loop + call_soon_threadsafe).
• Device seams: device_health_changed (core/processing/device_health.py, on transition) + device_discovered/device_lost (core/devices/discovery_watcher.py, zeroconf thread).
• No API-key create/rotate/revoke routes exist (only GET /system/api-keys) → those auth events are DESCOPED.
• Existing Log Viewer (utils/log_broadcaster.py) = ephemeral debug-log tail; the audit log is a different, persistent, structured feature. Differentiate; do not duplicate.

Frozen contracts (fill as phases complete)

• ActivityLogEntry fields / dict shape: frozen — see phase-1-storage.md Handoff section. 11 fields: id, ts, category, action, severity, actor, message, entity_type, entity_id, entity_name, metadata. seq is DB-only (not on dataclass).
• ActivityLogFilters shape: frozen — 8 optional fields: categories, severities, actor, entity_type, entity_id, since, until, message_like. See phase-1-storage.md Handoff.
• recorder.record(...) signature + actor ContextVar import path: frozen — see phase-2-recorder-retention.md Handoff section. Signature: record(category, action, *, severity="info", actor=None, entity_type=None, entity_id=None, entity_name=None, message, metadata=None, _bypass_enabled=False). ContextVar: from ledgrab.core.activity_log.context import current_actor. Module accessor: from ledgrab.core.activity_log.recorder import get_module_recorder. Event payload: {"type": "activity_logged", "entry": {11-field dict with ts as ISO string, metadata as dict}}. DI getters: get_activity_recorder(), get_activity_log_repo(), get_activity_log_retention_engine().
• API endpoints + query params + page envelope + settings bounds: frozen — see phase-4-api.md Handoff section. Endpoints: GET /api/v1/activity-log (list, AuthRequired), GET /api/v1/activity-log/export (stream CSV/JSON, require_authenticated), GET|PUT /api/v1/activity-log/settings (AuthRequired), DELETE /api/v1/activity-log (clear, require_authenticated). Page envelope: entries, next_before_seq, has_more, total. Settings fields: enabled (bool), max_days (0–3650), max_entries (0–10_000_000). Export: ?format=csv|json.

Failed approaches / rejected designs

• Buffered async-writer subsystem (asyncio.Queue): REJECTED — unsafe from the zeroconf thread and adds a shutdown-flush ordering hazard. Using direct synchronous-on-loop writes with call_soon_threadsafe marshaling instead (simpler + correct).
• Using BaseSqliteStore for the log: REJECTED — loads all rows into memory.
• Separate activity-log backup subsystem: REJECTED — whole-DB backup already covers the table; export-on-demand is the portability story.

Deferred / open

• Setup-scaffold first-run noise suppression (batch/suppress flag) — deferred (nice-to-have).

Phase progress notes

Phase 1 landed (2026-06-09): activity_log.py (dataclass + enums + filters + codec), AddActivityLogTableMigration (002_add_activity_log) appended to ALL_MIGRATIONS, ActivityLogRepository (record/query/count/prune/clear/iter_export), 41 new tests — all green. Full suite 2226 passed, 0 failed. Schema and method signatures frozen in phase-1-storage.md Handoff. Gotcha: Database.execute takes a positional tuple — use ? placeholders (not :name), otherwise Python 3.14 will raise ProgrammingError. Phase 2 landed (2026-06-09): core/activity_log/ package (context.py, recorder.py, retention.py, __init__.py); actor ContextVar set in api/auth.py (both branches); ActivityLogRetentionEngine mirroring AutoBackupEngine; full wiring in main.py (repo at module level, recorder+engine in lifespan, server.shutting_down first shutdown action, engine stop before db.close); DI getters in api/dependencies.py; activity_logged added to _ALLOWED_SERVER_EVENT_TYPES in events-ws.ts; set_module_recorder exposes recorder to non-DI sites; 24 new tests — all green. Full suite 2309 passed, 2 skipped, 0 failed. Ruff clean. Phase 4 landed (2026-06-09): schemas (api/schemas/activity_log.py), routes (api/routes/activity_log.py: list/export/settings/clear), router registration in api/__init__.py, get_seq_for_id helper on ActivityLogRepository. 49 new tests — all green. Full suite 2486 passed, 2 skipped, 0 failed. Ruff clean. Pagination bug found and fixed (limit+1 probe must drop oldest row when has_more, not tail). Phase 3 landed (2026-06-09): instrumented all four categories — entity CRUD via fire_entity_event choke-point (dependencies.py), auth failures + WS session in auth.py, device online/offline in device_health.py, device discovered/lost in discovery_watcher.py, ADB connect/disconnect in system_settings.py, capture start/stop (individual + bulk) in output_targets_control.py, scene/playlist/automation activate in their respective route/engine files, backup/restore/delete + restart/shutdown/update/calibration/settings in backup.py/update.py/calibration.py; all 11 entity delete handlers pass entity_name to fire_entity_event; 22 new tests (security: token never in any field, explicitly asserted) — all green. Full suite 2369 passed, 2 skipped, 0 failed. Ruff clean. Complete (category, action) inventory in phase-3-instrumentation.md Handoff section. Phase 5 landed (2026-06-09): Activity tab frontend — features/activity-log.ts (loadActivityLog, filter toolbar with category/severity chips + presets + debounced search + date range + actor/entity-type text fields, keyset-paginated list, expandable detail drawer, live-prepend via server:activity_logged, authed CSV/JSON blob export); core/ui.ts additions (formatTimestamp, formatRelativeTime); core/icon-paths.ts + core/icons.ts additions (scrollText/audit, circleAlert, info, filter, xCircle); core/tab-registry.ts registered; templates/index.html tab button + panel; app.ts + global.d.ts wired; static/css/activity-log.css (precision-ledger design, category color rail, live-dot pulse, detail drawer, responsive breakpoints); all three locales updated (activity_log.* + time.* relative-time keys + tour.activity_log); features/tutorials.ts getting-started tour extended. tsc --noEmit clean, npm run build passes. Reusable helpers for Phase 6 documented in phase-5-frontend-tab.md Handoff section. Phase 6 landed (2026-06-09): Dashboard "Recent Activity" widget (live SSE, View-all link, .dal-* CSS, loading/empty states) + Settings "Activity Log" panel (enabled toggle, max_days/max_entries, Save toast, authed CSV/JSON export, confirmed Clear, audit-vs-debug cross-links) + 32 i18n keys per locale + README "Activity Log" section. tsc --noEmit clean, npm run build passes. All six phases complete — feature ready for final review.