Commit Graph

15 Commits

Author SHA1 Message Date
alexei.dolgolyov 10d30fc956 feat: production readiness — security, perf, bug fixes, bridge self-monitoring
Comprehensive multi-area pass driven by a parallel 8-agent production
review. Frontend, backend, database, security, performance, operational,
plus a new self-monitoring feature.

## Critical fixes
- Planka webhook: reads bounded raw body (was NameError on every call)
- HA quiet hours: ha_state_changed/automation_triggered/service_called/
  event_fired added to deferrable set (were silently dropped)
- DNS-rebinding SSRF: PinnedResolver wired into shared aiohttp session
- Telegram inbound webhook: secret now mandatory (401 without)
- Generic webhook: auth_mode="none" requires explicit
  acknowledge_unauthenticated=true; per-IP rate limit 60/min
- svelte-check: 5 null-narrowing errors in EventDetailModal fixed
- Provider hardcoding: Immich-only block extracted to descriptor
  featureDiscoveryHint
- command_sync: snapshot+expunge bot before exiting AsyncSession

## Bug fixes
- notifier asyncio.gather(return_exceptions=True) — one bad chat no longer
  cancels peer sends
- NotificationDispatcher hoisted out of per-tracker loop
- Provider credential resolution unified across all 5 dispatch sites
- HA asyncio.shield now drains inner task on cancellation
- Provider construction switched from if/elif ladder to factory registry
- NUT first poll seeds silently (no spurious ups_on_battery)
- Quiet-hours gate: event-type-disabled now wins over deferral
- APScheduler drain job ID resolution upgraded to seconds
- HA on_status_change wired through to EventLog
- Webhook payload rollback failures now logged (not swallowed)
- Batched receivers/chats/bots in load_link_data (was per-target N+1)
- flag_modified on JSON column reassignments in deferred_dispatch

## Database
- UNIQUE indexes on service_provider.webhook_token,
  telegram_bot.webhook_path_id, partial UNIQUE on telegram_bot.bot_id,
  telegram_chat(bot_id, chat_id), notification_tracker_target unique link,
  partial UNIQUE on bridge_self provider per user
- Composite ix_event_log_user_event_type_created index
- save_chat_from_webhook switched to ON CONFLICT DO UPDATE
- ondelete=CASCADE on user-id FKs (model annotation; app-side cascade
  delete added for existing data)
- delete_notification_tracker converted from N+1 to bulk DELETE/UPDATE
- Module-level asyncio.Lock replaced with lazy _get_lock() pattern
- VACUUM INTO snapshot now PRAGMA integrity_check verified

## Performance
- Jinja2 template compilation LRU cached (lru_cache maxsize=512)
- Per-locale render cache in NotificationDispatcher (skips re-rendering
  identical content for receivers sharing a locale)
- Tracker list cached per provider_id with 5s TTL + explicit invalidation
  on tracker CRUD (relieves HA chat-bus rate query pressure)
- Nav-counts collapsed from 16 round-trips to single UNION ALL
- HA event_log: skip persisting empty assets_added/removed events

## Security hardening
- Mass-assignment guard on Action create/update; cron sub-minute reject
- Backup JSON depth/node-count cap (depth ≤ 10, nodes ≤ 100k)
- _sanitize_config extended to all JSON-typed fields on backup import
- Telegram _safe_get walks redirects manually with SSRF revalidation
- Bcrypt 72-byte password length cap with clear 422
- Webhook payload body redaction; sensitive substring set extended with
  oauth/client_secret/webhook_secret/csrf in both header filter and
  template extras filter

## Frontend
- 76 catch (err: any) sites converted to errMsg(err) helper
- globalProviderFilter: pure getter; reconciliation moved to one-time
  $effect in +layout
- Provider-filter binding: removed paired $effects + _syncingFilter flag,
  now one-way derived
- entity-cache: separate _refreshing flag for background re-fetches
- api.ts 401 handling: AuthRedirectError class + dedup _redirecting flag,
  goto() instead of window.location.href
- a11y: aria-expanded on mobile More, role=switch + aria-checked on
  Telegram bot toggles

## Tests & operations
- CI pytest gate added to .gitea/workflows/build.yml + release.yml
  (wheel-built install to dodge editable-install slowness)
- /api/ready upgraded to deep healthcheck (db SELECT 1, scheduler.running,
  HA supervisor presence) returning {ready, checks, errors, version}
- /api/metrics endpoint with prometheus_client (deferred_pending,
  event_log_total, dispatch_duration, poll_failures, send_failures)
- New OPERATIONS.md covering deploy, healthchecks, metrics, backup/restore
  procedures, log handling, common scenarios, upgrade flow
- New tests: test_bridge_self (11), test_gitea_parser (9),
  test_planka_parser (6), test_immich_change_detector (6),
  test_backup_roundtrip (1)

## New feature: bridge self-monitoring
- New bridge_self provider type — internal sink for bridge health events
- Three event types: bridge_self_poll_failures (consecutive tracker poll
  failures), bridge_self_deferred_backlog (pending count crosses
  threshold), bridge_self_target_failures (consecutive 5xx/network
  failures per target)
- Per-user thresholds (defaults: 3 / 100 / 5) configurable via the
  provider config form
- Auto-seeded on user create + /setup + boot backfill for existing users
- Anti-spam: counters reset after emission; backlog uses transition latch
- Self-loop guard: bridge_self failures don't count toward target-failure
  thresholds (logged only) — wire to your own Telegram/Email/Matrix to
  get notified when polls/dispatches/sends fail
- 6 default templates (3 events × 2 locales), tracking config columns
  with backfill migration, frontend descriptor (excluded from "create
  provider" wizard since auto-managed)

Operator-visible behavior changes (call out in release notes):
- NOTIFY_BRIDGE_TELEGRAM_WEBHOOK_SECRET now REQUIRED for webhook mode
- Existing webhook providers with auth_mode="none" need explicit opt-in
- Generic webhook endpoint rate-limited 60/min per source IP
- HA disconnect/reconnect writes ha_status_* EventLog rows
- Every user gets a bridge_self provider — wire it to a target to
  receive failure alerts

Pre-existing test failures (test_ssrf, test_release_provider) on
Python 3.13 are unrelated; CI runs on 3.12.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-16 02:16:49 +03:00
alexei.dolgolyov 22127e2a59 feat: Home Assistant provider — WebSocket subscription + bot commands
Adds Home Assistant as a service provider with two coordinated surfaces:

Notifications (subscription):
- Long-lived WebSocket client (aiohttp ws_connect) with auth handshake,
  exponential-backoff reconnect, bounded event queue, and area-registry
  enrichment cached per (re)connect
- ServiceProvider ABC gains an optional `subscribe()` method for push-style
  providers; HomeAssistantServiceProvider uses it via a per-provider
  supervisor task started in the FastAPI lifespan
- 4 event types (state_changed, automation_triggered, call_service,
  event_fired), 4 default Jinja templates (en + ru), HA-specific
  tracker filters (entity_glob, domain_allowlist, exact entity ids)
- Extracted shared dispatch pipeline (api/webhooks.py → services/
  event_dispatch.py) so subscription and webhook ingest share the same
  event_log + deferred-dispatch + quiet-hours code path

Bot commands:
- /status, /entities [glob], /state <entity_id>, /areas
- Multi-command WS session so /status and /areas cost one handshake
- Sensitive-attribute blocklist (camera access_token, entity_picture, etc.)
  and 30-attribute cap to keep /state output safe and within Telegram's
  message size
- Error-message redaction strips URL userinfo before surfacing to chat

Frontend:
- HA descriptor with toggle ConfigField type (new) and tag-input filter
  mode for free-text glob/domain lists (new TagInput component)
- 15 command slots + 4 notification slots wired into the existing
  template-config UI
2026-05-13 14:31:56 +03:00
alexei.dolgolyov b170c2b792 feat(frontend): smoother event refresh, localized crumbs, template config deep-link
- Auto-refresh ticker is now silent: skips ``eventsLoading`` so the
  loading placeholder no longer flashes, uses ``(event.id)`` key on
  the events ``{#each}`` so unchanged rows reuse their DOM nodes, and
  short-circuits the array reassignment when the visible page is
  identical to what we already rendered. No-op refreshes leave the
  list completely untouched.
- ``PageHeader`` crumbs (Routing · Notification, Operators · Bots, …)
  were hard-coded literals. Moved to a new ``crumbs`` i18n namespace
  with 9 keys; updated all 15 call sites to ``t('crumbs.*')`` so they
  switch with the language.
- Tracker form's Immich feature-discovery banner now exposes both
  ``Open Tracking Config`` and ``Open Template Config``. Added the
  ``?edit=<id>`` auto-open hook to ``/template-configs`` (mirrors the
  existing one on ``/tracking-configs``) so the new link lands users
  directly on the editor.
2026-05-07 22:34:24 +03:00
alexei.dolgolyov 5bd63a2191 feat(frontend): autogenerate entity names from type/provider
Mirror the providers form pattern (defaultName tied to type) across
bots, targets, trackers, actions, and configs. Each form now derives
form.name from the selected type or provider while the user hasn't
manually edited it; switching to edit-mode flips the manualEdited
flag so existing names are preserved.

Defaults: bots → "<Type> Bot"; targets → type label; notification
trackers → "<provider> Tracker"; command trackers → "<provider>
Commands"; actions → "<provider> <Action Type>"; tracking/template/
command/command-template configs → "<descriptor.defaultName>
<Suffix>". TargetForm and TrackerForm grew an optional onnameinput
prop so parents can flag manual edits in subform inputs.
2026-05-07 13:01:52 +03:00
alexei.dolgolyov 42af7a6551 feat(trackers): user filters for Gitea, webhook polling cleanup, dashboard navigability
- Gitea: NotificationTracker now exposes sender allowlist / blocklist filters
  via MultiEntitySelect, populated from Gitea /users/search merged with past
  EventLog senders so the picker is useful before the first webhook arrives.
- Webhook providers (gitea, planka, webhook): stop scheduling interval polling
  jobs on tracker create/update/startup; hide the "every Xs" indicator in the
  tracker list since there is no polling.
- Dashboard: stat cards are now <a> links that route to providers, trackers,
  targets, command-trackers, or scroll to the events panel. Provider deck
  rows highlight the target provider on click.
- Command trackers / command configs: auto-reselect the right config when the
  provider type changes (matches notification-tracker behavior).
- Migration: drop legacy batch_duration column from notification_tracker —
  the field is gone from the model but its NOT NULL constraint blocked
  inserts on older DBs.
- Docs: refresh entity-relationships.md with current NotificationTracker
  fields (filters, adaptive_max_skip, default_*_config_id).
2026-04-27 15:24:44 +03:00
alexei.dolgolyov 711f218622 fix(redesign): a11y, mobile, perf polish for production push
Comprehensive pre-production sweep across the Aurora redesign — drives
svelte-check to 0 errors / 0 warnings (was 61) without changing visual
intent. Highlights:

- Mobile: hero title shrinks at 480px, signal-list stacks timestamp
  under sentence below 640px, sidebar icon buttons bumped to 40x40
- Light theme: muted-foreground darkened to #3a3560 to clear WCAG AA
  on glass surfaces and the modal close button
- Perf: topbar backdrop-filter 28→14px, mobile-more sheet 28→12px to
  cut concurrent blur layers on mid-tier mobile
- a11y: prefers-reduced-motion mute for aurora drift / pulses /
  shimmer / stagger; aria-label on every icon-only button;
  aria-describedby on Hint; combobox/listbox/aria-activedescendant on
  SearchPalette; modal dialog tabindex; 47 label-without-control
  warnings across 14 form pages cleaned up via for=/id= or label→div
- Dashboard derived state split into topology- vs status-bound layers
  so polling no longer re-runs the full provider/wires computation
- Mobile bottom nav derived from baseNavEntries by key lookup so
  adding a top-level nav entry keeps the two trees in sync
- Bug: template-configs page now respects the global provider filter
  for both the count meter and the type pill (was reading the
  unfiltered cache)
- Misc: portal EventChart tooltip and switch its swatches to Aurora
  tokens; CollapsibleSlot warning state uses warning-fg/-bg tokens
  instead of #d97706; Hint z-index 99999→9999; element refs across
  Modal/EntitySelect/MultiEntitySelect/SearchPalette/IconGridSelect/
  Hint/targets converted to \$state for reactivity; 4 dead
  .topbar-cta selectors removed
2026-04-25 14:41:12 +03:00
alexei.dolgolyov ab621b6abc feat: wire tracking-config display filters + per-tracker adaptive polling
Display filters (Immich tracking config):
- favorites_only drops events with no favorited new assets, or filters
  added_assets to favorites only
- assets_order_by/assets_order sort the rendered list
  (date / name / rating / random / none)
- max_assets_to_show caps rendered+attached media (default 5 -> 10)
- include_tags strips people from event extras and tags from each asset
- include_asset_details strips city/country/state/lat/lon/is_favorite/
  rating/description; load-bearing fields (thumbhash, file_size,
  playback_size, cache keys) preserved
- New apply_tracking_display_filters helper in dispatch_helpers; wired
  into watcher, webhooks, scheduled/periodic/memory, and manual
  test-dispatch
- Targets sharing a TrackingConfig dispatch together; targets with
  different TCs each see their own shaped event

Adaptive polling:
- Replace NotificationTracker.batch_duration with adaptive_max_skip
- Per-tracker opt-in: NULL/0 disables back-off (every tick runs);
  positive N caps the skip factor at (N-1)-in-N after long idle
- Scheduler caches the cap in module state for the tick fast-path
- Migration adds the new column; API schemas/responses, frontend types,
  i18n, and the tracker form updated to match
2026-04-24 21:12:10 +03:00
alexei.dolgolyov b61394f057 feat(immich): per-album scheduled/memory dispatch + template tooling
Dispatch: honor {kind}_collection_mode on TrackingConfig — "per_collection"
fans out one event per album; "combined" pools assets as before. Extract
build_immich_dispatch_events shared by cron and test paths.

Assets: collect_scheduled_assets attaches album_name/album_url/album_public_url
to each asset so combined-mode templates can attribute rows to their source
album. Default scheduled_assets templates render a multi-album header with
inline album list and per-row album link; memory_mode follows the same pattern.

UI: "Reset to default" buttons on notification and command template slots
(per-slot and whole-template), backed by new GET /*-template-configs/defaults
endpoints. tracking-configs "Preview template" now opens an inline preview
modal with locale tabs instead of navigating away; Edit button deep-links
with ?edit_slot=<name> so the destination auto-opens the config and scrolls
to the slot. Reset confirmations use ConfirmModal instead of window.confirm.

Fixes:
* NotificationDispatcher._session_ctx infinite recursion when no shared
  aiohttp.ClientSession was passed — broke test dispatch for periodic/
  scheduled/memory (cron path was unaffected).
* telegram-bots /chats/{id}/test now resolves chat.language_override /
  language_code instead of using the raw ?locale query param, matching
  the resolution the tracker-target test endpoint already used.
* scheduled_assets default template no longer emits a blank line between
  header and the first asset when the multi-album branch is taken.
2026-04-24 19:15:54 +03:00
alexei.dolgolyov 6e35926772 feat: default tracker configs, email validation, expandable target links
- Tracker now has default_tracking_config_id and default_template_config_id
  that apply to all linked targets unless overridden per-target
- Dispatch falls back to tracker defaults when per-link configs are null
- Email bot creation validates SMTP connection before saving
- Email notifications sent as HTML (links render properly)
- Linked target items are expandable: collapsed shows config CrossLinks,
  expanded shows config selectors; action buttons always visible
- Fix email bot test button icon (mdiEmailSend → mdiSend)
- Fix target type icons in LinkedTargetsSection for all types
- Provider filter moved above search in sidebar
2026-03-24 22:32:37 +03:00
alexei.dolgolyov 8cb836e16c refactor: provider descriptor registry — eliminate provider-specific hardcoding
Replace all if/else chains keyed on provider type strings with a
descriptor-driven architecture. Each provider type (immich, gitea,
planka, scheduler, nut, google_photos) has a descriptor in
frontend/src/lib/providers/ that declares config fields, event
tracking fields, collection metadata, validation, and hooks.

Components now use getDescriptor(type) and render dynamically.
Dashboard provider card shows provider name + type when global
filter is active. Grid-items derived from registry.
2026-03-24 12:40:33 +03:00
alexei.dolgolyov 68ac13b452 feat: NUT (Network UPS Tools) service provider + provider-agnostic UI
Add full NUT support as a polling-based service provider:
- Async TCP client for upsd protocol (port 3493, configurable)
- 8 event types: online, on_battery, low_battery, battery_restored,
  comms_lost, comms_restored, replace_battery, overload
- 3 bot commands: /status, /devices, /battery
- 38 Jinja2 templates (EN+RU notification + command templates)
- Database: tracking config fields, migration, seeds
- Frontend: provider form with host/port/credentials, grid items, i18n

Provider-agnostic UI improvements:
- Remove hardcoded 'immich' defaults from all config forms
- Dynamic collection labels per provider type (Albums/Repos/Boards/UPS Devices)
- Capability-driven test types instead of provider type checks
- Template variable helpers for all providers (was Immich-only)
- Guard Immich-only shared link check to Immich providers
- Auto-clear stale global provider filter from localStorage
- EntitySelect search placeholder shows current selection
- Fix noneLabel in linked target config selectors

New CLAUDE.md rule #8: no provider-specific hardcoding
2026-03-23 23:23:58 +03:00
alexei.dolgolyov 6a559bfcd2 feat: Actions system — scheduled mutations on external services
Full-stack implementation of provider-scoped Actions with extensible
executor architecture. First action type: Immich auto_organize (sort
assets into albums by person, CLIP search, date range, favorites).

Core:
- ActionTypeDefinition registry + ActionExecutor ABC with execute/validate/dry-run
- ImmichActionExecutor with multi-album support and client-side filtering
- ImmichClient write methods: add/remove assets, create album, paginated search

Server:
- Action, ActionRule, ActionExecution DB models
- Full CRUD API + manual execute + dry-run + execution history endpoints
- APScheduler integration (interval + cron) for automated execution
- Action type discovery API + provider people endpoint

Frontend:
- Actions page with CRUD, execute/dry-run buttons, inline rule editor
- RuleEditor: person/album MultiEntitySelect pickers, criteria config
- ExecutionHistory: expandable per-rule result details
- MultiEntitySelect reusable component (searchable multi-pick palette)
- Notification tracker album picker migrated to MultiEntitySelect
- Fixed MdiIcon race condition (icons missing after cache-clearing reload)
2026-03-23 16:59:20 +03:00
alexei.dolgolyov 63437c1841 refactor: provider-agnostic bot command system + Gitea commands
Refactored the monolithic command handler (707 lines) into a pluggable
provider-handler architecture:

- Abstract ProviderCommandHandler interface (base.py)
- Handler dispatch registry routes commands by provider type
- Extracted all Immich logic into ImmichCommandHandler
- New GiteaCommandHandler with /status, /repos, /issues, /prs, /commits
- Multi-provider routing: groups context by provider type, finds handler
- handler.py reduced to ~280 line thin orchestrator

Gitea commands:
- Extended GiteaClient with get_repo_issues, get_repo_pulls, get_repo_commits
- 30 Jinja2 command templates (15 EN + 15 RU)
- Gitea capabilities updated with 6 commands + 15 command_slots
- Default command config + command template config seeded on startup
- Rate limiting: Gitea API commands share "api" category (15s cooldown)

Also:
- Command configs API accepts "gitea" provider type
- System command configs (user_id=0) visible to all users
- Webhook URL shown on Gitea provider card and edit form
- Scan interval hidden for webhook-based providers
2026-03-22 17:44:47 +03:00
alexei.dolgolyov 0562f78b35 feat: add Scheduler provider + multi-provider UX fixes
Scheduler provider:
- Virtual provider (no external service) that emits SCHEDULED_MESSAGE
  events on user-defined intervals or cron expressions
- Custom variables stored in tracker filters, flattened into template context
- fire_count persists across triggers via tracker state
- APScheduler CronTrigger support for cron-mode schedules
- Default templates (EN+RU), seeded on startup

Multi-provider UX fixes:
- Tracking config hides Immich-specific sections (periodic, scheduled,
  memory, asset display) for non-Immich providers
- Command config driven by provider capabilities — hides commands/settings
  for providers without bot commands
- Template config hides empty "Scheduled Messages" group
- Test menu on tracker targets is provider-aware (Immich shows all 4 test
  types, others show only basic)
- Removed redundant Test button from tracker card
- System-owned tracking configs (user_id=0) seeded for Gitea + Scheduler
- Fixed ownership checks to allow system configs in tracker-target links
- Capabilities cache shared across template-configs and command-configs
- Command tracker bot selector uses EntitySelect instead of raw select
- Sample context includes Gitea + Scheduler variables for template preview
2026-03-22 15:50:51 +03:00
alexei.dolgolyov 751097b347 feat: comprehensive code review fixes + receivers-only architecture
Security:
- Refuse startup with default secret_key in production (was just logging)
- Settings endpoint now requires admin role
- Password validation on initial setup
- DOM-based HTML sanitizer replaces regex in template previews
- Add *.log to .gitignore

Performance & reliability:
- Token refresh deduplication prevents race condition on concurrent 401s
- Theme media query listener registered once (no leak)
- IconPicker uses $derived instead of function call per render
- Snackbar uses single-batch state update instead of while loop
- Replace 11 inline hover handlers with CSS :hover in layout

Architecture - receivers-only:
- Delivery endpoints (chat_id, email, url, room_id, topic) now stored
  exclusively in TargetReceiver rows, never in target.config
- Migration extracts existing delivery fields to receiver rows
- Notifier and dispatcher remove all config fallbacks
- Frontend targets page shows receivers list per target with
  add/remove/toggle/test per receiver
- Single-receiver test endpoint: POST /targets/{id}/receivers/{id}/test

Code quality:
- Extract AuthLayout.svelte from login/setup (150 lines CSS dedup)
- Split telegram-bots page (754→51 lines + 3 tab components)
- Split notification-trackers page (547→432 lines + 4 components)
- Deduplicate _send_reply into shared handler.send_reply()
- Add locale column to template models, replace name-based detection
- Fix delete_notification_tracker dead protection check
- Fix check_telegram_bot query (filter by type, remove bogus OR)
- Add graceful scheduler shutdown in lifespan
- Consistent /bots?tab=telegram URLs across all nav links

i18n:
- Error page, chat actions, target types, provider types internationalized
- All new receiver UI strings in EN + RU
2026-03-22 02:19:31 +03:00