36 Commits

Author SHA1 Message Date
alexei.dolgolyov 6a8f374678 feat: observability, per-receiver Telegram options, oversized-video fallback
Operability:
- Correlation IDs end-to-end: shared dispatch_id between log lines and
  EventLog rows (event/watcher/scheduled/deferred/action/HA/command paths)
  and a new X-Request-Id middleware that normalizes inbound ids and binds
  request_id into log context.
- dispatch_summary block merged into EventLog.details: per-target
  success/failure counts plus Telegram media delivered/skipped/failed and
  truncated error lists, so partial outcomes surface in the UI.
- Diagnostic mode: admin can flip one module to DEBUG for a bounded
  window with auto-revert (in-memory only; setup_logging() resets on
  boot, lifespan reverts on shutdown). New /diagnostic-mode endpoints
  plus DiagnosticsCassette UI on the settings page.

Telegram:
- Per-receiver options: disable_notification (silent send) and
  message_thread_id (forum-topic routing), wired through the dispatcher
  via a ContextVar so all four send sites (sendMessage / sendPhoto-Video-
  Document / sendMediaGroup / cache-hit POST) pick them up.
- send_large_videos_as_documents target setting: bypass the 50 MB
  sendVideo cap by falling back to sendDocument for oversized videos.
- sendMediaGroup byte-budget enforcement (TELEGRAM_MAX_GROUP_TOTAL_BYTES,
  45 MB) with per-item fallback on chunk failure so a stale file_id no
  longer silently drops a cached asset.

Tests:
- New: diagnostic_mode, dispatch_summary, request_correlation,
  telegram_media_group_partial, telegram_per_send_options.

Docs:
- .claude/reviews/: six-axis production-readiness review of v0.8.1.
- .claude/docs/functional-review-2026-05-28.md: focused review of
  Telegram/Immich/logging subsystems.
2026-05-28 15:19:31 +03:00
alexei.dolgolyov 10d30fc956 feat: production readiness — security, perf, bug fixes, bridge self-monitoring
Comprehensive multi-area pass driven by a parallel 8-agent production
review. Frontend, backend, database, security, performance, operational,
plus a new self-monitoring feature.

## Critical fixes
- Planka webhook: reads bounded raw body (was NameError on every call)
- HA quiet hours: ha_state_changed/automation_triggered/service_called/
  event_fired added to deferrable set (were silently dropped)
- DNS-rebinding SSRF: PinnedResolver wired into shared aiohttp session
- Telegram inbound webhook: secret now mandatory (401 without)
- Generic webhook: auth_mode="none" requires explicit
  acknowledge_unauthenticated=true; per-IP rate limit 60/min
- svelte-check: 5 null-narrowing errors in EventDetailModal fixed
- Provider hardcoding: Immich-only block extracted to descriptor
  featureDiscoveryHint
- command_sync: snapshot+expunge bot before exiting AsyncSession

## Bug fixes
- notifier asyncio.gather(return_exceptions=True) — one bad chat no longer
  cancels peer sends
- NotificationDispatcher hoisted out of per-tracker loop
- Provider credential resolution unified across all 5 dispatch sites
- HA asyncio.shield now drains inner task on cancellation
- Provider construction switched from if/elif ladder to factory registry
- NUT first poll seeds silently (no spurious ups_on_battery)
- Quiet-hours gate: event-type-disabled now wins over deferral
- APScheduler drain job ID resolution upgraded to seconds
- HA on_status_change wired through to EventLog
- Webhook payload rollback failures now logged (not swallowed)
- Batched receivers/chats/bots in load_link_data (was per-target N+1)
- flag_modified on JSON column reassignments in deferred_dispatch

## Database
- UNIQUE indexes on service_provider.webhook_token,
  telegram_bot.webhook_path_id, partial UNIQUE on telegram_bot.bot_id,
  telegram_chat(bot_id, chat_id), notification_tracker_target unique link,
  partial UNIQUE on bridge_self provider per user
- Composite ix_event_log_user_event_type_created index
- save_chat_from_webhook switched to ON CONFLICT DO UPDATE
- ondelete=CASCADE on user-id FKs (model annotation; app-side cascade
  delete added for existing data)
- delete_notification_tracker converted from N+1 to bulk DELETE/UPDATE
- Module-level asyncio.Lock replaced with lazy _get_lock() pattern
- VACUUM INTO snapshot now PRAGMA integrity_check verified

## Performance
- Jinja2 template compilation LRU cached (lru_cache maxsize=512)
- Per-locale render cache in NotificationDispatcher (skips re-rendering
  identical content for receivers sharing a locale)
- Tracker list cached per provider_id with 5s TTL + explicit invalidation
  on tracker CRUD (relieves HA chat-bus rate query pressure)
- Nav-counts collapsed from 16 round-trips to single UNION ALL
- HA event_log: skip persisting empty assets_added/removed events

## Security hardening
- Mass-assignment guard on Action create/update; cron sub-minute reject
- Backup JSON depth/node-count cap (depth ≤ 10, nodes ≤ 100k)
- _sanitize_config extended to all JSON-typed fields on backup import
- Telegram _safe_get walks redirects manually with SSRF revalidation
- Bcrypt 72-byte password length cap with clear 422
- Webhook payload body redaction; sensitive substring set extended with
  oauth/client_secret/webhook_secret/csrf in both header filter and
  template extras filter

## Frontend
- 76 catch (err: any) sites converted to errMsg(err) helper
- globalProviderFilter: pure getter; reconciliation moved to one-time
  $effect in +layout
- Provider-filter binding: removed paired $effects + _syncingFilter flag,
  now one-way derived
- entity-cache: separate _refreshing flag for background re-fetches
- api.ts 401 handling: AuthRedirectError class + dedup _redirecting flag,
  goto() instead of window.location.href
- a11y: aria-expanded on mobile More, role=switch + aria-checked on
  Telegram bot toggles

## Tests & operations
- CI pytest gate added to .gitea/workflows/build.yml + release.yml
  (wheel-built install to dodge editable-install slowness)
- /api/ready upgraded to deep healthcheck (db SELECT 1, scheduler.running,
  HA supervisor presence) returning {ready, checks, errors, version}
- /api/metrics endpoint with prometheus_client (deferred_pending,
  event_log_total, dispatch_duration, poll_failures, send_failures)
- New OPERATIONS.md covering deploy, healthchecks, metrics, backup/restore
  procedures, log handling, common scenarios, upgrade flow
- New tests: test_bridge_self (11), test_gitea_parser (9),
  test_planka_parser (6), test_immich_change_detector (6),
  test_backup_roundtrip (1)

## New feature: bridge self-monitoring
- New bridge_self provider type — internal sink for bridge health events
- Three event types: bridge_self_poll_failures (consecutive tracker poll
  failures), bridge_self_deferred_backlog (pending count crosses
  threshold), bridge_self_target_failures (consecutive 5xx/network
  failures per target)
- Per-user thresholds (defaults: 3 / 100 / 5) configurable via the
  provider config form
- Auto-seeded on user create + /setup + boot backfill for existing users
- Anti-spam: counters reset after emission; backlog uses transition latch
- Self-loop guard: bridge_self failures don't count toward target-failure
  thresholds (logged only) — wire to your own Telegram/Email/Matrix to
  get notified when polls/dispatches/sends fail
- 6 default templates (3 events × 2 locales), tracking config columns
  with backfill migration, frontend descriptor (excluded from "create
  provider" wizard since auto-managed)

Operator-visible behavior changes (call out in release notes):
- NOTIFY_BRIDGE_TELEGRAM_WEBHOOK_SECRET now REQUIRED for webhook mode
- Existing webhook providers with auth_mode="none" need explicit opt-in
- Generic webhook endpoint rate-limited 60/min per source IP
- HA disconnect/reconnect writes ha_status_* EventLog rows
- Every user gets a bridge_self provider — wire it to a target to
  receive failure alerts

Pre-existing test failures (test_ssrf, test_release_provider) on
Python 3.13 are unrelated; CI runs on 3.12.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-16 02:16:49 +03:00
alexei.dolgolyov 22127e2a59 feat: Home Assistant provider — WebSocket subscription + bot commands
Adds Home Assistant as a service provider with two coordinated surfaces:

Notifications (subscription):
- Long-lived WebSocket client (aiohttp ws_connect) with auth handshake,
  exponential-backoff reconnect, bounded event queue, and area-registry
  enrichment cached per (re)connect
- ServiceProvider ABC gains an optional `subscribe()` method for push-style
  providers; HomeAssistantServiceProvider uses it via a per-provider
  supervisor task started in the FastAPI lifespan
- 4 event types (state_changed, automation_triggered, call_service,
  event_fired), 4 default Jinja templates (en + ru), HA-specific
  tracker filters (entity_glob, domain_allowlist, exact entity ids)
- Extracted shared dispatch pipeline (api/webhooks.py → services/
  event_dispatch.py) so subscription and webhook ingest share the same
  event_log + deferred-dispatch + quiet-hours code path

Bot commands:
- /status, /entities [glob], /state <entity_id>, /areas
- Multi-command WS session so /status and /areas cost one handshake
- Sensitive-attribute blocklist (camera access_token, entity_picture, etc.)
  and 30-attribute cap to keep /state output safe and within Telegram's
  message size
- Error-message redaction strips URL userinfo before surfacing to chat

Frontend:
- HA descriptor with toggle ConfigField type (new) and tag-input filter
  mode for free-text glob/domain lists (new TagInput component)
- 15 command slots + 4 notification slots wired into the existing
  template-config UI
2026-05-13 14:31:56 +03:00
alexei.dolgolyov ba199f24bd feat: deferred dispatch, release-check provider, settings polish
- Defer quiet-hours dispatches into new deferred_dispatch table; drain
  job + periodic catch-up scan re-fire at window end with coalescing on
  (link, event_type, collection_id).
- Add ON DELETE SET NULL migration on event_log_id and partial unique
  index on (link_id, collection_id, event_type) WHERE status='pending'.
- Add release-check provider abstraction (Gitea/GitHub) with SSRF-safe
  URL validation, settings UI cassette, and scheduled polling.
- Replace importlib-only version lookup with version.py helper that
  prefers the higher of installed metadata vs source pyproject so stale
  editable dev installs stop misreporting.
- Aurora frontend polish: MetaStrip component, ReleaseCassette,
  EventDetailModal expansion, and i18n additions.
2026-05-12 02:58:07 +03:00
alexei.dolgolyov 72dd611f8c fix(telegram): respect chat_action UI choice, drop phantom indicator
chat_action was stored in two places — the model column and config JSON —
and dispatch_helpers unconditionally overrode the config value with the
column. The frontend only ever wrote the JSON path, so the UI choice
silently had no effect on outgoing chat actions.

Make the column the single source of truth: frontend sends chat_action
top-level, dispatch_helpers reads from the column, and a one-time
backfill migrates existing config values to the column and strips the
legacy key.

Also fix a long-standing race where the keepalive's bare sleep(4) +
finally cancel could fire one last sendChatAction after the response
already arrived, leaving a phantom indicator for ~5s. Replace with a
stop event + wait_for so callers can signal stop cleanly via the new
stop_keepalive helper.
2026-04-27 18:20:50 +03:00
alexei.dolgolyov 8f0346ea03 fix(csp): allow unsafe-inline scripts for SvelteKit hydration bootstrap
The static-adapter build emits an inline <script> with the hydration
payload; ``script-src 'self'`` alone blocks the SPA from starting
(browser error: "Executing inline script violates the following Content
Security Policy directive").

Mirrors the 'unsafe-inline' already present for style-src. Primary XSS
protection still comes from Svelte's auto-escaping and
frontend/src/lib/sanitize.ts for the {@html} paths that render user
content. CSP still blocks remote scripts (no https: in script-src),
framing (frame-ancestors 'none'), base-uri hijacking, and form
exfiltration.
2026-04-23 21:08:17 +03:00
alexei.dolgolyov 7cbb02b1ef feat(db): pre-migration SQLite snapshots via VACUUM INTO
Build and Test / test-backend (push) Successful in 2m38s
Build and Test / test-frontend (push) Successful in 9m44s
Build and Test / build-image (push) Failing after 17m9s
Take a consistent, atomic copy of the DB at lifespan startup BEFORE
migrations run, so a botched future upgrade is recoverable by restoring
a single file instead of a data-loss incident.

Uses SQLite's VACUUM INTO — safe under WAL, cannot tear against
concurrent writes. Best-effort: failures are logged, never raised —
the main DB remains the source of truth.

Configurable via NOTIFY_BRIDGE_PRE_MIGRATE_SNAPSHOT_KEEP (default 5;
0 disables). Snapshots land in ``data_dir/backups/pre-migrate-<ts>.db``
and the N oldest are pruned each boot.
2026-04-23 19:53:15 +03:00
alexei.dolgolyov 920920bc67 feat: production-readiness hardening across security, async, DB, ops
Build and Test / test-frontend (push) Successful in 9m37s
Build and Test / test-backend (push) Successful in 10m53s
Build and Test / build-image (push) Failing after 14m52s
Security
- SSRF: async DNS resolver; allow_redirects=False on all outbound clients;
  matrix homeserver_url validated on create/update/test; update_provider
  and email_bot merge incoming config and reject ***-masked secrets.
- Auth: bcrypt offloaded to asyncio.to_thread; JWT now carries iss/aud +
  leeway and rejects missing claims; setup TOCTOU closed inside a
  transaction; rate limits extended (default 600/min, 10/min on password
  change, 30/min on needs-setup); constant-time login to prevent username
  enumeration.
- Config: rejects known dev secret keys; validates CORS origin schemes,
  port range, token lifetimes.
- Webhook handlers stream-read body with a 1 MiB cap; Discord 429 retries
  bounded (3 attempts, Retry-After capped at 60 s).
- CSP + HSTS added to SecurityHeadersMiddleware.

Async / runtime
- SQLite engine: WAL, synchronous=NORMAL, foreign_keys=ON, busy_timeout,
  pool_pre_ping, dispose on shutdown.
- Lifespan shutdown now stops scheduler before closing HTTP session and
  disposing the engine.
- Shared aiohttp session locked against concurrent first-caller races;
  core NotificationDispatcher accepts and reuses it.
- Storage and scheduled backup writes wrapped in asyncio.to_thread.
- NUT client writes bounded by asyncio.wait_for.
- Telegram poller switched from 3 s short-poll to 30 s interval + 25 s
  long-poll (~10x fewer API calls).

Database
- New performance-indexes migration covers every FK/owner column and
  hot-path composite (notification_tracker(provider_id, enabled);
  event_log(user_id, created_at DESC); webhook_payload_log(provider_id,
  created_at DESC); action_execution(action_id, started_at DESC)).
- New schema_version table for future upgrade gating.
- __system__ placeholder user (id=0) seeded so user_id=0 system defaults
  satisfy the newly enforced FK; filtered out of /auth/needs-setup,
  /api/users, and setup.
- list_notification_trackers rewritten to batched loads (was 1+N+N*M).
- Retention job extended to event_log, webhook_payload_log, and
  action_execution; retention days exposed as a setting.

Scheduler
- AsyncIOScheduler job_defaults: coalesce, misfire_grace_time=300,
  max_instances=1.

Ops
- uvicorn runs with proxy_headers, forwarded_allow_ips,
  timeout_graceful_shutdown; access log suppressed in non-debug.
- FastAPI version string now reads from importlib.metadata.
- New /api/ready endpoint separate from /api/health.
- docker-compose drops the ALLOW_PRIVATE_URLS=1 default, adds mem/cpu/pid
  limits, read_only + tmpfs, cap_drop:ALL, no-new-privileges; healthcheck
  targets /api/ready.
- CI now runs on push/PR with backend pytest, frontend svelte-check +
  build, and a non-push image build; release workflow gated on tests,
  publishes immutable sha-<commit> image tag, adds Trivy scan.

Tests
- New packages/server/tests/ with 29 passing tests: config validation,
  JWT round-trip + aud/alg=none rejection, SSRF scheme and private-range
  enforcement (sync + async), Discord bounded retry, and a lifespan-level
  /api/health + /api/ready smoke check.
- Renamed the misnamed services/test_dispatch.py to manual_dispatch.py so
  pytest never auto-collects production code.

Frontend
- /login now redirects already-authenticated users to /, shows a distinct
  'backend unreachable' banner (en/ru) when /auth/needs-setup fails.
2026-04-23 19:44:56 +03:00
alexei.dolgolyov f50d465c0e feat(logging): production-grade logging with context vars, secret masking, and runtime level control
Boot-time logging was a three-line basicConfig stub with no timestamps, no
correlation, and silent drops at every layer of the Telegram send path — a
/random command that delivered text but no media left zero evidence in the
log. This replaces the setup and closes every silent drop encountered end-to-end.

New infrastructure:
- notify_bridge_core.log_context: request_id/command/chat_id/bot_id/dispatch_id
  ContextVars with a bind_log_context() context manager so deep call sites
  (TelegramClient, NotificationDispatcher) inherit the correlation tag without
  threading args through.
- notify_bridge_server.logging_setup: dictConfig-based setup with a
  LogRecordFactory that tags every record, a SecretMaskingFilter that redacts
  /botN:TOKEN plus Authorization/x-api-key/password/secret in messages AND
  tracebacks, a JSON formatter for aggregators, text formatter with grep-friendly
  [req=... cmd=... bot=... chat=... disp=...] prefix, and default dampening
  for sqlalchemy/aiohttp/apscheduler/urllib3/PIL.

Runtime control:
- NOTIFY_BRIDGE_LOG_LEVEL / _FORMAT / _LEVELS env vars (boot).
- DB-backed log_level / log_format / log_levels AppSettings, applied on
  boot after migrations and live via apply_log_levels() when edited in
  the settings UI (format still requires restart, logs a WARN).
- Frontend settings page gains a Logging card (level dropdown, format
  dropdown, per-module overrides); en/ru i18n keys added.

Call-site fixes (/random media-group blind spot and adjacent):
- TelegramClient._fetch_asset: every silent drop now WARN-logs with reason
  (missing url, HTTP non-200, size/dimension limits, ClientError).
- TelegramClient._send_media_group: WARN on "chunk had N items but 0 usable",
  ERROR on sendMediaGroup non-ok/transport with full context; returns
  success=False + "no_items_delivered" instead of success=True with an empty
  message_ids list so callers can distinguish.
- TelegramClient.send_message / _upload_media / _send_from_cache: ERROR on
  non-ok + transport failures with status/code/desc; DEBUG for cache-hit
  fallbacks.
- NotificationDispatcher.dispatch: generates a dispatch_id, binds it, logs
  start/finish with failure count, uses exc_info for target failures.
- commands/handler: missing/failed templates -> ERROR + exc_info; send_reply
  and send_media_group errors upgraded WARNING -> ERROR with chat/error_code
  context; rate-limit and truncation cases logged with full context.
- commands/webhook and services/telegram_poller: bind_log_context(request_id
  =tg:<update_id>, command, chat_id, bot_id), INFO on receive/dispatch/
  completion with duration, exc_info on raise, INFO when commands disabled.
- commands/immich: INFO when album scope is empty; WARN per asset dropped
  from media payload and a summary WARN when "N assets in, 0 out".
2026-04-23 14:41:26 +03:00
alexei.dolgolyov a7a2b4efa4 feat: large polish pass — UX fixes, per-chat scope, restore/backup, action events
Backend
- Per-chat album scope for Immich commands (search/latest/memory/...): new
  allowed_album_ids on CommandTrackerListener, threaded listener/page kwargs
  through ProviderCommandHandler.handle; PATCH listener-scope endpoint.
- /search and /find accept a trailing page number; Immich client search_smart
  / search_metadata take a page param.
- Immich person-asset lookup switched from removed GET /api/people/{id}/assets
  to POST /api/search/metadata with personIds (fixes /person command and
  auto_organize rules silently returning zero candidates on Immich 1.106+).
- Auto_organize rule now sets the target album's thumbnail to the first added
  image when missing (falls back to any asset type); failures do not fail the
  rule. add_assets_to_album surfaces the Immich error body on non-2xx.
- EventLog.user_id / action_id / action_name columns with defensive migration
  + backfill. Status query filters by user_id directly; Immich/webhook paths
  emit user_id explicitly. action_runner writes an action_success/partial/
  failed event on each non-dry-run.
- Dashboard DELETE /api/status/events (scoped to user_id) + rendering live
  tracker/provider/action names via FK join with snapshot fallback.
- PATCH /api/users/{id} for username/role change with last-admin guard.
- Deletion protection returns structured {message, entity, blocked_by}
  (ApiError carries .blockedBy; frontend opens BlockedByModal).
- Backup prepare-restore → AppSetting markers + atomic write of
  pending_restore.json; lifespan hook applies on next startup and archives
  under data/applied_restores/. apply-restart sends SIGTERM so the lifespan
  shutdown runs; NOTIFY_BRIDGE_SUPERVISED env override gates the button.
  Manual POST /api/backup/files (same format as scheduled).
- New periodic-summary test path reuses shared collect_scheduled_assets
  (limit=0) so test and future production code go through one primitive.
- Per-receiver locale for Telegram test messages (resolves
  TelegramChat.language_override per chat instead of applying the first
  receiver's locale to everyone).
- Bounded concurrency (semaphores) in NotificationDispatcher._preload_asset_data
  and _refresh_telegram_chat_titles; chat title sweep extended to 24h since
  save_chat_from_webhook covers active chats opportunistically.
- Telegram poller detects the \"webhook is active\" 409 and auto-calls
  deleteWebhook for bots whose DB update_mode is polling (throttled per bot).
- TelegramClient.get_chat added (CLAUDE.md rule 6); set_album_thumbnail added.
- Seeds: rename \"Default Commands\" → \"Default Immich Commands\";
  track_assets_removed default False.

Frontend
- Global provider selector visible when there is only one provider.
- Clear-events button + i18n + ConfirmModal on the dashboard; new icons/
  labels/filters/colors for action_success / action_partial / action_failed.
- Auto-select first available tracking/template/command/config + bot on
  create forms (trackers, command-trackers, targets, template/command
  configs).
- Telegram target disable_url_preview defaults to true.
- BlockedByModal wired into 8 deletion flows; fetchAuth helper for
  multipart/binary calls (reuses api()'s refresh + ApiError mapping).
- Immich tracker 'Checking links' parallelised (concurrency cap 6).
- Backup page: pending-restore banner + Apply-now / Apply-later modal,
  restarting overlay polling /api/health, manual 'Create backup' button.
- Command-trackers listener row gets an 'Edit album scope' modal with
  inherit/explicit multiselect.
- Users page: Edit user modal (username + role).
- parseDate helper for consistent UTC date rendering.

Migrations / schema
- event_log: + user_id, action_id, action_name (+ backfill user_id from
  notification_tracker).
- command_tracker_listener: + allowed_album_ids.
2026-04-22 01:13:11 +03:00
alexei.dolgolyov 28465f56f9 fix(spa): serve index.html for SvelteKit client-side routes
Release / release (push) Successful in 1m7s
Deep-linking to non-root URLs like /settings or /notification-trackers
returned {"detail":"Not Found"} because StaticFiles(html=True) only
serves index.html for directory roots, not for arbitrary SPA routes.

Subclass StaticFiles with an SPA fallback: any 404 on a non-/api path
serves the root index.html, letting the SvelteKit router hydrate the
correct view on the client. Real /api/* 404s still bubble up as JSON
from FastAPI.
2026-04-21 20:57:39 +03:00
alexei.dolgolyov f0739ca949 feat: security hardening — SSRF guard, template sandbox timeout, webhook log prune, auth & backup polish
- Add outbound URL validation (SSRF) for webhook/Discord/Slack/ntfy/Matrix dispatch
- Template renderer: input/output caps and thread-based render timeout
- Webhook log filter: strip Authorization/signature/token-like headers; atomic prune
- Auth/JWT/backup/config tightening; misc frontend UX fixes
2026-04-16 03:21:45 +03:00
alexei.dolgolyov 6b2211353d feat: person excludes for auto-organize rules, backup & restore system
Add person exclude criteria to Immich auto-organize — assets containing
excluded persons are filtered out after candidate gathering. Also adds
full backup/restore system with export, import, scheduled backups, and
retention management.
2026-04-02 14:13:42 +03:00
alexei.dolgolyov 6113a0039c feat: webhook payload history — store and display recent incoming payloads
Backend:
- WebhookPayloadLog model (provider_id, method, headers, body, status, extracted_fields, error_message)
- Auto-log payloads in generic_webhook() with matched/unmatched/error status
- Auto-prune beyond max_stored_payloads per provider
- Header filtering (only Content-Type, User-Agent, X-* stored; no Authorization)
- GET/DELETE /api/providers/{id}/webhook-logs endpoints
- store_payloads + max_stored_payloads in WebhookProviderConfig

Frontend:
- WebhookPayloadHistory.svelte — expandable log viewer with status badges, JSON body, headers, extracted fields
- payloadHistory flag on webhook provider descriptor
- max_stored_payloads config field (0 = disabled)
- Password confirmation field on change password modal
- i18n keys for webhook logs (en + ru)
2026-03-28 13:54:54 +03:00
alexei.dolgolyov b803d004e1 refactor: comprehensive codebase review — security, performance, quality, UX
Security:
- Fix NUT protocol command injection (validate names against safe regex)
- Enable Jinja2 autoescape=True to prevent HTML injection via external data
- Add WebhookProviderConfig validation model

Performance:
- Shared aiohttp.ClientSession singleton (replaces 40+ per-request sessions)
- Fix 4 N+1 queries with batch IN loads (poller, scheduler, memory, broadcast)
- asyncio.gather for Gitea commands and notification dispatcher
- Add DB indexes on NotificationTrackerState.tracker_id, CommandTrackerListener
- LRU cache for compiled Jinja2 templates
- Daily EventLog cleanup job (90-day retention)
- 30s HTTP timeout on all external calls
- GROUP BY for target type counts (replaces 7 sequential queries)

Code quality:
- Extract get_owned_entity() helper (replaces 11 duplicate functions)
- Extract slot_helpers.py (load_slots, save_slots, render_template_preview)
- Extract command_utils.py (tracker lookup, last event, collection IDs)
- Extract http_session.py (shared session lifecycle)
- Provider connection validation dedup (3x → 1 helper)
- Command dispatch tables replacing if/elif chains
- Album+links fetch helper (fetch_albums_with_links)
- Provider dispatch polymorphism (list_provider_collections)
- Immutable _enrich_assets (no longer mutates in-place)
- Fix _format_assets return type + handler unpacking

Frontend:
- Fix 18+ hardcoded English strings → t() with new i18n keys (en + ru)
- Mobile "More" nav panel with provider filter and search
- Shared Button.svelte component (4 variants, 2 sizes)
- Shared ErrorBanner.svelte component (8 pages updated)
- SvelteKit goto() replacing window.location.href
- Dashboard grid fixed for 4 cards, paginator opacity consistency

Functionality:
- max_instances=1 on scheduler jobs (prevents duplicate events)
- Webhook provider in watcher (prevents error spam)
- Fix stale SQLModel reference in poller
- Gitea get_repo() direct API call
2026-03-28 13:22:26 +03:00
alexei.dolgolyov 307871cae5 feat: Google Photos provider backend + API hardening
- Add Google Photos provider: client, models, change detector, capabilities
- Add notification templates (en/ru) for all GP event slots
- Add command templates (en/ru) for GP bot commands
- Register GP in slot/command loaders, capabilities, and seeds
- Harden provider API: validate OAuth credentials on create/update
- Add internal URL rewriting for asset fetches (LAN optimization)
- Fix template renderer to handle missing variables gracefully
- Improve webhook command routing for multi-provider support
- Add provider health check endpoint and watcher improvements
2026-03-25 22:07:03 +03:00
alexei.dolgolyov 37388c430c feat: locale-aware notification templates + UX improvements
- Add locale support to notification templates (matching command template
  pattern): TemplateSlot now has locale field with (config_id, slot_name,
  locale) uniqueness, nested API format {slot: {locale: template}}
- Migration merges separate EN/RU system configs into unified per-provider
  configs; seeds create one config per provider with multi-locale slots
- Locale-aware dispatch with EN fallback in NotificationDispatcher
- Frontend locale tabs (EN/RU) on template config editor
- Fix tracking config cards not showing default provider icons
- Global provider filter, search palette, and various UX polish
2026-03-23 19:08:48 +03:00
alexei.dolgolyov 6a559bfcd2 feat: Actions system — scheduled mutations on external services
Full-stack implementation of provider-scoped Actions with extensible
executor architecture. First action type: Immich auto_organize (sort
assets into albums by person, CLIP search, date range, favorites).

Core:
- ActionTypeDefinition registry + ActionExecutor ABC with execute/validate/dry-run
- ImmichActionExecutor with multi-album support and client-side filtering
- ImmichClient write methods: add/remove assets, create album, paginated search

Server:
- Action, ActionRule, ActionExecution DB models
- Full CRUD API + manual execute + dry-run + execution history endpoints
- APScheduler integration (interval + cron) for automated execution
- Action type discovery API + provider people endpoint

Frontend:
- Actions page with CRUD, execute/dry-run buttons, inline rule editor
- RuleEditor: person/album MultiEntitySelect pickers, criteria config
- ExecutionHistory: expandable per-rule result details
- MultiEntitySelect reusable component (searchable multi-pick palette)
- Notification tracker album picker migrated to MultiEntitySelect
- Fixed MdiIcon race condition (icons missing after cache-clearing reload)
2026-03-23 16:59:20 +03:00
alexei.dolgolyov 1ac6a17f6f feat: Docker deployment + Gitea CI/CD workflow
- Multi-stage Dockerfile: Node frontend build → Python wheel build → slim runtime
- Backend serves SvelteKit static output via FastAPI StaticFiles mount
- docker-compose.yml with named volume for /data persistence
- Gitea Actions workflow: build/push Docker image + create release on v* tags
- Add NOTIFY_BRIDGE_STATIC_DIR config for frontend path
- Fix run() to use configurable host/port
2026-03-23 02:14:14 +03:00
alexei.dolgolyov e0bae394ee feat: comprehensive code review fixes — security, performance, quality
Backend security:
- Reject Gitea webhooks when webhook_secret is empty (was silently skipping)
- Add slowapi rate limiting on login (5/min) and setup (3/min) endpoints
- Add CORS middleware with configurable origins
- Mask telegram_webhook_secret in settings API response
- Protect system-owned command template configs from regular user modification
- Increase minimum password length to 8 characters

Backend performance:
- Batch queries in _resolve_command_context (3 queries instead of 3N)
- Concurrent album fetching with asyncio.gather in immich commands
- Singleton Jinja2 SandboxedEnvironment (reuse instead of per-render creation)
- TTLCache for rate limits (bounded memory, auto-eviction)
- Optional aiohttp session reuse in send_reply/send_media_group

Backend code quality:
- Extract dispatch_helpers.py (shared link_data loading + event filtering)
- Extract database/seeds.py from main.py (490 lines → dedicated module)
- Split immich_handler.py (415 lines) into commands/immich/ subpackage
- Replace bare except blocks with logged warnings
- Add per-provider config validation (Pydantic models)
- Truncate command input to 512 chars
- Expose usage_* and desc_* slots in capabilities and variables API

Frontend security:
- CSS.escape() for user-controlled querySelector in highlight.ts
- Client-side password min 8 chars validation on setup and password change

Frontend code quality:
- Replace any types with proper interfaces across top files
- Decompose targets/+page.svelte into TargetForm + ReceiverSection
- Fix $derived.by usage, $state mutation patterns
- Add console.warn to empty catch blocks

Frontend UX:
- Auth redirect via goto() with "Redirecting..." state
- Platform-aware Ctrl/Cmd K keyboard hint
- Remove stat-card hover transform

Frontend accessibility:
- Modal: role=dialog, aria-modal, focus trap, restore focus
- EntitySelect/IconGridSelect: listbox/option roles, aria-selected/disabled
2026-03-23 01:59:51 +03:00
alexei.dolgolyov 63437c1841 refactor: provider-agnostic bot command system + Gitea commands
Refactored the monolithic command handler (707 lines) into a pluggable
provider-handler architecture:

- Abstract ProviderCommandHandler interface (base.py)
- Handler dispatch registry routes commands by provider type
- Extracted all Immich logic into ImmichCommandHandler
- New GiteaCommandHandler with /status, /repos, /issues, /prs, /commits
- Multi-provider routing: groups context by provider type, finds handler
- handler.py reduced to ~280 line thin orchestrator

Gitea commands:
- Extended GiteaClient with get_repo_issues, get_repo_pulls, get_repo_commits
- 30 Jinja2 command templates (15 EN + 15 RU)
- Gitea capabilities updated with 6 commands + 15 command_slots
- Default command config + command template config seeded on startup
- Rate limiting: Gitea API commands share "api" category (15s cooldown)

Also:
- Command configs API accepts "gitea" provider type
- System command configs (user_id=0) visible to all users
- Webhook URL shown on Gitea provider card and edit form
- Scan interval hidden for webhook-based providers
2026-03-22 17:44:47 +03:00
alexei.dolgolyov 0562f78b35 feat: add Scheduler provider + multi-provider UX fixes
Scheduler provider:
- Virtual provider (no external service) that emits SCHEDULED_MESSAGE
  events on user-defined intervals or cron expressions
- Custom variables stored in tracker filters, flattened into template context
- fire_count persists across triggers via tracker state
- APScheduler CronTrigger support for cron-mode schedules
- Default templates (EN+RU), seeded on startup

Multi-provider UX fixes:
- Tracking config hides Immich-specific sections (periodic, scheduled,
  memory, asset display) for non-Immich providers
- Command config driven by provider capabilities — hides commands/settings
  for providers without bot commands
- Template config hides empty "Scheduled Messages" group
- Test menu on tracker targets is provider-aware (Immich shows all 4 test
  types, others show only basic)
- Removed redundant Test button from tracker card
- System-owned tracking configs (user_id=0) seeded for Gitea + Scheduler
- Fixed ownership checks to allow system configs in tracker-target links
- Capabilities cache shared across template-configs and command-configs
- Command tracker bot selector uses EntitySelect instead of raw select
- Sample context includes Gitea + Scheduler variables for template preview
2026-03-22 15:50:51 +03:00
alexei.dolgolyov 6d28cfb8d8 feat: add Gitea as webhook-based service provider
First webhook-based provider integration (Immich uses polling).
Gitea pushes events via POST /api/webhooks/gitea/{provider_id} with
HMAC-SHA256 signature validation.

- 9 event types: push, issue opened/closed/commented, PR opened/closed/merged/commented, release published
- Generic filters system on NotificationTracker (collections, senders, exclude_senders)
- Provider capabilities include supported_filters and webhook_based flag
- Gitea API client for connection testing and repository listing
- 18 default Jinja2 notification templates (EN + RU)
- Frontend: conditional provider forms, Gitea event toggles in tracking config
- Auto-migration for filters column and Gitea tracking flags
2026-03-22 12:58:35 +03:00
alexei.dolgolyov 1167d138a3 feat: locale-aware command templates, debounced auto-sync, entity pickers
- Locale-aware templates: CommandTemplateSlot now has a locale column,
  allowing each slot to have per-language variants (EN/RU). Templates
  are resolved at runtime from the Telegram user's language_code.

- Merged system configs: "Default Commands (EN)" and "(RU)" merged
  into a single "Default Commands" config with locale-aware slots.
  Migration handles existing data automatically.

- Configurable command descriptions: hardcoded COMMAND_DESCRIPTIONS
  replaced with desc_* template slots (desc_status, desc_help, etc.)
  that users can customize per locale. setMyCommands registers all
  locales explicitly.

- Removed locale from CommandConfig: no longer needed since locale
  is derived from the Telegram user's language at runtime.

- Debounced command auto-sync: after command config/tracker changes,
  affected bots are marked dirty and synced after a 30s debounce
  window. Manual "Sync with Telegram" button still works.

- Entity pickers in LinkedTargetsSection: replaced 6 plain <select>
  elements with EntitySelect components (search, icons, keyboard nav).
  Added onselect callback and size="sm" props to EntitySelect.
2026-03-22 03:14:51 +03:00
alexei.dolgolyov 751097b347 feat: comprehensive code review fixes + receivers-only architecture
Security:
- Refuse startup with default secret_key in production (was just logging)
- Settings endpoint now requires admin role
- Password validation on initial setup
- DOM-based HTML sanitizer replaces regex in template previews
- Add *.log to .gitignore

Performance & reliability:
- Token refresh deduplication prevents race condition on concurrent 401s
- Theme media query listener registered once (no leak)
- IconPicker uses $derived instead of function call per render
- Snackbar uses single-batch state update instead of while loop
- Replace 11 inline hover handlers with CSS :hover in layout

Architecture - receivers-only:
- Delivery endpoints (chat_id, email, url, room_id, topic) now stored
  exclusively in TargetReceiver rows, never in target.config
- Migration extracts existing delivery fields to receiver rows
- Notifier and dispatcher remove all config fallbacks
- Frontend targets page shows receivers list per target with
  add/remove/toggle/test per receiver
- Single-receiver test endpoint: POST /targets/{id}/receivers/{id}/test

Code quality:
- Extract AuthLayout.svelte from login/setup (150 lines CSS dedup)
- Split telegram-bots page (754→51 lines + 3 tab components)
- Split notification-trackers page (547→432 lines + 4 components)
- Deduplicate _send_reply into shared handler.send_reply()
- Add locale column to template models, replace name-based detection
- Fix delete_notification_tracker dead protection check
- Fix check_telegram_bot query (filter by type, remove bogus OR)
- Add graceful scheduler shutdown in lifespan
- Consistent /bots?tab=telegram URLs across all nav links

i18n:
- Error page, chat actions, target types, provider types internationalized
- All new receiver UI strings in EN + RU
2026-03-22 02:19:31 +03:00
alexei.dolgolyov db7aac5fe8 fix: re-create missing EN default template, provider type as IconGridSelect
- Template seed now re-creates missing system templates on every startup
  (not just first boot), using raw SQL to handle legacy NOT NULL columns
- Tracking configs: add provider type selector (was missing)
- All config forms: provider type uses IconGridSelect during creation,
  read-only text during editing (immutable after creation)
- Pages: tracking-configs, command-configs, command-template-configs,
  template-configs
2026-03-22 00:36:15 +03:00
alexei.dolgolyov 3e3a6f0777 feat: Discord/Slack/ntfy/Matrix targets, command templates, delete protection, email/matrix bots
- Discord, Slack, ntfy, Matrix notification target types with clients and dispatch
- MatrixBot model + API + frontend in Bots tab
- Command template system fully wired into all handler commands
- Default command templates seeded (EN/RU, 14 slots each)
- Command template editor with variables reference including child fields
- Delete protection on all 10 entity types (409 with consumer details)
- Provider type selector on template config forms
- Target type selector as dropdown with all 7 types
- Response template selector on command config form
- CLAUDE.md: mandatory server restart rule, child properties rule
2026-03-21 20:36:12 +03:00
alexei.dolgolyov 846d480d38 feat: provider-strict configs, slot-based templates, broadcast targets, email bots, command templates
Major architectural improvements:
- Provider-type enforcement: configs validated against provider type at assignment
- TemplateConfig migrated to slot-based pattern (TemplateSlot child table)
- Broadcast targets: TargetReceiver child table for multi-receiver dispatch
- EmailBot: first-class email sender entity with SMTP config, test connection
- CommandTemplateConfig: generic slot-based command response templates
- Provider capability registry: dynamic slot/event/command definitions per provider
- CommandTracker play/pause button matches NotificationTracker style
2026-03-21 16:33:24 +03:00
alexei.dolgolyov 371ea70756 feat: fix template preview links, default chat action, update default templates
- Fix sanitizePreview regex to match literal quotes instead of &quot; entities
- Default telegram chat_action to "typing" in model and frontend
- Change "photo(s)" to "file(s)" in default templates (EN/RU)
- Remove redundant album URL line from assets_added templates
- Auto-refresh system-owned templates from files on server startup

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 01:37:58 +03:00
alexei.dolgolyov 1d445f3980 feat: entity relationship refactor — notification trackers, command system, chat actions
Rework entity schema: rename Tracker→NotificationTracker, add CommandConfig/
CommandTracker/CommandTrackerListener entities for decoupled command handling.
Commands now resolve through CommandTracker→CommandConfig instead of
TelegramBot.commands_config. Smart ref-counted bot polling based on active
listeners. Add chat_action to telegram targets. Full frontend CRUD pages
for command configs and command trackers. Idempotent SQLite migrations.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 01:27:20 +03:00
alexei.dolgolyov 03ec9b3c86 feat: telegram commands, app settings, bot polling, webhook handling, UI improvements
Adds telegram bot command system with 13 commands (search, latest, random, etc.),
webhook/polling handlers, rate limiting, app settings page, and various UI/UX
improvements across all entity pages.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 23:11:42 +03:00
alexei.dolgolyov 03c5c66eed feat: UX & notification improvements — icons, events, chat names, link validation, templates
- Show entity icons on all cards with fallback defaults (providers, trackers, targets, bots)
- Enrich EventLog with provider_name, tracker_name, assets_count; add DB migration
- Dashboard events: filtering (type, provider, search), sorting, pagination, dynamic page size
- Friendly chat names on telegram target cards (resolve from TelegramChat table)
- Test message button on bot chat items with locale-aware messages
- Album public link validation on tracker save with auto-create dialog
- Support albums without public links: conditional <a href> in templates
- Fetch shared links during poll, enrich events with public_url/protected_url
- Per-asset public_url in template context ({share_url}/photos/{asset_id})
- Common date/location detection: common_date + common_location context vars
- Dual date formats: date_format (datetime) + date_only_format (date only)
- Template clone button, HTML link rendering in template preview
- Fix Telegram asset download 401: pass x-api-key headers through client
- Fix provider external_url matching for API key scoping
- Fix event timestamp timezone (append Z suffix for UTC)
- Localize event filter controls, test messages (EN/RU)
- Template variable UI helpers updated with all new fields
- CLAUDE.md: template system sync rules documentation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 16:18:03 +03:00
alexei.dolgolyov 9eec21a5b2 feat: port full CRUD API routes and frontend pages from Immich Watcher
Backend API (38 routes):
- providers: full CRUD + test connection + list collections + API key masking
- trackers: full CRUD + trigger + history + test-periodic/memory
- tracking-configs: full CRUD with Pydantic models, provider_type filter
- template-configs: full CRUD + preview + preview-raw with two-pass validation
- targets: full CRUD + test notification + config masking
- telegram-bots: full CRUD + chat discovery + token endpoint
- users: full admin CRUD + password reset + self-delete protection
- status: dashboard endpoint with providers/trackers/targets/events counts

Frontend pages updated:
- Dashboard with animated stat cards and event timeline
- Providers with proper components, delete confirm, snackbar
- Trackers/targets/tracking-configs/template-configs/telegram-bots/users
  all use PageHeader, Card, Loading, MdiIcon with correct i18n keys

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 00:49:40 +03:00
alexei.dolgolyov 08814e9ae2 feat(notify-bridge): phase 8 - integration and wiring
Wire all components into a working application:
- Scheduler service: APScheduler loads enabled trackers, polls at intervals
- Watcher service: orchestrates poll -> detect -> notify flow
  - Eagerly loads DB data, then creates aiohttp session for provider
  - Saves tracker state after each poll
  - Logs events to EventLog table
  - Dispatches notifications to targets with template rendering
- Manual trigger endpoint: POST /api/trackers/{id}/trigger
- Scheduler starts on app lifespan startup
- Full end-to-end flow verified: server starts cleanly

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 23:55:15 +03:00
alexei.dolgolyov 7f99c895a4 feat(notify-bridge): phase 6 - database models and server API
New database schema with ServiceProvider abstraction:
- ServiceProvider (replaces ImmichServer): type + JSON config
- Tracker (replaces AlbumTracker): owns tracking_config_id
- TrackingConfig: provider_type scoped, owned by Tracker
- TemplateConfig: provider_type scoped, owned by Target
- NotificationTarget: owns template_config_id (not tracking_config_id)
- TrackerState, EventLog, User, TelegramBot, TelegramChat

Full FastAPI server:
- /api/providers: CRUD + test connection + list collections
- /api/trackers: CRUD
- /api/tracking-configs: CRUD with provider_type filter
- /api/template-configs: CRUD with provider_type filter, system defaults
- /api/targets: CRUD
- /api/template-vars: variable docs filtered by provider type
- /api/auth: setup, login, refresh, me, password change
- /api/health: health check
- Default template seeding on first startup (EN/RU for Immich)
- pydantic-settings with NOTIFY_BRIDGE_ env prefix

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 23:39:23 +03:00
alexei.dolgolyov b724447f4d feat(notify-bridge): phase 1 - project scaffolding
Set up the Notify Bridge project structure:
- packages/core (notify_bridge_core) with provider, model, notification, template packages
- packages/server (notify_bridge_server) with FastAPI skeleton and health endpoint
- frontend with SvelteKit 2, Svelte 5, Tailwind CSS v4, static adapter
- Root configs: .gitignore, README.md, CLAUDE.md

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 22:30:06 +03:00