Commit Graph

18 Commits

Author SHA1 Message Date
alexei.dolgolyov 920920bc67 feat: production-readiness hardening across security, async, DB, ops
Build and Test / test-frontend (push) Successful in 9m37s
Build and Test / test-backend (push) Successful in 10m53s
Build and Test / build-image (push) Failing after 14m52s
Security
- SSRF: async DNS resolver; allow_redirects=False on all outbound clients;
  matrix homeserver_url validated on create/update/test; update_provider
  and email_bot merge incoming config and reject ***-masked secrets.
- Auth: bcrypt offloaded to asyncio.to_thread; JWT now carries iss/aud +
  leeway and rejects missing claims; setup TOCTOU closed inside a
  transaction; rate limits extended (default 600/min, 10/min on password
  change, 30/min on needs-setup); constant-time login to prevent username
  enumeration.
- Config: rejects known dev secret keys; validates CORS origin schemes,
  port range, token lifetimes.
- Webhook handlers stream-read body with a 1 MiB cap; Discord 429 retries
  bounded (3 attempts, Retry-After capped at 60 s).
- CSP + HSTS added to SecurityHeadersMiddleware.

Async / runtime
- SQLite engine: WAL, synchronous=NORMAL, foreign_keys=ON, busy_timeout,
  pool_pre_ping, dispose on shutdown.
- Lifespan shutdown now stops scheduler before closing HTTP session and
  disposing the engine.
- Shared aiohttp session locked against concurrent first-caller races;
  core NotificationDispatcher accepts and reuses it.
- Storage and scheduled backup writes wrapped in asyncio.to_thread.
- NUT client writes bounded by asyncio.wait_for.
- Telegram poller switched from 3 s short-poll to 30 s interval + 25 s
  long-poll (~10x fewer API calls).

Database
- New performance-indexes migration covers every FK/owner column and
  hot-path composite (notification_tracker(provider_id, enabled);
  event_log(user_id, created_at DESC); webhook_payload_log(provider_id,
  created_at DESC); action_execution(action_id, started_at DESC)).
- New schema_version table for future upgrade gating.
- __system__ placeholder user (id=0) seeded so user_id=0 system defaults
  satisfy the newly enforced FK; filtered out of /auth/needs-setup,
  /api/users, and setup.
- list_notification_trackers rewritten to batched loads (was 1+N+N*M).
- Retention job extended to event_log, webhook_payload_log, and
  action_execution; retention days exposed as a setting.

Scheduler
- AsyncIOScheduler job_defaults: coalesce, misfire_grace_time=300,
  max_instances=1.

Ops
- uvicorn runs with proxy_headers, forwarded_allow_ips,
  timeout_graceful_shutdown; access log suppressed in non-debug.
- FastAPI version string now reads from importlib.metadata.
- New /api/ready endpoint separate from /api/health.
- docker-compose drops the ALLOW_PRIVATE_URLS=1 default, adds mem/cpu/pid
  limits, read_only + tmpfs, cap_drop:ALL, no-new-privileges; healthcheck
  targets /api/ready.
- CI now runs on push/PR with backend pytest, frontend svelte-check +
  build, and a non-push image build; release workflow gated on tests,
  publishes immutable sha-<commit> image tag, adds Trivy scan.

Tests
- New packages/server/tests/ with 29 passing tests: config validation,
  JWT round-trip + aud/alg=none rejection, SSRF scheme and private-range
  enforcement (sync + async), Discord bounded retry, and a lifespan-level
  /api/health + /api/ready smoke check.
- Renamed the misnamed services/test_dispatch.py to manual_dispatch.py so
  pytest never auto-collects production code.

Frontend
- /login now redirects already-authenticated users to /, shows a distinct
  'backend unreachable' banner (en/ru) when /auth/needs-setup fails.
2026-04-23 19:44:56 +03:00
alexei.dolgolyov 734e5c9340 feat: UX improvements — secure webhooks, locale fixes, dynamic languages, UI polish
- Remove top paginator from dashboard events, keep only bottom
- Fix test message locale: pass UI locale to email/matrix bot tests
- Convert webhook auth mode from text input to icon grid selector
- Generate secure UUID tokens for webhook URLs instead of sequential IDs
- Move Recent Payloads into per-provider expandable container (lazy-loaded)
- Make template config languages dynamic via app settings instead of hardcoded
- Change default dev port to 5175
2026-04-11 02:14:15 +03:00
alexei.dolgolyov 6113a0039c feat: webhook payload history — store and display recent incoming payloads
Backend:
- WebhookPayloadLog model (provider_id, method, headers, body, status, extracted_fields, error_message)
- Auto-log payloads in generic_webhook() with matched/unmatched/error status
- Auto-prune beyond max_stored_payloads per provider
- Header filtering (only Content-Type, User-Agent, X-* stored; no Authorization)
- GET/DELETE /api/providers/{id}/webhook-logs endpoints
- store_payloads + max_stored_payloads in WebhookProviderConfig

Frontend:
- WebhookPayloadHistory.svelte — expandable log viewer with status badges, JSON body, headers, extracted fields
- payloadHistory flag on webhook provider descriptor
- max_stored_payloads config field (0 = disabled)
- Password confirmation field on change password modal
- i18n keys for webhook logs (en + ru)
2026-03-28 13:54:54 +03:00
alexei.dolgolyov b803d004e1 refactor: comprehensive codebase review — security, performance, quality, UX
Security:
- Fix NUT protocol command injection (validate names against safe regex)
- Enable Jinja2 autoescape=True to prevent HTML injection via external data
- Add WebhookProviderConfig validation model

Performance:
- Shared aiohttp.ClientSession singleton (replaces 40+ per-request sessions)
- Fix 4 N+1 queries with batch IN loads (poller, scheduler, memory, broadcast)
- asyncio.gather for Gitea commands and notification dispatcher
- Add DB indexes on NotificationTrackerState.tracker_id, CommandTrackerListener
- LRU cache for compiled Jinja2 templates
- Daily EventLog cleanup job (90-day retention)
- 30s HTTP timeout on all external calls
- GROUP BY for target type counts (replaces 7 sequential queries)

Code quality:
- Extract get_owned_entity() helper (replaces 11 duplicate functions)
- Extract slot_helpers.py (load_slots, save_slots, render_template_preview)
- Extract command_utils.py (tracker lookup, last event, collection IDs)
- Extract http_session.py (shared session lifecycle)
- Provider connection validation dedup (3x → 1 helper)
- Command dispatch tables replacing if/elif chains
- Album+links fetch helper (fetch_albums_with_links)
- Provider dispatch polymorphism (list_provider_collections)
- Immutable _enrich_assets (no longer mutates in-place)
- Fix _format_assets return type + handler unpacking

Frontend:
- Fix 18+ hardcoded English strings → t() with new i18n keys (en + ru)
- Mobile "More" nav panel with provider filter and search
- Shared Button.svelte component (4 variants, 2 sizes)
- Shared ErrorBanner.svelte component (8 pages updated)
- SvelteKit goto() replacing window.location.href
- Dashboard grid fixed for 4 cards, paginator opacity consistency

Functionality:
- max_instances=1 on scheduler jobs (prevents duplicate events)
- Webhook provider in watcher (prevents error spam)
- Fix stale SQLModel reference in poller
- Gitea get_repo() direct API call
2026-03-28 13:22:26 +03:00
alexei.dolgolyov 307871cae5 feat: Google Photos provider backend + API hardening
- Add Google Photos provider: client, models, change detector, capabilities
- Add notification templates (en/ru) for all GP event slots
- Add command templates (en/ru) for GP bot commands
- Register GP in slot/command loaders, capabilities, and seeds
- Harden provider API: validate OAuth credentials on create/update
- Add internal URL rewriting for asset fetches (LAN optimization)
- Fix template renderer to handle missing variables gracefully
- Improve webhook command routing for multi-provider support
- Add provider health check endpoint and watcher improvements
2026-03-25 22:07:03 +03:00
alexei.dolgolyov 68ac13b452 feat: NUT (Network UPS Tools) service provider + provider-agnostic UI
Add full NUT support as a polling-based service provider:
- Async TCP client for upsd protocol (port 3493, configurable)
- 8 event types: online, on_battery, low_battery, battery_restored,
  comms_lost, comms_restored, replace_battery, overload
- 3 bot commands: /status, /devices, /battery
- 38 Jinja2 templates (EN+RU notification + command templates)
- Database: tracking config fields, migration, seeds
- Frontend: provider form with host/port/credentials, grid items, i18n

Provider-agnostic UI improvements:
- Remove hardcoded 'immich' defaults from all config forms
- Dynamic collection labels per provider type (Albums/Repos/Boards/UPS Devices)
- Capability-driven test types instead of provider type checks
- Template variable helpers for all providers (was Immich-only)
- Guard Immich-only shared link check to Immich providers
- Auto-clear stale global provider filter from localStorage
- EntitySelect search placeholder shows current selection
- Fix noneLabel in linked target config selectors

New CLAUDE.md rule #8: no provider-specific hardcoding
2026-03-23 23:23:58 +03:00
alexei.dolgolyov 37388c430c feat: locale-aware notification templates + UX improvements
- Add locale support to notification templates (matching command template
  pattern): TemplateSlot now has locale field with (config_id, slot_name,
  locale) uniqueness, nested API format {slot: {locale: template}}
- Migration merges separate EN/RU system configs into unified per-provider
  configs; seeds create one config per provider with multi-locale slots
- Locale-aware dispatch with EN fallback in NotificationDispatcher
- Frontend locale tabs (EN/RU) on template config editor
- Fix tracking config cards not showing default provider icons
- Global provider filter, search palette, and various UX polish
2026-03-23 19:08:48 +03:00
alexei.dolgolyov 6a559bfcd2 feat: Actions system — scheduled mutations on external services
Full-stack implementation of provider-scoped Actions with extensible
executor architecture. First action type: Immich auto_organize (sort
assets into albums by person, CLIP search, date range, favorites).

Core:
- ActionTypeDefinition registry + ActionExecutor ABC with execute/validate/dry-run
- ImmichActionExecutor with multi-album support and client-side filtering
- ImmichClient write methods: add/remove assets, create album, paginated search

Server:
- Action, ActionRule, ActionExecution DB models
- Full CRUD API + manual execute + dry-run + execution history endpoints
- APScheduler integration (interval + cron) for automated execution
- Action type discovery API + provider people endpoint

Frontend:
- Actions page with CRUD, execute/dry-run buttons, inline rule editor
- RuleEditor: person/album MultiEntitySelect pickers, criteria config
- ExecutionHistory: expandable per-rule result details
- MultiEntitySelect reusable component (searchable multi-pick palette)
- Notification tracker album picker migrated to MultiEntitySelect
- Fixed MdiIcon race condition (icons missing after cache-clearing reload)
2026-03-23 16:59:20 +03:00
alexei.dolgolyov 0fde3c6b3d feat: add Planka service provider with full notification and command support
Webhook-based provider for Planka (self-hosted Kanban board) with:
- 15 event types (cards, boards, lists, comments, tasks, attachments, labels)
- Bearer token webhook authentication
- Async API client for boards/cards/lists
- 30 notification templates (en/ru) + 26 command templates (en/ru)
- Bot commands: /status, /boards, /cards, /lists
- Default tracking config, template config, command config seeded on startup
- DB migration for 15 new tracking_config columns
- Frontend: provider config UI with auto-name, Planka-specific hints
- Frontend: tracking config event toggles for all 15 Planka events
2026-03-23 15:54:00 +03:00
alexei.dolgolyov e0bae394ee feat: comprehensive code review fixes — security, performance, quality
Backend security:
- Reject Gitea webhooks when webhook_secret is empty (was silently skipping)
- Add slowapi rate limiting on login (5/min) and setup (3/min) endpoints
- Add CORS middleware with configurable origins
- Mask telegram_webhook_secret in settings API response
- Protect system-owned command template configs from regular user modification
- Increase minimum password length to 8 characters

Backend performance:
- Batch queries in _resolve_command_context (3 queries instead of 3N)
- Concurrent album fetching with asyncio.gather in immich commands
- Singleton Jinja2 SandboxedEnvironment (reuse instead of per-render creation)
- TTLCache for rate limits (bounded memory, auto-eviction)
- Optional aiohttp session reuse in send_reply/send_media_group

Backend code quality:
- Extract dispatch_helpers.py (shared link_data loading + event filtering)
- Extract database/seeds.py from main.py (490 lines → dedicated module)
- Split immich_handler.py (415 lines) into commands/immich/ subpackage
- Replace bare except blocks with logged warnings
- Add per-provider config validation (Pydantic models)
- Truncate command input to 512 chars
- Expose usage_* and desc_* slots in capabilities and variables API

Frontend security:
- CSS.escape() for user-controlled querySelector in highlight.ts
- Client-side password min 8 chars validation on setup and password change

Frontend code quality:
- Replace any types with proper interfaces across top files
- Decompose targets/+page.svelte into TargetForm + ReceiverSection
- Fix $derived.by usage, $state mutation patterns
- Add console.warn to empty catch blocks

Frontend UX:
- Auth redirect via goto() with "Redirecting..." state
- Platform-aware Ctrl/Cmd K keyboard hint
- Remove stat-card hover transform

Frontend accessibility:
- Modal: role=dialog, aria-modal, focus trap, restore focus
- EntitySelect/IconGridSelect: listbox/option roles, aria-selected/disabled
2026-03-23 01:59:51 +03:00
alexei.dolgolyov 0562f78b35 feat: add Scheduler provider + multi-provider UX fixes
Scheduler provider:
- Virtual provider (no external service) that emits SCHEDULED_MESSAGE
  events on user-defined intervals or cron expressions
- Custom variables stored in tracker filters, flattened into template context
- fire_count persists across triggers via tracker state
- APScheduler CronTrigger support for cron-mode schedules
- Default templates (EN+RU), seeded on startup

Multi-provider UX fixes:
- Tracking config hides Immich-specific sections (periodic, scheduled,
  memory, asset display) for non-Immich providers
- Command config driven by provider capabilities — hides commands/settings
  for providers without bot commands
- Template config hides empty "Scheduled Messages" group
- Test menu on tracker targets is provider-aware (Immich shows all 4 test
  types, others show only basic)
- Removed redundant Test button from tracker card
- System-owned tracking configs (user_id=0) seeded for Gitea + Scheduler
- Fixed ownership checks to allow system configs in tracker-target links
- Capabilities cache shared across template-configs and command-configs
- Command tracker bot selector uses EntitySelect instead of raw select
- Sample context includes Gitea + Scheduler variables for template preview
2026-03-22 15:50:51 +03:00
alexei.dolgolyov 6d28cfb8d8 feat: add Gitea as webhook-based service provider
First webhook-based provider integration (Immich uses polling).
Gitea pushes events via POST /api/webhooks/gitea/{provider_id} with
HMAC-SHA256 signature validation.

- 9 event types: push, issue opened/closed/commented, PR opened/closed/merged/commented, release published
- Generic filters system on NotificationTracker (collections, senders, exclude_senders)
- Provider capabilities include supported_filters and webhook_based flag
- Gitea API client for connection testing and repository listing
- 18 default Jinja2 notification templates (EN + RU)
- Frontend: conditional provider forms, Gitea event toggles in tracking config
- Auto-migration for filters column and Gitea tracking flags
2026-03-22 12:58:35 +03:00
alexei.dolgolyov 3e3a6f0777 feat: Discord/Slack/ntfy/Matrix targets, command templates, delete protection, email/matrix bots
- Discord, Slack, ntfy, Matrix notification target types with clients and dispatch
- MatrixBot model + API + frontend in Bots tab
- Command template system fully wired into all handler commands
- Default command templates seeded (EN/RU, 14 slots each)
- Command template editor with variables reference including child fields
- Delete protection on all 10 entity types (409 with consumer details)
- Provider type selector on template config forms
- Target type selector as dropdown with all 7 types
- Response template selector on command config form
- CLAUDE.md: mandatory server restart rule, child properties rule
2026-03-21 20:36:12 +03:00
alexei.dolgolyov 846d480d38 feat: provider-strict configs, slot-based templates, broadcast targets, email bots, command templates
Major architectural improvements:
- Provider-type enforcement: configs validated against provider type at assignment
- TemplateConfig migrated to slot-based pattern (TemplateSlot child table)
- Broadcast targets: TargetReceiver child table for multi-receiver dispatch
- EmailBot: first-class email sender entity with SMTP config, test connection
- CommandTemplateConfig: generic slot-based command response templates
- Provider capability registry: dynamic slot/event/command definitions per provider
- CommandTracker play/pause button matches NotificationTracker style
2026-03-21 16:33:24 +03:00
alexei.dolgolyov 03ec9b3c86 feat: telegram commands, app settings, bot polling, webhook handling, UI improvements
Adds telegram bot command system with 13 commands (search, latest, random, etc.),
webhook/polling handlers, rate limiting, app settings page, and various UI/UX
improvements across all entity pages.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 23:11:42 +03:00
alexei.dolgolyov 03c5c66eed feat: UX & notification improvements — icons, events, chat names, link validation, templates
- Show entity icons on all cards with fallback defaults (providers, trackers, targets, bots)
- Enrich EventLog with provider_name, tracker_name, assets_count; add DB migration
- Dashboard events: filtering (type, provider, search), sorting, pagination, dynamic page size
- Friendly chat names on telegram target cards (resolve from TelegramChat table)
- Test message button on bot chat items with locale-aware messages
- Album public link validation on tracker save with auto-create dialog
- Support albums without public links: conditional <a href> in templates
- Fetch shared links during poll, enrich events with public_url/protected_url
- Per-asset public_url in template context ({share_url}/photos/{asset_id})
- Common date/location detection: common_date + common_location context vars
- Dual date formats: date_format (datetime) + date_only_format (date only)
- Template clone button, HTML link rendering in template preview
- Fix Telegram asset download 401: pass x-api-key headers through client
- Fix provider external_url matching for API key scoping
- Fix event timestamp timezone (append Z suffix for UTC)
- Localize event filter controls, test messages (EN/RU)
- Template variable UI helpers updated with all new fields
- CLAUDE.md: template system sync rules documentation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 16:18:03 +03:00
alexei.dolgolyov 9eec21a5b2 feat: port full CRUD API routes and frontend pages from Immich Watcher
Backend API (38 routes):
- providers: full CRUD + test connection + list collections + API key masking
- trackers: full CRUD + trigger + history + test-periodic/memory
- tracking-configs: full CRUD with Pydantic models, provider_type filter
- template-configs: full CRUD + preview + preview-raw with two-pass validation
- targets: full CRUD + test notification + config masking
- telegram-bots: full CRUD + chat discovery + token endpoint
- users: full admin CRUD + password reset + self-delete protection
- status: dashboard endpoint with providers/trackers/targets/events counts

Frontend pages updated:
- Dashboard with animated stat cards and event timeline
- Providers with proper components, delete confirm, snackbar
- Trackers/targets/tracking-configs/template-configs/telegram-bots/users
  all use PageHeader, Card, Loading, MdiIcon with correct i18n keys

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 00:49:40 +03:00
alexei.dolgolyov 7f99c895a4 feat(notify-bridge): phase 6 - database models and server API
New database schema with ServiceProvider abstraction:
- ServiceProvider (replaces ImmichServer): type + JSON config
- Tracker (replaces AlbumTracker): owns tracking_config_id
- TrackingConfig: provider_type scoped, owned by Tracker
- TemplateConfig: provider_type scoped, owned by Target
- NotificationTarget: owns template_config_id (not tracking_config_id)
- TrackerState, EventLog, User, TelegramBot, TelegramChat

Full FastAPI server:
- /api/providers: CRUD + test connection + list collections
- /api/trackers: CRUD
- /api/tracking-configs: CRUD with provider_type filter
- /api/template-configs: CRUD with provider_type filter, system defaults
- /api/targets: CRUD
- /api/template-vars: variable docs filtered by provider type
- /api/auth: setup, login, refresh, me, password change
- /api/health: health check
- Default template seeding on first startup (EN/RU for Immich)
- pydantic-settings with NOTIFY_BRIDGE_ env prefix

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 23:39:23 +03:00