Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 5604c733d1 | |||
| 3b7808aa9c |
+11
-24
@@ -1,36 +1,23 @@
|
||||
# v0.3.0 (2026-04-22)
|
||||
# v0.3.1 (2026-04-22)
|
||||
|
||||
Major polling perf overhaul for large Immich libraries plus a UX fix for
|
||||
slow bot commands. Combined impact on idle albums: per-tick cost drops
|
||||
from ~150 MB fetched to a few hundred bytes; active albums now fetch
|
||||
O(changes) instead of O(library). Tested against a ~200k-asset library.
|
||||
|
||||
**Schema change:** adds a `meta_fingerprint` JSON column to
|
||||
`notification_tracker_state` — applied automatically by the startup
|
||||
migration, no manual step required.
|
||||
Follow-up perf pass on top of v0.3.0's polling overhaul — extends the same
|
||||
caching discipline to the bot-command read paths so repeat `/random`,
|
||||
`/latest`, `/memory`, etc. against the same album don't each refetch a
|
||||
multi-megabyte album body or pay for a full server-wide `/api/shared-links`
|
||||
listing.
|
||||
|
||||
## Performance
|
||||
|
||||
- **Skip full album fetch on idle ticks** — new `ImmichAlbumMeta` + `get_album_meta()` probe using `?withoutAssets=true` as a cheap change-detection fingerprint. When the fingerprint matches and no pending assets are outstanding, `poll()` short-circuits and does no asset fetch at all. ([fe38d20](https://git.dolgolyov-family.by/alexei.dolgolyov/notify-bridge/commit/fe38d20))
|
||||
- **Delta-fetch active albums** — when the fingerprint changes, poll with `updatedAfter` instead of refetching the whole album; falls back to a full fetch only on count decrease or mixed add+remove that delta can't reconcile. ([fe38d20](https://git.dolgolyov-family.by/alexei.dolgolyov/notify-bridge/commit/fe38d20))
|
||||
- **Parallel meta probes** — `asyncio.gather` over album meta probes so a 20-album tracker pays one round-trip of latency instead of 20. ([fe38d20](https://git.dolgolyov-family.by/alexei.dolgolyov/notify-bridge/commit/fe38d20))
|
||||
- **Tick-scoped shared-links cache** — new `get_all_shared_links_by_album()` coalesces to one `/api/shared-links` request per tick instead of one per changed album. ([fe38d20](https://git.dolgolyov-family.by/alexei.dolgolyov/notify-bridge/commit/fe38d20))
|
||||
- **Module-level users cache** — 1 h TTL, sha256-keyed, shared across providers that target the same Immich server. ([fe38d20](https://git.dolgolyov-family.by/alexei.dolgolyov/notify-bridge/commit/fe38d20))
|
||||
- **Skip `asset_ids` DB rewrite on idle ticks** — watcher no longer rewrites the (potentially ~8 MB for huge albums) JSON column when the fingerprint didn't change. ([fe38d20](https://git.dolgolyov-family.by/alexei.dolgolyov/notify-bridge/commit/fe38d20))
|
||||
- **Adaptive polling** — after 10 empty ticks the scheduler skips 1-in-2, after 30 empty ticks skips 1-in-4; resets on the first detected change or any schedule edit. ([fe38d20](https://git.dolgolyov-family.by/alexei.dolgolyov/notify-bridge/commit/fe38d20))
|
||||
- **APScheduler jitter** — `interval/4`, capped at 30 s, to smooth thundering-herd bursts when many trackers share the same `scan_interval`. ([fe38d20](https://git.dolgolyov-family.by/alexei.dolgolyov/notify-bridge/commit/fe38d20))
|
||||
- **Event payload cap** — 50 added / 200 removed assets per event so a bulk import can't explode a Jinja template or exceed Telegram message limits. ([fe38d20](https://git.dolgolyov-family.by/alexei.dolgolyov/notify-bridge/commit/fe38d20))
|
||||
|
||||
## Features
|
||||
|
||||
- **Chat-action hint stays alive during slow command fetches** — Telegram chat actions expire after ~5 s, so slow bot commands (`/latest`, `/random`, `/favorites`, `/memory`, `/search`, `/find`, `/person`, `/place`, `/summary`) previously showed a hint that vanished long before the media arrived and users saw nothing happening. New `telegram_chat_action` async context manager starts a keep-alive task that re-posts the action every 4 s until it exits; `classify_command_chat_action` maps each command to the right action (`upload_photo` for media-returning commands, `typing` for `/summary`, none for fast DB-only commands like `/status` / `/events`). Wired into both the webhook and long-poll paths. ([69711bb](https://git.dolgolyov-family.by/alexei.dolgolyov/notify-bridge/commit/69711bb))
|
||||
- **TTL-cache `GET /api/albums/{id}` responses** — 60 s TTL, 32-entry FIFO cap, keyed by `(server_digest, album_id)`. Module-scoped rather than instance-scoped because `ImmichClient` is constructed fresh per request in several places (`api/providers.py`, `services/action_runner.py`, command handlers), so an instance cache would never survive a second caller. Mirrors the existing `_users_cache` pattern. ([3b7808a](https://git.dolgolyov-family.by/alexei.dolgolyov/notify-bridge/commit/3b7808a))
|
||||
- **TTL-cache the bucketed shared-links map** — 60 s TTL, keyed by server digest. `/api/shared-links` has no per-album filter, so every `get_shared_links(album_id)` call was already paying for the full server-wide list; now one fetch serves every album until the TTL elapses. ([3b7808a](https://git.dolgolyov-family.by/alexei.dolgolyov/notify-bridge/commit/3b7808a))
|
||||
- **Collapse concurrent cache misses to one fetch** — async lock with an under-lock re-check around the album / shared-links populate step, so a burst of parallel commands hitting the same cold key issues one HTTP call instead of N. ([3b7808a](https://git.dolgolyov-family.by/alexei.dolgolyov/notify-bridge/commit/3b7808a))
|
||||
- **`use_cache=False` escape hatch on mutation / event-detection paths** — `ImmichActionExecutor.execute` (which diffs the current album state to decide what to add) and `ImmichServiceProvider.poll`'s full-fetch path (where a stale entry would silently delay asset-removal events) explicitly bypass the cache. Non-cached fetches still populate it for subsequent readers. ([3b7808a](https://git.dolgolyov-family.by/alexei.dolgolyov/notify-bridge/commit/3b7808a))
|
||||
|
||||
---
|
||||
|
||||
<details>
|
||||
<summary>All Commits</summary>
|
||||
|
||||
- [69711bb](https://git.dolgolyov-family.by/alexei.dolgolyov/notify-bridge/commit/69711bb) — feat(commands): keep chat-action hint alive during slow command fetches *(alexei.dolgolyov)*
|
||||
- [fe38d20](https://git.dolgolyov-family.by/alexei.dolgolyov/notify-bridge/commit/fe38d20) — perf(immich): skip full album fetch on idle ticks; delta-fetch for active ones *(alexei.dolgolyov)*
|
||||
- [3b7808a](https://git.dolgolyov-family.by/alexei.dolgolyov/notify-bridge/commit/3b7808a) — perf(immich): TTL cache for album bodies and shared-link listings *(alexei.dolgolyov)*
|
||||
|
||||
</details>
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "notify-bridge-frontend",
|
||||
"private": true,
|
||||
"version": "0.3.0",
|
||||
"version": "0.3.1",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite dev",
|
||||
|
||||
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "notify-bridge-core"
|
||||
version = "0.3.0"
|
||||
version = "0.3.1"
|
||||
description = "Core library for Notify Bridge — service provider abstractions, models, notifications, and templates"
|
||||
requires-python = ">=3.12"
|
||||
dependencies = [
|
||||
|
||||
@@ -177,7 +177,9 @@ class ImmichActionExecutor(ActionExecutor):
|
||||
needs_thumbnail = album_id in album_created_now
|
||||
|
||||
if album_id and album_id != "__dry_run_new__":
|
||||
album = await self._client.get_album(album_id)
|
||||
# Actions diff the current album state to decide what to
|
||||
# add — must observe fresh data, not a cached view.
|
||||
album = await self._client.get_album(album_id, use_cache=False)
|
||||
if album is None and create_if_missing and create_album_name:
|
||||
if not dry_run:
|
||||
created = await self._client.create_album(create_album_name)
|
||||
|
||||
@@ -2,8 +2,11 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
import aiohttp
|
||||
@@ -18,6 +21,51 @@ _LOGGER = logging.getLogger(__name__)
|
||||
MAX_SEARCH_QUERY_LEN = 256
|
||||
MAX_SEARCH_PERSON_IDS = 50
|
||||
|
||||
# Module-level TTL caches for album bodies and shared-link listings. The
|
||||
# Immich ``GET /api/albums/{id}`` response can be tens or hundreds of MB on a
|
||||
# large album, and bot commands like /random, /latest, /memory all refetch
|
||||
# the same album in quick succession. A short TTL makes repeat runs nearly
|
||||
# instant and deduplicates concurrent fetches so a burst of commands issues
|
||||
# one HTTP call instead of N.
|
||||
#
|
||||
# Caches are module-scoped (not instance-scoped) because ``ImmichClient`` is
|
||||
# constructed fresh per request in several places (api/providers.py,
|
||||
# services/action_runner.py, command handlers), so an instance cache would
|
||||
# never survive to serve a second caller. This mirrors ``_users_cache`` in
|
||||
# ``provider.py``.
|
||||
_ALBUM_CACHE_TTL_SECONDS = 60
|
||||
_SHARED_LINKS_CACHE_TTL_SECONDS = 60
|
||||
# Guard rail against runaway memory — a 200k-asset album response can be
|
||||
# ~150 MB, so even modest caps bound the worst case.
|
||||
_ALBUM_CACHE_MAX_ENTRIES = 32
|
||||
_album_cache_lock = asyncio.Lock()
|
||||
# key = (server_digest, album_id); value = (monotonic_ts, raw_api_dict)
|
||||
# Store the raw dict rather than the parsed ``ImmichAlbumData`` so callers
|
||||
# that pass a ``users_cache`` still get owner-name enrichment on cache hits.
|
||||
_album_cache: dict[tuple[str, str], tuple[float, dict[str, Any]]] = {}
|
||||
_shared_links_cache_lock = asyncio.Lock()
|
||||
# key = server_digest; value = (monotonic_ts, {album_id: [SharedLinkInfo, ...]})
|
||||
# The underlying ``/api/shared-links`` endpoint has no per-album filter, so
|
||||
# every call was already paying for the full server-wide list. Caching the
|
||||
# bucketed result once per server turns N per-album calls into one fetch.
|
||||
_shared_links_cache: dict[str, tuple[float, dict[str, list[SharedLinkInfo]]]] = {}
|
||||
|
||||
|
||||
def _server_digest(url: str, api_key: str) -> str:
|
||||
"""Hashed key that avoids putting raw api_key into cache dict keys."""
|
||||
return hashlib.sha256(f"{url}|{api_key}".encode("utf-8")).hexdigest()[:32]
|
||||
|
||||
|
||||
def invalidate_album_cache() -> None:
|
||||
"""Drop every cached album body. Call after mutations that invalidate
|
||||
the cached view (e.g. integration tests, manual /refresh commands)."""
|
||||
_album_cache.clear()
|
||||
|
||||
|
||||
def invalidate_shared_links_cache() -> None:
|
||||
"""Drop every cached shared-link listing."""
|
||||
_shared_links_cache.clear()
|
||||
|
||||
# User-facing error bodies — Immich responses may leak internal paths,
|
||||
# hostnames, or headers injected by intermediary proxies. These helpers keep
|
||||
# only a short, scrubbed summary; full bodies are logged server-side only.
|
||||
@@ -184,22 +232,30 @@ class ImmichClient:
|
||||
return {}
|
||||
|
||||
async def get_shared_links(self, album_id: str) -> list[SharedLinkInfo]:
|
||||
links: list[SharedLinkInfo] = []
|
||||
try:
|
||||
async with self._session.get(
|
||||
f"{self._url}/api/shared-links",
|
||||
headers=self._headers,
|
||||
) as response:
|
||||
if response.status == 200:
|
||||
data = await response.json()
|
||||
for link in data:
|
||||
album = link.get("album")
|
||||
key = link.get("key")
|
||||
if album and key and album.get("id") == album_id:
|
||||
links.append(SharedLinkInfo.from_api_response(link))
|
||||
except aiohttp.ClientError as err:
|
||||
_LOGGER.warning("Failed to fetch shared links: %s", err)
|
||||
return links
|
||||
bucketed = await self._get_shared_links_bucketed()
|
||||
return list(bucketed.get(album_id, []))
|
||||
|
||||
async def _get_shared_links_bucketed(self) -> dict[str, list[SharedLinkInfo]]:
|
||||
"""Return ``{album_id: [SharedLinkInfo, ...]}`` for the server, hitting
|
||||
the module-level TTL cache first. Underlying Immich endpoint has no
|
||||
per-album filter, so one server-wide fetch serves every caller until
|
||||
the TTL elapses.
|
||||
"""
|
||||
digest = _server_digest(self._url, self._api_key)
|
||||
now = time.monotonic()
|
||||
entry = _shared_links_cache.get(digest)
|
||||
if entry is not None and (now - entry[0]) < _SHARED_LINKS_CACHE_TTL_SECONDS:
|
||||
return entry[1]
|
||||
|
||||
async with _shared_links_cache_lock:
|
||||
# Re-check under the lock — another coroutine may have refreshed
|
||||
# while we waited.
|
||||
entry = _shared_links_cache.get(digest)
|
||||
if entry is not None and (time.monotonic() - entry[0]) < _SHARED_LINKS_CACHE_TTL_SECONDS:
|
||||
return entry[1]
|
||||
fresh = await self.get_all_shared_links_by_album()
|
||||
_shared_links_cache[digest] = (time.monotonic(), fresh)
|
||||
return fresh
|
||||
|
||||
async def get_all_shared_links_by_album(self) -> dict[str, list[SharedLinkInfo]]:
|
||||
"""Fetch every shared link on the server, bucketed by album id.
|
||||
@@ -247,7 +303,29 @@ class ImmichClient:
|
||||
self,
|
||||
album_id: str,
|
||||
users_cache: dict[str, str] | None = None,
|
||||
*,
|
||||
use_cache: bool = True,
|
||||
) -> ImmichAlbumData | None:
|
||||
"""Fetch an album by id, optionally serving from the module-level
|
||||
TTL cache. Pass ``use_cache=False`` from paths that must observe the
|
||||
current server state (e.g. the notification poll loop's full-fetch
|
||||
path, where a stale cached entry would delay asset-removal events).
|
||||
Non-cached fetches still populate the cache for subsequent readers.
|
||||
"""
|
||||
cache_key = (_server_digest(self._url, self._api_key), album_id)
|
||||
if use_cache:
|
||||
entry = _album_cache.get(cache_key)
|
||||
if entry is not None and (time.monotonic() - entry[0]) < _ALBUM_CACHE_TTL_SECONDS:
|
||||
# Rehydrate per-call so ``users_cache`` enrichment is applied
|
||||
# with the caller's dict, not whichever one was live when the
|
||||
# cache was populated.
|
||||
return ImmichAlbumData.from_api_response(entry[1], users_cache)
|
||||
|
||||
# Deliberately fetch without holding a lock so concurrent calls for
|
||||
# *different* album_ids (the common case from asyncio.gather in
|
||||
# fetch_albums_with_links) stay parallel. The worst case is a small
|
||||
# duplicate-fetch stampede when two requests miss the same album at
|
||||
# the same instant — acceptable for our scale.
|
||||
try:
|
||||
async with self._session.get(
|
||||
f"{self._url}/api/albums/{album_id}",
|
||||
@@ -260,10 +338,18 @@ class ImmichClient:
|
||||
f"Error fetching album {album_id}: HTTP {response.status}"
|
||||
)
|
||||
data = await response.json()
|
||||
return ImmichAlbumData.from_api_response(data, users_cache)
|
||||
except aiohttp.ClientError as err:
|
||||
raise ImmichApiError(f"Error communicating with Immich: {err}") from err
|
||||
|
||||
async with _album_cache_lock:
|
||||
# Evict the oldest entry if we're at the cap — simple FIFO is fine
|
||||
# for our access pattern (commands touch a small working set).
|
||||
if len(_album_cache) >= _ALBUM_CACHE_MAX_ENTRIES and cache_key not in _album_cache:
|
||||
oldest = min(_album_cache.items(), key=lambda kv: kv[1][0])[0]
|
||||
_album_cache.pop(oldest, None)
|
||||
_album_cache[cache_key] = (time.monotonic(), data)
|
||||
return ImmichAlbumData.from_api_response(data, users_cache)
|
||||
|
||||
async def get_album_meta(self, album_id: str) -> ImmichAlbumMeta | None:
|
||||
"""Fetch album metadata without the assets array.
|
||||
|
||||
|
||||
@@ -292,7 +292,13 @@ class ImmichServiceProvider(ServiceProvider):
|
||||
# the full-fetch path so removals get detected.
|
||||
|
||||
# Full fetch: first tick, or count-decreased, or delta-unsafe.
|
||||
album = await self._client.get_album(album_id, self._users_cache)
|
||||
# Bypass the module-level album cache — this path runs when we
|
||||
# specifically need the current server state (e.g. to detect
|
||||
# asset removals), so a stale cached entry would silently delay
|
||||
# the event.
|
||||
album = await self._client.get_album(
|
||||
album_id, self._users_cache, use_cache=False,
|
||||
)
|
||||
if album is None:
|
||||
# Album was deleted between meta probe and full fetch — handle
|
||||
# the deletion the same way as above.
|
||||
|
||||
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "notify-bridge-server"
|
||||
version = "0.3.0"
|
||||
version = "0.3.1"
|
||||
description = "Standalone Notify Bridge server — FastAPI REST API with SQLite database"
|
||||
requires-python = ">=3.12"
|
||||
dependencies = [
|
||||
|
||||
Reference in New Issue
Block a user