Adds the user-visible piece of the Workload refactor:
- web/src/lib/types.ts — Workload, Container, ContainerView,
App, WorkloadKind, ContainerState
- web/src/lib/api.ts — listWorkloads, getWorkload,
listWorkloadContainers, setWorkloadAppID,
listContainers (with filter),
CRUD for apps
- web/src/lib/i18n/{en,ru}.json — nav.containers
- web/src/routes/+layout.svelte — "Containers" nav item between Stacks
and Deploy, IconContainer
- web/src/routes/containers/+page.svelte — global Containers table:
* filter chips for kind (project/stack/site) and state
* client-side search across workload name / role / image /
subdomain / container ID prefix
* Workload column links to the kind-specific detail page,
resolved through a one-time /api/workloads call to map
workload_id → ref_id
* existing /containers/stale route untouched
The page renders against the live database now — boot backfill
populated workload rows from existing projects/stacks/sites,
the deployer dual-writes containers on every deploy, and the
30s reconciler keeps the index in sync with `docker ps`.
Persists every inbound webhook hit (project + site) so users can debug
"why didn't my deploy fire?" without grepping daemon logs. Surfaces a
14-day rolling history under the WebhookPanel on each project + site
detail page; refreshes every 30s while open. Daily cron prunes records
older than 14 days alongside the existing event log prune.
Schema:
- webhook_deliveries(id, target_type, target_id, target_name, received_at,
source_ip, signature_state, status_code, outcome, detail, body_size)
- indexes on (target_type,target_id,received_at) and (received_at)
Backend:
- store: WebhookDelivery model + Insert/List/Prune helpers
- webhook/handler: deferred recordDelivery() captures the final outcome
on every return path including HMAC rejects, image mismatch, no-stage,
auto_deploy=false, and successful deploys; signatureStateFor()
classifies "unconfigured" vs "missing" vs "invalid" vs "valid"
- api: GET /api/{projects,sites}/{id}/webhook/deliveries with
parseLimit() helper (default 50, max 200)
- main: daily prune cron retains the last 14 days
Frontend:
- WebhookDeliveryLog.svelte: panel with refresh button, status code +
outcome + signature badges, relative time tooltip-on-hover for
absolute time, source IP column
- Mounted below WebhookPanel on project + site detail pages
- en/ru i18n strings for outcome/signature enums and column labels
Adds an opt-in inbound HMAC scheme so a leaked URL alone is not enough
to forge deploy/sync requests — the caller must also know a separate
signing secret. Header format is X-Hub-Signature-256, matching the
Gitea/GitHub/GitLab convention so existing CI integrations work without
custom code.
Behaviour:
- per-project / per-site signing_secret is independent of the URL secret
- require_signature flag does a hard 401 on missing/invalid signatures
- even when require_signature is off, an *invalid* submitted signature
returns 401 — surfaces CI misconfiguration instead of silently passing
- comparison uses subtle/hmac.Equal (constant time)
Backend:
- store: webhook_signing_secret + webhook_require_signature columns on
projects + static_sites; scanProject helper, scan helpers updated; new
Set* helpers for both fields
- webhook/handler: verifyHMAC helper, body read once, integrated into
both project and site handlers
- api: per-entity signing-secret rotate / disable / require-toggle
endpoints under /api/{projects,sites}/{id}/webhook/...
Frontend:
- WebhookPanel gains optional signing handlers (no breaking change for
existing callers; signing UI hides when handlers aren't wired)
- one-shot reveal of the issued secret with copy + dismiss
- ToggleSwitch for require-signature, disabled until a secret is issued
- en/ru i18n strings
Tests:
- HMACRequiredAndValid (200 + deploy fires)
- HMACRequiredButMissing (401, no deploy)
- HMACPresentButWrong (401 even when require_signature=false)
- HMACOptionalUnsignedAccepted (200 when neither configured)
Outgoing notifications were bare POSTs with no auth and no way to verify
they came from Tinyforge. They also went out from one global URL only,
even though stages had a notification_url field, and static-site sync
emitted no events at all.
Schema: add notification_url + notification_secret (lazy-generated) to
settings, projects, stages and static_sites. Migrations are additive.
Notifier: SendSigned computes HMAC-SHA256 over the exact body bytes and
sends X-Hub-Signature-256 (GitHub-compatible — receivers built for
GitHub/Gitea/Forgejo verify out of the box). Aux headers
X-Tinyforge-Event/Delivery/Timestamp/Tier are advisory and not signed.
Empty secret => unsigned send for back-compat.
Resolution: deploys fall through stage > project > settings, sites fall
through site > settings. The secret travels with the URL that sourced
it, so any tier can sign even when its parents are unsigned. Site sync
events now actually emit (site_sync_success / site_sync_failure).
API: 12 new endpoints — {GET secret, POST regenerate, POST disable,
POST test} for each of the 4 tiers. SendSyncForTest returns
status_code/latency_ms/signature_sent/delivery_id/response_snippet so
the UI surfaces receiver feedback inline.
UI: shared OutgoingWebhookPanel.svelte fits the existing card aesthetic.
Signing-state pill, secret reveal-on-demand, regenerate/disable behind
ConfirmDialog modals (not inline strips — too easy to misclick), send-
test result card with colour-coded status. Wired into Settings →
Integrations, project edit form, per-stage edit, and per-site detail.
EN + RU i18n.
Tests: round-trip (sender signs, receiver verifies), tampered-body and
wrong-secret rejection, unsigned-send omits header, send-test surfaces
4xx, concurrent fan-out via Drain. Resolver precedence locked for both
deploy and site paths.
Docs: docs/webhooks.md with header reference, verifier snippets in
Node/Python/Go, and a recipe for the service-to-notification-bridge
generic webhook provider.
Security:
- rate limit /api/webhook routes per-IP and cap concurrent site syncs
- global SSE connection cap (256) with new sse_gate
- validate ?tail= and cap JSON log responses at 4 MiB
- strip ANSI/CSI/OSC and control bytes from streamed log lines
- redact webhook secret from request log middleware
- scrub host details from /api/health for non-admin viewers
- drop container_id from /api/system/stats/top for non-admins
- generate webhook secrets via crypto/rand; require >=32 chars on insert
- verify iid path consistency in streamContainerLogs
- LimitReader on site webhook body; reject malformed non-empty bodies
Concurrency / correctness:
- stats collector: Stop() no longer hangs without Start(), semaphore
acquired in parent loop so ctx cancellation short-circuits the queue,
in-flight tick cancellable via shared base context, zero-ts guard
- webhook handler: replace fire-and-forget goroutine with WaitGroup-tracked
workers + Drain() wired into graceful shutdown
- $derived(() => ...) mis-idiom fixed in ContainerStats / InstanceCard /
ProjectCard (returned function instead of value)
- SystemResourcesCard: rename `window` and `t` locals to avoid shadowing
globalThis.window and the i18n `t` import
Quality / performance:
- replace O(n^2) insertion sort with sort.Slice in stats top
- runMigrations only swallows duplicate-column / already-exists errors
- PruneStatsSamplesBefore wrapped in a transaction
- collapse N+1 in unusedImageStats / pruneImages to one ListAllInstances
pass; surface DB errors instead of silently treating them as inactive
- run Docker Info + DiskUsage in parallel via errgroup
- container log SSE emits `: ping` heartbeat every 20 s
- imageMatches case-insensitive on registry host (RFC behaviour)
- log warning on invalid stage tag pattern instead of silent skip
- reject malformed non-empty site webhook payloads
Frontend / i18n:
- shared formatBytes utility replaces three local copies
- statsInterval store drives dynamic "no samples / collection disabled"
copy across ContainerStats and SystemResourcesCard
- top consumers row now shows owner_name (project/stage or site name)
- drop seven `as any` casts on the Settings type; add cloudflare_api_token
write-only field
- move "Service status", "Docker daemon", "Docker unreachable",
"Proxy unreachable", "reachable", and "Docker daemon is not reachable."
strings into en/ru i18n bundles
Background collector samples CPU/memory/network/block I/O for every
instance and site on a configurable interval (default 15s, range
5-300s), persists samples to SQLite with a configurable retention
window (default 2h, range 0-24h), and skips ticks gracefully when
the Docker daemon is unreachable. Settings are reloadable without
a restart — each tick re-reads them.
New API endpoints:
- GET /api/system/stats (host snapshot: info + df)
- GET /api/system/stats/history
- GET /api/system/stats/top?by=cpu|memory
- GET /api/projects/{id}/stages/{s}/instances/{iid}/stats/history
- GET /api/sites/{id}/stats[/history]
- GET /api/sites/{id}/logs (SSE + JSON, reuses instance log streamer)
Frontend:
- ECharts added with tree-shaken imports (~180KB gzip) for
future-proof time-series/gantt/graph visualizations
- CollapsibleSection wraps all dashboard sections (system health,
daemons, system resources, static sites, projects) with
localStorage-persisted open state
- SystemResourcesCard shows capacity tiles, workload utilization
chart with 30m/2h/6h/24h window picker, disk breakdown with
reclaimable callouts, and top 5 consumers
- ContainerStats and ContainerLogs take a source discriminated union
so sites reuse the same components as instances; sites detail page
embeds both for Deno backend debugging
- Settings › Maintenance exposes collection interval + retention
- Docker-unavailable state returns 503 and renders an amber banner
instead of a generic 500
Full i18n coverage (en + ru) for all new strings.
Replace the single global webhook secret with entity-scoped secrets stored
on each project and static site. Webhook-driven project autocreate is
removed — projects must exist before their URL can trigger deploys.
Also wires static-site webhooks (sync_trigger=push|tag), turning the
previously inert "push" trigger into a functional one: POST the site's
webhook URL from a Git provider and Tinyforge re-syncs on matching refs.
- Adds webhook_secret columns + unique indexes to projects and static_sites
- Per-entity GET/regenerate endpoints under /api/projects/{id}/webhook
and /api/sites/{id}/webhook (admin-only)
- Removes /api/settings/webhook-url and the global webhook panel
- Reusable WebhookPanel Svelte component on both detail pages, i18n in en/ru
- Tests for matcher (siteRefMatches, ParseImageRef) and handler (project
match/mismatch/404 and site push/manual/branch-skip)
Nav & UI polish
- Sidebar nav items show monospace count badges (projects, sites, stacks,
proxies). Events badge shows error count only, styled red as actionable
- New $lib/stores/navCounts.ts polls all counts in parallel every 60s and
refreshes on route change so badges track mutations
- Login page gets a dynamic forge backdrop: rotating conic glow, drifting
embers, dot-grid texture, vignette — all pure CSS, reduced-motion safe
- main element gets scrollbar-gutter: stable so Settings tab switching no
longer shifts horizontally when content heights differ
Events i18n
- events.source.* dictionary rewritten to match actually-emitted backend
sources (deploy, static_site, stale_scanner, stale_cleanup, admin);
dead keys (container, proxy, system) removed
- EventLogFilter.allSources + /events default sources state updated to match
- Localize "{N} total" via events.totalCount in the page hero toolbar
Backend
- Stage API accepts enable_proxy on create/update (defaults to true) so
proxy registration can be opted out per stage
Concurrency
- api.ts: queued request waiters no longer double-increment the inflight
counter; releasing a slot hands it off directly
Reactive effects
- project detail / env / volumes pages wrap side-effect calls in untrack()
to prevent $effect feedback loops when their loaders mutate tracked state
Adds a new Stacks feature: upload/edit docker-compose YAML,
deploy as atomic units, browse revisions, roll back, and
stream logs. Backend in internal/stack + internal/api/stacks.go,
persistent storage in internal/store/stacks.go.
Stacks pages (list, new, detail) use a modern Forge aesthetic —
Instrument Serif display type, JetBrains Mono for meta/code,
indigo ember accents, dot-grid hero, registration marks on
hover, terminal panel for logs. Palette is sourced from the
app's existing design tokens so the feature remains consistent
with the rest of Tinyforge.
Fonts self-hosted via @fontsource/instrument-serif and
@fontsource/jetbrains-mono to satisfy the strict CSP.
- Add concurrency limiter (max 4 GET requests) to API layer, leaving
slots for SSE and health checks. Write ops bypass the limiter.
- Add AbortController to ContainerStats, project detail page, and
dashboard to cancel in-flight requests on navigation/unmount.
- Move global SSE connection from layout to events page (only consumer).
- Add 30s heartbeat to SSE endpoint to detect zombie connections.
- Serialize dashboard project fetches to avoid parallel burst.
- Rebuild frontend in dev-server.sh so go:embed stays in sync.
- Add GET /api/projects/{id}/stages/{stage}/instances/{iid}/logs endpoint
- Supports JSON mode (returns array of lines) and SSE mode (streams in real-time)
- Docker log stream header (8-byte prefix) stripped automatically
- ContainerLogs component with:
- Tail line selector (50/200/500/1000)
- Follow button for real-time streaming via SSE
- Auto-scroll to bottom
- Dark terminal-style display
- Close button
- Logs button (events icon) on each instance card
- i18n keys in EN and RU
- Add GET /api/projects/{id}/images endpoint returning local images matching the project
- Add ListImagesByRef with tag, size, and created timestamp to Docker client
- Display images table on project page with tag, ID (truncated), size (MB), and created date
- Only shown when Docker is available and images exist locally
- Image prune only removes images matching project image refs, skips active instances
- Add ListImagesByRef and RemoveImage to Docker client
- Fix 409 conflict: use listProjects instead of duplicate POST
- Add "Deploy immediately" toggle to Quick Deploy (off by default)
- Replace raw access list ID with EntityPicker on project edit form
- Trigger proxy resync on access list change
- Fix stage form layout: single responsive row
- Fix empty port default on project creation
- Improve inspect error message for remote Docker
- Replace raw NPM access list ID input with EntityPicker on project edit form
- Resolve access list name from NPM API when editing project
- Add "Deploy immediately" toggle to Quick Deploy (off by default)
- Fix stage form layout: all fields on same row with toggles
- Fix empty port default on project creation (placeholder instead of pre-filled)
- Improve inspect error message when Docker is unavailable
- Trigger proxy resync when NPM access list changes
- Resolve access list name on NPM settings page load
Resource limits:
- Add cpu_limit (cores) and memory_limit (MB) fields to Stage model
- Pass limits to Docker container via NanoCPUs and Memory in HostConfig
- Add CPU/Memory fields to stage creation form in project detail
- 0 = unlimited (default)
NPM access list:
- Add npm_access_list_id to Settings (global default) and Project (per-project override)
- Per-project overrides global when > 0
- NPM provider passes access_list_id when configuring proxy hosts
- Add GET /api/settings/npm-access-lists endpoint to list NPM access lists
- Add access list picker on NPM settings page (global)
- Add access list ID field on project edit form (per-project)
- DB migrations for all new columns
- Add enable_proxy toggle to Quick Deploy form (defaults to on)
- Add DELETE /api/events/log/{id} and DELETE /api/events/log endpoints
- Add Clear All button with confirmation on Events page
- Rename "NPM Proxy" to "Enable Proxy" on stage form (provider-agnostic)
- Fix polling interval validation (min 60s) and number input trim errors
- Fix domain field no longer required in settings
- Add /proxies page showing deploy-managed proxy routes with project/stage links, search, and status
- Add GET /api/proxies endpoint joining instances with project/stage names
- Add POST /api/settings/npm/test endpoint for NPM connection validation
- Add GET /api/auth/mode public endpoint for auth mode detection
- Add NPM Test Connection button with validation on save
- Fix OIDC SSO button only shown when auth_mode is oidc
- Fix webhook URL showing empty when domain not set (fallback to request host)
- Fix quick deploy double-tag (image:latest:latest) by splitting tag from image URL
- Fix trim() errors on number inputs in deploy and settings forms
- Fix NPM client auto-append /api to base URL
- Sanitize NPM test error messages (no raw HTML)
- Remove healthcheck field from Quick Deploy form
- Fix env vars placeholder newline
- Make domain field optional in settings
- Set polling interval minimum to 60s
- Add Proxies and Events to sidebar navigation
- Fix SSL cert name flash on NPM settings page
- Fix empty state icon on proxies page
- Add proxy provider selector (None/NPM) to settings page with radio cards
- Show warning when switching to None about existing NPM routes
- Conditionally hide SSL certificate picker and NPM credentials when provider is None
- Fix webhook URL regenerate not updating UI (field name mismatch: url vs webhook_url)
- Fix dev-server.sh to persist ENCRYPTION_KEY across restarts via data/.dev-key
- Fix selected radio card background visibility in dark mode
- Detect existing projects with same image on quick deploy; show conflict dialog with options
- Move logout button to sidebar header as icon-only
- Replace backup checkbox with ToggleSwitch component
- Allow unsafe-inline in CSP script-src for SvelteKit hydration
- Guard SSE connection behind isAuthenticated() check
- Add notification_url ALTER TABLE migration for existing databases
- Restore RegisterPersistentLogger on event bus
Add backup/restore functionality for the SQLite database. Users can
trigger manual backups, configure automatic backups on an interval
with retention policies, list/download/delete backups, and restore
from any backup.
- Backup engine using VACUUM INTO (safe with WAL mode)
- Backup metadata tracked in DB, files stored in DATA_DIR/backups/
- Settings: backup_enabled, backup_interval_hours, backup_retention_count
- API: POST/GET/DELETE /api/backups, download, restore endpoints
- Autobackup via cron scheduler with configurable interval
- Retention: prune on startup, after each backup (manual and auto)
- Orphan cleanup: removes backup files without metadata on startup
- Restore: replaces DB and triggers graceful server shutdown
- Settings UI: /settings/backup with toggle, interval, retention config
- Backup list with download, delete, restore actions
- i18n: English and Russian translations
- CRITICAL: Change DNS zones endpoint from GET to POST to avoid
leaking API token in URL query parameters
- HIGH: Add sync.RWMutex to protect dnsProvider field in Server,
Deployer, and proxy Manager against concurrent read/write races
- HIGH: Capture old DNS provider reference synchronously before
launching background cleanup goroutine
- HIGH: Use getDNS()/getDNSProviderLocked() accessors instead of
direct field reads in all DNS operations
Add flexible DNS management to Docker Watcher. By default, wildcard DNS
is assumed (current behavior). When disabled, users can configure a
Cloudflare DNS provider with API token and zone selection. DNS A records
are automatically created/updated/deleted in sync with proxy consumers
(deployed instances and standalone proxies).
- Settings: wildcard_dns toggle, dns_provider, cloudflare credentials
- Cloudflare client: Provider interface with EnsureRecord/DeleteRecord/ListRecords
- DNS lifecycle hooks in deployer and proxy manager (best-effort)
- Settings UI: DNS config section with provider picker, zone selector, test button
- DNS Records page at /dns with filtering, sync status, reconciliation
- Records visible in both wildcard and managed modes
- Cleanup on provider change: removes old records when switching modes
Replace confusing shared/isolated volume modes with explicit scopes:
- instance: per-deploy isolated directory
- stage: shared within a stage across deploys
- project: shared across all stages
- project_named: named group within a project
- named: global named volume across projects
- ephemeral: tmpfs in-memory mount
Includes schema migration (shared→project, isolated→instance),
backward-compatible deployer resolution, scope metadata API endpoint,
and redesigned volume editor UI with scope guide cards and hints.
- Classify Docker errors into categories (socket_not_found, connection_refused,
permission_denied, timeout, tls_error) with platform-specific hints
- Enrich GET /api/health with structured diagnostics (category, hints, platform)
- Expandable hints panel in sidebar when Docker is disconnected
- "Retry now" button for immediate re-check
- Collapsible raw error details for advanced users
- GET /api/health endpoint returning Docker connectivity status
- Sidebar shows Docker connection dot (green=connected, red=disconnected)
- Stale scanner returns store-only results when Docker is unavailable
- Polls health every 30s
Add all frontend pages for observability & proxy management:
- Proxy Viewer: /proxies with grouped view, filtering, health indicators
- Proxy Creation: form with live validation, diagnostic hints, edit/delete
- Stale Containers: /containers/stale with dashboard widget, cleanup actions
- Event Log: /events with filters, pagination, real-time SSE streaming
- Navigation: proxies and events links in sidebar
- i18n: full EN/RU translations for all new features
- Settings: stale threshold configuration
Add standalone proxy management:
- Multi-step validation pipeline (DNS, TCP, HTTP) with diagnostic hints
- Proxy lifecycle: create/update/delete via NPM API with SSL auto-assign
- Periodic health monitoring (5min) with event log on status transitions
- Unified /api/proxies/all endpoint merging standalone + managed proxies
- Frontend types and API functions for downstream UI phases
Add database foundation for observability features:
- event_log table with severity/source filtering and pagination
- standalone_proxies table for user-created reverse proxies
- stale_threshold_days setting (default 7 days)
- Auto-persist warn/error events from event bus to database
- SSE broadcast of persistent events for real-time UI updates
- Frontend types and API functions for downstream UI phases
- Add Stage button with inline form (name, tag pattern, max instances, auto deploy)
- Delete stage button per stage header
- Add createStage/updateStage/deleteStage to API client
- Fix null stages crash on env page
- Fix null slices globally in respondJSON via reflection
- Add ListImages() to registry interface, implement for Gitea
- Add owner field to registry config (needed for Gitea packages API)
- GET /api/registries/:id/images endpoint
- "Browse Images" button on Projects and Quick Deploy pages
- Image dropdown with registry grouping and search
- i18n support (EN/RU) for all new UI strings
- Add auth guard in API client: 401 responses redirect to /login
- Fix go:embed with all: prefix to include _app/ directory
- Move jsonContentType middleware to /api routes only
- Use http.ServeContent for correct MIME type detection
Per-stage env var overrides with encryption for secrets.
Volume mounts with shared/isolated modes (isolated appends
/{stage}-{tag}/ to source path). Store CRUD, API endpoints,
and frontend editors for both. Env merge during deploy.
Remove webhook secret from logs and API response.
Add auth-pending note to router. Fix decrypt fallback that
would use ciphertext as auth token on decrypt failure.
alexei.dolgolyov