tiny-forge/PLAN.md

Docker Watcher — Implementation Plan

Overview

A self-hosted tool that automates Docker container deployment with Nginx Proxy Manager integration. Detects new images from Gitea/GitHub registries, deploys containers, and configures reverse proxy routing — all from a web dashboard. Supports multiple simultaneous versions of the same project. DNS is handled by a Cloudflare wildcard record (*.dolgolyov-family.by) — no per-project DNS management needed.

Architecture

Gitea CI → pushes image → Registry
     │                        ↓
     │              Docker Watcher (Go)
     │              ├── Secret webhook URL (instant)
     │              └── Registry poller (fallback)
     │                        ↓
     └── or: POST /api/webhook/<secret-uuid>
              with {"image": "registry/org/app:tag"}
                              ↓
                  Known project? ──────────────────┐
                  ↓ yes                             ↓ no
           Match tag → stage              Auto-create project
                  ↓                       with defaults from
           auto_deploy?                   image inspection
           ↓ yes       ↓ no              (EXPOSE, labels)
     Deploy now    Notify, wait                ↓
           ↓       for UI trigger         Deploy with defaults
           ↓           ↓
     Pull image
     Start new container on shared network
     (old container stays if multi-instance)
           ↓
     NPM API: create proxy host (if first deploy for this subdomain)
     (DNS already handled by Cloudflare wildcard *.domain)
           ↓
     Health check
     → success: done, notify
     → failure: remove new container, alert

Decisions

Decision	Choice	Rationale
Language	Go	Single binary, excellent Docker SDK, low resource usage
Web UI	SvelteKit (embedded in Go binary)	User's existing stack, lightweight
Reverse proxy	Nginx Proxy Manager	Already deployed, API available
DNS	Cloudflare wildcard *.{domain}	One-time setup, all subdomains auto-resolve
Routing	Subdomain-based	No sub-path issues with SPAs
Image detection	Secret webhook URL + polling	Webhook for speed, polling as fallback
Config storage	SQLite (YAML for initial seed only)	Editable via UI, no manual file editing
Credentials	Encrypted in SQLite (AES-256)	Single ENCRYPTION_KEY env var
Webhook auth	Secret UUID in URL	No tokens needed, simple CI integration
Multi-instance	Yes	Multiple tags of same project can run simultaneously
Deployment target	Same TrueNAS host	Docker socket mounted

Subdomain Convention

Type	Pattern	Example
Dev (default)	stage-dev-{project}.{domain}	stage-dev-web-app-launcher.dolgolyov-family.by
Dev (specific tag)	stage-dev-{project}-{tag}.{domain}	stage-dev-web-app-launcher-abc123.dolgolyov-family.by
Release (default)	stage-rel-{project}.{domain}	stage-rel-web-app-launcher.dolgolyov-family.by
Release (specific tag)	stage-rel-{project}-{tag}.{domain}	stage-rel-web-app-launcher-v1-2-0.dolgolyov-family.by
Production	{custom}.{domain}	launcher.dolgolyov-family.by

Tags are sanitized for DNS: dots → dashes, lowercase, truncated to fit DNS limits.

Configuration

First Launch

YAML seed file exists? → import into SQLite → done
No YAML?               → empty state, configure everything via UI

After import, all configuration lives in SQLite and is managed via the Web UI. YAML is never read again unless user clicks "Re-import config" or "Export config".

Seed Config Format (optional)

global:
  domain: dolgolyov-family.by
  server_ip: 93.84.96.191
  network: staging-net
  subdomain_pattern: "stage-{stage}-{project}"
  notification_url: https://notify.dolgolyov-family.by/webhook
  npm:
    url: http://npm:81
    email: docker-watcher@dolgolyov-family.by
    password: "npm-password-here"
registries:
  gitea:
    url: https://git.dolgolyov-family.by
    type: gitea
    token: "gitea-token-here"

projects:
  web-app-launcher:
    registry: gitea
    image: git.dolgolyov-family.by/alexei/web-app-launcher
    port: 3000
    healthcheck: /api/health
    env:
      NODE_ENV: production
    stages:
      dev:
        tag_pattern: "dev-*"
        auto_deploy: true
        max_instances: 5
      rel:
        tag_pattern: "v*"
        auto_deploy: false
        max_instances: 2
      prod:
        tag_pattern: "v*"
        auto_deploy: false
        confirm: true
        promote_from: rel
        max_instances: 2
        subdomain: launcher

Web UI Sections

Dashboard

Overview of all projects with their running instances:

• Project name, running instance count, latest activity
• Quick status indicators (healthy / stopped / failing)
• "Quick Deploy" button for ad-hoc image deployment

Project Detail

Per-project view with stages and instances:

• Each stage shows all running instances with: tag, status, URL, uptime
• Controls per instance: Stop, Start, Restart, Remove
• "Deploy new version" dropdown — lists available tags from registry
• Deploy history log

Quick Deploy

For deploying images not yet configured as projects:

1. Paste image URL (e.g., git.dolgolyov-family.by/alexei/my-app:dev-abc123)
2. Docker Watcher pulls and inspects image (EXPOSE port, HEALTHCHECK, labels)
3. Pre-fills form with sensible defaults (project name, port, stage, subdomain)
4. User reviews, tweaks, clicks "Deploy"
5. Project is auto-created in the DB for future use

Settings

• Registries — add/edit/delete registries, test connection
• Credentials — NPM, registry tokens (encrypted, shown as ••••••••)
• Global — domain, server IP, Docker network, subdomain pattern, polling interval
• Notifications — webhook URL
• Webhook URL — shows the secret deploy URL, "Regenerate" button

Projects Config

• Add / edit / delete projects via UI
• Configure image, port, healthcheck, env vars, volumes per project
• Add / remove stages, set tag patterns, auto-deploy, subdomain overrides, max instances

Project Structure

docker-watcher/
├── cmd/
│   └── server/
│       └── main.go                 # Entry point
├── internal/
│   ├── config/
│   │   ├── config.go               # YAML seed parsing
│   │   └── config_test.go
│   ├── docker/
│   │   ├── client.go               # Docker Engine API wrapper
│   │   ├── container.go            # Create, start, stop, remove, inspect
│   │   └── client_test.go
│   ├── npm/
│   │   ├── client.go               # NPM API client (auth, CRUD proxy hosts)
│   │   └── client_test.go
│   ├── registry/
│   │   ├── registry.go             # Interface
│   │   ├── gitea.go                # Gitea registry implementation
│   │   ├── github.go               # GitHub Container Registry (future)
│   │   ├── poller.go               # Periodic tag polling
│   │   └── registry_test.go
│   ├── deployer/
│   │   ├── deployer.go             # Orchestrates full deploy flow
│   │   ├── rollback.go             # Rollback on failure
│   │   └── deployer_test.go
│   ├── health/
│   │   ├── checker.go              # HTTP health checks with retries
│   │   └── checker_test.go
│   ├── notify/
│   │   ├── notifier.go             # Webhook notifications
│   │   └── notifier_test.go
│   ├── webhook/
│   │   ├── handler.go              # Secret URL webhook receiver
│   │   └── handler_test.go
│   ├── api/
│   │   ├── router.go               # HTTP API for web UI
│   │   ├── projects.go             # Project CRUD endpoints
│   │   ├── registries.go           # Registry CRUD endpoints
│   │   ├── settings.go             # Global settings endpoints
│   │   ├── instances.go            # Instance start/stop/restart/remove
│   │   ├── deploys.go              # Deploy + quick deploy endpoints
│   │   └── middleware.go           # Auth, logging, CORS
│   ├── store/
│   │   ├── store.go                # SQLite schema, migrations
│   │   ├── projects.go             # Project queries
│   │   ├── instances.go            # Instance queries
│   │   ├── registries.go           # Registry queries
│   │   ├── settings.go             # Settings queries
│   │   ├── deploys.go              # Deploy history queries
│   │   └── store_test.go
│   └── crypto/
│       └── crypto.go               # AES-256 encrypt/decrypt for credentials
├── web/                            # SvelteKit frontend
│   ├── src/
│   │   ├── routes/
│   │   │   ├── +page.svelte        # Dashboard
│   │   │   ├── projects/
│   │   │   │   ├── +page.svelte    # Projects list + add
│   │   │   │   └── [id]/
│   │   │   │       └── +page.svelte # Project detail + instances
│   │   │   ├── deploy/
│   │   │   │   └── +page.svelte    # Quick deploy
│   │   │   └── settings/
│   │   │       ├── +page.svelte    # Global settings
│   │   │       ├── registries/
│   │   │       │   └── +page.svelte
│   │   │       └── credentials/
│   │   │           └── +page.svelte
│   │   ├── lib/
│   │   │   ├── api.ts              # API client
│   │   │   ├── types.ts            # Shared types
│   │   │   └── components/         # Reusable UI components
│   │   └── app.html
│   ├── package.json
│   ├── svelte.config.js
│   └── vite.config.ts
├── docker-watcher.example.yaml     # Example seed config
├── Dockerfile
├── docker-compose.yml
├── go.mod
└── go.sum

Implementation Phases

Phase 1: Foundation ✅

Core infrastructure — store, config import, Docker client, NPM client.

1. Go project init — go.mod, directory structure, dependencies
2. SQLite store — schema, migrations, CRUD for projects/registries/settings/instances/deploys
3. Crypto — AES-256 encrypt/decrypt for credential storage
4. Config seed loader — parse YAML, import into SQLite on first launch
5. Docker client — connect to socket, pull image, inspect image, list/start/stop/remove containers, manage networks
6. NPM client — authenticate (JWT), create/update/delete proxy hosts, list existing hosts

Phase 2: Detection & Deployment (Registry & Poller ✅, Webhook ✅, Deployer ✅)

The core loop — detecting new images and deploying them.

8. Registry client ✅ — Gitea registry API: list tags for an image, detect new tags
9. Poller ✅ — periodic check for new tags matching configured patterns
10. Secret webhook handler ✅ — UUID-based URL, receives image push notifications, auto-creates unknown projects
11. Deployer ✅ — orchestrate: pull → start container → NPM proxy → health check
12. Multi-instance support ✅ — multiple versions per project/stage, tag-based subdomains, max_instances limit
13. Health checker ✅ — HTTP GET with retries and timeout (3 retries, 5s interval, 10s timeout)
14. Rollback ✅ — on health check failure: remove new container, clean up NPM, alert
15. Notifications ✅ — send webhook on deploy success/failure (fire-and-forget)

Phase 3: Web UI

Full dashboard for visibility, manual control, and configuration.

16. API layer — REST endpoints for all CRUD operations + deploy/control actions
17. SvelteKit dashboard — project overview, instance status, quick status indicators
18. Project detail view — stages, instances, controls (stop/start/restart/remove), deploy history
19. Quick Deploy page — paste image URL, auto-inspect, pre-fill form, one-click deploy
20. Settings pages — registries, credentials, global settings, webhook URL management
21. Project config pages — add/edit/delete projects and stages via UI
22. Embed in Go ✅ — build SvelteKit to static, embed with go:embed, serve from Go
23. Real-time updates ✅ — SSE for deploy progress and instance status changes

Phase 4: Volumes & Environment (Phase 13) -- COMPLETED

Persistent storage and app-specific configuration for deployed containers.

24. Environment variables per project — key/value pairs stored in SQLite, sensitive values encrypted
25. Per-stage env overrides — e.g., NODE_ENV=development for dev, NODE_ENV=production for prod
26. Volume mounts per project — configurable source/target paths with shared/isolated modes
27. Shared volumes — all instances of a project mount the same host path (for stateless apps or shared uploads)
28. Isolated volumes — each instance gets its own subdirectory: {source}/{stage}-{tag}/ → {target} (for stateful apps with local DBs/files)
29. UI for volumes & env — project settings page with key/value editor, volume list, shared/isolated toggle, per-stage override support

Phase 13 Handoff Notes

• New tables: stage_env (id, stage_id, key, value, encrypted, timestamps), volumes (id, project_id, source, target, mode, timestamps)
• stage_env has UNIQUE(stage_id, key) constraint to prevent duplicate keys per stage
• Volume mode is either "shared" or "isolated"; default is "shared"
• Encrypted env values are encrypted with crypto.Encrypt before storage and decrypted at deploy time
• API masks encrypted env values as "••••••••" in responses
• Env merge order in deployer: project-level JSON env field parsed first, then stage-level stage_env records overlay (stage wins on key conflict)
• computeVolumeMounts appends /{stage}-{tag}/ to source for isolated volumes
• Docker ContainerConfig now has Mounts []mount.Mount field, passed to HostConfig.Mounts
• Both executeDeploy and blueGreenDeploy updated to use mergeEnvVars and computeVolumeMounts
• API routes: GET/POST /api/projects/{id}/stages/{stage}/env, PUT/DELETE .../env/{envId}, GET/POST /api/projects/{id}/volumes, PUT/DELETE .../volumes/{volId}
• Frontend pages: /projects/[id]/env (per-stage env editor with inherited/overridden indicators), /projects/[id]/volumes (volume editor with shared/isolated toggle)
• Project detail page now has navigation links to env and volumes pages

Volume config per project:

env:
  NODE_ENV: production
  DATABASE_URL: postgres://db:5432/myapp    # shared external DB
  SECRET_KEY: "..."                          # encrypted in SQLite
volumes:
  - source: /data/my-app/uploads
    target: /app/uploads
    mode: shared        # all instances share this path
  - source: /data/my-app/data
    target: /app/data
    mode: isolated      # auto-appends /{stage}-{tag}/ to source

Stage-level env overrides:

stages:
  dev:
    env:
      NODE_ENV: development       # overrides project-level
      DATABASE_URL: postgres://db:5432/myapp_dev
  prod:
    env:
      NODE_ENV: production        # uses project-level default

Phase 5: Hardening (Phase 12) -- COMPLETED

30. Blue-green deploys -- start new, health check, swap, stop old (zero downtime)
31. Promote flow -- enforce promote_from for production deploys
32. Auth on dashboard -- two modes, configurable via settings:
    • Local auth -- username/password stored in SQLite (bcrypt hashed), JWT session tokens
    • OAuth2 / OpenID Connect -- integration with any OIDC provider (configurable client ID/secret/discovery URL)
33. Graceful shutdown -- drain in-progress deploys on SIGTERM, close DB, stop poller
34. Structured logging -- JSON logs via log/slog with deploy context
35. Config export -- download current SQLite state as YAML
36. Dockerfile -- multi-stage build (Node.js 20 + Go 1.23 build, alpine runtime)
37. docker-compose.yml -- production-ready compose with volumes, network, env
38. Auth middleware -- protects all /api/* routes except webhook and auth endpoints
39. Auth settings UI -- settings page to toggle auth mode, configure OIDC, manage users
40. Login page -- username/password form with OIDC SSO option
41. Final wiring -- all services properly initialized and shut down in main.go

Phase 12 Handoff Notes

• Auth: auth.LocalAuth handles JWT generation/validation, auth.OIDCProvider handles OIDC flow
• Default admin user created on first launch (ADMIN_PASSWORD env var, default: "admin")
• JWT secret derived from ENCRYPTION_KEY via HMAC-SHA256
• Blue-green: triggered automatically when stage has max_instances=1; otherwise standard deploy
• Promote: validated in TriggerDeploy before deploy begins
• Graceful shutdown: deployer.Drain() waits for in-progress deploys; poller stopped; HTTP server drained; DB closed
• Structured logging: all API, deployer, and main.go use log/slog JSON handler
• New dependencies: github.com/golang-jwt/jwt/v5, golang.org/x/crypto/bcrypt, github.com/coreos/go-oidc/v3, golang.org/x/oauth2
• New tables: users (id, username, password_hash, email, role, timestamps), auth_settings (single-row: auth_mode, OIDC config)
• Auth middleware applied to all /api/* routes except /api/auth/login, /api/auth/oidc/*, /api/webhook/*, /api/config/export
• Frontend: token stored in localStorage, sent as Authorization: Bearer header
• Run go mod tidy after checkout to resolve transitive dependencies

Key Dependencies (Go)

• github.com/docker/docker — Docker Engine API
• github.com/go-chi/chi or net/http — HTTP routing
• gopkg.in/yaml.v3 — YAML seed config
• modernc.org/sqlite — SQLite (CGo-free)
• github.com/robfig/cron — Polling scheduler
• github.com/google/uuid — Webhook secret URL generation

Docker Compose (self-deployment)

services:
  docker-watcher:
    image: docker-watcher:latest
    container_name: docker-watcher
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./docker-watcher.yaml:/app/seed.yaml:ro  # optional, first launch only
      - ./data:/app/data                          # SQLite DB
    environment:
      - ENCRYPTION_KEY=${ENCRYPTION_KEY}           # protects all credentials in DB
    networks:
      - staging-net

networks:
  staging-net:
    external: true

API Endpoints

# Projects
GET    /api/projects                                    — list all projects with instance counts
POST   /api/projects                                    — create project
GET    /api/projects/:id                                — project detail + stages + instances
PUT    /api/projects/:id                                — update project config
DELETE /api/projects/:id                                — delete project + all instances

# Stages
POST   /api/projects/:id/stages                         — add stage to project
PUT    /api/projects/:id/stages/:stage                   — update stage config
DELETE /api/projects/:id/stages/:stage                   — delete stage + its instances

# Stage Env Overrides
GET    /api/projects/:id/stages/:stage/env                 — list stage env vars (secrets masked)
POST   /api/projects/:id/stages/:stage/env                 — create stage env var
PUT    /api/projects/:id/stages/:stage/env/:envId          — update stage env var
DELETE /api/projects/:id/stages/:stage/env/:envId          — delete stage env var

# Project Volumes
GET    /api/projects/:id/volumes                           — list project volumes
POST   /api/projects/:id/volumes                           — create project volume
PUT    /api/projects/:id/volumes/:volId                    — update project volume
DELETE /api/projects/:id/volumes/:volId                    — delete project volume

# Instances (running containers)
GET    /api/projects/:id/stages/:stage/instances          — list instances for stage
POST   /api/projects/:id/stages/:stage/instances          — deploy new instance (pick tag)
DELETE /api/projects/:id/stages/:stage/instances/:iid     — remove instance (container + NPM proxy)
POST   /api/projects/:id/stages/:stage/instances/:iid/stop    — stop container
POST   /api/projects/:id/stages/:stage/instances/:iid/start   — start stopped container
POST   /api/projects/:id/stages/:stage/instances/:iid/restart — restart container

# Quick Deploy
POST   /api/deploy/inspect                               — pull + inspect image, return defaults
POST   /api/deploy/quick                                 — create project + deploy in one step

# Registry
GET    /api/registries                                   — list registries
POST   /api/registries                                   — add registry
PUT    /api/registries/:id                               — update registry
DELETE /api/registries/:id                               — delete registry
POST   /api/registries/:id/test                          — test connection
GET    /api/registries/:id/tags/:image                   — list available tags

# Settings
GET    /api/settings                                     — get global settings
PUT    /api/settings                                     — update global settings
GET    /api/settings/webhook-url                          — get secret webhook URL
POST   /api/settings/webhook-url/regenerate               — regenerate webhook URL

# Deploy history
GET    /api/deploys                                      — recent deploys across all projects
GET    /api/deploys/:id/logs                             — deploy log stream (SSE)

# Webhook (secret URL — no auth needed)
POST   /api/webhook/:secret-uuid                         — receive image push notification

User Workflows

Auto-Deploy (zero effort)

Push code → CI builds → pushes tag → Docker Watcher detects →
auto_deploy: true → deployed → notification with URL

Manual Deploy via UI (one click)

Open dashboard → project → stage → "Deploy new version" →
pick tag from dropdown → click Deploy

Quick Deploy (new project, paste image URL)

Open dashboard → "Quick Deploy" → paste image URL →
review auto-filled defaults → click Deploy →
project auto-created + deployed

Deploy via CI Webhook (zero effort after CI setup)

# In .gitea/workflows/build.yml
- name: Notify Docker Watcher
  run: |
    curl -X POST https://watcher.dolgolyov-family.by/api/webhook/d8f2a1e9-... \
      -d '{"image": "git.dolgolyov-family.by/alexei/my-app:dev-${{ github.sha }}"}'

Known project → deploys per stage config. Unknown project → auto-creates with defaults from image inspection, deploys.

Production Deploy (two clicks)

Open dashboard → project → prod stage → "Deploy new version" →
dropdown shows only tags running in "rel" stage (promote_from) →
pick tag → confirmation dialog → Deploy