diff --git a/PLAN.md b/PLAN.md index af7d1bb..8660633 100644 --- a/PLAN.md +++ b/PLAN.md @@ -2,34 +2,39 @@ ## Overview -A self-hosted tool that automates Docker container deployment with Nginx Proxy Manager integration. Detects new images from Gitea/GitHub registries, deploys containers, and configures reverse proxy routing — all from a web dashboard. +A self-hosted tool that automates Docker container deployment with Nginx Proxy Manager integration. Detects new images from Gitea/GitHub registries, deploys containers, and configures reverse proxy routing — all from a web dashboard. Supports multiple simultaneous versions of the same project. DNS is handled by a Cloudflare wildcard record (`*.dolgolyov-family.by`) — no per-project DNS management needed. ## Architecture -``` +```text Gitea CI → pushes image → Registry + │ ↓ + │ Docker Watcher (Go) + │ ├── Secret webhook URL (instant) + │ └── Registry poller (fallback) + │ ↓ + └── or: POST /api/webhook/ + with {"image": "registry/org/app:tag"} ↓ - Docker Watcher (Go) - ├── Webhook listener (instant) - └── Registry poller (fallback) - ↓ - Match tag → project + stage - ↓ - ┌── auto_deploy? ──┐ - ↓ ↓ - Deploy now Notify, wait for - manual trigger (UI) - ↓ ↓ - Pull image - Stop old container - Start new container on shared network - ↓ - NPM API: create/update proxy host - → stage-dev-project.example.com - ↓ - Health check - → success: done, notify - → failure: rollback, alert + Known project? ──────────────────┐ + ↓ yes ↓ no + Match tag → stage Auto-create project + ↓ with defaults from + auto_deploy? image inspection + ↓ yes ↓ no (EXPOSE, labels) + Deploy now Notify, wait ↓ + ↓ for UI trigger Deploy with defaults + ↓ ↓ + Pull image + Start new container on shared network + (old container stays if multi-instance) + ↓ + NPM API: create proxy host (if first deploy for this subdomain) + (DNS already handled by Cloudflare wildcard *.domain) + ↓ + Health check + → success: done, notify + → failure: remove new container, alert ``` ## Decisions @@ -39,79 +44,138 @@ Gitea CI → pushes image → Registry | Language | Go | Single binary, excellent Docker SDK, low resource usage | | Web UI | SvelteKit (embedded in Go binary) | User's existing stack, lightweight | | Reverse proxy | Nginx Proxy Manager | Already deployed, API available | +| DNS | Cloudflare wildcard `*.{domain}` | One-time setup, all subdomains auto-resolve | | Routing | Subdomain-based | No sub-path issues with SPAs | -| Image detection | Webhook + polling | Webhook for speed, polling as fallback | -| Config | YAML | Version-controlled, easy to edit | +| Image detection | Secret webhook URL + polling | Webhook for speed, polling as fallback | +| Config storage | SQLite (YAML for initial seed only) | Editable via UI, no manual file editing | +| Credentials | Encrypted in SQLite (AES-256) | Single ENCRYPTION_KEY env var | +| Webhook auth | Secret UUID in URL | No tokens needed, simple CI integration | +| Multi-instance | Yes | Multiple tags of same project can run simultaneously | | Deployment target | Same TrueNAS host | Docker socket mounted | ## Subdomain Convention -| Stage | Pattern | Example | -|-------|---------|---------| -| Dev | `stage-dev-{project}.{domain}` | `stage-dev-web-app-launcher.dolgolyov-family.by` | -| Release | `stage-rel-{project}.{domain}` | `stage-rel-web-app-launcher.dolgolyov-family.by` | +| Type | Pattern | Example | +|------|---------|---------| +| Dev (default) | `stage-dev-{project}.{domain}` | `stage-dev-web-app-launcher.dolgolyov-family.by` | +| Dev (specific tag) | `stage-dev-{project}-{tag}.{domain}` | `stage-dev-web-app-launcher-abc123.dolgolyov-family.by` | +| Release (default) | `stage-rel-{project}.{domain}` | `stage-rel-web-app-launcher.dolgolyov-family.by` | +| Release (specific tag) | `stage-rel-{project}-{tag}.{domain}` | `stage-rel-web-app-launcher-v1-2-0.dolgolyov-family.by` | | Production | `{custom}.{domain}` | `launcher.dolgolyov-family.by` | -## Config Format +Tags are sanitized for DNS: dots → dashes, lowercase, truncated to fit DNS limits. + +## Configuration + +### First Launch + +```text +YAML seed file exists? → import into SQLite → done +No YAML? → empty state, configure everything via UI +``` + +After import, all configuration lives in SQLite and is managed via the Web UI. +YAML is never read again unless user clicks "Re-import config" or "Export config". + +### Seed Config Format (optional) ```yaml global: domain: dolgolyov-family.by + server_ip: 93.84.96.191 network: staging-net + subdomain_pattern: "stage-{stage}-{project}" + notification_url: https://notify.dolgolyov-family.by/webhook npm: url: http://npm:81 email: docker-watcher@dolgolyov-family.by - password_env: NPM_PASSWORD - subdomain_pattern: "stage-{stage}-{project}" - notification_url: https://notify.dolgolyov-family.by/webhook - + password: "npm-password-here" registries: gitea: url: https://git.dolgolyov-family.by type: gitea - token_env: GITEA_TOKEN - # github: - # url: https://ghcr.io - # type: github - # token_env: GITHUB_TOKEN + token: "gitea-token-here" projects: web-app-launcher: registry: gitea - image: gitea.dolgolyov-family.by/alexei/web-app-launcher + image: git.dolgolyov-family.by/alexei/web-app-launcher port: 3000 healthcheck: /api/health - env_file: ./envs/web-app-launcher.env - volumes: [] + env: + NODE_ENV: production stages: dev: tag_pattern: "dev-*" auto_deploy: true + max_instances: 5 rel: tag_pattern: "v*" auto_deploy: false + max_instances: 2 prod: tag_pattern: "v*" auto_deploy: false confirm: true promote_from: rel - subdomain: launcher # overrides pattern → launcher.dolgolyov-family.by + max_instances: 2 + subdomain: launcher ``` +## Web UI Sections + +### Dashboard + +Overview of all projects with their running instances: +- Project name, running instance count, latest activity +- Quick status indicators (healthy / stopped / failing) +- "Quick Deploy" button for ad-hoc image deployment + +### Project Detail + +Per-project view with stages and instances: +- Each stage shows all running instances with: tag, status, URL, uptime +- Controls per instance: Stop, Start, Restart, Remove +- "Deploy new version" dropdown — lists available tags from registry +- Deploy history log + +### Quick Deploy + +For deploying images not yet configured as projects: +1. Paste image URL (e.g., `git.dolgolyov-family.by/alexei/my-app:dev-abc123`) +2. Docker Watcher pulls and inspects image (EXPOSE port, HEALTHCHECK, labels) +3. Pre-fills form with sensible defaults (project name, port, stage, subdomain) +4. User reviews, tweaks, clicks "Deploy" +5. Project is auto-created in the DB for future use + +### Settings + +- **Registries** — add/edit/delete registries, test connection +- **Credentials** — NPM, registry tokens (encrypted, shown as `••••••••`) +- **Global** — domain, server IP, Docker network, subdomain pattern, polling interval +- **Notifications** — webhook URL +- **Webhook URL** — shows the secret deploy URL, "Regenerate" button + +### Projects Config + +- Add / edit / delete projects via UI +- Configure image, port, healthcheck, env vars, volumes per project +- Add / remove stages, set tag patterns, auto-deploy, subdomain overrides, max instances + ## Project Structure -``` +```text docker-watcher/ ├── cmd/ │ └── server/ │ └── main.go # Entry point ├── internal/ │ ├── config/ -│ │ ├── config.go # YAML parsing, validation +│ │ ├── config.go # YAML seed parsing │ │ └── config_test.go │ ├── docker/ │ │ ├── client.go # Docker Engine API wrapper -│ │ ├── container.go # Create, start, stop, remove +│ │ ├── container.go # Create, start, stop, remove, inspect │ │ └── client_test.go │ ├── npm/ │ │ ├── client.go # NPM API client (auth, CRUD proxy hosts) @@ -133,33 +197,51 @@ docker-watcher/ │ │ ├── notifier.go # Webhook notifications │ │ └── notifier_test.go │ ├── webhook/ -│ │ ├── handler.go # Receive push events from Gitea/GitHub +│ │ ├── handler.go # Secret URL webhook receiver │ │ └── handler_test.go │ ├── api/ │ │ ├── router.go # HTTP API for web UI -│ │ ├── projects.go # Project endpoints -│ │ ├── deploys.go # Deploy endpoints +│ │ ├── projects.go # Project CRUD endpoints +│ │ ├── registries.go # Registry CRUD endpoints +│ │ ├── settings.go # Global settings endpoints +│ │ ├── instances.go # Instance start/stop/restart/remove +│ │ ├── deploys.go # Deploy + quick deploy endpoints │ │ └── middleware.go # Auth, logging, CORS -│ └── store/ -│ ├── store.go # Deploy history, state (SQLite) -│ └── store_test.go +│ ├── store/ +│ │ ├── store.go # SQLite schema, migrations +│ │ ├── projects.go # Project queries +│ │ ├── instances.go # Instance queries +│ │ ├── registries.go # Registry queries +│ │ ├── settings.go # Settings queries +│ │ ├── deploys.go # Deploy history queries +│ │ └── store_test.go +│ └── crypto/ +│ └── crypto.go # AES-256 encrypt/decrypt for credentials ├── web/ # SvelteKit frontend │ ├── src/ │ │ ├── routes/ │ │ │ ├── +page.svelte # Dashboard │ │ │ ├── projects/ +│ │ │ │ ├── +page.svelte # Projects list + add │ │ │ │ └── [id]/ -│ │ │ │ └── +page.svelte # Project detail + deploy controls +│ │ │ │ └── +page.svelte # Project detail + instances +│ │ │ ├── deploy/ +│ │ │ │ └── +page.svelte # Quick deploy │ │ │ └── settings/ -│ │ │ └── +page.svelte # Config view +│ │ │ ├── +page.svelte # Global settings +│ │ │ ├── registries/ +│ │ │ │ └── +page.svelte +│ │ │ └── credentials/ +│ │ │ └── +page.svelte │ │ ├── lib/ │ │ │ ├── api.ts # API client -│ │ │ └── types.ts # Shared types +│ │ │ ├── types.ts # Shared types +│ │ │ └── components/ # Reusable UI components │ │ └── app.html │ ├── package.json │ ├── svelte.config.js │ └── vite.config.ts -├── docker-watcher.example.yaml +├── docker-watcher.example.yaml # Example seed config ├── Dockerfile ├── docker-compose.yml ├── go.mod @@ -170,50 +252,58 @@ docker-watcher/ ### Phase 1: Foundation -Core infrastructure — config, Docker client, NPM client. +Core infrastructure — store, config import, Docker client, NPM client. 1. **Go project init** — go.mod, directory structure, dependencies -2. **Config loader** — parse YAML, validate, resolve env vars -3. **Docker client** — connect to socket, pull image, list/start/stop/remove containers, manage networks -4. **NPM client** — authenticate (JWT), create/update/delete proxy hosts, list existing hosts -5. **Store** — SQLite for deploy history and state tracking +2. **SQLite store** — schema, migrations, CRUD for projects/registries/settings/instances/deploys +3. **Crypto** — AES-256 encrypt/decrypt for credential storage +4. **Config seed loader** — parse YAML, import into SQLite on first launch +5. **Docker client** — connect to socket, pull image, inspect image, list/start/stop/remove containers, manage networks +6. **NPM client** — authenticate (JWT), create/update/delete proxy hosts, list existing hosts ### Phase 2: Detection & Deployment The core loop — detecting new images and deploying them. -6. **Registry client** — Gitea registry API: list tags, compare with last known -7. **Poller** — periodic check for new tags matching configured patterns -8. **Webhook handler** — HTTP endpoint receiving push events from Gitea CI -9. **Deployer** — orchestrate: pull → stop old → start new → NPM update → health check -10. **Health checker** — HTTP GET with retries and timeout -11. **Rollback** — on health check failure: stop new, restart old, revert NPM -12. **Notifications** — send webhook on deploy success/failure +8. **Registry client** — Gitea registry API: list tags for an image, detect new tags +9. **Poller** — periodic check for new tags matching configured patterns +10. **Secret webhook handler** — UUID-based URL, receives image push notifications, auto-creates unknown projects +11. **Deployer** — orchestrate: pull → start container → NPM proxy → health check +12. **Multi-instance support** — multiple versions per project/stage, tag-based subdomains, max_instances limit +13. **Health checker** — HTTP GET with retries and timeout +14. **Rollback** — on health check failure: remove new container, clean up NPM, alert +15. **Notifications** — send webhook on deploy success/failure ### Phase 3: Web UI -Dashboard for visibility and manual control. +Full dashboard for visibility, manual control, and configuration. -13. **API layer** — REST endpoints for projects, stages, deploys, manual trigger -14. **SvelteKit app** — dashboard, project detail, deploy log, manual deploy button -15. **Embed in Go** — build SvelteKit to static, embed with `go:embed`, serve from Go -16. **Real-time updates** — SSE or WebSocket for deploy progress +16. **API layer** — REST endpoints for all CRUD operations + deploy/control actions +17. **SvelteKit dashboard** — project overview, instance status, quick status indicators +18. **Project detail view** — stages, instances, controls (stop/start/restart/remove), deploy history +19. **Quick Deploy page** — paste image URL, auto-inspect, pre-fill form, one-click deploy +20. **Settings pages** — registries, credentials, global settings, webhook URL management +21. **Project config pages** — add/edit/delete projects and stages via UI +22. **Embed in Go** — build SvelteKit to static, embed with `go:embed`, serve from Go +23. **Real-time updates** — SSE for deploy progress and instance status changes ### Phase 4: Hardening -17. **Blue-green deploys** — start new, health check, swap, stop old (zero downtime) -18. **Promote flow** — enforce `promote_from` for production deploys -19. **Auth on dashboard** — basic auth or token-based -20. **Graceful shutdown** — drain in-progress deploys on SIGTERM -21. **Structured logging** — JSON logs with deploy context +24. **Blue-green deploys** — start new, health check, swap, stop old (zero downtime) +25. **Promote flow** — enforce `promote_from` for production deploys +26. **Auth on dashboard** — basic auth or token-based +27. **Graceful shutdown** — drain in-progress deploys on SIGTERM +28. **Structured logging** — JSON logs with deploy context +29. **Config export** — download current SQLite state as YAML ## Key Dependencies (Go) - `github.com/docker/docker` — Docker Engine API - `github.com/go-chi/chi` or `net/http` — HTTP routing -- `gopkg.in/yaml.v3` — YAML config -- `github.com/mattn/go-sqlite3` or `modernc.org/sqlite` — SQLite (CGo-free) +- `gopkg.in/yaml.v3` — YAML seed config +- `modernc.org/sqlite` — SQLite (CGo-free) - `github.com/robfig/cron` — Polling scheduler +- `github.com/google/uuid` — Webhook secret URL generation ## Docker Compose (self-deployment) @@ -227,12 +317,10 @@ services: - "8080:8080" volumes: - /var/run/docker.sock:/var/run/docker.sock - - ./docker-watcher.yaml:/app/config.yaml:ro - - ./envs:/app/envs:ro - - ./data:/app/data # SQLite DB + - ./docker-watcher.yaml:/app/seed.yaml:ro # optional, first launch only + - ./data:/app/data # SQLite DB environment: - - NPM_PASSWORD=${NPM_PASSWORD} - - GITEA_TOKEN=${GITEA_TOKEN} + - ENCRYPTION_KEY=${ENCRYPTION_KEY} # protects all credentials in DB networks: - staging-net @@ -241,16 +329,96 @@ networks: external: true ``` -## API Endpoints (Phase 3) +## API Endpoints +```text +# Projects +GET /api/projects — list all projects with instance counts +POST /api/projects — create project +GET /api/projects/:id — project detail + stages + instances +PUT /api/projects/:id — update project config +DELETE /api/projects/:id — delete project + all instances + +# Stages +POST /api/projects/:id/stages — add stage to project +PUT /api/projects/:id/stages/:stage — update stage config +DELETE /api/projects/:id/stages/:stage — delete stage + its instances + +# Instances (running containers) +GET /api/projects/:id/stages/:stage/instances — list instances for stage +POST /api/projects/:id/stages/:stage/instances — deploy new instance (pick tag) +DELETE /api/projects/:id/stages/:stage/instances/:iid — remove instance (container + NPM proxy) +POST /api/projects/:id/stages/:stage/instances/:iid/stop — stop container +POST /api/projects/:id/stages/:stage/instances/:iid/start — start stopped container +POST /api/projects/:id/stages/:stage/instances/:iid/restart — restart container + +# Quick Deploy +POST /api/deploy/inspect — pull + inspect image, return defaults +POST /api/deploy/quick — create project + deploy in one step + +# Registry +GET /api/registries — list registries +POST /api/registries — add registry +PUT /api/registries/:id — update registry +DELETE /api/registries/:id — delete registry +POST /api/registries/:id/test — test connection +GET /api/registries/:id/tags/:image — list available tags + +# Settings +GET /api/settings — get global settings +PUT /api/settings — update global settings +GET /api/settings/webhook-url — get secret webhook URL +POST /api/settings/webhook-url/regenerate — regenerate webhook URL + +# Deploy history +GET /api/deploys — recent deploys across all projects +GET /api/deploys/:id/logs — deploy log stream (SSE) + +# Webhook (secret URL — no auth needed) +POST /api/webhook/:secret-uuid — receive image push notification ``` -GET /api/projects — list all projects with current status -GET /api/projects/:id — project detail + stage statuses -GET /api/projects/:id/stages/:stage — stage detail + deploy history -POST /api/projects/:id/stages/:stage/deploy — trigger manual deploy -POST /api/projects/:id/stages/:stage/rollback — rollback to previous -GET /api/deploys — recent deploys across all projects -GET /api/deploys/:id/logs — deploy log stream (SSE) -POST /api/webhook/gitea — Gitea push event receiver -POST /api/webhook/github — GitHub push event receiver (future) + +## User Workflows + +### Auto-Deploy (zero effort) + +```text +Push code → CI builds → pushes tag → Docker Watcher detects → +auto_deploy: true → deployed → notification with URL +``` + +### Manual Deploy via UI (one click) + +```text +Open dashboard → project → stage → "Deploy new version" → +pick tag from dropdown → click Deploy +``` + +### Quick Deploy (new project, paste image URL) + +```text +Open dashboard → "Quick Deploy" → paste image URL → +review auto-filled defaults → click Deploy → +project auto-created + deployed +``` + +### Deploy via CI Webhook (zero effort after CI setup) + +```text +# In .gitea/workflows/build.yml +- name: Notify Docker Watcher + run: | + curl -X POST https://watcher.dolgolyov-family.by/api/webhook/d8f2a1e9-... \ + -d '{"image": "git.dolgolyov-family.by/alexei/my-app:dev-${{ github.sha }}"}' +``` + +Known project → deploys per stage config. +Unknown project → auto-creates with defaults from image inspection, deploys. + +### Production Deploy (two clicks) + +```text +Open dashboard → project → prod stage → "Deploy new version" → +dropdown shows only tags running in "rel" stage (promote_from) → +pick tag → confirmation dialog → Deploy ```