alexei.dolgolyov/tiny-forge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: revise implementation plan

Browse Source 
Add multi-instance support, UI-managed config (SQLite), secret webhook
URLs, quick deploy flow, and Cloudflare wildcard DNS strategy. Remove
per-project DNS automation in favor of wildcard record.
This commit is contained in:
Admin
2026-03-27 20:38:51 +03:00
parent ae6735b05c
commit 17206933f3
1 changed files with 263 additions and 95 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PLAN.md
+263 -95
View File
@@ -2,34 +2,39 @@
## Overview ## Overview
A self-hosted tool that automates Docker container deployment with Nginx Proxy Manager integration. Detects new images from Gitea/GitHub registries, deploys containers, and configures reverse proxy routing — all from a web dashboard. A self-hosted tool that automates Docker container deployment with Nginx Proxy Manager integration. Detects new images from Gitea/GitHub registries, deploys containers, and configures reverse proxy routing — all from a web dashboard. Supports multiple simultaneous versions of the same project. DNS is handled by a Cloudflare wildcard record (`*.dolgolyov-family.by`) — no per-project DNS management needed.
## Architecture ## Architecture
``` ```text
Gitea CI → pushes image → Registry Gitea CI → pushes image → Registry
│ ↓
│ Docker Watcher (Go)
│ ├── Secret webhook URL (instant)
│ └── Registry poller (fallback)
│ ↓
└── or: POST /api/webhook/<secret-uuid>
with {"image": "registry/org/app:tag"}
↓ ↓
Docker Watcher (Go) Known project? ──────────────────┐
├── Webhook listener (instant) ↓ yes ↓ no
└── Registry poller (fallback) Match tag → stage Auto-create project
↓ ↓ with defaults from
Match tag → project + stage auto_deploy? image inspection
↓ ↓ yes ↓ no (EXPOSE, labels)
┌── auto_deploy? ──┐ Deploy now Notify, wait ↓
↓ ↓ ↓ for UI trigger Deploy with defaults
Deploy now Notify, wait for ↓ ↓
manual trigger (UI) Pull image
↓ ↓ Start new container on shared network
Pull image (old container stays if multi-instance)
Stop old container ↓
Start new container on shared network NPM API: create proxy host (if first deploy for this subdomain)
↓ (DNS already handled by Cloudflare wildcard *.domain)
NPM API: create/update proxy host ↓
→ stage-dev-project.example.com Health check
↓ → success: done, notify
Health check → failure: remove new container, alert
→ success: done, notify
→ failure: rollback, alert
``` ```
## Decisions ## Decisions
@@ -39,79 +44,138 @@ Gitea CI → pushes image → Registry
| Language | Go | Single binary, excellent Docker SDK, low resource usage | | Language | Go | Single binary, excellent Docker SDK, low resource usage |
| Web UI | SvelteKit (embedded in Go binary) | User's existing stack, lightweight | | Web UI | SvelteKit (embedded in Go binary) | User's existing stack, lightweight |
| Reverse proxy | Nginx Proxy Manager | Already deployed, API available | | Reverse proxy | Nginx Proxy Manager | Already deployed, API available |
| DNS | Cloudflare wildcard `*.{domain}` | One-time setup, all subdomains auto-resolve |
| Routing | Subdomain-based | No sub-path issues with SPAs | | Routing | Subdomain-based | No sub-path issues with SPAs |
| Image detection | Webhook + polling | Webhook for speed, polling as fallback | | Image detection | Secret webhook URL + polling | Webhook for speed, polling as fallback |
| Config | YAML | Version-controlled, easy to edit | | Config storage | SQLite (YAML for initial seed only) | Editable via UI, no manual file editing |
| Credentials | Encrypted in SQLite (AES-256) | Single ENCRYPTION_KEY env var |
| Webhook auth | Secret UUID in URL | No tokens needed, simple CI integration |
| Multi-instance | Yes | Multiple tags of same project can run simultaneously |
| Deployment target | Same TrueNAS host | Docker socket mounted | | Deployment target | Same TrueNAS host | Docker socket mounted |
## Subdomain Convention ## Subdomain Convention
| Stage | Pattern | Example | | Type | Pattern | Example |
|-------|---------|---------| |------|---------|---------|
| Dev | `stage-dev-{project}.{domain}` | `stage-dev-web-app-launcher.dolgolyov-family.by` | | Dev (default) | `stage-dev-{project}.{domain}` | `stage-dev-web-app-launcher.dolgolyov-family.by` |
| Release | `stage-rel-{project}.{domain}` | `stage-rel-web-app-launcher.dolgolyov-family.by` | | Dev (specific tag) | `stage-dev-{project}-{tag}.{domain}` | `stage-dev-web-app-launcher-abc123.dolgolyov-family.by` |
| Release (default) | `stage-rel-{project}.{domain}` | `stage-rel-web-app-launcher.dolgolyov-family.by` |
| Release (specific tag) | `stage-rel-{project}-{tag}.{domain}` | `stage-rel-web-app-launcher-v1-2-0.dolgolyov-family.by` |
| Production | `{custom}.{domain}` | `launcher.dolgolyov-family.by` | | Production | `{custom}.{domain}` | `launcher.dolgolyov-family.by` |
## Config Format Tags are sanitized for DNS: dots → dashes, lowercase, truncated to fit DNS limits.
## Configuration
### First Launch
```text
YAML seed file exists? → import into SQLite → done
No YAML? → empty state, configure everything via UI
```
After import, all configuration lives in SQLite and is managed via the Web UI.
YAML is never read again unless user clicks "Re-import config" or "Export config".
### Seed Config Format (optional)
```yaml ```yaml
global: global:
domain: dolgolyov-family.by domain: dolgolyov-family.by
server_ip: 93.84.96.191
network: staging-net network: staging-net
subdomain_pattern: "stage-{stage}-{project}"
notification_url: https://notify.dolgolyov-family.by/webhook
npm: npm:
url: http://npm:81 url: http://npm:81
email: docker-watcher@dolgolyov-family.by email: docker-watcher@dolgolyov-family.by
password_env: NPM_PASSWORD password: "npm-password-here"
subdomain_pattern: "stage-{stage}-{project}"
notification_url: https://notify.dolgolyov-family.by/webhook
registries: registries:
gitea: gitea:
url: https://git.dolgolyov-family.by url: https://git.dolgolyov-family.by
type: gitea type: gitea
token_env: GITEA_TOKEN token: "gitea-token-here"
# github:
# url: https://ghcr.io
# type: github
# token_env: GITHUB_TOKEN
projects: projects:
web-app-launcher: web-app-launcher:
registry: gitea registry: gitea
image: gitea.dolgolyov-family.by/alexei/web-app-launcher image: git.dolgolyov-family.by/alexei/web-app-launcher
port: 3000 port: 3000
healthcheck: /api/health healthcheck: /api/health
env_file: ./envs/web-app-launcher.env env:
volumes: [] NODE_ENV: production
stages: stages:
dev: dev:
tag_pattern: "dev-*" tag_pattern: "dev-*"
auto_deploy: true auto_deploy: true
max_instances: 5
rel: rel:
tag_pattern: "v*" tag_pattern: "v*"
auto_deploy: false auto_deploy: false
max_instances: 2
prod: prod:
tag_pattern: "v*" tag_pattern: "v*"
auto_deploy: false auto_deploy: false
confirm: true confirm: true
promote_from: rel promote_from: rel
subdomain: launcher # overrides pattern → launcher.dolgolyov-family.by max_instances: 2
subdomain: launcher
``` ```
## Web UI Sections
### Dashboard
Overview of all projects with their running instances:
- Project name, running instance count, latest activity
- Quick status indicators (healthy / stopped / failing)
- "Quick Deploy" button for ad-hoc image deployment
### Project Detail
Per-project view with stages and instances:
- Each stage shows all running instances with: tag, status, URL, uptime
- Controls per instance: Stop, Start, Restart, Remove
- "Deploy new version" dropdown — lists available tags from registry
- Deploy history log
### Quick Deploy
For deploying images not yet configured as projects:
1. Paste image URL (e.g., `git.dolgolyov-family.by/alexei/my-app:dev-abc123`)
2. Docker Watcher pulls and inspects image (EXPOSE port, HEALTHCHECK, labels)
3. Pre-fills form with sensible defaults (project name, port, stage, subdomain)
4. User reviews, tweaks, clicks "Deploy"
5. Project is auto-created in the DB for future use
### Settings
- **Registries** — add/edit/delete registries, test connection
- **Credentials** — NPM, registry tokens (encrypted, shown as `••••••••`)
- **Global** — domain, server IP, Docker network, subdomain pattern, polling interval
- **Notifications** — webhook URL
- **Webhook URL** — shows the secret deploy URL, "Regenerate" button
### Projects Config
- Add / edit / delete projects via UI
- Configure image, port, healthcheck, env vars, volumes per project
- Add / remove stages, set tag patterns, auto-deploy, subdomain overrides, max instances
## Project Structure ## Project Structure
``` ```text
docker-watcher/ docker-watcher/
├── cmd/ ├── cmd/
│ └── server/ │ └── server/
│ └── main.go # Entry point │ └── main.go # Entry point
├── internal/ ├── internal/
│ ├── config/ │ ├── config/
│ │ ├── config.go # YAML parsing, validation │ │ ├── config.go # YAML seed parsing
│ │ └── config_test.go │ │ └── config_test.go
│ ├── docker/ │ ├── docker/
│ │ ├── client.go # Docker Engine API wrapper │ │ ├── client.go # Docker Engine API wrapper
│ │ ├── container.go # Create, start, stop, remove │ │ ├── container.go # Create, start, stop, remove, inspect
│ │ └── client_test.go │ │ └── client_test.go
│ ├── npm/ │ ├── npm/
│ │ ├── client.go # NPM API client (auth, CRUD proxy hosts) │ │ ├── client.go # NPM API client (auth, CRUD proxy hosts)
@@ -133,33 +197,51 @@ docker-watcher/
│ │ ├── notifier.go # Webhook notifications │ │ ├── notifier.go # Webhook notifications
│ │ └── notifier_test.go │ │ └── notifier_test.go
│ ├── webhook/ │ ├── webhook/
│ │ ├── handler.go # Receive push events from Gitea/GitHub │ │ ├── handler.go # Secret URL webhook receiver
│ │ └── handler_test.go │ │ └── handler_test.go
│ ├── api/ │ ├── api/
│ │ ├── router.go # HTTP API for web UI │ │ ├── router.go # HTTP API for web UI
│ │ ├── projects.go # Project endpoints │ │ ├── projects.go # Project CRUD endpoints
│ │ ├── deploys.go # Deploy endpoints │ │ ├── registries.go # Registry CRUD endpoints
│ │ ├── settings.go # Global settings endpoints
│ │ ├── instances.go # Instance start/stop/restart/remove
│ │ ├── deploys.go # Deploy + quick deploy endpoints
│ │ └── middleware.go # Auth, logging, CORS │ │ └── middleware.go # Auth, logging, CORS
│ └── store/ │ ├── store/
│ ├── store.go # Deploy history, state (SQLite) │ │ ├── store.go # SQLite schema, migrations
│ └── store_test.go │ │ ├── projects.go # Project queries
│ │ ├── instances.go # Instance queries
│ │ ├── registries.go # Registry queries
│ │ ├── settings.go # Settings queries
│ │ ├── deploys.go # Deploy history queries
│ │ └── store_test.go
│ └── crypto/
│ └── crypto.go # AES-256 encrypt/decrypt for credentials
├── web/ # SvelteKit frontend ├── web/ # SvelteKit frontend
│ ├── src/ │ ├── src/
│ │ ├── routes/ │ │ ├── routes/
│ │ │ ├── +page.svelte # Dashboard │ │ │ ├── +page.svelte # Dashboard
│ │ │ ├── projects/ │ │ │ ├── projects/
│ │ │ │ ├── +page.svelte # Projects list + add
│ │ │ │ └── [id]/ │ │ │ │ └── [id]/
│ │ │ │ └── +page.svelte # Project detail + deploy controls │ │ │ │ └── +page.svelte # Project detail + instances
│ │ │ ├── deploy/
│ │ │ │ └── +page.svelte # Quick deploy
│ │ │ └── settings/ │ │ │ └── settings/
│ │ │ └── +page.svelte # Config view │ │ │ ├── +page.svelte # Global settings
│ │ │ ├── registries/
│ │ │ │ └── +page.svelte
│ │ │ └── credentials/
│ │ │ └── +page.svelte
│ │ ├── lib/ │ │ ├── lib/
│ │ │ ├── api.ts # API client │ │ │ ├── api.ts # API client
│ │ │ └── types.ts # Shared types │ │ │ ├── types.ts # Shared types
│ │ │ └── components/ # Reusable UI components
│ │ └── app.html │ │ └── app.html
│ ├── package.json │ ├── package.json
│ ├── svelte.config.js │ ├── svelte.config.js
│ └── vite.config.ts │ └── vite.config.ts
├── docker-watcher.example.yaml ├── docker-watcher.example.yaml # Example seed config
├── Dockerfile ├── Dockerfile
├── docker-compose.yml ├── docker-compose.yml
├── go.mod ├── go.mod
@@ -170,50 +252,58 @@ docker-watcher/
### Phase 1: Foundation ### Phase 1: Foundation
Core infrastructure — config, Docker client, NPM client. Core infrastructure — store, config import, Docker client, NPM client.
1. **Go project init** — go.mod, directory structure, dependencies 1. **Go project init** — go.mod, directory structure, dependencies
2. **Config loader** — parse YAML, validate, resolve env vars 2. **SQLite store** — schema, migrations, CRUD for projects/registries/settings/instances/deploys
3. **Docker client** — connect to socket, pull image, list/start/stop/remove containers, manage networks 3. **Crypto** — AES-256 encrypt/decrypt for credential storage
4. **NPM client** — authenticate (JWT), create/update/delete proxy hosts, list existing hosts 4. **Config seed loader** — parse YAML, import into SQLite on first launch
5. **Store** — SQLite for deploy history and state tracking 5. **Docker client** — connect to socket, pull image, inspect image, list/start/stop/remove containers, manage networks
6. **NPM client** — authenticate (JWT), create/update/delete proxy hosts, list existing hosts
### Phase 2: Detection & Deployment ### Phase 2: Detection & Deployment
The core loop — detecting new images and deploying them. The core loop — detecting new images and deploying them.
6. **Registry client** — Gitea registry API: list tags, compare with last known 8. **Registry client** — Gitea registry API: list tags for an image, detect new tags
7. **Poller** — periodic check for new tags matching configured patterns 9. **Poller** — periodic check for new tags matching configured patterns
8. **Webhook handler** — HTTP endpoint receiving push events from Gitea CI 10. **Secret webhook handler** — UUID-based URL, receives image push notifications, auto-creates unknown projects
9. **Deployer** — orchestrate: pull → stop old → start new → NPM update → health check 11. **Deployer** — orchestrate: pull → start container → NPM proxy → health check
10. **Health checker** — HTTP GET with retries and timeout 12. **Multi-instance support** — multiple versions per project/stage, tag-based subdomains, max_instances limit
11. **Rollback** — on health check failure: stop new, restart old, revert NPM 13. **Health checker** — HTTP GET with retries and timeout
12. **Notifications** — send webhook on deploy success/failure 14. **Rollback** — on health check failure: remove new container, clean up NPM, alert
15. **Notifications** — send webhook on deploy success/failure
### Phase 3: Web UI ### Phase 3: Web UI
Dashboard for visibility and manual control. Full dashboard for visibility, manual control, and configuration.
13. **API layer** — REST endpoints for projects, stages, deploys, manual trigger 16. **API layer** — REST endpoints for all CRUD operations + deploy/control actions
14. **SvelteKit app** — dashboard, project detail, deploy log, manual deploy button 17. **SvelteKit dashboard** — project overview, instance status, quick status indicators
15. **Embed in Go** — build SvelteKit to static, embed with `go:embed`, serve from Go 18. **Project detail view** — stages, instances, controls (stop/start/restart/remove), deploy history
16. **Real-time updates** — SSE or WebSocket for deploy progress 19. **Quick Deploy page** — paste image URL, auto-inspect, pre-fill form, one-click deploy
20. **Settings pages** — registries, credentials, global settings, webhook URL management
21. **Project config pages** — add/edit/delete projects and stages via UI
22. **Embed in Go** — build SvelteKit to static, embed with `go:embed`, serve from Go
23. **Real-time updates** — SSE for deploy progress and instance status changes
### Phase 4: Hardening ### Phase 4: Hardening
17. **Blue-green deploys** — start new, health check, swap, stop old (zero downtime) 24. **Blue-green deploys** — start new, health check, swap, stop old (zero downtime)
18. **Promote flow** — enforce `promote_from` for production deploys 25. **Promote flow** — enforce `promote_from` for production deploys
19. **Auth on dashboard** — basic auth or token-based 26. **Auth on dashboard** — basic auth or token-based
20. **Graceful shutdown** — drain in-progress deploys on SIGTERM 27. **Graceful shutdown** — drain in-progress deploys on SIGTERM
21. **Structured logging** — JSON logs with deploy context 28. **Structured logging** — JSON logs with deploy context
29. **Config export** — download current SQLite state as YAML
## Key Dependencies (Go) ## Key Dependencies (Go)
- `github.com/docker/docker` — Docker Engine API - `github.com/docker/docker` — Docker Engine API
- `github.com/go-chi/chi` or `net/http` — HTTP routing - `github.com/go-chi/chi` or `net/http` — HTTP routing
- `gopkg.in/yaml.v3` — YAML config - `gopkg.in/yaml.v3` — YAML seed config
- `github.com/mattn/go-sqlite3` or `modernc.org/sqlite` — SQLite (CGo-free) - `modernc.org/sqlite` — SQLite (CGo-free)
- `github.com/robfig/cron` — Polling scheduler - `github.com/robfig/cron` — Polling scheduler
- `github.com/google/uuid` — Webhook secret URL generation
## Docker Compose (self-deployment) ## Docker Compose (self-deployment)
@@ -227,12 +317,10 @@ services:
- "8080:8080" - "8080:8080"
volumes: volumes:
- /var/run/docker.sock:/var/run/docker.sock - /var/run/docker.sock:/var/run/docker.sock
- ./docker-watcher.yaml:/app/config.yaml:ro - ./docker-watcher.yaml:/app/seed.yaml:ro # optional, first launch only
- ./envs:/app/envs:ro - ./data:/app/data # SQLite DB
- ./data:/app/data # SQLite DB
environment: environment:
- NPM_PASSWORD=${NPM_PASSWORD} - ENCRYPTION_KEY=${ENCRYPTION_KEY} # protects all credentials in DB
- GITEA_TOKEN=${GITEA_TOKEN}
networks: networks:
- staging-net - staging-net
@@ -241,16 +329,96 @@ networks:
external: true external: true
``` ```
## API Endpoints (Phase 3) ## API Endpoints
```text
# Projects
GET /api/projects — list all projects with instance counts
POST /api/projects — create project
GET /api/projects/:id — project detail + stages + instances
PUT /api/projects/:id — update project config
DELETE /api/projects/:id — delete project + all instances
# Stages
POST /api/projects/:id/stages — add stage to project
PUT /api/projects/:id/stages/:stage — update stage config
DELETE /api/projects/:id/stages/:stage — delete stage + its instances
# Instances (running containers)
GET /api/projects/:id/stages/:stage/instances — list instances for stage
POST /api/projects/:id/stages/:stage/instances — deploy new instance (pick tag)
DELETE /api/projects/:id/stages/:stage/instances/:iid — remove instance (container + NPM proxy)
POST /api/projects/:id/stages/:stage/instances/:iid/stop — stop container
POST /api/projects/:id/stages/:stage/instances/:iid/start — start stopped container
POST /api/projects/:id/stages/:stage/instances/:iid/restart — restart container
# Quick Deploy
POST /api/deploy/inspect — pull + inspect image, return defaults
POST /api/deploy/quick — create project + deploy in one step
# Registry
GET /api/registries — list registries
POST /api/registries — add registry
PUT /api/registries/:id — update registry
DELETE /api/registries/:id — delete registry
POST /api/registries/:id/test — test connection
GET /api/registries/:id/tags/:image — list available tags
# Settings
GET /api/settings — get global settings
PUT /api/settings — update global settings
GET /api/settings/webhook-url — get secret webhook URL
POST /api/settings/webhook-url/regenerate — regenerate webhook URL
# Deploy history
GET /api/deploys — recent deploys across all projects
GET /api/deploys/:id/logs — deploy log stream (SSE)
# Webhook (secret URL — no auth needed)
POST /api/webhook/:secret-uuid — receive image push notification
``` ```
GET /api/projects — list all projects with current status
GET /api/projects/:id — project detail + stage statuses ## User Workflows
GET /api/projects/:id/stages/:stage — stage detail + deploy history
POST /api/projects/:id/stages/:stage/deploy — trigger manual deploy ### Auto-Deploy (zero effort)
POST /api/projects/:id/stages/:stage/rollback — rollback to previous
GET /api/deploys — recent deploys across all projects ```text
GET /api/deploys/:id/logs — deploy log stream (SSE) Push code → CI builds → pushes tag → Docker Watcher detects →
POST /api/webhook/gitea — Gitea push event receiver auto_deploy: true → deployed → notification with URL
POST /api/webhook/github — GitHub push event receiver (future) ```
### Manual Deploy via UI (one click)
```text
Open dashboard → project → stage → "Deploy new version" →
pick tag from dropdown → click Deploy
```
### Quick Deploy (new project, paste image URL)
```text
Open dashboard → "Quick Deploy" → paste image URL →
review auto-filled defaults → click Deploy →
project auto-created + deployed
```
### Deploy via CI Webhook (zero effort after CI setup)
```text
# In .gitea/workflows/build.yml
- name: Notify Docker Watcher
run: |
curl -X POST https://watcher.dolgolyov-family.by/api/webhook/d8f2a1e9-... \
-d '{"image": "git.dolgolyov-family.by/alexei/my-app:dev-${{ github.sha }}"}'
```
Known project → deploys per stage config.
Unknown project → auto-creates with defaults from image inspection, deploys.
### Production Deploy (two clicks)
```text
Open dashboard → project → prod stage → "Deploy new version" →
dropdown shows only tags running in "rel" stage (promote_from) →
pick tag → confirmation dialog → Deploy
``` ```