alexei.dolgolyov/tiny-forge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: revise implementation plan

Browse Source 
Add multi-instance support, UI-managed config (SQLite), secret webhook
URLs, quick deploy flow, and Cloudflare wildcard DNS strategy. Remove
per-project DNS automation in favor of wildcard record.
This commit is contained in:
Admin
2026-03-27 20:38:51 +03:00
parent ae6735b05c
commit 17206933f3
1 changed files with 263 additions and 95 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PLAN.md
+263 -95
View File
@@ -2,34 +2,39 @@
## Overview
A self-hosted tool that automates Docker container deployment with Nginx Proxy Manager integration. Detects new images from Gitea/GitHub registries, deploys containers, and configures reverse proxy routing — all from a web dashboard.
A self-hosted tool that automates Docker container deployment with Nginx Proxy Manager integration. Detects new images from Gitea/GitHub registries, deploys containers, and configures reverse proxy routing — all from a web dashboard. Supports multiple simultaneous versions of the same project. DNS is handled by a Cloudflare wildcard record (`*.dolgolyov-family.by`) — no per-project DNS management needed.
## Architecture
```
```text
Gitea CI → pushes image → Registry
│ ↓
│ Docker Watcher (Go)
│ ├── Secret webhook URL (instant)
│ └── Registry poller (fallback)
│ ↓
└── or: POST /api/webhook/<secret-uuid>
with {"image": "registry/org/app:tag"}
↓
Docker Watcher (Go)
├── Webhook listener (instant)
└── Registry poller (fallback)
↓
Match tag → project + stage
↓
┌── auto_deploy? ──┐
↓ ↓
Deploy now Notify, wait for
manual trigger (UI)
↓ ↓
Pull image
Stop old container
Start new container on shared network
↓
NPM API: create/update proxy host
→ stage-dev-project.example.com
↓
Health check
→ success: done, notify
→ failure: rollback, alert
Known project? ──────────────────┐
↓ yes ↓ no
Match tag → stage Auto-create project
↓ with defaults from
auto_deploy? image inspection
↓ yes ↓ no (EXPOSE, labels)
Deploy now Notify, wait ↓
↓ for UI trigger Deploy with defaults
↓ ↓
Pull image
Start new container on shared network
(old container stays if multi-instance)
↓
NPM API: create proxy host (if first deploy for this subdomain)
(DNS already handled by Cloudflare wildcard *.domain)
↓
Health check
→ success: done, notify
→ failure: remove new container, alert
```
## Decisions
@@ -39,79 +44,138 @@ Gitea CI → pushes image → Registry
| Language | Go | Single binary, excellent Docker SDK, low resource usage |
| Web UI | SvelteKit (embedded in Go binary) | User's existing stack, lightweight |
| Reverse proxy | Nginx Proxy Manager | Already deployed, API available |
| DNS | Cloudflare wildcard `*.{domain}` | One-time setup, all subdomains auto-resolve |
| Routing | Subdomain-based | No sub-path issues with SPAs |
| Image detection | Webhook + polling | Webhook for speed, polling as fallback |
| Config | YAML | Version-controlled, easy to edit |
| Image detection | Secret webhook URL + polling | Webhook for speed, polling as fallback |
| Config storage | SQLite (YAML for initial seed only) | Editable via UI, no manual file editing |
| Credentials | Encrypted in SQLite (AES-256) | Single ENCRYPTION_KEY env var |
| Webhook auth | Secret UUID in URL | No tokens needed, simple CI integration |
| Multi-instance | Yes | Multiple tags of same project can run simultaneously |
| Deployment target | Same TrueNAS host | Docker socket mounted |
## Subdomain Convention
| Stage | Pattern | Example |
|-------|---------|---------|
| Dev | `stage-dev-{project}.{domain}` | `stage-dev-web-app-launcher.dolgolyov-family.by` |
| Release | `stage-rel-{project}.{domain}` | `stage-rel-web-app-launcher.dolgolyov-family.by` |
| Type | Pattern | Example |
|------|---------|---------|
| Dev (default) | `stage-dev-{project}.{domain}` | `stage-dev-web-app-launcher.dolgolyov-family.by` |
| Dev (specific tag) | `stage-dev-{project}-{tag}.{domain}` | `stage-dev-web-app-launcher-abc123.dolgolyov-family.by` |
| Release (default) | `stage-rel-{project}.{domain}` | `stage-rel-web-app-launcher.dolgolyov-family.by` |
| Release (specific tag) | `stage-rel-{project}-{tag}.{domain}` | `stage-rel-web-app-launcher-v1-2-0.dolgolyov-family.by` |
| Production | `{custom}.{domain}` | `launcher.dolgolyov-family.by` |
## Config Format
Tags are sanitized for DNS: dots → dashes, lowercase, truncated to fit DNS limits.
## Configuration
### First Launch
```text
YAML seed file exists? → import into SQLite → done
No YAML? → empty state, configure everything via UI
```
After import, all configuration lives in SQLite and is managed via the Web UI.
YAML is never read again unless user clicks "Re-import config" or "Export config".
### Seed Config Format (optional)
```yaml
global:
domain: dolgolyov-family.by
server_ip: 93.84.96.191
network: staging-net
subdomain_pattern: "stage-{stage}-{project}"
notification_url: https://notify.dolgolyov-family.by/webhook
npm:
url: http://npm:81
email: docker-watcher@dolgolyov-family.by
password_env: NPM_PASSWORD
subdomain_pattern: "stage-{stage}-{project}"
notification_url: https://notify.dolgolyov-family.by/webhook
password: "npm-password-here"
registries:
gitea:
url: https://git.dolgolyov-family.by
type: gitea
token_env: GITEA_TOKEN
# github:
# url: https://ghcr.io
# type: github
# token_env: GITHUB_TOKEN
token: "gitea-token-here"
projects:
web-app-launcher:
registry: gitea
image: gitea.dolgolyov-family.by/alexei/web-app-launcher
image: git.dolgolyov-family.by/alexei/web-app-launcher
port: 3000
healthcheck: /api/health
env_file: ./envs/web-app-launcher.env
volumes: []
env:
NODE_ENV: production
stages:
dev:
tag_pattern: "dev-*"
auto_deploy: true
max_instances: 5
rel:
tag_pattern: "v*"
auto_deploy: false
max_instances: 2
prod:
tag_pattern: "v*"
auto_deploy: false
confirm: true
promote_from: rel
subdomain: launcher # overrides pattern → launcher.dolgolyov-family.by
max_instances: 2
subdomain: launcher
```
## Web UI Sections
### Dashboard
Overview of all projects with their running instances:
- Project name, running instance count, latest activity
- Quick status indicators (healthy / stopped / failing)
- "Quick Deploy" button for ad-hoc image deployment
### Project Detail
Per-project view with stages and instances:
- Each stage shows all running instances with: tag, status, URL, uptime
- Controls per instance: Stop, Start, Restart, Remove
- "Deploy new version" dropdown — lists available tags from registry
- Deploy history log
### Quick Deploy
For deploying images not yet configured as projects:
1. Paste image URL (e.g., `git.dolgolyov-family.by/alexei/my-app:dev-abc123`)
2. Docker Watcher pulls and inspects image (EXPOSE port, HEALTHCHECK, labels)
3. Pre-fills form with sensible defaults (project name, port, stage, subdomain)
4. User reviews, tweaks, clicks "Deploy"
5. Project is auto-created in the DB for future use
### Settings
- **Registries** — add/edit/delete registries, test connection
- **Credentials** — NPM, registry tokens (encrypted, shown as `••••••••`)
- **Global** — domain, server IP, Docker network, subdomain pattern, polling interval
- **Notifications** — webhook URL
- **Webhook URL** — shows the secret deploy URL, "Regenerate" button
### Projects Config
- Add / edit / delete projects via UI
- Configure image, port, healthcheck, env vars, volumes per project
- Add / remove stages, set tag patterns, auto-deploy, subdomain overrides, max instances
## Project Structure
```
```text
docker-watcher/
├── cmd/
│ └── server/
│ └── main.go # Entry point
├── internal/
│ ├── config/
│ │ ├── config.go # YAML parsing, validation
│ │ ├── config.go # YAML seed parsing
│ │ └── config_test.go
│ ├── docker/
│ │ ├── client.go # Docker Engine API wrapper
│ │ ├── container.go # Create, start, stop, remove
│ │ ├── container.go # Create, start, stop, remove, inspect
│ │ └── client_test.go
│ ├── npm/
│ │ ├── client.go # NPM API client (auth, CRUD proxy hosts)
@@ -133,33 +197,51 @@ docker-watcher/
│ │ ├── notifier.go # Webhook notifications
│ │ └── notifier_test.go
│ ├── webhook/
│ │ ├── handler.go # Receive push events from Gitea/GitHub
│ │ ├── handler.go # Secret URL webhook receiver
│ │ └── handler_test.go
│ ├── api/
│ │ ├── router.go # HTTP API for web UI
│ │ ├── projects.go # Project endpoints
│ │ ├── deploys.go # Deploy endpoints
│ │ ├── projects.go # Project CRUD endpoints
│ │ ├── registries.go # Registry CRUD endpoints
│ │ ├── settings.go # Global settings endpoints
│ │ ├── instances.go # Instance start/stop/restart/remove
│ │ ├── deploys.go # Deploy + quick deploy endpoints
│ │ └── middleware.go # Auth, logging, CORS
│ └── store/
│ ├── store.go # Deploy history, state (SQLite)
│ └── store_test.go
│ ├── store/
│ │ ├── store.go # SQLite schema, migrations
│ │ ├── projects.go # Project queries
│ │ ├── instances.go # Instance queries
│ │ ├── registries.go # Registry queries
│ │ ├── settings.go # Settings queries
│ │ ├── deploys.go # Deploy history queries
│ │ └── store_test.go
│ └── crypto/
│ └── crypto.go # AES-256 encrypt/decrypt for credentials
├── web/ # SvelteKit frontend
│ ├── src/
│ │ ├── routes/
│ │ │ ├── +page.svelte # Dashboard
│ │ │ ├── projects/
│ │ │ │ ├── +page.svelte # Projects list + add
│ │ │ │ └── [id]/
│ │ │ │ └── +page.svelte # Project detail + deploy controls
│ │ │ │ └── +page.svelte # Project detail + instances
│ │ │ ├── deploy/
│ │ │ │ └── +page.svelte # Quick deploy
│ │ │ └── settings/
│ │ │ └── +page.svelte # Config view
│ │ │ ├── +page.svelte # Global settings
│ │ │ ├── registries/
│ │ │ │ └── +page.svelte
│ │ │ └── credentials/
│ │ │ └── +page.svelte
│ │ ├── lib/
│ │ │ ├── api.ts # API client
│ │ │ └── types.ts # Shared types
│ │ │ ├── types.ts # Shared types
│ │ │ └── components/ # Reusable UI components
│ │ └── app.html
│ ├── package.json
│ ├── svelte.config.js
│ └── vite.config.ts
├── docker-watcher.example.yaml
├── docker-watcher.example.yaml # Example seed config
├── Dockerfile
├── docker-compose.yml
├── go.mod
@@ -170,50 +252,58 @@ docker-watcher/
### Phase 1: Foundation
Core infrastructure — config, Docker client, NPM client.
Core infrastructure — store, config import, Docker client, NPM client.
1. **Go project init** — go.mod, directory structure, dependencies
2. **Config loader** — parse YAML, validate, resolve env vars
3. **Docker client** — connect to socket, pull image, list/start/stop/remove containers, manage networks
4. **NPM client** — authenticate (JWT), create/update/delete proxy hosts, list existing hosts
5. **Store** — SQLite for deploy history and state tracking
2. **SQLite store** — schema, migrations, CRUD for projects/registries/settings/instances/deploys
3. **Crypto** — AES-256 encrypt/decrypt for credential storage
4. **Config seed loader** — parse YAML, import into SQLite on first launch
5. **Docker client** — connect to socket, pull image, inspect image, list/start/stop/remove containers, manage networks
6. **NPM client** — authenticate (JWT), create/update/delete proxy hosts, list existing hosts
### Phase 2: Detection & Deployment
The core loop — detecting new images and deploying them.
6. **Registry client** — Gitea registry API: list tags, compare with last known
7. **Poller** — periodic check for new tags matching configured patterns
8. **Webhook handler** — HTTP endpoint receiving push events from Gitea CI
9. **Deployer** — orchestrate: pull → stop old → start new → NPM update → health check
10. **Health checker** — HTTP GET with retries and timeout
11. **Rollback** — on health check failure: stop new, restart old, revert NPM
12. **Notifications** — send webhook on deploy success/failure
8. **Registry client** — Gitea registry API: list tags for an image, detect new tags
9. **Poller** — periodic check for new tags matching configured patterns
10. **Secret webhook handler** — UUID-based URL, receives image push notifications, auto-creates unknown projects
11. **Deployer** — orchestrate: pull → start container → NPM proxy → health check
12. **Multi-instance support** — multiple versions per project/stage, tag-based subdomains, max_instances limit
13. **Health checker** — HTTP GET with retries and timeout
14. **Rollback** — on health check failure: remove new container, clean up NPM, alert
15. **Notifications** — send webhook on deploy success/failure
### Phase 3: Web UI
Dashboard for visibility and manual control.
Full dashboard for visibility, manual control, and configuration.
13. **API layer** — REST endpoints for projects, stages, deploys, manual trigger
14. **SvelteKit app** — dashboard, project detail, deploy log, manual deploy button
15. **Embed in Go** — build SvelteKit to static, embed with `go:embed`, serve from Go
16. **Real-time updates** — SSE or WebSocket for deploy progress
16. **API layer** — REST endpoints for all CRUD operations + deploy/control actions
17. **SvelteKit dashboard** — project overview, instance status, quick status indicators
18. **Project detail view** — stages, instances, controls (stop/start/restart/remove), deploy history
19. **Quick Deploy page** — paste image URL, auto-inspect, pre-fill form, one-click deploy
20. **Settings pages** — registries, credentials, global settings, webhook URL management
21. **Project config pages** — add/edit/delete projects and stages via UI
22. **Embed in Go** — build SvelteKit to static, embed with `go:embed`, serve from Go
23. **Real-time updates** — SSE for deploy progress and instance status changes
### Phase 4: Hardening
17. **Blue-green deploys** — start new, health check, swap, stop old (zero downtime)
18. **Promote flow** — enforce `promote_from` for production deploys
19. **Auth on dashboard** — basic auth or token-based
20. **Graceful shutdown** — drain in-progress deploys on SIGTERM
21. **Structured logging** — JSON logs with deploy context
24. **Blue-green deploys** — start new, health check, swap, stop old (zero downtime)
25. **Promote flow** — enforce `promote_from` for production deploys
26. **Auth on dashboard** — basic auth or token-based
27. **Graceful shutdown** — drain in-progress deploys on SIGTERM
28. **Structured logging** — JSON logs with deploy context
29. **Config export** — download current SQLite state as YAML
## Key Dependencies (Go)
- `github.com/docker/docker` — Docker Engine API
- `github.com/go-chi/chi` or `net/http` — HTTP routing
- `gopkg.in/yaml.v3` — YAML config
- `github.com/mattn/go-sqlite3` or `modernc.org/sqlite` — SQLite (CGo-free)
- `gopkg.in/yaml.v3` — YAML seed config
- `modernc.org/sqlite` — SQLite (CGo-free)
- `github.com/robfig/cron` — Polling scheduler
- `github.com/google/uuid` — Webhook secret URL generation
## Docker Compose (self-deployment)
@@ -227,12 +317,10 @@ services:
- "8080:8080"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./docker-watcher.yaml:/app/config.yaml:ro
- ./envs:/app/envs:ro
- ./data:/app/data # SQLite DB
- ./docker-watcher.yaml:/app/seed.yaml:ro # optional, first launch only
- ./data:/app/data # SQLite DB
environment:
- NPM_PASSWORD=${NPM_PASSWORD}
- GITEA_TOKEN=${GITEA_TOKEN}
- ENCRYPTION_KEY=${ENCRYPTION_KEY} # protects all credentials in DB
networks:
- staging-net
@@ -241,16 +329,96 @@ networks:
external: true
```
## API Endpoints (Phase 3)
## API Endpoints
```text
# Projects
GET /api/projects — list all projects with instance counts
POST /api/projects — create project
GET /api/projects/:id — project detail + stages + instances
PUT /api/projects/:id — update project config
DELETE /api/projects/:id — delete project + all instances
# Stages
POST /api/projects/:id/stages — add stage to project
PUT /api/projects/:id/stages/:stage — update stage config
DELETE /api/projects/:id/stages/:stage — delete stage + its instances
# Instances (running containers)
GET /api/projects/:id/stages/:stage/instances — list instances for stage
POST /api/projects/:id/stages/:stage/instances — deploy new instance (pick tag)
DELETE /api/projects/:id/stages/:stage/instances/:iid — remove instance (container + NPM proxy)
POST /api/projects/:id/stages/:stage/instances/:iid/stop — stop container
POST /api/projects/:id/stages/:stage/instances/:iid/start — start stopped container
POST /api/projects/:id/stages/:stage/instances/:iid/restart — restart container
# Quick Deploy
POST /api/deploy/inspect — pull + inspect image, return defaults
POST /api/deploy/quick — create project + deploy in one step
# Registry
GET /api/registries — list registries
POST /api/registries — add registry
PUT /api/registries/:id — update registry
DELETE /api/registries/:id — delete registry
POST /api/registries/:id/test — test connection
GET /api/registries/:id/tags/:image — list available tags
# Settings
GET /api/settings — get global settings
PUT /api/settings — update global settings
GET /api/settings/webhook-url — get secret webhook URL
POST /api/settings/webhook-url/regenerate — regenerate webhook URL
# Deploy history
GET /api/deploys — recent deploys across all projects
GET /api/deploys/:id/logs — deploy log stream (SSE)
# Webhook (secret URL — no auth needed)
POST /api/webhook/:secret-uuid — receive image push notification
```
GET /api/projects — list all projects with current status
GET /api/projects/:id — project detail + stage statuses
GET /api/projects/:id/stages/:stage — stage detail + deploy history
POST /api/projects/:id/stages/:stage/deploy — trigger manual deploy
POST /api/projects/:id/stages/:stage/rollback — rollback to previous
GET /api/deploys — recent deploys across all projects
GET /api/deploys/:id/logs — deploy log stream (SSE)
POST /api/webhook/gitea — Gitea push event receiver
POST /api/webhook/github — GitHub push event receiver (future)
## User Workflows
### Auto-Deploy (zero effort)
```text
Push code → CI builds → pushes tag → Docker Watcher detects →
auto_deploy: true → deployed → notification with URL
```
### Manual Deploy via UI (one click)
```text
Open dashboard → project → stage → "Deploy new version" →
pick tag from dropdown → click Deploy
```
### Quick Deploy (new project, paste image URL)
```text
Open dashboard → "Quick Deploy" → paste image URL →
review auto-filled defaults → click Deploy →
project auto-created + deployed
```
### Deploy via CI Webhook (zero effort after CI setup)
```text
# In .gitea/workflows/build.yml
- name: Notify Docker Watcher
run: |
curl -X POST https://watcher.dolgolyov-family.by/api/webhook/d8f2a1e9-... \
-d '{"image": "git.dolgolyov-family.by/alexei/my-app:dev-${{ github.sha }}"}'
```
Known project → deploys per stage config.
Unknown project → auto-creates with defaults from image inspection, deploys.
### Production Deploy (two clicks)
```text
Open dashboard → project → prod stage → "Deploy new version" →
dropdown shows only tags running in "rel" stage (promote_from) →
pick tag → confirmation dialog → Deploy
```