feat(gitops): config-as-code via .tinyforge.yml for repo-backed workloads
A dockerfile or static workload can opt in to reading its deploy config from a
.tinyforge.yml in its own repo. Tinyforge fetches the file, shows field-level
drift vs the live config, and an admin applies it with an explicit Sync. The
repo becomes the source of truth for the declared fields. Manual-sync only;
no auto-apply on deploy, no multi-workload reconcile, no create/delete in v1.
Scope is deliberately source-aware and source_config-resident: dockerfile
declares port/healthcheck/deploy_strategy, static declares deploy_strategy.
The file never carries repo coords or secrets (those stay in the encrypted
DB), which keeps credentials out of the repo.
Backend:
- internal/gitops: Spec/ParseSpec (KnownFields rejects unknown keys), a
source-aware ApplyPlan/BuildPlan, MergeAndValidate (omitted-field-preserving
deep merge + validate-the-merged-result-then-commit — never a partial
config), declared-only Drift with normalization, and Fetch with
ok/no_file/fetch_failed/invalid statuses and token-redacted messages.
- staticsite: DownloadFile added to GitProvider + Gitea/GitHub/GitLab impls,
reusing each provider's SSRF-safe client; 64 KiB cap; ErrFileNotFound.
- store: 4 additive gitops_* columns + setters (disjoint from UpdateWorkload
so the edit-form save and a sync never clobber each other).
- api: GET /workloads/{id}/gitops (status + raw + live drift + managed_fields),
PUT /gitops (admin, enable/path, traversal-safe), POST /gitops/sync (admin,
per-workload locked read->merge->validate->write, audited to event_log).
Frontend:
- GitOpsPanel.svelte: status pill, a purpose-built field-level drift view,
.tinyforge.yml preview, enable ToggleSwitch, Sync via ConfirmDialog; all five
statuses handled, admin affordances gated on the real viewer role.
- GitOps-managed badge (list + detail hero) and a read-only edit-form banner.
- api.ts fetchers + types; i18n apps.detail.gitops.* (en + ru parity).
Built phase-by-phase with an adversarial plan review (caught 5 design flaws
pre-implementation) and an independent review per phase (go / security / ts /
final) — all APPROVE, 0 CRITICAL/HIGH. docs/gitops.md documents the schema and
what's intentionally out of v1. Plan: plans/gitops/.
This commit is contained in:
@@ -0,0 +1,115 @@
|
||||
# Tinyforge GitOps v1 — config-as-code for repo-backed workloads
|
||||
|
||||
**Status:** ✅ Complete (squash-merged to main 2026-06-21)
|
||||
**Branch:** `feat/gitops-config-as-code`
|
||||
**Mode:** Automated · Orchestrator (hybrid: backend built direct, Phase 3 via frontend implementer) · Incremental
|
||||
**Started:** 2026-06-21
|
||||
|
||||
## Summary
|
||||
|
||||
A `dockerfile` or `static` workload can opt in to reading its **deploy config** from a
|
||||
`.tinyforge.yml` file in its own repo. Tinyforge fetches the file, shows it, computes
|
||||
**drift** vs the live config, and lets an admin **sync** (repo → live) with one explicit
|
||||
action. The repo becomes the source of truth for the *declared* fields; the UI locks
|
||||
those fields and renders a drift view.
|
||||
|
||||
### v1 scope (deliberate)
|
||||
|
||||
- **In:** opt-in per workload (dockerfile/static only); `.tinyforge.yml` declares only
|
||||
**source_config-resident** fields (`port`, `healthcheck`, `deploy_strategy`,
|
||||
`resources.{cpu_limit,memory_limit}`, `max_instances`); manual explicit sync;
|
||||
declared-field drift view; GitOps-managed badge + read-only gate.
|
||||
- **Out (documented future seams):** `env`/`faces` declaration (separate stores —
|
||||
needs typed multi-target apply); auto-apply-on-deploy (must be a Source-plugin concern,
|
||||
not a dispatch concern); multi-workload reconcile with create/delete (Framing B);
|
||||
image/compose sources (not git-backed / overlapping config surface).
|
||||
|
||||
### `.tinyforge.yml` v1 schema
|
||||
|
||||
```yaml
|
||||
version: 1
|
||||
deploy:
|
||||
port: 8080
|
||||
healthcheck: /healthz
|
||||
deploy_strategy: blue-green # "" | recreate | blue-green (validated per source)
|
||||
resources: { cpu_limit: 0.5, memory_limit: 256 }
|
||||
max_instances: 1
|
||||
```
|
||||
|
||||
No repo location, no tokens, no secrets — those stay in the encrypted DB.
|
||||
|
||||
## Design constraints (from the adversarial review — non-negotiable)
|
||||
|
||||
- **C1** Overlay is a typed `ApplyPlan{SourceConfigPatch}` routed to `source_config` only.
|
||||
env/faces are NOT in source_config (they live in `workload_env` / `public_faces`), so
|
||||
they are cut from v1; the typed plan reserves their slots for later.
|
||||
- **C2** No `env` in v1 → no secrets-in-repo hole.
|
||||
- **C3** No auto-apply-on-deploy in v1 (SHA is resolved *inside* `src.Deploy`; image has
|
||||
no repo). Future auto-apply lands as a Source-plugin concern.
|
||||
- **C4** Sync is explicit-action only, with a hard gate:
|
||||
parse → build overlay → **omitted-field-preserving** deep-merge onto a fresh copy of the
|
||||
live source_config → run `Source.Validate` on the *merged* result → persist in one
|
||||
transaction only if valid.
|
||||
- **C5** Drift is computed **only over declared leaves**, post-normalization
|
||||
(`deploy_strategy:"" == "recreate"`; YAML int vs JSON coercion). Omitted = unmanaged.
|
||||
- Reuse `staticsite.NewGitProvider` (inherits SSRF defense); add a size-capped
|
||||
`DownloadFile`. Route all fetch errors through the existing `sanitizeError(msg, token)`.
|
||||
Distinct `no_file` status. Sync audit is NOT `deploy_history` (rollback assumes
|
||||
deployable rows). Gate enable to `dockerfile|static`. Derive read-only fields from the
|
||||
declared overlay leaves (no provenance column). 4 additive `gitops_*` columns only.
|
||||
|
||||
## Phases
|
||||
|
||||
| # | Title | Subplan | Status |
|
||||
|---|-------|---------|--------|
|
||||
| 1 | GitOps core (backend, no UI/mutation) | [phase-1-core.md](phase-1-core.md) | ✅ Done |
|
||||
| 2 | Store + API (manual sync) | [phase-2-api.md](phase-2-api.md) | ✅ Done |
|
||||
| 3 | Frontend experience (UI/UX showcase) | [phase-3-frontend.md](phase-3-frontend.md) | ✅ Done |
|
||||
| 4 | Hardening + docs + final review | [phase-4-hardening.md](phase-4-hardening.md) | ✅ Done |
|
||||
|
||||
## Phase progress log
|
||||
|
||||
- **Phase 1 — Done (2026-06-21).** Migration (4 additive `gitops_*` columns) + `Workload`
|
||||
read path. New `internal/gitops` package: `Spec`/`ParseSpec` (KnownFields rejects
|
||||
unknown keys incl. env/faces attempts), source-aware `ApplyPlan`/`BuildPlan`
|
||||
(dockerfile: port/healthcheck/deploy_strategy; static: deploy_strategy only — `resources`/
|
||||
`max_instances` dropped after confirming they aren't on dockerfile/static configs),
|
||||
`MergeAndValidate` (omitted-field-preserving + validate-then-commit), `Drift`
|
||||
(declared-only, normalized), `Fetch` (no_file/fetch_failed/invalid statuses, token-redacted).
|
||||
`DownloadFile` added to the `GitProvider` interface + 3 impls (64 KiB cap, ErrFileNotFound,
|
||||
SSRF-safe client reused, GitHub raw media type). Independent go-review: **APPROVE**, no
|
||||
CRITICAL/HIGH; M1 (GitLab doc comment) fixed; M2 (validate GitOpsPath at write) carried
|
||||
into Phase 2. 28/28 packages green.
|
||||
- **Phase 2 — Done (2026-06-21).** Store setters `SetWorkloadGitOps` / `RecordGitOpsSync`
|
||||
(targeted column updates — disjoint from `UpdateWorkload`, so neither writer clobbers the
|
||||
other). API: `GET /gitops` (single rich payload: status + raw + live drift + meta — folded
|
||||
the separate `/drift` endpoint in to avoid a double fetch), `PUT /gitops` (admin,
|
||||
enable/disable + path, rejects non-eligible source + traversal/URL-injection paths),
|
||||
`POST /gitops/sync` (admin: fetch → MergeAndValidate → UpdateWorkload → RecordGitOpsSync →
|
||||
event-log audit). Sync recorded to `event_log` (not `deploy_history` — review S6). Tests:
|
||||
store round-trip + `validGitOpsPath` + `planFields`. Independent **security review:
|
||||
clean, no CRITICAL/HIGH** (token never leaks, SSRF locked by safe dialer, authZ correct,
|
||||
no field loss); LOW-1 (path query/fragment injection) hardened in `validGitOpsPath`. Full
|
||||
backend suite green.
|
||||
- **Phase 3 — Done (2026-06-21).** Built by a frontend implementation agent.
|
||||
`GitOpsPanel.svelte` (self-fetching panel: status pill, purpose-built field-level drift
|
||||
view — repo→live per declared field on the forge/ember palette, `.tinyforge.yml` preview,
|
||||
enable `ToggleSwitch`, "Sync now" via `ConfirmDialog`, all five status states). api.ts
|
||||
fetchers + `GitOpsStatus`/`GitOpsDriftEntry`; `gitops_*` on the `Workload` TS type;
|
||||
GitOps-managed badge on the detail hero + apps list (payload already carries
|
||||
`gitops_enabled`); read-only edit-form banner (banner-only — hard-disabling inputs would
|
||||
need prop-threading through all 4 source forms; deferred). Backend `managed_fields` added
|
||||
to `GET /gitops` for the gate. i18n `apps.detail.gitops.*` en+ru (parity 1804/1804).
|
||||
Independent ts-review: one HIGH (`isAdmin` hardcoded true) + 2 MEDIUM — **all fixed**:
|
||||
real role wired via `getCurrentUser()` (panel default now `false`), stale-guard on the
|
||||
edit-open fetch, misleading `eligible` comment trimmed. check 0 errors · build · 26/26.
|
||||
- **Phase 4 — Done (2026-06-21).** Concurrent-sync guard (review S5): a per-workload
|
||||
`keyedMutex` on `Server`; `syncWorkloadGitOps` locks by id and loads the row inside the
|
||||
lock, serializing the read→merge→write so two syncs can't race. Docs: `docs/gitops.md`
|
||||
(enable flow, v1 schema, drift/sync semantics, explicit "not in v1": env/faces, auto-apply,
|
||||
multi-workload, image/compose). Backend green. Final comprehensive review + merge gate
|
||||
next.
|
||||
|
||||
## Amendment log
|
||||
|
||||
_(plan changes require approval + an entry here)_
|
||||
@@ -0,0 +1,57 @@
|
||||
# Phase 1 — GitOps core (backend, no UI, no mutations)
|
||||
|
||||
Pure logic + fetch. No HTTP endpoints, no DB writes to workloads yet (migration only).
|
||||
|
||||
## Tasks
|
||||
|
||||
- [ ] **Migration**: append 4 additive columns to the workloads-table migration list in
|
||||
`internal/store/store.go` (idempotent `ALTER TABLE workloads ADD COLUMN`):
|
||||
- `gitops_enabled INTEGER NOT NULL DEFAULT 0`
|
||||
- `gitops_path TEXT NOT NULL DEFAULT '.tinyforge.yml'`
|
||||
- `gitops_last_sync_at TEXT NOT NULL DEFAULT ''`
|
||||
- `gitops_commit_sha TEXT NOT NULL DEFAULT ''`
|
||||
- Add the fields to the `Workload` struct in `internal/store/models.go` + the
|
||||
scan/insert/update column lists in `internal/store/workloads.go` (read path now;
|
||||
write path for the setters lands in Phase 2).
|
||||
- [ ] **`internal/gitops` package — `spec.go`**: `Spec` struct mirroring the v1 schema
|
||||
(`Version int`, `Deploy DeploySpec{Port *int, Healthcheck *string, DeployStrategy *string,
|
||||
Resources *ResourceSpec{CpuLimit *float64, MemoryLimit *int}, MaxInstances *int}`).
|
||||
Pointers so "omitted" is distinguishable from "zero". `ParseSpec([]byte) (Spec, error)`
|
||||
using `gopkg.in/yaml.v3` with `KnownFields(true)` to reject unknown keys; reject
|
||||
`version != 1`.
|
||||
- [ ] **`apply.go`**: typed `ApplyPlan{ SourceConfigPatch map[string]any }` (env/faces
|
||||
slots reserved in a comment). `BuildPlan(spec) ApplyPlan` maps only the **present**
|
||||
(non-nil) declared fields to their `source_config` JSON keys (`port`, `healthcheck`,
|
||||
`deploy_strategy`, `cpu_limit`, `memory_limit`, `max_instances`).
|
||||
- [ ] **`merge.go`**: `MergeAndValidate(liveConfig json.RawMessage, plan ApplyPlan,
|
||||
validate func(json.RawMessage) error) (json.RawMessage, error)` — deep-copy live →
|
||||
overlay only the patch keys (omitted-field-preserving) → marshal → run `validate` on the
|
||||
**merged** result → return merged or error. Never returns a partial/empty config.
|
||||
- [ ] **`drift.go`**: `Drift(spec Spec, liveConfig json.RawMessage) ([]DriftEntry, error)`
|
||||
where `DriftEntry{Field string /*dotted path*/, RepoValue string, LiveValue string}`.
|
||||
Compare **only declared leaves**, post-normalization:
|
||||
- `deploy_strategy` via the source's effective-default rule (`"" == "recreate"` for
|
||||
dockerfile/static) — import or replicate `effectiveStrategy` semantics.
|
||||
- numeric coercion (YAML int vs JSON number) compared by value, not raw string.
|
||||
- [ ] **Provider `DownloadFile`**: add `DownloadFile(ctx, owner, repo, ref, path string,
|
||||
maxBytes int64) ([]byte, error)` to the `GitProvider` interface in
|
||||
`internal/staticsite/provider.go` and implement for Gitea, GitHub, GitLab using each
|
||||
provider's existing raw-file endpoint + the **safe HTTP client**. Cap at 64 KiB.
|
||||
Distinguish 404 (file absent) from transport/5xx errors.
|
||||
- [ ] **`fetch.go`**: `Fetch(ctx, deps, w) (Result, error)` where
|
||||
`Result{ Raw []byte, Spec Spec, CommitSHA string, Status string /* ok|no_file|fetch_failed */ }`.
|
||||
Decrypt `access_token`, build provider via `NewGitProvider`, `GetLatestCommitSHA`, then
|
||||
`DownloadFile(gitops_path)`. Missing file → `no_file` (NOT an error). All errors routed
|
||||
through the existing `sanitizeError(msg, token)` so the token never leaks.
|
||||
- [ ] **Unit tests** (`*_test.go`): ParseSpec (valid/unknown-key/bad-version);
|
||||
MergeAndValidate (omitted-field preserved, invalid merged config rejected, no clobber);
|
||||
Drift (declared-only, deploy_strategy normalization, numeric coercion, no false positive
|
||||
on undeclared keys); a redaction test mirroring `static/helpers_test.go`.
|
||||
|
||||
## Verify
|
||||
|
||||
- `go build ./...`, `go vet ./internal/...`, `go test ./internal/...` green.
|
||||
|
||||
## Handoff notes
|
||||
|
||||
_(filled after implementation)_
|
||||
@@ -0,0 +1,35 @@
|
||||
# Phase 2 — Store + API (manual sync, explicit-action)
|
||||
|
||||
## Tasks
|
||||
|
||||
- [ ] **Store setters** (`internal/store/workloads.go` or a new `gitops.go`):
|
||||
- `SetWorkloadGitOps(id string, enabled bool, path string) error` — gated to
|
||||
dockerfile/static at the API layer.
|
||||
- `RecordGitOpsSync(id, commitSHA, syncedAt string) error`.
|
||||
- All writes re-read the row first / use targeted column updates (avoid full-row
|
||||
clobber races — review S5).
|
||||
- [ ] **Sync audit** (NOT deploy_history): a small `gitops_sync_audit` table
|
||||
(`id, workload_id, outcome, commit_sha, drift_count, error, created_at`) with an insert
|
||||
helper. Errors stored as generic markers only (secret-safe). _(Or reuse the event log if
|
||||
cleaner — pick one and note it.)_
|
||||
- [ ] **API handlers** (`internal/api/gitops.go`, wired in `internal/api/router.go`):
|
||||
- `GET /api/workloads/{id}/gitops` → `{ enabled, path, status, raw, parsed, commit_sha,
|
||||
last_sync_at, drift_count }` (calls `gitops.Fetch` + `gitops.Drift`).
|
||||
- `GET /api/workloads/{id}/gitops/drift` → `[]DriftEntry`.
|
||||
- `POST /api/workloads/{id}/gitops/sync` (`auth.AdminOnly`) → `Fetch` →
|
||||
`MergeAndValidate` → `UpdateWorkload` (single txn) → `RecordGitOpsSync` + audit.
|
||||
Returns the applied summary. Secret-safe errors.
|
||||
- `PUT /api/workloads/{id}/gitops` (`auth.AdminOnly`) → enable/disable + path; **reject
|
||||
if source_kind ∉ {dockerfile, static}** with a clear 400.
|
||||
- [ ] **Validation**: path must be a repo-relative file (no `..`, no leading `/`, sane
|
||||
length); `enabled` only when the source is git-backed and has repo coords.
|
||||
|
||||
## Verify
|
||||
|
||||
- `go build ./...`, `go vet ./internal/...`, `go test ./internal/...` green.
|
||||
- Handler tests: admin-gate on sync/put, no_file path, secret-safe error on a failed
|
||||
fetch, drift_count surfaced, non-git source rejected by PUT.
|
||||
|
||||
## Handoff notes
|
||||
|
||||
_(filled after implementation)_
|
||||
@@ -0,0 +1,38 @@
|
||||
# Phase 3 — Frontend experience (frontend-design + UI/UX agent showcase)
|
||||
|
||||
Built by the **frontend implementer agent** under the frontend-design skill. Must follow
|
||||
project conventions: Svelte 5 runes, `ToggleSwitch` for booleans, `ConfirmDialog` for the
|
||||
sync action, `$t` with **en+ru parity**, the `.panel` vocabulary from `DeployHistoryPanel`.
|
||||
|
||||
## Tasks
|
||||
|
||||
- [ ] **`web/src/lib/api.ts`**: `GitOpsStatus` + `DriftEntry` interfaces; `fetchWorkloadGitOps(id)`,
|
||||
`fetchWorkloadDrift(id)`, `syncWorkloadGitOps(id)`, `setWorkloadGitOps(id, {enabled, path})`
|
||||
following the existing `get<T>`/`post<T>` typed-fetcher pattern (mirror `fetchWorkloadDeploys`).
|
||||
- [ ] **`GitOpsPanel.svelte`** (mounted on `apps/[id]` near the other panels): the
|
||||
centerpiece. Sections:
|
||||
- Header: title + status pill (`synced` / `N changes` / `no file` / `fetch failed`) +
|
||||
last-sync/commit meta + enable/disable `ToggleSwitch`.
|
||||
- **Drift view** — the design focus. For each declared field show repo-value vs
|
||||
live-value with a clean/changed state. Distinctive, legible, on-brand (forge tokens,
|
||||
`--forge-mono`, the `--color-warning`/`--color-success` hues already used). No diff
|
||||
library exists — design a purpose-built field-level diff (NOT a generic `<pre>` dump).
|
||||
- Rendered `.tinyforge.yml` preview (the `.code-area`/editor frame vocabulary).
|
||||
- "Sync now" button → `ConfirmDialog` ("apply repo config to live") → `syncWorkloadGitOps`
|
||||
→ toast + refresh. Admin-only affordance.
|
||||
- `no_file` / `fetch_failed` empty states (clear, not alarming).
|
||||
- [ ] **GitOps-managed badge** on apps list rows (`apps/+page.svelte`, only dockerfile/static)
|
||||
and the detail hero — reuse the `.badge` chip vocabulary.
|
||||
- [ ] **Read-only gate** on the source-config edit form: when managed, lock exactly the
|
||||
fields the synced overlay declares (derive from the drift/parsed payload) + a banner
|
||||
("managed by `.tinyforge.yml` — edit the file and sync").
|
||||
- [ ] **i18n**: `apps.detail.gitops.*` in BOTH `en.json` and `ru.json` (verify parity).
|
||||
|
||||
## Verify
|
||||
|
||||
- `npm run check` (0 errors), `npm run build`, `npm run test` green; i18n key parity equal.
|
||||
- Restart dev server (`./scripts/dev-server.sh`).
|
||||
|
||||
## Handoff notes
|
||||
|
||||
_(filled after implementation)_
|
||||
@@ -0,0 +1,25 @@
|
||||
# Phase 4 — Hardening + docs + final review
|
||||
|
||||
## Tasks
|
||||
|
||||
- [ ] **Concurrent-sync guard** (review S5): per-workload sync mutex (or re-read-then-apply
|
||||
with a compare) so a sync racing the edit-form save / another sync can't silently lose
|
||||
writes.
|
||||
- [ ] **File-size + path hardening**: confirm the 64 KiB `DownloadFile` cap is enforced
|
||||
across all three providers; confirm `gitops_path` validation rejects traversal.
|
||||
- [ ] **Security-reviewer pass**: SSRF (verify the fetch goes through `NewGitProvider`/the
|
||||
safe client, never raw `http.Get`), secret handling (token never logged/persisted/leaked
|
||||
in errors — `sanitizeError`), admin-gating on sync + put.
|
||||
- [ ] **Docs**: `docs/gitops.md` (or extend `docs/plans/`): the `.tinyforge.yml` v1 schema
|
||||
reference, how to enable, the sync flow, and an explicit **"not in v1"** section
|
||||
(env/faces, auto-apply-on-deploy, multi-workload Framing B) with the future seams noted.
|
||||
- [ ] **Final comprehensive review** + (if triggered) security review, then present for the
|
||||
merge gate.
|
||||
|
||||
## Verify
|
||||
|
||||
- Full backend + frontend build/test/vet green; dev server healthy on :8090.
|
||||
|
||||
## Handoff notes
|
||||
|
||||
_(filled after implementation)_
|
||||
Reference in New Issue
Block a user