tiny-forge/plans/volume-snapshot-restore/phase-2-lifecycle-locking.md

Phase 2: Engine.Restore orchestration + lifecycle/locking + rollback

Status: ✅ Complete Parent plan: PLAN.md Domain: backend

Objective

Wire the Phase 1 primitives into the full stop → swap → redeploy sequence under a per-workload lock, with crash-safe rollback (journal + recovery sweep) and a durable pre-restore auto-capture. Define the Lifecycle seam; modify the Deployer for per-workload locking + an unlocked redeploy.

Tasks

• internal/keyedmutex/keyedmutex.go — extract the gitops.go pattern into a shared package: type Mutex with Lock(key string) func() and TryLock(key string) (func(), bool) (the Try variant serves the Phase 3 API single-flight → 409). Unit test both.
• Deployer locking (C1) in internal/deployer/:
  • add workloadLocks keyedmutex.Mutex field.
  • refactor DispatchPlugin → unlock := d.workloadLocks.Lock(w.ID); defer unlock(); return d.dispatchLocked(ctx, w, intent); move the current body into unexported dispatchLocked.
  • wrap DispatchTeardown in the same per-workload lock.
  • do NOT lock DispatchReconcile (periodic; image Reconcile is a no-op; reconciler markMissingRows only flips labels = benign; locking it would stall the reconcile loop behind long deploys).
  • expose func (d *Deployer) LockWorkload(id string) func() and func (d *Deployer) RedeployLocked(ctx, w, intent) error (= dispatchLocked, doc: "caller already holds the workload lock; calling DispatchPlugin would deadlock").
• volsnap.Lifecycle interface (in volsnap):
  • Lock(workloadID string) func()
  • StopContainers(ctx, workloadID string) (runningTag string, err error) — stop every running container for the workload; return the newest-running container's ImageTag (so redeploy pins the same version; empty ⇒ source default). Mark stopped rows State="stopped".
  • Redeploy(ctx, w store.Workload, reference string) error — unlocked re-dispatch, Reason "restore", Reference=tag.
• Engine.Restore(ctx, snapshotID, workloadID string) error in internal/volsnap/restore.go (engine owns it). Sequence — does NOT hold e.mu (R1):
  1. load snap; verify snap.WorkloadID == workloadID; load workload + settings; require source_kind=="image".
  2. parseManifest; preflightResolve (C3 — abort if any fails); archiveUncompressedSize + per-filesystem freeDiskBytes pre-check (C5/R4 — abort).
  3. unlock := lc.Lock(workloadID); defer unlock() (C1).
  4. re-validate the workload still exists (R4 — teardown may have won the lock); abort if gone.
  5. tag, _ := lc.StopContainers(ctx, workloadID) (C4 stop).
  6. durably capture pre-restore snapshot: e.Create(w, settings, "pre-restore") (folded; AFTER stop = quiesced; BEFORE any rename = R3). Create takes its own e.mu — Restore must hold none.
  7. write restore journal <snapDir>/restore-<workloadID>.json (snapshotID, per-volume {live, old, tmp, swapped:false}).
  8. extract ALL volumes to their tmp staging dirs (safeExtractIndex) — R3 (shrinks the destructive window to pure renames).
  9. swap each volume (swapVolumeDir), updating the journal swapped=true per volume.
  10. on ANY error in 8–9 → rollbackSwaps + lc.Redeploy(ctx, w, tag) + delete journal + return wrapped error.
  11. success → lc.Redeploy(ctx, w, tag) (C4 redeploy); remove .old staging dirs (reclaim disk); delete journal; best-effort audit event (store.InsertEvent source "volsnap").
  • Engine.SetLifecycle(lc Lifecycle) setter; Restore errors clearly if lifecycle is nil.
• Engine.RecoverInterruptedRestores() (int, error) (R3) — startup sweep, mirrors CleanOrphans: for each restore-*.json journal, per volume: if swapped → remove old+tmp; else if live missing && old exists → rename old→live (revert mid-rename crash), remove tmp; else (live present, not swapped) → remove tmp. Delete journal. Log loudly. (Wiring at startup happens in Phase 3's main.go change, beside CleanOrphans.)

Files to Modify/Create

• internal/keyedmutex/keyedmutex.go (+ _test.go) — shared lock (new)
• internal/deployer/deployer.go, internal/deployer/dispatch.go — workloadLocks, dispatchLocked, LockWorkload, RedeployLocked, locked Teardown
• internal/volsnap/restore.go — Lifecycle interface, Engine.Restore, RecoverInterruptedRestores, SetLifecycle, journal type
• internal/volsnap/restore_test.go — fake-Lifecycle orchestration tests (extends Phase 1 file)
• internal/api/gitops.go — (optional, low-risk) migrate keyedMutex→keyedmutex.Mutex for DRY

Acceptance Criteria

• Lock re-entrancy: Engine.Restore → RedeployLocked does NOT re-acquire the workload lock (no deadlock). All existing deployer tests still pass (lock is externally transparent).
• Happy-path orchestration test uses the REAL Engine.Create (real store + t.TempDir()) for the pre-restore capture so the e.mu deadlock (R1) would fail go test, not prod. Asserts call order: preflight → lock → stop → create → extract-all → swap-all → redeploy → cleanup.
• Rollback test: a swap fails midway → originals restored, redeploy called, journal deleted, error returned.
• Preflight-fail test: lock/stop NEVER called (abort before lock).
• Disk-pre-check-fail test: abort before lock.
• RecoverInterruptedRestores test: simulate journals in each crash state → correct revert/keep/cleanup.
• go build ./..., go vet ./internal/..., go test ./internal/... green.

Notes

• ⚠️ The Deployer lock change touches the hot deploy path — verify no existing path re-enters DispatchPlugin under a held lock (webhook preview = sequential teardown-then-deploy on the child, not nested — confirmed safe).
• The API single-flight (Phase 3) is a fast 409 reject; the deployer lock is the real mutex — they compose (document).

Review Checklist

• All tasks completed
• Code follows project conventions
• No unintended side effects (existing deploy/teardown behavior unchanged externally)
• Build passes
• Tests pass (new + existing)

Handoff to Next Phase

Implemented: internal/keyedmutex (Lock+TryLock, tested); deployer workloadLocks + dispatchLocked + LockWorkload + RedeployLocked, DispatchPlugin/DispatchTeardown now per-workload-locked (reconciler intentionally NOT). volsnap.Lifecycle interface, Engine.Restore, restoreJournal (atomic write — W1), RecoverInterruptedRestores, recoverVolume, checkDiskSpace, SetLifecycle. Tests: restore_engine_test.go (happy/real-Create, redeploy-fail, preflight-abort, extract-fail-after-lock, nil-lifecycle, wrong-workload, recovery×3 states), keyedmutex_test.go. Full go test ./internal/... green.

Review (go-reviewer, APPROVE WITH NOTES): no functional blockers in this diff. Verified: no lock re-entrancy/e.mu self-deadlock, no prune-race (extract-all precedes e.Create), recovery state machine doesn't revert good data. Addressed in-phase: W1 (atomic journal), W3 (extract-failure orchestration test). Residual W3 (mid-swap fault injection) accepted.

🔴 HARD PREREQUISITES for Phase 3 (B1 + N1 from review):

1. Wire snapshotEngine.RecoverInterruptedRestores() at startup in cmd/server/main.go, BEFORE the API server serves — beside the existing CleanOrphans() call (~main.go:333). Without it the journal/WAL protects nothing — a crash mid-restore is unrecovered.
2. Wire snapshotEngine.SetLifecycle(adapter) strictly BEFORE serving (same place as SetSnapshotEngine) so the e.lifecycle field is safely published (no race).
3. The restore endpoint MUST NOT be reachable until both are wired.

Lifecycle adapter (Phase 3, main.go) maps: Lock→deployer.LockWorkload; StopContainers→store.ListContainersByWorkload + docker.StopContainer each running + UpdateContainerState(...,"stopped") + return newest-running ImageTag; Redeploy→deployer.RedeployLocked with a restore-reason intent (Reference=tag).