feat(cutover): hard legacy cutover — drop projects/stacks/sites/deploys
Build / build (push) Successful in 10m39s
Build / build (push) Successful in 10m39s
The clean-break delete that closes the workload-first refactor arc.
Net diff: ~30 backend files deleted, ~20 modified, ~12k LOC removed
on the Go side; entire /projects /stacks /sites /deploy frontend
trees gone; ~6.7k LOC removed on the Svelte/TypeScript side.
Backend
- API handlers gone: internal/api/{projects,stages,stage_env,stacks,
static_sites,deploys,instances,volume_browser}.go
- Store CRUD + tests gone: internal/store/{projects,stages,stage_env,
stacks,static_sites,static_site_secrets,deploys,poll_state,volumes,
workload_sync}.go (+ _test.go siblings)
- Legacy deployer pipeline gone: internal/deployer/{bluegreen,promote,
rollback,subdomain,resolver_test}.go; deployer.go trimmed to just the
dispatch surface used by the plugin pipeline
- internal/staticsite/{manager,healthcheck}.go and
internal/stack/manager.go gone (the rest of those packages stay as
helpers imported by the static + compose plugins)
- internal/registry/poller.go gone (legacy registry poller)
- internal/volume.ResolvePath gone; ResolveWorkloadPath stays
- internal/webhook: handleWebhook (project) + handleSiteWebhook (site)
gone; only POST /api/webhook/triggers/{secret} remains
- workload-side webhook URL handlers (getWorkloadWebhook +
regenerateWorkloadWebhook + EnsureWorkloadWebhookSecret +
SetWorkloadWebhookSecret + GetWorkloadByWebhookSecret) gone — they
minted URLs that would 404 against the new trigger-only ingress
- cmd/server/main.go: dropped staticsite.Manager, stack.Manager,
staticsite.HealthChecker, registry poller, SetSiteSyncTriggerer,
SetStaticSiteManager, SetStackManager, wireStaticBackend
- store/store.go: idempotent DROP TABLE IF EXISTS for every legacy
table (projects, stages, stage_env, volumes, deploys, deploy_logs,
poll_states, stacks, stack_revisions, stack_deploys, static_sites,
static_site_secrets); FK order children-then-parents
- store/models.go: dropped Project, Stage, Deploy, DeployLog, StageEnv,
Volume, StaticSite, StaticSiteSecret, Stack, StackRevision,
StackDeploy types; kept WorkloadKind constants as documented strings
- internal/store/helpers.go (new): BoolToInt, rowScanner,
GenerateWebhookSecret extracted from deleted CRUD files
- internal/api/secrets.go (new): forwards to store.GenerateWebhookSecret
so api + store paths share one secret-generation impl (no
panic-vs-UUID-fallback divergence)
- internal/reconciler/reconciler.go: dropped legacy stack-by-compose
+ static-site label paths; only canonical tinyforge.workload.id
dispatch remains
- providers (gitea_content/github_provider/gitlab_provider) gained
path-traversal rejection on every tree entry
- internal/webhook ParsedImage / ParseImageRef demoted to package-
private (no external callers)
Frontend
- /projects /stacks /sites /deploy routes deleted (entire trees)
- ProjectCard / InstanceCard / StaleContainerCard components deleted
- api.ts: dropped every project/stage/stack/site/deploy/instance
helper + types (Project, Stage, Stack, StaticSite, Deploy,
Instance, Volume, etc.); kept Workload, Container, App, Settings,
Registry, EventTrigger, LogScanRule, webhook envelopes
- WorkloadWebhook type + getWorkloadWebhook/regenerateWorkloadWebhook
api functions gone (mirror of the backend deletion above)
- web/src/routes/+layout.svelte: dropped /projects /sites /stacks
/deploy nav entries, trimmed quick-nav keymap
- web/src/routes/+page.svelte: dashboard rewrite — reads
listWorkloads + listContainers only; 4-card stat grid
(workloads/running/failed/stale) + recent workloads strip
- navCounts.ts, SystemHealthCard.svelte, ContainerLogs.svelte,
ContainerStats.svelte, StatusBadge.svelte, TagCombobox.svelte,
proxies/+page.svelte, containers/+page.svelte all rewired to the
workload-first surface
- AbortController plumbing on dashboard, nav-counts, stale page,
SystemHealthCard so navigation doesn't leave dangling fetches
- i18n: dropped projects.*, projectDetail.*, envEditor.*,
volumeEditor.*, volumeBrowser.*, quickDeploy.*, sites.*, stacks.*,
instance.*, confirm.* namespaces; en/ru parity preserved (1042
keys each)
Hardening from go-reviewer + security-reviewer + typescript-reviewer
subagent passes (0 CRITICAL across all three; 1 HIGH + ~12 MEDIUM
addressed inline before commit):
- Sec H1: dead-end workload webhook URL handlers (would mint URLs
that 404 the new trigger-only ingress) deleted across backend +
frontend
- Go M1: IsTerminalDeployStatus dropped (no production callers)
- Go M2: ParsedImage/ParseImageRef lowercased (in-package only)
- Go M6: generateWebhookSecret unified — api shim forwards to
store.GenerateWebhookSecret
- Doc/comment freshness: stage_id (no longer FK), ProxyRoute legacy
field names, workloadIDRow rationale, webhook_deliveries.target_type
enum, WebhookDeliveryLog component header
Doc
- WORKLOAD_REFACTOR_TODO: cutover marked DONE; all three Priority 1
items are now shipped. Next focus is Priority 3 polish (apps.* i18n
+ codemap entries) and Priority 4 tests.
Behavioral notes for operators upgrading from a pre-cutover build
- Existing rows in the dropped tables disappear on first boot.
- Legacy webhook URLs at /api/webhook/{secret} and
/api/webhook/sites/{secret} return 404; CI configs must repoint to
/api/webhook/triggers/{secret} (the trigger-split boot backfill
lifted any embedded workload secret onto a Trigger row, so the
secret value itself carries over).
- Frontend routes /projects /stacks /sites /deploy are gone; nav
links replaced with /apps and /triggers.
This commit is contained in:
+59
-128
@@ -3,7 +3,6 @@ package api
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
@@ -14,17 +13,15 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
|
||||
"github.com/alexei/tinyforge/internal/store"
|
||||
)
|
||||
|
||||
// Limits and constants for the log endpoints.
|
||||
const (
|
||||
defaultLogTail = 200
|
||||
maxLogTail = 5000
|
||||
maxJSONLogBytes = 4 << 20 // 4 MiB cap for non-streaming log responses
|
||||
maxLogLineBytes = 1 << 20 // 1 MiB max line length for the bufio.Scanner
|
||||
defaultLogTail = 200
|
||||
maxLogTail = 5000
|
||||
maxJSONLogBytes = 4 << 20 // 4 MiB cap for non-streaming log responses
|
||||
maxLogLineBytes = 1 << 20 // 1 MiB max line length for the bufio.Scanner
|
||||
logHeartbeatPeriod = 20 * time.Second
|
||||
)
|
||||
|
||||
@@ -37,82 +34,8 @@ var (
|
||||
ctlBytePattern = regexp.MustCompile(`[\x00-\x08\x0b-\x1a\x1c-\x1f\x7f]`)
|
||||
)
|
||||
|
||||
// listProjectImages handles GET /api/projects/{id}/images.
|
||||
// Returns all local Docker images matching the project's image reference.
|
||||
func (s *Server) listProjectImages(w http.ResponseWriter, r *http.Request) {
|
||||
id := chi.URLParam(r, "id")
|
||||
|
||||
project, err := s.store.GetProjectByID(id)
|
||||
if err != nil {
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
respondNotFound(w, "project")
|
||||
return
|
||||
}
|
||||
slog.Error("failed to get project", "error", err)
|
||||
respondError(w, http.StatusInternalServerError, "internal server error")
|
||||
return
|
||||
}
|
||||
|
||||
if s.docker == nil || project.Image == "" {
|
||||
respondJSON(w, http.StatusOK, []any{})
|
||||
return
|
||||
}
|
||||
|
||||
images, err := s.docker.ListImagesByRef(r.Context(), project.Image)
|
||||
if err != nil {
|
||||
slog.Warn("list project images", "project", project.Name, "error", err)
|
||||
respondJSON(w, http.StatusOK, []any{})
|
||||
return
|
||||
}
|
||||
|
||||
respondJSON(w, http.StatusOK, images)
|
||||
}
|
||||
|
||||
// streamContainerLogs handles GET /api/projects/{id}/stages/{stage}/instances/{iid}/logs.
|
||||
// Streams container logs via SSE. {iid} is the container row ID. Ownership is
|
||||
// verified by joining through workload + stage so an attacker cannot stream
|
||||
// logs for a foreign container by guessing IDs under the wrong project URL.
|
||||
func (s *Server) streamContainerLogs(w http.ResponseWriter, r *http.Request) {
|
||||
projectID := chi.URLParam(r, "id")
|
||||
stageID := chi.URLParam(r, "stage")
|
||||
containerRowID := chi.URLParam(r, "iid")
|
||||
|
||||
c, err := s.store.GetContainerByID(containerRowID)
|
||||
if err != nil {
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
respondNotFound(w, "container")
|
||||
return
|
||||
}
|
||||
slog.Error("failed to get container", "error", err)
|
||||
respondError(w, http.StatusInternalServerError, "internal server error")
|
||||
return
|
||||
}
|
||||
|
||||
wl, err := s.store.GetWorkloadByID(c.WorkloadID)
|
||||
if err != nil {
|
||||
respondNotFound(w, "container")
|
||||
return
|
||||
}
|
||||
stage, err := s.store.GetStageByID(stageID)
|
||||
if err != nil || stage.ProjectID != projectID {
|
||||
respondNotFound(w, "container")
|
||||
return
|
||||
}
|
||||
if wl.Kind != string(store.WorkloadKindProject) || wl.RefID != projectID || c.Role != stage.Name {
|
||||
respondNotFound(w, "container")
|
||||
return
|
||||
}
|
||||
|
||||
if c.ContainerID == "" {
|
||||
respondError(w, http.StatusBadRequest, "container row has no docker container bound")
|
||||
return
|
||||
}
|
||||
|
||||
s.streamLogsForContainer(w, r, c.ContainerID)
|
||||
}
|
||||
|
||||
// streamLogsForContainer streams logs for an arbitrary container ID using the
|
||||
// shared SSE/JSON dual-mode pattern. Owner-specific handlers (instance, site)
|
||||
// shared SSE/JSON dual-mode pattern. Owner-specific handlers (workload-container)
|
||||
// should validate ownership and then delegate here.
|
||||
func (s *Server) streamLogsForContainer(w http.ResponseWriter, r *http.Request, containerID string) {
|
||||
if s.docker == nil {
|
||||
@@ -255,11 +178,7 @@ func sanitizeDockerLogLine(line string) string {
|
||||
// by any container, computed in a single DB pass against the normalized
|
||||
// containers index. Returning an error (rather than swallowing) prevents
|
||||
// prune logic from treating a transient DB failure as "nothing is active".
|
||||
func buildActiveImagesSet(st *store.Store, projects []store.Project) (map[string]bool, error) {
|
||||
// `projects` is unused now — kept in the signature for back-compat with
|
||||
// callers that already happen to have the slice. The image_ref column
|
||||
// holds the full "image:tag" string written by the deployer.
|
||||
_ = projects
|
||||
func buildActiveImagesSet(st *store.Store) (map[string]bool, error) {
|
||||
containers, err := st.ListContainers(store.ContainerFilter{})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("list containers: %w", err)
|
||||
@@ -274,8 +193,43 @@ func buildActiveImagesSet(st *store.Store, projects []store.Project) (map[string
|
||||
return active, nil
|
||||
}
|
||||
|
||||
// unusedImageStats handles GET /api/docker/unused-images.
|
||||
// Returns the total size of unused project images and whether the threshold is exceeded.
|
||||
// workloadImageBases returns the set of "image" strings (no tag) that
|
||||
// some workload currently mounts to, derived from container.image_ref.
|
||||
// This replaces the legacy "list all projects → projects[].Image" view
|
||||
// after the workload-first cutover.
|
||||
func workloadImageBases(st *store.Store) (map[string]bool, error) {
|
||||
containers, err := st.ListContainers(store.ContainerFilter{})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("list containers: %w", err)
|
||||
}
|
||||
bases := make(map[string]bool, len(containers))
|
||||
for _, c := range containers {
|
||||
if c.ImageRef == "" {
|
||||
continue
|
||||
}
|
||||
ref, _ := splitImageTag(c.ImageRef)
|
||||
if ref != "" {
|
||||
bases[ref] = true
|
||||
}
|
||||
}
|
||||
return bases, nil
|
||||
}
|
||||
|
||||
// splitImageTag splits "image:tag" into image and tag parts. Returns the
|
||||
// full string and empty tag if no colon separator is found. Inlined here
|
||||
// because the legacy deploys.go that owned it was removed.
|
||||
func splitImageTag(ref string) (string, string) {
|
||||
if idx := strings.LastIndex(ref, ":"); idx != -1 {
|
||||
afterColon := ref[idx+1:]
|
||||
if !strings.Contains(afterColon, "/") {
|
||||
return ref[:idx], afterColon
|
||||
}
|
||||
}
|
||||
return ref, ""
|
||||
}
|
||||
|
||||
// unusedImageStats handles GET /api/docker/unused-images. Returns the total
|
||||
// size of unused workload images and whether the threshold is exceeded.
|
||||
func (s *Server) unusedImageStats(w http.ResponseWriter, r *http.Request) {
|
||||
if s.docker == nil {
|
||||
respondJSON(w, http.StatusOK, map[string]any{
|
||||
@@ -291,32 +245,25 @@ func (s *Server) unusedImageStats(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
|
||||
projects, err := s.store.GetAllProjects()
|
||||
imageBases, err := workloadImageBases(s.store)
|
||||
if err != nil {
|
||||
slog.Error("unused images: list projects", "error", err)
|
||||
slog.Error("unused images: list workload images", "error", err)
|
||||
respondError(w, http.StatusInternalServerError, "internal server error")
|
||||
return
|
||||
}
|
||||
|
||||
// Build set of active image refs in one DB pass instead of N×K queries.
|
||||
// A flaky read here previously masqueraded as "no images are active",
|
||||
// which on the prune endpoint would have deleted *running* images.
|
||||
activeImages, err := buildActiveImagesSet(s.store, projects)
|
||||
activeImages, err := buildActiveImagesSet(s.store)
|
||||
if err != nil {
|
||||
slog.Error("unused images: build active set", "error", err)
|
||||
respondError(w, http.StatusInternalServerError, "internal server error")
|
||||
return
|
||||
}
|
||||
|
||||
// Sum unused image sizes.
|
||||
ctx := r.Context()
|
||||
var totalSize int64
|
||||
var count int
|
||||
for _, p := range projects {
|
||||
if p.Image == "" {
|
||||
continue
|
||||
}
|
||||
images, err := s.docker.ListImagesByRef(ctx, p.Image)
|
||||
for base := range imageBases {
|
||||
images, err := s.docker.ListImagesByRef(ctx, base)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
@@ -339,69 +286,53 @@ func (s *Server) unusedImageStats(w http.ResponseWriter, r *http.Request) {
|
||||
})
|
||||
}
|
||||
|
||||
// pruneImages handles POST /api/docker/prune-images.
|
||||
// Only removes images that belong to Tinyforge projects (not all system images).
|
||||
// pruneImages handles POST /api/docker/prune-images. Only removes images that
|
||||
// some workload references (via container.image_ref), never arbitrary host
|
||||
// images.
|
||||
func (s *Server) pruneImages(w http.ResponseWriter, r *http.Request) {
|
||||
if s.docker == nil {
|
||||
respondError(w, http.StatusServiceUnavailable, "Docker is not available")
|
||||
return
|
||||
}
|
||||
|
||||
// Collect all image references from our projects.
|
||||
projects, err := s.store.GetAllProjects()
|
||||
imageBases, err := workloadImageBases(s.store)
|
||||
if err != nil {
|
||||
slog.Error("prune: failed to list projects", "error", err)
|
||||
slog.Error("prune: list workload images", "error", err)
|
||||
respondError(w, http.StatusInternalServerError, "internal server error")
|
||||
return
|
||||
}
|
||||
|
||||
// Build a set of image refs used by active instances. Bail out on error
|
||||
// — silently treating a DB blip as "no active images" would prune
|
||||
// images currently in use by running containers.
|
||||
activeImages, err := buildActiveImagesSet(s.store, projects)
|
||||
activeImages, err := buildActiveImagesSet(s.store)
|
||||
if err != nil {
|
||||
slog.Error("prune: build active set", "error", err)
|
||||
respondError(w, http.StatusInternalServerError, "internal server error")
|
||||
return
|
||||
}
|
||||
|
||||
// Collect all unique image bases from projects (without tags).
|
||||
projectImages := make(map[string]bool)
|
||||
for _, p := range projects {
|
||||
if p.Image != "" {
|
||||
projectImages[p.Image] = true
|
||||
}
|
||||
}
|
||||
|
||||
if len(projectImages) == 0 {
|
||||
if len(imageBases) == 0 {
|
||||
respondJSON(w, http.StatusOK, map[string]any{
|
||||
"images_removed": 0,
|
||||
"space_reclaimed_mb": 0,
|
||||
"message": "No project images to clean up",
|
||||
"message": "No workload images to clean up",
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
// List all local Docker images and find ones matching our projects but not actively used.
|
||||
ctx := r.Context()
|
||||
removed := 0
|
||||
var reclaimedBytes int64
|
||||
|
||||
for imageBase := range projectImages {
|
||||
// List all tags for this image.
|
||||
images, err := s.docker.ListImagesByRef(ctx, imageBase)
|
||||
for base := range imageBases {
|
||||
images, err := s.docker.ListImagesByRef(ctx, base)
|
||||
if err != nil {
|
||||
slog.Warn("prune: list images", "image", imageBase, "error", err)
|
||||
slog.Warn("prune: list images", "image", base, "error", err)
|
||||
continue
|
||||
}
|
||||
|
||||
for _, img := range images {
|
||||
// Skip images that are actively used by running instances.
|
||||
if activeImages[img.Ref] {
|
||||
continue
|
||||
}
|
||||
|
||||
// Remove unused image.
|
||||
if err := s.docker.RemoveImage(ctx, img.ID); err != nil {
|
||||
slog.Warn("prune: remove image", "image", img.Ref, "error", err)
|
||||
continue
|
||||
|
||||
Reference in New Issue
Block a user