tiny-forge/docs/plans/workload-refactor.md

# Workload Refactor — Compressed Plan

Status: Draft, pre-implementation
Owner: alexei.dolgolyov
Date: 2026-05-07

## Goal

Unify `Project`, `Stack`, and `StaticSite` under a single `Workload` primitive, and introduce a normalized `containers` index so every Tinyforge-managed container has one canonical row. This unblocks a global Containers view today and lets future workload kinds (cron jobs, one-shot tasks, databases-as-resource, functions) plug in without another tab/store/deployer branch.

## Why this is the compressed plan

The original 8-PR plan was designed for a live system with dual-writes and soak periods. Tinyforge has no production users yet, so all defenses against live runtime state collapse: no external label consumers, no third-party CI hitting webhook URLs, no orphaned containers to recover. Everything ships in 3 PRs against a clean slate. Solo-dev reversibility is preserved by branching, not by dual-write gymnastics.

## Target architecture

- `Workload` is the unifying primitive with `kind ∈ {project, stack, site, …}`. Each existing Project/Stack/StaticSite becomes a Workload row.
- `containers` is a normalized index: every Tinyforge-managed container has one row with `workload_id`, `workload_kind`, `role`, Docker container ID, host, state, last_seen.
- Optional `apps` table (thin nullable `app_id` on Workload) added empty; UI gated behind a feature flag, defer indefinitely until pull.
- Stable Docker labels: `tinyforge.workload.id`, `tinyforge.workload.kind`, `tinyforge.role`, `tinyforge.managed`. Legacy `tinyforge.project` / `tinyforge.stage` / `tinyforge.instance-id` are removed in the same wave.
- Global `/containers` UI route; per-workload container panel becomes a shared `<WorkloadContainers>` component reused by project, stack, and site detail pages.

## Schema

Appended to `internal/store/store.go::runMigrations()` as additive `CREATE TABLE` statements (idempotent via `CREATE TABLE IF NOT EXISTS`).

```sql
CREATE TABLE IF NOT EXISTS workloads (
    id              TEXT PRIMARY KEY,
    kind            TEXT NOT NULL,                 -- 'project' | 'stack' | 'site'
    ref_id          TEXT NOT NULL,                 -- FK into projects/stacks/static_sites by kind
    name            TEXT NOT NULL,
    app_id          TEXT,                          -- nullable FK into apps.id
    notification_url        TEXT NOT NULL DEFAULT '',
    notification_secret     TEXT NOT NULL DEFAULT '',
    webhook_secret          TEXT NOT NULL DEFAULT '',
    webhook_signing_secret  TEXT NOT NULL DEFAULT '',
    webhook_require_signature INTEGER NOT NULL DEFAULT 0,
    created_at      TEXT NOT NULL,
    updated_at      TEXT NOT NULL,
    UNIQUE(kind, ref_id)
);
CREATE INDEX IF NOT EXISTS idx_workloads_app_id ON workloads(app_id);
CREATE INDEX IF NOT EXISTS idx_workloads_kind   ON workloads(kind);

CREATE TABLE IF NOT EXISTS containers (
    id              TEXT PRIMARY KEY,
    workload_id     TEXT NOT NULL,
    workload_kind   TEXT NOT NULL,                 -- denormalized for filtered queries
    role            TEXT NOT NULL,                 -- stage name (project), service name (stack), '' (site)
    container_id    TEXT NOT NULL DEFAULT '',      -- Docker ID, '' between create+start
    image_ref       TEXT NOT NULL DEFAULT '',
    host            TEXT NOT NULL DEFAULT 'local',
    state           TEXT NOT NULL DEFAULT '',      -- running | stopped | failed | removing | missing
    port            INTEGER NOT NULL DEFAULT 0,
    last_seen_at    TEXT NOT NULL DEFAULT '',
    extra_json      TEXT NOT NULL DEFAULT '{}',    -- {subdomain, npm_proxy_id, proxy_route_id, ...}
    created_at      TEXT NOT NULL,
    updated_at      TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_containers_workload     ON containers(workload_id);
CREATE INDEX IF NOT EXISTS idx_containers_state        ON containers(state);
CREATE INDEX IF NOT EXISTS idx_containers_container_id ON containers(container_id);

CREATE TABLE IF NOT EXISTS apps (
    id          TEXT PRIMARY KEY,
    name        TEXT NOT NULL,
    description TEXT NOT NULL DEFAULT '',
    created_at  TEXT NOT NULL,
    updated_at  TEXT NOT NULL
);
```

`extra_json` carries kind-specific fields (`subdomain`, `npm_proxy_id`, `proxy_route_id`) so the spine stays narrow. SQLite JSON1 is required for queries against `extra_json`; verify the driver in `go.mod` supports it before committing — fall back to dedicated columns if not.

## PR 1 — Spine: schema, Workload package, reconciler

Single PR, lands the data layer end-to-end. No dual-writes; project/stack/site CRUD writes directly to `workloads`.

### New files

- `internal/store/workloads.go` — `CreateWorkload`, `GetWorkloadByID`, `GetWorkloadByRef(kind, refID)`, `ListWorkloads`, `UpdateWorkload`, `DeleteWorkload`.
- `internal/store/containers.go` — `UpsertContainer`, `GetContainerByDockerID`, `ListContainersByWorkload`, `ListContainers(filter)`, `MarkContainerMissing`, new `ListProxyRoutes` (mirrors the join shape from `internal/store/instances.go::ListProxyRoutes`, reading `extra_json` via `json_extract`).
- `internal/store/apps.go` — minimal CRUD; not wired anywhere yet.
- `internal/workload/workload.go` — `Workload` interface (`ID`, `Kind`, `Name`, `Deploy`, `Stop`, `Start`, `Delete`, `Containers`).
- `internal/workload/adapters/project_adapter.go` — wraps `internal/deployer`.
- `internal/workload/adapters/stack_adapter.go` — wraps `internal/stack/manager.go`.
- `internal/workload/adapters/site_adapter.go` — wraps `internal/staticsite/manager.go`.
- `internal/reconciler/reconciler.go` — single writer to `containers`. Reads `docker ps --filter label=tinyforge.managed`, groups by `(workload.id, role)`, upserts rows, marks absent rows `state='missing'`. Boot-time one-shot run + 30s tick.
- `internal/reconciler/reconciler_test.go` — table-driven tests with a fake Docker client.

### Modified files

- `internal/store/store.go::runMigrations` — append the three `CREATE TABLE` statements (after line ~165 where the existing migrations end).
- `internal/store/models.go` — add `Workload`, `Container`, `App` structs.
- `internal/store/projects.go` — `CreateProject`, `UpdateProject`, `DeleteProject` wrap the write in `s.db.Begin()` and also write the matching `workloads` row. Webhook/notification secret setters update `workloads.webhook_secret` / `webhook_signing_secret` / `notification_secret` directly.
- `internal/store/stacks.go` — same Workload write on `CreateStack` / `UpdateStack` / `DeleteStack`.
- `internal/store/static_sites.go` — same.
- `internal/docker/client.go` — add label constants `LabelWorkloadID`, `LabelWorkloadKind`, `LabelRole`, `LabelManaged`. **Remove** the old `LabelProject`, `LabelStage`, `LabelInstanceID` writes from the deployer.
- `internal/deployer/deployer.go` (label injection ~line 388) — emit only the new labels.
- `internal/deployer/bluegreen.go` (~line 97) — same.
- `internal/stack/manager.go` — after `docker compose up`, stamp new labels on each compose-managed container via `docker container update --label-add`. Compose's own `com.docker.compose.service` becomes `role`.
- `internal/staticsite/manager.go` — stamp new labels at container start.
- `internal/store/instances.go` — **delete this file**. The deployer no longer creates instance rows; reconciler owns container state.
- `internal/api/instances.go` — **delete or alias** to `/api/containers` filtered by workload. Solo dev → delete is cleaner.
- `internal/api/proxies.go` — switch the `ListProxyRoutes` import to `containers.ListProxyRoutes`.
- `internal/api/docker.go::buildActiveImagesSet` (~line 251) — replace the `ListAllInstances` walk with a single `containers.image_ref` query.
- `internal/api/stale.go`, `internal/stale/scanner.go` — read from `containers` instead of `instances`.
- `internal/webhook/matcher.go` — query `workloads.webhook_secret` directly.
- `cmd/server/main.go` — start the reconciler goroutine after `store.New`. Drop any startup code that touched `instances`.

### Tests

- Extend `internal/store/store_test.go` with `TestCreateProjectAlsoCreatesWorkload`, `TestDeleteProjectCascadesWorkload`, `TestUpsertContainerIdempotent`, `TestListProxyRoutesShape`.
- New `internal/reconciler/reconciler_test.go` with a `dockerClient` interface and a fake — assert that a slice of `types.Container` produces the expected `containers` upserts.
- Run the existing test suite under `-race`.

### Deliverable

System builds, deploys a project end-to-end, deploys a stack end-to-end, deploys a static site end-to-end. `containers` table reflects reality after each deploy and after a 30s reconciler tick. The legacy `instances` table is gone.

## PR 2 — API + frontend

### New files

- `internal/api/workloads.go` — `GET /api/workloads`, `GET /api/workloads/{id}`, `GET /api/workloads/{id}/containers`, `PATCH /api/workloads/{id}` (sets `app_id` and notification/webhook config).
- `internal/api/containers.go` — `GET /api/containers?workload_id=&kind=&state=&app_id=`, `GET /api/containers/{id}`.
- `internal/api/apps.go` — `GET /api/apps`, `POST /api/apps`, `PATCH /api/apps/{id}`, `DELETE /api/apps/{id}` (gated by settings flag `features.apps_grouping=true`).
- `web/src/routes/containers/+page.svelte` — global filterable table. Reuses table patterns from `web/src/routes/proxies/+page.svelte` and `web/src/routes/containers/stale/+page.svelte` (the existing `stale/` route stays untouched).
- `web/src/lib/components/WorkloadContainers.svelte` — shared container panel. Takes `workloadId` prop, hits `/api/workloads/{id}/containers`. Handles 1..N container rows.

### Modified files

- `internal/api/router.go` — register the new endpoints. Remove `/api/instances` registration.
- `web/src/routes/projects/[id]/+page.svelte` — replace the inline instance list with `<WorkloadContainers workloadId={...}/>`.
- `web/src/routes/stacks/[id]/+page.svelte` — same.
- `web/src/routes/sites/[id]/+page.svelte` — same.
- Top nav component (find under `web/src/lib/components/`) — insert a "Containers" tab between "Projects" and "Stacks". Existing tabs stay.
- `web/src/lib/api.ts` (or wherever API client functions live) — add `listWorkloads`, `getWorkload`, `listContainers`, `getContainer`, `listApps`. Remove instance-shaped helpers.
- `web/src/lib/types.ts` — add `Workload`, `Container`, `App` types. Remove `Instance` once unreferenced.

### Deliverable

User-visible: a `Containers` tab in the top nav showing every running container with kind/state/workload filters, links into the owning project/stack/site detail page, and a per-workload container panel that looks identical on all three detail pages.

## PR 3 — Polish + optional Apps UI

Defer indefinitely if no pull. Lands as a single PR when wanted.

### Scope

- Apps UI: `web/src/routes/apps/+page.svelte`, `[id]/+page.svelte`. Workload detail pages get an "App" dropdown to assign `app_id`. Gated by `features.apps_grouping=true` in settings.
- Drop any leftover dead code referencing `Instance` types.
- Documentation: update `CLAUDE.md` and `README.md` to describe the Workload model.
- Optional: consolidate `internal/deployer` and `internal/stack/manager` into a single orchestrator. **Out of scope for this refactor** — adapters wrap the existing kind-specific code and that's fine. Revisit only if the duplication starts hurting.

## What's explicitly deferred

- Deployer + stack-manager consolidation.
- Apps UI (schema added in PR 1, UI in PR 3 behind flag).
- Multi-host containers (`containers.host` exists but is always `'local'`).
- Workload-kind plugin model — the adapter registry has three hardcoded entries.
- Webhook secret handling for old per-project URLs that may already be in CI configs (no users yet → don't care).

## Risks (compressed)

- **SQLite JSON1 availability.** Verify the driver in `go.mod` supports `json_extract` before committing to `extra_json`. If not, hoist `subdomain`, `npm_proxy_id`, `proxy_route_id` to dedicated columns on `containers`.
- **`ListProxyRoutes` shape regression.** The new query reads from `containers` + `workloads` instead of `instances` + `projects` + `stages`. Worth a golden-output test before flipping `internal/api/proxies.go` over.
- **Stack containers and label stamping.** `docker container update --label-add` is required to label compose-managed containers post-up. If the local Docker engine version doesn't support it, fall back to relying on `com.docker.compose.project` + `com.docker.compose.service` for reconciler joins.
- **Boot-time backfill from `docker ps`.** First run needs to populate `containers` from currently-running containers using the legacy `tinyforge.instance-id` and `com.docker.compose.project` labels (since pre-refactor containers don't have the new labels). Solo-dev workaround: `docker compose down` test workloads, run the new binary against an empty Docker host, redeploy.

## Concrete file paths

Modified:
- `internal/store/store.go` (migrations at line ~75–165)
- `internal/store/projects.go`, `stacks.go`, `static_sites.go`, `models.go`, `store_test.go`
- `internal/docker/client.go`
- `internal/deployer/deployer.go` (~line 388), `internal/deployer/bluegreen.go` (~line 97)
- `internal/stack/manager.go`, `internal/staticsite/manager.go`
- `internal/api/router.go`, `proxies.go`, `docker.go` (`buildActiveImagesSet` at line 251), `stale.go`
- `internal/stale/scanner.go`, `internal/webhook/matcher.go`
- `cmd/server/main.go`
- `web/src/routes/projects/[id]/+page.svelte`, `stacks/[id]/+page.svelte`, `sites/[id]/+page.svelte`
- `web/src/lib/api.ts`, `web/src/lib/types.ts`
- Top nav component in `web/src/lib/components/`

Created:
- `internal/store/workloads.go`, `containers.go`, `apps.go`
- `internal/workload/workload.go`, `adapters/project_adapter.go`, `adapters/stack_adapter.go`, `adapters/site_adapter.go`
- `internal/reconciler/reconciler.go`, `reconciler_test.go`
- `internal/api/workloads.go`, `containers.go`, `apps.go`
- `web/src/routes/containers/+page.svelte`
- `web/src/lib/components/WorkloadContainers.svelte`

Deleted:
- `internal/store/instances.go`
- `internal/api/instances.go`