alexei.dolgolyov/tiny-forge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
188 Commits 3 Branches 0 Tags
7733e64b08b0db79e320d1f6c5617fc157668897
Commit Graph

9 Commits

Author SHA1 Message Date
alexei.dolgolyov 7733e64b08 feat(gitops): config-as-code via .tinyforge.yml for repo-backed workloads 
A dockerfile or static workload can opt in to reading its deploy config from a
.tinyforge.yml in its own repo. Tinyforge fetches the file, shows field-level
drift vs the live config, and an admin applies it with an explicit Sync. The
repo becomes the source of truth for the declared fields. Manual-sync only;
no auto-apply on deploy, no multi-workload reconcile, no create/delete in v1.

Scope is deliberately source-aware and source_config-resident: dockerfile
declares port/healthcheck/deploy_strategy, static declares deploy_strategy.
The file never carries repo coords or secrets (those stay in the encrypted
DB), which keeps credentials out of the repo.

Backend:
- internal/gitops: Spec/ParseSpec (KnownFields rejects unknown keys), a
  source-aware ApplyPlan/BuildPlan, MergeAndValidate (omitted-field-preserving
  deep merge + validate-the-merged-result-then-commit — never a partial
  config), declared-only Drift with normalization, and Fetch with
  ok/no_file/fetch_failed/invalid statuses and token-redacted messages.
- staticsite: DownloadFile added to GitProvider + Gitea/GitHub/GitLab impls,
  reusing each provider's SSRF-safe client; 64 KiB cap; ErrFileNotFound.
- store: 4 additive gitops_* columns + setters (disjoint from UpdateWorkload
  so the edit-form save and a sync never clobber each other).
- api: GET /workloads/{id}/gitops (status + raw + live drift + managed_fields),
  PUT /gitops (admin, enable/path, traversal-safe), POST /gitops/sync (admin,
  per-workload locked read->merge->validate->write, audited to event_log).

Frontend:
- GitOpsPanel.svelte: status pill, a purpose-built field-level drift view,
  .tinyforge.yml preview, enable ToggleSwitch, Sync via ConfirmDialog; all five
  statuses handled, admin affordances gated on the real viewer role.
- GitOps-managed badge (list + detail hero) and a read-only edit-form banner.
- api.ts fetchers + types; i18n apps.detail.gitops.* (en + ru parity).

Built phase-by-phase with an adversarial plan review (caught 5 design flaws
pre-implementation) and an independent review per phase (go / security / ts /
final) — all APPROVE, 0 CRITICAL/HIGH. docs/gitops.md documents the schema and
what's intentionally out of v1. Plan: plans/gitops/.
 2026-06-21 23:32:02 +03:00
alexei.dolgolyov 0c4c338bfe feat(apps): per-workload deploy history, rollback, and resource metrics 
Two additions to the app detail page, each backed by a per-workload
endpoint.

Deploy history + rollback:
- New deploy_history table — a structured, version-pinned ledger of every
  dispatch (success AND failure), distinct from the free-text event_log.
  Recorded at the single DispatchPlugin choke point so every source kind
  is covered. The raw deploy error is never persisted (it can carry
  registry-auth / compose-stdout secrets) — only a generic marker, with
  detail going to slog. Pruned to the newest N per workload; cascade-
  deleted with the workload.
- GET /api/workloads/{id}/deploys lists the ledger; POST .../rollback
  (admin) replays a prior successful deploy's pinned reference as a
  rollback-reason dispatch. Phase 1 is image-source only (RollbackCapable);
  git-built sources need checkout-by-commit, a later phase.
- DeployHistoryPanel.svelte renders the ledger with confirm-gated rollback.

Per-workload metrics:
- ListContainerStatsSamplesByWorkload joins the existing container stats
  samples through the containers index; GET /api/workloads/{id}/stats/history
  aggregates CPU/memory per timestamp across the workload's containers.
- WorkloadMetricsPanel.svelte reuses ResourceChart (CPU% + memory MiB,
  windowed, 15s poll).

en/ru i18n added with parity. Tests: store CRUD + cascade + workload-scoped
join, deployer recording (incl. secret-non-leak on failure), API rollback
guards, and per-timestamp aggregation. Plans under docs/plans/.
 2026-06-19 16:22:12 +03:00
alexei.dolgolyov 410a131cec feat(apps): stepped creation wizard, branch previews, and app-creation fixes 
This session (frontend focus):
- Rebuild /apps/new as a 4-step wizard (Basics → Configure → Trigger → Review):
  WizardRail, SourceKindPicker card grid, AppManifest review, per-step validation,
  ConfirmDialog-based unsaved-changes guard.
- Extract lib/workload/sourceForms.ts (single source of truth for source_config)
  + {Image,Compose,Static,Dockerfile}SourceForm + StaticDiscoveryWizard; fold the
  /apps/[id] edit form onto the same components (removes the duplication). Add
  vitest + sourceForms unit tests.
- Branch preview environments UI: /chain is_preview/preview_branch + a Preview
  environments panel on /apps/[id] (per-branch URLs, ConfirmDialog teardown, armed
  state); RegistryImagePicker on the registry trigger and the image source.
- Fixes: image-inspect 404 -> admin-gated POST /api/discovery/image/inspect;
  conflict-panel blur flicker; friendly localized discovery errors; CPU/Memory
  label hints; dashboard + /apps "Total workloads" count only source_kind workloads
  (drop stale trigger_kind gate); NPM cert/access-list name cache; EntityPicker
  empty-list guard.
- Update CLAUDE.md frontend conventions + add a Build & Test section.

Also captures pre-existing in-progress platform work (not from this session):
workload notifications, Prometheus metrics export, store lockfile, health probes,
backup hardening, and related store/webhook/scheduler changes.
 2026-05-29 02:09:54 +03:00
alexei.dolgolyov e3c7b13d58 chore(workload): close the workload-first arc — apps i18n + codemap + tests
Build / build (push) Successful in 10m36s
Details 
Closes the workload-first refactor by landing the Priority 3 polish
items and the Priority 4 test gap. Net: ~2,400 lines added,
~350 lines modified across 13 files.

Priority 3 — polish
- apps.* i18n namespace: 276 new keys across apps.list.* (27),
  apps.new.* (91, sibling of existing apps.new.triggers.*), and
  apps.detail.* (158, sibling of existing apps.detail.bindings.*).
  EN+RU at 1314 keys each, perfectly in sync. /apps, /apps/new,
  /apps/[id] now render entirely from i18n.
- New codemap docs/CODEMAPS/workload-plugin.md (238 lines):
  Source × Trigger contract, dispatch seam, webhook fan-out path,
  recipes for adding a new Source or Trigger kind. Plus
  docs/CODEMAPS/INDEX.md gateway.

Priority 4 — tests
- internal/api/workloads_test.go (new, ~30 subtests): /api/workloads
  CRUD + deploy + delete + env + volumes + chain + promote-from +
  triggers list/inline-bind + auth gating + standalone /api/triggers
  CRUD (create / dup-409 / kind filter / delete). Uses real
  POST handlers via httptest.NewServer + a fake plugin source
  registered under "testfakesource".
- internal/deployer/dispatch_test.go (new, 11 tests):
  DispatchPlugin / DispatchTeardown / DispatchReconcile happy +
  unknown-kind + propagated-error each; PluginDeps wiring; a real
  2s-bounded RWMutex deadlock probe on PluginDeps vs SetDNSProvider.
- internal/workload/plugin/source/compose/compose_test.go (new,
  ~26 subtests): composeProjectName sanitization,
  writeYAML / writeYAMLIfChanged hash short-circuit, Validate happy
  + bad inputs, Kind / SchemaSample.

Coverage delta on the workload-plugin path:
- internal/api: 1.1% → 16.0%
- internal/deployer: 0% → 54.1%
- internal/workload/plugin/source/compose: 0% → 38.5%
- Trigger plugins already at 87-95% from the trigger-split work.

Production fix surfaced by the tests
- store.CreateWorkload now self-references RefID = ID when caller
  leaves RefID empty (the typical plugin-native path). The api
  layer's broken backfill loop (called UpdateWorkload, which
  deliberately omits ref_id) is gone. Multiple sibling plugin
  workloads can now coexist under the UNIQUE(kind, ref_id) constraint.

Review fixes addressed before commit
- CRITICAL: deadlock-detect test gained a real 2s time.After (was
  selecting on context.Background().Done() which never fires).
- HIGH: happy-path test now hard-asserts RefID = ID (was a t.Logf
  that would silently pass after a production fix).
- HIGH: standalone /api/triggers CRUD coverage added (was bypassed
  by the workload-side bind flow).
- HIGH: seedWorkload bypass deleted; tests now go through the
  real POST /api/workloads handler.
- MEDIUM: withTempDir restore is a no-op (t.Setenv auto-restores);
  dead `old := os.Getenv(...)` capture removed.
- MEDIUM: list-workloads test now asserts ID membership, not just
  count.

Doc
- WORKLOAD_REFACTOR_TODO: all three Priority 1 items, Priority 3
  polish, and Priority 4 tests marked DONE. The workload-first arc
  is closed.
 2026-05-16 06:42:43 +03:00
alexei.dolgolyov 739b67856a feat(cutover): hard legacy cutover — drop projects/stacks/sites/deploys
Build / build (push) Successful in 10m39s
Details 
The clean-break delete that closes the workload-first refactor arc.
Net diff: ~30 backend files deleted, ~20 modified, ~12k LOC removed
on the Go side; entire /projects /stacks /sites /deploy frontend
trees gone; ~6.7k LOC removed on the Svelte/TypeScript side.

Backend
- API handlers gone: internal/api/{projects,stages,stage_env,stacks,
  static_sites,deploys,instances,volume_browser}.go
- Store CRUD + tests gone: internal/store/{projects,stages,stage_env,
  stacks,static_sites,static_site_secrets,deploys,poll_state,volumes,
  workload_sync}.go (+ _test.go siblings)
- Legacy deployer pipeline gone: internal/deployer/{bluegreen,promote,
  rollback,subdomain,resolver_test}.go; deployer.go trimmed to just the
  dispatch surface used by the plugin pipeline
- internal/staticsite/{manager,healthcheck}.go and
  internal/stack/manager.go gone (the rest of those packages stay as
  helpers imported by the static + compose plugins)
- internal/registry/poller.go gone (legacy registry poller)
- internal/volume.ResolvePath gone; ResolveWorkloadPath stays
- internal/webhook: handleWebhook (project) + handleSiteWebhook (site)
  gone; only POST /api/webhook/triggers/{secret} remains
- workload-side webhook URL handlers (getWorkloadWebhook +
  regenerateWorkloadWebhook + EnsureWorkloadWebhookSecret +
  SetWorkloadWebhookSecret + GetWorkloadByWebhookSecret) gone — they
  minted URLs that would 404 against the new trigger-only ingress
- cmd/server/main.go: dropped staticsite.Manager, stack.Manager,
  staticsite.HealthChecker, registry poller, SetSiteSyncTriggerer,
  SetStaticSiteManager, SetStackManager, wireStaticBackend
- store/store.go: idempotent DROP TABLE IF EXISTS for every legacy
  table (projects, stages, stage_env, volumes, deploys, deploy_logs,
  poll_states, stacks, stack_revisions, stack_deploys, static_sites,
  static_site_secrets); FK order children-then-parents
- store/models.go: dropped Project, Stage, Deploy, DeployLog, StageEnv,
  Volume, StaticSite, StaticSiteSecret, Stack, StackRevision,
  StackDeploy types; kept WorkloadKind constants as documented strings
- internal/store/helpers.go (new): BoolToInt, rowScanner,
  GenerateWebhookSecret extracted from deleted CRUD files
- internal/api/secrets.go (new): forwards to store.GenerateWebhookSecret
  so api + store paths share one secret-generation impl (no
  panic-vs-UUID-fallback divergence)
- internal/reconciler/reconciler.go: dropped legacy stack-by-compose
  + static-site label paths; only canonical tinyforge.workload.id
  dispatch remains
- providers (gitea_content/github_provider/gitlab_provider) gained
  path-traversal rejection on every tree entry
- internal/webhook ParsedImage / ParseImageRef demoted to package-
  private (no external callers)

Frontend
- /projects /stacks /sites /deploy routes deleted (entire trees)
- ProjectCard / InstanceCard / StaleContainerCard components deleted
- api.ts: dropped every project/stage/stack/site/deploy/instance
  helper + types (Project, Stage, Stack, StaticSite, Deploy,
  Instance, Volume, etc.); kept Workload, Container, App, Settings,
  Registry, EventTrigger, LogScanRule, webhook envelopes
- WorkloadWebhook type + getWorkloadWebhook/regenerateWorkloadWebhook
  api functions gone (mirror of the backend deletion above)
- web/src/routes/+layout.svelte: dropped /projects /sites /stacks
  /deploy nav entries, trimmed quick-nav keymap
- web/src/routes/+page.svelte: dashboard rewrite — reads
  listWorkloads + listContainers only; 4-card stat grid
  (workloads/running/failed/stale) + recent workloads strip
- navCounts.ts, SystemHealthCard.svelte, ContainerLogs.svelte,
  ContainerStats.svelte, StatusBadge.svelte, TagCombobox.svelte,
  proxies/+page.svelte, containers/+page.svelte all rewired to the
  workload-first surface
- AbortController plumbing on dashboard, nav-counts, stale page,
  SystemHealthCard so navigation doesn't leave dangling fetches
- i18n: dropped projects.*, projectDetail.*, envEditor.*,
  volumeEditor.*, volumeBrowser.*, quickDeploy.*, sites.*, stacks.*,
  instance.*, confirm.* namespaces; en/ru parity preserved (1042
  keys each)

Hardening from go-reviewer + security-reviewer + typescript-reviewer
subagent passes (0 CRITICAL across all three; 1 HIGH + ~12 MEDIUM
addressed inline before commit):

- Sec H1: dead-end workload webhook URL handlers (would mint URLs
  that 404 the new trigger-only ingress) deleted across backend +
  frontend
- Go M1: IsTerminalDeployStatus dropped (no production callers)
- Go M2: ParsedImage/ParseImageRef lowercased (in-package only)
- Go M6: generateWebhookSecret unified — api shim forwards to
  store.GenerateWebhookSecret
- Doc/comment freshness: stage_id (no longer FK), ProxyRoute legacy
  field names, workloadIDRow rationale, webhook_deliveries.target_type
  enum, WebhookDeliveryLog component header

Doc
- WORKLOAD_REFACTOR_TODO: cutover marked DONE; all three Priority 1
  items are now shipped. Next focus is Priority 3 polish (apps.* i18n
  + codemap entries) and Priority 4 tests.

Behavioral notes for operators upgrading from a pre-cutover build
- Existing rows in the dropped tables disappear on first boot.
- Legacy webhook URLs at /api/webhook/{secret} and
  /api/webhook/sites/{secret} return 404; CI configs must repoint to
  /api/webhook/triggers/{secret} (the trigger-split boot backfill
  lifted any embedded workload secret onto a Trigger row, so the
  secret value itself carries over).
- Frontend routes /projects /stacks /sites /deploy are gone; nav
  links replaced with /apps and /triggers.
 2026-05-16 06:00:21 +03:00
alexei.dolgolyov 8d6a527a2b refactor(workload): plugin architecture wave + apps UI + volume scopes 
Completes the workload-first refactor's plugin layer:

- internal/workload/plugin/ — Source/Trigger plugin contract,
  registry, types (Workload, DeploymentIntent, InboundEvent,
  PublicFace). Self-registering init() pattern + blank-import
  in cmd/server/main.go.
- Source plugins: image (blue-green with multi-face proxy routing),
  compose, static. Trigger plugins: registry, git, manual.
- internal/deployer/dispatch.go — DispatchPlugin/Teardown/Reconcile
  seam routing the legacy deployer through plugins.
- internal/api/workload_*.go — REST surface: workloads, env,
  volumes, chain (parent/children), promote-from. hooks.go
  serves /api/hooks/kinds/{kind}/schema for the wizard.
- internal/store: workload_env (encrypt-at-rest secrets) and
  workload_volumes tables, keyed on workload_id.
- cmd/server/static_backend.go — phantom-row adapter delegating
  the static source plugin to the legacy staticsite.Manager
  (deleted at hard cutover once the static inline port lands).
- web/src/routes/apps/ — /apps list + /apps/new wizard +
  /apps/[id] detail with kind-aware compose / image / static
  forms (Advanced JSON toggle), env panel, volumes panel,
  webhook panel, chain panel, manual deploy.

Volume scope generalization (v2 resolver):

- internal/volume.ResolveWorkloadPath (workload-keyed, sits
  next to legacy ResolvePath). Honors all VolumeScope values:
  absolute, ephemeral, instance, stage, project, project_named,
  named. internal/workload/plugin/source/image/image.go
  computeMounts wires settings + imageTag through. Coverage in
  internal/volume/resolver_test.go (portable Linux/Windows via
  t.TempDir).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 22:17:41 +03:00
alexei.dolgolyov cba2149aa9 refactor(workload): finalize containers index + post-review hardening 
Wraps up the workload refactor with the fixes that came out of the multi-agent
code review (see docs/plans/workload-refactor.md "What actually shipped").

Backend:
- store.ReconcileContainer: separate write path so the 30s reconciler tick no
  longer overwrites deployer-owned fields (subdomain, proxy_route_id,
  npm_proxy_id, image_tag).
- Container.stage_id column + index; ListProxyRoutes / ListContainersByStageID
  join via stage_id (survives stage rename), with legacy fallback to
  (project_id, role=stage_name).
- Reconciler: workload-existence check (rejects forged tinyforge.workload.id
  labels), skips inventing project-kind rows, child-context cancel before
  wg.Wait() on shutdown.
- Transactional CRUD across projects / stacks / static_sites: parent UPDATE
  and workload sync land in one transaction so secret rotations are durable.
- Webhook routing reads exclusively through workloads.webhook_secret; legacy
  GetProjectByWebhookSecret / GetStaticSiteByWebhookSecret fallback removed.
- store.GetStackByComposeProjectName + indexed lookup (no more full-table
  stack scan per compose container per tick).
- store.ListMissingSweepRows: filtered query for the missing-sweep.
- /api/instances/* handlers verify (workload_id, role) match URL
  (project_id, stage_name) before mutating — closes the cross-project
  hijack the security review flagged.
- extra_json no longer referenced from Go (column kept on disk for now).

Frontend:
- WorkloadContainers.svelte: generic detail-page panel reusable by stack and
  site detail pages.
- Containers page polish: client-side kind/state filters over an unfiltered
  fetch, URL-synced filters, race-safe loads via sequence number, EN+RU i18n,
  sidebar counter via navCounts.containers.

Misc:
- scripts/dev-server.sh: tolerate empty netstat grep result.
- .gitignore: ignore docker-watcher binaries, .claude/worktrees/, .facts-sync.json.
 2026-05-09 15:44:41 +03:00
alexei.dolgolyov db235c1412 feat(workload): write-through workload sync + boot-time backfill 
CRUD on Project / Stack / StaticSite now keeps a paired Workload
row in sync. Secret setters (webhook secret, signing secret,
require-signature toggle, notification secret) all re-sync after
mutating the source-of-truth row so the workload row always
reflects the canonical state.

Delete cascades: DeleteProject/Stack/StaticSite now drop the
matching workload row plus any container index entries owned by
it, so global views don't show ghost rows.

Boot-time BackfillWorkloads scans every project/stack/site and
ensures each has a workload row. Idempotent — safe to run on
every restart, recovers from a deleted/missing workload row.

Behavior unchanged for existing call sites; the workloads table
just starts being populated. Deployer / reconciler / consumer
switchover land in the next commit.
 2026-05-09 13:28:20 +03:00
alexei.dolgolyov f54a6ecee3 feat(workload): add Workload/Container/App store foundation 
Introduces the data layer for the Workload refactor (see
docs/plans/workload-refactor.md): three new tables and store
methods, no behavior changes elsewhere yet.

- workloads: unifying primitive over Project/Stack/StaticSite,
  paired via UNIQUE(kind, ref_id). Notification + webhook config
  hosted here so it lives in one place across kinds.
- containers: normalized index of every Tinyforge-managed
  container with first-class subdomain/proxy_route_id/npm_proxy_id
  columns (heavily queried by ListProxyRoutes / stale detection).
- apps: optional grouping of workloads; schema only, no UI in v1.

Foundation only — deployer surgery, reconciler, and consumer
switchover land in the next commit.
 2026-05-09 13:22:25 +03:00