Files
tiny-forge/internal/api/discovery.go
T
alexei.dolgolyov ea55d31177
Build / build (push) Successful in 10m43s
feat(discovery+runtime): restore static-site wizard discovery + close /sites/[id] feature parity
Two-stage feature arc closing the gaps left by the hard legacy cutover.
The static-site creation wizard regains its auto-discovery + connection-test
flow; /apps/[id] grows the runtime/storage/lifecycle surface the legacy
/sites/[id] page used to expose.

Backend (Go)
- internal/api/discovery.go: six admin-gated endpoints wrapping
  staticsite.GitProvider — POST /api/discovery/git/{detect-provider,
  test-connection,repos,branches,tree} + GET /api/discovery/image/conflicts.
  Identifier validation (validateGitIdent / validateGitBranch) at the
  boundary so provider URL interpolation cannot be hijacked via `..`.
  Upstream errors scrubbed: detailed slog on the server, generic 502 to
  the client (mitigates token-reflection-in-error-page).
- internal/api/workload_runtime.go: four endpoints —
  GET /api/workloads/{id}/runtime-state decodes containers.extra_json for
  static workloads; GET /api/workloads/{id}/storage execs `du -sb /app/data`
  with a 30s in-process cache (storageProbeCache) so polling can't turn
  into per-request execs; POST /api/workloads/{id}/{stop,start} iterate
  ListContainersByWorkload and call docker.StopContainer / StartContainer,
  returning 200 / 409 (nothing to act on) / 502 (all failed).
- internal/staticsite/safehttp.go: NewSafeHTTPClient + ValidateBaseURL +
  blockReason. DialContext re-resolves hostnames and refuses loopback /
  link-local / multicast / unspecified addresses. RFC1918 + ULA explicitly
  allowed (self-hosted Gitea on LAN is the dominant deployment).
  Replaced four raw &http.Client{} constructions in the provider files.
- internal/staticsite/gitlab_provider.go: url.PathEscape each segment in
  the raw-file URL builder for parity with projectPath().
- Test coverage: 26 cases in discovery_test.go (image-tag stripping,
  source-config decoding, conflict scenarios, validator boundaries,
  scheme rejection), 14 in workload_runtime_test.go (404 / 409 / nil-docker
  / probe-cache), 16 in safehttp_test.go (URL validation + block-reason
  policy matrix + live dial against loopback + AWS metadata literals).

Frontend (Svelte 5 + runes)
- web/src/lib/api.ts: typed wrappers for every endpoint, AbortSignal
  threaded through post(); ApiError exported so callers can narrow on
  e.status; new DetectedGitProvider narrow union.
- web/src/routes/apps/new/+page.svelte: static-form discovery controls
  (auto-detect provider, test connection, repo / branch / folder
  EntityPickers, Deno auto-detect); image-form conflict panel with
  debounced lookup + double-click submit guard ("Forge anyway") + Inspect
  button that pre-fills port/healthcheck; English error fallbacks routed
  through apps.new.errors.* (en + ru).
- web/src/routes/apps/[id]/+page.svelte: runtime-state panel + storage
  panel + Stop / Start / Open-site toolbar; universal live-state badge
  in the hero lede for image/compose/static (RUNNING / TRANSITIONING /
  STOPPED / NOT DEPLOYED / MIXED · n/m RUNNING); ContainerStats panel
  per row (auto-collapsing native <details> when N > 2); read-only
  webhook bindings summary card; responsive toolbar overflow with native
  <details> at <640px (z-index 100 above sticky nav).
- web/src/app.css: project-wide .forge-btn-ghost:focus-visible outline.

Hardening from go-reviewer + security-reviewer + typescript-reviewer +
frontend-design UI/UX subagents (0 CRITICAL, all HIGH/BLOCKER addressed
inline, IMPORTANT applied before commit):
- AbortController + per-call sequence tokens on every long-running
  fetch (loadRuntimeState / loadStorage / loadTriggerMeta / inspectImage /
  listImageConflicts) plus onDestroy cleanup so late resolves cannot
  mutate dead component state.
- doStop / doStart snapshot and restore `error` across the finally-block
  reload so a load()-cleared message doesn't hide a real failure.
- triggersById refreshed after inline trigger creation so the webhook
  card doesn't silently exclude the just-created trigger.
- Live-state badge wraps in role=status / aria-live=polite (no redundant
  aria-label).
- Webhook row has a single click target (was two pointing at the same URL).
- Empty webhook section hides entirely.
- Dropped role=menu / role=menuitem from the overflow menu (they would
  promise arrow-key nav we don't wire; native Tab + ESC carry it).

Doc
- docs/CODEMAPS/INDEX.md + new docs/CODEMAPS/discovery-and-runtime.md
  map the endpoint surface, security posture, frontend integration
  patterns, and an "add a new probe" recipe.

Verification
- svelte-check: 0 errors, 3 pre-existing a11y warnings.
- go build + go vet + go test ./...: all green.
- i18n parity: en + ru at 1413 keys each.
- Live smoke against :8090: 404 / 409 / 502 envelopes correct, discovery
  sanity passes, ProbeError surfaces on no-container path.
2026-05-16 21:35:51 +03:00

404 lines
13 KiB
Go

package api
import (
"context"
"encoding/json"
"log/slog"
"net/http"
"regexp"
"strings"
"time"
"github.com/alexei/tinyforge/internal/staticsite"
)
// Discovery endpoints feed the /apps/new wizard's auto-discovery and
// connection-test flow. They wrap staticsite.GitProvider so the form
// can validate a repo + token before the workload is created, browse
// repos/branches/folders without leaving the page, and warn the operator
// when an image is already in use by another workload.
//
// The endpoints are workload-agnostic on purpose — they are scoped under
// /api/discovery rather than tied to the static_sites table the cutover
// dropped. Any future Git-driven source plugin can reuse them.
// Per-request budget for outbound calls. Short enough that a malicious
// or stuck upstream cannot pin a worker for long; long enough for slow
// self-hosted Gitea instances to respond.
const discoveryTimeout = 15 * time.Second
// gitProviderRequest is the shared request body for the four Git
// discovery endpoints. Token is plaintext over HTTPS — the wizard has
// not yet persisted it, so there is nothing to decrypt server-side.
// Empty Provider triggers DetectProviderWithProbe.
type gitProviderRequest struct {
Provider string `json:"provider"`
BaseURL string `json:"base_url"`
AccessToken string `json:"access_token"`
RepoOwner string `json:"repo_owner"`
RepoName string `json:"repo_name"`
Branch string `json:"branch"`
Query string `json:"query"`
}
// gitIdentRe accepts Git owner / repo identifiers as the major hosts
// (GitHub, GitLab, Gitea/Forgejo) accept them: alphanumeric plus dot,
// underscore, hyphen. Rejecting other characters at the API boundary
// prevents `..` traversal and URL injection in the provider code that
// interpolates these segments into request paths.
var gitIdentRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]*$`)
// gitBranchRe is more permissive than gitIdentRe: branches may contain
// `/` (e.g. `feature/foo`) but still cannot contain `..` or control
// characters. The check below pairs this regex with an explicit `..`
// reject so a `feature/../admin` value cannot slip through.
var gitBranchRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._/-]*$`)
// validateGitIdent guards owner / repo path segments at the boundary
// so the provider code can interpolate them with fmt.Sprintf without
// risking traversal. Empty input is reported with the supplied field
// name so the error message is actionable.
func validateGitIdent(field, value string) error {
v := strings.TrimSpace(value)
if v == "" {
return &apiError{msg: field + " is required"}
}
if !gitIdentRe.MatchString(v) {
return &apiError{msg: field + " contains invalid characters"}
}
return nil
}
// validateGitBranch is the branch-shaped variant of validateGitIdent.
// Branches legitimately contain `/`; the extra `..` reject covers the
// one traversal vector the regex still admits.
func validateGitBranch(value string) error {
v := strings.TrimSpace(value)
if v == "" {
return &apiError{msg: "branch is required"}
}
if strings.Contains(v, "..") {
return &apiError{msg: "branch contains invalid sequence '..'"}
}
if !gitBranchRe.MatchString(v) {
return &apiError{msg: "branch contains invalid characters"}
}
return nil
}
// apiError is a small typed error so handlers can distinguish a
// validation failure (→ 400) from any other error (→ 500/502). The
// type lives in this file because nothing outside discovery uses it
// yet — promote to response.go if other handlers need the same shape.
type apiError struct{ msg string }
func (e *apiError) Error() string { return e.msg }
// providerType normalizes the provider string into the typed enum used
// by staticsite.NewGitProvider. Empty input falls through to provider
// auto-detection inside NewGitProvider.
func (req gitProviderRequest) providerType() staticsite.ProviderType {
switch strings.ToLower(strings.TrimSpace(req.Provider)) {
case "github":
return staticsite.ProviderGitHub
case "gitlab":
return staticsite.ProviderGitLab
case "gitea":
return staticsite.ProviderGitea
default:
return ""
}
}
// newProvider constructs the GitProvider for the request, or writes a
// 400 to w and returns nil if the inputs are invalid. BaseURL is fully
// validated here (scheme + host shape); connect-time IP filtering is
// enforced inside the safe-HTTP transport the provider receives.
func (req gitProviderRequest) newProvider(w http.ResponseWriter) staticsite.GitProvider {
if err := staticsite.ValidateBaseURL(req.BaseURL); err != nil {
respondError(w, http.StatusBadRequest, err.Error())
return nil
}
provider, err := staticsite.NewGitProvider(req.providerType(), req.BaseURL, req.AccessToken)
if err != nil {
respondError(w, http.StatusBadRequest, err.Error())
return nil
}
return provider
}
// upstreamError logs the detailed upstream failure server-side and
// writes a generic 502 to the client. Echoing the raw error string
// would leak any access token reflected by a misconfigured or
// attacker-controlled upstream into the response body.
func upstreamError(w http.ResponseWriter, op string, err error) {
slog.Warn("discovery upstream call failed", "op", op, "error", err)
respondError(w, http.StatusBadGateway, "upstream git provider returned an error")
}
// detectGitProviderRequest is the body for POST /api/discovery/git/detect-provider.
type detectGitProviderRequest struct {
BaseURL string `json:"base_url"`
}
// detectGitProvider probes the base URL for known Git provider API
// signatures so the wizard can auto-fill the provider dropdown.
// POST /api/discovery/git/detect-provider.
func (s *Server) detectGitProvider(w http.ResponseWriter, r *http.Request) {
var req detectGitProviderRequest
if !decodeJSON(w, r, &req) {
return
}
if err := staticsite.ValidateBaseURL(req.BaseURL); err != nil {
respondError(w, http.StatusBadRequest, err.Error())
return
}
ctx, cancel := context.WithTimeout(r.Context(), discoveryTimeout)
defer cancel()
provider := staticsite.DetectProviderWithProbe(ctx, req.BaseURL)
respondJSON(w, http.StatusOK, map[string]string{"provider": string(provider)})
}
// testGitConnection verifies the configured base URL + token + repo
// reach the provider successfully so the wizard can fail fast.
// POST /api/discovery/git/test-connection.
func (s *Server) testGitConnection(w http.ResponseWriter, r *http.Request) {
var req gitProviderRequest
if !decodeJSON(w, r, &req) {
return
}
if err := validateGitIdent("repo_owner", req.RepoOwner); err != nil {
respondError(w, http.StatusBadRequest, err.Error())
return
}
if err := validateGitIdent("repo_name", req.RepoName); err != nil {
respondError(w, http.StatusBadRequest, err.Error())
return
}
provider := req.newProvider(w)
if provider == nil {
return
}
ctx, cancel := context.WithTimeout(r.Context(), discoveryTimeout)
defer cancel()
if err := provider.TestConnection(ctx, req.RepoOwner, req.RepoName); err != nil {
upstreamError(w, "test_connection", err)
return
}
respondJSON(w, http.StatusOK, map[string]string{"status": "ok"})
}
// listGitRepos returns repositories accessible with the supplied token,
// optionally filtered by a name query.
// POST /api/discovery/git/repos.
func (s *Server) listGitRepos(w http.ResponseWriter, r *http.Request) {
var req gitProviderRequest
if !decodeJSON(w, r, &req) {
return
}
provider := req.newProvider(w)
if provider == nil {
return
}
ctx, cancel := context.WithTimeout(r.Context(), discoveryTimeout)
defer cancel()
repos, err := provider.ListRepos(ctx, req.Query)
if err != nil {
upstreamError(w, "list_repos", err)
return
}
if repos == nil {
repos = []staticsite.RepoInfo{}
}
respondJSON(w, http.StatusOK, repos)
}
// listGitBranches returns the branch list for a repo.
// POST /api/discovery/git/branches.
func (s *Server) listGitBranches(w http.ResponseWriter, r *http.Request) {
var req gitProviderRequest
if !decodeJSON(w, r, &req) {
return
}
if err := validateGitIdent("repo_owner", req.RepoOwner); err != nil {
respondError(w, http.StatusBadRequest, err.Error())
return
}
if err := validateGitIdent("repo_name", req.RepoName); err != nil {
respondError(w, http.StatusBadRequest, err.Error())
return
}
provider := req.newProvider(w)
if provider == nil {
return
}
ctx, cancel := context.WithTimeout(r.Context(), discoveryTimeout)
defer cancel()
branches, err := provider.ListBranches(ctx, req.RepoOwner, req.RepoName)
if err != nil {
upstreamError(w, "list_branches", err)
return
}
if branches == nil {
branches = []string{}
}
respondJSON(w, http.StatusOK, branches)
}
// listGitTree returns the full directory tree for a branch so the
// wizard can render the folder picker.
// POST /api/discovery/git/tree.
func (s *Server) listGitTree(w http.ResponseWriter, r *http.Request) {
var req gitProviderRequest
if !decodeJSON(w, r, &req) {
return
}
if err := validateGitIdent("repo_owner", req.RepoOwner); err != nil {
respondError(w, http.StatusBadRequest, err.Error())
return
}
if err := validateGitIdent("repo_name", req.RepoName); err != nil {
respondError(w, http.StatusBadRequest, err.Error())
return
}
if err := validateGitBranch(req.Branch); err != nil {
respondError(w, http.StatusBadRequest, err.Error())
return
}
provider := req.newProvider(w)
if provider == nil {
return
}
ctx, cancel := context.WithTimeout(r.Context(), discoveryTimeout)
defer cancel()
tree, err := provider.ListTree(ctx, req.RepoOwner, req.RepoName, req.Branch)
if err != nil {
upstreamError(w, "list_tree", err)
return
}
if tree == nil {
tree = []staticsite.FolderEntry{}
}
respondJSON(w, http.StatusOK, tree)
}
// imageConflict is a slim projection of Workload, scoped to what the
// /apps/new conflict dialog needs to render.
type imageConflict struct {
ID string `json:"id"`
Name string `json:"name"`
Image string `json:"image"`
AppID string `json:"app_id,omitempty"`
}
// listImageConflicts finds existing image-source workloads whose
// configured image matches the supplied ref, with or without tag.
// GET /api/discovery/image/conflicts?image=<ref>.
//
// Matching mirrors the legacy quickDeploy behavior: collide on
// repository-without-tag so nginx:1.25 surfaces nginx, nginx:latest,
// and nginx:1.26 as conflicts. This is intentionally permissive — the
// wizard surfaces matches but lets the operator decide.
func (s *Server) listImageConflicts(w http.ResponseWriter, r *http.Request) {
image := strings.TrimSpace(r.URL.Query().Get("image"))
if image == "" {
respondError(w, http.StatusBadRequest, "image query parameter is required")
return
}
target := stripImageTag(image)
if target == "" {
respondError(w, http.StatusBadRequest, "image is empty after tag strip")
return
}
workloads, err := s.store.ListWorkloads("")
if err != nil {
slog.Error("list workloads for conflict check", "error", err)
respondError(w, http.StatusInternalServerError, "internal server error")
return
}
conflicts := []imageConflict{}
for _, wl := range workloads {
if wl.SourceKind != "image" {
continue
}
ref := imageRefFromSourceConfig(wl.SourceConfig)
if ref == "" {
continue
}
if stripImageTag(ref) != target {
continue
}
conflicts = append(conflicts, imageConflict{
ID: wl.ID,
Name: wl.Name,
Image: ref,
AppID: wl.AppID,
})
}
respondJSON(w, http.StatusOK, conflicts)
}
// stripImageTag returns the image reference with the trailing :tag
// removed, taking care to leave a registry port (e.g. registry:5000/foo)
// intact. Digest references (image@sha256:...) are returned unchanged.
func stripImageTag(ref string) string {
ref = strings.TrimSpace(ref)
if ref == "" {
return ""
}
// Digest reference: keep as-is so two pinned-by-digest workloads do
// not collide with each other or with tag-based refs unless the
// caller asks for exact-match (we currently don't).
if at := strings.Index(ref, "@"); at >= 0 {
return ref[:at]
}
// Strip a :tag suffix only when the colon is in the final path
// segment — earlier colons belong to a registry port.
lastSlash := strings.LastIndex(ref, "/")
tail := ref
if lastSlash >= 0 {
tail = ref[lastSlash+1:]
}
if colon := strings.LastIndex(tail, ":"); colon >= 0 {
// Only strip if the tag part looks like a tag (no slashes,
// non-empty). Otherwise leave alone. When lastSlash is -1 the
// arithmetic still yields the right cut point (-1 + 1 + colon
// == colon), so no special case is needed.
tag := tail[colon+1:]
if tag != "" && !strings.ContainsAny(tag, "/") {
return ref[:lastSlash+1+colon]
}
}
return ref
}
// imageRefFromSourceConfig extracts the "image" field from a workload's
// source_config JSON. Returns "" when the blob is missing, malformed,
// or has no image field — those workloads simply do not contribute to
// conflict detection.
func imageRefFromSourceConfig(raw string) string {
if raw == "" {
return ""
}
var cfg struct {
Image string `json:"image"`
}
if err := json.Unmarshal([]byte(raw), &cfg); err != nil {
return ""
}
return strings.TrimSpace(cfg.Image)
}