Files
tiny-forge/internal/store/models.go
T
alexei.dolgolyov 7a9ff7ad54 feat(observability): event triggers + log scanner backend
Two paired backends sharing the events.Bus seam:

Event triggers (consumer-side):
- internal/store/event_triggers.go — CRUD with action_secret
  redaction on read (placeholder echo treated as "no change" on
  PATCH so secrets aren't accidentally wiped).
- internal/events/dispatcher.go — bus subscriber, AND-composed
  filters (severity CSV, source CSV, message regex with memoized
  compile cache). Structural loop-prevention: never writes to
  event_log. Sends via notifier.SendPayload.
- internal/notify: SendPayload + SendSyncForTestPayload methods,
  TierEventTrigger constant, doSendRaw shared with the legacy
  Event-shaped path.
- internal/api/event_triggers.go — admin-gated CRUD + /test
  sending the real TriggerWebhookPayload shape. SSRF guard
  rejects loopback / link-local / unspecified targets. PATCH
  uses pointer-typed DTO for partial updates.

Log scanner (producer-side):
- internal/logscanner/ — engine (per-rule cooldown +
  per-container token bucket, atomic drop counters), tail
  (multiplexed docker frame demuxer with TTY fallback + 16 MiB
  payload cap + 1 MiB reassembly cap + RFC3339Nano-validated
  timestamp strip + UTF-8-safe message truncation), manager
  (5s container polling, atomic.Pointer[Snapshot] hot-reload,
  HitEmitter writes event_log + publishes EventLog so the
  trigger dispatcher picks them up immediately).
- internal/docker/container.go — ContainerLogsOpts exposes
  stream selection for stderr-only / stdout-only rules.
- internal/store: log_scan_rules table + CRUD with
  EffectiveLogScanRules resolver (globals minus per-workload
  overrides plus workload-only additions). Transactional
  cascade-delete of overrides when a global rule is removed.
- internal/api/log_scan_rules.go — admin-gated CRUD + /test
  (sample_line → matched/captures) + /stats (drop counters +
  active tail count + last-snapshot compile errors) +
  GET /api/workloads/{id}/effective-rules.

cmd/server/main.go wires both subsystems next to the existing
RegisterPersistentLogger. Coverage spans engine cooldown / bucket
counter tests, snapshot effective-set semantics, manager compile-
error capture, dispatcher matching, store validation +
cascade-delete, API URL validator + secret redaction.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 22:18:11 +03:00

523 lines
23 KiB
Go

package store
// Project represents a deployable application.
type Project struct {
ID string `json:"id"`
Name string `json:"name"`
Registry string `json:"registry"`
Image string `json:"image"`
Port int `json:"port"`
Healthcheck string `json:"healthcheck"`
Env string `json:"env"` // JSON-encoded map
Volumes string `json:"volumes"` // JSON-encoded map
NpmAccessListID int `json:"npm_access_list_id"` // per-project override, 0 = use global
WebhookSecret string `json:"-"` // per-project webhook secret (URL identifier); never serialized
WebhookSigningSecret string `json:"-"` // HMAC-SHA256 key for inbound webhook signature verification; never serialized
WebhookRequireSignature bool `json:"webhook_require_signature"` // if true, reject unsigned/invalid-sig webhook requests
NotificationURL string `json:"notification_url"` // outgoing webhook target; empty = inherit from settings
NotificationSecret string `json:"-"` // outgoing-webhook signing secret; never serialized directly
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
}
// Stage represents a deployment stage within a project (e.g. dev, rel, prod).
type Stage struct {
ID string `json:"id"`
ProjectID string `json:"project_id"`
Name string `json:"name"`
TagPattern string `json:"tag_pattern"`
AutoDeploy bool `json:"auto_deploy"`
MaxInstances int `json:"max_instances"`
Confirm bool `json:"confirm"`
EnableProxy bool `json:"enable_proxy"`
PromoteFrom string `json:"promote_from"`
Subdomain string `json:"subdomain"`
NotificationURL string `json:"notification_url"`
NotificationSecret string `json:"-"` // outgoing-webhook signing secret; never serialized directly
CpuLimit float64 `json:"cpu_limit"` // CPU cores (e.g., 0.5, 1, 2), 0 = unlimited
MemoryLimit int `json:"memory_limit"` // megabytes, 0 = unlimited
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
}
// Registry represents a container image registry.
type Registry struct {
ID string `json:"id"`
Name string `json:"name"`
URL string `json:"url"`
Type string `json:"type"`
Token string `json:"token"`
Owner string `json:"owner"`
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
}
// Settings holds global application configuration (single-row pattern).
type Settings struct {
Domain string `json:"domain"`
ServerIP string `json:"server_ip"` // Docker host IP (for NPM remote forwarding)
PublicIP string `json:"public_ip"` // Public-facing IP for DNS A records (e.g., NPM/proxy host)
Network string `json:"network"`
SubdomainPattern string `json:"subdomain_pattern"`
NotificationURL string `json:"notification_url"`
NotificationSecret string `json:"-"` // outgoing-webhook signing secret; never serialized directly
NpmURL string `json:"npm_url"`
NpmEmail string `json:"npm_email"`
NpmPassword string `json:"npm_password"`
PollingInterval string `json:"polling_interval"`
BaseVolumePath string `json:"base_volume_path"`
SSLCertificateID int `json:"ssl_certificate_id"`
StaleThresholdDays int `json:"stale_threshold_days"`
AllowedVolumePaths string `json:"allowed_volume_paths"` // JSON array of allowed absolute paths
WildcardDNS bool `json:"wildcard_dns"`
DNSProvider string `json:"dns_provider"`
CloudflareAPIToken string `json:"cloudflare_api_token"`
CloudflareZoneID string `json:"cloudflare_zone_id"`
NpmRemote bool `json:"npm_remote"`
NpmAccessListID int `json:"npm_access_list_id"`
ProxyProvider string `json:"proxy_provider"`
TraefikEntrypoint string `json:"traefik_entrypoint"`
TraefikCertResolver string `json:"traefik_cert_resolver"`
TraefikNetwork string `json:"traefik_network"`
TraefikAPIURL string `json:"traefik_api_url"`
ImagePruneThresholdMB int `json:"image_prune_threshold_mb"`
BackupEnabled bool `json:"backup_enabled"`
BackupIntervalHours int `json:"backup_interval_hours"`
BackupRetentionCount int `json:"backup_retention_count"`
// AutoBackupBeforeDeploy creates a "pre-deploy" Tinyforge DB backup
// at the start of every project deploy. Independent of BackupEnabled
// (which governs the periodic auto-backup cron).
AutoBackupBeforeDeploy bool `json:"auto_backup_before_deploy"`
StatsIntervalSeconds int `json:"stats_interval_seconds"` // 0 disables collection
StatsRetentionHours int `json:"stats_retention_hours"` // 0 disables collection
UpdatedAt string `json:"updated_at"`
}
// ContainerStatsSample is one persisted sample of container resource usage.
// Cumulative counters (network, block I/O) require differencing two samples
// to get rates; CPU is already a percent-since-previous-sample value.
type ContainerStatsSample struct {
ContainerID string `json:"container_id"`
OwnerType string `json:"owner_type"` // "instance" or "site"
OwnerID string `json:"owner_id"`
TS int64 `json:"ts"` // Unix seconds UTC
CPUPercent float64 `json:"cpu_percent"`
MemoryUsage int64 `json:"memory_usage"`
MemoryLimit int64 `json:"memory_limit"`
NetworkRxBytes int64 `json:"network_rx_bytes"`
NetworkTxBytes int64 `json:"network_tx_bytes"`
BlockReadBytes int64 `json:"block_read_bytes"`
BlockWriteBytes int64 `json:"block_write_bytes"`
}
// SystemStatsSample is one persisted host-level snapshot that aggregates
// workload usage across all containers plus daemon capacity + disk totals.
type SystemStatsSample struct {
TS int64 `json:"ts"` // Unix seconds UTC
NCPU int `json:"ncpu"`
MemoryTotal int64 `json:"memory_total"`
WorkloadCPUPercent float64 `json:"workload_cpu_percent"`
WorkloadMemUsage int64 `json:"workload_mem_usage"`
ContainersRunning int `json:"containers_running"`
DiskTotalBytes int64 `json:"disk_total_bytes"`
}
// Backup represents a backup metadata record.
type Backup struct {
ID string `json:"id"`
Filename string `json:"filename"`
SizeBytes int64 `json:"size_bytes"`
BackupType string `json:"backup_type"` // "manual" or "auto"
CreatedAt string `json:"created_at"`
}
// DNSRecord tracks a DNS record managed by the application.
type DNSRecord struct {
ID string `json:"id"`
FQDN string `json:"fqdn"`
ProviderRecordID string `json:"provider_record_id"`
ConsumerType string `json:"consumer_type"` // "instance" or "standalone"
ConsumerID string `json:"consumer_id"`
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
}
// ProxyRoute is a proxy-enabled container row joined with its project + stage
// names, shaped for the Proxies page. Source is "instance" for project
// containers and "static_site" for site rows — the names are historical
// (the table itself was renamed to containers in the workload refactor).
type ProxyRoute struct {
Source string `json:"source"`
InstanceID string `json:"instance_id"`
ProjectID string `json:"project_id"`
ProjectName string `json:"project_name"`
StageID string `json:"stage_id"`
StageName string `json:"stage_name"`
ImageTag string `json:"image_tag"`
Subdomain string `json:"subdomain"`
Domain string `json:"domain"`
ContainerID string `json:"container_id"`
Port int `json:"port"`
ProxyRouteID string `json:"proxy_route_id"`
NpmProxyID int `json:"npm_proxy_id"`
Status string `json:"status"`
CreatedAt string `json:"created_at"`
}
// Deploy represents a deployment attempt.
type Deploy struct {
ID string `json:"id"`
ProjectID string `json:"project_id"`
StageID string `json:"stage_id"`
InstanceID string `json:"instance_id"`
ImageTag string `json:"image_tag"`
Status string `json:"status"` // pending, pulling, starting, configuring_proxy, health_checking, success, failed, rolled_back
StartedAt string `json:"started_at"`
FinishedAt string `json:"finished_at"`
Error string `json:"error"`
}
// DeployLog is a single log entry for a deploy.
type DeployLog struct {
ID int64 `json:"id"`
DeployID string `json:"deploy_id"`
Message string `json:"message"`
Level string `json:"level"` // info, warn, error
CreatedAt string `json:"created_at"`
}
// StageEnv represents a per-stage environment variable override.
type StageEnv struct {
ID string `json:"id"`
StageID string `json:"stage_id"`
Key string `json:"key"`
Value string `json:"value"`
Encrypted bool `json:"encrypted"`
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
}
// WorkloadVolume is the plugin-shape equivalent of legacy Volume: a
// per-workload mount declaration. The Scope enum matches the existing
// VolumeScope contract so the legacy resolver can be reused once its
// project_id assumption is loosened.
type WorkloadVolume struct {
ID string `json:"id"`
WorkloadID string `json:"workload_id"`
Source string `json:"source"`
Target string `json:"target"`
Scope string `json:"scope"`
Name string `json:"name"`
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
}
// WorkloadEnv is the plugin-shape equivalent of StageEnv: per-workload
// environment variable overrides, optionally encrypted at rest. Read by
// the Source plugin at deploy time, merged on top of source_config.env.
type WorkloadEnv struct {
ID string `json:"id"`
WorkloadID string `json:"workload_id"`
Key string `json:"key"`
Value string `json:"value"`
Encrypted bool `json:"encrypted"`
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
}
// VolumeScope defines the sharing scope for a volume mount.
// Valid scopes: instance, stage, project, project_named, named, ephemeral.
type VolumeScope string
const (
VolumeScopeInstance VolumeScope = "instance"
VolumeScopeStage VolumeScope = "stage"
VolumeScopeProject VolumeScope = "project"
VolumeScopeProjectNamed VolumeScope = "project_named"
VolumeScopeNamed VolumeScope = "named"
VolumeScopeEphemeral VolumeScope = "ephemeral"
VolumeScopeAbsolute VolumeScope = "absolute"
)
// ValidVolumeScopes contains all valid scope values for validation.
var ValidVolumeScopes = []VolumeScope{
VolumeScopeInstance, VolumeScopeStage, VolumeScopeProject,
VolumeScopeProjectNamed, VolumeScopeNamed, VolumeScopeEphemeral,
VolumeScopeAbsolute,
}
// IsValidVolumeScope returns true if the given string is a valid scope.
func IsValidVolumeScope(s string) bool {
for _, v := range ValidVolumeScopes {
if string(v) == s {
return true
}
}
return false
}
// Volume represents a volume mount configuration for a project.
type Volume struct {
ID string `json:"id"`
ProjectID string `json:"project_id"`
Source string `json:"source"`
Target string `json:"target"`
Mode string `json:"mode,omitempty"` // legacy: shared/isolated — kept for DB compat
Scope string `json:"scope"` // instance, stage, project, project_named, named, ephemeral
Name string `json:"name"` // required for project_named and named scopes
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
}
// StaticSite represents a static site deployed from a Git repository folder.
type StaticSite struct {
ID string `json:"id"`
Name string `json:"name"`
Provider string `json:"provider"` // "gitea", "github", "gitlab"; empty = autodetect
GiteaURL string `json:"gitea_url"` // base URL, e.g. https://git.example.com
RepoOwner string `json:"repo_owner"`
RepoName string `json:"repo_name"`
Branch string `json:"branch"`
FolderPath string `json:"folder_path"` // path within repo, e.g. "Pages"
AccessToken string `json:"access_token"` // encrypted; optional for public repos
Domain string `json:"domain"` // full domain for proxy
Mode string `json:"mode"` // "static" or "deno"
RenderMarkdown bool `json:"render_markdown"`
SyncTrigger string `json:"sync_trigger"` // "push", "tag", "manual"
TagPattern string `json:"tag_pattern"` // glob pattern for tag-based sync
ContainerID string `json:"container_id"`
ProxyRouteID string `json:"proxy_route_id"`
Status string `json:"status"` // idle, syncing, deployed, failed
LastSyncAt string `json:"last_sync_at"`
LastCommitSHA string `json:"last_commit_sha"`
Error string `json:"error"`
StorageEnabled bool `json:"storage_enabled"`
StorageLimitMB int `json:"storage_limit_mb"` // 0 = unlimited
WebhookSecret string `json:"-"` // per-site webhook secret (URL identifier); never serialized
WebhookSigningSecret string `json:"-"` // HMAC-SHA256 key for inbound webhook signature verification; never serialized
WebhookRequireSignature bool `json:"webhook_require_signature"` // if true, reject unsigned/invalid-sig webhook requests
NotificationURL string `json:"notification_url"` // outgoing webhook target; empty = inherit from settings
NotificationSecret string `json:"-"` // outgoing-webhook signing secret; never serialized directly
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
}
// StaticSiteSecret represents an encrypted environment variable for a static site's Deno backend.
type StaticSiteSecret struct {
ID string `json:"id"`
SiteID string `json:"site_id"`
Key string `json:"key"`
Value string `json:"value"`
Encrypted bool `json:"encrypted"`
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
}
// Stack represents a docker-compose stack managed as a single deployable unit.
type Stack struct {
ID string `json:"id"`
Name string `json:"name"`
Description string `json:"description"`
ComposeProjectName string `json:"compose_project_name"` // `-p` arg for docker compose
Status string `json:"status"` // stopped, deploying, running, failed
Error string `json:"error"`
CurrentRevisionID string `json:"current_revision_id"`
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
}
// StackRevision is an append-only record of a YAML version for a stack.
// Rollback = insert a new revision whose YAML is copied from an older one.
type StackRevision struct {
ID string `json:"id"`
StackID string `json:"stack_id"`
Revision int `json:"revision"` // monotonic per stack
YAML string `json:"yaml"`
Author string `json:"author"`
DeployID string `json:"deploy_id"`
Status string `json:"status"` // pending, success, failed
CreatedAt string `json:"created_at"`
}
// StackDeploy records a deployment attempt of a specific revision.
type StackDeploy struct {
ID string `json:"id"`
StackID string `json:"stack_id"`
RevisionID string `json:"revision_id"`
Status string `json:"status"` // pending, deploying, success, failed, rolled_back
Log string `json:"log"`
Error string `json:"error"`
StartedAt string `json:"started_at"`
FinishedAt string `json:"finished_at"`
}
// EventLog represents a persistent event log entry.
type EventLog struct {
ID int64 `json:"id"`
Source string `json:"source"`
Severity string `json:"severity"` // info, warn, error
Message string `json:"message"`
Metadata string `json:"metadata"` // JSON-encoded structured data
CreatedAt string `json:"created_at"`
}
// EventTrigger is a filter+action rule evaluated against EventLog
// entries published on the bus. When all non-empty filters match, the
// trigger fires its configured action (webhook today, additional action
// types extensible via the ActionType enum).
//
// Filter fields use a comma-separated list shape for multi-value
// filters (severity, source) to keep the schema flat — empty string
// means "no filter on this dimension." FilterMessageRegex is a single
// regex evaluated against EventLog.Message.
//
// Loop-prevention: deliveries are recorded in webhook_deliveries (the
// existing audit trail). The dispatcher MUST NOT write to event_log
// or it will recurse.
type EventTrigger struct {
ID int64 `json:"id"`
Name string `json:"name"`
FilterSeverity string `json:"filter_severity"` // comma list: "warn,error"; "" = any
FilterSource string `json:"filter_source"` // comma list: "logscan,deploy"; "" = any
FilterMessageRegex string `json:"filter_message_regex"` // "" = any
ActionType string `json:"action_type"` // "webhook" today
ActionTarget string `json:"action_target"` // URL for webhook
ActionSecret string `json:"action_secret"` // optional HMAC secret for signed delivery
Enabled bool `json:"enabled"`
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
}
// EventTriggerActionType enumerates the supported action_type values.
// Adding a new action is additive — old triggers keep working, the
// dispatcher just learns a new branch.
const (
EventTriggerActionWebhook = "webhook"
)
// LogScanRule is one regex-based pattern the log scanner evaluates
// against container log lines. The (workload_id, overrides_id) pair
// implements the "global rule with optional per-workload override"
// pattern documented in docs/LOGSCAN_AND_TRIGGERS_TODO.md:
//
// - WorkloadID == "" && OverridesID == 0 → global rule, applies to
// every workload unless overridden.
// - WorkloadID != "" && OverridesID == 0 → workload-only addition.
// - WorkloadID != "" && OverridesID != 0 → override of the named
// global rule for one workload (Enabled=false to disable globally
// for this workload).
type LogScanRule struct {
ID int64 `json:"id"`
WorkloadID string `json:"workload_id"` // "" = global
OverridesID int64 `json:"overrides_id"` // 0 = not an override
Name string `json:"name"`
Pattern string `json:"pattern"` // regex, compiled at load
Severity string `json:"severity"` // info|warn|error
Streams string `json:"streams"` // all|stdout|stderr
CooldownSeconds int `json:"cooldown_seconds"`
Enabled bool `json:"enabled"`
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
}
// Log scan stream filter values. "all" reads both streams; "stdout"
// or "stderr" filter to one. Used both for store validation and at
// docker-side log read time.
const (
LogScanStreamAll = "all"
LogScanStreamStdout = "stdout"
LogScanStreamStderr = "stderr"
)
// Log scan severity values mirror the event_log enum so a matched
// rule lands as an event_log row with the rule's severity verbatim.
const (
LogScanSeverityInfo = "info"
LogScanSeverityWarn = "warn"
LogScanSeverityError = "error"
)
// WorkloadKind enumerates the kinds of things that own containers.
// Each kind has a corresponding row in projects/stacks/static_sites referenced via Workload.RefID.
type WorkloadKind string
const (
WorkloadKindProject WorkloadKind = "project"
WorkloadKindStack WorkloadKind = "stack"
WorkloadKindSite WorkloadKind = "site"
)
// Workload is the unifying primitive that abstracts Project, Stack, and StaticSite.
// Each row is paired with exactly one project/stack/site via (Kind, RefID).
// Notification + webhook config moves here so it lives in one place across kinds.
//
// SourceKind / SourceConfig / TriggerKind / TriggerConfig / PublicFaces /
// ParentWorkloadID populate the unified plugin model from the Workload-first
// refactor. Existing rows keep these empty until they are explicitly migrated
// or replaced — the legacy Kind/RefID columns continue to point at
// project/stack/site rows in parallel during the cutover.
type Workload struct {
ID string `json:"id"`
Kind string `json:"kind"` // project | stack | site (legacy discriminator)
RefID string `json:"ref_id"`
Name string `json:"name"`
AppID string `json:"app_id"` // nullable; "" = unassigned (a.k.a. GroupID after rename)
SourceKind string `json:"source_kind"` // "" until plugin-mode populated
SourceConfig string `json:"source_config"` // JSON-encoded, decoded by the matching Source
TriggerKind string `json:"trigger_kind"`
TriggerConfig string `json:"trigger_config"` // JSON-encoded, decoded by the matching Trigger
PublicFaces string `json:"public_faces"` // JSON-encoded []PublicFace
ParentWorkloadID string `json:"parent_workload_id"` // "" = root; non-empty = stage chain
NotificationURL string `json:"notification_url"`
NotificationSecret string `json:"-"` // never serialized
WebhookSecret string `json:"-"` // URL-identifier secret; never serialized
WebhookSigningSecret string `json:"-"` // HMAC key; never serialized
WebhookRequireSignature bool `json:"webhook_require_signature"`
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
}
// Container is the normalized index of every Tinyforge-managed container.
// Replaces the project-specific Instance table after migration. Subdomain/
// proxy fields are hoisted as first-class columns because ListProxyRoutes,
// stale detection, and dashboard queries filter on them frequently.
//
// StageID is populated by the deployer for project containers so ListProxyRoutes
// survives stage renames; it stays empty for stack and site rows.
type Container struct {
ID string `json:"id"`
WorkloadID string `json:"workload_id"`
WorkloadKind string `json:"workload_kind"` // denormalized for filtered queries
Role string `json:"role"` // stage name (project), service name (stack), '' (site)
StageID string `json:"stage_id"` // project containers only; '' otherwise
ContainerID string `json:"container_id"` // Docker container ID; '' between create+start
ImageRef string `json:"image_ref"` // "image:tag" as scheduled
ImageTag string `json:"image_tag"` // just the tag, for ListProxyRoutes
Host string `json:"host"`
State string `json:"state"` // running | stopped | failed | removing | missing
Port int `json:"port"`
Subdomain string `json:"subdomain"`
ProxyRouteID string `json:"proxy_route_id"`
NpmProxyID int `json:"npm_proxy_id"`
LastSeenAt string `json:"last_seen_at"`
// ExtraJSON carries source-specific metadata that isn't promoted to a
// first-class column — currently per-face proxy route IDs for
// multi-face image deploys. Stored as a JSON object; '{}' on empty
// rows. Sources own the shape; consumers should tolerate unknown
// keys.
ExtraJSON string `json:"extra_json"`
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
}
// App is an optional grouping of workloads (e.g., "my-saas" = web project + worker stack + redis stack).
// Schema lives here from day one so future UI work is unblocked, but no UI is wired in v1.
type App struct {
ID string `json:"id"`
Name string `json:"name"`
Description string `json:"description"`
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
}