tiny-forge/docs/CODEMAPS/container-extra-json.md

containers.extra_json — Evolution Policy

Last Updated: 2026-05-16

extra_json is a TEXT column on the containers table that source plugins use to persist source-specific runtime state that hasn't been promoted to a first-class column. It is the single forward-compatibility seam between the canonical container row and per-source needs that arise after a schema is in production.

This doc captures the rules every reader and writer must follow so new sources can extend the blob without breaking older ones.

Schema position

• Column: containers.extra_json TEXT NOT NULL DEFAULT '{}' (internal/store/store.go:233).
• All four write paths (CreateContainer, UpsertContainer, ReconcileContainer, UpdateContainer) normalize "" → '{}' before the SQL exec — readers can assume a non-empty JSON object string and never need to handle SQL NULL or the empty-string edge.
• Defined on the Container model: internal/store/models.go:342-347.

Ownership model

One container row → one owning source. Sources never write to a row that belongs to another source. In practice:

Source kind	Row key	Number of rows per workload	Writes extra_json today?
static	deterministic <workloadID>:site	exactly 1	yes (preserve-unknown-keys)
image	UUID per deployed container	1 + N (blue-green rolls)	yes (wholesale-overwrite)
compose	deterministic <workloadID>:<service>	N (one per compose service)	no — left at '{}' default

Two sources cannot contend on the same row, so the policy below is concerned with forward compatibility across versions of the same source, not cross-source contention. When compose (or any future source) starts writing extra_json, the same rules apply.

Reader rules — ALL readers

1. Tolerate unknown keys. Decode into a typed struct using encoding/json; Go's default unmarshaller silently drops unknown keys, which is the desired behaviour. Never use json.Decoder.DisallowUnknownFields() on extra_json.
2. Tolerate decode failure as non-fatal where the row's first-class columns are useful. A corrupted extra_json is debug-logged and the reader falls back to zero state — see workload_runtime.go:118-133 for the canonical pattern. The container's ContainerID, State, ProxyRouteID, etc. live in their own columns and are still trustworthy.
3. Tolerate '' and '{}'. Both are equivalent to "no extras yet". Readers must short-circuit before json.Unmarshal to avoid unexpected end of JSON input on the empty case.

Writer rules — by mutation style

Two distinct write patterns live in the codebase today. Pick the one that matches your source's needs.

Wholesale-overwrite (image source pattern)

When the writer owns 100% of the blob's shape and discards old contents on every write:

// internal/workload/plugin/source/image/image.go:341-343
extra := containerExtra{ProxyRoutes: faceRoutes}
if b, err := json.Marshal(extra); err == nil {
    created.ExtraJSON = string(b)
}

• Cheap and simple.
• Loses unknown keys written by future versions of the same source. Only use when you are certain no other writer (including a future version of this code) needs to round-trip an unknown key.
• The containerExtra struct must be additive-only: never rename or remove a field once shipped, and never change its JSON type. Mark new fields with omitempty so older readers downgrading to an older codebase don't see surprise nulls.

Preserve-unknown-keys (static source pattern)

When future versions of the source (or sibling writers) may add fields and the current writer must round-trip them:

// internal/workload/plugin/source/static/state.go saveState
//   1. Decode existing blob into map[string]json.RawMessage.
//   2. Strip every key the current typed-state struct owns
//      (runtimeStateKeys) so a cleared field actually drops.
//   3. Apply caller's mutate() to the typed state.
//   4. Re-marshal typed state, splice its keys back into the
//      generic map (overwriting any historical sibling).
//   5. Marshal the merged map back into extra_json.

• Slightly more expensive (two round-trips through json).
• Preserves keys the current writer doesn't know about — required for safe rolling deploys where a newer instance writes a new key, an older instance then reads, mutates, and writes back.
• Must declare the typed key set explicitly (runtimeStateKeys) so step 2 can strip them. This invariant is fenced by TestRuntimeState_JSONTagsRoundTrip in state_integration_test.go.

Default to preserve-unknown-keys for any new source. Wholesale-overwrite is acceptable for the image source today because the row's lifetime is short (replaced on every blue-green roll) and only one writer touches it. Sources whose container rows are long-lived (static, future compose-with-stateful-services) should preserve unknown keys.

Concurrency

UpsertContainer is atomic at the SQL layer — SQLite serializes statements through one connection (internal/store/store.go:55 SetMaxOpenConns(1)) with WAL mode enabled (store.go:60). That guarantees no torn write on a single row, and concurrent readers see a consistent snapshot — they read either the pre- or post-write state, never a half-applied one.

What that does not guarantee is atomic read-modify-write across two Go goroutines. The static source serializes its RMW through a per-workload sync.Mutex keyed by workload ID (internal/workload/plugin/source/static/state.go lockFor + saveState). Any source that does its own read-modify-write on extra_json must do the same — verified in TestSaveState_ConcurrentWritesDoNotLoseUpdates (which loses 15+ markers per 20-writer run when the mutex is disabled, as confirmed in commit ef62a41).

If a future source is purely wholesale-overwrite from a single writer, no lock is needed.

What extra_json is NOT for

• Workload-level config. Workload config goes in workloads.source_config and is the operator's surface.
• Cross-source state. If two sources need the same data, promote it to a column.
• Anything queryable. SQLite can JSON-path extra_json but no index supports it; readers always pull the column wholesale and parse in Go.
• Secrets. Anything sensitive lives in workload_env (per-entry encrypt flag) or another encrypted table.

Adding a new field — checklist

1. Add the field to your source's typed struct with omitempty and a stable json:"snake_case" tag.
2. If you use the preserve-unknown-keys pattern, add the JSON key to your *Keys slice (the equivalent of runtimeStateKeys).
3. Confirm older readers (older deploys of the same binary) still parse the blob — encoding/json should drop the unknown key silently. Add a regression test if there's any doubt.
4. Document the new field in this codemap if it's load-bearing for cross-source code (e.g., the proxy_routes map drives ListProxyRoutes).

Pointers

• Container model + ExtraJSON comment: internal/store/models.go:342-347
• Schema declaration: internal/store/store.go:233
• Store-level normalization ('{}' default) across all four write paths: internal/store/containers.go:42-43 (CreateContainer), :77-78 (UpsertContainer), :129-130 (ReconcileContainer), :321-322 (UpdateContainer).
• Wholesale-overwrite writer + struct: image.go:341-343 writes; image.go:481-487 defines containerExtra; image.go:449-456 reads it back in Teardown.
• Preserve-unknown-keys example + concurrency lock: internal/workload/plugin/source/static/state.go.
• Canonical "decode-and-tolerate" consumer (the only cross-source reader in tree today): internal/api/workload_runtime.go:118-133 decodes the static-only typed fields and falls back to first-class columns when the blob is empty, missing keys, or malformed.

Note: no cross-source consumer reads extra_json in internal/store/. The proxy/route data exposed by ListProxyRoutes (containers.go:196) comes from first-class columns (proxy_route_id, subdomain, port); the proxy_routes map inside extra_json is read only by the image source's own Teardown for cleanup.