tiny-forge/docs/CODEMAPS/container-extra-json.md

# `containers.extra_json` — Evolution Policy

**Last Updated:** 2026-05-16

`extra_json` is a TEXT column on the `containers` table that source plugins use to persist source-specific runtime state that hasn't been promoted to a first-class column. It is the single forward-compatibility seam between the canonical container row and per-source needs that arise after a schema is in production.

This doc captures the rules every reader and writer must follow so new sources can extend the blob without breaking older ones.

## Schema position

- Column: `containers.extra_json TEXT NOT NULL DEFAULT '{}'` ([`internal/store/store.go:233`](../../internal/store/store.go#L233)).
- All four write paths (`CreateContainer`, `UpsertContainer`, `ReconcileContainer`, `UpdateContainer`) normalize `""` → `'{}'` before the SQL exec — readers can assume a non-empty JSON object string and never need to handle SQL `NULL` or the empty-string edge.
- Defined on the `Container` model: [`internal/store/models.go:342-347`](../../internal/store/models.go#L342-L347).

## Ownership model

**One container row → one owning source.** Sources never write to a row that belongs to another source. In practice:

| Source kind | Row key                                | Number of rows per workload | Writes `extra_json` today?  |
| ----------- | -------------------------------------- | --------------------------- | --------------------------- |
| `static`    | deterministic `<workloadID>:site`      | exactly 1                   | yes (preserve-unknown-keys) |
| `image`     | UUID per deployed container            | 1 + N (blue-green rolls)    | yes (wholesale-overwrite)   |
| `compose`   | deterministic `<workloadID>:<service>` | N (one per compose service) | no — left at `'{}'` default |

Two sources cannot contend on the same row, so the policy below is concerned with **forward compatibility across versions of the same source**, not cross-source contention. When compose (or any future source) starts writing `extra_json`, the same rules apply.

## Reader rules — ALL readers

1. **Tolerate unknown keys.** Decode into a typed struct using `encoding/json`; Go's default unmarshaller silently drops unknown keys, which is the desired behaviour. Never use `json.Decoder.DisallowUnknownFields()` on `extra_json`.
2. **Tolerate decode failure as non-fatal where the row's first-class columns are useful.** A corrupted `extra_json` is debug-logged and the reader falls back to zero state — see `workload_runtime.go:118-133` for the canonical pattern. The container's `ContainerID`, `State`, `ProxyRouteID`, etc. live in their own columns and are still trustworthy.
3. **Tolerate `''` and `'{}'`.** Both are equivalent to "no extras yet". Readers must short-circuit before json.Unmarshal to avoid `unexpected end of JSON input` on the empty case.

## Writer rules — by mutation style

Two distinct write patterns live in the codebase today. Pick the one that matches your source's needs.

### Wholesale-overwrite (image source pattern)

When the writer owns 100% of the blob's shape and discards old contents on every write:

```go
// internal/workload/plugin/source/image/image.go:341-343
extra := containerExtra{ProxyRoutes: faceRoutes}
if b, err := json.Marshal(extra); err == nil {
    created.ExtraJSON = string(b)
}
```

- Cheap and simple.
- **Loses unknown keys written by future versions of the same source.** Only use when you are certain no other writer (including a future version of this code) needs to round-trip an unknown key.
- The `containerExtra` struct must be **additive-only**: never rename or remove a field once shipped, and never change its JSON type. Mark new fields with `omitempty` so older readers downgrading to an older codebase don't see surprise nulls.

### Preserve-unknown-keys (static source pattern)

When future versions of the source (or sibling writers) may add fields and the current writer must round-trip them:

```go
// internal/workload/plugin/source/static/state.go saveState
//   1. Decode existing blob into map[string]json.RawMessage.
//   2. Strip every key the current typed-state struct owns
//      (runtimeStateKeys) so a cleared field actually drops.
//   3. Apply caller's mutate() to the typed state.
//   4. Re-marshal typed state, splice its keys back into the
//      generic map (overwriting any historical sibling).
//   5. Marshal the merged map back into extra_json.
```

- Slightly more expensive (two round-trips through `json`).
- Preserves keys the current writer doesn't know about — required for safe rolling deploys where a newer instance writes a new key, an older instance then reads, mutates, and writes back.
- Must declare the typed key set explicitly (`runtimeStateKeys`) so step 2 can strip them. This invariant is fenced by `TestRuntimeState_JSONTagsRoundTrip` in [`state_integration_test.go`](../../internal/workload/plugin/source/static/state_integration_test.go).

**Default to preserve-unknown-keys for any new source.** Wholesale-overwrite is acceptable for the image source today because the row's lifetime is short (replaced on every blue-green roll) and only one writer touches it. Sources whose container rows are long-lived (static, future compose-with-stateful-services) should preserve unknown keys.

## Concurrency

`UpsertContainer` is atomic at the SQL layer — SQLite serializes statements through one connection ([`internal/store/store.go:55`](../../internal/store/store.go#L55) `SetMaxOpenConns(1)`) with WAL mode enabled ([`store.go:60`](../../internal/store/store.go#L60)). That guarantees no torn write on a single row, and concurrent readers see a consistent snapshot — they read either the pre- or post-write state, never a half-applied one.

What that does **not** guarantee is atomic read-modify-write across two Go goroutines. The static source serializes its RMW through a per-workload `sync.Mutex` keyed by workload ID (`internal/workload/plugin/source/static/state.go` `lockFor` + `saveState`). Any source that does its own read-modify-write on `extra_json` must do the same — verified in `TestSaveState_ConcurrentWritesDoNotLoseUpdates` (which loses 15+ markers per 20-writer run when the mutex is disabled, as confirmed in commit `ef62a41`).

If a future source is purely wholesale-overwrite from a single writer, no lock is needed.

## What `extra_json` is NOT for

- **Workload-level config.** Workload config goes in `workloads.source_config` and is the operator's surface.
- **Cross-source state.** If two sources need the same data, promote it to a column.
- **Anything queryable.** SQLite can JSON-path `extra_json` but no index supports it; readers always pull the column wholesale and parse in Go.
- **Secrets.** Anything sensitive lives in `workload_env` (per-entry encrypt flag) or another encrypted table.

## Adding a new field — checklist

1. Add the field to your source's typed struct with `omitempty` and a stable `json:"snake_case"` tag.
2. If you use the **preserve-unknown-keys** pattern, add the JSON key to your `*Keys` slice (the equivalent of `runtimeStateKeys`).
3. Confirm older readers (older deploys of the same binary) still parse the blob — `encoding/json` should drop the unknown key silently. Add a regression test if there's any doubt.
4. Document the new field in this codemap if it's load-bearing for cross-source code (e.g., the proxy_routes map drives `ListProxyRoutes`).

## Pointers

- Container model + `ExtraJSON` comment: [`internal/store/models.go:342-347`](../../internal/store/models.go#L342-L347)
- Schema declaration: [`internal/store/store.go:233`](../../internal/store/store.go#L233)
- Store-level normalization (`'{}'` default) across all four write paths: [`internal/store/containers.go:42-43`](../../internal/store/containers.go#L42-L43) (CreateContainer), `:77-78` (UpsertContainer), `:129-130` (ReconcileContainer), `:321-322` (UpdateContainer).
- Wholesale-overwrite writer + struct: [`image.go:341-343`](../../internal/workload/plugin/source/image/image.go#L341-L343) writes; [`image.go:481-487`](../../internal/workload/plugin/source/image/image.go#L481-L487) defines `containerExtra`; [`image.go:449-456`](../../internal/workload/plugin/source/image/image.go#L449-L456) reads it back in Teardown.
- Preserve-unknown-keys example + concurrency lock: [`internal/workload/plugin/source/static/state.go`](../../internal/workload/plugin/source/static/state.go).
- Canonical "decode-and-tolerate" consumer (the only cross-source reader in tree today): [`internal/api/workload_runtime.go:118-133`](../../internal/api/workload_runtime.go#L118-L133) decodes the static-only typed fields and falls back to first-class columns when the blob is empty, missing keys, or malformed.

Note: no cross-source consumer reads `extra_json` in `internal/store/`. The proxy/route data exposed by `ListProxyRoutes` ([`containers.go:196`](../../internal/store/containers.go#L196)) comes from first-class columns (`proxy_route_id`, `subdomain`, `port`); the `proxy_routes` map inside `extra_json` is read only by the image source's own Teardown for cleanup.