feat(processed-audio-sources): phase 1 - audio filter framework

Add the foundation for audio processing filters, mirroring the existing
picture filter/postprocessing template system:
- AudioFilter base class, AudioFilterRegistry, AudioFilterOptionDef
- AudioProcessingTemplate dataclass + SQLite-backed store
- audio_filter_template meta-filter with recursive resolution
- Full REST API: CRUD templates + filter registry discovery
- Dependency injection wired in dependencies.py and main.py
This commit is contained in:
2026-03-31 17:35:39 +02:00
parent c59107c7c7
commit 86a9d344e6
40 changed files with 1498 additions and 1251 deletions
+103
View File
@@ -0,0 +1,103 @@
# Feature Context: Processed Audio Sources
## Configuration
- **Development mode:** Automated
- **Execution mode:** Orchestrator
- **Strategy:** Big Bang
- **Build (Python):** `cd server && ruff check src/ tests/ --fix`
- **Build (TypeScript):** `cd server && npx tsc --noEmit && npm run build`
- **Test:** `cd server && py -3.13 -m pytest tests/ --no-cov -q`
## Current State
Phase 1 (Audio Filter Framework) implemented. Core framework is in place:
- `AudioFilter` base class, `AudioFilterRegistry`, `AudioFilterOptionDef` in `core/audio/filters/`
- `AudioProcessingTemplate` dataclass + `AudioProcessingTemplateStore` (SQLite-backed) in `storage/`
- `audio_filter_template` meta-filter with recursive resolution
- Full REST API: CRUD templates + filter registry discovery
- Dependency injection wired in `dependencies.py` and `main.py`
## Key Architecture Reference
### Existing Pattern to Mirror: Processed Picture Sources
- `ProcessedPictureSource` references `source_stream_id` + `postprocessing_template_id`
- `PostprocessingTemplate` contains ordered `List[FilterInstance]`
- `FilterInstance` = `filter_id` (string) + `options` (dict)
- `FilterRegistry` handles registration, lookup, instantiation
- `filter_template` meta-filter embeds one template inside another
- `PostprocessingTemplateStore` has `resolve_filter_instances()` for recursive expansion
- Picture filters transform images; audio filters will transform `AudioAnalysis`
### Current Audio Source Types (to be replaced)
- `MultichannelAudioSource` → renamed to `CaptureAudioSource`
- `MonoAudioSource` → removed, replaced by channel_extract filter
- `BandExtractAudioSource` → removed, replaced by band_extract filter
### AudioAnalysis Structure (filter input/output)
```python
AudioAnalysis:
timestamp: float
rms: float # Overall RMS level
peak: float # Peak amplitude
spectrum: np.ndarray[64] # Log-spaced FFT bands
beat: bool # Beat detected
beat_intensity: float # 0-1 beat strength
left_rms: float # Left channel RMS
left_spectrum: np.ndarray # Left channel spectrum
right_rms: float # Right channel RMS
right_spectrum: np.ndarray # Right channel spectrum
```
### Key Existing Files
- `storage/audio_source.py` — current source dataclasses
- `storage/audio_source_store.py` — CRUD + resolve_audio_source()
- `core/audio/analysis.py` — AudioAnalyzer, AudioAnalysis
- `core/audio/band_filter.py` — existing band filtering logic
- `core/processing/audio_stream.py` — AudioColorStripStream
- `core/processing/value_stream.py` — AudioValueStream
- `core/filters/base.py` — PostprocessingFilter (picture filter base class)
- `core/filters/registry.py` — FilterRegistry (picture filters)
- `storage/postprocessing_template.py` — PostprocessingTemplate dataclass
- `storage/postprocessing_template_store.py` — template store with resolve_filter_instances()
## Temporary Workarounds
_(none yet)_
## Cross-Phase Dependencies
- Phase 2 depends on Phase 1 (filter framework)
- Phase 3 depends on Phase 1 (template store for ProcessedAudioSource)
- Phase 4 depends on Phases 1-3 (all backend pieces)
- Phase 5 depends on Phase 1 (template API)
- Phase 6 depends on Phase 3 (source type API)
- Phase 7 depends on all prior phases
## Deferred Work
_(none yet)_
## Failed Approaches
_(none yet)_
## Review Findings Log
_(none yet)_
## Phase Execution Log
| Phase | Agent Used | Test Writer | Parallel | Notes |
|-------|-----------|-------------|----------|-------|
| Phase 1 | impl-agent | — | No | Tasks 7+8 skipped (SQLite migration made them obsolete) |
| Phase 2 | — | — | — | — |
| Phase 3 | — | — | — | — |
| Phase 4 | — | — | — | — |
| Phase 5 | — | — | — | — |
| Phase 6 | — | — | — | — |
| Phase 7 | — | — | — | — |
## Environment & Runtime Notes
- Platform: Windows 10
- Python: 3.13
- Server port: 8080
- Shell: bash (Git Bash on Windows)
## Implementation Notes
- Clean-slate approach: no migration of existing MonoAudioSource/BandExtractAudioSource data
- 5 of 11 filters are stateful (peak hold, envelope follower, spectral smoothing, compressor, delay) — need per-stream instance lifecycle
- Audio filters operate on AudioAnalysis snapshots, not raw audio samples
- Big Bang strategy: intermediate phases may break the build; only Phase 7 enforces build/tests
+55
View File
@@ -0,0 +1,55 @@
# Feature: Processed Audio Sources
**Branch:** `feature/processed-audio-sources`
**Base branch:** `master`
**Created:** 2026-03-31
**Status:** 🟡 In Progress
**Strategy:** Big Bang
**Mode:** Automated
**Execution:** Orchestrator
## Summary
Replace hardcoded `MonoAudioSource` and `BandExtractAudioSource` types with a composable
**ProcessedAudioSource + AudioProcessingTemplate + AudioFilter** system — mirroring the
existing processed picture source pattern. Rename `MultichannelAudioSource` to
`CaptureAudioSource`. Adds 11 audio filters: channel extract, band extract, peak hold,
gain, noise gate, envelope follower, spectral smoothing, compressor, inverter, beat gate,
and delay.
Clean-slate approach: no data migration for old source types.
## Build & Test Commands
- **Build (Python):** `cd server && ruff check src/ tests/ --fix`
- **Build (TypeScript):** `cd server && npx tsc --noEmit && npm run build`
- **Test:** `cd server && py -3.13 -m pytest tests/ --no-cov -q`
## Phases
- [ ] Phase 1: Audio Filter Framework [domain: backend] → [subplan](./phase-1-audio-filter-framework.md)
- [ ] Phase 2: Audio Filters [domain: backend] → [subplan](./phase-2-audio-filters.md)
- [ ] Phase 3: Processed Audio Source Model [domain: backend] → [subplan](./phase-3-processed-audio-source-model.md)
- [ ] Phase 4: Runtime Integration [domain: backend] → [subplan](./phase-4-runtime-integration.md)
- [ ] Phase 5: Frontend — Audio Processing Templates [domain: frontend] → [subplan](./phase-5-frontend-templates.md)
- [ ] Phase 6: Frontend — Source Types [domain: frontend] → [subplan](./phase-6-frontend-source-types.md)
- [ ] Phase 7: Testing & Polish [domain: backend] → [subplan](./phase-7-testing-polish.md)
- [ ] Phase 8: Frontend Design Consistency Review [domain: frontend] → [subplan](./phase-8-frontend-design-review.md)
## Phase Progress Log
| Phase | Domain | Status | Review | Build | Committed |
|-------|--------|--------|--------|-------|-----------|
| Phase 1: Audio Filter Framework | backend | 🔨 In Progress | ⬜ | ⬜ | ⬜ |
| Phase 2: Audio Filters | backend | ⬜ Not Started | ⬜ | ⬜ | ⬜ |
| Phase 3: Processed Audio Source Model | backend | ⬜ Not Started | ⬜ | ⬜ | ⬜ |
| Phase 4: Runtime Integration | backend | ⬜ Not Started | ⬜ | ⬜ | ⬜ |
| Phase 5: Frontend — Audio Processing Templates | frontend | ⬜ Not Started | ⬜ | ⬜ | ⬜ |
| Phase 6: Frontend — Source Types | frontend | ⬜ Not Started | ⬜ | ⬜ | ⬜ |
| Phase 7: Testing & Polish | backend | ⬜ Not Started | ⬜ | ⬜ | ⬜ |
| Phase 8: Frontend Design Review | frontend | ⬜ Not Started | ⬜ | ⬜ | ⬜ |
## Final Review
- [ ] Comprehensive code review
- [ ] Full build passes
- [ ] Full test suite passes
- [ ] Merged to `master`
@@ -0,0 +1,106 @@
# Phase 1: Audio Filter Framework
**Status:** 🔨 In Progress
**Parent plan:** [PLAN.md](./PLAN.md)
**Domain:** backend
## Objective
Build the foundation for audio processing: base filter class, registry, template storage, and API endpoints. This mirrors the existing picture filter/postprocessing template system.
## Tasks
- [x] Task 1: Create `AudioFilter` base class in `core/audio/filters/base.py`
- Abstract `process(analysis: AudioAnalysis) -> AudioAnalysis` method
- `is_stateful` property (False by default, overridden by stateful filters)
- `reset()` method for stateful filters
- `AudioFilterOptionDef` class for declaring filter option schemas (mirrors `FilterOptionDef` from picture filters)
- Class-level `filter_id`, `name`, `description`, `options` declarations
- [x] Task 2: Create `AudioFilterRegistry` in `core/audio/filters/registry.py`
- `register(filter_class)` — register by filter_id
- `get_filter_class(filter_id)` — lookup
- `create_instance(filter_id, options)` → instantiated AudioFilter
- `get_available_filters()` → list of filter metadata + option schemas
- Mirrors `FilterRegistry` from `core/filters/registry.py`
- [x] Task 3: Create `AudioProcessingTemplate` dataclass in `storage/audio_processing_template.py`
- Fields: `id`, `name`, `filters: List[FilterInstance]`, `description`, `tags`, `created_at`, `updated_at`
- Reuse existing `FilterInstance` from `core/filters/filter_instance.py`
- ID prefix: `apt_` (audio processing template)
- [x] Task 4: Create `AudioProcessingTemplateStore` in `storage/audio_processing_template_store.py`
- SQLite-backed CRUD (same pattern as `PostprocessingTemplateStore`)
- `resolve_filter_instances()` — recursive expansion of `audio_filter_template` meta-filter
- Cycle detection in template composition
- Reference validation (check filter_ids exist in registry)
- [x] Task 5: Create `audio_filter_template` meta-filter in `core/audio/filters/audio_filter_template.py`
- Option: `template_id` referencing another AudioProcessingTemplate
- Never instantiated at runtime — expanded during resolution
- Mirrors `filter_template` from picture filters
- [x] Task 6: Create `core/audio/filters/__init__.py` — register all filters with the registry
- [x] Task 7: ~~Add `audio_processing_templates_file` to `StorageConfig` in `config.py`~~ SKIPPED — storage uses SQLite now, not JSON files. No config change needed.
- [x] Task 8: ~~Add audio processing templates to `STORE_MAP` in `api/routes/system.py`~~ SKIPPED — STORE_MAP no longer exists. Backup/restore works at the SQLite database level; new tables are automatically included.
- [x] Task 9: Create API schemas in `api/schemas/audio_processing.py`
- `AudioProcessingTemplateCreate`, `AudioProcessingTemplateUpdate`, `AudioProcessingTemplateResponse`
- `AudioFilterInstanceSchema` (reuse `FilterInstanceSchema` from existing schemas)
- [x] Task 10: Create API routes in `api/routes/audio_processing_templates.py`
- `GET /api/v1/audio-processing-templates` — list all
- `POST /api/v1/audio-processing-templates` — create
- `GET /api/v1/audio-processing-templates/{template_id}` — get one
- `PUT /api/v1/audio-processing-templates/{template_id}` — update
- `DELETE /api/v1/audio-processing-templates/{template_id}` — delete (with ref checks placeholder)
- [x] Task 11: Create filter registry endpoint in `api/routes/audio_filters.py`
- `GET /api/v1/audio-filters` — returns available filters with option schemas
- Dynamically populates `audio_filter_template` options with current template IDs
- [x] Task 12: Register new routes in the FastAPI app
## Files to Modify/Create
- `core/audio/filters/base.py`**created** — AudioFilter base class + AudioFilterOptionDef
- `core/audio/filters/registry.py`**created** — AudioFilterRegistry
- `core/audio/filters/audio_filter_template.py`**created** — meta-filter
- `core/audio/filters/__init__.py`**created** — filter registration
- `storage/audio_processing_template.py`**created** — dataclass
- `storage/audio_processing_template_store.py`**created** — store
- `api/schemas/audio_processing.py`**created** — Pydantic schemas
- `api/routes/audio_processing_templates.py`**created** — template CRUD routes
- `api/routes/audio_filters.py`**created** — filter registry endpoint
- `api/dependencies.py`**modified** — added getter + init param for audio_processing_template_store
- `api/__init__.py`**modified** — registered new routers
- `main.py`**modified** — created store instance, imported audio filter package, wired to init_dependencies
## Acceptance Criteria
- AudioFilter base class exists with process/reset/options API
- AudioFilterRegistry can register, lookup, and instantiate filters
- AudioProcessingTemplate can be created, read, updated, deleted via API
- `audio_filter_template` meta-filter is handled during resolution (recursive expansion)
- Filter registry endpoint returns available filters with option schemas
- Backup/restore includes audio processing templates (automatic via SQLite)
## Notes
- Reuse `FilterInstance` from `core/filters/filter_instance.py` — no need to create a separate one for audio
- The `audio_filter_template` meta-filter option should dynamically list available template IDs (like the picture filter version)
- Phase 2 will register the actual filters; this phase just sets up the framework with only the meta-filter registered
- Tasks 7 and 8 from original plan were skipped — the codebase migrated from JSON files to SQLite since the plan was written
## Review Checklist
- [ ] All tasks completed
- [ ] Code follows project conventions
- [ ] No unintended side effects
- [ ] Build passes
- [ ] Tests pass (new + existing)
## Handoff to Next Phase
### What was built
- Complete audio filter framework: `AudioFilter` base class, `AudioFilterRegistry`, `AudioFilterOptionDef`
- `AudioProcessingTemplate` dataclass + `AudioProcessingTemplateStore` (SQLite-backed CRUD)
- `audio_filter_template` meta-filter with recursive resolution and cycle detection
- Full REST API: CRUD for templates + filter registry discovery endpoint
- Dependency injection wired through `dependencies.py` and `main.py`
### What Phase 2 needs to know
- Register new audio filters by importing them in `core/audio/filters/__init__.py` and decorating with `@AudioFilterRegistry.register`
- Each filter implements `process(analysis: AudioAnalysis) -> AudioAnalysis`
- Stateful filters override `is_stateful` property to return `True` and implement `reset()`
- The `AudioFilterOptionDef` class mirrors `FilterOptionDef` exactly (same option_type values, same validation logic)
### Known deviations from plan
- Tasks 7 and 8 skipped: `StorageConfig` no longer holds per-entity file paths (SQLite migration), and `STORE_MAP` no longer exists (database-level backup/restore)
- Delete endpoint has a TODO placeholder for reference checks — Phase 3 will add `ProcessedAudioSource` which references templates
@@ -0,0 +1,97 @@
# Phase 2: Audio Filters
**Status:** ⬜ Not Started
**Parent plan:** [PLAN.md](./PLAN.md)
**Domain:** backend
## Objective
Implement all 11 audio filters and register them with the AudioFilterRegistry.
## Tasks
- [ ] Task 1: **Channel Extract** filter (`core/audio/filters/channel_extract.py`)
- Options: `channel` (select: mono | left | right)
- Stateful: No
- Behavior: Replaces main rms/spectrum with selected channel data. If "mono", averages L+R. If "left"/"right", copies that channel's data to the main fields.
- [ ] Task 2: **Band Extract** filter (`core/audio/filters/band_extract.py`)
- Options: `band` (select: bass | mid | treble | custom), `freq_low` (float, 20-20000), `freq_high` (float, 20-20000)
- Stateful: No
- Behavior: Computes a band mask for the 64 log-spaced bins, applies it to spectrum, recomputes RMS from in-band data. Reuse logic from existing `core/audio/band_filter.py`.
- Presets: bass=20-250Hz, mid=250-4000Hz, treble=4000-20000Hz
- [ ] Task 3: **Peak Hold** filter (`core/audio/filters/peak_hold.py`)
- Options: `decay_rate` (float, 0.1-50.0, dB/s), `per_bin` (bool, default true)
- Stateful: Yes
- Behavior: For each spectrum bin (if per_bin) or for rms/peak, retains the maximum value seen and decays it over time. Outputs the max of current value and held peak.
- [ ] Task 4: **Gain** filter (`core/audio/filters/gain.py`)
- Options: `factor` (float, 0.1-10.0, default 1.0)
- Stateful: No
- Behavior: Multiplies rms, peak, spectrum, and per-channel values by factor. Clamps to [0, 1] for rms/peak.
- [ ] Task 5: **Noise Gate** filter (`core/audio/filters/noise_gate.py`)
- Options: `threshold` (float, 0.0-1.0), `hysteresis` (float, 0.0-0.2, default 0.05)
- Stateful: No (hysteresis is stateless — it's a secondary threshold, not temporal)
- Behavior: If rms < threshold, zeros out all levels and spectrum. Hysteresis means: if gate was open and rms drops below (threshold - hysteresis), close it; if gate was closed and rms rises above threshold, open it.
- Actually stateful for hysteresis tracking: needs to remember gate open/closed state.
- [ ] Task 6: **Envelope Follower** filter (`core/audio/filters/envelope_follower.py`)
- Options: `attack_ms` (float, 1-500, default 10), `release_ms` (float, 10-2000, default 200)
- Stateful: Yes
- Behavior: Smooths rms and peak with asymmetric time constants. When signal rises, uses attack rate. When signal falls, uses release rate. Applied per-bin to spectrum optionally.
- Fast attack + slow release = punchy transients that fade smoothly.
- [ ] Task 7: **Spectral Smoothing** filter (`core/audio/filters/spectral_smoothing.py`)
- Options: `factor` (float, 0.0-0.99, default 0.5)
- Stateful: Yes (maintains previous spectrum state)
- Behavior: Applies exponential moving average per-bin: `smoothed[i] = factor * prev[i] + (1-factor) * current[i]`. Higher factor = smoother/slower.
- [ ] Task 8: **Compressor** filter (`core/audio/filters/compressor.py`)
- Options: `threshold` (float, 0.0-1.0, default 0.5), `ratio` (float, 1.0-20.0, default 4.0), `makeup_gain` (float, 0.0-2.0, default 1.0)
- Stateful: Yes (envelope tracking for gain reduction)
- Behavior: When signal exceeds threshold, reduces by ratio. `output = threshold + (input - threshold) / ratio`. Apply makeup_gain after. Applied to rms, peak, and spectrum.
- [ ] Task 9: **Inverter** filter (`core/audio/filters/inverter.py`)
- Options: none (or `invert_spectrum` bool, default true)
- Stateful: No
- Behavior: `rms = 1.0 - rms`, `peak = 1.0 - peak`, spectrum bins inverted if option set. Beat fields unchanged.
- [ ] Task 10: **Beat Gate** filter (`core/audio/filters/beat_gate.py`)
- Options: `hold_ms` (float, 10-500, default 50) — how long to hold signal after beat
- Stateful: Yes (tracks last beat timestamp)
- Behavior: When beat detected, passes signal through for `hold_ms` milliseconds. Between beats, zeros out rms/peak/spectrum. Beat fields themselves always pass through.
- [ ] Task 11: **Delay** filter (`core/audio/filters/delay.py`)
- Options: `delay_ms` (float, 10-2000, default 100)
- Stateful: Yes (ring buffer of AudioAnalysis snapshots)
- Behavior: Buffers incoming AudioAnalysis snapshots and outputs the one from `delay_ms` ago. Ring buffer sized based on ~30Hz update rate.
- [ ] Task 12: Register all 11 filters in `core/audio/filters/__init__.py`
- [ ] Task 13: Update Noise Gate to be stateful (hysteresis requires gate state tracking)
## Files to Modify/Create
- `core/audio/filters/channel_extract.py`**create**
- `core/audio/filters/band_extract.py`**create**
- `core/audio/filters/peak_hold.py`**create**
- `core/audio/filters/gain.py`**create**
- `core/audio/filters/noise_gate.py`**create**
- `core/audio/filters/envelope_follower.py`**create**
- `core/audio/filters/spectral_smoothing.py`**create**
- `core/audio/filters/compressor.py`**create**
- `core/audio/filters/inverter.py`**create**
- `core/audio/filters/beat_gate.py`**create**
- `core/audio/filters/delay.py`**create**
- `core/audio/filters/__init__.py`**modify** — register all filters
## Acceptance Criteria
- All 11 filters are implemented and registered
- Each filter correctly transforms AudioAnalysis according to its specification
- Stateful filters (peak hold, envelope follower, spectral smoothing, compressor, beat gate, delay, noise gate) properly maintain and reset state
- Filter option schemas are complete and accurate
- All filters are accessible via `GET /api/v1/audio-filters`
## Notes
- 6 stateful filters: peak hold, envelope follower, spectral smoothing, compressor, beat gate, delay. Noise gate is also stateful due to hysteresis.
- Band extract can reuse math from existing `core/audio/band_filter.py``compute_band_mask()` and `apply_band_filter()`
- Filters must produce a NEW AudioAnalysis (immutability principle), not mutate the input
- For delay filter, ring buffer size = `delay_ms / (1000 / update_rate)`. At 30Hz, 2000ms delay = 60 slots.
## Review Checklist
- [ ] All tasks completed
- [ ] Code follows project conventions
- [ ] No unintended side effects
- [ ] Build passes
- [ ] Tests pass (new + existing)
## Handoff to Next Phase
<!-- Filled in by the implementation agent after completing this phase. -->
@@ -0,0 +1,78 @@
# Phase 3: Processed Audio Source Model
**Status:** ⬜ Not Started
**Parent plan:** [PLAN.md](./PLAN.md)
**Domain:** backend
## Objective
Add the `ProcessedAudioSource` type, rename `MultichannelAudioSource` to `CaptureAudioSource`, remove `MonoAudioSource` and `BandExtractAudioSource`, and update the store's resolution logic.
## Tasks
- [ ] Task 1: Rename `MultichannelAudioSource``CaptureAudioSource` in `storage/audio_source.py`
- Change class name, update `source_type` default to `"capture"`
- Same fields: `device_index`, `is_loopback`, `audio_template_id`
- [ ] Task 2: Add `ProcessedAudioSource` dataclass in `storage/audio_source.py`
- Fields: `audio_source_id: str` (input source), `audio_processing_template_id: str`
- `source_type` = `"processed"`
- Inherits standard base fields (id, name, description, tags, created_at, updated_at)
- [ ] Task 3: Remove `MonoAudioSource` class entirely
- [ ] Task 4: Remove `BandExtractAudioSource` class entirely
- [ ] Task 5: Update `create_audio_source()` factory function to handle new types
- [ ] Task 6: Update `AudioSourceStore` resolution logic:
- `resolve_audio_source()` now returns: device info (from CaptureAudioSource at chain end) + ordered list of filter chains (from AudioProcessingTemplates along the chain)
- Walk chain: ProcessedAudioSource → ... → CaptureAudioSource
- Collect all audio_processing_template_ids in order
- Cycle detection for ProcessedAudioSource chains
- [ ] Task 7: Update `ResolvedAudioSource` dataclass:
- Remove `channel` and `freq_low`/`freq_high` fields (handled by filters now)
- Add `filter_instances: List[FilterInstance]` — flattened, ordered list of all filters to apply
- Or add `template_ids: List[str]` and resolve at runtime
- [ ] Task 8: Update reference validation in store:
- `ProcessedAudioSource.audio_source_id` must reference an existing audio source
- `ProcessedAudioSource.audio_processing_template_id` must reference an existing template
- Delete checks: can't delete a source referenced by another ProcessedAudioSource
- Delete checks: can't delete a template referenced by a ProcessedAudioSource
- [ ] Task 9: Update API schemas in `api/schemas/audio_sources.py`
- Remove `MonoAudioSourceCreate/Update/Response` schemas
- Remove `BandExtractAudioSourceCreate/Update/Response` schemas
- Add `CaptureAudioSourceCreate/Update/Response` (rename from Multichannel)
- Add `ProcessedAudioSourceCreate/Update/Response`
- Update discriminated union to use new type literals
- [ ] Task 10: Update API routes in `api/routes/audio_sources.py`
- Handle new source types in create/update endpoints
- Remove handling of old types
- Update WebSocket test endpoint to work with ProcessedAudioSource
- [ ] Task 11: Update any imports/references across the codebase that reference the old types
## Files to Modify/Create
- `storage/audio_source.py`**modify** — rename, add, remove dataclasses
- `storage/audio_source_store.py`**modify** — new resolution logic, validation
- `api/schemas/audio_sources.py`**modify** — new schemas
- `api/routes/audio_sources.py`**modify** — handle new types
- Any files importing `MultichannelAudioSource`, `MonoAudioSource`, `BandExtractAudioSource`**modify**
## Acceptance Criteria
- `CaptureAudioSource` replaces `MultichannelAudioSource` (same behavior, new name/type)
- `ProcessedAudioSource` can be created referencing a source + template
- `MonoAudioSource` and `BandExtractAudioSource` are fully removed
- Chain resolution walks ProcessedAudioSource → ... → CaptureAudioSource correctly
- Cycle detection prevents circular source references
- Reference validation prevents dangling references
- API accepts/returns new type discriminators
## Notes
- Clean-slate: no migration of existing data. Old source type records will be lost.
- The `source_type` string changes from `"multichannel"` to `"capture"` — this is a breaking change but acceptable for clean-slate.
- `ResolvedAudioSource` is consumed by `AudioColorStripStream` and `AudioValueStream` — they will need updates in Phase 4.
- Template reference checks in the store need coordination with `AudioProcessingTemplateStore` — may need to pass it as a dependency.
## Review Checklist
- [ ] All tasks completed
- [ ] Code follows project conventions
- [ ] No unintended side effects
- [ ] Build passes
- [ ] Tests pass (new + existing)
## Handoff to Next Phase
<!-- Filled in by the implementation agent after completing this phase. -->
@@ -0,0 +1,71 @@
# Phase 4: Runtime Integration
**Status:** ⬜ Not Started
**Parent plan:** [PLAN.md](./PLAN.md)
**Domain:** backend
## Objective
Wire the audio filter pipeline into the runtime audio streaming system so that ProcessedAudioSources actually apply their filter chains to live audio data.
## Tasks
- [ ] Task 1: Create filter pipeline executor in `core/audio/filters/pipeline.py`
- `AudioFilterPipeline` class:
- `__init__(filter_instances: List[FilterInstance], registry: AudioFilterRegistry)`
- Instantiates all filters from FilterInstance specs
- `process(analysis: AudioAnalysis) -> AudioAnalysis` — runs analysis through all filters in order
- `reset()` — resets all stateful filters
- `close()` — cleanup resources
- Handles stateful filter lifecycle (create on init, reset on demand, close on cleanup)
- [ ] Task 2: Update `AudioColorStripStream` in `core/processing/audio_stream.py`
- On construction: if source is ProcessedAudioSource, resolve the full chain:
- Walk to CaptureAudioSource for device info
- Collect all AudioProcessingTemplates along the chain
- Resolve all filter instances (with template expansion)
- Create AudioFilterPipeline
- In render loop: after getting AudioAnalysis from ManagedAudioStream, run it through the filter pipeline before visualization
- Remove old inline channel selection and band filtering code (now handled by filters)
- On stop: close the filter pipeline
- [ ] Task 3: Update `AudioValueStream` in `core/processing/value_stream.py`
- Same pattern: resolve ProcessedAudioSource chain, create filter pipeline, apply in get_value()
- Remove old inline channel/band handling
- [ ] Task 4: Hot-update support for filter templates
- When an AudioProcessingTemplate is updated, running streams that use it should re-resolve their filter pipeline
- Listen for template update events (or implement a refresh mechanism)
- Re-create AudioFilterPipeline with updated filter instances
- Reset stateful filter state on pipeline refresh
- [ ] Task 5: Update WebSocket test endpoint in `api/routes/audio_sources.py`
- For ProcessedAudioSource: resolve chain, create pipeline, apply filters to test stream data
- Return filtered analysis in real-time over WebSocket
- [ ] Task 6: Update any code that calls `AudioSourceStore.resolve_audio_source()` to handle the new return shape
## Files to Modify/Create
- `core/audio/filters/pipeline.py`**create** — AudioFilterPipeline
- `core/processing/audio_stream.py`**modify** — integrate filter pipeline
- `core/processing/value_stream.py`**modify** — integrate filter pipeline
- `api/routes/audio_sources.py`**modify** — update WebSocket test
- Any other consumers of `resolve_audio_source()`**modify**
## Acceptance Criteria
- ProcessedAudioSource chains are resolved and filter pipelines created at stream start
- AudioAnalysis passes through the filter chain before visualization/value extraction
- Stateful filters maintain correct state across frames
- Hot-update of templates refreshes running filter pipelines
- WebSocket test endpoint works with processed sources
- Old inline channel/band code removed from stream classes
## Notes
- ⚠️ Temporary breakage: Removing inline channel/band code from AudioColorStripStream breaks existing MonoAudioSource/BandExtractAudioSource flows — but those types were already removed in Phase 3.
- Filter pipeline must be thread-safe: AudioColorStripStream and AudioValueStream run in background threads.
- For hot-update: consider using an event callback from the template store rather than polling.
- The filter pipeline should produce a new AudioAnalysis each time (immutability), not mutate the shared snapshot from ManagedAudioStream.
## Review Checklist
- [ ] All tasks completed
- [ ] Code follows project conventions
- [ ] No unintended side effects
- [ ] Build passes
- [ ] Tests pass (new + existing)
## Handoff to Next Phase
<!-- Filled in by the implementation agent after completing this phase. -->
@@ -0,0 +1,70 @@
# Phase 5: Frontend — Audio Processing Templates
**Status:** ⬜ Not Started
**Parent plan:** [PLAN.md](./PLAN.md)
**Domain:** frontend
## Objective
Build the frontend UI for managing Audio Processing Templates — list, create, edit, delete, with a filter editor and real-time preview.
## Tasks
- [ ] Task 1: Create TypeScript module `static/js/features/audio-processing-templates.ts`
- Fetch/cache audio processing templates via DataCache
- CRUD operations using fetchWithAuth
- CardSection for template list with reconciliation
- [ ] Task 2: Create template editor modal
- Name, description, tags fields
- Ordered filter list with add/remove/reorder controls
- Per-filter option controls (sliders, selects, toggles) driven by option schemas from `GET /api/v1/audio-filters`
- Template composition support: `audio_filter_template` shows EntitySelect for sub-template
- Dirty check via snapshotValues()
- [ ] Task 3: Add Audio Processing Templates section to the Streams tab
- New sub-tab or section alongside existing Audio Sources
- CardSection rendering with template name, filter count, description
- Create/Edit/Delete actions per card
- [ ] Task 4: Real-time audio preview
- "Test" button on template editor that opens a WebSocket connection
- Shows spectrum visualization (reuse existing audio test pattern)
- Applies template's filters to a selected source in real-time
- Source picker (EntitySelect) for choosing input audio source
- [ ] Task 5: Add i18n keys for all new UI strings (en.json, ru.json, zh.json)
- Template section labels, filter names, option labels, buttons, errors
- [ ] Task 6: Register module in `app.js` / global exports if needed for inline onclick handlers
- [ ] Task 7: Fetch and cache audio filter registry data (for building filter option UIs)
## Files to Modify/Create
- `static/js/features/audio-processing-templates.ts`**create** — main module
- `static/js/features/audio-processing-template-modal.ts`**create** — editor modal
- `static/css/dashboard.css`**modify** — styles for template editor
- `static/js/app.js`**modify** — register module, add window exports
- `templates/dashboard.html` (or relevant Jinja template) — **modify** — add section
- `static/js/core/i18n/en.json`**modify** — new keys
- `static/js/core/i18n/ru.json`**modify** — new keys
- `static/js/core/i18n/zh.json`**modify** — new keys
## Acceptance Criteria
- Audio Processing Templates section visible in Streams tab
- Templates can be created, edited, deleted
- Filter editor shows all 11 available filters with correct option controls
- Template composition (audio_filter_template) works via EntitySelect
- Real-time preview shows filtered audio data
- All strings are internationalized
## Notes
- Follow existing patterns from postprocessing template UI
- Use IconSelect for filter type selection (if icon set supports it) or a custom filter picker
- NEVER use plain HTML `<select>` — use project custom selectors (CRITICAL project rule)
- NEVER use emoji — use SVG icons from `core/icons.ts`
- Use fetchWithAuth for ALL API calls (project rule)
- Call cache.invalidate() before load functions in save/delete handlers
## Review Checklist
- [ ] All tasks completed
- [ ] Code follows project conventions
- [ ] No unintended side effects
- [ ] Build passes
- [ ] Tests pass (new + existing)
## Handoff to Next Phase
<!-- Filled in by the implementation agent after completing this phase. -->
@@ -0,0 +1,67 @@
# Phase 6: Frontend — Source Types
**Status:** ⬜ Not Started
**Parent plan:** [PLAN.md](./PLAN.md)
**Domain:** frontend
## Objective
Update the audio source UI to support the new `ProcessedAudioSource` and `CaptureAudioSource` types, and remove the old `MonoAudioSource` and `BandExtractAudioSource` UI.
## Tasks
- [ ] Task 1: Update audio source TypeScript types/interfaces for new source types
- Add `ProcessedAudioSource` type with `audio_source_id` + `audio_processing_template_id`
- Rename `MultichannelAudioSource` type to `CaptureAudioSource` (source_type: "capture")
- Remove `MonoAudioSource` and `BandExtractAudioSource` types
- [ ] Task 2: Create `ProcessedAudioSource` card component
- EntitySelect for input audio source (any audio source)
- EntitySelect for audio processing template
- Show resolved chain info (which capture source at the end)
- Create/Edit/Delete actions
- [ ] Task 3: Update `CaptureAudioSource` card (relabeled from Multichannel)
- Same fields (device selector, loopback toggle, template selector)
- Updated label/icon to say "Capture Audio Source"
- [ ] Task 4: Remove `MonoAudioSource` card component/rendering
- [ ] Task 5: Remove `BandExtractAudioSource` card component/rendering
- [ ] Task 6: Update audio source creation dialog/flow
- Source type picker now shows: Capture, Processed (instead of Multichannel, Mono, Band Extract)
- Type-specific form fields
- [ ] Task 7: Update EntitySelect dropdowns that list audio sources
- Show type badges (Capture vs Processed) for clarity
- Audio source selectors in CSS editor, value source editor, etc.
- [ ] Task 8: Update i18n keys for renamed/new source types
- [ ] Task 9: Update any inline onclick handlers or window exports in app.js
## Files to Modify/Create
- `static/js/features/audio-sources.ts`**modify** — new types, remove old types
- `static/js/features/audio-source-modal.ts` (or equivalent) — **modify** — updated editor
- `static/css/dashboard.css`**modify** — any style updates
- `static/js/app.js`**modify** — update exports if needed
- `static/js/core/i18n/en.json`**modify** — updated keys
- `static/js/core/i18n/ru.json`**modify** — updated keys
- `static/js/core/i18n/zh.json`**modify** — updated keys
## Acceptance Criteria
- ProcessedAudioSource can be created/edited/deleted from the UI
- CaptureAudioSource shows correctly with updated label
- MonoAudioSource and BandExtractAudioSource UI is completely removed
- EntitySelect for audio sources shows type badges
- Source type picker shows only Capture and Processed
- All strings are internationalized
## Notes
- NEVER use plain HTML `<select>` — use project custom selectors
- NEVER use emoji — use SVG icons
- Use fetchWithAuth for ALL API calls
- Call cache.invalidate() before load in save/delete handlers
- Check DOM ID conflicts when adding new card types (project checklist)
## Review Checklist
- [ ] All tasks completed
- [ ] Code follows project conventions
- [ ] No unintended side effects
- [ ] Build passes
- [ ] Tests pass (new + existing)
## Handoff to Next Phase
<!-- Filled in by the implementation agent after completing this phase. -->
@@ -0,0 +1,85 @@
# Phase 7: Testing & Polish
**Status:** ⬜ Not Started
**Parent plan:** [PLAN.md](./PLAN.md)
**Domain:** backend
## Objective
Verify the full feature works end-to-end, write tests, fix any remaining issues, and clean up dead code.
## Tasks
- [ ] Task 1: Write unit tests for each audio filter
- Test each filter transforms AudioAnalysis correctly
- Test stateful filters maintain and reset state
- Test edge cases: empty spectrum, zero rms, all-zero input
- Test filter option validation
- [ ] Task 2: Write unit tests for AudioFilterPipeline
- Test chain of filters produces expected output
- Test empty pipeline passes through unchanged
- Test reset() resets all stateful filters
- [ ] Task 3: Write integration tests for AudioProcessingTemplateStore
- CRUD operations
- Template composition (audio_filter_template) expansion
- Cycle detection in template composition
- Reference validation
- [ ] Task 4: Write integration tests for ProcessedAudioSource in AudioSourceStore
- CRUD operations
- Chain resolution (ProcessedAudioSource → ... → CaptureAudioSource)
- Cycle detection in source chains
- Reference validation (source + template must exist)
- Cascade delete checks
- [ ] Task 5: Write API tests for audio processing template endpoints
- Create, read, update, delete
- Validation errors (missing fields, invalid filter_ids)
- Reference check on delete
- [ ] Task 6: Write API tests for updated audio source endpoints
- Create/update ProcessedAudioSource and CaptureAudioSource
- Reject old types (mono, band_extract)
- Validation of source/template references
- [ ] Task 7: Verify backup/restore includes audio processing templates
- Create templates, backup, restore, verify they survive
- [ ] Task 8: Full build verification
- `ruff check src/ tests/ --fix` passes
- `npx tsc --noEmit` passes
- `npm run build` succeeds
- `py -3.13 -m pytest tests/ --no-cov -q` — all tests pass
- [ ] Task 9: Clean up dead code
- Remove any remaining imports of `MonoAudioSource`, `BandExtractAudioSource`
- Remove old `band_filter.py` if fully superseded by the band_extract filter (or keep if still used elsewhere)
- Remove unused schema classes
- Verify no orphaned i18n keys
- [ ] Task 10: Update system health/info endpoints if they enumerate audio source types
## Files to Modify/Create
- `tests/test_audio_filters.py`**create** — filter unit tests
- `tests/test_audio_filter_pipeline.py`**create** — pipeline tests
- `tests/test_audio_processing_template_store.py`**create** — store tests
- `tests/test_audio_source_store.py`**modify** — updated for new types
- `tests/test_audio_processing_templates_api.py`**create** — API tests
- `tests/test_audio_sources_api.py`**modify** — updated for new types
- Various files — **modify** — dead code cleanup
## Acceptance Criteria
- All new tests pass
- All existing tests pass (no regressions)
- Full build (Python + TypeScript) succeeds
- Ruff linting passes
- No dead code referencing removed types
- Backup/restore round-trips correctly with audio processing templates
## Notes
- This is the Big Bang verification phase — the first time build + tests are enforced
- Expect some breakage from Phases 1-6 that needs fixing here
- Focus on fixing real issues, not cosmetic cleanup
- `band_filter.py` may still be used by the band_extract filter — check before removing
## Review Checklist
- [ ] All tasks completed
- [ ] Code follows project conventions
- [ ] No unintended side effects
- [ ] Build passes
- [ ] Tests pass (new + existing)
## Handoff to Next Phase
<!-- N/A — final phase -->
@@ -0,0 +1,65 @@
# Phase 8: Frontend Design Consistency Review
**Status:** ⬜ Not Started
**Parent plan:** [PLAN.md](./PLAN.md)
**Domain:** frontend
## Objective
Review all new frontend UI created in Phases 5-6 for visual consistency, design quality, and UX polish using the frontend-design skill agent. Fix any issues found.
## Tasks
- [ ] Task 1: Review Audio Processing Templates section for design consistency
- Card layout, spacing, typography alignment with existing sections
- Filter editor modal — controls alignment, visual hierarchy, grouping
- Responsive behavior at different viewport widths
- [ ] Task 2: Review Processed Audio Source cards for design consistency
- Card style matches existing source cards (capture, picture, value sources)
- EntitySelect pickers are visually consistent
- Type badges/icons are clear and distinguishable
- [ ] Task 3: Review Capture Audio Source card (relabeled)
- Label/icon updates look correct
- No visual regressions from the rename
- [ ] Task 4: Review source type picker/creation flow
- Type selector is clear and accessible
- Transition between types is smooth
- Empty states handled properly
- [ ] Task 5: Review real-time audio preview UI
- Spectrum visualization looks polished
- Source picker and controls are well-placed
- Loading/error states
- [ ] Task 6: Fix all design issues found in Tasks 1-5
- CSS adjustments for spacing, alignment, typography
- Icon/color consistency
- Dark mode compatibility (if applicable)
- Hover/focus/active states on interactive elements
- [ ] Task 7: Cross-browser spot-check (if applicable)
## Files to Modify/Create
- `static/css/dashboard.css`**modify** — design fixes
- `static/js/features/audio-processing-templates.ts`**modify** — UX fixes
- `static/js/features/audio-processing-template-modal.ts`**modify** — UX fixes
- `static/js/features/audio-sources.ts`**modify** — UX fixes
- Any template/HTML files — **modify** — structural fixes
## Acceptance Criteria
- All new UI sections are visually consistent with existing sections
- No orphaned styles or visual regressions
- Filter editor is intuitive and well-organized
- Cards, modals, and controls follow existing design language
- Interactive elements have proper hover/focus/active states
## Notes
- This phase uses the frontend-design skill agent for review
- Focus on consistency with existing UI, not a complete redesign
- The project uses vanilla CSS (no framework) — fixes must use the existing stylesheet approach
## Review Checklist
- [ ] All tasks completed
- [ ] Code follows project conventions
- [ ] No unintended side effects
- [ ] Build passes
- [ ] Tests pass (new + existing)
## Handoff to Next Phase
<!-- N/A — final phase -->