alexei.dolgolyov
ab43578049
Add AudioFilterPipeline for chained filter execution on AudioAnalysis. Wire filter pipelines into AudioColorStripStream, AudioValueStream, and WebSocket test endpoint. Add hot-update support via ProcessorManager.refresh_audio_filter_pipelines(). Thread AudioProcessingTemplateStore through dependency injection hierarchy.
112 lines
5.0 KiB
Markdown
112 lines
5.0 KiB
Markdown
# Feature Context: Processed Audio Sources
|
|
|
|
## Configuration
|
|
- **Development mode:** Automated
|
|
- **Execution mode:** Orchestrator
|
|
- **Strategy:** Big Bang
|
|
- **Build (Python):** `cd server && ruff check src/ tests/ --fix`
|
|
- **Build (TypeScript):** `cd server && npx tsc --noEmit && npm run build`
|
|
- **Test:** `cd server && py -3.13 -m pytest tests/ --no-cov -q`
|
|
|
|
## Current State
|
|
Phases 1-4 implemented. Phase 4 (Runtime Integration) wired the audio filter pipeline into the stream runtime.
|
|
|
|
Phase 1 framework:
|
|
- `AudioFilter` base class, `AudioFilterRegistry`, `AudioFilterOptionDef` in `core/audio/filters/`
|
|
- `AudioProcessingTemplate` dataclass + `AudioProcessingTemplateStore` (SQLite-backed) in `storage/`
|
|
- `audio_filter_template` meta-filter with recursive resolution
|
|
- Full REST API: CRUD templates + filter registry discovery
|
|
- Dependency injection wired in `dependencies.py` and `main.py`
|
|
|
|
Phase 2 filters (12 total registered, 11 real + 1 meta):
|
|
- Stateless: `channel_extract`, `band_extract`, `gain`, `inverter`
|
|
- Stateful: `peak_hold`, `noise_gate`, `envelope_follower`, `spectral_smoothing`, `compressor`, `beat_gate`, `delay`
|
|
- All produce new `AudioAnalysis` via `dataclasses.replace()` (immutability preserved)
|
|
|
|
## Key Architecture Reference
|
|
|
|
### Existing Pattern to Mirror: Processed Picture Sources
|
|
- `ProcessedPictureSource` references `source_stream_id` + `postprocessing_template_id`
|
|
- `PostprocessingTemplate` contains ordered `List[FilterInstance]`
|
|
- `FilterInstance` = `filter_id` (string) + `options` (dict)
|
|
- `FilterRegistry` handles registration, lookup, instantiation
|
|
- `filter_template` meta-filter embeds one template inside another
|
|
- `PostprocessingTemplateStore` has `resolve_filter_instances()` for recursive expansion
|
|
- Picture filters transform images; audio filters will transform `AudioAnalysis`
|
|
|
|
### Current Audio Source Types (Phase 3 complete)
|
|
- `CaptureAudioSource` (source_type="capture") — wraps a physical audio device
|
|
- `ProcessedAudioSource` (source_type="processed") — references audio_source_id + audio_processing_template_id
|
|
- `MonoAudioSource` — removed, replaced by channel_extract filter
|
|
- `BandExtractAudioSource` — removed, replaced by band_extract filter
|
|
|
|
### AudioAnalysis Structure (filter input/output)
|
|
```python
|
|
AudioAnalysis:
|
|
timestamp: float
|
|
rms: float # Overall RMS level
|
|
peak: float # Peak amplitude
|
|
spectrum: np.ndarray[64] # Log-spaced FFT bands
|
|
beat: bool # Beat detected
|
|
beat_intensity: float # 0-1 beat strength
|
|
left_rms: float # Left channel RMS
|
|
left_spectrum: np.ndarray # Left channel spectrum
|
|
right_rms: float # Right channel RMS
|
|
right_spectrum: np.ndarray # Right channel spectrum
|
|
```
|
|
|
|
### Key Existing Files
|
|
- `storage/audio_source.py` — current source dataclasses
|
|
- `storage/audio_source_store.py` — CRUD + resolve_audio_source()
|
|
- `core/audio/analysis.py` — AudioAnalyzer, AudioAnalysis
|
|
- `core/audio/band_filter.py` — existing band filtering logic
|
|
- `core/processing/audio_stream.py` — AudioColorStripStream
|
|
- `core/processing/value_stream.py` — AudioValueStream
|
|
- `core/filters/base.py` — PostprocessingFilter (picture filter base class)
|
|
- `core/filters/registry.py` — FilterRegistry (picture filters)
|
|
- `storage/postprocessing_template.py` — PostprocessingTemplate dataclass
|
|
- `storage/postprocessing_template_store.py` — template store with resolve_filter_instances()
|
|
|
|
## Temporary Workarounds
|
|
_(none yet)_
|
|
|
|
## Cross-Phase Dependencies
|
|
- Phase 2 depends on Phase 1 (filter framework)
|
|
- Phase 3 depends on Phase 1 (template store for ProcessedAudioSource)
|
|
- Phase 4 depends on Phases 1-3 (all backend pieces)
|
|
- Phase 5 depends on Phase 1 (template API)
|
|
- Phase 6 depends on Phase 3 (source type API)
|
|
- Phase 7 depends on all prior phases
|
|
|
|
## Deferred Work
|
|
_(none yet)_
|
|
|
|
## Failed Approaches
|
|
_(none yet)_
|
|
|
|
## Review Findings Log
|
|
_(none yet)_
|
|
|
|
## Phase Execution Log
|
|
| Phase | Agent Used | Test Writer | Parallel | Notes |
|
|
|-------|-----------|-------------|----------|-------|
|
|
| Phase 1 | impl-agent | — | No | Tasks 7+8 skipped (SQLite migration made them obsolete) |
|
|
| Phase 2 | impl-agent | — | No | All 11 filters implemented, no deviations |
|
|
| Phase 3 | impl-agent | — | No | All 11 tasks done; channel/band logic deferred to Phase 4 |
|
|
| Phase 4 | impl-agent | — | No | All 6 tasks done; dependency injection threaded through |
|
|
| Phase 5 | — | — | — | — |
|
|
| Phase 6 | — | — | — | — |
|
|
| Phase 7 | — | — | — | — |
|
|
|
|
## Environment & Runtime Notes
|
|
- Platform: Windows 10
|
|
- Python: 3.13
|
|
- Server port: 8080
|
|
- Shell: bash (Git Bash on Windows)
|
|
|
|
## Implementation Notes
|
|
- Clean-slate approach: no migration of existing MonoAudioSource/BandExtractAudioSource data
|
|
- 7 of 11 filters are stateful (peak hold, noise gate, envelope follower, spectral smoothing, compressor, beat gate, delay) — need per-stream instance lifecycle
|
|
- Audio filters operate on AudioAnalysis snapshots, not raw audio samples
|
|
- Big Bang strategy: intermediate phases may break the build; only Phase 7 enforces build/tests
|