alexei.dolgolyov
353c090b42
Replace MultichannelAudioSource with CaptureAudioSource, add ProcessedAudioSource (audio_source_id + audio_processing_template_id), remove MonoAudioSource and BandExtractAudioSource entirely. Update store resolution to walk processed chains collecting template IDs. Update all API schemas, routes, and frontend references.
112 lines
5.0 KiB
Markdown
112 lines
5.0 KiB
Markdown
# Feature Context: Processed Audio Sources
|
|
|
|
## Configuration
|
|
- **Development mode:** Automated
|
|
- **Execution mode:** Orchestrator
|
|
- **Strategy:** Big Bang
|
|
- **Build (Python):** `cd server && ruff check src/ tests/ --fix`
|
|
- **Build (TypeScript):** `cd server && npx tsc --noEmit && npm run build`
|
|
- **Test:** `cd server && py -3.13 -m pytest tests/ --no-cov -q`
|
|
|
|
## Current State
|
|
Phase 1 (Audio Filter Framework), Phase 2 (Audio Filters), and Phase 3 (Processed Audio Source Model) implemented.
|
|
|
|
Phase 1 framework:
|
|
- `AudioFilter` base class, `AudioFilterRegistry`, `AudioFilterOptionDef` in `core/audio/filters/`
|
|
- `AudioProcessingTemplate` dataclass + `AudioProcessingTemplateStore` (SQLite-backed) in `storage/`
|
|
- `audio_filter_template` meta-filter with recursive resolution
|
|
- Full REST API: CRUD templates + filter registry discovery
|
|
- Dependency injection wired in `dependencies.py` and `main.py`
|
|
|
|
Phase 2 filters (12 total registered, 11 real + 1 meta):
|
|
- Stateless: `channel_extract`, `band_extract`, `gain`, `inverter`
|
|
- Stateful: `peak_hold`, `noise_gate`, `envelope_follower`, `spectral_smoothing`, `compressor`, `beat_gate`, `delay`
|
|
- All produce new `AudioAnalysis` via `dataclasses.replace()` (immutability preserved)
|
|
|
|
## Key Architecture Reference
|
|
|
|
### Existing Pattern to Mirror: Processed Picture Sources
|
|
- `ProcessedPictureSource` references `source_stream_id` + `postprocessing_template_id`
|
|
- `PostprocessingTemplate` contains ordered `List[FilterInstance]`
|
|
- `FilterInstance` = `filter_id` (string) + `options` (dict)
|
|
- `FilterRegistry` handles registration, lookup, instantiation
|
|
- `filter_template` meta-filter embeds one template inside another
|
|
- `PostprocessingTemplateStore` has `resolve_filter_instances()` for recursive expansion
|
|
- Picture filters transform images; audio filters will transform `AudioAnalysis`
|
|
|
|
### Current Audio Source Types (Phase 3 complete)
|
|
- `CaptureAudioSource` (source_type="capture") — wraps a physical audio device
|
|
- `ProcessedAudioSource` (source_type="processed") — references audio_source_id + audio_processing_template_id
|
|
- `MonoAudioSource` — removed, replaced by channel_extract filter
|
|
- `BandExtractAudioSource` — removed, replaced by band_extract filter
|
|
|
|
### AudioAnalysis Structure (filter input/output)
|
|
```python
|
|
AudioAnalysis:
|
|
timestamp: float
|
|
rms: float # Overall RMS level
|
|
peak: float # Peak amplitude
|
|
spectrum: np.ndarray[64] # Log-spaced FFT bands
|
|
beat: bool # Beat detected
|
|
beat_intensity: float # 0-1 beat strength
|
|
left_rms: float # Left channel RMS
|
|
left_spectrum: np.ndarray # Left channel spectrum
|
|
right_rms: float # Right channel RMS
|
|
right_spectrum: np.ndarray # Right channel spectrum
|
|
```
|
|
|
|
### Key Existing Files
|
|
- `storage/audio_source.py` — current source dataclasses
|
|
- `storage/audio_source_store.py` — CRUD + resolve_audio_source()
|
|
- `core/audio/analysis.py` — AudioAnalyzer, AudioAnalysis
|
|
- `core/audio/band_filter.py` — existing band filtering logic
|
|
- `core/processing/audio_stream.py` — AudioColorStripStream
|
|
- `core/processing/value_stream.py` — AudioValueStream
|
|
- `core/filters/base.py` — PostprocessingFilter (picture filter base class)
|
|
- `core/filters/registry.py` — FilterRegistry (picture filters)
|
|
- `storage/postprocessing_template.py` — PostprocessingTemplate dataclass
|
|
- `storage/postprocessing_template_store.py` — template store with resolve_filter_instances()
|
|
|
|
## Temporary Workarounds
|
|
_(none yet)_
|
|
|
|
## Cross-Phase Dependencies
|
|
- Phase 2 depends on Phase 1 (filter framework)
|
|
- Phase 3 depends on Phase 1 (template store for ProcessedAudioSource)
|
|
- Phase 4 depends on Phases 1-3 (all backend pieces)
|
|
- Phase 5 depends on Phase 1 (template API)
|
|
- Phase 6 depends on Phase 3 (source type API)
|
|
- Phase 7 depends on all prior phases
|
|
|
|
## Deferred Work
|
|
_(none yet)_
|
|
|
|
## Failed Approaches
|
|
_(none yet)_
|
|
|
|
## Review Findings Log
|
|
_(none yet)_
|
|
|
|
## Phase Execution Log
|
|
| Phase | Agent Used | Test Writer | Parallel | Notes |
|
|
|-------|-----------|-------------|----------|-------|
|
|
| Phase 1 | impl-agent | — | No | Tasks 7+8 skipped (SQLite migration made them obsolete) |
|
|
| Phase 2 | impl-agent | — | No | All 11 filters implemented, no deviations |
|
|
| Phase 3 | impl-agent | — | No | All 11 tasks done; channel/band logic deferred to Phase 4 |
|
|
| Phase 4 | — | — | — | — |
|
|
| Phase 5 | — | — | — | — |
|
|
| Phase 6 | — | — | — | — |
|
|
| Phase 7 | — | — | — | — |
|
|
|
|
## Environment & Runtime Notes
|
|
- Platform: Windows 10
|
|
- Python: 3.13
|
|
- Server port: 8080
|
|
- Shell: bash (Git Bash on Windows)
|
|
|
|
## Implementation Notes
|
|
- Clean-slate approach: no migration of existing MonoAudioSource/BandExtractAudioSource data
|
|
- 7 of 11 filters are stateful (peak hold, noise gate, envelope follower, spectral smoothing, compressor, beat gate, delay) — need per-stream instance lifecycle
|
|
- Audio filters operate on AudioAnalysis snapshots, not raw audio samples
|
|
- Big Bang strategy: intermediate phases may break the build; only Phase 7 enforces build/tests
|