alexei.dolgolyov
86a9d344e6
Add the foundation for audio processing filters, mirroring the existing picture filter/postprocessing template system: - AudioFilter base class, AudioFilterRegistry, AudioFilterOptionDef - AudioProcessingTemplate dataclass + SQLite-backed store - audio_filter_template meta-filter with recursive resolution - Full REST API: CRUD templates + filter registry discovery - Dependency injection wired in dependencies.py and main.py
104 lines
4.3 KiB
Markdown
104 lines
4.3 KiB
Markdown
# Feature Context: Processed Audio Sources
|
|
|
|
## Configuration
|
|
- **Development mode:** Automated
|
|
- **Execution mode:** Orchestrator
|
|
- **Strategy:** Big Bang
|
|
- **Build (Python):** `cd server && ruff check src/ tests/ --fix`
|
|
- **Build (TypeScript):** `cd server && npx tsc --noEmit && npm run build`
|
|
- **Test:** `cd server && py -3.13 -m pytest tests/ --no-cov -q`
|
|
|
|
## Current State
|
|
Phase 1 (Audio Filter Framework) implemented. Core framework is in place:
|
|
- `AudioFilter` base class, `AudioFilterRegistry`, `AudioFilterOptionDef` in `core/audio/filters/`
|
|
- `AudioProcessingTemplate` dataclass + `AudioProcessingTemplateStore` (SQLite-backed) in `storage/`
|
|
- `audio_filter_template` meta-filter with recursive resolution
|
|
- Full REST API: CRUD templates + filter registry discovery
|
|
- Dependency injection wired in `dependencies.py` and `main.py`
|
|
|
|
## Key Architecture Reference
|
|
|
|
### Existing Pattern to Mirror: Processed Picture Sources
|
|
- `ProcessedPictureSource` references `source_stream_id` + `postprocessing_template_id`
|
|
- `PostprocessingTemplate` contains ordered `List[FilterInstance]`
|
|
- `FilterInstance` = `filter_id` (string) + `options` (dict)
|
|
- `FilterRegistry` handles registration, lookup, instantiation
|
|
- `filter_template` meta-filter embeds one template inside another
|
|
- `PostprocessingTemplateStore` has `resolve_filter_instances()` for recursive expansion
|
|
- Picture filters transform images; audio filters will transform `AudioAnalysis`
|
|
|
|
### Current Audio Source Types (to be replaced)
|
|
- `MultichannelAudioSource` → renamed to `CaptureAudioSource`
|
|
- `MonoAudioSource` → removed, replaced by channel_extract filter
|
|
- `BandExtractAudioSource` → removed, replaced by band_extract filter
|
|
|
|
### AudioAnalysis Structure (filter input/output)
|
|
```python
|
|
AudioAnalysis:
|
|
timestamp: float
|
|
rms: float # Overall RMS level
|
|
peak: float # Peak amplitude
|
|
spectrum: np.ndarray[64] # Log-spaced FFT bands
|
|
beat: bool # Beat detected
|
|
beat_intensity: float # 0-1 beat strength
|
|
left_rms: float # Left channel RMS
|
|
left_spectrum: np.ndarray # Left channel spectrum
|
|
right_rms: float # Right channel RMS
|
|
right_spectrum: np.ndarray # Right channel spectrum
|
|
```
|
|
|
|
### Key Existing Files
|
|
- `storage/audio_source.py` — current source dataclasses
|
|
- `storage/audio_source_store.py` — CRUD + resolve_audio_source()
|
|
- `core/audio/analysis.py` — AudioAnalyzer, AudioAnalysis
|
|
- `core/audio/band_filter.py` — existing band filtering logic
|
|
- `core/processing/audio_stream.py` — AudioColorStripStream
|
|
- `core/processing/value_stream.py` — AudioValueStream
|
|
- `core/filters/base.py` — PostprocessingFilter (picture filter base class)
|
|
- `core/filters/registry.py` — FilterRegistry (picture filters)
|
|
- `storage/postprocessing_template.py` — PostprocessingTemplate dataclass
|
|
- `storage/postprocessing_template_store.py` — template store with resolve_filter_instances()
|
|
|
|
## Temporary Workarounds
|
|
_(none yet)_
|
|
|
|
## Cross-Phase Dependencies
|
|
- Phase 2 depends on Phase 1 (filter framework)
|
|
- Phase 3 depends on Phase 1 (template store for ProcessedAudioSource)
|
|
- Phase 4 depends on Phases 1-3 (all backend pieces)
|
|
- Phase 5 depends on Phase 1 (template API)
|
|
- Phase 6 depends on Phase 3 (source type API)
|
|
- Phase 7 depends on all prior phases
|
|
|
|
## Deferred Work
|
|
_(none yet)_
|
|
|
|
## Failed Approaches
|
|
_(none yet)_
|
|
|
|
## Review Findings Log
|
|
_(none yet)_
|
|
|
|
## Phase Execution Log
|
|
| Phase | Agent Used | Test Writer | Parallel | Notes |
|
|
|-------|-----------|-------------|----------|-------|
|
|
| Phase 1 | impl-agent | — | No | Tasks 7+8 skipped (SQLite migration made them obsolete) |
|
|
| Phase 2 | — | — | — | — |
|
|
| Phase 3 | — | — | — | — |
|
|
| Phase 4 | — | — | — | — |
|
|
| Phase 5 | — | — | — | — |
|
|
| Phase 6 | — | — | — | — |
|
|
| Phase 7 | — | — | — | — |
|
|
|
|
## Environment & Runtime Notes
|
|
- Platform: Windows 10
|
|
- Python: 3.13
|
|
- Server port: 8080
|
|
- Shell: bash (Git Bash on Windows)
|
|
|
|
## Implementation Notes
|
|
- Clean-slate approach: no migration of existing MonoAudioSource/BandExtractAudioSource data
|
|
- 5 of 11 filters are stateful (peak hold, envelope follower, spectral smoothing, compressor, delay) — need per-stream instance lifecycle
|
|
- Audio filters operate on AudioAnalysis snapshots, not raw audio samples
|
|
- Big Bang strategy: intermediate phases may break the build; only Phase 7 enforces build/tests
|