ledgrab/plans/music-sync/PLAN.md

# music_sync Color Strip Source — Implementation Plan

## Overview

A higher-level music-reactive CSS that provides semantic audio analysis (BPM detection, beat tracking, energy envelope, drop detection, frequency band energy) and multiple visualization modes. Builds on existing `AudioCaptureManager` and `AudioAnalysis` infrastructure. No new external dependencies — uses numpy-only analysis.

## Requirements

- BPM estimation from real-time audio
- Beat onset detection with configurable threshold
- Smoothed RMS energy envelope with attack/release
- Drop detection: energy drops/buildups
- Frequency band energy: bass (20-250 Hz), mid (250-4k Hz), treble (4k-20k Hz)
- Four visualization modes: `pulse_on_beat`, `energy_gradient`, `spectrum_bands`, `strobe_on_drop`
- Uses existing audio engine infrastructure (no new audio capture code)
- No external dependencies beyond numpy

## Phase 1: Music Analysis Engine

**File: `server/src/wled_controller/core/audio/music_analyzer.py`** (new)

### 1.1 `MusicFeatures` dataclass (frozen=True)
- `bpm: float` — estimated BPM (0 if unknown)
- `beat: bool` — beat detected this frame
- `beat_intensity: float` — 0.0-1.0
- `beat_phase: float` — 0.0-1.0 position in beat cycle
- `energy: float` — smoothed RMS 0.0-1.0
- `energy_delta: float` — rate of change
- `bass_energy, mid_energy, treble_energy: float` — 0.0-1.0
- `drop_state: str` — "idle"|"buildup"|"drop"|"recovery"
- `drop_intensity: float` — 0.0-1.0

### 1.2 `MusicAnalyzer` class
- **State**: rolling energy buffer (~4s at ~43 Hz = ~172 samples), beat history timestamps (last 30), smoothed band energies, BPM estimate, drop state machine
- **`update(analysis: AudioAnalysis) -> MusicFeatures`**: main entry point
- **BPM estimation**: Track beat timestamps, compute median inter-beat interval, exponential smoothing, clamp 40-220 BPM
- **Beat tracking**: Pass through `AudioAnalysis.beat` + compute `beat_phase` (position in current beat cycle)
- **Energy envelope**: Smoothed RMS with configurable attack/release
- **Drop detection**: State machine: `idle -> buildup` (energy rising steadily 1-2s), `buildup -> drop` (energy drops >50% within 100ms), `drop -> recovery` (after 500ms), `recovery -> idle`
- **Frequency bands**: Sum spectrum bins into 3 bands from 64-band spectrum

## Phase 2: Storage Model

**File: `server/src/wled_controller/storage/color_strip_source.py`**
- Add `MusicSyncColorStripSource` dataclass after `AudioColorStripSource`
- Fields:
  - `visualization_mode: str` — default `"pulse_on_beat"`
  - `audio_source_id: str` — references AudioSource
  - `sensitivity: BindableFloat` — default 1.0
  - `smoothing: BindableFloat` — default 0.3
  - `palette: str` — default `"rainbow"`
  - `gradient_id: Optional[str]`
  - `color: BindableColor` — primary color
  - `color_secondary: BindableColor` — for two-color modes
  - `beat_decay: BindableFloat` — default 0.15
  - `led_count: int` — 0 = auto-size
  - `mirror: bool`
- Add `"music_sync": MusicSyncColorStripSource` to `_SOURCE_TYPE_MAP`

## Phase 3: Stream Implementation

**File: `server/src/wled_controller/core/processing/music_sync_stream.py`** (new)
- Class `MusicSyncColorStripStream(ColorStripStream)` following `AudioColorStripStream` pattern
- Constructor: accept source + audio_capture_manager + stores. Create `MusicAnalyzer` instance
- `start()`: Acquire audio stream, start background thread
- `stop()`: Release audio stream, stop thread
- `_animate_loop()`:
  1. Get `AudioAnalysis` from audio stream
  2. Apply audio filter pipeline (if any)
  3. Feed to `MusicAnalyzer.update()` → `MusicFeatures`
  4. Dispatch to visualization renderer
  5. Double-buffer output

### Visualization Renderers

- **`pulse_on_beat`**: Full-strip flash on beat with exponential decay. Between beats: sine-wave pulsing synced to BPM. Color from palette indexed by beat_intensity.
- **`energy_gradient`**: Maps bass→warm, treble→cool. Overall brightness from energy. Gradient scrolls with beat_phase.
- **`spectrum_bands`**: 3 zones (bass/mid/treble), each fills proportionally to band energy. Mirror mode: bass center, treble edges.
- **`strobe_on_drop`**: Idle=gentle breathing. Buildup=increasing pulse. Drop=rapid strobe at 10 Hz. Recovery=fade back.

## Phase 4: API Integration

**File: `server/src/wled_controller/api/schemas/color_strip_sources.py`**
- Add `MusicSyncCSSResponse`, `MusicSyncCSSCreate`, `MusicSyncCSSUpdate`
- Add to all three union types

**File: `server/src/wled_controller/api/routes/color_strip_sources.py`**
- Import + add to `_RESPONSE_MAP`

**File: `server/src/wled_controller/core/processing/color_strip_stream_manager.py`**
- Add `music_sync` branch in `acquire()` (same pattern as `audio` branch)
- Update `refresh_audio_filter_pipelines()` to include `MusicSyncColorStripStream`

## Phase 5: Frontend

**File: `server/src/wled_controller/static/js/core/icons.ts`**
- Add `music_sync: _svg(P.radio)` icon

**File: `server/src/wled_controller/templates/modals/css-editor.html`**
- Add `<div id="css-editor-music-sync-section">` with:
  - Visualization mode selector (IconSelect)
  - Audio source dropdown
  - Sensitivity, Smoothing, Beat Decay (BindableScalarWidget containers)
  - Palette/gradient selector (EntitySelect)
  - Primary + Secondary color (BindableColorWidget)
  - Mirror checkbox

**File: `server/src/wled_controller/static/js/features/color-strips-music-sync.ts`** (new, extracted)
- Editor logic, widget factories, card renderer

**File: `server/src/wled_controller/static/js/features/color-strips.ts`**
- Register type in `CSS_TYPE_KEYS`, `CSS_SECTION_MAP`, `CSS_TYPE_SETUP`, `NON_PICTURE_TYPES`
- Import and register from `color-strips-music-sync.ts`

**Files: `en.json`, `ru.json`, `zh.json`**
- Add i18n keys for type, visualization modes, all field labels

## Phase 6: Testing

**File: `server/tests/core/audio/test_music_analyzer.py`** (new)
- BPM estimation from regular beats (within 5 BPM accuracy)
- BPM handles no beats gracefully
- Beat phase progression
- Energy envelope attack/release
- Frequency band splitting
- Drop detection state machine transitions
- No false drops on steady signal

**File: `server/tests/core/processing/test_music_sync_stream.py`** (new)
- Stream lifecycle (start/stop)
- Produces valid colors
- Hot-update parameters
- Auto-size from device
- All 4 visualization modes produce valid (n,3) uint8 arrays
- Mirror mode symmetry

**Storage + API tests:**
- `from_dict` roundtrip
- CRUD via test client

## Risks & Mitigations

- **BPM accuracy** — Median-of-recent-IBIs is robust for dance/electronic. For ambient, BPM noisy but `energy_gradient` and `spectrum_bands` don't depend on BPM
- **Drop detection false positives** — Require minimum energy threshold + sustained increase before buildup state
- **Frontend file size** — Extract to `color-strips-music-sync.ts` (following composite/notification pattern)
- **Strobe photosensitivity** — Cap at 10 Hz (below 15-25 Hz danger zone), add UI warning
- **Thread safety** — `MusicAnalyzer` owned exclusively by one stream thread, no shared access

## Dependency Order

`math_wave` should be implemented first (simpler, no audio dependency), then `music_sync`.