ledgrab/plans/music-sync/PLAN.md

music_sync Color Strip Source — Implementation Plan

Overview

A higher-level music-reactive CSS that provides semantic audio analysis (BPM detection, beat tracking, energy envelope, drop detection, frequency band energy) and multiple visualization modes. Builds on existing AudioCaptureManager and AudioAnalysis infrastructure. No new external dependencies — uses numpy-only analysis.

Requirements

• BPM estimation from real-time audio
• Beat onset detection with configurable threshold
• Smoothed RMS energy envelope with attack/release
• Drop detection: energy drops/buildups
• Frequency band energy: bass (20-250 Hz), mid (250-4k Hz), treble (4k-20k Hz)
• Four visualization modes: pulse_on_beat, energy_gradient, spectrum_bands, strobe_on_drop
• Uses existing audio engine infrastructure (no new audio capture code)
• No external dependencies beyond numpy

Phase 1: Music Analysis Engine

File: server/src/wled_controller/core/audio/music_analyzer.py (new)

1.1 MusicFeatures dataclass (frozen=True)

• bpm: float — estimated BPM (0 if unknown)
• beat: bool — beat detected this frame
• beat_intensity: float — 0.0-1.0
• beat_phase: float — 0.0-1.0 position in beat cycle
• energy: float — smoothed RMS 0.0-1.0
• energy_delta: float — rate of change
• bass_energy, mid_energy, treble_energy: float — 0.0-1.0
• drop_state: str — "idle"|"buildup"|"drop"|"recovery"
• drop_intensity: float — 0.0-1.0

1.2 MusicAnalyzer class

• State: rolling energy buffer (~4s at ~43 Hz = ~172 samples), beat history timestamps (last 30), smoothed band energies, BPM estimate, drop state machine
• update(analysis: AudioAnalysis) -> MusicFeatures: main entry point
• BPM estimation: Track beat timestamps, compute median inter-beat interval, exponential smoothing, clamp 40-220 BPM
• Beat tracking: Pass through AudioAnalysis.beat + compute beat_phase (position in current beat cycle)
• Energy envelope: Smoothed RMS with configurable attack/release
• Drop detection: State machine: idle -> buildup (energy rising steadily 1-2s), buildup -> drop (energy drops >50% within 100ms), drop -> recovery (after 500ms), recovery -> idle
• Frequency bands: Sum spectrum bins into 3 bands from 64-band spectrum

Phase 2: Storage Model

File: server/src/wled_controller/storage/color_strip_source.py

• Add MusicSyncColorStripSource dataclass after AudioColorStripSource
• Fields:
  • visualization_mode: str — default "pulse_on_beat"
  • audio_source_id: str — references AudioSource
  • sensitivity: BindableFloat — default 1.0
  • smoothing: BindableFloat — default 0.3
  • palette: str — default "rainbow"
  • gradient_id: Optional[str]
  • color: BindableColor — primary color
  • color_secondary: BindableColor — for two-color modes
  • beat_decay: BindableFloat — default 0.15
  • led_count: int — 0 = auto-size
  • mirror: bool
• Add "music_sync": MusicSyncColorStripSource to _SOURCE_TYPE_MAP

Phase 3: Stream Implementation

File: server/src/wled_controller/core/processing/music_sync_stream.py (new)

• Class MusicSyncColorStripStream(ColorStripStream) following AudioColorStripStream pattern
• Constructor: accept source + audio_capture_manager + stores. Create MusicAnalyzer instance
• start(): Acquire audio stream, start background thread
• stop(): Release audio stream, stop thread
• _animate_loop():
  1. Get AudioAnalysis from audio stream
  2. Apply audio filter pipeline (if any)
  3. Feed to MusicAnalyzer.update() → MusicFeatures
  4. Dispatch to visualization renderer
  5. Double-buffer output

Visualization Renderers

• pulse_on_beat: Full-strip flash on beat with exponential decay. Between beats: sine-wave pulsing synced to BPM. Color from palette indexed by beat_intensity.
• energy_gradient: Maps bass→warm, treble→cool. Overall brightness from energy. Gradient scrolls with beat_phase.
• spectrum_bands: 3 zones (bass/mid/treble), each fills proportionally to band energy. Mirror mode: bass center, treble edges.
• strobe_on_drop: Idle=gentle breathing. Buildup=increasing pulse. Drop=rapid strobe at 10 Hz. Recovery=fade back.

Phase 4: API Integration

File: server/src/wled_controller/api/schemas/color_strip_sources.py

• Add MusicSyncCSSResponse, MusicSyncCSSCreate, MusicSyncCSSUpdate
• Add to all three union types

File: server/src/wled_controller/api/routes/color_strip_sources.py

• Import + add to _RESPONSE_MAP

File: server/src/wled_controller/core/processing/color_strip_stream_manager.py

• Add music_sync branch in acquire() (same pattern as audio branch)
• Update refresh_audio_filter_pipelines() to include MusicSyncColorStripStream

Phase 5: Frontend

File: server/src/wled_controller/static/js/core/icons.ts

• Add music_sync: _svg(P.radio) icon

File: server/src/wled_controller/templates/modals/css-editor.html

• Add <div id="css-editor-music-sync-section"> with:
  • Visualization mode selector (IconSelect)
  • Audio source dropdown
  • Sensitivity, Smoothing, Beat Decay (BindableScalarWidget containers)
  • Palette/gradient selector (EntitySelect)
  • Primary + Secondary color (BindableColorWidget)
  • Mirror checkbox

File: server/src/wled_controller/static/js/features/color-strips-music-sync.ts (new, extracted)

• Editor logic, widget factories, card renderer

File: server/src/wled_controller/static/js/features/color-strips.ts

• Register type in CSS_TYPE_KEYS, CSS_SECTION_MAP, CSS_TYPE_SETUP, NON_PICTURE_TYPES
• Import and register from color-strips-music-sync.ts

Files: en.json, ru.json, zh.json

• Add i18n keys for type, visualization modes, all field labels

Phase 6: Testing

File: server/tests/core/audio/test_music_analyzer.py (new)

• BPM estimation from regular beats (within 5 BPM accuracy)
• BPM handles no beats gracefully
• Beat phase progression
• Energy envelope attack/release
• Frequency band splitting
• Drop detection state machine transitions
• No false drops on steady signal

File: server/tests/core/processing/test_music_sync_stream.py (new)

• Stream lifecycle (start/stop)
• Produces valid colors
• Hot-update parameters
• Auto-size from device
• All 4 visualization modes produce valid (n,3) uint8 arrays
• Mirror mode symmetry

Storage + API tests:

• from_dict roundtrip
• CRUD via test client

Risks & Mitigations

• BPM accuracy — Median-of-recent-IBIs is robust for dance/electronic. For ambient, BPM noisy but energy_gradient and spectrum_bands don't depend on BPM
• Drop detection false positives — Require minimum energy threshold + sustained increase before buildup state
• Frontend file size — Extract to color-strips-music-sync.ts (following composite/notification pattern)
• Strobe photosensitivity — Cap at 10 Hz (below 15-25 Hz danger zone), add UI warning
• Thread safety — MusicAnalyzer owned exclusively by one stream thread, no shared access

Dependency Order

math_wave should be implemented first (simpler, no audio dependency), then music_sync.