docs(android): remove ANDROID-REVIEW planning/review docs
The Android feature-gap assessment and per-feature design docs have served their purpose — notification, audio, webcam, and the foreground-app automation condition are all implemented and merged, so no gaps remain to track. The implementation is documented in the code, commit messages, and git history; the review docs are now obsolete. No committed files referenced them (only the local-only plans/ archives, left as point-in-time records).
This commit is contained in:
@@ -1,308 +0,0 @@
|
||||
# Plan: Android on-device audio capture
|
||||
|
||||
> Status: proposed plan (not yet approved). No code changes. Last updated 2026-06-01.
|
||||
|
||||
## Context
|
||||
|
||||
LedGrab's audio-reactive features (music analyzer, audio value sources, band filters)
|
||||
depend on capturing an audio stream and running it through `AudioAnalyzer`
|
||||
(`server/src/ledgrab/core/audio/analysis.py`). On desktop this is fed by **WASAPI**
|
||||
(Windows) or **Sounddevice/PortAudio** (cross-platform). On the **experimental
|
||||
Android-TV build** neither is available — `sounddevice` has no Chaquopy wheel and PortAudio
|
||||
isn't bundled — so `core/audio/__init__.py` registers only `DemoAudioEngine`, and
|
||||
audio-reactive lighting is effectively dead on Android.
|
||||
|
||||
Android does not need PortAudio: the platform exposes **`AudioPlaybackCapture`** (API 29+),
|
||||
which captures system playback audio and **takes a `MediaProjection` token — the very token
|
||||
the app already obtains for screen capture** (`ScreenCapture(projection, …)`). This plan adds
|
||||
a push-based Android audio engine so the TV box can drive sound-reactive lighting from its own
|
||||
media playback, at parity with how desktop audio feeds the analyzer.
|
||||
|
||||
The design mirrors the working screen-capture bridge
|
||||
(`mediaprojection_engine.py` ↔ `ScreenCapture.kt` ↔ `PythonBridge`) and the existing audio
|
||||
engine abstraction (`AudioCaptureEngine` / `AudioCaptureStreamBase` /
|
||||
`AudioEngineRegistry`). **No new Python dependencies** (`numpy` is already bundled) → no
|
||||
Chaquopy / `build.gradle.kts` `pip {}` changes.
|
||||
|
||||
---
|
||||
|
||||
## Approach
|
||||
|
||||
A new **push-based** audio engine registered in the existing `AudioEngineRegistry`:
|
||||
|
||||
- **Python:** `AndroidAudioEngine` + `AndroidAudioCaptureStream` mirroring `SounddeviceEngine`,
|
||||
but `read_chunk()` pops PCM from a module-level queue that **Kotlin fills** (mirror of
|
||||
`mediaprojection_engine.push_frame`). High `ENGINE_PRIORITY` so
|
||||
`AudioEngineRegistry.get_best_available_engine()` selects it on Android. The existing
|
||||
`ManagedAudioStream` capture loop and `AudioAnalyzer` consume `read_chunk()` unchanged.
|
||||
- **Android:** an `AudioCapture` helper using `AudioRecord` + `AudioPlaybackCaptureConfiguration`
|
||||
(reusing `CaptureService`'s `MediaProjection`), pushing float32 PCM to Python. Mic
|
||||
(`AudioSource.MIC`) fallback. Wired into `CaptureService` next to `ScreenCapture`.
|
||||
|
||||
```
|
||||
[media playback] → AudioRecord (AudioPlaybackCapture, reuses MediaProjection)
|
||||
→ AudioCapture.kt → PythonBridge.pushAudio(pcmFloat32, frames, channels)
|
||||
→ android_audio_engine.push_samples() [module-level queue]
|
||||
→ AndroidAudioCaptureStream.read_chunk() → ManagedAudioStream → AudioAnalyzer [unchanged]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part A — Python (server)
|
||||
|
||||
**New file: `server/src/ledgrab/core/audio/android_audio_engine.py`** — mirror
|
||||
`mediaprojection_engine.py` (queue + configure + push) and `sounddevice_engine.py` (engine/stream shape):
|
||||
|
||||
```python
|
||||
import queue
|
||||
import numpy as np
|
||||
from typing import Any, Dict, List
|
||||
from ledgrab.core.audio.base import AudioCaptureEngine, AudioCaptureStreamBase, AudioDeviceInfo
|
||||
from ledgrab.utils import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
_pcm_queue: "queue.Queue[np.ndarray]" = queue.Queue(maxsize=8)
|
||||
_sample_rate = 48000
|
||||
_channels = 2
|
||||
_chunk_size = 1024
|
||||
_active = False
|
||||
|
||||
def configure(sample_rate: int, channels: int, chunk_size: int) -> None:
|
||||
"""Called from Kotlin before audio frames start flowing. Drains stale PCM."""
|
||||
global _sample_rate, _channels, _chunk_size, _active
|
||||
while not _pcm_queue.empty():
|
||||
try: _pcm_queue.get_nowait()
|
||||
except queue.Empty: break
|
||||
_sample_rate, _channels, _chunk_size = sample_rate, channels, chunk_size
|
||||
_active = True
|
||||
|
||||
def push_samples(pcm_float32: bytes) -> None:
|
||||
"""Push one interleaved float32 PCM chunk from Kotlin. Drops oldest if full."""
|
||||
samples = np.frombuffer(pcm_float32, dtype=np.float32)
|
||||
try:
|
||||
_pcm_queue.put_nowait(samples)
|
||||
except queue.Full:
|
||||
try: _pcm_queue.get_nowait()
|
||||
except queue.Empty: pass
|
||||
try: _pcm_queue.put_nowait(samples)
|
||||
except queue.Full: pass
|
||||
|
||||
def shutdown() -> None:
|
||||
global _active
|
||||
_active = False
|
||||
|
||||
|
||||
class AndroidAudioCaptureStream(AudioCaptureStreamBase):
|
||||
@property
|
||||
def channels(self) -> int: return _channels
|
||||
@property
|
||||
def sample_rate(self) -> int: return _sample_rate
|
||||
@property
|
||||
def chunk_size(self) -> int: return _chunk_size
|
||||
def initialize(self) -> None:
|
||||
if not _active:
|
||||
raise RuntimeError("Android audio engine not configured (only valid in-app).")
|
||||
self._initialized = True
|
||||
def cleanup(self) -> None:
|
||||
self._initialized = False
|
||||
def read_chunk(self) -> np.ndarray | None:
|
||||
try:
|
||||
return _pcm_queue.get(timeout=0.1) # 1-D float32 interleaved
|
||||
except queue.Empty:
|
||||
return None
|
||||
|
||||
|
||||
class AndroidAudioEngine(AudioCaptureEngine):
|
||||
ENGINE_TYPE = "android_playback"
|
||||
ENGINE_PRIORITY = 100 # highest on Android (demo is lower)
|
||||
@classmethod
|
||||
def is_available(cls) -> bool:
|
||||
from ledgrab.utils.platform import is_android
|
||||
return is_android() and _active
|
||||
@classmethod
|
||||
def get_default_config(cls) -> Dict[str, Any]:
|
||||
return {"sample_rate": _sample_rate, "channels": _channels, "chunk_size": _chunk_size}
|
||||
@classmethod
|
||||
def enumerate_devices(cls) -> List[AudioDeviceInfo]:
|
||||
if not cls.is_available(): return []
|
||||
return [AudioDeviceInfo(index=0, name="Android playback (system audio)",
|
||||
is_input=True, is_loopback=True,
|
||||
channels=_channels, default_samplerate=float(_sample_rate))]
|
||||
@classmethod
|
||||
def create_stream(cls, device_index, is_loopback, config) -> AndroidAudioCaptureStream:
|
||||
return AndroidAudioCaptureStream(device_index, is_loopback, {**cls.get_default_config(), **config})
|
||||
```
|
||||
|
||||
**Modify `server/src/ledgrab/core/audio/__init__.py`** — register behind a guarded import,
|
||||
matching the existing `_has_wasapi` / `_has_sounddevice` pattern:
|
||||
|
||||
```python
|
||||
try:
|
||||
from ledgrab.core.audio.android_audio_engine import AndroidAudioEngine
|
||||
_has_android_audio = True
|
||||
except ImportError:
|
||||
_has_android_audio = False
|
||||
...
|
||||
if _has_android_audio:
|
||||
AudioEngineRegistry.register(AndroidAudioEngine)
|
||||
```
|
||||
|
||||
**Reused, unchanged:** `AudioEngineRegistry.get_best_available_engine()` (picks by priority),
|
||||
`ManagedAudioStream._capture_loop()` (`audio_capture.py`), `AudioAnalyzer`, the audio value
|
||||
sources, and the device-enumeration endpoints. The Android engine appears as one loopback
|
||||
device named "Android playback (system audio)".
|
||||
|
||||
---
|
||||
|
||||
## Part B — Android (Kotlin + manifest)
|
||||
|
||||
**New file: `android/app/src/main/java/com/ledgrab/android/AudioCapture.kt`**
|
||||
|
||||
Mirrors `ScreenCapture.kt`, taking the same `MediaProjection`:
|
||||
|
||||
```kotlin
|
||||
class AudioCapture(
|
||||
private val projection: MediaProjection,
|
||||
private val bridge: PythonBridge,
|
||||
private val sampleRate: Int = 48000,
|
||||
private val channels: Int = 2,
|
||||
private val chunkFrames: Int = 1024,
|
||||
)
|
||||
```
|
||||
|
||||
- `start()` (API 29+, MediaProjection mode):
|
||||
- Build `AudioPlaybackCaptureConfiguration(projection)` adding usages
|
||||
`USAGE_MEDIA`, `USAGE_GAME`, `USAGE_UNKNOWN` (the capturable set).
|
||||
- `AudioRecord.Builder().setAudioPlaybackCaptureConfig(cfg)` with
|
||||
`AudioFormat(ENCODING_PCM_FLOAT, sampleRate, CHANNEL_IN_STEREO)`.
|
||||
- On a dedicated `HandlerThread`, loop `audioRecord.read(floatBuf, …, READ_BLOCKING)` →
|
||||
wrap into a little-endian float32 `ByteArray` (reusable buffer, like `ScreenCapture`'s
|
||||
`frameBuffer`) → `bridge.pushAudio(bytes, framesRead, channels)`.
|
||||
- `stop()`: stop/release `AudioRecord`, quit the thread.
|
||||
- **Mic fallback** (`startMic()`): `AudioSource.MIC` for root mode (no MediaProjection) or
|
||||
API < 29. Used only when playback capture is unavailable.
|
||||
|
||||
**Modify `android/app/src/main/java/com/ledgrab/android/PythonBridge.kt`** — add the audio
|
||||
push path (same shape as `pushFrame`, with a cached PyObject handle):
|
||||
|
||||
```kotlin
|
||||
@Volatile private var androidAudioEngine: PyObject? = null
|
||||
|
||||
fun configureAudio(sampleRate: Int, channels: Int, chunkFrames: Int) {
|
||||
val engine = Python.getInstance().getModule("ledgrab.core.audio.android_audio_engine")
|
||||
engine.callAttr("configure", sampleRate, channels, chunkFrames)
|
||||
androidAudioEngine = engine
|
||||
}
|
||||
fun pushAudio(pcmFloat32: ByteArray, frames: Int, channels: Int) {
|
||||
if (!running) return
|
||||
androidAudioEngine?.let {
|
||||
try { it.callAttr("push_samples", pcmFloat32) }
|
||||
catch (e: Exception) { Log.w(TAG, "pushAudio failed: ${e.message}") }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Modify `android/app/src/main/java/com/ledgrab/android/CaptureService.kt`** — in the
|
||||
MediaProjection start path (where `ScreenCapture` is created with the projection), if
|
||||
`RECORD_AUDIO` is granted and API ≥ 29, also `bridge.configureAudio(...)` and start an
|
||||
`AudioCapture(projection, bridge)`. Stop/release it in `onDestroy` alongside `ScreenCapture`.
|
||||
Root path → optional mic fallback (or skip; see Risks).
|
||||
|
||||
**Modify `android/app/src/main/AndroidManifest.xml`:**
|
||||
```xml
|
||||
<uses-permission android:name="android.permission.RECORD_AUDIO" />
|
||||
<!-- For mic-mode foreground capture on API 34+ (playback capture is covered by the
|
||||
existing mediaProjection FGS type): -->
|
||||
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_MICROPHONE" />
|
||||
```
|
||||
The existing `CaptureService` already declares `foregroundServiceType="mediaProjection|specialUse"`
|
||||
and holds `FOREGROUND_SERVICE_MEDIA_PROJECTION`; add `microphone` to the type only if mic
|
||||
fallback is implemented.
|
||||
|
||||
**Modify `MainActivity.kt`** — request `RECORD_AUDIO` at runtime alongside the existing
|
||||
`ensureNotificationPermission()` (POST_NOTIFICATIONS) flow, before starting capture. Capture
|
||||
proceeds without audio if denied (graceful degradation).
|
||||
|
||||
---
|
||||
|
||||
## Orchestration decision (the main trade-off)
|
||||
|
||||
Desktop starts audio capture **on demand** when an audio-reactive source is acquired
|
||||
(`AudioCaptureManager.acquire`). On Android, PCM only flows if Kotlin has set up `AudioRecord`.
|
||||
|
||||
- **MVP (recommended):** start `AudioCapture` when `CaptureService` starts (if `RECORD_AUDIO`
|
||||
granted + MediaProjection mode + API ≥ 29) and push continuously; the bounded queue drops
|
||||
frames when no audio source consumes them. Simplest; modest extra CPU.
|
||||
- **Future optimization:** on-demand start/stop signaled Python→Kotlin (Chaquopy can call
|
||||
Kotlin, as `BleBridge`/`UsbSerialBridge` show) so `AudioRecord` runs only while an
|
||||
audio-reactive source is active. Defer unless CPU/battery on low-end boxes warrants it.
|
||||
|
||||
---
|
||||
|
||||
## What does NOT change
|
||||
|
||||
- **Frontend / API** — audio engine + device selection, the music analyzer UI, and audio value
|
||||
sources are engine-agnostic; the Android engine shows up via the existing device enumeration.
|
||||
- **`build.gradle.kts` / Chaquopy pip block** — no new Python packages.
|
||||
- **Audio analysis pipeline** — `AudioAnalyzer`, band filters, `ManagedAudioStream` untouched.
|
||||
|
||||
---
|
||||
|
||||
## Files
|
||||
|
||||
**Create**
|
||||
- `server/src/ledgrab/core/audio/android_audio_engine.py`
|
||||
- `android/app/src/main/java/com/ledgrab/android/AudioCapture.kt`
|
||||
- `server/tests/core/audio/test_android_audio_engine.py`
|
||||
|
||||
**Modify**
|
||||
- `server/src/ledgrab/core/audio/__init__.py` — guarded import + registry registration.
|
||||
- `android/app/src/main/java/com/ledgrab/android/PythonBridge.kt` — `configureAudio` + `pushAudio`.
|
||||
- `android/app/src/main/java/com/ledgrab/android/CaptureService.kt` — start/stop `AudioCapture`.
|
||||
- `android/app/src/main/java/com/ledgrab/android/MainActivity.kt` — request `RECORD_AUDIO`.
|
||||
- `android/app/src/main/AndroidManifest.xml` — `RECORD_AUDIO` (+ mic FGS if mic fallback).
|
||||
|
||||
---
|
||||
|
||||
## Tests (Python — run on desktop CI, no Android device needed)
|
||||
|
||||
New `server/tests/core/audio/test_android_audio_engine.py`:
|
||||
|
||||
- `configure()` then `push_samples()` → `read_chunk()` returns the same float32 samples;
|
||||
queue drops oldest when full (push > maxsize).
|
||||
- `AndroidAudioEngine.is_available()` is `False` until `configure()` and only on Android
|
||||
(monkeypatch `ledgrab.utils.platform.is_android`); `True` after.
|
||||
- `enumerate_devices()` returns exactly one loopback device when active, `[]` otherwise.
|
||||
- Integration: with `is_android()` patched true + `configure()`, `get_best_available_engine()`
|
||||
returns `"android_playback"` (priority beats demo), and a stream created via
|
||||
`AudioEngineRegistry.create_stream("android_playback", 0, True, {})` yields pushed chunks.
|
||||
- Registry isolation: use `AudioEngineRegistry.clear_registry()` / re-register in fixtures so
|
||||
desktop engines aren't disturbed.
|
||||
|
||||
## Verification
|
||||
|
||||
1. **Python:** `py -3.13 -m pytest tests/core/audio/test_android_audio_engine.py --no-cov -q`
|
||||
(from `server/`), then the full suite.
|
||||
2. **Lint:** `ruff check src/ tests/ --fix` (from `server/`).
|
||||
3. **Android build:** `./gradlew :app:assembleDebug` (from `android/`).
|
||||
4. **On device/emulator (manual):** install APK → grant `RECORD_AUDIO` + screen-capture consent
|
||||
→ start capture → play non-DRM media (e.g. a local video / YouTube web) → create an
|
||||
audio-reactive value source bound to a strip → confirm the LEDs react to the audio, and the
|
||||
Android playback device appears in audio device enumeration.
|
||||
|
||||
## Risks / notes
|
||||
|
||||
- **DRM opt-out:** Netflix/Disney+/etc. set audio as non-capturable; `AudioPlaybackCapture`
|
||||
yields silence for them. Works for non-DRM media and the device's own audio. Document in UI.
|
||||
- **API 29 minimum** for playback capture (minSdk is 24). API 24–28 and root mode (no
|
||||
MediaProjection) → mic fallback only, or audio unsupported. Gate cleanly + log.
|
||||
- **`RECORD_AUDIO`** is a runtime "dangerous" permission — must be requested; capture must
|
||||
degrade gracefully when denied.
|
||||
- **Format:** request `ENCODING_PCM_FLOAT` so Kotlin pushes float32 matching
|
||||
`read_chunk()`'s contract (1-D interleaved float32, length = frames × channels). If a device
|
||||
rejects float, capture 16-bit PCM and convert (`/32768.0`) before pushing.
|
||||
- **Latency/CPU:** small `chunkFrames` (e.g. 1024 @ 48 kHz ≈ 21 ms) keeps reactivity tight;
|
||||
continuous capture (MVP) adds modest CPU on low-end boxes — see the orchestration trade-off.
|
||||
- **R8/ProGuard:** minify is disabled and the Python module is resolved by string from Kotlin;
|
||||
no new keep-rules needed.
|
||||
@@ -1,94 +0,0 @@
|
||||
# Android foreground-app automation condition — implementation notes
|
||||
|
||||
> Status: implemented on `feature/android-foreground-app-automation`. Last updated 2026-06-02.
|
||||
|
||||
## What & why
|
||||
|
||||
The desktop build has an **Application** automation rule (`ApplicationRule`): activate a scene
|
||||
when given apps are running / foreground / fullscreen. It was already wired end-to-end on
|
||||
Android (engine, storage, API, editor) but **silently never fired**, because the two
|
||||
Windows-only ctypes paths return empty off-Windows:
|
||||
|
||||
1. **Detection** — `PlatformDetector._get_topmost_process_sync()` (and the running/fullscreen
|
||||
variants) returned `(None, False)` / `set()` on Android.
|
||||
2. **The app picker** — populated from `GET /api/v1/system/processes` →
|
||||
`get_running_processes()`, also empty on Android, so users couldn't even choose an app.
|
||||
|
||||
This feature fills both holes using in-platform Android APIs and the established Kotlin↔Python
|
||||
bridge pattern. **Zero new Python or Gradle dependencies.**
|
||||
|
||||
## Design decision: one implicit "foreground" mode on Android
|
||||
|
||||
Android exposes exactly one obtainable signal — the **current foreground app package**. The
|
||||
desktop rule's four match types (`running` / `topmost` / `fullscreen` / `topmost_fullscreen`)
|
||||
are either unobtainable (`running` — `getRunningTasks` is restricted) or identical (a foreground
|
||||
TV app effectively *is* fullscreen). So on Android:
|
||||
|
||||
- The editor **hides the match-type selector** and the collector forces `match_type="topmost"`.
|
||||
- `_get_topmost_process_sync()` returns `(package, True)`; the running/fullscreen detectors
|
||||
return the foreground app as a best-effort single-element set so legacy rules still behave.
|
||||
|
||||
This avoided touching the existing plain `<select>` (forbidden for new UI) and removed a
|
||||
misleading 4-way choice — a simplification surfaced by the pre-implementation plan review.
|
||||
|
||||
## Detection — `ForegroundAppBridge` (Kotlin) ↔ `platform_detector.py`
|
||||
|
||||
`android/app/src/main/java/com/ledgrab/android/ForegroundAppBridge.kt` (an `object` singleton,
|
||||
mirroring `CameraBridge`, context bound in `LedGrabApp.onCreate`):
|
||||
|
||||
- `getForegroundPackage()` — `UsageStatsManager.queryEvents(now - 10s, now)`, returns the package
|
||||
of the most recent `MOVE_TO_FOREGROUND` / `ACTIVITY_RESUMED` event (the two constants share a
|
||||
value; the ~10s window absorbs event lag against the ~1s automation tick). `queryEvents` is the
|
||||
right call — `queryUsageStats` gives aggregate durations, not "current app".
|
||||
- `hasUsageAccess()` — `AppOpsManager` `OPSTR_GET_USAGE_STATS` check (`unsafeCheckOpNoThrow` on
|
||||
API 29+, `checkOpNoThrow` below).
|
||||
- `listLaunchableApps()` — `LauncherApps.getActivityList` → JSON `[{package,label}]` for the
|
||||
picker. The sanctioned launchable-app API; **no `QUERY_ALL_PACKAGES`**.
|
||||
|
||||
`server/src/ledgrab/core/automations/platform_detector.py`:
|
||||
|
||||
- Module-level guarded wrappers `get_foreground_package()` / `has_usage_access()` /
|
||||
`list_installed_apps()` resolve `jclass("com.ledgrab.android.ForegroundAppBridge").INSTANCE`
|
||||
lazily (never at import — the module loads on desktop CI). These are the **test monkeypatch
|
||||
surface**, mirroring `android_camera_engine`.
|
||||
- The `is_android()` branch is placed **ahead of** the import-time `if not _IS_WINDOWS:`
|
||||
early-return in each detector — the critical fix from plan review (a naive wiring would no-op
|
||||
behind the Windows guard yet still pass tests). The Windows ctypes path is unchanged
|
||||
(regression-tested).
|
||||
- A one-time `logger.warning` fires when Usage Access is missing.
|
||||
|
||||
## App picker — `/system/installed-apps` + platform signal
|
||||
|
||||
- `GET /api/v1/system/installed-apps` → `{apps:[{package,label}], count}` (empty off-Android).
|
||||
- `GET /api/v1/system/info` → `{is_android, app_match_kind, usage_access_granted}` — the editor
|
||||
reads it to pick the app source + matching semantics and to show the Usage-Access banner.
|
||||
- Frontend: the command-palette picker (`core/process-picker.ts`) gained label→value support; a
|
||||
new `AppPalette` shows the human label and inserts the package name. On Android the app-rule
|
||||
editor uses it (`attachAppPicker`) instead of the process picker, plus a package-name hint and
|
||||
the Usage-Access banner.
|
||||
|
||||
## Value semantics (no migration)
|
||||
|
||||
`ApplicationRule.apps` are **package names** on Android (`com.netflix.mediaclient`) vs **process
|
||||
names** on Windows (`chrome.exe`). Same field, same matching code — **no storage migration** —
|
||||
but rules are **not portable across platforms**. Documented in the model/schema docstrings and a
|
||||
user-facing editor hint.
|
||||
|
||||
## Permission UX
|
||||
|
||||
`PACKAGE_USAGE_STATS` is a special access (can't be granted at runtime):
|
||||
|
||||
- Manifest declares it with `tools:ignore="ProtectedPermissions"`.
|
||||
- MainActivity shows a passive **"Grant usage access"** button (opens
|
||||
`ACTION_USAGE_ACCESS_SETTINGS`, with a generic-Settings fallback) only while access is missing.
|
||||
**No blanket prompt at capture start** — most users have no foreground-app rule.
|
||||
- The web-UI rule editor shows a banner when an Android Application rule lacks access.
|
||||
|
||||
## Limitations
|
||||
|
||||
- Foreground-app only; no full window-title or arbitrary process enumeration on Android.
|
||||
- Detection rides the existing ~1s automation poll; `queryEvents` can lag a few seconds.
|
||||
- Rules authored on desktop don't match on Android and vice-versa (package vs process names).
|
||||
- The on-device "Grant usage access" button currently shows whenever access is missing (not
|
||||
gated on whether an Android Application rule exists), to avoid Activity↔server coupling; the
|
||||
web-UI banner provides the contextual guidance.
|
||||
@@ -1,196 +0,0 @@
|
||||
# Android (TV) — Missing Functionality Assessment
|
||||
|
||||
> Status: review/feasibility document. No code changes. Last updated 2026-06-01.
|
||||
|
||||
## Context
|
||||
|
||||
LedGrab ships an **experimental on-device Android-TV build**: a Kotlin shell that
|
||||
embeds the Python FastAPI server via **Chaquopy**, with Kotlin↔Python **bridges**
|
||||
(`PythonBridge`, `BleBridge`, `UsbSerialBridge`). Several desktop features are
|
||||
unavailable on this build because their Python backends rely on native libraries
|
||||
that have no Android/Chaquopy wheels (`mss`, `dxcam`, `sounddevice`/PortAudio,
|
||||
`opencv`, `nvidia-ml-py`, `winrt`, `dbus-next`), or on OS facilities Android
|
||||
sandboxes differently.
|
||||
|
||||
The README "Feature support by OS" table now carries an Android column reflecting
|
||||
this. This document assesses **whether each missing feature can be added**, how, and
|
||||
whether it's worth it.
|
||||
|
||||
### The enabling pattern (why most of this is feasible)
|
||||
|
||||
Every desktop capability that's "missing" on Android is missing only because of a
|
||||
*native dependency*, not because the capability is impossible. Android exposes the
|
||||
same capability through a platform API, and the codebase already has the bridge
|
||||
shape to plug it in:
|
||||
|
||||
> **Bridge pattern:** a Kotlin component captures an event/buffer → pushes it across
|
||||
> the Chaquopy JNI boundary into a **module-level receiver** in a small Python engine
|
||||
> → an existing engine/stream consumes it unchanged.
|
||||
|
||||
Reference implementation: `server/src/ledgrab/core/capture_engines/mediaprojection_engine.py`
|
||||
(`configure()` + `push_frame()` + a bounded `queue.Queue`) ↔
|
||||
`android/app/src/main/java/com/ledgrab/android/ScreenCapture.kt` ↔
|
||||
`PythonBridge.pushFrame()`. Screen capture already works on Android this exact way.
|
||||
|
||||
So for most missing features the work is: **add a Kotlin capture source + a thin
|
||||
Python receiver engine mirroring that pattern.**
|
||||
|
||||
---
|
||||
|
||||
## Current Android capability matrix
|
||||
|
||||
| Feature | Desktop | Android (TV) today | Missing? |
|
||||
| ------- | ------- | ------------------ | -------- |
|
||||
| Screen capture | DXCam/WGC/MSS | ✅ MediaProjection + root `screenrecord` | No |
|
||||
| LED transports (network/USB-serial/BLE) | ✅ | ✅ (USB via Android driver, BLE via Android bridge) | No |
|
||||
| System metrics | psutil | ✅ CPU/RAM/battery/thermal via `/proc`, `/sys` (`AndroidMetricsProvider`) | No |
|
||||
| **Audio capture** | WASAPI / Sounddevice | ❌ no PortAudio | **Yes** |
|
||||
| Notification capture | WinRT / D-Bus | ✅ NotificationListenerService → `push_notification()` | No (implemented) |
|
||||
| Webcam capture | OpenCV | ✅ Camera2 + on-demand bridge (`AndroidCameraEngine`) | No (implemented) |
|
||||
| GPU monitoring | NVML | ❌ no NVIDIA GPU | Marginal |
|
||||
| Capture from *another* Android phone | scrcpy/ADB | ❌ | Skip (redundant) |
|
||||
| Automation: foreground-app condition | Windows ctypes (running/topmost/fullscreen) | ✅ foreground app via UsageStatsManager (`ForegroundAppBridge`) | No (implemented) |
|
||||
| Monitor names / multi-display | WMI / generic | Single built-in display | Low value |
|
||||
|
||||
---
|
||||
|
||||
## Per-feature feasibility
|
||||
|
||||
### 🔊 Audio capture — **FEASIBLE, HIGH VALUE** ⭐ (detailed plan exists)
|
||||
|
||||
- **Blocker:** only `sounddevice`/PortAudio is missing — not the capability.
|
||||
- **Android path:** `AudioPlaybackCapture` (API 29+) captures system playback audio and
|
||||
**takes a `MediaProjection` token — which the app already obtains for screen capture.**
|
||||
Kotlin `AudioRecord` → push PCM (float32) → a new push-based `AndroidAudioEngine`
|
||||
mirroring `mediaprojection_engine.py`, registered in `core/audio/__init__.py`, feeding
|
||||
the existing `AudioAnalyzer` unchanged. Mic (`AudioSource.MIC`) is the fallback.
|
||||
- **Effort:** moderate. **Value:** high — music/sound-reactive lighting is a flagship use
|
||||
on a TV box. **No new Python deps.**
|
||||
- ⚠️ DRM-protected apps (Netflix etc.) opt out of playback capture; works for non-DRM
|
||||
media and the device's own audio. Root mode (no MediaProjection) → mic-only.
|
||||
- 📄 **See `android-audio-capture-plan.md`** for the full implementation plan.
|
||||
|
||||
### 🔔 Notification capture — **IMPLEMENTED** ✅ (shipped)
|
||||
|
||||
- **Android is the *best* platform for this:** `NotificationListenerService` is the native,
|
||||
event-push mechanism (no polling).
|
||||
- **Path:** a `NotificationListenerService` resolves the posting app's display label and
|
||||
pushes it via a module-level `push_notification()` into the existing
|
||||
`os_notification_listener.py` pipeline (a new push-based `_AndroidBackend` alongside
|
||||
`_WindowsBackend`/`_LinuxBackend`). Existing `NotificationColorStripSource` filters,
|
||||
per-app colors/sounds, and the history endpoint all work unchanged. **No new Python deps.**
|
||||
- **Permission:** user enables "Notification access" in Settings (`ACTION_NOTIFICATION_LISTENER_SETTINGS`);
|
||||
no runtime-permission popup.
|
||||
- **Effort:** moderate. **Value:** high.
|
||||
- ✅ **Implemented** on branch `feature/android-notification-capture`: a push-based
|
||||
`_AndroidBackend` + module-level `push_notification()` in `os_notification_listener.py`,
|
||||
a Kotlin `LedGrabNotificationListener` (NLS), and prompt-once permission UX. App-name
|
||||
parity — only the resolved app label crosses the JNI boundary, never the notification
|
||||
title/body. ⚠️ App labels can differ across OSes (Windows `display_name` / Linux D-Bus
|
||||
`app_name` / Android `getApplicationLabel`), so desktop-configured per-app colors/filters
|
||||
may need re-matching on Android.
|
||||
|
||||
### 📷 Webcam capture — **IMPLEMENTED** ✅ (shipped)
|
||||
|
||||
- **Blocker** was `opencv-python-headless` (no Chaquopy cp311 wheel) — but capture doesn't
|
||||
*need* OpenCV. Implemented with **Camera2** + `ImageReader` in Kotlin pushing RGB frames
|
||||
through the same bridge as MediaProjection into a new `AndroidCameraEngine`.
|
||||
- **Path:** a Kotlin `CameraBridge` singleton (Camera2) enumerates cameras and **opens the
|
||||
camera on demand** (only while a capture source is active — driven Python→Kotlin via the
|
||||
`BleBridge`/`UsbSerialBridge` pattern), converts each frame YUV_420_888→RGB, and pushes it
|
||||
into a push-based `AndroidCameraEngine` (`core/capture_engines/android_camera_engine.py`)
|
||||
that mirrors `mediaprojection_engine.py`. Cameras surface as selectable "displays" exactly
|
||||
like the desktop OpenCV `CameraEngine`; the data-driven capture-template UI (engine list +
|
||||
`resolution` config + display picker) needs **no changes**. **No new Python deps; no new
|
||||
Gradle deps** (Camera2 is in-platform).
|
||||
- **Permission:** `CAMERA` requested at capture-start, gated on `FEATURE_CAMERA_ANY` so
|
||||
camera-less TV boxes never see the prompt; graceful degradation when denied. The service is
|
||||
promoted with the `camera` FGS type (+ `FOREGROUND_SERVICE_CAMERA`) **only when CAMERA is
|
||||
already granted**, so backgrounded capture keeps working without risking a failed service
|
||||
start on camera-less boxes. (Unlike audio playback capture, the camera can't ride the
|
||||
MediaProjection token, so it needs its own FGS type to survive backgrounding.)
|
||||
- **Effort:** moderate. **Value:** low (TVs rarely have cameras), but the implementation reuses
|
||||
existing infrastructure end-to-end. **Priority `0`** so it's never auto-selected over
|
||||
MediaProjection — chosen explicitly via `engine_type="android_camera"`.
|
||||
- ⚠️ **MVP scope / limitations:** webcam capture works **while LedGrab capture is running**
|
||||
(no camera-only server path on Android); one camera active at a time; `"auto"` picks a
|
||||
balanced output size (not the sensor max) to keep per-frame YUV→RGB cheap; USB-UVC webcams
|
||||
appear only if the device routes them through Camera2 (varies by box); no frame-rotation
|
||||
correction.
|
||||
- 📄 **See `android-webcam-capture-plan.md`** for the full implementation notes.
|
||||
|
||||
### 🎮 GPU monitoring — **MARGINAL, SKIP FOR NOW**
|
||||
|
||||
- NVML is desktop-NVIDIA only. Android GPU load lives in **vendor-specific sysfs**
|
||||
(Adreno `/sys/class/kgsl/kgsl-3d0/gpubusy`, Mali `/sys/class/devfreq/*.mali/...`),
|
||||
inconsistent and often root-only.
|
||||
- CPU/RAM/battery/thermal are **already** covered by `AndroidMetricsProvider`. A best-effort
|
||||
GPU-load reader could be added to that provider, but reliability is poor and value is low.
|
||||
|
||||
### 🪟 Automation: foreground-app condition — **IMPLEMENTED** ✅ (shipped)
|
||||
|
||||
- Android forbids full window/process enumeration (`getRunningTasks` restricted since API 21+),
|
||||
but the *current foreground app package* is obtainable via `UsageStatsManager` (needs the
|
||||
`PACKAGE_USAGE_STATS` special access).
|
||||
- **Path:** a Kotlin `ForegroundAppBridge` (UsageStatsManager `queryEvents` over a ~10s trailing
|
||||
window + `LauncherApps` for the picker + an `AppOpsManager` access check) bridged into
|
||||
`automations/platform_detector.py` via the guarded-`jclass` pattern, ahead of the Windows-only
|
||||
ctypes path. The existing `ApplicationRule` / `AutomationEngine` / storage / deactivation modes
|
||||
are unchanged — only the detection + the picker's data source were filled in. **No new Python
|
||||
or Gradle deps** (UsageStatsManager + LauncherApps are in-platform; matching only string-compares
|
||||
the package name, so no `QUERY_ALL_PACKAGES` / package visibility is needed).
|
||||
- **UI:** the automation editor's app picker lists launchable apps by human label (storing the
|
||||
package name) via a new `GET /api/v1/system/installed-apps`; on Android the match-type selector
|
||||
is hidden and `match_type` is forced to `topmost` (the only obtainable signal), with a
|
||||
cross-platform value caveat — `apps` are **package names** on Android (`com.netflix.mediaclient`)
|
||||
vs **process names** on Windows (`chrome.exe`), so rules are not portable across platforms.
|
||||
- **Permission:** `PACKAGE_USAGE_STATS` is a special access (Settings deep-link via
|
||||
`ACTION_USAGE_ACCESS_SETTINGS`); the device shows a "Grant usage access" button when missing,
|
||||
and the web-UI rule editor shows a banner (driven by `/system/info`'s `usage_access_granted`).
|
||||
No blanket prompt at capture start. Detection degrades gracefully (rule never matches, warned
|
||||
once) until access is granted. **Effort:** moderate. **Value:** moderate (per-app scenes on a
|
||||
TV box). Full window-title matching remains out of scope (Android does not expose it).
|
||||
- 📄 **See `android-foreground-app-automation-plan.md`** for the full implementation notes.
|
||||
|
||||
### 📱 Capture from *another* Android phone (scrcpy/ADB) — **SKIP**
|
||||
|
||||
- Impractical and redundant: no `adb` binary in Chaquopy, TV boxes can't reliably host an
|
||||
adb server, and the device already captures its **own** screen via MediaProjection.
|
||||
|
||||
### 🖥️ Monitor names / multi-display — **LOW VALUE**
|
||||
|
||||
- `DisplayManager` can report a better display name and enumerate secondary (HDMI) displays,
|
||||
but MediaProjection captures the default display; capturing a secondary display is more
|
||||
involved and rarely useful on a single-screen box.
|
||||
|
||||
---
|
||||
|
||||
## Prioritization
|
||||
|
||||
| Priority | Feature | Effort | Value | New Python deps | Status |
|
||||
| -------- | ------- | ------ | ----- | --------------- | ------ |
|
||||
| 1 | Notification capture | Moderate | High | None | **✅ Implemented** |
|
||||
| 2 | Audio capture | Moderate | High | None | **✅ Implemented** |
|
||||
| 4 | Webcam capture (Camera2) | Moderate | Low | None | **✅ Implemented** |
|
||||
| 3 | Automation: foreground-app condition | Moderate | Moderate | None | **✅ Implemented** |
|
||||
| — | GPU load (vendor sysfs) | Low–Med | Low | None | Not recommended |
|
||||
| — | Capture from another phone | — | — | — | Won't do |
|
||||
| — | Multi-display / monitor names | Low | Low | None | Not recommended |
|
||||
|
||||
**Status:** notifications, audio, webcam, **and the foreground-app automation condition** are all
|
||||
shipped — each reuses existing infrastructure (the Kotlin↔Python bridge pattern, the
|
||||
MediaProjection consent token / process-global `Python.getInstance()`, the
|
||||
capture/audio/notification/automation pipelines) and adds **zero** Python dependencies, so none
|
||||
risks the Chaquopy `--no-deps` build constraint documented in `CLAUDE.md`. No prioritized ideas
|
||||
remain; GPU load, another-phone capture, and multi-display remain not-recommended / won't-do.
|
||||
|
||||
## Cross-cutting notes
|
||||
|
||||
- **No `build.gradle.kts` / Chaquopy pip impact** for notifications or audio — both use Android
|
||||
platform APIs (Kotlin) + stdlib/`numpy` (already bundled) on the Python side.
|
||||
- **Per-instance `PythonBridge`:** `PythonBridge` is created per `CaptureService` instance, so
|
||||
system-bound services (e.g. a `NotificationListenerService`) call Python via the
|
||||
process-global `Python.getInstance()` rather than borrowing that bridge.
|
||||
- **Permissions are the recurring friction**, not the capture: audio needs `RECORD_AUDIO` +
|
||||
(for playback capture) a MediaProjection token; notifications need the "Notification access"
|
||||
settings toggle; foreground-app automation needs `PACKAGE_USAGE_STATS`.
|
||||
@@ -1,168 +0,0 @@
|
||||
# Plan: Android on-device webcam capture
|
||||
|
||||
> Status: **implemented** on branch `feature/android-webcam-capture`. Last updated 2026-06-02.
|
||||
|
||||
## Context
|
||||
|
||||
LedGrab captures webcams on desktop through OpenCV (`cv2.VideoCapture`) in
|
||||
`server/src/ledgrab/core/capture_engines/camera_engine.py`. On the **experimental Android-TV
|
||||
build**, `opencv-python-headless` has no Chaquopy cp311 wheel, so the camera engine never
|
||||
loads and cameras are unusable on-device.
|
||||
|
||||
Android doesn't need OpenCV to capture a camera: the platform exposes **Camera2**
|
||||
(`android.hardware.camera2`), and the codebase already has the bridge shape to plug a Kotlin
|
||||
capture source into a push-based Python engine. This feature adds an on-device camera engine
|
||||
so a USB/integrated camera can drive ambient lighting, at parity with how the desktop OpenCV
|
||||
camera engine feeds the pipeline.
|
||||
|
||||
The design mirrors the working screen-capture bridge
|
||||
(`mediaprojection_engine.py` ↔ `ScreenCapture.kt`) and the just-shipped audio engine
|
||||
(`android_audio_engine.py` ↔ `AudioCapture.kt`). **No new Python dependencies** (numpy already
|
||||
bundled) and **no new Gradle dependencies** (Camera2 is in-platform) → no Chaquopy /
|
||||
`build.gradle.kts` changes.
|
||||
|
||||
## Approach
|
||||
|
||||
A new **push-based** capture engine registered in the existing `EngineRegistry`, plus a Kotlin
|
||||
`CameraBridge` that opens the camera **on demand**:
|
||||
|
||||
```
|
||||
[capture source acquired] → AndroidCameraCaptureStream.initialize()
|
||||
→ android_camera_engine.start_camera(index, w, h) [guarded jclass]
|
||||
→ CameraBridge.startCamera(index, w, h) [Camera2 open + session]
|
||||
→ onImageAvailable → YUV_420_888→RGB (stride-aware) → push_frame(rgbBytes, w, h)
|
||||
→ android_camera_engine [module-level queue] → AndroidCameraCaptureStream.capture_frame()
|
||||
→ ScreenCaptureLiveStream → processing pipeline [unchanged]
|
||||
|
||||
[capture source released] → AndroidCameraCaptureStream.cleanup()
|
||||
→ android_camera_engine.stop_camera() → CameraBridge.stopCamera() [releases the camera]
|
||||
```
|
||||
|
||||
The camera is **only open while a camera source is active** — the camera-in-use indicator and
|
||||
battery cost are bounded to actual use, unlike always-on screen/audio capture. This on-demand
|
||||
control reuses the synchronous Python→Kotlin singleton pattern of `BleBridge`/`UsbSerialBridge`.
|
||||
|
||||
## Selection path (why nothing downstream changes)
|
||||
|
||||
Webcams on desktop are a `ScreenCapturePictureSource` (`stream_type="raw"`) bound to a capture
|
||||
template whose `engine_type="camera"` + a `display_index`. `live_stream_manager`
|
||||
`_create_screen_capture_live_stream` reads `engine_type` from the template and calls
|
||||
`EngineRegistry.create_stream(engine_type, display_index, config)`. Android adds
|
||||
`engine_type="android_camera"` — the **same path**. The frontend
|
||||
(`static/js/features/streams-capture-templates.ts`) is fully data-driven: the engine list,
|
||||
the `resolution` config dropdown (keyed by field name), and the camera picker
|
||||
(`/config/displays?engine_type=android_camera`, since `HAS_OWN_DISPLAYS=True`) all work with
|
||||
no frontend changes.
|
||||
|
||||
## Part A — Python (`core/capture_engines/android_camera_engine.py`)
|
||||
|
||||
Mirrors `mediaprojection_engine.py` (module-level `queue.Queue` + `push_frame` + `_last_frame`
|
||||
fallback + drop-oldest) and the desktop `CameraEngine` shape (cameras as displays,
|
||||
`resolution` config).
|
||||
|
||||
- `_camera_bridge()` — lazy, `is_android()`-guarded `from java import jclass;
|
||||
jclass("com.ledgrab.android.CameraBridge").INSTANCE`. **Never imported at module load** (this
|
||||
module imports on desktop CI). Mirrors `core/devices/android_ble_transport.py`.
|
||||
- `list_cameras()` → parses `CameraBridge.listCameras()` JSON into
|
||||
`[{"index","name","facing"}]`; `_enumerate_cameras()` caches it (30 s TTL).
|
||||
- `push_frame(rgb_bytes, w, h)` → `np.frombuffer(...uint8)` reshape **`(h, w, 3)`** (RGB, 3
|
||||
B/px — NOT the RGBA `(h,w,4)` of the screen engine) → `.copy()` → drop-oldest enqueue. A
|
||||
short/malformed buffer is dropped, never reshape-crashes.
|
||||
- `start_camera(index, w, h) -> bool` / `stop_camera(index)` → guarded bridge calls.
|
||||
- `AndroidCameraEngine`: `ENGINE_TYPE="android_camera"`, `ENGINE_PRIORITY=0` (never
|
||||
auto-selected over MediaProjection=100 — explicit `engine_type` only), `HAS_OWN_DISPLAYS=True`,
|
||||
`is_available()=is_android() and ≥1 enumerated camera`, `get_config_choices()` exposes
|
||||
`resolution` (same presets as desktop).
|
||||
- `AndroidCameraCaptureStream`: `initialize()` parses `resolution` → `start_camera(...)` (raises
|
||||
if it returns False), drains stale frames; `capture_frame()` pops queue / returns `_last_frame`;
|
||||
`cleanup()` → `stop_camera(...)`.
|
||||
|
||||
Registered in `capture_engines/__init__.py` behind a guarded import (mirrors the
|
||||
mediaprojection block).
|
||||
|
||||
## Part B — Android (`CameraBridge.kt`)
|
||||
|
||||
`object CameraBridge` (mirrors `BleBridge`):
|
||||
|
||||
- `init(context)` — from `LedGrabApp.onCreate` (context only, no camera opened).
|
||||
- `listCameras(): String` — JSON array from `CameraManager.cameraIdList` + `LENS_FACING`
|
||||
(front/back/external). No CAMERA permission needed.
|
||||
- `startCamera(index, width, height): Boolean` — checks CAMERA permission; resolves cameraId;
|
||||
picks the supported YUV size closest to the request (balanced default ≤1280×720 for "auto");
|
||||
opens device + capture session on a private `HandlerThread`, blocking until configured
|
||||
(`runBlocking { withTimeout { ... } }` over `suspendCancellableCoroutine`-wrapped Camera2
|
||||
callbacks); sets a repeating preview request. Returns false (no throw across JNI) on
|
||||
permission/range/configure failure. Closes any prior camera first.
|
||||
- `onImageAvailable` → paced (≈20 fps) → stride-aware **YUV_420_888→RGB** (BT.601 fixed-point,
|
||||
reused plane + RGB buffers) → push to the cached `android_camera_engine` module handle.
|
||||
- `stopCamera()` — stops repeating, closes session/device/reader, idempotent.
|
||||
|
||||
## Part C — Wiring + permission + manifest
|
||||
|
||||
- `LedGrabApp.kt` — `CameraBridge.init(this)` next to `BleBridge.init`.
|
||||
- `MainActivity.kt` — `ensureCameraPermission()` (mirror `ensureAudioPermission`): request
|
||||
`CAMERA` iff `hasSystemFeature(FEATURE_CAMERA_ANY)`; called from both `startCaptureService`
|
||||
(MediaProjection path) and `startRootCaptureService` (root path). Fire-and-forget.
|
||||
- `AndroidManifest.xml` — `<uses-permission CAMERA>` + `<uses-feature camera.any required=false>`
|
||||
+ `<uses-permission FOREGROUND_SERVICE_CAMERA>`, and `camera` added to the `CaptureService`
|
||||
`foregroundServiceType` union (`mediaProjection|specialUse|camera`).
|
||||
- `CaptureService.onStartCommand` — on API 34+, OR `FOREGROUND_SERVICE_TYPE_CAMERA` into the
|
||||
promotion type **only when CAMERA is already granted**. Unlike audio playback capture (which
|
||||
rides the MediaProjection token under the mediaProjection type), the camera has no such
|
||||
coupling, so without its own FGS type Android 14+ revokes camera access once the app is
|
||||
backgrounded. The conditional guard avoids a failed `startForeground` (which would kill the
|
||||
whole service) on a camera-less / not-yet-granted box. If CAMERA is granted later, the camera
|
||||
type takes effect on the next Start.
|
||||
- No `proguard-rules.pro` change — the blanket `-keep class com.ledgrab.android.** { *; }`
|
||||
already covers `CameraBridge`, and R8/minify is disabled.
|
||||
|
||||
## What does NOT change
|
||||
|
||||
- **Frontend / API** — data-driven engine list, config, and display picker.
|
||||
- **`build.gradle.kts` / Chaquopy pip block** — no new Python or Gradle packages.
|
||||
- **Processing pipeline** — `ScreenCaptureLiveStream`, filters, color-strip sources unchanged.
|
||||
|
||||
## Files
|
||||
|
||||
**Create**
|
||||
- `server/src/ledgrab/core/capture_engines/android_camera_engine.py`
|
||||
- `android/app/src/main/java/com/ledgrab/android/CameraBridge.kt`
|
||||
- `server/tests/core/test_android_camera_engine.py`
|
||||
|
||||
**Modify**
|
||||
- `server/src/ledgrab/core/capture_engines/__init__.py` — guarded import + registration.
|
||||
- `android/app/src/main/java/com/ledgrab/android/LedGrabApp.kt` — `CameraBridge.init`.
|
||||
- `android/app/src/main/java/com/ledgrab/android/MainActivity.kt` — `ensureCameraPermission`.
|
||||
- `android/app/src/main/AndroidManifest.xml` — `CAMERA` + `camera.any`.
|
||||
|
||||
## Tests (Python — desktop CI, no device)
|
||||
|
||||
`server/tests/core/test_android_camera_engine.py`: push→capture round-trips RGB `(h,w,3)`;
|
||||
drop-oldest when full; `_last_frame` fallback on empty; short-buffer never crashes;
|
||||
`initialize()` opens with parsed/auto resolution and raises on open-failure / off-Android;
|
||||
`cleanup()` closes once (idempotent); `is_available()` gating (android + cameras); display
|
||||
enumeration; priority 0 never beats MediaProjection; create-via-registry yields a pushed frame.
|
||||
|
||||
## Verification
|
||||
|
||||
1. **Python:** `py -3.13 -m pytest tests/core/test_android_camera_engine.py --no-cov -q`, then
|
||||
the full suite (1880 passed, 2 skipped; 15 new).
|
||||
2. **Lint:** `ruff check src/ tests/ --fix` — clean.
|
||||
3. **Android build:** `./gradlew :app:assembleDebug` — BUILD SUCCESSFUL.
|
||||
4. **On device (manual):** install APK → Start capture → grant CAMERA → create a capture
|
||||
template with engine `android_camera` + a camera display + a ScreenCapture source bound to
|
||||
a strip → confirm LEDs react to the camera feed and the camera indicator only lights while
|
||||
the source is active.
|
||||
|
||||
## Risks / notes
|
||||
|
||||
- **MVP scope:** webcam works **while LedGrab capture is running** (the Python server only runs
|
||||
inside `CaptureService`; there is no camera-only start path on Android).
|
||||
- **One camera at a time:** `startCamera` closes any previously-open camera first.
|
||||
- **`"auto"` resolution** picks a balanced output size (~720p), not the sensor max, to keep the
|
||||
per-frame YUV→RGB conversion cheap on low-end TV boxes.
|
||||
- **USB-UVC webcams** appear only if the device exposes them through Camera2 (`LENS_FACING_EXTERNAL`),
|
||||
which varies by box; an explicit UVC library would be a separate, larger effort.
|
||||
- **No frame-rotation correction** — sensor orientation is not applied (ambient color sampling
|
||||
is largely orientation-tolerant); could be added later.
|
||||
- **CAMERA denied** → the engine reports no usable camera and capture proceeds without it.
|
||||
Reference in New Issue
Block a user