ledgrab/ANDROID-REVIEW/android-webcam-capture-plan.md

Plan: Android on-device webcam capture

Status: implemented on branch feature/android-webcam-capture. Last updated 2026-06-02.

Context

LedGrab captures webcams on desktop through OpenCV (cv2.VideoCapture) in server/src/ledgrab/core/capture_engines/camera_engine.py. On the experimental Android-TV build, opencv-python-headless has no Chaquopy cp311 wheel, so the camera engine never loads and cameras are unusable on-device.

Android doesn't need OpenCV to capture a camera: the platform exposes Camera2 (android.hardware.camera2), and the codebase already has the bridge shape to plug a Kotlin capture source into a push-based Python engine. This feature adds an on-device camera engine so a USB/integrated camera can drive ambient lighting, at parity with how the desktop OpenCV camera engine feeds the pipeline.

The design mirrors the working screen-capture bridge (mediaprojection_engine.py ↔ ScreenCapture.kt) and the just-shipped audio engine (android_audio_engine.py ↔ AudioCapture.kt). No new Python dependencies (numpy already bundled) and no new Gradle dependencies (Camera2 is in-platform) → no Chaquopy / build.gradle.kts changes.

Approach

A new push-based capture engine registered in the existing EngineRegistry, plus a Kotlin CameraBridge that opens the camera on demand:

[capture source acquired] → AndroidCameraCaptureStream.initialize()
   → android_camera_engine.start_camera(index, w, h)   [guarded jclass]
   → CameraBridge.startCamera(index, w, h)              [Camera2 open + session]
   → onImageAvailable → YUV_420_888→RGB (stride-aware) → push_frame(rgbBytes, w, h)
   → android_camera_engine [module-level queue] → AndroidCameraCaptureStream.capture_frame()
   → ScreenCaptureLiveStream → processing pipeline      [unchanged]

[capture source released] → AndroidCameraCaptureStream.cleanup()
   → android_camera_engine.stop_camera() → CameraBridge.stopCamera()  [releases the camera]

The camera is only open while a camera source is active — the camera-in-use indicator and battery cost are bounded to actual use, unlike always-on screen/audio capture. This on-demand control reuses the synchronous Python→Kotlin singleton pattern of BleBridge/UsbSerialBridge.

Selection path (why nothing downstream changes)

Webcams on desktop are a ScreenCapturePictureSource (stream_type="raw") bound to a capture template whose engine_type="camera" + a display_index. live_stream_manager _create_screen_capture_live_stream reads engine_type from the template and calls EngineRegistry.create_stream(engine_type, display_index, config). Android adds engine_type="android_camera" — the same path. The frontend (static/js/features/streams-capture-templates.ts) is fully data-driven: the engine list, the resolution config dropdown (keyed by field name), and the camera picker (/config/displays?engine_type=android_camera, since HAS_OWN_DISPLAYS=True) all work with no frontend changes.

Part A — Python (core/capture_engines/android_camera_engine.py)

Mirrors mediaprojection_engine.py (module-level queue.Queue + push_frame + _last_frame fallback + drop-oldest) and the desktop CameraEngine shape (cameras as displays, resolution config).

• _camera_bridge() — lazy, is_android()-guarded from java import jclass; jclass("com.ledgrab.android.CameraBridge").INSTANCE. Never imported at module load (this module imports on desktop CI). Mirrors core/devices/android_ble_transport.py.
• list_cameras() → parses CameraBridge.listCameras() JSON into [{"index","name","facing"}]; _enumerate_cameras() caches it (30 s TTL).
• push_frame(rgb_bytes, w, h) → np.frombuffer(...uint8) reshape (h, w, 3) (RGB, 3 B/px — NOT the RGBA (h,w,4) of the screen engine) → .copy() → drop-oldest enqueue. A short/malformed buffer is dropped, never reshape-crashes.
• start_camera(index, w, h) -> bool / stop_camera(index) → guarded bridge calls.
• AndroidCameraEngine: ENGINE_TYPE="android_camera", ENGINE_PRIORITY=0 (never auto-selected over MediaProjection=100 — explicit engine_type only), HAS_OWN_DISPLAYS=True, is_available()=is_android() and ≥1 enumerated camera, get_config_choices() exposes resolution (same presets as desktop).
• AndroidCameraCaptureStream: initialize() parses resolution → start_camera(...) (raises if it returns False), drains stale frames; capture_frame() pops queue / returns _last_frame; cleanup() → stop_camera(...).

Registered in capture_engines/__init__.py behind a guarded import (mirrors the mediaprojection block).

Part B — Android (CameraBridge.kt)

object CameraBridge (mirrors BleBridge):

• init(context) — from LedGrabApp.onCreate (context only, no camera opened).
• listCameras(): String — JSON array from CameraManager.cameraIdList + LENS_FACING (front/back/external). No CAMERA permission needed.
• startCamera(index, width, height): Boolean — checks CAMERA permission; resolves cameraId; picks the supported YUV size closest to the request (balanced default ≤1280×720 for "auto"); opens device + capture session on a private HandlerThread, blocking until configured (runBlocking { withTimeout { ... } } over suspendCancellableCoroutine-wrapped Camera2 callbacks); sets a repeating preview request. Returns false (no throw across JNI) on permission/range/configure failure. Closes any prior camera first.
• onImageAvailable → paced (≈20 fps) → stride-aware YUV_420_888→RGB (BT.601 fixed-point, reused plane + RGB buffers) → push to the cached android_camera_engine module handle.
• stopCamera() — stops repeating, closes session/device/reader, idempotent.

Part C — Wiring + permission + manifest

• LedGrabApp.kt — CameraBridge.init(this) next to BleBridge.init.
• MainActivity.kt — ensureCameraPermission() (mirror ensureAudioPermission): request CAMERA iff hasSystemFeature(FEATURE_CAMERA_ANY); called from both startCaptureService (MediaProjection path) and startRootCaptureService (root path). Fire-and-forget.
• AndroidManifest.xml — <uses-permission CAMERA> + <uses-feature camera.any required=false>
  • <uses-permission FOREGROUND_SERVICE_CAMERA>, and camera added to the CaptureService foregroundServiceType union (mediaProjection|specialUse|camera).
• CaptureService.onStartCommand — on API 34+, OR FOREGROUND_SERVICE_TYPE_CAMERA into the promotion type only when CAMERA is already granted. Unlike audio playback capture (which rides the MediaProjection token under the mediaProjection type), the camera has no such coupling, so without its own FGS type Android 14+ revokes camera access once the app is backgrounded. The conditional guard avoids a failed startForeground (which would kill the whole service) on a camera-less / not-yet-granted box. If CAMERA is granted later, the camera type takes effect on the next Start.
• No proguard-rules.pro change — the blanket -keep class com.ledgrab.android.** { *; } already covers CameraBridge, and R8/minify is disabled.

What does NOT change

• Frontend / API — data-driven engine list, config, and display picker.
• build.gradle.kts / Chaquopy pip block — no new Python or Gradle packages.
• Processing pipeline — ScreenCaptureLiveStream, filters, color-strip sources unchanged.

Files

Create

• server/src/ledgrab/core/capture_engines/android_camera_engine.py
• android/app/src/main/java/com/ledgrab/android/CameraBridge.kt
• server/tests/core/test_android_camera_engine.py

Modify

• server/src/ledgrab/core/capture_engines/__init__.py — guarded import + registration.
• android/app/src/main/java/com/ledgrab/android/LedGrabApp.kt — CameraBridge.init.
• android/app/src/main/java/com/ledgrab/android/MainActivity.kt — ensureCameraPermission.
• android/app/src/main/AndroidManifest.xml — CAMERA + camera.any.

Tests (Python — desktop CI, no device)

server/tests/core/test_android_camera_engine.py: push→capture round-trips RGB (h,w,3); drop-oldest when full; _last_frame fallback on empty; short-buffer never crashes; initialize() opens with parsed/auto resolution and raises on open-failure / off-Android; cleanup() closes once (idempotent); is_available() gating (android + cameras); display enumeration; priority 0 never beats MediaProjection; create-via-registry yields a pushed frame.

Verification

1. Python: py -3.13 -m pytest tests/core/test_android_camera_engine.py --no-cov -q, then the full suite (1880 passed, 2 skipped; 15 new).
2. Lint: ruff check src/ tests/ --fix — clean.
3. Android build: ./gradlew :app:assembleDebug — BUILD SUCCESSFUL.
4. On device (manual): install APK → Start capture → grant CAMERA → create a capture template with engine android_camera + a camera display + a ScreenCapture source bound to a strip → confirm LEDs react to the camera feed and the camera indicator only lights while the source is active.

Risks / notes

• MVP scope: webcam works while LedGrab capture is running (the Python server only runs inside CaptureService; there is no camera-only start path on Android).
• One camera at a time: startCamera closes any previously-open camera first.
• "auto" resolution picks a balanced output size (~720p), not the sensor max, to keep the per-frame YUV→RGB conversion cheap on low-end TV boxes.
• USB-UVC webcams appear only if the device exposes them through Camera2 (LENS_FACING_EXTERNAL), which varies by box; an explicit UVC library would be a separate, larger effort.
• No frame-rotation correction — sensor orientation is not applied (ambient color sampling is largely orientation-tolerant); could be added later.
• CAMERA denied → the engine reports no usable camera and capture proceeds without it.