feat: Android TV app embedding Python server via Chaquopy
Lint & Test / test (push) Successful in 2m10s

Adds a native Android TV application that runs the full LedGrab Python
server in-process via Chaquopy. Captures the TV box screen using the
MediaProjection API and exposes the existing web UI on the device's
local network — users configure via phone/tablet browser.

Android (new /android/ module):
- Kotlin shell: MainActivity, CaptureService (foreground service),
  ScreenCapture (MediaProjection + ImageReader), PythonBridge (Chaquopy).
- Polished Leanback-themed UI with QR code for easy web UI access.
- AGP 8.9 + Chaquopy 17 + Gradle 8.11 (avoids the AGP 8.7 thread-lock bug).
- Pre-built pydantic-core wheels for arm64-v8a, x86_64, x86 cross-compiled
  with maturin + Android NDK, linked against Chaquopy's libpython3.11.so.

Python server platform guards:
- New utils/platform.py with is_android()/is_windows()/is_linux() helpers.
- Guard every top-level import of desktop-only packages (mss, psutil,
  sounddevice, pyserial, PyAudioWPatch, etc.) with try/except ImportError.
- Android-incompatible calls gated with None-checks so the server runs on
  reduced capabilities on Android (no CPU/RAM metrics, no mss displays).
- utils/image_codec.py gains a Pillow fallback for resize + JPEG encode
  when cv2 is unavailable; all internal cv2.resize callers migrated.
- New android_entry.py start_server/stop_server invoked from Kotlin.
- get_displays API falls back to best available engine when mss fails.

New capture engines:
- MediaProjectionEngine: receives RGBA frames pushed from Kotlin through
  a thread-safe queue; caches last frame for static-screen previews.
- ScrcpyClientEngine: optional H.264 streaming via scrcpy-client library
  (priority 10, overrides the ADB-screencap engine when installed).

Frontend:
- Tab loaders previously required an apiKey; now correctly treat
  "auth disabled" as authenticated (Android has no auth by default).
- Re-trigger the active tab's loader after loadServerInfo resolves
  authRequired, since initTabs runs earlier.
- Add i18n keys for the demo / mediaprojection / scrcpy_client engines.

Docs:
- TODO.md: follow-ups for multi-ABI wheel rebuilds, CI pipeline, USB
  serial LED controllers, root-only capture, perf metrics abstraction.
- CLAUDE.md: Android dependency sync policy (pip --exclude doesn't exist).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-04-14 03:11:43 +03:00
parent a0b65e3fcb
commit 8574424fb7
56 changed files with 2443 additions and 126 deletions
+85 -22
View File
@@ -1,34 +1,97 @@
# Composite Nesting Support
# LedGrab TODO
## Phase 1: Store — Cycle & Depth Validation
## Android — Restore Multi-ABI Wheels
- [x] Add `get_transitive_dependencies()` to ColorStripStore
- [x] Add `validate_nesting()` with cycle detection + depth limit (MAX_DEPTH=4)
During emulator testing, we switched the build to **x86 only** (see `android/app/build.gradle.kts` `abiFilters`) to avoid having to keep the arm64-v8a / x86_64 pydantic-core wheels current. Before shipping, restore all three ABIs:
## Phase 2: API — Validation in Create/Update
- [ ] Rebuild `pydantic-core` wheels for all three ABIs with the current SOABI + libpython linking settings:
- `arm64-v8a` (real TV boxes — the primary target)
- `x86_64` (modern emulators)
- `x86` (legacy emulators)
- [ ] Verify wheels with `readelf -d`: SONAME must be `libpython3.11.so` in NEEDED
- [ ] Restore `abiFilters += listOf("arm64-v8a", "x86_64", "x86")` in `build.gradle.kts`
- [ ] Re-test on real ARM64 Android TV hardware
- [x] Call `validate_nesting()` in create handler for composite sources
- [x] Call `validate_nesting()` in update handler for composite sources
Build cache + scripts live in `android/build-scripts/` and `android/.build-cache/` (junction host + sysconfigdata for each ABI).
## Phase 3: Runtime — Depth Guard in Stream
## Android CI Pipeline
- [x] Add `depth` parameter to CompositeColorStripStream
- [x] Pass depth through ColorStripStreamManager.acquire()
- [x] Cap depth at runtime to prevent runaway nesting
Build the Android APK automatically on push/tag.
## Phase 4: Frontend — Allow Composites in Layer Dropdown
- [ ] Generate Gradle wrapper (`gradlew`) and commit it
- [ ] Create CI workflow (`.gitea/workflows/build-android.yaml` or `.github/workflows/`)
- JDK 17 + Android SDK + NDK setup
- Python 3.11 for Chaquopy build
- Recreate the directory junction (`ln -s` on Linux, `mklink /J` on Windows)
- `./gradlew assembleRelease`
- Upload APK as artifact
- [ ] Commit pre-built pydantic-core wheels to `android/wheels/` (arm64, x86, x86_64)
- [ ] APK signing for release builds (keystore setup)
- [ ] Consider: publish APK to GitHub/Gitea releases on tag push
- [x] Remove `source_type !== 'composite'` filter (keep self-exclusion)
- [x] Update docstring in composite_stream.py
## Android Root Capture (No Permission Dialog, No System Indicator)
## Phase 5: Tests
MediaProjection shows a mandatory system overlay/indicator while capturing — unavoidable on stock Android. Many cheap Android TV boxes ship pre-rooted, so an alternative root-only path would give much better UX.
- [x] Cycle detection tests (A→B→A, self-reference)
- [x] Depth limit tests (chain exceeding MAX_DEPTH)
- [x] Valid nesting tests (A→B both composite, no cycle)
- [ ] Detect root at runtime: check for `su` binary, `Superuser.apk`, etc.
- [ ] Implement `SurfaceControlCaptureEngine` (new capture engine) using hidden `SurfaceControl.screenshot()` API via reflection
- No permission dialog
- No system capture indicator
- Direct bitmap output (no encoder/decoder roundtrip)
- [ ] Engine priority: higher than MediaProjection when root detected
- [ ] Fallback chain: `SurfaceControl` (root) → `MediaProjection` (stock) → `adb screencap` (last resort)
- [ ] Handle Android version differences in `SurfaceControl` API surface (renamed/moved across API 29, 30, 33)
- [ ] Alternative: shell out to `screenrecord --output-format=h264 -` as root (same H.264 decode as scrcpy_client_engine, but local instead of remote ADB)
## Phase 6: Lint & Build
Known projects using this approach for reference: scrcpy-hidden-api, shizuku, commercial scrcpy-derived apps.
- [x] Ruff check passes
- [x] TypeScript build passes
- [x] All existing tests pass (715/715)
## Android USB Serial Support
Drive USB LED controllers (APA102, WS2812) connected directly to the Android TV box via USB-to-serial adapters.
- [ ] Add [usb-serial-for-android](https://github.com/mik3y/usb-serial-for-android) dependency to `android/app/build.gradle.kts`
- [ ] Create Kotlin `UsbSerialBridge` class that:
- Enumerates USB serial devices via Android USB Host API
- Requests user permission for USB device access
- Opens a serial connection (baud rate configurable)
- Exposes a write method callable from Python via Chaquopy
- [ ] Create Python `AndroidSerialProvider` in `server/src/ledgrab/core/devices/` that:
- Replaces `pyserial` on Android (which can't access USB ports)
- Calls `UsbSerialBridge` via Chaquopy to send LED data
- Registers as an alternative serial transport when `is_android()` is True
- [ ] Add USB device permission dialog to `MainActivity` (auto-triggered on device connect)
- [ ] Test with common USB-to-serial chips: CH340, CP2102, FTDI
- [ ] Document supported USB LED controllers in README
## Android App — Known Issues
Issues discovered during first end-to-end test on emulator (2026-04-14):
- [ ] **Stop/Start capture produces no frames on second try.** First capture works, but after tapping Stop and Start again, no frames reach the Python pipeline. Likely causes:
- Global MediaProjection engine state (`_active`, `_frame_queue`) not reset on service restart
- ScreenCapture listener not properly detached from ImageReader on stop
- Python uvicorn port reuse issue when server restarts
- [ ] **Web UI tabs show persistent spinners** on Dashboard, Automation, Sources, Integrations, Graph tabs (Targets tab works). All API calls return 200 OK, no console errors. Probably waiting on a specific WebSocket event that never fires when no output targets are configured/active.
- [ ] **Dead keyboard/IME handling** — password inputs lack autocomplete attributes (minor accessibility warning in browser console)
## Performance Metrics Abstraction
The codebase has direct `psutil.*` calls scattered across `api/routes/system.py` and `core/processing/metrics_history.py`, with ad-hoc `if psutil is not None` guards sprinkled in to support Android. This couples Android platform handling to every call site.
- [ ] Refactor: introduce `MetricsProvider` protocol in `utils/metrics.py` with methods like `cpu_percent()`, `memory_info()`, `process_info()`
- [ ] Implement `PsutilMetricsProvider` (desktop) and `NullMetricsProvider` (fallback when psutil missing)
- [ ] Later: `AndroidMetricsProvider` reading from `/proc` (see section below)
- [ ] Replace all direct `psutil.*` calls with the provider; only one factory location knows about psutil availability
## Android Performance Metrics
Currently `psutil` (used for CPU/RAM monitoring in the web UI) is not available on Android via Chaquopy. Metrics calls are guarded with `if psutil is not None` so they return no data on Android.
- [ ] Implement Android-native metrics collection:
- CPU usage via `/proc/self/stat` + `/proc/stat` parsing (no psutil needed)
- RAM usage via `/proc/meminfo` or `ActivityManager.getMemoryInfo()` through Chaquopy bridge
- App-specific memory via `Debug.getMemoryInfo()` (Kotlin → Python)
- [ ] Create `AndroidMetricsProvider` in `server/src/ledgrab/utils/` that implements the same interface as the psutil-based provider
- [ ] Wire into existing metrics endpoints (`/api/v1/system/metrics`) with platform detection
- [ ] Consider: device battery/temperature readings for TV boxes (some have thermal throttling)
- [ ] Optional: GPU usage via `/sys/class/kgsl/kgsl-3d0/gpubusy` on Adreno, Mali-specific paths for Mali GPUs