alexei.dolgolyov
1c1bbe2551
Make the existing Application automation rule (foreground app -> activate scene) work on the Android-TV build. A Kotlin ForegroundAppBridge reads the foreground app via UsageStatsManager and lists launchable apps via LauncherApps; PlatformDetector bridges it in (ahead of the Windows-only ctypes guard) so the existing AutomationEngine / ApplicationRule / storage / deactivation modes are unchanged. New /system/installed-apps + /system/info endpoints feed an app picker that stores package names (vs process names on desktop); on Android the editor hides the match-type selector since the foreground app is the only obtainable signal. PACKAGE_USAGE_STATS is granted via an on-device button + a web-UI banner (no blanket prompt at capture start); detection degrades gracefully until granted. Zero new Python/Gradle deps (UsageStatsManager + LauncherApps are in-platform; matching only string-compares the package name, so no QUERY_ALL_PACKAGES). assembleDebug + 1897 pytest + ruff + tsc + npm build all green; independent final review (0 blockers) + security review (no critical issues).
95 lines
5.4 KiB
Markdown
95 lines
5.4 KiB
Markdown
# Android foreground-app automation condition — implementation notes
|
|
|
|
> Status: implemented on `feature/android-foreground-app-automation`. Last updated 2026-06-02.
|
|
|
|
## What & why
|
|
|
|
The desktop build has an **Application** automation rule (`ApplicationRule`): activate a scene
|
|
when given apps are running / foreground / fullscreen. It was already wired end-to-end on
|
|
Android (engine, storage, API, editor) but **silently never fired**, because the two
|
|
Windows-only ctypes paths return empty off-Windows:
|
|
|
|
1. **Detection** — `PlatformDetector._get_topmost_process_sync()` (and the running/fullscreen
|
|
variants) returned `(None, False)` / `set()` on Android.
|
|
2. **The app picker** — populated from `GET /api/v1/system/processes` →
|
|
`get_running_processes()`, also empty on Android, so users couldn't even choose an app.
|
|
|
|
This feature fills both holes using in-platform Android APIs and the established Kotlin↔Python
|
|
bridge pattern. **Zero new Python or Gradle dependencies.**
|
|
|
|
## Design decision: one implicit "foreground" mode on Android
|
|
|
|
Android exposes exactly one obtainable signal — the **current foreground app package**. The
|
|
desktop rule's four match types (`running` / `topmost` / `fullscreen` / `topmost_fullscreen`)
|
|
are either unobtainable (`running` — `getRunningTasks` is restricted) or identical (a foreground
|
|
TV app effectively *is* fullscreen). So on Android:
|
|
|
|
- The editor **hides the match-type selector** and the collector forces `match_type="topmost"`.
|
|
- `_get_topmost_process_sync()` returns `(package, True)`; the running/fullscreen detectors
|
|
return the foreground app as a best-effort single-element set so legacy rules still behave.
|
|
|
|
This avoided touching the existing plain `<select>` (forbidden for new UI) and removed a
|
|
misleading 4-way choice — a simplification surfaced by the pre-implementation plan review.
|
|
|
|
## Detection — `ForegroundAppBridge` (Kotlin) ↔ `platform_detector.py`
|
|
|
|
`android/app/src/main/java/com/ledgrab/android/ForegroundAppBridge.kt` (an `object` singleton,
|
|
mirroring `CameraBridge`, context bound in `LedGrabApp.onCreate`):
|
|
|
|
- `getForegroundPackage()` — `UsageStatsManager.queryEvents(now - 10s, now)`, returns the package
|
|
of the most recent `MOVE_TO_FOREGROUND` / `ACTIVITY_RESUMED` event (the two constants share a
|
|
value; the ~10s window absorbs event lag against the ~1s automation tick). `queryEvents` is the
|
|
right call — `queryUsageStats` gives aggregate durations, not "current app".
|
|
- `hasUsageAccess()` — `AppOpsManager` `OPSTR_GET_USAGE_STATS` check (`unsafeCheckOpNoThrow` on
|
|
API 29+, `checkOpNoThrow` below).
|
|
- `listLaunchableApps()` — `LauncherApps.getActivityList` → JSON `[{package,label}]` for the
|
|
picker. The sanctioned launchable-app API; **no `QUERY_ALL_PACKAGES`**.
|
|
|
|
`server/src/ledgrab/core/automations/platform_detector.py`:
|
|
|
|
- Module-level guarded wrappers `get_foreground_package()` / `has_usage_access()` /
|
|
`list_installed_apps()` resolve `jclass("com.ledgrab.android.ForegroundAppBridge").INSTANCE`
|
|
lazily (never at import — the module loads on desktop CI). These are the **test monkeypatch
|
|
surface**, mirroring `android_camera_engine`.
|
|
- The `is_android()` branch is placed **ahead of** the import-time `if not _IS_WINDOWS:`
|
|
early-return in each detector — the critical fix from plan review (a naive wiring would no-op
|
|
behind the Windows guard yet still pass tests). The Windows ctypes path is unchanged
|
|
(regression-tested).
|
|
- A one-time `logger.warning` fires when Usage Access is missing.
|
|
|
|
## App picker — `/system/installed-apps` + platform signal
|
|
|
|
- `GET /api/v1/system/installed-apps` → `{apps:[{package,label}], count}` (empty off-Android).
|
|
- `GET /api/v1/system/info` → `{is_android, app_match_kind, usage_access_granted}` — the editor
|
|
reads it to pick the app source + matching semantics and to show the Usage-Access banner.
|
|
- Frontend: the command-palette picker (`core/process-picker.ts`) gained label→value support; a
|
|
new `AppPalette` shows the human label and inserts the package name. On Android the app-rule
|
|
editor uses it (`attachAppPicker`) instead of the process picker, plus a package-name hint and
|
|
the Usage-Access banner.
|
|
|
|
## Value semantics (no migration)
|
|
|
|
`ApplicationRule.apps` are **package names** on Android (`com.netflix.mediaclient`) vs **process
|
|
names** on Windows (`chrome.exe`). Same field, same matching code — **no storage migration** —
|
|
but rules are **not portable across platforms**. Documented in the model/schema docstrings and a
|
|
user-facing editor hint.
|
|
|
|
## Permission UX
|
|
|
|
`PACKAGE_USAGE_STATS` is a special access (can't be granted at runtime):
|
|
|
|
- Manifest declares it with `tools:ignore="ProtectedPermissions"`.
|
|
- MainActivity shows a passive **"Grant usage access"** button (opens
|
|
`ACTION_USAGE_ACCESS_SETTINGS`, with a generic-Settings fallback) only while access is missing.
|
|
**No blanket prompt at capture start** — most users have no foreground-app rule.
|
|
- The web-UI rule editor shows a banner when an Android Application rule lacks access.
|
|
|
|
## Limitations
|
|
|
|
- Foreground-app only; no full window-title or arbitrary process enumeration on Android.
|
|
- Detection rides the existing ~1s automation poll; `queryEvents` can lag a few seconds.
|
|
- Rules authored on desktop don't match on Android and vice-versa (package vs process names).
|
|
- The on-device "Grant usage access" button currently shows whenever access is missing (not
|
|
gated on whether an Android Application rule exists), to avoid Activity↔server coupling; the
|
|
web-UI banner provides the contextual guidance.
|