feat(android): foreground-app automation condition
Make the existing Application automation rule (foreground app -> activate scene) work on the Android-TV build. A Kotlin ForegroundAppBridge reads the foreground app via UsageStatsManager and lists launchable apps via LauncherApps; PlatformDetector bridges it in (ahead of the Windows-only ctypes guard) so the existing AutomationEngine / ApplicationRule / storage / deactivation modes are unchanged. New /system/installed-apps + /system/info endpoints feed an app picker that stores package names (vs process names on desktop); on Android the editor hides the match-type selector since the foreground app is the only obtainable signal. PACKAGE_USAGE_STATS is granted via an on-device button + a web-UI banner (no blanket prompt at capture start); detection degrades gracefully until granted. Zero new Python/Gradle deps (UsageStatsManager + LauncherApps are in-platform; matching only string-compares the package name, so no QUERY_ALL_PACKAGES). assembleDebug + 1897 pytest + ruff + tsc + npm build all green; independent final review (0 blockers) + security review (no critical issues).
This commit is contained in:
@@ -0,0 +1,94 @@
|
||||
# Android foreground-app automation condition — implementation notes
|
||||
|
||||
> Status: implemented on `feature/android-foreground-app-automation`. Last updated 2026-06-02.
|
||||
|
||||
## What & why
|
||||
|
||||
The desktop build has an **Application** automation rule (`ApplicationRule`): activate a scene
|
||||
when given apps are running / foreground / fullscreen. It was already wired end-to-end on
|
||||
Android (engine, storage, API, editor) but **silently never fired**, because the two
|
||||
Windows-only ctypes paths return empty off-Windows:
|
||||
|
||||
1. **Detection** — `PlatformDetector._get_topmost_process_sync()` (and the running/fullscreen
|
||||
variants) returned `(None, False)` / `set()` on Android.
|
||||
2. **The app picker** — populated from `GET /api/v1/system/processes` →
|
||||
`get_running_processes()`, also empty on Android, so users couldn't even choose an app.
|
||||
|
||||
This feature fills both holes using in-platform Android APIs and the established Kotlin↔Python
|
||||
bridge pattern. **Zero new Python or Gradle dependencies.**
|
||||
|
||||
## Design decision: one implicit "foreground" mode on Android
|
||||
|
||||
Android exposes exactly one obtainable signal — the **current foreground app package**. The
|
||||
desktop rule's four match types (`running` / `topmost` / `fullscreen` / `topmost_fullscreen`)
|
||||
are either unobtainable (`running` — `getRunningTasks` is restricted) or identical (a foreground
|
||||
TV app effectively *is* fullscreen). So on Android:
|
||||
|
||||
- The editor **hides the match-type selector** and the collector forces `match_type="topmost"`.
|
||||
- `_get_topmost_process_sync()` returns `(package, True)`; the running/fullscreen detectors
|
||||
return the foreground app as a best-effort single-element set so legacy rules still behave.
|
||||
|
||||
This avoided touching the existing plain `<select>` (forbidden for new UI) and removed a
|
||||
misleading 4-way choice — a simplification surfaced by the pre-implementation plan review.
|
||||
|
||||
## Detection — `ForegroundAppBridge` (Kotlin) ↔ `platform_detector.py`
|
||||
|
||||
`android/app/src/main/java/com/ledgrab/android/ForegroundAppBridge.kt` (an `object` singleton,
|
||||
mirroring `CameraBridge`, context bound in `LedGrabApp.onCreate`):
|
||||
|
||||
- `getForegroundPackage()` — `UsageStatsManager.queryEvents(now - 10s, now)`, returns the package
|
||||
of the most recent `MOVE_TO_FOREGROUND` / `ACTIVITY_RESUMED` event (the two constants share a
|
||||
value; the ~10s window absorbs event lag against the ~1s automation tick). `queryEvents` is the
|
||||
right call — `queryUsageStats` gives aggregate durations, not "current app".
|
||||
- `hasUsageAccess()` — `AppOpsManager` `OPSTR_GET_USAGE_STATS` check (`unsafeCheckOpNoThrow` on
|
||||
API 29+, `checkOpNoThrow` below).
|
||||
- `listLaunchableApps()` — `LauncherApps.getActivityList` → JSON `[{package,label}]` for the
|
||||
picker. The sanctioned launchable-app API; **no `QUERY_ALL_PACKAGES`**.
|
||||
|
||||
`server/src/ledgrab/core/automations/platform_detector.py`:
|
||||
|
||||
- Module-level guarded wrappers `get_foreground_package()` / `has_usage_access()` /
|
||||
`list_installed_apps()` resolve `jclass("com.ledgrab.android.ForegroundAppBridge").INSTANCE`
|
||||
lazily (never at import — the module loads on desktop CI). These are the **test monkeypatch
|
||||
surface**, mirroring `android_camera_engine`.
|
||||
- The `is_android()` branch is placed **ahead of** the import-time `if not _IS_WINDOWS:`
|
||||
early-return in each detector — the critical fix from plan review (a naive wiring would no-op
|
||||
behind the Windows guard yet still pass tests). The Windows ctypes path is unchanged
|
||||
(regression-tested).
|
||||
- A one-time `logger.warning` fires when Usage Access is missing.
|
||||
|
||||
## App picker — `/system/installed-apps` + platform signal
|
||||
|
||||
- `GET /api/v1/system/installed-apps` → `{apps:[{package,label}], count}` (empty off-Android).
|
||||
- `GET /api/v1/system/info` → `{is_android, app_match_kind, usage_access_granted}` — the editor
|
||||
reads it to pick the app source + matching semantics and to show the Usage-Access banner.
|
||||
- Frontend: the command-palette picker (`core/process-picker.ts`) gained label→value support; a
|
||||
new `AppPalette` shows the human label and inserts the package name. On Android the app-rule
|
||||
editor uses it (`attachAppPicker`) instead of the process picker, plus a package-name hint and
|
||||
the Usage-Access banner.
|
||||
|
||||
## Value semantics (no migration)
|
||||
|
||||
`ApplicationRule.apps` are **package names** on Android (`com.netflix.mediaclient`) vs **process
|
||||
names** on Windows (`chrome.exe`). Same field, same matching code — **no storage migration** —
|
||||
but rules are **not portable across platforms**. Documented in the model/schema docstrings and a
|
||||
user-facing editor hint.
|
||||
|
||||
## Permission UX
|
||||
|
||||
`PACKAGE_USAGE_STATS` is a special access (can't be granted at runtime):
|
||||
|
||||
- Manifest declares it with `tools:ignore="ProtectedPermissions"`.
|
||||
- MainActivity shows a passive **"Grant usage access"** button (opens
|
||||
`ACTION_USAGE_ACCESS_SETTINGS`, with a generic-Settings fallback) only while access is missing.
|
||||
**No blanket prompt at capture start** — most users have no foreground-app rule.
|
||||
- The web-UI rule editor shows a banner when an Android Application rule lacks access.
|
||||
|
||||
## Limitations
|
||||
|
||||
- Foreground-app only; no full window-title or arbitrary process enumeration on Android.
|
||||
- Detection rides the existing ~1s automation poll; `queryEvents` can lag a few seconds.
|
||||
- Rules authored on desktop don't match on Android and vice-versa (package vs process names).
|
||||
- The on-device "Grant usage access" button currently shows whenever access is missing (not
|
||||
gated on whether an Android Application rule exists), to avoid Activity↔server coupling; the
|
||||
web-UI banner provides the contextual guidance.
|
||||
@@ -49,7 +49,7 @@ Python receiver engine mirroring that pattern.**
|
||||
| Webcam capture | OpenCV | ✅ Camera2 + on-demand bridge (`AndroidCameraEngine`) | No (implemented) |
|
||||
| GPU monitoring | NVML | ❌ no NVIDIA GPU | Marginal |
|
||||
| Capture from *another* Android phone | scrcpy/ADB | ❌ | Skip (redundant) |
|
||||
| Automation: window/process conditions | Windows ctypes | ❌ sandboxed | Partial |
|
||||
| Automation: foreground-app condition | Windows ctypes (running/topmost/fullscreen) | ✅ foreground app via UsageStatsManager (`ForegroundAppBridge`) | No (implemented) |
|
||||
| Monitor names / multi-display | WMI / generic | Single built-in display | Low value |
|
||||
|
||||
---
|
||||
@@ -127,15 +127,30 @@ Python receiver engine mirroring that pattern.**
|
||||
- CPU/RAM/battery/thermal are **already** covered by `AndroidMetricsProvider`. A best-effort
|
||||
GPU-load reader could be added to that provider, but reliability is poor and value is low.
|
||||
|
||||
### 🪟 Automation: window/process conditions — **PARTIAL**
|
||||
### 🪟 Automation: foreground-app condition — **IMPLEMENTED** ✅ (shipped)
|
||||
|
||||
- Android forbids full window/process enumeration (`getRunningTasks` restricted since API 21+).
|
||||
- **Obtainable:** the *current foreground app package* via `UsageStatsManager` (needs the
|
||||
`PACKAGE_USAGE_STATS` special access) or an `AccessibilityService`.
|
||||
- So "when <app> is in the foreground → scene X" is feasible (mirrors
|
||||
`automations/platform_detector.py`, which currently returns empty off-Windows); full
|
||||
window-title matching is **not**. **Effort:** moderate. **Value:** moderate (per-app scenes
|
||||
on a TV box).
|
||||
- Android forbids full window/process enumeration (`getRunningTasks` restricted since API 21+),
|
||||
but the *current foreground app package* is obtainable via `UsageStatsManager` (needs the
|
||||
`PACKAGE_USAGE_STATS` special access).
|
||||
- **Path:** a Kotlin `ForegroundAppBridge` (UsageStatsManager `queryEvents` over a ~10s trailing
|
||||
window + `LauncherApps` for the picker + an `AppOpsManager` access check) bridged into
|
||||
`automations/platform_detector.py` via the guarded-`jclass` pattern, ahead of the Windows-only
|
||||
ctypes path. The existing `ApplicationRule` / `AutomationEngine` / storage / deactivation modes
|
||||
are unchanged — only the detection + the picker's data source were filled in. **No new Python
|
||||
or Gradle deps** (UsageStatsManager + LauncherApps are in-platform; matching only string-compares
|
||||
the package name, so no `QUERY_ALL_PACKAGES` / package visibility is needed).
|
||||
- **UI:** the automation editor's app picker lists launchable apps by human label (storing the
|
||||
package name) via a new `GET /api/v1/system/installed-apps`; on Android the match-type selector
|
||||
is hidden and `match_type` is forced to `topmost` (the only obtainable signal), with a
|
||||
cross-platform value caveat — `apps` are **package names** on Android (`com.netflix.mediaclient`)
|
||||
vs **process names** on Windows (`chrome.exe`), so rules are not portable across platforms.
|
||||
- **Permission:** `PACKAGE_USAGE_STATS` is a special access (Settings deep-link via
|
||||
`ACTION_USAGE_ACCESS_SETTINGS`); the device shows a "Grant usage access" button when missing,
|
||||
and the web-UI rule editor shows a banner (driven by `/system/info`'s `usage_access_granted`).
|
||||
No blanket prompt at capture start. Detection degrades gracefully (rule never matches, warned
|
||||
once) until access is granted. **Effort:** moderate. **Value:** moderate (per-app scenes on a
|
||||
TV box). Full window-title matching remains out of scope (Android does not expose it).
|
||||
- 📄 **See `android-foreground-app-automation-plan.md`** for the full implementation notes.
|
||||
|
||||
### 📱 Capture from *another* Android phone (scrcpy/ADB) — **SKIP**
|
||||
|
||||
@@ -157,17 +172,17 @@ Python receiver engine mirroring that pattern.**
|
||||
| 1 | Notification capture | Moderate | High | None | **✅ Implemented** |
|
||||
| 2 | Audio capture | Moderate | High | None | **✅ Implemented** |
|
||||
| 4 | Webcam capture (Camera2) | Moderate | Low | None | **✅ Implemented** |
|
||||
| 3 | Automation: foreground-app condition | Moderate | Moderate | None | Idea (only remaining) |
|
||||
| 3 | Automation: foreground-app condition | Moderate | Moderate | None | **✅ Implemented** |
|
||||
| — | GPU load (vendor sysfs) | Low–Med | Low | None | Not recommended |
|
||||
| — | Capture from another phone | — | — | — | Won't do |
|
||||
| — | Multi-display / monitor names | Low | Low | None | Not recommended |
|
||||
|
||||
**Status:** notifications, audio, **and webcam** are all shipped — each reuses existing
|
||||
infrastructure (bridge pattern, the MediaProjection consent token / process-global
|
||||
`Python.getInstance()`, the capture/audio/notification pipelines) and adds **zero** Python
|
||||
dependencies, so none risks the Chaquopy `--no-deps` build constraint documented in
|
||||
`CLAUDE.md`. The only remaining idea is the **foreground-app automation condition** (moderate
|
||||
value); GPU load, another-phone capture, and multi-display remain not-recommended / won't-do.
|
||||
**Status:** notifications, audio, webcam, **and the foreground-app automation condition** are all
|
||||
shipped — each reuses existing infrastructure (the Kotlin↔Python bridge pattern, the
|
||||
MediaProjection consent token / process-global `Python.getInstance()`, the
|
||||
capture/audio/notification/automation pipelines) and adds **zero** Python dependencies, so none
|
||||
risks the Chaquopy `--no-deps` build constraint documented in `CLAUDE.md`. No prioritized ideas
|
||||
remain; GPU load, another-phone capture, and multi-display remain not-recommended / won't-do.
|
||||
|
||||
## Cross-cutting notes
|
||||
|
||||
|
||||
Reference in New Issue
Block a user