- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the problem that traditional mobile agents mainly perform single or short-term operations and struggle with complex tasks that require long-term persistence, cross-app coordination, and adjustment in response to changes in device state and external events. The authors propose an OS-native agent framework for mobile operating systems that integrates task management with the OS environment, enabling the agent to maintain persistent workflows and to perform task scheduling, state tracking, and execution adjustment according to environmental changes. They also construct the DynamicWorld benchmark to evaluate agents' continuous task execution capability in dynamic mobile environments. This work extends the focus of mobile agent research from single UI operations to persistent workflow management in dynamic environments, provides a corresponding system framework and evaluation benchmark, and lays a research foundation for improving the adaptability and reliability of mobile agents in real-world complex scenarios.
Critical strengths
- The research problem has practical value. It focuses on long-term task management in dynamic mobile environments, expanding the application scenarios of mobile agents.
- The system design is relatively complete. It combines OS-level state awareness, function invocation, and permission control to achieve persistent cross-app task management.
- A new evaluation benchmark is proposed. DynamicWorld covers a variety of dynamic scenarios and can evaluate agents' continuous task execution capability.
- The experimental results are promising. Task success rates are notably better than those of compared methods while reducing the number of model calls.
Critical weaknesses
- There is a lack of ablation studies on key components, making it impossible to determine the true contribution of each part of the OS-Native framework.
- What are the fundamental architectural differences between OS-Native Agents and AOHP, beyond persistent task management and the introduction of DynamicWorld?
- Does the advantage of OS-Native Agents in DynamicWorld depend on the benchmark apps exposing more convenient interfaces specifically for this framework?
- Security and privacy mechanisms mainly rely on existing Android capabilities, and there is insufficient security evaluation.
- There is a lack of repeated experiments and statistical stability analysis.
Questions for authors
First, what is the main source of the performance improvement? The authors need to conduct ablation experiments within the same system to clarify how much the native state interface, AppFunctions, event recovery, and persistent task context each contribute to the performance gain. Second, what is the real technical innovation of OS-Native Agents relative to existing systems such as AOHP? If the main novelty lies in persistent task management and the DynamicWorld benchmark, these two contributions should be argued more focusedly, rather than presenting existing Android OS mechanisms themselves as the main innovation.
4 comments
W2 / Q2. What is the fundamental difference from AOHP?
We agree with the reviewer's reading: the main novelty lies in persistent task management and DynamicWorld, and the Android mechanisms are not the innovation. We argue the two as one contribution: autonomous task management is the design, the OS mechanisms are the means, and DynamicWorld is the test.
| Android | AOHP | Ours | |
|---|---|---|---|
| Proactive | Notification listeners, broadcasts, alarms | Event streams: a notification buffer and sensor streams the agent subscribes to | Monitors bound to tasks; a change log that keeps every notification, app event, and device change; event triage that decides per task to ignore, record, or act, and wakes only the affected task |
| Efficient | AppFunctions, content providers, system services | System-exposed tools through API, CLI, structured UI, or GUI as fallback; skills package reusable capabilities | System-exposed tools: app functions, OS tools for calendar, contacts, messages, notes, photos, and files, and system actions, with GUI as fallback; skills load only the functions they declare |
| Controlled | Permissions, identity, trusted confirmation | Per-operation information-flow policy, trusted vault, taint tracking | Calls sorted by declared side effect; grants issued outside the model, per task, revoked when it closes; notification redaction and on-device profile filling |
| Persistent | Storage, keep-alive | System memory for preferences, task state, histories, and policy decisions (profile, task-local, sensitive) | Task records (goal, constraints, progress, linked app records); routing each message to its task's session; rebuilding context from the record on wake-up; closing the task, which removes its monitors and grants |
Both systems build on largely the same OS mechanisms. They differ in what the mechanisms are tied to: AOHP ties them to the agent while it handles a request; we tie them to the user's task, from delegation until it closes. We also do not modify the OS: Android already gives apps every mechanism this needs. As W1 shows, tying events and state to the task is what keeps the agent's success level as changes accumulate.
W3. Does the advantage depend on the benchmark apps exposing more convenient interfaces for this framework?
No. Without native interfaces, our agent reads and changes app records through the same screens as MobileClaw, with the same model:
| Comparison (GPT-5.4) | Δ | 95% CI | p |
|---|---|---|---|
| OS-native without native interfaces (− Efficient) vs. MobileClaw | +24.6 pp | [+11.1, +37.3] | 0.004 |
The interfaces also were not designed around our agent:
| Item | How it is built |
|---|---|
| AppFunction schemas | Follow published Skill and API references, such as Variflight, AMap, and DiDi |
| App logic | GUI pages and AppFunctions call the same business operations on the same records under the same rules |
| Task construction | Every task was executed once through the GUI and once through AppFunctions, and kept only if both completed it |
| Scoring | Reads app records, not which interface was used |
W4. Security and privacy mechanisms mainly rely on existing Android capabilities; the security evaluation is insufficient.
The mechanisms do come from Android. What we add is that they serve the task: a task's monitors and checks last only as long as the task. To evaluate them, we ran checks on a real device in the form of AOHP's security evaluation (its Table 2). Each check runs the full code path, from the harness to the agent application to the app. A scripted model issues the calls, so the outcome depends only on the checks: they hold even if the model is misled, for example by injected text, into making the call.
| Check | Expected behavior | Result |
|---|---|---|
| Ordinary actions | Writing a note goes through without asking the user | Pass |
| Sensitive actions | A tap on a payment button requires the user's trusted confirmation; after a refusal, a coordinate tap on the same button requires it again, and the order stays unpaid | Pass |
| Unsupported access | A function outside the declared interface set is refused before it reaches the app | Pass |
| Sensitive events | Card numbers, ID numbers, one-time codes, and passwords in notifications are redacted before they reach the model | Pass |
| Task closing | Closing a task removes its monitors; a later matching notification wakes nothing | Pass |
W5. Repeated experiments and statistical stability.
Every system runs three times with every model on all 88 tasks with the same event timelines, 27 full runs in total.
| Model | System | Success | Model calls / scenario | Tokens / scenario | User requests / scenario |
|---|---|---|---|---|---|
| GPT-5.4 | MobileWorld | 21.2 ± 2.9% | 102.8 | 1.52M | 1.72 |
| MobileClaw | 47.0 ± 5.7% | 290.7 | 3.40M | 2.29 | |
| OS-native | 78.4 ± 3.0% | 142.2 | 2.48M | 1.63 | |
| GPT-5.4-mini | MobileWorld | 4.5 ± 2.3% | 61.1 | 0.80M | 1.12 |
| MobileClaw | 17.0 ± 3.4% | 196.5 | 1.68M | 2.71 | |
| OS-native | 36.7 ± 5.7% | 100.0 | 1.64M | 1.10 | |
| Gemini 3.5 Flash | MobileWorld | 44.3 ± 2.3% | 120.5 | 2.17M | 1.38 |
| MobileClaw | 51.9 ± 3.5% | 340.9 | 5.19M | 2.37 | |
| OS-native | 73.9 ± 1.1% | 170.0 | 4.46M | 1.25 |
Each task is averaged over its three runs and paired with the same task in the baseline:
| Model | Comparison | Δ | 95% CI | p |
|---|---|---|---|---|
| GPT-5.4 | OS-native vs. MobileClaw | +31.4 pp | [+20.0, +43.0] | 0.0004 |
| GPT-5.4 | OS-native vs. MobileWorld | +57.2 pp | [+45.8, +68.1] | <0.0001 |
| GPT-5.4-mini | OS-native vs. MobileClaw | +19.7 pp | [+13.6, +25.4] | 0.0002 |
| GPT-5.4-mini | OS-native vs. MobileWorld | +32.2 pp | [+23.5, +40.3] | 0.0001 |
| Gemini 3.5 Flash | OS-native vs. MobileClaw | +22.0 pp | [+9.0, +36.9] | 0.007 |
| Gemini 3.5 Flash | OS-native vs. MobileWorld | +29.5 pp | [+15.9, +43.6] | 0.002 |
All six differences are significant, and every interval lies above zero. We will add the per-run tables and the tests to the appendix.
Reviewer response to authors?
Dear reviewer JLTf,
Author-reviewer discussions end on November 18th. If possible, please respond to author comments by that time so that authors have opportunity for any additional responses.
Thank you.
We thank the reviewer for the comments.
Updated system. Since submission, we have improved the agent's harness: its context management, its skill texts, and its handling of user interaction. The OS mechanisms, the task design, the benchmark, and the baselines are unchanged. We also add Gemini 3.5 Flash as a third model, ablations of each module, and repeated runs for stability. We will update the paper with all results below.
Metrics. Success of each system is the mean ± SD over three runs; each ablation is one full run. Every comparison reports the paired difference Δ, its 95% confidence interval from a bootstrap over scenarios, and p from an exact permutation test over scenarios.
W1 / Q1. How much do the native state interface, AppFunctions, event recovery, and persistent task context each contribute?
Each ablation turns off one part of the GPT-5.4 system and keeps everything else fixed: the model, the harness, the apps, the event timelines, the simulated user, the judge, and the scoring. Native app-record reads and AppFunctions are removed together, because both let the agent read and change app records without the screen; device state and notifications stay, because they serve event handling.
A change is any app, system, or third-party event after delegation that concerns the task; the user's own later instructions are not counted. Success by the number of changes (GPT-5.4):
OS-native's success falls by 7% (relative) from no changes to three or more, MobileClaw's by 46%, and MobileWorld's by 91%. Without event delivery or without the task layer, OS-native falls by 31% and 36%; without native interfaces, by only 5%.