- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses long-horizon mobile GUI tasks that require processing multiple data objects across apps, retaining required values while tracking each object's processing progress. The authors propose DOMES, an object-centric execution state that binds each data unit's structured values to its evolving processing status (remaining / finished / skipped) under task constraints, injected into the prompt and updated jointly with actions. The agent is trained via verified online trajectory synthesis (final-state-verified SFT samples) followed by STR-GRPO, which contrasts DOMES-enabled and DOMES-disabled rollouts of the same task and penalizes nonempty updates to learn selective state maintenance. For evaluation, the paper introduces DataScope, an online benchmark of 128 data-centric tasks with same-schema distractors, plus two metrics: App-Level Progress and Scope-Aware F1. The 8B agent achieves 76.7% SR on AndroidWorld, 23.3% on unseen MobileWorld, and outperforms same-sized baselines on DataScope; STR-GRPO reduces nonempty update rates by ~14 points over standard GRPO while slightly improving success.
Critical strengths
-
Verification is closed-loop at both ends. Training trajectories are retained only after programmatic final-state verification, and DataScope's verifiers are calibrated against controlled terminal states with expected SR / App-Prog / Scope-F1 scores covering five failure categories, regenerated until all checks pass—giving the metrics a concrete correctness guarantee.
-
The ablation methodology is rigorous. Representation ablations include status-free and length-matched controls; GRPO and STR-GRPO share the SFT checkpoint, RL tasks, rollout budgets, and optimization steps; and the statistical comparison honestly reports that the SR gain of STR-GRPO over GRPO is not statistically significant (95% CI [−1.7, +5.8] on AndroidWorld) while the update-rate reduction is.
-
The two-level metrics and distractor design probe a real blind spot of binary success—partial progress and out-of-scope modifications—via same-schema distractors that require value-level discrimination rather than type-level filtering.
Critical weaknesses
-
The similarity between the synthesized training tasks and the AndroidWorld evaluation tasks is never quantified. The training scenarios are built on the AndroidWorld environment and app suite, and the training task families are visibly isomorphic to AW task families. The paper declares instance-level disjointness only with respect to DataScope; for AndroidWorld, the headline benchmark, no such analysis exists. The 76.7% headline is therefore an in-distribution result whose overlap with training is unmeasured. The authors need to quantify the similarity between training instances and AW's 116 evaluation tasks.
-
No scenario-level characterization of when state maintenance helps, despite this being the core motivation of STR-GRPO. The enabled-vs-disabled reward differences (Fig. 4c) are reported only as training-time aggregates, never bucketed by task attributes (cross-app degree, object count, information-retention distance); the ~68% of steps on which the trained model emits an empty state block are not analyzed for the scenario characteristics they correspond to; and DOMES-specific failure modes—the schema errors of Appendix C.6, present in 12.3% of DataScope tasks using DOMES—are treated as reliability footnotes rather than as evidence about when maintenance should be avoided. Readers cannot judge whether the learned update rate (31.9%) reflects appropriate scenario selection or merely penalty-induced frequency compression.
Questions for authors
-
How many of the 20 training apps overlap with AndroidWorld's apps, and how similar are the training tasks to the AW evaluation tasks?
-
What is the AndroidWorld success rate after excluding tasks that share task families with the training templates?
-
What scenario characteristics distinguish the ~68% of steps on which the model outputs an empty state block?
-
The Introduction and Abstract are too verbose and should be condensed.
-
For tasks that pass programmatic final-state verification, were the success labels manually spot-checked on actual agent rollouts (for evaluation or retained training trajectories)? The paper's human validation covers benchmark instances (B.2) and verifier calibration on controlled states (B.3), but neither audits the true-positive rate on real executions.