acceptodds
Under review as a conference paper at ICLR 2027

Remembering What to Act On: Object-Centric Execution State for Mobile GUI Agents

Abstract

Long-horizon mobile GUI tasks require agents to process multiple data objects across pages and applications. For reliable execution, agents must retain the required values while tracking how task data are organized, what has been done to each object, and which operations the task constraints allow. In this paper, we introduce DOMES, a method for Dynamic Object-centric Maintenance of Execution State, which unifies data retention and progress tracking by binding each object’s structured data to its evolving processing status and task constraints. We train our agent with an online trajectory synthesis pipeline that combines template-based task expansion, automatic environment initialization, and programmatic completion verification. This enables automated collection of successful trajectories that jointly supervise state updates and GUI actions. Considering that state maintenance does not benefit every task, we introduce STR-GRPO, which contrasts rollouts with and without state maintenance on the same task while penalizing non-empty updates. This encourages the agent to maintain state only when it contributes to task success, while improving generalization to unseen tasks. To evaluate agents' ability to complete required data operations and respect task scope, we introduce DataScope, a scalable online mobile benchmark. Its parameterized templates vary the number of target objects and out-of-scope distractors while preserving workflow logic and providing executable environment initialization and final-state verification. Since binary success does not capture partial completion or scope violations, we propose app-level and data-level metrics to measure workflow progress and the completeness and precision of data operations, respectively. Experiments show that our 8B agent achieves a 76.7% success rate on AndroidWorld, outperforming UI-TARS-2-230B by 3.4 percentage points. On DataScope, it consistently outperforms the strongest same-sized baseline across app-level, data-level, and binary task-success metrics. Compared with standard GRPO, STR-GRPO substantially reduces state-update rates while maintaining comparable observed task success. We will release our agent and curated benchmark.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Digital Agent, Multimodal learning, Large-Multimodal Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_SdiXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses long-horizon mobile GUI tasks that require processing multiple data objects across apps, retaining required values while tracking each object's processing progress. The authors propose DOMES, an object-centric execution state that binds each data unit's structured values to its evolving processing status (remaining / finished / skipped) under task constraints, injected into the prompt and updated jointly with actions. The agent is trained via verified online trajectory synthesis (final-state-verified SFT samples) followed by STR-GRPO, which contrasts DOMES-enabled and DOMES-disabled rollouts of the same task and penalizes nonempty updates to learn selective state maintenance. For evaluation, the paper introduces DataScope, an online benchmark of 128 data-centric tasks with same-schema distractors, plus two metrics: App-Level Progress and Scope-Aware F1. The 8B agent achieves 76.7% SR on AndroidWorld, 23.3% on unseen MobileWorld, and outperforms same-sized baselines on DataScope; STR-GRPO reduces nonempty update rates by ~14 points over standard GRPO while slightly improving success.

Critical strengths
  1. Verification is closed-loop at both ends. Training trajectories are retained only after programmatic final-state verification, and DataScope's verifiers are calibrated against controlled terminal states with expected SR / App-Prog / Scope-F1 scores covering five failure categories, regenerated until all checks pass—giving the metrics a concrete correctness guarantee.

  2. The ablation methodology is rigorous. Representation ablations include status-free and length-matched controls; GRPO and STR-GRPO share the SFT checkpoint, RL tasks, rollout budgets, and optimization steps; and the statistical comparison honestly reports that the SR gain of STR-GRPO over GRPO is not statistically significant (95% CI [−1.7, +5.8] on AndroidWorld) while the update-rate reduction is.

  3. The two-level metrics and distractor design probe a real blind spot of binary success—partial progress and out-of-scope modifications—via same-schema distractors that require value-level discrimination rather than type-level filtering.

Critical weaknesses
  1. The similarity between the synthesized training tasks and the AndroidWorld evaluation tasks is never quantified. The training scenarios are built on the AndroidWorld environment and app suite, and the training task families are visibly isomorphic to AW task families. The paper declares instance-level disjointness only with respect to DataScope; for AndroidWorld, the headline benchmark, no such analysis exists. The 76.7% headline is therefore an in-distribution result whose overlap with training is unmeasured. The authors need to quantify the similarity between training instances and AW's 116 evaluation tasks.

  2. No scenario-level characterization of when state maintenance helps, despite this being the core motivation of STR-GRPO. The enabled-vs-disabled reward differences (Fig. 4c) are reported only as training-time aggregates, never bucketed by task attributes (cross-app degree, object count, information-retention distance); the ~68% of steps on which the trained model emits an empty state block are not analyzed for the scenario characteristics they correspond to; and DOMES-specific failure modes—the schema errors of Appendix C.6, present in 12.3% of DataScope tasks using DOMES—are treated as reliability footnotes rather than as evidence about when maintenance should be avoided. Readers cannot judge whether the learned update rate (31.9%) reflects appropriate scenario selection or merely penalty-induced frequency compression.

Questions for authors
  1. How many of the 20 training apps overlap with AndroidWorld's apps, and how similar are the training tasks to the AW evaluation tasks?

  2. What is the AndroidWorld success rate after excluding tasks that share task families with the training templates?

  3. What scenario characteristics distinguish the ~68% of steps on which the model outputs an empty state block?

  4. The Introduction and Abstract are too verbose and should be condensed.

  5. For tasks that pass programmatic final-state verification, were the success labels manually spot-checked on actual agent rollouts (for evaluation or retained training trajectories)? The paper's human validation covers benchmark instances (B.2) and verifier calibration on controlled states (B.3), but neither audits the true-positive rate on real executions.

Reviewer_nQUbOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on mobile GUI tasks, particularly those where some operations need to be applied to every member of a set of inputs while satisfying some constraints. It proposes a structure called DOMES for tracking progress on such tasks. To train open models to work with this new structure, the paper uses a synthetic task generation method to create new task instances from a template, while handling environment initialization and programmatic evaluation. The training method is a variant of GRPO that contrasts trajectories with and without the DOMES structure. Post-trained Qwen3-VL-8B outperforms UI-TARS-2-230B on AndroidWorld and MobileWorld, and generalizes to a newly introduced benchmark called DataScope.

Critical strengths

This work covers a lot of ground: synthetic task generation, new benchmark creation, model training, and gets an 8B model with better performance than an existing 230B model on three tested benchmarks (AndroidWorld, MobileWorld, DataScope).

As described, the technical implementation of each individual part makes sense.

Critical weaknesses
  1. The very first words in the abstract grossly exaggerate the problem. Abstract uses the phrase "long-horizon mobile GUI tasks". From the original paper, AndroidWorld tasks can be done within 10 steps, most below 20, and the max step limit allowed is 2x the number required by humans to do it. This is not long at all by current standards (e.g. OSWorld 2.0).
  2. The problem of retaining information over multiple steps has a trivial solution: at each step, let the LLM planner generate the planned action(s) represented as code in some programming language and store values as variables in the runtime; the runtime can keep the variables across multiple LLM steps. So you get all the machinary of variable scope and control flow without needing any extra work. Just assign values to variables with clear names, then any decent LLM will be able to use it correctly. Practical GUI/browser automation languages like Playwright, BrowserUse, Simulang, etc. already allow LLMs to generate GUI actions represented in code, and their runtime will execute such code and preserve values across agent steps. Any decent model can already do this including various open models, not just frontier models. And various production-level CUA products (you should know them already, I won't name them to avoid giving them free marketing) already support generating code (of various length) and it's trivial to make the code runtime maintain variables across multiple calls of the planner.
  3. The problem of repeating a procedure over a set of input data has a trivial solution: represent the plan in a script that has control flow built in, and use some LLM calls for decision-making. In the example of Figure 1, the task is to get all event data, iterate through each one, decide whether it is work-related, and if yes then add it to calendar. The entire plan can be written as a script with some model calls, so the program runtime gives you "execution progress tracking" for free.
  4. Overall, the writing of Section 3.1 is a mess, does not give the reader a clear global picture of the overall approach. It does not clearly say where DOMES is applicable or not applicable, and what new information it adds that is not already available in the state (the agent's original screenshot or accessibility tree observation). How much extra latency does the separation of "harvest" and "execute" cause? If I need to wait 5s for the harvest phase, very few users would tolerate it on a mobile phone. Section 3.1 says the "agent constructs Schema.fields". How accurate is this? If the schema is constructed from the information that is already contained in the model's observation of the state, and you don't trust the model to do planning correctly with the raw observation (that's the argument you make in this paper), then how are we supposed to trust the same model's ability to produce a correct schema from the observation?
  5. Regarding the cost of using DOMES. It is unclear whether this "cost" is a token cost or latency, or both. You include this cost term in the reward (4) with a scalar α\alpha and don't say how the coefficient can be tuned. Your uiu_i is the fraction of steps in a trajectory that has non-empty DOMES, so the cost term does not depend on the raw token cost or latency. Unclear how this fraction is related to actual token cost or latency.
  6. The DataScope evaluation set introduced in this paper has "out-of-scope distractors". It is not clearly stated why including these distractors is useful for benchmarking, as it is unclear what tasks in practice will have such distractors.
  7. The abstract motivates the work by saying "For reliable execution...". But nothing in the experiment section measures reliability, in the sense of pass^k. It just says "success rate", which is unclear. It is not clear how many runs were done per task and whether success is measured by pass@k or pass^k.

Minor:

  1. Intro "which objects still require action": Unclear what you mean by objects here. What objects, in what context? And why "still require"? This is just unclear writing. You probably have some special scenario or task in mind where there are "objects that require action", but this is not made clear to the reader.
  2. "task data, represented through their structure, values, and processing states": You should just write these things formally, even in the introduction. These terms are nonstandard so it's impossible to understand from the intro what you mean precisely. Trying to describe them in normal language in the intro is useless.
  3. Intro lines 91-92: Unclear what has maintenance cost. This term appears out of nowhere. And this cost is used to motivated the "STR-GRPO", so if you don't tell me where/how this cost appears, how am I supposed to understand why you need STR-GRPO?
  4. Unclear what "STR" means.
Questions for authors

What information does DOMES include, that is not already contained in the agent's trajectory, that justifies the effort of further training? Why can't you just make the agent generate code, store values as variables in code, and use the built-in control flow for free to guarantee that every object that needs to be processed iteratively will be processed for sure?

Reviewer_14icOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DOMES, an object-centric execution-state representation for mobile GUI agents to tracking the multi-object status for long-term trajectorys. To reduce useless state update opeartions, this paper proposes STR-GRPO to penalize nonempty state updates. This paper also introduces DataScope, a benchmark that varies target objects and distractors. Evaluation on AndroidWorld, MobileWorld, and DataScope shows improved task performance.

Critical strengths
  • The design of object-centric state representation is clear and well-motivated. DOMES explicitly associates each actionable object with its processing status and task constraints in a structured JSON representation. This enables the GUI agent to distinguish completed, pending, and excluded objects without reconstructing these associations from interaction history.

  • The Execution-state ablation in Table 4 verify the the contribution of explicit state tracking in the JSON schema of DOMES.

Critical weaknesses
  • Lack of explicit evidence to prove the of effectiveness of DOMES for multi-object tasks, which motivates the method. Table 4 reports aggregated results on AndroidWorld and MobileWorld without separating single-object from multi-object tasks or examining how gains vary with object count and cross-step dependencies. Table 6 further compares state-management mechanisms using a fixed planner and executor, but likewise does not provide this task-level breakdown. Therefore, these experiments do not fully explain the sources of the overall gains.

  • It is unclear that how much diagnostic value the DataScope benchmark can provide beyond existing benchmarks. Its controlled task generation pipeline could help to examine how object-tracking and filtering demands affect execution. However, ablation studies are not conducted on this benchmark. Consequently, the evaluation establishes that DataScope is challenging, but does not clearly demonstrate what additional insight it provides into the proposed mechanism.

  • The json schema of DOMES needs more justification. The design in Section 3.1 is reasonable for collecting information followed by batch processing, but the suitability is less clear for tasks that interleave collection and execution, discover additional required fields during execution, or require multiple operations on each object. Table 4 removes execution-status information as a whole; it does not isolate these design choices or compare DOMES with alternative representations carrying equivalent status information.

Questions for authors
  • Which DOMES schema was used in the experiments? Section 3.1 and Figure 2 describe Constraints, RemainingFiles, CompletedFiles, and Schema.locked, whereas Appendix E.1 presents a different structure without explicitly including these fields.
  • Please provide further details of the GRPO and STR-GRPO training stages, particularly the number of unique training tasks/initializations, rollout trajectories collected, and trajectories used for optimization over the full training run.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.