acceptodds
Under review as a conference paper at ICLR 2027

Reuse, Rewrite, or Reject: Guarded Replay for Mobile GUI Agents

Abstract

Mobile GUI agents frequently perform repetitive tasks with similar interaction procedures. Reusing previous executions can reduce redundant LLM inference and execution overhead, but variations in user instructions and execution environments may invalidate previously successful actions. We systematically characterize these variations into six representative categories and observe that such changes do not necessarily invalidate entire procedures: individual transitions can remain reusable when validated against the current task context and interface state. Based on this insight, we propose graph–text templates for selective execution-trace reuse. The graph captures UI states and action transitions, while the text encodes task intent, procedural structure, and typed slots for variable content. A record–template–replay framework constructs templates from verified executions and leverages the current instruction and live UI state to reuse, adapt, or discard individual transitions. Given an execution trace, our framework reduces LLM inference token consumption by approximately 80% on average across five base agents on AndroidWorld, with most agents achieving at least 1.5 execution speedup over LLM-only execution. Real-device evaluations on Android and HarmonyOS demonstrate over 2 execution speedup in most evaluated configurations.

Primary area:
applications to robotics, autonomy, planning
Keywords:
mobile GUI agents, action replay, workflow memory, grounded execution, agent efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_siYyOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a record-template-replay controller for reusing historical execution trajectories in mobile GUI agents. Its key idea is to determine whether a recorded action should be reused, rewritten, or rejected based on the current instruction and environment state. The evaluation covers five base agents on AndroidWorld and real-device experiments on Android and Harmony OS.

Critical strengths
  • The paper addresses a practically important problem: how to reuse historical GUI interactions efficiently.
  • The distinction between environmental changes and instruction changes provides a useful view for understanding replay failures.
Critical weaknesses
  • Insufficiently specified method

The key mechanism is deciding whether a recorded action can be reused, requires rewriting, or should be rejected based on the environment and instruction changes. But, the paper does not explain sufficiently how the relevant environmental state and instruction semantics are represented and compared.

Lines 197-208 describe six categories of environment and instruction changes, while their role in the implemented controller remains unclear. How are these changes are identified, and how does each category affect the admission checks? Are the pre-checks and post-checks rul-based, model-based, or a combination of both?

In addition, various implementation details are missing. For example, how typed slots are extracted from instructions, how states are identified and compatibly merged, how retrieval candidates are ranked, and what thresholds govern matching and rejection. Terms such as parameter generalization'', app-level semantic retrieval'', ``soft task matching'' are introduced without enough operational detail to understand or reproduce them.

Even worse, as stated by the authors, the search thresholds and retry counts are not explicitly reported (Appendix A.4) and the provided materials ``do not constitute a complete reproduction package'' (Line 494).

  • Unclear experimental settings

Regarding the metrics, the success metric in Table 1 is difficult to interpret. Rather than directly reporting the fraction of tasks successfully completed by the replay-assisted controller, the paper constructs a value by multiplying an original success rate by a reply success rate. The probabilistic justification of this multiplication is. unclear, particularly in the real-device experiments. The authors should report the controller's directly measured end-to-end success rate and clearly separate record availability, success conditional on an available record, and success after fallback.

The meaning of the original success rate also changes across settings. It refers to the LLM-only baseline on AndroidWorld but to reply-only execution in the real-device experiments. Why are different reference configurations used for the same reported metric? Why is the LLM-only baseline not used consistently for success-rate comparisons?

These unclear metrics make it difficult to judge the effectiveness of the proposed controller.

Regarding the settings, first, although the paper states that successful base-agent executions supply the historical records, it does not adequately describe the record-collection procedure, the relationship between recording and evaluation tasks, or the number of tasks and repeated runs for each configuration. The appendix clarifies that 116 tasks is the size of the original AndroidWorld suite, rather than the evaluated subset of every agent. Consequently, percentages such as 21.0, 59.1, ... in Table 1 cannot be interpreted without the actual denominators, repetition counts and aggregation procedure. In addition, the rationale for selecting the five agents and their evaluation configurations should also be provided.

Second, the human evaluation protocol requires further clarification, including the number of evaluators, the criteria they applied, and how disagreements were resolved. If the judgments were conducted by a single evaluator, the absence of independent verification limits confidence in their reliability.

Finally, the provenance of the PreAct results in Figure 4 is unclear. The cited PreAct result concerns a 15-task AndroidWorld evaluation, whereas Figure 4 presents comparisons under several real-device conditions. Therefore, the comparison is unfair.

  • Insufficient ablation studies

The ablations are cumulative, progressing from C0 through C1 and C2 to the full controller. In Figure 4, the largest improvement appears when the full controller is introduced. However, Appendix B.4 states that final increment adds the remaining controller components, so its gain cannot be attributed to any individual mechanism.

Therefore, more targeted ablations are needed to distinguish the effects of pre-checks and post-checks, rewriting versus rejection followed by fallback, and graph-based versus text-based retrieval. Also, beyond the success rates, the token usage or execution time should be included to better establish the cost-benefit trade-off of the individual components.

Questions for authors

See the comments on weaknesses.

Reviewer_3bkBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how mobile GUI agents can reuse successful executions when instructions or interface states change. Recorded trajectories are converted into graph–text templates containing UI transitions, task descriptions, and typed parameters. During execution, a controller retains compatible actions, updates explicitly specified values, or returns control to the base agent, with checks before and after each action. Experiments with five agents report lower inference costs and faster execution on AndroidWorld and physical Android/HarmonyOS devices. The evaluation focuses on execution with available verified records, and the reported speedups exclude recording and template-construction costs.

Critical strengths
  1. Addresses an important practical problem. Repeated mobile tasks create opportunities to avoid redundant model inference, but changes in requested values or interface conditions can invalidate previously successful actions. The paper addresses this challenge through selective action reuse, explicit parameter adaptation, and fallback to the original agent. Its distinction between retrieving a relevant procedure and determining whether an action remains appropriate is well motivated.

  2. Evaluates efficiency across multiple agents and physical devices. The study includes five base agents and four phone models spanning Android and HarmonyOS. It reports token reductions of 78.0–89.6% and execution speedups of 1.2–1.9× on AndroidWorld, alongside speedups of 1.8–3.9× in the real-device experiments. These results provide encouraging evidence that stored execution traces can reduce online execution costs across several tested configurations.

Critical weaknesses
  1. The reported success scores are difficult to interpret consistently across evaluation settings. In the real-device experiments, Table 1 multiplies replay-only success by the full controller’s success rate, but the operational meaning of this product is unclear. For example, GUI-Owl’s background-state result is reported as 49.0%, obtained from 51.0% × 96.0%, while the full controller itself achieves 96.0%. These quantities describe different aspects of performance. Reporting observed success rates separately, alongside a matched LLM-only baseline, would make it possible to assess whether execution reuse preserves reliability while improving efficiency.

  2. Important experimental details are missing, limiting reproducibility and interpretation of the results. The paper does not report the evaluated AndroidWorld task subsets, episode counts for each model and configuration, repeated-run results, or the proportion of task pairs supporting the efficiency comparisons. The procedure for constructing and selecting record–live pairs is also insufficiently specified. Consequently, readers cannot determine how representative the evaluated variations are or assess uncertainty in the reported gains. Appendix D.3 additionally leaves timing boundaries and speedup aggregation unspecified, making the execution-time comparisons difficult to verify.

  3. The core decision and verification mechanisms need more concrete implementation details. Section 3 and Algorithm 1 explain the overall execution loop, but several essential operations remain described mainly through their intended behavior. These include extracting typed parameters, generating semantic preconditions and postconditions, merging compatible states, and checking whether a live target supports a stored action. The paper should explain how these operations are implemented and which require rules, model inference, or manual specifications. Without this information, it is difficult to reproduce the controller or assess how its checks achieve reliable reuse with reduced inference.

  4. The comparative and ablation results do not clearly isolate the benefit of the proposed mechanisms. The paper acknowledges that parameter adaptation, state checking, and fallback already appear in closely related methods. However, Figure 4 includes PreAct results without adequately documenting their source or experimental configuration, leaving the comparability of these results unclear. The cumulative ablations also leave intermediate guard and fallback settings incompletely specified, while the final configuration introduces several mechanisms together. These experiments suggest that the complete controller improves over basic replay, but provide limited evidence about which distinctive mechanisms account for that improvement.

Questions for authors
  1. Why are replay-only and full-controller success rates multiplied in the real-device results? Could you report the full controller’s success directly alongside a matched LLM-only baseline?
  2. How were record–live pairs selected, and how many tasks and repeated runs support each result? Please also clarify how execution time was measured.
  3. How are parameter binding, action matching, and semantic checks implemented? Which steps require model calls or manual specifications?
  4. How were the PreAct results in Figure 4 obtained? Could you clarify the ablation settings and which mechanisms account for the full controller’s gains?
Reviewer_a2y4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how mobile GUI agents can reuse successful execution traces when instructions or interface states change. It organizes these changes into six categories and proposes graph–text templates that combine UI transitions with task intent, typed slots, and pre/postconditions. A controller selectively reuses, rewrites, or rejects individual transitions, with fallback to the base agent. Given verified execution records, the paper reports 1.2–1.9× execution-time speedups and roughly 80% lower token consumption across five agents on AndroidWorld, and 1.8–3.9× speedups in evaluations on four physical Android/HarmonyOS phones.

Critical strengths

The six-category taxonomy provides a useful explanation of why a previously successful trajectory can become invalid. Separating environment changes from instruction changes gives a coherent motivation for state rechecking, value rebinding, and selective fallback. It also helps organize the experiments, which quantitatively examine background-state changes, parameter changes, and paraphrases.

The reductions in model calls and runtime across several base agents, together with physical-device experiments spanning Android and HarmonyOS, suggest practical value for repeated mobile tasks. This breadth makes the efficiency results worth considering, although the success–cost relationship needs clearer reporting.

Critical weaknesses

The central evidence-checking mechanism remains insufficiently specified. Section 3 and Appendix A describe contracts, control flow, and an illustrative timer example, but do not explain concretely how conditions are generated or which semantic checks use rules, models, or task-specific checkers. Since selective replay, parameter adaptation, and fallback are already established capabilities, these details are essential to assessing the contribution.

The success and cost reporting does not adequately explain the measured speedups. The paper identifies UI acquisition as a bottleneck, reporting that pre/post-action preparation and acquisition can account for 40.02% of AndroidWorld runtime, but provides no full breakdown of inference savings, retrieval/checking costs, or fallback and recovery overhead. Table 1 also uses different success baselines across settings and a separate LLM-only efficiency reference, yet the real-device product score's interpretation, episode counts, paired-success coverage, and precise timing boundaries remain unclear, limiting assessment of the success–efficiency tradeoff.

Questions for authors

Please specify how pre/postconditions are generated and how slot binding, live target grounding, and postcondition checks are implemented. Which operations invoke models?

Can you separate AndroidWorld and physical-device results into tables with explicit success and efficiency reference labels? For GUI-Owl under background-state changes, what task set and denominator yield the full configuration's 96%, and what evaluation event justifies multiplying it by the replay-only 51% to obtain 49%?

On the existing paired tasks, please report actual runtime breakdowns for LLM-only execution and the full method, separately for AndroidWorld and physical devices, such as inference, UI acquisition, device actions/waits, retrieval/checks, and recovery overhead. Please include fallback frequency, timing boundaries, and paired-success coverage, and clarify how the physical-device UI-acquisition optimizations affect the speedup comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.