- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes a record-template-replay controller for reusing historical execution trajectories in mobile GUI agents. Its key idea is to determine whether a recorded action should be reused, rewritten, or rejected based on the current instruction and environment state. The evaluation covers five base agents on AndroidWorld and real-device experiments on Android and Harmony OS.
Critical strengths
- The paper addresses a practically important problem: how to reuse historical GUI interactions efficiently.
- The distinction between environmental changes and instruction changes provides a useful view for understanding replay failures.
Critical weaknesses
- Insufficiently specified method
The key mechanism is deciding whether a recorded action can be reused, requires rewriting, or should be rejected based on the environment and instruction changes. But, the paper does not explain sufficiently how the relevant environmental state and instruction semantics are represented and compared.
Lines 197-208 describe six categories of environment and instruction changes, while their role in the implemented controller remains unclear. How are these changes are identified, and how does each category affect the admission checks? Are the pre-checks and post-checks rul-based, model-based, or a combination of both?
In addition, various implementation details are missing. For example, how typed slots are extracted from instructions, how states are identified and compatibly merged, how retrieval candidates are ranked, and what thresholds govern matching and rejection. Terms such as parameter generalization'', app-level semantic retrieval'', ``soft task matching'' are introduced without enough operational detail to understand or reproduce them.
Even worse, as stated by the authors, the search thresholds and retry counts are not explicitly reported (Appendix A.4) and the provided materials ``do not constitute a complete reproduction package'' (Line 494).
- Unclear experimental settings
Regarding the metrics, the success metric in Table 1 is difficult to interpret. Rather than directly reporting the fraction of tasks successfully completed by the replay-assisted controller, the paper constructs a value by multiplying an original success rate by a reply success rate. The probabilistic justification of this multiplication is. unclear, particularly in the real-device experiments. The authors should report the controller's directly measured end-to-end success rate and clearly separate record availability, success conditional on an available record, and success after fallback.
The meaning of the original success rate also changes across settings. It refers to the LLM-only baseline on AndroidWorld but to reply-only execution in the real-device experiments. Why are different reference configurations used for the same reported metric? Why is the LLM-only baseline not used consistently for success-rate comparisons?
These unclear metrics make it difficult to judge the effectiveness of the proposed controller.
Regarding the settings, first, although the paper states that successful base-agent executions supply the historical records, it does not adequately describe the record-collection procedure, the relationship between recording and evaluation tasks, or the number of tasks and repeated runs for each configuration. The appendix clarifies that 116 tasks is the size of the original AndroidWorld suite, rather than the evaluated subset of every agent. Consequently, percentages such as 21.0, 59.1, ... in Table 1 cannot be interpreted without the actual denominators, repetition counts and aggregation procedure. In addition, the rationale for selecting the five agents and their evaluation configurations should also be provided.
Second, the human evaluation protocol requires further clarification, including the number of evaluators, the criteria they applied, and how disagreements were resolved. If the judgments were conducted by a single evaluator, the absence of independent verification limits confidence in their reliability.
Finally, the provenance of the PreAct results in Figure 4 is unclear. The cited PreAct result concerns a 15-task AndroidWorld evaluation, whereas Figure 4 presents comparisons under several real-device conditions. Therefore, the comparison is unfair.
- Insufficient ablation studies
The ablations are cumulative, progressing from C0 through C1 and C2 to the full controller. In Figure 4, the largest improvement appears when the full controller is introduced. However, Appendix B.4 states that final increment adds the remaining controller components, so its gain cannot be attributed to any individual mechanism.
Therefore, more targeted ablations are needed to distinguish the effects of pre-checks and post-checks, rewriting versus rejection followed by fallback, and graph-based versus text-based retrieval. Also, beyond the success rates, the token usage or execution time should be included to better establish the cost-benefit trade-off of the individual components.
Questions for authors
See the comments on weaknesses.