- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces GMA to test agents' ability to complete complex mobile tasks. The authors adapt seven open-source applications, self-host their backend services, and build 300 tasks across four tiers: Atomic, Compositional, Cross-Application, and Realistic. The system uses programmatic checks to determine whether the full task and each subtask have been completed.
The authors also compare different implementations and combinations of context management, within-task memory, planning, and verification modules. The results show that the same module can have different effects on different tasks. Combining more modules does not always bring further gains.
Critical strengths
-
The application environment and task verification are fairly well developed.The seven applications and self-hosted backend services broaden the coverage of mobile evaluation. The initialization tools restore both the phone and backend states. The tasks have been checked through manual execution, and the system can record which subtasks have been completed. This makes it easier to compare agent performance and see which requirements remain unmet.
-
The comparisons between modules are useful.The authors compare specific implementations under a shared setup, which helps make the differences between modules clearer. There are experiments on history retention, memory formats, planning, and verification methods. The appendix also includes results for module combinations and task trajectories. These materials help researchers judge which tasks a given module may be useful f
Critical weaknesses
-
The reasons for differences in module performance are not fully explained.Tables 6–7 show results for different implementations and combinations, but do not fully explain why structured memory performs better overall or why performance drops with some module combinations. The appendix includes failure examples, but their link to differences in module performance is still unclear. The authors could use the existing trajectories to show which errors become less or more common when different memory methods or module combinations are used on the same task.
-
Task tiers and execution length need to be distinguished more clearly.The four tiers involve several factors, including the applications used, task length, the number of requirements, and how spread out the information is. Appendix E.1 also groups tasks by the actual number of steps taken by the baseline, including both successful and failed runs. However, the opening of Appendix E calls this “the interaction length required for successful execution.” These two descriptions do not fully match. The actual step count depends on both the task requirements and how the agent carries out the task. These results therefore do not directly show that the differences come from the task's long-horizon dependencies. The wording should be made consistent, and the authors should explain what conclusions each analysis supports.
Questions for authors
-
Based on the existing trajectories, what different errors occur with different memory methods and module combinations? Which changes are supported by examples, and which causes are still unclear?
-
How does ''the interaction length required for successful execution'' at the start of Appendix E relate to the actual step counts in E.1, which include both successful and failed runs? Can existing records of manual execution help distinguish the actions required by the task from extra actions caused by agent errors?
2 comments
Thank you again for your helpful feedback. We would appreciate any further thoughts on whether our rebuttal has addressed your concerns. If any questions remain, we would be happy to clarify them. Thank you for your time and consideration.
Rebuttal Part 1
We sincerely thank you for the careful reading, constructive feedback, and thoughtful suggestions. We appreciate the opportunity to clarify our contributions, address the concerns, and provide additional analyses.
W1 & Q1 Trajectory-Level Analyses of Harness Effects
We agree that detailed trajectory analysis helps explain how different harness modules affect agent behavior. Our case study examines actions, module states, and evaluation outcomes, with representative examples below.
M2 keeps unresolved requirements explicit during recovery. The task requires transferring a food recommendation into an order and then sending a message. When search results remain blank, M2 keeps the unresolved search goal and downstream ordering and messaging requirements explicit. The actor retries the search and switches to the Products tab while the memory preserves the remaining workflow. The run succeeds in 44 steps, whereas the baseline repeatedly waits and reaches 150 steps with a score of 0.33. This comparison illustrates explicit task state maintenance alongside recovery from an unproductive action pattern.
M2 can record procedural completion without validating the target. The task requires adding "Lego House" to a specified playlist. The actor selects an acoustic version, but M2 records the menu and playlist selection actions as complete and empties its remaining task list. The actor then declares success despite failing final verification. In contrast, the baseline selects the required song and succeeds. This case exposes a gap between tracking completed actions and establishing that they satisfy the intended requirement. Memory preserves the agent's mistaken completion state rather than correcting it.
E3 preserves requirements that remain usable after execution errors. The task requires creating two same name contacts with distinct attributes. E3's summaries explicitly associate each contact with its required phone number, labels, and email. After the actor enters information into the wrong field and accidentally clears the second contact's phone number, these requirements remain available in context. The actor identifies the missing number, re-enters the correct value, and resumes the required configuration. E3 succeeds in 44 steps, while the baseline reaches 150 steps with a score of 0.50. The useful behavior is recovery with the correct requirements intact, rather than error-free execution.
V1 can redirect both memory and action through an incorrect judgment. The task requires configuring the initially earliest inactive, non-repeating alarm. In an E3 + M2 + V1 run, the actor correctly selects and enables the 7:15 alarm. V1 then judges this target incorrect because it is now active and identifies the still inactive 9:30 alarm as the target. M2 replaces its remaining requirements for 7:15 with requirements for 9:30, and the actor follows this change. The run ends with a score of 0.33. Here, verification incorrectly re-evaluates an initial state selection condition after the agent changes that state, and the resulting error propagates through memory into subsequent actions.
Together, these cases show how harness modules shape execution by maintaining requirements, recording progress, and supplying corrective feedback. These mechanisms support recovery when their representations remain accurate, but can also preserve mistaken completion judgments or propagate incorrect feedback into subsequent actions. The trajectories therefore help explain why adding harness support does not necessarily improve performance.
W2 & Q2 Clarification Regarding Task Tiers and Execution Length
The four difficulty tiers are defined by task design, independently of agent execution. In contrast, Appendix E.1 groups tasks by the actual number of interaction steps taken by Qwen3.7-Plus under the baseline harness, including both successful and failed episodes. These analyses examine performance across task structures and observed baseline execution lengths, respectively. The latter reflects both task demands and agent behavior, rather than isolating inherent long-horizon dependencies.
The phrase "the interaction length required for successful execution" at the beginning of Appendix E is a wording error. The analysis uses observed baseline step counts, including both successful and failed episodes, as explicitly defined in E.1. We will correct the phrase to "the observed baseline execution length" for consistency. We again thank you for pointing this out.