- Rating
- 2
- Confidence
- 4(+2)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces TRAVEC, a training framework for mobile GUI agents based on evidence-grounded transition contracts. Each contract specifies the current state and preconditions, an action, its expected state change, observable verification evidence, and optional failure-conditioned recovery. The authors construct a reported 550K supervision items and train role-specific Action, Contract, and Verifier policies. During training, these auxiliary policies and a trajectory Critic audit collected interactions, which support action-policy optimization through SFT and GRPO. Deployment uses only an action-only policy without auxiliary verification or free-form reasoning. Evaluation covers four offline and two online benchmarks, with component and reward ablations and manual audits of annotation and verifier quality.
Critical strengths
-
The paper addresses an important practical problem: a syntactically valid and well-grounded GUI action does not necessarily produce the intended task-relevant transition. Organizing expected changes, observable evidence, and recovery conditions into a common training interface is a useful system design. Using auxiliary components only during training also preserves the low inference cost of an action-only deployment policy.
-
The evaluation covers both offline action prediction/grounding and online task completion. The reported ScreenSpot-Pro and MobileWorld improvements are encouraging. Appendix A.5 explicitly requires candidate-specific execution evidence collected from the same decision-state snapshot, avoiding the inappropriate reuse of a reference action's outcome for unexecuted alternatives. The manual annotation and verifier audits provide additional, although limited, evidence about supervision quality.
Critical weaknesses
-
The training/evaluation separation and supervision counts are not sufficiently clear to establish the experimental protocol. Section 4.2 states that the 400 AndroidWorld training tasks are disjoint from the evaluation task templates, while Appendix B.3 reports evaluation on all 116 described AndroidWorld templates. The paper does not explain which additional templates generated the training tasks or whether “disjoint” instead refers to different parameter instantiations of shared templates. Separately, Figure 5(b) includes 50,000 parser-only MobileViews records and 2,853 evaluation-only ScreenSpot/ScreenSpot-Pro records within the reported 350,000 contract/state items. Excluding these leaves 297,147 items. These observations do not establish leakage, but they leave both the generalization setting and the actual training-data count unresolved.
-
The experiments do not sufficiently isolate or directly validate the central contribution. A matched Qwen3.5-9B Action-SFT-only baseline is missing, and the action-only reward ablation does not report online task completion. Consequently, it is difficult to separate the contribution of structured contracts from domain adaptation, trajectory filtering, and standard action-alignment optimization. Moreover, on AndroidWorld, 35.345% versus 34.483% corresponds to 123 versus 120 successful episodes out of 348; without paired statistical analysis or repeated training runs, this small margin does not establish a reliable advantage over the strongest baseline. The paper also lacks quantitative measurements of silent transition failures, state drift, and recovery effectiveness, so the aggregate benchmark results do not directly demonstrate the proposed mechanism.
Questions for authors
Clarification: What templates generate the 400 AndroidWorld training tasks? Please specify their relationship to the 116 evaluation templates and distinguish template-level separation from parameter-instance separation. Are the training templates newly constructed?
Clarification: Does the 350K contract/state count, and consequently the 550K total, include parser-only and evaluation-only records? Please provide the number of examples actually used to optimize each policy and explain how the 60K preference pairs enter training.
4 comments
Response to Reviewer Ca3K (2/3)
A2: We thank the reviewer for these detailed and constructive suggestions. To address these concerns, we have added two controlled training comparisons and a direct evaluation of transition failures and recovery.
Matched controls. We have added a Qwen3.5-9B Action-SFT-only baseline and report online results for the action-only reward configuration in Table 4. All three configurations share the same backbone, Action-SFT data, and evaluation settings.
| Training configuration | AndroidControl High ↑ | AndroidControl Low ↑ | ScreenSpot ↑ | ScreenSpot-Pro ↑ | AndroidWorld ↑ | MobileWorld ↑ |
|---|---|---|---|---|---|---|
| Action-SFT-only | 50.259 | 77.532 | 85.456 | 47.818 | 12.931 | 5.983 |
| SFT + GRPO with action-only rewards | 54.890 | 78.816 | 92.060 | 62.998 | 16.667 | 8.547 |
| TRAVEC-9B | 64.309 | 93.337 | 92.689 | 67.805 | 35.345 | 15.385 |
All values are percentages. Performance increases from Action-SFT-only to action-only GRPO and then to full TRAVEC across all six metrics. Compared with Action-SFT-only, TRAVEC improves AndroidWorld and MobileWorld success by 22.414 and 9.402 percentage points, respectively. Compared with action-only GRPO, which uses the same SFT initialization, rollout data, and GRPO training schedule, TRAVEC improves the two online success rates by 18.678 and 6.838 percentage points. These results provide more direct evidence of the additional benefits of process-aware training beyond standard action supervision and action-only GRPO. The full comparison is reported in Section 4.7 and Appendix E.1 of the revised manuscript.
AndroidWorld comparison. We appreciate the reviewer's observation regarding the small numerical margin between TRAVEC and ForgeQwen3-8B on AndroidWorld. We interpret their performance on this benchmark as comparable, while the controlled comparisons above and the results on MobileWorld, which is not used during training, provide additional evidence of TRAVEC's effectiveness.
Direct evaluation of transition failures and recovery. We inspected the complete execution trajectories of each model on all 117 MobileWorld tasks, including task goals, action sequences, and interface evidence before and after each step.
| Model | Task success ↑ | Transition failure rate ↓ | Recovery rate after first failure ↑ | Task success after failure ↑ |
|---|---|---|---|---|
| Qwen3.5-9B | 5.13% (6/117) | 76.11% | 2.63% (3/114) | 2.63% (3/114) |
| OpenMobile-8B | 8.55% (10/117) | 49.59% | 13.27% (15/113) | 5.31% (6/113) |
| TRAVEC-9B (ours) | 15.38% (18/117) | 39.59% | 35.65% (41/115) | 13.91% (16/115) |
The transition failure rate measures the proportion of executed steps that fail to produce their intended task-relevant state changes. The recovery rate measures the proportion of trajectories with at least one confirmed transition failure in which subsequent actions successfully resolve the first failure, while task success after failure is measured on the same trajectories.
TRAVEC achieves the lowest transition failure rate and the highest recovery rate and post-failure task success among the three models. Compared with the base Qwen3.5-9B model sharing the same backbone, TRAVEC improves the recovery rate and task success after failure by 33.02 and 11.28 percentage points, respectively. These results provide direct quantitative evidence of the deployed Action Policy's failure recovery capability, complementing the qualitative examples of silent transition failures and state drift in Appendix C.
We further break down these metrics by grouping each model's trajectories by the number of steps it executed (1–20 and 21+ steps).
| Model | Execution length | Trajectories | Task success ↑ | Transition failure rate ↓ | Recovery rate after first failure ↑ | Task success after failure ↑ |
|---|---|---|---|---|---|---|
| Qwen3.5-9B | 1–20 steps | 88 | 5.68% (5/88) | 64.93% | 2.35% (2/85) | 2.35% (2/85) |
| Qwen3.5-9B | 21+ steps | 29 | 3.45% (1/29) | 93.59% | 3.45% (1/29) | 3.45% (1/29) |
| OpenMobile-8B | 1–20 steps | 68 | 11.76% (8/68) | 42.96% | 15.63% (10/64) | 6.25% (4/64) |
| OpenMobile-8B | 21+ steps | 49 | 4.08% (2/49) | 55.54% | 10.20% (5/49) | 4.08% (2/49) |
| TRAVEC-9B (ours) | 1–20 steps | 60 | 16.67% (10/60) | 28.17% | 36.21% (21/58) | 13.79% (8/58) |
| TRAVEC-9B (ours) | 21+ steps | 57 | 14.04% (8/57) | 47.04% | 35.09% (20/57) | 14.04% (8/57) |
Transition failures become more frequent in longer trajectories for all three models, while TRAVEC maintains similar recovery rates across the two groups (36.21% and 35.09%) and achieves the lowest transition failure rate and the highest task success in both. These evaluations are reported in Section 4.7 and Appendix E of the revised manuscript.
Response to Reviewer Ca3K (3/3)
Q1: What templates generate the 400 AndroidWorld training tasks? Please specify their relationship to the 116 evaluation templates and distinguish template-level separation from parameter-instance separation. Are the training templates newly constructed?
A3: As clarified in A1, the 400 AndroidWorld training tasks were not instantiated from a predefined set of templates. Instead, they consist of newly generated task instructions produced from exploration trajectories using MobileForge's public task-generation pipeline, without using the 116 official evaluation templates as generation references. The separation is therefore at the task-template level rather than merely between parameter instantiations of shared templates. Training and evaluation use the same AndroidWorld applications, so we characterize the setting as in-environment adaptation.
Q2: Does the 350K contract/state count, and consequently the 550K total, include parser-only and evaluation-only records? Please provide the number of examples actually used to optimize each policy and explain how the 60K preference pairs enter training.
A4: Yes. As detailed in A1, the 350K contract/state count includes 50,000 parser-only and 2,853 evaluation-only records, and the reported 550K total comprises 497,147 constructed records and pairs together with these 52,853 records.
Examples used to optimize each policy. The SFT training inputs for each policy are summarized below.
| Policy | Training inputs | Supervision target |
|---|---|---|
| Action Policy | 297,147 base records | Normalized actions |
| Contract Policy | The same 297,147 base records | Complete contracts, including applicable recovery branches |
| Verifier | Candidate supervision constructed from 60,000 preference pairs and 20,000 calibration records | Quality scores, verdicts, and confidence |
Role of the preference pairs. Each preference pair contrasts an accepted contract with a negative candidate under the same decision input. For Verifier SFT, the candidate contracts in these pairs, together with their structured scores and positive or negative labels, are converted into pointwise supervision and used alongside the 20,000 calibration records. The pairs are not used to train the Action or Contract policies.
GRPO data. The GRPO stage further optimizes the Action Policy using rollouts on the 400 training tasks, with four attempts per task (1,600 attempts in total). Following Section 3.4 and Appendix A.3, we retain tasks that are not solved in all four attempts, select one attempt per retained task, and keep the decision steps that pass the Critic, execution-validity, and Verifier filters. For each retained step, five candidate actions are sampled and evaluated using execution evidence collected from the same restored decision state, and the resulting records are fixed for the GRPO updates.
We have stated these training inputs and clarified the use of the preference pairs explicitly in the revised manuscript.
A gentle reminder that the deadline for the Author/Reviewer discussion is approaching
Dear Reviewer Ca3K,
We sincerely appreciate the time and effort you dedicated to reviewing our submission and providing valuable and thoughtful feedback. Your comments have been instrumental in helping us improve the clarity and rigor of our work.
We have carefully considered each of your comments and have done our best to provide comprehensive responses, clarifications, and the requested experiments. As the author–reviewer discussion period is drawing to a close, we would be truly grateful to receive any feedback you might have on our responses before the deadline.
Best regards,
The Authors
Response to Reviewer Ca3K (1/3)
We are grateful to Reviewer Ca3K for the careful and detailed review. We also appreciate the reviewer's recognition of the importance of the problem we study and our use of candidate-specific execution evidence. In response to the comments, we have clarified the experimental protocol and supervision counts, added the requested matched training controls, and conducted a direct evaluation of transition failures and recovery. We address each comment individually below.
W1: The training/evaluation separation and supervision counts are not sufficiently clear to establish the experimental protocol. Section 4.2 states that the 400 AndroidWorld training tasks are disjoint from the evaluation task templates, while Appendix B.3 reports evaluation on all 116 described AndroidWorld templates. The paper does not explain which additional templates generated the training tasks or whether “disjoint” instead refers to different parameter instantiations of shared templates. Separately, Figure 5(b) includes 50,000 parser-only MobileViews records and 2,853 evaluation-only ScreenSpot/ScreenSpot-Pro records within the reported 350,000 contract/state items. Excluding these leaves 297,147 items. These observations do not establish leakage, but they leave both the generalization setting and the actual training-data count unresolved.
A1: We thank the reviewer for the careful reading. We clarify both points below.
Training/evaluation separation. The 400 AndroidWorld training tasks were independently generated using MobileForge's public task-generation pipeline and exploration trajectories, without using the official evaluation templates as generation references. These tasks are newly constructed instructions rather than parameter instantiations of the 116 official evaluation templates. The term "disjoint" in Section 4.2 refers to task-template-level separation, rather than different parameter instantiations of shared templates. Since training and evaluation take place within the same AndroidWorld application ecosystem, we characterize this setting as in-environment adaptation. MobileWorld is reserved exclusively for evaluation, providing an additional assessment on a different mobile benchmark. We have clarified the task-generation procedure and the evaluation setting in Section 4.2.
Supervision counts. As the reviewer notes, the 350,000 contract/state items in Figure 5(b) include 50,000 MobileViews records used only for parser construction and 2,853 ScreenSpot/ScreenSpot-Pro records reserved for evaluation. Excluding these sources leaves 297,147 records used to construct training supervision, as detailed below.
The reported 550K total comprises 497,147 constructed records and pairs, together with 52,853 parser-only and evaluation-only records. Figure 5(b) identifies the purposes of these sources, but the main text did not separate them from the training supervision. We have made this distinction explicit and report the counts consistently throughout the revised manuscript.
W2: The experiments do not sufficiently isolate or directly validate the central contribution. A matched Qwen3.5-9B Action-SFT-only baseline is missing, and the action-only reward ablation does not report online task completion. Consequently, it is difficult to separate the contribution of structured contracts from domain adaptation, trajectory filtering, and standard action-alignment optimization. Moreover, on AndroidWorld, 35.345% versus 34.483% corresponds to 123 versus 120 successful episodes out of 348; without paired statistical analysis or repeated training runs, this small margin does not establish a reliable advantage over the strongest baseline. The paper also lacks quantitative measurements of silent transition failures, state drift, and recovery effectiveness, so the aggregate benchmark results do not directly demonstrate the proposed mechanism.