- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces AndroidFlux, a benchmark for evaluating failure recovery in mobile GUI agents by resuming execution from checkpoints around an annotated first error. It also proposes Latent Peer Voting (LPV), which combines experts’ action likelihoods to generate preferences for reward-model training. The experiments report stronger recovery with agentic frameworks and GUI-oriented reasoning, better action ranking with LPV, and gains in recovery success from offline training on mixed normal and error contexts.
Critical strengths
-
AndroidFlux provides a valuable benchmark for mobile-agent error recovery, with separate evaluations of recovery performance and recovery-action ranking. Its checkpoints around the first error, allow agent behavior to be examined under different recovery conditions.
-
The experiments cover many agent families and settings. The results suggest that LPV improves action ranking, while training on mixtures of normal and error contexts can improve both recovery success.
Critical weaknesses
-
Are the automatic error labels reliable enough to serve as benchmark ground truth, given the 90.8% step-level precision reported in L191? A mislabeled error would change the recovery problem being evaluated.
-
Candidate-action scores mix action quality with the ability of the expert that continues the task. A strong expert may succeed despite a poor action, while a weaker expert, or one unfamiliar with the action, may fail after a useful one. LPV adds likelihood-based signals, but its voting weights still depend on task completion rates and thus on expert ability.
-
Table 2 should clarify that and are computed over different sample sets and cannot be directly compared.
-
What explains the large gap in task completion rates between replay and snapshot cases in Table 2? Why not restore all error contexts from snapshots to keep the loading procedure consistent?
-
In Section 6.4, rewinding lets the target agent avoid an error it might not have made itself, so success need not involve recovery. Injection supplies the next action externally and does not test whether the agent can choose a recovery action on its own. The reported success rates may therefore overstate how well the agent can recover from its own mistakes in practice.
-
Does off-policy error-fixing training improve the agent’s ability to recover from its own mistakes during execution? The clean-start gains in Section 6.6 do not directly answer this question, since higher task success rates could also come from making fewer errors.
-
LPV uses five scorers, so its gain over a single scorer may simply come from ensembling. Table 6 should compare LPV with equal-weight averaging or majority voting using the same five models. The Naive baseline uses only the generator’s success rate and does not provide this comparison.
Questions for authors
see weaknesses