acceptodds
Withdrawn from ICLR 2027

AndroidFlux: Evaluating and Reward Modeling Failure Recovery Capabilities in Mobile-Use Multimodal Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mobile-use multimodal agents are typically evaluated by task success from a clean initial state, which does not isolate their ability to recover from intermediate errors. This capability also matters when a stronger agent takes over an unfinished task and inherits another agent's mistakes. Hence, We introduce AndroidFlux, a failure-recovery benchmark built on AndroidWorld. Recorded trajectories are restored to controlled checkpoints before, at, and after the first recoverable error, allowing different agents to continue from the same device state and history. A complementary reward-modeling benchmark evaluates recovery-action preferences using downstream rollout outcomes. Moreover, we release a reward-modeling training dataset of 1.8k first-error contexts with candidate recovery actions and an offline reinforcement learning (RL) dataset of 8.5k contexts to enhance both error recovery and overall task success. We also introduce Latent Peer Voting (LPV): each expert scores actions proposed by its peers using conditional likelihoods, and these scores are normalized within each candidate set and combined with performance-based weights to produce preference supervision. Our evaluations show that recovery performance only partially correlates with clean-start success, and that explicit reasoning does not consistently improve recovery. Also, we show that LPV aggregates the internal action preference representation in expert policies and its collective preference can be distilled into a single reward model.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Mobile Use Agent, GUI Agent, Vision Language Model, Reward Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_FcaKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces AndroidFlux, a benchmark for evaluating failure recovery in mobile GUI agents by resuming execution from checkpoints around an annotated first error. It also proposes Latent Peer Voting (LPV), which combines experts’ action likelihoods to generate preferences for reward-model training. The experiments report stronger recovery with agentic frameworks and GUI-oriented reasoning, better action ranking with LPV, and gains in recovery success from offline training on mixed normal and error contexts.

Critical strengths
  1. AndroidFlux provides a valuable benchmark for mobile-agent error recovery, with separate evaluations of recovery performance and recovery-action ranking. Its checkpoints around the first error, allow agent behavior to be examined under different recovery conditions.

  2. The experiments cover many agent families and settings. The results suggest that LPV improves action ranking, while training on mixtures of normal and error contexts can improve both recovery success.

Critical weaknesses
  1. Are the automatic error labels reliable enough to serve as benchmark ground truth, given the 90.8% step-level precision reported in L191? A mislabeled error would change the recovery problem being evaluated.

  2. Candidate-action scores mix action quality with the ability of the expert that continues the task. A strong expert may succeed despite a poor action, while a weaker expert, or one unfamiliar with the action, may fail after a useful one. LPV adds likelihood-based signals, but its voting weights still depend on task completion rates and thus on expert ability.

  3. Table 2 should clarify that tallt_{\mathrm{all}} and (t+1)all(t+1)_{\mathrm{all}} are computed over different sample sets and cannot be directly compared.

  4. What explains the large gap in task completion rates between replay and snapshot cases in Table 2? Why not restore all error contexts from snapshots to keep the loading procedure consistent?

  5. In Section 6.4, rewinding lets the target agent avoid an error it might not have made itself, so success need not involve recovery. Injection supplies the next action externally and does not test whether the agent can choose a recovery action on its own. The reported success rates may therefore overstate how well the agent can recover from its own mistakes in practice.

  6. Does off-policy error-fixing training improve the agent’s ability to recover from its own mistakes during execution? The clean-start gains in Section 6.6 do not directly answer this question, since higher task success rates could also come from making fewer errors.

  7. LPV uses five scorers, so its gain over a single scorer may simply come from ensembling. Table 6 should compare LPV with equal-weight averaging or majority voting using the same five models. The Naive baseline uses only the generator’s success rate and does not provide this comparison.

Questions for authors

see weaknesses

Reviewer_79rwOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an AndroidWorld-based benchmark that assesses mobile control agents’ failure recovery ability. It also introduces Latent Peer Voting, a method that leverages multiple models to score preferences over actions and obtain labels, alongside a reward-modeling training dataset comprising 1.8k first-error contents with candidate recovery actions and an RL training dataset comprising 8.5k contexts.

Using the proposed benchmark and datasets, the authors conduct experiments on major LLMs and mobile control frameworks, analyze the results, and draw conclusions.

This is a well-motivated paper with sound experiments, results, and analysis. I recommend acceptance.

Critical strengths
  • The motivation is clear and encouraging, addressing a precise pain point in current GUI agent research: failure recovery is one of the biggest challenges, and GUI agents’ poor performance in this area requires improvement. This research offers interesting perspectives and provides a benchmark/dataset as a starting point.
  • The dataset/benchmark construction and data collection are meticulously designed and executed. For example, the authors use both humans and LLMs as sources, leverage diverse models through Latent Peer Voting to reduce biases/outliers to some extent, and optimize for precision over recall to ensure dataset quality. Many of the conclusions align with experience from real use cases, including industrial applications of mobile control agents. For example, some models perform better from scratch but degrade when given an incorrect step, indicating a lack of error recovery ability.
Critical weaknesses
  • One major concern is the scale and diversity of this benchmark in at least two respects. First, it relies solely on AndroidWorld, constraining its diversity to that of AndroidWorld’s environments/tasks and inheriting their biases. Second, the benchmark contains only 100 recovery states across 26 AndroidWorld tasks, making it sensitive to the dominance of particular tasks.
  • Error attribution raises two concerns: (1) human bias and (2) the inherent difficulty of capturing certain errors in a single step. How are such errors captured to ensure that the error distribution is balanced and unbiased?
Questions for authors

How do the authors justify the real-world ecological validity of this work? AndroidWorld primarily uses open-source, non-mainstream, stable apps, rather than the top X real apps people use. These apps are very stable and may differ from real-world apps with dynamic content and failures. This dynamic nature is particularly important given the focus on error recovery.

Reviewer_QjHwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AndroidFlux, a benchmark for evaluating how mobile GUI agents recover from errors. It also proposes Latent Peer Voting to combine action preferences from multiple agents. Experiments show that recovery performance differs from clean-start success, LPV improves action ranking, and training on failure states can improve agent performance.

Critical strengths
  1. The paper addresses an important problem that is currently overlooked by existing GUI benchmarks. Instead of evaluating agents only from the initial state, AndroidFlux measures whether they can recover from errors made during task execution. The use of trajectory replay, restored device states, and standardized histories allows different agents to be compared under the same conditions.

  2. The recovery action benchmark uses actual task outcomes to determine action preferences, rather than relying solely on LLM judgments. The paper also provides experiments on agent evaluation, reward modeling, and offline RL. These results provide useful insights into agent recovery behavior and demonstrate the potential value of training on failure states.

Critical weaknesses
  1. The reliability and representativeness of the first error annotations need further validation. The LLM verified set relies on an automated detector and verifier, which achieves 90.8% precision but only 50.8% recall at the step level. While the verifier reduces false positives, the low recall may introduce selection bias, as some types of errors may be easier to detect than others. Recall also varies considerably across source models, from 70.1% for Qwen3-VL-2B to 37.8% for Mobile-Agent-v3-32B. The error type distributions differ substantially between the human verified and LLM verified sets, although the two sets also have different source model distributions. Furthermore, the reported detection metrics do not directly show how accurately the first error is located in the final 312 cases. These issues raise questions about whether the selected cases represent the broader distribution of agent failures and how reliable the resulting recovery evaluation is.

  2. The experimental evidence for LPV is still limited. The reward modeling benchmark contains only 100 preference pairs. LPV achieves 65% accuracy compared with 60% for the best individual expert, which amounts to only five additional correct predictions. The paper does not provide sufficient statistical analysis to establish how reliable this improvement is. After distillation, BT-LPV achieves 55% accuracy, only slightly higher than the original reward model at 52%. Although the authors suggest that LPV could be useful for policy optimization, the offline RL experiments use the original UI-Genie reward model rather than BT-LPV. It therefore remains unclear whether LPV can provide meaningful improvements beyond action ranking.

Questions for authors
  1. The detector has relatively low recall, and the recall varies significantly across source models. What causes these differences? Could the authors report precision and recall for different error types and difficulty levels? In particular, is there any relationship between how easily an error can be detected and how difficult it is to recover from?

  2. Since the verifier only examines errors proposed by the detector, earlier errors may still be missed. How often does the selected first error checkpoint agree with human annotations? Could the authors evaluate this on a manually reviewed subset of the 312 LLM verified cases?

  3. Could the authors provide more details about how the 312 cases were selected from the original agent trajectories? How many cases were discarded, and for what reasons? How do the error types and difficulty levels of the selected cases compare with those in the original failure trajectories?

  4. The improvement from LPV over the best individual expert is only five correct predictions on 100 pairs. Could the authors provide a statistical significance analysis or evaluate LPV on a larger set? Also, would using BT-LPV instead of the original UI-Genie reward model lead to better results in the offline RL experiments?

  5. All interactive experiments are conducted in AndroidWorld. Why was this environment chosen over other mobile GUI benchmarks such as AndroidLab and MobileWorld? Do the authors expect the findings to hold in other Android environments with different tasks and applications?

Reviewer_UMvoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents AndroidFlux, a failure-recovery benchmark built on AndroidWorld. Recorded trajectories are restored to checkpoints before, at, and after the first recoverable error (t−n, t, t+1+/t+1−), so that different agents can continue from the same device state and history; the history is stored schema-neutrally and re-rendered in each target policy's own format. The error recovery benchmark has a human-verified set of 80 cases from 6 source models and an LLM-verified set of 312 replay-based cases from 21 source models, whose first errors are labeled by a Gemini Flash 3.0 detector–verifier. The recovery-action reward modeling benchmark contains 100 pairs, labeled by the success counts of a twelve-agent panel after each candidate is executed. The authors also propose LPV, which scores peers' actions by length-normalized likelihood with performance-based weights, train a BT reward model on LPV labels, and release 1.8k reward-modeling and 8.5k offline RL contexts. Validation covers about 30 agents, pairwise accuracy on 100 pairs, and offline GRPO on Qwen3-VL-4B.

Critical strengths
  1. Takeover from a broken state is a useful evaluation target. The paper argues that "Whether a stronger agent can complete a failed task cannot be inferred from its clean-start success", and that in routing or step-level cascades the stronger model is handed "a foreign context and a device state it did not produce". I find this convincing, and no mobile benchmark measures it. Replay or snapshot restoration combined with per-policy re-rendering of the history (Appendix C.8) is what makes the setting testable. This is the main reason I see value in the paper.
  2. The outcome-grounded recovery labels expose a real gap. In the reward modeling benchmark, preference "follows from consequence rather than rule or judge": each candidate is executed and a fixed panel continues from the resulting state. With these labels, Table 6 shows that the UI-Genie BT model, which reaches 0.915 on UI-Genie, drops to 0.52 on AndroidFlux (0.486 on error-path pairs). This negative result is useful regardless of how well LPV works.
Critical weaknesses

1.The checkpoint comparisons are not made on the same episodes, so the main findings cannot be separated from selection effects. The t+1 conditions are only built for tasks where some model eventually recovered (52 of 65 human-verified tasks, 208 of 312 LLM-verified episodes), while t uses all cases. As a result, injecting the first action of a failed recovery still gives higher success than t for almost every model (e.g., Qwen3-VL-8B 29.49 vs. 43.27 on the LLM-verified set), which points to an easier subset. This affects the rebound result in Fig. 3 and the rewind-vs-injection contrast in Sec. 6.4, where the two gains are computed on 312 and 196 episodes respectively. The first-error labels have a related problem. The detector–verifier has 50.8 step-level recall and 71.0 trajectory-level accuracy (Table 24), and its prompt drops a candidate when a later screenshot shows the agent recovering (C.10), so whether a case is kept depends on what happened afterwards. The headline claim that recovery "only partially correlates" with clean-start success is also not quantified.

2.The evidence for LPV and for the offline RL results is too weak for the claims in the abstract. On 100 pairs, LPV is five pairs above the best single voter, and BT-LPV is three pairs above its initialization, while half of the pairs sit at the margin-2 release threshold (Table 27). α appears to be chosen on the same test pairs, and the voters are also the candidate generators and panel members. For RL, every run uses the UI-Genie reward model that is near chance on error-path pairs. The gain may therefore come from the terminate guard, which uses Gemini labels only for AndroidFlux contexts. Each mixture has one seed, and the best mixture improves error-t by three tasks over the base model, which is smaller than the four-task gap the authors report between two seeds (E.2).

Questions for authors
  1. Compared to GUI-RobustEval, what conclusions from AndroidFlux are either unobtainable in a desktop environment or contradict those found in that context?
  2. How would the recovery rate change if the target agent were explicitly informed—at the moment of takeover—that "previous execution attempts have failed"? Would this approach be more realistic for routing or cascaded execution scenarios?
  3. How does LPV compare to the following baselines: (a) using Gemini 3.x or GPT-5.x as a VLM judge; (b) majority voting on canonicalized actions; (c) existing GUI PRMs; and (d) a likelihood-ratio variant that subtracts the likelihood of a reference/base model?
  4. Does the +12.22pp comparison presented in Figure 5 and Section 6.4 still hold when calculated across the same set of episodes for all conditions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.