acceptodds
Withdrawn from ICLR 2027

World Model-Induced Dual-View Inconsistency for GUI Agent Trajectory Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Benefiting from large-scale, high-quality interaction trajectories, GUI agents have achieved substantial progress in perception, planning, and interaction. As trajectory synthesis and automated collection scale up, data quantity is becoming less of a bottleneck, while trajectory quality is increasingly important. However, large-scale trajectory collections often contain substantial redundancy and quality variation, making reliable supervision difficult to identify.Trajectory evaluation is further complicated by rich interface and state information and unexpected factors during execution. Existing filtering methods mainly assess realized state transitions and cannot effectively distinguish action quality from deviations in execution outcomes, potentially discarding valuable supervision.Since identifying whether a realized transition matches the expected outcome requires predicting future states, we introduce WM-IncPRM, a world-model-guided inconsistency-aware process reward framework for GUI trajectory evaluation. WM-IncPRM combines a world model with PRM to predict future semantic transitions and compare them with realized transitions. Reliable and supported prediction–reality discrepancies are used to calibrate step-level quality, which is further aggregated with recoverable supervision coverage to classify trajectories into Keep, Repair, or Reject. Experiments demonstrate that WM-IncPRM improves trajectory filtering quality and downstream performance on AndroidWorld.We will release the source code on GitHub in the future.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mobile World Model, Process Reward Model, GUI Agent

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_1A8xOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes WM-IncPRM, a method for GUI trajectory evaluation and filtering that combines a world model with a PRM. By comparing instruction-conditioned predictions, action-conditioned predictions, and actual state transitions, the method identifies actions that may be undervalued due to environmental failures. It then uses Support scores from a real-transition bank and Reliability scores from the world model to control score recovery and then combines step quality with the coverage of recoverable supervision to classify trajectories into three categories.

Critical strengths
  1. By incorporating world-model predictions as an additional reference, the framework attempts to distinguish action quality from execution outcomes.
  2. The method not only compares predicted and actual transitions, but also introduces two signals, Support and Reliability, to guide score recovery. It further combines step quality with the coverage of recoverable steps to filter trajectories at the trajectory level.
Critical weaknesses
  1. The paper argues that environmental failures can cause the PRM to undervalue correct actions, and that the corresponding scores should therefore be recovered to preserve useful supervision. This motivation is reasonable, but the current evidence does not fully establish the chain of reasoning from action undervaluation to the value of the supervision and, ultimately, to improved training. Trajectories with environmental errors may contain both correct and incorrect actions. Increasing their retention rate does not necessarily mean that more reliable supervision is being preserved. The reported EEPR primarily measures the retention rate of trajectories with environmental errors. It does not, by itself, reflect the accuracy of step-level recovery or quantify the risk of incorrectly recovering scores for erroneous actions.
  2. The experiments use OpenMobile for filtering and training, and evaluate downstream performance on AndroidWorld. However, the paper does not provide a clear evaluation on unseen applications or under low coverage of the real-transition bank. It therefore remains unclear how much the method's gains depend on coverage of existing applications and transition examples, and whether the gains persist on new applications or under different data collection distributions. The current experimental scope is insufficient to fully establish the method's applicability as a general-purpose trajectory evaluation approach.
Questions for authors
  1. Could the authors independently annotate step-level action correctness, quantify how much the PRM undervalues actions that are correct in themselves but fail due to environmental issues, and report both the accuracy of score recovery and the rate of erroneous actions being incorrectly recovered?
  2. Please clarify the overlap in applications and trajectories among OpenMobile, the real-transition bank, and AndroidWorld. Could the method also be evaluated on benchmarks beyond AndroidWorld to assess its generalization to unseen applications?
Reviewer_dPiSOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that PRMs scoring only realized transitions underestimate correct actions affected by environmental failures in mobile GUI trajectories. It proposes WM-IncPRM, which compares world-model predictions with realized transitions and recovers step scores based on the discrepancy, weighted by Support and Reliability, before classifying trajectories into Keep/Repair/Reject. On AndroidWorld, training on the 20% Keep subset reaches 26.0% success, versus 24.1% for the strongest baseline.

Critical strengths
  • A low realized score does not necessarily mean a wrong action, and discarding such steps can waste useful supervision. Figure 1 and the case studies in Figures 5–6 illustrate this problem clearly.
  • Using world-model predictions as a reference for what should happen after an action is a reasonable idea, and the framework requires no additional training on top of the frozen PRM and WM.
  • Evaluating both Keep and Reject subsets downstream gives a more complete view of filtering quality than Keep-only evaluation.
Critical weaknesses
  • Under the same 20% retention, Table 2 reports 26.0% success (30.2/116) for WM-IncPRM Keep, while Table 8 reports 7.76% (9/116). In Table 8, success increases monotonically with retention and peaks at 100%, which contradicts the claim in Section 5.1 that selecting higher-quality data is more effective than adding more data. Moreover, 100% retention should match Random 100% in Table 2 (16.2%), yet Table 8 reports 26.72%.
  • Δ in Eq. 10 includes r_exp, which comes from the instruction-conditioned branch and does not depend on the executed action. A wrong action with low r_pred can therefore still get a large Δ and be recovered, contrary to Section 3.4’s claim that erroneous rescue is prevented. Figure 6 is such a case: Δ = max(0.6, 1.0) − 0.0 is driven by r_exp. In addition, the thresholds τ_q and τ_w in Section 3.5 do not appear in Eq. 14, and Figure 5 recovers a step with w = 0.631 < τ_w = 0.82.
  • The PRM’s expected_change guides the WM prediction, and the same PRM then scores that prediction, so r_exp and r_pred are likely inflated by construction. Support measures similarity to retrieved real transitions and Reliability measures consistency across samples, so neither can detect a confidently wrong prediction. There is no direct evaluation of whether recovered steps are actually environmental failures, such as recovery precision against the human annotations.
  • With 3,884 trajectories, a 20% budget is about 777. However, the Reject sets of AgentTrek-VLM, AgentRM, and GUI-Critic-R1 contain only 681, 577, and 724 trajectories (Table 1), so the equal-sized Reject comparison in Table 3 cannot be built as described. AOC, the main ranking metric in Table 1, is never defined. The gain over AgentTrek-VLM is 1.9 points (about two tasks out of 116), with no significance test.
Questions for authors
  • Which result is correct for 20% retention, Table 2 or Table 8? Why does 100% retention in Table 8 differ from Random 100% in Table 2, and how is the trend in Table 8 consistent with the quality-over-quantity claim?
  • Why is r_exp included in Δ, and how does the method avoid recovering wrong actions when r_exp is high? Are τ_q and τ_w actually used, and is w computed as in Eq. 13 or as in Figures 5–6?
  • According to the human annotations, what fraction of recovered steps are true environmental failures?
  • How are the 20% Reject subsets built for baselines whose Reject sets are smaller than the budget? How are the binary outputs of AgentTrek-VLM and URST ranked? How is AOC defined?
Reviewer_FoowOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes WM-IncPRM for offline GUI trajectory evaluation and filtering. It combines instruction-conditioned and action-conditioned world-model predictions with observed transitions, using a PRM to score all three views. Discrepancies between these scores, weighted by support from retrieved real transitions and consistency across predictions, are used to increase step-level scores. The resulting scores are aggregated to classify trajectories into Keep, Repair, or Reject. Experiments filter OpenMobile trajectories and evaluate Qwen3.5-4B agents trained on the selected data using AndroidWorld.

Critical strengths
  1. The motivation is reasonable: in GUI environments, execution failures (e.g., network errors or UI freezes) can make a valid action look poor under realized-transition evaluation. Testing filtering quality through downstream agent training under a fixed data budget and five seeds is also a sensible evaluation protocol in principle.
Critical weaknesses
  1. The recovery rule can reward incorrect actions, and its implementation is unclear.

With r_exp = 1, r_pred = r_real = 0, and q = w = 1, Equations (10)–(14) raise the step score to 1. Both the predicted and observed consequences receive zero scores; the recovery comes entirely from the instruction-conditioned expectation, without evidence of an environmental failure. Support and prediction consistency do not establish that an action is correct.

The implementation is also inconsistently described. Equation (13) defines Reliability through repeated action-conditioned predictions, while Figures 5–6 use instruction–action agreement. Figure 5 applies recovery at w = 0.631, below the stated threshold of 0.82. The full gating rule and aggregation of multiple predictions into PRM scores remain unspecified. These choices affect the selected training data, making it unclear which procedure the experiments validate.

  1. The evaluation is incomplete and difficult to reproduce.

Repair contains over half the trajectories, but its downstream training value is not evaluated separately. EEPR reports retention of environmental-error trajectories without measuring how often genuinely incorrect actions are also recovered. The Keep results therefore leave important parts of the claimed recovery mechanism untested.

The experimental reporting also needs clarification. Table 8's 70% row lists 26 successes and 94 failures across 116 tasks, with a success rate of 18.97%; these numbers cannot all be correct. Tables 2 and 8 also leave the retention denominators and exact training-set sizes unclear.

Finally, AOC lacks a concrete definition, and key filtering parameters, the semantic similarity computation, and the transition bank's source and data-separation rules are not sufficiently specified. These gaps prevent a reliable assessment and reproduction of the reported results.

Questions for authors
  1. What exact recovery rule was used to produce the reported results? Does it include an action-quality gate beyond Equation (11), and how does it handle the counterexample above? If the rule needs modification, would the selected training data and downstream results change?

  2. Which definition of Reliability was used: repeated action-prediction consistency or instruction–action agreement? How is Figure 5's recovery with w < tau_w consistent with the implementation? Please also explain how the M predictions are aggregated into the PRM scores.

  3. How are Repair steps converted into training examples, including their historical context? Are there results showing their benefit beyond Keep-only training? What fraction of the steps receiving increased scores are genuinely incorrect actions rather than appropriate actions affected by environmental failures?

  4. Please provide the corrected 70% row of Table 8 and clarify the denominator and exact training-set size for each setting in Tables 2 and 8.

  5. How is AOC computed? Please specify M, K, tau_a, tau_c, beta, and tau_step, the semantic similarity and retrieval methods, and the transition bank's source and exclusion rules. What is the total filtering cost compared with PRM-only selection?

  6. How does the proposed recovery mechanism differ from direct action evaluation in UI-Genie [1] and world-model-based consequence assessment in WAC [2]? It would also help to clarify how the trajectory-level evaluation relates to Plan-RewardBench [3], particularly the differences in evaluation targets and treatment of planning errors and error recovery.

References

[1] Xiao et al. UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents. 2025.

[2] Shen et al. World-Model-Augmented Web Agents with Action Correction. 2026.

[3] Wang et al. Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling. ACL 2026.

Reviewer_D8TvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes WM-IncPRM for filtering offline mobile GUI agent trajectories. For each step, a world model predicts the next state twice: once from the task instruction and once from the executed action. A PRM then scores the predicted and the real next states. When the real score is lower than the predicted one, the step score is raised, weighted by a Support score and a Reliability score. Trajectories are then labelled Keep, Repair, or Reject. The paper evaluates on OpenMobile trajectories and fine-tunes Qwen3.5-4B, testing on AndroidWorld.

Critical strengths
  • The goal of finding useful supervision in failed trajectories is fair and makes sense. A failed trajectory does not mean all of its actions are wrong (L171–173).
  • The idea of studying the gap between the imagined outcome and the actual outcome, to evaluate actions separately from environmental errors, is interesting.
Critical weaknesses

1. No comparison on standardized benchmarks or with published numbers. All results come from the authors' own internal runs. That is fine by itself, but for credibility, and to place this work in the existing literature, it should be evaluated on standardized benchmarks. This makes me question the presented results. The evaluation the authors ran may somehow favor their method, so it does not necessarily show the significance of the proposed method.

  • There is no base-model reference. The paper reports no zero-shot Qwen3.5-4B number. Its model card reports 58.6 on AndroidWorld, while the paper's best is 26.0 ± 1.1 (Table 2) / 26.72 (Table 8).
  • There is no comparison with published AndroidWorld results. For example, OpenMobile (arXiv 2604.15093, Table 1) uses the same source dataset and reports 51.7 (7B) and 64.7 (8B).
  • Only one benchmark, AndroidWorld, is used. The training corpus covers the same 116 AndroidWorld task templates.
  • The comparisons are internal only. The baselines are re-implemented by the authors, and their configurations are not described.

The authors' own runs and tables could then follow the standard-benchmark results, to show the additional points they want to make.

2. The introduction does not build on the literature. The introduction has no citations at all, even for things it names directly: "conventional filtering methods" (L047), OpenMobile (L069), AndroidWorld (L070, L085), and "PRM-based mobile trajectory filtering" (L075). It needs references to prior work to support its motivation, for example works that use PRM-based mobile trajectory filtering (URST, OS-Genesis TRM, AgentTrek, UI-Genie-RM) and AndroidWorld. This is needed even if they are cited later in the related work. The introduction should be rewritten and reframed to show how this work builds on the existing literature in the field.

3. Figures are not referenced, explained or integrated.

  • Figures need to be referenced in the text and integrated into the presentation. Figure 1 and Figure 2 are never referenced in the text, and Figures 3–7 are referenced only in the appendix.
  • Figure 1 needs to be explained, in the caption or in the text. Its caption is a single sentence.
  • Figure 2 (method overview) is never used in the methodology section, and even its caption does not explain it (it does not cover panels (1)–(4) or the numbers shown). The reader is left to walk through it alone, which is a significant flaw in the presentation. Figure 2 needs to be revisited, explained well in the methodology, and checked so that it presents the idea in a simple, informative way.
Questions for authors

1. It is unclear how the contribution differs from prior work. Up to L143, the presentation does not make clear how the contribution builds on existing literature, how WM-IncPRM addresses the gap ("action-conditioned predictive view"), and how it is necessarily different from existing work. This makes the paper harder to read. For example, GUI-Critic-R1, one of the baselines, already judges an action before it is executed, and Sec. 2.2 does not discuss it.

2. The core premise is not motivated (L173–176). The claim that "a PRM that evaluates only the realized transition can assign a low score to such a step because the observed state shows little or no task progress" needs to be motivated. The authors could give examples from existing works, or explain why this happens in existing works, in the introduction and/or the related work. There is no citation or measurement for it at this point.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.