- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes WM-IncPRM, a method for GUI trajectory evaluation and filtering that combines a world model with a PRM. By comparing instruction-conditioned predictions, action-conditioned predictions, and actual state transitions, the method identifies actions that may be undervalued due to environmental failures. It then uses Support scores from a real-transition bank and Reliability scores from the world model to control score recovery and then combines step quality with the coverage of recoverable supervision to classify trajectories into three categories.
Critical strengths
- By incorporating world-model predictions as an additional reference, the framework attempts to distinguish action quality from execution outcomes.
- The method not only compares predicted and actual transitions, but also introduces two signals, Support and Reliability, to guide score recovery. It further combines step quality with the coverage of recoverable steps to filter trajectories at the trajectory level.
Critical weaknesses
- The paper argues that environmental failures can cause the PRM to undervalue correct actions, and that the corresponding scores should therefore be recovered to preserve useful supervision. This motivation is reasonable, but the current evidence does not fully establish the chain of reasoning from action undervaluation to the value of the supervision and, ultimately, to improved training. Trajectories with environmental errors may contain both correct and incorrect actions. Increasing their retention rate does not necessarily mean that more reliable supervision is being preserved. The reported EEPR primarily measures the retention rate of trajectories with environmental errors. It does not, by itself, reflect the accuracy of step-level recovery or quantify the risk of incorrectly recovering scores for erroneous actions.
- The experiments use OpenMobile for filtering and training, and evaluate downstream performance on AndroidWorld. However, the paper does not provide a clear evaluation on unseen applications or under low coverage of the real-transition bank. It therefore remains unclear how much the method's gains depend on coverage of existing applications and transition examples, and whether the gains persist on new applications or under different data collection distributions. The current experimental scope is insufficient to fully establish the method's applicability as a general-purpose trajectory evaluation approach.
Questions for authors
- Could the authors independently annotate step-level action correctness, quantify how much the PRM undervalues actions that are correct in themselves but fail due to environmental issues, and report both the accuracy of score recovery and the rate of erroneous actions being incorrectly recovered?
- Please clarify the overlap in applications and trajectories among OpenMobile, the real-transition bank, and AndroidWorld. Could the method also be evaluated on benchmarks beyond AndroidWorld to assess its generalization to unseen applications?