- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on the reliability of reward evaluation and the long-term trajectory credit allocation issues in GUI Agent reinforcement learning, and presents three main contributions. Firstly, an OS-Themis multi-agent reward evaluation framework is proposed. Through key Milestone selection, visual evidence verification, and multi-agent review, reliable trajectory-level rewards and Milestone verification results are generated. Secondly, an MC-GRPO optimization algorithm is proposed. By further utilizing the Milestone information generated by OS-Themis, the Advantage of the standard GRPO is finely re-weighted, thereby improving the credit allocation within the trajectory. Additionally, the author has constructed OGRBench, a benchmark across five types of GUI platforms, for evaluating the trajectory-level judgment ability of the reward model. In terms of experiments, the author verified the effectiveness of OS-Themis through multi-model, multi-platform comparisons and ablation experiments, and verified the additional performance benefits of MC-GRPO through the AndroidWorld online reinforcement learning experiment. At the same time, scale expansion experiments demonstrated the scalability of the overall solution.
Critical strengths
The research topic holds significant practical importance. The paper focuses on the problems of unreliable reward evaluation and difficult long-trajectory credit allocation in GUI Agent reinforcement learning, and has strong research value. There is a natural connection between OS-Themis and MC-GRPO, and the Milestone information obtained during the reward evaluation process is further utilized for strategy optimization. The overall research idea is relatively complete.
The experimental coverage is extensive, and the performance improvement is quite convincing. The authors evaluated on multiple GUI platforms and different-sized base models, and verified the practical effect of the method through online reinforcement learning experiments. OS-Themis overall outperforms existing reward evaluation methods on OGRBench, and MC-GRPO also achieves an additional 2.5–2.9 percentage points of performance improvement compared to standard GRPO, indicating the practical effectiveness of the proposed solution.
The ablation experiments are quite rich. The authors analyzed the roles of components such as Selector, Verifier, and Reviewer, and verified the effectiveness of the key step selection mechanism through experiments such as random replacement and deletion of some Milestones. These experiments provide certain empirical support for the design of different components in the framework.
It has certain systematic and engineering value. The authors constructed the cross-platform reward evaluation benchmark OGRBench and built an Android online reinforcement learning system that supports parallel interaction and asynchronous reward evaluation. The scale expansion experiments and trajectory filtering experiments further demonstrated the application potential of this framework in actual GUI Agent training. In addition, the overall writing of the paper is clear, the organization is reasonable, and the appendix provides relatively detailed implementation explanations.
Critical weaknesses
OS-Themis mainly accomplishes the key steps selection, verification, and final reward judgment through the collaboration of multiple evaluation Agents. While MC-GRPO essentially builds upon the standard GRPO framework and reweights the original Advantage based on the Milestone verification results. The overall approach is somewhat related to the existing research on process rewards and fine-grained credit allocation, but the methodological innovation is relatively limited.
The key design of MC-GRPO lacks more targeted experimental verification. Although the authors demonstrated the effectiveness of MC-GRPO through comparisons with standard GRPO and ADMIRE, they have not fully explained the advantages of its specific weight function over other simple reweighting strategies. The existing experiments can prove the practical value of the Milestone-guided credit allocation, but they are not sufficient to fully demonstrate the necessity and independent contribution of the weight function itself.
The data quality verification of OGRBench still has room for improvement. This Benchmark has certain cross-platform coverage capabilities, and the data sources and construction processes are relatively clear. However, the paper does not provide sufficient details on the specific standards for manual review, sample selection process, and label quality control. Considering that OGRBench is one of the main contributions of the paper, a more systematic analysis of data reliability will help enhance its persuasiveness as an independent evaluation benchmark.
The reliability of the Milestone supervisory signals lacks direct verification. The OGRBench proposed in the paper mainly provides the final task success label for the complete trajectory to evaluate the accuracy of the reward model's trajectory-level judgment. However, MC-GRPO further relies on the key Milestones selected from the trajectory and their verification results to allocate the Advantage for different steps. Although the authors indirectly proved the effectiveness of these information through Milestone selection ablation experiments and downstream reinforcement learning experiments, they have not directly evaluated the accuracy of the Milestone selection and verification results themselves. Therefore, it is still unclear how reliable these fine-grained credit allocation signals are, and to what extent potential Milestone judgment errors will affect the optimization effect of MC-GRPO.
Questions for authors
MC-GRPO reweighted the Advantage of the standard GRPO based on the verification results of Milestone, achieving certain performance improvements. However, it is not clear to what extent this improvement is attributed to the specific weight function proposed by the authors. Can the authors further compare this weight function with simpler Milestone weighting strategies (such as fixed weights or linear reweighting) and verify whether it has additional advantages under the same Milestone information and normalization conditions?
The paper validated the effectiveness of Milestone information through ablation experiments on Milestone selection and downstream reinforcement learning results. However, the credit allocation of MC-GRPO not only depends on the selection of key steps but also directly on the Verifier's judgment of the success or failure of each Milestone. Existing experiments mainly evaluated the impact of these information on the final task judgment and training effect, but have not directly measured the accuracy of the Milestone-level verification results themselves. Can the authors provide corresponding validation accuracy analysis or further discuss the impact of Milestone judgment errors on the performance of MC-GRPO?
OGRBench integrates the execution trajectories of multiple existing GUI Benchmarks, providing a unified test benchmark for cross-platform reward evaluation. However, the unique evaluation value of this paper compared to existing Reward Benchmarks can still be further clarified. Additionally, Appendix C.2 shows that OS-Themis improved Precision on AgentRewardBench, but Recall significantly decreased. Can the authors further analyze the reasons for this performance difference?
4 comments
Response to Weakness 2 and Question 1: Effectiveness of the MC-GRPO Weighting Function
We thank the reviewer for this constructive suggestion. To isolate the contribution of our weighting function, we conducted additional controlled ablations comparing MC-GRPO with simpler milestone-based weighting strategies.
Using Qwen3-VL-4B as the policy backbone, all variants reuse the same rollout trajectories and OS-Themis evaluation results, with identical milestone selections, verification outcomes, terminal rewards, training configurations, and token-level advantage normalization. Only the weighting strategy varies:
-
Fixed Weights: Assigns weights of 1.2 and 0.8 to milestones whose verification outcomes agree and disagree with the terminal outcome, respectively.
-
Fixed Strength: Replaces the trajectory-adaptive factor in Eq. with a fixed coefficient shared across trajectories, retaining the original outcome-conditioned, centered weighting structure.
-
Shuffled Milestone Labels: Randomly permutes milestone verification outcomes within each trajectory while retaining the original weighting function and normalization.
The results are summarized below :
| Method | Accuracy |
|---|---|
| GRPO | 51.3 ± 1.3 |
| MC-GRPO (Shuffled Milestone Labels) | 50.3 ± 1.7 |
| MC-GRPO (Fixed Weights) | 52.3 ± 1.3 |
| MC-GRPO | 52.7 ± 1.1 |
| MC-GRPO (Ours) | 54.2 ± 0.9 |
Shuffling milestone labels reduces accuracy to 50.3%, below standard GRPO , highlighting the importance of milestone-action correspondence. Both simpler weighting strategies outperform GRPO but remain below MC-GRPO , with gains of 1.9 and 1.5 percentage points over Fixed Weights and Fixed Strength, respectively. In particular, the Fixed Strength comparison highlights the benefit of adapting credit redistribution strength to each trajectory rather than using a constant coefficient.
Response to Weakness 3 and Question 3: OGRBench Data Quality, Benchmark Contributions, and AgentRewardBench Generalization
We thank the reviewer for the constructive comments. We clarify three aspects of OGRBench and its evaluation results.
1. Data quality control. As described in Appendix G, OGRBench collects real GUI trajectories from five established benchmarks, retaining original environment-based success labels. We manually screen potentially unreliable trajectories based on three criteria:
-
Ambiguous instructions: Task success criteria cannot be reliably determined.
-
Environment-related failures: Persistent environment or infrastructure issues make tasks infeasible, rather than agent errors.
-
Incomplete records: Missing or corrupted screenshots, interaction logs, or other essential records prevent reliable evaluation.
These procedures distinguish genuine agent failures from task ambiguity, environment issues, and data corruption. We further employ stratified sampling to preserve task diversity and balance label distribution. The resulting benchmark contains 1,409 trajectories, including 700 positive and 709 negative samples. We will clarify these screening criteria and quality control procedures in the revised manuscript.
2. Distinctive value of OGRBench. Beyond aggregating existing trajectories, OGRBench provides a unified evaluation setting for GUI outcome reward models across heterogeneous platforms, tasks, and agent behaviors. As shown in Table 10, existing benchmarks have more limited platform coverage: AgentRewardBench focuses on Web, while CUARewardBench primarily covers Ubuntu. In contrast, OGRBench covers five platforms: Ubuntu , Android , Windows , macOS , and Web . Its trajectories are generated by diverse GUI agents, including Qwen3-VL, UI-TARS, ScaleCUA, and Claude-Sonnet-4.5. Combined with consistent outcome annotations and a unified evaluation protocol, this diversity enables systematic evaluation of reward generalization across platforms and agent-generated trajectories.
3. Recall reduction on AgentRewardBench. To investigate this issue, we excluded trajectories generated by Claude-3.7-Sonnet, many of which lack explicit action rationales. The resulting subset contains 951 trajectories (243 positive and 708 negative). OS-Themis achieves 75.66% Precision and 70.37% Recall, improving Recall by 15.97 percentage points over the full benchmark while maintaining comparable Precision .
This supports our analysis in Appendix C.2 that incomplete action-level information can hinder milestone construction and verification, making it more difficult for the Selector to identify intended subgoals and formulate appropriate verification targets. In our primary application, GUI online RL, action descriptions and interaction histories can be recorded directly during trajectory collection, providing the contextual information required by OS-Themis.
Furthermore, as discussed in Appendix A, reward precision is particularly important for policy optimization, since false-positive rewards may reinforce incorrect behaviors and introduce misleading training signals. OS-Themis therefore prioritizes precision while maintaining sufficient recall for effective learning. The subset results show that substantially higher Recall can be achieved while maintaining high Precision when more informative trajectory records are available.
Response to Weakness 4 and Question 2: Reliability of Milestone Supervision and Its Impact on MC-GRPO
We thank the reviewer for raising this concern. To directly evaluate milestone-level supervision, we conducted a human evaluation of 1,219 milestones from 200 trajectories. We sampled 100 trajectories from actual online RL rollouts and 100 from OGRBench, all evaluated by OS-Themis . Annotators familiar with GUI environments were provided with the task goal, assignment goal, and before-and-after screenshots to assess milestone outcomes against the Verifier predictions.
The results are summarized below (accuracy is measured by agreement with human annotations):
| Data Source | Trajectories | Milestones | Correct | Incorrect | Accuracy |
|---|---|---|---|---|---|
| Online RL Rollouts | 100 | 414 | 371 | 43 | 89.6% |
| OGRBench | 100 | 805 | 726 | 79 | 90.2% |
| Overall | 200 | 1,219 | 1,097 | 122 | 90.0% |
The overall agreement with human annotations reaches 90.0%, with comparable accuracy on online RL rollouts and OGRBench , directly supporting the reliability of the milestone signals used by MC-GRPO.
Regarding the impact of milestone judgment errors on policy optimization, MC-GRPO constrains their influence through outcome-conditioned advantage redistribution. The terminal reward and trajectory-level GRPO advantage remain unchanged, while milestone outcomes only adjust the relative magnitudes of token-level advantages. The normalization in Eqs. – preserves each token's advantage sign and the total signed advantage within each trajectory. Thus, an incorrect milestone verdict affects relative credit allocation without changing the terminal reward or directly reversing the corresponding token-level optimization signal.
To further examine the importance of accurate milestone-action correspondence, we conducted an additional training ablation using the same Qwen3-VL-4B policy backbone, rollout trajectories, and OS-Themis evaluation results. Randomly shuffling milestone labels within each trajectory, while retaining the original weighting and normalization, reduces AndroidWorld accuracy from 54.2% to 50.3%, below standard GRPO . This highlights the importance of preserving meaningful correspondence between milestone outcomes and their associated actions.
Table 8 provides further evidence: removing 25% of selected milestones reduces Recall by 11.16 percentage points and F1 by 8.03 percentage points, while random and final-step replacement also degrade outcome evaluation performance.
Response to Weakness 1: Methodological Novelty and Contributions
We thank the reviewer for the opportunity to clarify our contributions. The novelty of OS-Themis and MC-GRPO lies in their overall integration rather than individual components in isolation. Our work unifies reliable reward construction, fine-grained trajectory understanding, and policy optimization into a framework for GUI agent reinforcement learning.
1. Research significance and practical contribution. Reliable and generalizable reward acquisition remains a major bottleneck in GUI online RL. Long trajectories, complex state transitions, and heterogeneous environments make accurate reward evaluation difficult without task-specific rules or specialized reward models. OS-Themis addresses this challenge by extracting task-relevant evidence directly from GUI trajectories to generate outcome rewards without environment-specific evaluation scripts or successful reference trajectories. We further integrate it into a scalable Android online RL infrastructure, connecting trajectory collection, reward evaluation, and policy optimization. Its effectiveness is demonstrated through cross-platform evaluation on OGRBench, online RL experiments, and trajectory filtering.
2. Methodological innovation through unified architecture and structured reward construction. Rather than simply combining multiple evaluators or modifying GRPO, OS-Themis establishes a unified evidence-driven architecture that connects global outcome evaluation with local process supervision. Conventional outcome-based methods provide global task feedback but limited fine-grained guidance, whereas process-based methods provide intermediate signals that may not reflect final task completion. OS-Themis uses the same verified milestone evidence for both purposes, establishing consistency between reward determination and policy supervision.
More importantly, OS-Themis systematically organizes long GUI trajectories into task-relevant, visually verifiable state transitions, with iterative evidence auditing to identify missing requirements and distinguish intermediate progress from actual task completion. This transforms reward evaluation from coarse trajectory-level prediction into structured evidence construction and verification. The ablations in Tables 3 and 8, together with the matched-call-budget comparison in Table 14, support the effectiveness of this evidence organization.
Crucially, MC-GRPO further connects reward construction with policy optimization. Instead of discarding milestone evidence after generating the terminal reward or introducing additional process rewards, MC-GRPO reuses the verified milestones to redistribute the original GRPO advantage across actions. It preserves the terminal optimization direction and total signed advantage while providing fine-grained credit assignment, without additional evaluator calls for this purpose. This shared evidence representation avoids relying on independently constructed and potentially inconsistent supervision signals. As shown in Table 2, MC-GRPO further improves AndroidWorld performance by 2.5–2.9 percentage points over standard GRPO using the same OS-Themis evaluator.