- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes OTAD, a method for credit assignment in long-horizon agent. It uses a VLM to construct hindsight for scoring individual actions. These scores guide reweighting of advantages within GRPO. The scores also guide budget-constrained turn selection across different action types and adaptive task sampling. On OSWorld-Verified, OTAD achieves a substantial improvement over budget-matched GRPO. Ablation studies support the contributions of transition information and credit weighting.
Critical strengths
- This method is well-motivated. I can understand it clearly. Credit assignment is particularly challenging in long-horizon CUA tasks. I find several details reflect careful designs from authors, such as the nonnegative weights and fallback handling for uncertain scores.
- The experiments support the effectiveness of the method. OTAD improves success on OSWorld-Verified over budget-matched GRPO, and the ablations support the contributions of transition information and credit weighting.
Critical weaknesses
- OTAD introduces bias in GRPO. Although the reported results are promising, it remains unclear whether the gains would persist with larger models or longer training runs. This limits confidence in the method’s effectiveness at scale.
- Hindsights are well established in RL. A clearer discussion of prior work, such as Hindsight Credit Assignment, would help distinguish OTAD’s specific methodological contribution.
Questions for authors
-
Could the authors clarify how OTAD differs methodologically from prior hindsight-based credit-assignment approaches, particularly which aspects of the ordinal scoring and advantage reweighting constitute the main novelty?
-
Given the bias introduced by turn-level advantage reweighting, do the authors have existing evidence that OTAD’s gains remain stable over longer training? Do the reported learning curves indicate that both OTAD and the GRPO baseline have reached a performance plateau?
4 comments
Q1 / W2. Difference from hindsight credit assignment
Section 2 now discusses Hindsight Credit Assignment (HCA; Harutyunyan et al., 2019), counterfactual credit assignment (Mesnard et al., 2021) and HER (Andrychowicz et al., 2017).
How OTAD differs from HCA. HCA learns a hindsight distribution and uses the advantage , which is unbiased when is exact. By Bayes' rule, , so with a binary verifier HCA needs calibrated success probabilities before and after each action. Section 3.1 explains why these are hard to obtain in GUI environments.
| HCA | OTAD | |
|---|---|---|
| What is judged | likelihood of the action, given the state and the return | the observed transition (pre-screen, action, post-screen), given task and outcome |
| Source of credit | a hindsight model trained with the policy | a frozen VLM; nothing extra is trained |
| How credit enters the update | ratio-corrected advantage of either sign | non-negative weight on the verifier-signed advantage |
HCA-style baseline (Appendix F.10). Our variant obtains the hindsight probabilities without a learned hindsight model: the frozen 27B scorer estimates success probabilities before and after each transition, the ratio above is formed from them, clipped at a threshold , and used as , with OTAD's turn selection and task sampling. The best threshold reaches 45.2%, against 43.8% for unit credit and 52.1% for OTAD. A frozen VLM's ordinal ranking is thus more useful than its probability estimates, which are not calibrated.
Main novelty. (i) Turn credit from a frozen VLM's ordinal, outcome-conditioned judgment of the observed pre/post transition, which needs neither action likelihoods nor calibrated probabilities; removing the post-action screen costs 6.8 points (Table 8). (ii) An outcome-conditioned mapping from these judgments to a non-negative weight on the verifier-signed advantage. As Section 2 notes, concurrent work (Xu et al., 2026; Huang et al., 2026) also lets evidence set only the update strength. In OTAD the evidence is the visible interface change, judged with the outcome known, and the mapping itself depends on the outcome, so the same score is weighted differently in successful and failed trajectories (48.9% with an outcome-independent mapping, Table 13). (iii) Reusing the same judgments for turn selection and task sampling (−3.9 and −2.7 points when removed, Table 5).
To make the presentation easier to follow, the revision adds a notation table (Table R5, Appendix A) and a worked numerical example of the mapping (Section 3.2).
We hope the extended runs and the clarified relation to hindsight methods address the reviewer's questions.
References
- Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. Hindsight experience replay. In NeurIPS, 2017. arXiv:1707.01495.
- Harutyunyan, A., Dabney, W., Mesnard, T., Azar, M. G., Piot, B., Heess, N., van Hasselt, H., Wayne, G., Singh, S., Precup, D., and Munos, R. Hindsight credit assignment. In NeurIPS, 2019. arXiv:1912.02503.
- Huang, M., Xue, T., Peng, C., Ding, J., Yang, J., Fan, S., Hong, J., Gao, Y., Zhang, X., Guo, L., Yang, X., et al. EvoCUA-1.5: Online reinforcement learning for multi-turn computer-use agents. arXiv:2607.09773, 2026.
- Mesnard, T., Weber, T., Viola, F., Thakoor, S., Saade, A., Harutyunyan, A., Dabney, W., Stepleton, T., Heess, N., Guez, A., Moulines, É., Hutter, M., Buesing, L., and Munos, R. Counterfactual credit assignment in model-free reinforcement learning. In ICML, PMLR 139:7654–7664, 2021. arXiv:2011.09464.
- Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300, 2024. (GRPO)
- Xu, Y., Zhou, Z., Sang, H., Li, X., Zhang, J., Du, X., Na, S., Wang, Z., and Geramifard, A. TRIAGE: Role-typed credit assignment for agentic reinforcement learning. arXiv:2606.32017, 2026.
Thank you for the detailed response and additional experiments. The distinction from prior hindsight-based methods and the evidence on training stability are now clear to me, and my main concerns have been addressed.
We sincerely thank the reviewer for the positive feedback and for acknowledging that our responses and additional experiments have successfully addressed your concerns. Your constructive suggestions have been invaluable in improving our work, and we will ensure all these revisions are fully integrated into the final version.
Response to Reviewer adjq
We thank the reviewer for the positive assessment and for noting design details such as the non-negative weights and the fallback for unscorable outputs. Tables R1–R4 are in the General Response, and all changes in the revised PDF are in blue. In short:
Q2 / W1. Bias and stability over longer training
The form of the bias is constrained. OTAD's update is a biased estimate of the gradient of expected success: it optimizes a reweighted surrogate (Appendix A.3). OTAD does not change the reward, and it never reverses the sign that the verifier assigns to a turn (Proposition A.3 in Appendix A.3; Appendix A.4). In credit assignment, its only deviation from GRPO is to rescale each turn's update by qi,t∈[0,1.4]. This trades a constrained bias for a much less noisy within-trajectory signal. Whether that helps over a long run is an empirical question, so we tested it.
Longer training. The training-reward curves in Appendix F.8 (Figure 7) are noisy and cannot show whether a plateau has been reached. Up to step 40, the runs below are the two runs of the paper, which reports their step-40 checkpoints. Training continued with the same settings, and we have now evaluated every 20th step on OSWorld, so the step-40 values are those of Tables 2 and 4. Both use the same learning rate (3×10⁻⁶), KL coefficient (10⁻⁴) and batch (Table 11); these are common settings for RL fine-tuning and were not tuned for OTAD. The revision adds this as Table R4 (Appendix F.8):
OTAD has plateaued by step 40 and does not degrade; GRPO degrades. After step 40, every checkpoint is within 1.3 points of 52.1%, the same order as the standard deviation across three evaluation passes (±1.0, Table 2). The 40-step budget is the one fixed in the paper (Table 11) for both methods. The weighting also does not collapse: at step 120 the negative-level share is 8% and mean Ci is 0.79 (Appendix F.7).
GRPO's decline follows the motivation of the paper. With one terminal bit per trajectory, GRPO gives every turn the same advantage, so it reinforces the detours inside successful trajectories (12% of turns in successful training trajectories are detours or regressions, Section 4.5) and penalizes the correct steps inside failed ones. The SFT model already imitates successful trajectories, so what RL can still add is mainly this within-trajectory discrimination, and here GRPO's signal is noisy; over many updates these errors accumulate. Human annotation of GRPO's own rollouts supports this: among 400 turns sampled from its successful trajectories at each of steps 40 and 120, the detour share rises from 13% to 21%. OTAD down-weights such detours (q=0.6 for −0.5), masks harmful moves (−1), and reduces the penalty on useful turns in failed trajectories, which is why its training stays stable.
Larger policy. All our results use an 8B policy, and larger policies remain untested; Appendix G now lists this as a limitation. One related result: for the 8B policy, the scorer does not need to be larger than the policy. A frozen copy of the 8B initial policy gives 50.8%, a 7B scorer 49.6% (Table 15), and a new 2B scorer, at 0.2× the scoring cost of the 27B scorer, 48.3%.