acceptodds
Under review as a conference paper at ICLR 2027

Seeing What Changed: Transition-Guided Credit Assignment for GUI Agents

Abstract

Reinforcement learning with verifiable rewards (RLVR) for multi-turn GUI agents provides terminal outcome supervision but leaves credit assignment across individual turns unresolved. Uniformly broadcasting a trajectory-level advantage can reinforce both useful actions and mistakes that the agent subsequently corrects. Screenshots before and after each action expose state changes that help distinguish progress from detours, even within successful trajectories. We propose Observable-Transition Advantage Decomposition (OTAD), which uses this evidence to guide turn-level credit assignment. A vision-language model provides hindsight ordinal judgments of each transition conditioned on the verified outcome. OTAD maps these judgments to non-negative weights on the trajectory-level advantage rather than additional rewards: transition evidence sets the magnitude of each turn's advantage while the terminal verifier fixes its sign. This design removes the need for a learned value model and for extra environment interaction. The same judgments guide budget-constrained turn selection and prioritize tasks with discriminative transition structure, improving training-data efficiency. On OSWorld-Verified, OTAD raises success from 42.2% under budget-matched GRPO to 52.1%; at 8B parameters, it surpasses ComputerRL-9B and OpenCUA-72B, while also improving GUI grounding.

Primary area:
reinforcement learning
Keywords:
GUI Agent, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_adjqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes OTAD, a method for credit assignment in long-horizon agent. It uses a VLM to construct hindsight for scoring individual actions. These scores guide reweighting of advantages within GRPO. The scores also guide budget-constrained turn selection across different action types and adaptive task sampling. On OSWorld-Verified, OTAD achieves a substantial improvement over budget-matched GRPO. Ablation studies support the contributions of transition information and credit weighting.

Critical strengths
  1. This method is well-motivated. I can understand it clearly. Credit assignment is particularly challenging in long-horizon CUA tasks. I find several details reflect careful designs from authors, such as the nonnegative weights and fallback handling for uncertain scores.
  2. The experiments support the effectiveness of the method. OTAD improves success on OSWorld-Verified over budget-matched GRPO, and the ablations support the contributions of transition information and credit weighting.
Critical weaknesses
  1. OTAD introduces bias in GRPO. Although the reported results are promising, it remains unclear whether the gains would persist with larger models or longer training runs. This limits confidence in the method’s effectiveness at scale.
  2. Hindsights are well established in RL. A clearer discussion of prior work, such as Hindsight Credit Assignment, would help distinguish OTAD’s specific methodological contribution.
Questions for authors
  1. Could the authors clarify how OTAD differs methodologically from prior hindsight-based credit-assignment approaches, particularly which aspects of the ordinal scoring and advantage reweighting constitute the main novelty?

  2. Given the bias introduced by turn-level advantage reweighting, do the authors have existing evidence that OTAD’s gains remain stable over longer training? Do the reported learning curves indicate that both OTAD and the GRPO baseline have reached a performance plateau?

4 comments
AuthorsOpenReview ↗

Response to Reviewer adjq

We thank the reviewer for the positive assessment and for noting design details such as the non-negative weights and the fallback for unscorable outputs. Tables R1–R4 are in the General Response, and all changes in the revised PDF are in blue. In short:

  • Q2 / W1 (bias, longer training): OTAD's gains are stable over 3× longer training, while GRPO degrades. OTAD peaks at step 40 (52.1%) and stays within 50.8–52.0% up to step 120; GRPO peaks at 42.4% and falls to 36.8%. All results use an 8B policy, and larger policies remain untested (Appendix G).
  • Q1 / W2 (hindsight): hindsight is a well-established idea, and outcome conditioning itself is not our contribution. OTAD differs from HCA in what is judged, how credit is obtained and how it enters the update. An HCA-style baseline using the same scorer reaches 45.2%, against 52.1% for OTAD.

Q2 / W1. Bias and stability over longer training

The form of the bias is constrained. OTAD's update is a biased estimate of the gradient of expected success: it optimizes a reweighted surrogate (Appendix A.3). OTAD does not change the reward, and it never reverses the sign that the verifier assigns to a turn (Proposition A.3 in Appendix A.3; Appendix A.4). In credit assignment, its only deviation from GRPO is to rescale each turn's update by qi,t∈[0,1.4]q_{i,t}\in[0,1.4]. This trades a constrained bias for a much less noisy within-trajectory signal. Whether that helps over a long run is an empirical question, so we tested it.

Longer training. The training-reward curves in Appendix F.8 (Figure 7) are noisy and cannot show whether a plateau has been reached. Up to step 40, the runs below are the two runs of the paper, which reports their step-40 checkpoints. Training continued with the same settings, and we have now evaluated every 20th step on OSWorld, so the step-40 values are those of Tables 2 and 4. Both use the same learning rate (3×10⁻⁶), KL coefficient (10⁻⁴) and batch (Table 11); these are common settings for RL fine-tuning and were not tuned for OTAD. The revision adds this as Table R4 (Appendix F.8):

Step020406080100120
GRPO41.742.042.242.440.938.636.8
OTAD41.748.652.151.452.050.851.5

OTAD has plateaued by step 40 and does not degrade; GRPO degrades. After step 40, every checkpoint is within 1.3 points of 52.1%, the same order as the standard deviation across three evaluation passes (±1.0, Table 2). The 40-step budget is the one fixed in the paper (Table 11) for both methods. The weighting also does not collapse: at step 120 the negative-level share is 8% and mean CiC_i is 0.79 (Appendix F.7).

GRPO's decline follows the motivation of the paper. With one terminal bit per trajectory, GRPO gives every turn the same advantage, so it reinforces the detours inside successful trajectories (12% of turns in successful training trajectories are detours or regressions, Section 4.5) and penalizes the correct steps inside failed ones. The SFT model already imitates successful trajectories, so what RL can still add is mainly this within-trajectory discrimination, and here GRPO's signal is noisy; over many updates these errors accumulate. Human annotation of GRPO's own rollouts supports this: among 400 turns sampled from its successful trajectories at each of steps 40 and 120, the detour share rises from 13% to 21%. OTAD down-weights such detours (q=0.6q=0.6 for −0.5-0.5), masks harmful moves (−1-1), and reduces the penalty on useful turns in failed trajectories, which is why its training stays stable.

Larger policy. All our results use an 8B policy, and larger policies remain untested; Appendix G now lists this as a limitation. One related result: for the 8B policy, the scorer does not need to be larger than the policy. A frozen copy of the 8B initial policy gives 50.8%, a 7B scorer 49.6% (Table 15), and a new 2B scorer, at 0.2× the scoring cost of the 27B scorer, 48.3%.

Q1 / W2. Difference from hindsight credit assignment

Section 2 now discusses Hindsight Credit Assignment (HCA; Harutyunyan et al., 2019), counterfactual credit assignment (Mesnard et al., 2021) and HER (Andrychowicz et al., 2017).

How OTAD differs from HCA. HCA learns a hindsight distribution h(a∣x,Z)h(a\mid x,Z) and uses the advantage (1−π(a∣x)/h(a∣x,Z))Z(1-\pi(a\mid x)/h(a\mid x,Z))Z, which is unbiased when hh is exact. By Bayes' rule, π/h=P(Z∣x)/P(Z∣x,a)\pi/h=P(Z\mid x)/P(Z\mid x,a), so with a binary verifier HCA needs calibrated success probabilities before and after each action. Section 3.1 explains why these are hard to obtain in GUI environments.

HCAOTAD
What is judgedlikelihood of the action, given the state and the returnthe observed transition (pre-screen, action, post-screen), given task and outcome
Source of credita hindsight model trained with the policya frozen VLM; nothing extra is trained
How credit enters the updateratio-corrected advantage of either signnon-negative weight on the verifier-signed advantage

HCA-style baseline (Appendix F.10). Our variant obtains the hindsight probabilities without a learned hindsight model: the frozen 27B scorer estimates success probabilities before and after each transition, the ratio above is formed from them, clipped at a threshold cc, and used as (1−ratio)(Ri−b)(1-\text{ratio})(R_i-b), with OTAD's turn selection and task sampling. The best threshold reaches 45.2%, against 43.8% for unit credit and 52.1% for OTAD. A frozen VLM's ordinal ranking is thus more useful than its probability estimates, which are not calibrated.

Main novelty. (i) Turn credit from a frozen VLM's ordinal, outcome-conditioned judgment of the observed pre/post transition, which needs neither action likelihoods nor calibrated probabilities; removing the post-action screen costs 6.8 points (Table 8). (ii) An outcome-conditioned mapping from these judgments to a non-negative weight on the verifier-signed advantage. As Section 2 notes, concurrent work (Xu et al., 2026; Huang et al., 2026) also lets evidence set only the update strength. In OTAD the evidence is the visible interface change, judged with the outcome known, and the mapping itself depends on the outcome, so the same score is weighted differently in successful and failed trajectories (48.9% with an outcome-independent mapping, Table 13). (iii) Reusing the same judgments for turn selection and task sampling (−3.9 and −2.7 points when removed, Table 5).

To make the presentation easier to follow, the revision adds a notation table (Table R5, Appendix A) and a worked numerical example of the mapping (Section 3.2).

We hope the extended runs and the clarified relation to hindsight methods address the reviewer's questions.


References

  • Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. Hindsight experience replay. In NeurIPS, 2017. arXiv:1707.01495.
  • Harutyunyan, A., Dabney, W., Mesnard, T., Azar, M. G., Piot, B., Heess, N., van Hasselt, H., Wayne, G., Singh, S., Precup, D., and Munos, R. Hindsight credit assignment. In NeurIPS, 2019. arXiv:1912.02503.
  • Huang, M., Xue, T., Peng, C., Ding, J., Yang, J., Fan, S., Hong, J., Gao, Y., Zhang, X., Guo, L., Yang, X., et al. EvoCUA-1.5: Online reinforcement learning for multi-turn computer-use agents. arXiv:2607.09773, 2026.
  • Mesnard, T., Weber, T., Viola, F., Thakoor, S., Saade, A., Harutyunyan, A., Dabney, W., Stepleton, T., Heess, N., Guez, A., Moulines, É., Hutter, M., Buesing, L., and Munos, R. Counterfactual credit assignment in model-free reinforcement learning. In ICML, PMLR 139:7654–7664, 2021. arXiv:2011.09464.
  • Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300, 2024. (GRPO)
  • Xu, Y., Zhou, Z., Sang, H., Li, X., Zhang, J., Du, X., Na, S., Wang, Z., and Geramifard, A. TRIAGE: Role-typed credit assignment for agentic reinforcement learning. arXiv:2606.32017, 2026.

Thank you for the detailed response and additional experiments. The distinction from prior hindsight-based methods and the evidence on training stability are now clear to me, and my main concerns have been addressed.

We sincerely thank the reviewer for the positive feedback and for acknowledging that our responses and additional experiments have successfully addressed your concerns. Your constructive suggestions have been invaluable in improving our work, and we will ensure all these revisions are fully integrated into the final version.

Reviewer_x9SDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies turn-level credit assignment for multi-turn GUI agents trained with reinforcement learning from verifiable rewards (RLVR). The authors propose OTAD, which uses a frozen vision-language model, conditioned on the verified outcome, to assign a five-level ordinal score to each pre-action/post-action screenshot transition. The score is mapped to a non-negative weight on the trajectory-level advantage, so the verifier fixes the sign while the transition fixes the magnitude. The same scores are reused for budgeted turn selection and task sampling. On OSWorld-Verified, OTAD improves a Qwen3-VL-8B policy from 42.2% under budget-matched GRPO to 52.1%, a substantial gain. However, the main weaknesses lie in the experimental design and the missing comparison to existing turn-level credit-assignment or GUI-specific RL methods.

Critical strengths
  1. Important and well-motivated problem. Long-horizon GUI RL with a single terminal reward is a real bottleneck, and detour statistic makes the failure mode concrete rather than hypothetical.
  2. Large and consistent main result. The gain is consistent across OSWorld categories, and the horizon analysis shows the expected qualitative pattern. For a credit-assignment method, this is the right kind of evidence.
Critical weaknesses
  1. The main comparison in Table 2 does not control for the cost of the strong scorer. According to the Qwen3.8-27B model card[1], the scorer Qwen3.6-27B reaches 63.9 on OSWorld, well above the 52.1 of the OTAD-trained 8B policy. The paper uses this model only as a training-time judge. Since both OTAD and OPD pay for a stronger model and extra training-time inference, an OPD baseline using the same 27B teacher on the student’s on-policy states is the natural cost-matched comparison.

  2. No comparison to turn-level credit-assignment or GUI-specific RL methods on the same SFT checkpoint. The current experiments only show OTAD > uniform GRPO. Without comparison on the same SFT checkpoint and task pool, the paper does not establish that its specific form of turn-level credit is better than existing turn-level credit method like Process Reward Model-base RL or GUI specific RL method like ARPO[1].

  3. No absolute GPU-hours, FLOPs, or wall-clock comparison to GRPO is reported. The paper states that scoring accounts for 12% of end-to-end training wall-clock, but this is a share of OTAD’s own runtime, not the overhead relative to GRPO. A compute-matched GRPO (same total wall-clock/FLOPs) would be needed to attribute the gain to credit assignment rather than extra compute

[1] https://huggingface.co/Qwen/Qwen3.8-27B [2] Arpo: End-to-end policy optimization for gui agents with experience replay

Questions for authors
  1. Why does GRPO improve only +0.5 percentage points over SFT in Table 4? Is the RL signal too sparse, or is the run too short? If sparsity is the issue, a stronger RL baseline such as DAPO (or a longer/tuned GRPO run) would help isolate whether the gain comes from credit assignment or from a weak baseline.
6 comments
AuthorsOpenReview ↗

Response to Reviewer x9SD

We thank the reviewer for recognizing the importance of the problem and the consistency of the main result across categories and horizons. Tables R1–R4 are in the General Response, and all changes in the revised PDF are in blue. In short:

  • W1: OTAD uses the 27B model only as a judge of each turn, and this judgment is close to human annotation (88.5% direction agreement, Table 7). A stronger agent is not required: a frozen copy of the 8B SFT policy itself already gives 50.8% (Table 15). Under our protocol the 27B model reaches 59.8% as an agent (63.9 in the model card, which uses a different context setting, including the screenshots in context and the prompt). OPD with this teacher, up to the same GPU-hours as OTAD, reaches at most 47.9%, 4.2 points below OTAD.
  • W2: the submission already compares against PRM-style use of the same scores: Table 6 adds them as per-turn rewards (40.8%) and Table 13 tests a signed mapping (47.5%). New runs on the same checkpoint reach 46.8% for ARPO, 44.1% for DAPO, 44.8% for GiGPO (47.0% with OTAD's selection and sampling) and at most 46.6% for tuned PRM-style variants, all at least 5.1 points below OTAD (52.1%).
  • W3: Appendix F.6 reported 12% as a share of OTAD's own wall-clock, and the revision now gives absolute GPU-hours (Table R2). OTAD uses 38% more GPU-hours than GRPO, and GRPO with 1.5× and 3× the GPU-hours reaches only 42.4% and 36.8%.
  • Q1: tuning GRPO's learning rate, KL coefficient or group size gives 40.3–42.8%, and DAPO reaches 44.1%. Longer training does not help: at 120 steps GRPO falls to 36.8% (38.1% with a larger KL coefficient) and DAPO stays at 43.9%. GRPO's small gain comes mainly from credit dilution, the problem OTAD targets.

W1. The scorer's strength and an OPD baseline

One premise needs a correction: OTAD does not require a model that is a stronger agent than the policy. It uses the 27B model as a judge, not as a source of actions. Judging whether an observed transition advanced the task, with the outcome known, is easier than choosing the action: the scorer agrees with human direction labels on 88.5% of turns (Table 7), so it provides turn-level credit close to human annotation at low cost (Table R2). With a frozen copy of the 8B SFT checkpoint (41.7% as an agent) as scorer, OTAD still reaches 50.8%, keeping 87% of the gain (Table 15). Under the same setting OPD gets no signal, as explained below.

On the cited 63.9: under our evaluation protocol, Qwen3.6-27B (Qwen Team, 2026) reaches 59.8%. The remaining gap comes from a different context setting (the screenshots in context and the prompt). Our evaluation uses the official OSWorld agent loop for Qwen3-VL (mm_agents/qwen3vl_agent.py) with the prompt of Appendix F.2 and the context of Table 11: at most three screenshots, earlier actions kept as a text-only summary, and the 50-step limit. All our evaluations, including our policy's, use this protocol; the external baselines in Table 2 are taken from their own reports, as its caption states. The revision adds this number and protocol to Appendix F.6.

OPD with the same 27B model. We ran OPD with this teacher, using the standard on-policy distillation recipe (Agarwal et al., 2024; Lu, 2025). The student samples one rollout per task, the teacher computes the log-probabilities of the sampled tokens in the same context, and the negative per-token reverse KL, log⁡πT(yk∣x,y<k)−log⁡πθ(yk∣x,y<k)\log\pi_{\mathrm{T}}(y_k\mid x,y_{\lt k})-\log\pi_\theta(y_k\mid x,y_{\lt k}), is used as the per-token advantage. OPD uses no verifier reward and the same task pool as GRPO. Results are in Table R1 (Section 4.3) and Table R7 (Appendix F.10).

Method (same SFT checkpoint)Rollouts per taskGPU-hours (40-step GRPO = 1)OSWorld
GRPO, 40 steps81.00×42.2
OPD, 27B teacher, 40 steps10.23×46.9
OTAD, 27B scorer, 40 steps81.38×52.1

OPD samples one rollout per task (eight for GRPO), so 40 steps are cheap. To compare at equal compute, we trained OPD until its GPU-hours match OTAD's (Table R7):

OPD steps4080120160200240
GPU-hours (40-step GRPO = 1)0.23×0.46×0.69×0.92×1.15×1.38×
OSWorld46.947.447.847.647.947.5

From step 120 on, OPD stays at 47.5–47.9%, against 52.1% for OTAD at the same 1.38×. OPD thus closes about a third of the gap between the SFT checkpoint (41.7%) and its teacher (59.8%); OTAD closes more than half. OPD fits the teacher's distribution on every token, mistakes included, and the verified outcome plays no role. OTAD instead uses the teacher only to judge the student's own transitions and lets the verifier fix the sign of every update. This is also why OTAD needs no stronger agent. With a frozen copy of the initial policy as teacher, the reverse KL is zero at the start of training and OPD gets no signal, whereas OTAD with this model as scorer reaches 50.8%. Swapping only OTAD's credit into GRPO gives 47.8%, on par with OPD at any budget (Table R1); full OTAD also uses the scores to select turns and tasks, which adds another 4.3 points. Section 2 now discusses on-policy distillation.

W2. Comparison with PRM-based and GUI-specific RL on the same SFT checkpoint

The submission already contains PRM-style baselines that use the same model as process reward model, on the same SFT checkpoint and task pool. Table 6 adds the ordinal scores as per-turn rewards (40.8%), with shuffled scores as a control (43.5%), and Table 13 tests a signed mapping (47.5%). This separates how the signal is used from which model produces it.

We have added the following runs, all on the same SFT checkpoint, task pool and budget (Table R1 in Section 4.3, Table R6 in Appendix F.4, details in Appendix F.10):

MethodGRPO selection & samplingOTAD selection & sampling
PRM-style per-turn rewards, best of scale/normalization sweep–44.7
Additive process advantage Aiep+λzi,tA_i^{\mathrm{ep}}+\lambda z_{i,t}–46.6
ARPO (success replay buffer + filtering of uniform-outcome tasks)46.8–
DAPO44.1–
GiGPO (best anchor key)44.847.0
OTAD47.852.1

Full OTAD is at least 5.1 points above every method in the table. With only the credit module swapped in, OTAD (47.8%) is already the best in the GRPO column, 3.0–3.7 points above GiGPO and DAPO and on par with ARPO (46.8%); the full gain needs the same scores to drive credit, turn selection and task sampling together. Under the same OTAD selection and sampling, OTAD is still 5.1–7.4 points above the PRM-style variants and GiGPO. The PRM-style variants fall short because the added scores change the return that RL optimizes. The scores are ordinal levels that rank turns, not calibrated reward units, and their sum grows with trajectory length, so a long failed trajectory with many +0.5 turns can earn about as much return as a short successful one (Appendix A.4). The additive process advantage keeps the return, but its score term can reverse the sign of some turns' advantages, as the signed mapping does, and with a frozen scorer reversal does not help (Q4 of our reply to sNBs). OTAD leaves the verifier's reward unchanged, never reverses a sign, and uses the scores only to weight its advantage across turns. GiGPO is limited on desktops because its anchor states rarely recur (only 5.8% of non-initial turns share an exact screenshot within their group); details are in our reply to sNBs.

W3. Absolute cost and compute-matched GRPO

Table R2 (Appendix F.1) gives absolute GPU-hours. OTAD's 40-step run uses the 32 training GPUs of Table 11 plus 8 separately deployed scorer GPUs for 48 h, i.e. 1,920 GPU-hours; GRPO uses 1,389 GPU-hours on the 32 training GPUs. OTAD therefore adds 38% GPU-hours, with the scorer GPUs charged for the whole run, including idle time between scoring requests. Appendix F.6 now states explicitly that the 12% is a share of OTAD's own wall-clock and points to Table R2.

The gain is not explained by extra compute (Table R3). GRPO with 1.5× the GPU-hours of its 40-step run, more than OTAD's 1.38×, reaches 42.4%, the best checkpoint of that run; with 3× it falls to 36.8%. Spending the extra compute on a larger group instead of more steps changes little: GRPO with G=12 for 40 steps (about 1.5×) reaches 42.8%. OTAD at 0.69× already reaches 48.6%, above GRPO at any budget. OPD at the same 1.38× as OTAD reaches 47.5% (W1).

Reviewer_sNBsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Observable-Transition Advantage Decomposition (OTAD) for reinforcement learning of GUI agents. A frozen vision-language judge assigns ordinal scores to individual transitions using pre- and post-action screenshots, action history, and the verified terminal outcome. These scores determine non-negative weights on trajectory-level GRPO advantages, adjusting turn-level update magnitudes without reversing their signs. The scores also guide budget-constrained turn selection and adaptive task sampling. On OSWorld-Verified, OTAD improves an 8B policy’s success rate from 42.2% under interaction- and optimization-budget-matched GRPO to 52.1%. The paper further evaluates GUI grounding, component ablations, score-use controls, scorer agreement with human annotations, alternative judges, and continuation-based estimates of transition credit.

Critical strengths
  1. The paper addresses a consequential limitation of trajectory-level reinforcement learning for GUI agents: uniform advantages cannot distinguish useful actions from detours within the same trajectory. Using observable state transitions to modulate turn-level updates is well motivated. The improvement from 42.2% to 52.1% over interaction- and optimization-budget-matched GRPO is substantial and provides meaningful evidence of practical value in the evaluated 8B setting.

  2. The internal empirical analysis provides converging evidence that the gains depend on informative transition judgments and their alignment with individual turns. Component ablations, shuffled-score controls, scorer input ablations, human agreement measurements, and continuation-based analyses support this interpretation. These controls strengthen my confidence that the proposed transition evidence contributes to the improvement beyond simply introducing non-uniform update weights.

Critical weaknesses
  1. Insufficient comparison with existing step-level credit-assignment methods. The paper identifies GiGPO as its closest related method, but controlled comparisons primarily involve trajectory-level GRPO and internal variants. These demonstrate improvement over uniform advantage broadcasting without establishing competitiveness against existing approaches to the same problem. Comparisons with published GUI-agent results do not close this gap because their training configurations differ. This limits my confidence in the significance of the methodological contribution.

  2. Insufficient validation of the proposed advantage mapping. The added-reward and signed-mapping baselines lack sufficient implementation details, particularly regarding reward scaling, return normalization, and signed-score mapping. These choices can materially affect their competitiveness. Moreover, OTAD's non-negative weighting cannot correct the direction of trajectory-level advantages, limiting its ability to assign genuinely fine-grained credit. The current evidence does not sufficiently establish the advantages or limitations of the proposed mapping relative to these alternatives.

Questions for authors
  1. Comparative evidence. A controlled comparison with GiGPO or another established fine-grained credit-assignment method is needed to substantiate the methodological contribution. Any obstacles to applying GiGPO in GUI environments should be explained, with an appropriate alternative identified.

  2. Baseline clarification. The exact formulations, normalization procedures, and reward scaling of the added-reward and signed-mapping baselines should be specified to establish the fairness of these comparisons.

  3. Dependence on heavyweight judges and computational cost. OTAD relies on relatively large judges (7B/27B) to train an 8B actor. It remains unclear whether substantially smaller judges (e.g., 2B–3B) can provide comparable benefits, or whether judge capacity must increase with actor size. The additional computational costs of judge inference should be quantified, and the efficiency gains should be evaluated under matched end-to-end compute budgets.

  4. Sign-preserving credit assignment. OTAD cannot assign positive advantages to useful actions within negatively advantaged trajectories. The implications of this restriction for fine-grained credit assignment should be clarified, including whether the signed-mapping baseline adequately evaluates this limitation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer sNBs

We thank the reviewer for the clear summary and for noting that our controls show "the proposed transition evidence contributes to the improvement beyond simply introducing non-uniform update weights." Tables R1–R4 are in the General Response, and all changes in the revised PDF are in blue. In short:

  • Q1 (GiGPO): we implemented GiGPO on our checkpoint. In the same pipeline, OTAD reaches 52.1% and GiGPO 47.0%. With only the credit module swapped into GRPO, OTAD is 3.0 points higher (47.8% vs. 44.8%).
  • Q2 (baselines): exact formulas are below and in Appendix F.4. After tuning scale and normalization, reward/advantage-space variants reach at most 46.6% and sign-reversing variants at most 50.3%, against 52.1%.
  • Q3 (judge size, cost): OTAD does not need a judge larger than the actor. Besides the 7B and 27B judges, Table 15 already includes a frozen copy of the 8B actor itself, which gives 50.8%; a new 2B judge at 0.2× the scoring cost of the 27B judge gives 48.3%. OTAD costs 38% more GPU-hours than GRPO, and GRPO with 3× the GPU-hours stays far below it.
  • Q4 (sign preservation): this is a deliberate design choice. The judge is frozen and used without training, so it is cheap and mostly right, but it can misjudge some transitions. The verified outcome of online RL therefore fixes the sign of each update, and the judge only scales it.

W1 / Q1. Controlled comparison with GiGPO

We ran GiGPO and other baselines on the same SFT checkpoint with our task pool and budget (Table R1; Section 4.3 and Appendix F.10 of the revision).

Obstacle and adaptation. GiGPO computes step advantages within groups of anchor states that recur across a rollout group. Such states are common in text environments like ALFWorld (Shridhar et al., 2021) but rare on desktops: only 5.8% of our non-initial turns share an exact screenshot with another trajectory in their group. We therefore tried two anchor keys, an exact hash after masking the clock and cursor (14.8% coverage) and a perceptual hash with Hamming distance ≤ 6 (31.5%). Following GiGPO, the step return γTi−tRi\gamma^{T_i-t}R_i is standardized within each anchor group and the advantage is Aiep+ωAi,tstepA_i^{\mathrm{ep}}+\omega A_{i,t}^{\mathrm{step}}; we report the best setting (γ=0.95, ω=1).

SettingGiGPOOTAD
Credit module only (GRPO selector, uniform tasks)43.3 (exact) / 44.8 (similarity)47.8
Full pipeline, only the credit module swapped47.0 (similarity)52.1

With the credit module alone, OTAD is 3.0 points above GiGPO with similarity anchors and 4.5 points above exact anchors. When GiGPO also uses OTAD's score-driven turn selection and task sampling, the gap grows to 5.1 points. OTAD's selection and credit come from the same scores, so every selected turn carries its own pre/post evidence. GiGPO's step advantage covers only turns with an anchor (31.5% even with similarity anchors), and the other turns still receive the uniform broadcast. Similarity anchors also merge states that differ in one task-relevant detail, such as a spreadsheet cell. The revision also corrects our description of GiGPO in Section 2.

W2 / Q2. Exact formulations of the baselines

The following is now in Appendix F.4 (Table R6). si,ts_{i,t} is the ordinal score (⊥↦0\bot\mapsto0) and ρi,t\rho_{i,t} the token weight of Appendix A.2. Scorer, scores, turn selection, task sampling and budget are fixed as in Table 6; only how the scores enter the update changes.

  • Added rewards (Table 6). Ri′=Ri+η∑tsi,tR_i'=R_i+\eta\sum_t s_{i,t} with η=0.1\eta=0.1, summed over all scored turns, then standardized within the group as in Eq. (2) and broadcast to the selected turns. We also test a turn-level return-to-go form, Ri+η∑k≥tsi,kR_i+\eta\sum_{k\ge t}s_{i,k}.
  • Signed mapping (Table 13). qi,t±=1+κ(2Ri−1)si,tq_{i,t}^{\pm}=1+\kappa(2R_i-1)s_{i,t} with κ=1.5\kappa=1.5, rescaled per trajectory so that ∑tρi,t∣qi,t±∣\sum_t\rho_{i,t}\lvert q_{i,t}^{\pm}\rvert equals OTAD's ZiZ_i. A −1-1 turn in a successful trajectory and a +1+1 turn in a failed one get a reversed sign.
  • Flip-only (new). OTAD's qq everywhere, except that ∣s∣=1\lvert s\rvert=1 turns whose direction disagrees with the outcome get qflip<0q_{\mathrm{flip}}<0. The signed mapping matches OTAD's total weight per trajectory, but the relative weights of non-reversed turns still differ from OTAD's; this variant changes only the reversals and so isolates sign reversal.

For each variant we tuned its scale (η\eta, λ\lambda, κ\kappa or qflipq_{\mathrm{flip}}) and report the best result, as was done for OTAD's α and B (Table 14).

VariantOSWorld
Added rewards, trajectory sum / return-to-go (Table 6: 40.8)44.1 / 44.7
Process advantage Aiep+λzi,tA_i^{\mathrm{ep}}+\lambda z_{i,t}, zz = group-standardized ss46.6
Signed mapping (Table 13: 47.5)48.1
Flip-only50.3
OTAD52.1

Tuning raises the baselines by 0.6–3.9 points over the paper's settings, but all stay below OTAD. Reward-space use loses most, because ordinal scores rank turns but do not give a calibrated reward scale (Appendix A.4). Flip-only comes closest, since it keeps OTAD's weights on all but a few turns; Q4 explains why reversing those few turns does not help.

Q3. Judge size and compute

  • Judge size. The submission does not rely only on 7B/27B judges: Table 15 also includes a frozen copy of our 8B initial policy (50.8%, 0.5× the scoring cost of the 27B judge), i.e., the actor itself before RL with no extra training, and a 7B judge (49.6%, 0.4×). We have also tested a 2B judge (Qwen3-VL-2B, 0.2× the scoring cost), added to Table 15 (Appendix F.6). Its direction agreement drops to 75.2%, and OTAD reaches 48.3%, 6.1 points above GRPO and 62% of the gain. This is below the frozen 8B copy, which needs no extra model and no extra training, so for an 8B actor the actor's own frozen copy is the better judge. Among judges with the same inputs, final success rises with direction agreement: 48.3% at 75.2% (2B), 49.6% at 77.5% (7B), 50.8% at 79.0% (frozen 8B), 52.1% at 88.5% (27B) and 52.4% at 89.1% (72B).
  • Judge versus actor size. For the 8B actor, a judge of the same size suffices. All results use an 8B policy, and larger policies remain untested (Appendix G).
  • Compute. OTAD costs 38% more GPU-hours than GRPO with the 27B judge (Table R2, Appendix F.1). Under matched GPU-hours (Table R3), GRPO reaches 42.4% at 1.5× and 36.8% at 3×, against 48.6% for OTAD at 0.69×.

Q4. Implications of sign preservation

The restriction exists, and Section 3.2 and Appendix A.3 state it: OTAD cannot give a positive advantage to a useful action in a failed trajectory. It is a deliberate trade-off. OTAD's judge is a frozen VLM used without any training. This makes turn-level credit cheap to obtain, and its direction judgments are mostly correct (88.5% agreement with humans, Table 7), but it can misjudge some transitions, for example when an action's effect becomes visible only later. If such a judgment could reverse the sign, the error would push the policy the wrong way. The verified outcome therefore remains the supervising signal of online RL: it fixes the sign of every update, and the judge only sets the size, so a judge error can shrink an update but not reverse it. The flip-only variant, which differs from OTAD only in these reversals, reaches 50.3% against 52.1% (W2), so reversal does not help; Figure 4 adds that the signed mapping falls further behind as judge noise grows. Instead of reversing, OTAD shrinks these updates: a +1+1 turn in a failed trajectory gets q=0.2q=0.2, removing 80% of its penalty, and a −1-1 turn in a successful trajectory is masked.

To measure the cost, we reused the continuation data of Section 4.5. This analysis is now in Appendix A.3 ("Cost of the restriction"):

  • How often it binds. After Benjamini–Hochberg control (FDR 0.1; Benjamini & Hochberg, 1995), 6.3% of turns in failed trajectories have a significantly positive increment and 4.1% of turns in successful ones a significantly negative increment.
  • What OTAD does on these turns. It scores 72% and 77% of them in the matching direction, lowering their mean weight from 1 (GRPO) to 0.55 and 0.48. OTAD therefore roughly halves the wrong-direction update on exactly the turns the restriction concerns.
  • Why reversal does not help with a frozen judge. The continuation estimates contradict 26% of the signed mapping's reversals, so about a quarter of its reversed updates point the wrong way. The restriction concerns few turns (6.3% and 4.1%), OTAD already halves the wrong-direction update on them, and a quarter of reversals add errors, so little room is left for reversal to help; flip-only (50.3%) indeed does not exceed OTAD.

Reversal would pay off only with a judge whose errors are rare on these turns. With a cheap frozen judge, letting the outcome decide the sign is the safer choice. A judge trained for this purpose could relax the restriction, which Appendix G now notes.

We hope these controlled comparisons address the reviewer's concerns about the significance of the contribution, and we would be glad to provide further details.

References

  • Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al. Qwen3-VL technical report. arXiv:2511.21631, 2025.
  • Benjamini, Y. and Hochberg, Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1):289–300, 1995.
  • Feng, L., Xue, Z., Liu, T., and An, B. Group-in-group policy optimization for LLM agent training. In NeurIPS 38, 2026. arXiv:2505.10978.
  • Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300, 2024. (GRPO)
  • Shridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M. ALFWorld: Aligning text and embodied environments for interactive learning. In ICLR, 2021. arXiv:2010.03768.

We sincerely thank the reviewer for the time and care spent on our submission. The reviewer's questions led us to run the comparisons and analyses in the revision, and they have improved the paper. If any point in our response remains unclear, we would be glad to discuss it before the discussion period closes. We look forward to hearing from the reviewer.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

We thank the reviewers for their careful reviews. All three agree that turn-level credit assignment is a real bottleneck for long-horizon GUI RLVR and that the gain over budget-matched GRPO (42.2% → 52.1%) is substantial. We ran the requested experiments and uploaded a revised PDF in which every change is in blue; the list of changes with their locations is at the end of this response. All new runs start from the same SFT checkpoint (41.7%) with the same task pool and, unless the number of steps is stated, the same 40-step budget. The policy trains on 32 H800 GPUs, and the scorer (or OPD teacher) runs on 8 additional GPUs.

Main new results

  1. Full OTAD is at least 4.2 points above every existing method we compared on the same checkpoint (Tables R1, R3): GiGPO, ARPO, DAPO and on-policy distillation (OPD) from the 27B scorer model. GiGPO given OTAD's turn selection and task sampling is still 5.1 points below, and OPD trained up to the same GPU-hours as OTAD reaches at most 47.9%. Using the same scores as PRM-style rewards or as a process advantage reaches at most 46.6% (G1).
  2. The gain is not explained by extra compute (Tables R2, R3). OTAD uses 38% more GPU-hours than GRPO, counting the 8 scorer GPUs for the whole run. GRPO given 1.5× and 3× the GPU-hours reaches 42.4% and 36.8%, while OTAD at 0.69× already reaches 48.6%. OPD stays at 47.5–47.9% from 0.69× to 1.38×, against 52.1% for OTAD at 1.38×.
  3. OTAD is stable over 3× longer training, while GRPO degrades (Table R4). After step 40, OTAD stays within 50.8–52.0% up to step 120; GRPO peaks at 42.4% and falls to 36.8% (38.1% with a larger KL coefficient), and DAPO stays at 43.9%.

Points the submission already addresses

  • OTAD does not need a scorer stronger than the policy. Section 4.4 and Table 15 show that a frozen copy of the 8B initial policy, i.e. the RL starting checkpoint itself, gives 50.8% (87% of the gain).
  • The submission already tests the scores as process rewards. Table 6 adds them as per-turn rewards (40.8%), which is the PRM-style use of the same model, and Table 13 tests a signed mapping (47.5%). We have now also tuned their scale and normalization (G1).
  • Sign preservation is a deliberate design choice (Section 3.2, Appendix A.3). The scorer is a frozen model used without training. It is cheap and judges direction correctly on most turns (88.5% agreement with humans, Table 7), but it can misjudge some transitions. The verified outcome of online RL therefore fixes the sign of every update and the scorer only sets its size, so a scorer error can shrink an update but never reverse it (details in the reply to sNBs).
AuthorsOpenReview ↗

Summary for the Area Chair

We sincerely thank the AC for handling our submission and the reviewers for their constructive feedback.

Paper. We find that even successful GUI trajectories contain detours and regressions (12% of their turns), which uniform trajectory-level credit reinforces, and that the screen change each action causes reveals which turns made progress. Building on this, we propose OTAD, which turns judgments of these screen changes into turn-level credit and reuses them to select training turns and tasks. On OSWorld-Verified, OTAD raises an 8B policy from 42.2% under budget-matched GRPO to 52.1%, surpassing ComputerRL-9B and OpenCUA-72B.

Strengths noted by the reviewers. All three reviewers consider turn-level credit assignment a real bottleneck for long-horizon GUI agents and the gain over GRPO substantial. x9SD finds the gain consistent across categories and horizons. sNBs finds that the controls attribute the gain to the transition evidence rather than to non-uniform weights alone. adjq notes the careful design of the non-negative weights and the fallback for uncertain scores.

Concerns addressed in the revision (changes in blue)

  • Comparison with existing methods (sNBs, x9SD). On the same SFT checkpoint, task pool and budget, OTAD is at least 4.2 points above GiGPO, ARPO, DAPO and on-policy distillation from the 27B scorer (Section 4.3, Table R1).
  • Scorer cost and compute (x9SD, sNBs). OTAD uses 38% more GPU-hours than GRPO, yet GRPO given 3× the GPU-hours reaches only 36.8% on OSWorld-Verified, against 52.1% for OTAD. OTAD does not need a stronger scorer: a frozen copy of the 8B policy gives 50.8% (Appendices F.1, F.6).
  • Baseline details and sign preservation (sNBs). With exact formulas and tuned scales, every alternative use of the scores, including sign-reversing ones, stays below OTAD (at most 50.3% vs. 52.1%), so reversing signs does not help with a frozen scorer (Appendices F.4, A.3).
  • Training stability and hindsight methods (adjq, x9SD). Over 3× longer training, OTAD stays within 50.8–52.0% while GRPO degrades (Appendix F.8). Section 2 now discusses hindsight credit assignment, and an HCA-style baseline reaches 45.2% (Appendix F.10).
  • Discussion status. Reviewers x9SD and adjq both stated that our response addressed their main concerns, and x9SD raised the rating to 3. We have responded to every question from reviewer sNBs but have not yet received a reply.

Thank you again for your time and effort in overseeing the discussion. We are also grateful to all three reviewers, whose questions led to the comparisons and analyses that have improved the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.