acceptodds
Under review as a conference paper at ICLR 2027

Reward Reasoning for Its Gain over a Snap Judgment: Inverse-RL Cold Start of GUI Agents with an Adversarial Critic

Abstract

Cold-starting GUI agents requires learning from expert demonstrations before sparse task-completion rewards can support effective online reinforcement learning. Supervised fine-tuning (SFT) provides this initialization, but fitting expert traces can come at the cost of general capability. Learning a reward from demonstrations offers another route: the agent trains on its own decisions, guided by a critic that reasons about whether a decision came from the expert or the policy. The challenge is that source labels supervise the critic's final answer, not whether its reasoning helped. Obvious errors and stylistic cues can make a pair easy to classify, allowing uninformative reasoning to receive the same reward as reasoning that improves the judgment. We introduce ReGain, which trains the critic on the improvement its reasoning provides over a direct, no-reasoning judgment. For each policy-expert pair, we score the verdict distributions with and without chain-of-thought reasoning using a detached Brier score and reward their difference. The continuous score captures improvements even when the selected verdict is unchanged, while centring rewards across distinct pairs gives greater relative credit to reasoning that improves the comparison. Confidently correct direct judgments leave little room for positive gain, reducing the reward for reasoning that merely repeats obvious source cues. On a 27B GUI agent trained on 20,480 demonstrations, ReGain outperforms the adversarial baseline RARO and SFT on the same data and delays the critic's reliance on stylistic cues. It matches SFT trained on nine times as many demonstrations in pass@1 and pass@4, without the 14% loss of general capability incurred by either SFT baseline.

Primary area:
reinforcement learning
Keywords:
GUI agents, inverse reinforcement learning, cold start, learning from demonstrations, adversarial critic, verifier-free reinforcement learning, reward design, proper scoring rules, chain-of-thought, GRPO

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_i5wzOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces ReGain, which is an adversarial learning approach to improve reasoning in multimodal large language models used for GUI agents. Instead of learning only a critic over the trace source, they also distinguish between Chain-of-Thought responses and responses without Chain-of-Thought. They report results on a single model for a GUI task, where it mostly outperforms the RARO and SFT baselines.

Critical strengths

S1) Motivation: Overall, I like the paper and the approach. It is well motivated by related work (e.g., DAPO) and its current failure modes, and it explains how to address them. I find their reward gain approach sensible.

S2) Scientific Reporting: I appreciate that the authors are reporting standard errors to demonstrate the uncertainty of their experiments. Unfortunately, this is often not the case in reviews today. I thank the authors for this, as it makes the results easier to interpret.

S3) Baselines: I appreciate the baselines the authors use. While there are obviously more adversarial reasoning approaches (e.g., rePIRL, GAD, R-AIRL, GAR, ...), it makes sense to compare against RARO, which has different tuning parameters. However, I would maybe discuss these works in the related work section.

Critical weaknesses

W1) Dataset / Task: My biggest concern with this paper is that it tests only a single model on a single GUI dataset. I almost misinterpreted the appendix results as ReGain being trained on different datasets, but I understand it is only tested after being trained on the GUI Agent dataset. I believe, however, that this method is not necessarily limited to GUI Agents, and it would be interesting to see whether it also generalises to other reasoning datasets, such as MATH, MMLU-Pro, etc., and what the gains are over SFT and RARO.

Questions for authors

Q1) Stability What stability methods were used to address the non-stationarity of the method (e.g. replay buffer, KL regularisation, etc.)

3 comments
AuthorsOpenReview ↗

Response to Reviewer i5wz

Thank you for the positive evaluation and helpful references.

S3. There are more adversarial reasoning approaches (e.g., rePIRL, GAD, R-AIRL, GAR, ...) ... I would maybe discuss these works in the related work section.

[S3] Related adversarial and inverse-RL reasoning works. We added these works in §2 (Related Work): rePIRL (Wu et al., 2026) and R-AIRL (Fanconi et al., 2025) use inverse RL to learn process-level reward models from expert trajectories or CoTs, GAD (Ye et al., 2025) trains a sequence-level discriminator as an on-policy reward for black-box distillation, and GAR (Liu et al., 2025) co-trains a reasoner with a slice-level adversarial discriminator. In all of these methods, the discriminator or reward model is scored only on its final classification, without comparing against a no-reasoning judgment.

W1. It tests only a single model on a single GUI dataset ... it would be interesting to see whether it also generalises to other reasoning datasets, such as MATH, MMLU-Pro.

[W1] Evaluation beyond a single dataset. During the rebuttal period we prioritized adding public evaluation and online RL continuation on MobileGym-Bench [1] (256 test tasks, 4 trials per task; new §5.5, Figure 3):

Online RL step01020304050
ReGain (ours)55.658.260.260.161.661.3
ReGain, Brier-only (ours)52.854.154.557.258.458.3
SFT, full data39.843.444.440.744.547.9

We leave extending the gain-from-reasoning formulation to general text-reasoning benchmarks (such as MATH and MMLU-Pro) as a direction for future work, as the present study focuses on demonstration cold starts for GUI agents ahead of sparse-reward online RL.

Q1. What stability methods were used to address the non-stationarity of the method (e.g. replay buffer, KL regularization, etc.)?

[Q1] Stabilization techniques. Across all adversarial runs, we address co-training non-stationarity through six mechanisms (detailed in Appendix J, Table 3):

  • Critic experience replay: half of the critic groups are rebuilt from a FIFO buffer of the last 1,024 policy rollouts (following RARO), preventing the critic from overfitting solely to the latest policy.
  • KL regularization: a low-variance KL loss (coefficient 0.001) anchors updates to the initial checkpoint.
  • Conservative optimization: asymmetric PPO-style ratio clipping (0.2 low, 0.28 high, dual clip 3.0), gradient norm clipping at 1.0, and a constant 10−610^{-6} learning rate with one update per step.
  • Balanced role weighting: fixed loss weights of 1/3 for the policy and 2/3 for the critic.
  • Unscaled group centering: advantages are mean-centered within each group without standard-deviation normalization, avoiding variance amplification when scores concentrate.
  • Detached proper scoring: Brier scores are detached before verdict sampling so rewards do not backpropagate directly through token logits.

Beyond these shared settings, the gain reward itself helps prevent critic collapse: at step 160, ReGain maintains higher critic advantage variance (0.11 vs. 0.05 for discrete c=0.75c=0.75 and Brier-only) and avoids tie collapse (35% ties vs. 64% for Brier-only). Late-stage style overfitting (after step 320) is controlled via a fixed training budget.

[1] Wu et al. 2026. MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research. In Proceedings of EMNLP 2026, pages 23223-23247.

Thank you to the authors for their response. Overall, I think the paper is in good shape, with a well-motivated, clear method and results to validate their claims.

I understand that the authors do not have time to test their method beyond GUI control, and they do not claim to do so in the paper. I would nevertheless encourage the authors to expand their experiments to more general use cases, as this could make the paper even stronger. Despite this point, they clarified my questions, and I increased my score.

I wish the authors good luck with their work.

Thank you very much for reviewing our response and revised manuscript, and for raising your score to 4. Your pointers on related adversarial and inverse-RL reasoning works, stabilization details, and broader evaluation helped us make the positioning and empirical section much clearer. Your clear acceptance with high confidence has indeed greatly encouraged us.

Reviewer_kzF8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReGain, an adversarial learning method for cold-starting GUI agents from expert demonstrations. A shared model acts as both policy and critic, and the critic is rewarded for the improvement of its post-reasoning judgment over a direct judgment, measured using a detached Brier score. Experiments show that ReGain uses fewer demonstrations to achieve performance comparable to full-data SFT while better preserving general capabilities.

Critical strengths
  1. The paper addresses a practical problem: using limited expert demonstrations to initialize GUI agents without substantially degrading their general capabilities.

  2. Rewarding the contribution of reasoning rather than the final verdict alone is an interesting approach. The Brier-only ablation and training diagnostics help examine the effect of the proposed reward.

  3. The paper clearly discusses its limitations, including the persistence of stylistic shortcuts, imperfect critic calibration, and the computational cost of adversarial training.

Critical weaknesses
  1. The benefits for subsequent online RL are not directly evaluated. Although the paper targets cold-start initialization, the evaluation covers single-step pass@1 and pass@4, trajectory success before online training, and general-capability retention. Better initial performance does not necessarily imply faster learning or a higher final success rate after online RL. I would like to see the different initializations followed by the same online RL procedure and interaction budget, comparing learning curves, the interactions needed to reach a given success rate, and final performance. This would directly establish the method's value for the training stage it aims to support.

  2. The critic's lower average score after reasoning needs further explanation. Section 5.3 reports a mean snap-judgment score of 0.722 at step 160, compared with 0.620 after reasoning, giving an average gain of approximately -0.1. Thus, reasoning does not improve source judgments on average, which deserves closer examination given the method's motivation. Reporting the gain distribution, the proportion of positive-gain pairs, and their evolution during training would help clarify this result. An analysis of negative-gain cases would also help distinguish improved reasoning from optimization benefits caused by redistributing advantages across pairs.

  3. The demonstration-efficiency claim needs a comparison that also accounts for computation. ReGain uses 20,480 demonstrations to achieve performance comparable to SFT on 190k demonstrations, but each demonstration requires multiple policy and critic generations plus snap-judgment forward passes. The paper acknowledges and reports this cost, yet demonstration counts alone do not establish the practical trade-off under limited resources. I would like to see performance reported jointly against demonstration count and total training compute, including compute-matched comparisons that allow SFT to reuse the smaller dataset over multiple epochs. This would clarify when the reduction in demonstration requirements justifies the additional computation.

Questions for authors

Refer to the weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer kzF8 (Part 1/2)

Thank you for the constructive review and experimental suggestions.

W1. The benefits for subsequent online RL are not directly evaluated ... I would like to see the different initializations followed by the same online RL procedure and interaction budget, comparing learning curves, the interactions needed to reach a given success rate, and final performance.

[W1] Direct evaluation of subsequent online RL. We ran the online RL continuation experiment you suggested (§5.5, Figure 3; setup in Appendix J). Starting from ReGain (ours), ReGain Brier-only (ours), and full-data SFT, we run 50 steps of GRPO with binary task-completion rewards under identical hyperparameters (12 tasks x 8 rollouts = 96 episodes per step, one seed per initialization) on the public MobileGym-Bench training split [1], and evaluate on its public 256-task test split (4 trials per task, official step budgets, no checkpoint selection):

Online RL step01020304050
ReGain (ours)55.658.260.260.161.661.3
ReGain, Brier-only (ours)52.854.154.557.258.458.3
SFT, full data39.843.444.440.744.547.9

ReGain achieves the highest success rate at every evaluated step, outperforming our Brier-only ablation (61.3% vs. 58.3% at step 50) and already surpassing Brier-only's final 58.3% score at step 15 (60.0%), using 1,440 vs. 3,840 episodes (about 26k vs. 60k environment steps, 0.43x, given mean rollout lengths of 18.0 vs. 15.6 actions). On this external task suite, full-data SFT starts much lower (39.8%, consistent with its 14% general-capability loss) and failed episodes often repeat actions until reaching the step limit (19.2 to 23.3 actions), consuming 1.19x ReGain's environment steps while finishing 13.4 points lower (still below both adversarial cold starts at step 0). Figure 3 plots the test-split learning curves, and §5.5 reports the episode and environment-step counts.

[1] Wu et al. 2026. MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research. In Proceedings of EMNLP 2026, pages 23223-23247.

AuthorsOpenReview ↗

Response to Reviewer kzF8 (Part 2/2)

W2.** The critic's lower average score after reasoning needs further explanation (0.722 vs 0.620, mean gain about -0.1) ... Reporting the gain distribution, the proportion of positive-gain pairs, and their evolution ... An analysis of negative-gain cases would also help.

[W2] Explaining the lower average score after reasoning. Below we analyze Δ=sfull−ssnap\Delta = s_{\text{full}} - s_{\text{snap}} across critic rollouts with a snap probe and parsed verdict (§5.3 is updated to this subset, with step-160 means ssnap=0.732s_{\text{snap}} = 0.732 and sfull=0.640s_{\text{full}} = 0.640; Appendix Q, Table 8):

StepPairsMeanMedian>0.05>0.05≈0\approx 0<−0.05<-0.05Snap rightCoT rightCorrected / reversed
1209-0.115+0.00027.8%29.2%43.1%51.2%54.1%22 / 16
40643-0.033+0.01039.3%28.1%32.5%54.6%65.3%148 / 79
80497-0.107+0.00535.4%26.6%38.0%55.9%58.6%89 / 76
120467-0.064+0.00939.4%25.1%35.5%50.3%60.6%94 / 46
160490-0.092+0.00633.5%33.7%32.9%57.1%60.2%66 / 51
  1. Distribution and transitions. Treating s>0.75s > 0.75 as favoring the expert, median Δ\Delta stays near zero, pairs split into rough thirds across the bins, and CoT corrects more wrong snap judgments than it reverses right ones at every step, lifting expert-favoring reports by 2.7 to 10.7 points.
  2. Why the mean is negative without hurting updates. At step 160, only 51/490 pairs (10.4%) reverse an expert-favoring snap judgment; another 110 negative-gain pairs started with ssnap≤0.75s_{\text{snap}} \le 0.75 and committed sharply to the policy (sfull≈0s_{\text{full}} \approx 0), while gain on the 229 pairs where both favor the expert is capped by 1−ssnap<0.251 - s_{\text{snap}} < 0.25. Under the quadratic Brier score, sharp wrong verdicts pull Q1Q_1 to −0.30-0.30 vs. Q3=+0.11Q_3 = +0.11, matching the miscalibration term −E[(r1−q1)2∣r0]-\mathbb{E}[(r_1 - q_1)^2 \mid r_0] in Eq. (10) (Appendix C). Group-centered advantages Δi−Δˉ\Delta_i - \bar{\Delta} cancel the negative mean and shift positive credit from glance-decided pairs (40% to 20%) to overturned pairs (37% to 56%; Figure 8).
  3. Sources of negative-gain pairs. Inspecting sampled GUI-action pairs with Δ<−0.05\Delta < -0.05 identifies two main sources of sfull≈0s_{\text{full}} \approx 0: (i) critic errors on fine-grained UI mechanics (e.g., inverting time-picker scroll direction or misestimating progress-bar and seat positions), where negative Δ\Delta penalizes overconfident wrong CoTs as intended; and (ii) valid or more complete policy actions (e.g., searching directly from the home screen, selecting an alternative entry button, or filling multiple form fields at once), where task reasoning diverges from demonstration authorship (Origin B, §4.1; Appendix Q).

W3. The demonstration-efficiency claim needs a comparison that also accounts for computation ... including compute-matched comparisons that allow SFT to reuse the smaller dataset over multiple epochs.

[W3] Compute cost and SFT baselines. Below we summarize cold-start compute (identical A800 nodes) alongside demonstration count and performance (§5.1, §5.5, §6):

Cold startDemosCold-start GPU-h (A800)Single-step @1 / @4Gen. cap.MobileGym RL (0 →\to 50)
ReGain (ours)20,480 (1 pass)1,900 (64x29.6h)70.0 / 86.0-0.9%55.6 →\to 61.3
ReGain, Brier-only (ours)20,480 (1 pass)1,900 (same budget)65.9 / 79.8+0.2%52.8 →\to 58.3
SFT, full data190,000 (1 epoch)640 (32x20h)66.1 / 86.8-14.1%39.8 →\to 47.9

Three factors contextualize this roughly 3x cold-start cost: (i) we prioritized 1-epoch full-data SFT as our strongest SFT reference because 1-epoch training outperformed multi-epoch runs in prior experiments on this dataset (extra epochs degraded general capability faster), and we will add multi-epoch SFT runs on the 20,480 subset (e.g., 3 and 9 epochs) in the camera-ready; (ii) the rollout cost is shared with RARO and Brier-only (the snap probe is one forward pass without generation); and (iii) cold start is a one-time initialization before multi-turn online RL, and extra SFT training neither recovers the 14% general-capability loss nor closes the 13.4-point online RL gap.

Reviewer_WEVXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to cold-start a GUI agent from expert demonstrations before sparse task-completion rewards are available. Supervised fine-tuning fits the demonstration text and can reduce general capability, while an adversarial critic such as RARO can classify the expert from style or obvious errors and still receive full credit. ReGain scores each policy-expert pair twice: after a chain of thought and as a no-reasoning snap judgment. The critic is trained on the detached Brier-score difference, centered over a group of distinct pairs, while the policy is still rewarded for being judged expert. On a 27B agent and 20,480 demonstrations, ReGain improves over data-matched SFT, retains general capability, and is competitive with SFT trained on the full demonstration set.

Critical strengths
  1. The failure diagnosis is specific and the control is the right one. Verdict collapse, unused reasoning, and the short-decision shortcut are separated, and the Brier-only ablation holds the score fixed so the comparison isolates the snap-judgment baseline.

  2. The data-matched SFT comparison supports the cold-start claim. SFT on 20,480 demonstrations falls below the untouched backbone, and both SFT runs lose about 14% of general capability. ReGain stays near the backbone on the eleven multimodal benchmarks and reaches a similar pass@4 to SFT trained on nine times as many demonstrations.

Critical weaknesses
  1. The improvement over the adversarial baselines is within the reported noise. At the fixed 160-step budget, ReGain's pass@1 is 70.0 against 65.6 and 65.9 for discrete RARO and Brier-only. The paper states that neither the 4.1-point single-step gap nor the 4.6-point trajectory gap exceeds 1.4 standard errors. ReGain and Brier-only use two seeds; every other run uses one, and the training curves are single runs.

  2. The stated test of a cold start is missing. The introduction says an initialization should be judged by the online RL that can be reached from it. No online stage is run. Pass@4, general-capability retention, and zero-shot success on 100 tasks are proxies. Capability retention is also shared by the other adversarial runs, so it does not by itself establish the gain reward.

  3. The empirical comparison cannot be checked outside the authors' data. The 190k demonstrations, the 250-state benchmark, and the 100 trajectory tasks are internal and will not be released. The single-step metric scores one reference action, which the paper notes favors SFT, and the trajectory set is small enough that a 4-point standard error remains.

Questions for authors
  1. Can the authors evaluate pass@1, pass@4, and trajectory success at each run's validation peak, not only at step 160? The other adversarial runs peak at steps 40-80, so the fixed budget may compare ReGain before its peak with baselines after theirs.

  2. Does a later online RL stage, using the same task-completion reward and budget, reach different performance from ReGain, RARO, Brier-only, and the two SFT initializations? That is the handoff the introduction uses to motivate the cold start.

  3. Because the mean gain is negative, what fraction of critic traces improve the snap judgment, and do policy improvements concentrate on those pairs? Please also report this over more than two seeds.

  4. Can the single-step or trajectory comparison be repeated on a public GUI benchmark, with the action-matching rule and the harness specified tightly enough to reproduce the table?

4 comments
AuthorsOpenReview ↗

Response to Reviewer WEVX (Part 1/2)

Thank you for the thorough review and for highlighting both the Brier-only control and the areas to strengthen in our evaluation.

W2. The stated test of a cold start is missing ... No online stage is run. Pass@4, general-capability retention, and zero-shot success on 100 tasks are proxies. Capability retention is also shared by the other adversarial runs.

Q2. Does a later online RL stage, using the same task-completion reward and budget, reach different performance from ReGain, RARO, Brier-only, and the two SFT initializations?

[W2, Q2] Online RL continuation. We added an online RL stage starting from the step-160 checkpoints of ReGain (ours), ReGain Brier-only (ours), and full-data SFT (new §5.5, Figure 3; hyperparameters in Appendix J). All three runs use the same GRPO setup with binary task-completion rewards on the public MobileGym-Bench training split (Wu et al., EMNLP 2026; 50 RL steps, 12 tasks x 8 rollouts = 96 episodes per step, one seed per initialization) and are evaluated on the public 256-task test split (4 trials per task, official step budgets, reporting step 50 without checkpoint selection):

Online RL step01020304050
ReGain (ours)55.658.260.260.161.661.3
ReGain, Brier-only (ours)52.854.154.557.258.458.3
SFT, full data39.843.444.440.744.547.9

ReGain achieves the highest success rate at every step in the table (and across all five-step checkpoints in Figure 3), outperforming our Brier-only ablation at step 50 (61.3% vs. 58.3%) and already surpassing Brier-only's final 58.3% score at step 15 (60.0%) with 0.43x the environment steps (1,440 vs. 3,840 episodes, or about 26k vs. 60k environment steps; Appendix J). On this external benchmark, full-data SFT starts much lower (39.8%, consistent with its 14% general-capability loss), consumes 1.19x ReGain's environment steps as failed rollouts repeat actions until reaching the step limit (19.2 to 23.3 actions per episode), and finishes at 47.9%, still below both adversarial cold starts at step 0. During the rebuttal period we prioritized ReGain, the Brier-only ablation, and full-data SFT (our strongest SFT baseline), and we will include discrete RARO and data-matched SFT in the camera-ready.

Regarding general capability, all four adversarial runs preserve general capability similarly, so retention reflects on-policy adversarial training in general. We revised §5.2 to state this explicitly. What distinguishes the gain reward from Brier-only and discrete RARO is higher single-step pass@1/pass@4 and trajectory success at matched data, avoiding tie collapse during training, shifting positive critic credit away from glance-decided pairs (Figure 8), and stronger online RL performance.

W1. The improvement over the adversarial baselines is within the reported noise ... ReGain and Brier-only use two seeds; every other run uses one, and the training curves are single runs.

[W1] Statistical uncertainty and seeds. On single-step pass@1 and in-house trajectory success at step 160, the gap to Brier-only or discrete RARO is within 1.4 standard errors (§5.3, §6). What supports the comparison is that the same ordering holds across checkpoints, metrics, and stages: ReGain leads Brier-only and both discrete RARO runs in both pass@1 and pass@4 at steps 40, 80, 120, and 160 on the single-step benchmark (Table 7 in Part 2/2; +3.5 to +6.2 points in pass@4 at every step, and +1.1 growing to +4.1 points over Brier-only in pass@1); it ranks first in in-house trajectory success at step 160; and on the 256-task public MobileGym-Bench test split, it stays ahead of Brier-only both zero-shot and across all online RL evaluation steps. Given the roughly 1,900 A800 GPU-hours per cold-start run, we prioritized multi-checkpoint evaluations and online RL continuation during the rebuttal period, and we will expand the cold-start comparison with additional random seeds in the camera-ready.

AuthorsOpenReview ↗

Response to Reviewer WEVX (Part 2/2)

Q1. Can the authors evaluate pass@1, pass@4, and trajectory success at each run's validation peak, not only at step 160? The other adversarial runs peak at steps 40-80.

[Q1] Evaluation at earlier checkpoints. Below we report single-step pass@1 and pass@4 (%) at steps 40, 80, 120, and 160 (pooling two seeds for ReGain and Brier-only, matching Table 1; Appendix Q, Table 7):

RunStep 40Step 80Step 120Step 160
RARO, c=0.7557.3 / 77.260.8 / 78.064.3 / 79.665.6 / 79.6
RARO, c=0.5557.5 / 78.259.2 / 78.862.8 / 79.863.3 / 80.4
ReGain, Brier-only (ours)59.2 / 78.260.2 / 78.163.8 / 79.865.9 / 79.8
ReGain (ours)60.3 / 83.263.3 / 82.367.8 / 85.770.0 / 86.0

Although the three other adversarial runs reach their validation peaks around steps 40 to 80, all three achieve higher test pass@1 and pass@4 at step 160 than at steps 40 to 80 (tying step 120 in pass@4 for c=0.75 and Brier-only), so step 160 compares every run at its best evaluated checkpoint. We also clarified in §5.4 how step 160 was set: it came from earlier development runs where single-step test accuracy peaked around step 160, and was applied as a shared budget without per-method tuning. ReGain's validation score continues to rise until step 320, and MobileGym-Bench was not used in selecting step 160. We evaluated trajectory success at step 160 (both in-house and on MobileGym-Bench), and we will include intermediate trajectory checkpoints in the camera-ready.

Q3. Because the mean gain is negative, what fraction of critic traces improve the snap judgment, and do policy improvements concentrate on those pairs? Please also report this over more than two seeds.

[Q3] Critic gain distribution and negative mean gain. Below we report Δ=sfull−ssnap\Delta = s_{\text{full}} - s_{\text{snap}} across critic rollouts with a snap probe and a parsed verdict (Appendix Q, Table 8; s>0.75s > 0.75 favors the expert):

StepPairsMeanMedian>0.05>0.05≈0\approx 0<−0.05<-0.05Snap rightCoT rightCorrected / reversed
1209-0.115+0.00027.8%29.2%43.1%51.2%54.1%22 / 16
40643-0.033+0.01039.3%28.1%32.5%54.6%65.3%148 / 79
80497-0.107+0.00535.4%26.6%38.0%55.9%58.6%89 / 76
120467-0.064+0.00939.4%25.1%35.5%50.3%60.6%94 / 46
160490-0.092+0.00633.5%33.7%32.9%57.1%60.2%66 / 51

At every logged step, CoT corrects more wrong snap judgments than it reverses right ones, raising expert-favoring reports by 2.7 to 10.7 points. Only 10.4% of pairs (51/490 at step 160) reverse from right to wrong; the mean is negative because confident post-CoT verdicts for the policy score sfull≈0s_{\text{full}} \approx 0 (Q1=−0.30Q_1 = -0.30 vs. Q3=+0.11Q_3 = +0.11; full breakdown in our reply to Reviewer kzF8, W2). To test directly whether downstream policy improvements concentrate on states with corrected critic pairs, we will add per-state pair-to-policy tracking and report this analysis across additional seeds in the camera-ready.

W3. The empirical comparison cannot be checked outside the authors' data ... the trajectory set is small enough that a 4-point standard error remains.

Q4. Can the single-step or trajectory comparison be repeated on a public GUI benchmark, with the action-matching rule and the harness specified tightly enough to reproduce the table?

[W3, Q4] Public benchmark evaluation and reproducibility. To provide an externally checkable comparison on public tasks alongside our internal benchmarks, we added zero-shot evaluation and online RL continuation on the public MobileGym-Bench splits [1] (256 test tasks, scored by MobileGym's public deterministic evaluators), which also avoids the single-reference bias of single-step pass@1. While the 190k demonstrations, the in-house task sets, and the action harness remain proprietary internal assets, every compared checkpoint runs through an identical harness, and we will release our verl training recipe.

[1] Wu et al. 2026. MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research. In Proceedings of EMNLP 2026, pages 23223-23247.

Thank you for the additional experiments. The public MobileGym-Bench continuation shows that the step-160 ReGain initialization stays ahead of Brier-only and full-data SFT throughout online RL, and the earlier checkpoints show that step 160 is the strongest evaluated test point for the other adversarial runs as well. The gain distribution also clarifies that the chain of thought corrects more wrong snap judgments than it reverses.

So I am raising my score to 3.

Thank you very much for taking the time to review our additional experiments and for raising your score. Your questions on the downstream online RL stage, the earlier checkpoint comparisons, and the critic gain distribution really strengthened the paper a lot.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.