acceptodds
Under review as a conference paper at ICLR 2027

Uncertainty-Gated Policy Optimization for Robust Reasoning

Abstract

Recent group-based Reinforcement Learning (RL) methods have greatly improved LLMs' reasoning capabilities. However, relying solely on sparse outcome-level rewards risks misaligned credit assignment: models may be rewarded for “lucky” trajectories that arrive at correct answers despite uncertain and potentially flawed intermediate reasoning. We introduce Uncertainty-Gated Advantage (UGA), a plug-and-play mechanism that reshapes trajectory-level advantages using entropy-based uncertainty estimation. UGA directly dampens the advantage of high-uncertainty correct trajectories, reducing their contribution to policy updates while preserving learning signals from confident generations. The method requires no auxiliary reward model and integrates seamlessly into existing RL pipelines such as GRPO and DAPO. Experiments on challenging math benchmarks (AMC23, AIME24, AIME25) show that UGA consistently outperforms strong baselines across two model scales, achieving up to 11.6% relative improvement in mean@32 and 20.6% in maj@32. On knowledge-intensive agent tasks, UGA further improves accuracy by 5.0% (relative) over the baseline using step-level uncertainty gating.

Primary area:
reinforcement learning
Keywords:
Large Language Model, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_a7mfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Uncertainty-Gated Advantage (UGA), which rescales detached GRPO/DAPO advantages using mean top-50 token entropy and a rollout group's empirical pass rate. It also adapts entropy gating to a self-search agent. On Qwen3-1.7B/4B mathematical reasoning, the paper reports improved macro mean@32 and maj@32 in four training configurations, plus agent and cross-domain results. A blinded, ProcessBench-calibrated judge finds that high-entropy, verifier-accepted base-model solutions are enriched for process errors. This diagnostic motivates UGA, but does not directly measure whether UGA reduces such errors after training.

Critical strengths
  1. Simple method with informative controls. UGA requires no new learned reward model and adds roughly 10% reported training time. Table 1 shows positive mean@32 changes in 11 of 12 benchmark-by-training comparisons, with positive macro gains for both GRPO and DAPO at both sizes. This correlated sign pattern is favorable descriptive evidence. Table 4's batch-mean entropy, constant entropy, and reversed pass-rate controls all weaken mean@32. They support the full reweighting configuration, although they also change gate magnitudes and do not independently identify a process-reliability mechanism.

  2. A useful, cautiously presented diagnostic. The process audit is blinded to entropy and model identity, calibrated against ProcessBench, and explicitly presented as an association rather than a causal proof. The authors also report mixed outcomes, including a higher maj@32 for the fixed-pass-rate ablation and some decreases in large-k coverage. This candor makes the evidence easier to interpret.

Critical weaknesses
  1. The reliability interpretation is not established by the entropy audit or the gate's algebra. Mean token entropy measures uncertainty over token choices, not whether the reasoning is valid. More specifically, Eq. (8) also implements outcome-dependent difficulty reweighting: at the same entropy, a high pass rate suppresses correct trajectories more than incorrect ones, and a low pass rate reverses this relation. This can affect optimization without identifying lucky guesses. The audit covers only verifier-correct base-model trajectories; it does not validate the entropy/error association conditional on the gate's pass-rate signal, or justify the negative-advantage weighting. Thus the full configuration's benefit is compatible with difficulty-dependent update rebalancing, while the more distinctive claim of reliability-aware credit remains unverified.

  2. Figure 1's 7.8% versus 45.3% association is useful motivation, but its 314 traces cover only 41 questions, with 290 from AMC23, and entropy quintiles are formed within benchmarks rather than within questions. The critic's precision is 0.783 on its correct-answer calibration subset (138 false positives, 22 false negatives), so calibration does not eliminate annotation concerns. Blinding does not control question difficulty or visible solution properties such as length. Most importantly, no controlled comparison shows fewer process errors among verifier-correct UGA outputs than GRPO outputs after training. The authors acknowledge the audit's correlational scope; nevertheless, this missing link limits the evidence for the paper's central motivation, rather than merely omitting a peripheral implementation detail.

  3. Appendix E reports some large-k coverage losses, including 1.7B GRPO AIME25 pass@32 falling from 33.33 to 23.33 with UGA; this qualifies the robust-reasoning interpretation while remaining compatible with improved average single-sample accuracy. Table 5's cross-domain transfer is also mixed, rather than evidence of a broadly validated reliability signal. Finally, the agent extension changes the gate and advantage-sign treatment and adds a fixed gamma = 0.5 discount. Without retaining those changes and removing only entropy dependence, its accuracy gains cannot independently validate the math gate's proposed mechanism. The paper reports several of these qualifications candidly, but the remaining support is for a useful reweighting heuristic with limited demonstrated advantage over the closest alternative.

Questions for authors
  1. Please clarify whether the entropy/error association persists after accounting for question, pass rate, and solution length, and reconcile the cohort counts: Table 1 implies approximately 348 accepted base-model rollouts, whereas Appendix F.1 reports 314 after excluding 12. Different verification or filtering may explain this. A small process audit of verifier-correct GRPO and UGA outputs on the same questions, for one representative trained configuration, would directly test the reliability interpretation and could change my assessment.
  2. How were beta and checkpoints selected, and what uncertainty do the existing paired question-level outputs imply for the close Table 2 comparisons? To distinguish trajectory-specific entropy alignment from average reweighting strength, a useful targeted control would permute entropies within each question and reward outcome, preserving the outcome-specific gate distributions. One representative setting would suffice; it is an alternative way to strengthen the mechanism evidence, rather than a request for a broad new experimental suite.
  3. With the fixed gamma = 0.5 discount and the advantage-sign treatment retained, what happens if the entropy-dependent factors in Eqs. (10)–(11) are replaced by constants matched to their average strength? This would distinguish uncertainty-dependent weighting from the overall reduction in positive updates, and materially affect how I interpret the agent extension.
4 comments
AuthorsOpenReview ↗

Response to Reviewer a7mf (Part 1/2)

We thank the reviewer for the comments. We respond to each point below.

W1. Interpretation of the reliability mechanism

We agree that mean token entropy is not a certificate of valid reasoning. The original diagnostic establishes an association between entropy and process errors among verifier-accepted base-model responses. UGA uses this association as a noisy training signal.

The pass-rate term regulates the allocation of credit: it attenuates rare successes less on low-pass-rate questions and retains more of the negative penalty for rare failures on high-pass-rate questions. This does not establish that a rare success is process-valid. Likewise, reduced process-error rates after training would not, by themselves, uniquely identify the gate's mechanism.

The new matched-weight and entropy-shuffling controls test whether trajectory-specific weighting contributes beyond average gate scaling, while the new trained-policy audit evaluates the resulting outputs. These provide complementary evidence. A conditional analysis accounting for question, pass rate, and solution length would further strengthen the reliability interpretation, and the rationale for negative-advantage gating remains distinct from the audit of correct outputs. We do not regard these questions as resolved solely by the gate's algebra.

W2. Judge error and the post-training comparison

We answer the two points in turn.

1. Judge error. We agree that a precision of 0.783 on the original calibration subset is not sufficient to settle this concern. The new audit uses Claude Opus 5.5 and is recalibrated on ProcessBench: on the answer-correct subset, binary accuracy is 90.4%. Re-auditing all correct outputs of Qwen3-1.7B-Base with this judge gives a process-error rate of 23.3%, close to the 22.0% reported in the paper.

2. Post-training comparison. This is precisely the experiment the reviewer identified as potentially changing the assessment. On the representative 1.7B GRPO configuration, we blind-judged the outputs of GRPO and GRPO w/ UGA. The table below reports the rate of process errors among outputs whose final answers are already correct.

PolicyAMC23AIME24AIME25
Qwen3-1.7B-Base21.19%39.29%33.33%
+ GRPO21.44%30.77%32.47%
+ GRPO w/ UGA20.85%21.21%24.44%

After training, the UGA outputs accepted by the verifier do contain fewer process errors than those of GRPO.

W3. Coverage, cross-domain transfer, and the agent extension

Regarding coverage, the 1.7B GRPO AIME25 pass@32 decrease from 33.33% to 23.33% is real. Appendix E also reports decreases on AMC23 for 4B GRPO and 4B DAPO. Across the 12 benchmark-by-training comparisons, pass@32 improves or remains unchanged in nine and decreases in three; we therefore do not claim uniformly preserved exploration.

We also retain the mixed cross-domain results and limit the transfer claim to the evaluated configurations where improvements are observed. The agent results support the combined variant's answer accuracy, but do not isolate the contributions of entropy gating, advantage-sign treatment, and the fixed γ = 0.5 discount.

AuthorsOpenReview ↗

Response to Reviewer a7mf (Part 2/2)

Q1. Audit cohort counts and trained-policy outputs

Thank you for identifying the discrepancy. You are correct that Table 1 implies approximately 348 verifier-accepted base-model outputs. With 32 responses per question, the counts are approximately 302 for AMC23 (40 questions × 32 × 23.59%), 28 for AIME24 (30 × 32 × 2.92%), and 18 for AIME25 (30 × 32 × 1.88%). The recorded counts are exactly 302, 28, and 18, totaling 348.

The original process audit did not cover all of these outputs. It covered 326 candidates, excluded 12 with uncertain answer or process status, and reported the remaining 314. Thus, 22 of the 348 verifier-accepted outputs were not covered by that original audit. Figure 1's total of 314 is consistent with the retained audit cohort, but does not account for those 22 unaudited outputs.

For this rebuttal, we completed the audit of all 348 verifier-accepted base-model outputs.

The trained-policy comparison is reported in Table in W2 section.

Q2. β and checkpoint selection; entropy permutation

We used β = 0.3, with the sensitivity results reported in Appendix D. The checkpoint selection rule is as follows: we train to convergence and then select the best checkpoint according to validation performance, applying the same procedure to all experiments. Regarding the entropy-shuffling control the reviewer suggested, we ran it under the Qwen3-1.7B-Base + GRPO setting, with the following results.

MethodAMC23 mean/majAIME24 mean/majAIME25 mean/majAvg. mean@32Avg. maj@32
GRPO43.36 / 47.5012.29 / 23.338.02 / 10.0021.2226.94
+ entropy-shuffling40.94 / 57.5012.08 / 16.677.08 / 13.3320.0329.17
+ UGA47.97 / 57.5013.75 / 20.009.38 / 20.0023.7032.50

Shuffling reduces macro mean@32 by 3.67 points and maj@32 by 3.33 points relative to UGA. This supports the value of the original trajectory–entropy correspondence in this setting.

Q3. A constant-gate control for the agent

This is an interesting control, and it would indeed separate entropy-dependent weighting from an overall reduction of positive updates, but we did not have time to complete the agent-side training during the rebuttal period. We did run the equivalent control on the mathematical tasks, which may serve as a reference. Under the Qwen3-1.7B-Base + GRPO setting we ran a control we call matched-weight: within each batch, each side is rescaled by the mean UGA gate over positive- or negative-advantage trajectories respectively, so that the average update strength on both sides matches UGA. It performs worse than GRPO w/ UGA, and on mean@32 it falls below GRPO as well. In other words, on the mathematical tasks a constant matched to the average strength does not reproduce the gain of UGA.

MethodAMC23 mean/majAIME24 mean/majAIME25 mean/majAvg. mean@32Avg. maj@32
GRPO43.36 / 47.5012.29 / 23.338.02 / 10.0021.2226.94
+ matched-weight42.19 / 55.0013.96 / 20.005.52 / 13.3320.5629.44
+ UGA47.97 / 57.5013.75 / 20.009.38 / 20.0023.7032.50

I thank the authors for the detailed responses and additional experiments. These results alleviate several of my concerns. However, I still consider the novelty of the work insufficient. I therefore maintain my original score.

We thank the reviewer for confirming that the additional experiments alleviate several concerns.

On novelty, we would welcome the chance to respond, since this consideration does not appear among the four weaknesses listed in the original review — those concerned the reliability interpretation, the audit cohort, coverage losses, and the agent ablations, and are the points our rebuttal was directed at.

Briefly, our claim is not that entropy weighting or difficulty weighting is new, but that the gain comes from their conditional coupling. Two controls added during this period test exactly that. Matched-weight preserves UGA’s mean gate separately on positive and negative advantages and reaches 20.56 mean@32, below unweighted GRPO at 21.22, so the gain is not reduced update magnitude. Entropy-shuffling preserves the gate distribution but breaks the trajectory correspondence and reaches 20.03. Against the closest prior methods trained under an identical setting, SEED weighting gives 21.03 and the EDA component of EDGE gives 20.56, versus 23.70 for UGA. We take the novelty question to be empirical in this form: the proposed rule outperforms both of the signals it is said to combine, and neither preserving its average strength nor its distribution reproduces it.

We would be glad to address any remaining aspect of the novelty concern if the reviewer can say which element of the design they view as already present in prior work.

Reviewer_XEGqOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Uncertainty-Gated Advantage (UGA), which rescales GRPO/DAPO advantages using trajectory-level predictive entropy and an outcome-dependent estimate of problem difficulty. It also introduces a step-level variant for self-search agents. A diagnostic audit finds an association between higher entropy and process errors among verifier-accepted base-model responses. Experiments report improvements in aggregate answer accuracy across several mathematical reasoning and knowledge QA settings.

Critical strengths
  1. Simple and practical design. UGA can be integrated into existing policy optimization pipelines without an auxiliary reward model or architectural changes.

  2. Positive results across several configurations. Average mathematical accuracy improves across both GRPO and DAPO at two model scales. The agent and cross-domain evaluations provide additional evidence of potential applicability.

Critical weaknesses
  1. SEED-GRPO is a particularly relevant omitted reference and baseline. SEED-GRPO [1] is a directly relevant method that already uses uncertainty to multiplicatively rescale GRPO advantages. Its public implementation uses 1 / (1 + alpha * semantic_entropy), which has the same reciprocal gating form as UGA’s basic gate in Eq. (6). Although the uncertainty estimator and weighting granularity differ, the basic gating function itself does not distinguish UGA from this prior approach. The absence of discussion and comparison with SEED-GRPO therefore leaves the contribution of the proposed estimator, trajectory-level weighting, and additional difficulty coupling unclear. Related trajectory-entropy-based advantage reweighting, such as the EDA component of EDGE-GRPO [2], also warrants comparison.

  2. The difficulty-aware gate relies on an important but unvalidated assumption. At a fixed entropy level, UGA applies a smaller discount to successful trajectories on low-pass-rate problems while more strongly suppressing their failures. However, a rare success may reflect either productive exploration near the model’s competence boundary or an accidental correct answer obtained through invalid reasoning. The paper does not establish when its gate preferentially preserves the former. On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models [3] emphasizes the importance of training near the competence boundary, rather than treating lower success rates as inherently indicating greater learning value. This does not invalidate UGA’s design, but makes its applicability conditions important to establish. The current audit does not test the interaction between entropy, difficulty, and process validity.

  3. The experiments do not isolate the proposed reliability mechanism or clearly establish the benefit of difficulty awareness. The process audit examines base-model outputs, rather than comparing the process reliability of policies trained with and without UGA. Consequently, it does not demonstrate that UGA reduces the reinforcement of invalid reasoning. Moreover, Table 4 provides mixed evidence for the difficulty term: fixing the pass rate to 0.5 reduces average mean@32 from 23.70 to 23.07, but improves maj@32 from 32.50 to 33.89. Replacing instance entropy with a constant or batch average also changes the distribution and magnitude of the gates, so these ablations do not fully separate instance-specific reliability information from changes in update strength. A matched-weight or entropy-shuffling control would be more informative.

  4. The advantages over strong baselines lack sufficient uncertainty analysis, and the results reveal a coverage trade-off. No multi-seed results or confidence intervals are reported. In Table 2, UGA exceeds GRPO-LEAD by only 0.13 percentage points in average mean@32 and S-GRPO by 0.83 points in average maj@32. Sampling 32 responses per question does not establish robustness to training randomness. Furthermore, on AIME25 with 1.7B GRPO, pass@32 decreases from 33.33 to 23.33 despite improvements in mean@32 and maj@32. Although this limitation is acknowledged, it constrains the interpretation of the gains as improved reasoning coverage or preserved exploration.

  5. The agent extension contains additional changes whose effects are not disentangled. Besides uncertainty gating, the agent variant introduces positive-only gating, an observation-update floor, and a fixed reasoning discount of gamma = 0.5. Without ablations, the improvement cannot be specifically attributed to uncertainty-dependent weighting. The gain over the stronger SSRL baseline with information masking is only 0.6 percentage points, and observation factuality is not directly evaluated. Constant-gate and gamma = 1 controls would help establish whether uncertainty selectivity is responsible for the improvement.

[1] SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization. [2] EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity.

Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer XEGq (Part 1/2)

We thank the reviewer for the comments. We respond to each point below.

W1. SEED-GRPO and EDGE-GRPO

We thank the reviewer for pointing us to these two closely related works, and we agree that they should be discussed and compared against. To make the comparison fair, we reimplemented the weighting components of both papers and trained and evaluated them under an identical setting, namely Qwen3-1.7B-Base + GRPO, i.e. the first block of experiments in Table 1 of the paper. The results are as follows.

MethodAMC23 mean/majAIME24 mean/majAIME25 mean/majAvg. mean@32Avg. maj@32
GRPO43.36 / 47.5012.29 / 23.338.02 / 10.0021.2226.94
+ SEED weighting42.89 / 57.5012.40 / 20.007.81 / 16.6721.0331.39
+ EDA weighting (the weighting component of EDGE)45.94 / 60.0012.40 / 13.333.33 / 3.3320.5625.56
+ UGA47.97 / 57.5013.75 / 20.009.38 / 20.0023.7032.50

SEED weighting follows the official implementation; EDA follows Eqs. (6)–(7) of the EDGE paper. Full EDGE-GRPO additionally includes Guided Error Correction, which is not part of its weighting mechanism, so we label the row EDA weighting.

The three differ in who receives the weight. SEED's weight is shared across responses to the same question. EDA weights responses within a group by their entropy, but does not distinguish trajectory outcomes and is independent of the group pass rate. The UGA gate depends jointly on a trajectory's own entropy and on where its outcome sits relative to the group pass rate, and it also achieves the best overall performance.

W2. Rare successes on low-pass-rate questions

On this point we respectfully disagree. The concern rests on the premise that the role of the difficulty term should be to identify which successes are "worth keeping". In UGA, however, the entropy term and the difficulty term serve two different purposes, and that division of labour is itself part of the design.

The entropy term judges whether the process is trustworthy: among trajectories that share both the question and the outcome, the one with higher entropy receives a smaller gate. The difficulty term protects the optimization process rather than making a reliability judgement. Its effects are:

  1. On hard groups, it preserves the scarce correct trajectories. These are the only positive signal that such a group contains, and once they are diluted by ordinary weighting, nothing can be learned from hard questions at all.
  2. On easy groups, it sharpens the penalty on the few incorrect trajectories, since an error on a question the model has already mastered deserves a heavier penalty.

This is a choice about the scarcity of learning signal, not a verdict on whether a given success was lucky. The complete behaviour emerges only when the two terms are combined: scarce successes on hard questions are preserved, but those among them with high entropy are still attenuated by the entropy term. The scenario in which high-entropy successes on low-pass-rate questions are amplified indiscriminately therefore does not arise in UGA. The ablation in Table 4 that fixes p = 0.5 (mean@32 −0.63, maj@32 +1.39) shows that removing the difficulty term lowers single-sample accuracy while slightly raising majority voting, which is consistent with the division of labour described above.

AuthorsOpenReview ↗

Response to Reviewer XEGq (Part 2/2)

W3. Isolating the mechanism and evaluating trained-policy reliability

The reviewer makes a fair point: replacing the entropy with a constant or with a batch mean changes both the distribution and the strength of the gate, so the two ablations in Table 4 are not sufficient to distinguish the two explanations. We have added one control along each of the two lines suggested.

MethodAMC23 mean/majAIME24 mean/majAIME25 mean/majAvg. mean@32Avg. maj@32
GRPO43.36 / 47.5012.29 / 23.338.02 / 10.0021.2226.94
+ matched-weight42.19 / 55.0013.96 / 20.005.52 / 13.3320.5629.44
+ entropy-shuffling40.94 / 57.5012.08 / 16.677.08 / 13.3320.0329.17
+ UGA47.97 / 57.5013.75 / 20.009.38 / 20.0023.7032.50

Matched-weight: within each batch, each side is rescaled by the mean UGA gate over positive- or negative-advantage trajectories respectively, so that the average update strength on both sides matches UGA. Entropy-shuffling: the correspondence between entropies and trajectories is randomly shuffled. Both variants fall short of UGA.

On post-training reliability, we have added an audit of process errors in the outputs of the RL-trained models. The audit procedure is the same as in the paper, except that the judge is now Claude Opus 5.5, calibrated on ProcessBench (answer-correct subset: binary accuracy 90.4%, invalid precision 81.6%, recall 88.5%). The table below reports the rate of process errors among outputs whose final answers are already correct.

PolicyAMC23AIME24AIME25
Qwen3-1.7B-Base21.19%39.29%33.33%
+ GRPO21.44%30.77%32.47%
+ GRPO w/ UGA20.85%21.21%24.44%

UGA has lower audited process-error rates than GRPO on all three benchmarks, with the largest reductions on AIME24 and AIME25.

W4. Statistical uncertainty and coverage

We distinguish evaluation sampling variability from training-seed variability. The evaluation uses 32 responses per question, with mean@32 measuring average correctness and maj@32 measuring majority-vote accuracy. We additionally report the evaluation sampling standard deviations below.

ConfigurationAMC23AIME24AIME25
Qwen3-1.7B + GRPO43.36 ± 5.9512.29 ± 4.128.02 ± 3.21
Qwen3-1.7B + GRPO + UGA47.97 ± 6.1713.75 ± 3.979.38 ± 3.77
Qwen3-1.7B + DAPO45.47 ± 5.9812.19 ± 3.598.33 ± 3.91
Qwen3-1.7B + DAPO + UGA47.03 ± 6.4814.69 ± 4.159.69 ± 3.67
Qwen3-4B + GRPO63.36 ± 4.6223.65 ± 4.9620.21 ± 3.99
Qwen3-4B + GRPO + UGA62.66 ± 4.3726.25 ± 5.8222.81 ± 5.34
Qwen3-4B + DAPO67.11 ± 7.6629.38 ± 5.3625.52 ± 4.69
Qwen3-4B + DAPO + UGA69.92 ± 6.3031.15 ± 5.6326.98 ± 4.81

These statistics concern evaluation sampling at fixed checkpoints. The training experiments use a single seed, and we acknowledge that independent training repetitions are needed to assess that source of uncertainty.

Regarding coverage, the 1.7B GRPO AIME25 pass@32 decrease from 33.33% to 23.33% is real. Across the 12 benchmark-by-training comparisons in Appendix E, pass@32 improves or remains unchanged in nine and decreases in three. We therefore limit our conclusion to improved macro mean@32 and maj@32 in the evaluated configurations; these gains do not imply uniformly preserved exploration or expanded reasoning coverage.

W5. Agent ablations and the SSRL baselines

Regarding the positive-only gating and the discount factor, these are indeed ablations that could be added, and because of compute limitations we were unable to isolate each variable during the rebuttal period. On the choice of the SSRL baseline, however, we would like to clarify our position.

SSRL's info mask excludes the retrieved observation tokens from the loss. This is a form of avoidance: since updating on hallucinated observations is risky, the masked variant simply does not update on observations at all. UGA addresses the same problem, but by down-weighting according to uncertainty rather than excluding wholesale. The like-for-like comparison is therefore against SSRL without the mask (+1.7 points), which is the condition under which observations participate in the update. The masked variant is a strong reference point from a different setting; we report it as well (+0.6 points) for completeness, not because it is the appropriate control.

Could you please add an experiment on Qwen3-4B + DAPO + SEED-GRPO?

Thank you for the suggestion. We will run it.

To be transparent about timing: the discussion period closes today, and this configuration requires a full Qwen3-4B DAPO run with SEED weighting plus 32-sample decoding on all three benchmarks. We cannot finish that before the deadline, and we would rather not post numbers from an under-trained run, which would be unfair to the baseline you are asking us to compare against.

We will include this experiment in a future version of the paper. If this discussion thread still accepts updates after the deadline, we will post the result here as soon as the run completes.

Reviewer_vsyqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Uncertainty-Gated Advantage (UGA), a lightweight advantage-reweighting mechanism for RLVR. UGA combines token entropy with problem difficulty to reweight the advantages used in GRPO and DAPO, aiming to reduce reinforcement of potentially unreliable correct trajectories. The method is also extended to step-level optimization in Self-Search Agents. Experiments show improvements on mathematical and knowledge-intensive QA benchmarks.

Critical strengths
  1. UGA uses a lightweight advantage-reweighting mechanism, which can be easily integrated into existing policy opyimization algorithm.
  2. Experiments show the effectiveness of the proposed UGA on both GRPO and DAPO, and against existing advantage-reweighting methods.
Critical weaknesses
  1. GRPO-LEAD already incorporates difficulty-dependent advantage weighting, while ERPO uses entropy-aware gating. UGA primarily combines these two signals through a new weighting function. Moreover, its average mean@32 improvement over GRPO-LEAD is only 0.13 points in Table 2, making the practical benefit of the proposed method somewhat unclear, and limited novelty over existing advantage-reweighting methods.
  2. The claimed improvement in reasoning reliability is not directly validated. Sec. 2.3 establishes a correlation between entropy and process errors using trajectories from base model, but does not examine whether UGA actually reduces process errors after RL training. Consequently, the main results demonstrate improved answer accuracy rather than improved reasoning-process reliability.
  3. Insufficient analysis of the difficulty-dependent gating function. Eq. (8) becomes undefined when r(τ)=p(q)r(\tau)=p(q), which occurs in all-success/all failed groups. The numerical treatment of these cases is unspecified. Moreover, the gate may strongly suppress advantages when the group pass rate approaches 0 or 1, but the resulting optimization behavior is not systematically analyzed.
  4. The agent improvement is not clearly attributable to uncertainty gating. Eq. (11) applies an additional discount factor γ=0.5\gamma=0.5 to positive reasoning advantages, substantially reducing their magnitude even without entropy-based modulation. Without a corresponding constant-discount ablation, the improvements in Table 3 cannot be clearly attributed to the proposed uncertainty mechanism.
Questions for authors
  1. Since the proposed gate always satisfies ψ≤1\psi \leq 1, UGA systematically reduces advantage magnitudes. Could the authors compare against a baseline with a constant scaling factor matched to UGA's average gate value, to distinguish uncertainty-aware reweighting from reduced effective update magnitude?
  2. Positive vs. Negative Advantage Gating: Eq. (8) reweights both positive and negative advantages, while the agent extension only gates positive advantages. Could the authors provide an ablation comparing positive-only, negative-only, and bidirectional gating to clarify which component contributes most to the improvement?
  3. Eqs.4–5 compute trajectory uncertainty using mean top-k token entropy. How sensitive are the results to the choice of k?
4 comments
AuthorsOpenReview ↗

Response to Reviewer vsyq (Part 1/2)

Thank you for your review. We address each concern below.

W1. Novelty and practical benefit

We agree that difficulty-aware and entropy-aware weighting both have precedents. UGA's specific contribution is to couple trajectory-level uncertainty with the trajectory's outcome and the current policy's group pass rate, so that the strength of entropy-based attenuation depends on the success/failure context.

GRPO-LEAD's difficulty-weighting component depends on the pass rate and advantage sign; it does not distinguish responses to the same question with the same outcome using their individual entropies. ERPO assigns higher relative weight to high-entropy decision points at the token level. UGA instead attenuates the original advantage using aggregate trajectory entropy, conditional on outcome and pass rate. For successful trajectories with the same entropy, UGA applies less attenuation on low-pass-rate questions and more attenuation on high-pass-rate questions. Our contribution is this conditional weighting rule, rather than either signal in isolation.

To test whether the gains arise only from average advantage scaling and whether the trajectory–entropy correspondence matters, we added two controls using the Qwen3-1.7B-Base + GRPO setting, consistent with the first block of experiments in Table 1 of the paper.

  • Matched-weight: Within each batch, we compute the mean UGA gate separately over positive- and negative-advantage trajectories, then use the corresponding mean to scale every advantage on that side. This preserves the mean gate for each sign while removing within-sign trajectory selectivity.
  • Entropy-shuffling: We randomly shuffle the correspondence between entropy values and trajectories before computing the gates.
MethodAMC23 mean/majAIME24 mean/majAIME25 mean/majAvg. mean@32Avg. maj@32
GRPO43.36 / 47.5012.29 / 23.338.02 / 10.0021.2226.94
+ matched-weight42.19 / 55.0013.96 / 20.005.52 / 13.3320.5629.44
+ entropy-shuffling40.94 / 57.5012.08 / 16.677.08 / 13.3320.0329.17
+ UGA47.97 / 57.5013.75 / 20.009.38 / 20.0023.7032.50

Full UGA improves over both matched-weight and entropy-shuffling. In other words, under this setting, neither preserving only the average gate coefficients on the two sides nor shuffling the entropy-to-trajectory correspondence reproduces the performance of full UGA. These results support the view that the selective weighting guided by per-trajectory uncertainty carries real value, and that the particular coupling used by UGA is not simply a combination of two pre-existing signals.

W2. Reasoning-process reliability after RL training

We have added a post-training process audit using the same rubric and blinded protocol as in the paper, with Claude Opus 5.5 as the judge. We recalibrated this judge on ProcessBench. On its final-answer-correct subset, binary accuracy is 90.4%, with 81.6% precision and 88.5% recall for the process-invalid class. The table below reports the rate of process errors among outputs whose final answers are already correct.

PolicyAMC23AIME24AIME25
Qwen3-1.7B-Base21.19%39.29%33.33%
+ GRPO21.44%30.77%32.47%
+ GRPO w/ UGA20.85%21.21%24.44%

The direction is consistent across all three datasets, with the largest reductions on the two AIME sets. This indicates that UGA does reduce process errors after RL training, making the reasoning process more reliable.

W3. The boundary case r = p and extreme pass rates

This point requires a clarification. Under binary rewards, r ∈ {0,1}, so r = p can only occur in groups that are entirely correct or entirely incorrect. In such groups the GRPO advantage is identically zero to begin with, and DAPO's dynamic sampling filters them out beforehand in any case. It is therefore not an undefined point that can affect training.

W4. Attribution of the agent improvement

Equation (11) uses the entropy of the immediately preceding observation to modulate a positive reasoning advantage: uncertain observations can make the following reasoning less reliable. However, your concern is also valid: the fixed factor γ = 0.5 reduces positive reasoning advantages even when entropy-dependent modulation is removed. Explaining the entropy term does not isolate its contribution from this fixed discount.

We have not completed the corresponding constant-discount ablation during the rebuttal period. We therefore interpret the agent result as evidence for the combined design, rather than attribute the entire gain specifically to uncertainty gating.

AuthorsOpenReview ↗

Response to Reviewer vsyq (Part 2/2)

Q1. A constant scaling baseline matched to the average gate

The matched-weight control in Table in W1 addresses this question by matching the mean gate separately for positive and negative advantages within each batch. This is a more targeted control than a single shared constant, although matching mean gates does not imply identical gradient magnitudes or parameter updates. It reaches 20.56 mean@32 and 29.44 maj@32, compared with UGA's 23.70 and 32.50. Its mean@32 is also below unweighted GRPO's 21.22. These results support selective weighting beyond sign-specific average scaling.

Q2. Positive-only, negative-only, and bidirectional gating

This is a useful ablation. The mathematical setting uses bidirectional gating, whereas the Self-Search Agent variant gates only positive advantages. The agent design aims to avoid assigning excessive positive credit to potentially hallucinated observations while retaining negative penalties. We did not have sufficient compute during the rebuttal period to complete this ablation, and we will explore the contribution of the individual components in future work.

Q3. Sensitivity to top-k

We recomputed trajectory uncertainty for 3,200 trajectories at different values of k and measured Spearman rank correlation against k = 50. The table below reports the stability of the trajectory-uncertainty ranking across top-k.

k102050100Full vocabulary
Spearman ρ vs. k = 500.999650.999961.000001.000000.99999

We find that the choice of k is not particularly sensitive. During generation, probability mass is typically concentrated on a small number of highest-probability tokens, while tokens beyond roughly the top 10 carry very low probability; this is also one of the reasons we adopt a top-k truncation.

Thank you for the detailed rebuttal and additional experiments. The matched-weight and entropy-shuffling controls strengthen the evidence for trajectory-specific uncertainty weighting. However, the contribution of the proposed difficulty–entropy interaction over entropy-only gating remains unclear, particularly given the modest difference from the fixed-pass-rate variant in Table 4. And the agent experiments still do not isolate entropy gating from the fixed advantage discount.

Overall, I will maintain my current score.

Thank you for the follow-up. On the agent experiments we agree: the fixed γ = 0.5 discount is not isolated from entropy gating, and we will add the constant-discount ablation.

On the interaction, one clarification about Table 4 bears on the comparison. Fixing p = 0.5 does not reduce UGA to entropy-only gating: under binary rewards it sets (r − p)² ≡ 0.25, a uniform difficulty prior in which every trajectory is treated as a median case. That variant remains a member of UGA’s gate family rather than a difficulty-free baseline, so its distance from full UGA understates what the coupling contributes. The control that does break the difficulty signal while staying in the same family is the inverted row: swapping the difficulty factor between correct and incorrect trajectories within each group reuses the same set of gate values and reaches 20.63 mean@32, below unweighted GRPO at 21.22. A term carrying no information would not become actively harmful when its direction is reversed.

That said, we will scope the claim in the revision rather than overstate it: against the uniform-prior variant the pass-rate term adds roughly 0.6 mean@32, concentrated on the low-pass-rate benchmarks, with maj@32 moving the other way on AMC23.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.