- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Uncertainty-Gated Advantage (UGA), which rescales detached GRPO/DAPO advantages using mean top-50 token entropy and a rollout group's empirical pass rate. It also adapts entropy gating to a self-search agent. On Qwen3-1.7B/4B mathematical reasoning, the paper reports improved macro mean@32 and maj@32 in four training configurations, plus agent and cross-domain results. A blinded, ProcessBench-calibrated judge finds that high-entropy, verifier-accepted base-model solutions are enriched for process errors. This diagnostic motivates UGA, but does not directly measure whether UGA reduces such errors after training.
Critical strengths
-
Simple method with informative controls. UGA requires no new learned reward model and adds roughly 10% reported training time. Table 1 shows positive mean@32 changes in 11 of 12 benchmark-by-training comparisons, with positive macro gains for both GRPO and DAPO at both sizes. This correlated sign pattern is favorable descriptive evidence. Table 4's batch-mean entropy, constant entropy, and reversed pass-rate controls all weaken mean@32. They support the full reweighting configuration, although they also change gate magnitudes and do not independently identify a process-reliability mechanism.
-
A useful, cautiously presented diagnostic. The process audit is blinded to entropy and model identity, calibrated against ProcessBench, and explicitly presented as an association rather than a causal proof. The authors also report mixed outcomes, including a higher maj@32 for the fixed-pass-rate ablation and some decreases in large-k coverage. This candor makes the evidence easier to interpret.
Critical weaknesses
-
The reliability interpretation is not established by the entropy audit or the gate's algebra. Mean token entropy measures uncertainty over token choices, not whether the reasoning is valid. More specifically, Eq. (8) also implements outcome-dependent difficulty reweighting: at the same entropy, a high pass rate suppresses correct trajectories more than incorrect ones, and a low pass rate reverses this relation. This can affect optimization without identifying lucky guesses. The audit covers only verifier-correct base-model trajectories; it does not validate the entropy/error association conditional on the gate's pass-rate signal, or justify the negative-advantage weighting. Thus the full configuration's benefit is compatible with difficulty-dependent update rebalancing, while the more distinctive claim of reliability-aware credit remains unverified.
-
Figure 1's 7.8% versus 45.3% association is useful motivation, but its 314 traces cover only 41 questions, with 290 from AMC23, and entropy quintiles are formed within benchmarks rather than within questions. The critic's precision is 0.783 on its correct-answer calibration subset (138 false positives, 22 false negatives), so calibration does not eliminate annotation concerns. Blinding does not control question difficulty or visible solution properties such as length. Most importantly, no controlled comparison shows fewer process errors among verifier-correct UGA outputs than GRPO outputs after training. The authors acknowledge the audit's correlational scope; nevertheless, this missing link limits the evidence for the paper's central motivation, rather than merely omitting a peripheral implementation detail.
-
Appendix E reports some large-k coverage losses, including 1.7B GRPO AIME25 pass@32 falling from 33.33 to 23.33 with UGA; this qualifies the robust-reasoning interpretation while remaining compatible with improved average single-sample accuracy. Table 5's cross-domain transfer is also mixed, rather than evidence of a broadly validated reliability signal. Finally, the agent extension changes the gate and advantage-sign treatment and adds a fixed gamma = 0.5 discount. Without retaining those changes and removing only entropy dependence, its accuracy gains cannot independently validate the math gate's proposed mechanism. The paper reports several of these qualifications candidly, but the remaining support is for a useful reweighting heuristic with limited demonstrated advantage over the closest alternative.
Questions for authors
- Please clarify whether the entropy/error association persists after accounting for question, pass rate, and solution length, and reconcile the cohort counts: Table 1 implies approximately 348 accepted base-model rollouts, whereas Appendix F.1 reports 314 after excluding 12. Different verification or filtering may explain this. A small process audit of verifier-correct GRPO and UGA outputs on the same questions, for one representative trained configuration, would directly test the reliability interpretation and could change my assessment.
- How were beta and checkpoints selected, and what uncertainty do the existing paired question-level outputs imply for the close Table 2 comparisons? To distinguish trajectory-specific entropy alignment from average reweighting strength, a useful targeted control would permute entropies within each question and reward outcome, preserving the outcome-specific gate distributions. One representative setting would suffice; it is an alternative way to strengthen the mechanism evidence, rather than a request for a broad new experimental suite.
- With the fixed gamma = 0.5 discount and the advantage-sign treatment retained, what happens if the entropy-dependent factors in Eqs. (10)–(11) are replaced by constants matched to their average strength? This would distinguish uncertainty-dependent weighting from the overall reduction in positive updates, and materially affect how I interpret the agent extension.
4 comments
Response to Reviewer a7mf (Part 2/2)
Q1. Audit cohort counts and trained-policy outputs
Thank you for identifying the discrepancy. You are correct that Table 1 implies approximately 348 verifier-accepted base-model outputs. With 32 responses per question, the counts are approximately 302 for AMC23 (40 questions × 32 × 23.59%), 28 for AIME24 (30 × 32 × 2.92%), and 18 for AIME25 (30 × 32 × 1.88%). The recorded counts are exactly 302, 28, and 18, totaling 348.
The original process audit did not cover all of these outputs. It covered 326 candidates, excluded 12 with uncertain answer or process status, and reported the remaining 314. Thus, 22 of the 348 verifier-accepted outputs were not covered by that original audit. Figure 1's total of 314 is consistent with the retained audit cohort, but does not account for those 22 unaudited outputs.
For this rebuttal, we completed the audit of all 348 verifier-accepted base-model outputs.
The trained-policy comparison is reported in Table in W2 section.
Q2. β and checkpoint selection; entropy permutation
We used β = 0.3, with the sensitivity results reported in Appendix D. The checkpoint selection rule is as follows: we train to convergence and then select the best checkpoint according to validation performance, applying the same procedure to all experiments. Regarding the entropy-shuffling control the reviewer suggested, we ran it under the Qwen3-1.7B-Base + GRPO setting, with the following results.
| Method | AMC23 mean/maj | AIME24 mean/maj | AIME25 mean/maj | Avg. mean@32 | Avg. maj@32 |
|---|---|---|---|---|---|
| GRPO | 43.36 / 47.50 | 12.29 / 23.33 | 8.02 / 10.00 | 21.22 | 26.94 |
| + entropy-shuffling | 40.94 / 57.50 | 12.08 / 16.67 | 7.08 / 13.33 | 20.03 | 29.17 |
| + UGA | 47.97 / 57.50 | 13.75 / 20.00 | 9.38 / 20.00 | 23.70 | 32.50 |
Shuffling reduces macro mean@32 by 3.67 points and maj@32 by 3.33 points relative to UGA. This supports the value of the original trajectory–entropy correspondence in this setting.
Q3. A constant-gate control for the agent
This is an interesting control, and it would indeed separate entropy-dependent weighting from an overall reduction of positive updates, but we did not have time to complete the agent-side training during the rebuttal period. We did run the equivalent control on the mathematical tasks, which may serve as a reference. Under the Qwen3-1.7B-Base + GRPO setting we ran a control we call matched-weight: within each batch, each side is rescaled by the mean UGA gate over positive- or negative-advantage trajectories respectively, so that the average update strength on both sides matches UGA. It performs worse than GRPO w/ UGA, and on mean@32 it falls below GRPO as well. In other words, on the mathematical tasks a constant matched to the average strength does not reproduce the gain of UGA.
| Method | AMC23 mean/maj | AIME24 mean/maj | AIME25 mean/maj | Avg. mean@32 | Avg. maj@32 |
|---|---|---|---|---|---|
| GRPO | 43.36 / 47.50 | 12.29 / 23.33 | 8.02 / 10.00 | 21.22 | 26.94 |
| + matched-weight | 42.19 / 55.00 | 13.96 / 20.00 | 5.52 / 13.33 | 20.56 | 29.44 |
| + UGA | 47.97 / 57.50 | 13.75 / 20.00 | 9.38 / 20.00 | 23.70 | 32.50 |
I thank the authors for the detailed responses and additional experiments. These results alleviate several of my concerns. However, I still consider the novelty of the work insufficient. I therefore maintain my original score.
We thank the reviewer for confirming that the additional experiments alleviate several concerns.
On novelty, we would welcome the chance to respond, since this consideration does not appear among the four weaknesses listed in the original review — those concerned the reliability interpretation, the audit cohort, coverage losses, and the agent ablations, and are the points our rebuttal was directed at.
Briefly, our claim is not that entropy weighting or difficulty weighting is new, but that the gain comes from their conditional coupling. Two controls added during this period test exactly that. Matched-weight preserves UGA’s mean gate separately on positive and negative advantages and reaches 20.56 mean@32, below unweighted GRPO at 21.22, so the gain is not reduced update magnitude. Entropy-shuffling preserves the gate distribution but breaks the trajectory correspondence and reaches 20.03. Against the closest prior methods trained under an identical setting, SEED weighting gives 21.03 and the EDA component of EDGE gives 20.56, versus 23.70 for UGA. We take the novelty question to be empirical in this form: the proposed rule outperforms both of the signals it is said to combine, and neither preserving its average strength nor its distribution reproduces it.
We would be glad to address any remaining aspect of the novelty concern if the reviewer can say which element of the design they view as already present in prior work.
Response to Reviewer a7mf (Part 1/2)
We thank the reviewer for the comments. We respond to each point below.
W1. Interpretation of the reliability mechanism
We agree that mean token entropy is not a certificate of valid reasoning. The original diagnostic establishes an association between entropy and process errors among verifier-accepted base-model responses. UGA uses this association as a noisy training signal.
The pass-rate term regulates the allocation of credit: it attenuates rare successes less on low-pass-rate questions and retains more of the negative penalty for rare failures on high-pass-rate questions. This does not establish that a rare success is process-valid. Likewise, reduced process-error rates after training would not, by themselves, uniquely identify the gate's mechanism.
The new matched-weight and entropy-shuffling controls test whether trajectory-specific weighting contributes beyond average gate scaling, while the new trained-policy audit evaluates the resulting outputs. These provide complementary evidence. A conditional analysis accounting for question, pass rate, and solution length would further strengthen the reliability interpretation, and the rationale for negative-advantage gating remains distinct from the audit of correct outputs. We do not regard these questions as resolved solely by the gate's algebra.
W2. Judge error and the post-training comparison
We answer the two points in turn.
1. Judge error. We agree that a precision of 0.783 on the original calibration subset is not sufficient to settle this concern. The new audit uses Claude Opus 5.5 and is recalibrated on ProcessBench: on the answer-correct subset, binary accuracy is 90.4%. Re-auditing all correct outputs of Qwen3-1.7B-Base with this judge gives a process-error rate of 23.3%, close to the 22.0% reported in the paper.
2. Post-training comparison. This is precisely the experiment the reviewer identified as potentially changing the assessment. On the representative 1.7B GRPO configuration, we blind-judged the outputs of GRPO and GRPO w/ UGA. The table below reports the rate of process errors among outputs whose final answers are already correct.
After training, the UGA outputs accepted by the verifier do contain fewer process errors than those of GRPO.
W3. Coverage, cross-domain transfer, and the agent extension
Regarding coverage, the 1.7B GRPO AIME25 pass@32 decrease from 33.33% to 23.33% is real. Appendix E also reports decreases on AMC23 for 4B GRPO and 4B DAPO. Across the 12 benchmark-by-training comparisons, pass@32 improves or remains unchanged in nine and decreases in three; we therefore do not claim uniformly preserved exploration.
We also retain the mixed cross-domain results and limit the transfer claim to the evaluated configurations where improvements are observed. The agent results support the combined variant's answer accuracy, but do not isolate the contributions of entropy gating, advantage-sign treatment, and the fixed γ = 0.5 discount.