- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies diversity collapse in RLVR, where Pass@1 improves while high-budget Pass@k declines, through the lens of overtraining. It distinguishes training beyond validation accuracy gains from continued optimization of individual problems whose high-k coverage is already nearly saturated. A Beta–Binomial analysis estimates the marginal coverage utility from a small number of rollouts. The paper evaluates early stopping and success-bucket filtering, and proposes Bayesian Boundary Gating (BBG), which suppresses updates on higher-success problems while retaining equal weights for the remaining problems. Experiments on Qwen2.5-Math models report improved coverage retention, often at the cost of lower Pass@1 than standard baselines.
Critical strengths
The distinction between single-attempt accuracy and high-budget coverage is useful. The analysis explains clearly why a problem can retain substantial Pass@1 headroom while offering little additional Pass@k benefit. This provides a practical perspective on allocating training effort under different inference budgets.
The paper goes beyond aggregate performance by tracking coverage gains and losses and intervening on training duration and rollout-success buckets. These experiments provide useful evidence that the scope and duration of optimization affect coverage retention.
Critical weaknesses
The stability of the negative-reinforcement-heavy training recipe is insufficiently established. BBG builds on W-RF, which downweights successful trajectories and thereby increases the relative contribution of negative reinforcement. In my own RL experiments, increasing the relative weight of negative samples has sometimes made learning substantially harder: excessively long incorrect responses can dominate gradient contributions and cause training collapse, depending on the loss reduction and length normalization. This is not evidence that the authors’ runs exhibit the same failure, but it motivates an important robustness concern. The paper does not adequately examine sensitivity to trajectory length, positive/negative weighting, or baseline estimation. Appendix A.4 specifies an EMA baseline for REINFORCE; its interaction with asymmetric weighting and gating needs clearer specification and validation. Without these analyses and independent training repetitions, the practical robustness of the proposed recipe remains uncertain.
The contribution of gating is not cleanly isolated from the underlying update rule and effective optimization budget. Table 2 compares BBG against REINFORCE, GRPO, and MaxRL, but omits W-RF alone, even though BBG builds on W-RF. BBG also buffers active prompts, and the POLARIS comparison aligns rollout steps rather than optimizer updates. Consequently, coverage retention could partly reflect asymmetric reinforcement or reduced optimization rather than the particular selection rule. A standalone W-RF baseline and random filtering with matched effective update budgets are important controls. Moreover, the marginal-utility analysis explains saturation of the coverage objective but does not establish why updates on higher-success problems harm lower-success problems through shared parameters. These gaps weaken the attribution of the observed gains to the proposed overtraining mechanism.
Questions for authors
Implementation clarification: What are the exact advantage formulas and baseline-update rules for W-RF and BBG? Is the EMA baseline computed before or after gating, and how is the loss normalized across tokens and trajectories?
Evidence needed to assess robustness: How sensitive are the results to the positive-reinforcement weight and maximum response length? Can the authors provide training-stability and gradient-contribution diagnostics for long incorrect trajectories, together with variation across independent training runs?
Evidence needed to isolate the contribution: Does BBG outperform W-RF alone and random filtering when rollout budgets, effective batch sizes, and optimizer-update counts are explicitly controlled or separately reported?
Reporting clarification: Section 4.1 attributes MinervaMath Pass@256 of 77.9 to full REINFORCE, whereas Table 1 lists 77.9 under W-RF and 72.4 under REINFORCE. Which underlying method is used for the early-stopping and progressive-masking comparisons?
6 comments
Response to Reviewer FHjd (Part2/2)
2. Stability of negative-reinforcement-heavy training (W1, Q1, Q2)
Our experience differs from yours on this point, and we would like to discuss it.
Subjectively, across the close to 100+ times W-RF and negative-only runs we have trained, we did not observe a clear difference in training stability from GRPO or positive-only training. As you suggest, we do observe transient spikes in the pre-clipping gradient norm; however, we observe similar spikes in positive-only runs, and all of these runs completed successfully. Looking at this more objectively: in the W-RF training traces we inspected (7B on MATH, and 1.5B on MATH and POLARIS-53K), we observed no sustained training collapse. In every run, the last recorded MATH validation accuracy was within 0.5 percentage points of the best value recorded in that run.
In our understanding, the failure mode you describe may be more closely tied to how the loss is reduced over response length. With per-trajectory length normalization, each token of a long incorrect response is penalized less than each token of a short one, which rewards being long when wrong and can lead to runaway response length (Reference: Understanding R1-Zero-Like Training: A Critical Perspective); with a sum over tokens, long incorrect responses dominate the gradient directly. We follow the reference implementation of W-RF, which takes a masked mean of the clipped policy loss over valid response tokens within each actor microbatch, so every token of an incorrect response receives the same penalty regardless of the response's length.
Following your suggestion, we went back to the training logs to check for the specific symptoms of this failure mode. Incorrect responses are indeed longer than correct ones on average (about 1,210 versus 562 tokens for W-RF at the first update), so the precondition for the failure mode was present. Their length nonetheless decreased during training, consistent with the token-mean reduction:
| Qwen2.5-Math-7B, MATH, (first update → last validated checkpoint) | W-RF | Gate-only BBG |
|---|---|---|
| Mean length of incorrect responses (tokens) | ~1,210 → 1,056 | ~1,187 → 1,062 |
| Fraction of responses reaching the 3,072-token cap | 8.0% → 0.3% | 10.6% → 2.4% |
Implementation details (Q1): neither W-RF nor BBG uses the EMA baseline described for REINFORCE in Appendix A.4. With binary verifier scores, , and no KL penalty, W-RF assigns each valid response token an advantage of on a correct rollout and on an incorrect rollout. BBG keeps this rule and multiplies the policy loss by the binary problem-level gate in Eq. (11).
The lengths by correctness are derived from logged aggregates; we did not log per-trajectory gradient contributions; all reported experiments use a 3,072-token response cap.
3. Reporting (Q4)
The early-stopping and masking rows in Table 1 come from the W-RF run; Section 4 and Appendix A.7 incorrectly stated REINFORCE. We have corrected this in the revised manuscript.
We thank you again for the discussion. It has helped us separate the roles of the selection rule, the update rule, and the optimization budget in the paper, which has improved the manuscript. We hope our responses have addressed your main concerns, and we would be grateful if you could reconsider your assessment of our paper accordingly.
Best regards,
The Authors
Thank you for the detailed response and the additional results. I have read the rebuttal and the replies to the other reviewers.
The implementation details, the corrected W-RF/REINFORCE reporting, and the added W-RF baseline are helpful. My main concerns remain. The random-filtering control at matched update counts was not provided, and the early-stopping comparison is not conclusive. BBG’s update count (448) lies between the 5- and 10-epoch W-RF checkpoints (280 and 560), and its Pass@256 margin over them is 0 to 3.6 points on MATH-500, OlympiadBench and MinervaMath, with the AIME 2025 gap amounting to two or three problems. These are single runs, and the response to Reviewer Gb7c confirms that multi-seed results are deferred to a later version. On stability, the response relies on informal experience and aggregate length statistics. No sensitivity results for the positive-reinforcement weight or the response cap were provided, and per-trajectory gradient contributions were not logged.
I am therefore keeping my score.
Response to Reviewer FHjd (Part1/2)
Dear Reviewer FHjd,
We thank you for the careful review. We understand your two main question to be: (1) whether coverage retention can be attributed to the selection rule, rather than to the underlying W-RF update or simply to less optimization (W2, Q3); and (2) whether the negative-reinforcement-heavy recipe is stable (W1, Q1, Q2). We have considered both carefully, and your practical experience with negative reinforcement is a useful point of contrast with ours. Since attribution bears most directly on the paper's claim, we discuss it first, then stability, and finally the reporting inconsistency (Q4).
To respect the conference's call for brevity and reduce the reviewers' reading burden, we keep our response concise.
1. Attribution of coverage retention (W2, Q3)
For clarity, we break it into three sub-questions:
(i) whether gating adds to W-RF itself;
(ii) whether coverage retention simply reflects less optimization;
(iii) why updates on high-success problems would reduce coverage on other problems.
The 7B comparison below answers (i) and (ii). All rows use the W-RF update rule, and BBG adds only the problem-level gate, and the early-stopping rows stop the same full W-RF run earlier, with no problem selection. Each cell reports Pass@1 / Pass@256.
(i) Gating on top of W-RF. Compared with W-RF alone, BBG raises Pass@256 by 2.2, 4.8, 3.2, and 10.0 points on the four benchmarks. As you note, W-RF already mitigates diversity collapse, but it does not remove it, and the gate further mitigates the remaining loss.
(ii) Less optimization. The reviewer is right that BBG performs less optimization than full W-RF. A core argument of our paper is that the standard training recipe overtrains: after validation Pass@1 plateaus, further optimization mostly costs coverage (Section 3.1). Section 4.3 defines BBG as early stopping at the problem level: it stops optimizing only the problems whose remaining coverage benefit is small, while training continues on the rest.
Nonetheless, we agree with the reviewer that this confound should be ablated to isolate the contribution of BBG. We can therefore compare directly with W-RF in which optimization is reduced by early stopping. Relative to full W-RF, both early stopping and BBG recover coverage; at similar Pass@1, BBG's per-problem stopping recovers more than stopping the whole run.
(iii) Effects on other problems. We agree that the marginal-utility analysis answers a within-problem question and does not predict how updating on one problem affects other problems. Section 3.2 states this part as a condition (Eq. (10)): if continued optimization also lowers success probabilities on low-coverage problems, these losses can outweigh the gains on already-covered problems. The bucket-restricted experiment (Figure 5) tests this condition empirically. After 28 updates restricted to B8/8, validation token entropy falls from 0.112 to 0.037, and Pass@256 falls below the base model on both MinervaMath and OlympiadBench. Restricting updates to B0/8 gives the opposite pattern.