acceptodds
Under review as a conference paper at ICLR 2027

Understanding Diversity Collapse in RLVR via the Lens of Overtraining

Abstract

Reinforcement learning with verifiable rewards (RLVR) often suffers from diversity collapse: Pass@ improves while high- Pass@ degrades. We study this divergence through the lens of overtraining. We first show that validation Pass@ largely plateaus while training-set success continues to climb and high- coverage continues to decline. We then show that a problem can retain substantial Pass@ headroom after its direct contribution to high- coverage has nearly saturated. A Bayesian analysis quantifies this distinction under few-rollout uncertainty. Early stopping reduces coverage loss. We also find that stopping updates on high-success problems preserves or improves Pass@ relative to the base model while retaining Pass@ gains, even as training continues on the remaining problems. Together, these results identify overtraining as a source of diversity collapse in RLVR: continued optimization outlasts its coverage benefit, and restricting its period or scope reduces the resulting coverage loss.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning with verifiable rewards, RLVR, diversity collapse, mode collapse, post-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FHjdOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies diversity collapse in RLVR, where Pass@1 improves while high-budget Pass@k declines, through the lens of overtraining. It distinguishes training beyond validation accuracy gains from continued optimization of individual problems whose high-k coverage is already nearly saturated. A Beta–Binomial analysis estimates the marginal coverage utility from a small number of rollouts. The paper evaluates early stopping and success-bucket filtering, and proposes Bayesian Boundary Gating (BBG), which suppresses updates on higher-success problems while retaining equal weights for the remaining problems. Experiments on Qwen2.5-Math models report improved coverage retention, often at the cost of lower Pass@1 than standard baselines.

Critical strengths

The distinction between single-attempt accuracy and high-budget coverage is useful. The analysis explains clearly why a problem can retain substantial Pass@1 headroom while offering little additional Pass@k benefit. This provides a practical perspective on allocating training effort under different inference budgets.

The paper goes beyond aggregate performance by tracking coverage gains and losses and intervening on training duration and rollout-success buckets. These experiments provide useful evidence that the scope and duration of optimization affect coverage retention.

Critical weaknesses

The stability of the negative-reinforcement-heavy training recipe is insufficiently established. BBG builds on W-RF, which downweights successful trajectories and thereby increases the relative contribution of negative reinforcement. In my own RL experiments, increasing the relative weight of negative samples has sometimes made learning substantially harder: excessively long incorrect responses can dominate gradient contributions and cause training collapse, depending on the loss reduction and length normalization. This is not evidence that the authors’ runs exhibit the same failure, but it motivates an important robustness concern. The paper does not adequately examine sensitivity to trajectory length, positive/negative weighting, or baseline estimation. Appendix A.4 specifies an EMA baseline for REINFORCE; its interaction with asymmetric weighting and gating needs clearer specification and validation. Without these analyses and independent training repetitions, the practical robustness of the proposed recipe remains uncertain.

The contribution of gating is not cleanly isolated from the underlying update rule and effective optimization budget. Table 2 compares BBG against REINFORCE, GRPO, and MaxRL, but omits W-RF alone, even though BBG builds on W-RF. BBG also buffers active prompts, and the POLARIS comparison aligns rollout steps rather than optimizer updates. Consequently, coverage retention could partly reflect asymmetric reinforcement or reduced optimization rather than the particular selection rule. A standalone W-RF baseline and random filtering with matched effective update budgets are important controls. Moreover, the marginal-utility analysis explains saturation of the coverage objective but does not establish why updates on higher-success problems harm lower-success problems through shared parameters. These gaps weaken the attribution of the observed gains to the proposed overtraining mechanism.

Questions for authors

Implementation clarification: What are the exact advantage formulas and baseline-update rules for W-RF and BBG? Is the EMA baseline computed before or after gating, and how is the loss normalized across tokens and trajectories?

Evidence needed to assess robustness: How sensitive are the results to the positive-reinforcement weight and maximum response length? Can the authors provide training-stability and gradient-contribution diagnostics for long incorrect trajectories, together with variation across independent training runs?

Evidence needed to isolate the contribution: Does BBG outperform W-RF alone and random filtering when rollout budgets, effective batch sizes, and optimizer-update counts are explicitly controlled or separately reported?

Reporting clarification: Section 4.1 attributes MinervaMath Pass@256 of 77.9 to full REINFORCE, whereas Table 1 lists 77.9 under W-RF and 72.4 under REINFORCE. Which underlying method is used for the early-stopping and progressive-masking comparisons?

6 comments
AuthorsOpenReview ↗

Response to Reviewer FHjd (Part1/2)

Dear Reviewer FHjd,

We thank you for the careful review. We understand your two main question to be: (1) whether coverage retention can be attributed to the selection rule, rather than to the underlying W-RF update or simply to less optimization (W2, Q3); and (2) whether the negative-reinforcement-heavy recipe is stable (W1, Q1, Q2). We have considered both carefully, and your practical experience with negative reinforcement is a useful point of contrast with ours. Since attribution bears most directly on the paper's claim, we discuss it first, then stability, and finally the reporting inconsistency (Q4).

To respect the conference's call for brevity and reduce the reviewers' reading burden, we keep our response concise.

1. Attribution of coverage retention (W2, Q3)

For clarity, we break it into three sub-questions:

(i) whether gating adds to W-RF itself;

(ii) whether coverage retention simply reflects less optimization;

(iii) why updates on high-success problems would reduce coverage on other problems.

The 7B comparison below answers (i) and (ii). All rows use the W-RF update rule, and BBG adds only the problem-level gate, and the early-stopping rows stop the same full W-RF run earlier, with no problem selection. Each cell reports Pass@1 / Pass@256.

Qwen2.5-Math-7BMATH-500OlympiadBenchMinervaMathAIME 2025
Base Model64.5 / 96.228.5 / 78.524.4 / 80.97.4 / 53.3
W-RF (20 ep)76.5 / 94.438.5 / 73.738.9 / 77.99.7 / 53.3
W-RF + early stop (10 ep)75.6 / 96.037.3 / 74.937.6 / 78.310.2 / 56.7
W-RF + early stop (5 ep)74.0 / 96.636.3 / 77.436.0 / 79.49.1 / 53.3
BBG (W-RF + gate)75.6 / 96.637.1 / 78.537.1 / 81.19.9 / 63.3

(i) Gating on top of W-RF. Compared with W-RF alone, BBG raises Pass@256 by 2.2, 4.8, 3.2, and 10.0 points on the four benchmarks. As you note, W-RF already mitigates diversity collapse, but it does not remove it, and the gate further mitigates the remaining loss.

(ii) Less optimization. The reviewer is right that BBG performs less optimization than full W-RF. A core argument of our paper is that the standard training recipe overtrains: after validation Pass@1 plateaus, further optimization mostly costs coverage (Section 3.1). Section 4.3 defines BBG as early stopping at the problem level: it stops optimizing only the problems whose remaining coverage benefit is small, while training continues on the rest.

Nonetheless, we agree with the reviewer that this confound should be ablated to isolate the contribution of BBG. We can therefore compare directly with W-RF in which optimization is reduced by early stopping. Relative to full W-RF, both early stopping and BBG recover coverage; at similar Pass@1, BBG's per-problem stopping recovers more than stopping the whole run.

(iii) Effects on other problems. We agree that the marginal-utility analysis answers a within-problem question and does not predict how updating on one problem affects other problems. Section 3.2 states this part as a condition (Eq. (10)): if continued optimization also lowers success probabilities on low-coverage problems, these losses can outweigh the gains on already-covered problems. The bucket-restricted experiment (Figure 5) tests this condition empirically. After 28 updates restricted to B8/8B_{8/8}, validation token entropy falls from 0.112 to 0.037, and Pass@256 falls below the base model on both MinervaMath and OlympiadBench. Restricting updates to B0/8B_{0/8} gives the opposite pattern.

AuthorsOpenReview ↗

Response to Reviewer FHjd (Part2/2)

2. Stability of negative-reinforcement-heavy training (W1, Q1, Q2)

Our experience differs from yours on this point, and we would like to discuss it.

Subjectively, across the close to 100+ times W-RF and negative-only runs we have trained, we did not observe a clear difference in training stability from GRPO or positive-only training. As you suggest, we do observe transient spikes in the pre-clipping gradient norm; however, we observe similar spikes in positive-only runs, and all of these runs completed successfully. Looking at this more objectively: in the W-RF training traces we inspected (7B on MATH, and 1.5B on MATH and POLARIS-53K), we observed no sustained training collapse. In every run, the last recorded MATH validation accuracy was within 0.5 percentage points of the best value recorded in that run.

In our understanding, the failure mode you describe may be more closely tied to how the loss is reduced over response length. With per-trajectory length normalization, each token of a long incorrect response is penalized less than each token of a short one, which rewards being long when wrong and can lead to runaway response length (Reference: Understanding R1-Zero-Like Training: A Critical Perspective); with a sum over tokens, long incorrect responses dominate the gradient directly. We follow the reference implementation of W-RF, which takes a masked mean of the clipped policy loss over valid response tokens within each actor microbatch, so every token of an incorrect response receives the same penalty regardless of the response's length.

Following your suggestion, we went back to the training logs to check for the specific symptoms of this failure mode. Incorrect responses are indeed longer than correct ones on average (about 1,210 versus 562 tokens for W-RF at the first update), so the precondition for the failure mode was present. Their length nonetheless decreased during training, consistent with the token-mean reduction:

Qwen2.5-Math-7B, MATH, w+=0.3w^+ = 0.3 (first update → last validated checkpoint)W-RFGate-only BBG
Mean length of incorrect responses (tokens)~1,210 → 1,056~1,187 → 1,062
Fraction of responses reaching the 3,072-token cap8.0% → 0.3%10.6% → 2.4%

Implementation details (Q1): neither W-RF nor BBG uses the EMA baseline described for REINFORCE in Appendix A.4. With binary verifier scores, γ=1\gamma = 1, and no KL penalty, W-RF assigns each valid response token an advantage of +w++w^+ on a correct rollout and −1-1 on an incorrect rollout. BBG keeps this rule and multiplies the policy loss by the binary problem-level gate α(m)\alpha(m) in Eq. (11).

The lengths by correctness are derived from logged aggregates; we did not log per-trajectory gradient contributions; all reported experiments use a 3,072-token response cap.

3. Reporting (Q4)

The early-stopping and masking rows in Table 1 come from the W-RF run; Section 4 and Appendix A.7 incorrectly stated REINFORCE. We have corrected this in the revised manuscript.


We thank you again for the discussion. It has helped us separate the roles of the selection rule, the update rule, and the optimization budget in the paper, which has improved the manuscript. We hope our responses have addressed your main concerns, and we would be grateful if you could reconsider your assessment of our paper accordingly.

Best regards,

The Authors

Thank you for the detailed response and the additional results. I have read the rebuttal and the replies to the other reviewers.

The implementation details, the corrected W-RF/REINFORCE reporting, and the added W-RF baseline are helpful. My main concerns remain. The random-filtering control at matched update counts was not provided, and the early-stopping comparison is not conclusive. BBG’s update count (448) lies between the 5- and 10-epoch W-RF checkpoints (280 and 560), and its Pass@256 margin over them is 0 to 3.6 points on MATH-500, OlympiadBench and MinervaMath, with the AIME 2025 gap amounting to two or three problems. These are single runs, and the response to Reviewer Gb7c confirms that multi-seed results are deferred to a later version. On stability, the response relies on informal experience and aggregate length statistics. No sensitivity results for the positive-reinforcement weight or the response cap were provided, and per-trajectory gradient contributions were not logged.

I am therefore keeping my score.

Reviewer_Yr2QOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary
  1. This paper analyzes the divergence between improving Pass@1 and declining high-k Pass@k in RLVR through the lens of overtraining, showing that a problem’s coverage benefit can nearly saturate while its single-attempt success probability remains low.
  2. Through training trajectories, Bayesian analysis, and bucket-based interventions, the authors show that early stopping or excluding high-success problems from updates can reduce coverage loss while retaining some accuracy gains.
Critical strengths
  1. The paper clearly demonstrates the relationship between continued optimization, declining Pass@256, and entropy contraction through training trajectories and bucket-based interventions, and shows that early stopping and masking high-success problems can reduce coverage loss.
  2. The paper introduces Bayesian boundary utility to quantify the posterior expected marginal gain in Pass@k under uncertainty from limited rollouts, clearly explaining why coverage gains saturate before single-attempt accuracy.
Critical weaknesses
  1. The improvement in Pass@256 supports the effectiveness of restricting updates on low-marginal-utility problems, but the paper does not directly establish whether this improvement comes from enhanced learning on high-marginal-utility problems.
  2. Although the method preserves coverage while retaining Pass@1 gains over the base model, Pass@1 drops substantially in some settings compared with training baselines without early stoping.
Questions for authors

Q1: Could you provide validation token entropy curves throughout training for bucket-restricted RLVR and the original method, to assess whether excluding high-success problems consistently mitigates entropy collapse and how this relates to changes in coverage?

Q2: Can bucket-restricted RLVR achieve Pass@1 comparable to the original method? If so, does it require substantially much more rollouts or wall-clock training time?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Yr2Q

Dear Reviewer Yr2Q,

We thank you for the careful review. To respect the conference's call for brevity and reduce the reviewers' reading burden, we keep our response concise.

1. Where the coverage improvement comes from (W1)

You point out a useful distinction. We attribute the improvement mainly to avoided coverage loss. Under standard training, the decline in Pass@k is a net effect: some problems with no base-model success become covered, but more previously covered problems lose their coverage. The bucket-restricted experiment points to the same conclusion: updates restricted to B8/8B_{8/8} lower Pass@k on held-out problems below the base model (Figure 5), so removing such updates can preserve coverage even if learning on the retained problems is unchanged.

2. Pass@1 relative to training without early stopping (W2, Q2)

We agree that BBG gives up some Pass@1 (with milder masking the cost can be much smaller; e.g., masking only B8/8B_{8/8}). Whether training is stopped for the whole run or for individual problems, part of this Pass@1 gain is given up to avoid a large drop in coverage. Notably, at similar Pass@1, BBG (which can be viewed as early stopping at the problem level) retains more coverage than stopping the whole run.

On Q2: BBG does not buy its coverage with additional budget. In the POLARIS-53K comparison, it uses the same number of rollout steps as the baselines (Table 2), and since problems excluded by the gate are not backpropagated, it performs fewer updates.

3. Entropy throughout training (Q1)

We have added validation token entropy curves for the bucket-restricted runs to the revised paper:

Trainer stepB0/8B_{0/8} onlyB1/8B_{1/8} onlyB4/8B_{4/8} onlyB8/8B_{8/8} onlyFull training
00.1120.1120.1120.1120.112
140.1110.1080.0940.0730.094
280.1230.1070.0760.0370.073

We thank you again for your constructive comments, which have helped us clarify the source and the cost of coverage retention in the paper.

Best regards,

The Authors

Reviewer_Gb7cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why RLVR can improve Pass@1 while reducing high-kk Pass@kk value. It examines continued training after validation accuracy has largely plateaued, and continued updates on problems that offer little further gain. A Beta-Binomial model estimates the remaining marginal benefit from a small rollout group. The experiments track newly covered and lost problems, then test early stopping and success-based masking. The paper also introduces Bayesian Boundary Gating (BBG), which applies a utility-based gate to W-RF. Result on Qwen2.5-Math models suggest that these interventions can recover coverage while retaining accuracy gains, although BBG often achieves lower Pass@11 than the comparison methods.

Critical strengths
  1. The analysis in Section 4.1 and Appendix C is very useful. Tracking both Pass@256256 curve and Entry and loss curve makes that behavior easier to understand. Appendix C also distinguishes weak one-success transitions from stronger multi-success transitions, and explains the selection and multiple-comparison limits of its one-sided scores. The reported 3838 entries and 152152 losses provide substantial evidence of coverage loss in the examined evaluation.

  2. The Bayesian calculation gives a clear way to discuss how much coverage a problem can still gain when its success rate is estimated from only a few rollouts. The posterior utility u(m;n,k)u(m;n,k) and the ratio u(1;n,k)/u(0;n,k)u(1;n,k)/u(0;n,k) in (8) and (9) are correct under the uniform prior. Appendix B also explains why the zero-success bucket is a noisy proxy for low coverage. These calculations connect the coverage argument to a concrete training decision.

Critical weaknesses
  1. The intervention results is inconsistent with the stated baselines. Section 4.1 describes MinervaMath Pass@256256 increasing from 77.9 under full REINFORCE to 79.4 with early stopping. In Table 1, 77.9 belongs to W-RF; REINFORCE is 72.4. The associated Pass@1 value, 38.9, also comes from W-RF rather than REINFORCE's 37.3. Section 4.2 uses W-RF's values again when discussing masking, including the OlympiadBench baseline of 73.7. Yet Section 4 and Appendix A.7 identify REINFORCE as the method used for these interventions. Due to this, the source of benefits becomes confusing. Similarly, the introduction gives a MinervaMath+OlympiadBench average Pass@256256 change of 78.4 to 75.8, while Section 3.1 and Figure 2 give 75.2 to 70.6. Averaging the corresponding epoch 5 and REINFORCE rows in Table 1 gives approximately 78.4 to 70.6. The epoch 5 Pass@1 average in that table is about 36.2, rather than the 37.3 quoted in the text.

  2. Section 4.3 says that the controlled 7B BBG experiment preserves or improves coverage on all four benchmarks. I could not find those numbers: Table 1 has no BBG row, and Table 2 reports 1.5B models. Please include the comparison supporting this statement.

  3. BBG adds a gate to W-RF, using an advantage weight of 0.3 for correct rollouts, but Table 2 omits W-RF alone. W-RF already mitigates diversity collapse, as the paper notes. Without such comparison, it is difficult to tell how much of BBG's coverage retention comes from selecting which problems receive updates.

  4. I would also like to know whether the reported gains hold across different runs. Table 2 reports higher Pass@256256 for BBG than for the base model in all eight settings, but some improvements are small. On AIME 2025, the increases from 50.0 to 56.7 and 60.0 correspond to only two and three additional covered questions. Appendix A.8 notes that the reported comparisons do not quantify variation across independent training runs.

Questions for authors
  1. Which runs produced the early-stopping and masking rows in Table 1? Please check the REINFORCE descriptions with the W-RF baseline values and give a consistent comparison for epoch 5 to epoch 20.

  2. Can you provide the controlled 7B results for BBG mentioned in Section 4.3, including both Pass@11 and Pass@256256?

  3. Have you repeated the main experiments with different random seeds? Are the improvements in Pass@11 and Pass@256256 consistent across different runs, especially for the smaller gains in Table 2?

  4. How does BBG compare with W-RF alone under the same experimental settings? Please clarify the effective batch sizes, rollout budgets, and numbers of optimizer updates.

  5. What happens to Pass@11 in the experiments in Figure 5? Since these experiments only run up to step 28, would the observations still hold with longer training?

8 comments
AuthorsOpenReview ↗

Response to Reviewer Gb7c (Part1/2)

Dear Reviewer Gb7c,

We thank you for the careful review, in particular for checking Section 4.1, Appendix C, and the Bayesian analysis item by item. We respond to each of your points below.

To respect the conference's call for brevity and reduce the reviewers' reading burden, we keep our response concise.

1. Consistency between the text and Table 1 (W1, Q1)

Thank you for the careful reading. We did mix up W-RF and REINFORCE in the writing, and we have corrected three places in the revised manuscript:

(1) Sections 4.1 and 4.2 now state that the baseline in these comparisons is W-RF.

(2) The Pass@256 values in the Introduction now use the REINFORCE numbers, 75.2 → 70.6, consistent with Section 3.1.

(3) The opening of Section 4 now states that coverage tracking and bucket-restricted training use REINFORCE, while early stopping and progressive masking use W-RF (Appendix A.7 is corrected accordingly). This error also caused the epoch-5 mismatch you pointed out: the early stop (5 ep) row in Table 1 comes from the W-RF run, whereas the 37.3 in the text is the epoch-5 value of the REINFORCE run.

2. BBG results at 7B (W2, Q2)

Thank you for pointing out this omission. We have added the 7B BBG results to Table 1. In brief (each cell reports Pass@1 / Pass@256 / Avg.):

Qwen2.5-Math-7BMATH-500OlympiadBenchMinervaMathAIME 2025
Base Model64.5 / 96.2 / 86.828.5 / 78.5 / 58.024.4 / 80.9 / 57.67.4 / 53.3 / 27.5
W-RF (Zhu et al., 2025)76.5 / 94.4 / 87.938.5 / 73.7 / 59.438.9 / 77.9 / 61.29.7 / 53.3 / 31.3
BBG (W-RF + gate)75.6 / 96.6 / 89.537.1 / 78.5 / 61.337.1 / 81.1 / 62.19.9 / 63.3 / 34.3

3. Comparison with W-RF alone and training budget (W3, Q4)

In Table 1, BBG differs from W-RF only by the problem-level gate. Relative to W-RF alone, BBG raises Pass@256 by 2.2, 4.8, 3.2, and 10.0 points on the four benchmarks, which isolates the contribution of the gate. We have also added W-RF to Table 2, where the conclusion is similar.

On the budget of BBG: both methods sample 8 rollouts per problem. In W-RF, each rollout batch contains 1,024 prompts, all of which are used for the update; in BBG, only problems that pass the gate are backpropagated.

7B, MATHPrompts per trainer stepMini-batchTrainer stepsGradient updates
W-RF (20 ep)1,0241281401,120
W-RF + early stop (10 ep)1,02412870560
W-RF + early stop (5 ep)1,02412835280
BBG512 (active)6456448

The early-stopping rows can help us separate the effect of optimizing less from the effect of selecting problems, with no problem selection: at similar Pass@1 (e.g., the 10-epoch checkpoint), BBG has higher Pass@256.

AuthorsOpenReview ↗

Response to Reviewer Gb7c (Part2/2)

4. Consistency across random seeds (W4, Q3)

We agree with your assessment. Due to limited computational resources, we will add a comparison across multiple seeds in the next version.

5. Pass@1 in Figure 5 and longer training (Q5)

The Pass@1 of each bucket-restricted run in Figure 5 at step 28 is as follows:

Updates restricted toMinervaMath Pass@1OlympiadBench Pass@1
B0/8B_{0/8}32.4535.54
B1/8B_{1/8}36.0536.16
B4/8B_{4/8}37.2036.93
B8/8B_{8/8}37.0336.52

B0/8B_{0/8} has the lowest Pass@1 on both benchmarks, and B4/8B_{4/8} the highest.

These runs are short diagnostic interventions that compare the direction of different update scopes. Their counterparts at full training length are the masking and BBG experiments in Table 1: BBG updates only B0/8B_{0/8} through B3/8B_{3/8} throughout training and retains coverage. The long-horizon case of updating only high-success buckets (e.g., B8/8B_{8/8}) is closest to the later stage of standard training: the combined share of B7/8B_{7/8} and B8/8B_{8/8} rises from 16.0% to 64.3% (Section 3.1) while coverage continues to decline.


We thank you again for your constructive comments, especially the careful check of numerical consistency; these changes have made the paper more rigorous. We hope our responses have addressed your concerns, and we would be grateful if you could reconsider your assessment of our paper accordingly.

Best regards,

The Authors

Thank you for the clarifications and additional results. These address my concerns about the inconsistent experiment results. My remaining concern is on the contribution of the selection rule in BBG. From your new experiment, I believe BBG has better performance than those baselines, especially better than simply using early stopping on W-RF. However, BBG uses fewer updates and smaller mini-batches than W-RF. Maybe part of the coverage improvement comes from these differences rather than which problems are selected. Can you clarify this?

Comments on the paper

1 comment

Dear Reviewers,

We hope you had a great weekend!

As the discussion period progresses, we wanted to gently follow up on our responses and the revised manuscript posted last week. We would love to hear from Reviewers regarding whether our newly added analyses and clarifications have addressed your questions.

We remain fully available to answer any remaining questions and would greatly appreciate your feedback.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.