acceptodds
Withdrawn from ICLR 2027

Diversity Collapse in RLVR: Even Good Strategies Die if They Start Rare

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves reasoning accuracy, but it can also narrow the set of strategies a model uses. We ask which strategies survive this process and why. Across a full RLVR trajectory on Countdown and three base/RL model pairs on MATH-500, we find a consistent rich-get-richer effect: strategies that are already common become more dominant after training, while rare strategies are much more likely to disappear. Strategy quality provides weaker and less consistent protection; even reliable strategies can be lost when they begin uncommon. On Countdown, the effective number of strategies per problem falls from about to nearly , while problems whose strategies start at similar frequencies retain substantially more diversity. We further find that finite training samples can amplify this imbalance; once a strategy becomes sufficiently rare, it may stop appearing in sampled rollouts and receive no direct reinforcement. These results show that RLVR can favor what is already common over what is merely effective, causing early frequency differences to compound into the loss of useful reasoning strategies.

Primary area:
reinforcement learning
Keywords:
LLM reasoning, RLVR, evaluation metrics, diversity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Fp8TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks which reasoning strategies survive RLVR: those that are initially common, or those that are reliable. It groups sampled responses into strategy families (Countdown: first operand+operator via Zhou & Li's parser; MATH-500: normalized first calculation via a new parser), measures each family's frequency q and success rate mu, and fits multinomial selection models q_t ∝ q0^alpha exp(mu0/beta) across six checkpoints of a public GRPO Countdown run (1000 problems x 64 samples) and three SimpleRL-Zoo base/RL pairs on MATH-500 (100 problems x 256 samples). It finds alpha > 1 everywhere (cumulative alpha ≈ 3 late in Countdown; 1.10-1.27 on MATH-500), that current frequency strongly predicts a family becoming unobserved at the next checkpoint while quality does not, that problems with more even initial strategy distributions retain more diversity (AUC 0.70) without an accuracy penalty, and that a simplified finite-sampling (G=8, permanent removal) simulation explains part but not all of the late concentration. No models are trained and no method is proposed.

Critical strengths
  1. The question is well-posed and the analysis is statistically careful. Comparing frequency against quality among families competing on the same problem (within-problem standardization, fixed step-25 family set, synthetic check that the cumulative-alpha estimator does not drift, problem-level bootstrap, split-half reliability of mu, duplicate-text control) is considerably more rigorous than most diversity-collapse analyses, and the paper is unusually honest about what "disappearance at n=64" does and does not mean.

  2. The resistant-problem result (Section 4.3, Table 12) is the most novel piece: initial entropy of the strategy distribution predicts retention of multiple strategies, the raw number of families does not once evenness is controlled, and retaining strategies carries no detectable final-accuracy cost. This is an actionable observation for curriculum or initialization design.

Critical weaknesses
  1. The headline "frequency beats quality" result is not separated from two simpler explanations. (a) Uniform sharpening: if RLVR merely lowers the entropy of the policy (a well-documented effect), then q_t ∝ q0^alpha with alpha > 1 holds for every problem with no strategy-level selection at all, so alpha > 1 largely restates "diversity collapses" rather than demonstrating rich-get-richer dynamics. (b) Detection floor: under any such shrinkage the rarest families are the ones nearest the 64-sample detection threshold, so a negative frequency coefficient in the disappearance regression is close to guaranteed. The paper needs a null that applies the same multiplicative sharpening to all families (or the fitted alpha without any quality term) and shows that the observed extinction-vs-frequency curve and the quality null effect are not reproduced by that null alone. Without this, the mechanism claim and the title are stronger than the evidence.

  2. The quality signal is too weak and too noisily measured for the negative result to be informative, especially on Countdown. Within-problem quality SD is 0.058 on Countdown at a base accuracy of ~9%, mu is estimated from as few as 5-6 samples (the Table 18 "reliable" strategies are 5/6 and 3/5), and errors-in-variables attenuation is not corrected. Showing that a noisy, low-variance predictor has a smaller standardized coefficient than frequency does not establish that RLVR fails to select on quality; the within-problem Shapley/first-order analyses in the cited Zhou & Li work in fact attribute most of the solve-rate gain to quality-driven changes. Combined with a single public Countdown trajectory, small MATH-500 effect sizes (12-19% Neff loss, extinction too rare to regress), and a MATH-500 parser validated on 20 hand-inspected examples per run, the generalizable contribution beyond Zhao et al. (2025), Fan et al. (2026) and Zhou & Li (2026) is modest.

Questions for authors

Clarification requests:

  1. Please add a sharpening-only null: for each problem, apply q_t ∝ q0^alpha_t with the fitted alpha_t and no quality term, simulate 64-sample draws, and report the resulting extinction-by-frequency-decile curve and the frequency/quality logistic coefficients. How much of Figure 3 and Table 1 does this null reproduce?
  2. Is there any problem-level evidence that selection is strategy-specific rather than global, e.g., do families gain share relative to the uniform-sharpening prediction in proportion to (mu_b - mu_bar), as a policy-gradient account would predict?
  3. For the quality coefficients, please report estimates after correcting for measurement error in mu (e.g., using the split-half reliabilities in Table 7), and restrict to families with ≥20 observations on Countdown as was done for MATH-500.
  4. How were the 20 inspected MATH-500 parses evaluated, and what fraction merged distinct approaches or split one approach? Could the 24% unparsed responses be systematically different (e.g., longer, more exploratory) between base and RL?

Suggestion for future work: A controlled experiment that varies the starting frequency of a fixed-quality strategy (e.g., via SFT warm-start) and then runs RLVR would directly test the causal claim in the title.

Reviewer_ZAABOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates reasoning-strategy diversity collapse during RLVR, focusing on whether strategy survival is more strongly associated with initial frequency or strategy quality. The authors analyze a Countdown training trajectory and three base/RL model pairs on MATH-500. Their results show that initially common strategies tend to become more dominant, while rare strategies are more likely to disappear from sampled outputs, even when they have high success rates. The paper also finds that balanced initial strategy distributions help preserve diversity and that limited rollout sampling may further amplify strategy concentration.

Critical strengths
  1. The paper addresses an important issue in RLVR: why some reasoning strategies survive training while others disappear from sampled outputs. Distinguishing the effects of initial strategy frequency and strategy quality provides a useful perspective on reasoning diversity collapse.
  2. The authors analyze intermediate checkpoints, compare multiple predictive models, and investigate strategy disappearance and diversity preservation. The additional experiments on MATH-500 and the finite-sampling simulations provide useful supporting evidence. The paper also carefully distinguishes empirical non-observation from the complete disappearance of a strategy from the model's distribution.
Critical weaknesses
  1. The finding that initially frequent strategies become more dominant is closely related to existing observations about diversity collapse and the rich-get-richer effect in RLVR. Although the paper provides additional empirical evidence, its novelty beyond prior studies is somewhat limited. Moreover, the analysis is mainly correlational and does not establish that initial frequency causally determines strategy survival independently of other training dynamics.
  2. The paper defines strategy disappearance as observing zero occurrences in a limited number of sampled responses. However, rare strategies are inherently more likely to be missed under finite sampling, even without any change in their underlying probabilities. This makes the strong relationship between frequency and observed disappearance partly expected by construction. A more rigorous comparison against a stationary-probability baseline would help establish how much disappearance is actually caused by RLVR.
  3. The strongest evidence comes from a single Countdown training trajectory, while MATH-500 is evaluated only through base/RL endpoint comparisons. Moreover, strategy families are defined using initial calculations rather than complete reasoning approaches, and the MATH-500 parser covers only approximately 76% of responses without extensive human validation. These limitations raise concerns about the robustness and generalizability of the conclusions.
Questions for authors
  1. Since rare strategies are naturally more likely to be unobserved under finite sampling, could the authors compare the observed disappearance rates with a stationary-probability baseline that preserves each strategy's frequency? How much additional disappearance can be attributed to RLVR beyond this sampling effect?
  2. The paper shows a strong correlation between initial frequency and later strategy survival. Can the authors provide additional evidence that this relationship reflects a causal effect of initial frequency rather than other correlated properties of the strategies? For example, would controlled changes to initial strategy frequencies lead to predictable differences in survival?
  3. How sensitive are the findings to the definition of strategy families, particularly for MATH-500, where families are identified using the first calculation? Would the main conclusions remain consistent under alternative strategy grouping methods, larger sampling budgets, or additional RLVR training trajectories?
Reviewer_hCYyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper tests whether a solution strategy's (a) initial frequency or (b) quality better predicts its eventual frequency under repeated sampling after RL training. By fitting the parameters of a few simple models, the authors find that the parameters corresponding to (a) are consistently larger in magnitude, and conclude that initial frequency has a larger effect. They also show that strategies which are above median quality at a given initial frequency level are no more likely to survive than strategies with below median frequency at the same level.

As the authors cite in related work, the fact that diversity can collapse in LM RL training is well-known, as well as the fact that initial frequency of a response can determine how much it is sampled -- rare samples have fewer opportunities for reinforcement. The approach of partitioning solutions into strategy families is also cited as prior work. The primary contribution thus appear to be trying to empirically disentangle the factors of quality and diversity in an actual training run.

Critical strengths

The authors provide extensive details for reproduction, as well as anonymized code.

Critical weaknesses

I tend towards rejection for a few key reasons. Overall, the findings are rather unsurprising to me, and I feel limited in applicability as the results are purely observational based on specific model training runs.

First, it's unclear to me that initial strategy reliability is a meaningful notion of quality in the domains studied, where the reward is strictly binary (MATH500, Countdown). The paper's notion of quality is saying that strategy A > B iff p(correct | strategy A) > p(correct | strategy B), where the probabilities of success is computed at a fixed early checkpoint during training. The work claims that if A > B but p(B) > p(A), A may be suppressed over time. But p(correct | strategy B) is not static - the model's ability to execute strategy B can improve over training. Moreover, B can still get to the right answer -- is it really meaningful or interesting to say that B has lower quality then A? It would be more surprising to me if, say, the initial model has some strategy A that gets strictly better reward than B, but A is suppressed over training in favor of a more frequent but less-rewarding B, such that the final model converges to a reward lower than A. The results do not yet support this stronger claim.

Second, the idea that a rare, good strategy can lose to a common, adequate one is well documented both empirically and theoretically, in the works that the authors themselves cite.

Third, all of these results are based off of observing a few training runs, but in practice small details like hyperparameters (e.g. training length, rollout batch size, see ProRL from Liu et al.) can an have a huge impact on the behavior of RL algorithms. It's hard for me to know what takeaways I can draw generally.

Finally, much of the text has AI-esque language, and can be unclear at times.

Questions for authors
  1. Can you show how the quality of the strategies change over training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.