- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks which reasoning strategies survive RLVR: those that are initially common, or those that are reliable. It groups sampled responses into strategy families (Countdown: first operand+operator via Zhou & Li's parser; MATH-500: normalized first calculation via a new parser), measures each family's frequency q and success rate mu, and fits multinomial selection models q_t ∝ q0^alpha exp(mu0/beta) across six checkpoints of a public GRPO Countdown run (1000 problems x 64 samples) and three SimpleRL-Zoo base/RL pairs on MATH-500 (100 problems x 256 samples). It finds alpha > 1 everywhere (cumulative alpha ≈ 3 late in Countdown; 1.10-1.27 on MATH-500), that current frequency strongly predicts a family becoming unobserved at the next checkpoint while quality does not, that problems with more even initial strategy distributions retain more diversity (AUC 0.70) without an accuracy penalty, and that a simplified finite-sampling (G=8, permanent removal) simulation explains part but not all of the late concentration. No models are trained and no method is proposed.
Critical strengths
-
The question is well-posed and the analysis is statistically careful. Comparing frequency against quality among families competing on the same problem (within-problem standardization, fixed step-25 family set, synthetic check that the cumulative-alpha estimator does not drift, problem-level bootstrap, split-half reliability of mu, duplicate-text control) is considerably more rigorous than most diversity-collapse analyses, and the paper is unusually honest about what "disappearance at n=64" does and does not mean.
-
The resistant-problem result (Section 4.3, Table 12) is the most novel piece: initial entropy of the strategy distribution predicts retention of multiple strategies, the raw number of families does not once evenness is controlled, and retaining strategies carries no detectable final-accuracy cost. This is an actionable observation for curriculum or initialization design.
Critical weaknesses
-
The headline "frequency beats quality" result is not separated from two simpler explanations. (a) Uniform sharpening: if RLVR merely lowers the entropy of the policy (a well-documented effect), then q_t ∝ q0^alpha with alpha > 1 holds for every problem with no strategy-level selection at all, so alpha > 1 largely restates "diversity collapses" rather than demonstrating rich-get-richer dynamics. (b) Detection floor: under any such shrinkage the rarest families are the ones nearest the 64-sample detection threshold, so a negative frequency coefficient in the disappearance regression is close to guaranteed. The paper needs a null that applies the same multiplicative sharpening to all families (or the fitted alpha without any quality term) and shows that the observed extinction-vs-frequency curve and the quality null effect are not reproduced by that null alone. Without this, the mechanism claim and the title are stronger than the evidence.
-
The quality signal is too weak and too noisily measured for the negative result to be informative, especially on Countdown. Within-problem quality SD is 0.058 on Countdown at a base accuracy of ~9%, mu is estimated from as few as 5-6 samples (the Table 18 "reliable" strategies are 5/6 and 3/5), and errors-in-variables attenuation is not corrected. Showing that a noisy, low-variance predictor has a smaller standardized coefficient than frequency does not establish that RLVR fails to select on quality; the within-problem Shapley/first-order analyses in the cited Zhou & Li work in fact attribute most of the solve-rate gain to quality-driven changes. Combined with a single public Countdown trajectory, small MATH-500 effect sizes (12-19% Neff loss, extinction too rare to regress), and a MATH-500 parser validated on 20 hand-inspected examples per run, the generalizable contribution beyond Zhao et al. (2025), Fan et al. (2026) and Zhou & Li (2026) is modest.
Questions for authors
Clarification requests:
- Please add a sharpening-only null: for each problem, apply q_t ∝ q0^alpha_t with the fitted alpha_t and no quality term, simulate 64-sample draws, and report the resulting extinction-by-frequency-decile curve and the frequency/quality logistic coefficients. How much of Figure 3 and Table 1 does this null reproduce?
- Is there any problem-level evidence that selection is strategy-specific rather than global, e.g., do families gain share relative to the uniform-sharpening prediction in proportion to (mu_b - mu_bar), as a policy-gradient account would predict?
- For the quality coefficients, please report estimates after correcting for measurement error in mu (e.g., using the split-half reliabilities in Table 7), and restrict to families with ≥20 observations on Countdown as was done for MATH-500.
- How were the 20 inspected MATH-500 parses evaluated, and what fraction merged distinct approaches or split one approach? Could the 24% unparsed responses be systematically different (e.g., longer, more exploratory) between base and RL?
Suggestion for future work: A controlled experiment that varies the starting frequency of a fixed-quality strategy (e.g., via SFT warm-start) and then runs RLVR would directly test the causal claim in the title.