acceptodds
Under review as a conference paper at ICLR 2027

Measuring and Mitigating Mode Reasoning Collapse in RLVR

Abstract

A language model (LM) can usually answer the same question in more than one way, but reinforcement learning with verifiable rewards (RLVR) is indifferent to which correct answer a model produces. A solution will earn the same reward whether it is the thousandth copy of a familiar answer or one the model has never produced before. Yet, there is potential value in having the model retain multiple correct solutions as it is trained. For instance, multiple modes may give users a choice and provide problem-solving strategies that improve overall model performance. Here, we introduce ModeBench, a benchmark of multi-solution tasks in which the verifier returns both correctness and mode discovered. We then use ModeBench to measure how solution diversity changes under RLVR post-training. We find that RLVR post-training concentrates probability onto fewer correct modes even as accuracy holds or improves, and moreover, that frontier models are already highly concentrated. We then introduce our solution, Re:Max, which stores one verified example per discovered mode in a replay buffer and trains on those stored modes uniformly. A solution found once is, therefore, practiced as often as one found repeatedly. Across three model scales, two RL objectives, and harder task constructions, replay improves both how often a policy succeeds and how many different ways it can succeed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning with verifiable rewards, mode collapse, solution diversity, language model reasoning, experience replay

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_SaPGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a strategy to mitigate the mode collapse problem during RL post-training of language models. They highlight that this problem is prevalent even for frontier models, where on simple tasks frontier models are also not able to sample diverse enough modes.

Critical strengths
  • The work studies an important problem of lack of diversity in LLMs and collapse of diversity during post-training.
  • On the tasks studied in the paper the proposed method provides strong improvement over the baselines.
Critical weaknesses

I would be willing to reconsider my assessment if the authors could provide clarifications about the below points:

  1. It is unclear to me how much of a pass@1 vs pass@N tradeoff is happening out here. Is it the case that the proposed approach achieves a higher pass@8 but lower pass@1 than something like GRPO or MaxRL? It's fine for a tradeoff to exist. In which case I think reporting a graph which has pass@N (or PCMD) on the Y axis and pass@1 on the X axis and different methods as points on this graph might provide better understanding about whether the proposed approach has a better tradeoff than the baselines?

  2. If the goal is to optimize for sampling multiple modes, how would approaches like multi-turn sequential sampling work? For e.g., if the model samples a solution, it is just told to "sample a solution different from ones sampled in previous turns". I expect this to improve the diversity of the outputs from different turns.

  3. It would be helpful to discuss / compare against approaches which have aimed to address the mode collapse problem without using mode labels such as [1]. It seems to me that the problem highlighted in the paper could be fixed by changing the sampling strategy (above point), or by making changes to the objective that do not need extra labels for the mode. The use of these extra labels could restrict the broader applicability of the method


[1] GX-Chen, A., Prakash, J., Guo, J., Fergus, R., and Ranganath, R. "KL-Regularized Reinforcement Learning for Generative Modelling is Designed to Mode Collapse." ICLR 2026. arXiv:2510.20817.

Questions for authors

Refer above weaknesses

6 comments

We thank you for your thoughtful review and for recognizing that our work "studies an important problem" and that Re:Max "provides strong improvement over the baselines." We address your questions below.

"It is unclear to me how much of a pass@1 vs pass@N tradeoff is happening out here... reporting a graph which has pass@N (or PCMD) on the Y axis and pass@1 on the X axis... might provide better understanding."

We thank the reviewer for this suggestion and have added this figure to our Appendix, plotting per-response correctness (pass@1) against pass@8 and PCMD for every method and model scale.

Interestingly, the tradeoff the reviewer anticipates appears in the baselines rather than in replay. Without replay, we find that per-response correctness increases and pass@8 decreases over training in Graph and PantryPlan, falling below the untrained model (as we show in Table 1). As RLVR moves the policy towards a single answer, pass@1 increases at the expense of pass@N. Figure 1 shows this on a single problem, where Re:Max produces fewer invalid answers than Dr.GRPO and retains more solution modes.

"If the goal is to optimize for sampling multiple modes, how would approaches like multi-turn sequential sampling work?"

We thank the reviewer for this suggestion, which we view as complementary to Re:Max. At inference time, we tested several related interventions. Raising temperature, changing sampling, and increasing the sample budget. Under each of these interventions, we do not recover the diversity lost under RLVR (App. I), and moreover, explicit diversity prompting helps inconsistently across domains (App. L.3). Even frontier models remain highly concentrated, with GPT-5.6 Sol producing a single solution mode on harder MathIR problems. We have added multi-turn sequential sampling as a future work idea to the main body of the paper. However, we anticipate that mode collapse would still emerge under this setting, but would perhaps take longer to fully converge towards a singular solution.

"It would be helpful to discuss / compare against approaches which have aimed to address the mode collapse problem without using mode labels such as [1]... The use of these extra labels could restrict the broader applicability of the method."

We thank the reviewer for this point and agree that applicability matters. Notably, a substantial part of replay's benefit does not require mode labels. Our single-slot variant (App. P.5) stores only the first verified success per prompt, needing nothing beyond a binary verifier, and increases PCMD from .004 to .218 and pass@8 from .444 to .707, recovering 58% of the diversity gain of full Re:Dr. Mode labels add a further improvement by balancing rehearsal across the solutions found. We have moved this result into the main text as per your suggestion.

We thank the reviewer for pointing us to GX-Chen et al. (2026), which we have since added to our citations in the current version. Their work characterizes how the KL-regularized objective itself can favor collapsed solutions, while our analysis focuses on the setting of binary rewards, where a rarely sampled correct solution receives a vanishing on-policy gradient. Because replay acts on stored solutions rather than the objective, the two approaches should be compatible. A sweep over KL regularization to the initial policy (App. E.7) preserves more diversity in PantryPlan, where the base model is already highly diverse, but achieves a worse correctness–diversity tradeoff than replay in Graph and MathIR. We also compare against several label-free approaches in our Appendix. UCPO and RLEP improve PCMD by only .009 over Dr.GRPO, compared with .320 for Re:Max.

Thank you for your questions and helping us improve the clarity of our paper. We’ve aimed to answer every question thoroughly. If any details remain unclear, please do not hesitate to let us know.

Reviewer_SaPGOpenReview ↗

Reviewer Response

Thanks for the clarifications and references to the relevant sections of the appendix! I would like to follow up on the authors' responses

1. Should one address the diversity collapse problem through training, or do scaffold-like methods (e.g., multi-turn sampling) subsume training methods?

I would like more intuition / evidence that training-based solutions are orthogonal to solutions that build scaffolds. My current intuition is that many of these diversity issues can be fixed with simple tricks like the multi-turn sampling I suggested in my review.

I ran an experiment on the graph coloring problem comparing (i) sampling 8 solutions per problem, and (ii) sampling 4 solutions, then generating the 5th by conditioning on the 1st and instructing the model to give a different solution, the 6th similarly from the 2nd, and so on. The experiment used the 96 problems generated by running

python ops/make_exact_answer_mode_data.py \
    --output-root data/released_exact_answer \
    --eval-splits multi_answer

Results on Gemini-4-Carbon and Qwen2.5-7B-Instruct are below. Conditioning improves PCMD for both models:

Gemini-4-Carbon pass@8Gemini-4-Carbon PCMDQwen2.5-7B-Instruct pass@8Qwen2.5-7B-Instruct PCMD
Parallel1.0000.0710.7190.336
Conditioned (4 + 4)1.0000.6420.6880.438

Given these results I would like to hear the authors' thoughts on should one focus more on training based methodologies to address the problem or inference.

2. If training is a desired route to address such issues, I feel the need for labels restricts the generality of the method

I noticed that in response to other reviewers' the authors have added more baselines and Replay seems to works well, but is there any reason why methods like DMPO which have been designed to improve diversity have worse PCMD than MaxRL?

Also, I couldn't find the pass@1 vs pass@N / PCMD plot in the appendix, could you please direct me to which is the plot? Thanks!

(I used Claude Opus 5.5 to write code for above experiments.)

1. Should one address the diversity collapse problem through training, or do scaffold-like methods (e.g., multi-turn sampling) subsume training methods?

Thank you for following up on this comment. The reason that we do not recommend multi-turn / inference time approaches, is because inference cannot "make up" for the things that training has already removed. As we say in our intro: "what makes models useful at deployment depends on having different candidates to choose from, i.e. test-time search and reranking (Brown et al., 2024; Snell et al., 2024; Cobbe et al., 2021; Wang et al., 2023). As RLVR improves accuracy, what happens to the diversity of correct solutions?"

Your suggestions about inference time calls for diversity / multi-turn sampling fall into this bucket. A model will not suggest something that has already been "collapsed" or "forgotten". If we post-train on a set of problems with RLVR (in a way that is commonly done today), without any interventions, then: the set of outputs a model will produce will collapse (and eventually leave you in a situation where you cannot actually take advantage of inference-time techniques) -- the very thing we would like to promote.

Thank you for running the experiment! We appreciate the time you have taken to throughly engage with our research. Your findings are consistent with our own for parallel, and we are not at all surprised by your findings using an inference-time technique.

"Given these results... should one focus more on training based methodologies to address the problem or inference."

It depends on what you want to do! We think that the key distinction here is RLVR post-training:

  1. If your goal is to take an existing model and find the full extent of all diverse solutions the given model can currently provide, then inference methodologies is the right answer.
  2. If your goal is to have a model become "better" at a task through RLVR post-training (e.g. come up with things that the pre-trained model didn't consider), then you will need to make sure you maintain diversity while working on becoming "more accurate".

Our paper is about the later, and so we propose a solution to address the issue in the later category. The former differs from our efforts. To test / further your experiment, we encourage you to:

  1. Post-train with RLVR the Qwen2.5-7B-Instruct with GRPO.
  2. Compare your post-trained Qwen2.5-7B-Instruct against your pre-trained Qwen2.5-7B-Instruct with inference time methods.

Our research indicates that your post-trained Qwen2.5-7B-Instruct will have poorer PCMD on both parallel and conditioned.

2. If training is a desired route to address such issues, I feel the need for labels restricts the generality of the method

We want to separate two things:

  1. ModeBench, the benchmark: if the goal is to show how much of the solution space your algorithm can find, then you need to have a way to show-off the diversity of your model vs the complete diversity space. Thus, you need labels.
  2. Re:Max, the algorithm: our algorithm prefers labels, where they exist, but (1) does not require them (as we discuss with another reviewer) and (2) another model could be deployed to determine whether a given solution is unique, thus providing the set of labels.

Also, I couldn't find the pass@1 vs pass@N / PCMD plot in the appendix, could you please direct me to which is the plot? Thanks!

Apologies for the inconvenience. The figure is not available in the appendix you can see on this page (and we cannot update the appendix during the review period). We have added it to our updated appendix (held locally). In words, it show that: "without replay, we find that per-response correctness increases and pass@8 decreases over training in Graph and PantryPlan, falling below the untrained model (as we show in Table 1). As RLVR moves the policy towards a single answer, pass@1 increases at the expense of pass@N."

"I noticed that in response to other reviewers' the authors have added more baselines and Replay seems to works well, but is there any reason why methods like DMPO which have been designed to improve diversity have worse PCMD than MaxRL?" (We assume you mean Re:Max (our algorithm) here, rather than MaxRL. If not let us know.)

The issue is that DMPO (and all the others) are on-policy approaches. Meaning, each algorithm samples from what the model can currently output, and each algorithm will only update based upon the "n" traces that the model undertook at that time step. This is the origin of the "winner-takes-all" collapse that regular RLVR faces. If you only train on the outputs your model gave you on this time step, then ( Sinha et al. (2026) and Lochab et al. (2026)), you will collapse. Our contribution is that we show via Re:Max (and prove in our appendix) that if you move to an off-policy setting (meaning you train over all unique solution modes), you will not collapse.

Reviewer_n2W4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies solution-mode collapse in reinforcement learning with verifiable rewards (RLVR), where post-training improves correctness but concentrates probability on a smaller set of correct solutions. It introduces ModeBench, a five-domain benchmark with executable verifiers that identify both correctness and discrete solution modes, together with PCMD for measuring diversity among correct responses. The authors show that several RLVR objectives substantially reduce solution-mode diversity across models and task difficulties. To mitigate this, they propose mode replay, which stores one exemplar per discovered correct mode and rehearses stored modes uniformly. Combined with Dr. GRPO or MaxRL, mode replay generally improves both correctness and solution diversity.

Critical strengths
  • The paper is generally well written.

  • The proposed ModeBench and the PCMD measure seem reasonable and effective.

Critical weaknesses
  • A particular related prior appears to be missing. DMPO (Beyond Mode Collapse: Distribution Matching for Diverse Reasoning, ICML26) identifies solution mode collapse via quality ratio and proposes to solve it through distribution matching. Moreover, the theoretical intuition discussed in Section 3.1 has also been similarly discussed in DMPO. A thorough discussion on the identification methods and a direct comparison with DMPO would therefore be necessary.

  • Some other related prior works on diversity collapse are missing in the empirical comparisons, e.g., G$^2$RPO: Geometric GRPO Escaping LLMs’ Reasoning Ruts to Break the Accuracy–Entropy Trade-off(ICML26)and The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward (ICLR26).

  • Although Table 1 includes a broad set of comparators, most of the evaluated training methods still rely on group-based objectives or are built on top of such backbones. Therefore, the claim about general RLVR in the abstract and introduction seems a bit overstated.

Questions for authors
  • How is mode replay compared empirically with DMPO, G$^2$RPO, and DPH-RL?

  • Could the authors include a non-group-based RLVR objective, such as PPO with a non-group-relative advantages in the empirical comparisons?

  • Can mode replay further enhance GAPO?

3 comments

Thank you for your review. We appreciate your finding that the proposed ModeBench and the PCMD measure are “reasonable and effective”.

To address your comments, we have performed additional experiments:

  1. Run the additional reviewer-recommended baselines, including DMPO, G²RPO, and DPH-RL.
  2. Run a non-group-based RLVR comparison (PPO).
  3. Discussed the benefit of mode replay against all on-policy objectives, including GAPO.

We’re grateful for your suggestion that additional comparisons are needed (beyond the pre-existing 7 baselines already explored in the paper), and we have put substantial effort into addressing them.

We have run open-source versions of DMPO, G²RPO, DPH-F, and PPO (the non-group comparative). We used the same Qwen2.5-0.5B-Instruct base, Level 1, as for our other baselines. Here are our results, over 5 seeds (where time permitted, 1 where not; we will update as our seeds continue to come in per objective/domain), reporting both pass@8 and PCMD:

First, here are the pass@8 values after training:

DomainDr.GRPOMaxRLReplay Dr.GRPOReplay MaxRLDMPOG²RPODPH-FPPO
Graph Coloring32.3% (5)54.2% (5)97.3% (5)94.1% (5)33.1% (5)84.1% (5)60.6% (5)31.6% (5)
Countdown48.1% (5)58.1% (5)67.2% (5)66.6% (5)64.4% (5)61.6% (5)71.9% (warm start) (5)55.5% (5)
Python Factors17.2% (5)17.2% (5)52.8% (5)68.1% (5)17.2% (5)17.3% (5)100.0% (warm start) (5)17.2% (5)
MathIR50.6% (5)44.7% (5)79.2% (5)79.2% (5)43.0% (5)53.8% (5)94.8% (warm start) (5)5.6% (5)
PantryPlan52.2% (5)53.1% (5)75.3% (5)74.8% (5)54.5% (5)61.1% (5)56.1% (5)65.2% (5)

Second, here are the PCMD values after training:

DomainDr.GRPOMaxRLReplay Dr.GRPOReplay MaxRLDMPOG²RPODPH-FPPO
Graph Coloring.001 (5).113 (5).740 (5).701 (5).014 (5).454 (5).310 (5)0.0037 (5)
Countdown.010 (5).023 (5).554 (5).579 (5).007 (5).003 (5)0.0190 (warm start) (5).015 (5)
Python Factors0 (5)0 (5)0 (5).309 (5)0 (5)0 (5)0.0020 (warm start) (5)0 (5)
MathIR.001 (5).000 (5).017 (5).007 (5).000 (5).002 (5)0.0042 (warm start) (5)0 (5)
PantryPlan.000 (5).000 (5).431 (5).415 (5).000 (5).006 (5).028 (5)0.1916 (5)

We note that DPH-F requires a frozen rehearsal pool of examples that the initial policy solves reliably. Under the original initialization, this pool was empty for all five seeds on Countdown, Python Factors, and MathIR. This prevented us from running the prescribed retention mechanism. Graph Coloring and PantryPlan had eligible examples under the original screening, so their reported runs did not require a warm start. To evaluate DPH-F on the three remaining domains, we therefore conducted a separate, matched warm-start experiment. The comparison (from an accuracy perspective) is therefore not quite fair, but we also felt that it would be not true to the spirit of the DPH-F authors / algorithm to do it without the warm start in those domains. We have not bolded their "wins" in accuracy (because they are zero percent accuracy under an equivalent training setting).

We find that empirically, we outperform all other baselines. We believe this is sufficient to address the empirical comparisons requested by this reviewer. We have added these results to Appendix Q.

We note that our contribution is not that binary-reward RL concentrates probability on certain outcomes (as this has been found in other papers, which we cite thoroughly). Instead, our contribution is that we:

  1. Provide a benchmark to evaluate the impact of mode collapse on a given objective.
  2. Show that mode collapse is a direct result of on-policy training setups in RLVR.
  3. Propose a potential way to mitigate mode collapse.

Unlike GAPO / other on-policy distribution-matching approaches, we directly optimize over off-policy, previously discovered modes. On-policy RLVR is guaranteed to concentrate, and we show this both experimentally and theoretically.

We thank the reviewer for helping us improve the paper!

We have updated our results since Friday (see above) to produce all baselines requested by this reviewer with a full 5-seeds of comparison. We are grateful for this reviewer's suggestion to include these additional comparatives (particularly the PPO baseline), and we hope that they demonstrate our algorithm's performance sufficiently under the expanded comparison.

Thank you for the rebuttal. I appreciate the authors' efforts and the stronger empirical evidence.

I have raised my rating from 2 to 3.

Reviewer_1d9rOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes MODEBENCH, a benchmark of multi-solution tasks that evaluates not only correctness but also the diversity of solution modes discovered by models, measured using pairwise correct-mode diversity (PCMD). The authors show that RL post-trained models tend to concentrate probability mass on specific modes, reducing solution diversity, particularly in frontier models. To address this issue, they introduce Re:Max, a method that stores one verified example per discovered mode in a replay buffer and trains on these stored examples to preserve solution diversity.

Critical strengths
  • The proposed PCMD metric, to my knowledge, is novel and insightful, providing a useful way to quantify solution-mode diversity beyond correctness.
Critical weaknesses
  • Although the authors introduce MODEBENCH as a new benchmark, most of its constituent datasets already exist (e.g., Countdown, Graph). The primary novelty lies in the proposed PCMD metric rather than the datasets themselves. I find the claim of introducing a new benchmark somewhat overstated.
  • The finding that RLVR substantially reduces solution-mode diversity across domains (Contribution 2) is not entirely novel, as similar observations have already been reported in prior work [1, 2].
  • The proposed method relies on identifying which solution mode each response belongs to, which may be difficult to generalize to other domains or more complex tasks, such as Olympiad-level mathematics and competitive programming.
Questions for authors
  • In Fig.4, the authors claim that "as task difficulty increases, solution-mode diversity generally declines, including in settings where correctness remains near saturation." However, for smaller models, solution-mode diversity appears to increase. Could the authors explain why smaller models exhibit increasing diversity, in contrast to larger frontier models?
  • Could the authors explain how Re:Max could be applied to standard RLVR training datasets, such as DeepScaler and DeepCoder, where identifying distinct solution modes may be challenging?
  • To what extent do models trained with Re:Max on tasks where solution-mode diversity can be measured generalize (as measured by Pass@k with large k values) to standard reasoning benchmarks, such as AIME, MATH-500, and LiveCodeBench-v5?

[1] Assessing Diversity Collapse in Reasoning. SSI-FM 2025.
[2] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? NeurIPS 2025 Oral.
3 comments

Thank you for your review. We’re glad you find our PCMD metric novel and insightful for quantifying solution mode diversity beyond correctness. We address your questions below.

“Most of its constituent datasets already exist…” Thank you for your question – we agree! We did not invent these problem domains, nor do we claim to be the only paper to use them. However, we differ from existing works in two ways. First, we construct a set that shows not only the problems but also different solution modes (across 5 problem domains and 5 difficulty levels). Second, we include a verifier that uses our benchmark to ensure each mode corresponds to a normalized computation tree. These distinctions are fundamental to constructing a substantive, publicly available evaluation setup for assessing solution mode collapse on new and existing RLVR objectives.

“Finding that RLVR substantially reduces solution-mode diversity across domains (Contribution 2) is not entirely novel….” Absolutely, we agree (and both of the works you provide are already referenced in the Related Works of our paper – page 9; 445-446). However, we argue that our work provides two novel contributions, namely ways to (1) measure and (2) mitigate the problem of mode collapse. On the measurement side, we introduce a benchmark that allows the precise quantification of solution mode collapse. On the mitigation side, we propose a new algorithm that vastly outperforms existing approaches.

“The proposed method relies on identifying which solution mode each response belongs to, which may be difficult to generalize to other domains [...] such as Olympiad-level mathematics and competitive programming.” It is really hard to explore diversity in existing benchmarks. You’ve pointed out a key problem, which is that both (1) defining solution mode and (2) assigning solution mode to each model output is tough. So, we propose our benchmark as a unified way to evaluate this phenomenon on constrained domains. Even in this constrained setting, we show that frontier models suffer from solution mode collapse. Moreover, we find that post-training smaller models with our algorithm results raises pass@8 by 90% relative to Dr.GRPO with 1.9x its effective verified modes.

“... for smaller models, solution-mode diversity appears to increase…” Great point. We do claim that “as task difficulty increases, solution-mode diversity generally declines”. Our hypothesis is that frontier models find a single mode and always use that (correct) response. Whereas, small models are not often “right”, but demonstrate less evidence of collapse / produce more ways of being right. This inspired us to consider that (1) accuracy is not the right metric for evaluation in this setting and (2) that perhaps something in the way larger models are trained results in mode collapse. This insight then dictates our exploration of “why” models collapse.

“Could the authors explain how Re:Max could be applied to standard RLVR training datasets” Re:Max could be applied through a separate model that indicates whether a solution differs from the pre-existing path (LLM-as-judge) to then include in our set of solution modes. We feel that this goes beyond the purpose of this paper, but would make a great future contribution.

“To what extent do models trained with Re:Max on tasks where solution-mode diversity can be measured generalize (as measured by Pass@k with large k values) to standard reasoning benchmarks, such as AIME, MATH-500, and LiveCodeBench-v5?” In math domains, such as AIME and MATH-500, there usually is only one way to solve the problem (we’ve tried this). So, adding diversity in this setting does not work well – because there is no need for diversity within those specifically designed benchmarks (which really only have at most 2 ways to solve). For coding problems, we’ve included Codeforces as our benchmark (see Appendix G) and post-trained w/ RLVR Qwen2.5-Coder-7B-Instruct. We find that replay produces a small but consistent training-side increase in the fraction of fresh rollouts that verify. We believe this would hold for other coding benchmarks as well. Check out Appendix G, and let us know if any questions on this one remain.

Thank you for your questions and helping us improve the clarity of our paper. We’ve aimed to answer every question thoroughly. If any details remain unclear, please do not hesitate to let us know.

As we wrap up the review period, we wanted to ensure that our answers to your points above were complete (and that new questions did not arise as you considered our responses).

  1. "The primary novelty lies in the proposed PCMD metric rather than the datasets." -- I'm glad we can agree that PCMD is a novel idea! However, in order to calculate PCMD, you must be able to have a sense of how many modes exist (or theoretically exist) for a given problem. We did the work, for each of those domains, to write a substantive procedure to find the full set of solution modes (or unique reasoning traces in the math domain). This work, designed to figure out, per-objective (a) how to measure mode collapse is really occurring over post-training (beyond naming the phenomena) is new. Moreover, now easy to do for any future objectives. Additionally (b) we show that given the measurement, that mode collapse is not an inherent property of RLVR and can be mitigated without loss of accuracy.

  2. "Relies on identifying which solution mode each response belongs to" -- (a) We'd argue that we make a case for why one might want to find individual solution modes / why this labor of identifying diverse solutions is important. (b) Additionally, even if you naively use the "first" mode (and thus no label), we show that our Re:Max method offers improvements over a slew of objectives.

We believe our responses to be sufficient to address your critical weaknesses concerns, but we're eager to address any lingering comments / questions that you may have during this rebuttal period. Thank you for your consideration!

Thank you for your response. As I mentioned, the finding that RLVR substantially reduces solution-mode diversity is not novel. While the PCMD metric is indeed novel, it cannot be directly applied to existing benchmarks. Furthermore, my main concern is whether RL training on tasks where solution-mode diversity can be measured (e.g., Countdown) leads to improvements that transfer to existing benchmarks (AIME, etc.), rather than simply adding diversity on those benchmarks. Therefore, I will maintain my original score.

Reviewer_o5WNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how RLVR affects the diversity of correct solutions. The authors (1) introduce ModeBench, five executable domains whose verifiers return both correctness and solution mode, with a metric, PCMD, for the probability that two correct answers differ; (2) show that RLVR (Dr.GRPO, GRPO, MaxRL) reduces PCMD across domains and model scales (0.5B–3B), and that frontier models are accurate but concentrated; and (3) propose Re:Max, which stores one verified exemplar per discovered mode and replays all modes uniformly. Over five seeds, replay improves both pass@8 and PCMD relative to no-replay controls and eight diversity-oriented baselines, with gains persisting at harder difficulty levels.

Critical strengths
  1. ModeBench’s verifiers identify solution modes directly, and PCMD separates diversity from accuracy without a judge model, making the mode-collapse finding precise and reproducible.

  2. Re:Max improves both pass@8 and PCMD across domains, model scales, and RL objectives, rather than trading one for the other.

Critical weaknesses
  1. Re:Max needs mode labels that typical RLVR tasks lack. Standard math or code verifiers return only correct or incorrect, so the method’s practical relevance is unclear, and no approximate mode criterion is tested.

  2. Evidence is limited to small models (0.5B–3B) and synthetic tasks. It is unclear whether the gains hold where RLVR is used in practice.

Questions for authors

Questions:

  1. How does Re:Max perform without ground-truth mode labels, e.g., deduplicating by answer string or embedding clusters? With compute matched for the extra replay updates, e.g., an equal number of updates on randomly chosen correct samples, do the gains remain?

  2. How were baseline hyperparameters tuned? Why is Dr.GRPO’s pass@8 below the untrained checkpoint in some domains (Table 1)?

Suggestion for future work:

Results on a larger model (7B+) or a standard reasoning benchmark (e.g., MATH or a code benchmark).

3 comments

We thank you for your thoughtful review and for recognizing that ModeBench and PCMD make the mode-collapse finding "precise and reproducible," and that Re:Max improves both pass@8 and PCMD "rather than trading one for the other." We address your questions below.

"Re:Max needs mode labels that typical RLVR tasks lack... How does Re:Max perform without ground-truth mode labels, e.g., deduplicating by answer string or embedding clusters?"

We thank the reviewer for this important question. We agree that applicability beyond mode labels is important, and moreover, we would like to highlight our existing appendix experiment that answers your question. In P.5, we train a single-slot variant of replay that stores only the first verified success for each prompt. This requires nothing beyond a standard binary verifier, and increases terminal PCMD from .004 to .218 and pass@8 from .444 to .707, recovering 58% of the diversity gain of full Re:Dr (PCMD .372, pass@8 .796). Replaying verified successes is therefore useful with any binary verifier, and mode labels contribute an additional gain by balancing rehearsal across the solutions a model has found. We thank you for your feedback and will move this result into the main text.

Our current solution finds that determining diversity is separate from the problem of maintaining model diversity. Knowing this, we will add future work to explore ways to separate the problem of finding appropriate diverse solution modes through embedding clusters / LLM-as-judge.

“With compute matched for the extra replay updates, e.g., an equal number of updates on randomly chosen correct samples, do the gains remain?"

We thank the reviewer for this suggestion, and we would like to reference you to this exact experiment in Appendix P. Our replay and baselines already share the same optimizer updates and fresh rollout settings. , The baselines run the same buffer and scoring code with the replay gradient set to zero. We show that replay adds only 2.4% learner time (compared to MaxRL and Dr.GRPO). We further gave Dr.GRPO 50% more fresh rollouts, either through 12 rather than 8 training passes or through larger rollout groups, and find that control PCMD remains at .013 and .008 respectively, against .376 for Re:Dr, with pass@8 near .43 against .75. So, even if we gave our baselines equal compute, they would not perform as well as Re:Dr or Re:Max.

"How were baseline hyperparameters tuned? Why is Dr.GRPO's pass@8 below the untrained checkpoint in some domains (Table 1)?"

We thank the reviewer for this question. We would like to clarify that all methods share the same learning-rate schedule, optimizer, group size and rollout budget, and our baselines use the best settings as reported in their papers. The lower pass@8 for Dr.GRPO in Graph and PantryPlan is indicative of severe mode collapse in these domains after a set (consistent across all objectives) number of training steps (which impacts accuracy as well as diversity). As the policy concentrates on a single answer per prompt, its eight samples become near-identical, so when that answer is wrong, all eight fail together. This is why pass@1 can rise while pass@8 falls, consistent with Yue et al. (2025), and these two domains are where we observe the steepest loss of solution diversity. We will add this explanation alongside Table 1 in the final revision.

"Evidence is limited to small models (0.5B–3B) and synthetic tasks... Results on a larger model (7B+) or a standard reasoning benchmark."

We would like to highlight two existing results on larger models in our existing Appendix. First, we found that on Qwen2.5-7B, Re:Max raises pass@8 from .342 to .789 relative to MaxRL and PCMD from .028 to .245 (App. E.2). Next, we train Qwen2.5-Coder-7B-Instruct on human-authored Codeforces problems whose checkers were not designed for this study (App. G), where replay produces a small but consistent improvement in verified rollouts (+.014, positive in 9 of 10 seed pairs). As mode collapse is a pervasive issue that is harder to study in complex domains, we focused our benchmark on classical math reasoning tasks with solutions that are easy to interpret. Moreover, we demonstrate that in inference settings mode collapse happens even in frontier models / is not limited to our small model training setting (see Figure 4 & 5) .

Thank you for your questions and helping us improve the clarity of our paper. We’ve aimed to answer every question thoroughly. If any details remain unclear, please do not hesitate to let us know.

As we are coming to the end of the review period, we wanted to ensure there were not any additional questions / lingering doubts on your end. We believe that your core issues are that:

  1. "Re:Max needs mode labels that typical RLVR tasks lack". To resolve, we've clarified that (a) our benchmark contribution (ModeBench) explicitly provides the labels needed, in order to fairly evaluate diversity given a known set of solutions. (b.1) while Re:Max works best when the labels are known, just saving one right label per target recovers 58% of the diversity gain of full Re:Dr. and (b.2) future work could employ an LLM-as-judge architecture to label outputs. In this current work, we show that finding unique labels is worthwhile (i.e. given you have a way to label, our Re:Max objective will result in a both more accurate and more diverse RLVR post-trained model).

  2. "Evidence is limited to small models (0.5B–3B) and synthetic tasks." We include (0.5B-3B) in our full experiment size given that each model takes ~1 day to train per seed, per domain on a GPU. So, at 5 seeds, for 5 domains, for 3 scales, per objective. We have done 5 * 5 * 3 * (>= 4) days of training per model. (1) We show 7B comparative in the appendix for both our synthetic tasks (one seed) and (2) a real set of non-synthetic Codeforce problems at the 7B size.

We believe that this addresses both your listed concerns. Please let us know!

Thank you for the clarifications. I will maintain my original score.

Comments on the paper

1 comment

Thank you again to all our reviewers. We have undertaken extensive additional work in response to the valuable feedback and suggestions provided during this review cycle, including:

Reviewer o5WN: We added 7B-scale experiments, human-authored Codeforces tasks, and evaluations on standard mathematical reasoning benchmarks, alongside additional compute-budget and hyperparameter-tuned controls. We also demonstrated that replay can improve performance without explicit mode labels.

Reviewer 1d9r: We clarified ModeBench's novelty relative to existing problem domains and prior work on mode collapse. Our contributions go beyond identifying the phenomenon to providing a systematic framework for its measurement and a method for its mitigation.

Reviewer n2W4: We implemented all four explicitly requested baselines (DMPO, G$^2$RPO, DPH-F, and non-group-relative PPO), reporting pass@8 and PCMD across five domains with five seeds where eligible. These comparisons substantially expand our seven pre-existing baselines. We also strengthened our related-work discussion and clarified the distinction between on-policy diversity objectives and verified-mode replay.

Reviewer SaPG: We added pass@1 versus pass@8 comparisons across all five domains and three model scales, demonstrated the effectiveness of replay without explicit mode labels, and expanded our discussion of when inference-time versus training-time approaches to mode collapse are appropriate and how they may complement one another.

We believe these additions substantially address the concerns raised during review. We sincerely appreciate the reviewers' suggestions, which have materially strengthened our work. We care deeply about the scientific questions that motivate this paper. We believe that it is important to train not just for accuracy, but in a way that ensures the model does not collapse to a subset of potential solutions / reasoning paths. We believe that our benchmark for measurement and our algorithm for mitigation represent a valuable contribution to the RLVR community, LLM output robustness studies, and researchers focused on exploration during post-training.

"it is better to solve one problem five different ways than five problems one way." -- George Pólya

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.