- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a strategy to mitigate the mode collapse problem during RL post-training of language models. They highlight that this problem is prevalent even for frontier models, where on simple tasks frontier models are also not able to sample diverse enough modes.
Critical strengths
- The work studies an important problem of lack of diversity in LLMs and collapse of diversity during post-training.
- On the tasks studied in the paper the proposed method provides strong improvement over the baselines.
Critical weaknesses
I would be willing to reconsider my assessment if the authors could provide clarifications about the below points:
-
It is unclear to me how much of a pass@1 vs pass@N tradeoff is happening out here. Is it the case that the proposed approach achieves a higher pass@8 but lower pass@1 than something like GRPO or MaxRL? It's fine for a tradeoff to exist. In which case I think reporting a graph which has pass@N (or PCMD) on the Y axis and pass@1 on the X axis and different methods as points on this graph might provide better understanding about whether the proposed approach has a better tradeoff than the baselines?
-
If the goal is to optimize for sampling multiple modes, how would approaches like multi-turn sequential sampling work? For e.g., if the model samples a solution, it is just told to "sample a solution different from ones sampled in previous turns". I expect this to improve the diversity of the outputs from different turns.
-
It would be helpful to discuss / compare against approaches which have aimed to address the mode collapse problem without using mode labels such as [1]. It seems to me that the problem highlighted in the paper could be fixed by changing the sampling strategy (above point), or by making changes to the objective that do not need extra labels for the mode. The use of these extra labels could restrict the broader applicability of the method
[1] GX-Chen, A., Prakash, J., Guo, J., Fergus, R., and Ranganath, R. "KL-Regularized Reinforcement Learning for Generative Modelling is Designed to Mode Collapse." ICLR 2026. arXiv:2510.20817.
Questions for authors
Refer above weaknesses
6 comments
Reviewer Response
Thanks for the clarifications and references to the relevant sections of the appendix! I would like to follow up on the authors' responses
1. Should one address the diversity collapse problem through training, or do scaffold-like methods (e.g., multi-turn sampling) subsume training methods?
I would like more intuition / evidence that training-based solutions are orthogonal to solutions that build scaffolds. My current intuition is that many of these diversity issues can be fixed with simple tricks like the multi-turn sampling I suggested in my review.
I ran an experiment on the graph coloring problem comparing (i) sampling 8 solutions per problem, and (ii) sampling 4 solutions, then generating the 5th by conditioning on the 1st and instructing the model to give a different solution, the 6th similarly from the 2nd, and so on. The experiment used the 96 problems generated by running
python ops/make_exact_answer_mode_data.py \
--output-root data/released_exact_answer \
--eval-splits multi_answer
Results on Gemini-4-Carbon and Qwen2.5-7B-Instruct are below. Conditioning improves PCMD for both models:
| Gemini-4-Carbon pass@8 | Gemini-4-Carbon PCMD | Qwen2.5-7B-Instruct pass@8 | Qwen2.5-7B-Instruct PCMD | |
|---|---|---|---|---|
| Parallel | 1.000 | 0.071 | 0.719 | 0.336 |
| Conditioned (4 + 4) | 1.000 | 0.642 | 0.688 | 0.438 |
Given these results I would like to hear the authors' thoughts on should one focus more on training based methodologies to address the problem or inference.
2. If training is a desired route to address such issues, I feel the need for labels restricts the generality of the method
I noticed that in response to other reviewers' the authors have added more baselines and Replay seems to works well, but is there any reason why methods like DMPO which have been designed to improve diversity have worse PCMD than MaxRL?
Also, I couldn't find the pass@1 vs pass@N / PCMD plot in the appendix, could you please direct me to which is the plot? Thanks!
(I used Claude Opus 5.5 to write code for above experiments.)
1. Should one address the diversity collapse problem through training, or do scaffold-like methods (e.g., multi-turn sampling) subsume training methods?
Thank you for following up on this comment. The reason that we do not recommend multi-turn / inference time approaches, is because inference cannot "make up" for the things that training has already removed. As we say in our intro: "what makes models useful at deployment depends on having different candidates to choose from, i.e. test-time search and reranking (Brown et al., 2024; Snell et al., 2024; Cobbe et al., 2021; Wang et al., 2023). As RLVR improves accuracy, what happens to the diversity of correct solutions?"
Your suggestions about inference time calls for diversity / multi-turn sampling fall into this bucket. A model will not suggest something that has already been "collapsed" or "forgotten". If we post-train on a set of problems with RLVR (in a way that is commonly done today), without any interventions, then: the set of outputs a model will produce will collapse (and eventually leave you in a situation where you cannot actually take advantage of inference-time techniques) -- the very thing we would like to promote.
Thank you for running the experiment! We appreciate the time you have taken to throughly engage with our research. Your findings are consistent with our own for parallel, and we are not at all surprised by your findings using an inference-time technique.
"Given these results... should one focus more on training based methodologies to address the problem or inference."
It depends on what you want to do! We think that the key distinction here is RLVR post-training:
- If your goal is to take an existing model and find the full extent of all diverse solutions the given model can currently provide, then inference methodologies is the right answer.
- If your goal is to have a model become "better" at a task through RLVR post-training (e.g. come up with things that the pre-trained model didn't consider), then you will need to make sure you maintain diversity while working on becoming "more accurate".
Our paper is about the later, and so we propose a solution to address the issue in the later category. The former differs from our efforts. To test / further your experiment, we encourage you to:
- Post-train with RLVR the Qwen2.5-7B-Instruct with GRPO.
- Compare your post-trained Qwen2.5-7B-Instruct against your pre-trained Qwen2.5-7B-Instruct with inference time methods.
Our research indicates that your post-trained Qwen2.5-7B-Instruct will have poorer PCMD on both parallel and conditioned.
2. If training is a desired route to address such issues, I feel the need for labels restricts the generality of the method
We want to separate two things:
- ModeBench, the benchmark: if the goal is to show how much of the solution space your algorithm can find, then you need to have a way to show-off the diversity of your model vs the complete diversity space. Thus, you need labels.
- Re:Max, the algorithm: our algorithm prefers labels, where they exist, but (1) does not require them (as we discuss with another reviewer) and (2) another model could be deployed to determine whether a given solution is unique, thus providing the set of labels.
Also, I couldn't find the pass@1 vs pass@N / PCMD plot in the appendix, could you please direct me to which is the plot? Thanks!
Apologies for the inconvenience. The figure is not available in the appendix you can see on this page (and we cannot update the appendix during the review period). We have added it to our updated appendix (held locally). In words, it show that: "without replay, we find that per-response correctness increases and pass@8 decreases over training in Graph and PantryPlan, falling below the untrained model (as we show in Table 1). As RLVR moves the policy towards a single answer, pass@1 increases at the expense of pass@N."
"I noticed that in response to other reviewers' the authors have added more baselines and Replay seems to works well, but is there any reason why methods like DMPO which have been designed to improve diversity have worse PCMD than MaxRL?" (We assume you mean Re:Max (our algorithm) here, rather than MaxRL. If not let us know.)
The issue is that DMPO (and all the others) are on-policy approaches. Meaning, each algorithm samples from what the model can currently output, and each algorithm will only update based upon the "n" traces that the model undertook at that time step. This is the origin of the "winner-takes-all" collapse that regular RLVR faces. If you only train on the outputs your model gave you on this time step, then ( Sinha et al. (2026) and Lochab et al. (2026)), you will collapse. Our contribution is that we show via Re:Max (and prove in our appendix) that if you move to an off-policy setting (meaning you train over all unique solution modes), you will not collapse.
We thank you for your thoughtful review and for recognizing that our work "studies an important problem" and that Re:Max "provides strong improvement over the baselines." We address your questions below.
"It is unclear to me how much of a pass@1 vs pass@N tradeoff is happening out here... reporting a graph which has pass@N (or PCMD) on the Y axis and pass@1 on the X axis... might provide better understanding."
We thank the reviewer for this suggestion and have added this figure to our Appendix, plotting per-response correctness (pass@1) against pass@8 and PCMD for every method and model scale.
Interestingly, the tradeoff the reviewer anticipates appears in the baselines rather than in replay. Without replay, we find that per-response correctness increases and pass@8 decreases over training in Graph and PantryPlan, falling below the untrained model (as we show in Table 1). As RLVR moves the policy towards a single answer, pass@1 increases at the expense of pass@N. Figure 1 shows this on a single problem, where Re:Max produces fewer invalid answers than Dr.GRPO and retains more solution modes.
"If the goal is to optimize for sampling multiple modes, how would approaches like multi-turn sequential sampling work?"
We thank the reviewer for this suggestion, which we view as complementary to Re:Max. At inference time, we tested several related interventions. Raising temperature, changing sampling, and increasing the sample budget. Under each of these interventions, we do not recover the diversity lost under RLVR (App. I), and moreover, explicit diversity prompting helps inconsistently across domains (App. L.3). Even frontier models remain highly concentrated, with GPT-5.6 Sol producing a single solution mode on harder MathIR problems. We have added multi-turn sequential sampling as a future work idea to the main body of the paper. However, we anticipate that mode collapse would still emerge under this setting, but would perhaps take longer to fully converge towards a singular solution.
"It would be helpful to discuss / compare against approaches which have aimed to address the mode collapse problem without using mode labels such as [1]... The use of these extra labels could restrict the broader applicability of the method."
We thank the reviewer for this point and agree that applicability matters. Notably, a substantial part of replay's benefit does not require mode labels. Our single-slot variant (App. P.5) stores only the first verified success per prompt, needing nothing beyond a binary verifier, and increases PCMD from .004 to .218 and pass@8 from .444 to .707, recovering 58% of the diversity gain of full Re:Dr. Mode labels add a further improvement by balancing rehearsal across the solutions found. We have moved this result into the main text as per your suggestion.
We thank the reviewer for pointing us to GX-Chen et al. (2026), which we have since added to our citations in the current version. Their work characterizes how the KL-regularized objective itself can favor collapsed solutions, while our analysis focuses on the setting of binary rewards, where a rarely sampled correct solution receives a vanishing on-policy gradient. Because replay acts on stored solutions rather than the objective, the two approaches should be compatible. A sweep over KL regularization to the initial policy (App. E.7) preserves more diversity in PantryPlan, where the base model is already highly diverse, but achieves a worse correctness–diversity tradeoff than replay in Graph and MathIR. We also compare against several label-free approaches in our Appendix. UCPO and RLEP improve PCMD by only .009 over Dr.GRPO, compared with .320 for Re:Max.
Thank you for your questions and helping us improve the clarity of our paper. We’ve aimed to answer every question thoroughly. If any details remain unclear, please do not hesitate to let us know.