acceptodds
Under review as a conference paper at ICLR 2027

Rethinking diversity in RLVR: from policy entropy to semantic coverage

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard paradigm for improving the reasoning capabilities of Large Language Models. However, we find that successful responses become increasingly concentrated in fewer semantic reasoning modes as training proceeds. Under a finite rollout budget, this concentration causes on-policy sampling to repeatedly revisit dominant modes, reducing coverage of alternative successful regions and limiting further policy improvement. Theoretically, we attribute this depressing phenomenon to the finite-sample policy gradient estimation, which theoretically leads the policy model collapse on a single response. Motivated by our analysis, to preserve the diverse response modes, we propose SCARS, a framework that organizes discovered responses and intermediate reasoning states in a hierarchical semantic tree and redirects rollout budget toward under-covered successful regions through prefix-conditioned generation. Experiments across multiple reasoning benchmarks show that our method improves overall reasoning performance while maintaining substantially higher successful-response diversity than other baseline methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM post-training, semantic diversity, exploration

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_6aEXOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the loss of semantic diversity among successful responses during reinforcement learning with verifiable rewards. It distinguishes token-level policy entropy from coverage of successful reasoning modes and analyzes concentration dynamics in a simplified response-level policy model. The authors propose SCARS, which organizes historical responses and intermediate prefixes in a hierarchical semantic tree, preferentially samples under-covered successful regions, and trains on prefix-conditioned continuations with segment-wise credit assignment. Experiments report improvements in mathematical reasoning performance and successful-response semantic entropy, together with gains on code-generation benchmarks. Component ablations examine the contributions of prefix-conditioned exploration, coverage-aware selection, and credit reassignment.

Critical strengths

By distinguishing token-level randomness from diversity among successful reasoning paths, the paper identifies a limitation of using policy entropy as a proxy for effective exploration. Appendix A.2 further shows that, under the studied parameterization, more probable successful responses receive larger expected logit updates, amplifying existing imbalances. Within its assumptions, this analysis helps explain why improving reward need not preserve multiple successful solutions.

Maintaining historical successful trajectories gives the current policy access to intermediate states from previously discovered solutions, even when their complete trajectories have become difficult to sample. The association between complete-response modes and intermediate prefixes turns coverage-based selection into a concrete generation procedure. This provides a plausible way to reuse successful experience while allowing the current policy to explore new continuations.

The reported results show improvements in both mathematical performance and measured successful-response diversity. On the primary backbone, adding SCARS to GRPO increases average mathematical pass@32 from 46.08 to 48.34 and Math12K test-set successful semantic entropy from 0.359 to 0.630. Average performance also improves when SCARS is combined with DAPO and when evaluated on the second backbone, supporting its usefulness beyond a single training configuration.

The experimental study includes relevant exploration-oriented baselines and ablations that separate several components of the method. Comparing prefix-conditioned generation with historical response replay, coverage-aware selection with uniform selection, and the proposed credit assignment with standard response-level advantages helps assess what each component contributes. In particular, the uniform-selection comparison provides evidence that coverage-aware allocation improves average mathematical performance beyond prefix reuse alone.

Critical weaknesses

The characterization of finite-sample gradient estimation as the root of collapse is stronger than the analysis supports. Appendix A.2 already demonstrates a concentration tendency under exact expected-gradient updates, while Appendix A.3 uses sampling noise to exclude convergence to equilibria supported on multiple successful responses. These are distinct contributions to the argument. Moreover, the KL-regularized optimum discussed in Section 3.1 and the subsequent unregularized dynamics concern different objectives. The main text should distinguish these roles more carefully rather than attributing the concentration mechanism to finite sampling alone.

The robustness argument also requires a more explicit connection to the earlier analysis. Earlier sections define semantic classes through generated responses to a fixed prompt, while Appendix A.4 treats classes as a policy-independent component of the data distribution. Without establishing how these notions correspond, the worst-case prior-shift result does not directly explain why preserving multiple solutions per prompt should produce the reported cross-task generalization benefits.

A prefix associated with a rare historical mode may lead the current policy to a different mode, including a dominant one. The paper does not directly quantify how often selected prefixes yield new successful responses in their intended modes. The initialization case study and aggregate diversity results leave this aspect of the proposed mechanism unresolved.

Both tree construction and diversity evaluation also rely on the same embedding model. Although the semantic-tree case study provides qualitative examples of interpretable clusters, the paper does not report a systematic independent assessment of whether the evaluation clusters correspond to meaningfully different valid solution strategies. The CMIMC25 failure case described in Appendix D.4 illustrates that irrelevant text can inflate apparent diversity despite verifier acceptance. Differences in successful sample counts may further affect the entropy estimates. These issues limit how confidently higher measured entropy can be interpreted as preservation of distinct successful reasoning strategies.

Questions for authors

Could the authors revise the causal interpretation to distinguish the concentration tendency already present under expected-gradient updates from the role of sampling noise in excluding non-singleton equilibria? Please also clarify how the KL-regularized optimum in Section 3.1 relates to the subsequent unregularized analysis, and which aspects of the observed collapse this comparison is intended to explain.

How is the policy-independent variable (C) in Appendix A.4 related to the response-dependent mapping (\psi_x(y)) when a prompt admits multiple correct solution strategies? What connection is established, rather than hypothesized, between Theorem 2 and the empirical semantic-entropy and cross-task results?

Using existing rollout records, could the authors report how often continuations remain in the selected historical mode, switch to a dominant mode, or enter another mode, distinguishing short and long prefixes where possible? An independently assessed sample of these transitions, together with a comparison controlling successful sample counts, would help determine whether the diversity gains reflect distinct valid reasoning strategies rather than clustering or sampling effects.

Reviewer_9UQeOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper explains the semantic concentration problem in RLVR from the perspective of response diversity (instead of the popular entropy ). It proposes a hierarchical sampling framework with semantics-aware resampling, together with a credit assignment design that reinforces outputs from under-covered successful regions, so that diversity is preserved.

Critical strengths

The motivation is clear, the experimental and theoretical analysis behind it is thorough, the ablations are complete, and the paper was a pleasure to read. I’m positive about this work, so I only have a few remaining concerns.

Critical weaknesses
  1. The paper shows that preserving entropy does not necessarily improve diversity, and so does not guarantee better performance either. For example, R1-Zero-Div retains higher entropy than SCARS, yet its diversity is not better. This runs counter to the common view. I’m curious whether there are counterexamples, i.e., methods that preserve entropy better and achieve higher diversity than SCARS, and whether such a method could outperform GRPO+SCARS. This would not diminish the contribution of SCARS, which is a simple and direct way to optimize diversity.
  2. Is the number of tree categories fixed? And is the effectiveness limited by the diversity of the base model itself?
  3. Figure 2 is not very easy to read. The current style makes it hard to match the figure with the results described in the text.
Questions for authors

See weaknesses above.

Reviewer_E4YuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper finds that successful responses become increasingly concentrated in fewer semantic reasoning modes as training proceeds. It argues that token-level entropy does not fully capture this problem. The proposed SCARS stores historical responses and prefixes in a semantic tree, and applies separate credit assignment to prefixes and continuations. Experiments on two models show higher semantic diversity and modest gains on math and code benchmarks.

Critical strengths
  1. The distinction between token entropy and successful reasoning diversity is well motivated.
  2. Reusing historical prefixes offers a practical way to revisit rare successful paths.
  3. The proposed SCARS improves the average math score with both GRPO and DAPO.
Critical weaknesses
  1. As shown in Table 8, SCARS needs 32 initial responses per prompt and an 8B embedding model. The paper does not report the full training cost when compared with traditional RL methods.

  2. In Table 4, removing prefix credit reassignment sharply reduces semantic diversity. It remains unclear how much of the gain comes from semantic coverage rather than reinforcing successful prefixes.

  3. In Table 2, the math improvement on Qwen3-4B-Base is only 0.83 points. The authors may report math pass@1 and results from multiple training seeds. Furthermore, the performance on larger models needs more justification.

Questions for authors

Please refer to the weakness.

2 comments

Dear Reviewer,

As the rebuttal period is approaching its end, we would like to follow up on our previous responses. We have carefully considered the questions raised in your review and provided detailed responses to each of them.

We look forward to further feedback from all reviewers, as it will help us improve the quality of our work. If you have any additional questions or concerns, we would be happy to address them.

Best regards,
The Authors of Paper 17645

Area_Chair_C26eOpenReview ↗

Reminder: Author discussions end

Dear Reviewer,

This is a brief reminder that responding to rebuttal comments and engaging in discussions is a core part of a reviewer's responsibilities. Without your participation, it will be much more difficult to provide a well-informed recommendation for the submission.

I therefore kindly ask you to acknowledge the authors' rebuttal, indicate whether it has addressed your concerns and whether it affects your overall assessment. Specifically, please clarify which points have not been sufficiently addressed and respond to any questions raised by the authors.

Best regards, AC

Comments on the paper

1 comment

Dear Area Chair,

I hope this message finds you well.

As the rebuttal period is approaching its end, we would like to follow up on the current status of our discussion. We are glad that Reviewers 9UQe and 6aEX have engaged with our responses and provided positive feedback. However, we have not yet received any follow-up response from Reviewer E4Yu.

We have thoroughly addressed the concerns raised by all reviewers through detailed analyses, clarifications, and additional experiments. In particular:

  • Theoretical Analysis. We further clarified the underlying mechanism of semantic diversity collapse in RLVR, distinguishing the rich-get-richer effect in expected on-policy policy-gradient dynamics from the role of finite-sample stochasticity in driving convergence toward a single successful response mode.

  • Methodological Validation. We conducted additional experiments to demonstrate the feasibility of guiding semantic exploration through prefix-conditioned continuation, clarified the complementary roles of coverage-aware selection and credit reassignment, and further assessed the reliability of semantic clustering and diversity evaluation through LLM-as-a-Judge and human-expert verification.

  • Experimental Results and Computational Efficiency. We provided a detailed analysis of SCARS's training overhead compared with vanilla GRPO, evaluated lightweight embedding models, and discussed engineering optimizations including asynchronous pipelining and parallel tree maintenance. We also supplemented the experimental results across different model backbones and multiple random seeds.

We hope that all reviewers have the opportunity to fully consider our responses before the discussion phase concludes.

We would appreciate your assistance in encouraging Reviewer E4Yu to review our rebuttal and share any further feedback. We would be happy to address any remaining questions or concerns.

Thank you for your time and consideration.

Best regards,

The Authors of Paper 17645

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.