- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the loss of semantic diversity among successful responses during reinforcement learning with verifiable rewards. It distinguishes token-level policy entropy from coverage of successful reasoning modes and analyzes concentration dynamics in a simplified response-level policy model. The authors propose SCARS, which organizes historical responses and intermediate prefixes in a hierarchical semantic tree, preferentially samples under-covered successful regions, and trains on prefix-conditioned continuations with segment-wise credit assignment. Experiments report improvements in mathematical reasoning performance and successful-response semantic entropy, together with gains on code-generation benchmarks. Component ablations examine the contributions of prefix-conditioned exploration, coverage-aware selection, and credit reassignment.
Critical strengths
By distinguishing token-level randomness from diversity among successful reasoning paths, the paper identifies a limitation of using policy entropy as a proxy for effective exploration. Appendix A.2 further shows that, under the studied parameterization, more probable successful responses receive larger expected logit updates, amplifying existing imbalances. Within its assumptions, this analysis helps explain why improving reward need not preserve multiple successful solutions.
Maintaining historical successful trajectories gives the current policy access to intermediate states from previously discovered solutions, even when their complete trajectories have become difficult to sample. The association between complete-response modes and intermediate prefixes turns coverage-based selection into a concrete generation procedure. This provides a plausible way to reuse successful experience while allowing the current policy to explore new continuations.
The reported results show improvements in both mathematical performance and measured successful-response diversity. On the primary backbone, adding SCARS to GRPO increases average mathematical pass@32 from 46.08 to 48.34 and Math12K test-set successful semantic entropy from 0.359 to 0.630. Average performance also improves when SCARS is combined with DAPO and when evaluated on the second backbone, supporting its usefulness beyond a single training configuration.
The experimental study includes relevant exploration-oriented baselines and ablations that separate several components of the method. Comparing prefix-conditioned generation with historical response replay, coverage-aware selection with uniform selection, and the proposed credit assignment with standard response-level advantages helps assess what each component contributes. In particular, the uniform-selection comparison provides evidence that coverage-aware allocation improves average mathematical performance beyond prefix reuse alone.
Critical weaknesses
The characterization of finite-sample gradient estimation as the root of collapse is stronger than the analysis supports. Appendix A.2 already demonstrates a concentration tendency under exact expected-gradient updates, while Appendix A.3 uses sampling noise to exclude convergence to equilibria supported on multiple successful responses. These are distinct contributions to the argument. Moreover, the KL-regularized optimum discussed in Section 3.1 and the subsequent unregularized dynamics concern different objectives. The main text should distinguish these roles more carefully rather than attributing the concentration mechanism to finite sampling alone.
The robustness argument also requires a more explicit connection to the earlier analysis. Earlier sections define semantic classes through generated responses to a fixed prompt, while Appendix A.4 treats classes as a policy-independent component of the data distribution. Without establishing how these notions correspond, the worst-case prior-shift result does not directly explain why preserving multiple solutions per prompt should produce the reported cross-task generalization benefits.
A prefix associated with a rare historical mode may lead the current policy to a different mode, including a dominant one. The paper does not directly quantify how often selected prefixes yield new successful responses in their intended modes. The initialization case study and aggregate diversity results leave this aspect of the proposed mechanism unresolved.
Both tree construction and diversity evaluation also rely on the same embedding model. Although the semantic-tree case study provides qualitative examples of interpretable clusters, the paper does not report a systematic independent assessment of whether the evaluation clusters correspond to meaningfully different valid solution strategies. The CMIMC25 failure case described in Appendix D.4 illustrates that irrelevant text can inflate apparent diversity despite verifier acceptance. Differences in successful sample counts may further affect the entropy estimates. These issues limit how confidently higher measured entropy can be interpreted as preservation of distinct successful reasoning strategies.
Questions for authors
Could the authors revise the causal interpretation to distinguish the concentration tendency already present under expected-gradient updates from the role of sampling noise in excluding non-singleton equilibria? Please also clarify how the KL-regularized optimum in Section 3.1 relates to the subsequent unregularized analysis, and which aspects of the observed collapse this comparison is intended to explain.
How is the policy-independent variable (C) in Appendix A.4 related to the response-dependent mapping (\psi_x(y)) when a prompt admits multiple correct solution strategies? What connection is established, rather than hypothesized, between Theorem 2 and the empirical semantic-entropy and cross-task results?
Using existing rollout records, could the authors report how often continuations remain in the selected historical mode, switch to a dominant mode, or enter another mode, distinguishing short and long prefixes where possible? An independently assessed sample of these transitions, together with a comparison controlling successful sample counts, would help determine whether the diversity gains reflect distinct valid reasoning strategies rather than clustering or sampling effects.
Dear Reviewer,
As the rebuttal period is approaching its end, we would like to follow up on our previous responses. We have carefully considered the questions raised in your review and provided detailed responses to each of them.
We look forward to further feedback from all reviewers, as it will help us improve the quality of our work. If you have any additional questions or concerns, we would be happy to address them.
Best regards,
The Authors of Paper 17645