- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether the diversity of reasoning routes in SFT data affects generalization after subsequent RL. The authors propose a rule-based fingerprint that represents the structure of a reasoning trace and use it to select route-diverse solutions from a fixed candidate pool and SFT budget. The selected checkpoints are then trained with the same RL recipe. Across synthetic reasoning environments, mathematics, executable tasks, and three released reasoning corpora, route-diverse SFT generally improves post-RL coverage and pass@k. The paper also shows that the effect remains when all candidate traces are generated by a single model. Pre-RL analysis finds that diverse SFT produces mixed rewards on more prompts, which the authors propose as one reason it provides a better starting point for group-relative RL.
Critical strengths
-
The empirical study is broad and well controlled. The paper evaluates the hypothesis across multiple student models, SFT budgets, task domains, OOD settings, and released reasoning corpora. Most comparisons match the student initialization, SFT budget, RL recipe, evaluation setup, and checkpoint. The single-model experiment is particularly useful because it shows that the gain does not simply come from mixing different teacher models.
-
The proposed selector is simple and practical. On OpenThoughts3, INTELLECT-3, and Nemotron-Cascade 2, the proposed selection method outperforms random, topology, gradient-diversity, embedding, and lexical baselines in mean post-RL performance. It also avoids model inference during selection and scales to million-example pools with much lower computational cost than the model-based alternatives.
Critical weaknesses
-
The incremental contribution over the closest prior work is not fully established. The paper itself identifies prior work showing that divergent reasoning paths can improve reasoning and remain beneficial after RL, especially Ju et al. (2025), which also studies divergent solutions under a fixed demonstration budget. The main additional contributions here appear to be the scalable rule-based selector and a much broader empirical study, but there is no direct comparison with the closest prior selection method. This makes it difficult to judge how much of the contribution is a new empirical finding versus a more scalable implementation and validation of an existing idea.
-
The experiments do not completely isolate route diversity from other changes in the selected SFT data. In the direct selection experiments, Diverse and Similar are selected globally under domain-level budgets rather than using exactly the same prompts with the same number of trajectories per prompt. Thus the two datasets may also differ in prompt coverage and other correlated trace properties. The paper provides useful controls, including the single-model setting and some fixed-per-prompt teacher-source experiments, but these do not fully isolate the effect of the proposed route-diversity selector itself. A matched within-prompt comparison would make the causal attribution much cleaner.
Questions for authors
- How does the proposed selector compare directly with the closest prior route-diversity selection method, especially Ju et al. (2025), under a comparable candidate pool and SFT/RL budget?
- Does the route-diversity advantage remain when Diverse and Similar use exactly the same prompts and the same number of trajectories per prompt?
4 comments
W2 excerpt. The experiments do not completely isolate route diversity from other changes in the selected SFT data.
Q2. Does the route-diversity advantage remain when Diverse and Similar use exactly the same prompts and the same number of trajectories per prompt?
Yes. Per-prompt matching was already the default in our original study: unless a comparison is explicitly identified as non-per-prompt-controlled, both conditions use exactly the same training questions and retain the same number of solutions for each question. This applies to the controlled direct-selection comparisons, not only the teacher-source experiments. Thank you for highlighting the need to make this protocol clearer. We will state the default matching rule in the main text and label its exceptions alongside the corresponding results.
The original controlled results already show the benefit. Figure 5 compares Diverse and Similar within a single teacher's candidate pool, matching the retained questions and their solution counts. Diverse improves mean post-RL pass@8 by 3.39–6.17 points across three SFT budgets. Figure 3a provides complementary teacher-source evidence with twelve retained solutions for every question in both conditions (35.2% versus 28.0% post-RL pass@64). We will show how the per-question constraints enter the selection procedure.
We additionally tested the same selection question with a much larger pool of alternatives per prompt, using Qwen3-1.7B-Base as the student model for SFT followed by RL. For each of 1,000 fixed RLVE questions, we constructed 1,000 correct candidate responses. Diverse and Similar each retain exactly eight or 256 responses per question. We compare 8,000 versus 8,000 training trajectories, then 256,000 versus 256,000. Each pair fixes the training questions and their multiplicities, including the question-defined task and difficulty composition.
| Traces per question | Selection | ID pass@32 after SFT (%) | ID pass@32 after RL (%) | OOD pass@64 after SFT (%) | OOD pass@64 after RL (%) |
|---|---|---|---|---|---|
| 8 | Diverse | 25.73 | 25.67 | 14.00 | 14.81 |
| 8 | Similar | 25.24 | 23.51 | 11.56 | 13.00 |
| 256 | Diverse | 29.50 | 41.10 | 12.63 | 18.25 |
| 256 | Similar | 29.59 | 34.72 | 11.56 | 17.50 |
ID denotes in-domain evaluation; OOD denotes out-of-distribution evaluation. All scores are percentages.
Diverse leads after RL on both evaluation splits at both matched budgets. The in-domain gains are 2.16 points at eight traces and 6.38 points at 256; the corresponding OOD gains are 1.81 and 0.75 points. At 256 traces, in-domain SFT scores start nearly equal (29.50% versus 29.59%) and reach 41.10% versus 34.72% after RL. This additional experiment strengthens the original finding that the retained solutions matter at fixed question coverage and demonstration counts.
The explicitly identified non-per-prompt-controlled comparisons test practical selection from available data. Released corpora do not provide a uniformly large set of correct alternatives for every question. In those practical comparisons, we use the released responses directly under matched total trajectory budgets and domain quotas. Generating many extra solutions would change the available corpus and add a data-generation stage. Restricting the experiment to response-rich questions would change the population on which the selector is tested. These comparisons therefore evaluate the usefulness of the full selection method under existing data availability. The controlled comparisons establish the fixed-coverage finding.
The Qwen3-4B 50K pair behind Table 3 is a separately disclosed pooled-selection exception (Appendix A.4). Exceptions are identified locally; per-prompt matching remains the default for the other paired comparisons. The 1,000-response pools in the additional experiment enable a detailed controlled test, but are not an input requirement of our selector.
We will include the complete additional-experiment table in the revised paper and make the default protocol and its exceptions explicit. The accompanying accounting will document the existing question-level constraints and report actual SFT token exposure separately from trajectory counts.
Thank you again for these constructive suggestions. The direct RPD comparison strengthens the method's empirical positioning. The original per-prompt controls and additional matched experiment support the contribution of retained-solution choice to post-RL generalization.
Thank you for the additional experiments and clarifications. I appreciate the authors’ efforts to address the concerns raised in my review. After carefully considering the new evidence, I do not find sufficient reason to revise my initial assessment, and therefore maintain my original score.
Dear Reviewer eFyG,
Thank you for your positive assessment of our work and suggesting & carefully reviewing additional evidence! Your suggestions have helped us strengthen the paper.
We have now incorporated the direct RPD comparison, the additional matched per-prompt experiment, and the protocol clarifications into the revised manuscript. These additions provide the requested comparison with the closest prior method and test whether the selection advantage persists with identical questions and trajectories per question.
We would welcome the opportunity to clarify any remaining concerns and further strengthen our work!
Thank you very much again for your time and constructive feedback!
Thank you for recognizing the breadth of our study and the value of the single-teacher control. We appreciate your suggestions for clarifying the contribution and testing the selection effect directly.
TL;DR.
Our contribution is a careful study of how the structure of demonstrated reasoning affects post-RL generalization. We strengthen the evidence through the two comparisons you requested:
Comparison with Ju et al. 2025. Our selector achieves higher post-RL pass@8 on all three released corpora, with gains of 4.09–7.14 percentage points, using CPU-only selection without model inference. This shows our approach is both more efficient and more effective, in selecting SFT reasoning traces for RL.
Per-prompt control. Our original paired comparisons already match the questions and retained trajectories per question, except where explicitly identified as non-per-prompt-controlled. We clarify that protocol below and provide an additional matched experiment. The separately identified practical comparisons test the selector on released data without requiring large candidate pools for every question.
W1 / Q1. Contribution and direct comparison with Ju et al.
Our selector achieves higher post-RL pass@8 than RPD on all three corpora. Thank you for recommending this direct comparison.
ot3, int3 and casc2 denote OpenThoughts3, INTELLECT-3 and Nemotron-Cascade 2. Differences are Ours minus RPD in percentage points.
The pass@8 gains are 4.09, 5.16 and 7.14 points, respectively. Avg@1 point estimates also favor our method on each corpus. These results establish a post-RL performance benefit alongside the selector's computational practicality.
Our contribution is a careful study of how the structure of demonstrated reasoning affects post-RL generalization. We build on the prior diversity results by testing which correct solutions to retain, then measuring the successful strategies that the trained policy actually produces. The controlled comparisons hold question coverage and demonstration counts fixed. The single-teacher setting further shows that the selection benefit can arise within one model's candidate pool.
The selector makes this criterion practical at million-answer scale. Our full-pool selection takes approximately 2.6–2.9 hours on one CPU node, with zero model calls and zero GPU-hours. We also provide RPD total time costs.
Our method therefore combines stronger post-RL results with selection that needs no model-based annotation pass. We will include the performance comparison in Section 4 and the cost breakdown in Appendix C.4, with the reproduction settings. We will also include the generated-strategy analysis in the revised paper.