acceptodds
Under review as a conference paper at ICLR 2027

Selecting Diverse SFT Traces Improves Post-RL Generalization

Abstract

Verified solutions are not equally useful for preparing reasoning models for reinforcement learning (RL). We present a comprehensive study of route diversity, the variation in the sequences of reasoning steps in supervised fine-tuning (SFT) data, and propose a lightweight, rule-based fingerprint to select for it. From one pool at one budget, with matched training recipes and checkpoints, selecting diverse rather than similar routes improves post-RL problem coverage across puzzles and mathematics, including on problems harder than those seen in either training stage. In synthetic experiments, route-diverse SFT improves OLMo3-7B's pass@8 by 16.9 points on environments held out from SFT. In a single-model condition, where one model writes every candidate, diverse selection gains up to 6.2 points of mean pass@8 across 10 mathematics benchmarks. Pre-RL diagnostics suggest why: diverse SFT can produce both successful and failed attempts on more prompts despite slightly lower mean accuracy, giving group-relative RL more prompts with a learning signal. On 3 open-source corpora, our CPU-only selector, without model calls, outperforms more expensive alteratives in every comparison of mean post-RL performance. These results identify reasoning-route diversity as a practical criterion for selecting SFT data that better prepares models for RL.

Primary area:
foundation or frontier models, including LLMs
Keywords:
interplay between SFT stage and RL, reasoning exploration, empirical study on SFT-data

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_eFyGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether the diversity of reasoning routes in SFT data affects generalization after subsequent RL. The authors propose a rule-based fingerprint that represents the structure of a reasoning trace and use it to select route-diverse solutions from a fixed candidate pool and SFT budget. The selected checkpoints are then trained with the same RL recipe. Across synthetic reasoning environments, mathematics, executable tasks, and three released reasoning corpora, route-diverse SFT generally improves post-RL coverage and pass@k. The paper also shows that the effect remains when all candidate traces are generated by a single model. Pre-RL analysis finds that diverse SFT produces mixed rewards on more prompts, which the authors propose as one reason it provides a better starting point for group-relative RL.

Critical strengths
  1. The empirical study is broad and well controlled. The paper evaluates the hypothesis across multiple student models, SFT budgets, task domains, OOD settings, and released reasoning corpora. Most comparisons match the student initialization, SFT budget, RL recipe, evaluation setup, and checkpoint. The single-model experiment is particularly useful because it shows that the gain does not simply come from mixing different teacher models.

  2. The proposed selector is simple and practical. On OpenThoughts3, INTELLECT-3, and Nemotron-Cascade 2, the proposed selection method outperforms random, topology, gradient-diversity, embedding, and lexical baselines in mean post-RL performance. It also avoids model inference during selection and scales to million-example pools with much lower computational cost than the model-based alternatives.

Critical weaknesses
  1. The incremental contribution over the closest prior work is not fully established. The paper itself identifies prior work showing that divergent reasoning paths can improve reasoning and remain beneficial after RL, especially Ju et al. (2025), which also studies divergent solutions under a fixed demonstration budget. The main additional contributions here appear to be the scalable rule-based selector and a much broader empirical study, but there is no direct comparison with the closest prior selection method. This makes it difficult to judge how much of the contribution is a new empirical finding versus a more scalable implementation and validation of an existing idea.

  2. The experiments do not completely isolate route diversity from other changes in the selected SFT data. In the direct selection experiments, Diverse and Similar are selected globally under domain-level budgets rather than using exactly the same prompts with the same number of trajectories per prompt. Thus the two datasets may also differ in prompt coverage and other correlated trace properties. The paper provides useful controls, including the single-model setting and some fixed-per-prompt teacher-source experiments, but these do not fully isolate the effect of the proposed route-diversity selector itself. A matched within-prompt comparison would make the causal attribution much cleaner.

Questions for authors
  1. How does the proposed selector compare directly with the closest prior route-diversity selection method, especially Ju et al. (2025), under a comparable candidate pool and SFT/RL budget?
  2. Does the route-diversity advantage remain when Diverse and Similar use exactly the same prompts and the same number of trajectories per prompt?
4 comments

Thank you for recognizing the breadth of our study and the value of the single-teacher control. We appreciate your suggestions for clarifying the contribution and testing the selection effect directly.


TL;DR.

Our contribution is a careful study of how the structure of demonstrated reasoning affects post-RL generalization. We strengthen the evidence through the two comparisons you requested:

  • Comparison with Ju et al. 2025. Our selector achieves higher post-RL pass@8 on all three released corpora, with gains of 4.09–7.14 percentage points, using CPU-only selection without model inference. This shows our approach is both more efficient and more effective, in selecting SFT reasoning traces for RL.

  • Per-prompt control. Our original paired comparisons already match the questions and retained trajectories per question, except where explicitly identified as non-per-prompt-controlled. We clarify that protocol below and provide an additional matched experiment. The separately identified practical comparisons test the selector on released data without requiring large candidate pools for every question.


W1 / Q1. Contribution and direct comparison with Ju et al.

W1 excerpt. The incremental contribution over the closest prior work is not fully established.

Q1. How does the proposed selector compare directly with the closest prior route-diversity selection method, especially Ju et al. (2025), under a comparable candidate pool and SFT/RL budget?

Our selector achieves higher post-RL pass@8 than RPD on all three corpora. Thank you for recommending this direct comparison.

DatasetMethodavg@1 (%)pass@8 (%)
ot3RPD37.3556.78
Ours38.3760.87
Difference+1.01 (p = 0.22)+4.09 (p = 0.022)
int3RPD31.9551.39
Ours33.0756.55
Difference+1.12 (p = 0.13)+5.16 (p = 0.004)
casc2RPD32.6653.55
Ours34.7360.69
Difference+2.07 (p = 0.007)+7.14 (p = 0.001)

ot3, int3 and casc2 denote OpenThoughts3, INTELLECT-3 and Nemotron-Cascade 2. Differences are Ours minus RPD in percentage points.

The pass@8 gains are 4.09, 5.16 and 7.14 points, respectively. Avg@1 point estimates also favor our method on each corpus. These results establish a post-RL performance benefit alongside the selector's computational practicality.

Our contribution is a careful study of how the structure of demonstrated reasoning affects post-RL generalization. We build on the prior diversity results by testing which correct solutions to retain, then measuring the successful strategies that the trained policy actually produces. The controlled comparisons hold question coverage and demonstration counts fixed. The single-teacher setting further shows that the selection benefit can arise within one model's candidate pool.

The selector makes this criterion practical at million-answer scale. Our full-pool selection takes approximately 2.6–2.9 hours on one CPU node, with zero model calls and zero GPU-hours. We also provide RPD total time costs.

ResourceOpenThoughts3INTELLECT-3Nemotron-Cascade 2
Answers in the full pool456,1862,174,1672,142,332
RPD model calls0.85 million4.10 million4.13 million
RPD input tokens5.4 billion26.7 billion39.7 billion
RPD GPU-hoursabout 135about 700about 1,070
Ours model calls / GPU-hours0 / 00 / 00 / 0
Ours CPU-node wall timeabout 2.6 hoursabout 2.9 hoursabout 2.9 hours

Our method therefore combines stronger post-RL results with selection that needs no model-based annotation pass. We will include the performance comparison in Section 4 and the cost breakdown in Appendix C.4, with the reproduction settings. We will also include the generated-strategy analysis in the revised paper.

W2 excerpt. The experiments do not completely isolate route diversity from other changes in the selected SFT data.

Q2. Does the route-diversity advantage remain when Diverse and Similar use exactly the same prompts and the same number of trajectories per prompt?

Yes. Per-prompt matching was already the default in our original study: unless a comparison is explicitly identified as non-per-prompt-controlled, both conditions use exactly the same training questions and retain the same number of solutions for each question. This applies to the controlled direct-selection comparisons, not only the teacher-source experiments. Thank you for highlighting the need to make this protocol clearer. We will state the default matching rule in the main text and label its exceptions alongside the corresponding results.

The original controlled results already show the benefit. Figure 5 compares Diverse and Similar within a single teacher's candidate pool, matching the retained questions and their solution counts. Diverse improves mean post-RL pass@8 by 3.39–6.17 points across three SFT budgets. Figure 3a provides complementary teacher-source evidence with twelve retained solutions for every question in both conditions (35.2% versus 28.0% post-RL pass@64). We will show how the per-question constraints enter the selection procedure.

We additionally tested the same selection question with a much larger pool of alternatives per prompt, using Qwen3-1.7B-Base as the student model for SFT followed by RL. For each of 1,000 fixed RLVE questions, we constructed 1,000 correct candidate responses. Diverse and Similar each retain exactly eight or 256 responses per question. We compare 8,000 versus 8,000 training trajectories, then 256,000 versus 256,000. Each pair fixes the training questions and their multiplicities, including the question-defined task and difficulty composition.

Traces per questionSelectionID pass@32 after SFT (%)ID pass@32 after RL (%)OOD pass@64 after SFT (%)OOD pass@64 after RL (%)
8Diverse25.7325.6714.0014.81
8Similar25.2423.5111.5613.00
256Diverse29.5041.1012.6318.25
256Similar29.5934.7211.5617.50

ID denotes in-domain evaluation; OOD denotes out-of-distribution evaluation. All scores are percentages.

Diverse leads after RL on both evaluation splits at both matched budgets. The in-domain gains are 2.16 points at eight traces and 6.38 points at 256; the corresponding OOD gains are 1.81 and 0.75 points. At 256 traces, in-domain SFT scores start nearly equal (29.50% versus 29.59%) and reach 41.10% versus 34.72% after RL. This additional experiment strengthens the original finding that the retained solutions matter at fixed question coverage and demonstration counts.

The explicitly identified non-per-prompt-controlled comparisons test practical selection from available data. Released corpora do not provide a uniformly large set of correct alternatives for every question. In those practical comparisons, we use the released responses directly under matched total trajectory budgets and domain quotas. Generating many extra solutions would change the available corpus and add a data-generation stage. Restricting the experiment to response-rich questions would change the population on which the selector is tested. These comparisons therefore evaluate the usefulness of the full selection method under existing data availability. The controlled comparisons establish the fixed-coverage finding.

The Qwen3-4B 50K pair behind Table 3 is a separately disclosed pooled-selection exception (Appendix A.4). Exceptions are identified locally; per-prompt matching remains the default for the other paired comparisons. The 1,000-response pools in the additional experiment enable a detailed controlled test, but are not an input requirement of our selector.

We will include the complete additional-experiment table in the revised paper and make the default protocol and its exceptions explicit. The accompanying accounting will document the existing question-level constraints and report actual SFT token exposure separately from trajectory counts.

Thank you again for these constructive suggestions. The direct RPD comparison strengthens the method's empirical positioning. The original per-prompt controls and additional matched experiment support the contribution of retained-solution choice to post-RL generalization.

Thank you for the additional experiments and clarifications. I appreciate the authors’ efforts to address the concerns raised in my review. After carefully considering the new evidence, I do not find sufficient reason to revise my initial assessment, and therefore maintain my original score.

Dear Reviewer eFyG,

Thank you for your positive assessment of our work and suggesting & carefully reviewing additional evidence! Your suggestions have helped us strengthen the paper.

We have now incorporated the direct RPD comparison, the additional matched per-prompt experiment, and the protocol clarifications into the revised manuscript. These additions provide the requested comparison with the closest prior method and test whether the selection advantage persists with identical questions and trajectories per question.

We would welcome the opportunity to clarify any remaining concerns and further strengthen our work!

Thank you very much again for your time and constructive feedback!

Reviewer_kd8bOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies which verified solutions to keep for SFT, when it is followed by GRPO-style RLVR under a fixed budget. The author proposes a rule-based and model-free, named “topology fingerprint” of reasoning steps and selects route, which means diverse traces via cluster-proportional farthest-point sampling. Against a nearest-centroid “Similar” selection from the same pool, diverse selection improves post-RL coverage. Examples include +16.9 pp pass@8 on SFT-unseen RLVE environments of OLMo3-7B model and +3.4–6.2 pp in a single-teacher pool. On three released corpora, the selector overpass random, gradient, embedding and lexical baselines by 1.2–10.8%.

Critical strengths

1.The question is practically important, and the study uses a matched protocol: same pool, budget, student, recipe and RL checkpoint.

2.The evaluation is broad: puzzles, math, OMEGA OOD, Sokoban and released corpora. Use multiple students in SFT-unseen environments and harder difficulty levels.

3.The selector is CPU-only, more than 2M candidates, no model calls, so it is cheap and scalable, easy to adopt.

Critical weaknesses

1.Diversity is confounded with prompt coverage. RLVE selection has no environment quota. Table 3 shows Similar has nearly twice as many prompts with ≥2 solutions, so the number of unique prompts differs. Unique prompts, task/difficulty distribution and token counts are not reported, so the gains may come from coverage or data volume rather than route diversity.

2.The fingerprint and selector are not validated. Step labels come from lexical rules, and the two selections have identical pattern entropy and pattern counts. With 40K clusters for 100K rows, the method is close to stratified sampling. The baselines use pure farthest-point sampling (gradient-FPS is worse than random), which confounds feature space with selection rule.

3.The statistics and the RL claim are weak.

  • Every condition is a single run.
  • RL is very short (30–75 steps).
  • Readout steps are inconsistent: step 8/16 vs. 64.
  • No pre-RL/post-RL curves are reported, so “better prepares for RL” is not separated from “better SFT endpoint”.
Questions for authors

1.Please report unique prompts, task/difficulty distribution and total SFT tokens for each selection.

2.Please report full RL curves with pre-RL scores. And why do readout steps differ across comparisons?

3.Can you ablate the selector (cluster allocation vs. within-cluster FPS) and apply the same rule to embedding and gradient features?

4.Can policy entropy explain the higher mixed-reward share? For example, does sampling the Similar checkpoint at a higher temperature reproduce it?

9 comments
AuthorsOpenReview ↗

Response to Reviewer kd8b (Part 1/6)

Thank you for the careful review and the concrete suggestions. We completed additional experiments to test the coverage, representation, and RL explanations you raised.

TL;DR

  • W1/Q1: We ran an exact question-matched comparison. Diverse and Similar use the same questions and the same number of demonstrations per question. Diverse achieves higher post-RL performance on both ID and OOD evaluations at both tested budgets.
  • W2: We validated the fingerprint and the trained models' strategies. Fingerprint distance correlates with annotated path differences in both settings. The diverse-trained model also generates more distinct strategies among equal numbers of correct answers, including when answer lengths are matched.
  • W2/Q3: We tested the representation and the within-cluster selection rule separately. Under the same selector, our fingerprint improves overall mean pass@8 by 5.10 over embeddings and 3.98 over gradient-surrogate features, with lower feature-construction cost. In a separate fixed-cluster, fixed-quota analysis, our within-cluster rule produces less redundant selections and better fingerprint-space pool coverage than stratified random sampling in all three domains.
  • W3/Q2: We compared SFT and RL endpoints, extended training, and repeated runs. The matched experiment gives a 6.38-point ID advantage after RL from nearly identical SFT scores. Countdown results favor Diverse at 600 RL steps from both initializations. Repeated-run means favor Diverse at every reported difficulty and sampling budget.
  • Q4: We ran the proposed temperature intervention. Increasing Similar's temperature did not recover Different's higher mixed-reward share at any tested higher temperature.

The revision will include these results with their protocols. Full learning curves, data/token accounting, and the cluster-occupancy breakdown remain agreed additions. The fixed-cluster selection analysis is now complete.

W1 / Q1. Controlling question coverage and demonstration counts

W1. Diversity is confounded with prompt coverage. RLVE selection has no environment quota.

Q1. Please report unique prompts, task/difficulty distribution and total SFT tokens for each selection.

We ran an additional comparison with identical questions and retained counts per question. Diverse leads after RL on both evaluation splits at both budgets. This directly tests whether the benefit remains when question coverage and demonstration multiplicity are fixed.

Existing controls

Our original paired comparisons use per-prompt matching unless explicitly identified otherwise. Figure 5 also fixes the teacher and reports 3.39-6.17-point improvements in mean post-RL pass@8 across three SFT budgets. Figure 3a provides a complementary teacher-source control with exactly twelve retained solutions per question (Appendix B). We will state the matching constraint explicitly in the selection procedure.

Additional matched-question experiment

We trained Qwen3-1.7B-Base using the same 1,000 RLVE questions, each with 1,000 correct candidate responses. Diverse and Similar retain exactly eight or 256 responses per question. Within each budget, the datasets therefore contain the same 8,000 or 256,000 SFT demonstrations. Exact question-level matching also fixes the question-defined environment and difficulty composition.

Traces per questionSelectionID pass@32 after SFTID pass@32 after RLOOD pass@64 after SFTOOD pass@64 after RL
8Diverse25.7325.6714.0014.81
8Similar25.2423.5111.5613.00
256Diverse29.5041.1012.6318.25
256Similar29.5934.7211.5617.50

Scores are percentages. Compare Diverse with Similar within each trace budget.

Result. Diverse's post-RL ID gains are 2.16 and 6.38 percentage points at the two budgets. Its OOD gains are 1.81 and 0.75 points. The retained questions and their demonstration counts are identical within each comparison.

The one-trace condition remains a contextual reference. The diversity comparison uses equal trace budgets. The large candidate set enables this controlled experiment, while the selector itself can operate on existing candidate pools.

AuthorsOpenReview ↗

Response to Reviewer kd8b (Part 2/6)

W1 / Q1 (continued): data accounting

Table 3 and the requested data accounting

We will clarify the scope of Table 3. Its counts identify questions with at least two retained solutions, which qualify for the within-question spread analysis. Appendix A.4 identifies the Qwen3-4B 50K pair in Figure 19 as the pooled-selection comparison used for that analysis. We will label this exception alongside the question-matched comparisons.

Revision. We will include the new result table and the requested accounting for the final SFT inputs. The accounting will report exact question overlap and retained multiplicities, together with environment-difficulty distributions. It will also report loss-bearing tokens after preprocessing. Equal demonstration counts establish the coverage control, while the token totals document actual supervision volume.

W2. Validating the fingerprint and generated strategies

The fingerprint and selector are not validated. Step labels come from lexical rules, and the two selections have identical pattern entropy and pattern counts.

We added annotated-path validation and a direct check of generated strategies. Fingerprint distance agrees with the annotated path measures in both settings. The diverse-trained model also produces more distinct strategies among correct answers.

1. We compared fingerprint distance with annotated path differences

We sampled correct solutions to the same problem across the fingerprint-distance ranking. An annotator labeled each solution separately using one-sentence definitions of the step types. We compared the resulting outlines with a five-point path-difference rating, sequence edit distance, and step-transition distance. The analysis uses complete traces.

Distance fifthPairs (% of total)Mean rating (1-5)Rating >=3Transition distanceEdit distance
1 (nearest)20.6%2.87 (ref.)77.1% (ref.)0.629 (ref.)0.540 (ref.)
220.6%+8.0%87.1% (+10.0 pts)+7.6%+8.7%
319.9%+11.1%90.1% (+13.0 pts)+11.1%+14.3%
419.8%+13.9%93.7% (+16.6 pts)+13.0%+15.7%
5 (farthest)19.1%+17.8%93.8% (+16.7 pts)+14.9%+18.1%

Pair shares are fractions of all analyzed complete-trace pairs in the multi-model pool. The nearest fifth supplies the reference values. Later mean-rating and distance entries give relative changes from that reference. Rating >=3 is the fraction within each fifth. Parenthesized changes are percentage points.

Result. The fraction rated at least 3 rises from 77.1% in the nearest fifth to 93.8% in the farthest, an increase of 16.7 percentage points. Larger fingerprint distances therefore correspond to larger annotated path differences.

We repeated the validation on RLVE with its eighteen step types and examined agreement across trace-length thirds.

SettingAnnotated-path measureOverall [95% interval]ShortMiddleLong
Multi-modelPath-difference rating0.30 [0.25, 0.35]0.370.200.31
Multi-modelEdit distance0.26 [0.21, 0.31]0.330.240.22
Multi-modelTransition distance0.19 [0.14, 0.25]0.200.150.22
RLVEPath-difference rating0.21 [0.16, 0.27]0.160.180.30
RLVEEdit distance0.17 [0.12, 0.21]0.240.080.18
RLVETransition distance0.16 [0.11, 0.21]0.210.070.19

Entries are within-problem rank correlations with fingerprint distance. Brackets contain 95% intervals. Short, Middle, and Long denote the reported trace-length thirds. Correlations remain coefficients, not percentages.

Agreement is positive in both settings and across all reported length groups. The longest-third rating correlations are 0.31 for the multi-model pool and 0.30 for RLVE. These associations support the fingerprint as a procedural-distance proxy. The annotations share the step vocabulary and label each solution separately.

AuthorsOpenReview ↗

Response to Reviewer kd8b (Part 3/6)

W2 (continued): generated strategies

2. We measured distinct strategies in the trained models' correct answers

We compared the post-RL OLMo3-7B policies on every eligible RLVE question where both produced at least four correct answers in 32 attempts. Each model contributed four correct answers. A model-blinded rater grouped each set by solution strategy while ignoring wording and length.

Result. Diverse averages 1.45 strategies per question, compared with 1.33 for Similar. The difference is 0.12 [95% interval: 0.08, 0.17]. The eight-answer comparison with matched answer lengths also favors Diverse, by 0.20 [0.06, 0.35]. A second rater gives the same direction.

Equal correct-answer counts make this a comparison of the strategies within successful outputs. It provides a behavioral check beyond Figure 7's lexical measure.

3. We will distinguish pattern occurrence from route structure

Pattern counts and entropy summarize which coarse patterns occur and how evenly they occur. Transition and path features describe how the steps are arranged (Appendix A.2). Equal pattern summaries can therefore coexist with different route structure.

Revision. We will include the path-validation tables and Table R5 with their annotation protocols. We will also clarify the role of Table 3's pattern summaries alongside the new evidence.

W2 / Q3. Isolating the representation and within-cluster selection

W2 excerpt. The baselines use pure farthest-point sampling (gradient-FPS is worse than random), which confounds feature space with selection rule.

Q3. Can you ablate the selector (cluster allocation vs. within-cluster FPS) and apply the same rule to embedding and gradient features?

We completed both requested same-rule representation comparisons and a separate fixed-cluster selection analysis. Our fingerprint improves overall mean post-RL pass@8 by 5.10 over embedding features and 3.98 over gradient-surrogate features. With the clusters and quotas fixed, our within-cluster rule also improves both measured properties of the selected data over stratified random sampling.

1. We replaced the feature vector and kept the selector unchanged

Selection componentOURSOURS-embOURS-grad
Per-solution vectorRule-based route-topology fingerprint, 1,738 dimensionsQwen3-Embedding-8B embedding, 4,096 dimensionsOlmo-3-7B output-head gradient surrogate, 4,096 dimensions
Feature preprocessingStandardize; clip at +/-5; unit-normalize; Gaussian projection to 96 dimensions, seed 42; unit-normalizeSame recipeSame recipe
SelectorEven math/code/science budgets; k-means with 40,000 clusters; cluster-size-proportional quotas; within-cluster FPS; seed 42Same code, unchangedSame code, unchanged
Pool and budgetOpenThoughts3: 456,186 candidates -> 100,000 selected solutionsSameSame

Both ablations retain 33,334 math, 33,333 code, and 33,333 science solutions. All representations are projected to the same 96-dimensional selection space.

2. The representation advantage remains after RL

At the common RL-step-32 checkpoint, the overall mean pass@8 gains are 5.10 over OURS-emb and 3.98 over OURS-grad. Our fingerprint also achieves higher pass@8 on each evaluated benchmark against both alternatives.

ComparisonΔ pass@8, benchmark mean [95% CI], pΔ avg@1, benchmark mean, pΔ pass@8, pooled, pΔ avg@1, pooled, p
OURS − OURS-emb+5.10 [+1.54, +8.75], 0.005+2.32, <0.001+2.18, 0.003+1.49, <0.001
OURS − OURS-grad+3.98 [+0.98, +7.10], 0.011+1.65, 0.016+1.79, 0.012+1.59, <0.001

Differences are OURS minus the named ablation. "Benchmark mean" weights benchmarks equally. "Pooled" weights evaluation questions equally.

Result. Our fingerprint improves both pass@8 and avg@1 under benchmark-mean and pooled aggregation. Because these comparisons use the same selection code, they support a contribution from the representation within the shared pipeline. We will include the full protocol and benchmark-level results in the revision.

3. The fingerprint also has lower feature-construction cost

OURS-emb runs Qwen3-Embedding-8B over the candidate solutions. OURS-grad uses Olmo-3-7B to construct an output-head gradient surrogate. Our fingerprint is computed directly from trace text on CPUs, with no model inference.

Result. The higher-performing representation also avoids the alternatives' model-based extraction cost. We will report feature construction separately from the common selection stages. The gradient comparison uses the stated output-head surrogate.

Reviewer_4eW5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines whether selecting varied correct reasoning traces for SFT improves performance after RL. It proposes a rule-based fingerprint for selecting traces and tests the approach across synthetic tasks, mathematics, and released reasoning corpora.

Critical strengths

I found the paper clear and well executed. The authors test the idea across a wide range of tasks and corpora, and the shared-pool and single-teacher experiments make the empirical result convincing.

Critical weaknesses
  1. The main idea feels somewhat expected.

The main concern I have is that the conceptual contribution is somewhat limited. Prior work has already shown that selecting divergent solutions at a fixed SFT budget can yield gains that persist after RL. This paper more thoroughly demonstrated the effect through a more extensive and controlled evaluation and used an inexpensive selector, but the fundamental insight is not particularly new.

  1. The interpretation of fingerprint diversity remains uncertain.

Table 3 reports larger within-problem distances between selected SFT traces in the proposed feature space for one RLVE setting. However, these distances are not an independent measure of distinct solution strategies. For example, the fingerprint can also reflect wording or how a solution is presented. Additionally, the method works on several corpora, but its rules are partly domain-specific, and their reliability on new domains and longer traces remains unclear.

  1. The longer-term RL effect is unclear.

The RLVR training steps used in various comparison and results experiments are generally too short. The impact of increasing the training steps in comparisons and experiments needs to be further examined (especially, comparison on Diverse-Similar).

Questions for authors
  1. For correct solutions to the same problem, how well does fingerprint distance align with human- or LLM-judged strategy differences? Does this agreement change with trace length?

  2. How much would the labeling rules need to change for a new domain? Beyond the lexical diversity in Figure 7, do the trained models generate more semantically distinct solution strategies?

  3. Could the authors provide longer RL learning curves? Do the gains persist?

  4. The paper used a base model as the student model, and while I understand the intention, I am simply curious that when performing the same RL after applying an additional SFT to an instruction-tuned reasoning model, would selecting various tracking paths still be helpful?

8 comments
AuthorsOpenReview ↗

Response 1/4

Thank you for the thoughtful review and for recognizing the value of the shared-pool and single-teacher experiments. Your questions prompted direct checks of what our fingerprint measures and what the trained models generate.

TL;DR.

Contribution (W1). We carefully study reasoning topology: how the demonstrated search proceeds, rather than just which answer it reaches. Diverse outperforms Similar after RL on both in-domain and OOD evaluations at both tested budgets, with questions and demonstration counts fixed. Our selector also improves on RPD. The reward diagnostic identifies an initialization property that mean accuracy does not capture.

Validity and behavior (W2/Q1-Q2). Fingerprint distance tracks separately annotated path differences in both tested settings, including long traces. More directly, the trained model generates more distinct successful strategies among equal-sized sets of correct answers. This result remains positive with length matching and on questions both models solve frequently. We clarify which parts of the representation need domain adaptation.

Longer RL and initialization (W3/Q3-Q4). We completed controlled Countdown comparisons through 600 RL steps. Diverse leads at the endpoint on pass@8 and pass@32 across all three evaluation splits, from both base and instruction-tuned initialization. We will add the complete curves before the rebuttal deadline. Further experiments and supporting data will follow in the camera-ready version if accepted.

W1. What the study adds beyond earlier diversity results

The main idea feels somewhat expected.

Prior work has already shown that selecting divergent solutions at a fixed SFT budget can yield gains that persist after RL.

Our contribution is a careful study of how the structure of demonstrated reasoning affects post-RL generalization. We build on the prior diversity results by testing which correct solutions to retain, then measuring the successful strategies that the trained policy actually produces.

The choice of solutions matters at fixed question coverage. In the additional RLVE experiment, every condition uses the same 1,000 questions, each with 1,000 correct candidate responses. We train Qwen3-1.7B-Base using the same training set-up. Within each budget, Diverse and Similar retain exactly the same number of solutions per question.

Matched-question RLVE results (%). ID uses pass@32; OOD uses pass@64.

SelectionID after SFTID after RLOOD after SFTOOD after RL
Diverse 825.7325.6714.0014.81
Similar 825.2423.5111.5613.00
Diverse 25629.5041.1012.6318.25
Similar 25629.5934.7211.5617.50

Diverse outperforms Similar after RL on both ID and OOD evaluations at both matched budgets. At 256 traces per question, Diverse starts slightly lower on ID pass@32 after SFT but finishes higher after RL. Its post-RL advantage therefore does not simply reflect a higher SFT-only score on this metric.

The direct RPD comparison establishes practical value beyond selection cost.

CorpusMethodavg@1 (%)pass@8 (%)
OpenThoughts3RPD37.3556.78
Ours38.3760.87
Difference+1.01 (p=.22)+4.09 (p=.022)
INTELLECT-3RPD31.9551.39
Ours33.0756.55
Difference+1.12 (p=.13)+5.16 (p=.004)
Cascade-2RPD32.6653.55
Ours34.7360.69
Difference+2.07 (p=.007)+7.14 (p=.001)

Differences are percentage points. Pass@8 improves on every corpus while selection requires no additional model inference. We will include the reproduction settings and statistical procedure with this comparison.

Mean accuracy does not fully describe the learning opportunities available to RL. In the pre-RL diagnostic (Section 3.4), Diverse produces mixed rewards on 54.7% of 64 mathematics prompts versus 46.9% for Similar, despite slightly lower mean accuracy. These groups provide nonzero outcome-based advantages under the group-relative objective. This supports a proposed explanation for the effect: SFT can change where RL receives a learning signal, not only its starting accuracy.

The annotations in Q1 test the procedural meaning of the fingerprint. Q2 then tests the trained models' successful strategies directly. We will foreground these findings in the contribution statement and include the new tables in the paper.

AuthorsOpenReview ↗

response 2/4

The interpretation of fingerprint diversity remains uncertain.

For correct solutions to the same problem, how well does fingerprint distance align with human- or LLM-judged strategy differences? Does this agreement change with trace length?

Thank you for suggesting this direct check. Larger fingerprint distances correspond to greater differences in separately annotated reasoning paths. The association appears in both settings, including their longest-trace groups.

We followed your suggestion to compare against annotated paths rather than measure spread only in our own feature space. For 250 multi-model problems, we sampled 2,500 pairs of correct solutions to the same problem across within-problem fingerprint distances. An annotator labeled each solution separately using one-sentence definitions of the ten step types. We compared fingerprint distance with a 1-5 path-difference rating based on the two outlines. Sequence edit distance and step-transition distance provide complementary checks.

Judged path differences increase across the fingerprint-distance ranking.

Distance fifthPairs (% of total)Mean rating (1-5)Rating >=3Transition distanceEdit distance
1 (nearest)20.6%2.87 (ref.)77.1% (ref.)0.629 (ref.)0.540 (ref.)
220.6%+8.0%87.1% (+10.0 pts)+7.6%+8.7%
319.9%+11.1%90.1% (+13.0 pts)+11.1%+14.3%
419.8%+13.9%93.7% (+16.6 pts)+13.0%+15.7%
5 (farthest)19.1%+17.8%93.8% (+16.7 pts)+14.9%+18.1%

All three measures increase monotonically across these sampled fifths. This supports using the fingerprint to rank procedural differences among correct solutions to the same question.

The association is not confined to short traces. We repeated the check on 2,000 pairs from 200 RLVE questions with its eighteen step types. The table reports within-problem rank correlations; short, middle and long denote the reported trace-length thirds.

SettingAnnotated-path measureOverall [95% interval]ShortMiddleLong
Multi-modelPath-difference rating0.30 [0.25, 0.35]0.370.200.31
Multi-modelEdit distance0.26 [0.21, 0.31]0.330.240.22
Multi-modelTransition distance0.19 [0.14, 0.25]0.200.150.22
RLVEPath-difference rating0.21 [0.16, 0.27]0.160.180.30
RLVEEdit distance0.17 [0.12, 0.21]0.240.080.18
RLVETransition distance0.16 [0.11, 0.21]0.210.070.19

All six overall intervals exclude zero. Every reported length-group estimate is positive. These results support the fingerprint as a proxy for annotated procedural differences, including on long traces. Q2 complements this candidate-pool validation with direct judgments of the trained models' generated strategies.

We will include the results and annotation rubric in the revision. We will document complete-trace eligibility and the sampling protocol, with the uncertainty calculations alongside them.

AuthorsOpenReview ↗

Response 3/4

Q2. Domain adaptation and distinct generated strategies

How much would the labeling rules need to change for a new domain?

The selection procedure is shared; adapting a new domain primarily concerns how its traces are mapped to events and features. The three released reasoning corpora use the same ten step types, also used for the Dolci-Think and single-teacher pools. A separate vocabulary is not required for each corpus.

The existing domain-specific implementations illustrate the changes. RLVE uses eighteen step types. OMEGA reads recorded step annotations and strategy labels; Sokoban instead uses move and board-path signatures. Appendix A.2 and Table 2 specify how each dataset instantiates the shared procedure, including feature blocks and preprocessing choices.

For a new domain, the practical task is to define the event mapping and check that it captures the relevant procedural distinctions. We will make these adaptation points explicit and add a worked real-data example. The validation in Q1 covers two existing vocabularies; a new vocabulary should receive its own check rather than be assumed reliable.

Beyond the lexical diversity in Figure 7, do the trained models generate more semantically distinct solution strategies?

Yes. The diverse-trained model generates more distinct successful strategies even when both models contribute the same number of correct answers to the same question. This directly tests the requested behavior rather than using lexical spread as its substitute.

For every RLVE question where both post-RL OLMo3-7B policies produced at least four correct answers out of 32 samples, we drew four correct answers from each. A rater grouped each set by solution strategy while ignoring wording and length, without knowing which model produced it. The additional eight-answer comparisons probe sensitivity to sample size, answer length and solve rate.

Mean distinct strategies per question, OLMo3-7B after RL. Relative gain is Diverse over Similar.

Correct-answer sampleQuestions (% of eligible)DiverseSimilarRelative gain95% interval
4; all eligible100.0%1.451.33+9.0%[+6.0%, +12.8%]
844.0%1.711.53+11.1%[+5.9%, +17.0%]
8; lengths within 15%14.3%1.851.65+12.1%[+3.6%, +21.2%]
8; both correct >=16/3224.0%1.621.44+12.5%[+6.3%, +19.4%]

The difference remains positive with length matching and among questions both policies solve frequently. Equal-sized correct-answer sets prevent the strategy count from increasing merely because one model supplies more successful samples. Questions solved only by Diverse do not enter this comparison. A second rater gives the same direction.

On 300 shared questions, a separate path-difference rating among correct answers also favors Diverse by 0.16 [0.11, 0.21]. Together, these checks connect the selected supervision to differences in the trained model's successful reasoning, beyond Figure 7's wording measure.

We will include the table in the revised paper, with the strategy-grouping instructions and rater details. The protocol will specify subset construction and the confidence-interval calculation.

Reviewer_rvtZOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the effect of diverse SFT traces on post-RL generalization. Basically if we want the model to excel during the RL stage, we need to feed correct solutions that solves the problems in structurally different ways during SFT. Diversity in this study is defined by the topology of a solution which measures route diversity. Massive gains on unseen and harder problems in new domain (RLVE) were shown after RL when trained with traces with higher diversity, and a simple way to achieve diversity is to mix different teachers.

Critical strengths
  • It presents very useful insight on how much improvement one can expect from simply filtering SFT data for diversity
  • I like how simple the diversity heuristic is and the process is rule-based without needing any GPU.
  • The multi-teacher diversity is an expected results but showing that we can achieve same level of diversity from a single teacher by using the method proposed here is great

Overall I appreciate the simplicity and effectiveness of the data method proposed in this paper. I also appreciate the thorough experiments and analysis done in this work.

Critical weaknesses

The main weakness i found in this paper is somehow the writing (particularly in Section 3) a bit hard to follow. It describes the entire method in high-level prose, and from a reader's perspective i read it several times still couldn't grasp the method until digging into Appendix. Also the toy figure in Fig 4 does not represent the pipeline on real data. I would greatly appreciate if the author could describe the actual pipeline more clearly in the main text.

Questions for authors

N/A

1 comment

Thank you for your supportive assessment and for recognizing the simplicity of the selector, the single-teacher result, and the breadth of the experiments! We sincerely appreciate the suggestion you provided to help improving the presentation of our paper! We will work towards enhancing the clarity and readability in our next revision!

Thank you again for the constructive guidance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.