acceptodds
Under review as a conference paper at ICLR 2027

Not Every Success Is Worth Imitating: Rethinking Data Selection for Reasoning

Abstract

Rejection-sampling supervised fine-tuning (RS-SFT) improves a language model by training on its own generated responses that reach the correct answer. Despite its simplicity and low cost, RS-SFT is commonly regarded as a weaker alternative to reinforcement learning (RL) for reasoning. We identify an overlooked limitation in its data selection: a response that eventually succeeds is not necessarily a desirable trajectory to imitate. Standard outcome-based selection checks only the final answer and can therefore admit responses that make failed attempts before reaching the correct answer or continue generating meaningless text after already answering. Under iterative self-training, these undesirable behaviors can be learned, regenerated, and selected again; in our experiments, post-answer degeneration among selected responses increases from 5.2% to 34.7%. Motivated by this observation, we propose Imitation-Aware Trajectory Selection (IMIT), a lightweight selection method that retains a correct response only if it contains exactly one final answer and terminates immediately after it. IMIT changes neither the training objective nor the training budget. On Qwen3-1.7B-Base and MATH-500, IMIT improves first-answer pass@1 over standard RS-SFT by 2.12 percentage points in one-round training and by 4.37 points under iterative self-training. Controlled experiments show that these gains arise primarily from which responses are selected, rather than from the reduced number of training problems. Despite its more selective training data, IMIT preserves solution diversity close to that of the base model and reaches 89.0% first-answer pass@32, compared with 86.4% for GRPO, although GRPO achieves substantially higher single-sample accuracy. Our results show that outcome correctness alone is insufficient for selecting self-generated responses for imitation, and suggest that the selection criterion is a critical component of effective self-training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, rejection sampling fine-tuning, self-training, data selection, supervised fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oUaiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

RS-SFT selects self-generated responses by final answer correctness alone, which admits flawed trajectories that make failed attempts or keep generating after answering, and iterative self-training amplifies these behaviors. IMIT fixes this by keeping a correct response only if it has exactly one final answer and stops immediately after it. On Qwen3-1.7B-Base and MATH-500, IMIT beats standard RS-SFT by 2.12 points in one round and 4.37 points iteratively, while preserving solution diversity close to the base model. Outcome correctness alone is therefore insufficient for selecting responses to imitate.

Critical strengths
  • The paper makes a compelling case that outcome-only selection amplifies undesirable behaviors across self-training rounds, and that last-answer evaluation can conceal this by rewarding within-sample retries. This is consequential because it re-examines a widely used pipeline and turns the observation into a concrete, actionable criterion.

  • IMIT operates purely at the data-selection stage and adds only a simple trajectory-level condition on top of the existing verifier, so it can be dropped into standard RS-SFT pipelines without extra models, process reward models, or additional compute.

Critical weaknesses
  • All evidence comes from a single small model on a single benchmark, and the central improvement is not robust under greedy decoding, where the authors report no significant difference. Combined with the acknowledged validity limits of the primary first-answer metric, the practical significance of the reported gains remains unclear.

  • IMIT only enforces a single-box, immediate-termination condition and explicitly does not verify reasoning. A crucial ablation is absent, namely whether simply truncating accepted responses at the first box before SFT would recover most of the gain. Without it, it remains unclear whether the selection framing adds anything beyond a simple preprocessing rule, which materially affects the assessment of novelty and significance.

Questions for authors
  • Could the authors report greedy-decoding results with confidence intervals for both settings, and clarify the practical interpretation of a gain that holds only under sampling?

  • What happens if, instead of rejecting multi-box or continued responses, the authors truncate each accepted RS-SFT response at its first \boxed{} before SFT? If this recovers most of IMIT's gain, the contribution reduces to a preprocessing trick. The authors should report this ablation.

  • The conservative analysis bounds rather than quantifies the intermediate-box bias. Could the authors provide a human- or model-verified estimate of the true rate on a sample, so the bias of the primary metric is measured rather than bounded?

  • Do the authors have preliminary results on a second model or benchmark showing the same iterative amplification trend?

  • IMIT discards a substantial share of otherwise-solvable problems simply because no sampled response satisfies its trajectory criterion. Would performance change if, under the same total generation budget, the authors kept those problems and continued sampling until a criterion-compliant response was found?

8 comments

Thank you for suggesting the first-box truncation experiment. This comparison strengthened the paper by showing how much of the early-pool gain a simple preprocessing step could recover. Verifying the truncated answer recovered nearly all of IMIT's early-pool first-box gain. The late-pool results depended on both the choice of training target and which answer was scored.

AuthorsOpenReview ↗

W1. What does the first-box gain establish in practice?

We found no reliable greedy gain on the submitted checkpoints. Crediting intermediate first boxes when the last answer was correct narrowed the sampled accuracy gap. With a fixed token budget on retrained late-pool models, final-answer voting favored IMIT over RS. Stopping alone scored similarly to IMIT. IMIT's tendency to stop after the answer also appeared on SVAMP. On the two 8B models, however, IMIT showed no clear last-box accuracy advantage over simpler rules. The clearest change is how the model presents its answer. A general reasoning gain remains unproven.

Revised manuscript. Appendix E.2, p. 23, lines 1221~1227 (voting). Appendix E.5, p. 25, lines 1298~1312 (8B models and SVAMP). Appendix E.6, p. 25, lines 1316~1327 (first-box audit, scoring sensitivity and greedy decoding).

AuthorsOpenReview ↗

W2. How much of the gain can simple truncation explain?

IMIT checks answer format and stopping. It does not verify each reasoning step. Truncation alone modestly raised sampled first-box accuracy and lowered last-box accuracy in both pools. Verifying the answer at the cut recovered nearly all of IMIT's early-pool first-box gain. IMIT retained a larger first-box lead in the late pool, where a stopping-only rule did best on last-box accuracy in the matched-question comparison.

Revised manuscript. Appendix E.3, p. 24, lines 1244~1247 (selection rules), Table 10, lines 1249~1256 (truncation results), and lines 1258~1263 (matched-question controls).

Reviewer_LEqZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies trajectory selection for rejection-sampling SFT. Its central observation is that a response with a correct final answer can still contain earlier failed attempts or unnecessary continuation, all of which receive positive supervision when the whole response is used for SFT. IMIT supplements outcome correctness with a simple rule: retain responses with exactly one boxed answer that terminate immediately afterward. Experiments on Qwen3-1.7B-Base and MATH-500 compare IMIT with outcome-only selection on shared generation pools. The main observed benefits are higher first-answer accuracy, shorter responses, and less degeneration.

Critical strengths
  1. The paper identifies a concrete issue and proposes an inexpensive intervention. The distinction between final-answer correctness and suitability for imitation is useful. The analysis shows that outcome-only selection admits increasing amounts of undesirable continuation during iterative RS-SFT: the proportion of selected responses with meaningless text after the answer rises from 5.2% to 34.7% across rounds. IMIT is straightforward to implement and does not require a learned process verifier or a new training objective. Its effect on output length and degeneration gives the results an observable behavioral interpretation.
  2. The matched-problem controls make the first-answer result more convincing. The authors restrict outcome-only selection to the same problems selected by IMIT, matching the training budget and repetition count. IMIT still improves first-answer pass@1 by 2.03 and 3.86 percentage points in the two settings (Table 4). This addresses the explanation that the gain is merely due to training on a smaller or different question set. The paper also reports both first- and last-answer results, paired uncertainty estimates, and limitations of its answer-reading rules.
Critical weaknesses

Critical Weaknesses

  1. The practical gain is limited when final-answer correctness is the objective. The first-answer improvements are real under the reported evaluation, but they do not translate into a consistent advantage under conventional last-answer pass@1. In the one-round setting, accuracy changes from 32.66% to 33.05%, with an interval for the difference that includes zero. In the iterative-pool setting, it decreases from 42.02% to 40.86% (Tables 2–3). The matched-problem control does show a positive one-round last-answer effect, but the iterative comparison does not show a significant improvement, so the evidence remains mixed. The output-length reduction is a genuine benefit: in the iterative setting, mean length falls from 656 to 430 tokens. However, I do not find the demonstrated accuracy–length trade-off sufficiently compelling by itself. If an application allows self-correction and evaluates the final answer, the main first-answer result establishes a narrower benefit than improved overall problem solving. The authors pre-specified first-answer accuracy and analyze intermediate-box misclassification, so my concern is about the practical significance of the chosen objective rather than the validity of reporting it. A comparison of final-answer success under matched inference-token budgets would help establish whether the efficiency benefit changes this assessment. The lack of a statistically significant improvement under greedy decoding further limits the robustness of the accuracy benefit.
  2. The iterative experiment does not evaluate an IMIT-driven self-training loop. Figure 2 applies IMIT to R1 and R4, which are generated by the outcome-only RS-SFT lineage, and then retrains from the base model. This is a reasonable controlled experiment for isolating trajectory selection on a shared pool. It does not evaluate what happens when IMIT is used at every generation–selection–training round. This missing experiment matters because the motivation concerns undesirable behavior accumulating through iterative self-training. Once IMIT changes a model, its next-round generations may differ in question coverage, trajectory quality, and the availability of acceptable solutions. The current experiment cannot establish whether repeated use of IMIT would sustain its gains or prevent degeneration over time. I consider this end-to-end validation important to the paper’s contribution.
Questions for authors
  • Does IMIT improve final-answer success when inference is compared at a matched total generation-token budget? Reporting that trade-off, rather than only a fixed number of samples, would help establish the practical value of the shorter outputs.

  • What happens in a complete multi-round self-training loop that applies IMIT at every round? A comparison with outcome-only RS-SFT under matched generation and training budgets, tracking coverage, degeneration, output length, and both answer metrics, would directly test whether IMIT addresses the iterative failure mode motivating the paper.

6 comments

Thank you for asking whether shorter responses improve the final answer when the model can spend its remaining tokens on another attempt. Your suggestion gave us a practical way to test when shorter responses help the user. On the submitted late-pool checkpoints, sampled last-box pass@1 was 42.02% for RS and 40.86% for IMIT. The greedy difference was −1.8 percentage points.

Revised manuscript. Appendix D.1, Table 2 (p. 18, lines 923~936), and E.6 (p. 25, lines 1325~1327).

AuthorsOpenReview ↗

W1 / Q1. Final-answer voting under a common output-token cap

We separately retrained models on the late pool. On each MATH-500 question, they generated up to 32 candidates within 2,048 output tokens and returned the most frequent final-answer string. Across two learner seeds, accuracy was 48.4% for RS, 53.9% for IMIT and 53.4% for STOP, which permits earlier boxes if the response ends at the last box. IMIT gained 5.5 pp over RS. Its difference from STOP was +0.5 pp.

IMIT generated 9.41 candidates per question versus 4.55 for RS. Repeated prompts used 2.06× as many input tokens for IMIT, and mean request time was similar. In a separate replay of 32 stored responses per question, accuracy at output caps of 4,096 and 8,192 tokens was 56.7% and 63.3% for RS, 59.8% and 63.3% for IMIT, and 62.0% and 66.8% for STOP. IMIT's advantage over RS narrowed and vanished at the largest cap. At 8,192 tokens IMIT had used all 32 stored responses in 37.7% of cases (STOP 14.0%, RS 4.2%), so the larger caps are not a clean equal-budget comparison.

Revised manuscript. Section 5.5 (p. 9, lines 440~442) and Appendix E.2 (p. 23, lines 1221~1227).

We also evaluated the fifth-round checkpoints from the independent lineages below under the 2,048-token cap. In this additional comparison for learner seed 43, IMIT reached 54.6% versus 52.6% for RS, so the advantage under this cap was uncertain for that lineage. At 4,096 and 8,192 tokens the differences were +0.6 and −2.2 pp.

AuthorsOpenReview ↗

W2 / Q2. Applying IMIT at every round

We also appreciate your request to follow the two methods across rounds. Starting from a common base-generated pool, each method generated its own subsequent pools. Each round used eight candidates per question on the same 6,488 questions and trained a fresh base model for 256 updates of 32 responses. Actual token counts differed. From the training logs, we additionally report that in round five RS supervised 3.01× as many tokens as IMIT for learner seed 43 and 2.77× for seed 44.

Each cell gives RS / IMIT, averaged over learners. Coverage counts questions contributing a training target. The lexical detector checks all generated responses, including rejected ones. Output tokens are per response.

Training poolQuestions covered (%)Responses flagged (%)Output tokens
R0 (shared)48.37 / 28.0121.03 / 21.03571 / 571
R165.14 / 65.8425.21 / 1.29549 / 254
R269.84 / 70.7637.26 / 1.53636 / 260
R373.70 / 73.3849.37 / 1.49748 / 273
R475.29 / 75.6158.14 / 0.95850 / 284

After five rounds, MATH-500 responses averaged 717 tokens for RS and 294 for IMIT. IMIT gained 6.58 pp in sampled first-box pass@1. Last-box pass@1 differed by +0.50 pp, with opposite signs across learners. Greedy last-box accuracy differed by −0.30 pp, and last-box pass@8 fell by 2.59 pp. A five-round STOP lineage would be needed to isolate the contribution of requiring exactly one box.

Revised manuscript. Appendix E.1 (p. 23, lines 1192~1217), including Table 9 (lines 1200~1208).

Reviewer_hXwsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines a failure mode of outcome-based rejection-sampling supervised fine-tuning (RS-SFT): a response may contain multiple answer attempts or degenerate continuation yet still be selected because its final answer is correct. The authors show that such trajectories become increasingly frequent across rounds of standard iterative self-training. They propose Imitation-Aware Trajectory Selection (IMIT), which keeps correct responses only when they contain exactly one boxed answer and end immediately afterward. On Qwen3-1.7B-Base and MATH-500, IMIT improves first-answer pass@1 by 2.12 and 4.37 percentage points in the one-round and iterative-pool settings, respectively. The paper provides matched-problem controls, bootstrap intervals, and analyses of answer retries, degeneration, and output diversity.

Critical strengths

S1. Concrete diagnosis of a self-training failure mode. The paper goes beyond reporting accuracy by tracking the content of selected trajectories across rounds. The increase in post-answer degeneration from 5.2% to 34.7% is a concrete observation that can inform future work on training-data selection.

S2. Useful controls for a simple intervention. The authors match candidate pools, use the same SFT optimization budget, control the training-problem set, and provide paired uncertainty estimates. These controls help show that the observed gain on first-answer accuracy is not merely a consequence of selecting fewer training problems.

Critical weaknesses

W1. The main improvement is strongly tied to a nonstandard, method-aligned evaluation metric. IMIT explicitly filters responses with multiple boxed answers, while its primary metric grades the first boxed answer. This makes the reported first-answer gain plausible but does not by itself establish better mathematical reasoning. Crucially, in the iterative setting conventional last-answer pass@1 decreases from 42.02% (RS-SFT) to 40.86% (IMIT), a statistically significant difference according to Appendix D.2. Under greedy decoding, neither setting shows a significant improvement. The paper therefore lacks convincing evidence of a robust gain under standard use conditions; the method may primarily teach a particular output convention rather than improve problem-solving ability.

W2. Limited empirical scope and incomplete evidence for the iterative claim. The core results use only one 1.7B base model and one mathematical benchmark. More importantly, the “iterative” IMIT model is trained after filtering R4, a pool generated by a standard RS-SFT lineage; IMIT does not govern the intermediate self-training rounds. The experiment demonstrates the value of filtering a late-stage pool, not that a fully iterative IMIT system avoids the alleged feedback loop. The paper also does not isolate the multiple-answer and post-answer conditions from simpler length, truncation, or answer-format filters, and it may reject legitimate self-correction.

Questions for authors

Q1. What happens when IMIT is applied at every generation-selection-training round, starting from R0? Please compare full IMIT and full standard RS-SFT lineages rather than only filtering the latter’s R4 pool.

Q2. Do the results persist for a larger or stronger instruction-tuned model, another reasoning dataset, and greedy or conventional last-answer evaluation? What is the intended deployment metric if the standard metric worsens?

Q3. Can the authors distinguish genuine multi-attempt correction from degenerate retry trajectories without assuming that every earlier boxed expression represents an undesirable final-answer attempt?

7 comments

Thank you for distinguishing selection on a shared pool from selection that shapes subsequent training data. Your question helped us explain more precisely what the original experiment could establish. The submitted R4 models used a fixed RS-generated pool. We now also compare five-round RS and IMIT lineages in which each learner generated its own subsequent pool.

Revised manuscript. Section 3.3, p. 5, lines 219~224 (fixed early and late pools).

AuthorsOpenReview ↗

Q1. What happens when IMIT selects the data at every round?

Both lineages began with the same base-generated pool. At each of five rounds, each branch selected its targets and trained a fresh student from the base model. For each of two learner seeds, we matched the number of updates and sampled eight responses per training question. Supervised token counts differed. This five-round comparison tests the full IMIT rule. The contribution of the single-box requirement was not isolated across rounds.

In the pool entering round five, RS and IMIT obtained a training target for 75.29% and 75.61% of questions, respectively. At this similar coverage, the lexical degeneration detector flagged 58.14% of all generated RS responses and 0.95% of IMIT responses. On MATH-500, the fifth-round IMIT models generated shorter responses on average (294 versus 717 tokens) and improved sampled first-box pass@1 by 6.58 percentage points over RS. The last-box pass@1 difference was small and uncertain (+0.50 points), while last-box pass@8 was 2.59 points lower.

Revised manuscript. Section 5.5, p. 9, lines 435~440. Appendix E.1, p. 23, lines 1192~1217, including Table 9, lines 1200~1208.

AuthorsOpenReview ↗

W1 / Q2. What does the first-box gain establish, and how do the results change under standard evaluation?

We agree that higher first-box accuracy alone does not establish better mathematical reasoning, especially because IMIT explicitly selects responses with a single boxed answer.
For a user who reads the model's final answer, we would report final-answer accuracy at the intended decoding budget. On the submitted 1.7B checkpoints, IMIT minus RS in sampled last-box pass@1 was +0.39 pp in R1 and −1.15 pp in R4. Neither pool showed a significant greedy gain. We separately retrained late-pool models and tested voting under a cap of 2,048 output tokens per question. Final-answer accuracy was 53.9% for IMIT, 48.4% for RS and 53.4% for stopping only. First-box accuracy measures accuracy when a user takes the first boxed output directly.

Revised manuscript. Section 6.1, p. 9, lines 467~468 (deployment metric). Appendix D.2, Table 3, p. 18, lines 941~951 (sampled differences). Appendix E.6, p. 25, lines 1325~1327 (greedy), and E.2, p. 23, lines 1221~1227 (voting).

On Qwen3-8B-Base, one round of IMIT yielded a 1.56~1.57 pp higher MATH-500 sampled last-box pass@1 than question-matched RS (the range reflects two unresolved verdicts). The stopping-only and verified-cut controls scored 51.76% and 51.38%, versus 51.23~51.24% for IMIT. On GSM8K, strict final-answer accuracy was 76.90% for IMIT, 73.60% for matched RS, 77.37% for stopping only and 78.99% for the untuned base. On Llama-3.1-8B-Instruct, MATH-500 accuracy was 33.08% for IMIT and 33.12% for matched RS. On Llama, all three comparisons of IMIT with matched RS, stopping only and verified first-box truncation had Holm-adjusted p = 1.00.

Revised manuscript. Appendix E.5, p. 25, lines 1298~1306 (Qwen3-8B, GSM8K and Llama-3.1-8B-Instruct).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.