acceptodds
Under review as a conference paper at ICLR 2027

TailSFT: Filtered Fine-Tuning Improves Post-Training Performance

Abstract

Reinforcement learning post-training drives reasoning and agentic capabilities in modern AI systems, yet a growing body of work shows that it is most effective when used to fine-tune an already capable base model. We question whether existing pipelines yield models that are most suitable for reinforcement learning. Building on prior work highlighting the role of coverage and pass@K as predictors of post-RL performance, we design a simple modification to supervised fine-tuning, TailSFT, which filters out already fit sequences during training, thereby focusing learning on under-modeled regions, or the tail, of the data distribution. We justify and validate the design choices in TailSFT, particularly the specific filtering criteria, through a combination of controlled experiments and theoretical analysis. On OLMo-3 7B, TailSFT improves pass@16 in 15 of 18 math and coding comparisons, with gains up to 16.8 percentage points, while incurring minimal computational overhead. Initializing GRPO from TailSFT checkpoints yields up to 3.9 percentage points of pass@1 improvement in the reported comparisons, consistent with the hypothesis that broader coverage supports subsequent RL. We further introduce a lightweight diagnostic for identifying settings where TailSFT is most likely to help. More broadly, our results motivate a principled, stage-aware approach to model development, in which intermediate checkpoints are judged by how effectively they support subsequent training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post-training, reinforcement learning, language modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uQ4bOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TailSFT, which drops, at each step, the sequences whose loss has fallen most relative to the initial policy, to preserve coverage (large-k pass@k) for later RL. Support: a toy graph task, a stylized theorem, and OLMo-3 7B math/code experiments where pass@16 improves in most settings and GRPO from TailSFT checkpoints gains 1–4 points of pass@1 in all five comparisons.

Critical strengths
  • The GRPO follow-through is the real evidence: TailSFT initializations beat standard SFT after RL in all five matched settings, even where pre-RL pass@1 was lower.
  • Thorough reporting (three seeds, paired CIs, full grids).
Critical weaknesses
  • The main confound is uncontrolled: TailSFT discards a quarter to half of the gradients and uses a lower learning rate, so it may simply be "less SFT" where standard SFT over-fits. No fewer-epochs, early-stopping or random-drop baseline is reported.
  • The defining choice, filtering relative to the initial policy, is never ablated on LMs, and in the toy task it is slightly worse than plain quantile filtering. Math configurations are selected per benchmark and math gains are small.
  • Unlike traditional meta-learning algorithms such as Adaboost, the choices here is rather intuitively motivated and thus lack theoretical rigor. In particular, the notion of "coverage" is very loosely defined based on SFT objective (i.e. LM cross-entropy) which does not necessarily have clear mechanistic association with any possible downstream RL processes--what if, a different heuristic of "coverage" is used, e.g. ranking temperature instead of cross-entropy (i.e. uncertainty regardless of correctness)?
Questions for authors
  1. Could you add an ablation where standard SFT is carried out with matched effective sample count at the same learning rate?
  2. Could you ablate quantile filtering vs the offset criterion on one code and one math setting?
  3. Could you report math with a single pre-registered filter configuration?
  4. Does the RL advantage persist with more rollouts and longer training?
1 comment

We appreciate your identification of the central result: “The GRPO follow-through is the real evidence.” On OLMo-3 7B, all five main-table comparisons improve post-GRPO pass@1, including the code settings where TailSFT starts with lower pre-RL pass@1.

Could TailSFT’s gains come simply from avoiding overfitting? We provide evidence to the contrary. TailSFT retains coverage gains at matched learning rates (math, OCI, and Tables 8–9), under early stopping checkpoint selection (83.4% vs. 82.0% pass@16 on OMI→MATH-500), and after multiple epochs. Algorithm 1 also normalizes by the retained target-token count, so filtering changes which examples drive the update rather than simply scaling down the gradient.

Does filtering relative to the initial model help beyond current-loss filtering or random dropping? Our new comparison matches learning rates, update budgets, and drop counts across several variants: TailSFT achieves 70.90% pass@16, versus 62.69% for current-loss quantiles and 68.66% for random dropping. Our theorem also establishes why TailSFT is the most direct way to improve coverage out of all of these variants.

What formally justifies the filtering rule, and how is coverage relevant to RL? Our coverage definition follows the formal framework of The Coverage Principle (ICLR 2025), which connects coverage to RL sample complexity. Coverage, as we define it, is not an ad hoc uncertainty heuristic. Our new theorem shows why measuring improvement relative to the initial model helps preserve correct responses during SFT, providing a formal justification for TailSFT’s filtering rule. Coverage concerns access to correct responses, unlike uncertainty, which can be high even when every response is incorrect.

Do the gains persist with more rollouts and longer RL training? The submitted appendix reports similar gains with 8 rollouts and an additional GRPO configuration, indicating that the gains are fairly general. Whether the advantage persists over substantially longer RL horizons remains an open question.

Reviewer_wxJgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1(−1)
Presentation
2
Soundness
1
Read review
Paper summary

This work observes that greedily, individually optimizing each stage of post-training: SFT and RLVR could lead to ultimately sub-optimal models due to the tension between minimizing cross-entropy loss for SFT and maximizing coverage for RL. They propose TailSFT, which is a tweak to normal SFT that filters out samples that are already well-fit and instead focuses on the less well-fit samples in order to improve coverage for subsequent RL.

Critical strengths
  • I appreciate the simplicity of TailSFT. This approach is easy to understand and quite intuitive: focus learning effort on where the model is weakest rather than where it is already improving/strong.
  • I appreciate the theoretical analysis and the smaller experiments used to establish the tension between cross-entropy and coverage.
Critical weaknesses
  • The choice to go with Figure 1 doesn't make much sense to me. First, you qualify it in the caption with "Illustration of the hypothesis that broader SFT coverage supports subsequent RL" and "Curves are illustrative," but if this method indeed works, why not just show us the actual results as opposed to an illustrative diagram? Second, it's AI-generated (I verified it's OpenAI SynthID), and this wasn't disclosed in the AI Use statement. This is really unfortunate because, aside from this, your paper looks decent, but now I doubt its integrity. What is the point of generating a graphical figure with AI? Why not give your numbers to the AI agent and ask it to make a graph in LaTeX or Matplotlib?
  • I think the decision not to include more models (beyond OLMo-3 7B) significantly weakens your findings for two reasons. First, we don't know how your findings interact with scale. If we have a smaller or bigger model from the same family, does that change anything? Second, do different model families behave differently? The fact that your findings are only confirmed on one model means that they could be a fluke.
  • A brief note on my clear rejection rating: I think that in this form, this paper should definitely be rejected. The idea itself seems solid, intuitive, and worth exploring, but the execution is lacking. Specifically, experimenting on a single model, especially one that isn't even popular, does not inspire confidence that this approach is at all generalizable or can be useful to the community. If the authors are able to convincingly remedy this within the rebuttal period, I am open to increasing my score.
Questions for authors
  • Why OLMo-3 7B? It seems like an arbitrary choice to me. See also my second weakness.
  • Why are we focusing on the ones that have improved rather than the ones that are already strong? I imagine a case where a model has improved a lot, but is still generally weak. Shouldn't we want to learn more still in that case?
1 comment

We thank the reviewer for recognizing that our idea is "solid" and "intuitive," and for noting that the theoretical analysis and small-scale experiments clearly demonstrate the promise of the algorithm.

To address concerns about the breadth of our experiments, we have added substantial new experiments on Llama-3.1-8B base. At matched learning rates, TailSFT improves pass@16 in 8 of 15 code comparisons, including +6.79 points on Magicoder–CruxEval-O, +3.76 on Magicoder–LiveCodeBench, and +3.22 on OCI–MBPP+. All eight tested math filtering configurations improve mean AIME pass@16. We hope you will increase your score to account for these additional experiments.

Figure 1 and the experimental evidence. Figure 1 is a conceptual schematic, explicitly labeled “Curves are illustrative” in the original caption. We have now disclosed its generation with ChatGPT in both the caption and AI-use statement. This disclosure issue does not affect the experimental evidence, which is reported separately in tables and plots programmatically generated from measured results.

Why OLMo? Our initial experiments found that fine-tuning Qwen on math data reduced training loss but degraded benchmark performance, consistent with reports of SFT-induced degradation and forgetting in this model family (Karan et al., 2026; Wang et al., 2025; Jin et al., 2025; Chen et al., 2025). This motivated us to use OLMo, whose openly documented training data and checkpoints let us inspect prior exposure to related data and better understand the effects of additional SFT. OLMo is actually a quite popular model with many monthly downloads, and we are not sure how to address the reviewer's subjective claim that it is not. We have also added Llama-3.1-8B experiments, where TailSFT improves pass@16 in 13 of 17 dataset–benchmark pairs, corroborating its utility beyond OLMo.

Why filter by relative improvement rather than absolute current loss? A response that is rare can have a dramatic probability increase but still have a high absolute loss. Relative filtering instead prioritizes responses that have improved least, which our theory connects directly to undercoverage. Under expert conditioning, our new theorem guarantees that the relative-quantile optimum has coverage no worse than current-loss quantiles at the same retention rate. Our newly added experiment further supports our algorithm design: TailSFT achieves 70.90% pass@16 versus 62.69% for current-loss quantiles.

Karan, Aayush, Sitan Chen, and Yilun Du. 2026. Finetuning with Sampling: SFT Learns Better Than You Think. arXiv:2610.02140.

Wang, Yubo, Xiang Yue, and Wenhu Chen. 2025. Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate. COLM 2025.

Jin, Hangzhan, Sitao Luan, Tianwei Ni, Sicheng Lyu, Guillaume Rabusseau, Reihaneh Rabbany, Doina Precup, and Mohammad Hamdaqa. 2025. RL Fine-Tuning Heals OOD Forgetting in SFT. arXiv:2509.12235.

Chen, Feng, Allan Raventós, Nan Cheng, Surya Ganguli, and Shaul Druckmann. 2025. Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning. NeurIPS 2025.

Reviewer_ZiMAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TailSFT, which filters training sequences according to their loss reduction relative to the initial model. It aims to preserve useful response coverage during SFT and improve subsequent reinforcement learning. The paper presents a graph-navigation study, a theoretical analysis under expert conditioning, and math and code experiments with OLMo-3 7B.

Critical strengths

It aims to preserve useful response coverage A theoretical analysis under expert conditioning

Critical weaknesses
  • Without comparisons to several closely related methods, the specific contributions cannot be clearly established. [1] already uses reference-relative sequence selection, and [2] includes online filtering for CoT SFT.
  • Simpler absolute and current-loss quantile filters are not evaluated in the language-model SFT-to-GRPO experiments.
  • Algorithm 1 uses minibatch quantile filtering, and the paper does not establish that it inherits the theorem’s coverage advantage.
  • The BigCode and Magicoder post-GRPO comparisons use initializations trained with different SFT learning rates. [1] Thirukovalluru et al. Sequence Reducible Holdout Loss for Language Model Pretraining. LREC-COLING 2024. [2] Chen et al. Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning. NeurIPS 2025.
Questions for authors
  • Does TailSFT outperform current-loss quantile filtering when both retain the same fraction of sequences?
  • What connects the fixed-threshold optimization in Theorem 3.1 to the minibatch quantile rule in Algorithm 1? What evidence shows that the practical rule preserves the coverage advantage established in the theorem?
  • Do the post-GRPO gains on BigCode or Magicoder persist when Standard SFT and TailSFT use the same SFT learning rate?
3 comments

Thank you for your review and questions. The theoretical connection between coverage and TailSFT has been strengthened in our revision, and we discuss the baselines further below.

Relationship to S-RHOL and confidence limiting. Both cited papers were discussed in the original submission and, in short, require a substantial amount of additional compute compared to TailSFT, which is why we did not include them in our paper.

  • S-RHOL selects sequences using losses relative to an auxiliary model trained on held-out data, and it is focused on improving pretraining efficiency, not SFT performance. We would need an alternative pre-trained model to serve as the auxiliary model. Thirukovalluru et al., 2024.
  • Applying DCO directly to complete reasoning traces essentially amounts to ordinary SFT: their gradient multiplier approaches one when (Kp\ll1), where (p) is the probability of the entire trace. The authors acknowledge this issue. Their second method, DCOa, instead estimates answer probabilities through online sampling. This requires substantially more compute: for their 64-sample configuration, the authors estimate sampling costs roughly 10x the compute of a forward/backward pass. Chen et al., 2025.

How does the theory relate to TailSFT? Our updated theorem directly analyzes quantile filtering: under expert conditioning, for every retained fraction (\rho\in(0,1]) and coverage scale (N>1), the offset-quantile optimum has coverage deficit no greater than either Standard SFT or current-loss quantile filtering. Intuitively, the expert preserves the initial model’s relative probabilities among rewarding responses, so rare and common responses are equally well covered when their probabilities increase by the same factor. Offset filtering measures this relative progress, whereas current-loss filtering can prioritize a rare response simply because its absolute probability remains low.

Could the gains come from using a lower SFT learning rate? The submitted appendix already contains same-learning-rate comparisons: at (2\times10^{-5}), TailSFT improves MBPP+ pass@16 by 2.12 points on BigCode and 3.53 points on Magicoder (Tables 8–9). These establish that their SFT coverage gains are not solely due to using a lower learning rate. The math and OCI post-GRPO comparisons already match the SFT learning rate and show positive pass@1 gains, so the reported downstream benefit is also present without that mismatch.

Thank you for your revisions. Several of my concerns have been addressed.

However, the benefit of TailSFT's specific filtering strategy for downstream RL, compared with simpler alternatives, remains insufficiently demonstrated. Its generalizability across model families also remains unclear.

Thank you for your quick response to our rebuttal. We request that you raise your score as we have addressed your concerns around improving the theory and providing additional experiments. Our theory now directly motivates TailSFT, and we also did demonstrate that TailSFT "outperforms current-loss quantile filtering when both retain the same fraction of sequences."

We would like to highlight our top-level comment which indeed demonstrates "generalizability across model families" by demonstrating that TailSFT is beneficial for the LLaMA-3.1-8B model, in addition to the OLMo-3 model that we ran our original experiments on. Our added experiments span several SFT datasets, many common benchmarks, and cover both the math and coding domains.

Due to the limited time period in the rebuttal, we are unable to perform the proper hyperparameter tuning to do the RL stage of the training pipeline as well. However, we note, as we did in our original submission, that extensive prior work has established that pre-RL pass@k predicts downstream RL outcomes (Kang et al., 2026; Wu et al., 2026; Sun et al., 2026), and that interventions preserving or expanding solution coverage, reasoning diversity, or exploratory behaviors can improve subsequent reinforcement learning (Cen et al., 2025; Dang et al., 2025; Fan et al., 2026; Wang et al., 2026; Xiang et al., 2026; Yao et al., 2026; Zhang et al., 2026).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.