- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents TailSFT, which drops, at each step, the sequences whose loss has fallen most relative to the initial policy, to preserve coverage (large-k pass@k) for later RL. Support: a toy graph task, a stylized theorem, and OLMo-3 7B math/code experiments where pass@16 improves in most settings and GRPO from TailSFT checkpoints gains 1–4 points of pass@1 in all five comparisons.
Critical strengths
- The GRPO follow-through is the real evidence: TailSFT initializations beat standard SFT after RL in all five matched settings, even where pre-RL pass@1 was lower.
- Thorough reporting (three seeds, paired CIs, full grids).
Critical weaknesses
- The main confound is uncontrolled: TailSFT discards a quarter to half of the gradients and uses a lower learning rate, so it may simply be "less SFT" where standard SFT over-fits. No fewer-epochs, early-stopping or random-drop baseline is reported.
- The defining choice, filtering relative to the initial policy, is never ablated on LMs, and in the toy task it is slightly worse than plain quantile filtering. Math configurations are selected per benchmark and math gains are small.
- Unlike traditional meta-learning algorithms such as Adaboost, the choices here is rather intuitively motivated and thus lack theoretical rigor. In particular, the notion of "coverage" is very loosely defined based on SFT objective (i.e. LM cross-entropy) which does not necessarily have clear mechanistic association with any possible downstream RL processes--what if, a different heuristic of "coverage" is used, e.g. ranking temperature instead of cross-entropy (i.e. uncertainty regardless of correctness)?
Questions for authors
- Could you add an ablation where standard SFT is carried out with matched effective sample count at the same learning rate?
- Could you ablate quantile filtering vs the offset criterion on one code and one math setting?
- Could you report math with a single pre-registered filter configuration?
- Does the RL advantage persist with more rollouts and longer training?
We appreciate your identification of the central result: “The GRPO follow-through is the real evidence.” On OLMo-3 7B, all five main-table comparisons improve post-GRPO pass@1, including the code settings where TailSFT starts with lower pre-RL pass@1.
Could TailSFT’s gains come simply from avoiding overfitting? We provide evidence to the contrary. TailSFT retains coverage gains at matched learning rates (math, OCI, and Tables 8–9), under early stopping checkpoint selection (83.4% vs. 82.0% pass@16 on OMI→MATH-500), and after multiple epochs. Algorithm 1 also normalizes by the retained target-token count, so filtering changes which examples drive the update rather than simply scaling down the gradient.
Does filtering relative to the initial model help beyond current-loss filtering or random dropping? Our new comparison matches learning rates, update budgets, and drop counts across several variants: TailSFT achieves 70.90% pass@16, versus 62.69% for current-loss quantiles and 68.66% for random dropping. Our theorem also establishes why TailSFT is the most direct way to improve coverage out of all of these variants.
What formally justifies the filtering rule, and how is coverage relevant to RL? Our coverage definition follows the formal framework of The Coverage Principle (ICLR 2025), which connects coverage to RL sample complexity. Coverage, as we define it, is not an ad hoc uncertainty heuristic. Our new theorem shows why measuring improvement relative to the initial model helps preserve correct responses during SFT, providing a formal justification for TailSFT’s filtering rule. Coverage concerns access to correct responses, unlike uncertainty, which can be high even when every response is incorrect.
Do the gains persist with more rollouts and longer RL training? The submitted appendix reports similar gains with 8 rollouts and an additional GRPO configuration, indicating that the gains are fairly general. Whether the advantage persists over substantially longer RL horizons remains an open question.