acceptodds
Under review as a conference paper at ICLR 2027

From Checkpoint Variation to Selection Gains in Supervised Fine-Tuning

Abstract

Checkpoint selection is a routine decision in supervised fine-tuning (SFT): training produces multiple checkpoints, but only one is retained. Yet fixed-budget comparisons do not by themselves distinguish three empirical claims: whether more validation data improve checkpoint selection, whether a selection rule outperforms validation-loss selection, and whether it improves over simply retaining the final checkpoint. We therefore treat checkpoint selection as a finite-information decision problem: the decision must be made using limited validation data. Holding completed training trajectories, candidate checkpoints, and independent test items fixed, we vary the validation budget and separately measure improvement from additional validation data, gain over negative log-likelihood (NLL) selection, and gain over the final checkpoint. Across 60 mathematical SFT trajectories and 19 configurations, increasing the validation budget from 32 to 305–313 examples raises independent-test accuracy by percentage points (pp) for generated-accuracy selection and pp for checkpoint agreement, with 95% configuration-bootstrap CIs of and , respectively. At the full validation budget, the two generation-based rules outperform matched NLL selection by and pp, respectively, yet their gains over the final checkpoint remain unresolved. A cross-domain replication on 12 newly trained Commonsense trajectories, with its protocol frozen before training and checkpoint choices frozen before independent-test evaluation, shows the same qualitative separation: increasing the validation budget from 32 to 1,024 questions improves generated-accuracy and checkpoint-agreement selection by and pp, while their gains over the final checkpoint again remain unresolved. Together, these results show that benefiting from more validation data, outperforming NLL selection, and outperforming the final checkpoint are distinct empirical claims that require separate evidence. **Anonymous code:** https://anonymous.4open.science/r/checkpoint-selection-reproduction-C02F/

Primary area:
foundation or frontier models, including LLMs
Keywords:
checkpoint selection, supervised fine-tuning, validation budget, model selection, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_G5ZkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how to pick a checkpoint after a supervised fine-tuning run has finished. It does not propose a new selector. Instead it separates three questions: does more validation data help, does a selector beat validation NLL, and does it beat just keeping the final checkpoint. It measures all three on math and commonsense fine-tunes across several models and validation budgets, with paired comparisons and a prospective replication. The main finding is that more validation data gives a small gain for generation-based selection, but this mostly closes the gap to the final checkpoint rather than beating it.

Critical strengths

The paper provides empirical evidence that beating validation-NLL selection does not establish a benefit over simply keeping the final checkpoint. Its main experiments show that this distinction matters in practice: additional validation improves selection, yet the resulting selectors still have no established advantage over the final checkpoint.

Critical weaknesses
  • The main practical question stays unresolved. The gain over keeping the final checkpoint is not established. The one positive finding, that more validation data helps generation-based selection, is small. It mostly just brings selection up to the free final-checkpoint baseline. So the paper's own motivation, whether extra validation effort is worth it, comes out close to "not clearly."

  • The strongest confirmatory result uses four configurations from two base models. The main grid is nonfactorial and shares datasets and model families, and some older runs have incomplete provenance. The contribution is empirical, so this limits how far the finding reaches. Overall the paper offers limited explanation of why the final checkpoint is so competitive or when this pattern changes.

Questions for authors
  1. Do the central results survive if you drop the imported runs with incomplete train/validation provenance and keep only corroborated ones?
  2. How often do the selectors pick the final checkpoint or its neighbors, and how often do low-budget scores tie? How much does a deterministic latest-on-tie rule change the result?
  3. In the Commonsense replication, how are the 32 validation examples sampled across the eight tasks, and how many subsets are averaged?
Reviewer_cE2kOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies SFT checkpoint selection under limited validation data. Across mathematical experiments and a new Commonsense replication, larger validation budgets improve generation-based selection and outperform NLL selection, while gains over the final checkpoint remain inconclusive.

Critical strengths
  1. The paired design clearly separates validation-budget gains, gains over NLL, and gains over the final checkpoint.

  2. Newly trained Commonsense trajectories and targeted sensitivity analyses strengthen the empirical evidence.

Critical weaknesses
  1. The study does not distinguish limited genuine headroom over the final checkpoint from failure to identify better checkpoints using finite validation data, limiting its explanatory contribution.

  2. Appendix C.2 reports incomplete run-level provenance for 18 of 81 trajectories, but the paper does not establish whether its main findings remain robust after excluding unverifiable runs.

Questions for authors
  1. Which incompletely documented trajectories enter the main 60-trajectory analysis, and do the conclusions hold after excluding them?

  2. Can an independent analysis distinguish limited headroom from unreliable identification of better checkpoints?

Reviewer_dHNZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies checkpoint selection after supervised fine-tuning. It varies validation-set size while keeping training trajectories, candidate checkpoints, and test items fixed. It compares validation accuracy, checkpoint agreement, and NLL selection against retaining the final checkpoint. More validation data improve generation-based selection, and these rules outperform NLL selection, but gains over the final checkpoint remain inconclusive. A separate commonsense replication supports this distinction.

Critical strengths
  1. The paper makes a useful distinction between benefiting from more validation data, beating NLL selection, and beating the final checkpoint. The results show why these claims should be tested separately. This is relevant to everyday SFT evaluation, even without a new selection algorithm.

  2. The experimental controls are careful. Selection and test items are separated, comparisons are paired, and repeated subsets are not treated as independent runs. The paper includes multiple seeds, sensitivity checks, and a commonsense replication whose protocol was fixed before training. It also correctly treats confidence intervals crossing zero as inconclusive.

Critical weaknesses
  1. The dependence on the training endpoint needs clearer analysis. Most runs use about 2,700 training examples, mainly LoRA, with 249-756 optimization steps. Longer runs and completed-decay controls are included, but it remains unclear how selection gains depend on whether task accuracy is still rising, has plateaued, or has declined near the final checkpoint. This limits the practical guidance on when checkpoint selection is worth doing.

  2. Limited novelty and explanatory insight. The experiments carefully quantify familiar model-selection issues, including finite-validation noise and same-item optimism. The more interesting result is that NLL selection underperforms the final checkpoint, but the paper does not explain why this happens or when checkpoint selection becomes useful. Differences across configurations are mainly descriptive. Without connecting these differences to training progress or overfitting, the paper provides limited new understanding or practical guidance.

Questions for authors
  1. What explains the gap between NLL-selected and final checkpoints? Can the existing shorter and longer runs show when selection helps, especially as task accuracy approaches a plateau or starts to decline?

  2. Do the main findings remain when restricting the analysis to trajectories with verified training-validation splits, excluding runs whose historical source data or split membership cannot be established?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.