- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies how to pick a checkpoint after a supervised fine-tuning run has finished. It does not propose a new selector. Instead it separates three questions: does more validation data help, does a selector beat validation NLL, and does it beat just keeping the final checkpoint. It measures all three on math and commonsense fine-tunes across several models and validation budgets, with paired comparisons and a prospective replication. The main finding is that more validation data gives a small gain for generation-based selection, but this mostly closes the gap to the final checkpoint rather than beating it.
Critical strengths
The paper provides empirical evidence that beating validation-NLL selection does not establish a benefit over simply keeping the final checkpoint. Its main experiments show that this distinction matters in practice: additional validation improves selection, yet the resulting selectors still have no established advantage over the final checkpoint.
Critical weaknesses
-
The main practical question stays unresolved. The gain over keeping the final checkpoint is not established. The one positive finding, that more validation data helps generation-based selection, is small. It mostly just brings selection up to the free final-checkpoint baseline. So the paper's own motivation, whether extra validation effort is worth it, comes out close to "not clearly."
-
The strongest confirmatory result uses four configurations from two base models. The main grid is nonfactorial and shares datasets and model families, and some older runs have incomplete provenance. The contribution is empirical, so this limits how far the finding reaches. Overall the paper offers limited explanation of why the final checkpoint is so competitive or when this pattern changes.
Questions for authors
- Do the central results survive if you drop the imported runs with incomplete train/validation provenance and keep only corroborated ones?
- How often do the selectors pick the final checkpoint or its neighbors, and how often do low-budget scores tie? How much does a deterministic latest-on-tie rule change the result?
- In the Commonsense replication, how are the 32 validation examples sampled across the eight tasks, and how many subsets are averaged?