- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper aims to predict the test-validation gap of a model at the end of post-training with early training dynamics, claiming that the test-validation gap at 50 epochs can be effectively predicted at epoch 4. It utilize roughness, how far the training losses sit from a smooth descent, and drift, how far the validation curve pulls away, as well as the ratio between them to characterize the extent of overfitting. The paper also shows that the checkpoints of distilled models are easier to overfit on downstream task.
Critical strengths
- The problem of predicting the future training results with early training dynamics is interesting.
Critical weaknesses
- The paper is poorly written and very hard to follow. Taking the abstract alone as an example:
- At Line 11 the paper says 'for every run', but in the next sentence it begins with 'The remaining pair', which is very confusing.
- Right after that sentence, at Line 12, the paper says 'Both threshold were locked in advance', but the previous context did not give any clue about even a single threshold.
- Right after that, at Line 13, the paper claims that 'The results uncover a clear pattern...' and begins to compare distilled model with standard instruction tuned models. However, as there are no additional settings or experiments in the previous context, if 'the results' refer to the previous 'thresholds were locked in advance', it gives no clue about the comparison between distilled and standard models.
- At Line 17, the paper says 'fine-tune like the twin', but doesn't specify what does 'twin' stand for and what does it mean to fine-tune like it.
- At Line 25 the paper says 'No single-epoch reading of the same curves is right at both.' but doesn't specify what is 'read' from single epoch, and what is 'right at both'.
- This is outside of abstract but super confusing. At Line 190 the paper mentions 'audit.py'. Also at Line 191, the paper says 'One number indexes; three numbers diagnose'. What is one number and three numbers here?
- The paper contains many basic mistakes. Taking the first few paragraphs of introduction as example:
- At Line 36, the paper says 'Fine-tune the wrong one and the annotation budget is gone, hardest where labels are most expensive'. It does not make sense because the author supposed we already hold the labeled data at Line 34, and there is no annotation cost after we have already obtained the data.
- At Line 37, the paper says 'Overfitting is normally legible only once a run is over', which is not true as we can always monitor the training and validation curve and don't have to wait until the end of run.
- At Line 38, the paper says 'A small fine-tune drives the training loss to zero, ...', which is not likely to be true, depending on the definition of 'small'. A few epochs of SFT usually would not drive the loss to zero and usually people don't 50 epochs of post-training as the paper does.
- 50 epochs of training is an uncommon setting of LLM post-training. Typically standard instruction tuning only takes 3-5 epochs[1], reasoning post-training like s1[2] also use 5 epochs. It does not make sense to me to do 50 epochs of training and measure overfitting based on that.
[1] Taori et al. (2023). Stanford Alpaca: An Instruction-following LLaMA model.
[2] Muennighoff et al. (2025). s1: Simple test-time scaling.
Questions for authors
At Line 37, the paper asks 'Which of two released checkpoints will overfit my data sooner?', but the paper seems to try to answer which models will overfit 'more', rather than 'sooner'. Can you clarify on that?
Since we have early stopping approach at select when to stop training based on the lowest validation loss or best validation performance, why do we need to select checkpoints based on the final test-validation gap?
7 comments
Summary of paper updates: epoch-15 truth 22/22, accuracy 32/32
We thank all reviewers and the Area Chair. The revised PDF is uploaded; every change is in blue and the main text stays within 10 pages.
-
Ground truth at earlier epochs (WvNW Q2, Dxe2). Read at epoch 15, the test gap gives the same grouping on all 22 pair runs, and the epoch-4 rule agrees on 22 of 22. With a pair's two seeds averaged, the two groups are apart at every epoch from 6 to 50. A run stopped at epoch 15 still carries the difference: 1.18 to 1.60 times the twin's test gap. (Section 3, p. 3, lines 130–133; Appendix X)
-
Accuracy (WvNW Q1). On the full GSM8K test set, all four separating pairs, eight checkpoints: the flagged member is the less accurate one at 32 of 32 readings; at epoch 50, pooled, 74.0% against 79.4% (p = 2 × 10⁻¹¹). (Section 5.2, p. 6, lines 315–324; Appendix Y)
-
Simple aggregates (es6e W1, WvNW Q3). The rule is right on all 20 runs at epoch 4 and at each of the five later horizons; no single-epoch reading matches it. Paired with ORI-AUC, which this paper introduces, the RMS gap gives the same numbers, as Theorem 2 predicts. Calibrated out of family, the drift reads 19/20 with one false alarm, the validation-loss ratio 17/20 with three. (Section 5.3, p. 7, lines 352–367)
-
Other protocols (es6e W2). At rank 64 and on a difficulty-matched split, the thresholds fixed at rank 16 flag all three re-run pairs at epoch 4. The verdict holds across four families, three tasks and 1.5B to 32B parameters; a new paragraph names full-parameter fine-tuning as the next step. (Figure 5e; Section 6, p. 10, lines 524–538)
-
Writing (WvNW W1, Dxe2). The abstract names the two thresholds, the pair read at epoch 5 and the twins; "index" means ORI(t) only; Section 5 has numbered heads 5.1 to 5.4 tied to the contributions; new Figure 6 puts the epoch-4 reading beside every check made on it.
Official Comments by Reviewer Dxe2
Thanks for your rebuttal. However, I still have significant concern over the motivation and presentation quality of the paper.
- I understand the authors are trying to predict the train-test gap (test loss minus training loss), at 50 or 15 epochs based on the information in first 4 epochs, but why it is practical is still unclear to me. If our goal is to select which checkpoint to start tuning with, then the correct objective of this is to predict the test loss or downstream performance of the model before it starts to overfit. Which means, for every checkpoint, we use early stopping to terminate the run when the model is overfitting. The validation loss or the downstream performance of model we get through this process are valid objectives to predict and compare with, rather than how each model overfits at a same epoch across different models. Whether a fine-tuned model will overfit on a task and how much will the model overfit on the task depend on the training schema and are not properties determined by the base model checkpoint. In another word, if you tune and stop wisely (or stop more aggressively), the worst base model in your experiment may don't overfit at all.
- The current abstract is still difficult to understand. For one example, in line 13 the revised paper says 'the rule uses two thresholds', but nowhere before or near line 13 does the paper explain what the thresholds are or what are the principles of these thresholds. Only at line 21 the paper mentions two numbers before it explains in highly descriptive ways about a, c, and the ratio between them, which are three numbers and only a/c is one of the thresholds that the paper use. The reader can only speculate what are the thresholds.
Due to the reasons above, I cannot raise my score.
The cited "simpler metric" uses our own ORI-AUC; epoch-15 ground truth leaves all 22 verdicts unchanged
We thank the reviewer for the comments and answer each with the reviewer's words quoted. Page and line numbers refer to the uploaded revision, where changes are in blue; the review's own line numbers refer to the original.
1. Fifty epochs
Review: "It does not make sense to me to do 50 epochs of training and measure overfitting based on that."
Epoch 50 is only "the settled endpoint the verdict read from the first E = 4 epochs is checked against" (p. 3, lines 129–130); the paper cites s1, LIMA and LIMO at up to 15 epochs (p. 3, lines 127–128). We re-read the ground truth earlier, from the released per-epoch test losses (new Appendix X, Table A22):
Entries are test-gap ratios A/B. At epoch 15 the assignments are identical and the early rule is right on 22 of 22. At epoch 10 they are the same with a thin margin. The revision states the scope: the claim covers horizons from epoch 6, and a run stopped at epoch 15 still carries the difference, the separating members' test gap being 1.18 to 1.60 times their twins' (p. 3, lines 130–133).
2. Early stopping (Q2)
Review: "why do we need to select checkpoints based on the final test-validation gap?"
Early stopping decides when to stop one run. The paper asks which of two checkpoints to start from; the revision says so in Section 3: "Early stopping decides when to stop one run; the rule decides which checkpoint to start from" (p. 2, lines 102–103). The quantity is also different: the paper's truth is test loss minus training loss (Eq. 1, p. 3, lines 111–112). The term "test-validation gap" does not occur in the paper.
3. The three "basic mistakes"
Review: "'Overfitting is normally legible only once a run is over', which is not true as we can always monitor the training and validation curve"
Table A9 gives what that monitoring shows at epoch 4. The raw validation-gap ratio of the separating pair P8 is 1.06–1.08, while the matched pairs P3 and P7 are at 1.07–1.12. The ranges overlap, so the curves do not yet tell the two groups apart, which is the point of the sentence. The revision now carries this evidence in the sentence itself (p. 1, lines 040–041; Figure 6e).
Review: "'A small fine-tune drives the training loss to zero, ...', which is not likely to be true"
The words left out are "numerically zero at the final epoch in every run here" (p. 1, lines 041–042). It is a measured fact about these runs, and "small" refers to the thousand examples of line 036.
Review (line 36, annotation budget): A fair point. The sentence now reads "Fine-tune the wrong one and the labels buy a model that overfits them" (p. 1, lines 036–037).
4. The abstract
The other fixes are made: the abstract names the two thresholds ("The rule uses two thresholds, both locked in advance", line 013); audit.py is introduced as "The released script audit.py" at first use (p. 4, line 193); and "sooner" now reads "more" (p. 1, line 038; Q1).
Request
Every line the review cites lies in lines 11–38 or 190–191 of the original 42-page paper. No experiment, table or theorem is questioned, yet soundness is rated as having concerns that "do not appear easy to address". The one soundness point, the 50-epoch horizon, is answered in Section 1 with the released logs. We would be grateful if the reviewer would reconsider the scores.
P.S. To the Area Chair
This review rates soundness 1 without discussing any result, table or theorem; its line references stop at line 191. Its main objection (50 epochs) is answered above: at epoch 15 the grouping of all 22 runs is unchanged. We would be grateful if this were taken into account.