- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that the terminal learning-rate schedule and the output estimator should be designed jointly rather than defaulting to the raw final iterate. It proposes Terminal Shrinkage Averaging, which interpolates between the last iterate and a uniform average of the last K checkpoints. Under a local quadratic model, the authors derive an exact risk that is quadratic in α and decomposes into a lag term and a noise term, and they give conditions for an interior optimum. Controlled depth-12 NanoChat experiments show that raising the terminal LR floor hurts the raw iterate but helps TSA, and a depth-22 time-to-GPT-2 run improves validation BPB.
Critical strengths
-
Separating the trajectory-generating schedule from the deployed estimator is clean, and it encourages practitioners to tune cooldown and averaging together.
-
TSA requires no change to the optimizer and only a few stored checkpoints.
Critical weaknesses
-
The controlled experiments are very small and short. The depth-12 runs train 286M parameters for only 98M tokens.
-
Effect sizes are very small and practical significance is unclear. The main gains are on the order of 1e-3 BPB.
Questions for authors
-
Does the schedule–estimator interaction also hold across different floors when the estimator is EWA or checkpoint EMA instead of TSA?
-
Can you express the depth-12 and depth-22 gains as an equivalent compute saving, e.g., how many extra steps the raw-iterate baseline needs to match TSA's BPB?
4 comments
Less than 48 hours until the discussion window closes.
Thank you again for your review.
We address your comments and questions in our answers above.
We hope we've clarified that the role of the interaction we explore is not unique to the kind of averaging we focused on and that the seemingly small improvement in BPB provides a nontrivial improvement in pretraining compute efficiency.
If you have further questions, we are very happy to answer them.
Thank you for the additional experiments. They do not change my assessment. The scale limitation remains. The new table also weakens the case for TSA itself. From the 5% to the 15% floor, EWA improves by up to 0.001615 and checkpoint EMA by 0.001579, against 0.000622 for TSA, while Fig. 7 shows TSA ahead of the best EWA by only 0.000159 at the 10% floor.
Thank you for your follow-up and careful review. We would like to clarify two points.
On scale: We agree that experiments on frontier LLMs would provide stronger evidence, but full-scale pretraining is well beyond our compute budget. The time-to-GPT-2 benchmark is a meaningful and competitive pretraining setting that the community actively works on. NanoChat's official leaderboard recorded six successive records between January 29 and March 14, 2026, with more submissions continuing into August and September [1]. While not frontier-scale, reproducing GPT-2-level performance more efficiently is substantially more than a toy experiment.
On TSA versus EWA: Our central contribution is the joint design of the learning-rate schedule and output estimator, and the interaction between them, not that TSA is the best possible averaging method under this framing. We chose this kind of averaging because its simple interpolation makes this interaction easy to isolate, analyze theoretically, and demonstrate experimentally. We believe that EWA and EMA exhibiting an even stronger interaction reinforces our main finding rather than weakening it.
We appreciate the reviewer's feedback and agree that optimizing the estimator and scaling are both valuable future directions.
[1] https://github.com/karpathy/nanochat/blob/master/README.md#time-to-gpt-2-leaderboard
We thank the reviewer for emphasizing generality across output estimators and practical significance.
W1
We agree that the approximately 286M-parameter, 98M-token D12 experiments are short controlled studies rather than demonstrations of compute-optimal large-scale pretraining. Their role is to isolate the interaction; D22 provides a separate scale-transfer test.
Q1.
Yes. The schedule–estimator interaction is not specific to TSA. We extended the comparison in Appendix B.3 across all three terminal floors (5%, 10%, 15%) using three paired D12 streams. Each estimator's hyperparameters were held fixed across floors, and all estimators were evaluated on the same training trajectories.
Both EWA and checkpoint EMA reproduce the interaction at both higher floors. At 15%, raising the floor worsens raw BPB by +0.003801, yet improves BPB under every tested averaging estimator, reversing the sign of the schedule effect. Several EWA configurations and checkpoint EMA exhibit even larger interactions than fixed TSA. Rather than weakening our contribution, this supports its broader premise: the effect of terminal learning rate scheduling depends on the estimator used to construct the returned model. TSA is a simple, interpretable realization of this interaction.
Mean BPB differences across three paired streams. TSA and EWA use K=8,s=32; checkpoint EMA uses K=16,s=16. All coefficients are held fixed across floors. Positive interaction means increasing the floor is more favorable for the estimator than for the raw iterate; negative BPB change means the averaged model improves when the floor increases.
Q2.
We measured BPB-matched update work in D12 using three additional 3,300-step raw-baseline runs, with the learning-rate schedule recalibrated to the longer horizon. So the comparison is 3000 steps with a 15% floor + TSA versus 3,300 step runs with a 5% floor calibrated for the 3,300 endpoint + raw iterate being returned. These are independently trained longer-horizon baselines, not continuations of the original 3,000-step trajectories. Using linear interpolation of their validation learning curves, we determine when each raw baseline matches the BPB achieved by the 15%-floor + TSA recipe at 3,000 updates (α=0.50,K=8,s=32).
Across the three paired streams, the raw baseline requires 172–192 additional updates, averaging approximately 183 extra updates to reach the TSA target. This corresponds to 5.42%–6.00% fewer training updates, averaging 5.74%, for the TSA recipe at matched validation BPB.
Targets are measured on sampled validation learning curves, with linear interpolation between evaluation points. Savings are expressed as a fraction of the raw baseline's updates required to attain the target, excluding checkpoint construction, evaluation, and other overhead.
Even modest BPB improvements can be economically meaningful in pretraining: a small quality gain can correspond to a nontrivial reduction in the training work needed to reach a given loss. At large training scales, such differences could have substantial computational value, although we do not extrapolate the measured D12 percentage to larger models.