acceptodds
Under review as a conference paper at ICLR 2027

Terminal Shrinkage Averaging Reveals a Schedule-Estimator Interaction in LLM Pretraining

Abstract

Large language model (LLM) pretraining conventionally returns the raw final iterate. This couples two design choices: the learning-rate schedule that generates the parameter trajectory and the estimator that constructs the deployed model (e.g. the raw final iterate or a checkpoint average). A schedule that promotes optimization progress may differ from one that minimizes variation in the raw final iterate. Separating these choices creates an opportunity to maintain progress late in training while reducing variation in the returned model. To this end, we propose Terminal Shrinkage Averaging (TSA), which interpolates between the raw final iterate and the average of recent checkpoints to balance recent progress against terminal variation. We analyze how TSA changes the preferred terminal learning-rate schedule under a local quadratic approximation and test this interaction through a sequence of controlled NanoChat experiments. Finally, we demonstrate that the resulting gains transfer to depth-22 NanoChat, where the combined schedule and estimator improve validation quality. A qualifying time-to-GPT-2 run also finishes faster than the public baseline used in our experiments, providing preliminary evidence of benchmark acceleration.

Primary area:
optimization
Keywords:
Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PvvQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that the terminal learning-rate schedule and the output estimator should be designed jointly rather than defaulting to the raw final iterate. It proposes Terminal Shrinkage Averaging, which interpolates between the last iterate and a uniform average of the last K checkpoints. Under a local quadratic model, the authors derive an exact risk that is quadratic in α and decomposes into a lag term and a noise term, and they give conditions for an interior optimum. Controlled depth-12 NanoChat experiments show that raising the terminal LR floor hurts the raw iterate but helps TSA, and a depth-22 time-to-GPT-2 run improves validation BPB.

Critical strengths
  1. Separating the trajectory-generating schedule from the deployed estimator is clean, and it encourages practitioners to tune cooldown and averaging together.

  2. TSA requires no change to the optimizer and only a few stored checkpoints.

Critical weaknesses
  1. The controlled experiments are very small and short. The depth-12 runs train 286M parameters for only 98M tokens.

  2. Effect sizes are very small and practical significance is unclear. The main gains are on the order of 1e-3 BPB.

Questions for authors
  1. Does the schedule–estimator interaction also hold across different floors when the estimator is EWA or checkpoint EMA instead of TSA?

  2. Can you express the depth-12 and depth-22 gains as an equivalent compute saving, e.g., how many extra steps the raw-iterate baseline needs to match TSA's BPB?

4 comments
AuthorsOpenReview ↗

We thank the reviewer for emphasizing generality across output estimators and practical significance.

W1

We agree that the approximately 286M-parameter, 98M-token D12 experiments are short controlled studies rather than demonstrations of compute-optimal large-scale pretraining. Their role is to isolate the interaction; D22 provides a separate scale-transfer test.

Q1.

Yes. The schedule–estimator interaction is not specific to TSA. We extended the comparison in Appendix B.3 across all three terminal floors (5%, 10%, 15%) using three paired D12 streams. Each estimator's hyperparameters were held fixed across floors, and all estimators were evaluated on the same training trajectories.

Both EWA and checkpoint EMA reproduce the interaction at both higher floors. At 15%, raising the floor worsens raw BPB by +0.003801, yet improves BPB under every tested averaging estimator, reversing the sign of the schedule effect. Several EWA configurations and checkpoint EMA exhibit even larger interactions than fixed TSA. Rather than weakening our contribution, this supports its broader premise: the effect of terminal learning rate scheduling depends on the estimator used to construct the returned model. TSA is a simple, interpretable realization of this interaction.

EstimatorInteraction, 10% vs. 5%Interaction, 15% vs. 5%Averaged BPB change, 15% vs. 5%
TSA, α=0.50\alpha=0.50+0.001409+0.004424−0.000622
EWA, β=0.50\beta=0.50+0.001499+0.003909−0.000108
EWA, β=0.75\beta=0.75+0.001693+0.005133−0.001332
EWA, β=0.90\beta=0.90+0.001605+0.005415−0.001614
EWA, β=0.95\beta=0.95+0.001548+0.005417−0.001615
Checkpoint EMA, β=0.95\beta=0.95+0.001561+0.005381−0.001579

Mean BPB differences across three paired streams. TSA and EWA use K=8,s=32K=8,s=32; checkpoint EMA uses K=16,s=16K=16,s=16. All coefficients are held fixed across floors. Positive interaction means increasing the floor is more favorable for the estimator than for the raw iterate; negative BPB change means the averaged model improves when the floor increases.

Q2.

We measured BPB-matched update work in D12 using three additional 3,300-step raw-baseline runs, with the learning-rate schedule recalibrated to the longer horizon. So the comparison is 3000 steps with a 15% floor + TSA versus 3,300 step runs with a 5% floor calibrated for the 3,300 endpoint + raw iterate being returned. These are independently trained longer-horizon baselines, not continuations of the original 3,000-step trajectories. Using linear interpolation of their validation learning curves, we determine when each raw baseline matches the BPB achieved by the 15%-floor + TSA recipe at 3,000 updates (α=0.50,K=8,s=32\alpha=0.50,K=8,s=32).

Across the three paired streams, the raw baseline requires 172–192 additional updates, averaging approximately 183 extra updates to reach the TSA target. This corresponds to 5.42%–6.00% fewer training updates, averaging 5.74%, for the TSA recipe at matched validation BPB.

D12 seedRaw updates to match TSAAdditional raw updatesEquivalent update-work saving
212013,172.07+172.075.42%
212023,185.02+185.025.81%
212033,191.57+191.576.00%
Mean3,182.89+182.895.74%

Targets are measured on sampled validation learning curves, with linear interpolation between evaluation points. Savings are expressed as a fraction of the raw baseline's updates required to attain the target, excluding checkpoint construction, evaluation, and other overhead.

Even modest BPB improvements can be economically meaningful in pretraining: a small quality gain can correspond to a nontrivial reduction in the training work needed to reach a given loss. At large training scales, such differences could have substantial computational value, although we do not extrapolate the measured D12 percentage to larger models.

AuthorsOpenReview ↗

Less than 48 hours until the discussion window closes.

Thank you again for your review.

We address your comments and questions in our answers above.

We hope we've clarified that the role of the interaction we explore is not unique to the kind of averaging we focused on and that the seemingly small improvement in BPB provides a nontrivial improvement in pretraining compute efficiency.

If you have further questions, we are very happy to answer them.

Thank you for the additional experiments. They do not change my assessment. The scale limitation remains. The new table also weakens the case for TSA itself. From the 5% to the 15% floor, EWA improves by up to 0.001615 and checkpoint EMA by 0.001579, against 0.000622 for TSA, while Fig. 7 shows TSA ahead of the best EWA by only 0.000159 at the 10% floor.

Thank you for your follow-up and careful review. We would like to clarify two points.

On scale: We agree that experiments on frontier LLMs would provide stronger evidence, but full-scale pretraining is well beyond our compute budget. The time-to-GPT-2 benchmark is a meaningful and competitive pretraining setting that the community actively works on. NanoChat's official leaderboard recorded six successive records between January 29 and March 14, 2026, with more submissions continuing into August and September [1]. While not frontier-scale, reproducing GPT-2-level performance more efficiently is substantially more than a toy experiment.

On TSA versus EWA: Our central contribution is the joint design of the learning-rate schedule and output estimator, and the interaction between them, not that TSA is the best possible averaging method under this framing. We chose this kind of averaging because its simple interpolation makes this interaction easy to isolate, analyze theoretically, and demonstrate experimentally. We believe that EWA and EMA exhibiting an even stronger interaction reinforces our main finding rather than weakening it.

We appreciate the reviewer's feedback and agree that optimizing the estimator and scaling are both valuable future directions.

[1] https://github.com/karpathy/nanochat/blob/master/README.md#time-to-gpt-2-leaderboard

Reviewer_XifnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TSA, a method jointly designed by the terminal schedule and the estimator for LLM pretraining. TSA interpolates between the raw final iterate and the average of recent checkpoints to balance recent progress against terminal variation. The authors' analysis found that TSA was used as the output estimator, which could employ a higher terminal learning rate in the later stages of training to continue effective optimization, while averaging is used to suppress the variance of random noise.

Critical strengths
  • (S1) This paper proposes a novel insight to optimize the LLM pre-training stage by schedule‑estimator interaction. Unlike other methods, which use weighted averaging (SWA/LAWA) follows a fixed training trajectory, with averaging performed after the fact. This paper proposes that the choice of output estimator determines the type of terminal learning rate scheduling you should use. The proposed TSA method allows for the use of a higher terminal learning rate, enabling continued optimization in the later stages. The results are reproduced under both the Muon+AdamW and pure AdamW optimizers.

  • (S2) The TSA is straightforward, easy to implement, and simple to deploy. It doesn't modify the optimizer; it is a plug-and-play method with the optimizer.

Critical weaknesses
  • (W1) This paper relies on the assumption of a local quadratic approximation and describes only the window at the end of training. Some hyperparameters, e.g., the endpoint alphaalpha, checkpoint number KK, and optimizer steps ss, rely on manual setting. The TSA needs to save multiple checkpoints, which will require more memory.

  • (W2) The experiment environment of the paper is limited in scope and scale. All experiments were conducted using the NanoChat training framework and were not validated on mainstream LLMs or large-scale model sizes. Furthermore, the experimental dataset for the 22-layer model is small, with only 3 seeds, and only 1 seed achieved the passing threshold for the DCLM-CORE metric. The authors acknowledge that this is only preliminary evidence. Also, no evaluations were conducted for downstream fine-tuning tasks, lacking downstream task metrics to support the demonstrated benefits. The magnitude of the gains is relatively small; the improvement in BPB is on the order of 10−310^{-3}, which constitutes a relatively minor enhancement. In many real-world pre-training noise environments, gains of this magnitude may be masked by training randomness.

Questions for authors
  • (Q1) The theoretical analysis in this paper relies on a local quadratic approximation that applies only to a brief terminal window. How reliable are these conclusions when applied to real-world, non-convex loss surfaces of large models? Can empirical evidence be provided to demonstrate that this quadratic assumption indeed holds during the cooldown phase in the late stages of training?

  • (Q2) All experiments are conducted on NanoChat. Could the authors provide some results on mainstream LLMs, e.g., LLaMA, Qwen?

  • (Q3) As I mentioned in W2, could the authors provide more experiments, tasks, or domains to demonstrate the effect of TSA?

  • (Q4) The “Time-to-GPT-2” experiment with depth-22 used only 3 random seeds, and only 1 seed exceeded the DCLM-CORE passing threshold. Could the authors provide more replicated experiments to statistically confirm that TSA can consistently improve downstream zero-shot performance, rather than merely enhancing validation BPB?

  • (Q5) Could the authors provide some failure modes associated with this TSA-and-high-floor formulation?

3 comments
AuthorsOpenReview ↗

We thank the reviewer for recognizing the schedule–estimator perspective and the simplicity of TSA. We address the questions below and additionally clarify the implementation and hyperparameter concerns.

Q1

We ask that you refer to our note to reviewer 1nPD for a detailed explanation on this topic. There is support for our approximation both in past literature and in our experiments.

Q2

Full-scale Qwen3 or Llama pretraining is beyond our computational budget. The smallest released Qwen3 model has 0.6B parameters and was pretrained on approximately 36 trillion tokens. Our cost estimate of that is over $100k in compute.

Instead, we conducted a smaller, controlled from-scratch pretraining experiment using a 125M-parameter model with the Qwen3 architecture and tokenizer. We trained two paired seeds for 98.3M tokens per schedule, comparing a 5% terminal-floor baseline against a 15% floor with TSA. We fixed the TSA parameters at α=0.50\alpha=0.50, K=8K=8, and s=32s=32. Both seeds show a positive schedule–estimator interaction, averaging +0.00482 nats/token. The combined high-floor + TSA recipe also improves held-out cross-entropy over the low-floor raw baseline by 0.03324 nats/token on average.

This provides evidence that the interaction is not specific to NanoChat's architecture, although it doesn't show if it persists at the original Qwen3 model or training scale. Unlike the NanoChat experiment, the higher floor also improves the raw Qwen3-architecture model; TSA provides an additional benefit rather than reversing a raw-model degradation.

Seed5% raw5% TSA15% raw15% TSAInteraction
232014.5535234.5531434.5247054.519656+0.004669
232024.5275414.5270724.5003604.494927+0.004964
Mean————+0.004817

Held-out cross-entropy in nats/token (lower is better). Improvements are positive loss reductions; the interaction measures the additional benefit of TSA at the higher floor relative to its benefit at the 5% floor.

Q3

The original D22 experiments evaluate downstream zero-shot performance through DCLM-CORE. To additionally test whether the pretraining gains transfer to downstream adaptation, we fine-tuned one matched D22 pair on BoolQ: the calibrated raw baseline and the 15% floor + TSA model. Both used the same one-epoch supervised fine-tuning procedure, optimizer settings, and data ordering.

Pretraining checkpointBoolQ dev accuracy ↑Correct / 3,270Fine-tuning time
Calibrated baseline77.217%2,5254m 23s
15% floor + TSA77.554%2,5364m 39s

The combined schedule–estimator recipe improves BoolQ accuracy by 0.336 percentage points (77.554% vs. 77.217%), corresponding to 11 additional correct predictions out of 3,270 examples. This is encouraging evidence that the pretraining improvement survives fine-tuning. However, one paired pretraining seed and one downstream task doesn't establish a reliable improvement across seeds or tasks.

Q4

We ask that you refer to our response to Q1 of reviewer 1nPD. We do not claim that our results are reliable evidence of a record. We intend them to be reliable evidence of scaling. We've also done one more seed to confirm scaling; however, its not on the same hardware required for the record.

Q5

The benefit is not unconditional. Raising the floor from 5% to 15% worsens the raw final iterate (Sec. 4.2), while full uniform LAWA is worse than raw at the 5% floor (Sec. 4.1). A longer SWA-style averaging window (K=32,s=16K=32,s=16) also worsens BPB by 0.000775 relative to raw (Appendix B.3). Our theory is explicitly local to a short, late-training window: averaging can become counterproductive when checkpoint lag outweighs noise reduction. We therefore do not extrapolate our conclusions to early training or much longer, nonstationary windows, where the local approximation may not hold. These are plausible limitations, not experimentally established TSA failures; TSA remains beneficial across all nine tested checkpoint windows (Appendix B.2). Although, we did see that the benefit of TSA started decreasing as the window got too long (Figure 6). Finally, storing multiple checkpoints incurs memory overhead, partially mitigated by the hidden-matrix-only results (Appendix B.5), but we didn't explore this beyond a surface level.

Thank you to the author for the detailed response. Many of my concerns have been addressed, and I apologize for not having considered the cost of the experiment earlier. However, the random seed and performance issues with larger window sizes remain limitations of this method. So I maintain my inital score.

Thank you for your positive perspective on our work. We appreciate the reply and the broader review. It prompted us to provide key clarifications.

Reviewer_1nPDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that the terminal learning-rate schedule and the output estimator should be designed jointly. It introduces TSA, which interpolates between the raw final iterate and a uniform average of recent checkpoints. A local quadratic analysis decomposes the risk into a lag cost and a noise-reduction benefit, explaining why an interior a can be optimal. This is shown in paired depth-12 NanoChat experiments under Muon+AdamW and pure AdamW.

Critical strengths
  1. The schedule–estimator interaction is a useful framing, it changes how one thinks about terminal learning-rate design once the returned model is an average. The paired depth-12 design is clean, and the direction replicates under two optimizers.
  2. TSA is simple and practical. It only uses saved checkpoints, does not change the optimizer, and has a clear bias–variance interpretation.
Critical weaknesses
  1. The scale evidence is weak to support the main claim. The depth-22 result uses three seeds, and only one seed individually passes the CORE threshold. The BPB gain is also small.
  2. The paper’s main contribution is the interaction observation, but the experiments do not fully separate the terminal-floor effect from the TSA window length and the choice of a. The theory is a local quadratic analysis of a fixed trajectory; it does not explain why the interaction should persist at scale.
Questions for authors
  1. For the depth-22 experiment, can you report paired CORE differences with confidence intervals over more seeds, and clarify whether the seed-level qualification is robust?
  2. Can you provide ablations that separate the terminal-floor effect from the TSA window length K and a, including a fixed untuned a
4 comments
AuthorsOpenReview ↗

We thank the reviewer for recognizing the value of jointly designing the terminal learning-rate schedule and output estimator, and for highlighting the paired experimental design. We address the replication, estimator-hyperparameter, and theoretical-scope concerns below.

Q1

The current D22 evidence don't establish reliable time-to-GPT-2 qualification. Our intention was to show TSA can scale. However, establishing a record would require optimizing the stopping point. Due to limited access to 8xH100 nodes, which records often use, we prioritized scaling the schedule–estimator effect rather than a search for the fastest qualifying configuration. That said, our method does provide a speed up, except that we don't know exactly how much.

Across the original experiments, the intervention improves validation BPB by 0.000654, with a 95% interval of [0.000470,0.000838]. The paired CORE differences are (+0.0033,+0.0124,+0.0108). Mean improvement is +0.0088 with a 95% interval of approximately [-0.0032,+0.0209]. To strengthen evidence of scaling, we ran one more paired D22 repetition, comparing the calibrated baseline against the same schedule with a our experimental interventions. These experiments use 4xGH200 nodes, so they don't directly compare to the other runs.

D22 configurationValidation BPB ↓CORE ↑Time
Baseline0.7242140.2435643h16m29s
15% floor + TSA0.7235710.2449663h14m53s

The new pair reproduces the schedule–estimator effect at D22. Raising the floor worsens raw BPB from 0.724214 to 0.725814, but applying TSA reduces it to 0.723571, outperforming the calibrated baseline by 0.000643 BPB. CORE also improves by +0.001402, although neither run qualifies. This provides another favorable paired replication of the validation improvement. That said, we don't interpret the small timing difference as an algorithmic speedup.

Q2

Section 4.3 already varies terminal floor 5%, 10%, 15% with fixed TSA parameters α=0.55,K=8,s=32\alpha=0.55,K=8,s=32; the interaction doesn't arise from tuning TSA separately for each floor. Our appendices characterize effects of these hyperparameters on our method. Appendix B.1 shows the preferred α\alpha increases with the floor, while Appendix B.2 finds TSA is robust among nine checkpoint-count/spacing configurations at the 10% floor.

To directly test whether the interaction survives changes to these choices, we ran three new paired D12 streams, varying K∈4,8,16K\in{4,8,16} at fixed, untuned α=0.50,s=32\alpha=0.50,s=32, and separately varying α∈0.25,0.50,0.55,0.75,1.00\alpha\in{0.25,0.50,0.55,0.75,1.00} at fixed K=8,s=32K=8,s=32. All estimators reuse the same training trajectories. The interaction is positive at both higher floors in all tested configurations. At the 15% floor, the raw iterate worsens by +0.003801 BPB. With α=0.50\alpha=0.50, K=8K=8 and K=16K=16 reverse this degradation, while K=4K=4 substantially reduces but does not reverse it. Similarly, α≥0.50\alpha\geq0.50 produces a sign reversal at K=8K=8, whereas α=0.25\alpha=0.25 does not. Thus, the positive interaction is robust across these settings, while reversing the raw schedule penalty requires sufficiently strong averaging.

α\alphaKKΔLraw\Delta L_{\mathrm{raw}}ΔLTSA\Delta L_{\mathrm{TSA}}I(10\%)(15\%)$
0.504+0.003801+0.000335+0.001355+0.003466
0.508+0.003801−0.000622+0.001409+0.004424
0.5016+0.003801−0.000541+0.001300+0.004343
0.258+0.003801+0.001130+0.000862+0.002672
0.558+0.003801−0.000858+0.001465+0.004659
0.758+0.003801−0.001515+0.001611+0.005317
1.008+0.003801−0.001583+0.001522+0.005384

Mean BPB differences across three paired streams; (s=32). ΔLe(f):=Le(f)−Le(5\Delta L_e(f):=L_e(f)-L_e(5\\%) and I(f)=ΔLraw(f)−ΔLTSA(f)I(f)=\Delta L_{\mathrm{raw}}(f)-\Delta L_{\mathrm{TSA}}(f). Negative ΔL\Delta L is improvement; positive II shows the higher floor is more favorable under TSA than under the raw estimate. Raw 15%-vs-5% change is +0.003801 BPB. For the primary α=0.50,K=8\alpha=0.50,K=8 setting, I(15I(15\\%) has 95% paired CI [0.004312, 0.004536] (p=0.000104).

Note: Quadratic Theory

Our theory is explicitly local: it models the terminal checkpoint window, not the full nonconvex pretraining trajectory. Prior work provides empirical support for local quadratic approximations late in language-model training [1]; quadratic models are also established tools for studying averaging and optimizer behavior [2,3]. As an empirical consistency check, Appendix B.9 finds that quadratic fits to the mean BPB response as a function of α\alpha achieve R2=0.99950R^2=0.99950–0.999930.99993 across six optimizer–floor conditions, which supports the model's usefulness.

[1] Meterez et al. “A Defense of the Quadratic Model.”

[2] Jain et al. “Parallelizing Stochastic Gradient Descent for Least Squares Regression: Mini-batching, Averaging, and Model Misspecification.”

[3] Zhang et al. “Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model.”

AuthorsOpenReview ↗

Less than 48 hours until the discussion window closes.

Thank you again for your review.

We address your comments and questions in our answers above.

We hope we've clarified the role of the D22 experiments in establishing the scaling of our method, shown through our ablations how the TSA interaction is affected by the relevant hyperparameters, and explained where the local quadratic approximation comes from.

If you have further questions, we are very happy to answer them.

Thank you for the detailed response and additional experiments. However, the D22 evidence remains a single additional pair, the CORE result still doesn't qualify, and the BPB gain is small. Scale evidence remains my main concern, and I will retain my initial score.

Thank you for your follow-up. We would like to clarify two points.

First, our D22 experiments aim to demonstrate scaling, not to establish a benchmark record. Adding our method to the best recipe from that benchmark improves both BPB and CORE on every paired seed, including the additional D22 pair. Pooling these pairs gives a mean BPB improvement of 0.000651 (p ≈ 0.00022) and a mean CORE improvement of +0.00698 (p ≈ 0.083). These gains require no additional training updates. The consistent improvements over the same-seed baseline are the relevant evidence for scaling, rather than crossing an absolute CORE threshold. Also, the new paired seeds we ran are not comparable on the CORE to the original due to the change in hardware, which is why both the baseline and ours have lower CORE.

Second, small BPB gains can be practically meaningful. As shown in our response to Reviewer PvvQ, our D12 experiments demonstrate that the baseline requires approximately 183 additional updates to match our method's BPB, equivalent to 5.74% fewer training updates.

We acknowledge the limitations in scale, but believe these results are well beyond a toy setting and consistently show improvement.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviews and Rebuttals for AC

Dear Area Chair,

We thank the reviewers for their constructive feedback. We briefly summarize how our rebuttal addressed their main concerns.

Reviewer 1nPD (2): We explained that the role of the D22 experiments was to provide evidence of scaling rather than establish a new time-to-GPT-2 benchmark record. We then strengthened the scaling evidence with an additional paired depth-22 experiment, again showing improved validation BPB and CORE. We also provided new ablations varying TSA's checkpoint window and shrinkage coefficient, demonstrating that the schedule–estimator interaction persists across multiple fixed hyperparameter choices and providing some clarity as to how the interaction is affected by these hyperparameters. Finally, we clarified the scope of our local quadratic theory and provided references in the literature, as well as our own empirical support for it.

Reviewer Xifn (3): We addressed the generalization concerns through new Qwen3-architecture pretraining experiments, reproducing the positive interaction across two seeds. We also added a downstream BoolQ fine-tuning experiment showing improved accuracy and separately clarified practical failure modes of our method. The reviewer subsequently acknowledged that many concerns had been addressed and maintained their positive score.

Reviewer PvvQ (2): We directly addressed both questions with new experiments. First, we demonstrated that the interaction extends to other averaging estimators, establishing that our central finding is a general schedule–estimator interaction rather than something TSA-specific. Second, we showed that the small BPB boost our method provides yields a nontrivial improvement in training efficiency. On longer-horizon baseline runs, our method achieves matched validation BPB with approximately 5.74% fewer training updates at depth 12. We clarified that TSA is an interpretable realization of the broader principle, not a claim of superiority over every averaging method.

We thank the AC for their consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.