acceptodds
Under review as a conference paper at ICLR 2027

Forecasting Post-Checkpoint Training with Moving-Frame Dynamics

Abstract

We study how far the future trajectory of a training run can be forecast from its current checkpoint. Because learned features continue to evolve, coordinates fixed at the checkpoint become progressively inaccurate. We introduce a moving-frame predictor that transports the checkpoint-conditioned mean and covariance with the evolving representation, using optimizer-induced feature motion to update the frame. This models the forecastable component of the continuation. We then establish a complementary, method-independent limit: the irreducible state-prediction MSE induced by post-checkpoint stochasticity is locally linear in the forecast horizon, whereas the additional error from freezing feature evolution admits a local quartic bound. The resulting error budget explains how inaccuracies accumulate over a completed forecast. Experiments on end-to-end multilayer perceptrons and Transformers for next-token prediction on real text verify the predicted error structure and show that the moving-frame predictor substantially outperforms checkpoint-fixed and learning-curve extrapolation baselines. Together, these results separate forecast error that can be reduced by modeling representation dynamics from uncertainty intrinsic to the future training path.

Primary area:
optimization
Keywords:
Training dynamics, post-checkpoint forecasting, moving-frame analysis, stochastic optimization, conformal prediction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_oyrZOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work primarily investigates the issue of future trajectory prediction during the model training phase. Since fixed coordinates shift as feature learning progresses, accurately evaluating these coordinates requires real-time updates, a significant aspect for model learning. To address these, the authors propose a moving-frame predictor that updates the feature frame by propagating checkpoint-conditioned mean and covariance. Furthermore, the authors identify two key findings regarding predictions across different time horizons. In the short-horizon regime, model error is difficult to eliminate due to post-checkpoint stochasticity, growing roughly at a rate of O(T). Conversely, in the long-horizon regime, freezing the feature frame introduces an additional avoidable state-MSE that grows approximately as O(T⁴), which the proposed moving-frame predictor aims to reduce. Experiments further demonstrate that the proposed predictor achieves superior performance compared to competitive baselines.

Critical strengths

In my view, this paper clearly articulates their work motivation. By examining how changes in learning representations necessitate updates to fixed coordinates across different stages of the model, the authors identified two distinct effects within the learning process, specifically regarding error dynamics during short-term and long-term horizons. They distinguish between irreducible uncertainty and avoidable error induced by feature drift. Notably, the state-MSE components susceptible to modification or suppression were effectively mitigated using the moving-frame predictor proposed by the authors.

The derivations of key equations and the theoretical proofs are comprehensive, serving as a solid theoretical foundation that supports the final conclusions.

Critical weaknesses

Regarding the experimental setup, the authors' experiments appear to rely largely on controlled MLPs and small-scale character-level text tasks. While the authors conduct experiments using public text corpora and the Pythia-1B model, it would be beneficial for them to provide further evidence regarding the predictability behavior associated with the proposed moving-frame predictor and, if feasible, to demonstrate the framework's generalizability to more diverse or large-scale training scenarios.

Another point I consider is that post-checkpoint stochasticity brings a diffusion floor over long horizons, which ultimately limits prediction performance. This raises the concern of whether there is a critical threshold or a specific range where performance begins to degrade significantly during continuous prediction. I believe this is important for defining the learning paradigm of future learning. Indeed, incorporating a more practical selection rule regarding the prediction horizon is a vital component for closing the loop on the framework.

Although presentation is a relatively minor issue, the paper would be improved from greater attention to detail, particularly in the design and presentation of its figures.

Questions for authors

Please see the Critical Weakness to regarding my concerns.

3 comments

General Comment

Thank you for highlighting scale, practical horizon selection, and figure clarity. Because the question refers to the listed weaknesses, we answer each one directly.


W1 — Scale and generality.

We tested Moving Frame on five Pythia models from 410M to 12B parameters, forecasting 64 to 1,024 updates ahead during fine-tuning on the Dolly dataset. At a given horizon, forecast error relative to the variation across training runs has been stable, suggesting that forecast accuracy is not strongly affected by model size within the evaluated range. The raw forecast error increased with the forecast horizon in all five models (Fig. 3(e), p. 10, and Appendix D.8, pp. 46–48).

We also tested Pythia-1B on Dolly instruction tuning and code next-token prediction for up to 64 updates. Moving Frame had lower forecast error than Frozen on both tasks (Appendix D.7, pp. 45–47, and Fig. 13, p. 47).

In addition, we evaluated two uses of the forecasts in MLP experiments. For compute allocation, we used the forecasts to choose which training configuration should receive more compute. For early stopping, we used them to decide whether a run should continue (Table 1, p. 9, and Appendix D.6).


W2 — Practical horizon selection.

Thank you for highlighting the importance of selecting a reliable forecasting horizon. We have clarified the distinction between the theoretical predictability limit and the practical checkpoint-based decision procedure.

Theoretical horizon. Theorem 1 defines the intrinsic prediction horizon as the first point at which the irreducible state-MSE exceeds a prescribed tolerance. Proposition 2 additionally provides a local certified horizon for Frozen under its regularity assumptions. These results characterize how stochastic uncertainty and feature drift limit the useful forecasting range. However, they do not directly provide an optimal adaptive horizon-selection rule for Moving Frame.

Practical decision procedure. We implemented and evaluated a rolling checkpoint-based decision procedure in a practical early-stopping task. Rather than relying on a single long forecast, Moving Frame predicts the loss at the next designated checkpoint and compares the predicted improvement, accounting for compute cost, against a fixed threshold. Training continues to the next checkpoint if the predicted improvement exceeds the threshold; otherwise, it stops. After continuing, the forecast is re-anchored using the newly observed model and optimizer states. Figure 12 (p. 42) also demonstrates that rolling re-anchoring reduces the accumulation of long-horizon prediction error.

Early-stopping evaluation. Appendix D.6 evaluates this procedure on MLP training problems using predetermined decision checkpoints at steps 80, 160, and 320. The stopping threshold is selected on separate discovery problems and fixed before confirmation. Moving Frame achieves the lowest confirmation objective regret, 9.91×10−49.91\times10^{-4}, while saving 47.6% of the remaining training steps (Tables 1 and 10).

Together, these results connect the theoretical predictability limit to an experimentally validated early-stopping decision procedure. While our experiments use predetermined candidate horizons rather than optimizing the horizon itself, rolling re-anchoring provides a practical mechanism for limiting forecast error accumulation and making sequential training decisions.


W3 — Figure presentation.

We increased the font sizes in the existing figures and tables in the main text. We also removed small legends and simplified crowded panels so the main results are easier to see. Figure 1 now makes it easier to see when feature drift starts to dominate Frozen’s forecast error, and Figure 3 brings together the results on forecast error, interval coverage, and model size. We revised the captions to explain what each panel measures and how the errors are normalized.

Dear authors,

Thanks for your rebuttal with my concerns, I hope that my suggestions will be incorporated into the camera-ready version if accepted.

Strengthening the in-depth analysis of experimental results and refining the discussion on the limitations of the work are aspects that cannot be overlooked.

Therefore, my score remains here.

AuthorsOpenReview ↗

Reply to Reviewer

Dear Reviewer oyrZ

Thank you for maintaining your recommendation for acceptance. We will continue polishing our paper. Thank you again for the time you spent on our paper.

Best Authors

Reviewer_SeX5OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how to forecast future training trajectories from intermediate checkpoints. The authors propose a moving-frame predictor that tracks evolving feature representations and propagates the mean and covariance of model states. Theoretically, they distinguish irreducible prediction error caused by training stochasticity from additional error introduced by freezing feature evolution, establishing local O(T)O(T) and O(T4)O(T^4) error scalings, respectively. Experiments on MLPs and Transformer-based language models demonstrate improved forecasting accuracy over fixed-frame and learning-curve extrapolation baselines.

Critical strengths
  1. The paper proposes a moving-frame predictor that accounts for evolving feature representations, addressing a key limitation of checkpoint-fixed approximations.

  2. The theoretical analysis distinguishes irreducible stochasticity from avoidable feature-drift errors and provides interpretable error bounds.

  3. The experiments cover MLPs and Transformer-based language models, with comparisons against several forecasting baselines demonstrating improved prediction accuracy.

Critical weaknesses
  1. The presentation is poor. The abstract and introduction are difficult to follow, and the introduction lacks citations to establish the research context and clearly motivate the work. I do not think the paper is written in a way that is understandable to human readers.

  2. The theoretical guarantees rely on strong local regularity assumptions, and their applicability to realistic, nonstationary training dynamics remains unclear.

  3. The experiments mainly focus on small MLPs and shallow Transformers, with limited evidence on larger models and longer forecasting horizons.

Questions for authors
  1. I strongly encourage the authors to improve the presentation, particularly the abstract and introduction, to clarify the motivation, research gap, and key contributions.

  2. How robust are the theoretical results when the local regularity assumptions are violated or training dynamics change abruptly?

  3. Could the authors evaluate larger models and longer forecasting horizons to better demonstrate the scalability and generality of the proposed method?

4 comments
AuthorsOpenReview ↗

Response to Reviewer SeX5

General Comment

Thank you for the constructive suggestions. We have extensively revised the presentation.


Q1 (W1) — Presentation.

We rewrote Abstract (p. 1) and Introduction (pp. 2–3) from scratch to improve the presentation.

  • Motivation. Abstract now explains why forecasting both the learning curve and its uncertainty matters for understanding training dynamics and allocating limited compute across runs (p. 1, lines 011–014). The Introduction expands on the theoretical motivation by contrasting deterministic training, where a checkpoint determines a single continuation once the objective and update rule are fixed, with stochastic training, where different minibatch draws can produce different trajectories from the same checkpoint. This contrast motivates our study of which aspects of training are predictable from the checkpoint and which arise from subsequent stochastic updates (p. 2, lines 056–073). Introduction also makes the practical motivation concrete by explaining why practitioners compare random seeds and configurations in pre/post-training, and why completing every candidate run is costly (p. 2, lines 075–080).

  • Research gap. Abstract now explains that learning-curve extrapolation leaves the saved model and optimizer state unused, while a checkpoint-fixed NTK approximation uses checkpoint information but holds features fixed as training changes them (p. 1, lines 014–018). Introduction adds citations on stochastic optimization (p. 2, lines 064–069), learning-curve extrapolation (lines 079–084), and checkpoint-fixed NTK forecasting (lines 089–097). It explicitly asks how much prediction error is unavoidable when the checkpoint is available, learned features continue to evolve, and future minibatches are unknown (p. 2, lines 068–073).

  • Key contributions. Abstract now explains why future minibatch randomness gives rise to irreducible prediction error and distinguishes it from the avoidable error of freezing features (p. 1, lines 023–027). Introduction puts this theoretical result first in the contribution list, separating the locally linear irreducible readout-state MSE from Frozen’s excess MSE, which has a local quartic upper bound (p. 3, lines 146–150). We then explain that Moving Frame and Frozen use the same checkpoint information, so their comparison isolates the benefit of tracking feature evolution (p. 2, lines 099–106 and p. 3, lines 151–157). In the experiments, Moving Frame stays close to the measured diffusion floor and extends the median forecast horizon by 1.92× at the reported error tolerance (Fig. 1, p. 1, lines 050–053).

  • Visual summary. In addition, we added Figure 2 to Introduction (p. 3, lines 108–144) to improve readability. The figure gives readers an overview before they reach the formal setup. It shows how Moving Frame generates a loss forecast and uncertainty band from a checkpoint, distinguishes the two sources of forecast error, and illustrates how the forecast can inform compute allocation and early stopping.


Q2 (W2) — Violations of local regularity.

We have clarified what will occur when our assumption is violated, especially violation of local regularity. Proposition 1 (Section 4.1, p. 7; Appendix C.4.2, pp. 22–25) bounds Moving Frame's local loss error by its feature/subspace, operator, forcing, source-covariance, and discretization defects. Appendix C.4.2 further shows that, under anchor and first-order matching, local αH\alpha_{\rm H}-Hölder coefficient and feature-velocity mismatches give O(T1+αH)+ϵdiscO(T^{1+\alpha_{\rm H}})+\epsilon_{\rm disc} (recovering the quadratic continuous-path term at αH=1\alpha_{\rm H}=1); without matching, explicit O(T)O(T) terms remain, and an unannounced jump is not certified.

Under ordinary stochastic minibatch training with fixed training conditions, Moving Frame’s prediction intervals achieved 92.3% validation-loss coverage in our AdamW experiments. This supports reliability in the tested stationary setting, but the forecast can become unreliable when training conditions change abruptly and the rollout still assumes the original conditions. When we tripled the learning rate at step 64 without informing the forecast, coverage dropped to 43.0%; supplying the change restored it to 93.3%.

We now state this limitation in Section 6 (p. 10, lines 528–532).


Q3 (W3) — Larger models and longer horizons.

We tested Moving Frame on five Pythia models from 410M to 12B parameters, forecasting 64 to 1,024 updates ahead during fine-tuning on the Dolly dataset. At a fixed horizon, variance-normalized forecast errors remain comparable across model sizes, suggesting that forecast accuracy is not strongly affected by model size within the evaluated range (Fig. 3(e), p. 10). Raw forecast errors increase with the horizon, consistent with the theoretical prediction, while variance-normalized errors show only weak dependence on model scale within the evaluated range (Appendix D.8, pp. 46–48, and Fig. 15, p. 48).

Dear Reviewer SeX5

As the discussion period is about to end in 2 days, we would like to kindly follow up and ask whether our rebuttal has addressed your concerns. We have extensively revised our paper following Reviewer's instruction and we hope the feedback.

Best, Authors

Thank you for the detailed rebuttal and substantial revisions. I appreciate the authors' efforts to address my concerns.

In particular, the rewritten introduction improves the clarity of the motivation and contributions, and the new experiments on Pythia models up to 12B parameters over longer forecasting horizons considerably strengthen the empirical evaluation. The additional analyses of regularity violations are also helpful.

My main remaining concern is the gap between the theoretical guarantees, which rely on structured stochastic dynamics and regularity assumptions, and their applicability to general nonlinear neural network training. While the new experiments provide encouraging evidence, they do not fully resolve this concern.

Overall, I consider update the score to 2.

AuthorsOpenReview ↗

Thank you for your feedback

Dear Reviewer SeX5

Thank you for getting back to us and sharing your feedback. We deeply appreciate it. Thank you as well for raising the score (1->2). We understand that the gap between real networks and theory is an important limitation of this work. We believe this is the most reasonable claim that can be derived from mathematical theory with standard assumptions and is supported by extensive experiments, but we understand your view. This gap is a community-level hard problem in the theory of deep learning. Once again, we deeply appreciate your assessment of the most important issue regarding presentation.

Best Authors

Reviewer_HZx9OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies forecasting the future training trajectory of neural networks from an intermediate checkpoint. The authors introduce a moving-frame approach that models the evolution of learned feature representations and propagates the mean and covariance of readout parameters. The theoretical analysis separates irreducible uncertainty from future stochastic updates, with locally linear state-MSE growth, from the additional error introduced by freezing the training dynamics, which admits a local quartic bound. Experiments on MLPs, character-level Transformers, and Pythia-1B demonstrate improved forecasting accuracy over frozen-frame and learning-curve extrapolation baselines.

Critical strengths
  1. The moving-frame formulation provides an interesting way to account for evolving learned representations when forecasting training trajectories. The decomposition of forecast error into stochastic uncertainty and avoidable drift is mathematically insightful, and the distinction between locally linear and quartic state-MSE contributions is particularly interesting.

  2. The paper provides extensive theoretical analysis and empirical validation. The moving-frame predictor substantially improves forecasting accuracy over frozen-frame baselines across the evaluated models. The experiments also investigate prediction horizons, uncertainty calibration, and the contribution of individual approximation errors.

Critical weaknesses
  1. Theorem 1 assumes that future affine coefficient paths are measurable at the checkpoint, whereas in end-to-end stochastic training the feature representations and corresponding dynamics depend on future minibatches. The resulting diffusion bound is therefore established for a restricted surrogate process rather than general neural-network training. Similarly, Lemma 1 describes clean-gradient dynamics but does not directly justify deterministic mean propagation under adaptive optimizers such as AdamW. Furthermore, the predictability guarantees concern state MSE and do not directly establish corresponding bounds for forecasting scalar validation loss.

  2. The proposed method requires deterministic full-batch propagation of model parameters over the forecast horizon to estimate future features. The paper does not quantify the computational cost of this procedure or compare against directly evaluating the deterministically propagated checkpoint. Existing learning-curve baselines also receive substantially less information than the proposed method. The Pythia-1B experiments consider only relatively short horizons of up to 64 optimizer steps. Consequently, it is unclear whether the approach provides a practical computational advantage over continuing training or using simpler checkpoint-based forecasting methods.

Questions for authors
  1. How do the authors justify the checkpoint-measurability assumption for future coefficient paths when learned features evolve stochastically? Can the theoretical analysis be extended to coefficients that depend on future minibatch innovations?

  2. Could the authors quantify how accurately deterministic optimizer propagation approximates the conditional mean trajectory under AdamW, particularly in the Pythia-1B experiments?

  3. Could the authors compare against directly evaluating the full-batch-propagated model and report the computational cost of forecasting relative to actual training? This would help isolate the benefit of the moving-frame and covariance machinery.

  4. Can the authors demonstrate useful checkpoint-computable error certificates without relying on future trajectory measurements? Could they also clarify how the state-space diffusion guarantees translate into uncertainty bounds for validation-loss forecasts?

  5. How well does the method perform at substantially longer forecasting horizons, and can its predictions improve practical decisions such as early stopping or compute allocation?

3 comments
AuthorsOpenReview ↗

Response to Reviewer HZx9

General comment

Thank you for the constructive review. We are very appreciating the reviewer's positive review. We have improved our paper following the reviewer's comments as below.


Q1 (W1) — Stochastic future coefficients.

Thank you for your important suggestion. We have extended our theory to address the "assumption for future coefficient paths when learned features evolve stochastically."

  • We added Appendix C.6, Causal Predictability with Stochastic Future Coefficients (pp. 27–31).
  • Theorem 2 (p. 27) gives the locally optimal checkpoint-measurable risk without assuming that the realized future coefficient path is known at the checkpoint.
  • Proposition 3 in Appendix C.6.2 (pp. 30–31) gives sufficient conditions for quartic Frozen excess with state-dependent coefficients.

Q2 (W1) — AdamW.

We have extended our theory to AdamW and conducted additional experiments.

  • Appendix C.8 (p. 33) now distinguishes the deterministic central path from the exact conditional mean and propagates the complete saved AdamW state.
  • Appendix D.3 (pp. 39–40) provides the optimizer checks; Table 5 (p. 40) directly compares the deterministic path with same-anchor Pythia-1B continuations after 64 updates.

The theory identifies and bounds the sources of discrepancy between deterministic AdamW propagation and the conditional mean.

Our experiments show that validation-NLL optimism is only 0.011480.01148 nats/token on Dolly and 0.008990.00899 on Code NTP at this horizon.

Together, these theoretical and empirical results support deterministic AdamW propagation as a practically effective approximation for forecasting stochastic training continuations over the evaluated horizon, despite its nonzero conditional-mean bias.


Q3 (W2) — Direct propagation and cost.

We added a direct-propagation ablation and explicit computational-cost measurements.

In Table 6, Moving full reduces the median prediction error to 0.761 times that of Direct full, with a standalone predictor-time ratio of 1.029.

To address the latter comparison, Figure 14 reports a compute accuracy frontier for Pythia-1B, normalizing the major computation by the measured cost of a training continuation. The results quantify the trade-off between frame-refresh frequency, prediction accuracy, and actual training cost. In practice, multiple training runs are often needed to reduce the influence of stochastic outliers and obtain reliable results, whereas Moving Frame predicts the mean and covariance in a single forecasting computation.

Finally, Table 1 and Appendix D.6 evaluate whether the forecasts support downstream decisions. Moving Frame achieves 8/8 correct compute-allocation selections and the lowest confirmation objective regret in the tested MLP early-stopping setting. These results demonstrate practical value in the evaluated decision problems, especially for real-world tasks that require multiple training runs.


Q4 (W1) — Checkpoint-time certificates and loss forecasts.

Thank you for your suggestion. We now provide checkpoint-time loss intervals and theoretical state-to-loss results.

  • Section 4.2 (p. 8) and Appendix C.4.3 (p. 25) introduce a split-conformal certificate issued before the target endpoint is observed, with simultaneous run-level coverage under complete-run exchangeability.
  • Figure 3(d) (p. 10) reports coverage of 77/8077/80 complete runs and 957/960957/960 windows.
  • Appendix C.7 (pp. 32–33; Lemma 2, p. 32) gives the one-way state-to-loss upper-bound transfer. Appendix C.6.1 (pp. 28–30; Corollary 1, p. 28) proves the irreducible realized-loss risk κt0T+o(T)\kappa_{t_0}T+o(T) under loss–diffusion alignment.

Q5 (W2) — Long horizons and practical decisions.

Thank you for the suggestion. We have substantially extended the model scales and forecast horizons.

  • Section 5 (pp. 9–10) and Appendix D.8 (pp. 46–48) extend Pythia-410M–12B over 64–1,024 updates. Normalized errors remain comparable across model scales, suggesting that forecast accuracy is not strongly affected by model size within the evaluated range, while raw errors increase with the forecast horizon as the theory predicted.
  • Figure 12 in Appendix D.5 (p. 42) tests rolling re-anchoring.

The practical role of Moving Frame is to turn a forecast close to the checkpoint-conditioned limit into a decision about whether additional training compute is worth spending.

  • Table 1 (p. 9) and Appendix D.6 (pp. 42–46) test this role on MLPs through compute allocation and prospective early stopping. Moving Frame makes the correct compute-allocation choice in all 8/8 confirmation cases, with zero regret.
  • For early stopping, we select the threshold on discovery problems (MLP problem seeds used to tune the threshold), then fix it before evaluating on separate, held-out confirmation problems. Table 10 (p. 46) reports the lowest confirmation early-stopping regret, 9.91×10−49.91\times10^{-4}, with 47.6% savings in the remaining training steps.

I've read authors comments and keep my high score.

Dear Reviewer HZx9

Thank you for keeping high score and maintaining your recommendation for clear acceptance. In addition, thank you again for the time you spent on our paper.

Best Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Remark from Authors

Dear AC and reviewers

We thank the reviewers and the AC for their time and constructive feedback. Our discussion has been successfully concluded, and we believe major concerns have been resolved.

Our main contribution is to characterize the limits of forecasting training from a checkpoint by separating the irreducible error due to future minibatch randomness from the avoidable error due to freezing evolving features. In addition, we have proposed a real implementation of the predictor called Moving Frame to achieve near this limit. The discussion helped us clarify this contribution and strengthen the empirical evaluation, including experiments with Pythia models up to 12B parameters.

We will address any additional points raised after the author discussion period in the camera-ready version if accepted. Thank you again for your consideration.

Best, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.