- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work primarily investigates the issue of future trajectory prediction during the model training phase. Since fixed coordinates shift as feature learning progresses, accurately evaluating these coordinates requires real-time updates, a significant aspect for model learning. To address these, the authors propose a moving-frame predictor that updates the feature frame by propagating checkpoint-conditioned mean and covariance. Furthermore, the authors identify two key findings regarding predictions across different time horizons. In the short-horizon regime, model error is difficult to eliminate due to post-checkpoint stochasticity, growing roughly at a rate of O(T). Conversely, in the long-horizon regime, freezing the feature frame introduces an additional avoidable state-MSE that grows approximately as O(T⁴), which the proposed moving-frame predictor aims to reduce. Experiments further demonstrate that the proposed predictor achieves superior performance compared to competitive baselines.
Critical strengths
In my view, this paper clearly articulates their work motivation. By examining how changes in learning representations necessitate updates to fixed coordinates across different stages of the model, the authors identified two distinct effects within the learning process, specifically regarding error dynamics during short-term and long-term horizons. They distinguish between irreducible uncertainty and avoidable error induced by feature drift. Notably, the state-MSE components susceptible to modification or suppression were effectively mitigated using the moving-frame predictor proposed by the authors.
The derivations of key equations and the theoretical proofs are comprehensive, serving as a solid theoretical foundation that supports the final conclusions.
Critical weaknesses
Regarding the experimental setup, the authors' experiments appear to rely largely on controlled MLPs and small-scale character-level text tasks. While the authors conduct experiments using public text corpora and the Pythia-1B model, it would be beneficial for them to provide further evidence regarding the predictability behavior associated with the proposed moving-frame predictor and, if feasible, to demonstrate the framework's generalizability to more diverse or large-scale training scenarios.
Another point I consider is that post-checkpoint stochasticity brings a diffusion floor over long horizons, which ultimately limits prediction performance. This raises the concern of whether there is a critical threshold or a specific range where performance begins to degrade significantly during continuous prediction. I believe this is important for defining the learning paradigm of future learning. Indeed, incorporating a more practical selection rule regarding the prediction horizon is a vital component for closing the loop on the framework.
Although presentation is a relatively minor issue, the paper would be improved from greater attention to detail, particularly in the design and presentation of its figures.
Questions for authors
Please see the Critical Weakness to regarding my concerns.
3 comments
Dear authors,
Thanks for your rebuttal with my concerns, I hope that my suggestions will be incorporated into the camera-ready version if accepted.
Strengthening the in-depth analysis of experimental results and refining the discussion on the limitations of the work are aspects that cannot be overlooked.
Therefore, my score remains here.
Reply to Reviewer
Dear Reviewer oyrZ
Thank you for maintaining your recommendation for acceptance. We will continue polishing our paper. Thank you again for the time you spent on our paper.
Best Authors
General Comment
Thank you for highlighting scale, practical horizon selection, and figure clarity. Because the question refers to the listed weaknesses, we answer each one directly.
W1 — Scale and generality.
We tested Moving Frame on five Pythia models from 410M to 12B parameters, forecasting 64 to 1,024 updates ahead during fine-tuning on the Dolly dataset. At a given horizon, forecast error relative to the variation across training runs has been stable, suggesting that forecast accuracy is not strongly affected by model size within the evaluated range. The raw forecast error increased with the forecast horizon in all five models (Fig. 3(e), p. 10, and Appendix D.8, pp. 46–48).
We also tested Pythia-1B on Dolly instruction tuning and code next-token prediction for up to 64 updates. Moving Frame had lower forecast error than Frozen on both tasks (Appendix D.7, pp. 45–47, and Fig. 13, p. 47).
In addition, we evaluated two uses of the forecasts in MLP experiments. For compute allocation, we used the forecasts to choose which training configuration should receive more compute. For early stopping, we used them to decide whether a run should continue (Table 1, p. 9, and Appendix D.6).
W2 — Practical horizon selection.
Thank you for highlighting the importance of selecting a reliable forecasting horizon. We have clarified the distinction between the theoretical predictability limit and the practical checkpoint-based decision procedure.
Theoretical horizon. Theorem 1 defines the intrinsic prediction horizon as the first point at which the irreducible state-MSE exceeds a prescribed tolerance. Proposition 2 additionally provides a local certified horizon for Frozen under its regularity assumptions. These results characterize how stochastic uncertainty and feature drift limit the useful forecasting range. However, they do not directly provide an optimal adaptive horizon-selection rule for Moving Frame.
Practical decision procedure. We implemented and evaluated a rolling checkpoint-based decision procedure in a practical early-stopping task. Rather than relying on a single long forecast, Moving Frame predicts the loss at the next designated checkpoint and compares the predicted improvement, accounting for compute cost, against a fixed threshold. Training continues to the next checkpoint if the predicted improvement exceeds the threshold; otherwise, it stops. After continuing, the forecast is re-anchored using the newly observed model and optimizer states. Figure 12 (p. 42) also demonstrates that rolling re-anchoring reduces the accumulation of long-horizon prediction error.
Early-stopping evaluation. Appendix D.6 evaluates this procedure on MLP training problems using predetermined decision checkpoints at steps 80, 160, and 320. The stopping threshold is selected on separate discovery problems and fixed before confirmation. Moving Frame achieves the lowest confirmation objective regret, 9.91×10−4, while saving 47.6% of the remaining training steps (Tables 1 and 10).
Together, these results connect the theoretical predictability limit to an experimentally validated early-stopping decision procedure. While our experiments use predetermined candidate horizons rather than optimizing the horizon itself, rolling re-anchoring provides a practical mechanism for limiting forecast error accumulation and making sequential training decisions.
W3 — Figure presentation.
We increased the font sizes in the existing figures and tables in the main text. We also removed small legends and simplified crowded panels so the main results are easier to see. Figure 1 now makes it easier to see when feature drift starts to dominate Frozen’s forecast error, and Figure 3 brings together the results on forecast error, interval coverage, and model size. We revised the captions to explain what each panel measures and how the errors are normalized.