- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates how accurately the training dynamics of neural networks can be predicted from their initialization and how long such predictions remain valid. The authors use the initial output-space Gauss–Newton operator and residual-weighted spectral information to construct a prediction of the training loss rate under squared-error optimization. A Krylov/Lanczos-based procedure allows this prediction to be computed without explicitly forming the full operator. The paper further studies the validity horizon of the linearized prediction, establishes a worst-case impossibility result for predicting this horizon from initial first-order information alone, and proposes early reliability tests based on operator drift and residual-direction growth. The authors evaluate their approach on several model families. The results show accurate rate predictions for standard-parameterized Transformers, different horizon trends under standard and maximal-update parameterizations, and promising results for detecting prediction failures early in training.
Critical strengths
The paper studies when predicting training dynamics from the initial spectrum ease to be accurate. The theoretical impossibility result and the analysis of operator motion provide useful insights into the limitations of initialization-based predictions. The paper evaluates its predictions across different model sizes, parameterizations, target scales, and example counts, including Transformers with up to 3B parameters. The Krylov/Lanczos procedure also reduces the cost of computing the spectral prediction. The early reliability tests provide an additional practical contribution by attempting to identify predictions that will become inaccurate later in training.
Critical weaknesses
The main Transformer experiments use squared-error loss, a fixed learning rate, and full-batch gradient descent, often with only 16 training sequences. Although the paper includes additional experiments with larger example counts and minibatch SGD, these settings remain different from typical large-scale neural network training. More importantly, the paper reports substantially weaker prediction accuracy under cross-entropy loss, where a simple baseline performs considerably better. The paper primarily evaluates prediction accuracy using median rate error over logarithmically spaced training steps, which places greater weight on early training.
Questions for authors
The paper reports that the initial spectral prediction performs substantially worse under cross-entropy loss than the simple baseline. Could the authors clarify whether the proposed prediction framework is primarily intended for squared-error training? Could the authors also provide evidence of its effectiveness under cross-entropy loss and commonly used optimizers such as AdamW?
Although the experiments include Transformers with up to 3B parameters, the main large-model results are obtained using only 16 training sequences. Could the authors evaluate the method with substantially larger training datasets and longer training runs, particularly for larger models?
The paper evaluates prediction accuracy using the median rate error over a logarithmically spaced grid, which gives relatively more weight to early training steps. Specifically, in the 3,000-step training window, half of the 60 evaluation points occur before step 235. This evaluation may favor predictions that are accurate early in training while giving less attention to errors at later stages. Could the authors clarify why the log-time median is chosen as the primary evaluation metric and provide a more balanced assessment of prediction accuracy across the full training trajectory?
The paper acknowledges that the practical drift threshold is empirical and that the main Transformer experiments do not satisfy all assumptions of Theorem 3. Could the authors clarify to what extent the theoretical results support the reliability guarantees claimed for these experiments?