acceptodds
Under review as a conference paper at ICLR 2027

The Rate Measure: How Far Linearisation Governs Training, and Why That Horizon Cannot Be Read

Abstract

The loss curve of a network trained by gradient descent at squared error can be predicted before it trains. The linearised loss is a Laplace transform of a measure carried by the output-space Gauss–Newton operator , and one Krylov pass before training returns it. We measure the horizon, how long real networks follow this forecast, and find it set by how far moves during training. From M to B parameters, standard-parameterised transformers follow the forecast through the whole window we score from M parameters up. Under maximal-update parameterisation the same networks leave it early. At one size and one initial operator, scaling the targets alone moves the horizon from the whole run to under two hundred steps. At a fixed example count, across size, parameterisation and target scale, runs whose moves further leave the forecast earlier. A tenth of the way into training, this motion flags the runs whose forecast will fail. On MNIST classifiers its relative size does, and every classifier it certifies holds. On transformers to B its growth along the residual does, withholding all but four of the runs that fail. Before the first step, the forecast tracks the measured rate curve of standard-parameterised transformers more closely, in median over log time, than every scaling law we fit. It matches the best predictor that sees none of the run, with no fitted parameter. We prove that in the worst case over networks sharing the linearisation's inputs, no function of them predicts the horizon, and that at cross-entropy the rate is asymptotically model-free.

Primary area:
optimization
Keywords:
loss curve prediction, training dynamics, Gauss-Newton operator, linearisation, neural tangent kernel, Lanczos quadrature, scaling laws, impossibility results

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_7hj3OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how accurately the training dynamics of neural networks can be predicted from their initialization and how long such predictions remain valid. The authors use the initial output-space Gauss–Newton operator and residual-weighted spectral information to construct a prediction of the training loss rate under squared-error optimization. A Krylov/Lanczos-based procedure allows this prediction to be computed without explicitly forming the full operator. The paper further studies the validity horizon of the linearized prediction, establishes a worst-case impossibility result for predicting this horizon from initial first-order information alone, and proposes early reliability tests based on operator drift and residual-direction growth. The authors evaluate their approach on several model families. The results show accurate rate predictions for standard-parameterized Transformers, different horizon trends under standard and maximal-update parameterizations, and promising results for detecting prediction failures early in training.

Critical strengths

The paper studies when predicting training dynamics from the initial spectrum ease to be accurate. The theoretical impossibility result and the analysis of operator motion provide useful insights into the limitations of initialization-based predictions. The paper evaluates its predictions across different model sizes, parameterizations, target scales, and example counts, including Transformers with up to 3B parameters. The Krylov/Lanczos procedure also reduces the cost of computing the spectral prediction. The early reliability tests provide an additional practical contribution by attempting to identify predictions that will become inaccurate later in training.

Critical weaknesses

The main Transformer experiments use squared-error loss, a fixed learning rate, and full-batch gradient descent, often with only 16 training sequences. Although the paper includes additional experiments with larger example counts and minibatch SGD, these settings remain different from typical large-scale neural network training. More importantly, the paper reports substantially weaker prediction accuracy under cross-entropy loss, where a simple baseline performs considerably better. The paper primarily evaluates prediction accuracy using median rate error over logarithmically spaced training steps, which places greater weight on early training.

Questions for authors

The paper reports that the initial spectral prediction performs substantially worse under cross-entropy loss than the simple baseline. Could the authors clarify whether the proposed prediction framework is primarily intended for squared-error training? Could the authors also provide evidence of its effectiveness under cross-entropy loss and commonly used optimizers such as AdamW?

Although the experiments include Transformers with up to 3B parameters, the main large-model results are obtained using only 16 training sequences. Could the authors evaluate the method with substantially larger training datasets and longer training runs, particularly for larger models?

The paper evaluates prediction accuracy using the median rate error over a logarithmically spaced grid, which gives relatively more weight to early training steps. Specifically, in the 3,000-step training window, half of the 60 evaluation points occur before step 235. This evaluation may favor predictions that are accurate early in training while giving less attention to errors at later stages. Could the authors clarify why the log-time median is chosen as the primary evaluation metric and provide a more balanced assessment of prediction accuracy across the full training trajectory?

The paper acknowledges that the practical drift threshold is empirical and that the main Transformer experiments do not satisfy all assumptions of Theorem 3. Could the authors clarify to what extent the theoretical results support the reliability guarantees claimed for these experiments?

Reviewer_yt2nOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies to which extent gradient decent training of neural networks can be described by the evolution of linearized models. It is observed that in some settings this time interval of approximate linearity is shorter than in others (for example, it is shorter with the μP\mu P parameterization than with the standard parameterization). The paper describes an experimental protocol to identify the linearized model and accordingly predict the loss evolution. As main theoretical results, the paper provides two theorems: one asserting that it is not generaly possible to predict the validity interval of the linearized model from the initial state, and another giving an upper bound on the deviation of the model loss from the predicted form given the deviations of the respective Gauss-Newton operators.

Critical strengths

The study has potential: the overall approach of studying loss dynamics through linearization snapshots is reasonable, and the paper has definitely made some progress in this direction. However, the present version is too poor for acceptance; major improvements are needed.

Critical weaknesses

Poor clarity

I found the paper very hard to read. It is written extremely densely, lumping together all kinds of information: conceptual ideas and technical details, theoretical, computational and experimental contributions, known facts and statements claimed as new - all without a clear separation between them.

There is no coherent narrative arc expected in a research paper: motivation - formulation of the research problem - proposed approach - results and their discussion. The paper essentially starts from the results in the beginning of the introduction, immediately jumping into what looks like implementation details. After reading the paper, I don't know what was the original motivation of the research and which significant new knowledge we have obtained.

All sections of the paper are filled with numerous specific numbers whose significance I don't understand. The "Contributions" subsection in the Introduction lists numerous specific values of parameters, seeds, sizes, correlations, heads, etc., but does not explain the overall conceptual takeway.

The content of the paper cannot be properly understood without careful examination of the appendices. This concerns even the theorems - even their formulations cannot be understood without reading the appendices. The formulations contain numerous undefined objects or ambiguous phrases such as "motion deciding expiry", "a predictor needs an ingredient of degree -1", "further passes of Corollary 11 evaluate second-order information", "largest relative motion of a rate", etc.

Lack of notable contributions

I don't see sufficiently important contributions of the work.

The theoretical contributions are limited to Theorems 1 and 3. I understand Theorem 1 as essentially saying that we cannot predict the interval of linearity based on the initial linearization data. This seems expected and unsurprising, Theorem 3 looks like a relatively simple perturbative bound on the loss evolution in terms of the eigenvalue deviations. I don't see why this result is important - the values appearing in the bound do not seem to be easier to compute than the loss itself, and the overall applicability of the theorem is very limited.

Regarding empirical and experimental contributions, the paper list several specific results, but they are not coherently and systematically organized. In particular, one of the emphasized experimental findings concerns the difference in the linearization horizons under the standard and μP\mu P parameterizations. This difference is shown in Figure 1 and reported in more detail in section 5.1. However, the paper never discusses why this dependence is important and how it can be explained. The effect is established only for a limited family of transformer models under particular training conditions, and its generality is questionable.

Questions for authors

N/A

Reviewer_qjv4OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The dynamics of the loss of a linearized neural network under GD can be predicted analytically. This paper tries to understand how accurately this dynamics predicts the non-linearized training under GD and the squared error loss function. The goal is to predict the loss and its decay using the spectrum of the initial output space Gauss-Newton operator, weighted by the initial residual. They empirically show that standard parameterized transformers follow the predicted loss dynamics longer than under maximal update parameterization.

Critical strengths

Understanding when the linearized model dynamics predict the nonlinear training is a relevant question. This can help to understand when feature learning happens.

Critical weaknesses

The paper is unintelligible. The Abstract uses several terms without defining them. They use sentences like “On MNIST classifiers its relative size does, and every classifier it certifies holds.”, which are not understandable. Furthermore, the first sentence of the Abstract is wrong, as this is only true for linearized networks.

The introduction starts with a copy of the abstract and then goes into experimental details without ever placing the paper into a context.

The novelty of this work is also not clear to me. The spectral description of linearized training is known, and the qualitative contrast with μP\mu P training is expected, as μP\mu P is designed to preserve feature learning for large widths.

The paper may contain valuable experimental results, but they do not present them in an understandable way. I believe that for a paper, the results are just as important as making them intelligible for other readers. A paper without a proper introduction and with very confusing language does not meet this basic requirement.

Questions for authors

I do not have any additional questions beyond the points raised above. The issues are not a few isolated clarifications, but a substantial revision improving the presentation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.