- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies whether the ordering of training controls can affect the learned predictor even two schedules use the same learning rate and noise control pairs. The proposed mechanism is covariance lag, which means that stochastic fluctuations do not instantaneously equilibrate after a control change, so an update depends on covariance inherited from earlier controls. For slowly varying cyclic schedules, the resulting order-dependent response is expressed as a curvature flux through the control loop.
The theoretical contribution is mainly a rank-two Gram-completion construction. The observed principal blocks determine the within-block geometry but leave a relative rotation unobserved. The paper proves that a control loop and its reversal can select different missing cross-block predictions while achieving the same training fit and exactly the same complete Gram spectrum after a common noiseless relaxation. A repeated-cycle theorem establishes an limiting separation as the noise amplitude vanishes and the number of cycles increases.
The paper then introduces Checkpoint Response, which uses checkpoint derivatives and covariance propagation to forecast and design schedule-order effects. A separate finite-horizon expansion is developed for native minibatch noise, momentum, and AdamW. Experiments are reported on controlled Gram completion, large regression models, and 304M/1.136B-parameter ImageNet classifiers.
Critical strengths
- The exact Gram construction is conceptually interesting.
The paper demonstrates that training fit and the complete spectrum need not determine unobserved predictions. The use of relative block orientation gives a concrete functional degree of freedom that is invisible to standard spectral summaries.
- The matched-occupation comparison isolates order effects.
The forward and reversed schedules contain exactly the same joint control pairs instead of merely matching average learning rates or marginal control distributions. This makes the distinction between control exposure and control ordering precise.
- The covariance-lag interpretation is intuitive.
Connecting schedule order response to geometric pumping provides a potentially useful perspective on stochastic training dynamics.
Critical weaknesses
- The paper is difficult to follow.
The difficulty goes beyond the technical nature of the material. The manuscript introduces many non-standard terms and custom quantities, including matched occupation, current, curvature flux, relaxed endpoints, collateral changes, and several different response constructions.
Notation is also heavily overloaded. Symbols such as , , , and are reused for projectors, covariances, prediction arrays, cycle counts, transition operators, cross-block matrices, constants, rotations, and responses in different sections.
The paper would benefit from: a notation table; a hierarchy explicitly relating the three theoretical settings; pseudocode for CR; explicit time and memory complexity.
- The paper does not clearly separate three substantially different results.
The paper repeatedly moves between a local small-noise, one-cycle response expansion; a long-horizon isospectral selection theorem for a specialized Gram learner with injected noise and noiseless relaxation; a finite-horizon forecasting approximation for general neural networks with native minibatch noise.
These results have different assumptions and conclusions. In particular, the neural experiments do not establish persistent selection, exact isospectrality, or the long-horizon limit of the Gram theorem. They demonstrate relatively small local output changes after 512 continuation updates.
- The submitted code does not appear capable of producing the large-model CR results.
The submitted native-CR implementation concatenates the parameters and optimizer memory into a full vector. It then explicitly maintains a covariance factor of shape state dimension * r with default rank r=256. For the reported 1.136B-parameter AdamW model, this factor alone requires approximately $3(1.136\times10^9)\times256\times4 \approx 3.5$ TB in FP32. Even rank 64 requires approximately 873 GB. The supplied code uses a single device and contains no distributed factor representation, CPU offloading, or matrix-free compression that avoids materializing these tensors. Therefore, as supplied, the code does not appear runnable for the model sizes reported in the paper. If a different distributed or genuinely matrix-free implementation was used to obtain the results, it is not included or described.
- Several presentation details further reduce confidence.
The abstract states that CR “reduces median forecast error from 7.3% to 11.7%,” which is inconsistent as written. Table 1 suggests that the intended claim is a reduction from a range of 7.3%-11.7% to 4.1%-6.4%. This is likely a typo, but it occurs in a central quantitative claim. Together with the complexity of the experimental pipeline and the absence of raw artifacts, it contributes to the difficulty of independently checking the results.
Questions for authors
-
What exact implementation produced the 304M and 1.136B CR forecasts? How was the parameter and optimizer-state covariance factor represented without requiring hundreds of gigabytes to several terabytes of memory?
-
Please provide the hardware configuration, total hours, peak memory, and number of full training runs for each major experiment.
-
Please try to modify the manuscript to make it more readable, especially in using standard terminology and clear notations.
3 comments
W2. Distinguishing the three results.
Section 3.1 (p. 3) and Table 19 (p. 46) now state the assumptions, conclusion and experiment for each result. We also extended the neural experiment to answer the question left open at update 512.
One cycle. Section 3 asks what happens when two runs start at the same checkpoint and visit the same control pairs in opposite orders. The covariance takes time to adjust as the controls change. For one slow cycle with small injected noise, Eq. (7) (p. 4) gives the resulting difference in expected outputs after final gradient flow, a last noise-free stage that removes the training error. This calculation covers one cycle.
Repeated cycles in Gram completion. Theorem 1 (p. 6) separately controls how the angle change and its fluctuations accumulate over many cycles. For the construction in Section 4, the missing entries remain separated as noise vanishes, while the observed entries and full Gram spectra agree exactly after final gradient flow. Section 6.1 (p. 8) tests the predicted separation: the predicted distance between missing blocks was 0.888, and the mean distance across independent runs was 0.891. Theorem 5 (Appendix C, p. 29) also proves separation for a fixed-data Gram problem in which minibatch sampling supplies all the noise.
Neural networks after the schedules end. Theorem 2 (p. 7) forecasts expected outputs after a fixed number of minibatch updates. Section 6.4 (p. 9) tests that forecast at update 512, where the change in class probabilities is 2.0–2.18 times the variation from repeating the same schedule. To find out whether that change lasts, we stopped varying the schedule at update 512 and continued both models with the same learning rate, batch size and minibatches. We then measured how much of the original probability change remained in its original direction (Appendix E.7, p. 42).
| Update | Fraction of the change that remains (95% interval) | Same-order control (95% interval) |
|---|---|---|
| 1,024 | 81% (74 to 88) | 0% (−3 to 3) |
| 2,048 | 62% (52 to 72) | 1% (−4 to 6) |
| 4,096 | 47% (34 to 60) | 1% (−5 to 7) |
These are averages across the four model and optimizer settings. About half of the original shift was still measurable after 3,584 further updates under identical training conditions. That is the persistence the neural experiment establishes. The vanishing-noise limit and exact equality of spectra are results for Gram completion.
W3. Large-model CR implementation.
Your 3.5 TB estimate is correct for the FP32 factor in the submitted code. The revised supplement includes a distributed implementation for both the 304M and 1.136B models. It stores each factor column in half precision and splits the columns across GPUs. For the 1.136B AdamW model at rank 256, each of 32 GPUs holds eight columns, or 54.5 GB of factor storage. New columns stay in CPU memory until compression. During compression, workers exchange blocks of rows, so no GPU holds the whole factor.
We checked rank-256 compression against a dense reference on two CPU workers. The resulting covariances differed by 0.029% in relative norm. The supplement's README gives the launch commands for both model sizes, and each run records peak memory on every GPU.
W4. Presentation and checking the results.
Thank you for pointing this out. The abstract had a typo. It now says CR lowers the median forecast error from 7.3–11.7% to 4.1–6.4% across the five regression families, as in Table 1 (p. 8).
To make the results easier to check, Table 12 (p. 37) lists every experiment with its runs and GPU-hours. The revised supplement has the code, the launch commands, the 80 checkpoint and loop pairs behind Table 14 (p. 40), and the script that scores a forecast. Every forecast run saves its configuration, seed, timing, peak memory and outputs, and the supplement includes these records from our implementation checks.
Q1. Implementation of the 304M and 1.136B forecasts.
Both sizes were run with the distributed implementation now in the supplement, described under W3. Each factor column is stored in half precision and the columns are split across GPUs. New columns, the weights and the AdamW moments stay in CPU memory. At rank 256 with AdamW:
| Model | GPUs | Factor columns per GPU | Factor storage per GPU |
|---|---|---|---|
| 304M | 8 | 32 | 58.4 GB |
| 1.136B | 32 | 8 | 54.5 GB |
Q2. Hardware, total hours, peak memory and full training runs.
Tables 11 and 12 (p. 37) now give this.
| Item | Value |
|---|---|
| Hardware, 304M models | One node with eight H100 80 GB GPUs |
| Hardware, 1.136B models | Four such nodes |
| All experiments | 12,631 GPU-hours |
| Full training, classifiers | 20 runs of 100 epochs, 1,886 GPU-hours |
| Full training, regression ViTs | 10 runs, 75 GPU-hours |
| Peak memory in the CR forecast | 51 to 75 GB per GPU, depending on the model and optimizer |
Table 12 splits the total by experiment and gives the number of runs for each.
Q3. Readability, terminology and notation.
We revised the paper for readability. Table 20 (pp. 47–48) lists every symbol in the main text with its type and where it is defined, and , , and now each have one meaning. Table 21 (p. 49) explains each term in plain words with its source. Table 19 (p. 46) shows how the three settings relate, Algorithms 1–3 give CR as pseudocode, and Table 22 (p. 50) gives their time and memory costs.
Thank you for your comments and feedback. We have addressed each of your concerns below and updated the paper accordingly. We apologize for the dense notation in the original version, and we hope the results and theory are now easier to see.
W1. The paper is difficult to follow.
Thank you for identifying the terms and transitions that caused this. We revised Sections 3–5 and the appendices so readers can trace how the formulas of Section 3 become a forecast.
Table 11 (p. 37) reports measured costs for all four model and optimizer settings. For ViT-1.1B with AdamW at rank 256: