acceptodds
Under review as a conference paper at ICLR 2027

Training Schedule Order Selects Isospectral Matrix Completions

Abstract

Training schedules can change predictions even when they use the same learning rates and noise levels equally often. We construct a rank-two Gram completion problem in which reversing a control cycle changes unobserved predictions while preserving training fit and the entire matrix spectrum after the same final noiseless gradient flow. The mechanism is covariance lag: fluctuations retain earlier controls, and Stokes' theorem expresses the slow-cycle response as curvature flux through the control loop. By canceling the exact drift produced by fixed controls, we prove persistent separation over many cycles, vanishing fluctuations and quantitative bounds for composing schedules, and certify the separation at the experimental noise level. We turn this response into Checkpoint Response (CR), a method for forecasting and directing schedule effects, and test it on matrix completion and billion-parameter neural models. Across regression families, CR reduces median forecast error from 7.3% to 11.7%, and in native classification, it reduces full-output forecast error by 49-53%.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
implicit bias, geometric phase, stochastic pumping, matrix completion, function selection, Stokes theorem

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1HdcOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether the ordering of training controls can affect the learned predictor even two schedules use the same learning rate and noise control pairs. The proposed mechanism is covariance lag, which means that stochastic fluctuations do not instantaneously equilibrate after a control change, so an update depends on covariance inherited from earlier controls. For slowly varying cyclic schedules, the resulting order-dependent response is expressed as a curvature flux through the control loop.

The theoretical contribution is mainly a rank-two Gram-completion construction. The observed principal blocks determine the within-block geometry but leave a relative rotation unobserved. The paper proves that a control loop and its reversal can select different missing cross-block predictions while achieving the same training fit and exactly the same complete Gram spectrum after a common noiseless relaxation. A repeated-cycle theorem establishes an O(1)O(1) limiting separation as the noise amplitude vanishes and the number of cycles increases.

The paper then introduces Checkpoint Response, which uses checkpoint derivatives and covariance propagation to forecast and design schedule-order effects. A separate finite-horizon expansion is developed for native minibatch noise, momentum, and AdamW. Experiments are reported on controlled Gram completion, large regression models, and 304M/1.136B-parameter ImageNet classifiers.

Critical strengths
  1. The exact Gram construction is conceptually interesting.

The paper demonstrates that training fit and the complete spectrum need not determine unobserved predictions. The use of relative block orientation gives a concrete functional degree of freedom that is invisible to standard spectral summaries.

  1. The matched-occupation comparison isolates order effects.

The forward and reversed schedules contain exactly the same joint control pairs instead of merely matching average learning rates or marginal control distributions. This makes the distinction between control exposure and control ordering precise.

  1. The covariance-lag interpretation is intuitive.

Connecting schedule order response to geometric pumping provides a potentially useful perspective on stochastic training dynamics.

Critical weaknesses
  1. The paper is difficult to follow.

The difficulty goes beyond the technical nature of the material. The manuscript introduces many non-standard terms and custom quantities, including matched occupation, current, curvature flux, relaxed endpoints, collateral changes, and several different response constructions.

Notation is also heavily overloaded. Symbols such as PP, MM, CC, and RR are reused for projectors, covariances, prediction arrays, cycle counts, transition operators, cross-block matrices, constants, rotations, and responses in different sections.

The paper would benefit from: a notation table; a hierarchy explicitly relating the three theoretical settings; pseudocode for CR; explicit time and memory complexity.

  1. The paper does not clearly separate three substantially different results.

The paper repeatedly moves between a local small-noise, one-cycle response expansion; a long-horizon isospectral selection theorem for a specialized Gram learner with injected noise and noiseless relaxation; a finite-horizon forecasting approximation for general neural networks with native minibatch noise.

These results have different assumptions and conclusions. In particular, the neural experiments do not establish persistent selection, exact isospectrality, or the long-horizon limit of the Gram theorem. They demonstrate relatively small local output changes after 512 continuation updates.

  1. The submitted code does not appear capable of producing the large-model CR results.

The submitted native-CR implementation concatenates the parameters and optimizer memory into a full vector. It then explicitly maintains a covariance factor of shape state dimension * r with default rank r=256. For the reported 1.136B-parameter AdamW model, this factor alone requires approximately $3(1.136\times10^9)\times256\times4 \approx 3.5$ TB in FP32. Even rank 64 requires approximately 873 GB. The supplied code uses a single device and contains no distributed factor representation, CPU offloading, or matrix-free compression that avoids materializing these tensors. Therefore, as supplied, the code does not appear runnable for the model sizes reported in the paper. If a different distributed or genuinely matrix-free implementation was used to obtain the results, it is not included or described.

  1. Several presentation details further reduce confidence.

The abstract states that CR “reduces median forecast error from 7.3% to 11.7%,” which is inconsistent as written. Table 1 suggests that the intended claim is a reduction from a range of 7.3%-11.7% to 4.1%-6.4%. This is likely a typo, but it occurs in a central quantitative claim. Together with the complexity of the experimental pipeline and the absence of raw artifacts, it contributes to the difficulty of independently checking the results.

Questions for authors
  1. What exact implementation produced the 304M and 1.136B CR forecasts? How was the parameter and optimizer-state covariance factor represented without requiring hundreds of gigabytes to several terabytes of memory?

  2. Please provide the hardware configuration, total hours, peak memory, and number of full training runs for each major experiment.

  3. Please try to modify the manuscript to make it more readable, especially in using standard terminology and clear notations.

3 comments

Thank you for your comments and feedback. We have addressed each of your concerns below and updated the paper accordingly. We apologize for the dense notation in the original version, and we hope the results and theory are now easier to see.

W1. The paper is difficult to follow.

Thank you for identifying the terms and transitions that caused this. We revised Sections 3–5 and the appendices so readers can trace how the formulas of Section 3 become a forecast.

  • Terminology. Section 3.1 (p. 3) writes out the forward and reverse orders. Both use every control pair equally often, and we now call such schedules matched instead of "matched occupation". The main text no longer uses "relaxed" and says "after final gradient flow" directly. We removed "current" and "collateral changes" as technical terms, and the introduction no longer uses "flux". The second derivative of an output is now the output curvature, and the integrand of the loop integral is the control-plane curvature. We use "response" only for half the difference in expected outputs between the two orders. Table 21 (p. 49) gives each term's meaning and origin.
  • Notation. PP denotes the observed Gram block, MM counts repeated cycles, CC with a subscript denotes a constant, and R(ϕ)R(\phi) denotes a rotation. The missing block is ZZ, class-probability arrays are Prob\mathrm{Prob}, the tangent projector is DΠ(θ0)D\Pi(\theta_0), and the covariance propagation operator is Γ\Gamma. Section 3.1 points to Table 20 in Appendix G (pp. 47–48), which gives each symbol's meaning, type and place of definition.
  • How the three settings fit together. Section 3 studies one cycle. Each Gram block in Section 4 is an instance of that setting. Theorem 1 gives the additional assumptions needed to repeat the cycle. Section 5.1 treats ordinary minibatch training. Its covariance recursion, Eq. (16), reduces to Eq. (2) for the residual dynamics at a fixed checkpoint. Table 19 (p. 46) links each setting to its algorithm and experiments.
  • From the formulas to CR. Algorithm 1 in Section 5 (p. 6) shows how Eqs. (2)–(3) produce a forecast. At each update, CR uses the current covariance to compute the expected output change, then advances the covariance. It sums the changes for each order and takes half their difference. Algorithms 2 and 3 in Appendix G (p. 49) give the steps for schedule design and minibatch forecasting. Each step cites the equation it uses.
  • Time and memory. Table 22 (p. 50) gives the cost by step. With neign_{\mathrm{eig}} retained modes and dGd_G outputs, Algorithm 1 takes O(NdGneig2)O(N d_G n_{\mathrm{eig}}^2) time per NN-step cycle after checkpoint setup. For pp parameters, it uses O(pneig+(1+dG)neig2)O(p n_{\mathrm{eig}}+(1+d_G)n_{\mathrm{eig}}^2) storage beyond the model and derivative workspace. Algorithm 2 reuses the checkpoint derivatives, forecasts each candidate loop once, solves Eq. (15), and makes one pass to write the assembled schedule. It stores one forecast vector per candidate, along with the solver workspace and the schedule. Algorithm 3 makes one TT-update pass and one final output evaluation per order. Its covariance factor stores one optimizer-state vector per factor column.

Table 11 (p. 37) reports measured costs for all four model and optimizer settings. For ViT-1.1B with AdamW at rank 256:

QuantityMeasured
CR forecast, per checkpoint0.74 GPU-hours
Each additional schedule0.027 GPU-hours
Direct measurement of one schedule pair, 16 runs per order10.7 GPU-hours
Peak memory per GPU75 GB

W2. Distinguishing the three results.

Section 3.1 (p. 3) and Table 19 (p. 46) now state the assumptions, conclusion and experiment for each result. We also extended the neural experiment to answer the question left open at update 512.

One cycle. Section 3 asks what happens when two runs start at the same checkpoint and visit the same control pairs in opposite orders. The covariance takes time to adjust as the controls change. For one slow cycle with small injected noise, Eq. (7) (p. 4) gives the resulting difference in expected outputs after final gradient flow, a last noise-free stage that removes the training error. This calculation covers one cycle.

Repeated cycles in Gram completion. Theorem 1 (p. 6) separately controls how the angle change and its fluctuations accumulate over many cycles. For the construction in Section 4, the missing entries remain separated as noise vanishes, while the observed entries and full Gram spectra agree exactly after final gradient flow. Section 6.1 (p. 8) tests the predicted separation: the predicted distance between missing blocks was 0.888, and the mean distance across independent runs was 0.891. Theorem 5 (Appendix C, p. 29) also proves separation for a fixed-data Gram problem in which minibatch sampling supplies all the noise.

Neural networks after the schedules end. Theorem 2 (p. 7) forecasts expected outputs after a fixed number of minibatch updates. Section 6.4 (p. 9) tests that forecast at update 512, where the change in class probabilities is 2.0–2.18 times the variation from repeating the same schedule. To find out whether that change lasts, we stopped varying the schedule at update 512 and continued both models with the same learning rate, batch size and minibatches. We then measured how much of the original probability change remained in its original direction (Appendix E.7, p. 42).

UpdateFraction of the change that remains (95% interval)Same-order control (95% interval)
1,02481% (74 to 88)0% (−3 to 3)
2,04862% (52 to 72)1% (−4 to 6)
4,09647% (34 to 60)1% (−5 to 7)

These are averages across the four model and optimizer settings. About half of the original shift was still measurable after 3,584 further updates under identical training conditions. That is the persistence the neural experiment establishes. The vanishing-noise limit and exact equality of spectra are results for Gram completion.

W3. Large-model CR implementation.

Your 3.5 TB estimate is correct for the FP32 factor in the submitted code. The revised supplement includes a distributed implementation for both the 304M and 1.136B models. It stores each factor column in half precision and splits the columns across GPUs. For the 1.136B AdamW model at rank 256, each of 32 GPUs holds eight columns, or 54.5 GB of factor storage. New columns stay in CPU memory until compression. During compression, workers exchange blocks of rows, so no GPU holds the whole factor.

We checked rank-256 compression against a dense reference on two CPU workers. The resulting covariances differed by 0.029% in relative norm. The supplement's README gives the launch commands for both model sizes, and each run records peak memory on every GPU.

W4. Presentation and checking the results.

Thank you for pointing this out. The abstract had a typo. It now says CR lowers the median forecast error from 7.3–11.7% to 4.1–6.4% across the five regression families, as in Table 1 (p. 8).

To make the results easier to check, Table 12 (p. 37) lists every experiment with its runs and GPU-hours. The revised supplement has the code, the launch commands, the 80 checkpoint and loop pairs behind Table 14 (p. 40), and the script that scores a forecast. Every forecast run saves its configuration, seed, timing, peak memory and outputs, and the supplement includes these records from our implementation checks.

Q1. Implementation of the 304M and 1.136B forecasts.

Both sizes were run with the distributed implementation now in the supplement, described under W3. Each factor column is stored in half precision and the columns are split across GPUs. New columns, the weights and the AdamW moments stay in CPU memory. At rank 256 with AdamW:

ModelGPUsFactor columns per GPUFactor storage per GPU
304M83258.4 GB
1.136B32854.5 GB

Q2. Hardware, total hours, peak memory and full training runs.

Tables 11 and 12 (p. 37) now give this.

ItemValue
Hardware, 304M modelsOne node with eight H100 80 GB GPUs
Hardware, 1.136B modelsFour such nodes
All experiments12,631 GPU-hours
Full training, classifiers20 runs of 100 epochs, 1,886 GPU-hours
Full training, regression ViTs10 runs, 75 GPU-hours
Peak memory in the CR forecast51 to 75 GB per GPU, depending on the model and optimizer

Table 12 splits the total by experiment and gives the number of runs for each.

Q3. Readability, terminology and notation.

We revised the paper for readability. Table 20 (pp. 47–48) lists every symbol in the main text with its type and where it is defined, and PP, MM, CC and RR now each have one meaning. Table 21 (p. 49) explains each term in plain words with its source. Table 19 (p. 46) shows how the three settings relate, Algorithms 1–3 give CR as pseudocode, and Table 22 (p. 50) gives their time and memory costs.

Reviewer_hocGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks whether schedule order can change predictions even when each control setting is used equally often. It compares forward and reversed schedules starting from the same checkpoint. In the main construction, the controls are learning rate and label-noise level.

The authors construct a rank-two Gram completion problem. The observed blocks fix their individual geometry but leave their relative rotation free. Reversing the schedule changes this rotation and the missing-entry predictions. After the same final noiseless relaxation, both endpoints have identical training fit and the same full matrix spectrum. The paper explains this through covariance lag. Fluctuations retain information about earlier controls, which affects later prediction changes. The analysis also studies repeated, increasingly slow cycles. Under its assumptions, it proves a nonzero prediction gap and vanishing fluctuations in a small-noise limit.

The paper develops Checkpoint Response (CR) from this analysis. CR uses checkpoint derivatives and covariance propagation to forecast the mean output difference between forward and reversed schedules. It combines candidate loop responses to design schedules for a chosen target difference. A separate finite-horizon analysis covers ordinary minibatch noise and optimizer memory.

Experiments include matrix completion, neural regression, and ImageNet classification with momentum SGD and AdamW. Forecasting and schedule design are compared under equal computational budgets. Across the regression families, CR reduces median forecast errors from 7.3%–11.7% for the calibration baseline to 4.1%–6.4%. In classification, it reduces full-output forecast errors by 49%–53%. The experiments also measure target attainment and changes outside the target outputs. The exact spectral identity applies to the Gram construction. Neural experiments test approximate preservation of the output spectrum.

Critical strengths

S1: The paper states a clear research question about schedule order.

S2: The response analysis motivates CR and explains the mechanism in the Gram example.

Critical weaknesses

W1: The presentation is hard to follow. The dense notation makes the analysis and method difficult to track, particularly the transition from the response formulas in Section 3 to the CR procedure in Section 5.

W2: The experiments show control over prescribed prediction changes. The connection between these targets and a concrete practical objective remains unclear.

W3: The neural forecast error bounds in Table 8 are difficult to verify. Appendix F.5 gives limited detail on how each error term is numerically bounded. It is also unclear how the conditions of Corollary 3 are verified in these experiments.

W4: The paper reports computational costs in work units, without runtime or peak memory measurements. This makes the practical cost of CR on larger models difficult to assess.

Questions for authors

Q1: The abstract says that CR reduces forecast error from 7.3% to 11.7%. Table 1 reports calibration errors of 7.3%–11.7% and CR errors of 4.1%–6.4%. Please correct this statement.

Q2: Is controlling the output difference between forward and reversed schedules enough to move a single model toward a specified target?

Q3: How are the neural error bounds in Table 8 computed and validated? Please provide one worked calculation. How are all reachable states and approximation errors covered?

3 comments

Thank you for your comments and feedback. We have addressed each of your concerns below and updated the paper accordingly. We apologize for the dense notation in the original version, and we hope the results and theory are now easier to see.

W1. Presentation. "The dense notation makes the analysis and method difficult to track, particularly the transition from the response formulas in Section 3 to the CR procedure in Section 5."

We rewrote Sections 3 to 5 around these two points.

Notation. We renamed the symbols that had several meanings, and each symbol is now defined before it is used. Section 3.1 (p. 3) introduces the setting in order: the residual, the control pair, a schedule, the forward and reverse orders, and the outputs we measure. Appendix G (pp. 46–49) lists the symbols in Table 20, explains the terms in Table 21, and compares the three settings of the paper in Table 19.

The response formulas. Section 3.2 (pp. 3–4) now derives each formula in turn. Eq. (2) follows from linearizing the update, Eq. (3) from the second-order term, and Eq. (4) is the fixed point of Eq. (2). We then explain why reversing the order changes the sign of the covariance lag, which gives Eq. (5).

From Section 3 to Section 5. Section 3.2 ends by saying how its results are used: Section 5 applies Eqs. (2)–(3) step by step. Algorithm 1 in Section 5 (p. 6) starts from the three quantities these equations need at a checkpoint: the eigenpairs of K, the output curvatures, and the noise covariance. It then gives the forecast as numbered steps. Algorithm 2 (Appendix G, p. 49) does the same for schedule design, and each step cites the equation it uses. Section 5.1 (p. 7) states how ordinary minibatch training differs from Section 3 and gives its recursion, Eq. (16), before Theorem 2.

We also added Figure 1 (p. 2), which shows the main result before the analysis begins.

W2. Practical objective. "The experiments show control over prescribed prediction changes. The connection between these targets and a concrete practical objective remains unclear."

In Section 6.3, a target is a prescribed change in a few output coordinates. We chose those targets to test the forecast in every direction of output space, because a method that can steer toward any direction can steer toward a useful one. The revision now shows this step directly, in theory and in experiments.

First, an objective determines a target. Take any smooth objective J of the model outputs, such as cross-entropy or the Brier score. The outputs lower J fastest by moving along −∇J. Proposition 9 (Appendix F, p. 43) shows that when the target in Eq. (15) is a step in this direction, Algorithm 2 minimizes an upper bound on J, and the forecast error limits how much of the decrease can be lost. For held-out cross-entropy on ImageNet, the proposition guarantees a decrease of at least 0.006 nats, and we measure 0.011.

Second, we test four objectives on held-out data (Section 6.5, Table 4, p. 10, and Table 16, p. 42). Every number comes from single trained models. For cross-entropy and the Brier score, the baseline is Bayesian schedule search, the search baseline from Table 2. For the other two rows, the baseline is the order that performs best on development data.

ObjectiveBaselineGain with CRPaired 95% interval
Cross-entropy, ImageNetBayesian schedule search0.007 nats lower0.004 to 0.010
Brier score, ImageNetBayesian schedule search0.0019 lower0.0008 to 0.0030
Correct answers at a matched wrong-answer rate, ImageNetBest fixed order2.1 more per 100 images1.2 to 3.0
Next-token loss, scientific textBest fixed order0.020 nats per token lower0.008 to 0.032

The ImageNet rows are at equal total GPU-hours: the baseline spends the time CR uses for forecasting on extra training. The language-model row is at equal training length. The third row counts answers from a classifier that responds only when its confidence passes a threshold set on development data. Because a confidence gain could come from recalibration alone, we also apply temperature scaling, a standard recalibration step, to both models. The gain remains 1.8 correct answers per 100 images (0.9 to 2.7). Top-1 accuracy moves by less than 0.1 percentage points in every row, so the gains come from the probabilities and not from changed predictions. For cross-entropy, CR beats the baseline at 17 of 20 checkpoints and reaches 76% of the gain obtained by running every candidate schedule and keeping the best.

Finally, the test targets and the practical objectives are linked in the data. Checkpoints where CR forecasts the Section 6.3 targets more accurately also show larger decreases in cross-entropy (Spearman correlation −0.68 between forecast error and decrease, Appendix E.8, p. 43).

W3. Verifying the bounds. "Appendix F.5 gives limited detail on how each error term is numerically bounded. It is also unclear how the conditions of Corollary 3 are verified in these experiments."

Corollary 3 bounds the forecast error by adding three kinds of error, provided that training stays inside a region around the full-gradient reference run. Appendix E.5 (pp. 39–40) now states how we compute each term for the networks in Table 14, works one bound out in full, and tests the procedure against a case where every step is certified.

The three terms. The initial term is the error from computing the forecast at finite rank. We read it from the residuals of the two recursions in Eq. (16). The per-update term measures how far one training update departs from the quadratic approximation behind the forecast. It has two sources: the curvature of the update, and the change in minibatch-noise covariance across the region. The final-output term is the curvature of the map from the final state to the class probabilities. The networks use GELU and softmax, so all of these derivatives exist.

The region and the maxima. At update k the region is a ball around the reference state. Its radius is twice the largest deviation among the 16 runs in each order. Each curvature and covariance term is a maximum over this ball. We search for it at 64 points, which are the sampled states and points found by gradient ascent on the quantity itself, and multiply the largest value by 1.5.

One bound in full. ViT-L with SGD, checkpoint 5, ellipse loop (row 5.1 of Table 14; units of 10⁻³ in the per-image probability norm):

TermForwardReverse
Initial0.310.29
Per-update, curvature1.471.49
Per-update, covariance2.452.47
Final output0.420.40
Total4.654.65

Corollary 3 takes half the sum of the two totals, so the bound is 4.65. The forecast has norm 34.1, and the relative bound is 4.65 / (34.1 − 4.65) = 15.8%, the entry in Table 14. The measured error is 9.5%. Table 14 (p. 40) gives the bound and the measured error for all 80 pairs.

Testing the procedure. We test the radius and the factor 1.5 in three ways.

  • Against a certified case (Appendix E.6, Table 15, p. 40). For a small tanh network with a fixed set of minibatches, interval arithmetic gives guaranteed enclosures of every reachable state and every term, as in Appendix D. The certified bound is 24.6%, our procedure gives 19.8%, and the measured error is 9.1%. The maxima found by search reach 88–97% of the certified ones, well inside the factor 1.5.
  • On runs not used to set the regions. None of 1,280 fresh runs on the large models leaves its region. The largest deviation is 0.58 of the radius.
  • Against measured error. In all 80 pairs, the measured error plus its sampling radius is below the bound.

W4. Runtime and memory. "The paper reports computational costs in work units, without runtime or peak memory measurements."

Appendix E.3 (Table 11, p. 37) now reports measured runtime and peak memory. One work unit is one forward and backward pass on one training image, so 262,144 units equal one 512-update continuation at batch size 512.

Model / optimizerOne continuation (GPU-hours)CR forecast, per checkpoint (GPU-hours)Direct measurement of one schedule pair (GPU-hours)Peak memory per GPU (GB)
ViT-L / SGD0.070.152.351
ViT-L / AdamW0.080.172.670
ViT-1.1B / SGD0.310.6610.056
ViT-1.1B / AdamW0.330.7410.775

ViT-L runs on one node with eight H100 80 GB GPUs, and ViT-1.1B on four such nodes. The forecast costs about twice one continuation and is computed once per checkpoint. Each further schedule adds about 8% of one continuation. A direct measurement uses the 16 runs per order of our evaluation, which is 32 continuations for one schedule pair. The forecast does not use the full-gradient runs. Those are needed only for scoring (Section 6.4). Memory is dominated by the rank-256 factor, whose size is the optimizer-state size times the rank. We store it in half precision across the GPUs. The largest, for ViT-1.1B with AdamW, is 1.75 TB. Runtime and memory grow linearly with model size and rank. Table 12 (p. 37) lists the runs and GPU-hours for every experiment, about 12,600 GPU-hours in total.

Q1. Abstract.

Thank you for pointing this out. The abstract now gives both ranges from Table 1 (p. 8). CR lowers the median forecast error from 7.3–11.7% to 4.1–6.4%.

Q2. Moving a single model toward a target.

Yes. Every continuation runs its control pairs in some order, so choosing the order is a choice about one model. If the reverse order gives expected outputs oo, the forward order gives o+do + d, with the same learning rates, batch sizes and number of updates. Controlling dd therefore controls where the single trained model ends. We report the difference between the two orders because the change that both orders share cancels, which isolates the effect of order.

Four results apply to one model directly.

Forecast. Theorem 2 (p. 7) forecasts the expected output of each order separately (Eq. (17)). The forecast of the difference is built from these two single-order forecasts.

Guarantee. Proposition 9 (Appendix F, p. 43) bounds the objective of the model trained in the forward order. For cross-entropy it guarantees a decrease of at least 0.006 nats, and the measured decrease is 0.011.

Measured outcomes. Section 6.5 scores single trained models. The model trained in the order chosen by CR gives 2.1 more correct answers per 100 images than the model trained in the best fixed order, at equal GPU-hours (Table 4, p. 10). For language models, when each order is run once, CR's choice has the lower loss in 30 of 40 pairs (Table 18, p. 45).

Prescribed targets. We added a single-run check for the targets of Section 6.3 (Appendix E.3, p. 36). For each of the 200 classification targets, we take one model trained in the designed order. Its output change points toward the target in 196 of 200 cases, with a median cosine of 0.92, where 1 means exactly along the target.

Q3. Error bounds.

Our reply to W3 answers this question in full. It describes how each term of the bound is computed, works one bound out step by step, and reports three tests of the procedure. The paragraphs on the region and on testing explain how reachable states and approximation errors are covered. The same material is in Appendix E.5 and E.6 (pp. 39–40).

Reviewer_BqMzOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies an interesting optimization bias problem, which you train with the same learning rate and noise levels, the train order change will change the predictions. It constructs a matrix completion problem where reversing a control loop selects different missing entries despite identical training fit and complete spectrum after noiseless relaxation. The authors proposes covariance lag and establishes its persistence over repeated cycles. A resulting Checkpoint Response method is proposed.

Critical strengths

I am not expert on this, but the problem is interesting and the approach here is clever.

  1. The controlled theoretical construction is compelling and interesting to me. The repeated-cycle analysis is particularly valuable.

  2. Move beyond the toy example to some real training and show the value.

Critical weaknesses
  1. What is the practical value of this study would be? Like the schedule steering didn't change the accuracy too much like in the ViT case, and it seems only change the probability outputs. I wonder whether the authors could share more on the practical value.

  2. Another question would be how we can extend the theory for larger model like LLMs.

Questions for authors
  1. It seems the abstract " median forecast error from 7.3% to 11.7%" didn't match with the experimental part (Table 1?)

  2. Schedule effects exceed continuation noise, but remain below seed variation. Is it bad?

1 comment

Thank you for your positive comments on the repeated-cycle result. Your questions helped us clarify the following, and we have accordingly updated the paper.

  1. Practical value. “I wonder whether the authors could share more on the practical value.”

Reversing the schedule changes the ViT’s predicted probabilities even when the predicted class stays the same. Consider a classifier that refers uncertain images to a human. Both schedules can give the same prediction, yet only one assigns it enough confidence for the classifier to answer. The practical question is how many more images it answers correctly and how many additional mistakes it makes. Section 6.5 (Table 4, p. 10) compares Checkpoint Response (CR), which chooses the order from the starting checkpoint, with always using the best fixed order. We select the fixed order and the confidence thresholds on development data. Per 100 held-out ImageNet images, the comparison at equal total GPU-hours gives 2.1 more correct answers with CR (paired 95% interval, 1.2 to 3.0) and 0.02 more wrong answers (−0.08 to 0.12). In this comparison, the fixed-order baseline spends CR’s forecasting time on extra training. At equal continuation length, the differences are 2.5 more correct answers and 0.04 more wrong answers per 100 images.

  1. Extending the theory to LLMs. “Another question would be how we can extend the theory for larger model like LLMs.”

Suppose we have one pretrained language model and enough compute for one continuation on scientific text, and we must choose the schedule. Two schedules can predict the same next token but assign different probabilities to the token that actually occurs. Held-out next-token loss captures that difference.

Appendix F.1 (Table 18, p. 45) compares four schedule pairs at each of five checkpoints for each model size, 1.4B and 6.9B parameters. Each schedule and its reverse use the same starting weights, AdamW state, training tokens, and learning-rate/batch-size pairs. CR chooses the order before either continuation begins. At equal training length, the mean gain over the best fixed order from development data is 0.020 nats per token on scientific text, about 2% lower perplexity. The paired 95% interval is 0.008–0.032 nats per token (Table 16, p. 42) and accounts for the four comparisons sharing each checkpoint. The upper confidence bound on the increase in general-text loss is below the prespecified tolerance of 0.005 nats per token. CR selects the order with lower mean loss in 34 of 40 comparisons, versus 25 for an equal-cost pilot.

We also consider when the model is confident enough to offer an autocomplete suggestion (Section 6.5, Table 4). Across 1.2 million positions in 400 held-out documents, CR gives 205 correct suggestions per 1,000 positions, compared with 180 for the fixed order. Wrong suggestions are 10.2 versus 9.9 per 1,000. Top-1 accuracy differs by less than 0.1 percentage point. The theory explains why order can affect next-token loss and when the forecast is reliable enough to choose an order. Proposition 10 shows that minibatch noise changes the loss in two ways. It shifts the average score of each token, and it makes the scores fluctuate, which can only raise the loss. Reversing the schedule can change both effects (Appendix F, p. 44).

For the decision itself, Proposition 11 shows that if the forecast error is at most BB, choosing the predicted order loses at most max⁡(0,B−∣forecast∣)\max(0, B-|\text{forecast}|) in expected loss relative to choosing the better order. At one checkpoint, CR favors the forward order by 0.050 nats per token. Subtracting the estimated error bound of 0.008 gives a lower bound of 0.042 nats per token for that choice (Table 17, p. 44). Averaging the checkpoint bounds gives a lower bound on CR’s gain over a fixed order. Of the 40 forecast intervals, 23 exclude zero (Table 18).

  1. Forecast errors in the abstract.

Thank you for pointing out the discrepancy. The revised abstract (p. 1) gives the two ranges separately. Across the five regression families, median forecast errors range from 7.3% to 11.7% for rational calibration and from 4.1% to 6.4% for finite-step CR (Section 6.2, Table 1, p. 8).

  1. Seed variation. “Schedule effects exceed continuation noise, but remain below seed variation. Is it bad?”

No. For someone resuming a checkpoint, the starting weights are already fixed. The relevant comparison is between reversing the schedule and repeating the same schedule from those weights. Section 6.4 (p. 9) and Appendix E.3 (p. 36) measure both on the same scale. For the change in class probabilities studied there, the order effect is 2.0–2.18 times the variation from repeating the same schedule.

The practitioner can choose the schedule in advance, and CR predicts which direction to take. To test a single run directly, Table 18 also evaluates CR’s choice on a fresh pair of continuations. The chosen order gives lower loss in 30 of 40 pairs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.