acceptodds
Under review as a conference paper at ICLR 2027

When Historical Energy Is Insufficient for Local Training Response

Abstract

A training trajectory can be nearly one-dimensional in historical energy, yet discarding its weak second direction can lose an order of local prediction accuracy. For smooth fixed-loss gradient descent over fixed finite windows, we characterize the necessary and sufficient gradient and curvature capture rates for preserving the unrestricted linearized model's normalized parameter-error guarantee. With nonzero gradient and transverse curvature, even the best line through the anchor, fixed across the window and chosen using the complete future, has sharp error. Both the gradient-curvature plane and the exact noiseless historical top-two space attain error. History therefore contains a sufficient response space, but every fixed explained-energy threshold below 100% eventually discards its necessary second direction, whose energy fraction vanishes quadratically. Multi-scale tests in selected LoRA coordinates of eight LLM model-data blocks support the predicted orders and assess the sharp best-line coefficient, with model-dependent finite-scale deviations. Across four families, matched experiments locate over 96% of the one-direction predictor's squared residual in a direction carrying under 2% of future update energy (family medians). A varying-driver motion law separates driver and curvature contributions to historical compression; shuffled-AdamW diagnostics examine direction selection beyond fixed-loss GD.

Primary area:
optimization
Keywords:
optimization dynamics, trajectory compression, low-rank trajectories, local training response, Hessian curvature, reduced-order modeling, large language models, LoRA

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_RUgWOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper characterizes when historical trajectory compression preserves local prediction accuracy under fixed-loss gradient descent. It establishes necessary and sufficient capture conditions, a sharp one-dimensional error bound, and two-dimensional sufficiency, showing why fixed energy thresholds discard necessary information. Controlled multi-scale experiments in selected LLM LoRA coordinates test the predicted error orders and coefficients.

Critical strengths
  1. Precise theoretical characterization. Necessary and sufficient capture conditions, a sharp optimum over anchored fixed lines, and exact-history two-dimensional sufficiency clarify why retaining historical energy can fail to preserve local prediction accuracy.
  2. Direct empirical validation. Matched-space comparisons isolate compression effects, while multi-scale experiments test predicted error orders and coefficients. These evaluations connect the theoretical claims to measured behavior rather than relying solely on aggregate performance improvements.
Critical weaknesses
  1. Limited demonstrated significance beyond local trajectory fidelity. Preserving the local model’s accuracy order has a clear numerical motivation, supported by the matched experiments. However, its importance for learning quality remains less established. Appendix G.1’s oracle historical-projection diagnostics report a median NLL gain of 0.00000425 nats/token and a median recovery ratio of 0.208%, with continuation effects also below the stated materiality threshold. These findings do not undermine the theory, but limit the demonstrated significance of the accuracy-order distinction beyond parameter prediction.
  2. Limited transfer to practical compression criteria. The guarantees concern fixed-loss, stateless GD over windows that shrink in optimization time, and historical sufficiency assumes noiseless updates. The required accuracy for recovering the weakening second direction from perturbed histories remains uncharacterized. The shuffled-AdamW and selective-extrapolation studies provide useful supporting evidence, but require additional information and computation; Appendix F.2 finds no favorable accuracy–cost window for the tested reduced predictors. Consequently, the results offer limited guidance for reliable, cost-justified compression outside the exact local regime.
Questions for authors
  1. Finite-scale implications. Within the fixed-loss setting, could you use the existing best-line bounds and K2/HistSVD2 errors to identify normalized parameter-error tolerances that a two-dimensional predictor satisfies but no anchored, window-fixed one-dimensional predictor can satisfy?
  2. Sensitivity to imperfect history. Keeping the anchor gradient and Hessian exact, could you provide a sufficient perturbation bound for the normalized history matrix (the update matrix divided by the learning rate times the initial gradient’s Euclidean norm), such that its recomputed top-two subspace preserves Corollary 3’s capture rates?
3 comments
AuthorsOpenReview ↗

Response to Reviewer RUgW

1. Does the separation hold at a concrete finite tolerance?

Yes. At the finest tested scale, a mean relative parameter-error tolerance of 0.05% separates both K2 and HistSVD2 from every anchored, window-fixed line on each of the eight model–data blocks. The comparison allows the line to see the full future and to use a different coefficient at every horizon.

For a recorded trajectory, let LdisjL_{\mathrm{disj}} be the disjoint-pair lower bound on the optimal line error (Appendix C.2). If max⁡(EK2,EHistSVD2)≤ε<Ldisj\max(E_{\mathrm{K2}},E_{\mathrm{HistSVD2}})\le\varepsilon<L_{\mathrm{disj}}, both two-dimensional forecasts meet the tolerance and every line misses it. All errors use the same true displacement at each horizon to normalize D0D_0. Joining the two released Figure 3 tables by condition gives these examples at P=8,H=24P=8,H=24 and δ=η∥g0∥\delta=\eta\|g_0\|. Each row uses the largest error of either two-dimensional method and the smallest lower bound across all eight blocks.

Initial step length δ\deltaLargest two-dimensional errorSmallest line lower boundA common tolerance
0.0018750.000151410.000843270.0005
0.0037500.000591160.001679360.0010
0.0075000.002180600.003329480.0025

Across all five scales, the paired ratio Ldisj/max⁡(EK2,EHistSVD2)L_{\mathrm{disj}}/\max(E_{\mathrm{K2}},E_{\mathrm{HistSVD2}}) is at least 1.68 in every one of the 40 conditions (eight blocks, five nested scales). At δ=0.03\delta=0.03, for example, a common tolerance of 1% works on all four Llama blocks and 2% on all four Qwen blocks. These are retrospective guarantees for the observed trajectories, since the lower bound uses their future directions. As η\eta tends to zero even the best-line error tends to zero, so the tolerance separating the accuracy orders must also shrink.

2. How accurate must an imperfect prefix be?

For A=U/(η∥g0∥)A=U/(\eta\|g_0\|) and A~=A+N\widetilde A=A+N, ∥N∥2=O(η2)\|N\|_2=O(\eta^2) in operator norm suffices for arbitrary additive perturbations, with the anchor gradient and Hessian kept exact as in your question. Let Π~\widetilde\Pi be the top-two right singular projector of A~\widetilde A and e=∥N∥2e=\|N\|_2. The rank-two residual and Weyl's inequality give ∥A(I−Π~)∥2≤e+σ3(A~)≤σ3(A)+2e=O(η2)+2e.\|A(I-\widetilde\Pi)\|_2\le e+\sigma_3(\widetilde A)\le \sigma_3(A)+2e=O(\eta^2)+2e . This bounds every row residual. In column notation let bt=ut/(η∥g0∥)b_t=u_t/(\eta\|g_0\|) and s=∥ga∥/∥g0∥→1s=\|g_a\|/\|g_0\|\to1. Expanding the last two updates gives 2bP−bP−1=sqa+O(η2)2b_P-b_{P-1}=s q_a+O(\eta^2) and (bP−1−bP)/η=sHaqa+O(η)(b_{P-1}-b_P)/\eta=s H_aq_a+O(\eta). Hence the distances of qaq_a and HaqaH_aq_a to the estimated plane are O(η2+e)O(\eta^2+e) and O(η+e/η)O(\eta+e/\eta), respectively. With e=O(η2)e=O(\eta^2) these are precisely the two capture rates in Theorem 1. The matched rollout retains D0=O(η2)D_0=O(\eta^2). In raw units the sufficient bound is ∥U~−U∥2=O(η3∥g0∥)\|\widetilde U-U\|_2=O(\eta^3\|g_0\|).

The distinction from simply resolving the second singular value matters. Take an exact rank-two quadratic trajectory embedded in three dimensions and apply the same orthogonal rotation to every observed update. The rotation tilts qaq_a by ϕ\phi toward the unused dimension and fixes the in-plane direction perpendicular to it. This preserves every singular value of the prefix. Yet the first true update points along the unrotated qaq_a, so even its best relative error in the rotated plane is ∣sin⁡ϕ∣|\sin\phi|, and D0≥∣sin⁡ϕ∣/HD_0\ge|\sin\phi|/H. The prefix has an order-one projection onto qaq_a, so ∥N∥2=Θ(∣ϕ∣)\|N\|_2=\Theta(|\phi|). Choosing ϕ=η3/2\phi=\eta^{3/2} gives noise smaller than σ2(A)=Θ(η)\sigma_2(A)=\Theta(\eta) that still loses second-order accuracy. Even an unchanged energy spectrum can conceal a tilt that matters for the forecast.

Dear Reviewer,

We’re nearing the end of the discussion period. This is a friendly reminder to leave a comment for the authors.

Thank you

~ AC

I thank the authors for their response. While both answers are informative, they reinforce rather than resolve my two primary concerns, and I am lowering my rating accordingly.

Q1 (Finite-scale tolerance): As the authors acknowledge, the separating tolerances (0.05%–0.25% mean relative parameter error) are retrospective: the line lower bound relies on observed future directions, and the separating tolerance necessarily shrinks to zero as η→0\eta \to 0. More importantly, no connection is established between these parameter-error tolerances and learning quality. Given the median held-out NLL gain of 4.25×10−64.25\times10^{-6} nats/token in Appendix G.1, along with the response to WPQ3 (the plane yields 1.8% of the line's NLL gain at a 9–17% higher computational cost, and under shuffled AdamW, the accurate predictor is 1.62$\times$ slower than direct training), the distinction in accuracy order still lacks demonstrated practical significance.

Q2 (Imperfect history): In normalized units, the sufficient condition ∥N∥2=O(η2)\Vert{}N\Vert{}_2 = O(\eta^2) (O(η3∥g0∥)O(\eta^3 \Vert{}g_0\Vert{}) in raw units) is one order of magnitude below σ2(A)=Θ(η)\sigma_2(A) = \Theta(\eta), and the authors' own construction demonstrates that a Θ(η3/2)\Theta(\eta^{3/2}) perturbation already destroys second-order accuracy. Under minibatch sampling, each normalized update carries gradient-noise perturbations of order ∥ξt∥/∥g0∥\Vert{}\xi_t\Vert{}/\Vert{}g_0\Vert{}, which do not vanish as η→0\eta \to 0. Consequently, the guarantee does not appear to cover any realistic stochastic training history, which is consistent with the batch-variance floor in Proposition 1. This further confirms that the results offer little guidance for practical compression criteria outside the exact, noiseless, fixed-loss regime.

Furthermore, the main derivation for Q2 failed to render properly in the posted comment (inline LaTeX is merged with the body text), preventing me from verifying it step by step.

Overall, while the theory is precise within its defined setting, the rebuttal makes it clearer that both its practical relevance (W1) and its applicability beyond noiseless, fixed-loss gradient descent (W2) are severely limited. I am lowering my rating to 1.

Reviewer_v4gcOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how the number of singular vectors retained when compressing training trajectories affects the approximation error.

Critical strengths

The paper mathematically proves that retaining the top two singular vectors is necessary to achieve a second-order error guarantee.

Critical weaknesses
  1. Although I am not an expert in this area, the paper appears to provide an in-depth analysis of the largely expected trade-off between the retained rank and approximation quality when using methods such as SVD to compress training trajectories.

  2. The conclusion that a rank-two approximation should be used, based on this analysis, also appears to offer a limited technical contribution.

Questions for authors

Have prior studies examined how performance varies with the retained rank in similar compression tasks?

2 comments
AuthorsOpenReview ↗

Response to Reviewer v4gc

Yes. Two rank sweeps in Section 8 illustrate why the choice of rank needs an accuracy target. Wei et al.'s RELEX uses the first 75 training steps of Qwen2.5-Math-1.5B to forecast step 500. Keeping 1, 5, or 10 temporal directions per weight tensor gives MATH accuracies of 71.6%, 70.6%, and 70.5%. Li et al. vary the dimension for projected SGD. On CIFAR-10 with 70% label corruption, increasing it from 10 to 20 raises test accuracy from 68.92% to 77.00%, with 77.22% at dimension 40. In a different setting, Ilak and Rowley showed that a mode with only 0.025% of the snapshot energy helped recover transient energy growth in channel flow. RELEX, Table 2 · Li et al., preprint Table 4 · Ilak and Rowley

Theorem 2 gives a one-dimensional predictor access to the entire future. It can choose the best line through the anchor and a separate coefficient at every horizon. Even with these choices, its minimum mean relative parameter error is cH∗β0η+O(η2)c_H^*\beta_0\eta+O(\eta^2). For the same smooth fixed-loss GD, with fixed P,H≥2P,H\ge2, nonzero initial gradient and nonzero transverse curvature, the top two directions of the exact history achieve O(η2)O(\eta^2) with the matched rollout (Corollary 3). Future displacement directions turn with the horizon. Even a perfectly chosen scalar coefficient can only move a forecast along the chosen line.

This gives a concrete failure mode for rank selection. As the step shrinks, the second direction's historical energy fraction falls as Θ(η2)\Theta(\eta^2). One direction then captures arbitrarily close to 100% of the historical update energy, while its best possible forecast still loses an order of accuracy. In Figure 3 the 99% rule selects one direction in all 40 block–scale conditions. In each condition both matched two-dimensional rollouts beat a lower bound on every anchored, window-fixed line. RELEX fits a coefficient per tensor and scores the resulting checkpoint on MATH. Those coefficients can trace a path outside a single joint line. For joint parameter displacements, our result identifies the minimum dimension for the stated accuracy order and how well the space must capture the gradient and its transverse turn.

I thank the authors for their response. However, my concern that the technical contribution of the rank-two approximation is incremental remains unresolved. Therefore, I will maintain my current score.

Reviewer_WPQ3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies local trajectory prediction and subspace compression over short windows (PP prefix steps, HH forecast steps) under fixed-loss, stateless gradient descent. The main question is what a low-dimensional subspace must retain to match the O(η2)O(\eta^2) relative parameter error of the unrestricted linearized model (LocalFull). Theorem 1 shows that the subspace projector needs to capture the anchor gradient qaq_a to O(η2)O(\eta^2) and the first Hessian action HaqaH_a q_a to O(η)O(\eta). Building on this, Theorem 2 proves that when transverse curvature is nonzero, any fixed 1D line through the anchor has Θ(η)\Theta(\eta) error (with a sharp constant cH∗=⌊H2/4⌋/(2H)c_H^* = \lfloor H^2/4 \rfloor / (2H)), even if the line direction and per-step coefficients are chosen with full access to future states. In contrast, the 2D Krylov plane (K2) and the noiseless historical top-2 SVD space (HistSVD2) both recover the O(η2)O(\eta^2) rate (Corollary 3). Because the second singular direction carries only Θ(η2)\Theta(\eta^2) of the historical update energy, any fixed explained-variance threshold τ<1\tau < 1 eventually drops it as η→0+\eta \to 0^+ (Theorem 4, Corollary 6). The authors test these rates on final-layer q/v LoRA-B weights across four LLM families, and include extensive appendix diagnostics on changing mini-batches and AdamW.

Critical strengths
  • The proof of Theorem 2 is quite neat. By reducing the oracle 1D line problem to minimizing L1L_1 deviation over {0,…,H−1}\{0, \dots, H-1\}, the paper gets an exact analytical constant cH∗c_H^* without needing numerical optimization over the Grassmannian. This shows convincingly that the Θ(η)\Theta(\eta) error floor comes from the 1D restriction itself rather than a suboptimal choice of direction.
  • The empirical protocol is very carefully controlled. Keeping the anchor, exact autograd HVPs, and rollout recurrence identical across K1, K2, and HistSVD1/2 cleanly isolates the effect of the subspace. I also appreciate the care taken with numerical precision (identifying the BF16 quantization floor in Appendix C.6 and running the main checks in FP32/FP64) and statistical grouping (treating model-blocks as the unit of replication rather than pooling nested step sizes).
  • The paper is refreshingly honest about what does and does not work outside the idealized regime. Appendices E, F, and G openly document where the local model breaks down under shuffled AdamW and where parameter-space improvements fail to move held-out NLL.
Critical weaknesses
  • The gap between the theoretical setup and actual LLM training is my main reservation. The theorems and primary experiments assume stateless GD on a single repeated 4-example batch, updating only the last-layer q/v LoRA-B matrices while freezing LoRA-A. Because LoRA-A is frozen, the trainable parameters act linearly before the loss, which artificially suppresses parameter-coupling curvature. Indeed, Table 20 (Appendix E.3) shows that once LoRA A+B are trained jointly or mini-batches are shuffled, the 99% rank-1 occupancy regime largely disappears. Similarly, Proposition 1 and Figure 4 show that with varying batches (λ>0\lambda > 0), the tail energy M(λ,η)M(\lambda, \eta) hits a floor dominated by batch variance Aλ2A\lambda^2 as η\eta shrinks, so the "nearly 1D history" premise no longer holds.
  • The practical message behind Corollary 3 (HistSVD2 sufficiency) feels somewhat circular regarding compute and information budgets. While the basis QQ of HistSVD2 is extracted from history, rolling out the prediction via Eq. (3) still requires computing the anchor gradient gag_a and two exact HVPs (HaQH_a Q). If we can afford exact HVPs at the anchor anyway, we might as well just form K2 (span{ga,Haga}\mathrm{span}\{g_a, H_a g_a\}), which only takes one HVP and consistently beats HistSVD2 at finite step sizes because it has zero forcing leakage (Appendix D.4). Meanwhile, genuinely operator-free historical methods like DMD2, RELEX-style extrapolation, or the secant approximation in Appendix F.3 do not achieve the O(η2)O(\eta^2) guarantee.
  • Capturing the weak second direction q2q_2 does not seem to matter functionally in the tested setups. Even though q2q_2 contains over 96% of K1's parameter-space residual, Appendix G.1 reports that projecting the true future displacement onto the top-2 historical directions improves held-out NLL by only 4.25×10−64.25 \times 10^{-6} nats/token over rank-1, well below the authors' own materiality threshold. Combined with Appendix F.2, where causal reduced-rank predictors under shuffled AdamW fail the 0.10 error budget at h=8h=8 and cost 2.51×2.51\times more compute than direct training, it is hard to see how this O(η)O(\eta) vs. O(η2)O(\eta^2) distinction translates into a practical training or extrapolation algorithm.
  • Presentation-wise, the paper leans too heavily on a 35-page appendix. Section 7 packs several important negative and boundary results into three dense paragraphs, leaving the reader to hunt through Appendices E-G to understand what predictors like "Prospective" or "CORR2" actually compute.
Questions for authors
  1. In Corollary 3, HistSVD2 only uses the prefix UU to pick the subspace QQ, but still relies on exact HVPs (HaQH_a Q) for the rollout recurrence. Can you get an O(η2)O(\eta^2)-accurate rollout purely from the noiseless prefix UU (for instance, by approximating the projected Hessian action on span{q1}\mathrm{span}\{q_1\} via finite differences of the historical updates) without any HVP calls at the anchor?
  2. In Figure 3 and Tables 4-5, Llama-3.2-1B matches the asymptotic slopes (pK1=0.976p_{K1} = 0.976) and sharp constant (0.9940.994) very closely, whereas Qwen3-1.7B shows a clear gap (pK1=0.768p_{K1} = 0.768, constant ratio 0.9510.951 at the smallest δ\delta, and much lower at δ=0.03\delta = 0.03). Yet in Table 6, Qwen3-8B is right back at the theoretical rates (1.0051.005 and 2.0102.010). Do you know what causes this finite-step deviation in Qwen3-1.7B (e.g., a larger parallel curvature term qa⊤Haqaq_a^\top H_a q_a or third-order derivative along the trajectory)?
  3. Given the very small NLL gains in Appendix G.1 and the negative compute savings in Appendix F.2, where do you see the O(η2)O(\eta^2) capture criterion making a tangible difference in practice? Is there a regime (perhaps longer horizons with periodic operator refresh, or second-order subspace optimizers) where retaining q2q_2 visibly improves loss or wall-clock speed?
2 comments
AuthorsOpenReview ↗

Response to Reviewer WPQ3

1. Can a noiseless prefix supply the rollout coefficients without HVPs?

Yes. For fixed-loss, constant-step GD, let ut=θt−θt−1u_t=\theta_t-\theta_{t-1}, a=θPa=\theta_P and d=uP−uP−1d=u_P-u_{P-1}. Predict the first unseen update by uP+du_P+d and the hh-step displacement by bh=huP+h(h+1)2d.b_h=h u_P+\tfrac{h(h+1)}2d. These two updates suffice. In the HistSVD2 basis QQ from the full prefix, the coefficients are Q⊤bhQ^\top b_h. Neither uses anchor gradients, HVPs or future states. No full projected Hessian is needed.

For fixed P,HP,H the iterates are O(η)O(\eta) from aa. A Lipschitz Hessian gives ut+1−ut=−η(gt−gt−1)=η2Haga+O(η3)u_{t+1}-u_t=-\eta(g_t-g_{t-1})=\eta^2H_ag_a+O(\eta^3). Thus uP+j=uP+jd+O(η3)u_{P+j}=u_P+jd+O(\eta^3). Summing fixed hh gives O(η3)O(\eta^3) absolute error; dividing by ∥Δh∥=hη∥ga∥+O(η2)\|\Delta_h\|=h\eta\|g_a\|+O(\eta^2) gives D0=O(η2)D_0=O(\eta^2) for nonzero limiting gradient. Projection preserves the order since each observed update lies within σ3(U)=O(η3)\sigma_3(U)=O(\eta^3) of QQ.

Using u7,u8u_7,u_8, both forecasts beat repeating u8u_8 in 40/40 Fig. 3 conditions over 24 future steps. Unprojected five-scale slopes were 1.751/1.539 (Qwen/Llama). Finest-three median block slopes (unprojected / HistSVD2-projected):

TrajectoriesQwen3-1.7BLlama-3.2-1B
Archived FP321.923 / 1.9291.096 / 1.848
Matched FP321.923 / 1.9281.107 / 1.827
Matched FP641.930 / 1.9311.985 / 1.987

With identical frozen-prefix activations, we recomputed the last block, head, loss and entire GD trajectory at each precision. On Llama, the finest halving reduced error 3.97-fold in FP64 versus 1.41-fold in matched FP32.

2. Why does Qwen3-1.7B deviate at finite steps?

At large steps Qwen's anchor loses more transverse curvature: βa/β0\beta_a/\beta_0 rises from 0.633 to 0.977 from coarse to fine (Llama: 0.958 to 0.997). Its K1 slope rises from 0.768 (all five scales) to 0.903 (finest three).

We tested both mechanisms at all 40 anchors. Write A=HaA=H_a, q=−ga/∥ga∥q=-g_a/\|g_a\|, α=q⊤Aq\alpha=q^\top Aq, v=(Aq−αq)/βav=(Aq-\alpha q)/\beta_a, γ=v⊤Av\gamma=v^\top Av and m=∥ga∥v⊤∇3L(a)[q,q]/βam=\|g_a\|v^\top\nabla^3 L(a)[q,q]/\beta_a. For these smooth losses and H=24H=24, the first correction to total K1 error normalized by T=5.75ηβaT=5.75\eta\beta_a is η(53α/18−44γ/9+97m/18)\eta(53\alpha/18-44\gamma/9+97m/18). We measured these terms by autodiff; centered differences at two increments agreed with the Qwen third-derivative contraction within 0.07%. At Qwen's coarsest step the corrections were +0.019 (parallel curvature), -0.162 (curvature along vv), and -0.177 (third derivative): componentwise medians. Parallel curvature alone predicts the wrong sign. Llama's corresponding terms are +0.007, -0.029 and -0.008.

Combining terms per block, without fitting, gives predicted/observed K1 ratios of 0.924 vs. 0.928 at δ=0.0075\delta=0.0075 and 0.690 vs. 0.780 at 0.03 (Llama: 0.969 vs. 0.969 at 0.03). Qwen's frozen-Hessian restriction alone gives 0.870 at 0.03. The same derivatives predict a best-line error ratio relative to 3ηβ03\eta\beta_0 of 0.9515 at the finest Qwen scale (bounds 0.95142–0.95145), but 0.343 at the coarsest (0.434–0.435). It overstates Qwen's coarse-step gap.

Across the eight separate Qwen3-8B blocks, Table 6's data give βa/β0\beta_a/\beta_0 near 1 even at the coarsest step (median 1.024; finest 1.002). Its slopes 1.005 and 2.010 measure the frozen-model restriction and historical tail energy, not total K1 error or the best-line constant. The cohorts do not isolate model size.

3. When does the second direction pay off in practice?

Periodic refresh. Adding HgHg to the gradient line lets one fit track the bend for more steps. On eight fixed-batch Qwen/Llama blocks (Fig. 3, last-layer LoRA), we forecast 48 GD steps at the middle scale, rebuilding at predicted weights. Refreshing the plane every 8 steps (six fits) met the 0.05% mean relative displacement-error target on 8/8 blocks (median 0.012%). The line missed on all eight even with 4-step refreshes. All methods cached the frozen layers. Counting cache setup and every refresh, the plane was 1.99× as fast as direct GD and 1.23× as fast as that line (paired medians, RTX 4090 D).

For shuffled AdamW we used its update dd and HdHd. Refreshing every 4 steps with the actual batches met the 10% endpoint-error target on 8/8 runs but took 1.62× as long as direct AdamW. Reusing the refresh batch gave a 1.74× speedup but missed on all eight (median error 21.9%).

Second-order optimization. From the Fig. 3 checkpoints we took 12 trust-region steps on the gradient line or in the g,Hgg,Hg plane. At radius 0.12 the plane gave lower training NLL on 8/8 blocks and lower held-out NLL on 5/8. At radius 0.48 (exploratory), it gave lower NLL on the eight held-out examples in all eight blocks (median extra reduction: 0.00154 nats/token, or 1.8% of the line's gain). At each radius, both methods reached a 0.01 nats/token held-out drop on the same step in every block. Including setup, curvature and validation, the plane took 17% longer at 0.12 and 9% at 0.48. We have not tested full-run acceleration.

Dear Reviewer,

We’re nearing the end of the discussion period. This is a friendly reminder to leave a comment for the authors.

Thank you

~ AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.