- Rating
- 1(−1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper characterizes when historical trajectory compression preserves local prediction accuracy under fixed-loss gradient descent. It establishes necessary and sufficient capture conditions, a sharp one-dimensional error bound, and two-dimensional sufficiency, showing why fixed energy thresholds discard necessary information. Controlled multi-scale experiments in selected LLM LoRA coordinates test the predicted error orders and coefficients.
Critical strengths
- Precise theoretical characterization. Necessary and sufficient capture conditions, a sharp optimum over anchored fixed lines, and exact-history two-dimensional sufficiency clarify why retaining historical energy can fail to preserve local prediction accuracy.
- Direct empirical validation. Matched-space comparisons isolate compression effects, while multi-scale experiments test predicted error orders and coefficients. These evaluations connect the theoretical claims to measured behavior rather than relying solely on aggregate performance improvements.
Critical weaknesses
- Limited demonstrated significance beyond local trajectory fidelity. Preserving the local model’s accuracy order has a clear numerical motivation, supported by the matched experiments. However, its importance for learning quality remains less established. Appendix G.1’s oracle historical-projection diagnostics report a median NLL gain of 0.00000425 nats/token and a median recovery ratio of 0.208%, with continuation effects also below the stated materiality threshold. These findings do not undermine the theory, but limit the demonstrated significance of the accuracy-order distinction beyond parameter prediction.
- Limited transfer to practical compression criteria. The guarantees concern fixed-loss, stateless GD over windows that shrink in optimization time, and historical sufficiency assumes noiseless updates. The required accuracy for recovering the weakening second direction from perturbed histories remains uncharacterized. The shuffled-AdamW and selective-extrapolation studies provide useful supporting evidence, but require additional information and computation; Appendix F.2 finds no favorable accuracy–cost window for the tested reduced predictors. Consequently, the results offer limited guidance for reliable, cost-justified compression outside the exact local regime.
Questions for authors
- Finite-scale implications. Within the fixed-loss setting, could you use the existing best-line bounds and K2/HistSVD2 errors to identify normalized parameter-error tolerances that a two-dimensional predictor satisfies but no anchored, window-fixed one-dimensional predictor can satisfy?
- Sensitivity to imperfect history. Keeping the anchor gradient and Hessian exact, could you provide a sufficient perturbation bound for the normalized history matrix (the update matrix divided by the learning rate times the initial gradient’s Euclidean norm), such that its recomputed top-two subspace preserves Corollary 3’s capture rates?
3 comments
Dear Reviewer,
We’re nearing the end of the discussion period. This is a friendly reminder to leave a comment for the authors.
Thank you
~ AC
I thank the authors for their response. While both answers are informative, they reinforce rather than resolve my two primary concerns, and I am lowering my rating accordingly.
Q1 (Finite-scale tolerance): As the authors acknowledge, the separating tolerances (0.05%–0.25% mean relative parameter error) are retrospective: the line lower bound relies on observed future directions, and the separating tolerance necessarily shrinks to zero as . More importantly, no connection is established between these parameter-error tolerances and learning quality. Given the median held-out NLL gain of nats/token in Appendix G.1, along with the response to WPQ3 (the plane yields 1.8% of the line's NLL gain at a 9–17% higher computational cost, and under shuffled AdamW, the accurate predictor is 1.62$\times$ slower than direct training), the distinction in accuracy order still lacks demonstrated practical significance.
Q2 (Imperfect history): In normalized units, the sufficient condition ( in raw units) is one order of magnitude below , and the authors' own construction demonstrates that a perturbation already destroys second-order accuracy. Under minibatch sampling, each normalized update carries gradient-noise perturbations of order , which do not vanish as . Consequently, the guarantee does not appear to cover any realistic stochastic training history, which is consistent with the batch-variance floor in Proposition 1. This further confirms that the results offer little guidance for practical compression criteria outside the exact, noiseless, fixed-loss regime.
Furthermore, the main derivation for Q2 failed to render properly in the posted comment (inline LaTeX is merged with the body text), preventing me from verifying it step by step.
Overall, while the theory is precise within its defined setting, the rebuttal makes it clearer that both its practical relevance (W1) and its applicability beyond noiseless, fixed-loss gradient descent (W2) are severely limited. I am lowering my rating to 1.
Response to Reviewer RUgW
1. Does the separation hold at a concrete finite tolerance?
Yes. At the finest tested scale, a mean relative parameter-error tolerance of 0.05% separates both K2 and HistSVD2 from every anchored, window-fixed line on each of the eight model–data blocks. The comparison allows the line to see the full future and to use a different coefficient at every horizon.
For a recorded trajectory, let Ldisj be the disjoint-pair lower bound on the optimal line error (Appendix C.2). If max(EK2,EHistSVD2)≤ε<Ldisj, both two-dimensional forecasts meet the tolerance and every line misses it. All errors use the same true displacement at each horizon to normalize D0. Joining the two released Figure 3 tables by condition gives these examples at P=8,H=24 and δ=η∥g0∥. Each row uses the largest error of either two-dimensional method and the smallest lower bound across all eight blocks.
Across all five scales, the paired ratio Ldisj/max(EK2,EHistSVD2) is at least 1.68 in every one of the 40 conditions (eight blocks, five nested scales). At δ=0.03, for example, a common tolerance of 1% works on all four Llama blocks and 2% on all four Qwen blocks. These are retrospective guarantees for the observed trajectories, since the lower bound uses their future directions. As η tends to zero even the best-line error tends to zero, so the tolerance separating the accuracy orders must also shrink.
2. How accurate must an imperfect prefix be?
For A=U/(η∥g0∥) and A=A+N, ∥N∥2=O(η2) in operator norm suffices for arbitrary additive perturbations, with the anchor gradient and Hessian kept exact as in your question. Let Π be the top-two right singular projector of A and e=∥N∥2. The rank-two residual and Weyl's inequality give ∥A(I−Π)∥2≤e+σ3(A)≤σ3(A)+2e=O(η2)+2e. This bounds every row residual. In column notation let bt=ut/(η∥g0∥) and s=∥ga∥/∥g0∥→1. Expanding the last two updates gives 2bP−bP−1=sqa+O(η2) and (bP−1−bP)/η=sHaqa+O(η). Hence the distances of qa and Haqa to the estimated plane are O(η2+e) and O(η+e/η), respectively. With e=O(η2) these are precisely the two capture rates in Theorem 1. The matched rollout retains D0=O(η2). In raw units the sufficient bound is ∥U−U∥2=O(η3∥g0∥).
The distinction from simply resolving the second singular value matters. Take an exact rank-two quadratic trajectory embedded in three dimensions and apply the same orthogonal rotation to every observed update. The rotation tilts qa by ϕ toward the unused dimension and fixes the in-plane direction perpendicular to it. This preserves every singular value of the prefix. Yet the first true update points along the unrotated qa, so even its best relative error in the rotated plane is ∣sinϕ∣, and D0≥∣sinϕ∣/H. The prefix has an order-one projection onto qa, so ∥N∥2=Θ(∣ϕ∣). Choosing ϕ=η3/2 gives noise smaller than σ2(A)=Θ(η) that still loses second-order accuracy. Even an unchanged energy spectrum can conceal a tilt that matters for the forecast.