acceptodds
Withdrawn from ICLR 2027

Gradient Descent Without Repeated Backpropagation via Lazy Low-Rank Jacobians

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Backpropagation is the standard and highly effective mechanism for computing gradient-descent updates in neural networks. We explore whether part of its repeated derivative computation can be amortized by exploiting structure that emerges in large, overparameterized models. Our approach builds on two such properties: the *lazy regime*, where the Jacobian changes only mildly during training, and substantial low-rank structure in the Jacobian. Together, these suggest that derivative information can be compressed once and reused across optimization. We develop practical tools for testing these properties without explicitly forming or storing the full Jacobian or Hessian, including a Hessian–Jacobian measure of laziness and a scalable rank-estimation procedure based on a small subset of model parameters. We further show that, under suitable parameterizations, parameter-efficient fine-tuning inherits the laziness of the base model. Building on these observations, we construct a training procedure that reuses a compact low-rank Jacobian representation to approximate standard gradient-descent updates without repeated network-wide backpropagation. The forward computation remains unchanged, while the costly network-wide backward pass is replaced by inexpensive updates over a compact Jacobian representation, making parameter updates nearly negligible compared with standard backpropagation. Our goal is not to replace backpropagation as a general-purpose training method, but to show that gradient-based learning can be reorganized around the mathematical structure of overparameterized networks, substantially reducing the need for repeated backpropagation. Experiments show that this alternative computational organization remains viable in practice on realistic architectures and datasets.

Primary area:
optimization
Keywords:
lazy training, low-rank Jacobians, gradient descent, backpropagation, neural tangent kernel, parameter-efficient fine-tuning, Jacobian compression

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nzHuOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes reusing a compressed initialization Jacobian to approximate parameter updates without repeated network-wide backpropagation. It combines laziness and rank diagnostics with a low-rank representation, which are two key assumptions of this work, and evaluates the approach for parameter-efficient fine-tuning of pretrained vision models.

Critical strengths
  • The motivation is clear, and reducing repeated backward computation is a relevant practical objective.
  • The experiments measure step time, memory, and test accuracy on large pretrained models, include activation-checkpointing comparisons, and report the main accuracy results over ten runs.
Critical weaknesses
  1. I think the novelty is not sufficient, especially on the diagnostic components. The laziness measure follows Chizat et al. (lines 236–246), while Jacobian spectral analysis and explained-variance rank criteria seem to be similar with that presented in the cited literature (Oymak et al., 2019& Mahowald et al., 2026). Proposition 3.1 is also an elementary Gram-matrix identity, as acknowledged by the authors (p. 5, lines 216–225). The contribution therefore appears to lie primarily in their scalable implementation and integration into the training pipeline. The claim of being the first practical, scalable diagnostic framework needs a more precise distinction from existing methods. The parameter-update requirement discussed in lines 130–131 does not by itself establish novelty of the diagnostics.

  2. The computational savings are not clear for me, since it is not sufficiently quantified. Algo 1 table shows that it requires reference selection, Jacobian construction, eigendecomposition, and projection accumulation before training. Appendix B.1 tells that selection itself requires one backward pass per training example, while Table 7 concerns individual training steps. Although Lines 397-399 states that the laziness diagnostic takes minutes, the manuscript does not report the complete preprocessing-plus-training time or the break-even training duration. The net benefit for the reported 30-epoch experiments therefore remains unclear.

Questions for authors
  1. Which specific methodological elements of the diagnostics are new relative to the prior work?

  2. Including preprocessing, what is the total runtime for 30 epochs, and when does the method break even with BP?

Reviewer_qEjWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper combines low-rank Jacobian compression with the lazy training regime assumption to reuse derivative information throughout fine-tuning, avoiding repeated network backpropagation after preprocessing. It also proposes diagnostics for these assumptions and analyzes their preservation under selected PEFT parameterizations.

Critical strengths

The Paper is interesting, presenting a framework that avoid back backpropagation, the most central computation in training, under very certain setup.

Critical weaknesses

My main concern is the limited novelty and applicability.

Novelty

The idea is basically mixing of:

  • Low-rank adaptation and gradient compression have been extensively explored by methods such as LoRA, GaLore, Fira, and APOLLO.
  • Linearized fine-tuning with lazy regime has been studied in works such as LQF and tangent-based approaches.

Moreover, once the lazy regime assumption holds, reusing derivatives to avoid repeated backpropagation is a natural consequence. As LQF states, the scenerio is restricted.

Applicability

The experimental setup is very restrictive, most experiment setup is non-standard specific experiment setup comparing to original LQF: EuroSAT and StanfordDogs are rare dataset. For CIFAR-100, the nearly two percentage point accuracy drop is substantial. The preprocess is very costy, the paper somehow intentionally ignores it.
Also, The theoretical results are very toy.

Overall, the limited methodological novelty and narrow empirical scope do not convince me that the paper meets the bar for ICLR.

Questions for authors

See weakness please.

Reviewer_gTqKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores whether the repeated backward computation of gradient descent can be amortized in large, overparameterized models. It relies on two properties: the lazy regime, in which the Jacobian changes only mildly during training, and the low-rank structure of the Jacobian. The authors propose scalable diagnostics for both properties, a Hessian–Jacobian laziness measure based on Hessian–vector products and Lanczos iterations, and a rank estimate computed on a small reference set of parameters. They further show that fixed-A LoRA and BitFit inherit the laziness of the base model, whereas two-factor LoRA need not. Based on these results, the method compresses the Jacobian once into a low-rank representation and reuses it to approximate gradient-descent updates without repeated backpropagation. Experiments on fine-tuning CLIP ViTs and EVA02-E on CIFAR-100 show that the update cost is nearly independent of sequence length, memory usage is much lower, and the accuracy stays within about 2 points of backpropagation.

Critical strengths
  1. A well-motivated idea with verifiable assumptions. Instead of simply assuming that large models are lazy and low-rank, the paper provides scalable diagnostics to test these properties before exploiting them, and the laziness-inheritance analysis clearly explains which PEFT parameterizations preserve them. The diagnostic also correlates well with the Jacobian drift observed during training, which makes the overall approach principled rather than heuristic.

  2. Clear computational benefits per training step. Once the Jacobian representation is built, the update cost is nearly independent of sequence length, the training step approaches the cost of inference, and the method supports much larger batches than backpropagation. These gains are measured in wall-clock time and memory rather than only in FLOPs.

Critical weaknesses
  1. Limited practical significance. The method only applies when the model is close to the lazy regime, the trainable parameter space is small, and the number of output units is small, since a separate representation must be built and stored for each output unit. This restricts the evaluation to image classification fine-tuning with up to 120 classes. It is unclear how the approach would extend to generative or language-modeling tasks, where the output dimension equals the vocabulary size, or to settings where full fine-tuning or feature learning matters. Moreover, the reported savings exclude the preprocessing phase, which requires computing projected Jacobians for every output unit over thousands of examples. Without reporting this one-time cost and the resulting end-to-end training time, it is difficult to judge whether the method actually saves compute in practice, especially since the per-epoch saving is only about 2× while the accuracy is 1 to 2 points lower on two of the three datasets.

  2. Missing simple and closely related baselines. Because the Jacobian is fixed, the method essentially trains a linearized model in a low-rank tangent subspace together with a linear readout. However, it is compared only with two-factor LoRA trained by backpropagation. Simpler and cheaper alternatives are not reported, including linear probing on frozen features, which the authors themselves describe as already strong on EuroSAT, backpropagation with the same fixed-A LoRA parameterization, and existing linearized fine-tuning methods based on tangent features. Without these baselines, it is unclear how much of the reported accuracy comes from the proposed Jacobian reuse rather than from the strong frozen CLIP features and the linear readout.

Questions for authors
  1. In which practical scenario would a practitioner choose this method over standard backpropagation or over cheaper alternatives such as linear probing? Could you give a concrete use case where the per-step savings outweigh the preprocessing cost and the 1 to 2 point accuracy loss?
  2. What is the end-to-end training time of the method compared with backpropagation, including the preprocessing phase, for each setting?
  3. Is the method applicable beyond classification fine-tuning with a small number of output units, for example to language modeling or other generative tasks with large output spaces? If not, what is the intended scope of the approach?
  4. How much of the reported accuracy comes from Jacobian reuse rather than from the frozen CLIP features and the linear readout, i.e., how does the method compare with linear probing and with backpropagation using the same fixed-A LoRA?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.