- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes reusing a compressed initialization Jacobian to approximate parameter updates without repeated network-wide backpropagation. It combines laziness and rank diagnostics with a low-rank representation, which are two key assumptions of this work, and evaluates the approach for parameter-efficient fine-tuning of pretrained vision models.
Critical strengths
- The motivation is clear, and reducing repeated backward computation is a relevant practical objective.
- The experiments measure step time, memory, and test accuracy on large pretrained models, include activation-checkpointing comparisons, and report the main accuracy results over ten runs.
Critical weaknesses
-
I think the novelty is not sufficient, especially on the diagnostic components. The laziness measure follows Chizat et al. (lines 236–246), while Jacobian spectral analysis and explained-variance rank criteria seem to be similar with that presented in the cited literature (Oymak et al., 2019& Mahowald et al., 2026). Proposition 3.1 is also an elementary Gram-matrix identity, as acknowledged by the authors (p. 5, lines 216–225). The contribution therefore appears to lie primarily in their scalable implementation and integration into the training pipeline. The claim of being the first practical, scalable diagnostic framework needs a more precise distinction from existing methods. The parameter-update requirement discussed in lines 130–131 does not by itself establish novelty of the diagnostics.
-
The computational savings are not clear for me, since it is not sufficiently quantified. Algo 1 table shows that it requires reference selection, Jacobian construction, eigendecomposition, and projection accumulation before training. Appendix B.1 tells that selection itself requires one backward pass per training example, while Table 7 concerns individual training steps. Although Lines 397-399 states that the laziness diagnostic takes minutes, the manuscript does not report the complete preprocessing-plus-training time or the break-even training duration. The net benefit for the reported 30-epoch experiments therefore remains unclear.
Questions for authors
-
Which specific methodological elements of the diagnostics are new relative to the prior work?
-
Including preprocessing, what is the total runtime for 30 epochs, and when does the method break even with BP?