- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper distinguishes the rank of individual LoRA updates from the dimension of a shared input subspace covering their training trajectory, and from the preservation of predictions after projection. In a linear-regression construction, gradient descent produces rank-one checkpoints whose shared hidden input space has dimension d−1, even though the initialization space already contains a solution. The paper also analyzes shared measurements and spectral approximation. Experiments across 24 language-model fine-tuning runs examine trajectory energy and the effects of shared-subspace projections on aggregate performance and individual predictions.
Critical strengths
-
The paper clearly separates instantaneous rank, shared trajectory dimension, and predictive fidelity. The gradient-descent construction is a nontrivial formal demonstration of their distinction, rather than an example assembled from unrelated checkpoints.
-
The empirical analysis connects the geometric quantities to actual model predictions through projection interventions and controls. The paper also acknowledges the limits of its conclusions, including the difference between preserving aggregate performance and preserving individual decisions.
Critical weaknesses
-
Limited significance of the theoretical and empirical insights. The distinction between checkpoint rank and trajectory-wide subspace dimension is useful, and I acknowledge the technical value of establishing it under gradient descent. Nevertheless, the underlying possibility—that low-rank updates can change direction during training—is intuitive. I am not convinced that the formal result yields sufficiently deep or consequential new understanding beyond establishing this possibility. Similarly, the findings that trajectory energy concentrates in a few principal directions and that preserving most matrix energy need not preserve predictions are informative, but do not substantially change my understanding of LoRA.
-
Unclear practical implications. The paper does not sufficiently explain when these analyses would change a concrete decision about LoRA training or deployment. It does not demonstrate improvements in training efficiency, parameter compression, or another practical application, and the value of analyzing the full trajectory rather than the final checkpoint remains unclear. An analysis paper need not introduce a new algorithm, but its conceptual or practical consequences should be sufficiently compelling. Despite the formal and empirical value of the work, I do not find that level of significance established here, which motivates my clear-rejection recommendation.
Questions for authors
-
Which conclusion from the gradient-descent analysis provides the most consequential insight beyond the intuitive distinction between instantaneous rank and trajectory-wide subspace dimension?
-
What evidence shows that analyzing the training trajectory, rather than the final checkpoint, provides information that changes a useful decision about LoRA training, compression, or preservation of model behavior?
These are clarification questions about the significance of the existing results; I am not requiring the paper to introduce a new training algorithm.