- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MatryoshkaLoRA, a training framework intended to produce a single LoRA checkpoint with usable lower-rank prefixes. It inserts a fixed diagonal matrix P between the LoRA factors, representing an aggregate of prefix updates while requiring only one model forward pass. At inference, P is discarded and a selected prefix is used. The paper analyzes the difference between this surrogate and a weighted multi-rank loss through a rank-dispersion term, and introduces AURAC and log-AURAC to summarize performance across ranks. Experiments on GSM8K with Llama-3.2-1B and Llama-3.1-8B show improvements in several LoRA and DyLoRA configurations, benefits for a DoRA extension, and comparable or slightly lower aggregate performance for PiSSA.
Critical strengths
A simple implementation with promising empirical gains. The method requires only fixed diagonal scaling within the standard LoRA computation, avoiding separate model evaluations for every training rank. Several GSM8K configurations show meaningful improvements, including the smaller Llama model and the DoRA extension. These results make the proposed parameterization worth investigating as an inexpensive way to improve prefix usability.
Critical weaknesses
-
The central explanation of hierarchical learning is insufficiently established. The method optimizes f(Σr λrΔr), rather than Σr λrf(Δr). Consequently, every component receiving a gradient does not mean that every prefix is trained against its own prediction loss. Moreover, P is fixed and invertible, so APB can be absorbed into a standard rank-R LoRA factorization. Any benefit must therefore arise through training dynamics or implicit bias, which the current controls do not isolate. The theoretical bound is conditional on learned rank dispersion being small, but the method neither explicitly controls nor empirically characterizes this quantity. Its vanishing-support-factor analysis also uses normalized weights and α/r scaling, whereas the main experiments use λr=sr=1.
-
The evaluation does not adequately establish the claimed multi-rank advantage. Validation is limited to one benchmark and two models. The results include ranks where standard LoRA is better and a PiSSA extension with slightly lower aggregate scores. Missing comparisons include independently optimized rank-specific LoRA adapters, a direct multi-rank-loss reference, and controls for the initialization and optimization changes induced by P. The paper also does not clearly specify a held-out validation protocol for selecting learning rates and rank-support configurations. These gaps make it difficult to determine whether the reported gains establish a broadly useful nested-adapter method.
Questions for authors
-
Evidence needed to assess the mechanism: What distinguishes the observed benefit from ordinary LoRA with matched initialization and optimization changes induced by absorbing P into its factors? A controlled comparison, together with a direct multi-rank-loss reference, would materially affect my assessment.
-
Clarification of theoretical applicability: How does the average-optimality argument apply to the experimental λr=sr=1 configuration? Please clarify the normalization of the actual training update and report or characterize the learned rank dispersion. Is there evidence that the conditional bound is informative for the trained checkpoints?
-
Clarification of the comparison protocol: Which split was used to select learning rates, training rank sets, and reported configurations? Were final scores measured on an untouched test set? Please also clarify whether standard LoRA was tuned for its trained rank or for prefix log-AURAC, and specify the DyLoRA sampling, freezing, and scaling settings.
2 comments
Thank you for the additional experiments and clarifications, which partially address my concerns. However, the key mechanistic concern remains unresolved: Table 6 suggests that the gains are primarily associated with the unnormalized weighting setting, while the normalized variants perform similarly to EQ-3, and a reparameterization-matched LoRA control is still missing. Furthermore, selecting hyperparameters based on test-set performance raises concerns about the reliability of the evaluation, so I maintain my original rating.
Rebuttal Comment for Reviewer afEr
We would like to thank the reviewer for their feedback, we address your concerns below.
Q1: Controlled comparison and Multi-Rank Loss
We implemented the formula Σr λrf(Δr) from Equation 3 (called EQ-3) and added the results in Table 6 in the new revision of the paper that we uploaded in response to the rebuttal. Our results show that using EQ-3 does not yield significantly higher accuracy for GSM-8k/8-shot on any of the choices of λr and S. In contrast, MatryoshkaLoRA with λr=1 achieves higher accuracy compared to any EQ-3 setting, which holds for both ranks 16 and 32.
Our results in Table 1 show comparisons with LoRA trained independently at each target rank. Regarding the dataset splitting protocol, in the new revision we explicitly mention that we perform learning rate sensitivity analysis by training on the entire training set of GSM-8k and report the accuracy on the test set.
Q2: Theoretical applicability
We compute the rank dispersion δS for MatryoshkaLoRA and DyLoRA using λr=1/∣S∣, according to Equation 14 in Appendix B. Since LoRA is a single rank method, we implement Equation 14 with the same λr to have the same order of magnitude. When computing δS, we average all rank dispersions computed at each layer for the three methods (the 1B model has 112 parameter matrices which we fine-tune). We report these values below for the 1B and GSM-8k:
We observe that all runs with MatryoshkaLoRA achieve lower rank dispersion than LoRA and DyLoRA and similar or lower than EQ-3. This is an indication that our surrogate loss matches the theory. We observe that despite using sr=1,λr=1, when using all ranks in S, we can achieve a lower rank dispersion, which in our opinion is quite surprising. We hope these ablations clarify the link between theory and practice.
Q3: Comparison protocol
We perform learning rate sensitivity runs for all methods by fine-tuning on the entire training set of GSM-8k and report the accuracy computed on the test set across multiple ranks. We do not isolate any validation set from the training set. We agree that this distinction should have been stated explicitly and we will fix this in the next revision.
For DyLoRA, we update the entire adapter prefix of size k, where k is the rank sampled from the uniform distribution over all ranks from 1 to R to scale the product of low-rank adapters.