acceptodds
Under review as a conference paper at ICLR 2027

MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning

Abstract

With the rise in scale for deep learning models to billions of parameters, the computational cost of fine-tuning remains a significant barrier to deployment. While Low-Rank Adaptation (LoRA) has become the standard for parameter-efficient fine-tuning, the need to set a predefined, static rank requires exhaustive grid searches to balance efficiency and performance. Existing rank-adaptive solutions such as DyLoRA mitigate this by sampling ranks during the training from a predefined distribution. However, they often yield sub-optimal results at higher ranks due to lack of consistent gradient signals across the full hierarchy of ranks, thus making these methods data-inefficient. We propose MatryoshkaLoRA, a general, Matryoshka-inspired training framework for LoRA-like techniques that learns accurate hierarchical low-rank representations by inserting a fixed, carefully crafted diagonal matrix between the existing LoRA adapters to scale their sub-ranks accordingly. By introducing this simple modification, our general framework recovers LoRA and DyLoRA only by changing and ensures all sub-ranks embed the available gradient information efficiently. Our MatryoshkaLoRA supports dynamic rank selection with minimal degradation in accuracy and show it is general enough by applying it to other LoRA-inspired techniques, such as DoRA and PiSSA. We theoretically justify a potential train–test mismatch by bounding the average optimality gap across the supported ranks, guaranteeing that the learned hierarchy remains close, on average, to independently optimized rank-specific adapters during inference. We further propose Area Under the Rank Accuracy Curve (AURAC), a metric that consistently evaluates the performance of hierarchical low-rank adapters. Our results demonstrate that MatryoshkaLoRA learns more accurate hierarchical low-rank representations than prior rank-adaptive approaches and achieves superior accuracy-performance trade-offs across ranks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LoRA, LLMs, fine-tuning, low-rank adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_afErOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MatryoshkaLoRA, a training framework intended to produce a single LoRA checkpoint with usable lower-rank prefixes. It inserts a fixed diagonal matrix P between the LoRA factors, representing an aggregate of prefix updates while requiring only one model forward pass. At inference, P is discarded and a selected prefix is used. The paper analyzes the difference between this surrogate and a weighted multi-rank loss through a rank-dispersion term, and introduces AURAC and log-AURAC to summarize performance across ranks. Experiments on GSM8K with Llama-3.2-1B and Llama-3.1-8B show improvements in several LoRA and DyLoRA configurations, benefits for a DoRA extension, and comparable or slightly lower aggregate performance for PiSSA.

Critical strengths

A simple implementation with promising empirical gains. The method requires only fixed diagonal scaling within the standard LoRA computation, avoiding separate model evaluations for every training rank. Several GSM8K configurations show meaningful improvements, including the smaller Llama model and the DoRA extension. These results make the proposed parameterization worth investigating as an inexpensive way to improve prefix usability.

Critical weaknesses
  1. The central explanation of hierarchical learning is insufficiently established. The method optimizes f(Σr λrΔr), rather than Σr λrf(Δr). Consequently, every component receiving a gradient does not mean that every prefix is trained against its own prediction loss. Moreover, P is fixed and invertible, so APB can be absorbed into a standard rank-R LoRA factorization. Any benefit must therefore arise through training dynamics or implicit bias, which the current controls do not isolate. The theoretical bound is conditional on learned rank dispersion being small, but the method neither explicitly controls nor empirically characterizes this quantity. Its vanishing-support-factor analysis also uses normalized weights and α/r scaling, whereas the main experiments use λr=sr=1.

  2. The evaluation does not adequately establish the claimed multi-rank advantage. Validation is limited to one benchmark and two models. The results include ranks where standard LoRA is better and a PiSSA extension with slightly lower aggregate scores. Missing comparisons include independently optimized rank-specific LoRA adapters, a direct multi-rank-loss reference, and controls for the initialization and optimization changes induced by P. The paper also does not clearly specify a held-out validation protocol for selecting learning rates and rank-support configurations. These gaps make it difficult to determine whether the reported gains establish a broadly useful nested-adapter method.

Questions for authors
  1. Evidence needed to assess the mechanism: What distinguishes the observed benefit from ordinary LoRA with matched initialization and optimization changes induced by absorbing P into its factors? A controlled comparison, together with a direct multi-rank-loss reference, would materially affect my assessment.

  2. Clarification of theoretical applicability: How does the average-optimality argument apply to the experimental λr=sr=1 configuration? Please clarify the normalization of the actual training update and report or characterize the learned rank dispersion. Is there evidence that the conditional bound is informative for the trained checkpoints?

  3. Clarification of the comparison protocol: Which split was used to select learning rates, training rank sets, and reported configurations? Were final scores measured on an untouched test set? Please also clarify whether standard LoRA was tuned for its trained rank or for prefix log-AURAC, and specify the DyLoRA sampling, freezing, and scaling settings.

2 comments
AuthorsOpenReview ↗

Rebuttal Comment for Reviewer afEr

We would like to thank the reviewer for their feedback, we address your concerns below.

Q1: Controlled comparison and Multi-Rank Loss

We implemented the formula Σr λrf(Δr) from Equation 3 (called EQ-3) and added the results in Table 6 in the new revision of the paper that we uploaded in response to the rebuttal. Our results show that using EQ-3 does not yield significantly higher accuracy for GSM-8k/8-shot on any of the choices of λr\lambda_r and SS. In contrast, MatryoshkaLoRA with λr=1\lambda_r = 1 achieves higher accuracy compared to any EQ-3 setting, which holds for both ranks 16 and 32.

Our results in Table 1 show comparisons with LoRA trained independently at each target rank. Regarding the dataset splitting protocol, in the new revision we explicitly mention that we perform learning rate sensitivity analysis by training on the entire training set of GSM-8k and report the accuracy on the test set.

Q2: Theoretical applicability

We compute the rank dispersion δS\delta_S for MatryoshkaLoRA and DyLoRA using λr=1/∣S∣\lambda_r = 1 / |S|, according to Equation 14 in Appendix B. Since LoRA is a single rank method, we implement Equation 14 with the same λr\lambda_r to have the same order of magnitude. When computing δS\delta_S, we average all rank dispersions computed at each layer for the three methods (the 1B model has 112 parameter matrices which we fine-tune). We report these values below for the 1B and GSM-8k:

  • δS(LoRA)=1.46\delta_S(LoRA) = 1.46
  • δS(DyLoRA)=0.57\delta_S(DyLoRA) = 0.57
  • δS(MatryoshkaLoRA:sr=1/r,λr=1/∣S∣,S=all)=0.19\delta_S(MatryoshkaLoRA: s_r = 1/r, \lambda_r=1 / |S|, S=all) = 0.19
  • δS(MatryoshkaLoRA:sr=1,λr=1,S=pow2)=1.2\delta_S(MatryoshkaLoRA: s_r = 1, \lambda_r=1, S=pow2) = 1.2
  • δS(MatryoshkaLoRA:sr=1,λr=1,S=all)=0.16\delta_S(MatryoshkaLoRA: s_r = 1, \lambda_r=1, S=all) = 0.16
  • δS(EQ−3)=0.19\delta_S(EQ-3) = 0.19

We observe that all runs with MatryoshkaLoRA achieve lower rank dispersion than LoRA and DyLoRA and similar or lower than EQ-3. This is an indication that our surrogate loss matches the theory. We observe that despite using sr=1,λr=1s_r=1, \lambda_r=1, when using all ranks in SS, we can achieve a lower rank dispersion, which in our opinion is quite surprising. We hope these ablations clarify the link between theory and practice.

Q3: Comparison protocol

We perform learning rate sensitivity runs for all methods by fine-tuning on the entire training set of GSM-8k and report the accuracy computed on the test set across multiple ranks. We do not isolate any validation set from the training set. We agree that this distinction should have been stated explicitly and we will fix this in the next revision.

For DyLoRA, we update the entire adapter prefix of size kk, where kk is the rank sampled from the uniform distribution over all ranks from 11 to RR to scale the product of low-rank adapters.

Thank you for the additional experiments and clarifications, which partially address my concerns. However, the key mechanistic concern remains unresolved: Table 6 suggests that the gains are primarily associated with the unnormalized weighting setting, while the normalized variants perform similarly to EQ-3, and a reparameterization-matched LoRA control is still missing. Furthermore, selecting hyperparameters based on test-set performance raises concerns about the reliability of the evaluation, so I maintain my original rating.

Reviewer_JD5qOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Matryoshka LoRA, a parameter-efficient fine-tuning framework designed to learn hierarchical low-rank adapters that support multiple inference ranks from a single training run. By introducing a fixed diagonal weighting matrix between the LoRA factors, the method enables different rank prefixes to contribute to training with minimal additional computational overhead.

Critical strengths
  1. Simple and efficient design. Matryoshka LoRA introduces a lightweight diagonal weighting mechanism that enables hierarchical low-rank adaptation with minimal modification to standard LoRA.
  2. Flexible rank selection. The method supports multiple inference ranks from a single trained adapter and demonstrates improved rank-wise performance over LoRA and DyLoRA, particularly at smaller ranks.
Critical weaknesses
  1. Limited practical motivation and novelty. Matryoshka LoRA relies on a relatively simple fixed diagonal weighting mechanism to support multiple ranks. Although dynamic rank selection offers flexibility, the paper does not sufficiently demonstrate its practical necessity or establish whether it provides meaningful deployment efficiency benefits.
  2. Insufficient experimental comparisons. The evaluation is restricted to two Llama models and a single GSM8K benchmark, limiting evidence of generalizability. Moreover, the paper lacks systematic comparisons with independently trained LoRA adapters at each target rank, including their accuracy and overall training costs.
Questions for authors
  1. Practical utility: Could the authors provide deployment experiments to demonstrate whether dynamic rank selection significantly reduces inference latency, memory usage, or computational costs?
  2. Comparison with independently trained LoRA: Could the authors compare Matryoshka LoRA with standard LoRA trained independently at each target rank, considering both accuracy and total training costs?
  3. Generalizability: Could the authors extend the evaluation to code generation, commonsense reasoning, or language understanding tasks to demonstrate effectiveness beyond GSM8K?
3 comments
AuthorsOpenReview ↗

Rebuttal Comment for Reviewer JD5q

We would like to thank the reviewer for their feedback, we address your concerns below.

Q1: About practical utility

We provide an example in the context of choosing a dynamic rank for serving multiple adapters. Next, we use the general notation X∈(m,n)X \in (m, n) to denote that XX has mm rows and nn columns. Considering the pretrained weight W∈(m,n)W \in (m, n), we want to serve NN LoRA adapters A∈(N,m,R),B∈(N,R,n)A \in (N, m, R), B \in (N, R, n) and the inputs X∈(N,d,m)X \in (N, d, m). The forward pass for LoRA is Y=X(W+AB)=XW+XABY = X (W + AB) = XW + XAB. When NN is large, the product XABXAB becomes expensive, as the multiplication cost is O(NdmR)O(NdmR) for C=XAC=XA and O(NdRn)O(NdRn) for C⋅BC \cdot B, resulting in a total complexity of O(NdR(m+n))O(NdR(m+n)). Given a target performance budget (for example throughput), the only way to reduce the complexity is by reducing the rank RR. Of course, this can be solved by using LoRA adapters independently trained with different ranks or using DyLoRA. First, LoRA would be expensive, as it would multiply the training costs by the total number of supported ranks and second, DyLoRA might not be accurate enough. Through our MatryoshkaLoRA work, we aim to solve both these issues by keeping the training cost low through the surrogate objective (via the scaling matrix PP), which allows obtaining accurate prefix adapters.

Q2: About comparison with independently trained LoRA

We showed comparisons between MatryoshkaLoRA and LoRA trained independently at each target rank in Table 1. We are kindly asking the reviewer to check our newer revision of the paper, where we marked the new changes in blue text.

Q3: About generalizability

We add some more experiments on the ARC-Challenge from the Commonsense Reasoning benchmark in addition to our existing GSM-8k results. For this experiment, we fine-tune Llama-3.1-8B using maximum rank R=32R = 32, scaling sr=1/rs_r = 1/r and experiment with S∈all,pow2S \in \\{all, pow2\\} and λr∈1,1/∥S∥\lambda_r \in \\{1, 1/ \|S\|\\} for MatryoshkaLoRA. We report the run with the highest log-AURAC metric. We use the same learning rate grid of 20 values and 3 seeds for each (60 runs in total). We retain the model-family generalization as a limitation of our work.

In our results shown below, we show the results for LoRA, DyLoRA and MatryoshkaLoRA, as well as differences between MatryoshkaLoRA and each baseline. Our approach consistently yields higher accuracy across all ranks compared to LoRA and DyLoRA baselines, which also translates to log-AURAC.

AdapterLRSsrs_rλr\lambda_rRank 1Rank 2Rank 4Rank 8Rank 16Rank 32Alog-A
LoRA1e-3all1/r1/r-56.0.456.0_{.4}57.1.557.1_{.5}57.8.857.8_{.8}58.8.358.8_{.3}58.7.258.7_{.2}59.1.259.1_{.2}58.6.258.6_{.2}58.0.358.0_{.3}
DyLoRA1e-3all1/r1/r-54.71.254.7_{1.2}57.0.657.0_{.6}56.9.456.9_{.4}58.3.458.3_{.4}58.8.358.8_{.3}58.0.558.0_{.5}58.2.358.2_{.3}57.5.357.5_{.3}
M-LoRA2e-3pow21/r1/r1157.8.357.8_{.3}58.4.758.4_{.7}59.3.259.3_{.2}59.5.659.5_{.6}60.3.660.3_{.6}60.4.960.4_{.9}59.9.659.9_{.6}59.3.559.3_{.5}
M-LoRA − LoRA+1.8+1.3+1.5+0.7+1.6+1.3+1.3+1.3
M-LoRA − DyLoRA+3.1+1.4+2.4+1.2+1.5+2.4+1.7+1.8

Regarding MatryoshkaLoRA, I appreciate the additional experiments and clarifications. However, my concerns about the limited methodological novelty still remain. Therefore, I will maintain my original rating of 2.

AuthorsOpenReview ↗

Reply to the Official Comment by Reviewer JD5q

Dear reviewer,

Our work does not mention SAE-based feature alignment at all. To the best of our knowledge stands for Sparse Auto Encoder, however our work is part of the LoRA fine-tuning literature. Your current comment seems orthogonal to our contribution. Could you please clarify your comment?

Thank you, The Authors

Reviewer_1P94OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

(All my reviews are rewritten)

This paper trains one LoRA adapter so that its rank prefixes can be used at inference. MatryoshkaLoRA puts a fixed diagonal matrix PP between the two LoRA factors. With a different PP, it becomes LoRA or DyLoRA. Theorem 3.1 bounds the gap between the multi-rank objective and the trained surrogate. Experiments use Llama 1B and 8B on GSM-8k, against LoRA and DyLoRA.

Critical strengths

S1. A simple idea with a clear effect at 1B. It adds no trainable parameters. In Table 1 at r = 16, M-LoRA gets 39.4, while DyLoRA gets 35.3 and LoRA 34.5.

S2. Open reporting. The paper shows that M-PiSSA does not beat PiSSA. It says clearly that a surrogate is trained.

Critical weaknesses

W1. Limited methodological novelty. Also, the gain is not shown to come from the nested structure.

My main concerns include: 1) LoRA and DyLoRA fit the same form with a different PP. The core change is a fixed reweighting of LoRA components; 2) The paper says PP works by scaling the gradients. But AdamW normalizes each coordinate, and the paper does not discuss this; 3) The 1B learning rate grid is narrow, from 1e-5 to 8e-5. The chosen rates are not reported. There is no LoRA control with a uniform PP of matched scale. The 1B gain may come from a larger step size, not the nested shape.


W2. The evidence does not support the claims at this level of detail.

My main concerns include: 1) At 1B, the baselines are hardly better than the pre-trained model, 34.7. The paper says a larger grid could help, and this may also hold for the baselines; 2) In the Tables 2 and 3, they give different numbers for the same 8B, 8-shot, R = 32 setting. Table 2 reports log-A 79.4, higher than both Table 3 rows; 3) The text does not match the tables. The 78.6% result comes from R = 256, not R = 32. There LoRA reaches 78.0; 4) Table 2 has no standard deviations. At 8-shot, the gaps over the best baseline are only 0.5 to 0.9 points; 5) M-LoRA is run with two choices of SS and the better one is reported. The baselines do not get this choice.

Questions for authors

Questions

Q1. Which data split did you use to select models? I could not find a validation split.

Q2. Which learning rate did you choose for each method and R in Table 1? Please add a wider 1B grid and a uniform PP control.

Q3. Does the theory cover any reported run? The vanishing result needs sr=α/rs_r = \alpha/r, but the experiments use sr=1s_r = 1. Please soften the guarantee in the abstract, or report δS\delta_S.

Q4. What happens if you train Eq. 3 directly? One 1B run with R = 32 is enough.

Q5. Do the results hold beyond GSM-8k and Llama? Please add one more task or model family, or soften the claim.


Suggestions

Suggestion 1 Please fix the text-table mismatches in W2. Table 4.1 should be Table 1.

Suggestion 2 Please explain how Tables 2 and 3 are related.

5 comments
AuthorsOpenReview ↗

Rebuttal Comment #1 for Reviewer 1P94 addressing weaknesses

We thank the reviewer for their detailed feedback, we address your concerns below.

W1-2: About AdamW scaling

During the forward pass, the weighting introduced by the matrix PP influences the activations and therefore the loss function. Even though the AdamW optimizer performs element-wise scaling, our matrix PP influences the loss in a non-trivial way. We agree with the reviewer that AdamW normalizes each coordinate and might reduce the effect of PP over the gradient, however it does not enforce the nesting structure at all and it cannot remove the effect over the activations and loss.

W1-3: Learning rate grid clarification

We would like to thank the reviewer for this observation. We want to clarify that our learning rate sensitivity started with the initial learning rates 1e−5,2e−5,4e−5,6e−5,8e−5\\{1e-5, 2e-5, 4e-5, 6e-5, 8e-5\\} and 3 seeds for each. We observed that the best performance was obtained for the learning rate 8e-5 and therefore we extended the experiments to the largest grid with 20 values which we also used for the 8B model. After having 3 seeds for each of the 20 learning rates (so 60 runs in total), we noticed that the runs with highest accuracy were still the ones in the initial grid with magnitude 10−510^{-5}, meaning that our initial grid captured the mode of the distribution. In the end, we omitted to change the learning rate grid in our paper. We fixed this in the new revision, as we believe it is crucial to provide clarity and transparency to be able to reproduce the experiments in our paper.

W2-1: Fixes to Table 2 (now split into Table 2 and Table 5)

We would like to thank the reviewer for these observations, we checked again our results in Table 2 against the logs from wandb and detected the source of the inconsistencies you mentioned.

We ran the LoRA baseline with the scaling sr∈1/r,1/rs_r \in \\{1/r, 1/\sqrt{r}\\} across the learning rate grid with 20 values and 3 seeds per run. For the setting 8B / 3-shot / R=32, we correctly added the scaling sr=1/rs_r=1/r in the table for LoRA. However, for the setting 8B / 8-shot / R=32, we incorrectly added the results for sr=1/rs_r = 1 / \sqrt{r} for LoRA. We updated these rows with the correct scaling sr=1/rs_r = 1/r in Table 2 to fix these inconsistencies and we keep the old results in black text and the new results in blue text. For MatryoshkaLoRA on 8B / 8-shot / R=32, we use sr=1/rs_r=1/r and λ=1/∣S∣\lambda=1/|S| to be consistent with the theory. Please check the results in the new revision, where Table 2 was split into two: Table 2 for R=32R=32 and Table 5 for R=256R=256.

Regarding the learning rates, we already provided an explanation for W1-3.

W2-2: Clarifications for Table 3

As part of our checks, we also investigated what is the reason for the mismatch between results in Table 2 and Table 3 and we found the mistake we did: when we manually choose the results from wandb for MatryoshkaLoRA for S=allS=all and S=pow2S=pow2, we have mistakenly sorted the runs based on AURAC instead of log-AURAC. We agree this is even more confusing when coupled with the mismatch in Table 2 mentioned in the previous bullet. Now we updated the values for MatryoshkaLoRA in Table 3 using sr=1/rs_r = 1 / r and λ=1/∣S∣\lambda = 1/|S|. We updated the paper with the correct values to fix these inconsistencies by adding more details.

W2-3: Fixes to results discussion for 3/8-shots and R∈{32,256}R \in \{32, 256\}

We agree with the reviewer that this paragraph requires some refactory to improve the clarity. However, we would like to mention that the paragraph called 3-shot setting aims to describe both settings with R∈32,256R \in \\{32, 256\\}. We wrote a dedicated paragraph for 3/8-shots and R=32,256 to provide individual discussion for the results in each category to have a clear presentation of our results. Please check Tables 2 and 5 in the current revision.

W2-4: Adding standard deviations for Table 2

At lines 301-302, we refer to the R=256R=256 runs as “large-scale” experiments because the evaluation on 20 learning rates, 3 seeds and 9 ranks is challenging for our compute budget on the 8B model, therefore for 8−shots/R=2568-shots/R=256 we ran only one seed during the preparation of the paper (we have 3 seeds for 3-shots and R=256R=256). We managed to run 2 more seeds for the 3-shot setting and still keep 1 seed for 8B/8-shot/R=256. For R=32R=32 we have 3 seeds for each experiment and we report the standard deviations in the updated table in the paper revision (see Tables 2 and 5).

W2-5: Clarifications on training ranks set SS

The LoRA baseline does not require two choices for SS. In the case of DyLoRA, we follow the standard recipe described in the paper and sample the ranks uniformly from the set ∣S∣|S| that contains all ranks from 11 to RR. Since MatryoshkaLoRA takes into consideration all prefixes of rank r∈Sr \in S, we experimented with two choices of SS: all and 2^k.

AuthorsOpenReview ↗

Rebuttal Comment #2 for Reviewer 1P94 addressing questions 1-4

Q1: Data Split

We perform learning rate sensitivity runs for all methods by fine-tuning on the entire training set of GSM-8k and report the accuracy computed on the test set across multiple ranks. We do not isolate any validation set from the training set. We agree that this distinction should have been stated explicitly and we will fix this in the next revision.

Q2: Selected learning rates

We added the learning rates to all tables in our paper in blue text. We provide a detailed set of results for sr=1/rs_r = 1/r and λr∈{1,1/∣S∣}\lambda_r \in \{1, 1/|S|\} for MatryoshkaLoRA in Table 6 in the new revision of the paper. In Table 6, we show how MatryoshkaLoRA behaves under different choices of srs_r and λr\lambda_r.

Concretely, using λ=1/∣S∣\lambda = 1 / |S| does not translate to higher accuracies for any of the settings R=16R=16 or R=32R=32. However, when using λ=1\lambda = 1, the accuracy increases significantly compared to the pretrained model. This holds for both choices of SS (all and pow2).

This way, we make sure the scales of MatryoshkaLoRA match the LoRA scales to present our results transparently.

Q3: Clarifying theory-experiments link and reporting rank dispersion

We add the results for sr=1/rs_r = 1/r and λr=1/∣S∣\lambda_r = 1 / |S| for MatryoshkaLoRA for the 1B model, which we report in the updated tables in the paper, emphasized in blue text. Therefore, we make sure the theory clearly covers the experiments. We thank the reviewer for pointing this out and we will fix this.

In addition, we compute the rank dispersion δS\delta_S for MatryoshkaLoRA and DyLoRA using λr=1/∣S∣\lambda_r = 1 / |S|, according to Equation 14 in Appendix B. Since LoRA is a single rank method, we implement Equation 14 with the same λr\lambda_r to have the same order of magnitude. When computing δS\delta_S, we average all rank dispersions computed at each layer for the three methods (the 1B model has 112 parameter matrices which we fine-tune). We report these values below for the 1B and GSM-8k:

  • δS(LoRA)=1.46\delta_S(LoRA) = 1.46
  • δS(DyLoRA)=0.57\delta_S(DyLoRA) = 0.57
  • δS(MatryoshkaLoRA:sr=1/r,λr=1/∣S∣,S=all)=0.19\delta_S(MatryoshkaLoRA: s_r = 1/r, \lambda_r=1 / |S|, S=all) = 0.19
  • δS(MatryoshkaLoRA:sr=1,λr=1,S=pow2)=1.2\delta_S(MatryoshkaLoRA: s_r = 1, \lambda_r=1, S=pow2) = 1.2
  • δS(MatryoshkaLoRA:sr=1,λr=1,S=all)=0.16\delta_S(MatryoshkaLoRA: s_r = 1, \lambda_r=1, S=all) = 0.16
  • δS(EQ−3)=0.19\delta_S(EQ-3) = 0.19

We observe that all runs with MatryoshkaLoRA achieve lower rank dispersion than LoRA and DyLoRA and similar or lower than EQ-3. This is an indication that our surrogate loss matches the theory. We observe that despite using sr=1,λr=1s_r=1, \lambda_r=1, when using all ranks in SS, we can achieve a lower rank dispersion, which in our opinion is quite surprising. We hope these ablations clarify the link between theory and practice.

Q4: Training Equation 3

We agree that directly optimizing Equation 3 would provide a useful reference for assessing the quality of the proposed surrogate. The computational cost of Equation 3 would scale with the number of supported ranks: for S=1,2,4,8,16,32S = \\{1, 2, 4, 8, 16, 32\\}, we would have to perform a total of ∣S∣=6|S|=6 forward and backward passes for each batch of data. The goal of our work is to use the surrogate loss, which allows us to perform only one forward and backward pass, thus being on par with LoRA and DyLoRA.

We implement Equation 3 (EQ-3) on the 1B model on GSM-8k with rank 32, sr=1/rs_r = 1 / r and λr∈1,1/∣S∣\lambda_r \in \\{1, 1 / |S|\\} (sum vs averaging) with S=allS = all and S=pow2S = pow2 and run it on the same grid of 20 learning rates with only one seed. For S=pow2S = pow2, EQ-3 is 6 times slower and for S=allS = all, it is 32 times slower. Despite using significantly more compute, the maximum log-AURAC value is at most 35.7%, which is an indication that implementing EQ-3 is both computationally expensive and inaccurate. We show these results in Table 6 in the new revision of our paper.

Implementation details: to avoid performing as many backward passes as forward passes, we can store the losses from each forward pass in a list and backpropagate only once through the mean of the losses. This would be mathematically equivalent to performing backwards after each forward (of course,with scaling by ∣S∣|S|). However, this would increase the memory usage and in our experiment we perform forward and backward ∣S∣|S| times each. Even if we apply this trick to reduce the number of backward passes from ∣S∣|S| to 11, we still have to perform ∣S∣|S| forward passes.

AuthorsOpenReview ↗

Rebuttal Comment #3 for Reviewer 1P94 addressing question 5

Q5: Additional tasks: ARC-Challenge

We add some more experiments on the ARC-Challenge from the Commonsense Reasoning benchmark in addition to our existing GSM-8k results. For this experiment, we fine-tune Llama-3.1-8B using maximum rank R=32R = 32, scaling sr=1/rs_r = 1/r and experiment with S∈{all,pow2}S \in \{all, pow2\} and λr∈1,1/∥S∥\lambda_r \in \\{1, 1/ \|S\| \\} for MatryoshkaLoRA. We report the run with the highest log-AURAC metric. We use the same learning rate grid of 20 values and 3 seeds for each (60 runs in total). We retain the model-family generalization as a limitation of our work.

In our results shown below, we show the results for LoRA, DyLoRA and MatryoshkaLoRA, as well as differences between MatryoshkaLoRA and each baseline. Our approach consistently yields higher accuracy across all ranks compared to LoRA and DyLoRA baselines, which also translates to log-AURAC.

AdapterLRSsrs_rλr\lambda_rRank 1Rank 2Rank 4Rank 8Rank 16Rank 32Alog-A
LoRA1e-3all1/r1/r-56.0.456.0_{.4}57.1.557.1_{.5}57.8.857.8_{.8}58.8.358.8_{.3}58.7.258.7_{.2}59.1.259.1_{.2}58.6.258.6_{.2}58.0.358.0_{.3}
DyLoRA1e-3all1/r1/r-54.71.254.7_{1.2}57.0.657.0_{.6}56.9.456.9_{.4}58.3.458.3_{.4}58.8.358.8_{.3}58.0.558.0_{.5}58.2.358.2_{.3}57.5.357.5_{.3}
M-LoRA2e-3pow21/r1/r1157.8.357.8_{.3}58.4.758.4_{.7}59.3.259.3_{.2}59.5.659.5_{.6}60.3.660.3_{.6}60.4.960.4_{.9}59.9.659.9_{.6}59.3.559.3_{.5}
M-LoRA − LoRA+1.8+1.3+1.5+0.7+1.6+1.3+1.3+1.3
M-LoRA − DyLoRA+3.1+1.4+2.4+1.2+1.5+2.4+1.7+1.8

We also softened the claim to avoid overstating our work.

Suggestions

We would like to thank the reviewer for pointing out these issues, we fixed them in the new revision that we uploaded during the rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.