acceptodds
Under review as a conference paper at ICLR 2027

How Much Adaptation Space Does LoRA Need?

Abstract

Low-rank adaptation (LoRA) and its variants provide memory- and compute-efficient alternatives to full-parameter fine-tuning. Still, it remains unclear how aggressively we can restrict their adaptation space without sacrificing performance and generalization. We study this question through structured extensions of LoRA—Cheap LoRA (cLA), which trains a single low-rank factor while fixing the other in a compute-efficient way, and LA, a chained-circulant variant of cLA. We interpret cLA as a structured form of asymmetric LoRA and, more broadly, as a controlled column-subspace restriction of full fine-tuning. We derive information-theoretic generalization error bounds for these restricted adaptation classes along with their base variants, providing a theoretical perspective on their generalization behavior. Empirically, we evaluate **11 fine-tuning methods** across **10 pre-trained models and 14 datasets** using downstream performance, generalization, loss-landscape, and spectral analyses. Across tasks, cLA and LA remain competitive with baselines matched on trainable parameters while additionally reducing training time relative to LoRA by up to 14.4% and peak GPU memory by up to 11%. Our results suggest that substantially restricting the adaptation subspace can preserve effective transfer while reducing fine-tuning cost as an additional benefit.

Primary area:
optimization
Keywords:
parameter-efficient fine-tuning, low-rank adaptation, LoRA, structured sparsity, asymmetric LoRA, generalization bounds, training efficiency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NuiGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Cheap LoRA (cLA), which fixes one LoRA factor as a column selector and only trains the other factor, thereby restricting the update to selected columns. Its chained variant, c3LA, shifts the selected columns across training stages to expand cumulative coverage while keeping the number of simultaneously trainable parameters fixed. Experiments on 10 pretrained models and 14 datasets show that substantial restriction to the adaptation space can often retain most of LoRA’s performance with lower training time and memory.

Critical strengths
  1. The paper evaluates 11 fine-tuning methods across 10 pretrained models and 14 datasets. This provides a systematic evaluation of the empirical applicability of the proposed method.
  2. The method is straightforward and easy to implement. On suitable tasks, it reduces training time and peak GPU memory while retaining competitive performance, and the paper also provides information-theoretic generalization bounds.
Critical weaknesses
  1. Limited new insight. The paper shows that LoRA’s adaptation space can often be strongly restricted, but it does not provide a transferable minimum space or a rule for different tasks. As acknowledged in Section 5, “the bounds therefore do not distinguish when a particular column restriction will help or hurt empirically.”
  2. Lack of empirical guarantees or guidance. The performance of the proposed method is not guaranteed, with substantial drops observed on some tasks, such as TinyLlama/OpenBookQA in Table 16. Since the trade-off varies substantially across tasks, the paper provides limited guidance on whether Cheap LoRA will be effective for a new task.
  3. No task-aware subspace selection. The columns are selected in advance, randomly sampled, or shifted according to a fixed schedule. A task-aware method for selecting important subspaces could be useful, but this is not considered or discussed in the proposed method.
Questions for authors

Section 4.2 reports the gap between the training and test losses but does not report the two losses separately. Could the authors report both losses, since a small gap may also result from underfitting or failed optimization?

3 comments

We thank the reviewer for their assessment and respond to each point below. We have revised the paper in response to the reviewers’ comments and marked the relevant changes in blue.

W1. No transferable minimum space or selection rule.

This is an excellent question. However, if this question were answered, there would be no further research for the field of parameter-efficient fine-tuning. Finding the task-dependent, minimal subspace to adapt any model to any task is the goal of PEFT as a whole. This paper takes a small step towards it by assessing the ability of a common PEFT method (LoRA) to adapt to tasks under heavily restricted adaptation spaces, and analyzes that ability theoretically, in terms of diagnostic tools (loss landscape, intruder dimensions), and empirically. We also highlight that leaving the minimum transferable space open is also common in the LoRA literature, where the rank is typically chosen high enough to ensure that the space can adapt to the task. The original LoRA paper calls the relationship between model size and the optimal rank an open question ([2], §H.2), and later work notes that optimizing the rank requires an exhaustive search (DyLoRA [3]) or allocates the rank budget across layers during training (AdaLoRA [4]).

The paper offers the following insights into LoRA's capacity under heavily restricted adaptation spaces:

(1) A column-restricted LoRA family that naturally arises from this goal and connects LoRA to another PEFT family (PaCA [1]; §2.3, p. 4, lines 185–211 and Appendix B, pp. 16–17, lines 848–884; Table 4, p. 17, lines 886–907).

(2) Information-theoretic generalization bounds for all considered parameterizations, showing that the column restriction does not enlarge the bound of its asymmetric parent (§3.2, p. 6, lines 295–299; Table 1, p. 5, lines 216–232).

(3) A matched-budget comparison of 11 methods on 10 models and 14 datasets covering performance, training time, memory, and empirical generalization (§4, pp. 6–9, lines 301–475) to assess the performance of the column-restricted variants and highlight a benefit in studying this direction: that the column restrictions can be used to increase compute efficiency. The restricted variants retain 97.3–98.1% of LoRA's performance at 6.3–7.8% lower step time (§4.1, p. 7, final paragraph beginning at line 375), where efficiency is an ad-hoc benefit.

The paper’s scientific contribution therefore lies in the analytical connections, controlled extensions, and verified performance–cost consequences of restricting adaptation—not in claiming that the selector’s identity creates a new update class. We respectfully ask that novelty be assessed against these contributions, which the equivalence alone neither establishes nor explains.

W2. No guarantee; large drop on TinyLlama/OpenBookQA.

The drop for adapting TinyLlama on OpenBookQA is not unique to column restriction, rather, it is shared among all variants that freeze the A matrix. Every method that freezes A loses to LoRA by roughly 5-7% on this task, as does full fine-tuning (FFT). For example, LoRA reaches 56.72% compared with 51.88% for Asymmetric LoRA and 51.12% for FFT. cLA and its variants achieve 50.04-50.80%, staying within 1.84% of Asymmetric LoRA here (Table 22). Of cLA's 6.68-point gap to LoRA, 4.84 points are already present in the Asymmetric LoRA baseline.

This gives the guidance the reviewer asks for. These results support using Asymmetric LoRA's performance in tasks as a practical guide to when column restriction is promising. cLA is within 2.2% of Asymmetric LoRA on 21 of the 23 non-perplexity model–dataset pairs (Table 22, p. 45, lines 2381–2419). On the same OpenBookQA task with LLaMA-3-8B, Asymmetric LoRA reaches 87.84% versus 87.64% for LoRA, and cLA reaches 88.32% (Table 22). Generally: the column-restricted variants serve to amplify the trade-off of Asymmetric LoRA relative to LoRA, giving more training speed for a small amount of performance while sometimes still being the best (§4.1).

W3. No task-aware subspace selection.

Please see our response to W1: adaptive selection is discussed as the main open direction (§5) and is outside the question the paper asks.

Question. Report training and test losses separately.

This is a great question. We now report the requested training and test losses in Appendix E.9 (Table 21, p. 44, lines 2322–2372), the corresponding generalization errors in Table 20 (p. 43, lines 2278–2317), and task performance in Appendix E.10 (Table 22, p. 45, lines 2381–2419). These results cover all 24 model–dataset pairs and all ten methods of Table 20.

If these responses address the reviewer's concerns, we kindly ask the reviewer to reconsider their score.

References

[1] Woo et al., PaCA: Partial Connection Adaptation for Efficient Fine-Tuning, ICLR 2025.

[2] Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022.

[3] Valipour et al., DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation, EACL 2023.

[4] Zhang et al., AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning, ICLR 2023.

AuthorsOpenReview ↗

Can Reviewer NuiG kindly let us know what else we need to do to get the reviewer's attention to our rebuttal and revised version?

Dear Reviewer NuiG,

We would greatly appreciate it if the reviewer could provide any comments on our work after we submitted the rebuttal and the revised version of the paper, as the author-reviewer discussion period will end in less than 24 hours.

We believe we have clarified all questions related to novelty and the task-specific subspace selection protocol. The other issues that the reviewer asked for further clarification on, that is,

  1. Large drop on TinyLlama/OpenBookQA.
  2. Report training and test losses separately,

are provided with objective details.

We would again like to reiterate that the paper’s scientific contribution therefore lies in the analytical connections, controlled extensions, and verified performance–cost consequences of restricting adaptation—not in claiming that the selector’s identity creates a new update class. We respectfully ask that novelty be assessed against these contributions, which the equivalence alone neither establishes nor explains.

If these responses address the reviewer's concerns, we kindly ask the reviewer to reconsider their score.

Sincerely,

The Authors

Reviewer_6YyMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores a basic issue, i.e., how aggressively we can restrict their adaptation space without sacrificing performance and generalization. Based on this issue, the authors study this question through structured extensions of LoRA. Besides, the authors interpret Cheap LoRA (cLA) as a structured form of asymmetric LoRA and provide a theoretical perspective on their generalization behavior. Finally, the authors conducted extensive experiments on 11 fine-tuning methods across 10 pre-trained models and 14 datasets using downstream performance, generalization, loss-landscape, and spectral analyses. The result denotes the proposed methods have promising competitive performance.

Critical strengths

S1: This paper studies a practically quesiton, i.e., how aggressively we can restrict their adaptation space without sacrificing performance and generalization.

S2: This paper provides an interesting view about the relationship between adaptation-space restrictions, downstream performance, and computational efficiency.

S3: This paper provide suffice experiments and results to verify the effectiveness of the proposed method.

Critical weaknesses

W1: I think the methodology of this paper is similar to that of recent works, e.g., PaCA and SBoRA, as both fix A using standard basis vectors and only optimize B. In this paper, although the author discusses the connection with these works, the author's own technical contribution does not seem sufficiently clear. I think the author should clarify the difference with these existing works.

W2: I think the theoretical analysis of this paper does not fully explain the main empirical findings. In Table 1, the generalization boundaries are the same for cLA and asymmetric LoRA, even though their search spaces differ. Therefore, I think the author should explain why certain subspace constraints are effective.

W3: The authors should provide some experimental comparison with SBoRA, which will be helpful to better demonstrate the contribution of the proposed methods.

W4: In the ablation section. I suggest the authors provide a controlled ablation to isolate these two factors. The current experiment does not clearly distinguish between the effects of changing the selected column and periodically resetting the adapter.

W5: In section 4.1, I think the parameter-matching strategy does not guarantee identical trainable parameter counts, particularly for decoder models. Comparing results under a fully matched parameter budget would make the reported efficiency gains more convincing.

Questions for authors

Please refer to W1, W2, and W5.

7 comments

We thank the reviewer for their time and respond to each point below. We have revised the paper in response to the reviewers’ comments and marked the relevant changes in blue.

W1. Similarity to PaCA and SBoRA; unclear contribution.

The column-restricted methods are the instruments used to study LoRA's capacity to update models under heavily restricted adaptation spaces, and to give an example of how to leverage that adaptability to increase compute efficiency throughout training. They "serve as vehicles for a central contribution of this work: a systematic theoretical and empirical analysis of the benefits and task-dependent limitations of restricting LoRA's adaptation space" (Appendix A, p. 16, lines 842–844).

Built on this column-restricted family of LoRA variants, our contributions are:

(1) Generalization bounds for every variant (Table 1, p. 5, lines 216–232) which show that restricting LoRA's adaptation space via A does not increase the information-theoretic bound of its asymmetric parent and that chaining (which is a way to offset the restricted adaptivity by moving the columns throughout training) enlarges it.

(2) A matched-budget comparison of 11 fine-tuning methods on 10 models and 14 datasets (Table 3, p. 8, lines 378–392; Table 20, p. 43, lines 2278–2317; Table 22, p. 45, lines 2381–2419), which validates the performance of the column-restricted variants and their runtime efficiency: they retain 97.3–98.1% of LoRA's performance while reducing training-step time by 6.3–7.8%.

(3) An analysis across types of restriction, chained and randomized (Table 11, p. 34, lines 1782–1819; Table 12, p. 35, lines 1836–1855; Table 22, p. 45, lines 2381–2419).

(4) A diagnostic analysis of the generalizability of the models via loss-landscape and intruder-dimension perspectives (§4.2.1, p. 9, lines 453–475; Figure 1, p. 2, lines 54–68; Figure 4, p. 9, lines 432–443), which explain the empirical generalization of the restricted variants less consistently than our theoretical bounds.

(5) The connection of these variants to current literature, such as PaCA [1], and how this family can act as a guide on how to leverage the benefits of future research in PaCA and LoRA together.

The difference from the two works the reviewer names is the substantial analyses (theoretical, empirical, and diagnostic) listed above, the chained variants (c³LA, r-c³LA, and C-PaCA) that have no counterpart in either paper, and the connection of the two families of PEFT methods.

Additionally, we note, as mentioned to Reviewer cUoT:

The connection to PaCA serves as an additional ad-hoc benefit of the study, but PaCA itself is less adaptive to future LoRA literature than our column-restricted variants, which preserve LoRA’s structure. The LoRA ecosystem is built on this structure: multi-adapter serving (S-LoRA [5], Punica [6]), quantized bases (QLoRA [7]), and extensions such as DoRA [8] all rely on it. Applying them to PaCA can require targeted, method-specific extensions for each method, as quantization did with QPaCA ([1], §4.3). This is the value of connecting PaCA to the LoRA family via these variants. We gain the benefits of PaCA, and preserve the benefits of LoRA.

The paper’s scientific contribution therefore lies in the analytical connections, controlled extensions, and verified performance–cost consequences of restricting adaptation—not in claiming that the selector’s identity creates a new update class. We respectfully ask that novelty be assessed against these contributions, which the equivalence alone neither establishes nor explains.

W2. Theory does not explain the empirical findings.

The theory addresses the concern that a column-restricted subspace variant does not produce a worse generalization upper bound compared to its dense counterpart; the restriction does not increase the information-theoretic upper bound (Table 1). We have explicitly mentioned it in §3.2 of the revised version of the paper (p. 6, lines 295–299).

Additionally, the bounds help predict empirical behavior more consistently compared to popular alternative diagnostic tools, such as loss-landscape analysis [9] or intruder-dimension analysis [10] (§4.2.1, p. 9, lines 453– 475). The theory shows that chained methods enlarge the theoretical bound, and empirically chained variants have the larger empirical generalization error compared to their non-chain counterparts (on 18 of 24 model–dataset pairs for c³LA against cLA, and on 17 of 24 for RAC-LoRA against Asymmetric LoRA; Table 20, p. 43, lines 2278–2317). For example, on DeBERTa-v2-XXL fine-tuned on MRPC, the generalization error rises from 0.243 for cLA to 0.426 for c³LA (Table 20).

Some discrepancy between theory and empirics is expected. The actual generalization error cannot be computed over the entire distribution, so we use the held-out test set to estimate the population loss (§4.2; Appendix E.9). Factors such as the dataset and optimizer can sometimes affect this measured error. The theory and experiments, together, give a more complete view of LoRA's capability under extreme restriction, as the conclusion states: "the theoretical and empirical results should be viewed as complementary rather than as a single criterion for predicting the performance of a restricted adaptation space" (§5).

W3. No experimental comparison with SBoRA.

We thank the reviewer for their suggestion. We additionally have run SBoRA-FA and SBoRA-FB [2] in a subset of our study over three seeds and report the performance and training-step time below (Table R3). We have added these results to the Appendix E.5 of the revised paper (Table 15, p. 36, lines 1922–1937).

Table R3: The performance of SBoRA-FA, SBoRA-FB, cLA, and r-cLA on adapting TinyLlama and LLaMA-3-8B to OpenBookQA. cLA and r-cLA are five seed main-experiment results (Table 22). SBoRA performance results are means ± sample standard deviations over three seeds. We report the test accuracy (%) at the best validation checkpoint and, in parentheses below it, the total training-step time in minutes. In Tables R3-R5, new performance values are three-seed means ± sample standard deviations; new training-step times are single seed-0 measurements.

Model / Dataset\textbf{Model / Dataset}SBoRA-FA\textbf{SBoRA-FA}cLA\textbf{cLA}r-cLA\textbf{r-cLA}SBoRA-FB\textbf{SBoRA-FB}
TinyLlamaOpenBookQA\begin{gathered}\text{TinyLlama}\\\\\text{OpenBookQA}\end{gathered}50.87 ± 1.10(7.7)\begin{gathered}\text{50.87 ± 1.10}\\\\\text{(7.7)}\end{gathered}50.04 ± 4.06(7.3)\begin{gathered}\text{50.04 ± 4.06}\\\\\text{(7.3)}\end{gathered}50.12 ± 6.59(7.4)\begin{gathered}\text{50.12 ± 6.59}\\\\\text{(7.4)}\end{gathered}50.87 ± 2.12(8.2)\begin{gathered}\text{50.87 ± 2.12}\\\\\text{(8.2)}\end{gathered}
LLaMA-3-8BOpenBookQA\begin{gathered}\text{LLaMA-3-8B}\\\\\text{OpenBookQA}\end{gathered}87.07 ± 0.83(11.4)\begin{gathered}\text{87.07 ± 0.83}\\\\\text{(11.4)}\end{gathered}88.32 ± 0.72(10.7)\begin{gathered}\text{88.32 ± 0.72}\\\\\text{(10.7)}\end{gathered}86.88 ± 1.36(10.9)\begin{gathered}\text{86.88 ± 1.36}\\\\\text{(10.9)}\end{gathered}78.33 ± 0.12(12.3)\begin{gathered}\text{78.33 ± 0.12}\\\\\text{(12.3)}\end{gathered}

r-cLA is our counterpart of SBoRA-FA (Appendix A). r-cLA reaches 50.12% accuracy versus 50.87% for SBoRA-FA on TinyLlama and 86.88% versus 87.07% on LLaMA-3-8B, with lower training-step time on both models. r-cLA uses a randomly positioned contiguous block of columns (Algorithm 2), enabling efficient slicing of the input activations. SBoRA-FB freezes B, the factor identified as more important in the analysis of [3] (their Appendix B). We emphasize that the focus of the study is not on the novelty of these methods, but the analysis of them (see our response to W1).

W4. Column changes and adapter resets not isolated.

This is a great observation, and we would like to clarify how these factors relate because, by construction, they are not independent. c³LA moves the trainable columns throughout training. It is necessary to merge the current BA update into the pretrained weights and re-initialize B to zero (and move A appropriately) to avoid incurring the increasing inference cost of having additional LoRA modules and to ensure each LoRA update changes the columns it targeted when optimizing the LoRA module (Algorithm 3). A reset without a column change would leave the update targeting the same columns that were just merged into the base model, so resetting can still be tested on its own, with the selected columns held fixed. We therefore ran an additional experiment on a subset of our study that applies the same reset that c³LA does but does not move the columns (Table R4). Resetting fixed columns marginally improved the performance of cLA when adapting TinyLlama to OpenBookQA, and degraded cLA when adapting LLaMA-3-8B to the same dataset.

Table R4: Reset without a column change, at rank 32. The fixed column reset control uses the reset schedule of c³LA and never moves the selected columns. Each cell reports test accuracy (%) at the best-validation checkpoint and, in parentheses below it, the total training-step time in minutes. Values from Table 22 in the paper are means ± sample standard deviations over five seeds. The recent experiments are for three seeds.

Model / Dataset\textbf{Model / Dataset}cLAno reset, fixed columns\begin{gathered}\textbf{cLA}\\\\\text{no reset, fixed columns}\end{gathered}Fixed Column Resetreset, fixed columns\begin{gathered}\textbf{Fixed Column Reset}\\\\\text{reset, fixed columns}\end{gathered}c³LAreset, moving columns\begin{gathered}\textbf{c³LA}\\\\\text{reset, moving columns}\end{gathered}
TinyLlamaOpenBookQA\begin{gathered}\text{TinyLlama}\\\\\text{OpenBookQA}\end{gathered}50.04 ± 4.06(7.3)\begin{gathered}\text{50.04 ± 4.06}\\\\\text{(7.3)}\end{gathered}50.80 ± 0.92(7.4)\begin{gathered}\text{50.80 ± 0.92}\\\\\text{(7.4)}\end{gathered}50.36 ± 4.10(7.4)\begin{gathered}\text{50.36 ± 4.10}\\\\\text{(7.4)}\end{gathered}
LLaMA-3-8BOpenBookQA\begin{gathered}\text{LLaMA-3-8B}\\\\\text{OpenBookQA}\end{gathered}88.32 ± 0.72(10.7)\begin{gathered}\text{88.32 ± 0.72}\\\\\text{(10.7)}\end{gathered}87.00 ± 0.92(10.6)\begin{gathered}\text{87.00 ± 0.92}\\\\\text{(10.6)}\end{gathered}87.92 ± 1.15(10.7)\begin{gathered}\text{87.92 ± 1.15}\\\\\text{(10.7)}\end{gathered}
Reviewer_cUoTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how restricting LoRA's adaptation space affects downstream performance and fine-tuning cost. It proposes Cheap LoRA (cLA), which fixes one low-rank factor to a column selector so that updates act only on selected columns of the pretrained weights, together with a circulant chained extension (c3LA) and randomized versions of both. The authors analyze these methods through information-theoretic generalization error bounds and compare them across 10 pretrained models and 14 datasets. Empirically, the proposed methods retain downstream performance close to standard LoRA on average while reducing training time and memory, though the effect varies across models and tasks.

Critical strengths

1. A valuable unifying view. The column-restricted parameterization W+BAPSW + BAP_S (p. 4) brings cLA, c3LA, asymmetric LoRA, RAC and PaCA under a single form, decoupling where an update may act from the parameter count and turning it into an independently adjustable variable. This framing is useful for comparative work on PEFT.

2. Solid evaluation coverage and cost reporting. The evaluation spans language, vision, code and reasoning tasks, the main results report means and standard deviations over five seeds, and training-step time, throughput and peak GPU memory are reported alongside task performance. These measurements help assess the practical performance–cost trade-off of the methods.

Critical weaknesses

1. Limited novelty of the proposed construction. cLA fixes one low-rank factor to a column selector, so its layer update is B∣0B \mid 0: only the rr selected columns of the pretrained weights are trained. Since a matrix with only rr nonzero columns already has rank at most rr, the low-rank factorization imposes no further constraint here, and the reachable set of updates coincides with that of directly updating those rr columns. The paper itself (p. 4) describes PaCA as the same construction without the low-rank constraint and attributes the difference to update parameterization and compute cost rather than adaptation geometry; Table 4 likewise shows broadly similar downstream performance between the corresponding methods. I find the unifying view valuable, but the paper still needs to establish what important and verified benefit this parameterization offers over direct column updates.

2. Efficiency gains lack the relevant baseline. Efficiency is one of the paper's central claims, with the abstract headlining reductions in training time and memory relative to LoRA. However, the main performance–cost comparisons use LoRA at r=16r=16 against single-factor methods at r=32r=32 to approximately match trainable parameter counts. Under this setting, p. 7 reports that the column-restricted variants retain 97.3–98.1% of LoRA's performance while reducing training-step time by 6.3–7.8%, which amounts to trading performance for cost. Once that trade-off is accepted, the most direct alternative is simply to lower the rank of standard LoRA, yet no such comparison is reported; Tables 7–8 do vary rank but do not report time and memory at the corresponding ranks. Without this comparison, gains attributable to the column restriction itself cannot be separated from gains attributable to any equivalent reduction in capacity.

Questions for authors
  1. Since ΔW↦W0+ΔW\Delta W \mapsto W_0 + \Delta W is a bijection for fixed W0W_0 and leaves the mutual information unchanged, why not apply the bound directly to the full adapted model instead of routing through the update-only model and adding ΦW0\Phi_{W_0}?
  2. Table 1 gives Asym-LoRA a tighter bound than LoRA at equal rr, yet under the parameter-matched protocol the two bounds coincide and no corresponding gap appears in the experiments — what does this ordering mean in practice?
  3. Could you report absolute training and test losses separately for the setting in Figure 1, to rule out underfitting as the reason for FFT's smaller generalization gap?
  4. Given that deterministic and random column selection perform comparably, which result supports the conclusion that the structure and scheduling of permitted updates matter?
5 comments

We thank the reviewer for the careful reading and respond to each point below. We have revised the paper in response to the reviewers’ comments and marked the relevant changes in blue.

W1. Novelty: cLA equals direct column updates (PaCA).

To justify the novelty, we first quote from the Conclusion of the paper, (§5, p. 9, lines 471–473), "This paper explicitly asks how much adaptation space LoRA needs for effective downstream adaptation. We study this question by controlling restrictions on the LoRA update. Cheap LoRA (cLA) and c3LA provide controlled ways to impose increasingly structured restrictions.”

Therefore, the central theme of the paper is to assess how far we can restrict LoRA's adaptation space, and what that restriction costs in terms of generalization, performance, and efficiency (§1, p.2, Lines 73–102). The column-restricted family sharply restricts where an update can act while retaining LoRA's parameterization, enabling us to analyze the restriction and measure its practical cost. The novelty is in the analysis (theoretical, empirical, and diagnostic), the connection between LoRA and PaCA [1], and the generation of natural column-restricted variants (C-PaCA, c³LA, r-c³LA). Current LoRA literature lacks a systematic account of how column restriction affects generalization, task performance, and training efficiency of some SotA LoRA variants.

We agree that fixing A=[Ir∣0]A=[I_r\mid0] gives BA=[B∣0]BA=[B\mid0], so cLA and direct updates of the same rr columns have identical reachable updates. The rank bound is automatic. This equivalence is explicitly established in §2.3 (p. 3, lines 157–160) and Appendix B (p. 16, lines 848–855). Our contribution is the controlled characterization of how the location and scheduling of permitted updates affect adaptation at comparable parameter budgets. We state in §2.3 (p. 4, lines 196–201) of the revised version of the paper that the distinction is primarily one of update parameterization and computational cost, rather than adaptation geometry.

The comparison with asymmetric LoRA makes the scientific question concrete. At the same rank, both methods train nrnr parameters per layer, but asymmetric LoRA uses a generally dense fixed projection, whereas cLA confines updates to selected coordinates. This allows us to study update structure independently of parameter count. The c³LA extension additionally separates simultaneous training capacity from cumulative coverage: across kk disjoint stages, its update can cover krkr columns and attain rank up to min⁡(n,m,kr)\min(n,m,kr), while only nrnr adapter parameters remain simultaneously trainable. The framework also produces direct-update counterparts, D-PaCA and C-PaCA, and a shared generalization analysis whose capacity terms depend explicitly on rank, chain length, precision, and sample size. Matching these terms does not imply identical actual generalization. (§1, p. 2, lines 80–91; Appendix B.1, pp. 16–17, lines 861–869; Table 1, p. 5, lines 216–232).

The computational benefit within the LoRA framework is concrete: the selector replaces the dense AxAx projection with indexing, avoiding r(2m−1)r(2m-1) floating-point operations per input vector. (Appendix E.7.1, p. 38, lines 2004–2015)

With our column-restricted family in place, we study this question from two sides:

Theoretically, the column-restricted variants preserve the information-theoretic upper bound of their asymmetric parents (see Q2), while chaining enlarges it. Thus, column restriction preserves the generalization guarantee of its asymmetric parent (Table 1, p. 5, lines 216–232).

Empirically, across 11 fine-tuning methods, 10 pretrained models, and 14 datasets, the column-restricted variants retain, on average, 97.3–98.1% of LoRA's performance while reducing training-step time by 6.3–7.8% (§4.1, p. 7, final paragraph beginning at line 375). This provides an example of how to leverage LoRA's capacity to adapt models under restricted spaces by restricting them; efficiency is an ad-hoc gain of this process.

The connection to PaCA serves as an additional ad-hoc benefit of the study, but PaCA itself is less adaptive to future LoRA literature than our column-restricted variants, which preserve LoRA’s structure. The LoRA ecosystem is built on this structure: multi-adapter serving (S-LoRA [2], Punica [3]), quantized bases (QLoRA [4]), and extensions such as DoRA [5] all rely on it. Applying them to PaCA can require targeted, method-specific extensions for each method, as quantization did with QPaCA ([1], §4.3). This is the value of connecting PaCA to the LoRA family via these variants. We gain the benefits of PaCA, and preserve the benefits of LoRA.

The paper’s scientific contribution therefore lies in the analytical connections, controlled extensions, and verified performance–cost consequences of restricting adaptation—not in claiming that the selector’s identity creates a new update class. We respectfully ask that novelty be assessed against these contributions, which the equivalence alone neither establishes nor explains.

W2. Efficiency baseline: lower-rank LoRA not compared.

We thank the reviewer for this observation. Considering this, we have run a targeted subset of our study on lower ranks of LoRA (r = 8, 4, 2) and report three-seed mean performance alongside seed-0 training-step time and peak GPU memory for the new runs (Table R1). For the reviewer's convenience, we also include the LoRA, cLA, and r-cLA results from Table 22 (p. 45, lines 2381–2419) to compare our methods with the lower-rank LoRA runs. These lower-rank controls now appear in Appendix E.4 (pp. 35–36, lines 1884–1891; Table 14, p. 36, lines 1893–1917).

Table R1: Standard LoRA at lower ranks compared with the column-restricted variants, used to adapt LLaMA-3-8B on OpenBookQA and ViT-Tiny on CIFAR-10. Each cell reports test accuracy (%) at the best-validation checkpoint and, in parentheses below it, the total training-step time in minutes followed by the peak GPU memory. LoRA at r = 16 and the column-restricted variants at r = 32 are the main-experiment runs (Table 22, p. 45, lines 2381–2419).

Model / Dataset\textbf{Model / Dataset}LoRAr = 16\begin{gathered}\textbf{LoRA}\\\\\text{r = 16}\end{gathered}LoRAr = 8\begin{gathered}\textbf{LoRA}\\\\\text{r = 8}\end{gathered}LoRAr = 4\begin{gathered}\textbf{LoRA}\\\\\text{r = 4}\end{gathered}LoRAr = 2\begin{gathered}\textbf{LoRA}\\\\\text{r = 2}\end{gathered}cLAr = 32\begin{gathered}\textbf{cLA}\\\\\text{r = 32}\end{gathered}r-cLAr = 32\begin{gathered}\textbf{r-cLA}\\\\\text{r = 32}\end{gathered}
LLaMA-3-8BOpenBookQA\begin{gathered}\text{LLaMA-3-8B}\\\\\text{OpenBookQA}\end{gathered}87.64 ± 1.23(11.7; 14.32)\begin{gathered}\text{87.64 ± 1.23}\\\\\text{(11.7; 14.32)}\end{gathered}88.13 ± 0.23(11.7; 14.3)\begin{gathered}\text{88.13 ± 0.23}\\\\\text{(11.7; 14.3)}\end{gathered} 87.33 ± 0.64(11.9; 14.29)\begin{gathered}\text{ 87.33 ± 0.64}\\\\\text{(11.9; 14.29)}\end{gathered}86.33 ± 2.14(11.8; 14.28)\begin{gathered}\text{86.33 ± 2.14}\\\\\text{(11.8; 14.28)}\end{gathered}88.32 ± 0.72(10.7; 14.32)\begin{gathered}\text{88.32 ± 0.72}\\\\\text{(10.7; 14.32)}\end{gathered}86.88 ± 1.36(10.9; 14.32)\begin{gathered}\text{86.88 ± 1.36}\\\\\text{(10.9; 14.32)}\end{gathered}
ViT-TinyCIFAR-10\begin{gathered}\text{ViT-Tiny}\\\\\text{CIFAR-10}\end{gathered}95.90 ± 0.11(12.9; 0.94)\begin{gathered}\text{95.90 ± 0.11}\\\\\text{(12.9; 0.94)}\end{gathered}95.68 ± 0.14(12.9; 0.93)\begin{gathered}\text{95.68 ± 0.14}\\\\\text{(12.9; 0.93)}\end{gathered}95.04 ± 0.18(12.6; 0.93)\begin{gathered}\text{95.04 ± 0.18}\\\\\text{(12.6; 0.93)}\end{gathered}94.49 ± 0.21(13.6; 0.93)\begin{gathered}\text{94.49 ± 0.21}\\\\\text{(13.6; 0.93)}\end{gathered}94.99 ± 0.27(12.1; 0.84)\begin{gathered}\text{94.99 ± 0.27}\\\\\text{(12.1; 0.84)}\end{gathered}95.11 ± 0.20(12.9; 0.84)\begin{gathered}\text{95.11 ± 0.20}\\\\\text{(12.9; 0.84)}\end{gathered}

In these comparisons, lowering LoRA's rank does not reproduce cLA's runtime savings. On LLaMA-3-8B/OpenBookQA, cLA achieves 88.32% accuracy with 10.7 minutes of total training-step time compared with 88.13% and 11.7 minutes for rank-8 LoRA. On ViT-Tiny/CIFAR-10, cLA and rank-4 LoRA achieve 94.99% and 95.04%, respectively, while cLA trains faster and uses less peak GPU memory. This is a byproduct of the column-restricted structure enabling users to very efficiently compute the matrix-vector product Ax compared to the LoRA variants; see Appendix E.7.1 (p. 38, lines 2004–2015).

We clarify that our paper focuses on whether the structure of the adaptation space matters at a fixed trainable budget from a performance and generalizability perspective and presents an example of how to leverage that adaptation to gain compute efficiency (§1, p. 2, lines 85–102; §4, pp. 6–9, lines 301–475). Lowering LoRA's rank changes the budget and would not separate the effect of structure (the restricted adaptation space) from the effect of capacity (the trainable parameter count).

Q1. Why not apply the bound to the full model directly?

This is a great question. For a fixed W0W_{0}, the map from ΔW\Delta W to W0+ΔWW_{0}+\Delta W is a bijection and preserves mutual information. This invariance does not make the generalization errors of the update-only model and of the adapted model identical.

Additionally, Theorem 1 (p. 5, lines 257–268) enables directly assessing the change in generalization error produced by the LoRA adapters, which addresses our study of restricted adaptation spaces. Theorem 1 gives bounds that relate the generalization error of the adapted model, G(W0+ΔW)G(W_{0}+\Delta W), to that of the pretrained model, G(W0)G(W_{0}), and to the update-only model, G(ΔW)G(\Delta W). These results carry model-dependent information that mutual-information invariance alone does not.

Q2. What the ordering of the bounds means in practice.

The ordering shows what restricting the adaptation space does not cost in the guarantee. Intuitively, one would expect a method whose adaptation space is more restricted could generalize worse than its unrestricted counterpart. Table 1 (p. 5, lines 216–232) shows that restricting a frozen A to a column selector does not increase the information-theoretic upper bound. The fixed-A restrictions considered here will not increase the bound.

Q3. Separate training and test losses for Figure 1.

Table R2 below reports the training loss, test loss, generalization error, and test accuracy for the Figure 1 runs. (p. 2, lines 63–68). These results now appear in Appendix E.8 (Table 18, p. 40, lines 2112–2121).

Table R2: Training loss, test loss, generalization error, and test accuracy of ViT-Base on CIFAR-10 for the runs behind Figure 1. The training loss is computed on the augmented training images. Values are three-seed means; sample standard deviations are shown for loss and accuracy. Bold marks the lowest generalization error and the best test accuracy. Gen-Error is the mean over the seeds of the absolute test – train loss difference.

Method\textbf{Method}Training loss\textbf{Training loss}Test loss\textbf{Test loss}Gen-Error\textbf{Gen-Error}\textbf{Test accuracy (\\%)}
FFT0.2617 ± 0.00500.1619 ± 0.00380.099898.06 ± 0.25
LoRA0.3274 ± 0.02380.1351 ± 0.00180.192398.71 ± 0.13
RAC-LoRA0.3657 ± 0.00880.1359 ± 0.00090.229898.73 ± 0.03

The test accuracies of the models (98.06% for FFT, 98.71% for LoRA, and 98.73% for RAC-LoRA) are in line with published results for ViT-Base on CIFAR-10. The original ViT paper reports 98.13% with ImageNet pretraining and 98.95% with ImageNet-21k pretraining under fine-tuning ([6], Table 5).

Q4. Evidence that structure and scheduling matter.

We thank the reviewer for this perspective and want to clarify the intent of our statement in the introduction (§1, p. 2, lines 97–100). "Structure" refers to the column restriction itself (thus the difference between Asymmetric LoRA and our column-restricted methods), rather than selecting which columns. "Scheduling" refers to applying chain-resets throughout adapting. The randomized variants of our column-restricted methods are included to ensure the study is robust to the selected columns we update in the models, rather than being focused on updating the first few columns of each layer (or having a particular order for the c³LA method). We have clarified this meaning in §1 of the revised paper.

The scheduling of permitted updates affects the generalizability of the models: chain methods tend to generalize worse than their non-chain counterparts, as suggested by the bound ordering in Table 1 (p. 5, lines 216–232) (on 18 of 24 model–dataset pairs for c³LA against cLA, and on 17 of 24 for RAC-LoRA against Asymmetric LoRA; Table 20, p. 43, lines 2278–2317). These definitions now appear in §1 of the revised paper.

We thank the reviewer for their time, and we kindly ask the reviewer to reconsider their score in light of these clarifications and additional experiments.

References

[1] Woo et al., PaCA: Partial Connection Adaptation for Efficient Fine-Tuning, ICLR 2025.

[2] Sheng et al., S-LoRA: Serving Thousands of Concurrent LoRA Adapters, MLSys 2024.

[3] Chen et al., Punica: Multi-Tenant LoRA Serving, MLSys 2024.

[4] Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs, NeurIPS 2023.

[5] Liu et al., DoRA: Weight-Decomposed Low-Rank Adaptation, ICML 2024.

[6] Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.