acceptodds
Under review as a conference paper at ICLR 2027

When Can Compression Improve LoRA? A Bias–Variance Theory of Restricted Adaptation

Abstract

Compressed variants of LoRA can sometimes outperform standard LoRA despite using far fewer trainable parameters. We study this effect as restricted estimation: compression reduces estimation variance but introduces task-dependent projection bias. A local quadratic decomposition and an exact sequence model yield an energy–noise condition for when restriction helps. At matched parameter budgets, sparse residual correction is beneficial when its joint recovery exceeds the opportunity cost of displaced compressed capacity, with an additional retained-noise term under adaptive selection. For Uni-LoRA-style parameter sharing, we characterize recovery after selected coordinates are released and the remaining shared parameters are refitted. This motivates Projected LoRA with Sparse Adapter (ProLoSA), which combines a compressed branch with selected residual coordinates. Sequence and Transformer controls test the mechanism and its limits, while downstream experiments on language understanding, reasoning, instruction tuning, and vision show matched-budget gains over compressed LoRA in several settings.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Parameter-Efficient Fine-Tuning (PEFT), Low-Rank Adaptation (LoRA), Compressed LoRA

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UrzmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies compressed LoRA through a bias–variance framework, deriving conditions for beneficial compression and sparse correction under a fixed parameter budget. It proposes ProLoSA, which combines shared LoRA-factor parameters with sparse residual coordinates selected using warmup SNIP.

Critical strengths
  • The matched-budget analysis explicitly accounts for displaced compressed capacity and adaptive-selection noise.
  • Controlled experiments include predictive bias–variance measurements and informative failure cases.
  • Paired multi-seed GLUE and vision comparisons provide stronger evidence than the quoted baseline results alone.
Critical weaknesses
  • The core theory largely builds on classical projection-risk identities; its predictive usefulness for realistic LoRA training remains limited.
  • SNIP is only loosely connected to the recovery objective and frequently fails in the controlled selector experiments (Appendix E.1).
  • Comparisons beyond Uni-LoRA are less controlled: most GLUE baselines are quoted, while RoSA uses a substantially different budget.
  • Math and IFEval gains are small relative to seed variability. Parameter savings also do not yield lower measured training time or peak memory versus LoRA.
  • The improvements are marginal based on the provided experimental results, shown in Table 1.
Questions for authors

Can the theory help choose the compressed-to-sparse budget ratio without an extensive hyperparameter search?

1 comment

Can the theory reduce the search for the budget split?

No. We select d:K by validation and have not shown that the theory reduces the number of splits we need to test. In the equal-rank sequence model, Eq. (8) specializes the classical projection-risk identity to a fixed budget. Moving K parameters to a fixed sparse support lowers risk precisely when joint refitting recovers enough residual signal to outweigh the opportunity cost of shrinking the shared branch. Eq. (29) also accounts for the change in retained noise when the fitting data select the support. We lack estimates of these quantities before fine-tuning on a new task. At B = 23,040, the highest means among the tested splits occurred at d:K = 5:3 on CoLA and 31:1 on MRPC (Table 12).

Warmup SNIP chooses coordinates given K. In Appendix E.1, pointwise paired 95% intervals favored compression alone over SNIP in 7 of 12 conditions for the concentrated teacher and 8 of 12 for the diffuse teacher. A target-informed selector improved on compression in all 12 conditions for each teacher.

What does the empirical evidence support?

On STS-B, increasing Gaussian training-label noise shifted the MSE gap toward compression, as predicted. From zero to the highest noise level, the gap on the original development labels (LoRA minus Uni-LoRA) increased by 0.2740 (five paired training resamples, 95% CI [0.1476, 0.4003]).

Table 1 includes published GLUE baselines. In reruns matched by budget, splits, and seeds, ProLoSA gained 0.511 points over Uni-LoRA on the six-task RoBERTa-large average (five seeds, paired 95% CI [0.042, 0.981], Table 6). Only MRPC's task-level interval excluded zero.

The eight-task mean gains were 1.194 points on ViT-Base and 0.352 on ViT-Large. StanfordCars and FGVC Aircraft accounted for about 95% of the net ViT-Base gain (Table 15). On GSM8K, MATH500, IFEval, and IFBench, all four three-seed paired 95% intervals included zero. RoSA provides a separate-budget reference because its rank-one factors alone exceed our budget on the same RoBERTa-large layers (442,368 versus 23,040 parameters, Appendix D.8).

Reviewer_apsiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper frames compressed LoRA variants (Uni-LoRA-style fixed linear reconstruction θ = Pz) as restricted estimators and asks when restriction lowers risk relative to standard LoRA. Section 2 gives a local quadratic bias–variance decomposition (Eq. 2–3), an exact energy–noise threshold in an isotropic Gaussian sequence model (Proposition 1: compression helps iff excluded target energy per discarded dimension is below the noise variance), and a matched-budget identity for adding K released coordinates to a d = B − K compressed branch (Proposition 2, Eq. 8; adaptive-selection extension Eq. 29; Corollary 1: target-independent supports give zero expected gain). This motivates ProLoSA, which combines Uni-LoRA's normalized one-hot sharing with K sparse LoRA-factor coordinates selected by warmup SNIP and then jointly refits both branches. Validation consists of (i) sequence-model experiments (Fig. 1, App. C/E) confirming the predicted crossover and the zero-gain of random supports over 30 tasks; (ii) a fixed single-block Transformer teacher where rank-4 LoRA, Uni-LoRA and ProLoSA are trained on 20 paired datasets at five label-noise levels, showing a noise-driven reversal (LoRA best at low noise, compressed methods best at high noise) and 10–26% risk reduction of ProLoSA over Uni-LoRA (Fig. 2, Table 3); (iii) recoverability-vs-selector-realizability controls (Fig. 3, App. E.1); and (iv) downstream results on GLUE (RoBERTa-base/large), MetaMathQA→GSM8K/MATH500 and SmolTalk→IFEval/IFBench (Llama-3.1-8B), and eight ViT transfer datasets, with matched-seed paired CIs against Uni-LoRA. Theory-derived selectors (RG, RF) are also evaluated and underperform SNIP (App. D.9). Code and recorded recipes are supplied.

Critical strengths
  1. A clean, exact matched-budget accounting that explains a real phenomenon (Prop. 2, Eq. 29, Cor. 1, App. A.4; verified in App. E.5). The decomposition of the hybrid-vs-compressed gain into recovered residual energy, an opportunity cost for displaced compressed capacity, and an adaptive retained-noise term is simple but non-obvious in its consequences: it predicts that random residual supports cannot help at matched budget, which the paper confirms over 30 independent tasks (Table 23, random-support gains ≈ 0), and it correctly separates "useful residual directions exist" from "a finite-data selector finds them." This is consequential because it gives practitioners the right question to ask about any sparse-plus-compressed PEFT variant.
  2. Controlled mechanism experiments with paired uncertainty, and a complete code package. Fig. 2 / Table 3 show, under actual rank-4 LoRA-factor training, a noise-driven reversal in which full LoRA wins at label-noise multiples 0–1 and both compressed estimators win at 2–4, with bootstrap intervals on bias, variance and risk; App. E.1–E.3 add recoverability-vs-realizability, adaptive-noise and matched-residual-energy controls. The supplement contains runnable recipes with recorded seeds and budgets. This is consequential because it is the only evidence in the paper that the bias–variance story survives nonlinear optimization, and it is done carefully.
Critical weaknesses
  1. The theory is elementary, is repeatedly declared inapplicable to real LoRA training by the authors themselves, and never confronts a real task; the selectors it derives lose to the heuristic actually used (Sec. 2, App. A.3–A.5, App. D.9, Table 18). Proposition 1 is the risk of an orthogonal projection in a Gaussian sequence model; Proposition 2 and Corollary 1 are least-squares projection algebra of the kind already in the cited compressed-regression literature (Maillard & Munos 2009; Kabán 2014; Thanei et al. 2017). The paper is candid that these identities hold "for exact sequence least squares, not directly to finite-step nonlinear training" (A.4) and that the weight-space teacher "does not certify the local factor-space assumptions" (C.4), with linearization error reaching 0.21 at realistic update amplitudes (Table 22). Crucially, the title question is never tested where it matters: no proxy for v⊥ or σ²_eff is estimated on any downstream task, and no downstream experiment varies training-set size or label noise to exhibit the predicted LoRA↔Uni-LoRA crossover; the only reversal (Fig. 2) comes from one fixed synthetic teacher on a width-64 single block. Finally, the recovery-oriented selectors RG and RF that approximate the paper's own objective (Eq. 32–40) underperform warmup SNIP by 1.6–2.2 points and even fall below Uni-LoRA (Table 18); the authors conclude "SNIP remains an empirical design choice." The theory therefore neither predicts real-task behaviour nor improves the method, which undercuts contributions (i) and (iii) as stated. This drives my recommendation: an ICLR theory-plus-method paper needs the theory to do work on at least one real problem.
  2. Downstream gains are small and the baseline tuning is asymmetric (Sec. 5, Tables 1–2, 6, 8, 15–16, App. D). Under the matched five-seed GLUE protocol the average gain over Uni-LoRA is +0.511 with CI [0.042, 0.981] and only MRPC is individually significant (Table 6); the math and instruction-tuning results are explicitly "descriptive" with three seeds (D.2, D.5); the ViT-Base +1.19 is carried by FGVC (+6.59) and StanfordCars (+2.47) while six of eight datasets have CIs covering zero, and ViT-Large is +0.35 (Table 15). Meanwhile ProLoSA's configuration is selected per task over d:K ∈ {8:1, 12:1}, η_z ∈ {0.002, 0.01, 0.05}, η_h ∈ {0.002, 0.01} (Table 16) and, on GLUE, over task-specific d/K, learning rate and epoch counts (Table 8; CoLA uses 14400/8640 versus 22320/720 elsewhere), whereas it is not stated whether the Uni-LoRA matched reruns received an equivalent search. With effect sizes this small, an unequal tuning budget can account for the entire reported margin. Fixable, but as submitted the empirical case for ProLoSA is not established.
Questions for authors

Clarifications that could change my assessment:

  1. For the matched Uni-LoRA reruns on RoBERTa-large (Table 6) and ViT (Table 15), was Uni-LoRA given the same hyperparameter search (learning rates, epochs, and for ViT the η_z × η_h grid) as ProLoSA, or were Uni-LoRA's published settings reused? Please state the search space per method.
  2. Can you test the energy–noise prediction on a real task, e.g., sweep training-set size and/or label-noise rate on one or two GLUE tasks and show where LoRA and Uni-LoRA cross, together with any measurable proxy for excluded-energy vs. noise (e.g., prediction-space bias/variance as in Fig. 2)?
  3. Given Table 18, in what concrete sense does the theory "motivate" ProLoSA beyond the generic statement that residual directions may exist? If the theory cannot select the selector, would you reframe the contribution as an explanatory account rather than a design principle?
  4. Fig. 2 and Table 3 use a single teacher draw (seed 13). Does the noise-driven reversal and the ProLoSA–Uni-LoRA gap persist across, say, 5–10 independent teacher draws?
  5. The code package's paper_settings/README.md and instruction_llama2.json refer to a "paper LLaMA2 / Cleaned Alpaca / MT-Bench experiment" that does not appear in the submission. Were those results removed, and what did they show?

Suggestions (not affecting the rating): resolve the E_I notation clash; move the definition of N_adapt into the main text; clarify that the implementation builds on the RoSA codebase (all sparse flags are rosa_*) given the emphasis in App. A.1 on distinguishing the two methods.

4 comments
AuthorsOpenReview ↗

Response to Reviewer apsi — Q1–Q2

Q1. Hyperparameter selection for Uni-LoRA and ProLoSA

We reran Uni-LoRA locally and selected hyperparameters by validation performance. For Table 6, the final search covered eight configurations per method on each GLUE task. We varied the head learning rate for Uni-LoRA. For ProLoSA, we crossed two sparse allocations with four head learning rates and counted the preliminary allocation sweep in its tuning budget. For Table 15, both methods evaluated 24 configurations per ViT backbone–dataset pair using the same search seed and validation protocol.

RoBERTa-large (Table 6). We applied the search from the released Uni-LoRA CoLA/MRPC launcher to all six GLUE tasks. We fixed the vector learning rate at ηz=0.005η_z = 0.005 and evaluated eight head learning rates (ηhη_h) of 0.0001, 0.0002, 0.0005, 0.001, 0.002, 0.005, 0.01, and 0.02. We selected the rate with the highest mean validation score over seeds 0–4 (40 runs per task).

For ProLoSA, we evaluated the nine allocations in Table 12, including the shared-only endpoint, at a fixed budget of 23,040 adapter parameters. Here B=d+KB = d + K, where dd counts shared parameters and KK counts sparse coordinates. The highest mean validation scores occurred at K=8640K = 8640 on CoLA and K=720K = 720 on MRPC. On all six tasks, we crossed both values of KK with head learning rates of 0.0002, 0.0005, 0.001, and 0.005, setting d=B−Kd = B - K, ηz=0.005η_z = 0.005, and ηs=0.001η_s = 0.001. Selection by mean validation performance over seeds 0–4 gave K=8640K = 8640 for CoLA and K=720K = 720 for the other five tasks. Both methods used fixed training schedules of 20 epochs on SST-2 and QNLI and 40 epochs on CoLA, MRPC, RTE, and STS-B.

ViT (Table 15). For Uni-LoRA, we retained the five vector learning rates and four head learning rates from Section 4.4 of the Uni-LoRA paper. We added ηz=0.001η_z = 0.001 from the commented base_lrs list in ViT/run_multi.sh. The resulting 6 × 4 grid crossed vector learning rates of 0.001, 0.002, 0.005, 0.01, 0.02, and 0.05 with head learning rates of 0.001, 0.002, 0.005, and 0.01. For ProLoSA, we crossed three vector learning rates (0.002, 0.01, 0.05), two head learning rates (0.002, 0.01), and four d:Kd:K allocations (2:1, 4:1, 8:1, 12:1), fixing ηs=0.2ηzη_s = 0.2η_z.

For each backbone–dataset pair, we trained all 24 candidates per method with training seed 0 and selected the configuration with the highest validation accuracy. Table 15 reports five reruns of each selected configuration, with seeds 42–46 paired across methods.

Q2. Training-label noise on STS-B

We conducted the suggested label-noise experiment on STS-B using RoBERTa-large. With RoBERTa-large, LoRA had the lowest mean development-set MSE at τ = 0 and 0.25. Both compressed methods had lower mean MSE than LoRA at τ = 0.5 and 1. Uni-LoRA had the lowest MSE at τ = 0.5. At τ = 1, ProLoSA's mean MSE was 0.2498 lower than LoRA's and 0.0080 lower than Uni-LoRA's.

We split the official STS-B training set into 4,599 fitting and 1,150 tuning examples and evaluated on 1,500 clean development examples. We added Gaussian noise to the training labels with standard deviation τ times the training-label standard deviation (1.4678). We selected one learning rate per method from six candidates using two separate tuning resamples at τ = 0 and 1. Each run used 5,760 updates with fixed random streams for initialization, projection, and training. The table reports mean MSE across five independent bootstrap resamples. All methods shared the same resamples and standardized noise draws.

Noise level τLoRAUni-LoRAProLoSA
00.3726600.4047720.407145
0.250.4119750.4178180.414703
0.50.5345770.4676080.479753
10.9156760.6738310.665837

We estimated prediction variance across the five training resamples and computed a finite-sample-corrected squared-bias proxy using the observed development labels.

τComponentLoRAUni-LoRAProLoSA
0Prediction variance0.05110.04210.0451
0Squared-bias proxy0.32150.36270.3620
1Prediction variance0.56790.24400.2529
1Squared-bias proxy0.34770.42990.4130

At τ = 0, Uni-LoRA and ProLoSA reduced prediction variance relative to LoRA by 0.0090 and 0.0060, respectively. These reductions were smaller than the corresponding increases in the squared-bias proxy (0.0412 and 0.0405). At τ = 1, the variance reductions grew to 0.3239 and 0.3150, outweighing the increases in the squared-bias proxy (0.0822 and 0.0653).

At this higher noise level, ProLoSA's squared-bias proxy was 0.0169 lower than Uni-LoRA's, more than offsetting its 0.0089 higher prediction variance.

AuthorsOpenReview ↗

Response to Reviewer apsi — Q3–Q4

Q3. Yes. We agree with the reviewer's framing and will revise the contribution statement to describe the theory as an explanatory account with architecture-level design implications. The projection argument is classical. Its role here is to explain when allocating K of B trainable scalars to individual factor coordinates improves on using all B for the shared bank.

In the sequence model, for a noise-independent support and equal effective rank B, Eq. (8) gives an exact criterion. The hybrid improves when the residual signal recovered by jointly refitting both branches exceeds the task-dependent opportunity cost of reducing the shared bank to B − K. For one-hot sharing, Eq. (9) shows why joint refitting matters. In the example in Appendix A.5, a group has residuals (2, 1, −1, −2). Releasing the entries 2 and −2 recovers 8 units of energy. Releasing 2 and 1 captures only 5 in those entries but recovers 9.5 once the remaining tied entries are refitted. This motivates jointly training the two branches and evaluating the gain against a full-budget shared bank. When the same data determine the support and fit the parameters, Eq. (29) also accounts for the signed change in retained noise.

Proposition 2 leaves the support ranking unspecified. Our choice of warmup SNIP and our task-specific values of K are empirical. With d = 22,320 and K = 720 fixed, warmup SNIP improves over random support from 66.58 to 69.62 on CoLA and from 90.28 to 91.69 on MRPC (Table 12). In Table 18, both recovery-oriented selectors, RG and RF, score below warmup SNIP and the shared-only Uni-LoRA baseline on both tasks. The scoring windows and some allocations differ, which limits attribution of these gaps to the ranking rule alone.

We will also state the scope of the nonlinear evidence explicitly. For the concentrated teacher, increasing the target output displacement from 10% to 100% raises the relative squared linearization error from 0.01452 to 0.21112 (Table 22). We therefore interpret the nonlinear experiments as diagnostics of the proposed mechanism.

Q4. We repeated the experiment with ten independently sampled Transformer teachers (seeds 100–109). The noise-driven LoRA–Uni-LoRA reversal appears in all ten teachers, and ProLoSA retains an average advantage over Uni-LoRA.

For each teacher, we ran twenty paired training-data/noise repetitions at each of five noise levels, totaling 3,000 fits. We kept the original selected configurations and 1,000-update training protocol fixed, without retuning on the new teachers. The original seed 13 is excluded from the following results.

Noise λLoRAUni-LoRAProLoSAProLoSA reduction (%)
00.32520.99530.868813.62
0.50.49091.04140.938710.71
10.92081.16051.08137.57
22.41251.50381.39177.66
47.17932.34822.050612.52

The table reports mean clean prediction risk in units of 10⁻³ over the ten new teachers. For each teacher, we compute the relative reduction as 100 × (1 − R(ProLoSA) / R(Uni-LoRA)) and then average these percentages across teachers.

The LoRA–Uni-LoRA ordering reverses between λ = 1 and λ = 2 in every teacher's repeat-averaged risks. The teacher-averaged LoRA-minus-Uni-LoRA difference changes from −0.2397 to 0.9087 in the table's units. The corresponding Bonferroni-adjusted intervals across the five noise levels are [−0.3691, −0.1102] and [0.6878, 1.1296], supporting the reversal across independent teachers.

ProLoSA achieves lower teacher-averaged risk at all five noise levels, with mean relative reductions of 7.57–13.62%. Averaging the risk improvement equally over the five noise levels gives a positive improvement for nine of ten teachers, with an overall mean of 0.1436 × 10⁻³ and a 95% confidence interval of [0.0400, 0.2472] × 10⁻³, using teachers as the independent units. Thus, the new experiment reproduces both the noise-driven reversal and the average ProLoSA–Uni-LoRA gap beyond the original teacher draw.

AuthorsOpenReview ↗

Response to Reviewer apsi — Q5 and minor comments

Q5. Archived LLaMA2 instruction-tuning results

Yes. The LLaMA2/Cleaned Alpaca experiment appeared in an earlier manuscript. For the submission, we replaced it with the Llama-3.1-8B/SmolTalk evaluation to use a more recent model and instruction-tuning dataset. The “paper experiment” labels in paper_settings/README.md and instruction_llama2.json carried over from the earlier manuscript. We will update the supplementary documentation to identify the LLaMA2 setup as archived and Llama-3.1-8B/SmolTalk as the experiment reported in the submission.

We fine-tuned LLaMA2-7B and LLaMA2-13B using the QLoRA recipe with training seed 0. A single GPT-4 judge scored the MT-Bench responses. Each pair in the table gives the 7B value first and the 13B value second.

MethodTrainable parametersMT-Bench single-turnMT-Bench multi-turn
No fine-tuning—1.31 / 1.461.11 / 1.06
LoRA#159.9M / 250.3M5.62 / 6.203.23 / 4.13
Uni-LoRA0.52M / 1.0M5.34 / 6.343.56 / 4.43
ProLoSA0.52M / 1.0M5.36 / 6.574.25 / 4.72

ProLoSA had the highest multi-turn score at both sizes. At the same trainable parameter count as Uni-LoRA, it scored 0.69 points higher at 7B and 0.29 points higher at 13B. On single-turn MT-Bench, standard LoRA led at 7B (5.62), where ProLoSA and Uni-LoRA were nearly tied (5.36 vs. 5.34). ProLoSA led at 13B (6.57).

Minor comments on notation and implementation

In Proposition 2, we renamed the matrix of selected coordinate vectors from EIE_I to RIR_I to distinguish it from the scalar recovered residual energy in Eq. (7). Section 2.3 now defines NadaptN_{adapt} in the main text as the signed change in retained noise relative to a fixed rank-BB fit.

We built ProLoSA by modifying the Uni-LoRA codebase. In our scripts, the rosa_* prefix serves only to distinguish different mask-selection strategies. Appendix A.1 explains where the sparse update acts in each method. RoSA adds a sparse update directly to the weight matrix, and ProLoSA releases selected LoRA factor coordinates from parameter sharing.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Reviewer_YhMpOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when compressed LoRA can outperform less restricted adaptation through a bias-variance analysis. Under a local quadratic model and an exact sequence model, it characterizes the trade-off between variance reduction and projection bias, and analyzes when sparse residual correction helps under a fixed parameter budget. It proposes ProLoSA, which combines a compressed branch with sparse residual coordinates selected using a SNIP-style criterion. The paper evaluates its analysis through controlled experiments and tests the method on language understanding, reasoning, instruction tuning, and vision tasks.

Critical strengths

S1. A useful explanation of when compression helps or hurts. I find the bias-variance perspective on Uni-LoRA-style restricted adaptation informative. In particular, the analysis makes explicit that sparse correction must compensate for the compressed capacity it displaces under a fixed budget. This gives a useful framework for understanding both the benefits and limitations of compression.

S2. A reasonable method with encouraging empirical support. Combining a compressed branch with a small set of independently adjustable residual coordinates is a sensible design. The experiments span several application domains, and the five-seed paired ViT comparisons provide positive evidence for the method. I view the explanatory contribution and practical design as sufficient to support weak acceptance, while the discrepancies below should be clarified.

Critical weaknesses

W1. Experimental configurations and uncertainty need clarification. Appendix D.7 and Table 16 specify d:K ratios in {8:1, 12:1}, whereas six of the sixteen released ViT configurations in paper_settings/vit.json use 2:1 or 4:1. These include ViT-Base FGVC (+6.59 percentage points) and StanfordCars (+2.47), which show the largest gains. The total adapter budgets remain matched, so this discrepancy does not establish that the gains are invalid, but it makes the actual settings and search space unclear. In addition, the LLM comparisons use only three seeds, with some gains small relative to the reported variability. These results provide limited evidence of a stable LLM advantage.

W2. The connection from theory to the practical method is difficult to follow. Although the analysis in Section 2 is useful, I find it hard to see how its conditions lead to the specific SNIP-style selection procedure in Section 3. The distinction between a theoretically motivated design and a practical heuristic should be made more explicit. The dense presentation also obscures the main message. I suggest keeping the key conditions and design implications in Section 2, moving secondary derivations to the appendix, and including a concise version of Appendix B's Algorithm 1 in Section 3. A workflow diagram showing warmup, scoring, support selection, and joint training would help. These are presentation suggestions rather than additional empirical acceptance conditions.

Questions for authors

Q1. Configuration clarification. Which exact d:K configurations generated the ViT results in Table 15? Please reconcile the six discrepancies with Table 16 and paper_settings/vit.json, state the actual search space and tuning budgets, and provide the corresponding configurations or logs.

Q2. Theory-to-method clarification. Which specific results in Section 2 motivate the SNIP-style scoring and support selection used in Section 3? Please distinguish design choices supported by the analysis from heuristic choices, and explain what the theory predicts about when the practical selector should help.

Q3. Uncertainty clarification. Can you provide the per-seed paired differences between ProLoSA and the matched baselines for the LLM experiments, together with an appropriate uncertainty summary? This would help assess whether the small average gains are consistent across runs; comparing the gains with individual standard deviations alone does not settle this question.

4 comments
AuthorsOpenReview ↗

Response to Reviewer YhMp — Q1–Q2

Q1.

Table 16 misreported six of the d:Kd:K ratios used for the Table 15 results. The table below gives the integer allocations used in all five runs for each affected backbone–dataset pair, as recorded in paper_settings/vit.json. The other ten ratios in Table 16 agree with the released configurations. Appendix D.7 also omitted 2:1 and 4:1 from the ProLoSA search space.

BackboneDatasetTable 16 d:KActual d:KActual dActual K
ViT-BaseStanfordCars12:14:157,60014,400
ViT-BaseCIFAR108:14:157,60014,400
ViT-BaseEuroSAT8:14:157,60014,400
ViT-BaseFGVC Aircraft8:12:148,00024,000
ViT-LargeCIFAR108:12:196,00048,000
ViT-LargeCIFAR1008:14:1115,20028,800

Both methods used an adapter budget of 72,000 parameters for ViT-Base and 144,000 for ViT-Large. At the corrected allocations, the five-seed mean gains over Uni-LoRA on ViT-Base were +2.472 percentage points on StanfordCars (4:1) and +6.590 on FGVC Aircraft (2:1).

ViT search space and budget. For our Uni-LoRA reruns, we retained the four head learning rates from Section 4.4 of the Uni-LoRA paper and combined its five projected-vector learning rates with the base_lrs listed in the commented grid in the authors' ViT/run_multi.sh. The script contributed one additional vector rate, 0.001. Our local grid crossed six projected-vector rates (ηzη_z = 0.001, 0.002, 0.005, 0.01, 0.02, 0.05) with four head rates (ηhη_h = 0.001, 0.002, 0.005, 0.01), giving 6 × 4 = 24 candidates per backbone–dataset pair.

For ProLoSA, we crossed three projected-vector rates (ηzη_z = 0.002, 0.01, 0.05) and two head rates (ηhη_h = 0.002, 0.01) with four d:Kd:K ratios (2:1, 4:1, 8:1, 12:1), giving 3 × 2 × 4 = 24 candidates per pair. We fixed the sparse learning rate at ηs=0.2ηzη_s = 0.2η_z.

For both methods, we ran each candidate with search seed 0 in our local reruns and selected the configuration with the highest validation accuracy for each backbone–dataset pair. This gave 16 × 24 = 384 candidate runs per method. We then evaluated the selected configurations using five matched training seeds (42, 43, 44, 45, 46). Both methods used the same fixed data splits and trained for 20 epochs. We will correct the six entries in Table 16 and report the full ProLoSA allocation grid in Appendix D.7.

Q2.

Section 2 motivates allocating part of a fixed budget to sparse residual coordinates and jointly refitting the compressed and sparse branches. We chose warmup SNIP empirically to decide which coordinates to release.

Proposition 1 explains when the variance reduction from compression outweighs the bias from excluding task-relevant signal. Proposition 2 then asks when it is worth reallocating some of that compressed capacity to residual coordinates. In the isotropic sequence model, for a fixed support independent of the observation noise and two estimators of equal effective rank, the hybrid lowers risk precisely when the residual signal recovered by joint refitting exceeds the task-specific cost of shrinking the compressed branch (Eq. (8)). Equation (9) explains why joint refitting matters under one-hot sharing. Releasing selected coordinates allows the shared parameter to better fit the entries that remain tied within the same group. When the fitting data also determine the support, the expected risk gain includes a signed correction for the change in retained noise (Eq. (29)).

These results specify what a useful support should achieve. Under the local quadratic approximation, the gain from releasing a support depends on its profiled gradient and residualized curvature, including interactions among selected coordinates (Eq. (32)). Warmup SNIP uses a cheaper coordinate-wise proxy. It accumulates the magnitude of the first-order loss change associated with zeroing each reconstructed coordinate during warmup (Eq. (11)). We therefore expect SNIP to help when high sensitivity identifies coordinates whose joint recovery yields a positive net gain. Whether it finds such a support in nonlinear fine-tuning is an empirical question.

With d=22,320d = 22{,}320 and K=720K = 720 fixed, replacing random support with warmup SNIP raises CoLA MCC from 66.58 to 69.62 (+3.04 points on the reported scale) and MRPC accuracy from 90.28% to 91.69% (+1.41 percentage points), as shown in Table 12. Figure 3 illustrates the limit of this proxy. Even when residual signal is recoverable, the SNIP-selected support improves on compression alone only in some settings.

AuthorsOpenReview ↗

Response to Reviewer YhMp — Q3

Q3. Uncertainty across paired training seeds

At the matched budget of 2.62M trainable adapter parameters, ProLoSA had higher mean accuracy than Uni-LoRA on all four benchmarks. IFBench was the only benchmark where ProLoSA scored higher for all three seeds (mean gain +2.33 percentage points, 95% CI [−1.27, +5.94]). All four paired intervals include zero, so the direction of the mean difference remains uncertain with three seeds per benchmark.

Standard LoRA serves as a higher-parameter reference with 10.49M trainable adapter parameters. All three methods use the same training seeds within each benchmark. IFEval reports prompt-strict accuracy and IFBench reports prompt-loose accuracy.

Table 1. Accuracy (%) for each training seed, in the order shown.

BenchmarkSeed orderLoRAUni-LoRAProLoSA
GSM8K42, 43, 4475.59, 75.89, 76.6575.13, 76.35, 76.8874.98, 76.72, 76.80
MATH50042, 43, 4426.80, 25.20, 27.2027.00, 26.80, 26.8026.20, 27.20, 28.60
IFEval0, 21, 4248.24, 44.18, 47.5141.77, 49.17, 48.9846.58, 44.36, 49.72
IFBench0, 21, 4217.67, 15.67, 17.3314.67, 16.33, 16.3318.00, 17.00, 19.33

For each seed, we subtract the second method's score from the first method's score using the two-decimal accuracies in Table 1. We compute each pointwise 95% paired t interval as mean difference ± t × SD / √3, where SD is the sample standard deviation of the three differences and t is the critical value for 2 degrees of freedom (approximately 4.30265). We round the means and interval endpoints to two decimals only after calculation.

Table 2. Paired differences in percentage points, in the seed order from Table 1.

BenchmarkComparisonWithin-seed differencesMean95% CI
GSM8KProLoSA − Uni-LoRA−0.15, +0.37, −0.08+0.05[−0.65, +0.75]
GSM8KProLoSA − LoRA−0.61, +0.83, +0.15+0.12[−1.67, +1.91]
GSM8KUni-LoRA − LoRA−0.46, +0.46, +0.23+0.08[−1.11, +1.27]
MATH500ProLoSA − Uni-LoRA−0.80, +0.40, +1.80+0.47[−2.77, +3.70]
MATH500ProLoSA − LoRA−0.60, +2.00, +1.40+0.93[−2.45, +4.32]
MATH500Uni-LoRA − LoRA+0.20, +1.60, −0.40+0.47[−2.08, +3.02]
IFEvalProLoSA − Uni-LoRA+4.81, −4.81, +0.74+0.25[−11.75, +12.24]
IFEvalProLoSA − LoRA−1.66, +0.18, +2.21+0.24[−4.57, +5.05]
IFEvalUni-LoRA − LoRA−6.47, +4.99, +1.470.00[−14.59, +14.58]
IFBenchProLoSA − Uni-LoRA+3.33, +0.67, +3.00+2.33[−1.27, +5.94]
IFBenchProLoSA − LoRA+0.33, +1.33, +2.00+1.22[−0.87, +3.31]
IFBenchUni-LoRA − LoRA−3.00, +0.66, −1.00−1.11[−5.67, +3.44]

Thank you for the detailed and candid responses, which address my main questions. My concerns have been largely resolved, and I will maintain my rating of 3.

Thank you for your confirmation and for taking the time to review our paper. We appreciate your valuable feedback and are glad our responses were helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.