- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work investigates whether factorization depth can serve as another regularization dimension for LoRA, alongside trainable parameter count and structural rank. Analyses of a mergeable -factor parameterization motivate a finite-horizon hypothesis that deeper factorization may selectively delay weak singular modes, promoting spectral concentration at moderate depths but hindering task fitting at excessive depths. Experiments on LLaMA-2-7B, including capacity controls, depth ablations, spectral diagnostics, and synthetic experiments, support this trade-off rather than a monotonic benefit from increasing depth. A nuclear-norm intervention recovers only part of the gain, suggesting that the depth effect is not fully explained by explicit nuclear-norm regularization.
Critical strengths
The main comparison controls capacity in two complementary ways. LoRA r8 matches the trainable parameter count of the default DR-LoRA, and LoRA r4 matches its structural-rank bound. DR-LoRA achieves higher task performance than both. The complementary ablation experiments (e.g., the structural-rank ablation) further enhance the value of the work. Additionally, the depth sweep reports the trade-off rather than only the best configuration (Table 3, Figure 3). As depth increases, effective rank falls monotonically, while accuracy peaks at a moderate depth and training loss rises sharply beyond it.
Critical weaknesses
The evidence that the gain is robust to the optimization setting is limited. The main six-task comparison uses a single learning rate shared by all methods (Section 4). Although a learning-rate check is provided, it covers only CommonsenseQA, and LoRA performs best at the lowest tested rate, so its optimal setting is not reached (Appendix B.3, Table 11).
The default configuration is not compared with the close prior method. Deep LoRA [1] already uses a three-factor adapter and reports benefits in reducing overfitting in limited-data fine-tuning. It would be a natural baseline and would strengthen this work.
[1] Can Yaras, Peng Wang, Laura Balzano, and Qing Qu. Compressible dynamics in deep overparameterized low-rank learning and adaptation. In ICML, 2024.
Questions for authors
-
Adapters are inserted only into q_proj and v_proj (Section 4). Does the depth effect persist when adapters are applied to all linear layers, including the MLP projections?
-
The intermediate dimensions used for and in the depth sweep are not reported. Could the authors provide them, together with the resulting structural-rank bound and parameter count?
-
Figure 2(b) reports the pre-clip gradient norm. Since Section 4 states that gradient clipping is not used, could the authors clarify whether any clipping was applied during training?
-
All methods are trained for a fixed 3 epochs without mentioned early stopping (Section 4). As LoRA already reaches near-zero training loss (Table 3), the gain may depend on this training length. Could the authors clarify how the training length and the default configuration were chosen, and report results with early stopping or different training lengths?
My score could be raised if the concerns in the Questions and Weaknesses sections are properly addressed.
3 comments
I want to thank the authors for the detailed clarifications, which addresses most of my concerns. I also encourage the authors to incorporate these clarifications into the revised version of the paper. I will consider raising my score from 2 to 3.
Thank you for the careful follow-up and for considering a higher score. As suggested, we have incorporated the key clarifications into the revised version of the paper. We are grateful for your constructive comments, which have substantially strengthened our work.
New experiments on learning rate, training length, and all-linear adapters, plus a Deep LoRA baseline, show that the gains hold; the clipping statement is corrected.
Thank you for the detailed reading. Below are new LLaMA-2-7B experiments and a Deep LoRA baseline; the revision adds Sec. 7.3 and 8.2 and fixes the clipping statement in Sec. 4.
Protocol. 10% of the CSQA training set is held out as a tune split for selecting learning rates and early-stopping epochs; validation is used only for reporting. All methods use s = 2, AdamW, cosine schedule, clipping 1.0, effective batch 16, and q/v adapters unless noted.
W1: Robustness to the optimization setting
Table R1. CSQA, 3 epochs; tune/report accuracy per learning rate (selected rate in bold).
LoRA's optimum is bracketed, and its tune-selected rate is the shared rate of Table 1. DR-LoRA leads by 4.0 points at the selected rates (0.809† vs. 0.769) and at the best report rates (0.816 vs. 0.776), and is less rate-sensitive (0.800–0.816 vs. 0.725–0.776).
Table R2. Six tasks at the selected rates (LoRA's rate equals the shared one, so its row is from Table 1).
DR-LoRA leads on all six tasks; the macro gain (+4.0) is no smaller than with the shared rate (+3.4, Table 1) and far above the seed std (0.3–0.4 points).
W2: Deep LoRA
Deep LoRA (official code, tune-selected 8e-4, seed 42) reaches a six-task macro-average of 0.844, vs. 0.849 for DR-LoRA and 0.809 for LoRA r8, and is within 0.8 points of DR-LoRA on every task. The two deep methods are comparable, so the gain comes from the added factor itself. Our contribution is depth as a tunable regularization dimension, not another three-factor adapter: Deep LoRA fixes L = 3, whereas our sweep (L = 2–5) shows falling effective rank, rising training loss (0.001 to 0.205), and accuracy that first rises then falls (Table 3), with the optimum shifting with the learning rate (Table 11).
Q1: All linear layers (Sec. 8.2, Table 7)
With adapters on all seven projections and the q/v learning rates, DR-LoRA leads by 3.1 points and stays spectrally concentrated in attention and MLP layers, while LoRA stays near its bound.
Q2: Depth-sweep dimensions
Each added factor is a 4×4 matrix at the input-side bottleneck, so the rank bound stays at 4 and each matrix gains 16 parameters: L = 3/4/5 are 4096→4→12→4096, 4096→4→4→12→4096, and 4096→4→4→4→12→4096 (65,584/65,600/65,616 params per matrix), vs. 65,536 for LoRA r8 (bound 8) and 32,768 for LoRA r4 (bound 4). In Eq. (1), r₁ = 12 (output side) and r₂ = 4 (input side). The rank-matched LoRA r4 (0.776) is also below DR-LoRA (0.816). The final version will add this table and a LoRA r4 row to Table 3.
Q3: Gradient clipping
Section 4 had a writing error: the code calls clip_grad_norm_(max_norm=1.0) before every update, so all runs of all methods clip the global adapter-gradient norm to 1.0 (corrected in the revision). Figure 2(b) and the peaks in Table 2 are the pre-clip norms returned by this function; they stay above 1.0, so almost every update is clipped.
Q4: Training length, early stopping, and defaults
The defaults follow from capacity constraints, not tuning: width 4 matches LoRA r4's rank bound, width 12 matches LoRA r8's parameter count, α gives s = 2, and L = 3 is the smallest depth beyond LoRA. Like Yaras et al. (2024), we evaluate the final checkpoint; to check whether this disadvantages LoRA, we trained for 5 epochs with and without early stopping on the tune split.
Table R6. CSQA report accuracy.
Longer training does not close the gap (+5.0 at 5 epochs, p < 1e-5), and early stopping picks the last or second-to-last epoch for every method. At 8e-4, L = 4 matches or beats L = 3 with lower effective rank (2.32 vs. 2.89), whereas at 2e-4 it underfits within 3 epochs (Table 3): the optimal depth depends on the effective training horizon.