- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PSR LoRA, which reduces redundancy between LoRA rank components through power based similarity regularization and adaptive weighting. This method has been improved on three downstream datasets.
Critical strengths
- This method is simple and easy to understand, and is an effective method for calculating rank similarity.
- Experiments have shown that under several LoRA baselines, the results are promising through useful rank and hyperparameter analysis.
Critical weaknesses
- The novelty is limited. Reducing redundancy between LoRA components is not a new thing. The proposed method is relatively close to existing methods based on diversity and orthogonality.
- Unfair comparison. Table 1 uses different LoRA levels in different methods, making it difficult to determine the actual improvement situation.
- Insufficient experimentation. This paper lacks clear provisions for separating regularization and adaptive weighting. It only evaluates one backbone on three specific domain datasets, without results from multiple random seeds.
Questions for authors
- Can the authors compare all methods under the same rank and parameter budget?
- How much does each proposed component contribute to the final performance?
- Since different LoRA factorizations can represent the same weight update, how reliable is the proposed rank-wise similarity measure?
Thank you for these thoughtful and rigorous comments, which help us better distinguish our method from prior work and strengthen our empirical validation.
Reply W1: We fully acknowledge that promoting component diversity for low-rank adapters is a known direction. However, our core novelty lies in a critical practical gap of prior orthogonality/diversity methods: Most existing orthogonal / diversity LoRA imposes hard orthogonality constraints or inner-product penalties directly on low-rank matrices. These constraints often break the native rank-one decomposition of LoRA, introduce heavy overhead, or cannot be naturally absorbed and merged into the base model after training. Our signed-power transformation is designed specifically for rank-one LoRA components: it computes similarity directly from the low-dimensional factors without constructing large matrices, and the regularization + adaptive weighting can be fully fused into LoRA weights at inference time with zero extra cost. This is a key distinction from prior orthogonal LoRA methods, which often retain extra constraints or cannot be cleanly merged.
Reply W2: We agree this is a major flaw in our original table. Mixed rank settings obscure whether gains come from our algorithm or simply different parameter budgets. We will rework Table 1 with fixed equal rank for all compared methods, under strictly matched parameter budgets. All main comparisons will be conducted at identical rank, and we will add an extra controlled sweep across multiple fixed ranks to show consistent trends. In addition, Figure 3 shows different results for the same rank.
Reply W3: We appreciate this suggestion. Our original experiments bundled the similarity regularization and adaptive component weighting together, and lacked robustness validation.We add a dedicated ablation table to isolate the two core modules: vanilla LoRA, LoRA + only similarity regularization, LoRA + only adaptive weighting, full PSR-LoRA. This quantifies the individual contribution of each component.We add results on an additional mainstream LLM backbone (e.g., Llama 3 8B) to verify generalizability beyond Qwen2.5 7B. All results will be reported as mean ± std over ≥5 random seeds across all datasets and settings. Q1: We will provide all main comparisons under identical rank and equal parameter budget, plus rank sweep experiments. Q2: We will add ablation experiments to quantify individual gains from the similarity regularization and adaptive weighting modules. Q3: We recognize factorization ambiguity. Our similarity regularizes the learned rank-one directions along the optimization path, rather than being invariant to equivalent delta reparameterizations. We will add this discussion and supporting sensitivity experiments.