acceptodds
Under review as a conference paper at ICLR 2027

PSR-LoRA: Power-Similarity Regularized Low-Rank Adaptation

Abstract

Low-Rank Adaptation (LoRA) has become a widely adopted approach for parameter-efficient fine-tuning of large language models. However, its rank components are typically optimized jointly without explicitly modeling their mutual dependencies, which may cause multiple components to learn highly correlated adaptation patterns and lead to inefficient utilization of the limited rank budget. In this work, we propose Power-Similarity Regularized LoRA (PSR-LoRA), a parameter-efficient adaptation framework that explicitly models and regulates the relationships among rank-wise update components. Specifically, PSR-LoRA first decomposes the LoRA update into rank-one components and maps them into a power-transformed space using a signed power function controlled by an exponent hyperparameter. Pairwise dependencies among the transformed components are then characterized through rank-wise similarity modeling. Based on the resulting similarity structure, we introduce a redundancy regularization objective to suppress excessive dependencies among rank components and further derive similarity-guided weights to adaptively adjust their contributions to the final low-rank update. To maintain efficiency, the proposed similarity computation is performed directly on the low-rank factors without explicitly constructing full rank-one update matrices. Extensive experiments across multiple downstream tasks demonstrate that PSR-LoRA consistently improves adaptation performance under constrained trainable parameter budgets while inducing less redundant rank-wise adaptation patterns. These results show that explicitly organizing the internal structure of a fixed low-rank space provides an effective alternative to simply increasing the LoRA rank.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Parameter-efficient fine-tuning, Power-similarity regularization, Low-rank adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CX6EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PSR LoRA, which reduces redundancy between LoRA rank components through power based similarity regularization and adaptive weighting. This method has been improved on three downstream datasets.

Critical strengths
  1. This method is simple and easy to understand, and is an effective method for calculating rank similarity.
  2. Experiments have shown that under several LoRA baselines, the results are promising through useful rank and hyperparameter analysis.
Critical weaknesses
  1. The novelty is limited. Reducing redundancy between LoRA components is not a new thing. The proposed method is relatively close to existing methods based on diversity and orthogonality.
  2. Unfair comparison. Table 1 uses different LoRA levels in different methods, making it difficult to determine the actual improvement situation.
  3. Insufficient experimentation. This paper lacks clear provisions for separating regularization and adaptive weighting. It only evaluates one backbone on three specific domain datasets, without results from multiple random seeds.
Questions for authors
  1. Can the authors compare all methods under the same rank and parameter budget?
  2. How much does each proposed component contribute to the final performance?
  3. Since different LoRA factorizations can represent the same weight update, how reliable is the proposed rank-wise similarity measure?
1 comment

Thank you for these thoughtful and rigorous comments, which help us better distinguish our method from prior work and strengthen our empirical validation.

Reply W1: We fully acknowledge that promoting component diversity for low-rank adapters is a known direction. However, our core novelty lies in a critical practical gap of prior orthogonality/diversity methods: Most existing orthogonal / diversity LoRA imposes hard orthogonality constraints or inner-product penalties directly on low-rank matrices. These constraints often break the native rank-one decomposition of LoRA, introduce heavy overhead, or cannot be naturally absorbed and merged into the base model after training. Our signed-power transformation is designed specifically for rank-one LoRA components: it computes similarity directly from the low-dimensional factors without constructing large matrices, and the regularization + adaptive weighting can be fully fused into LoRA weights at inference time with zero extra cost. This is a key distinction from prior orthogonal LoRA methods, which often retain extra constraints or cannot be cleanly merged.

Reply W2: We agree this is a major flaw in our original table. Mixed rank settings obscure whether gains come from our algorithm or simply different parameter budgets. We will rework Table 1 with fixed equal rank for all compared methods, under strictly matched parameter budgets. All main comparisons will be conducted at identical rank, and we will add an extra controlled sweep across multiple fixed ranks to show consistent trends. In addition, Figure 3 shows different results for the same rank.

Reply W3: We appreciate this suggestion. Our original experiments bundled the similarity regularization and adaptive component weighting together, and lacked robustness validation.We add a dedicated ablation table to isolate the two core modules: vanilla LoRA, LoRA + only similarity regularization, LoRA + only adaptive weighting, full PSR-LoRA. This quantifies the individual contribution of each component.We add results on an additional mainstream LLM backbone (e.g., Llama 3 8B) to verify generalizability beyond Qwen2.5 7B. All results will be reported as mean ± std over ≥5 random seeds across all datasets and settings. Q1: We will provide all main comparisons under identical rank and equal parameter budget, plus rank sweep experiments. Q2: We will add ablation experiments to quantify individual gains from the similarity regularization and adaptive weighting modules. Q3: We recognize factorization ambiguity. Our similarity regularizes the learned rank-one directions along the optimization path, rather than being invariant to equivalent delta reparameterizations. We will add this discussion and supporting sensitivity experiments.

Reviewer_cPZEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PSR-LoRA addresses redundancy among LoRA’s rank-one update components. It applies a signed power transformation to measure component similarity, penalizes correlated components, and adaptively weights their contributions. Similarities are computed efficiently from low-rank factors, preserving inference-time merging. Experiments using Qwen2.5-7B on three domain-specific tasks report improvements over standard LoRA and several variants.

Critical strengths

The signed power transformation fits LoRA well because it preserves the rank-one structure, so similarities can be computed from the small factors without building large matrices. The method is also easy to integrate into standard LoRA: it doesn’t need an extra trainable gating network, and the final update can still be merged into the original model, without involving extra inference steps.

Critical weaknesses
  • My main concern is that the paper doesn’t clearly show whether the improvements actually come from reducing redundancy between LoRA components. At r=1r=1, there’s only one component, so there’s no redundancy between components to reduce, and the method should behave like standard LoRA under the same training setup. But the paper still reports a noticeable performance gap. That raises the possibility that other implementation or training differences are driving some of the gains. Until the authors explain this gap and provide tightly controlled comparisons, it’s hard to attribute the improvements to their proposed mechanism.

  • Another concern I have is that every LoRA variant performs worse than standard LoRA across all three tasks, sometimes by a large margin. That’s possible, but such a consistent pattern needs an explanation. Different methods may need different learning rates or method-specific settings, so using the same training configuration doesn’t necessarily make the comparison fair. Without clearer implementation details and comparable tuning budgets, it’s hard to tell whether PSR-LoRA is genuinely better or whether the other baselines weren’t properly configured and tuned.

Questions for authors
  • Why do most of the LoRA variants perform worse than standard LoRA? How did you tune each baseline’s learning rate and method-specific settings?
  • Does PSR-LoRA still outperform the baselines under the same parameter budget and comparable tuning budgets?
  • Does PSR-LoRA still outperform the baselines under the same parameter budget and comparable tuning budgets?
Reviewer_aHnjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PSR-LoRA aims to make better use of a small LoRA rank by discouraging similar update components. It applies a signed power transformation to each rank-one component, then uses their similarities to compute a regularization loss and component weights. The similarities are calculated directly from the low-rank factors. The authors test the method on three domain-specific tasks with Qwen2.5-7B.

Critical strengths

S1. The similarity calculation is straightforward and avoids building large rank-one matrices. The signed power transformation makes this factorization possible.

S2. The weights can be absorbed into the LoRA factors after training, so the update can still be merged into the model for inference.

Critical weaknesses

W1. I do not think the experiments are enough to judge the gains reliably. Table 1 compares different ranks: on Patent and FaultQA, LoRA uses rank 2 and PSR-LoRA uses rank 4. Figure 3 includes same-rank comparisons, but the paper does not give results across seeds or complete training settings and data splits. The p=1 analysis already provides a useful comparison with linear similarity regularization. Together with lambda=0, it still does not show what adaptive weighting adds. A comparison with uniform weights at one representative rank, such as r=4, would address this more directly. I also noticed a mismatch in the SMRT results: Table 1 says p=0.8, while its precision, recall, and F1 match the p=0.7 column of Table 2. The authors need to clarify which configuration produced these results.

W2. Some of the low-rank results are hard to reconcile with the equations. Equations (6) and (11) are undefined at r=1. If the penalty is set to zero and g_1=1, the method becomes ordinary LoRA, so I do not understand the sizeable rank-1 gap in Figure 3. At r=2, the two redundancy scores are equal and the weights are always 1/2; only the regularizer can make a difference. The claim about better rank utilization also needs care. For example, M_1=[[1,1],[0,0]] and M_2=[[0,0],[1,1]] have zero power-space cosine similarity for any p>0, yet their sum still has rank 1. Lower component similarity alone is therefore not enough to show that the final update uses its rank more effectively.

Questions for authors

Q1. What does the implementation do at r=1, and what explains the gap from LoRA in Figure 3? At r=2, do you confirm that the weights are always uniform?

Q2. Were the SMRT results in Table 1 obtained with p=0.7 or p=0.8? Please provide the settings for that run, including data splits, optimizer, initialization, alpha, tau, and target modules. How do you handle values near zero when p<1?

Q3. For one representative same-budget setting, is the gain stable across seeds, and what changes when the weights are fixed to 1/r during training? For the rank-utilization claim, do the existing checkpoints provide supporting evidence, or is this intended as a qualitative explanation rather than a claim about the effective matrix rank?

1 comment

We thank you for acknowledging the efficiency of our factorized similarity computation and inference mergeable property of PSR LoRA.

Reply W1:

  1. In Table 1 we reported the best validated rank for each baseline following their original papers. Figure 3 already provides controlled comparisons under identical rank across standard LoRA and PSR LoRA for all rank values r ∈ {1,2,4,8,16}, where PSR LoRA consistently outperforms vanilla LoRA at every rank. This isolates rank as a controlled variable. We will add explicit text in revision to emphasize Figure3 is the core same budget comparison.
  2. You correctly point out that p=1 (linear similarity only) + λ_PSR=0 does not isolate adaptive weighting. In revision we will add an explicit ablation at fixed representative rank, PSR LoRA full (regularizer + adaptive weighting), PSR LoRA w/o adaptive weighting (fix uniform weights g_k=1/r, keep only L_PSR regularization), Vanilla LoRA This will quantitatively disentangle the contribution of redundancy regularizer versus similarity guided adaptive weighting.
  3. SMRT hyper parameter inconsistency between Table1 and Table2: This is a typographical error in Table 1 caption. The SMRT main result in Table 1 was run with p=0.7 (matching the best metric numbers shown in Table 2), while we incorrectly wrote p=0.8. We apologize for this typo. We will correct this mistake and add full configuration footnotes for Table 1 including: data split, optimizer, initialization, α, τ, target modules in the revised manuscript.
  4. Multiple random seeds: We agree results should be validated across seeds. In revision we will report mean and standard deviation over multiple random seeds for key main result configurations.

Reply W2:

  1. Edge case r=1: Equation (6) L_PSR and redundancy score ρ_k in Eq.(11) are mathematically undefined for r=1. In our implementation: • When r=1, we set L_PSR=0 and force weight g_1=1. In theory this reduces the formulation exactly to standard LoRA. We observe performance gap at r=1 in Figure3. After checking our code, this gap arises from numerical behaviour of the signed power transform applied during forward pass even without regularization loss. Even with L_PSR=0 and fixed g_1=1, the forward computation still applies ã_1=sgn(a_1)|a_1|^p, b̃_1=sgn(b_1)|b_1|^p inside our adaptive update formula Eq.(13). It is not identical to vanilla LoRA forward pass, even though there is no inter rank regularization. This subtle implementation detail was not clearly documented and will be fully clarified in Algorithm 1 and text.
  2. Edge case r=2: For r=2: ρ_1=|S_{12}|, ρ_2=|S_{21}|=|S_{12}| Hence ρ_1=ρ_2, which yields uniform softmax weights g_1=g_2=1/2. You are exactly correct: at r=2, adaptive weighting produces uniform weights, and all improvements must come purely from the PSR redundancy regularizer. We will explicitly state this observation in revision.
  3. Low pairwise component similarity does not guarantee high effective rank for summed update: You provide a correct counter example: two rank one components can be orthogonal in power transformed space yet their sum still yields rank 1 matrix. We fully agree that minimizing pairwise component similarity is a heuristic objective, not a rigorous guarantee on effective matrix rank of final ΔW. In our paper, "improved rank utilization" is intended as qualitative mechanism explanation rather than a strict mathematical theorem on effective rank. We will soften our wording in revision and add your counter example as a footnote to clarify this conceptual limitation.

Q1: For r=1: regularizer is zeroed and weight fixed to one; forward pass still applies power transform which creates numerical difference vs vanilla LoRA. For r=2 weights are always uniform. We will document all these details. Q2: The SMRT main result uses p=0.7 . In revision we will add full configuration block including data split, optimizer, initialization, α, τ, target modules. For values near zero under fractional p<1: since sgn(x)|x|^p is well defined for x→0, and we maintain float32 numerical stability; we add a tiny epsilon only inside similarity denominator, not for power transformation itself. Q3: We will add multi seed mean/std for key configurations, add ablation fixing weights to uniform 1/r to separate regularization and weighting. As discussed above, rank utilization is qualitative mechanism instead of strict mathematical claim, and we will tone down the corresponding statements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.