acceptodds
Under review as a conference paper at ICLR 2027

TriBLoRA: Tri-Factor Blockwise Low-Rank Adaptation

Abstract

Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning but couples input–output interactions through a common low-dimensional latent structure. Motivated by evidence of functional differentiation and importance heterogeneity within individual layers, we introduce TriBLoRA, a tri-factor block-wise parameterization that partitions the LoRA factors into local input and output blocks and inserts lightweight, dense pair-specific interaction matrices. The shared outer factors define reusable shared local subspaces, while each interaction matrix provides a pair-specific latent transformation. Beyond the core tri-factor parameterization, TriBLoRA also supports non-uniform block-wise rank allocation, enabling adaptation capacity to be redistributed across local regions under a controlled budget. We characterize its feasible update families and approximation properties and analyze when finer block-wise rank reallocation can lower approximation error. Experiments with RoBERTa-base on GLUE and LLaMA-2-7B instruction tuning show improved aggregate performance over rank-8 LoRA with modest parameter overhead, and further gains under appropriately configured block-rank topologies.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Low-Rank Adaptation, Tri-Factor Decomposition, Interaction Decoupling, Block-Rank Diversity

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_cRkPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new version of LoRA called TriBLoRA that splits its matrices into blocks and adds a trainable matrix between each pair of blocks. The authors also propose use different ranks for different blocks while keeping the same number of parameters. The paper includes theoretical results about the proposed method. The authors test it on GLUE with RoBERTa-base and on several tasks with LLaMA-2-7B. The results show small improvements over LoRA. The authors also study how different parts of the method affect the results.

Critical strengths

The main strength of the paper is detailed experiments. The authors compare TriBLoRA with several existing PEFT methods and provide many ablations: different interaction matrices, initialization methods, block numbers, and rank allocations. However, I think it is not enough to compensate for small improvements in performance and the fact that the method is quite similar to existing approaches.

Critical weaknesses

The proposed method is quite similar to existing block-based LoRA methods like BoRA. The main difference is use of dense matrices instead of diagonal. Theory results follow from standard matrix factorization and low-rank analysis, rank allocation considers a simplified problem, so theory is limited.

The improvements over LoRA are quite small on both GLUE and LLaMA-2-7B. Learned matrices replaced by identity does not affect the performance. Rank allocation shows no improvement over uniform. It seems it is additional complexity which is not justified and does not support accept at ICLR.

Questions for authors

I do not have specific questions. My main concerns are about the contribution and small improvements in the experiments, not unclear explanations.

Reviewer_7jRnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an extension of LoRA by dividing its A and B matrices into smaller blocks and introducing additional learnable interaction matrices between them. The method supports both uniform and non-uniform block ranks that allow different regions to have different adaptation capacities. The authors provide theoretical analysis that shows increased representational flexibility and potential improvements in approximation error under certain conditions. They evaluate the method on RoBERTa-base using GLUE benchmarks and LLaMA-2-7B on several downstream tasks, comparing it against multiple baseline methods. Overall, the experimental results show modest improvements over standard LoRA.

Critical strengths

The paper is well explained and organized, and the method is easy to understand. Comparisons with many baseline methods are provided, and they help to give a better understanding of how TriBLoRA performs compared to other approaches.

Critical weaknesses
  1. Some explanations are repeated several times throughout the paper.
  2. The theoretical analysis supports the increased flexibility of the proposed method, but it does not necessarily show that this approach leads to better practical performance.
  3. The improvements over existing methods are relatively small, and I am wondering whether the additional complexity is justified.
  4. The paper mainly focuses on the number of trainable parameters when discussing efficiency, but actual training time, GPU memory usage, and quantitative computational overhead are not reported, which are important factors for evaluating the practical efficiency of the method.
Questions for authors
  1. Could the authors provide a comparison of the actual computational cost?
  2. Could the authors further explain how these gains justify the additional complexity introduced by the method?
  3. Could the authors clarify how the theoretical results support the practical effectiveness of TriBLoRA?
Reviewer_QwSuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TriBLoRA studies how to accommodate functional heterogeneity and importance heterogeneity within a layer under a small adaptation budget. Each update block takes the form ΔWij=BiMijAj\Delta W_{ij}=B_iM_{ij}A_j: shared local outer factors capture reusable structure, while dense interaction matrices allow different transformations for each block pair. Block-rank diversity assigns different input- and output-side ranks using gradient saliency. The paper analyzes approximation constraints and budget preservation, then evaluates both design choices on RoBERTa-base with GLUE and Alpaca-tuned LLaMA-2-7B.

Critical strengths
  1. The method and analysis address a coherent allocation question. Dense interaction matrices allow each block pair to transform shared local features differently, while unequal side ranks move capacity between regions. Equation 13 distinguishes the global rank ceiling from the remaining local-subspace restrictions. Appendix A.4 explains why separate side-rank sums preserve the exact module budget and why the mean pairwise rank ceiling can nevertheless decrease.

  2. The controlled comparisons show that the changes can help. Dense cores improve the four-task mean from 88.06 to 88.28 over diagonal cores, including a 0.79-point RTE gain over ten seeds, with 896 additional parameters per matrix (Table 2). Under the same Kaiming–Frobenius initialization and core learning rate, block allocation improves ARC-C from 45.36 to 47.05 and the six-task mean from 40.10 to 40.40 (Table 3). One Alpaca-derived topology serves all downstream benchmarks. Table 12 holds rank multisets and budgets fixed; guided placement benefits CoLA and SST-2.

Critical weaknesses
  1. The residual idea is a direct structural extension rather than a substantive allocation principle. BoRA [1] already supplies shared local factors and pair-specific diagonal transformations; MoSLoRA [2] supplies dense latent mixing; GoRA [3] plans module ranks from gradients. TriBLoRA replaces the diagonal transformations with dense or rectangular matrices and applies importance-based allocation at finer granularity. Its projection identities characterize the enlarged family, and its budget identity follows by factoring the total core cost into two side-rank sums. Proposition 3 establishes the benefit of finer allocation for independent local approximation errors, leaving out the actual topology's coupled input/output constraints. Given the calibrated rank limits, Equation 160 fits the two sides separately to saliency targets; pairwise alignment enters only to break ties. The analysis identifies a bottleneck but does not produce a substantive new rule for managing it.

  2. The empirical findings do not establish a sufficiently consequential advance beyond this refinement. The placement control adds evidence that saliency-guided locations matter on some tasks. It establishes task-specific placement sensitivity, but does not show that the diagnosed loss of pairwise rank capacity explains the gains. The complete configuration exceeds layer-adaptive LoRA by 0.30 points on average, with +1.81 on ARC-C and −0.60 on WinoGrande; identity-initialized block adaptation reaches 40.13 against 40.10. GLUE block adaptation reaches 85.52 against uniform TriBLoRA's 85.53. Table 8's roughly halved parameters relative to rank-16 LoRA are favorable, but low-budget layer-adaptive LoRA already reaches 40.10 against uniform TriBLoRA's 40.06. This reference is distinct from complete GoRA. These results support local utility, while the additional capability and understanding remain limited.

References

[1] Shiwei Li, Xiandi Luo, Haozhao Wang, Xing Tang, Ziqiang Cui, Dugang Liu, Yuhua Li, Yichen Li, Xiuqiang He, and Ruixuan Li, "BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversity," ICLR, 2026. Paper

[2] Taiqiang Wu, Jiahao Wang, Zhe Zhao, and Ngai Wong, "Mixture-of-Subspaces in Low-Rank Adaptation," EMNLP, 2024. Paper

[3] Haonan He, Peng Ye, Yuchen Ren, Yuan Yuan, Luyang Zhou, Shucun Ju, and Lei Chen, "GoRA: Gradient-driven Adaptive Low Rank Adaptation," NeurIPS, 2025. Paper

Questions for authors
  1. Clarification. Once the maximum rank is fixed, Equations 159–160 fit the two sides separately to their saliency targets. For the block-adaptive topologies used in Table 3, how often does the residual path-alignment tie-break in Equation 161 resolve otherwise equivalent rank assignments? An example from those topologies, if available, would help explain its practical role. This would clarify how much pairwise coordination the implemented allocation adds beyond marginal importance.

  2. Clarification. For layer-adaptive LoRA and block-adaptive TriBLoRA (K-F) in Table 3(b), are the prescribed module ranks identical, or adjusted to match total trainable adapter parameters? Please clarify the budget convention with the total adapter counts for these two reported configurations. Appendix C.1 establishes budget equality between the two TriBLoRA allocation schemes, whereas dense cores add parameters relative to LoRA at identical ranks. This would make the quality–parameter trade-off behind the 40.10 versus 40.40 comparison precise.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.