TriBLoRA: Tri-Factor Blockwise Low-Rank Adaptation

Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning but couples input–output interactions through a common low-dimensional latent structure. Motivated by evidence of functional differentiation and importance heterogeneity within individual layers, we introduce TriBLoRA, a tri-factor block-wise parameterization that partitions the LoRA factors into local input and output blocks and inserts lightweight, dense pair-specific interaction matrices. The shared outer factors define reusable shared local subspaces, while each interaction matrix provides a pair-specific latent transformation. Beyond the core tri-factor parameterization, TriBLoRA also supports non-uniform block-wise rank allocation, enabling adaptation capacity to be redistributed across local regions under a controlled budget. We characterize its feasible update families and approximation properties and analyze when finer block-wise rank reallocation can lower approximation error. Experiments with RoBERTa-base on GLUE and LLaMA-2-7B instruction tuning show improved aggregate performance over rank-8 LoRA with modest parameter overhead, and further gains under appropriately configured block-rank topologies.