TensoLoRA: Learning Higher-Order Representations of Weight Updates

Parameter-efficient fine-tuning (PEFT) adapts pretrained models under a limited trainable-parameter budget. Low-rank adaptation (LoRA) parameterizes each weight update with two low-rank factors. However, reducing the number of trainable parameters in LoRA requires a smaller rank, which lowers the matrix-rank bound and limits the expressive capacity of the weight update. At a fixed parameter budget, weight-update parameterization determines the matrix-rank bound and shapes how update magnitude is distributed across singular directions, as characterized by the singular-value (SV) spectrum. We therefore introduce Tensorized Low-Rank Adaptation (TensoLoRA), which learns higher-order representations of weight updates by jointly optimizing all tensor factors within a fixed parameter budget. Each tensor slice is expressed as a linear combination of shared dense planes, with the slice-wise coefficients parameterized by two line factors per plane. Factorizing these coefficients reduces the parameter cost per component relative to dense coefficient matrices, allowing more jointly learned plane-line components within the same budget while retaining dense trainable planes. TensoLoRA learns less concentrated SV spectra than LoRA and retains higher performance even when the final matrix rank is matched, as shown by rank-controlled analysis. Experiments on video base-to-novel generalization, commonsense reasoning, and image classification across diverse architectures show that TensoLoRA remains competitive with fewer trainable parameters and outperforms LoRA at matched parameter budgets.