Lie-Tuning: Exact Low-Rank Matrix Flows for Rank-Preserving Adaptation

Parameter-efficient fine-tuning must balance adaptation capacity against preservation of a pretrained linear map. LoRA constrains the rank of an additive update, but Wθ = W₀ + BA does not guarantee rank(Wθ) = rank(W₀). Orthogonal multiplicative methods provide that guarantee, but they also preserve the entire singular spectrum and therefore exclude anisotropic scaling. We propose Lie-Tuning, a framework that decouples rank preservation from spectral rigidity. It adapts a frozen map through invertible general-linear actions, Wθ = exp(ξ_L)W₀ or Wθ = W₀ exp(ξ_R). Invertibility preserves the rank of W₀ exactly, while a general, non-skew-symmetric generator can change singular values through scaling and shearing. The two action sides provide complementary inductive biases: left multiplication post-composes the pretrained map in its output space, whereas right multiplication pre-composes it in its input space. We instantiate this framework with MExRA, whose generator ξ = BA is low rank, and MExBA, whose generator is block diagonal. For MExRA, we derive the exact identity exp(BA) = I + Bφ₁(AB)A. It retains all orders of the exponential while reducing an ambient d × d exponential to an r × r kernel, requiring O(dr) trainable parameters and O(dr² + r³) kernel construction. The same φ₁ representation has a second computational consequence: it forms exp(ξ) − I without explicitly subtracting the identity, avoiding near-identity cancellation and directly representing the first variation about exact identity initialization. This matrix analogue of expm1 makes the residual numerically resolvable during training. The learned action remains mergeable for inference. On LLaMA-2-7B, MExRA reaches 25.00% HumanEval Pass@1 in the regularized setting versus LoRA's 21.95%. On GSM8K, its three-seed mean accuracy is 35.28%, compared with 33.46% for LoRA and 29.11% for OFT. Right-acting MExBA achieves 46.11% German Belebele accuracy versus 43.78% for LoRA. MExRA also achieves the highest GSM8K accuracy among the compared methods throughout sequential adaptation, while numerical tests verify accurate exponential residual evaluation near identity.