Structured Algebraic Adapters for Fine-Tuning of Large Language Models

Parameter-efficient fine-tuning restricts the trainable update space of a pretrained model, commonly through low-rank factorization. We study Structured Algebraic Adapters (SAA), which instead learn coefficients over a fixed family of blockwise signed-permutation operators. This parameterization supports residual-state and weight-space adaptation without imposing an explicit low-rank bottleneck, while keeping the pretrained backbone frozen. We evaluate SAA against LoRA at approximately matched trainable-parameter budgets on mathematical supervised fine-tuning, data-to-text generation, and reinforcement learning with verifiable rewards. On Ministral-3B with canonical-boxed training targets, weight-space SAA reaches 76.6% GSM8K accuracy under a MetaMathQA-aligned evaluation prompt, compared with 74.2% for projection-level LoRA. However, improvements in answer formatting do not imply uniform gains in mathematical correctness, and neither method improves MATH performance over the frozen backbone in this setting. On GPT-2 Medium, selected SAA and LoRA configurations achieve comparable E2E generation scores. In our Qwen3-0.6B RL experiments, SAA obtains higher answer accuracy than LoRA and the frozen backbone on all four evaluations. These single-seed results motivate structured operator parameterizations as an alternative to low-rank adaptation, while revealing task- and placement-dependent trade-offs.