Decoupling Rotation and Deformation via Frozen Reservoirs for Efficient Fine-Tuning

Parameter-efficient fine-tuning (PEFT) is widely used for low-cost adaptation of pretrained models. From a geometric perspective, full fine-tuning permits flexible, high-rank updates that alter the orientation and scale of weight mappings, whereas representative PEFT methods restrict this flexibility through low-rank or orthogonality constraints. To enable high-rank weight updates and independently control rotation and deformation in weight transformations under a limited parameter budget, we propose Geometrically Decoupled Reservoir Adaptation (GeDRA). GeDRA decomposes a frozen operator reservoir into rotation-generator and deformation-generator templates and learns one modulation vector per template, enabling separate control over local rotation and deformation updates in the tangent space. Theoretical analysis reveals an expressivity advantage of geometric decoupling in terms of expected optimal approximation error under matched trainable parameter budgets. We further observe broad directional support for both rotation and deformation in GeDRA, qualitatively similar to full fine-tuning. Experiments on commonsense reasoning, image classification, and natural language understanding demonstrate favorable performance–parameter trade-offs against strong baselines. Qualitative comparisons with LoRA further illustrate GeDRA's effectiveness in subject-driven generation.