Parameter-Space Symmetry in Fine-Tuning

Parameter-efficient fine-tuning introduces structured trainable parameters into a frozen model, but distinct parameter settings can represent exactly the same function. We characterize the parameter-space symmetries of LoRA, prefix tuning, and prompt tuning in multihead attention, making explicit the redundancies in their parameterizations. These results provide a foundation for studying linear mode connectivity and model merging, where accounting for symmetries helps distinguish functional differences from differences in parameter representation.