First Fundamental Form Regularization for Robust LoRA Fine-Tuning of Diffusion Models

Low-Rank Adaptation (LoRA) has become the de facto standard for personalizing text-to-image diffusion models due to its efficiency and flexibility. Despite its popularity, LoRA fine-tuning often exhibits practical instability, including sensitivity to prompt variations and interference when multiple adapters are used together. We argue that these issues stem from uncontrolled distortion of the pretrained representation geometry during fine-tuning. In this work, we propose FF-LoRA, a relational-distillation regularizer for LoRA fine-tuning that matches pairwise cosine-similarity Gram matrices, computed over the spatial tokens of each training sample, between the frozen base model and the LoRA-adapted model. Methodologically, this is in the same family as relational knowledge-distillation losses (RKD (Park et al., 2019), SP (Tung & Mori, 2019)), transposed from cross-sample relations to within-sample token relations; the further delta over the closest baseline (SP) is the cosine normalization, which we ablate in Section 4.5. The regularizer is loosely motivated by the differential-geometric notion of preserving the first fundamental form of the representation manifold, but the quantity actually penalized is a token-level relational Gram matrix, not the metric tensor itself; we use the geometric interpretation as motivation rather than as a theoretical contribution (see Appendix L). This regularization encourages LoRA to adapt to new concepts while maintaining the local distance and angle structure of the pretrained representation. FF-LoRA is lightweight, easy to implement, and fully compatible with standard diffusion training pipelines. Experimental results demonstrate that FF-LoRA improves robustness under prompt variations, reduces overfitting to limited training captions, and significantly enhances compatibility when multiple LoRA adapters are used together.