acceptodds
Under review as a conference paper at ICLR 2027

First Fundamental Form Regularization for Robust LoRA Fine-Tuning of Diffusion Models

Abstract

Low-Rank Adaptation (LoRA) has become the de facto standard for personalizing text-to-image diffusion models due to its efficiency and flexibility. Despite its popularity, LoRA fine-tuning often exhibits practical instability, including sensitivity to prompt variations and interference when multiple adapters are used together. We argue that these issues stem from uncontrolled distortion of the pretrained representation geometry during fine-tuning. In this work, we propose FF-LoRA, a relational-distillation regularizer for LoRA fine-tuning that matches pairwise cosine-similarity Gram matrices, computed over the spatial tokens of each training sample, between the frozen base model and the LoRA-adapted model. Methodologically, this is in the same family as relational knowledge-distillation losses (RKD (Park et al., 2019), SP (Tung & Mori, 2019)), transposed from cross-sample relations to within-sample token relations; the further delta over the closest baseline (SP) is the cosine normalization, which we ablate in Section 4.5. The regularizer is loosely motivated by the differential-geometric notion of preserving the first fundamental form of the representation manifold, but the quantity actually penalized is a token-level relational Gram matrix, not the metric tensor itself; we use the geometric interpretation as motivation rather than as a theoretical contribution (see Appendix L). This regularization encourages LoRA to adapt to new concepts while maintaining the local distance and angle structure of the pretrained representation. FF-LoRA is lightweight, easy to implement, and fully compatible with standard diffusion training pipelines. Experimental results demonstrate that FF-LoRA improves robustness under prompt variations, reduces overfitting to limited training captions, and significantly enhances compatibility when multiple LoRA adapters are used together.

Primary area:
generative models
Keywords:
LoRA, diffusion models, fine-tuning, personalization, regularization, knowledge distillation, representation geometry, text-to-image, multi-adapter

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_me96OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FF-LoRA, a regularization method for LoRA-based personalization of text-to-image diffusion models. The method maintains a frozen copy of the base model and encourages the adapted model to preserve the base model’s pairwise cosine similarities among spatial feature tokens. Concretely, the authors construct cosine-normalized Gram matrices from an intermediate feature layer and minimize their Frobenius distance alongside the standard diffusion objective.

The method is motivated using the differential-geometric concept of the first fundamental form. However, it does not compute or match the actual induced metric. Instead, the proposed loss is a discrete relational surrogate that preserves normalized feature similarities. Empirically, the paper evaluates single-subject personalization, robustness to out-of-distribution prompts, and composition of independently trained adapters. Experiments are conducted primarily on SDXL, with smaller studies on SD-1.5 and SD3-medium. The authors report comparable in-distribution fidelity, modestly improved out-of-distribution CLIP-I, and higher two-adapter composition success relative to standard LoRA and relational-distillation baselines.

Critical strengths

Prompt sensitivity and interference between independently trained adapters are important limitations of diffusion personalization. The paper evaluates both subject-subject and subject-style composition rather than restricting evaluation to prompts closely resembling the training captions. The compatibility experiment with ZipLoRA also suggests that the proposed training-time regularizer may complement existing merge-time methods.

The proposed loss can be integrated into standard LoRA training without changing the adapter architecture or inference procedure. It requires an additional frozen-model feature pass during training but introduces no inference-time overhead. This simplicity could make the technique useful if its improvements are shown to be robust.

Critical weaknesses

The central first-fundamental-form interpretation is substantially stronger than what the proposed objective establishes. A first fundamental form is a local metric induced by derivatives of an embedding, typically represented by Jacobians. FF-LoRA neither estimates this Jacobian nor directly constrains the induced metric. It instead matches cosine similarities between features at different spatial locations.

The paper does not establish conditions under which similarity between finite token Gram matrices implies similarity between the corresponding pullback metrics. Spatial tokens are not shown to constitute infinitesimal tangent perturbations around a common manifold point. Moreover, cosine normalization removes feature-norm information, so the loss cannot generally preserve local distances. Claims concerning preservation of distances, angles, or the conformal class of the representation metric are therefore insufficiently supported.

Once the geometric terminology is removed, the method appears to be a relatively direct variant of similarity-preserving relational distillation: the base model acts as the teacher, the adapted model acts as the student, and an internal relation matrix is matched. The principal distinction from the closest baselines is the use of within-image spatial tokens and cosine normalization. The improvements over RKD and similarity-preserving distillation are modest. For example, the reported robustness gaps are 0.12, 0.13, and 0.14 for FF-LoRA, RKD, and SP, respectively, while composition success improves from 53% for SP to 58% for FF-LoRA. These results may support cosine normalization as a useful design choice, but they do not substantiate the broader claim of a new geometric framework.

This issue is decisive for my recommendation because the paper’s perceived novelty depends heavily on an interpretation that is neither formally derived nor directly evaluated.

The empirical evidence relies heavily on self-defined or threshold-sensitive metrics. The proposed “robustness gap” is the difference between in-distribution and out-of-distribution CLIP-I. This metric has an important degeneracy: a method can obtain a small gap by performing similarly poorly in both conditions. Although absolute scores are also reported, the OOD prompt construction protocol remains insufficiently controlled. The paper does not clearly establish a taxonomy of distribution shifts, difficulty calibration, or whether prompts were fixed before model development and output inspection.

Similarly, multi-adapter success is determined using thresholds on CLIP similarities for the two concepts. The paper does not provide adequate threshold-sensitivity analysis or validate these automatic labels against blinded human judgments. This is particularly problematic because the advantage over the closest baselines is only a few percentage points and might change under alternative thresholds. CLIP similarity also cannot reliably distinguish preservation of a specific personalized identity from generation of a generic instance of the same class.

The statistical testing is also insufficiently described. It is unclear whether the independent units in the reported tests are subjects, subject pairs, prompts, seeds, or individual generated images. Generated samples sharing an adapter or prompt are correlated, and treating them as independent would inflate statistical significance. The paper should report subject-level effect sizes and confidence intervals, not only selected p-values.

Overall, the evaluation provides preliminary evidence that the regularizer may be useful, but it does not convincingly establish that the reported gains are perceptually meaningful, statistically robust, or caused by preservation of the proposed geometry.

Questions for authors

What precise assumptions connect token-level cosine-Gram matching to preservation of the first fundamental form? If no such connection is claimed formally, should the contribution instead be characterized as within-sample similarity-preserving distillation?

How were the OOD prompts constructed and selected? Please provide the complete prompt sets, categories of distribution shifts, and results separated by shift type. Were these prompts fixed before selecting the regularization strength, layer, timestep, and warm-up schedule?

How stable are the multi-adapter rankings across CLIP thresholds? Please provide success-versus-threshold curves and agreement between the automatic metric and blinded human judgments.

What is the independent unit used in each statistical test? Please clarify how prompts and seeds are aggregated and report effect sizes and confidence intervals at the subject or subject-pair level.

Did RKD and SP receive the same hyperparameter-search budget as FF-LoRA? The critical comparison should use identical features, spatial tokens, timestep sampling, warm-up, and regularization-strength tuning.

Reviewer_8akoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FF-LoRA, which regularizes diffusion-model personalization by matching spatial-token cosine similarity matrices between the frozen base model and its LoRA-adapted version. Experiments report improved fidelity under prompt shifts and better multi-adapter composition, including complementary gains when combined with ZipLoRA.

Critical strengths
  1. Practical and simple approach. The regularizer addresses relevant problems in personalization and adapter composition without adding inference-time computation. Its complementary gains with ZipLoRA suggest practical value.

  2. Broad empirical investigation. The paper evaluates multiple baselines and backbones, includes DreamBooth-30, and studies regularization strength, feature location, and diffusion timesteps.

Critical weaknesses
  1. The methodological distinction and geometric claims need clarification. The intra-image component of Fine-Grained Manifold Distillation [1] already matches Gram matrices of L2-normalized patch features, closely resembling Eq. 3, but is not discussed. Although the paper acknowledges that its loss is a surrogate, §3.3 makes stronger claims about first-order geometry and the conformal class of the local metric without establishing this connection.

  2. Comparative and evaluation evidence remains incomplete. The experiments do not include ACCORD [2] for personalization regularization, LoRACLR [3] for multi-subject composition, or DuoLoRA [4] for subject–style merging. Comparisons accounting for their different training and merging requirements would help establish practical competitiveness. The manuscript also leaves composition success thresholds and leakage criteria insufficiently specified.

Questions for authors
  1. Methodological distinction. What methodological or empirical advances distinguish FF-LoRA from applying the intra-image relation-matching objective in [1] to diffusion LoRA? Given the surrogate interpretation acknowledged in the paper, could the authors justify or qualify the stronger geometric claims in §3.3?

  2. Comparisons and implementations. Could the authors provide comparisons with ACCORD, LoRACLR, and DuoLoRA in their respective settings, accounting for backbone, data, and computational differences, or explain the constraints preventing them? Please also clarify the SP normalization and whether the Mix-of-Show baseline includes ED-LoRA or only an adapted fusion component.

  3. Composition evaluation. What thresholds and decision rules define success and leakage in Table 2, and how were they selected? Are the conclusions stable across reasonable thresholds? Separate subject/style/text-alignment scores for the composition experiments, or a human-validated subset, would help assess the metric’s validity. Do the five mixing seeds in Table 2 vary generation only, or include independent adapter training?

[1] Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation. NeurIPS 2022.

[2] ACCORD: Alleviating Concept Coupling through Dependence Regularization for Text-to-Image Diffusion Personalization. ICLR 2026.

[3] LoRACLR: Contrastive Adaptation for Customization of Diffusion Models. CVPR 2025.

[4] DuoLoRA: Cycle-consistent and Rank-disentangled Content-Style Personalization. ICCV 2025.

Reviewer_HfyhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FF-LoRA, a regularizer for LoRA personalization of text-to-image diffusion models. For each training image, it computes cosine-similarity Gram matrices over spatial tokens of the U-Net mid-block and penalizes the difference between the frozen base model and the adapted model. The goal is better robustness to prompt rephrasing and better linear composition of independently trained adapters; the first-fundamental-form view is offered as motivation only. On SDXL, the authors report a smaller ID–OOD gap than LoRA-family baselines and higher two-concept success when mixing adapters, plus a small proof-of-concept on SD3-medium.

Critical strengths
  1. The paper is clearly written and candid about its scope: it places the loss in the RKD/SP family and states that the geometric framing is motivation.
  2. The regularizer is simple to add, keeps the LoRA parameterization unchanged, works at batch size 1, and adds no inference cost.
  3. On the reported metrics, FF-LoRA improves OOD subject fidelity over LoRA-family baselines while keeping ID fidelity.
Critical weaknesses
  1. Limited practical value over simple alternatives. The composition gains require both adapters to be FF-LoRA-trained; with one vanilla adapter, success (47%) falls below ZipLoRA on vanilla adapters (54%), and FF-LoRA alone roughly matches ZipLoRA (Table 2). Standard approaches such as DreamBooth prior preservation and caption augmentation, as well as encoder-based models like MS-Diffusion (ICLR 2025), are not compared.

  2. Overlap with prior work. Eq. 3 matches the pair-wise loss of Structured Knowledge Distillation (Liu et al., CVPR 2019), which already uses cosine similarities between spatial positions within one image. DCO (Lee et al., NeurIPS 2024) and Orthogonal Adaptation (Po et al., CVPR 2024) target the same two goals at training time; neither is cited or compared. The gain over the closest baselines is also small (Δrob 0.12 vs. 0.13 for RKD on DreamBooth-30, std 0.02, Table 1).

  3. Composition metric is unspecified and numbers disagree. The CLIP threshold behind two-concept success is not given, and the metric is not validated by humans or a VQA model. The gains over RKD/SP are stated as +4pp in Sec. 4.3 but are +7/+5pp in Table 2, and SP results differ between Table 2 and the Table 3 caption.

Overall, I lean toward rejection, mainly due to Weakness 2 and Weakness 3, with Weakness 1 limiting practical impact. Reconciling the composition numbers, validating the metric, and showing a clear advantage over DCO and Orthogonal Adaptation would raise my score.

Questions for authors
  1. At w=0.5, Fig. 5 shows about 73/84/91% success for LoRA/PaRa/FF-LoRA, while Table 2 reports 33/51/58% for the same setting. Also, why is two-concept success about 48% for FF-LoRA at w=0 and w=1, when only one adapter is active?

  2. Which layers receive LoRA? If only the U-Net is adapted, text-encoder features are identical for the base and adapted models, so how can the "Text encoder" row in Table 3 differ from vanilla LoRA? And what does "followed by spatial pooling" mean given the token-level Gram matrix in Eq. 3?

  3. Mixing uses wΔW1 + (1−w)ΔW2, so at w=0.5 each adapter runs at half strength. How do the methods compare under the standard additive merge with both adapters at full strength, as in DCO and Orthogonal Adaptation?

  4. For SD-1.5 at 512 resolution, the U-Net mid-block has 8×8 = 64 tokens. How are T=256 tokens sampled there (Appendix B)? And how does a full second U-Net forward pass add only 6% wall-clock time?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.