acceptodds
Under review as a conference paper at ICLR 2027

Your Few-shot Diffusion Models Secretly Localize to a Small Subspace: An Explanatory Framework in the Latent Space

Abstract

Few-shot diffusion personalization poses a puzzle: how can a handful of images induce subject or attribute changes without relearning a high-dimensional distribution? We propose SUDA-MoG, a shared–unshared framework that models adaptation as selective recalibration of factor-wise Mixture-of-Gaussians laws in frozen pretrained latents. Shared factors inherit source statistics, while unshared factors are calibrated from target examples. Under an ideally decoupled shared–unshared factorization, we derive a generative-error bound whose target-sample term depends on the free active score parameters under stated regularity conditions. For coupled scores, we bound the initial shared gradient through shared-domain mismatch and cross-block coupling. We instantiate the framework with a semantically guided Greedy construction that combines active-block selection with conditional residual completion. Across evaluated tasks, SUDA-MoG achieves target alignment comparable to DreamBooth-LoRA and Custom Diffusion. Matched interventions on DreamBooth-LoRA outputs show that task-aligned coordinates carry part of the adapter's task response under an independent latent evaluator. These results characterize a task-relative functional footprint of few-shot adaptation, connecting inherited structure, selective calibration, and observable task effects.

Primary area:
generative models
Keywords:
Diffusion Models, Few-shot Learning, Latent Manifold

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_M5WmOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SUDA-MoG, a framework for interpreting few-shot diffusion adaptation as a low-dimensional recalibration of latent probability laws. The main hypothesis is that a pretrained latent representation can be divided into shared factors, whose source-domain statistics remain unchanged, and unshared or active factors, whose Mixture-of-Gaussians parameters are updated using a small target support set. The encoder, decoder, and latent factor map are otherwise frozen.

The theoretical analysis provides two main results. Under an exactly factorized score model, the authors derive a finite-sample generative-error bound whose explicit estimation term depends on the number of active score parameters rather than the full latent dimension. Under a coupled score model, they bound the initial target gradient on shared parameters using a shared-domain mismatch term and a cross-block Hessian coupling term. The practical method, called Greedy, uses supervised directions to construct active coordinates and models the selected factor distributions with MoGs.

Experiments cover Dog-to-Cat transfer, facial-attribute editing, synthetic score-coupling tests, and interventions on DreamBooth-LoRA outputs. The strongest result is a matched intervention in which transplanting one task-aligned coordinate from a LoRA output into the corresponding base-model latent recovers 42.68% of the LoRA-induced change measured by an external Dog-versus-Cat classifier.

The paper presents an interesting descriptive perspective, but its central claim is significantly stronger than what the theory and experiments establish. The theory assumes that the relevant factorization and invariant marginals are already known, while most experiments evaluate supervised latent editing rather than explaining how real diffusion personalization methods learn.

Critical strengths

The distinction between inherited and recalibrated latent statistics provides a potentially useful vocabulary for analyzing personalization. Unlike work that only observes low-rank parameter updates, this paper attempts to describe which parts of the generated distribution change. The separation between coordinate extractors, external evaluators, active factors, and residual coordinates is also conceptually careful.

This is consequential because understanding why few-shot personalization works is an important problem. A validated account of which pretrained statistics are reused and which are modified could inform more efficient adaptation procedures.

The intervention on DreamBooth-LoRA outputs is stronger than a conventional probe experiment. The authors transplant the LoRA output’s task coordinate into the paired base output while retaining the remaining base coordinates. They evaluate the result using a separately trained classifier and include shuffled-label and rotation controls. These choices provide some evidence that the discovered coordinate functionally contributes to the measured adapter response.

However, this experiment supports only a narrow conclusion: one supervised coordinate captures part of one classifier’s response in a Dog-to-Cat task. It does not demonstrate that LoRA adaptation generally operates by recalibrating a low-dimensional MoG distribution.

Critical weaknesses

The principal theoretical result conditions on a fixed factor map, a correct shared–unshared partition, invariant shared marginals, known source distributions, regular score classes, and a stable sampler. Once an oracle has identified a low-dimensional active parameterization and declared the remaining target statistics invariant, it is unsurprising that estimating only the active parameters produces a complexity term depending on their number.

Crucially, the analysis omits the cost of discovering this representation. In the practical experiments, the chart, supervised task directions, residual statistics, and factor laws are estimated from the same limited support data. Therefore, the theorem does not analyze the procedure that supposedly provides the method’s few-shot advantage.

The favorable estimation term also appears alongside uncontrolled approximation terms covering factorization error, MoG misspecification, partition error, nonlinear coupling, VAE reconstruction, positive-noise truncation, and sampler discrepancy. The paper does not estimate these quantities. They may dominate the stated active-parameter term, leaving the bound without meaningful predictive content in the evaluated settings.

The shared-factor stability result is similarly weak. In the additive construction, score blocks are parameterized separately. Consequently, the cross-parameter Hessian is zero by construction. If the shared target marginal is additionally assumed to equal the source marginal, the shared gradient is expected to vanish. The coupled extension is essentially a generic smoothness argument based on the cross-Hessian. These results formalize consequences of the assumed decomposition rather than showing that learned diffusion adapters possess such a decomposition.

Most experiments evaluate a supervised latent editor. For example, facial-attribute directions are learned using explicit labels, and the active rank is chosen based on the number of targeted attributes. This setting supplies the algorithm with the semantic factors that ordinary subject-personalization methods must discover from a few unlabeled images.

The main comparison is also inadequate. On Dog-to-Cat, SUDA-MoG obtains CLIP-T 0.265, compared with 0.264 for DreamBooth-LoRA and Custom Diffusion. This difference is negligible, and no statistical comparison is reported. More importantly, SUDA-MoG appears to perform image-conditioned latent transfer while the baselines are normally intended for text-conditioned generation across novel prompts. A target-only CLIP score cannot establish comparable personalization quality without evaluating identity preservation, prompt generalization, diversity, carrier preservation, and memorization.

The CelebA effects are statistically positive but extremely small, with CLIP-T improvements on the order of 0.0016–0.0028. The paper provides no human evaluation establishing that these changes are perceptually meaningful. Evaluating drift on one untargeted binary attribute is also insufficient to support a broad claim of selective preservation.

Finally, the 42.68% LoRA recovery result is tied to one coordinate, one binary category transfer, and one class-discriminative evaluator. Recovering part of a classifier score does not mean that the same fraction of the adapter’s full generative behavior has been recovered. The paper does not evaluate texture, geometry, diversity, prompt response, or distributional correspondence. It also lacks comparisons with simpler alternatives such as LDA without MoG modeling, the mean paired base-to-LoRA difference, PCA of paired differences, partial least squares, or reduced-rank regression. The result may therefore reflect ordinary supervised linear separability rather than the proposed latent-law mechanism.

Questions for authors

Is SUDA-MoG intended as a competitive personalization method, a post hoc model of trained adapters, or a supervised latent-editing framework? The paper currently moves among these interpretations.

Can the authors provide a theoretical result that includes data-dependent discovery of the factor map and shared–unshared partition from the same few-shot support set?

Can the authors empirically estimate the approximation and VAE terms in the main bound? Without this information, how can readers determine whether the favorable active-dimensional term is consequential?

Does the fitted MoG predict the distribution of LoRA-induced coordinate changes across held-out prompts, carriers, and random seeds, rather than merely recovering one classifier response?

How does the reported intervention compare with simpler baselines, including LDA without MoG modeling, paired-difference PCA, a mean paired-difference direction, and reduced-rank regression?

Are all methods in the Dog-to-Cat comparison evaluated under matched image-conditioning, prompting, preservation, and sampling protocols?


POST REBUTTAL COMMENTS

The authors have addressed most of the comments. So, I raise my score to weak reject.

6 comments

Thank you for your careful reading and for distinguishing between discovering a factor and fitting its statistics once it is known. We respond to your concerns and questions below.

W1, W2 & Q2: Factor-map assumptions and the cost of discovery

Theorem 5.1 covers calibration on a fixed map, which is intended to highlight our core rationale: the non-shared components are what truly play a role during the fine-tuning process. We extend it to Theorem J.1, which adds the cost of learning the map and factor roles from the same target images. At resolution 1/n1/n, the estimation term counts the free unshared parameters, the log covering number of the map family and the log number of candidate partitions. A covering number at most (An)q(An)^q adds at most qlog⁡(An)q\log(An) to the numerator. This extension requires uniform control of the DSM loss and score error over candidate maps. Each selected map and partition has its own population approximation error.

For a mean shift, the cost of discovery has a simple interpretation. The reduction in squared transport error equals the squared true shift captured minus the squared sampling noise captured. When the true shift is zero and the noise is isotropic in DD dimensions, the observed mean-difference direction captures DD times as much squared noise in expectation as an independently fixed direction, although both updates have rank one (§5.1 and Appendix J.1).

We tested how this trade-off changes with sample size across eight CelebA transfers. With 16 supports, an axis learned on 1,536 separate labeled faces reduced squared mean mismatch by 40.1%, while fitting the axis on the 16 supports increased mismatch by 169.8%. With 128 supports, the support-fitted axis reduced mismatch by 54.3%, compared with 40.5% for the independently learned axis. To test factor discovery in the nonlinear Greedy editor, we used the same 32 supports to fit both the map and the factor distributions. Correct labels steered at least two target attributes on 56/64 new faces, compared with 24/64 after permuting the labels. Blue revisions: §§5.1 and 6.2 and Appendix J.

W3 & Q3: Estimation, approximation and sampling error

Thank you for asking how large the approximation and reconstruction errors are in practice. With four target images, estimating the center and width contributes over 15 times as much squared error as the shape residual on the tested factors. We fixed the factor and mixture shape and compared inheriting the source width with estimating it from the same supports. Across four CelebA factors in both transfer directions, 256 independent test images per direction give the following decomposition of one-dimensional squared Wasserstein error in squared source-SD units.

Target imagesShape residual (both)Center error (both)Width error: inherit / re-estimateTotal: inherit / re-estimate
40.0270.2610.048 / 0.1570.336 / 0.445
320.0270.0320.048 / 0.0190.107 / 0.078

At four supports, the fitted center and width contribute 0.261 + 0.157 = 0.418, compared with 0.027 for shape. Fitting width alone adds 0.109 over inheriting it, about four times the shape residual. At 32 supports it instead reduces total error by 0.029. The discrepancy between the 16-step sampler and the fitted law was below 0.0001 in all four settings.

We then tested whether inheriting the spread also helped predict the outputs of trained subject LoRAs. The observed DINO radius was 0.764, close to the base prediction of 0.778 and larger than the four-photo prediction of 0.493. Energy distance favored the base spread in all 15 classes. Empirical squared transport between two sets of 30 DINO features favored the narrower set (0.635 versus 0.837), a finite-sample effect also seen when splitting outputs from the same base generator (Appendix K.1).

To measure the reconstruction term in image space, we passed 32 real target faces through our frozen deep compression autoencoder (DC-AE). The RGB transport distance between the original and reconstructed faces was 0.031, about one tenth of the baseline transport distance of 0.297. Editing reduced the latter to 0.283. The fixed-factor comparison above isolates the estimation cost and shows when fitting one additional width parameter helps. Blue revisions: §§5.1 and 6.3 and Appendices J–K.

W4: Shared-factor stability under invariance and coupling

We appreciate your question about what the stability result predicts once the factors interact. At the same unshared displacement, stronger coupling produces a larger shared gradient. In the factorized theorem the shared gradient vanishes under shared marginal invariance and source stationarity. The coupled bound predicts how shared mismatch and cross-block curvature enter when those conditions are relaxed. Across 64 fitted joint score models, the high-coupling models had 2.57 times the shared-parameter gradient of the low-coupling models at matched displacement. In a trained SD1.5 LoRA, the change in cross-direction curvature predicted the orthogonal loss-gradient shift at a larger displacement, lowering RMSE from 0.00164 under the unchanged base response to 0.00073.

A separate experiment tests the functional role of a pretrained task direction during sampling. We use three independently trained rank-16 Dog→Cat LoRAs and the same 12 input dogs. At every DDIM step, we decompose the LoRA-minus-base noise prediction along the base U-Net’s Cat-versus-Dog score direction. Image, initial noise and prompt are identical, and the orthogonal control is matched to the task component’s norm for every image and step. Retaining the task component raises the final RGB target margin by 0.0188 ± 0.0053 and an independent real-image DINO task readout by 0.1089 ± 0.0417 relative to that control across the three seeds. Paired-base LPIPS differs by only 0.0060 ± 0.0044. This intervention measures the contribution of the task-aligned update during sampling in the trained denoiser. Blue revisions: §§5.2 and 6.1 and Appendix K.

W5 & Q1: SUDA-MoG as a framework for explaining diffusion personalization

Thank you for asking us to state the primary purpose of SUDA-MoG. SUDA-MoG is an analytical framework for explaining and predicting which output statistics change during few-shot diffusion fine-tuning. We use Greedy as a controlled intervention to isolate the effects of shared and unshared factors. Independently trained LoRAs let us test the resulting predictions on new outputs.

Figure 5 isolates the effect of the training photos in ordinary text-to-image personalization. We trained six LoRAs in three same-class pairs, changing only the four training photos within each pair, then generated outputs from matched Gaussian noise under two new prompts. The difference between the four-photo DINO centroids predicted the ordering of 144/144 paired outputs. On the harder test of identifying each image individually, nearest-centroid retrieval against eight unused real photos gave 244/288 correct, versus 134/288 for base.

Greedy uses supervised directions, as in InterFaceGAN [1]. Along a readable direction, the mixture lets us ask whether shifting its components or changing their frequencies better predicts the distribution of new LoRA outputs. Selecting component count on separate fitting images improved test log density over one Gaussian by 0.163 nats on AFHQ dog/cat and by 0.037 nats per coordinate across four CelebA attributes. Two of the four CelebA attributes selected a single Gaussian. Our reply to Q4 gives the held-out frequency forecasts, and our reply to W4 gives the denoising intervention. Blue revisions: §§2, 6.1, and 6.3 and Appendices J–K.

W6 & Q6: Matched evaluation protocols for Dog→Cat

We appreciate your concern about how the choice of inputs and prompts affects this comparison. Table 2 matches the ten target cats, 40 input dogs, prompts and sampling noise across all three methods and reports both target gain and input fidelity. Both trained adapters receive 300 updates, while Greedy uses 192 additional source dogs to fit its factor map. Greedy and DreamBooth-LoRA produce similar LPIPS distances from the paired base image, 0.254 and 0.247 respectively. Greedy gives a larger mean cat-prompt gain, +0.0392 versus +0.0229 for LoRA, while LoRA preserves more of the input dog’s appearance, with DINO similarity of 0.506 versus 0.420 for Greedy. Custom Diffusion gives the largest mean prompt gain at +0.0410 and the largest LPIPS distance from the base at 0.442. Figure 5 separately tests subject identity under new prompts in text-to-image generation. Blue revisions: §6.2 and Table 2.

W7: Target-attribute changes and drift on other CelebA attributes

Thank you for questioning what a small CLIP-T gain tells us about a facial attribute. Independent attribute classifiers detect the intended local changes on the same faces. Figure 4(c) compares a whole-face embedding with a short attribute phrase. Opening the mouth slightly or adding bangs changes only part of that image. We therefore also measured each attribute with an independent logistic classifier using histograms of oriented gradients (HOG). Mouth opening raised CLIP-T by 0.00160 and the HOG classifier probability by 0.0530. The corresponding gains were 0.00246 and 0.0554 for bangs, and 0.00283 and 0.0619 for smiling (Appendix H.4). Figure 4(a) illustrates separate eyeglasses and haircut edits.

On 64 new faces, we tested whether the edit consistently moved the intended attributes. We measured changes in 19 CLIP attribute scores in units of each attribute’s class gap. With correct labels, the mean signed change was +0.063 on the three targets and +0.001 on the 16 untargeted attributes. Mean absolute changes were 0.175 and 0.182, respectively. The corresponding untargeted change under shuffled support labels was 0.170. Correct labels made the target changes more consistent across support draws. On 56/64 faces, at least two target attributes each moved in the intended direction in at least six of ten draws, compared with 24/64 under shuffled labels. Blue revisions: §6.2, Figure 4 and Appendices H.4 and K.3.

W8 & Q4: Predicting LoRA output distributions across inputs, prompts and seeds

Thank you for asking what we can predict before seeing the adapted outputs. On new Dog→Cat LoRAs, mixture reweighting predicts the fraction of cat-positive outputs more accurately than a uniform shift with the same mean. Under the neutral prompt the two forecasts were 77.0% and 69.5%, respectively. The observed rate across three newly trained LoRAs was 77.3%.

We calibrated both forecasts on 64 inputs using two historical LoRAs and froze them before testing on 64 new dogs and 64 new cats. Translation moves every score by the same amount. Reweighting puts more probability on the more cat-like component, so the two forecasts can agree on the mean and still predict different fractions above the cat-positive threshold (Proposition J.3). We also compared the full score distributions on two readouts.

Measurement on the same 128 outputsTranslation W1Component reallocation W1
RGB cat-minus-dog CLIP score0.0233 ± 0.00070.0073 ± 0.0008
Frozen latent task coordinate (evaluated in Q5)0.536 ± 0.1390.396 ± 0.134

Values are means ± sample SD across the three test seeds. Reweighting reduced distribution error on both readouts in every seed. Even allowing a separate historical shift for dogs and for cats gave a higher mean W1W_1 on the pooled RGB scores (0.0246 versus 0.0073 for reweighting) at the same overall mean shift.

With calibration frozen across three additional prompts, reallocation beat uniform translation in 9/9 RGB comparisons (mean W1W_1 0.0102 versus 0.0247) and 8/9 latent comparisons (0.685 versus 0.719). Appendix K.2 also reports the affine-scale comparison. Blue revisions: §6.3 and Appendices J–K.

Reviewer_6uy9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes SUDA-MoG, a method that explains few-shot personalisation (similar to DreamBooth, Text inversion, etc). The latent is modelled as a Mixture of Gaussians. Shared factors keep the source statistics, while unshared factors are assumed to be active and updated by few target images. Experiments show comparable ClipScore to DreamBooth.

Critical strengths
  • Understanding which statistics change in few-shot adaptation is an interesting question and important for multiple tasks in diffusion models, beyond personalisation
  • The intervention-based check against a real trained LoRA (Sec. 6.3) is the right kind of experiment for an explanatory claim, even though it has problems in its current form.
  • Good controls (PCA, random directions etc.) are included in the paper
Critical weaknesses

The main issues of the paper are related to clarity and reproducibility. The core mechanism is never clearly and plainly stated in the main text. The method is written heavily in jargon and is obscured by heavy mathematical formalisation. However, in essence, what the method describes is a version of supervised linear latent editing (https://arxiv.org/abs/2005.09635) adapted from GANs.

Some additional issues:

  • Several symbols are used before or without definition (eg., η\eta, but there are many)
  • Key results are not self-contained: Theorem 5.1 holds “under the full hypotheses of Theorem C.8." (latter is in the appendix)
  • Table 2 shows 0.265 vs. 0.264 vs. 0.264 with no variance. These results seem statistically tied (even with no variance, it's unlikely that there is significance)
  • In Table 3, the ablation seems inconclusive as S/U and All seem to win for different splits (plus some 100% accuracy entries appear like either the test set is small or there is a mistake in the table)
Questions for authors
  • What explains the different CLIP-T values for Dog-to-Cat across Tables 1–3 and Fig. 4?
  • How does the method compare to InterFaceGAN-style or h-space direction editing using the same supports?
  • How are the target accuracies in Table 3 computed (which classifier, how many images), and what is their variance?
  • In a nutshell, can you explain your method plainly and cleanly? What are the key contributions and novelty?
4 comments

Thank you for asking us to explain the mechanism more plainly and clarify the experimental comparisons. We address your questions below.

W1 & Q4: Plain explanation of the mechanism, contributions, and novelty

How it works. SUDA-MoG models what changes when a pretrained diffusion model is personalized with a few photos. In a frozen latent space, we keep the pretrained distributions of shared factors, such as pose or background, and use the photos to recalibrate a few unshared factors, such as subject identity. We model each factor with a mixture of Gaussians (MoG) [1, 2]. Adding Gaussian noise during diffusion preserves the mixture form, so we can compute how recalibrating a factor changes its denoising score at each noise level (Eqs. 2 and 10).

What this adds beyond InterFaceGAN-style editing [3]. A linear edit shifts every sample by the same vector. We also model how outputs are distributed along that direction and test how this distribution changes with personalization.

  1. Theory. With the factor map and roles fixed, the estimation term depends on how many unshared parameters we fit from the target photos. Learning the map and roles from those same photos adds a discovery cost (Theorems 5.1 and J.1).
  2. Controlled editing. Greedy replaces selected factor coordinates while keeping each image’s conditional residual fixed. We compare edits to shared factors, unshared factors, or both to see which changes move the target attribute and which preserve the input image.
  3. Tests on trained LoRAs. Across 36 subject LoRAs, the training photos helped predict where outputs moved. For the image-conditioned LoRAs, the base model also predicted much of the spread (C1). On Dog→Cat, two forecasts matched the same mean shift. Reweighting predicted 77.0% cat-positive outputs and translation predicted 69.5%, with 77.3% observed (C2). A separate denoising intervention showed that the LoRA update along a pretrained task direction contributed to the shift (C3).

Blue revisions: §§2, 3.2, 4.1, 5.1, 6.1–6.3, and Appendices J–K.

W2 & W3: Notation and assumptions of Theorem 5.1

Thank you for flagging the undefined symbols and the assumptions that were hard to locate. The factor law is the distribution you get by encoding images and reading out their factor coordinates. Sections 3.2–4.1 define this law, its mixture parameters, and the shared and unshared roles. Theorem 5.1 states the conditions on the factor map, source fit and sampler in the main text, followed by the formulas for approximation and reconstruction error. With the map and roles fixed, the sample-size term counts the free unshared score parameters. Theorem J.1 adds the cost of selecting the map and roles from the target images. Blue revisions: §§3.2, 4.1 and 5.1 and Appendix J.1.

W4 & Q1: Dog→Cat CLIP-T scores and variability across protocols

Thank you for flagging the different Dog→Cat scores. Table 2 measures how much an edit raises the cat-prompt score over its paired base output. Tables 1 and 3 and Fig. 4(b) report the final cat-prompt score under different edits.

LocationWhat variesMetricTest inputs and conditionsReported score
Table 1DreamBooth-LoRA rankAbsolute CLIP-T20 images per rank0.275–0.281
Table 2Greedy, DreamBooth-LoRA, Custom DiffusionPaired CLIP-T gain over baseSame 40 dogs, prompts and noise across methods, 3 seedsGreedy +0.0392
Table 3Shared, unshared or all factorsAbsolute CLIP-T30 fixed dogs, three-component Greedy edit0.2669 (unshared), 0.2716 (all)
Fig. 4(b)LDA, PCA, random or shuffled-label directionAbsolute CLIP-TDog→Cat direction edits without per-image LPIPS matching0.2817 (LDA)

Blue revisions: §§4 and 6.2, Tables 1–3 and Fig. 4(b).

W5 & Q3: Role ablations and the two 100% accuracy entries

Thank you for asking what the two 100% entries mean and how much the scores vary. On all three tasks, unshared-only edits (U) reached the target more often than shared-only edits (S), while S preserved more of the input. Table 2 compares methods on base outputs for 40 dogs across three seeds. Table 3 compares U, S and All on 30 fixed inputs per task. CLIP-T is the mean ± sample SD across images. We give target accuracy as the count for which CLIP ViT-B/32 favors the target prompt. Both tables measure DINO similarity to the input. LPIPS uses the paired base output in Table 2 and the original input in Table 3.

TaskFactors editedTarget CLIP-T ↑Target countLAB sim. ↑Input DINO ↑Input LPIPS ↓
Dog→CatU0.2669 ± 0.014530/300.8310.04240.6872
Dog→CatS0.2277 ± 0.017110/300.9370.40920.4939
Dog→CatAll0.2716 ± 0.003230/300.8210.02340.6864
GlassesU0.2745 ± 0.012217/300.8230.32970.6834
GlassesS0.2263 ± 0.01251/300.9150.71370.4373
GlassesAll0.2774 ± 0.011416/300.8100.33770.6840
Young→OldU0.2459 ± 0.013423/300.7950.43780.6106
Young→OldS0.2243 ± 0.01515/300.9120.71340.4450
Young→OldAll0.2424 ± 0.012218/300.7820.41200.6342

All 30 dogs passed the threshold under both U and All. On Glasses, their counts differ by one face. On Young→Old, U reached the target on five more faces and preserved the input better on both fidelity measures. One explanation is that when the true shift on a shared coordinate is small, fitting it from the target photos can introduce sampling noise (Theorem 5.1 and Proposition J.2). Blue revisions: §6.2, Table 3 and Appendix J.

Q2: Direction editing on the same Dog→Cat supports

We appreciate the connection to InterFaceGAN and respectfully disagree with characterizing the whole framework as a linear editor. A supervised direction specifies an edit, while the mixture also predicts how often different outputs will occur after personalization. We fitted all three directions on the same ten cats and 192 source dogs. On 4040 test dogs, we calibrated the edit strength separately for each input and direction to match an LPIPS distance of 0.1210.121 from the base image. At this matched perceptual budget, Appendix K.3 shows that Greedy achieves the largest mean cat-prompt CLIP gain at 0.02408, outperforming both the class-mean direction at 0.01544 and the InterFaceGAN-style linear SVM at 0.00960. The separate U-Net bottleneck edit gave gains of 0.00498, 0.01419 and 0.02914 at LPIPS distances of 0.087, 0.154 and 0.237, respectively.

The same task also tests a distributional prediction. Two changes with the same mean shift can put different fractions of the outputs on the cat side of the decision boundary. We fixed the mean shift using historical LoRAs and forecast the fraction on the cat side for three new LoRAs. As shown in Common C2, shifting every score uniformly predicted 69.5%69.5\% cat-positive outputs, whereas reallocating the fitted mixture component weights predicted 77.0% against 77.3% observed, demonstrating the concrete predictive value added by the mixture formulation. Blue revisions: §6.2 and Appendix K.

References

[1] Ruofeng Yang, Yongcan Li, Bo Jiang, Cheng Chen, and Shuai Li. Multi-Subspace Multi-Modal Modeling for Diffusion Models: Estimation, Convergence and Mixture of Experts. ICLR, 2026.

[2] Kulin Shah, Sitan Chen, and Adam Klivans. Learning Mixtures of Gaussians Using the DDPM Objective. NeurIPS, 2023.

[3] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the Latent Space of GANs for Semantic Face Editing. CVPR, 2020.

Thanks for the rebuttal. I have also seen the updated PDF. Unfortunately, I don't think the clarity issues can be resolved within a review cycle, as the paper needs heavy rewriting rather than editing of sections. The added variances help with understanding, as well as the clarifications requested by other reviewers.

Thank you for reading the revised manuscript. We’re glad the added variability estimates and clarifications helped. Your initial review guided several changes.

  • Notation and theory (W2, W3). Sections 3.2–4.1 now explain how we obtain a factor distribution from encoded images and define its mixture parameters. Section 5.1 states the assumptions of Theorem 5.1 alongside the result.
  • Experimental comparisons (W4, W5, Q1, Q3). Table 2 reports the gain in CLIP-T over each input’s paired base output, together with variability and fidelity measures. Tables 1 and 3 report absolute CLIP-T scores. Table 3 also gives the variation in CLIP-T across images and explains how target accuracy is computed. Both 100% entries mean that CLIP favored the cat prompt on all 30 test dogs, once when editing only unshared factors and once when editing all factors.
  • Relation to InterFaceGAN-style editing (W1, Q2, Q4). On the same Dog→Cat supports and at matched LPIPS, Greedy gave a larger cat-prompt gain than an InterFaceGAN-style linear SVM direction (0.0241 vs. 0.0096). We then tested whether a uniform shift and a change in mixture weights make different predictions about the output distribution, even when both match the same mean shift. At the exact same mean shift, uniform translation predicted that 69.5% of outputs would score higher for cat than dog, while mixture reweighting predicted 77.0%. We fixed both forecasts before testing three new LoRAs and observed 77.3% on a balanced set of dog and cat inputs (§6.3, Appendix K).

The question connecting these experiments is what a few target photos change in a pretrained diffusion model and what the model can reuse. Our hypothesis is that the photos recalibrate a few unshared factors while shared factors retain their pretrained distributions. The theory asks how much data that calibration requires. Greedy lets us change the two groups separately and measure their effects. The tests on independently trained LoRAs then ask whether this account predicts how their outputs change.

Could you point us to a passage in the revised manuscript where a step in this argument, or the transition between two steps, was still unclear? We’d appreciate the opportunity to clarify it here.

Reviewer_5JBaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a personalization method that models the diffusion latent space as a mixture of Gaussians (MoG) to separate it into shared and unshared factors. SUDA-MoG uses supervision to identify the unshared factors to be adjusted for personalization. This allows the method to update only the factors associated with the desired attributes of the target concept while preserving the original distribution. The paper also provides a theoretical analysis of the shared and unshared factor structure, and experimental results on AFHQ and CelebA show that the proposed method can shift the source data distribution toward the target data distribution.

Critical strengths

The paper presents the problem setting in a well-structured manner and seeks to provide a mathematical characterization of the setting. The proposed approach to personalization is also reasonable, as it distinguishes shared and unshared factors and outlines how this distinction can be used.

Critical weaknesses

The authors present the results in Table 1 as evidence that only a small number of parameters need to be adjusted. However, the differences in CLIP-T scores are small. The experimental setup also needs to be described in more detail. In addition to CLIP-T, the authors should report changes in image fidelity metrics, as image fidelity is an important aspect of few-shot personalization. The same issue appears in the evaluations in Tables 2 and 3, making the motivation and interpretation of the proposed method unclear. Therefore, the authors should also report scores such as DINO and CLIP-I and provide more details on the experimental setup.

The proposed method has limitations in scalability and generalizability. It relies on task-specific labels or attribute supervision to identify active factors and determine their shared and unshared roles, and then adjusts the statistics of unshared factors using target examples. This task-specific preparation limits its extension beyond specific data domains such as AFHQ and CelebA to broader personalization settings, such as text-to-image diffusion models involving diverse concepts and open-ended conditioning. The authors should further develop the approach toward more general personalization by building on the observed controllability of shared and unshared factors.

Questions for authors

The authors should provide additional experiments and evidence to address the weaknesses discussed above. Further validation of the proposed method and discussion of potential directions for extending its applicability beyond controlled experimental settings would strengthen the paper.

5 comments

Thank you for your questions about image fidelity and how well our findings generalize. We address each point below.

W1 & Q1: LoRA rank, edit magnitude and image fidelity

Thank you for pointing out that target alignment alone tells us little about how much of the input image an edit preserves. Rank 1 already moves outputs toward cat, but the paired measurements reveal a larger edit at rank 16. Table 1 reports nearly identical absolute target CLIP-T scores (0.275–0.281) across 20 images per rank. On 40 paired dogs and three seeds, increasing rank from 1 to 16 raises the cat-minus-dog CLIP gain from 0.0113 to 0.0312 and LPIPS from the paired base from 0.105 to 0.247. DINO similarity to the input dog stays close (0.494 and 0.506).

Table 2 compares three methods on the same ten target cats and 40 input dogs, with matched prompts and noise. Both trained adapters received 300 updates. Greedy used 192 additional source dogs to fit its factor map.

MethodTarget CLIP-T gain ↑CLIP-I to input ↑DINO to input ↑LPIPS to base ↓
Greedy0.0392 ± 0.00070.727 ± 0.0580.420 ± 0.1440.254 ± 0.018
DreamBooth-LoRA0.0229 ± 0.00170.774 ± 0.0560.506 ± 0.1520.247 ± 0.071
Custom Diffusion0.0410 ± 0.00110.727 ± 0.0420.282 ± 0.1510.442 ± 0.064

CLIP-T gain is reported as mean ±\pm SD across the three seed means. For fidelity, we first averaged over seeds for each of the 4040 dogs and then computed the SD across dogs. Greedy used one fitted edit on the base images from each seed. Greedy and DreamBooth-LoRA alter the paired base outputs by nearly the same LPIPS distance of 0.2540.254 versus 0.2470.247: Greedy achieves a larger mean gain on the cat prompt, whereas LoRA preserves more of the input dog's appearance on both CLIP-I and DINO. Custom Diffusion achieves the largest prompt gain but also alters the base images the most, reaching a base LPIPS of 0.4420.442.

Table 3 varies which factors Greedy edits on 30 fixed inputs per task. Both DINO and LPIPS use the original input as the reference here. Table 2 measures LPIPS from the paired base output. Unshared-only edits reached the target more often than shared-only edits on Dog→Cat (30 vs. 10), Glasses (17 vs. 1) and Young→Old (23 vs. 5). Shared-only edits stayed closest to the original inputs on both fidelity measures in all three tasks. On Young→Old, editing only unshared factors also preserved more of the face than editing all factors (DINO 0.4378 vs. 0.4120, LPIPS 0.6106 vs. 0.6342) and reached the target on five more faces. Blue revisions: §§4 and 6.2, Tables 1–3, and Appendices J–K.

W2 & Q1: Supervision for factor discovery and transfer beyond AFHQ and CelebA

We appreciate your question about generalization beyond AFHQ and CelebA. The subject prediction extends to ordinary text-to-image LoRAs and uses their training photos without attribute labels. We tested three same-class pairs of backpacks, cats and sneakers. Within each pair we changed only the four training photos. The difference between the two photo centroids in DINO space predicts a direction before either LoRA generates an image.

For each subject pair, we generated from noise alone under two new prompts with 24 shared noise draws. Along the direction from subject B’s training photos to subject A’s, A’s generated image lay farther toward A than B’s in 144/144 paired comparisons across the three classes. We also assessed each image individually using real reference photos excluded from training. As shown in Figure 5 and Common C1, these references identified the correct subject in 84.7% of LoRA outputs versus 46.5% of base outputs across all six subjects, extending our framework's predictions to subject identity under new prompts.

For the Greedy editor, we measured how much attribute supervision it takes to find and use an editing direction. The same 8, 16 or 32 labeled CelebA images were used both to fit the directions and to calibrate their distributions. Independent logistic classifiers using histograms of oriented gradients (HOG) measured mean target-score gains of 0.0370.037, 0.0400.040, and 0.0570.057 across ten draws. We then shuffled the labels to test whether they identified the intended attributes. On 64 new faces, at least two of three target scores moved as intended in a majority of draws for 56 faces with correct labels and 24 with shuffled labels. A natural extension to open-ended prompts is to estimate a subject direction once from its photos and test whether it continues to predict identity as the prompt changes the subject’s pose, activity or surroundings. Blue revisions: §§6.2–6.3, Figure 5, and Appendices J–K.

Reviewer_5JBaOpenReview ↗

Thank you

Thank you for the response and additional experiments.

While the additional results address some of my concerns, the scalability and generalizability of the proposed framework to broader personalization settings remain unclear. The additional experiments are limited in scope and validation.

I will maintain my original rating.

AuthorsOpenReview ↗

Overview of the personalization experiments

Thank you for following up and for taking the time to read our response and the additional experiments. We appreciate your feedback on the scope of the validation.

Your initial review asked whether the framework could extend beyond AFHQ and CelebA to text-to-image personalization with diverse subjects, given the task-specific labels used in our editing experiments. Our reply to W2 focused on the six text-to-image LoRAs in Fig. 5. We reported the larger study of 30 subject LoRAs separately in Common Response C1 and Appendix K.1. We could have made the connection between these studies clearer.

The revised experiments cover 42 independently trained diffusion adapters, comprising 36 subject LoRAs, three Dog→Cat LoRAs, and three eyeglasses LoRAs. The first two rows below address subject generalization directly. The remaining rows summarize how we tested the mechanism and where attribute supervision enters.

SettingScope and supervisionMain result
Text-to-image subject personalization (§6.3, Fig. 5)Six LoRAs on three same-class subject pairs (backpacks, cats, and sneakers). Four training photos per subject, no attribute labels.Under two new prompts and 24 shared noise draws, the training photos predicted the relative ordering of 144/144 output pairs. Retrieval against unused real photos identified the correct subject in 84.7% of individual outputs, versus 46.5% for the base model.
Generalization across subject classes (§6.3, App. K.1)30 image-conditioned LoRAs across 15 classes, holding out one entire class at a time. Four photos per subject were used for calibration, without attribute labels.Calibrating the output center reduced prediction error in 14/15 held-out classes compared with using the shift learned from other classes alone. With the center fixed, retaining the base model’s spread gave a lower mean squared energy distance in DINO space than estimating spread from four photos (0.0726 vs. 0.1280), with lower error in all 15 classes.
Intervention during denoising (§6.1, App. K.2)Three Dog→Cat and three eyeglasses LoRAs. A pair of prompts defined the pretrained task direction at each denoising step.On Dog→Cat, the task-aligned component carried 5.1% of the update energy and retained 55% of the full LoRA’s CLIP-margin gain and 43% of its DINO gain. On both tasks, it produced a larger semantic change than an equal-norm orthogonal update in every seed.
Forecasting new LoRA outputs (§6.3, App. K.2)We fixed both forecasts before testing three new Dog→Cat LoRAs on 64 dog and 64 cat inputs.At the same mean shift, mixture reweighting predicted 77.0% cat-positive outputs under the neutral prompt, compared with 77.3% observed and 69.5% predicted by uniform translation. Reweighting also gave lower RGB score-distribution error in all nine prompt-seed comparisons across three additional prompts.
Few-shot attribute editing (§6.2, App. J.3, App. K.3)Greedy used 8, 16, or 32 labeled CelebA faces to learn editing directions and calibrate their distributions.Independent attribute classifiers measured mean target-score gains of 0.037, 0.040, and 0.057 with 8, 16, and 32 labeled faces, respectively. With 32 supports, 56/64 new faces met the following criterion: at least two of the three target attributes each moved in the intended direction in at least six of ten draws. With shuffled support labels, 24/64 faces met the same criterion.

Both subject-personalization studies used the subjects’ training photos without task-specific attribute labels. We used attribute labels when training Greedy to edit a named facial attribute, and the final row shows how its performance varied with the number of labeled examples.

We hope this gives a clearer picture of the evidence behind our generalization claim. If you had a particular personalization setting or supervision requirement in mind that these studies still leave open, we would appreciate an example and would be glad to discuss it here.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Common response

We thank all three reviewers for their feedback. To answer M5Wm Q1 and 6uy9 W1/Q4, SUDA-MoG is an analytical framework for explaining and predicting which output statistics change during few-shot diffusion personalization. Revisions are marked in blue.

  1. What changes. We hypothesize that shared factors, such as pose or background in subject personalization, retain their pretrained distributions. The photos recalibrate a few unshared factors that distinguish the subject or target attribute. Gaussian mixtures let us model these changes and compute the corresponding denoising score in closed form (Eqs. 2 and 10).
  2. What a distribution predicts beyond an editing direction. An InterFaceGAN-style edit [3] shifts each sample along a direction. A mixture can also change how often an existing mode occurs. At the same mean shift on Dog→Cat, reweighting predicted 77.0% cat-positive outputs, close to the 77.3% observed. Uniform translation predicted 69.5%. Reweighting reduced Wasserstein-1 error on the RGB CLIP score distribution by 69% (C2).
  3. How we test the framework. Greedy changes selected coordinates while holding conditional residuals fixed, so we can measure what each edit preserves (§6.2). C1 tests 36 subject LoRAs. Four photos per subject predict the relative ordering of outputs from six text-to-image LoRAs. For the other 30, which are image-conditioned, the photos help predict the center and the base model predicts the spread in DINO space. In three Dog→Cat and three eyeglasses LoRAs, we intervene during denoising to measure the contribution of a pretrained task direction (C3).

C1. The effect of training photos on generated images (Section 6.3, Appendix K.1)

We appreciate the request to test our predictions in ordinary subject personalization. Four training photos predict which way a subject's outputs move, and the base generator predicts much of their spread. Consider the two backpacks in Figure 5. We trained one LoRA on four photos of backpack A and one on four photos of backpack B, with the same prompts, initialization and training randomness. The direction between the two sets of photo features in a frozen DINO encoder gives a prediction before generation: under matched noise, A's output should lie farther toward A than B's.

We tested this across three same-class pairs covering backpacks, cats, and sneakers. The training photos correctly predicted the ordering of 144/144 pairs (3 subject pairs × 2 new prompts × 24 shared noise draws). We also scored the 288 outputs individually against reference centroids computed from eight unused real photos per subject; nearest-centroid retrieval in DINO space identified the correct subject in 244/288 adapted outputs versus 134/288 outputs from the shared base generator. Uniform guessing among six subjects gives 1/6 accuracy. Guessing between the two subjects of a known class gives 1/2.

Across 30 additional image-conditioned LoRAs, we held out one subject class at a time. Using four photos to calibrate the center reduced mean DINO endpoint error from 0.2054 to 0.1983 (3.5%), improving 14 of 15 classes over the historical shift alone. With that center fixed, keeping the base spread gave a mean squared energy distance of 0.0726, versus 0.1280 when the four photos set a single scale for the spread. Keeping the base spread gave lower error in all 15 classes.

C2. Predicting output frequencies beyond the mean shift (Section 6.1, Section 6.3, Appendix K.2)

Thank you for asking what a mixture can predict beyond a change in the mean. On Dog→Cat, changing the mixture weights predicts how often new outputs look more like cats than dogs. The mean cat-minus-dog CLIP score could rise because all outputs shift together or because an existing, more cat-like group becomes more common. Even when the two changes give the same mean, they can put different fractions of images above the classification threshold.

We model each unshared coordinate with a mixture of Gaussians (MoG). This lets us vary the locations and frequencies of modes separately and keeps the score analytic under Gaussian noising. Prior work models multiple modes within low-dimensional diffusion subspaces [1]. Shah et al. show when the DDPM objective can recover the parameters of spherical Gaussian mixtures [2]. On our own factor coordinates, BIC-selected mixtures improved test log-density over a single Gaussian by 0.163 nats on the pooled AFHQ dog/cat axis and by 0.037 nats per coordinate across four CelebA attributes, while automatically reducing to a single Gaussian on unimodal axes.

We calibrated both forecasts to the same mean shift on two historical Dog→Cat LoRAs and froze them before testing on 128 new inputs and three new LoRA seeds. Uniform translation predicted that CLIP would favor cat on 69.5% of outputs. Reweighting predicted 77.0%, close to the 77.3% observed. Reweighting also gave lower Wasserstein-1 error on the RGB scores than a global shift and scale fitted directly to each test adapter's outputs (0.0073 versus 0.0156). With calibration frozen across three further prompts, it beat uniform translation on RGB scores in all nine prompt–seed comparisons. Appendix K.2 gives the comparisons on both RGB and latent scores.

C3. The contribution of a pretrained task direction during denoising (§6.1, Appendix K.2)

We appreciate the emphasis on testing the mechanism in a trained denoiser. Keeping the part of the LoRA update aligned with the base model's task direction produces a stronger semantic change than keeping an equally large orthogonal part. At each denoising step, we query the base U-Net on the same noisy image under a cat prompt and a dog prompt. The difference defines the task direction for that step. We project the trained LoRA's update onto it and compare the final images under the two updates.

Across three independently trained Dog$\to$Cat LoRAs and 12 dog inputs starting from shared noise under a neutral prompt, projecting the LoRA's guided noise-prediction update onto this single base direction captured only 5.1 ± 1.5% of the step update energy. Even so, relative to the norm-matched orthogonal remainder, retaining only this rank-one projection increased the final cat-minus-dog CLIP score by +0.0188 ± 0.0053 and the independent real-image DINO score by +0.1089 ± 0.0417 across all three training seeds. Relative to the base output, the projection retained 55% of the full LoRA’s mean CLIP-margin gain (0.0201/0.0365) and 43% of its mean DINO gain (0.1076/0.2497). Performing the same stepwise intervention along the base U-Net's eyeglasses direction on 24 face inputs likewise produced higher CLIP and DINO attribute scores than the norm-matched orthogonal control in every seed.

References

[1] Ruofeng Yang, Yongcan Li, Bo Jiang, Cheng Chen, and Shuai Li. Multi-Subspace Multi-Modal Modeling for Diffusion Models: Estimation, Convergence and Mixture of Experts. ICLR, 2026.

[2] Kulin Shah, Sitan Chen, and Adam Klivans. Learning Mixtures of Gaussians Using the DDPM Objective. NeurIPS, 2023.

[3] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the Latent Space of GANs for Semantic Face Editing. CVPR, 2020.

Dear Area Chair,

Thank you for overseeing the review of our paper. We summarize the paper’s contribution, the evidence added during discussion, and the points still awaiting reviewer assessment.

Our central insight is that few-shot diffusion personalization can retain much of the variation learned during pretraining while recalibrating a few task-relevant factors. We develop SUDA-MoG as a tractable probabilistic model to analyze this insight. Its shared–unshared factorization and factor-wise Gaussian mixtures let us distinguish changes in centers, spread, and mode frequencies and derive the corresponding denoising scores. Under stated assumptions, the theory characterizes the sample cost of calibrating unshared parameters and how coupling affects the initial shared gradient. Greedy provides controlled interventions on these factors, and experiments on independently trained adapters then test the resulting predictions about output statistics and the contribution of pretrained task directions.

  1. The new text-to-image study tests the framework’s predictions under diverse prompts.

Following Reviewer 5JBa’s November 17 request, we evaluated DreamBooth-LoRA and Custom Diffusion on all 30 DreamBooth subjects and 25 official prompts per subject. Both methods used four training photos and 300 training steps, with four matched noise seeds per prompt and 3,000 outputs per method. We reported this evaluation in our discussion reply that day, extending the subject-personalization experiments in the manuscript.

Without attribute labels, the training-photo direction correctly ordered 98.4% of 3,700 same-class output comparisons paired by prompt and noise seed for LoRA and 67.6% for Custom Diffusion. Holding out one class at a time, adding four subject photos reduced overall mean DINO center error by 15.3% for LoRA and 1.7% for Custom Diffusion relative to the historical forecast learned from the other 14 classes, improving 14/15 and 15/15 classes, respectively. At the same predicted center, retaining the base spread yielded lower squared energy distance in DINO space than estimating spread from four photos in 13/15 classes for LoRA and 15/15 classes for Custom Diffusion.

  1. The Dog→Cat forecast tests what mixture weights add beyond a mean shift.

We calibrated two forecasts on historical adapters and fixed them before testing three new LoRAs. Under a neutral prompt and at the same mean shift, mixture reweighting predicted 77.0% cat-positive outputs, uniform translation predicted 69.5%, and the observed rate was 77.3%. Under three additional prompts, reweighting gave lower RGB score-distribution error than translation in all nine prompt–seed comparisons (§6.3, Appendix K.2).

A separate denoising intervention examines a pretrained task direction’s functional contribution. On Dog→Cat and eyeglasses LoRAs, retaining the update along that direction produced greater CLIP and DINO task-score gains than an equal-norm orthogonal update (§6.1, Appendix K.2).

  1. The revisions address factor discovery and the specific presentation questions.

Theorem J.1 extends the fixed-map bound to selecting both the map and the shared–unshared roles from the same target images, under uniform regularity assumptions. It accounts for the complexity of the map family and candidate role assignments while retaining the selected model’s approximation error. Appendix J also quantifies the effects of support-set direction discovery and decomposes squared transport error into shape, center, and width terms.

We also defined the factor-distribution notation, summarized Theorem 5.1’s assumptions in the main text, added variability and image-fidelity measurements, and clarified cross-table metrics and accuracy counts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.