- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SUDA-MoG, a framework for interpreting few-shot diffusion adaptation as a low-dimensional recalibration of latent probability laws. The main hypothesis is that a pretrained latent representation can be divided into shared factors, whose source-domain statistics remain unchanged, and unshared or active factors, whose Mixture-of-Gaussians parameters are updated using a small target support set. The encoder, decoder, and latent factor map are otherwise frozen.
The theoretical analysis provides two main results. Under an exactly factorized score model, the authors derive a finite-sample generative-error bound whose explicit estimation term depends on the number of active score parameters rather than the full latent dimension. Under a coupled score model, they bound the initial target gradient on shared parameters using a shared-domain mismatch term and a cross-block Hessian coupling term. The practical method, called Greedy, uses supervised directions to construct active coordinates and models the selected factor distributions with MoGs.
Experiments cover Dog-to-Cat transfer, facial-attribute editing, synthetic score-coupling tests, and interventions on DreamBooth-LoRA outputs. The strongest result is a matched intervention in which transplanting one task-aligned coordinate from a LoRA output into the corresponding base-model latent recovers 42.68% of the LoRA-induced change measured by an external Dog-versus-Cat classifier.
The paper presents an interesting descriptive perspective, but its central claim is significantly stronger than what the theory and experiments establish. The theory assumes that the relevant factorization and invariant marginals are already known, while most experiments evaluate supervised latent editing rather than explaining how real diffusion personalization methods learn.
Critical strengths
The distinction between inherited and recalibrated latent statistics provides a potentially useful vocabulary for analyzing personalization. Unlike work that only observes low-rank parameter updates, this paper attempts to describe which parts of the generated distribution change. The separation between coordinate extractors, external evaluators, active factors, and residual coordinates is also conceptually careful.
This is consequential because understanding why few-shot personalization works is an important problem. A validated account of which pretrained statistics are reused and which are modified could inform more efficient adaptation procedures.
The intervention on DreamBooth-LoRA outputs is stronger than a conventional probe experiment. The authors transplant the LoRA output’s task coordinate into the paired base output while retaining the remaining base coordinates. They evaluate the result using a separately trained classifier and include shuffled-label and rotation controls. These choices provide some evidence that the discovered coordinate functionally contributes to the measured adapter response.
However, this experiment supports only a narrow conclusion: one supervised coordinate captures part of one classifier’s response in a Dog-to-Cat task. It does not demonstrate that LoRA adaptation generally operates by recalibrating a low-dimensional MoG distribution.
Critical weaknesses
The principal theoretical result conditions on a fixed factor map, a correct shared–unshared partition, invariant shared marginals, known source distributions, regular score classes, and a stable sampler. Once an oracle has identified a low-dimensional active parameterization and declared the remaining target statistics invariant, it is unsurprising that estimating only the active parameters produces a complexity term depending on their number.
Crucially, the analysis omits the cost of discovering this representation. In the practical experiments, the chart, supervised task directions, residual statistics, and factor laws are estimated from the same limited support data. Therefore, the theorem does not analyze the procedure that supposedly provides the method’s few-shot advantage.
The favorable estimation term also appears alongside uncontrolled approximation terms covering factorization error, MoG misspecification, partition error, nonlinear coupling, VAE reconstruction, positive-noise truncation, and sampler discrepancy. The paper does not estimate these quantities. They may dominate the stated active-parameter term, leaving the bound without meaningful predictive content in the evaluated settings.
The shared-factor stability result is similarly weak. In the additive construction, score blocks are parameterized separately. Consequently, the cross-parameter Hessian is zero by construction. If the shared target marginal is additionally assumed to equal the source marginal, the shared gradient is expected to vanish. The coupled extension is essentially a generic smoothness argument based on the cross-Hessian. These results formalize consequences of the assumed decomposition rather than showing that learned diffusion adapters possess such a decomposition.
Most experiments evaluate a supervised latent editor. For example, facial-attribute directions are learned using explicit labels, and the active rank is chosen based on the number of targeted attributes. This setting supplies the algorithm with the semantic factors that ordinary subject-personalization methods must discover from a few unlabeled images.
The main comparison is also inadequate. On Dog-to-Cat, SUDA-MoG obtains CLIP-T 0.265, compared with 0.264 for DreamBooth-LoRA and Custom Diffusion. This difference is negligible, and no statistical comparison is reported. More importantly, SUDA-MoG appears to perform image-conditioned latent transfer while the baselines are normally intended for text-conditioned generation across novel prompts. A target-only CLIP score cannot establish comparable personalization quality without evaluating identity preservation, prompt generalization, diversity, carrier preservation, and memorization.
The CelebA effects are statistically positive but extremely small, with CLIP-T improvements on the order of 0.0016–0.0028. The paper provides no human evaluation establishing that these changes are perceptually meaningful. Evaluating drift on one untargeted binary attribute is also insufficient to support a broad claim of selective preservation.
Finally, the 42.68% LoRA recovery result is tied to one coordinate, one binary category transfer, and one class-discriminative evaluator. Recovering part of a classifier score does not mean that the same fraction of the adapter’s full generative behavior has been recovered. The paper does not evaluate texture, geometry, diversity, prompt response, or distributional correspondence. It also lacks comparisons with simpler alternatives such as LDA without MoG modeling, the mean paired base-to-LoRA difference, PCA of paired differences, partial least squares, or reduced-rank regression. The result may therefore reflect ordinary supervised linear separability rather than the proposed latent-law mechanism.
Questions for authors
Is SUDA-MoG intended as a competitive personalization method, a post hoc model of trained adapters, or a supervised latent-editing framework? The paper currently moves among these interpretations.
Can the authors provide a theoretical result that includes data-dependent discovery of the factor map and shared–unshared partition from the same few-shot support set?
Can the authors empirically estimate the approximation and VAE terms in the main bound? Without this information, how can readers determine whether the favorable active-dimensional term is consequential?
Does the fitted MoG predict the distribution of LoRA-induced coordinate changes across held-out prompts, carriers, and random seeds, rather than merely recovering one classifier response?
How does the reported intervention compare with simpler baselines, including LDA without MoG modeling, paired-difference PCA, a mean paired-difference direction, and reduced-rank regression?
Are all methods in the Dog-to-Cat comparison evaluated under matched image-conditioning, prompting, preservation, and sampling protocols?
POST REBUTTAL COMMENTS
The authors have addressed most of the comments. So, I raise my score to weak reject.
6 comments
W4: Shared-factor stability under invariance and coupling
We appreciate your question about what the stability result predicts once the factors interact. At the same unshared displacement, stronger coupling produces a larger shared gradient. In the factorized theorem the shared gradient vanishes under shared marginal invariance and source stationarity. The coupled bound predicts how shared mismatch and cross-block curvature enter when those conditions are relaxed. Across 64 fitted joint score models, the high-coupling models had 2.57 times the shared-parameter gradient of the low-coupling models at matched displacement. In a trained SD1.5 LoRA, the change in cross-direction curvature predicted the orthogonal loss-gradient shift at a larger displacement, lowering RMSE from 0.00164 under the unchanged base response to 0.00073.
A separate experiment tests the functional role of a pretrained task direction during sampling. We use three independently trained rank-16 Dog→Cat LoRAs and the same 12 input dogs. At every DDIM step, we decompose the LoRA-minus-base noise prediction along the base U-Net’s Cat-versus-Dog score direction. Image, initial noise and prompt are identical, and the orthogonal control is matched to the task component’s norm for every image and step. Retaining the task component raises the final RGB target margin by 0.0188 ± 0.0053 and an independent real-image DINO task readout by 0.1089 ± 0.0417 relative to that control across the three seeds. Paired-base LPIPS differs by only 0.0060 ± 0.0044. This intervention measures the contribution of the task-aligned update during sampling in the trained denoiser. Blue revisions: §§5.2 and 6.1 and Appendix K.
W5 & Q1: SUDA-MoG as a framework for explaining diffusion personalization
Thank you for asking us to state the primary purpose of SUDA-MoG. SUDA-MoG is an analytical framework for explaining and predicting which output statistics change during few-shot diffusion fine-tuning. We use Greedy as a controlled intervention to isolate the effects of shared and unshared factors. Independently trained LoRAs let us test the resulting predictions on new outputs.
Figure 5 isolates the effect of the training photos in ordinary text-to-image personalization. We trained six LoRAs in three same-class pairs, changing only the four training photos within each pair, then generated outputs from matched Gaussian noise under two new prompts. The difference between the four-photo DINO centroids predicted the ordering of 144/144 paired outputs. On the harder test of identifying each image individually, nearest-centroid retrieval against eight unused real photos gave 244/288 correct, versus 134/288 for base.
Greedy uses supervised directions, as in InterFaceGAN [1]. Along a readable direction, the mixture lets us ask whether shifting its components or changing their frequencies better predicts the distribution of new LoRA outputs. Selecting component count on separate fitting images improved test log density over one Gaussian by 0.163 nats on AFHQ dog/cat and by 0.037 nats per coordinate across four CelebA attributes. Two of the four CelebA attributes selected a single Gaussian. Our reply to Q4 gives the held-out frequency forecasts, and our reply to W4 gives the denoising intervention. Blue revisions: §§2, 6.1, and 6.3 and Appendices J–K.
W6 & Q6: Matched evaluation protocols for Dog→Cat
We appreciate your concern about how the choice of inputs and prompts affects this comparison. Table 2 matches the ten target cats, 40 input dogs, prompts and sampling noise across all three methods and reports both target gain and input fidelity. Both trained adapters receive 300 updates, while Greedy uses 192 additional source dogs to fit its factor map. Greedy and DreamBooth-LoRA produce similar LPIPS distances from the paired base image, 0.254 and 0.247 respectively. Greedy gives a larger mean cat-prompt gain, +0.0392 versus +0.0229 for LoRA, while LoRA preserves more of the input dog’s appearance, with DINO similarity of 0.506 versus 0.420 for Greedy. Custom Diffusion gives the largest mean prompt gain at +0.0410 and the largest LPIPS distance from the base at 0.442. Figure 5 separately tests subject identity under new prompts in text-to-image generation. Blue revisions: §6.2 and Table 2.
W7: Target-attribute changes and drift on other CelebA attributes
Thank you for questioning what a small CLIP-T gain tells us about a facial attribute. Independent attribute classifiers detect the intended local changes on the same faces. Figure 4(c) compares a whole-face embedding with a short attribute phrase. Opening the mouth slightly or adding bangs changes only part of that image. We therefore also measured each attribute with an independent logistic classifier using histograms of oriented gradients (HOG). Mouth opening raised CLIP-T by 0.00160 and the HOG classifier probability by 0.0530. The corresponding gains were 0.00246 and 0.0554 for bangs, and 0.00283 and 0.0619 for smiling (Appendix H.4). Figure 4(a) illustrates separate eyeglasses and haircut edits.
On 64 new faces, we tested whether the edit consistently moved the intended attributes. We measured changes in 19 CLIP attribute scores in units of each attribute’s class gap. With correct labels, the mean signed change was +0.063 on the three targets and +0.001 on the 16 untargeted attributes. Mean absolute changes were 0.175 and 0.182, respectively. The corresponding untargeted change under shuffled support labels was 0.170. Correct labels made the target changes more consistent across support draws. On 56/64 faces, at least two target attributes each moved in the intended direction in at least six of ten draws, compared with 24/64 under shuffled labels. Blue revisions: §6.2, Figure 4 and Appendices H.4 and K.3.
W8 & Q4: Predicting LoRA output distributions across inputs, prompts and seeds
Thank you for asking what we can predict before seeing the adapted outputs. On new Dog→Cat LoRAs, mixture reweighting predicts the fraction of cat-positive outputs more accurately than a uniform shift with the same mean. Under the neutral prompt the two forecasts were 77.0% and 69.5%, respectively. The observed rate across three newly trained LoRAs was 77.3%.
We calibrated both forecasts on 64 inputs using two historical LoRAs and froze them before testing on 64 new dogs and 64 new cats. Translation moves every score by the same amount. Reweighting puts more probability on the more cat-like component, so the two forecasts can agree on the mean and still predict different fractions above the cat-positive threshold (Proposition J.3). We also compared the full score distributions on two readouts.
| Measurement on the same 128 outputs | Translation W1 | Component reallocation W1 |
|---|---|---|
| RGB cat-minus-dog CLIP score | 0.0233 ± 0.0007 | 0.0073 ± 0.0008 |
| Frozen latent task coordinate (evaluated in Q5) | 0.536 ± 0.139 | 0.396 ± 0.134 |
Values are means ± sample SD across the three test seeds. Reweighting reduced distribution error on both readouts in every seed. Even allowing a separate historical shift for dogs and for cats gave a higher mean on the pooled RGB scores (0.0246 versus 0.0073 for reweighting) at the same overall mean shift.
With calibration frozen across three additional prompts, reallocation beat uniform translation in 9/9 RGB comparisons (mean 0.0102 versus 0.0247) and 8/9 latent comparisons (0.685 versus 0.719). Appendix K.2 also reports the affine-scale comparison. Blue revisions: §6.3 and Appendices J–K.
Thank you for your careful reading and for distinguishing between discovering a factor and fitting its statistics once it is known. We respond to your concerns and questions below.
W1, W2 & Q2: Factor-map assumptions and the cost of discovery
Theorem 5.1 covers calibration on a fixed map, which is intended to highlight our core rationale: the non-shared components are what truly play a role during the fine-tuning process. We extend it to Theorem J.1, which adds the cost of learning the map and factor roles from the same target images. At resolution 1/n, the estimation term counts the free unshared parameters, the log covering number of the map family and the log number of candidate partitions. A covering number at most (An)q adds at most qlog(An) to the numerator. This extension requires uniform control of the DSM loss and score error over candidate maps. Each selected map and partition has its own population approximation error.
For a mean shift, the cost of discovery has a simple interpretation. The reduction in squared transport error equals the squared true shift captured minus the squared sampling noise captured. When the true shift is zero and the noise is isotropic in D dimensions, the observed mean-difference direction captures D times as much squared noise in expectation as an independently fixed direction, although both updates have rank one (§5.1 and Appendix J.1).
We tested how this trade-off changes with sample size across eight CelebA transfers. With 16 supports, an axis learned on 1,536 separate labeled faces reduced squared mean mismatch by 40.1%, while fitting the axis on the 16 supports increased mismatch by 169.8%. With 128 supports, the support-fitted axis reduced mismatch by 54.3%, compared with 40.5% for the independently learned axis. To test factor discovery in the nonlinear Greedy editor, we used the same 32 supports to fit both the map and the factor distributions. Correct labels steered at least two target attributes on 56/64 new faces, compared with 24/64 after permuting the labels. Blue revisions: §§5.1 and 6.2 and Appendix J.
W3 & Q3: Estimation, approximation and sampling error
Thank you for asking how large the approximation and reconstruction errors are in practice. With four target images, estimating the center and width contributes over 15 times as much squared error as the shape residual on the tested factors. We fixed the factor and mixture shape and compared inheriting the source width with estimating it from the same supports. Across four CelebA factors in both transfer directions, 256 independent test images per direction give the following decomposition of one-dimensional squared Wasserstein error in squared source-SD units.
At four supports, the fitted center and width contribute 0.261 + 0.157 = 0.418, compared with 0.027 for shape. Fitting width alone adds 0.109 over inheriting it, about four times the shape residual. At 32 supports it instead reduces total error by 0.029. The discrepancy between the 16-step sampler and the fitted law was below 0.0001 in all four settings.
We then tested whether inheriting the spread also helped predict the outputs of trained subject LoRAs. The observed DINO radius was 0.764, close to the base prediction of 0.778 and larger than the four-photo prediction of 0.493. Energy distance favored the base spread in all 15 classes. Empirical squared transport between two sets of 30 DINO features favored the narrower set (0.635 versus 0.837), a finite-sample effect also seen when splitting outputs from the same base generator (Appendix K.1).
To measure the reconstruction term in image space, we passed 32 real target faces through our frozen deep compression autoencoder (DC-AE). The RGB transport distance between the original and reconstructed faces was 0.031, about one tenth of the baseline transport distance of 0.297. Editing reduced the latter to 0.283. The fixed-factor comparison above isolates the estimation cost and shows when fitting one additional width parameter helps. Blue revisions: §§5.1 and 6.3 and Appendices J–K.