Your Few-shot Diffusion Models Secretly Localize to a Small Subspace: An Explanatory Framework in the Latent Space

Few-shot diffusion personalization poses a puzzle: how can a handful of images induce subject or attribute changes without relearning a high-dimensional distribution? We propose SUDA-MoG, a shared–unshared framework that models adaptation as selective recalibration of factor-wise Mixture-of-Gaussians laws in frozen pretrained latents. Shared factors inherit source statistics, while unshared factors are calibrated from target examples. Under an ideally decoupled shared–unshared factorization, we derive a generative-error bound whose target-sample term depends on the free active score parameters under stated regularity conditions. For coupled scores, we bound the initial shared gradient through shared-domain mismatch and cross-block coupling. We instantiate the framework with a semantically guided Greedy construction that combines active-block selection with conditional residual completion. Across evaluated tasks, SUDA-MoG achieves target alignment comparable to DreamBooth-LoRA and Custom Diffusion. Matched interventions on DreamBooth-LoRA outputs show that task-aligned coordinates carry part of the adapter's task response under an independent latent evaluator. These results characterize a task-relative functional footprint of few-shot adaptation, connecting inherited structure, selective calibration, and observable task effects.