- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
Tbe paper studies the effects of class geometry to CFG generation. It writes the oracle CFG correction using inter class separation and intra class variance, relating noisy class posterior to CFG correction. Based on this understanding, it proposes Prototype Regularization (PR) that regularizes the class geometry during diffusion model training by a cosine prototype classifier. PR demonstrates that some of the generation metrics such as gFID and IS are improved against the baseline model.
Critical strengths
- The experiments are thorough, including pseudo label and video datasets.
- PR is a cheap method to implement.
- The paper is well-written and easy to follow.
- The appendix contains useful information.
Critical weaknesses
- The theoretical novelty seems to be compromised given the previous works on GMM score difference [1-4], class-conditional entropy [5] and posterior concentration under larger mean separation [6,7]. Zhong et al, as cited, already claims the connection between intra-class compactness as well as semantic separability and diffusion quality. The paper rederives these results without attribution and presents their conclusions as new (e.g., ...revealing a factor that remains largely overlooked).
- The practical novelty is undermined given the existing attempts on improving class separability during diffusion model training, such as in [8,9]. [8] also reports ImageNet CFG gain. I didn't find comparisons or discussions to those works, given that they are also shaping the latent class geometry during diffusion training.
- While there is experiment on text-conditioned generation, it is not the same as the real continuous conditioning signal, and can largely reduce to the class label. PR needs clean partition of the data in the given dataset, while modern diffusion models focus on multi-object scenes (e.g. COCO) and attribute-level prompts (e.g. GenEval) that have no single prototype.
- I don't understand why VA-VAE achieves around 2.5 gFID in the paper while in the original Table 1 it can achieve 2.1 gFID at 64 epoch. I assume this is due to the usage of DINOv3 instead of DINOv2 in the original paper, or some training settings. If this is the case, please also include the original VA-VAE as a baseline.
- According to Table 1, PR sacrifices the detail and spatial structure for the gain of overall quality, as demonstrated by the increase in sFID and decrease in gFID. This brings concern that class separation is not improving the overall model performance but trading the CFG gains with details. It can be undesirable in the generative modeling case, especially when local textures/structures/details are wanted (e.g., photorealistic image generation).
[1] Theoretical Insights for Diffusion Guidance: A Case Study for Gaussian Mixture Models. ICML 2024
[2] What Does Guidance Do? A Fine-Grained Analysis in a Simple Setting. NeurIPS 2024.
[3] Overshoot and Shrinkage in Classifier-Free Guidance: From Theory to Practice. ICLR 2026.
[4] Emergence of Distortions in High-Dimensional Guided Diffusion Models. arXiv 2026.
[5] The Entropic Signature of Class Speciation in Diffusion Models. ICML 2026.
[6] Spontaneous Symmetry Breaking in Generative Diffusion Models. NeurIPS 2023.
[7] Dynamical Regimes of Diffusion Models. Nature Communications 2024.
[8] No Alignment Needed for Generation: Learning Linearly Separable Representations in Diffusion Models. arXiv 2025.
[9] Prototype-Guided Diffusion: Visual Conditioning without External Memory. arXiv 2025.
Questions for authors
Please see Critical Weaknesses.
6 comments
Response to Reviewer RGod (2/2)
Q: Does PR trade spatial detail for CFG gains (gFID vs. sFID)?
We thank the reviewer for the careful reading. We respectfully argue that Tab. 1 does not show a systematic loss of local detail, for three reasons.
(1) The sFID change is marginal. PR raises sFID by only 0.19 over the LDM baseline (4.68 → 4.87), which lies within the range of the VA-VAE baselines (4.83 to 4.95). PR also preserves reconstruction fidelity (i.e., comparable rFID/PSNR/LPIPS to LDM) and improves Recall (0.50 → 0.59), suggesting no loss of local detail or diversity.
(2) A slight sFID increase alongside gFID improvement is commonly observed. It is a trade-off similar in spirit to precision vs. recall. For instance, REPA reports that as training proceeds (200 → 800 epochs), guided gFID keeps decreasing while sFID slowly rises from 4.49 to 4.70 (+0.21).
(3) The sFID gap closes under standard 250-step sampling. Our default uses 50 Euler steps. With the common 250-step setting, all methods reach nearly identical sFID (4.50 to 4.57), while PR keeps the best gFID (2.21).
| Checkpoint | gFID↓ (50 steps) | gFID↓ (250 steps) | ΔgFID | sFID↓ (50 steps) | sFID↓ (250 steps) | ΔsFID |
|---|---|---|---|---|---|---|
| LDM | 3.31 | 3.13 | -0.18 | 4.68 | 4.54 | -0.14 |
| VA-VAE (DINOv3-L) | 2.44 | 2.57 | +0.13 | 4.83 | 4.57 | -0.26 |
| VA-VAE (DINOv2-L, official) | 2.39 | 2.54 | +0.15 | 4.95 | 4.50 | -0.45 |
| LDM + PR (Ours) | 2.28 | 2.21 | -0.07 | 4.87 | 4.55 | -0.32 |
We hope our responses address your concerns. Please let us know if there are any further questions.
Thank you for your detailed rebuttal and the new results, especially regarding the clarifications to related works. I think that part resolves my concerns. I still have some reservations regarding the continuous-text conditional part as in the new results, PR needs pseudo-labels to work and there is no validation of how would that extend to other datasets (e.g. GenEval as I mentioned before), which means that it might still rely on clear prototypes in the dataset. However, I think my overall concerns are largely resolved and thus I raise my score to 3.
Response to Reviewer RGod (1/2)
Dear Reviewer RGod,
Thank you for your feedback and for taking the time to engage with our work. We address your questions below.
We thank the reviewer for bringing up these references, and have revised our attribution accordingly (revision details below). We first clarify the key difference: [1-7] treat the space in which diffusion operates as fixed. Specifically, [1,2,5,7] characterize the sampling dynamics, and [3,4,6] further adjust the sampling procedure. In comparison, our work studies the inverse question: how the latent space should be shaped so that guidance works better.
We acknowledge that these works share analytical tools with ours: [1-4] analyze guidance with a Gaussian mixture assumption; [5] uses the class posterior entropy as a diagnostic of when classes are decided during sampling; [6,7] characterize the critical time at which denoising commits to a class. We agree these connections deserve explicit credit at the point of use.
Regarding the work by Zhong et al., semantic separability is indeed one of the latent properties they explore. More specifically, they conduct a systematic correlational analysis of latent diffusability, primarily without guidance: they relate a range of latent properties to gFID across tokenizers trained with existing regularizers (largely by aligning to pretrained encoders), without varying any property in isolation.
In contrast to both lines of work, we directly intervene in the class geometry of the latent space to improve guided generation. We show that the per-sample gradient of a noisy-posterior objective with respect to the clean latent is proportional to the oracle CFG correction, making it a principled objective for shaping the latent. We introduce Prototype Regularization (PR) as a lightweight, tractable surrogate of this objective, which effectively shapes the class geometry and improves guided generation.
Revisions. We removed the "largely overlooked" phrasing, added in-line citations throughout Sec. 2, and expanded the related work to cover [1-9] and additional references.
We thank the reviewer for these references, which are now discussed in the related work section of the updated PDF. However, we respectfully note that neither of them shapes the latent class geometry:
PR instead shapes the latent distribution learned by the diffusion model. Like REPA, [8] acts on the denoiser and can be combined with a PR-shaped latent.
Please see the General Remarks (T2I results on MS-COCO).
The 2.1 gFID at 64 epochs reported in the VA-VAE paper is obtained with a stage-1 VAE trained for 130 epochs and 250-step Euler sampling. Our controlled experiments instead use a fixed budget for every method (20 epochs for stage 1, 80 epochs for stage 2) and 50 Euler sampling steps by default (see our response to the next question for the comparison across sampling steps).
As requested, we reproduced the original VA-VAE baseline with DINOv2-L. We followed the original implementation exactly and used the same training budget. Compared with the DINOv3-L variant (with CFG), it has slightly better gFID, IS, and Recall, but worse reconstruction quality, sFID, and Precision. It still falls behind LDM + PR.