acceptodds
Under review as a conference paper at ICLR 2027

Learning to Be Guided: Shaping Latent Geometry for Classifier-Free Guidance

Abstract

Classifier-free guidance (CFG) is a standard component of conditional diffusion models, and guided performance has become the de facto measure for state-of-the-art models. In latent diffusion, recent work designs latent spaces that are easier to model. However, under CFG, generation additionally depends on how conditions are arranged in the latent space, revealing a factor that remains largely overlooked. We study this arrangement under discrete class conditions, termed class geometry, and investigate what class geometry allows a latent space to work well with CFG. By analyzing the oracle CFG correction and class distinguishability under noise, we show that greater inter-class separation and smaller intra-class spread are favorable for CFG, and derive a noisy posterior objective whose gradient aligns with the oracle correction. Guided by the analysis, we propose **P**rototype **R**egularization (**PR**), a lightweight surrogate for the intractable noisy posterior, which applies a cosine prototype classifier to spatially pooled latents. Under a controlled training budget (80 epochs), adding PR to an LDM baseline reduces guided gFID on ImageNet from 3.31 to 2.28, surpassing a DINOv3-aligned VA-VAE (2.44) with negligible overhead. Through rich extensions, we show that PR generalizes to various label choices, data modalities, and condition signals. Together, our findings highlight class geometry as a critical factor in guided generation and take a first step toward CFG-compatible representation learning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Latent Diffusion Models, Classifier-Free Guidance, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RGodOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

Tbe paper studies the effects of class geometry to CFG generation. It writes the oracle CFG correction using inter class separation and intra class variance, relating noisy class posterior to CFG correction. Based on this understanding, it proposes Prototype Regularization (PR) that regularizes the class geometry during diffusion model training by a cosine prototype classifier. PR demonstrates that some of the generation metrics such as gFID and IS are improved against the baseline model.

Critical strengths
  1. The experiments are thorough, including pseudo label and video datasets.
  2. PR is a cheap method to implement.
  3. The paper is well-written and easy to follow.
  4. The appendix contains useful information.
Critical weaknesses
  1. The theoretical novelty seems to be compromised given the previous works on GMM score difference [1-4], class-conditional entropy [5] and posterior concentration under larger mean separation [6,7]. Zhong et al, as cited, already claims the connection between intra-class compactness as well as semantic separability and diffusion quality. The paper rederives these results without attribution and presents their conclusions as new (e.g., ...revealing a factor that remains largely overlooked).
  2. The practical novelty is undermined given the existing attempts on improving class separability during diffusion model training, such as in [8,9]. [8] also reports ImageNet CFG gain. I didn't find comparisons or discussions to those works, given that they are also shaping the latent class geometry during diffusion training.
  3. While there is experiment on text-conditioned generation, it is not the same as the real continuous conditioning signal, and can largely reduce to the class label. PR needs clean partition of the data in the given dataset, while modern diffusion models focus on multi-object scenes (e.g. COCO) and attribute-level prompts (e.g. GenEval) that have no single prototype.
  4. I don't understand why VA-VAE achieves around 2.5 gFID in the paper while in the original Table 1 it can achieve 2.1 gFID at 64 epoch. I assume this is due to the usage of DINOv3 instead of DINOv2 in the original paper, or some training settings. If this is the case, please also include the original VA-VAE as a baseline.
  5. According to Table 1, PR sacrifices the detail and spatial structure for the gain of overall quality, as demonstrated by the increase in sFID and decrease in gFID. This brings concern that class separation is not improving the overall model performance but trading the CFG gains with details. It can be undesirable in the generative modeling case, especially when local textures/structures/details are wanted (e.g., photorealistic image generation).

[1] Theoretical Insights for Diffusion Guidance: A Case Study for Gaussian Mixture Models. ICML 2024

[2] What Does Guidance Do? A Fine-Grained Analysis in a Simple Setting. NeurIPS 2024.

[3] Overshoot and Shrinkage in Classifier-Free Guidance: From Theory to Practice. ICLR 2026.

[4] Emergence of Distortions in High-Dimensional Guided Diffusion Models. arXiv 2026.

[5] The Entropic Signature of Class Speciation in Diffusion Models. ICML 2026.

[6] Spontaneous Symmetry Breaking in Generative Diffusion Models. NeurIPS 2023.

[7] Dynamical Regimes of Diffusion Models. Nature Communications 2024.

[8] No Alignment Needed for Generation: Learning Linearly Separable Representations in Diffusion Models. arXiv 2025.

[9] Prototype-Guided Diffusion: Visual Conditioning without External Memory. arXiv 2025.

Questions for authors

Please see Critical Weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer RGod (1/2)

Dear Reviewer RGod,

Thank you for your feedback and for taking the time to engage with our work. We address your questions below.


Q: Relation to [1-7] and Zhong et al.?

We thank the reviewer for bringing up these references, and have revised our attribution accordingly (revision details below). We first clarify the key difference: [1-7] treat the space in which diffusion operates as fixed. Specifically, [1,2,5,7] characterize the sampling dynamics, and [3,4,6] further adjust the sampling procedure. In comparison, our work studies the inverse question: how the latent space should be shaped so that guidance works better.

We acknowledge that these works share analytical tools with ours: [1-4] analyze guidance with a Gaussian mixture assumption; [5] uses the class posterior entropy as a diagnostic of when classes are decided during sampling; [6,7] characterize the critical time at which denoising commits to a class. We agree these connections deserve explicit credit at the point of use.

Regarding the work by Zhong et al., semantic separability is indeed one of the latent properties they explore. More specifically, they conduct a systematic correlational analysis of latent diffusability, primarily without guidance: they relate a range of latent properties to gFID across tokenizers trained with existing regularizers (largely by aligning to pretrained encoders), without varying any property in isolation.

In contrast to both lines of work, we directly intervene in the class geometry of the latent space to improve guided generation. We show that the per-sample gradient of a noisy-posterior objective with respect to the clean latent is proportional to the oracle CFG correction, making it a principled objective for shaping the latent. We introduce Prototype Regularization (PR) as a lightweight, tractable surrogate of this objective, which effectively shapes the class geometry and improves guided generation.

Revisions. We removed the "largely overlooked" phrasing, added in-line citations throughout Sec. 2, and expanded the related work to cover [1-9] and additional references.


Q: How does PR differ from LSEP [8] and PDM [9]?

We thank the reviewer for these references, which are now discussed in the related work section of the updated PDF. However, we respectfully note that neither of them shapes the latent class geometry:

  • [8] adds a linear-probe loss on the DiT's intermediate representations during diffusion training; the VAE latent is frozen.
  • [9] learns prototypes in an auxiliary encoder as conditioning tokens for a pixel-space DDPM; it involves no latent space or CFG.

PR instead shapes the latent distribution learned by the diffusion model. Like REPA, [8] acts on the denoiser and can be combined with a PR-shaped latent.


Q: Does PR work for scenes that have no single prototype?

Please see the General Remarks (T2I results on MS-COCO).


Q: Explanation of performance gap & original VA-VAE baseline?

The 2.1 gFID at 64 epochs reported in the VA-VAE paper is obtained with a stage-1 VAE trained for 130 epochs and 250-step Euler sampling. Our controlled experiments instead use a fixed budget for every method (20 epochs for stage 1, 80 epochs for stage 2) and 50 Euler sampling steps by default (see our response to the next question for the comparison across sampling steps).

As requested, we reproduced the original VA-VAE baseline with DINOv2-L. We followed the original implementation exactly and used the same training budget. Compared with the DINOv3-L variant (with CFG), it has slightly better gFID, IS, and Recall, but worse reconstruction quality, sFID, and Precision. It still falls behind LDM + PR.

MethodrFID↓PSNR↑LPIPS↓Steps/s↑gFID↓sFID↓IS↑Precision↑Recall↑
LDM0.3127.070.1012.163.314.68237.70.860.50
VA-VAE (DINOv3-B)0.2827.090.0991.652.644.94236.00.830.56
VA-VAE (DINOv3-L)0.2827.060.0991.612.444.83242.20.820.56
VA-VAE (DINOv2-L, official)0.3126.880.1031.642.394.95252.50.810.57
LDM + PR (Ours)0.2826.950.1012.132.284.87269.90.810.59
AuthorsOpenReview ↗

Response to Reviewer RGod (2/2)

Q: Does PR trade spatial detail for CFG gains (gFID vs. sFID)?

We thank the reviewer for the careful reading. We respectfully argue that Tab. 1 does not show a systematic loss of local detail, for three reasons.

(1) The sFID change is marginal. PR raises sFID by only 0.19 over the LDM baseline (4.68 → 4.87), which lies within the range of the VA-VAE baselines (4.83 to 4.95). PR also preserves reconstruction fidelity (i.e., comparable rFID/PSNR/LPIPS to LDM) and improves Recall (0.50 → 0.59), suggesting no loss of local detail or diversity.

(2) A slight sFID increase alongside gFID improvement is commonly observed. It is a trade-off similar in spirit to precision vs. recall. For instance, REPA reports that as training proceeds (200 → 800 epochs), guided gFID keeps decreasing while sFID slowly rises from 4.49 to 4.70 (+0.21).

(3) The sFID gap closes under standard 250-step sampling. Our default uses 50 Euler steps. With the common 250-step setting, all methods reach nearly identical sFID (4.50 to 4.57), while PR keeps the best gFID (2.21).

CheckpointgFID↓ (50 steps)gFID↓ (250 steps)ΔgFIDsFID↓ (50 steps)sFID↓ (250 steps)ΔsFID
LDM3.313.13-0.184.684.54-0.14
VA-VAE (DINOv3-L)2.442.57+0.134.834.57-0.26
VA-VAE (DINOv2-L, official)2.392.54+0.154.954.50-0.45
LDM + PR (Ours)2.282.21-0.074.874.55-0.32

We hope our responses address your concerns. Please let us know if there are any further questions.

Thank you for your detailed rebuttal and the new results, especially regarding the clarifications to related works. I think that part resolves my concerns. I still have some reservations regarding the continuous-text conditional part as in the new results, PR needs pseudo-labels to work and there is no validation of how would that extend to other datasets (e.g. GenEval as I mentioned before), which means that it might still rely on clear prototypes in the dataset. However, I think my overall concerns are largely resolved and thus I raise my score to 3.

Reviewer_3SobOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

the paper investigates the problem of representation of latent vectors in image/video generation models when using CFG. The main conflict is from the training of VAE models are in continuous while the guidance of CFG is based on discrete label classes. The authors propose a VAE training with label regularization and this scheme shows that it could perform better than other baselines on guidance.

Critical strengths
  1. The paper analyses the problem well
  2. The problem is sounded and worth investigtion
  3. The proposed method is straight forward.
Critical weaknesses
  1. The theory doesn't explain the empricial result. Eq. 2 makes the correction a posterior-weighted sum over competing classes. As qc→1q_c → 1 all competitor weights vanish and the correction goes to zero, Fig. 1(a) is captioned to this effect. So under their own model, better class separation shrinks the quantity CFG amplifies, yet their claim is that better separation improves guided generation.

  2. Every experiment is categorical by construction. T2I is ImageNet with synthetic Florence-2 captions. Video is UCF-101 (101 action classes). The “unsupervised” variant uses K=1000, matched to ImageNet. The thesis is that categorical priors produce good geometry, and it’s only tested where categorical structure is the data-generating structure. No CoCo, CC12M, LAION-scale T2I

  3. Missing a number of baselines:

  • newest CFG methods: CFG++, CFG-Zero*, Interval guidance
  • RAE, SVG, Aligntok, MAETok, VTP are note compared as baselines
Questions for authors
  1. Under the Gaussian class model, the oracle correction vanishes as the posterior concentrates. What is the mechanism by which PR improves guided generation, and can it be stated within your framework?

  2. Please see the others in the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3Sob (1/2)

Dear Reviewer 3Sob,

Thank you for your feedback and for taking the time to engage with our work. We address your questions below.


Q: If the oracle correction vanishes as the posterior concentrates (qc→1q_c \to 1), how does PR improve guided generation?

The correction in Eq. 2 vanishes only if two conditions hold: (i) classes barely overlap under noise, so that qc→1q_c \to 1, and (ii) the learned correction matches the oracle one, free of approximation and estimation errors. We examine this empirically on LDM and LDM+PR. Concretely, we run 100-step Euler sampling with each model's best guidance scale (ω⋆=2\omega^\star = 2 and 2.52.5). Along these trajectories, we report the empirical posterior E[qc]\mathbb{E}[q_c] with respect to the training latents (i.e., the exact posterior under the empirical training distribution) and the relative correction strength ∥vc−vu∥/∥vu∥\lVert v_c - v_u\rVert / \lVert v_u \rVert.

tt0.050.100.150.200.400.80
CFGoffoffonononon
E[qc]\mathbb{E}[q_c], LDM0.000.010.020.020.030.03
E[qc]\mathbb{E}[q_c], LDM+PR0.040.180.290.360.450.47
∥vc−vu∥/∥vu∥\lVert v_c - v_u \rVert / \lVert v_u \rVert, LDM0.180.140.120.100.070.04
∥vc−vu∥/∥vu∥\lVert v_c - v_u \rVert / \lVert v_u \rVert, LDM+PR0.350.140.080.060.050.04

In practice, E[qc]\mathbb{E}[q_c] stays far from 1 and the correction does not vanish. Moreover, both trends predicted by our analysis hold for the learned models: with better class geometry, LDM+PR attains a higher E[qc]\mathbb{E}[q_c] (Fig. 1(b)) and a correction that is stronger early and decays sooner (Fig. 1(a)). Note that a larger correction is not necessarily better. For both models, the correction is largest in the high-noise regime, yet the best CFG interval skips this range (see our interval sweep in a later response).

PR improves guided generation mainly because it explicitly shapes how conditions are organized in the latent space, i.e., a separated and compact class geometry, which in turn shapes the CFG correction. Our center transplantation experiment (Tab. 2) further supports the direct effect of class geometry: replacing only LDM's class centers with PR's, while keeping all other setup fixed, recovers most of the guided gFID improvement of LDM+PR over LDM (LDM 3.31 → transplant 2.52; LDM+PR 2.28).


Q: Does PR hold beyond categorical data?

Please see the General Remarks (T2I results on MS-COCO).


Q: Does PR work well under CFG variants (e.g., CFG++, CFG-Zero*, interval guidance)?

Yes. We evaluate all three suggested CFG variants. LDM + PR achieves the best guided gFID under every variant, and each method attains its best result with its own tuned guidance interval.

Interval guidance. Our default setting already uses the [0.11, 1.0] guidance interval inherited from the VA-VAE codebase. We sweep the CFG start time for each method and report guided gFID@50K below; each entry uses the best guidance scale ω* re-swept at that start time. At each method's best start time, the ranking is unchanged and the gap between LDM + PR and VA-VAE slightly widens (2.28 vs. 2.44 → 2.19 vs. 2.38).

CFG start time0.000.050.100.11 †0.150.200.250.30
LDM3.963.643.333.313.082.992.922.94
VA-VAE (DINOv3-L)3.712.752.522.442.542.382.392.40
LDM + PR3.692.972.382.282.192.222.292.33

† indicates default setting used in the paper.

We also observe that the best start time is earlier for latents with higher class separability (e.g., by Purity@10 in Tab. 1). This is consistent with our analysis in Sec. 2: when the noisy class posterior concentrates earlier along the trajectory, the CFG correction becomes informative earlier.

CFG++ and CFG-Zero*. Since CFG++ was originally derived under DDIM, we adapt it to the rectified flow framework with v-prediction and t ∈ [0, 1]. For CFG-Zero*, we adopt the optimized scale plus the officially recommended 4% zero-init steps (i.e., 2 of our 50 steps). Neither variant improves upon interval guidance: CFG++ yields at most marginal gains on top of it, CFG-Zero* is consistently worse, and LDM + PR remains the best in every setting.

Checkpointinterval guidance (default)interval guidance (best)CFG++CFG++ †CFG-Zero*CFG-Zero* †
LDM3.312.923.302.914.153.95
VA-VAE (DINOv3-L)2.442.382.492.343.124.45
LDM + PR2.282.192.462.202.963.73

† indicates combined with the best interval guidance config; otherwise, CFG++ and CFG-Zero* are applied over the full trajectory.

AuthorsOpenReview ↗

Response to Reviewer 3Sob (2/2)

Q: How does PR compare with other baselines (e.g., RAE, SVG, AlignTok, MAETok, and VTP)?

We thank the reviewer for suggesting these baselines. They are all representative tokenizers, but a direct comparison of their released models with ours on guided generation would not be fully fair. The main reason is that they differ substantially in latent shape and in pretraining data and supervision (see the table and explanations below).

Nevertheless, we can still compare the latent geometry of these methods (using their official checkpoints) against our controlled models. Specifically, we measure E[qc]\mathbb{E}[q_c]: the posterior of the true class given a noised validation latent, computed exactly from all ImageNet train latents and averaged over samples and tt. A higher value means the classes stay more distinguishable under noise, which benefits guided generation (Fig. 2, Tab. 2). To compensate for differences in latent dimension, we align all latents to the total SNR of an 8192-d latent via the SD3/RAE timestep shift. Results for all tokenizers with public checkpoints (i.e., all but AlignTok) are summarized below.

TokenizerLatent shapeTraining data / additional supervisionE[qc]\mathbb{E}[q_c] (averaged over tt)
LDM32×16×16ImageNet-1K / -0.05
VA-VAE-L32×16×16ImageNet-1K / DINOv3-L features0.16
LDM+PR (ours)32×16×16ImageNet-1K / ImageNet-1K class labels0.33
MAETok-B128×32ImageNet-1K / HOG, DINOv2-L, SigLIP features0.32
VTP-L64×16×16277M image-text pairs / CLIP + SSL0.31
SVG (frozen DINOv3-S)392×16×16LVD-1689M + ImageNet-1K residual encoder0.56
RAE (frozen DINOv2-B)768×16×16LVD-142M0.60

Using only ImageNet class labels, PR raises the same VAE from 0.05 to 0.33, on par with MAETok and VTP, which rely on multiple external features or 277M image-text pairs. Semantic alignment alone does not guarantee this geometry: VA-VAE reaches only 0.16 despite its DINOv3 alignment. RAE and SVG score higher as they directly use frozen DINO features, where strong class geometry emerges as a by-product of large-scale self-supervised pretraining rather than being explicitly targeted as in PR.

In addition, we would like to clarify the feasibility of controlled comparisons with these baselines (i.e., how PR compares against their latent regularizers). RAE and SVG use frozen DINO features as their latent space, so PR is not directly applicable to them. For VTP, such a comparison would require pretraining on its large-scale image-text corpus (277M pairs), which exceeds our available compute. We are working on a controlled comparison with MAETok, which is trained on ImageNet-1K, and will do our best to provide the results before the end of the discussion period.


We hope our responses address your concerns. Please let us know if there are any further questions.

AuthorsOpenReview ↗

Update: controlled comparison with MAETok

Dear Reviewer 3Sob,

Thank you for your valuable feedback and for taking the time to engage with our work.


We have completed a controlled comparison with MAETok on ImageNet-1K.

Setup. Starting from the official MAETok repo, we train MAETok and a variant with PR added, keeping everything else identical: the same training budget (20 epochs for tokenizer and 80 epochs for diffusion, matching our LDM experiments) and the same guided sampling setup. We report reconstruction and guided generation results.

TokenizerrFID↓PSNR↑LPIPS↓gFID↓sFID↓IS↑Precision↑Recall↑
MAETok1.6222.460.2313.625.72252.60.830.51
MAETok + PR1.6922.290.2363.185.99271.50.790.56
LDM (reference)0.2827.070.1013.314.68237.70.860.50
LDM + PR (reference)0.2826.950.1012.284.87269.90.810.59

Bold marks the better result between MAETok and MAETok + PR. LDM rows are included for reference only.

Findings.

(1) PR improves guided generation on MAETok as well. Adding PR lowers gFID from 3.62 to 3.18 and improves IS and Recall, at the cost of a slight degradation in reconstruction. This is the same behavior we observe on LDM.

(2) Reconstruction gap between LDM and MAETok. Under the same budget, our MAETok reproduction reconstructs less accurately than the LDM baseline. We note that it uses a 1D latent with half the dimensionality (128×32 vs. 32×16×16) and is normally trained for longer. This gap is a property of the base tokenizer and is orthogonal to the controlled comparison of PR.

In summary, PR yields consistent guided generation gains on two quite different tokenizers: a 2D convolutional VAE, and a 1D transformer tokenizer. This suggests that PR generalizes across tokenizer architectures and remains effective on top of existing latent regularization.


We hope these updates further address your concerns, and we would be glad to answer any follow-up questions before the end of the discussion period.

Reviewer_7TBFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how the geometric arrangement of classes in a Stage-1 autoencoder's latent space (class geometry) affects conditional diffusion generation under Classifier Free Guidance (CFG). Using a Gaussian class model, the authors show that reallocating intra-class variance to inter-class separation keeps classes distinguishable under higher noise and aligns the negative gradient of a noisy posterior cross-entropy objective with the oracle CFG correction. Motivated by this, they propose Prototype Regularization (PR) which is a cosine classifier on globally average-pooled clean latents. They validate their findings via several experiments.

Critical strengths
  1. Link between Stage-1 latent geometry and CFG effectiveness: Analyzing inter-class vs. intra-class variance allocation under noise cleanly explains why class separability specifically boosts CFG generation.

  2. PR adds a small Stage-1 overhead without needing external foundation models, preserves reconstruction quality by leaving spatial residuals unconstrained.

  3. Center transplantation confirms that global channel-wise center separation drives most of the guided gain.

Critical weaknesses
  1. Although the authors note that PR is a simplified version of their noisy-posterior theory, several jumps between Section 2 and Section 3 are not well explained. First, Section 2 analyzes oracle velocity predictors where unguided generation (ω=1\omega=1) already recovers the exact data distribution regardless of α\alpha. Second, the theory relies on noisy latents (t∈(0,1)t \in (0, 1)), yet when the authors actually apply PR to noisy latents in Table 7, generation fails badly compared to clean ony latents. The paper claims this might be due to over-regularization, but could it be that VAE latents are not standardized to unit variance during Stage-1 training, so the encoder can cheat fixed-variance noise?

  2. The continuous conditioning experiment (section 4.6, Table 5) only tests captioned ImageNet-1K images. Since this dataset still consists of 1000 categories, it naturally favors discrete prototypes and does not test whether pulling pooled latents toward a single prototype hurts complex scenes where unlike ImageNet multiple objects exist in the scene.

Questions for authors
  1. Is PR over discrete data classes similar to some kind of contrastive loss?

  2. How is the intractable posterior E[qc(zt)]\mathbb{E}[q_c(z_t)] in Fig. 2(b) estimated?

  3. Because clean PR only aligns the angles of 32 spatially pooled channels, it is unclear to me why it ends up shifting the full-space Euclidean variance in Fig. 2 so much?

  4. Fixing the CFG start time at t≥0.11t \ge 0.11 for all models (Table 6) may hurt models whose classes become distinguishable later in denoising than PR. Please clarify how this threshold is chosen.

  5. In Table 7, was noise injected into unstandardized Stage-1 latents before or after pooling? and does per-channel standardization before adding noise prevent the constant-channel collapse?

  6. In Table 4, can you also provide video reconstruction metrics such as PSNR/FVD?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7TBF (1/2)

Dear Reviewer 7TBF,

Thank you for your feedback and for taking the time to engage with our work. We address your questions below.


Q: Connection between Section 2 and Section 3 & implementation details

Thank you for the careful reading of our theory, method, and appendix. Regarding the jumps between Sec. 2 and 3, we have updated the PDF to highlight the theory takeaways and the theory-to-practice link, and will further polish them in the next version. We address each of your points below.

Role of the oracle analysis. With oracle velocity predictors, unguided generation indeed recovers the exact data distribution for any α\alpha. In practice, however, the learned correction is never perfect due to approximation and estimation errors (e.g., model capacity, sample size, optimization). Practitioners thus rely on ω>1\omega > 1 to amplify the correction, whose effect depends on how geometry shapes the learned CFG correction. Our oracle analysis aims to reveal how three quantities jointly shape the correction vector: the class means, the intra-class variance, and the noisy class posterior (Eq. 2). This further motivates our analysis of favorable class geometry (Eqs. 3 and 4, Fig. 1).

PR on noisy latents. Our noisy-posterior objective takes the general form Lρ=Et∼ρ[Lpost(t)]\mathcal{L}_ \rho = \mathbb{E}_ {t\sim\rho}[\mathcal{L}_ {\text{post}}(t)] given a time distribution ρ\rho. Fig. 1(c) shows two instantiations: clean-only (t=1t = 1) and noisy latents with t∼U(0,1)t \sim \mathcal{U}(0, 1). We attribute the weaker performance of t∼U(0,1)t \sim \mathcal{U}(0, 1) (Tab. 7, reduced training budget; full-budget results below) mainly to its requirement that latents be class-separable even at very low signal-to-noise ratio (SNR; see details below), which is what we meant by over-regularization (i.e., the highest IS but the worst reconstruction and gFID).

We further test two PR variants with t∼U(tstart,1)t \sim \mathcal{U}(t_{\text{start}}, 1), where tstart∈{0.11,0.15}t_{\text{start}} \in \{0.11, 0.15\} match our default and best CFG start times (more details in our response to a later question). Since CFG is only applied for t>tstartt > t_{\text{start}}, these variants require class separability only where guidance acts. With noise-dominated levels excluded, PR remains effective on noisy latents, improving the gFID of clean-only PR with comparable reconstruction. All variants below use the full training budget.

PR variantrFID↓PSNR↑LPIPS↓gFID↓sFID↓IS↑Precision↑Recall↑
Noisy (t∼U(0,1)t \sim \mathcal{U}(0, 1))0.3625.870.1163.055.74299.80.750.58
Noisy (t∼U(0.11,1)t \sim \mathcal{U}(0.11, 1))0.3026.840.1032.254.94268.30.790.61
Noisy (t∼U(0.15,1)t \sim \mathcal{U}(0.15, 1))0.2926.800.1032.174.98273.30.800.60
Clean (t = 1)0.2826.950.1012.284.87269.90.810.59

SNR perspective: We consider the SNR of the spatially pooled latents, which the PR head operates on. Given 16×1616\times16 spatial tokens and 38.6% of the signal variance retained after pooling (measured on the clean-only PR variant), the pooled SNR is approximately 100 t2/(1−t)2100\,t^2/(1-t)^2. It falls below 1 for t<0.09t < 0.09, and is roughly 1.5 and 3.1 at tstart=0.11t_{\text{start}} = 0.11 and 0.150.15 (both signal-dominated).

Latent standardization in PR. In noisy PR, we did standardize the VAE latent with detached per-channel running statistics before adding rectified-flow noise at t∼ρt \sim \rho by linear interpolation. Thus, the standardized latent roughly stays at unit variance, and the encoder cannot raise the effective SNR by rescaling its latent.


Q: Results under complex multi-object scenes

Please see the General Remarks (T2I results on MS-COCO).


Q: Relation between PR and contrastive losses

PR shares the target geometry of contrastive objectives (compact, well-separated classes), and can be viewed as a proxy-based contrastive loss (i.e., an InfoNCE-style objective whose keys are learnable class prototypes rather than paired samples).

Our geometric preference is originally formulated as minimizing the noisy-posterior objective (Eq. 5). Since this posterior is intractable over a latent space that keeps evolving during training, we instead build a tractable surrogate via a prototype classifier on pooled (clean or noisy) latents. Minimizing this surrogate likewise keeps classes distinguishable under noise corruption and empirically improves guided generation.

AuthorsOpenReview ↗

Response to Reviewer 7TBF (2/2)

Q: How is E[qc(zt)]\mathbb{E}[q_c(z_t)] in Fig. 2(b) estimated?

qc(zt)q_c(z_t) admits a closed form under the empirical distribution of the full ImageNet training set, and E[qc(zt)]\mathbb{E}[q_c(z_t)] averages it over ztz_t.

More concretely, let {(z(i),y(i))}i=1N\{(z^{(i)}, y^{(i)})\}_{i=1}^{N} denote the normalized training latents and their class labels. Under the forward process, their empirical distribution becomes a uniform mixture of NN Gaussians N(αtz(i),σt2I)\mathcal{N}(\alpha_t z^{(i)}, \sigma_t^2 I), so by Bayes' rule the posterior probability that ztz_t originates from a class-cc training latent is given by:

qc(zt)=∑i:y(i)=cN(zt;αtz(i),σt2I)∑i=1NN(zt;αtz(i),σt2I).q_c(z_t) = \frac{\sum_{i: y^{(i)} = c} \mathcal{N}(z_t; \alpha_t z^{(i)}, \sigma_t^2 I)}{\sum_{i=1}^{N} \mathcal{N}(z_t; \alpha_t z^{(i)}, \sigma_t^2 I)}.

We average this posterior over forward-noised validation latents (solid) and intermediate CFG sampling states (dotted) to get the plot in Fig. 2(b).


Q: Why does aligning 32 pooled channel angles shift the full-space variance so much (Fig. 2)?

Although PR is applied only to the 32 pooled channels, these global statistics largely carry class identity, as the spatial variations have been averaged out. Overall, the pooled subspace spans 0.4% of the dimensions but carries about 30 to 40% of the total energy (Pooled energy), and holds over 80% of the inter-class energy in all three autoencoders (Inter-class: Pooled / Inter-class: Total).

% of total energyPooled energyInter-class: TotalInter-class: PooledInter-class: Residual
LDM30.18.26.61.6
VA-VAE (DINOv3-L)29.216.413.62.8
LDM + PR38.624.322.22.0

We use angular alignment in PR primarily to avoid the norm-inflation shortcut, leaving the norm to reconstruction. Empirically, the pooled energy does not shrink under PR (30.1% → 38.6%), so the angular separation carries over to Euclidean distances between class means, as reflected in the inter-class columns above.


Q: How was the shared CFG interval t≥0.11t \ge 0.11 chosen (Tab. 6)?

The CFG start time of 0.11 is inherited from the official VA-VAE repository. Skipping the high-noise regime is likely helpful because amplifying the CFG correction too early can harm the formation of the image layout.

We sweep the CFG start time for each method and report the guided gFID@50K results below (every entry uses the best guidance scale ω* re-swept at that start time). At each method's best start time, the ranking is unchanged, and the gap between LDM + PR and VA-VAE slightly widens (2.28 vs. 2.44 → 2.19 vs. 2.38).

CFG start time0.000.050.100.11 †0.150.200.250.30
LDM3.963.643.333.313.082.992.922.94
VA-VAE (DINOv3-L)3.712.752.522.442.542.382.392.40
LDM + PR3.692.972.382.282.192.222.292.33

† default setting in the paper

We also observe that the best start time is earlier for latents with higher class separability (e.g., by Purity@10 in Tab. 1). This is consistent with our analysis in Sec. 2: when the noisy class posterior concentrates earlier along the trajectory, the CFG correction becomes informative earlier.


Q: Video reconstruction metrics for Tab. 4 (e.g., PSNR, rFVD)?

We report reconstruction metrics on the test split of UCF-101. Fine-tuning improves frame-level metrics (PSNR, SSIM, LPIPS) over the official checkpoint at a slight rFVD cost shared by both fine-tuned variants. Overall, PR matches the recon-only baseline on all metrics, so its generation gains in Tab. 4 (e.g., guided gFVD 152.7 → 140.1) come at negligible reconstruction cost.

Stage 1 (LeanVAE)PSNR ↑SSIM ↑LPIPS ↓rFVD ↓
Official33.760.9360.0933.36
Fine-Tuned (Recon Only)34.040.9430.0774.13
Fine-Tuned (+PR, Ours)34.030.9430.0774.18

We hope our responses address your concerns. Please let us know if there are any further questions.

Dear Reviewer 7TBF,

Thank you for your valuable feedback and for taking the time to engage with our work. We hope our rebuttal updates address your concerns, and we would be glad to answer any follow-up questions before the end of the discussion period.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Remarks

We thank all reviewers for their constructive feedback and for taking the time to engage with our work. For ease of reading and discussion, we have included all new results directly in our responses below; please see the Summary of updates for details.

Unless otherwise specified, all new experiments follow the same training budget as in the main text (20 epochs for stage-1 and 80 epochs for stage-2) and the same sampling config (50 Euler steps with CFG applied in the guidance interval [0.11, 1.0]).


TL;DR

  • We study how to learn latent spaces that work well with CFG and identify class geometry as a key factor.
  • We propose a lightweight regularizer that promotes favorable geometry and improves guided generation.

Core Contributions

  • How class geometry shapes CFG. Under a Gaussian class model, we derive the oracle CFG correction in closed form and show that shifting variance from intra- to inter-class keeps classes distinguishable at higher noise levels. Hence, the target class posterior concentrates earlier in denoising and the CFG correction becomes informative sooner. We empirically verify both trends on learned models (Fig. 2b; see also our response to Reviewer 3Sob).
  • Prototype Regularization (PR). We formalize the derived geometric preference as minimizing the entropy of the noisy class posterior, whose gradient aligns with the oracle CFG correction. We introduce PR as a tractable, lightweight surrogate for this objective, which consistently improves guided generation. Center transplantation confirms that the gains originate from class geometry.
  • Generalization. PR extends to cluster pseudo-labels, video latents, and text-to-image generation, which suggests it works as a general latent geometry regularizer for guided generation. Applying PR to noisy latents at signal-dominated noise levels further improves guided gFID with comparable reconstruction (see our response to Reviewer 7TBF).

Summary of updates

New experiments (detailed in our responses)

  • Common question: T2I experiments on multi-object scenes (this thread)
  • Reviewer 7TBF:
    • Applying PR on noisy latents
    • Energy allocation in pooled/flat latents
    • CFG interval sweep
    • Video reconstruction metrics
  • Reviewer 3Sob:
    • Latents' empirical posterior E[qc]\mathbb{E}[q_c] and CFG correction strength over time
    • Sampling with CFG variants (CFG++, CFG-Zero*, interval guidance)
    • Latent geometry comparison with RAE, SVG, MAETok, and VTP
    • [Update] Controlled comparison with MAETok
  • Reviewer RGod:
    • Reproduced VA-VAE baseline (DINOv2-L, official recipe)
    • 250-step sampling (gFID vs. sFID)

PDF revisions (highlighted in blue)

  • Abstract: revised the "largely overlooked" phrasing (RGod).
  • Fig.1 & Fig.2: added how E[qc]\mathbb{E}[q_c] is estimated (7TBF).
  • Sec. 2: clarified theory takeaways and the theory-to-practice link (7TBF, 3Sob); added in-line citations to prior work (RGod); generalized the posterior objective to time distributions ρ\rho.
  • Related work: expanded to cover the suggested references [1-9] (RGod).

T2I experiments on multi-object scenes (MS-COCO)

All three reviewers asked how PR performs on text-to-image (T2I) generation in more complex, multi-object scenes. To address this, we added experiments on MS-COCO.

Specifically, we train on MS-COCO train2014 (~83K images) and evaluate on a fixed 30K subset of val2014, with the same preprocessing and latent shape as ImageNet (256×256 images, 32×16×16 latents). For a fair comparison, all tokenizers start from the same ImageNet-pretrained LDM VAE and are finetuned on COCO for 32K steps with different objectives. Since COCO images do not have single class label, PR uses k-means pseudo-labels (k=128) on DINOv3-L [CLS] features. On each tokenizer, we train a LightningDiT-XL/1 for 64K steps conditioned on CLIP ViT-L/14 text embeddings, with captions sampled uniformly from the official COCO captions and Florence-2 captions. We evaluate the trained T2I models separately with the official and the Florence-2 captions. The reconstruction and guided generation results are shown below.

FT MethodrFID↓PSNR↑LPIPS↓gFID@30K↓ (Official / Florence)CLIP↑ (Official / Florence)
Recon-only0.4728.290.0865.57 / 5.720.2361 / 0.2443
Recon + VA-VAE (DINOv3-L)0.5027.970.0929.58 / 10.090.2370 / 0.2435
Recon + PR (Clean-only, k=128, Ours)0.4828.260.0875.28 / 5.590.2413 / 0.2486
Reference: real images----0.2509 / 0.2575

PR improves both gFID and CLIP score over recon-only under both caption sets, with nearly unchanged reconstruction. In contrast, VA-VAE matches recon-only in CLIP score but has substantially worse gFID. We set k=128 empirically so that it exceeds the 80 COCO object categories and did not tune it further, so a better choice of k may give additional gains.

Area_Chair_4fvCOpenReview ↗

Author-Reviewer Discussion

Dear Reviewers,

Thank you for your time and thoughtful reviews. With the author–reviewer discussion period ending on November 18 (AoE), please review the authors' rebuttal and share your updated assessment. Specifically, let us know whether your main concerns have been resolved, and if not, which points remain open.

Your input before the deadline is greatly appreciated!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.