acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Autoencoder Evaluation for Controllable Latent Diffusion

Abstract

Autoencoders (AEs) define the latent space in which latent diffusion models learn the data distribution, making them a critical component of image generation. Recent AE evaluation disproportionately emphasizes whether a representation is diffusion-friendly, using image quality after diffusion training to compare and select AEs. In this work, we first formulate the condition-alignment objective of controllable latent diffusion. By decomposing the generation process and the conditional score, we derive three complementary dimensions of AE evaluation: faithfulness of condition preservation through reconstruction, recoverability of conditions from noisy latents, and tractability of learning the induced latent distribution. Existing practice prioritizes tractability, provides incomplete coverage of faithfulness through reconstruction metrics, and leaves condition recoverability unevaluated in the studies we examine. We examine representative AE methods and their variants, combining model comparisons with downstream controllable-generation experiments to answer three questions: whether the dimensions provide distinct information, whether they inform controllable-generation performance, and how to evaluate them efficiently. Our results show that the three dimensions can favor different AEs, and that condition preservation and prediction complement generation quality in assessing downstream control. Their relationship to control alignment depends on both the condition and the generator. Our recoverability experiments further show largely stable AE rankings across predictor designs and noise levels, supporting efficient comparison with a lightweight predictor evaluated on clean latents. This work broadens AE evaluation beyond diffusion-friendly image synthesis, providing a theoretically grounded and practical perspective on representations for controllable generation.

Primary area:
generative models
Keywords:
autoencoders, latent diffusion models, controllable generation, representation learning, model evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RppCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes three dimensions for evaluating autoencoders in controllable latent diffusion: faithfulness, recoverability, and tractability. Through theoretical analysis and experiments across different AEs and control tasks, the authors show that generation quality alone does not fully reflect condition alignment and suggest a lightweight approach to evaluating recoverability.

Critical strengths

The paper addresses a relevant gap in AE evaluation, particularly for downstream controllable generation.

The analysis connects AE properties with downstream control performance and reveals task-dependent differences in AE preferences.

Critical weaknesses

The controllable generation experiments cover only three variants from one AE family and two control tasks. This limits the generality of the proposed evaluation framework.

Its theoretical connection to conditional score learning remains indirect, while the experiments do not establish a consistent relationship with downstream control performance or demonstrate its added value over existing reconstruction metrics.

Questions for authors

How well do the three evaluation dimensions predict downstream control performance across different AE families, beyond the VA-VAE variants?

What additional insight does recoverability provide beyond faithfulness and existing reconstruction metrics? Can its independent contribution to AE selection be demonstrated?

Can the connection between condition prediction accuracy and conditional score learning be established more directly?

How would the cross-family comparisons change under a unified diffusion architecture and training recipe, given that the reported gFID values come from different settings?

6 comments
AuthorsOpenReview ↗

Response to Reviewer RppC

Weakness 1: Limited AE families and control tasks.

We extended the matched controllable-generation experiment to a second AE family. The DINOv2-B, MAE-B and SigLIP2-B encoders of RAE use the 80-epoch generators released by RAE and one shared ControlNet recipe, and faithfulness and TCP are measured in the same way as for VA-VAE (the new RAE group of Table 3):

EncoderBase gFID ↓Faithfulness Canny F1 ↑Faithfulness Depth RMSE ↓TCP Canny F1 ↑TCP Depth RMSE ↓Control Canny F1 ↑Control Canny gFID ↓Control Depth RMSE ↓Control Depth gFID ↓
DINOv2-B2.160.29680.07940.19740.14440.23605.610.18075.34
MAE-B8.380.58590.04960.41430.13200.292513.430.164312.38
SigLIP2-B4.930.30260.08600.18460.16140.24948.640.18658.08

The base gFID is taken from the RAE paper (50K samples), gFID under control is computed on 10K samples, and bold marks the best value in the group. The generator of DINOv2-B is DiT^DH-XL, and that of MAE-B and SigLIP2-B is DiT-XL.

The conclusions of the paper hold in the second family. The ranking by base gFID is exactly the reverse of the order of Canny control alignment, so selecting the AE by generation quality misses control. The faithfulness ranking agrees with control alignment on both conditions, while gFID under control follows the order of base gFID, so alignment and image quality need to be evaluated separately.

The two families together give 9 combinations of AE and generator and 18 AE pairs. The TCP ranking agrees with control alignment on 16 pairs, faithfulness on 13 and base gFID on 6 (Sec. 4.2).

Regarding the control tasks. Canny and depth represent edge detail and geometric structure respectively. The best AE differs between the two tasks for the same set of AEs: on VA-VAE it is LDM for Canny and MAE or DINO for depth, which shows that AE evaluation needs to be done per condition. The AE-level measurements also cover segmentation: Sec. 4.1 compares the preservation of the three conditions Canny, depth and segmentation over 31 checkpoints from 6 families.

Weakness 2: Theory, downstream relevance, and added value.

We understand "its" here to refer to recoverability, and answer in four points below.

Theoretical connection. Eq. (7) of the paper splits the conditional score into the marginal score and the posterior term ∇ztlog⁡p(c∣zt)\nabla_{z_t}\log p(c\mid z_t). Written in denoiser form, the difference between the optimal conditional denoiser and the optimal unconditional denoiser is exactly ϵ^(zt,c)−ϵ^(zt)=−σt∇ztlog⁡p(c∣zt)\hat\epsilon(z_t,c)-\hat\epsilon(z_t)=-\sigma_t\nabla_{z_t}\log p(c\mid z_t), and the decrease of the optimal denoising loss after the condition is added equals σt2E∥∇ztlog⁡p(C∣Zt)∥2\sigma_t^2 \mathbb{E}\lVert\nabla_{z_t}\log p(C\mid Z_t)\rVert^2 (Eqs. (8) and (9) in Sec. 3.3, derivation in Appendix A.5). In other words, what a conditional denoiser has to learn beyond an unconditional one is exactly the gradient of the condition posterior with respect to the latent. Classifier guidance generates conditionally with a classifier on the basis of this relation (Dhariwal & Nichol, 2021). What the TCP predictor estimates at each noise level is exactly this posterior.

Experimental connection (new Table 4, Sec. 4.2). On VA-VAE and its released LightningDiT-XL, neither the generator nor the AE is changed, only the gradient of the TCP predictor is added to the sampler as the posterior term, and the evaluation uses 256 ImageNet validation images. At guidance scales 0, 0.1, 0.3 and 0.6, Canny F1 is 0.118, 0.148, 0.190 and 0.215 in turn, and depth RMSE is 0.343, 0.320, 0.281 and 0.241 in turn. The difference between the score of a generated image for its own target and for the targets of unrelated images also grows with it (from 0.005 to 0.094 for Canny and from 0.024 to 0.116 for depth). Alignment improves monotonically with the guidance scale, and the change comes only from the posterior term supplied by the TCP predictor.

Relationship with downstream control (Sec. 4.2). Over the three groups of results in Table 3 (the L/2 and XL/2 backbone groups of VA-VAE and the new RAE group, 18 AE pairs in total), the ranking by recoverability agrees with that by control alignment on 16 pairs, faithfulness on 13 and base gFID on 6. Base gFID instead agrees best with gFID under control (12 pairs, against 7 and 6 for faithfulness and TCP). The relationship between recoverability and control alignment is consistent across two families and two backbones.

Added value over reconstruction metrics. Recoverability can reflect changes that reconstruction metrics cannot see. Among the 64-channel AEs released by VA-VAE, three versions differ only in representation alignment (new Table 9 in Appendix D.3):

AE (64 channels)Mean condition drift ↓PSNR ↑TCP Canny F1 ↑TCP Depth RMSE ↓
No alignment (LDM)0.054027.090.45300.2417
MAE alignment0.053527.280.35660.1685
DINOv2 alignment0.054327.150.35270.1678

The mean condition drift of the three differs by only 0.0008 and their PSNR by less than 0.2 dB, yet TCP differs widely, in opposite directions for Canny and depth. This difference affects the choice of AE. On VA-VAE depth, faithfulness (depth RMSE of 0.0419, 0.0426 and 0.0429 in turn) and PSNR (25.21, 24.92 and 24.72 dB) both prefer LDM, and LDM has the worst depth alignment under both backbones. TCP prefers DINO (depth RMSE 0.1642, against 0.2411 for LDM), and DINO is exactly the AE with the best depth alignment under XL/2 (0.1842). Of these 6 pairs, faithfulness agrees with control alignment on 1 and TCP on 5. Looking at reconstruction metrics alone selects the AE with the worst alignment on depth.

Question 1: Predicting control performance across AE families.

Beyond VA-VAE, we ran the same matched experiment on the RAE family: DINOv2-B, MAE-B and SigLIP2-B use the 80-epoch generators released by RAE and one shared ControlNet recipe (the new RAE group of Table 3):

EncoderBase gFID ↓Faithfulness Canny F1 ↑Faithfulness Depth RMSE ↓TCP Canny F1 ↑TCP Depth RMSE ↓Control Canny F1 ↑Control Canny gFID ↓Control Depth RMSE ↓Control Depth gFID ↓
DINOv2-B2.160.29680.07940.19740.14440.23605.610.18075.34
MAE-B8.380.58590.04960.41430.13200.292513.430.164312.38
SigLIP2-B4.930.30260.08600.18460.16140.24948.640.18658.08

The base gFID is taken from the RAE paper (50K samples), gFID under control is computed on 10K samples, and bold marks the best value in the group. The generator of DINOv2-B is DiT^DH-XL, and that of MAE-B and SigLIP2-B is DiT-XL.

Within the RAE family, the number of pairs ordered as control alignment is 6/6 for faithfulness, 5/6 for TCP and 1/6 for base gFID. The ranking by base gFID is exactly the reverse of the order of Canny alignment, while gFID under control follows the order of base gFID exactly.

Over the two families together, the number of pairs that each dimension orders as control alignment and as gFID under control is as follows (Sec. 4.2):

DimensionPairs ordered as control alignmentPairs ordered as gFID under control
Tractability (base gFID)6/1812/18
Faithfulness13/187/18
Recoverability (TCP)16/186/18

Computed from Table 3: the L/2 and XL/2 groups of VA-VAE and the RAE group, 3 AE pairs per group, two conditions (Canny and depth), 18 pairs in total.

The three dimensions correspond to different outcomes of controllable generation. Faithfulness and TCP are most closely related to control alignment, agreeing on 13 and 16 pairs respectively, and the pairs on which they agree differ. On VA-VAE depth, faithfulness prefers LDM, which has the worst alignment, and TCP prefers DINO, which has the best alignment under XL/2. On RAE Canny, it is faithfulness that orders DINOv2-B and SigLIP2-B correctly. Base gFID agrees with alignment on 6 pairs, with all 9 Canny pairs reversed, but it agrees with gFID under control on 12 pairs, including all 6 pairs of the RAE group, more than faithfulness (7 pairs) and TCP (6 pairs).

Reviewer_V1mjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that autoencoder (AE) evaluation for latent diffusion focuses too much on gFID and ignores what controllable generation needs. Starting from a condition-alignment loss, it derives three evaluation dimensions: 1) Faithfulness: how much a condition (Canny, depth, segmentation) drifts through reconstruction. 2) Recoverability. 3) Tractability: how well a generator fits the latent distribution, measured by gFID.

It shows that condition drift equals alignment loss under perfect latent fitting, with a Lipschitz/Wasserstein bound for imperfect fitting. Across many ImageNet AE checkpoints, condition drift tracks PSNR and LPIPS closely but not reported gFID. ControlNet experiments on three VA-VAE variants with two DiT backbones show that the best AE depends on the condition and the generator.

Critical strengths

The question is timely and practical. AEs from T2I models are reused for control and editing, yet they are selected mainly by gFID. The paired vs distributional distinction is a clean point, and the rFID-swap example in A.2 shows it well.

The recoverability measurement is cheap and stable. Rankings hold in 77 of 80 predictor-noise comparisons, so a light probe on clean latents is a usable tool.

Critical weaknesses
  1. Can you extend the downstream study to more AEs, ideally different families such as RAE, UAE and REPA-E, plus the T2I AEs (SD-VAE, FLUX-VAE) that motivate the paper? i am curious about if the same family AEs have similar pros/cons?

  2. The current results didn't show std, etc. Can you add multiple seeds, and compute rank correlations between each dimension and control alignment? hard to tell the significance of the porposed method.

  3. not weakness but i do believe that this is a critical direction we should explore in this paper: There are some discussions on the diffusability of the AEs (especially for the AEs w/ various channel dimensions). I am curious if the authors explored the differences among the AEs w/ the same arch but different channel dimensions?

Questions for authors

i found that faithfulness tracks PSNR almost perfectly, what does condition drift tell you that PSNR or LPIPS doesn’t? Is there a case where they rank AEs differently?

3 comments
AuthorsOpenReview ↗

Response to Reviewer V1mj

Weakness 1: More AE families and within-family differences.

Thank you for the suggestion. We extended the downstream study to a second AE family, and on this basis answer whether AEs of the same family have similar pros and cons.

1. Matched experiment on the RAE family (the new RAE group of Table 3). DINOv2-B, MAE-B and SigLIP2-B use the 80-epoch generators released by RAE and one shared ControlNet recipe, and faithfulness and TCP are measured in the same way as for VA-VAE:

EncoderBase gFID ↓Faithfulness Canny F1 ↑Faithfulness Depth RMSE ↓TCP Canny F1 ↑TCP Depth RMSE ↓Control Canny F1 ↑Control Canny gFID ↓Control Depth RMSE ↓Control Depth gFID ↓
DINOv2-B2.160.29680.07940.19740.14440.23605.610.18075.34
MAE-B8.380.58590.04960.41430.13200.292513.430.164312.38
SigLIP2-B4.930.30260.08600.18460.16140.24948.640.18658.08

The base gFID is taken from the RAE paper (50K samples), gFID under control is computed on 10K samples, and bold marks the best value in the group. The generator of DINOv2-B is DiT^DH-XL, and that of MAE-B and SigLIP2-B is DiT-XL.

The results on the RAE family agree with the conclusions of the paper. The ranking by base gFID is exactly the reverse of the order of Canny control alignment, so selecting the AE by generation quality misses control. The faithfulness ranking agrees with control alignment on both conditions, while gFID under control follows the order of base gFID, so alignment and image quality are determined by different properties of the AE.

2. Whether AEs of the same family have similar pros and cons. AEs of the same family share the properties that come from the architecture, while the pros and cons related to control are determined by the encoder or the alignment target, and this pattern is the same in both families:

  • VA-VAE family. Variants with the same channel dimension have almost the same faithfulness (at 32 channels the mean condition drift differs by no more than 0.0024, new Table 9 in Appendix D.3), and the differences appear in recoverability and tractability. The DINOv2-aligned variant has the best base gFID and the most predictable depth, and the worst Canny prediction and Canny control alignment. The unaligned LDM is exactly the opposite.
  • RAE family. The three encoders differ more, in the same direction as for VA-VAE. The semantic encoder DINOv2-B has the best base gFID (2.16) and the worst Canny control alignment (0.2360). MAE-B, whose pretraining objective is reconstruction, has the worst base gFID (8.38) and the best faithfulness and control alignment on both conditions.
  • The pattern shared by the two families. AEs that use DINOv2 as the alignment target or the encoder have the best generation quality and the weakest Canny control, and AEs that use MAE as the target have better condition preservation and Canny control alignment. So the pros and cons within one family are not the same, and it is AEs that adopt the same representation that have similar pros and cons.

Weakness 2: Variability and rank correlations.

Thank you for the suggestion. Below we give the rank correlation and the standard deviation in turn.

Rank correlation (Sec. 4.2). We use Table 3 to compute the rank agreement of each dimension with the outcomes of controllable generation. Controllable generation has two outcomes: control alignment, and the quality of the images generated under control (gFID under control).

DimensionPairs ordered as control alignmentPairs ordered as gFID under control
Tractability (base gFID)6/1812/18
Faithfulness13/187/18
Recoverability (TCP)16/186/18

Computed from Table 3: the L/2 and XL/2 groups of VA-VAE and the RAE group, 3 AE pairs per group, two conditions (Canny and depth), 18 pairs in total. Converted to Kendall τ, the values with control alignment are −0.33, 0.44 and 0.78 in turn, and the values with gFID under control are 0.33, −0.22 and −0.33 in turn.

All three dimensions affect the final controllable generation. Faithfulness and TCP are most closely related to control alignment, and the pairs on which they agree with it differ: on VA-VAE depth, TCP agrees on 5/6 pairs and faithfulness on 1/6, and on RAE Canny, faithfulness agrees on 3/3 and TCP on 2/3. Base gFID is only weakly related to alignment, with all 9 Canny pairs reversed, but it is most closely related to image quality under control. The RAE group shows this most clearly: the base gFID of the three encoders is 2.16, 4.93 and 8.38, gFID under control follows exactly this order (Canny 5.61, 8.64, 13.43, depth 5.34, 8.08, 12.38), while control alignment is ordered by faithfulness, with MAE-B the best. Looking at gFID alone roughly predicts image quality, but gives the opposite alignment judgment on 12 of the 18 pairs. After faithfulness and recoverability are added, both alignment and image quality can be explained.

Standard deviation and significance. The four faithfulness and TCP columns are all computed on the 50K ImageNet validation images. We bootstrap over images (2,000 resamples) and add standard errors to these four columns for both the VA-VAE and RAE families in Table 3: 0.0002 to 0.0005 for the two faithfulness columns, and 0.0003 to 0.0004 for the two TCP columns. All AEs are evaluated on the same images, so we run a paired bootstrap on the difference of each pair of AEs. In the four columns, the AE ranking of each group stays unchanged in all 2,000 resamples, with one exception: the faithfulness depth RMSE of MAE and DINO in the VA-VAE group differs by only 0.0003, and their order holds in 98.5% of the resamples. The TCP predictors were additionally trained once with each of 3 training seeds. Across seeds, the standard deviation of the 12 predictors is no more than 0.0030 for Canny F1 and no more than 0.0003 for depth RMSE, and under every seed the AE ranking of both families is the same as in Table 3. These results are reported in Appendix B.2. The TCP ranking is equally stable across predictor designs and noise levels (Sec. 4.3): among the 40 combinations of 4 designs × 10 noise levels, the depth ranking is the same in all of them, the Canny ranking is the same in 37, and the other 3 are swaps between LDM and MAE smaller than 0.003 F1. The per-image bootstrap 95% intervals of the TCP guidance experiment are as follows: at scale 0.6, Canny F1 improves by 0.098 with interval [0.092, 0.103], and depth RMSE decreases by 0.102 with interval [0.090, 0.113]. The control alignment and gFID columns are results of single training and sampling runs, and their conclusions are supported by the consistency among the two backbone groups and the RAE group (Sec. 4.2).

Weakness 3: Effects of latent channel dimension.

Thank you for this suggestion. VA-VAE released a set of AEs with the same architecture and different channel dimensions: d16, and three versions each at d32 and d64 (no alignment, MAE alignment, DINOv2 alignment). We measured the three dimensions for these 7 AEs with the same protocol (new Table 9 in Appendix D.3):

AlignmentChannelsMean condition drift ↓PSNR ↑TCP Canny F1 ↑TCP Depth RMSE ↓FID-10K ↓
None (LDM)160.081523.180.41350.239116.24
None (LDM)320.067525.210.42970.241122.62
MAE320.068624.920.40810.220719.89
DINOv2320.069924.720.33360.164215.82
None (LDM)640.054027.090.45300.241736.83
MAE640.053527.280.35660.168523.58
DINOv2640.054327.150.35270.167824.00

Mean condition drift and PSNR represent faithfulness. TCP uses the Base predictor, with the same training and scoring setting as Table 3. FID-10K is taken from the VA-VAE paper (LightningDiT-B, 160 epochs) and represents tractability. Bold marks the best value in each column.

The more channels, the better the faithfulness and the worse the tractability. For the unaligned LDM from d16 to d32 to d64, the mean condition drift falls from 0.0815 to 0.0675 and 0.0540, while FID-10K rises from 16.24 to 22.62 and 36.83. Looking at gFID alone favors low-channel AEs, and low-channel AEs preserve less condition information.

Representation alignment mainly improves tractability and hardly changes faithfulness. At d64, MAE and DINOv2 alignment reduce FID-10K from 36.83 to 23.58 and 24.00, and the drift of the three versions differs by only 0.0008 (0.0024 at d32).

Representation alignment clearly changes recoverability, and the direction depends on the condition. For the unaligned LDM, the TCP F1 of Canny rises as the channels increase (0.4135, 0.4297, 0.4530), and the TCP RMSE of depth stays basically unchanged (0.2391, 0.2411, 0.2417). After alignment is added, Canny is harder to recover and depth is easier to recover at both channel dimensions: at d64, MAE and DINOv2 alignment lower Canny F1 from 0.4530 to 0.3566 and 0.3527 and lower depth RMSE from 0.2417 to 0.1685 and 0.1678, and DINOv2 alignment at d32 shows the same change (0.4297 to 0.3336, 0.2411 to 0.1642). These changes occur between AEs whose condition drift is almost unchanged, and cannot be inferred from faithfulness.

The three dimensions give different best AEs. The three d64 AEs have the best faithfulness, and among them LDM also has the best Canny recoverability but the worst tractability. The DINOv2-aligned d32 AE has the best tractability and depth recoverability, and the worst Canny recoverability. The choice of AE depends on the control task and requires looking at the three dimensions together. Sec. 4.1 adds this analysis.

Question 1: Condition drift beyond PSNR and LPIPS.

Faithfulness and PSNR agree almost perfectly, and this is exactly the finding we want to emphasize. Over the 31 checkpoints, the absolute Spearman correlation between mean condition drift and PSNR is 0.989. This agreement has a theoretical basis: under ideal latent fitting, the expected alignment loss equals the condition drift (Eq. (4)), and the paired reconstruction error bounds the condition drift through a Lipschitz relation. Paired reconstruction metrics such as PSNR are therefore very good proxies for faithfulness.

However, such metrics are often overlooked in AE work (Sec. 2, Table 1). Among the 8 representative AE studies we examined, only VA-VAE and UAE report PSNR, REPA-E, RAE, VTP and SVG report only rFID, and FAE and SVG-T2I report no reconstruction metric. In the ablations, the configuration selected by all 5 studies is the one with the best gFID. One direct effect of the paper's emphasis on faithfulness is to bring paired reconstruction metrics such as PSNR back into AE evaluation.

PSNR and condition drift can also give different rankings, especially for conditions that depend on higher-level structure. The absolute correlation of PSNR with Canny drift reaches 0.993, and with depth drift it is 0.963, lower than the 0.982 of LPIPS. In pairwise comparison, among the 465 checkpoint pairs, PSNR orders 10, 28 and 35 pairs opposite to Canny, depth and segmentation drift respectively (Appendix D.1). For example, the PSNR of RAE is 4.19 dB lower than that of SVG-T2I (R stage2), yet its depth and segmentation drift are smaller.

We therefore recommend (now stated in Sec. 4.3): in most cases, use paired reconstruction metrics such as PSNR as the main measure of faithfulness. For scenarios that depend on specific structure, such as text rendering where thin strokes must be preserved accurately, additionally measure the drift of the target condition.

Reviewer_DyfeOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an interesting approach for evaluation of autoencoders for controllable latent diffusion. This covers condition preservation through reconstruction and condition prediction from latents alognside generation quality. The analysis performed relates reconstructed images to errors in generated images, then shows how a generator that does not perfectly learn the latent distribution can change that relationship. The experiments show that good image generation quality does not necessarliy imply good control.

Critical strengths
  1. The paper highlights a useful limitation of generation focused evaluation, supported by experiments where the autoencoder with best generation FID acheives worse alignment. This comaprison is strong for smaller diffusion models.
  2. The reconstruction analysis provides a clear foundation for evaluating condition preservation and is ably supported by formal derivations.
  3. The appendix section shows strong a relation between paired reconstruction and condition preservation. This gives an accessible starting point for evaluation.
Critical weaknesses
  1. The generation quality comaprison is not controlled. It uses only 6 checkpoints with generation scores from different training settings. So differences cannot be tied to the autoencoder alone.
  2. Predicting a condition accurately is different from learning how to generate images with that condition. The authors need to show a direct relationship between these two abilities.
  3. The efficiency claim is partially supported. The proposed shortcut still uses predictors trained over different noise levels for 20 epochs. More details regarding how much computation is saved is needed.
  4. The analysis should models which have reconstruction FID above 3. Moreover it includes entries with identical reconstruction measurements, so verifying whether a different selection gives same conclusions is needed.
Questions for authors

Why were the control experiments limited to VA-VAE variants? Do the conclusions hold for RAE or UAE?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Dyfe

Weakness 1: Generation scores from different training settings.

We understand the reviewer's concern: when gFID is used to evaluate the generation ability of an AE, the result is coupled with the diffusion model. We would like to explain that this coupling is reasonable and is also the fairer way to compare.

Table 2 deliberately uses the gFID reported in the original paper of each AE, which are also the numbers the community actually relies on when selecting and evaluating AEs. Different AEs induce different latent spaces. The channel dimension ranges from 32 (VA-VAE, REPA-E) and 64 (VTP) to 384 (SVG) and 768 (RAE, UAE), so the architecture and training strategy of the diffusion model also differ in places. The generation quality of an AE also depends on a diffusion model that suits it. RAE reports that a standard DiT-XL has gFID 23.08 on its latents and 7.13 on SD-VAE. After the noise schedule is adjusted and noise augmentation is added to decoder training, DiT-XL drops to 4.28 on RAE. Switching further to DiT^DH-XL with a wide DDT head improves RAE from 4.28 to 2.16, while SD-VAE degrades from 7.13 to 11.70. Using one diffusion architecture and training recipe for all AEs would favor the latent space that matches this recipe, and the comparison would become less fair. The paper therefore defines tractability relative to the generator family (Sec. 3.4), and our other experiments that involve diffusion training also follow the original architecture and training strategy of each AE. Sec. 4.3 of the revised paper states this practice: AEs with the same latent shape are compared with one generator, and AEs with different latent spaces keep their own generators.

Regarding "only 6 checkpoints", we add nine gFID values obtained under matched settings (the Base gFID column of Table 3): the three VA-VAE variants with LightningDiT-L/2 and with LightningDiT-XL/2, and the three RAE encoders with the generators released by RAE. Within each VA-VAE group, the latent shape, the diffusion architecture and the training recipe are the same, so differences in gFID come only from the AE. These nine results are now included in Figure 1(a), and Sec. 4.1 adds the corresponding description. In both VA-VAE groups, the gFID ranking is exactly the reverse of the ranking by mean condition drift. DINO has the lowest gFID (4.73, 4.38) and the highest drift (0.0699). LDM has the highest gFID (6.29, 5.91) and the lowest drift (0.0675). In the RAE group, MAE-B has the highest gFID (8.38) and the lowest drift (0.0627), while DINOv2-B and SigLIP2-B, whose gFID is 2.16 and 4.93, have similar drift (0.1085 and 0.1096). In all three groups, the AE with the best gFID is not the one with the best condition preservation.

Weakness 2: Condition prediction versus conditional generation.

We agree that the two abilities are different, which is also why the paper separates recoverability and tractability into two dimensions. Below we first explain what TCP corresponds to in conditional generation, and then give the direct relationship between the two with two experiments.

What TCP corresponds to. Eq. (7) of the paper splits the conditional score into the marginal score and the posterior term ∇ztlog⁡p(c∣zt)\nabla_{z_t}\log p(c\mid z_t). Written in denoiser form, the difference between the optimal conditional denoiser and the optimal unconditional denoiser is exactly ϵ^(zt,c)−ϵ^(zt)=−σt∇ztlog⁡p(c∣zt)\hat\epsilon(z_t,c)-\hat\epsilon(z_t)=-\sigma_t\nabla_{z_t}\log p(c\mid z_t), and the decrease of the optimal denoising loss after the condition is added equals σt2E∥∇ztlog⁡p(C∣Zt)∥2\sigma_t^2 \mathbb{E}\lVert\nabla_{z_t}\log p(C\mid Z_t)\rVert^2 (Eqs. (8) and (9) in Sec. 3.3, derivation in Appendix A.5). In other words, what a conditional denoiser has to learn beyond an unconditional one is exactly the gradient of the condition posterior with respect to the latent. Classifier guidance generates conditionally with a classifier on the basis of this relation (Dhariwal & Nichol, 2021). What the TCP predictor estimates at each noise level is exactly this posterior. How much of it a generator can learn also depends on the fitting error (Proposition 1), so we test the relationship between prediction and generation directly with the two experiments below.

Experiment 1: generating directly with the TCP predictor (new Table 4, Sec. 4.2). On VA-VAE and its released LightningDiT-XL, neither the generator nor the AE is changed, and only the gradient of the TCP predictor with respect to the target condition is added to the sampler as the posterior term. The sampling details are in Appendix B.4. Each setting is evaluated on 256 ImageNet validation images:

ConditionGuidance scaleAlignmentGain over unguidedGap
Canny (F1 ↑)00.118—0.005
0.10.148+0.0300.025
0.30.190+0.0730.060
0.60.215+0.0980.094
Depth (RMSE ↓)00.343—0.024
0.10.320+0.0230.041
0.30.281+0.0620.075
0.60.241+0.1020.116

Gap is the difference between the score of a generated image for its own target and its score for the targets of unrelated images. For both conditions, alignment improves monotonically with the guidance scale: Canny F1 rises from 0.118 to 0.215, and depth RMSE falls from 0.343 to 0.241. The gap grows with it, from 0.005 to 0.094 for Canny and from 0.024 to 0.116 for depth, which shows that the generated images follow their own specified targets. Neither the generator nor the AE is changed, so the change comes only from the posterior term supplied by the TCP predictor. The ability to predict a condition can be used directly for conditional generation.

Experiment 2: across AEs, the AE whose conditions are predicted more accurately has better control alignment (Table 3, Sec. 4.2). Over the 18 AE pairs of the two VA-VAE backbone groups and the new RAE group, the TCP ranking agrees with the control alignment ranking on 16 pairs, faithfulness on 13 and base gFID on 6. On VA-VAE depth, reconstruction metrics prefer LDM, and LDM has the worst depth alignment under both backbones. TCP prefers DINO, and DINO is exactly the AE with the best depth alignment under XL/2.

Weakness 3: Computational cost of TCP evaluation.

Thank you for the suggestion. We have added the measured computation. The efficiency of the evaluation in the paper comes mainly from the fact that the predictor can be small: Sec. 4.3 compares four predictor designs and ten noise levels, and the AE ranking is consistent across designs and noise levels, so it is enough to compare with a small predictor at any single noise level. The table below gives the measured GPU time for training and scoring a predictor for one AE and one condition (new Table 5 in Sec. 4.3, one RTX 3090, batch 128, mixed precision):

PredictorTraining and scoringParametersGPU-hours
Base (wider + residual)all noise levels, 20 epochs7.89M6.22
Base (wider)all noise levels, 20 epochs7.72M4.95
Baseall noise levels, 20 epochs0.51M1.03
Base (shallower)all noise levels, 20 epochs0.48M0.52
Base (shallower)clean latents only, 10 epochs0.48M0.25

GPU-hours are the sum of the time for training and for scoring the 50K validation images. The first four rows are scored at ten noise levels, and the last row is scored at t = 0 only. The four designs give the same AE ranking. The Base predictor used in the paper takes 1.03 GPU-hours, and the shallower predictor takes 0.52 GPU-hours.

On this basis, we newly trained the shallower predictor with training and scoring at a single noise level (t = 0) only, for 10 epochs. It gives the same AE ranking as Base: Canny F1 is 0.4210 for LDM, 0.4178 for MAE and 0.3982 for DINO, depth RMSE is 0.1615 for DINO, 0.2213 for MAE and 0.2449 for LDM, and training and scoring take 0.25 GPU-hours in total. Epoch by epoch, the depth ranking no longer changes from the first epoch on, and the RMSE at epoch 5 differs from that at epoch 10 by no more than 0.006. On Canny, LDM and MAE finally differ by only 0.003. The two swap in the first few epochs and stay stable over the last 3 epochs. When the same setting is trained for 20 epochs, the ranking is unchanged and the scores differ from the 10-epoch ones by no more than 0.004. These results are reported in Appendix B.2.

For reference, measuring controllable generation directly requires training a base diffusion model for each AE and then a ControlNet for each AE and each condition, and our ControlNets took about 8 to 11 GPU-hours for 20k training steps. Although the TCP evaluation involves model training, its overall cost is small.

Weakness 4: Checkpoint exclusions and duplicate entries.

Appendix D follows the rule of the original comparison that excludes rFID > 3.0, and four checkpoints are excluded in total: the three stage weights of SVG-T2I trained for 512 and 1024 pixels, and the first-stage intermediate weight of UAE's progressive training. Their reconstructions at 256 pixels are clearly distorted (new Figure 3), and they do not represent AEs actually used for latent diffusion.

We thank the reviewer for pointing out the duplicated entries. After checking, two pairs of entries correspond to the same model: uae and uae-stage4 (the weight released by UAE is the final stage-4 weight, and the two sets of reconstructed images are byte-identical), and e2e-vae and e2e-vavae. The table below compares the Spearman correlations between reconstruction metrics and mean condition drift under three checkpoint selections (absolute value for PSNR):

Checkpoint setCheckpointsrFIDPSNRLPIPS
Submission330.9230.9890.953
Two duplicates removed310.9180.9890.950
Duplicates removed, four excluded checkpoints added350.9430.9910.966

Neither removing the duplicates nor adding the four excluded checkpoints changes the conclusions of the paper. After the duplicated entries are removed, the correlations change by no more than 0.005. After the four excluded checkpoints are added, the correlations are higher, because these checkpoints have both the largest reconstruction error and the largest condition drift. Under all three selections, paired reconstruction metrics (PSNR, LPIPS) agree with condition drift more closely than rFID does.

The revised paper removes the two duplicated entries, recomputes Figure 1 and Tables 6 to 8 of Appendix D on 31 checkpoints, and reports the correlations under the three selections in Appendix D.2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.