- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes three dimensions for evaluating autoencoders in controllable latent diffusion: faithfulness, recoverability, and tractability. Through theoretical analysis and experiments across different AEs and control tasks, the authors show that generation quality alone does not fully reflect condition alignment and suggest a lightweight approach to evaluating recoverability.
Critical strengths
The paper addresses a relevant gap in AE evaluation, particularly for downstream controllable generation.
The analysis connects AE properties with downstream control performance and reveals task-dependent differences in AE preferences.
Critical weaknesses
The controllable generation experiments cover only three variants from one AE family and two control tasks. This limits the generality of the proposed evaluation framework.
Its theoretical connection to conditional score learning remains indirect, while the experiments do not establish a consistent relationship with downstream control performance or demonstrate its added value over existing reconstruction metrics.
Questions for authors
How well do the three evaluation dimensions predict downstream control performance across different AE families, beyond the VA-VAE variants?
What additional insight does recoverability provide beyond faithfulness and existing reconstruction metrics? Can its independent contribution to AE selection be demonstrated?
Can the connection between condition prediction accuracy and conditional score learning be established more directly?
How would the cross-family comparisons change under a unified diffusion architecture and training recipe, given that the reported gFID values come from different settings?
6 comments
Weakness 2: Theory, downstream relevance, and added value.
We understand "its" here to refer to recoverability, and answer in four points below.
Theoretical connection. Eq. (7) of the paper splits the conditional score into the marginal score and the posterior term . Written in denoiser form, the difference between the optimal conditional denoiser and the optimal unconditional denoiser is exactly , and the decrease of the optimal denoising loss after the condition is added equals (Eqs. (8) and (9) in Sec. 3.3, derivation in Appendix A.5). In other words, what a conditional denoiser has to learn beyond an unconditional one is exactly the gradient of the condition posterior with respect to the latent. Classifier guidance generates conditionally with a classifier on the basis of this relation (Dhariwal & Nichol, 2021). What the TCP predictor estimates at each noise level is exactly this posterior.
Experimental connection (new Table 4, Sec. 4.2). On VA-VAE and its released LightningDiT-XL, neither the generator nor the AE is changed, only the gradient of the TCP predictor is added to the sampler as the posterior term, and the evaluation uses 256 ImageNet validation images. At guidance scales 0, 0.1, 0.3 and 0.6, Canny F1 is 0.118, 0.148, 0.190 and 0.215 in turn, and depth RMSE is 0.343, 0.320, 0.281 and 0.241 in turn. The difference between the score of a generated image for its own target and for the targets of unrelated images also grows with it (from 0.005 to 0.094 for Canny and from 0.024 to 0.116 for depth). Alignment improves monotonically with the guidance scale, and the change comes only from the posterior term supplied by the TCP predictor.
Relationship with downstream control (Sec. 4.2). Over the three groups of results in Table 3 (the L/2 and XL/2 backbone groups of VA-VAE and the new RAE group, 18 AE pairs in total), the ranking by recoverability agrees with that by control alignment on 16 pairs, faithfulness on 13 and base gFID on 6. Base gFID instead agrees best with gFID under control (12 pairs, against 7 and 6 for faithfulness and TCP). The relationship between recoverability and control alignment is consistent across two families and two backbones.
Added value over reconstruction metrics. Recoverability can reflect changes that reconstruction metrics cannot see. Among the 64-channel AEs released by VA-VAE, three versions differ only in representation alignment (new Table 9 in Appendix D.3):
| AE (64 channels) | Mean condition drift ↓ | PSNR ↑ | TCP Canny F1 ↑ | TCP Depth RMSE ↓ |
|---|---|---|---|---|
| No alignment (LDM) | 0.0540 | 27.09 | 0.4530 | 0.2417 |
| MAE alignment | 0.0535 | 27.28 | 0.3566 | 0.1685 |
| DINOv2 alignment | 0.0543 | 27.15 | 0.3527 | 0.1678 |
The mean condition drift of the three differs by only 0.0008 and their PSNR by less than 0.2 dB, yet TCP differs widely, in opposite directions for Canny and depth. This difference affects the choice of AE. On VA-VAE depth, faithfulness (depth RMSE of 0.0419, 0.0426 and 0.0429 in turn) and PSNR (25.21, 24.92 and 24.72 dB) both prefer LDM, and LDM has the worst depth alignment under both backbones. TCP prefers DINO (depth RMSE 0.1642, against 0.2411 for LDM), and DINO is exactly the AE with the best depth alignment under XL/2 (0.1842). Of these 6 pairs, faithfulness agrees with control alignment on 1 and TCP on 5. Looking at reconstruction metrics alone selects the AE with the worst alignment on depth.
Question 1: Predicting control performance across AE families.
Beyond VA-VAE, we ran the same matched experiment on the RAE family: DINOv2-B, MAE-B and SigLIP2-B use the 80-epoch generators released by RAE and one shared ControlNet recipe (the new RAE group of Table 3):
| Encoder | Base gFID ↓ | Faithfulness Canny F1 ↑ | Faithfulness Depth RMSE ↓ | TCP Canny F1 ↑ | TCP Depth RMSE ↓ | Control Canny F1 ↑ | Control Canny gFID ↓ | Control Depth RMSE ↓ | Control Depth gFID ↓ |
|---|---|---|---|---|---|---|---|---|---|
| DINOv2-B | 2.16 | 0.2968 | 0.0794 | 0.1974 | 0.1444 | 0.2360 | 5.61 | 0.1807 | 5.34 |
| MAE-B | 8.38 | 0.5859 | 0.0496 | 0.4143 | 0.1320 | 0.2925 | 13.43 | 0.1643 | 12.38 |
| SigLIP2-B | 4.93 | 0.3026 | 0.0860 | 0.1846 | 0.1614 | 0.2494 | 8.64 | 0.1865 | 8.08 |
The base gFID is taken from the RAE paper (50K samples), gFID under control is computed on 10K samples, and bold marks the best value in the group. The generator of DINOv2-B is DiT^DH-XL, and that of MAE-B and SigLIP2-B is DiT-XL.
Within the RAE family, the number of pairs ordered as control alignment is 6/6 for faithfulness, 5/6 for TCP and 1/6 for base gFID. The ranking by base gFID is exactly the reverse of the order of Canny alignment, while gFID under control follows the order of base gFID exactly.
Over the two families together, the number of pairs that each dimension orders as control alignment and as gFID under control is as follows (Sec. 4.2):
| Dimension | Pairs ordered as control alignment | Pairs ordered as gFID under control |
|---|---|---|
| Tractability (base gFID) | 6/18 | 12/18 |
| Faithfulness | 13/18 | 7/18 |
| Recoverability (TCP) | 16/18 | 6/18 |
Computed from Table 3: the L/2 and XL/2 groups of VA-VAE and the RAE group, 3 AE pairs per group, two conditions (Canny and depth), 18 pairs in total.
The three dimensions correspond to different outcomes of controllable generation. Faithfulness and TCP are most closely related to control alignment, agreeing on 13 and 16 pairs respectively, and the pairs on which they agree differ. On VA-VAE depth, faithfulness prefers LDM, which has the worst alignment, and TCP prefers DINO, which has the best alignment under XL/2. On RAE Canny, it is faithfulness that orders DINOv2-B and SigLIP2-B correctly. Base gFID agrees with alignment on 6 pairs, with all 9 Canny pairs reversed, but it agrees with gFID under control on 12 pairs, including all 6 pairs of the RAE group, more than faithfulness (7 pairs) and TCP (6 pairs).
Response to Reviewer RppC
Weakness 1: Limited AE families and control tasks.
We extended the matched controllable-generation experiment to a second AE family. The DINOv2-B, MAE-B and SigLIP2-B encoders of RAE use the 80-epoch generators released by RAE and one shared ControlNet recipe, and faithfulness and TCP are measured in the same way as for VA-VAE (the new RAE group of Table 3):
The base gFID is taken from the RAE paper (50K samples), gFID under control is computed on 10K samples, and bold marks the best value in the group. The generator of DINOv2-B is DiT^DH-XL, and that of MAE-B and SigLIP2-B is DiT-XL.
The conclusions of the paper hold in the second family. The ranking by base gFID is exactly the reverse of the order of Canny control alignment, so selecting the AE by generation quality misses control. The faithfulness ranking agrees with control alignment on both conditions, while gFID under control follows the order of base gFID, so alignment and image quality need to be evaluated separately.
The two families together give 9 combinations of AE and generator and 18 AE pairs. The TCP ranking agrees with control alignment on 16 pairs, faithfulness on 13 and base gFID on 6 (Sec. 4.2).
Regarding the control tasks. Canny and depth represent edge detail and geometric structure respectively. The best AE differs between the two tasks for the same set of AEs: on VA-VAE it is LDM for Canny and MAE or DINO for depth, which shows that AE evaluation needs to be done per condition. The AE-level measurements also cover segmentation: Sec. 4.1 compares the preservation of the three conditions Canny, depth and segmentation over 31 checkpoints from 6 families.