Rethinking Autoencoder Evaluation for Controllable Latent Diffusion

Autoencoders (AEs) define the latent space in which latent diffusion models learn the data distribution, making them a critical component of image generation. Recent AE evaluation disproportionately emphasizes whether a representation is diffusion-friendly, using image quality after diffusion training to compare and select AEs. In this work, we first formulate the condition-alignment objective of controllable latent diffusion. By decomposing the generation process and the conditional score, we derive three complementary dimensions of AE evaluation: faithfulness of condition preservation through reconstruction, recoverability of conditions from noisy latents, and tractability of learning the induced latent distribution. Existing practice prioritizes tractability, provides incomplete coverage of faithfulness through reconstruction metrics, and leaves condition recoverability unevaluated in the studies we examine. We examine representative AE methods and their variants, combining model comparisons with downstream controllable-generation experiments to answer three questions: whether the dimensions provide distinct information, whether they inform controllable-generation performance, and how to evaluate them efficiently. Our results show that the three dimensions can favor different AEs, and that condition preservation and prediction complement generation quality in assessing downstream control. Their relationship to control alignment depends on both the condition and the generator. Our recoverability experiments further show largely stable AE rankings across predictor designs and noise levels, supporting efficient comparison with a lightweight predictor evaluated on clean latents. This work broadens AE evaluation beyond diffusion-friendly image synthesis, providing a theoretically grounded and practical perspective on representations for controllable generation.