- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes interpolated FID (iFID), a simple VAE evaluation metric that measures the FID of images decoded from interpolated nearest-neighbor latent representations. The authors provide theoretical intuition based on diffusion generalization and demonstrate that iFID correlates strongly with diffusion generation FID (gFID) across 22 VAEs and 44 diffusion models, significantly outperforming conventional reconstruction metrics.
Critical strengths
-
Practical significance and simplicity. The paper addresses an important problem in latent diffusion models: reconstruction quality does not reliably predict generation quality. The proposed iFID is simple to implement and provides a practical way to evaluate VAEs without training expensive diffusion models.
-
Extensive empirical evaluation. The authors evaluate 22 diverse VAEs and compare iFID against 14 existing metrics. The consistently strong correlations across different diffusion model sizes, training settings, and additional generation benchmarks provide convincing empirical support for the proposed metric.
Critical weaknesses
-
Limited theoretical justification. The theoretical analysis establishes that diffusion samples may lie close to local affine lines connecting neighboring latent representations. However, the proposed metric uses fixed-midpoint SLERP interpolation, which is not directly justified by the theory. Furthermore, the analysis does not formally establish a relationship between interpolation FID and generation FID.
-
Potential confounding factors in the correlation analysis. The evaluated VAEs differ substantially in architecture, latent dimensionality, and training objectives. Consequently, the strong overall correlation may partly arise from systematic differences between VAE families rather than the ability of iFID to distinguish generation performance among similar VAEs. Additional controlled experiments are needed to establish its fine-grained predictive capability.
Questions for authors
-
Predictive power within VAE families ( Weakness 2 ). Does iFID remain strongly correlated with gFID when evaluated within the same VAE architecture or across different training checkpoints? Such experiments would help determine whether the observed correlation is primarily driven by differences between VAE families.
-
Theoretical consistency ( Weakness 1 ). The theoretical analysis focuses on linear interpolation between nearby latent samples, whereas SLERP substantially outperforms linear interpolation in Table 6. Can the authors explain this discrepancy and provide further justification for using SLERP with a fixed interpolation coefficient of 0.5?
-
Generalization to text-to-image generation. While Table 7 reports promising correlations between iFID and T2I benchmarks, it remains unclear whether these correlations reflect the intrinsic predictive capability of iFID or other differences across the evaluated models. Can the authors provide a more controlled evaluation where only the VAE is varied while keeping the T2I backbone and training configuration fixed?
Response to Reviewer DyWG
W1 & Q2. Theoretical justification and the discrepancy between linear interpolation and SLERP
W2 & Q1. Predictive capability within the same VAE family
We agree that correlations across heterogeneous VAEs may be influenced by differences in architecture, latent dimensionality, and training objectives. To directly address this concern, we conduct an additional controlled experiment involving 13 VA-VAE variants with the same VAE architecture and latent dimensionality, differing only in their foundation-model alignment configurations.
Specifically, we consider VAEs aligned with CLIP, DINOv2, DINOv3, and MAE using different alignment losses (cosine similarity and DMS), together with an unaligned baseline. The generation results are obtained from the publicly available benchmark [Diffusing in the Right Space: A Systematic Study of Latent Diffusability].
Correlations between iFID and gFID:
iFID achieves even stronger correlations within this controlled VAE family than across the diverse VAEs in our original benchmark. Since the VAE architecture and latent dimensionality are held fixed, these results substantially reduce the possibility that the observed predictive correlations arise primarily from architectural differences between VAE families.
Q3. Controlled evaluation for text-to-image generation