- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper points out that the scalar FID score alone cannot explain the reasons behind the score, such as why increasing the number of diffusion sampling steps improves image quality while resulting in an increased FID score. To provide more interpretable explanation of the FID score, the paper proposes directional FD.
Critical strengths
I agree that the direction-based FID analysis method proposed in this paper may provide an good way to analyze how much different feature directions in the latent space contribute to the resulting FID score.
Critical weaknesses
CW1. It is difficult to agree that this paper fully addresses the questions people may have regarding the interpretation of the FID score. For example, regarding the phenomenon “additional denoising steps improve ImageReward but instead increase FID score”, the paper argues that "about two-thirds of the FID increase lies along a single direction (simplicity direction)" based on an observed relationship with CLIP representations. However, I believe that the observed relationship between the FID score and CLIP based interpretation alone is insufficient to support the paper's causal interpretation of this relationship, such as the claim that "the generated set differs from COCO by a shift along it, toward simpler, single-object images".
CW2. Another important factor for the reliability of the paper's analysis is how the 24 concepts are selected, as they serve as one of the main tools for interpreting the FID score. However, the paper does not explain how these concepts are selected, nor does it provide sufficient justification or analysis to demonstrate that the selected concepts independently and adequately capture the broad semantic factors that should be considered when interpreting FID score. This lack of justification makes the resulting interpretation potentially subjective.
Questions for authors
Q1. The criteria used to select the “24 concepts” require clarification in the main text, and the analysis method based on them (Line 200-214) also appears to be described in a way that is difficult to fully understand.
Q2. “Fr’echet distance” in Line 018 is a typo.
3 comments
I view the idea of decomposing FID into directional components positively, as it provides additional interpretive insights into an otherwise single score. However, I remain uncertain about the extent of the paper's methodological contribution.
The primary role of FID is to rank models. Although lower FID values are also interpreted as indicating more realistic and diverse image generation, what matters most in practice is the relative ranking of models. However, rather than improving the accuracy or granularity of this ranking, the paper primarily adds a direction-based interpretation to already computed FID values.
I also have concerns about the validity of this interpretation. First, the semantic meaning of each direction does not originate from FID itself but is assigned through external tools, such as CLIP concept dictionaries. Second, from a user’s perspective, directions other than , as well as directions obtained when evaluating other generative models, still require separate validation before their interpretations can be trusted. This limits the reliability of the analysis without additional user-side verification.
Therefore, despite the authors' sincere efforts to address my concerns, my overall assessment of the methodological contribution has not changed substantially.
Thank you for the follow-up, and for viewing the decomposition positively. We answer your concerns in turn and point to the corresponding changes, marked in green in the revised PDF.
- The contribution relative to the ranking
We agree that ranking models is the primary role of FID, and directional FD leaves FID and its ranking unchanged. What it adds is an account of why the ranking comes out as it does. In practice, FID often acts as a black box. When a sampler setting raises FID, as in the sign reversal of Section 2.2, a developer can only try other settings, without knowing which difference FID charges. Directional FD shows which feature directions a ranking rests on, and so whether it reflects the property the developer cares about. We state this in the introduction.
- Where the semantics come from
We agree that the names of the directions come from outside FID, and the paper separates what is intrinsic from what is assigned. Assigning semantics with external concept data is the standard path in interpretability, as in concept activation vectors [1], SpLiCE [2], and TextSpan [3]. On the other hand, our ImageNet-class dictionary needs no tool outside FID, since its concept directions are the rows of the Inception classifier that computes the FID features (Section 4.1). For a new direction or generator, the revised paper gives checks a user can repeat before trusting a name (Section 4.1, Appendices D and E) and reports where names fail.
[1] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, et al. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). ICML, 2018.
[2] U. Bhalla, A. Oesterling, S. Srinivas, F. P. Calmon, and H. Lakkaraju. Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE). NeurIPS, 2024.
[3] Y. Gandelsman, A. A. Efros, and J. Steinhardt. Interpreting CLIP's Image Representation via Text-Based Decomposition. ICLR, 2024.
Thank you for the careful reading, which caught the unclear lines and the typo. We answer each of your comments below, and refer to the general response where it covers a concern shared across reviews.
We agree that the paper does not answer every question about FID, since it explains a selected phenomenon (G4). Please refer to general response G1 for the validation of the name with dictionaries we did not write and with the COCO annotations. The finding has two parts, which need different support. The first part, that two thirds of the step-driven FID increase lies on e1, uses no concept, since it is measured on M alone. The second part, the name of e1, rests on the CLIP concepts, and we agree that it needs support from outside them. The COCO instance annotations give this support without text prompts.
Please refer to general response G1.
Please refer to general response G1 for the selection criteria. We agree that Lines 200-214 are hard to follow, and we will rewrite them step by step with the notation of G1, explicitly defining concept dictionaries, scores, and atoms.
Thank you for catching it. We will correct "Fr'echet" to "Fréchet" in the abstract.