acceptodds
Under review as a conference paper at ICLR 2027

What Does Fréchet Distance Measure? A Directional Decomposition

Abstract

The Fréchet distance is a *de facto* standard for evaluating generative models across domains, appearing as FID for images and FVD for videos. It summarizes the discrepancy between generated and reference distributions in a single scalar, with lower values typically interpreted as better generation quality. However, this scalar view can obscure what drives the comparison. For example, in COCO dataset, increasing the number of diffusion sampling steps improves ImageReward scores yet worsens (increases) FID. Motivated by this mismatch, we seek to make the Fréchet distance more interpretable by uncovering *where* the discrepancy lies. To this end, we introduce *directional Fréchet distance*, the expected squared projection of the optimal transport displacement onto a given direction. Across our image, video, and protein case studies, we find that a small number of interpretable directions account for much of the distance. We use these directions to explain the FID increase in terms of semantic concepts represented by CLIP embeddings, quantify FVD's bias toward per-frame appearance, and revisit the interpretation of Protein FID.

Primary area:
generative models
Keywords:
Fréchet distance, Optimal transport, Generative model evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WPRMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper points out that the scalar FID score alone cannot explain the reasons behind the score, such as why increasing the number of diffusion sampling steps improves image quality while resulting in an increased FID score. To provide more interpretable explanation of the FID score, the paper proposes directional FD.

Critical strengths

I agree that the direction-based FID analysis method proposed in this paper may provide an good way to analyze how much different feature directions in the latent space contribute to the resulting FID score.

Critical weaknesses

CW1. It is difficult to agree that this paper fully addresses the questions people may have regarding the interpretation of the FID score. For example, regarding the phenomenon “additional denoising steps improve ImageReward but instead increase FID score”, the paper argues that "about two-thirds of the FID increase lies along a single direction (simplicity direction)" based on an observed relationship with CLIP representations. However, I believe that the observed relationship between the FID score and CLIP based interpretation alone is insufficient to support the paper's causal interpretation of this relationship, such as the claim that "the generated set differs from COCO by a shift along it, toward simpler, single-object images".

CW2. Another important factor for the reliability of the paper's analysis is how the 24 concepts are selected, as they serve as one of the main tools for interpreting the FID score. However, the paper does not explain how these concepts are selected, nor does it provide sufficient justification or analysis to demonstrate that the selected concepts independently and adequately capture the broad semantic factors that should be considered when interpreting FID score. This lack of justification makes the resulting interpretation potentially subjective.

Questions for authors

Q1. The criteria used to select the “24 concepts” require clarification in the main text, and the analysis method based on them (Line 200-214) also appears to be described in a way that is difficult to fully understand.

Q2. “Fr’echet distance” in Line 018 is a typo.

3 comments

Thank you for the careful reading, which caught the unclear lines and the typo. We answer each of your comments below, and refer to the general response where it covers a concern shared across reviews.

  • CW1. The causal reading of the simplicity direction

We agree that the paper does not answer every question about FID, since it explains a selected phenomenon (G4). Please refer to general response G1 for the validation of the name with dictionaries we did not write and with the COCO annotations. The finding has two parts, which need different support. The first part, that two thirds of the step-driven FID increase lies on e1e_1, uses no concept, since it is measured on MM alone. The second part, the name of e1e_1, rests on the CLIP concepts, and we agree that it needs support from outside them. The COCO instance annotations give this support without text prompts.

  • CW2. How the 24 concepts were chosen

Please refer to general response G1.

  • Q1. The selection criteria and the presentation of Lines 200-214

Please refer to general response G1 for the selection criteria. We agree that Lines 200-214 are hard to follow, and we will rewrite them step by step with the notation of G1, explicitly defining concept dictionaries, scores, and atoms.

  • Q2. The typo in the abstract

Thank you for catching it. We will correct "Fr'echet" to "Fréchet" in the abstract.

I view the idea of decomposing FID into directional components positively, as it provides additional interpretive insights into an otherwise single score. However, I remain uncertain about the extent of the paper's methodological contribution.

The primary role of FID is to rank models. Although lower FID values are also interpreted as indicating more realistic and diverse image generation, what matters most in practice is the relative ranking of models. However, rather than improving the accuracy or granularity of this ranking, the paper primarily adds a direction-based interpretation to already computed FID values.

I also have concerns about the validity of this interpretation. First, the semantic meaning of each direction does not originate from FID itself but is assigned through external tools, such as CLIP concept dictionaries. Second, from a user’s perspective, directions other than e1e_1, as well as directions obtained when evaluating other generative models, still require separate validation before their interpretations can be trusted. This limits the reliability of the analysis without additional user-side verification.

Therefore, despite the authors' sincere efforts to address my concerns, my overall assessment of the methodological contribution has not changed substantially.

Thank you for the follow-up, and for viewing the decomposition positively. We answer your concerns in turn and point to the corresponding changes, marked in green in the revised PDF.

  • The contribution relative to the ranking

We agree that ranking models is the primary role of FID, and directional FD leaves FID and its ranking unchanged. What it adds is an account of why the ranking comes out as it does. In practice, FID often acts as a black box. When a sampler setting raises FID, as in the sign reversal of Section 2.2, a developer can only try other settings, without knowing which difference FID charges. Directional FD shows which feature directions a ranking rests on, and so whether it reflects the property the developer cares about. We state this in the introduction.

  • Where the semantics come from

We agree that the names of the directions come from outside FID, and the paper separates what is intrinsic from what is assigned. Assigning semantics with external concept data is the standard path in interpretability, as in concept activation vectors [1], SpLiCE [2], and TextSpan [3]. On the other hand, our ImageNet-class dictionary needs no tool outside FID, since its concept directions are the rows of the Inception classifier that computes the FID features (Section 4.1). For a new direction or generator, the revised paper gives checks a user can repeat before trusting a name (Section 4.1, Appendices D and E) and reports where names fail.

[1] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, et al. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). ICML, 2018.

[2] U. Bhalla, A. Oesterling, S. Srinivas, F. P. Calmon, and H. Lakkaraju. Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE). NeurIPS, 2024.

[3] Y. Gandelsman, A. A. Efros, and J. Steinhardt. Interpreting CLIP's Image Representation via Text-Based Decomposition. ICLR, 2024.

Reviewer_b3B7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces directional Fréchet distance, an exact decomposition of Gaussian optimal-transport cost into contributions along feature-space directions, with the eigenvectors of the displacement second-moment matrix identifying where discrepancies concentrate. Through image, video, and protein case studies, the authors associate selected directions with domain-specific concepts to explain behavior that is obscured by the aggregate FD score.

Critical strengths
  • The paper addresses an important interpretability gap with a clear mathematical formulation. Rather than replacing FD with another scalar score, it provides an exact additive decomposition of the existing metric. This offers a principled way to investigate which feature-space discrepancies drive an evaluation result.
  • The cross-domain analyses provide useful and nontrivial diagnostic insights. The experiments examine image composition, video appearance versus flicker, and protein structural properties, with additional checks across reference datasets, feature embeddings, and distortion settings. These analyses demonstrate that similar aggregate scores can conceal substantially different sources of distributional discrepancy.
Critical weaknesses
  • The exact geometric decomposition does not establish equally reliable semantic attribution. Naming the directions depends on domain-specific concept dictionaries, grouping procedures, and sparse regression, while the resulting directions can mix correlated concepts and be only partially reconstructed. For example, Appendix D reports that the composition-simplicity concept direction accounts for approximately 39% of the step-related FID increase, compared with 66% for the eigenvector named after it. This supports an association but not an exclusive attribution to that concept. Stronger independent validation and sensitivity analysis are needed to establish how consistently the proposed labels reflect the discrepancies being measured.
  • The broader diagnostic utility remains insufficiently established within each domain. Although the cross-domain coverage and supplementary checks are valuable, the main analyses center on one generator per domain and use dictionaries or probes tailored to the selected phenomena. In the video study, for instance, the appearance and flicker bands cover only 18–38% of the actual generator’s FD, leaving most of its discrepancy outside the interpreted subspaces. Complete semantic coverage is not required, but the paper would be stronger with a systematic assessment of diagnostic coverage, failure cases, and transfer to additional generators or discrepancies not anticipated when constructing the probes. The current evidence establishes informative case studies more clearly than a broadly reliable procedure for interpreting generator differences.
Questions for authors
  • How stable are the semantic interpretations under changes to the concept dictionary, grouping thresholds, Lasso regularization, and sample resampling? For the image and protein studies, could the assigned labels be validated independently of the concept constructions used to obtain them?
  • Can the same predefined dictionaries or distortion probes be applied to additional generators to evaluate their diagnostic coverage and limitations without redesigning them for each observed discrepancy?
  • Appendix D reports that the dominant image eigenvector has cosine similarity 0.99 with the mean-difference direction and that its mean component accounts for 85% of its directional FD. Could the authors compare the diagnostic value of the proposed decomposition with simpler analyses of feature mean shifts, covariance differences, and directly measured concept statistics, clarifying which findings specifically require the directional formulation?
3 comments

Thank you for separating the exactness of the decomposition from the reliability of its semantic labels. We answer each of your comments below, and refer to the general response where it covers a concern shared across reviews.

  • W1. The 39% of the concept direction against the 66% of e1e_1

Please refer to general response G1 for the validation of the names, and to G3 for their sensitivity to the samples, the grouping quantile and the Lasso penalty. We agree that the 39% supports an association, not an exclusive attribution. The two numbers measure different directions. e1e_1 is the top eigenvector of MM, and carries 66% of the step-driven FID increase (slope 0.66 against FID). The composition-simplicity atom is built from the reference images without MM, has cosine 0.67 with e1e_1, and carries 39% (slope 0.39). The sparse fit reconstructs e1e_1 with cosine 0.74, with composition simplicity as its dominant atom (weight 0.55) beside smaller atoms such as depth of field (0.12) and furniture (0.08). e1e_1 thus mixes composition simplicity with correlated concepts, and the 24 atoms reconstruct it only in part. The name of e1e_1 therefore gives its dominant concept, not a complete account of it (G1, Scope). The evidence of G1 supports this dominant concept without our dictionary. Two external dictionaries describe the same contrast in their own words, and reference images further along e1e_1 have fewer annotated object categories. We will state in Section 4.1.1 that e1e_1 is only associated with composition simplicity.

  • W2. Diagnostic coverage and transfer

Please refer to general response G4 for the scope of the case studies and for other generators, and to G5 for the coverage of the video bands. We agree that our evidence establishes informative case studies more than a procedure that interprets every generator difference. The probes are tailored to each phenomenon, but they do not depend on the generator. We therefore assess their coverage, their failure cases, and their transfer with the probes unchanged.

The flicker band holds a small part of the FVD of the generated videos, but it tracks their temporal defects, measured independently by VBench (G5). The reconstruction cosine of the sparse fit (Table 1) and the image correlation (G1) report how much of an image direction the dictionary covers.

The five other references of Appendix E give discrepancies that the 24 concepts, written for COCO, did not anticipate. We apply the concepts to them unchanged. Composition simplicity leads the fit of e1e_1 on Flickr30K (weight 0.26) and MJHQ (0.21), and ties for the lead on ImageNet (0.28, with staged versus candid). The name of e1e_1 thus transfers to the three references of general images. The dictionary fails on the two face references. The leading atoms of e1e_1 there are people, vehicles and daytime on CelebA-HQ, and colorfulness and sports equipment on FFHQ, which do not describe how the generated faces differ from the reference faces. e1e_1 is the mean-difference direction on both references (∣cos⁡∣|\cos| 0.999), the gap between a general text-to-image model and a curated face set. A dictionary of scene and photographic concepts thus does not name the top direction of a face-only comparison.

We are running the COCO step sweep on the Flux models with the same dictionary, and will add the results during the discussion. We will report these failure cases in Appendix E.

  • Q1. Stability of the names, and validation independent of the concept constructions

Please refer to general response G1 for other concept dictionaries, and to G3 for the samples, the grouping quantile and the Lasso penalty. The COCO instance annotations validate the name of e1e_1 independently of the concept constructions, since they need no prompts and no atoms. Reference images further along e1e_1 have fewer annotated object categories (image correlation r=0.33r = 0.33), a test that also uses no CLIP. The annotations also measure the generated images: a ridge regression from CLIP image features to the log object count predicts about 20% fewer objects in every generated set than in COCO (G1).

In the protein study, the P-SEA [1] secondary structure of each structure validates the name of e1e_1 without the CATH atoms and the sparse fit. The Protein FID reference provides embeddings only, so we measure it across the 467 generated structures. Generated structures further along e1e_1 have less coil (Spearman correlation -0.72) and more helix (+0.75), as its coil-deficit name predicts. The same holds, more weakly, across the 31,885 real CATH domains (-0.37 and +0.32).

  • Q2. The same probes on other generators

Please refer to general response G4.

  • Q3. Comparison with simpler analyses

Please refer to general response G2.

[1] G. Labesse, N. Colloc'h, J. Pothier, and J.-P. Mornon. P-SEA: A New Efficient Assignment of Secondary Structure from Cα Trace of Proteins. Bioinformatics, 1997.

I thank the authors for their thoughtful and detailed response, which addresses several of my concerns and provides additional evidence supporting the paper's main claim. I appreciate the general applicability of the mathematical formulation across multiple domains. However, my main concern regarding the broader contribution and generalizable diagnostic utility of the proposed apporach remains insufficiently resolved. Therefore, I will maintain my original rating.

Thank you for reading our response, and for recognizing the generality of the formulation. On the generalizable diagnostic utility, we have since posted two results in the follow-up to the general response.

  • The same probes on other generators. We apply the image analysis, with the concept dictionaries unchanged, to FLUX.1-dev and its distilled variant FLUX.1-schnell. Each Flux e1e_1 lies close to SD1.5's (∣cos⁡∣\lvert\cos\rvert 0.87 and 0.88) and carries more of FID (23% and 26%). Composition simplicity leads its fit, and TextSpan and SpLiCE name the same contrast for all three models. e2e_2 also carries over, while e3e_3 is a small direction particular to each model.
  • The same analysis under a second seed. The top three eigenvectors and their names hold for a second draw of the sampling noise (∣cos⁡∣\lvert\cos\rvert 0.999, 0.994 and 0.988 with the first seed's).

The diagnostic thus transfers to new generators without redesigning the probes. We also report where a name does not transfer: e3e_3 across generators, and e1e_1 on the two face references (Appendix E, Table 9). Our contribution is an account of why a Fréchet metric moves, with the metric and its ranking unchanged; in each of our three case studies, it explains a discrepancy that the single number leaves open. Both results are now in Appendix D (Tables 7 and 8) of the revised PDF.

Reviewer_B6FMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript introduces directional Fréchet distance, which provides an interpretable perspective on what FID, FVD, and Protein-FID measure and how they measure it across image, video, and protein generation spaces. They propose an additive decomposition of Gaussian space using optimal-transport cost along the feature directions. They identify directions that contribute most to the distance and associate them with interpretable concepts.

Critical strengths
  • The paper presents an interesting take on Frechet Distance and the interpretability of one the most common metric used to evaluate generative models.
  • The directional decomposition is simple, they take the contributions over an orthonormal basis sum to the original Frechet distance.
  • It also separates differences in means from the differences in spread, which provides more information rather than just the scalar scores.
  • The authors show experiments across three domains - images, videos and proteins.
Critical weaknesses
  • The way each direction is assigned a meaning needs stronger validation. The directions are named using manually selected 24 concepts. These manually selected concepts may explain only part of what a direction captures, and it is unclear whether expanding the concept set would provide a complete interpretation. Showing images at the extremes and testing different concept dictionaries would help assess the proposed meanings, but these checks would not establish that the directions are fully explained by those concepts. The paper should clarify this limitation when drawing semantic conclusions.

  • In the image experiment, the dominant direction e1e_1 is almost aligned with the mean-difference direction, with a cosine similarity of 0.99. Its contribution is also largely driven by the mean shift. Comparing the proposed analysis with directly interpreting the mean-difference direction would clarify the additional insight provided by the decomposition.

  • Are the results consistent across different samples or generation seeds?

  • The experiments cover a limited set of generators. Do modern DiT-based models exhibit the same trends for image and video generation? It would also be useful to examine whether distilled and non-distilled models differ in how the distance concentrates across directions and what those directions represent.

  • For DINO-based FD, the paper only reports the interpretation of the dominant direction e1e_1, whose concept reconstruction is weaker than the corresponding Inception result. Reporting the contributions and interpretations of e2e_2 and e3e_3 would help clarify which findings persist across the embeddings.

  • Appendix E shows that different datasets exhibit different trends in FID. How does directional FD then isolate the effect of the reference distribution?

  • For generated videos, the appearance and flicker subspaces together account for only around 18–38% of the distance, leaving much of the discrepancy unexplained. Comparing their contributions with independently measured temporal defects would help establish whether the identified directions reflect actual problems in the generated videos.

  • The Protein FID interpretation relies on strongly correlated structural concepts, making their effects difficult to separate. A small contribution along the chosen alpha-versus-beta direction does not establish that missing beta sheets have little influence on the overall Protein FID. An analysis that considers these related properties jointly would strengthen the conclusion.

  • The authors should distinguish distribution mismatch from poor metric behavior more clearly. An increase in FID alongside an improvement in ImageReward does not by itself establish that FID is incorrect, since the two measure different properties. Similarly, concentration along a few directions may reflect genuine differences between the distributions rather than an undesirable metric bias.

Questions for authors

Please see the weakness section.

4 comments
AuthorsOpenReview ↗

Response to reviewer B6FM (1/2)

Thank you for the detailed review and the concrete checks you suggest for each case study. We answer each of your comments below, and refer to the general response where it covers a concern shared across reviews.

  • W1. Validating the concept names

Please refer to general response G1.

  • W2. e1e_1 versus the mean-difference direction

Please refer to general response G2.

  • W3. Consistency across samples and seeds

Please refer to general response G3 for a different set of samples. We are generating the SD1.5 COCO step sweep again with a second seed, and will add the results during the discussion.

  • W4. Modern and distilled generators

Please refer to general response G4.

  • W5. The second and third DINOv2 directions

The spectrum in DINOv2 space is steep, as in Inception space: e1e_1, e2e_2 and e3e_3 carry 14.0%, 3.9% and 3.3% of the distance. As in Inception space, e1e_1 is mostly a mean shift (89% of its directional FD), while e2e_2 and e3e_3 are almost entirely covariance part (1% and 3% mean part). The generated set spreads 3.4 and 1.8 times wider than the reference along e2e_2 and e3e_3. We read all three directions with the sparse fit over our concepts (Table 1), and with the image correlation over the external dictionaries of G1.

sharesparse fit (cos): surviving conceptsTextSpan, highest ∣r∣\lvert r \rvertSpLiCE, highest ∣r∣\lvert r \rvert
e1e_114.0%0.41: composition simplicity (+0.26), brightness (+0.11), has people (-0.07)"Nature macro photography" (+0.36), "An image of an interior of a room" (-0.25)"beautiful nature" (+0.39), "karaoke" (-0.26)
e2e_23.9%0.14: indoor/outdoor (+0.04), depth of field (-0.03)"An image of an interior of a room" (+0.17), "Energetic street scene" (-0.16)"new apartment" (+0.19), "street sign" (-0.20)
e3e_33.3%0.16: composition simplicity (-0.09)"Photograph taken in a cozy interior" (+0.27), "Close-up view" (-0.24)"night party" (+0.24), "team logo" (-0.20)

e1e_1 again reads as composition simplicity, with simple, nature-like images at the generated end and indoor scenes with people at the reference end. e2e_2 and e3e_3 read as indoor rooms against street scenes and close-ups, close to the indoor and outdoor contrast of the Inception e2e_2. Their fits are weaker than that of e1e_1 (cosine 0.14 and 0.16, against 0.41), so we state these readings with less confidence. We will add this table to Section 4.1.

AuthorsOpenReview ↗

Response to reviewer B6FM (2/2)

  • W6. What Appendix E isolates

Directional FD does not isolate the effect of the reference distribution alone in Appendix E. Each reference is compared with SD1.5 generations from its own captions, as in the standard protocol, so the six comparisons differ in the generated set as well as in the reference. What directional FD separates instead is the part of the step-driven change that the six comparisons share from the part that depends on each reference. The shared part is the motion of the generated mean: from 15 to 50 steps, it moves in a similar direction in all six comparisons (pairwise cosine 0.52 to 0.88), although their captions differ. The reference-specific part is how each reference's FID charges that motion, along its own top eigenvectors. The generated mean moves away from the reference along e1e_1 on COCO, Flickr30K, ImageNet and MJHQ, and toward it on CelebA-HQ and FFHQ. Directional FD also falls over the tail on the four non-face references (G2), so the head decides the sign of the FID change there. Appendix E thus shows how one shared change of the generator is charged differently by each reference, rather than the effect of changing the reference with the generator held fixed. We will state this distinction in Appendix E.

  • W7. Coverage of the video bands

Please refer to general response G5.

  • W8. Correlated structural concepts in Protein FID

We agree that the alpha-versus-beta direction alone does not settle how much Protein FID charges secondary structure, since the structural concepts are strongly correlated (Appendix G). We therefore measure the secondary-structure directions jointly: alpha versus beta, and the helix, strand and coil fractions. These four unit directions are nearly collinear: the top two singular directions of the matrix they form hold over 99% of its squared norm. We read their span by these two directions, since the other two are noise-like and would inflate the share. This two-dimensional subspace carries 31.9% of Protein FID. Secondary structure can thus be a significant part of the Protein FID gap. Most of this share comes from the second singular direction, which lies along e1e_1 (cosine 0.58), while the first carries 7.5%. The structural-regularity direction e1e_1 alone carries 66%, so it remains the main account of the gap, and secondary structure enters it mainly through the coil deficit that leads its fit. We will add the joint share to Section 4.3, and state there that secondary structure is a significant part of the gap.

  • W9. Distribution mismatch versus metric failure

We agree, and we will make this distinction explicit. An FID increase alongside an ImageReward improvement shows that the two metrics measure different properties, not that FID is wrong. Our image result supports this reading: two thirds of the step-driven FID increase lies on e1e_1, along which the generated images genuinely differ from COCO (G1). FID charges this difference in composition, while ImageReward rises with steps.

The contradiction of Section 2.2 arises from the common reading of a lower FID as a better generator, under which a method that lowers FID is taken to preserve quality. Under this reading, a disagreement between FID and a per-sample score looks as if one of the metrics were wrong, and prior work has sought to fix FID or to replace it with another scalar metric (Section 1). Our paper instead clarifies the disagreement by locating what FID charges, without assuming that either metric is wrong. Likewise, concentration on a few directions describes where the two distributions differ, and we do not read it as a bias of the metric. We use the term bias only for the content bias of FVD, where the distortions are controlled, so the embeddings can be compared on the same known difference. We will revise the introduction and the findings of Section 4.1 accordingly, stating that FID and ImageReward disagree rather than that FID fails.

I thank the authors for the detailed response and additional analyses. My concerns about the distinction between FID & ImageReward, protein and video interpretations, and the additional DINO directions are largely addressed.

I also read other reviewers comments and discussion and I strongly agree with Reviewer WPRM that the interpretations of other directions and their applicability to different generators still need separate validation. The additional dictionaries and COCO annotations strengthen the interpretation of the dominant image direction. However, the meanings assigned to these directions still depend on the concepts and validation used and I remain uncertain about how reliably these interpretations carry over to other cases.

The authors mention repeating the SD1.5 experiments with a second seed and running experiments with FLUX models, including a distilled variant. Can you point me to those results again? This would help assess stability across generation seeds and whether that the same interpretation procedure works across different generators.

For now, my score remains unchanged.

Thank you for reading the responses and the discussion. We have posted both results in the follow-up to the general response. In short:

  • Second seed. The second-seed SD1.5 sweep reproduces the analysis of Section 4.1. The top three eigenvectors have ∣cos⁡∣\lvert\cos\rvert 0.999, 0.994 and 0.988 with the first seed's, their shares match to the first decimal, and the fits of e1e_1 to e3e_3 keep their leading concepts, with composition simplicity leading e1e_1 (weight 0.56, against 0.55).
  • Other generators. We apply the same procedure, with the concept dictionaries unchanged, to FLUX.1-dev and its distilled variant FLUX.1-schnell. FID rises with the steps for both. Each Flux e1e_1 lies close to SD1.5's (∣cos⁡∣\lvert\cos\rvert 0.87 and 0.88) and carries more of FID (23% and 26%); composition simplicity leads its fit, and TextSpan and SpLiCE name the same contrast for all three models.

We agree that a name holds only as far as it is validated, and the same checks also show where a name does not carry over. e2e_2 carries over across the three generators, as a contrast of indoor scenes with outdoor public places. e3e_3 does not: it is a small direction particular to each model (about 2% of FID), and the dictionaries name it differently for each. Likewise, the dictionary fails to name e1e_1 on the two face references (Appendix E, Table 9). The checks of Appendix D thus tell a user when to trust a name, rather than assuming that it carries over.

Comments on the paper

6 comments
AuthorsOpenReview ↗

General Response (5/5): Coverage of the video bands

  • G5. Coverage of the video bands

The appearance and flicker bands hold 18% to 38% of the FVD of the generated videos (Appendix F), which raises the question of whether they reflect actual defects of those videos. The bands are probes for the content bias of FVD, built from controlled distortions, rather than a decomposition of the full distance (G4). A probe is not required to hold most of the distance to be useful, but it should track the defect it was built for. We therefore test whether the flicker band tracks the temporal defects of the generated clips, measured without the FVD embeddings.

Defects measured independently. We score each of the 2,048 ModelScope clips with two dimensions of VBench [1], motion smoothness and temporal flickering. They are computed by a frame-interpolation model and by frame differences, not by I3D or VideoMAE. Both scores also fall with large motion, so they measure temporal defects together with motion.

Per-clip agreement. Clips further along the top flicker direction score worse on both dimensions, under both embeddings. With the motion-blur bands, the Spearman correlation with motion smoothness is -0.47 under VideoMAE and -0.39 under I3D. With the elastic bands, it is -0.37 and -0.17.

The more and the less defective halves. We split the clips at the median motion smoothness and compute the FVD of each half against the same reference. Under VideoMAE, the more defective half has a larger FVD (430, against 323). It also puts 2.5 times more of its FVD on the flicker band (14.2%, against 5.6% at k=10k = 10), while its share on the appearance band stays at 9%. Under I3D, the more defective half has a smaller FVD (811, against 1062), and its flicker share does not rise (13.2%, against 19.6%). I3D-based FVD thus rates the more defective clips as closer to real videos, which is the content bias of Section 4.2 on generated videos. Splitting at the median temporal flickering gives the same picture.

Revision. We will add this test to Appendix F, and state in Section 4.2 that the bands are probes for the content bias rather than a decomposition of the full distance.

[1] Z. Huang et al. VBench: Comprehensive Benchmark Suite for Video Generative Models. CVPR, 2024.

AuthorsOpenReview ↗

General Response (4/5): Other generators and the scope of the claims

  • G4. Other generators and the scope of the claims

Each case study analyzes one generator, which raises the question of whether the analysis applies to other generators, including modern and distilled ones.

Why we present case studies. Directional FD is designed to explain a selected phenomenon of a Fréchet metric, rather than to interpret every generator in the same way. Each case study starts from a reported phenomenon that a single number does not explain: the sign reversal of FID with sampling steps, the content bias of FVD, and the reading of the Protein FID gap as missing beta sheets. The phenomenon decides which directions are worth interpreting. The image and protein studies ask which directions carry the distance, so they read the top eigenvectors of MM and name them with a concept dictionary. The video study asks whether the distance lies on the directions that flicker moves, so it builds those directions first, from controlled distortions, and measures the distance on them. For this reason the probes differ across the case studies, and we present each as its own study.

Within one phenomenon, the probes do not depend on the generator. The CLIP dictionary is built from the reference images, the video bands from distortions of the reference clips, and the protein dictionary from CATH domains.

New generators. We test whether the explanations carry over to other generators, with the probes unchanged. We are running the COCO step sweep on the Flux models, which are diffusion transformers and include a distilled variant. We will report the spectrum of MM, the name of e1e_1 under the same CLIP dictionary, and its slope against FID, and add the results during the discussion.

Revision. We will state in the introduction that directional FD is applied to a selected phenomenon, and that our findings are case studies of three such phenomena.

AuthorsOpenReview ↗

General Response (3/5): Stability of the semantic interpretation

  • G3. Stability of the semantic interpretation

We test whether the interpretation of the top image directions (Table 1) changes under three interventions on the analysis. In each case, composition simplicity remains the dominant concept of e1e_1, e2e_2 keeps its lighting and scene concepts, and e3e_3 keeps its food classes. The exception is a strong Lasso penalty, under which the fits of e2e_2 and e3e_3 keep no atom.

(a) A different set of samples. We subsample 10,000 of the 30,000 COCO captions once, without replacement, and repeat the analysis on their reference and generated images. We do not resample with replacement, since repeated samples bias the covariance estimates. The top three eigenvectors keep their directions (cosine 0.997, 0.983 and 0.970 with the full-data ones), and e1e_1 keeps its slope of 0.66 against FID. Its share falls to 18.8% from 24.0%, since the finite-sample bias of FID is larger at 10,000 samples (FID 18.73, against 14.64). The sparse concept fit of e1e_1 keeps its cosine (0.73, against 0.74) and its dominant atom, composition simplicity (weight 0.56, against 0.55). The fit of e2e_2 keeps its cosine (0.48, against 0.49) and its four leading atoms: indoor/outdoor, weather, brightness and crowd density. The ImageNet-class fit of e3e_3 keeps its cosine (0.38, against 0.39) and its leading food classes: Dungeness crab, mashed potato and burrito.

(b) The grouping quantile. Each atom aca_c is the difference of the mean features of the top and bottom qq of reference images by scs_c, with q=10q = 10\\% in the paper. The fit of e1e_1 keeps its cosine (0.74) at q=20q = 20\\% and 3030\\%, and composition simplicity remains its dominant atom (weight 0.58). The fit of e2e_2 also keeps its cosine (0.49) and its two leading atoms, brightness and weather. The ImageNet-class dictionary of e3e_3 is the rows of the Inception classifier, so it has no quantile.

(c) The Lasso penalty. We fit both dictionaries at α=2×10−5\alpha = 2 \times 10^{-5}, 5×10−55 \times 10^{-5}, 10−410^{-4} and 2×10−42 \times 10^{-4}. The fit of e1e_1 keeps between 10 and 2 atoms from α=2×10−5\alpha = 2 \times 10^{-5} to 10−410^{-4}, with reconstruction cosine 0.78 to 0.67. Composition simplicity is its dominant atom over this whole range (weight 0.46 to 0.55), brightness and weather lead the fit of e2e_2, and Dungeness crab, mashed potato and burrito lead the fit of e3e_3. At α=2×10−4\alpha = 2 \times 10^{-4}, composition simplicity is the only atom left for e1e_1, and e2e_2 and e3e_3 keep none.

Revision. We will add these checks to Appendix D.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.