acceptodds
Under review as a conference paper at ICLR 2027

Default Kernel Distances Miss Covariance Shape at the Sample Sizes in Use

Abstract

Practitioners compare generative models by distances between embedded sample sets: the Fr\'echet family, and kernel metrics such as the Kernel Inception Distance (KID) and the Kernel Audio Distance (KAD), which were proposed to avoid its Gaussian assumptions and each ship with a fixed default kernel inherited from the two-sample testing literature. Which differences these defaults can and cannot detect on real embedding distributions has not been measured, to our knowledge, so a practitioner reading a reported value cannot know which differences it reflects; we provide that measurement and the reporting protocol that follows from it. We treat each configuration as a statistical test and inject known distortions into real embedding distributions from symbolic music, audio, and vision. Neither default responds to a trace-preserving covariance change (a rotation) of equal or larger transport magnitude than a mean shift it always detects, and the two fail differently. KAD's median-heuristic bandwidth detects location and scale: it rejects an isotropic contraction matched in transport to the mean shift in every run in every space, and against rotations it is sample-inefficient rather than insensitive, reaching full power at sample sizes that vision datasets afford and per-genre symbolic subsets do not. KID's cubic kernel detects the mean shift and little else: it misses the matched contraction in most runs, shows no recovery at the sample sizes we reach, and does not reject on a real model against its own training data once the location component is removed. The Fr\'echet distance, run through the same permutation protocol, detects that difference. Aggregated multi-kernel tests restore power against shape at nominal level, except against coverage losses constructed to leave the mean unchanged, and near-duplicate samples corrupt split-based calibration unless the construction controls for them. These measurements give configuration choice a tested basis and yield a reporting protocol that separates three questions a single scalar conflates: whether distributions differ, how they differ, and whether samples are copied.

Primary area:
generative models
Keywords:
generative model evaluation, two-sample testing, maximum mean discrepancy, kernel methods, permutation tests, statistical power, Fréchet Inception Distance, Kernel Inception Distance, Kernel Audio Distance, music and audio generation, memorization.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_zy7gOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes to investigate how divergence measures as commonly applied with default implementation detect synthetically created shifts in the embeddings on public datasets. Namely trace preserving covariance changes. It shows that tests miss these synthetic shifts or require large sample sizes. MMDAug provides better power and a battery of tests can be used to identify specific tests.

Critical strengths

A number of different embeddings spaces are tested. The papers description of the methodology is reproducible.

Critical weaknesses

The first critique is that the entire paper reads like an modern LLM-AI wrote it. I find it hard to parse with strangely succient but unclear phrasing. For example what does "matched in transport" mean? The authors should work to expand the writing for clarity, using consistent and simple word choices for describing the methdology.

The second critique is that it is not clear why rotation in embedding space is a meaningful shift. Also some rotations may end up with data alignment. Likewise what physical meaning would 'isotropic contraction' have in the embddding space? Because this is not a physically meaningful change (i.e., there is no change to the actual input) it cannot be called groudn truth. When rotation is only for some principal component dimensions it is also not clear that the data distribution itself may be nearly iostropic in those directions so any rotations will create the same distribution.

Questions for authors

"real model against its own training data once the location component is removed" what does removing the location component mean?

what are 'coverage losses constructed to leave the mean unchanged'

why would near-duplicate samples exist ('controlled fraction of training-set copies')?

Why would rotation be present in per-genre symbolic subsets?

Its not clear why in the reproducibility statement that the earlier version of the harness was discussed, wouldn't only the corrected version be included in the results?

2 comments

Dear Reviewer,

You wrote that the paper was hard to parse and that its phrasing was unclear. We have revised the manuscript to improve its clarity, and we have adjusted its structure so that it better reflects our contribution and the method that follows from it. The original experiments and their findings are unchanged. Terms such as "matched in transport" are replaced by plain descriptions. Rotations and contractions are presented as known transformations, not as ground truth (Section 3.3), and Section 4.3 reports how much each one changes the distribution. Appendices A.3, A.13 and A.24 answer your questions about centering, cluster removal and injected copies.

Thank you for your questions and suggestions, which have made this a better paper. If you find the revision clearer, we would be grateful if you would consider raising your score towards acceptance.

The authors

I think the authors for the update. While the paper is certainly clearer, its significance and contribution in actual generative evaluation is wholly lacking. Section 7 proposes a way to reference calibration which to me seems like a form of dataset augmentation to enable kernel selection. However, it is only test on synthetic transformations and still does not discuss whether this for of reference-calibrated kernel selection actually improves power in practice compared to kernel learning/selection with train/test splits, or standard kernel size selections.

Reviewer_QsDoOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors develop a comparison framework for various kernel distances (primarily KID and KAD) to understand what distributional differences these distances can and cannot detect for finite sample sizes. This is supported by permutations tests applied in their experiments, which include synthetic perturbations to various embedding distributions from image, audio, and music domains. Empirical insights include that KID and KAD are less sensitive to covariance rotations for smaller sample sizes and that KAD's sensitivity scales better with sample size (in their tested settings); these results are qualified by intuition from the MMD kernels used. The authors additionally analyze aggregated kernel tests (MMDAgg) and FID, which they show can be more sensitive to discrepancies like covariance rotation and recentering compared to KID and KAD.

Critical strengths
  1. As the metrics examined are widely used (especially FID & KID), the pertinence of understanding what properties within realistic embeddings spaces they are sensitive to is high for practitioners.
  2. The synthetic perturbation tests on embeddings are done so for a variety of domains (image, audio, symbolic music) and multiple embedding models, which helps solidify the conclusions on metric sensitivity. This is also corroborated by the results from synthetic recentering in the generated vs reference data experiments from section 8.
Critical weaknesses
  1. It is unclear to me what the core contribution is, as the contributions section is dense and hard to parse through; there is no clear delineation of a unifying framework (typically instantiated in a "Methods" or "Framework" section prior to empirical results). Mentions of a proposed framework or tests are instead expanded on much later in the paper (the conclusion refers to section 8 as where recommendations lie for spectrum diagnostics) --- these proposals should be moved and consolidated earlier. I recommend revising "Protocol" to separate formalization and the experimental setup, the latter of which it is focused on currently.

  2. The paper would greatly benefit from better situating their results within the existing (and very extensive) literature on kernel distances. The distances focused on for interpretation (especially FID and KID) are well-established and have thus been examined significantly in prior work. For example, the referenced Bischoff et al. (2024) paper is described to have comparisons that "exist at one fixed kernel and sample size." However, reading this motivating paper, there are many ablations on sample size, bandwidth, dimensionality, choice of kernel, etc., which would be helpful to contrast to the author's conclusions.

Minor: The formatting for the contributions section is inconsistent -- is (5) intended to be highlighted? -- and should be shortened for clarity. I was confused by the inclusion of a "Retractions" paragraph in the concluding section. The way it is framed seems to be retracting two hypotheses proposed in the paper. It is unclear if that is the case (in which case the soundness of conclusions are weaker and they should not be framed affirmatively in the main text), if this is reflective of some possible confounding variables in the experiments (in which case should be expanded on), or if this is correcting intuition from the research process (which should not be included, or if so, provided justification for why interesting to mention). "In use" (as used in the title & conclusion) implies some correspondence of the conclusions within the paper and realistic evaluation setups for practitioners, which are not elaborated on.

Questions for authors
  1. What is the explicit delineation of previously established results and intuitions about FID, KID, & KAD and the author's findings? Which conclusions differ from prior expectations from existing work on MMD sensitivity to e.g., bandwidth or sample size?
  2. Can you expand upon the implications of the "Retractions" section?
  3. Regarding the generated vs reference data experiments (in section 8 and table 9), it seems that the metrics are sensitive to differences before recentering, and the highlighted failure modes are only apparent if embeddings are synthetically recentered. What are the implications of this choice for realistic settings (i.e., unmodified model outputs)?
2 comments

Dear Reviewer,

The revised manuscript is now uploaded, and its structure follows your suggestions. The introduction states the main finding and four contributions. Section 3 separates the methodology from the experimental setup, and the proposal is consolidated in Section 7. Related Work corrects our description of Bischoff et al. and separates established results from our measurements. The "Retractions" paragraph is removed, and Section 5.2 shows kernel-dependent detection without recentering.

Thank you for your suggestions on structure and related work, which have made this a better paper. If these changes resolve the points you raised, we would be grateful if you would consider raising your score towards acceptance.

The authors

I thank the authors for the effort in the revision and manuscript. Given the significance and extent of the changes, I believe another round of review is necessary to properly assess the revised work.

Reviewer_XBpTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the sensitivity of the default kernels used in KID and KAD to different kinds of distributional changes. The authors construct controlled distortions in real embedding spaces and evaluate detection power using permutation tests. They find that the default configurations are effective at detecting mean shifts but often have much lower power against covariance changes. The paper also examines sample-size effects, near-duplicate contamination, and alternatives based on aggregated kernels and the Fréchet distance.

Critical strengths

The experimental study is fairly thorough. Rather than relying only on synthetic Gaussian distributions, the authors test multiple embedding spaces across symbolic music, audio, and vision. Matching distortions by Wasserstein distance is a reasonable attempt to make the sensitivity comparisons fair. I also appreciate that they examine statistical power and empirical test level rather than interpreting raw metric differences alone.

The distinction between KID and KAD is useful. The results show that the median-bandwidth Gaussian kernel can eventually recover power with more samples, whereas the cubic kernel remains weak on several covariance alternatives over the sample sizes examined. The analysis of near-duplicates is another useful contribution, particularly the demonstration that conclusions based on split-normalized z-scores can be misleading.

Critical weaknesses

My main concern is the practical significance of the identified failures. The controlled distortions clearly demonstrate differences in statistical power, but it is less clear how frequently these differences affect actual generative-model evaluation. In Table 9, nearly all unmodified model comparisons are detected by the default configurations. The more interesting differences appear after removing the mean component through recentering. While this is a useful diagnostic experiment, recentering is not part of typical model evaluation. I would like to see stronger evidence that these weaknesses lead to incorrect conclusions about real generative models. The ranking reversals in Appendix A.13 are interesting, but without an independent quality criterion, it remains unclear which ranking is preferable.

I am also not fully convinced by the recommendation to use MMDAgg as the default test. Section 9 shows that the permutation-calibrated Fréchet distance successfully detects several differences missed by KID and KAD, including the recentered real-model comparisons. Meanwhile, MMDAgg also fails on some coverage-loss constructions in Appendix A.11. This raises the question of what important distributional failures MMDAgg detects that a properly calibrated Fréchet test does not. A comparison on higher-order distribution changes with matched first and second moments would help clarify the advantage of aggregation.

Another concern is the interpretation of effective dimensionality. Section 10 suggests an effective-dimensionality boundary for sensitivity to covariance changes, but Table 8 complicates this explanation. Even after reducing Inception embeddings to an effective dimensionality of 12.8, the median heuristic detects the rotation in only 4/20 runs. In contrast, the Adversarial-MidiBERT encoder, with an effective dimensionality of 11.2, achieves 20/20. This suggests that effective dimensionality alone may not explain the difference, and that other properties of embedding geometry are important.

Finally, the methodological novelty is somewhat limited. The dependence of MMD test power on kernel choice and bandwidth is well established, and MMDAgg is an existing method. The contribution is primarily the systematic empirical characterization of these effects in generative-model evaluation. This is useful, but the paper could make a stronger case for what genuinely new understanding its experiments provide beyond the existing two-sample testing literature and recent work on diagnosing generative evaluation metrics.

Questions for authors

Check Weaknesses

1 comment

Dear Reviewer, We have uploaded a revised manuscript. Your review asked for evidence of practical relevance, and the revision adds two experiments you proposed: detection on mixtures of unmodified generated and reference data (Section 5.2), and a comparison of distributions with matched low-order moments (Section 5.4). Our recommendation no longer relies on MMDAgg (Section 7). We removed the effective-dimension boundary, which your counterexample showed does not hold (Section 6).

Thank you for the experiments you proposed, which have strengthened the paper. If the revision addresses your concerns, we would be grateful if you would consider raising your score towards acceptance.

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.