acceptodds
Under review as a conference paper at ICLR 2027

When marginals match but structure fails: covariance fidelity in generative models

Abstract

Generative models are increasingly used to produce synthetic data for scientific analysis, but common evaluation criteria mainly assess likelihood, feature-wise marginal fit, or perceptual quality. These measures may miss changes in covariance structure, which directly affect procedures such as regression and principal component analysis. We study this issue using the covariance discrepancy between a target distribution and a generative distribution . Three results describe what this quantity can and cannot establish. First, identical univariate marginals can coexist with different copulas and covariance matrices, and the raw covariance discrepancy can grow arbitrarily under rescaling. Second, when marginal variances match, exactly determines the difference between two bivariate population regression slopes. Third, covariance perturbation bounds PCA eigenvalue and principal-subspace errors. The resulting Frobenius-based subspace certificate is informative when , where is the relevant population eigengap. Synthetic examples show that matched marginals can conceal a reversal in regression slope and differences in tail dependence that are not summarized by covariance alone. Applications to Fashion-MNIST and gene-expression data show that marginal, covariance, and downstream diagnostics can give different assessments. Observed sample errors are reported separately from theorem-based certificates. Since is scale-sensitive and captures only second-order structure, it is intended as a task-linked complement to FID, MMD, copula diagnostics, and direct downstream evaluation, rather than as a general measure of generative quality.

Primary area:
generative models
Keywords:
generative model evaluation, synthetic data, dependence fidelity, covariance fidelity, marginal fidelity, principal component analysis, matrix perturbation theory

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_s5qUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies evaluating models via the covariance discrepancy DΣ(P,Q)=∣ΣP−ΣQ∣FD_\Sigma(P,Q)=|\Sigma_P-\Sigma_Q|F, motivated by the observation that likelihood, marginal-fit, and perceptual criteria can miss covariance changes that directly affect regression and PCA. Three theoretical results characterize what DΣD_\Sigma can and cannot establish — marginals do not determine covariance, matched marginal variances make DΣD_\Sigma exactly determine the difference in bivariate population regression slopes, and covariance perturbation bounds PCA eigenvalue and principal-subspace errors, yielding a subspace certificate informative when DΣ(P,Q)/γr(P)<1/2D_\Sigma(P,Q)/\gamma_r(P)<1/2. Synthetic data show that matched marginals can conceal slope reversals and tail-dependence differences, and DΣD_\Sigma is positioned as a task-linked complement to FID, MMD, copula diagnostics, and direct downstream evaluation.

Critical strengths
  • The paper is relatively well structured and written, especially in the first half.
  • The authors make due effort to scope the work.
Critical weaknesses
  • There is a large gap between the linear theory and practical generative models.

    • Rezaei et al. (2026), the paper's main reference, concerns linear models, for which covariance-based analysis is natural and effective. A neural classifier often has a linear head, but that analysis's relevance to neural classifiers is a theoretically motivated heuristic supported by experiments. The theoretical gap remains substantial even with a linear final layer.
    • All of the above applies to the current submission, and even more so. Although the Limitations section acknowledges that "Covariance does not characterize nonlinear, conditional, higher-order, or tail dependence" and that the work is "for linear and spectral tasks," it would not be unreasonable to state bluntly that the current submission concerns only linear models, in both theory and experiments.
  • The experiments do not close the gap described above.

    • First, the experiments all appear to be purely synthetic (as suggested by the section title "SYNTHETIC EXAMPLES"); or are they?
    • Are neural networks used in any of them? Only the VAE-on-MNIST experiment seems to use neural networks — tho it may be drastically simplified.
    • The datasets appearing under EMPIRICAL SUMMARY are never explained, let alone the results. That subsection contains only a table, which is not appropriate for a main-text subsection.
  • The theory itself consists of elementary examples or straightforward corollaries of classical (textbook) results.

    • Theorem 1 is an elementary consequence of a stronger classical result. The underlying classical fact is that univariate marginals do not determine the joint distribution. Sklar’s theorem expresses the remaining freedom through the copula; with continuous marginals, that copula is unique. See Fermanian and Scaillet, §2, recalling Sklar’s theorem.

      Embrechts, P., McNeil, A. J., & Straumann, D. (2002). Correlation and dependence in risk management: Properties and pitfalls. Cambridge University Press. Theorem 4 in the author version addresses the possible correlations with fixed marginal distributions.

      Nelsen, R. B. (2006). An introduction to copulas (2nd ed.). Springer. https://doi.org/10.1007/0-387-28678-0. Relevant result: Sklar’s theorem, Theorem 2.3.3, §2.3.

    • Theorem 2 is an immediate algebraic identity. The standard population linear-projection formula is β(P)=Cov⁡P(X,Y)Var⁡P(X).\beta(P)=\frac{\operatorname{Cov}_P(X,Y)}{\operatorname{Var}_P(X)}. It requires finite second moments and positive predictor variance; it does not require Gaussianity, a linear conditional mean, or homoscedasticity. See Yen-Chi Chen’s regression notes, “Linear prediction”. And:

      Hansen, B. E. (2022). Econometrics. Princeton University Press. (This is an undergrad textbook.) §2.21, Theorem 2.10, p. 42, Eq. (2.40), gives the population regression coefficient with an intercept: β=Var⁡(X)−1Cov⁡(X,Y).\beta=\operatorname{Var}(X)^{-1}\operatorname{Cov}(X,Y).

    • Theorem 3 is also a standard consequence of known results: Eq. (18) is Weyl’s eigenvalue perturbation bound, and Eq. (20) follows from the Davis–Kahan theorem.

      Yu, Y., Wang, T., & Samworth, R. J. (2015). A useful variant of the Davis–Kahan theorem for statisticians. Biometrika, 102(2), 315–323. https://doi.org/10.1093/biomet/asv008. Relevant material: §1, Theorem 1 and the following operator-norm remark; §2, Theorem 2, develops bounds using only a population eigengap. This is particularly close to your formulation.

      Fan, J., Wang, K., Zhong, Y., & Zhu, Z. (2021). Robust high-dimensional factor models with applications to statistical machine learning. Statistical Science, 36(2), 303–327. https://doi.org/10.1214/20-STS785. §2.4 of the author manuscript states Weyl’s eigenvalue inequality and then Corollary 2.1, which gives the operator-norm sine-of-principal-angles bound with factor 2 and an eigengap from the unperturbed matrix only.

Questions for authors

See the questions about the experiments raised in the Critical Weaknesses section above.

Reviewer_BTpmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework for understanding covariance discrepancy across various tasks, such as regression and PCA. The proposed workflow is to (i) assess the marginal agreement between corresponding variables in real and generated data, (ii) assess that of covariance, and (iii) interpret the discrepancy relative to the downstream task type. The paper provides illustrative examples for when matching marginal distributions is insufficient for determining variable dependence. It also quantifies differences in regression slopes when the variances match and uses established bounds for interpreting PCA results. The framework is instantiated in various empirical settings, including experiments on Fashion-MNIST and gene expression data.

Critical strengths
  1. The focus on providing an interpretable diagnostic framework is useful for practitioners working with synthetic data, particularly whether aspects of the data distribution (like covariance) correspond to interpretable changes in e.g. regression slopes or PCA subspaces.
  2. The experiments are performed in a variety of domains, e.g. images and gene expression datasets, and are supported by intuitive and illustrative synthetic examples.
Critical weaknesses

The paper frames around the fact that matching marginals does not necessarily determine joint distribution. This is a well-established result from probability theory, and the framing of it as a main result (4.1) makes the novelty of the paper's contributions unclear. This is also corroborated by (4.2) following directly from the standard population regression formula, and (4.3) applying established perturbation bounds (from Weyl and Davis-Kahan). While the author's acknowledge their use of classical results, a significant portion of the paper is dedicated to such; it would have been stronger to focus more on downstream task analyses/implications rather than established theory results.

There also appear to be some discrepancies between the figures and prose or captions. For example, in Figure 10, most of the plotted KS distances are below 0.2, while the caption says the median is 0.26. Similar issue with Figure 15 and Table 5.

Questions for authors
  1. As 4.1-4.3 rely on classical results, what are the emphases of the paper to be focused on / novel methodological insights?
  2. What are the practical implications of the certificate analysis given that informative PCA bounds appear to be obtained only for fitted Gaussians?
  3. Is there a sense to when practitioners can determine whether a given discrepancy is acceptable? As even with an informative PCA bound there can still be significant changes in principal subspace
Reviewer_tu7SOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a new metric to evaluate the accuracy of samples produced by trained generative models for scientific purposes by looking at the Frobenius covariance. discrepancy DΣD_\Sigma and look at 1) marginal inadequacy, 2) regression sensitivity, 3) PCA and subspace stability.

Critical strengths
  • Clear motivation and problem statement: perceptual or marginal accuracy metrics for the quantity of produced samples is not sufficient for most scientific applications of generative modelling.

  • The math is clear and rigorous, and it establishes a solid foundation for the analytical identities used in the paper.

  • I appreciated the honesty of the authors in not overstating the sensitivity of their method and the rigorous with which they identified when upper bounds were vacuous.

Critical weaknesses
  • Appendix dumping: The organization of the manuscript should be severely reworked. The main text of the paper only contained a single, simple 2D synthetic example, while the entire empirical evaluation of the method (including Fashion MNIST, Alzheimer's gene expression, TCGA-BRCA BULK RNA-SEQ DATA, and all the associated heat maps, graphs, and tables) is dumped into appendices. I would expect a conference submission to be self-contained. Relegating core validation experiments to the supplement feels like an attempt to bypass the page limitations, and it is not acceptable formatting practice since it prevents referees to assess the paper's empirical substance within the body of the text.
  • Inadequate literature positioning and missing comparison baselines: while FID, MMD, and copulas are touched on, the paper ignores a vast literature on accuracy assessment and fails to systematically compare their method to state of the art metrics. Score and methods to test distributional accuracy are numerous in recent literature, for example Sliced Wasserstein Distance (SWD), Classifier Two-Sample Tests (C2ST/LC2ST), and Simulation-Based Calibration (SCB), especially projected along random axes to detect mis-matched covariances, and, importantly, recent scores like PQMass for unconditional generative models, MIRA for conditional generative models. Not including these comparisons systematically places the proposed method in a methodological vacuum and does not allow the reader to evaluate its advantages or disadvantages compared to existing methods.
  • Empirical and Diagnostic Scope Limitations: The evaluation presented fails to demonstrate performance on foundational stress-test required for a new generative model diagnostic tool beyond the ones already available, especially if the main motivation is for scientific applications. There is no demonstration of high-dimensional scaling, as the evaluation is bottlenecked on low-dimensional setups (Fashion MNIST flattened to 50 PCA components, small genomic feature sets with p=50 to 100), which doesn't show that it scales well and remains sensitive in truly high dimensional settings. It shows so study of cases of localized mode drop, minor structural modes missing in generative models, or sensitivity to memorization/overfitting behaviours. Moreover, while the authors note that resampling schemes may be needed when sample covariance is unstable, they completely omit any formal sample-complexity analysis, finite-sample concentration bounds, or empirical scaling curves showing how many samples are required across varying dimensions to reliably detect structural discrepancies.
Questions for authors
  • Can you provide comparisons to existing metrics used in the literature to compare the performance of your proposed method?
  • Could you provide guidance about number of samples needed for the metrics to be sensitive as a function of dimensionality of the data?
  • How would over fitting be detected and how sensitive would the metric be?
  • Can this metric be extended to conditional generative models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.