When marginals match but structure fails: covariance fidelity in generative models

Generative models are increasingly used to produce synthetic data for scientific analysis, but common evaluation criteria mainly assess likelihood, feature-wise marginal fit, or perceptual quality. These measures may miss changes in covariance structure, which directly affect procedures such as regression and principal component analysis. We study this issue using $D_{\Sigma }(P,Q)=\Vert {\Sigma }_P-{\Sigma }_Q{\Vert }_F,$ the covariance discrepancy between a target distribution $P$ and a generative distribution $Q$. Three results describe what this quantity can and cannot establish. First, identical univariate marginals can coexist with different copulas and covariance matrices, and the raw covariance discrepancy can grow arbitrarily under rescaling. Second, when marginal variances match, $D_{\Sigma }$ exactly determines the difference between two bivariate population regression slopes. Third, covariance perturbation bounds PCA eigenvalue and principal-subspace errors. The resulting Frobenius-based subspace certificate is informative when $D_{\Sigma }(P,Q)/{\gamma }_r(P)<1/2$, where ${\gamma }_r(P)$ is the relevant population eigengap. Synthetic examples show that matched marginals can conceal a reversal in regression slope and differences in tail dependence that are not summarized by covariance alone. Applications to Fashion-MNIST and gene-expression data show that marginal, covariance, and downstream diagnostics can give different assessments. Observed sample errors are reported separately from theorem-based certificates. Since $D_{\Sigma }$ is scale-sensitive and captures only second-order structure, it is intended as a task-linked complement to FID, MMD, copula diagnostics, and direct downstream evaluation, rather than as a general measure of generative quality.