- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a Geometry-Aware Ornstein–Uhlenbeck (GOU) diffusion process for structured and multimodal data. The main idea is to construct a fixed anisotropic drift matrix from the eigendecomposition of the empirical data covariance. Low-variance directions are contracted more rapidly, whereas high-variance directions are preserved for longer during the forward process. For reverse-time generation, the paper approximates the finite-time terminal distribution with a Gaussian mixture whose components are centered around transformed structural anchors. The paper derives the evolution of the mean and covariance, analyzes mode separability, and provides a Wasserstein bound on the terminal initialization error. The method is intuitive, interpretable, and analytically tractable. However, I have significant concerns regarding its novelty, the scope of its theoretical guarantees, the lack of sufficiently strong baselines, its unverified scalability to genuinely high-dimensional data, and the computational cost associated with retraining the score model.
Critical strengths
- The drift matrix is explicitly derived from the covariance eigensystem, making the role of each principal direction easy to understand.
- Since the proposed process remains a linear OU process, its mean, covariance, and reverse-time dynamics admit explicit characterization.
- Preserving multimodal and correlation structure is important for scientific data, particularly genomic and single-cell data.
- When important modes are separated primarily along high-variance principal directions, the proposed contraction schedule may delay mode mixing and simplify reverse-time learning.
Critical weaknesses
- The paper derives the exact evolution of the first two moments, a finite-time mode-separation result, and an initialization-error bound. These results support the intuition behind the method, but they do not establish convergence of the practical generation procedure to the target distribution. With the exact score, exact simulation, and initialization from the true terminal distribution , the reverse-time SDE recovers by standard time-reversal theory. However, this correctness property holds for general sufficiently regular forward diffusions and is not specific to the proposed choice of . The practical method instead uses:
- a finite-sample estimate of and ;
- a learned score network;
- a GMM approximation based on anchors;
- a numerical discretization of the reverse SDE. The paper does not provide an end-to-end error bound that combines these sources of error. In particular, the initialization bound alone does not guarantee that the final generated distribution is close to . The paper should distinguish more clearly between guarantees inherited from standard diffusion time reversal and guarantees specifically provided by the proposed drift construction.
- The current experiments do not convincingly demonstrate scalability to genuinely high-dimensional settings. The method requires estimating and decomposing a empirical covariance matrix. When , the empirical covariance is rank-deficient and potentially unstable, while a full eigendecomposition and dense representation of may incur substantial time and memory costs. This concern is particularly relevant because the paper motivates the method using genomic and biological applications, which commonly involve tens of thousands or more features. The paper should include:
- experiments on substantially higher-dimensional datasets;
- runtime and memory scaling with and ;
- experiments in the regime;
- sensitivity to covariance-estimation error;
- comparisons using shrinkage covariance, truncated PCA, low-rank approximations, and randomized SVD;
- analysis of how covariance regularization affects generation quality. Therefore, my concern is not that the method has been proven incapable of generating high-dimensional data, but rather that its claimed high-dimensional applicability is currently unsupported.
- The empirical evaluation relies mainly on an isotropic diffusion baseline. This is not sufficient to establish the benefit of the proposed approach. Comparisons should include methods designed to incorporate anisotropy, covariance information, structured priors, or preconditioning. In particular, I recommend comparisons against:
- preconditioned diffusion sampling [1];
- anisotropic or matrix-valued diffusion methods [2];
- learned noise or covariance schedules [3]
- Although and are computed directly from data statistics and introduce little additional parameter-learning cost, changing the forward process also changes all intermediate marginal distributions and their scores. Consequently, a score model trained for a standard isotropic diffusion cannot generally be reused by simply replacing the drift during sampling. Thus, the method is not a plug-and-play sampling-time modification: it requires retraining, or at least substantial fine-tuning, of the score network under the new perturbation kernel. This is an important practical cost, particularly when a strong pretrained diffusion model is already available.
[1] Ma, H., Zhu, X., Feng, J., and Zhang, L. Preconditioned Score-Based Generative Models. International Journal of Computer Vision, 2025. arXiv:2302.06504.
[2] Voleti, V., Pal, C., and Oberman, A. Score-Based Denoising Diffusion with Non-Isotropic Gaussian Noise Models. arXiv:2210.12254, 2022.
[3] Kingma, D. P., Salimans, T., Poole, B., and Ho, J. Variational Diffusion Models. Advances in Neural Information Processing Systems (NeurIPS), 2021. arXiv:2107.00630.
Questions for authors
-
Is the proposed GOU process mathematically equivalent to applying a global linear transformation or whitening operation, training a standard diffusion model in the transformed coordinates, and mapping samples back? If not, what is the precise difference?
-
How much of the reported improvement is attributable to the anisotropic drift, and how much is attributable to the anchor-based GMM initialization?
-
Does every reported GOU result use a score network retrained from scratch under the GOU perturbation kernel? Can a pretrained isotropic score model be reused or efficiently fine-tuned?
-
What are the runtime and memory costs of estimating, storing, and applying for large ? Is represented densely, or is the eigen decomposition used implicitly?
-
How stable is the method when and the empirical covariance is rank-deficient?
-
Why should global high-variance directions generally coincide with directions that are important for mode preservation? What happens when the discriminative structure lies in a low-variance subspace?
3 comments
To answer the questions raised by the reviewers regarding: Empirical Evaluation; Scalability to Genuinely High-dimensional; How much of the reported improvement is attributable to the anisotropic drift, and how much is attributable to the anchor-based GMM initialization? we have conducted complementary simulations in our revised version (Please see Appendix; Table 4 and Table 5):
We compare seven approaches: GOU, isotropic diffusion with Gaussian initialization, isotropic diffusion with Gaussian-mixture initialization, truncated-PCA diffusion, and tabular adaptations of PDS (Ma et al., 2025), NI-VP (Voletiet al., 2022), and VDM (Kingma et al., 2021). All methods use the same feature panel, data splits, and training and sampling budgets. Generation quality is evaluated using seven distributional and structural metrics. Results for Paul15 on 500 and 4000 features (high-dimensional setting as n=2700< <d=4000) are represented in Table 4 and Table 5 in the Appendix. GOU well outperforms other methods on most metrics and performs comparably to the strongest baselines on the rest. Our comparison shows that, although method rankings vary across metrics, GOU performs consistently well overall. Furthermore, comparison with isotropic diffusion using Gaussian-mixture initialization suggests that GOU’s advantages extend beyond the initialization strategy. In particular, GOU outperforms the isotropic GMM baseline especially on structural metrics, as it was expected. Similar observations are obtained for other datasets, which will be included in the final version of the paper. We sincerely thank the reviewer for their input.
Drift matrix construction on GOU performance: We have now also added a new section to our Appendix reporting our observations on the computation of matrix M as below:
Impact of drift matrix construction on GOU performance To examine how constructing the drift matrix M affects GOU, we compare full dense covariance eigendecomposition, full-spectrum dual decomposition, and randomized approximations with ranks and on a high-dimensional Paul15 gene panel with 2,700 cells and 4,000 genes; clearly a high-dimensional setup. Each configuration trains a separate score model while keeping preprocessing, drift mapping, architecture, anchor centers and weights, and training and sampling budgets fixed. We assess generation quality and paired changes relative to full dense GOU, alongside computation time and memory usage. Our empirical results showed that randomized approximations with a rank of 512 achieve the best average performance. While providing a marginal change in accuracy, it substantially reduces the storage and computational costs of matrix M (see also lines 212-215 of the paper for a quick discussion).
We hope that our responses have addressed the reviewer's questions and concerns. Please let us know if any points require further clarification.
We have also conducted a quick simulation to compare runtimes in different methods:
Comparing Runtimes
We compare methods using mean total computation time and generation time per 1,000 samples across random seeds. Total time includes method-specific setup, training, initialization, sampling, and evaluation, whereas generation time includes initialization and sampling. For biological data, GOU requires 380 s in total and 55 s per 1,000 generated samples. The corresponding times are 390 s and 58 s for NI-VP, 500 s and 50 s for VDM, 350 s and 50 s for isotropic diffusion with Gaussian initialization, and 50 s and 15 s for truncated PCA. These results show that GOU achieves competitive runtime relative to several baselines, with lower total computation time than NI-VP and VDM, although truncated PCA remains substantially faster but much weaker in performance overall.
We thank the reviewer for the careful assessment and for recognizing the interpretability and analytical tractability of our approach. We do our best to respond to your questions point by point:
Is the proposed GOU process mathematically equivalent to applying a global linear transformation or whitening operation?
In general, no. GOU assigns different contraction rates to different principal directions, whereas standard isotropic diffusion uses a shared contraction rate. A fixed invertible linear transformation, including whitening, cannot introduce these direction-dependent contraction rates. This selective contraction can help preserve mode separation when it aligns with slowly contracting directions. Thus, GOU changes the diffusion dynamics beyond a change of coordinates.
How much of the reported improvement is attributable to the anisotropic drift, and how much is attributable to the anchor-based GMM initialization? To address the reviewer's question, we have conducted additional empirical studies comparing GOU with alternative approaches, including isotropic diffusion with a mixture-of-Gaussians initialization. Our new findings clearly demonstrate that the performance advantage of GOU stems not merely from the initialization strategy, but fundamentally from respecting the underlying geometry of the data (please see Lemma 1 and Remark 1). We will integrate these results into the revised manuscript.
Does every reported GOU result use a score network retrained from scratch under the GOU perturbation kernel? Yes. All reported GOU results use score networks trained from scratch under the GOU perturbation kernel. Simply replacing the drift during sampling does not generally yield a valid GOU reverse process, because the intermediate marginal distributions and their scores change. Fine-tuning a pretrained model under the GOU objective is a potential alternative.
Costs of estimating, storing, and applying (M) The cost depends on how M is represented. For (d) features, a dense matrix requires O(d2) memory and O(d2) computation per drift evaluation. A full eigendecomposition costs O(d3), but is performed only once before training. For large datasets, (M) can instead be represented using (r≪d) leading covariance directions, with a shared contraction rate for the remaining directions. This reduces storage and per-vector drift computation to O(dr), while retaining all (d) dimensions. We have now mentioned these points in our revised manuscript; thanks for raising this question.
How stable is the method when d≫n: Rank deficiency does not make the GOU drift singular, because our construction does not invert the empirical covariance. Directions with zero empirical variance receive a contraction rate (-b). Thus, the forward dynamics remain stable. However, when d≫n, the estimated covariance may poorly represent the population geometry; shrinkage or low-rank covariance regularization can help address this statistical issue. We will make sure to include this discussion in the paper.
Why should global high-variance directions generally coincide with directions that are important for mode preservation : When clusters are far apart along a particular direction, their separation increases the variance in that direction. This motivates GOU’s slower contraction along high-variance directions, which can help keep clusters separated for longer. However, differences between clusters can also lie in low-variance directions, which the global drift may contract too rapidly. A promising direction for the future is to investigate locally adaptive diffusion using local means and covariances to better capture structure that the global covariance may overlook.
Convergence of GOU: Thank you for raising this point. As mentioned, GOU admits a corresponding backward process under standard time-reversal assumptions, providing a principled foundation for structured data generation. Our main contribution is a geometry-aware diffusion framework for structured data generation, using an anisotropic drift designed to mitigate the decay of feature correlations and delay mode merging during the forward process. Our empirical studies on biological data validate the power of this approach for generating highly structured data. The present analysis characterizes its moment dynamics and directional separation properties and provides a terminal initialization-error bound.
As already acknowledged in the manuscript, a comprehensive non-asymptotic analysis of the practical sampler is planned as a separate, in-depth future study. Further theoretical developments and extensions to higher-order structures remain directions for future research. We will make sure to more clearly state these points in the revised version.