- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies the question under which conditions a trained generative model can be used to estimate the intrinsic dimensionality of the data manifold. Perturbing this data-manifold with Gaussian (or uniform) noise with variance results in noisy observations with density . Existing methods estimate by calculating which equals to under suitable conditions. The paper extend this density-slope method to non-Gaussian noise kernels (Theorem 2.2). Moreover, they derive conditions under which the Jacobian of the posterior mean can be used for estimating (Corollary 2.4).
The authors derive explicit expressions of the density-slope estimator for different generative models (Table 1) and compare their performance on different toy benchmarks. A given generative model with non-Gaussian perturbation kernel can be used to estimate without retraining as demonstrated on two real-world examples from chemistry and biology.
Critical strengths
State-of-the-art generative models such as diffusion models add noise to the data manifold in a controllable manner; the noise kernel is typically Gaussian but can be non-Gaussian in some applications. In the latter case, using the density-slope method for estimating requires retraining models with different noise magnitudes. The master Theorem, Theorem 2.2, generalizes the density-slope method to non-Gaussian kernels allowing to estimate without retraining. For practitioners, Table 1 provides a useful look-up table to turn a trained generative model to an intrinsic dimensionality estimator. The decomposition of the density-slope estimator as response term plus correction is interesting and potentially useful to study differences of generative models.
Critical weaknesses
The paper lacks clarity on different levels. Mathematically, for the theory crucial assumptions are either only described or deferred to the exhaustive appendix (A1, A2, and A3). How restrictive those assumption are - and which models/kernels fulfill those assumption or not - is not discussed in the main text. Intuitions for the proofs are not provided such that without reading the appendix it is difficult to assess correctness of mathematical results.
Section 4.3 starts with the question whether a trained model for its own domain provide useful local geometry through response. But useful information can be deduced from the density-slope method alone with diffusion models providing the necessary score without retraining as well. In Table 2, for the Gaussian models, the density-slope method (referred to as full) performs better than the response method except for the rectified flow. A principled discussion when to use response or when to use the density-slope method is missing (see Q1 below). Figure 1 is a good starting point for providing intuition, but in practice, whether a point is interior or a boundary point is unclear.
The experimental results in Section 4 are difficult to comprehend. LID-Benchmarks are not explained. Justifications for choosing the specific model families (Table 1) are not provided. MAE score in Table 3 is not clearly explained. Results are of qualitative nature; knowing that lower or higher is better helps comparing the different models, but to which extend the task of estimating is solved is left unanswered. The only true reference is a constant estimator . Overall, Section 4 is very dense in information. For example, authors write "For known LID, a disjoint source-train holdout selects each checkpoint’s scale by primary-readout MAE." which I could only understand with the appendix. But this sentence is crucial as it basically says that the known is used for estimating itself. Other expression such as "primary readout”, “dependent readouts”, “inherit Full’s selected scales”, “suite-balanced relative-MAE aggregates”, “pointwise Kneedle” require plain language descriptions to be understandable. So does Section 4.3 which is inaccessible for non-expert on the crystals or proteins.
Questions for authors
-
Assume practitioners have a problem which can be solved using a generative model. They are interested in local informations as well. Which generative model should they use and why? Which ID method, density-slope or response, should they use and why? What are the determining factors of the underlying problem determining the choice?
-
In Section 4.3, why did you just report the response and not the density-slope? DiffCSP is a diffusion model so score should be available? Or do I misunderstand something?
-
Which noise kernels fulfill the necessary conditions for the theory to apply? You mentioned a few in the introduction (e.g. wrapped-normal), are those admissible? How restrictive are the assumptions more generally?
-
Please explain in very simple terms what experiments 4.3 is measuring. What is the data dimension , what is the ID . What are the noise kernels. Do they fulfill the necessary assumptions yes or no. What are the estimates . How do they compare with other methods.
Further clarification requests:
- Line 138-139. It is not clear to me why a projection onto yields a Jacboian trace of . Please explain
Suggestions/Remark:
- Line 124 is confusing as stated and I think even wrong. The local dimension does not describe how the probability of finding a data points changes as radius tends to zero; this depends on the probability itself (uniform with flat curvature will not change).
- I think the paper would benefit from providing more intuition. After Proposition 2.1, Figure 1 could explain the involved terms and their differences. Then, briefly introducing two important generative models - diffusions and flow matching - and explaining what they provide (score or velocity). This motivates why both estimators are important to be unbiased (Proposition 2.1 itself does not).
- I'm not sure how useful a Table 1 is for the main text. I'd rather see one clear derivation sketch for a standard model (e.g. diffusion) together with a discussion on the conditions under which it converges, computational complexity etc.
- I suggest motivating the different model variants. Why did you decide using those and not others?
- I'm not sure if the title accurately addresses the papers contribution. The density-slope method already suggests that generative models are data dimensionality estimators. .