- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This work investigates whether estimates derived from local covariance spectra can be interpreted as intrinsic dimension. It distinguishes estimating a spectral quantity from establishing that this quantity equals the true dimension. The paper studies the sample size needed to verify geometric assumptions and gives examples where different latent dimensions produce observed distributions with the same covariance matrix. Finally, it proposes a statistic for detecting deviation from a reference noise model and reports experiments on synthetic data and transformer representations.
Critical strengths
The paper addresses a interesting question. A stable estimate can still measure the wrong quantity, and this distinction matters when interpreting dimension estimates from latent representations. The experiments vary the noise model while keeping the structure fixed. This provides a useful way to illustrate how spectral estimates can change for reasons unrelated to a change in dimension.
Critical weaknesses
-
The presentation falls below the standard of clarity expected of an ICLR submission. Despite careful reading, I found the abstract and introduction very difficult to follow. From the start, the paper heavily relies on non-standard expressions such as “warrant,” “licenses the readout,” and “calibrated gate,” while moving rapidly between geometric assumptions, statistical guarantees, and empirical interpretations. The reader is then asked to interpret before the relevant objects and assumptions have been defined and explained.
-
About exposition, the order of definitions creates further difficulties. Section 3.1 introduces sample-size thresholds about , , , and before presenting the model. Some quantities are defined later in Theorem 3.1, its concentration assumption refers forward to Theorem 5.3, and the definition of effective rank appears in Appendix R. Hence, understanding any central statement requires to jump between the main text and several appendices.
-
Several claims are overstatred. Theorem 5.5 states that a high-probability failure claim for every covariance method. However, Theorem H.2 establishes that probability bound only for the max-gap rule. The general statement in H.2 and the probability guarantee for any method should to be clearly separated.I also find that the empirical claims need some clarification: the introduction claims that the experiments show that the spectral target fails to equal the dimension. However, later in the introduction and in Section 8, the authors acknowledge that stability has not been proved for real transformer data. Hence, the claims should distinguish clearly between the conclusions made by synthetic experiments and those by the real transformer data.
-
Proposition K.1 equivalence looks false: it states that , then concludes that iff all tangent directions are detectable and no curvature directions are detectable. I believe it is false because and also yield .
-
The lower-bound proof uses an incorrect probability calculation. Appendix F.2 suppose that the modified region contains a fraction of the points. This is true for a flat disk, but adding a bump changes the region’s surface area and therefore its sampling probability. I don't think the proof takes that into account, and Appendix G.3 repeats the same assumption. As a result, the exact lower bound claimed in the paper is not established. A similar bound may still be possible, but the proof needs to be corrected.
Overall assessment: Clear rejection.
The abstract and introduction rely heavily on abstract terminology and figurative language before clearly explaining the statistical problem and the assumptions behind the main claims. Definitions, constants, and conditions are repeatedly introduced after they are used or spread out across later sections and appendix. This put a lot of burden to the reader to move back and forth in order to simply understand the main results. Despite careful reading, I found it very difficult to determine precisely what is assumed, what is proved, and which conclusions are supported by the experiments.
More seriously, the main lower-bound proof uses an unjustified probability calculation. Appendix F.2 assigns the bumped region the same sampling probability as a flat region, ignoring that the bump changes its surface area. Appendix G.3 repeats this assumption. The exact lower bound is therefore not established by the provided proof. This gap in a central result, beyond the clarity issues, motivates my soundness score of 1. As clarity and precision are requirement for writing any accessible and assessable theoretical results by the research community, I do not consider the manuscript currently appropriate for acceptance at ICLR.
Questions for authors
- Does the probability bound in Theorem 5.5 apply to every covariance method? If so, could you explain where is this result proved? as theorem H.2 appears to prove it only for the max-gap rule.
- Given the stability limitations reported in Section 8, can you explain what can be concluded about incorrect dimension targets on the real transformer data? Could you distinguish this from the synthetic experiments ?