acceptodds
Under review as a conference paper at ICLR 2027

OUTPUT WITHOUT WARRANT: SPECTRAL DIMENSION READOUTS CANNOT CERTIFY THEIR OWN HYPOTHESES

Abstract

Local intrinsic dimension is routinely read off the local covariance spectra of neural representations. We distinguish a readout's output, the index it returns, from its warrant, the grounds for reading that index as a dimension. The output is cheap: for any data law, a spectral readout recovers its own population target at a sample size set by effective rank and spectral margin. The warrant is not. The hypotheses under which that target equals the dimension are lower bounds on reach, which no procedure can certify from a window of fewer than samples under any hypothesis-independent observation channel. Moreover, the covariance does not identify dimension at all, so a stable readout can be wrong for every dimension but one. In planted windows matched to BERT and GPT-2 representations, the readout tracks where a decaying spectrum crosses the noise edge, moving by two orders of magnitude at fixed structure. We replace the dimension claim with a narrower one: a profile-depth statistic, calibrated against an anisotropy-matched null with measured power, that certifies detectable structure but not its dimension.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
local intrinsic dimension, large language models, transformer representations, sample complexity, spectral dimension, null models, signal-to-noise ratio, local geometry

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iVH3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work investigates whether estimates derived from local covariance spectra can be interpreted as intrinsic dimension. It distinguishes estimating a spectral quantity from establishing that this quantity equals the true dimension. The paper studies the sample size needed to verify geometric assumptions and gives examples where different latent dimensions produce observed distributions with the same covariance matrix. Finally, it proposes a statistic for detecting deviation from a reference noise model and reports experiments on synthetic data and transformer representations.

Critical strengths

The paper addresses a interesting question. A stable estimate can still measure the wrong quantity, and this distinction matters when interpreting dimension estimates from latent representations. The experiments vary the noise model while keeping the structure fixed. This provides a useful way to illustrate how spectral estimates can change for reasons unrelated to a change in dimension.

Critical weaknesses
  1. The presentation falls below the standard of clarity expected of an ICLR submission. Despite careful reading, I found the abstract and introduction very difficult to follow. From the start, the paper heavily relies on non-standard expressions such as “warrant,” “licenses the readout,” and “calibrated gate,” while moving rapidly between geometric assumptions, statistical guarantees, and empirical interpretations. The reader is then asked to interpret before the relevant objects and assumptions have been defined and explained.

  2. About exposition, the order of definitions creates further difficulties. Section 3.1 introduces sample-size thresholds about C0C_0, ρT\rho_T, κT\kappa_T, and r(Σ)r(\Sigma) before presenting the model. Some quantities are defined later in Theorem 3.1, its concentration assumption refers forward to Theorem 5.3, and the definition of effective rank appears in Appendix R. Hence, understanding any central statement requires to jump between the main text and several appendices.

  3. Several claims are overstatred. Theorem 5.5 states that a high-probability failure claim for every covariance method. However, Theorem H.2 establishes that probability bound only for the max-gap rule. The general statement in H.2 and the probability guarantee for any method should to be clearly separated.I also find that the empirical claims need some clarification: the introduction claims that the experiments show that the spectral target fails to equal the dimension. However, later in the introduction and in Section 8, the authors acknowledge that stability has not been proved for real transformer data. Hence, the claims should distinguish clearly between the conclusions made by synthetic experiments and those by the real transformer data.

  4. Proposition K.1 equivalence looks false: it states that k^→d++m+\hat{k}\to d_+ + m_+, then concludes that k^→d\hat{k}\to d iff all tangent directions are detectable and no curvature directions are detectable. I believe it is false because d+=d−1d_+=d-1 and m+=1m_+=1 also yield k^→d\hat{k}\to d.

  5. The lower-bound proof uses an incorrect probability calculation. Appendix F.2 suppose that the modified region contains a fraction ℓ/r)d\ell/r)^d of the points. This is true for a flat disk, but adding a bump changes the region’s surface area and therefore its sampling probability. I don't think the proof takes that into account, and Appendix G.3 repeats the same assumption. As a result, the exact lower bound claimed in the paper is not established. A similar bound may still be possible, but the proof needs to be corrected.

Overall assessment: Clear rejection.

The abstract and introduction rely heavily on abstract terminology and figurative language before clearly explaining the statistical problem and the assumptions behind the main claims. Definitions, constants, and conditions are repeatedly introduced after they are used or spread out across later sections and appendix. This put a lot of burden to the reader to move back and forth in order to simply understand the main results. Despite careful reading, I found it very difficult to determine precisely what is assumed, what is proved, and which conclusions are supported by the experiments.

More seriously, the main lower-bound proof uses an unjustified probability calculation. Appendix F.2 assigns the bumped region the same sampling probability as a flat region, ignoring that the bump changes its surface area. Appendix G.3 repeats this assumption. The exact lower bound is therefore not established by the provided proof. This gap in a central result, beyond the clarity issues, motivates my soundness score of 1. As clarity and precision are requirement for writing any accessible and assessable theoretical results by the research community, I do not consider the manuscript currently appropriate for acceptance at ICLR.

Questions for authors
  1. Does the probability bound in Theorem 5.5 apply to every covariance method? If so, could you explain where is this result proved? as theorem H.2 appears to prove it only for the max-gap rule.
  2. Given the stability limitations reported in Section 8, can you explain what can be concluded about incorrect dimension targets on the real transformer data? Could you distinguish this from the synthetic experiments ?
Reviewer_dEKaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when local covariance spectra can be interpreted as intrinsic dimension. It distinguishes convergence to a spectral population target from the geometric assumptions needed to identify that target with dimension. The authors present stability results, a lower bound for certifying a reach condition from a local sample, and constructions where different latent dimensions yield the same observed covariance under different observation channels. They also propose a profile-depth statistic, calibrated against an anisotropy-matched null, to detect structure rather than estimate its dimension. The evaluation combines synthetic experiments, analyses of BERT and GPT-2 representations, and a re-analysis of a previous embedding study.

Critical strengths
  1. The distinction between a stable spectral output and a justified dimension estimate is useful and clearly motivated. The combination of the certification question and the covariance non-identifiability construction provides a framework for understanding why numerical stability alone is insufficient. This is the paper’s strongest conceptual contribution.

  2. The combination of controlled experiments with known structure and audits of real neural representations gives the methodological argument practical relevance. Calibrating the proposed diagnostic against anisotropic noise and explicitly examining its detection power are valuable steps towards more interpretable claims about representation geometry.

Critical weaknesses
  1. Theorem 4.2 is vacuous as stated. Soundness (S) ranges over every M∈M(d,L3)M \in \mathfrak{M}(d, L_3) with τ(M)<τ⋆\tau(M) < \tau^\star, where τ\tau is the global reach (Definition E.1(i)), but the test sees only the observations in Wr(x)W_r(x). Modifying any M0M_0 from (P) only outside Bˉ(x,r)\bar{B}(x,r) (e.g., placing the G.3 bump at a distant point) gives τ(M′)<τ⋆\tau(M') < \tau^\star with an identical window law, so PM′(T=1)=PM0(T=1)≥1−δ>δP_{M'}(T=1) = P_{M_0}(T=1) \geq 1-\delta > \delta. Hence no test satisfies (S) and (P) at any nrn_r: the bound (2d−1)log⁡12δ(2^d-1)\log\frac{1}{2\delta} holds only vacuously, and Remark G.3's claim that this constant is the problem's value does not hold. The intended result needs a local quantity in (S), e.g., curvature or local feature size on Wr(x)W_r(x).

  2. The exact lower bound needs a corrected justification. The constructions in Appendices F.2 and G.3 use a cap probability of at most (ℓ/r)d(\ell/r)^d, giving 2−d2^{-d} at ℓ=r/2\ell=r/2, under uniform sampling with respect to manifold volume. However, introducing a nonflat graph changes the volume element through its Jacobian, so the tangent-ball volume ratio does not by itself establish the required probability bound. The argument also needs to cover the baseline manifold allowed by Theorem 4.2. Exponential scaling may survive a repair, but the quantitative certification bound is a central contribution and is not fully justified as written.

  3. The interpretation of non-detection appears stronger than the reported calibration supports. Remark 7.2 and Appendix N.4 state that a noise-equivalent window either has no local structure or contains structure of dimension at least 32 within 1.75 times the noise edge. The reported tests on particular planted spectra, including dimensions 32 and 64, do not establish that these alternatives exhaust the possibilities. The scope of this conclusion matters because the calibrated gate is the paper’s proposed practical alternative to dimension estimates.

Questions for authors
  1. Could you provide a corrected cap-mass argument for Theorems 3.2 and 4.2 that accounts for the manifold volume element and verifies that the constructed alternatives satisfy the stated assumptions? Does the exact bound (2d−1)log⁡(1/(2δ))(2^d-1)\log(1/(2\delta)) remain unchanged?

  2. What precise class of alternatives is covered by the gate’s power calibration? In particular, what excludes undetected structures of dimension below 32, or with spectra different from those tested, in the disjunction of Remark 7.2 and Appendix N.4? Can this conclusion be justified for the stated class, or should it be restricted to the calibrated families?

Reviewer_xxonOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies local intrinsic dimension estimates obtained from the covariance spectrum of neural representations. Theoretically, it argues that the number itself can be estimated reliably, while the assumptions needed to interpret it as a dimension cannot be verified from the same data. Empirically, on synthetic data matched to BERT and GPT-2 representations, it shows that the estimate can depend on the noise level rather than on the underlying structure. As an alternative, the authors propose a statistic that detects whether structure is present, without estimating its dimension, calibrated against a null model matched to the data. They also re-analyse a published study and identify errors that explain part of its reported discrepancies.

Critical strengths

The paper addresses a relevant question, as intrinsic dimension estimates are widely used to analyse neural representations. The distinction between the estimate and its interpretation is clear, and the authors are careful about the scope of their claims. Additionally, the re-analysis of a published study is a useful contribution, as it identifies concrete errors that can be verified.

Critical weaknesses
  1. The main text might be difficult to follow, and some elements seem to be missing. The ball-mass exponent, used in Section 3 and 8, does not seem to be defined or referenced anywhere in the paper. Some labels, such as regime (R2) in Corollary 4.3 or condition (N) in Section 7, do not correspond to anything introduced in the main text. Central notions such as the reach are used without a definition or an intuitive explanation, which might make the main results hard to interpret for a broader audience. A more self-contained presentation would improve clarity, and it is currently difficult to assess some claims from the main text alone. Some statements also seem to refer to elements that do not exist in the paper. For instance, Section 3.2 refers to an “old constant” and Appendix F.1 to an “earlier form” of the bound, neither of which appears in the paper. Remark G.3 refers to a caveat in Section 8 that does not seem to be there.

  2. Minor remark: the abstract and Section 8 state that the readout varies by two orders of magnitude at constant structure, while the range reported in Section 8 (from 3 to 48) seems to correspond to roughly one order of magnitude. The authors should clarify this.

Questions for authors

Could the authors clarify the elements that are currently missing or unclear? (See Weaknesses)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.