acceptodds
Under review as a conference paper at ICLR 2027

What Does TwoNN Measure? Sample-Resolution Laws in High Dimensions

Abstract

TwoNN estimates intrinsic dimension from ratios of first- and second-nearest-neighbor distances. In high dimensions, even nearest neighbors may lie outside the small neighborhoods underlying manifold-dimension interpretations. We study the pooled inverse-mean-log-ratio estimator as sample size and dimension grow together. For anisotropic Gaussian data under stated spectral assumptions, we prove concentration around an explicit function of sample size and covariance eigenvalues, accounting for dependence between ratios computed from the same sample. For uniform points on a noiseless sphere, polynomially many samples make TwoNN report a vanishing fraction of the true dimension. We also bound spherical mean-spacing approximation error and shallow Gaussian target-estimation error. Gaussian and spherical experiments test the phase laws. In a separate controlled Gaussian study, the trace-moment estimator closely recovers its oracle shallow target, while observed TwoNN retains finite-resolution and sampling error. In five CIFAR-10/ResNet-18 models, covariance-aligned rescaling fitted on disjoint images increases TwoNN by a median factor of 2.20 while preserving every prediction; random orientations with identical singular values give a factor of 0.995. Direct simulations from fitted global-Gaussian and class-Gaussian-mixture nulls compare neural TwoNN with those nulls: the mixture moves closer to real TwoNN at one tested layer and farther away at the other. TwoNN can therefore consistently track a sample-resolution-dependent target without recovering population manifold dimension. In the tested neural representations, its value should be interpreted together with coordinate controls and explicit distributional nulls.

Primary area:
learning theory
Keywords:
intrinsic dimension, TwoNN, nearest-neighbor estimation, high-dimensional statistics, sample complexity, representation geometry

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_N6ZFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies what pooled TwoNN measures in high-dimensional finite-sample regimes. For anisotropic Gaussian data, the authors show that TwoNN concentrates around an explicit sample-size- and covariance-spectrum-dependent effective dimension, rather than necessarily recovering the population dimension. They further derive a corresponding sample-resolution law for uniform data on a noiseless sphere and show that polynomially many samples recover only a vanishing fraction of the true dimension as the dimension grows.

The paper validates these phase laws experimentally on Gaussian and spherical data, studies finite-sample target estimation, and examines neural representations from CIFAR-10 ResNet-18 models. In the neural experiments, function-preserving coordinate transformations can substantially change TwoNN while leaving predictions unchanged, and fitted Gaussian and class-Gaussian-mixture null models do not fully account for the observed TwoNN values. Overall, the paper argues that TwoNN should be interpreted as a sample-resolution-dependent quantity whose value depends on the representation geometry and metric, rather than as a direct estimate of population manifold dimension.

Critical strengths

S1. The paper provides an explicit characterization of what TwoNN measures in high-dimensional, finite-sample regimes.
Rather than merely pointing out finite-sample bias, it shows that pooled TwoNN converges, for Gaussian data, to a sample-size- and spectrum-dependent effective dimension deff(n,Σ)d_{\mathrm{eff}}(n,\Sigma). This gives a substantially more precise interpretation of the estimator.

S2. The paper clearly distinguishes population manifold dimension from sample-resolution-dependent effective dimension.
The noiseless-sphere result is particularly compelling: even when the true manifold dimension is known exactly, polynomially many samples may allow TwoNN to recover only a vanishing fraction of it in high dimensions. This provides a clean theoretical demonstration of the resolution limits of nearest-neighbor-based ID estimation.

S3. The theoretical results are supported by careful finite-sample experiments.
The Gaussian and spherical experiments test the proposed phase laws directly, while the later-neighbor analysis and neighbor-window correction help separate resolution effects from approximation and target-estimation errors.

S4. The neural representation experiments provide a strong practical stress test of the usual interpretation of TwoNN.
By applying invertible, function-preserving coordinate transformations and compensating the classifier accordingly, the paper shows that TwoNN can change by roughly a factor of two while all predictions remain unchanged. This makes the dependence of TwoNN on representation geometry and metric structure especially concrete.

Critical weaknesses

W1. The main theoretical results do not directly cover the low-intrinsic-rank regime that is often of primary interest in manifold-based LID estimation.
In the Gaussian analysis, the assumptions imply that the covariance rank grows as Θ(D)\Theta(D), while fixed-rank or sublinear-rank settings are explicitly outside the asymptotic regime considered. This leaves some uncertainty about how the proposed sample-resolution law extends to the more classical setting in which data lie near a manifold of dimension d≪Dd \ll D. Clarifying whether the same qualitative phenomenon is expected in such low-rank or noisy-manifold regimes would help establish the broader scope of the theory.

W2. The neural coordinate-intervention experiment is informative, but its interpretation depends on what invariances an intrinsic-dimension estimator is expected to satisfy.
The invertible transformations preserve the network function and the information contained in the representation, but they also change the Euclidean metric geometry on which TwoNN is based. Therefore, a substantial change in TwoNN under such transformations is not entirely surprising for a distance-based estimator. The result is still a useful stress test, but its implications would be clearer if the paper more explicitly discussed which classes of coordinate transformations an intrinsic-dimension estimate should ideally be invariant to. In addition, the neural experiments are limited to CIFAR-10, ResNet-18, five trained models, and a specific intervention site, so the generality of the observed effect remains open.

Questions for authors

Q1 (related to W1).
Do the authors expect the sample-resolution phenomenon identified in this paper to extend to low-intrinsic-rank or noisy-manifold settings with d≪Dd \ll D? A brief discussion of this regime would help clarify the scope of the theory.

Q2 (related to W2).
What class of coordinate transformations do the authors believe an intrinsic-dimension estimator should ideally be invariant to? Clarifying this point would help interpret the function-preserving coordinate-intervention experiment.

Q3.
The paper interprets TwoNN as measuring a sample-resolution-dependent finite-radius dimension rather than the population manifold dimension. This seems closely related to recent density-/diffusion-based LID estimators such as LHSD [1] and LIDL [2], where the effective resolution is instead controlled by a density-estimation or smoothing/noise scale. Could the authors discuss how these two lines of work are related, possibly in the Related Work or Discussion section? In particular, is the exponential sample requirement derived here specific to nearest-neighbor estimators, or should one expect an analogous resolution--sample-complexity tradeoff for density-/diffusion-based LID estimation?

[1] Osada, Genki. "Local Hessian Spectral Filtering for Robust Intrinsic Dimension Estimation." ICML 2026. [2] Tempczyk, Piotr, et al. "Lidl: Local intrinsic dimension estimation using approximate likelihood." ICML 2022.

2 comments
AuthorsOpenReview ↗

Low Rank, Coordinate Invariance, and Density-Based Estimation

We thank the reviewer for their positive assessment and the questions about the scope of the results. We use the review's W1/W2 and Q1/Q2/Q3 labels.

1. Low-rank Gaussians and noisy manifolds (W1, Q1)

The Gaussian theorem also covers data whose intrinsic rank is much smaller than ambient dimension, provided that rank grows. For a Gaussian supported on an rr-dimensional subspace of RD\mathbb R^D, orthonormal coordinates on that subspace preserve pairwise distances and TwoNN values. If r→∞r\to\infty and the restricted covariance satisfies Theorem 3.1's assumptions, the sample-resolution law applies with log⁡n/r\log n/r, even when r≪Dr\ll D. Thus, growing low-rank Gaussian data are covered by applying the theorem in the dimension of their support.

For curved or noisy manifolds, TwoNN can report different dimensions depending on the neighborhoods reached by the sample. Equation (5) relates neighbor ratios to probability-mass growth over those neighborhoods. When the sampled radii are too large to resolve the local manifold, TwoNN may report a different dimension. Noise can also change mass growth at small radii by introducing variation normal to the manifold. Establishing corresponding convergence laws would require assumptions on curvature, density, noise, and dependence between sampled neighbors.

2. What invariance should TwoNN satisfy? (W2, Q2)

An estimator of population manifold dimension should have the same population limit under smooth coordinate changes with smooth inverses. Such transformations preserve manifold dimension, but can change Euclidean neighbor distances. TwoNN is invariant to translations, orthogonal transformations, and uniform nonzero scaling because these preserve neighbor-distance ratios. Anisotropic transformations generally alter those ratios. If distances are adjusted to account for the coordinate transformation, all pairwise distances and TwoNN estimates remain unchanged. Our experiments keep Euclidean distance after changing feature coordinates to measure how much the reported TwoNN value changes.

The additional contribution is the measured size and alignment dependence of that effect. Across five ResNet-18 models, PCA-aligned rescaling increases pooled TwoNN by 2.199-fold, whereas random orientations with the same singular values yield 0.995 (Table 2). We compensate the classifier weights with the inverse transformation, preserving the model's predictions. The original empirical-CDF implementation gives corresponding factors 2.348 and 0.994. The control isolates the effect of aligning the rescaling with covariance directions rather than changing the rescaling magnitudes. Theorem 4.1 also constructs a rise and fall in pooled TwoNN under invertible Gaussian contractions while population dimension stays fixed. A further test on one frozen ViT-Base/16 gives pooled factors 1.641 and 0.997 under PCA alignment and matched random orientations (Appendix P). The experiments quantify coordinate sensitivity in both tested architectures.

3. Relation to LIDL and LHSD (Q3)

TwoNN and the density-based methods are connected through the probability mass around a point. Let Gx(r)=P(∥Y−x∥≤r)G_x(r)=P(\lVert Y-x\rVert\le r). The Gaussian-smoothed density used in LIDL can be written as

pσ(x)=(2πσ2)−D/2∫0∞e−r2/(2σ2) dGx(r).p_\sigma(x)=(2\pi\sigma^2)^{-D/2}\int_0^\infty e^{-r^2/(2\sigma^2)}\,dG_x(r).

Thus, TwoNN's neighbor ratios and LIDL's likelihood changes depend on the same radial mass distribution, but probe it differently. TwoNN evaluates mass growth around radii selected by nn. LIDL varies the smoothing scale σ\sigma and estimates dimension from the resulting change in log likelihood. For local mass growth Gx(r)∝rdG_x(r)\propto r^d, the smoothed density scales as σd−D\sigma^{d-D} in the local regime. LHSD examines the Hessian spectrum of the smoothed log density, filtering directions associated with normal rather than tangent variation. Its statistic uses second spatial derivatives of that smoothed distribution.

This connection suggests a finite-resolution comparison across estimators. TwoNN selects radii at probability mass near 1/n1/n, which drives its exponential threshold. Density-based methods set the smoothing scale separately, while estimating the smoothed density and its derivatives from finite samples. One could study how the mass profile, sample size, ambient dimension, and smoothing scale jointly determine the reported dimension. The Gaussian and spherical large-deviation calculations in this paper offer starting points for analyzing such scale transitions, with additional work needed for density-estimation error and Hessian estimation. We have added this connection to Related Work.

Thank you for your detailed response. The strengths of this paper, as noted in my original review, remain clear. I also appreciate the clarification of the connection between TwoNN and LIDL through their shared dependence on radial probability mass.

Given the growing interest in density- and diffusion-based LID estimators, I believe that advancing the theoretical understanding of a classical estimator like TwoNN is a valuable contribution that merits acceptance. I therefore maintain my positive assessment.

Reviewer_cx7YOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This study examines the pooled inverse-mean-log-ratio variant of TwoNN under joint sample-size and dimensional asymptotics. It derives an explicit Gaussian spectral target and establishes pooled concentration despite anchor dependence. A spherical law demonstrates that polynomial sample sizes may capture only a negligible fraction of the true dimension on a noiseless manifold. Additional results address neighbor-spacing calibration and shallow Gaussian target estimation. Simulations assess phase accuracy and target estimation. Five-model CIFAR-10/ResNet-18 experiments evaluate prediction-preserving coordinate transformations and compare observed TwoNN with directly simulated fitted Gaussian and class-mixture comparators. The analysis is exploratory, and the experiments do not yield a universal interpretation regarding population dimension..

Critical strengths
  1. The study replaces a qualitative undersampling warning with an explicit asymptotic target and a shared-sample concentration argument. This change is significant because a fixed-anchor calculation alone does not justify the pooled estimate commonly computed from a single dataset.
  2. The noiseless-sphere result and separated empirical controls provide a precise interpretation. The study distinguishes target estimation, finite resolution, coordinate sensitivity, and fitted-null mismatch, and reports negative findings, including no substantial random-basis effect and layer-dependent failure of the class to explain neural measurements using mixtures.
Critical weaknesses
  1. In practice, transfer is restricted to a specific estimator and is effective only under particular distributional conditions. The theoretical framework does not directly address the original empirical-CDF method, scenarios involving fixed or sublinear intrinsic-rank Gaussian sequences, or general neural representations. Furthermore, it does not offer finite-sample uncertainty quantification for observed pooled TwoNN. These limitations constrain immediate applicability, despite the theoretical contributions.
  2. The primary neural result is exploratory and is based on a single dataset, a single architecture, one intervention site, and five trained models. The fitted-null discrepancies remain sensitive to the covariance-estimation method and to shrinkage, which precludes identifying a population-level geometric explanation. These factors limit the broader applicability of the findings, although the central model-specific theorems remain intact.
  3. The mathematical precision of the asymptotic target surpasses that observed in finite-sample applications. The auxiliary bounds address target estimation and expected spherical log spacings, rather than the bias and uncertainty associated with the observed reciprocal statistic. For n = D = 512, the observed TwoNN remains approximately 12–13 percent below the shallow-target estimator, with smaller yet persistent discrepancies relative to the full phase target (p. 7; Table 17). Additionally, the neighbor-window approximation increases the actual first/second-neighbor median absolute relative error from 2.17 to 3.97 percent (Table 1). While these findings do not undermine consistency, they diminish the immediate practical benefits.
  4. A specific mathematical overstatement appears on p. 30, line 1604. The text states (n\asymp\exp{c_(\alpha)D}), although the threshold theorem determines only the leading exponential rate. At the limiting target fraction, the appropriate formulation is [ n=\exp{c_(\alpha)D+o(D)}. ] For instance, multiplying the nominal exponential sample size by (\exp(\sqrt D)) preserves the limiting exponent but causes the multiplicative ratio to diverge. Therefore, the displayed asymptotic-equivalence notation requires correction.
  5. The neural experiments do not provide quantitative validation for the proposed spectral explanation. The Gaussian comparator predicts a 3.338-fold principal component analysis effect, compared to the approximately 2.124-fold effect observed. Directly simulated fitted Gaussian and mixture comparators continue to exhibit substantial, layer-dependent discrepancies; mixtures improve agreement at one layer but worsen it at another. Covariance shrinkage significantly alters the comparator, and the reported surrogate Monte Carlo uncertainty does not account for covariance-fitting uncertainty.
  6. The experiments demonstrate coordinate sensitivity and disagreement with certain fitted distributions. However, they do not establish that the Gaussian theorem accounts for the neural statistic. The theorem pertains to the pooled inverse-mean-log-ratio estimator rather than the original empirical cumulative distribution function implementation, and its Gaussian assumptions exclude fixed or sublinear intrinsic-rank sequences. Consequently, the broader relevance of the paper requires more rigorous justification.
  7. A symbol table would help a wider audience.
Questions for authors

Clarification regarding practical scope: Are any of the retained comparisons evaluating the original empirical-CDF TwoNN implementation, including its standard fitting and trimming procedures, under identical sample-resolution regimes?

Which qualitative conclusions are empirically supported across methods, and which are unique to the pooled TwoNN approach?

Clarification regarding fitted-null interpretation: Does the retained data allow quantification of variability arising from refitting the Gaussian and class-mixture models, distinct from surrogate Monte Carlo error?

What is the stability of paired residuals across the previously examined covariance regularization choices?

Suggestions for future research, rather than clarification: investigation of finite-sample concentration rates for the dependent pooled statistic, extension to broader non-Gaussian classes, and independent confirmation across architectures would enhance the general applicability, although these are not prerequisites for the stated Gaussian and spherical results.

What is “d log” in equation (3)?

Do the main conclusions still hold for the original empirical-CDF estimator when it uses its standard fitting and trimming options?

2 comments
AuthorsOpenReview ↗

Clarifications on Estimator Choice, Finite-Sample Accuracy, and Experimental Scope

We thank the reviewer for the detailed comments.

We label the numbered critical weaknesses W1–W7 and the questions for authors Q1–Q7.

1. Original empirical-CDF TwoNN (W1, W6, Q1, Q2, Q7)

We tested the original empirical-CDF TwoNN with standard fitting and trimming on the same neighbor distances as pooled TwoNN. We uploaded the additional code to OpenReview. On Gaussian and spherical datasets, the median CDF-to-pooled estimate ratio is 0.994 (range 0.972–1.033). Increasing sample size at fixed distribution and dimension changes both estimates in the same direction in 49 of 50 comparisons (Section 5.1).

Across five ResNet-18 models, PCA-aligned rescaling increases TwoNN 2.199-fold with pooling and 2.348-fold with CDF fitting. Matched random-orientation controls give 0.995 and 0.994 (Table 2). We apply the inverse transformation to the classifier weights, preserving predictions while changing feature coordinates. Our experimental conclusions about sample-size dependence and coordinate sensitivity therefore also hold for empirical-CDF TwoNN in these tests. Regarding W1 and W6, the convergence proofs apply to pooled TwoNN. The CDF results remain empirical.

2. Finite-sample prediction errors (W1, W3)

At n=D=512n=D=512, the reported 12–13% gap uses the shallow approximation dUd_U. Against the full Gaussian phase target, observed TwoNN is lower by 5.0% for isotropic data and 6.4% for two-block data (Appendix H.2). The neighbor-window correction improves later-neighbor predictions, but its first/second-neighbor error rises from 2.17% to 3.97% (Table 1). The auxiliary bounds concern Gaussian target estimation and expected spherical log spacings, and do not bound finite-sample bias or variance of pooled TwoNN.

3. Gaussian comparisons and covariance uncertainty (W2, W5, Q3, Q4)

To assess how well Gaussian distributions predict TwoNN on ResNet-18 features, we simulated feature vectors from a global Gaussian and a class-wise Gaussian mixture fitted on separate images. At layers 3 and 4, observed-to-simulated TwoNN ratios are 0.658 and 0.907 for the global Gaussian, and 0.745 and 0.794 for the mixture (Section 5.4).

Q3, refitting uncertainty. The 0.22–0.65% Monte Carlo standard errors measure simulation variability with fitted parameters fixed. We did not repeatedly refit the models, so they do not measure covariance-fitting uncertainty.

Q4, covariance regularization. Across five models, observed-to-mixture ratios are 0.726–0.771 at layer 3 and 0.792–0.803 at layer 4. With 10% covariance shrinkage, they fall to 0.603–0.637 and 0.578–0.587 (Appendix Table 19). Shrinkage increases the gap, while both choices overpredict TwoNN.

W5, the PCA prediction. The fitted Gaussian predicts a 3.338-fold increase under PCA rescaling, whereas we measure 2.124-fold on the full ResNet-18 sample. The Gaussian model overpredicts the magnitude. Our 2.124-fold measurement comes directly from transformed ResNet-18 features, with classifier predictions preserved. It supports the coordinate-sensitivity claim despite the Gaussian prediction error.

4. Scope and additional architecture evidence (W1, W2, W6, Q5)

For W1 and W6, the Gaussian theorem permits zero eigenvalues but requires rank proportional to the dimension. Orthonormal coordinates preserve all distances within an rr-dimensional Gaussian subspace. If rr grows and the restricted covariance satisfies the assumptions, the theorem applies using rr.

In response to W2, we additionally tested a frozen ViT-Base/16. At its classifier input, PCA rescaling increases TwoNN 1.641-fold with pooling and 1.813-fold with CDF fitting. Matched random orientations give 0.997 and 0.996 (Table 2). The map is fitted on separate images, with predictions preserved. We observe coordinate sensitivity in both tested architectures.

The convergence laws apply to Gaussian and spherical models under the stated assumptions. Coordinate experiments concern the tested network representations. Following Q5, we leave non-Gaussian theory, finite-sample concentration rates, and independent evaluations across datasets and architectures for future work.

5. Presentation (W4, W7, Q6)

We corrected the threshold to log⁡n/D→c∗(α)\log n/D\to c_*(\alpha), equivalently n=exp⁡{c∗(α)D+o(D)}n=\exp\{c_*(\alpha)D+o(D)\} (Section 3.2). In Equation (3), dlog⁡\mathrm d\log denotes a logarithmic differential. Here βx(u)=dlog⁡Qx(u)/dlog⁡u\beta_x(u)=\mathrm d\log\sqrt{Q_x(u)}/\mathrm d\log u denotes the derivative of log radius with respect to log probability mass. We have rewritten Sections 2 and 3 to introduce notation and assumptions before use and Section 7 to clarify the scope of the theoretical and network experiments.

Thank you for your detailed response and clarifications. The revisions satisfactorily address my previous concerns, and I will update my evaluation accordingly.

Reviewer_CYDAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

TwoNN (two nearest neighbors) is a method for estimating the dimension of data: for each data point, it compares the distance to the nearest neighbor with the distance to the second-nearest neighbor, then combines these comparisons into one estimate. The paper shows that this estimate depends on both the data and the number of examples available. With too few examples, TwoNN can give a much lower dimension than the data really have.

The authors support this with mathematical proofs and computer simulations. They also test five image-recognition models. In those tests, they changed how the models’ internal data were represented without changing their predictions, and TwoNN changed a lot. The main takeaway is that TwoNN can be useful for comparing data, but its result should not automatically be treated as the data’s true dimension.

Critical strengths

The paper develops an interesting and potentially valuable theory of what TwoNN measures when sample size is limited. It proves a convergence result for shared-sample TwoNN on anisotropic Gaussian data, where the predicted value depends on sample size and the covariance spectrum. It also proves a spherical result showing that, even for noiseless data on a sphere with known dimension, polynomially many samples can make TwoNN report a vanishing fraction of that dimension. These results explain why TwoNN need not recover population manifold dimension and provide a useful framework for interpreting its estimates at finite sample sizes.

Critical weaknesses

The paper’s practical impact may be limited. Its theory clarifies how TwoNN behaves as sample size and dimension grow, but the paper offers limited evidence that these results lead to a practical method or concrete guidance for common applications. The neural-network experiments are restricted to five CIFAR-10/ResNet-18 models, and they show that TwoNN can change substantially under prediction-preserving coordinate transformations. This makes it difficult to know when practitioners can rely on TwoNN or how they should use it in practice.

Questions for authors
  1. The main convergence results assume Gaussian data or points uniformly sampled from a sphere, while neural representations are generally non-Gaussian. The paper compares them with fitted Gaussian and class-Gaussian-mixture nulls, but reports that these do not fully explain the observed TwoNN values. What interpretation of TwoNN do the authors consider justified for general non-Gaussian data, and which conclusions in the paper extend beyond the two theoretically analyzed distributions?

  2. The neural experiments use five CIFAR-10/ResNet-18 models. Could the authors test whether the coordinate effects and conclusions also hold for transformer representations or other architectures and datasets? Such evidence would help assess how broadly the theoretical and empirical findings apply in practice.

2 comments
AuthorsOpenReview ↗

Non-Gaussian Interpretation and Transformer Evidence

We thank the reviewer for recognizing the Gaussian and spherical results and for asking how they inform the use of TwoNN. We refer to the practical-impact concern as W1 and the numbered questions as Q1 and Q2.

1. Interpretation and practical use on non-Gaussian data (W1, Q1)

For non-Gaussian data, TwoNN describes how probability mass grows over the distances reached by the observed neighbors. Let Gx(r)G_x(r) be the probability mass within distance rr of a point xx. Its logarithmic growth rate dx(r)=rGx′(r)/Gx(r)d_x(r)=rG_x'(r)/G_x(r) equals the local dimension when Gx(r)∝rdG_x(r)\propto r^d. Equation (5) expresses each neighbor log ratio as an integral of 1/dx(r)1/d_x(r) over the mass interval between the two neighbors. This identity applies whenever the stated radial-CDF regularity conditions hold, including non-Gaussian distributions. Because the first two neighbors reach probability masses of order 1/n1/n, the pooled statistic combines information about finite-radius local dimension at a sample-dependent scale. A change in TwoNN as nn increases can therefore reflect a change in the radius being examined, even when the underlying distribution is unchanged.

The Gaussian and sphere results show how large this resolution effect can be. Theorems 3.1 and 3.2 establish concentration of pooled TwoNN at sample-dependent targets despite the reuse of neighbors across anchors. On a noiseless sphere with known manifold dimension, polynomially many samples recover a vanishing fraction of that dimension as dd grows. The practical implication is that a small TwoNN value alone does not establish a low-dimensional population manifold. Comparing estimates across sample sizes helps reveal whether the measurement changes with resolution.

Our experiments on ResNet-18 features test what additional information is needed beyond covariance. We fitted a global Gaussian and a Gaussian mixture with one component per class on separate images, generated synthetic feature vectors, and measured TwoNN on those vectors. The ratios of observed to simulated TwoNN under the global Gaussian are 0.658 at layer 3 and 0.907 at layer 4. Class-conditional fitting brings the simulated value closer at layer 3 but farther away at layer 4 (Section 5.4, Table 3). Thus, the observed nearest-neighbor ratios are not determined by the tested Gaussian covariance descriptions, even after fitting each class separately.

The coordinate intervention gives a second practical check that does not require Gaussian assumptions. Across five ResNet-18 models, PCA-aligned rescaling fitted on separate images increases pooled TwoNN by a median 2.199-fold, while matched random orientations with the same singular values give 0.995 (Section 5.3, Table 2). The inverse transformation is applied to the classifier so that every prediction is preserved. Since the implemented classifier function stays fixed, the change in TwoNN reflects the chosen feature coordinates and Euclidean distances. For comparisons of network representations, we therefore recommend reporting the sample size and distance metric, checking changes across sample sizes, and testing sensitivity to feature rescaling. These measurements make TwoNN useful as a description of representation geometry while avoiding unsupported conclusions about population manifold dimension.

2. Coordinate sensitivity in a transformer (W1, Q2)

We ran an additional experiment on a frozen ViT-Base/16 to test whether the coordinate effect also appears outside ResNet-18. Using the same CIFAR-10 images, we fit each PCA transformation on 4,000 images and measure TwoNN on the other 4,000, then reverse the split. At the ViT classifier input, PCA-aligned rescaling increases TwoNN by 1.641-fold with the pooled estimator and 1.813-fold with the original empirical-CDF estimator. Matched random orientations with the same singular values give 0.997 and 0.996 (Table 2 and Appendix P). Applying the inverse coordinate map to the classifier preserves every predicted class, with maximum logit difference 2.40×10−62.40\times10^{-6}. Both the tested transformer and the five ResNet-18 models show substantial TwoNN changes under prediction-preserving coordinate transformations.

The ViT result extends this test to a second architecture family on CIFAR-10. It uses one frozen checkpoint and one classifier-input representation, so further architectures and datasets would establish how effect sizes vary. We have clarified this scope in Section 7. Together, the experiments show why the sample resolution and coordinate metric must be specified when TwoNN is used to compare representations.

Reviewer_CYDAOpenReview ↗

Improve the rating

Thank you for the answer and improvements, I find the additional experiment on ViT convincing and therefore I increase the rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.