acceptodds
Under review as a conference paper at ICLR 2027

Intrinsic Dimension Reads Within-Class Structure Below the Class Size and Class-Mean Geometry Above It

Abstract

Intrinsic dimension is a popular one-number summary of a neural network’s representation, used to argue that networks compress their representations during training and that lower dimension means better generalisation. We show that on labelled data this number measures one of two different things, and a rarely reported choice decides which: how many neighbours the estimator looks at, compared with how many images each class contributes. With few neighbours the number describes the shape of each class; with many, how the classes are laid out. The two geometries relate to accuracy in opposite ways. On 178 ImageNet classifiers, changing only the neighbour count turns the well-known correlation between dimension and error from +0.80 to −0.66 on the same networks and images, and in training runs the same checkpoints read as compression at one count and expansion at another. We prove two short theorems: one derives the switch from how a query’s neighbours fill with classmates, the other shows that class means, class covariances and even the network’s own logits do not determine what the estimator reports. The switch holds across architecture families and network scale, on four out-of-distribution test sets, in self-supervised encoders and in 26 language models. We close with a two-count check that any study can run: read at half and at four times the class size, and say which geometry the conclusion rests on.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
intrinsic dimension, nearest-neighbour estimators, representation geometry, neural collapse, generalisation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_seokOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper considers a specific notion of intrinsic dimension of representations in image neural networks. This intrinsic dimension metric measures the ratio/difference of the distance from a representation vector to its 2n2n'th nearest neighbor, compared to the distance to its nn'th nearest neighbor. (For low intrinsic dimension, those distances differ a lot, while for high intrinsic dimension they only differ little.) The list of representation vector samples is taken from all image classes equally, mm images per class, and the paper studies the impact of the choice of nn (the neighbor count). They claim two regimes: When nn is small the described metric measures "in-class" intrinsic dimension, while when nn is large, it measures "inter-class" geometry. In the first regime, this intrinsic dimension measure correlates with error, as shown in existing literature. The authors find that when nn grows, and the "inter-class" regime is entered, the sign of the correlation flips. They also do a little theory, which however I couldn't fully parse.

Critical strengths
  • Some experiments show a clear transition in the measured correlation as the number of neighbors increases, supporting the model of an in-class and inter-class regime.
Critical weaknesses
  • The language often reads "claudish", for example overuse of titles/phrases like "What/where/why [...] does, and where/why it [...] .", or generally many sentences (statements) starting with what/where/why. But also other markers of potentially AI-generated text that do not follow usual conventions in scientific writing, for example many figure titles describe a conclusion instead of being a short descriptive title (e.g. see the bold-formatted titles of figures 3,4,5). Sometimes the text is also not written in full sentences, see for example the beginning of section 2: "Same networks, same images. Only the neighbour count changes, and the sign of the correlation with error changes with it." Some terms/symbols are also never defined (e.g. ψ\psi).
  • The theorem statements are not formal, also not in the appendix. What is a "labelled cloud"? What does "almost every t" mean? If it means "almost surely", then with respect to which probability measure? What is the intuitive meaning of the theorems?
Questions for authors
  • What is ψ\psi?
  • What exactly is the correlation that switches sign? (define top-5 error etc)
  • Can you please state your theorems formally?
2 comments
AuthorsOpenReview ↗

psi, the correlation, and the theorems stated precisely

Thank you — all three questions are answered in the revised PDF, and your examples were the quickest things to act on.

What psi is. It is the digamma function. The constant in the estimator, psi(2n) - psi(n), is the expected log ratio of the two shell radii under the Poisson model these estimators assume, so dividing by it puts the estimate in units of dimension. It is now defined where it first appears.

Which correlation switches sign. It is the Pearson correlation across networks, one point per network, between the log estimate and the log of top-5 error on ImageNet validation. Top-5 error is the share of images whose true label is not among the five classes a network scores highest. Both are now stated in the main text where the correlation is first used.

The theorems. Appendix A does state both as theorems with proofs, but you are right that the main-text statements were too informal and that the terms in them were never defined. The revision fixes that.

  • A labelled cloud is a finite set of distinct points in R^D, each carrying a class label, with at least two classes. Nothing is assumed about how they are distributed.
  • "Almost every t" was the wrong phrase and we have dropped it. We mean all but finitely many t in (0,1], and no probability measure is involved: along the contraction two neighbour distances are quadratics in t, so they cross at most twice, and the finitely many crossing points are what we exclude.
  • In words, Theorem 1 says the estimate responds to a shrinking of within-class scale only through the share of each shell that is a classmate, plus a remainder we measure rather than bound. At n = m that remainder is 0.31 of the classmate term, and the identity predicts the measured change to within 0.008 in log units.
  • Theorem 2 says that nothing computed from the class means, the within-class distances, the class covariances or the logits can tell you what the estimate will report, and it puts a floor on how far any such prediction must be off.

The writing. You are right, and we acted on each of your examples. The sentence you quote from the start of Section 2 is now a full sentence. In all six figures the titles say what the figure shows rather than what we concluded. The what/why headings are gone, from Sections 3, 3.3 and 4 and from two appendix subsections. Every symbol is defined where it first appears.

If anything is still unclear, or there is a definition or statement you would like us to tighten further, we are glad to do it while the discussion is open.

AuthorsOpenReview ↗

Follow-up on our response

Thank you again for reading our paper and for the specific examples, which were the quickest things to act on. In response we have defined psi and top-5 error where they first appear, stated both theorems precisely with "labelled cloud" and "all but finitely many t" spelled out, and fixed the writing you pointed to, including the sentence at the start of Section 2 and the figure titles. The revised PDF is uploaded. It would be great if you could take a moment to see whether these address your concerns, and we are happy to clarify anything further before the discussion closes.

Best, The authors

Reviewer_iJvsOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The contribution focuses on the interpretation of nn-based intrinsic dimension estimates for learned data representations in neural networks or other relevant learning machines. The Authors argue that on labelled data the estimate relates to class geometry or to the layout of class means, depending on the neighbour count relative to the class size. The work shows, through experiments, that these two readings may correlate with accuracy in opposite directions.

Critical strengths

The work shows, through extensive experiments, that the sign of dimension/error correlation may depend on the neighbour count relative to the class size.

Critical weaknesses

The major practically relevant impact of the paper is around a mere reporting convention. The Authors show that one number does not distinguish between two geometries. This is expected when the neighbourhood size is compared with the number of same-class points per point. It is still unclear which of these geometries are most relevant for generalisation. The results may not apply to models without labels. Overall, the practical gain is marginal or small.

Questions for authors
  1. It would be great to hear whether there is any novelty beyond the reported and largely expected dichotomy.
  2. Please clarify which geometry matters for generalisation?
  3. Can the results be generalised to cases without class labels? If so, how?
3 comments
AuthorsOpenReview ↗

The practical stake, which geometry tracks error, and the unlabelled case

Thank you. You judge the practical gain small, so we start there.

The reporting convention. A convention is what we propose, but what it asks people to report is not a formality. Without it the sign of a published result is undetermined on modern networks: the same 178 networks read on the same images give +0.80 at one neighbour and -0.66 at eighty. A small count is no protection, because the regime is set by n compared with m. On the design behind that result each class contributes about two images, so TwoNN at one neighbour is already reading at n/m = 1/2, with the first neighbour a classmate for a median 47.5% of queries and the second for 29.2%. When the count is chosen by a flatness rule instead, it lands at or above the class size for 90-100% of networks across five designs. And the choice costs something: picking a layer by the estimate read at n/m = 2 instead of 1/2 loses a median 36 points of probe accuracy, worse in 24 of 32 encoders and better in none. Compression during training becomes expansion on the same checkpoints.

Novelty beyond the dichotomy. You are right that the dichotomy itself is expected. Section 5 credits the scale dependence to five prior works and places the concurrent result of Osman et al. (2026) beside ours, and the revised abstract now says so first. What is not expected is that the two readings order the same networks by error in opposite directions. Four things we do claim: the reversal holds within architecture families, within one architecture under seven recipes, and given parameter count (Table 12); class means, covariances, the collapse statistics and even the logits do not determine the estimate, with a numerical floor on how wrong any such predictor must be; deletion and intervention on real features say what each regime measures; and the direction of the upper branch, which nobody predicted, since better networks space their class means more evenly and so read higher at 4m.

Which geometry matters for generalisation. We do not know, and we are careful not to claim it. Both readings track error here: the residuals-only estimate at +0.79 below m, the class-means-only one at -0.74 above it (Table 21). All our cross-network results are correlations, which is the first of the five limitations we list. We also report the result that cuts against our own statistic. On the same probe, 1-NN accuracy and nearest-class-mean accuracy track error better than any reading of the estimate, at 0.93 and 0.95, and whether the estimate adds anything beyond 1-NN error is unresolved: the held-out gain is 0.021 with an interval of [-0.013, +0.050] across architecture families. So our recommendation is not to use the layout reading. It is to say which geometry a conclusion rests on and, if the conclusion is about generalisation, to report those two accuracies beside the estimate. We would rather leave the causal question open than answer it with correlations.

Without labels. Yes for the class regime, partly for the layout regime, and both are measured. Replace the classes with k-means clusters (k = 100 on the centred probe, adjusted Rand index with the true classes 0.54 to 0.70). The residuals-only estimate then reproduces the full estimate at r = +0.977 to +0.983 for n up to 7, slightly better than with the true classes (+0.946 to +0.972), and tracks error at +0.715 against +0.678 with classes (Table 17). The cluster-means-only estimate keeps up to about 2m (+0.720 at n = 40) but not at 4m (+0.283), because clusters are not classes and differ in size. More generally, the knob is the count relative to the occupancy of whatever grouping the data carry. That is why the switch also appears in 26 language models on intent-labelled text, and why the depth peak of Cheng et al. (2025) on unlabelled text moves with n/m in 6 of 7 models. The revision says explicitly that the check is run with clusters when there are no labels.

If anything here is still unclear, or there is a check you would like us to run, we are glad to do it while the discussion is open.

AuthorsOpenReview ↗

Follow-up on our response

Thank you again for reading our paper and for your questions. In response we have set out what the reading costs in practice, answered which geometry tracks error and where we cannot claim more than a correlation, and shown that the class regime carries over without labels, using k-means clusters in place of classes. The revised PDF is uploaded. It would be great if you could take a moment to see whether these address your concerns, and we are happy to clarify anything further before the discussion closes.

Best, The authors

Thank you very much for the effort and response. It is clearer now where the novelty claim sits. Yet, I am still not convinced about the overall impact or relevance of this finding. That is why I am keeping my provisional score unchanged.

Reviewer_8q5UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that intrinsic dimension captures different geometries at different neighborhood sizes, reversing its relationship with generalization.

Critical strengths

The paper identifies a clear and consequential issue: on labeled data, nearest-neighbor intrinsic dimension can measure either within-class structure or class-mean geometry, depending on the neighbor count relative to class size. This finding is supported by comprehensive theoretical and empirical evidence across diverse settings and leads to a practical two-count diagnostic at m/2m/2 and 4m4m.

Critical weaknesses
  1. The novelty should be stated cautiously. The dependence of intrinsic-dimension estimates on neighborhood scale is not itself new; the main contribution is to connect this dependence with within-class and between-class geometry and show how it revises previous conclusions.

  2. The highlighted correlation reversal is not universal. It appears in many modern model collections, but not in the original fourteen ImageNet models or some CIFAR experiments, so the claim should be framed accordingly.

  3. Most dimension–error results remain correlational. Even with controls for architecture and training setup, potential confounding factors, such as model scale, training recipes, and data preprocessing, cannot be fully excluded.

Questions for authors

Could the authors clarify what is genuinely novel in this work beyond the previously known dependence of intrinsic-dimension estimates on neighborhood scale?

How often does the reported correlation reversal occur across all evaluated settings, and what factors determine whether it appears or not?

Does intrinsic dimension retain significant predictive value for generalization error after controlling for major confounders and comparing against simpler geometric baselines?

2 comments
AuthorsOpenReview ↗

How often the reversal appears and what decides it, novelty, and the baselines

Thank you — we have taken your framing into the revision, and all three questions have answers that were in the paper but buried in the appendix.

How often the reversal happens, and what decides it. This is the one we most wanted to answer, and the revision moves it into the main text.

It appears in each of the three ImageNet collections on its own, in the pooled 178 and the pooled 222, in each of the three ResNet recipe groups, within architecture families once family means are removed, given parameter count, on training images, on an independent re-extraction of the features, on four shifted test sets and the relabelled validation set, and across 26 language models at their last layer (Tables 6, 11-13, 25). It is absent or only partial in six settings, which Table 10 lists: the original fourteen networks, one ViT family, ImageNet at 64 pixels, the CIFAR-100 collection, two CIFAR-100 regulariser sweeps, and when local PCA replaces the estimator — a test we had registered in advance as one we expected to reverse, and which failed.

What decides it is the balance between two parts. Write the log estimate at 4m as a class-means part X plus a remainder Y. The sign of its correlation with error is then the sign of s_X r_XE + s_Y r_YE (Table 19):

groupNr(full, err)r(X, err)r(Y, err)var Xvar Y
ImageNet, all222-0.65-0.74+0.480.1590.046
ImageNet, recipe groups28-0.84-0.87+0.380.1650.022
ImageNet, ViT family22-0.55-0.38+0.01
CIFAR-100, weight decay21+0.99-0.97+0.980.0430.092
CIFAR-100, LS + weight decay20+0.39-0.92+0.970.1520.165

The class-means part relates to error negatively in every group, the two CIFAR sweeps included, where the full estimate nonetheless stays positive. The single number fails to reverse exactly when the remainder is both large and strongly positive, which is what a regulariser sweep produces: it moves within-class scale and class spacing together, so the two parts nearly cancel. The opposition between the two geometries is what carries across cohorts; whether one summary number flips depends on which part a cohort happens to vary more.

What is new. We have taken your wording. The revised abstract says in its second sentence that the scale dependence of these estimates is already known, and states the contribution as connecting that dependence to within-class and between-class geometry and showing which published conclusions change. What we do claim as new is the opposite-sign ordering of the same networks, the identification result with its numerical floor, the deletion and intervention results that say what each regime measures, and the two-count check.

Whether the estimate survives the controls and the baselines. Largely not, and the paper says so: Table 21 is a horse race we ran against ourselves.

On confounders, within architecture families and given log parameter count both branches keep their sign with intervals excluding zero (+0.71 and -0.63 on the published design, +0.56 and -0.70 on the stratified one; Table 12). Within one architecture trained with seven recipes the reversal runs from +0.85 to -0.91 (Table 14). Restricted to the 110 networks inside the interquartile error range it weakens to +0.46 and -0.44, and is not established collection by collection, which Table 10 reports.

On baselines, 1-NN accuracy on the same probe reaches r = 0.93 and nearest-class-mean 0.95, above any reading of the estimate, whose best is +0.80. The partial correlation of the estimate given 1-NN error is -0.44, and adding the estimate to 1-NN error gains 0.021 in held-out R2 with an interval of [-0.013, +0.050] across architecture families, so that gain is unresolved.

We draw the conclusion this supports: the estimate is not the best available predictor of generalisation, and a claim about generalisation should report the labelled baselines beside it. Our result is about what the number measures, not about it being the best thing to measure.

If anything here is still unclear, or there is a check you would like us to run, we are glad to do it while the discussion is open.

AuthorsOpenReview ↗

Follow-up on our response

Thank you again for reading our paper and for your questions. In response we have taken your framing on novelty into the abstract, moved the decomposition that says when the reversal appears out of the appendix and into the main text, and set out how often the reversal occurs and where it does not. The revised PDF is uploaded. It would be great if you could take a moment to see whether these address your concerns, and we are happy to clarify anything further before the discussion closes.

Best, The authors

Reviewer_5tR7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors identify that the intrinsic dimension estimation approaches for NN layers are sensitive to the choice of the number of nearest neighbor n. More specifically, they find that when n<m/2, where m is the average number of examples per class, then the trends show the expected dependence of ID to generalization error, but the trend flips when n>m (specifically at n=4m). There are some theoretical results provided that strive to identify how class distributions can disrupt and confound ID estimates. The authors demonstrate the correlation flip in more settings, including some language models.

Critical strengths
  • The idea of linking ID-generalization stability to the neighborhood size is a good starting point.
  • Experiments seem enough and done in varied settings/domains to showcase the intended dependency flip.
Critical weaknesses

The use of AI in writing is clear (as also declared by the authors) and unfortunately it is not vetted well: the explanations and terminology and wording used throughout is non-standard, inconsistent and confusing, making it unnecessarily hard to read.

The regimes seem to be defined independently of K. Is that okay? As in, I would think that lower K would probably need more cautious intervals than higher K, given a fixed m?

Also, the regimes assume certain rigid conditions that are very less likely to hold as the feature distribution curves more or increases its intrinsic dimension itself. For instance, it is assumed that n<m/2 would ensure both shells are within the same class on average. However, this can wildly change depending on the curvature and dimension of the feature distribution. HIgh curvature or high dimension would invoke the curse of dimensionality and essentially make it so that much before m/2 the shells would leak into other class distributions with high probability.

Importantly, it is unclear what the utility of changing the number of neighbors is, as fundamentally all previous works strive to estimate the intrinsic dimension and choose appropriate k which is generally not too large to ensure that the neighbors are part of the local manifold as much as possible. In fact Ansuini et al. 2019 just uses TwoNN which this paper reports, which simply uses k=1. What I don't connect with is as k is increased beyond the local manifold region, even beyond the class distribution, we are looking at a quantity which is not connect to intrinsic dimension at all, and is a different quantity altogether. So I'm not sure why this is interesting, or provides any new insights. The observation about keeping residuals reproducing the ordering below half the class size and similarly for class means is again not surprising and importantly not sure what we are learning from it. Especially as for the larger k, the fact that the internal class structures are ignored is not surprising, and nor is it relevant to somenone computing ID using small k to ensure local manifolds are only being captured.

I don't see the point of 3.2. They are in some ways erasing local within class structures and just keeping the broad covariance structure of the class, and arguing that that does not change the trend. However, looking at Figure 4, the trends are diferent in 3 of hte 4 tests, so I'm not sure what we are learning from this.

The theorems themselves are also hard to interpret in terms of what we actually learn from it. For instance, theorem 1 has an R(t) term whose only information provided is that it vanishes as the classes separate for low enough t. But the authors are primarily interested in the regime where the transition is happening from higher overlap to no overlap, and how that influences/corrupts the ID estimates. The theorem says nothing about the behaviour of R(t) in that critical phase, thus causing confusion w.r.t what exactly to gain from the Theorem. Furthermore, it is also not clear if any condition is imposed on the distributions themselves, if not, it should be clearly stated that this is a general result.

Theorem 2 is also quite weird, not sure what NC1 and NC2 is, they are just introduced without any definition. I mean it is quite clear that reflections preserve intra-class structure but change inter-class, so I'm not sure what the theorem is adding to this statement. "double reflections preserve logits": isn't this obvious? I'm not sure why this needs to be mentioned. The overall theorem presentation is very high-level and unclear. The appendix has more math in the statement, and I'm not sure why this formal statement isn't presented in the main part. Either way, even the appedix theorem statements are quite unclear: what is a "summary"? what is the concrete outcome of the theorem?

Lastly looking at the experiments, why even are we interested in the fact that the correlation flips when n grows high enough that it doesn't capture local structure anymore? I think this is a fundamental question that needs to be first answered. In fact the only outcome I see of the experiments is to choose n<m/2 in most cases. But for most methods this will be true as small n is typically chosen anyway (e.g. TwoNN). In my experience as well this is true, even constructing graphs that mirror the geodesic manifolds of simple distributions such as MNIST get intractable when n grows beyond a small number (much earlier than m/2).

Questions for authors

Please see weaknesses

2 comments
AuthorsOpenReview ↗

Where small k lands, the K dependence, and the remainder of Theorem 1, measured

Thank you — two of your points changed the paper.

Why a reading above the class size matters. What sets the regime is how large n is compared with m, not n on its own, and on the design you name the two are close. Ansuini et al. read 2,000 ImageNet images spread over 1,000 classes, so each class contributes about two, and TwoNN at k = 1 is already reading at n/m = 1/2. On 194 of those networks the first neighbour is a classmate for a median 47.5% of queries, the second for 29.2%.

The same happens when the count is chosen for you. A flatness rule over the scale scan reads at or above the class size for 90-100% of networks across five designs (Table 26). Three of eleven published designs we place on this axis already read there (Table 27), among them Ma et al. (2018) at n/m of 2 to 3, whose estimate drives training. And it costs something: a layer chosen at n/m = 2 instead of 1/2 loses a median 36 points of probe accuracy, worse in 24 of 32 encoders and better in none (Table 28).

We agree that at large n the estimator is no longer measuring the dimension of a local manifold. That is our point: we propose reading at 4m not to get a better dimension, but to find out whether a conclusion depends on the count.

What the deletion result is for. You are right that it is no surprise a large k ignores what happens inside a class. We measure it because it lets a reader tell which geometry their own number is reporting. Below m/2 the full estimate follows the residuals-only one (r = +0.97 at n = 1, +0.93 at n = 9); from 2m it follows the class-means-only one (+0.85 at 4m). Computing both alongside the full estimate names the geometry, without relying on our theory.

Leakage before m/2. You are right, and we overstated it. At n = m/2 the inner shell is a classmate for a median 71% of queries and the outer for 11%, and both shells are classmates for only 8% (Table 17). The revision replaces that sentence with the measured percentages.

You predict that curvature pulls the transition below m/2. The 20 probe arms in Table 20 check that, across K from 25 to 1,000 and m from 1 to 40. The crossing never falls below 0.93m and never rises above 2.52m, so every arm sits between the two readings our check uses. Leakage moves the crossing up rather than down: the arms with the smallest classes have the highest crossing of all. Where leakage is severe, on ImageNet at 64 pixels, a classmate is nearest for only 37% of queries, the class regime does break, and Table 10 reports it as a failure.

K dependence. They are not independent of K, and the law is now in the main text. Over the 19 arms with m of at least 2, the crossing is n*/m = 0.98 + 0.0016(K-1), with R2 = 0.99. Your intuition that a smaller K needs more caution is right in sign: with few classes the crossing sits essentially at n = m, leaving no margin above it.

The remainder in Theorem 1. We write R out explicitly instead of bounding it, so it can be evaluated, and we evaluate it in the transition you ask about. Shrinking every residual by 10% on 194 networks, the identity predicts the measured change at n = m to within 0.008 in log units, where the classmate term on its own is off by 0.066. There |R| is 0.31 of the classmate term, and only grows past it at 4m, by which time the response has faded to -0.008 (Table 5). Nothing is assumed about the distributions, and the revision now says so.

Section 3.2. It asks which information the negative branch needs, not whether the correlation is robust. The surrogate throws away everything inside a class except each point's distance to its class mean, and its curve still matches the real one: r = 0.99, 0.99, 0.82 and 0.87 in panels A to D. It beat three baselines we fixed in advance in 6, 7 and 7 of the 7 populations (Table 22), and those four numbers are now in the caption. The second half is an intervention, not a correlation: making every distance to the class centre equal removes the negative branch, while the same factors applied to permuted classes leave it intact (Table 23).

Theorem 2. NC1 and NC2 are now defined where they first appear. By a summary we mean any function of the class means, the within-class distances and the covariances, and for the double reflection the logits too. The reflection is only the construction. The content is that no such function can track the estimate: any predictor built from them is wrong by at least half the shift the reflection produces. That shift is 0.18 log units at 4m, so the floor is 0.09 — 30% of the spread of the estimate across these networks, against 10% at one neighbour. The logit version matters because two networks can give identical logits on every probe point and still differ this much in the dimension they report.

The writing. We agree. Half-sentences are now full sentences, figure titles say what the figure shows rather than what we concluded, the what/why headings are gone, and every symbol is defined where it first appears.

AuthorsOpenReview ↗

Follow-up on our response

Thank you again for reading our paper and for such a detailed review. In response we have replaced the claim that both neighbours are classmates with the measured percentages, put the dependence of the regimes on K into the main text as a fitted law, measured the remainder in Theorem 1 at the transition rather than leaving it unknown, and gone back over the writing you flagged. The revised PDF is uploaded. It would be great if you could take a moment to see whether these address your concerns, and we are happy to clarify anything further before the discussion closes.

Best, The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: the regime is set by neighbours per class, and a revised PDF is uploaded

We thank all four reviewers, and we have uploaded a revised PDF.

Two criticisms we accept. Our sentence that both neighbours are classmates below n = m/2 is too strong. The revision replaces it with the measured shares below. And two of you found the paper hard to read. We have gone back over the text: half-sentences are now full sentences, figure titles say what the figure shows instead of what we concluded, and every symbol is defined where it first appears.

The regime is set by neighbours per class, not by the neighbour count. A reader is inside the class regime only if n is small relative to m, the images each class contributes. The design behind the result we re-read spreads 2,000 ImageNet images over 1,000 classes, so m is about 2. Measured on 194 of those networks (medians):

designnn/mnth neighbour is a classmate2nth is a classmate
published, K = 1000, m = 21 (TwoNN)1/20.4750.292
stratified, K = 100, m = 2011/200.8900.873
stratified101/20.7100.111
stratified4020.0260.011

TwoNN at one neighbour on the published design therefore reads a mixture of the two geometries, not the shape of a class.

A reader who never chooses n can still land above m. With the count picked by a flatness rule, 90-100% of networks across five probe designs are read at or above the class size, where the correlation with error is negative (Table 26). Three of the eleven published designs we place on this axis already read there (Table 27).

The ambiguity costs accuracy. Choosing a layer by the estimate read at n/m = 2 rather than 1/2 costs a median 36 points of probe top-1, worse in 24 of 32 encoders and better in none (Table 28).

What the paper adds. The scale dependence of these estimates is not new, and Section 5 credits it with five citations; the revised abstract now says so in its second sentence. What we add is that on labelled data the two scales order the same networks by error in opposite directions. In every cohort we measured, the class-means term relates to error negatively, from -0.38 to -0.98, including the two CIFAR sweeps where the full estimate stays positive (Table 19).

What changed in the revision.

  • We removed the sentence saying that both neighbours are classmates below n = m/2, and put the measured percentages in its place.
  • We added a paragraph to the introduction on why the choice of neighbour count matters in practice.
  • We added how the crossing moves with the number of classes, as a fitted line.
  • We moved the split of the estimate into a class-means part and a remainder, which says when the reversal appears, out of the appendix and into Section 3.1.
  • We measured how large the remainder term in Theorem 1 is, and report it next to the theorem.
  • We stated both theorems precisely, and defined psi, NC1, NC2, Pearson and top-5 error where they first appear.

We reply to each of you below, and would be glad to know whether these changes address your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.