- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper considers a specific notion of intrinsic dimension of representations in image neural networks. This intrinsic dimension metric measures the ratio/difference of the distance from a representation vector to its 'th nearest neighbor, compared to the distance to its 'th nearest neighbor. (For low intrinsic dimension, those distances differ a lot, while for high intrinsic dimension they only differ little.) The list of representation vector samples is taken from all image classes equally, images per class, and the paper studies the impact of the choice of (the neighbor count). They claim two regimes: When is small the described metric measures "in-class" intrinsic dimension, while when is large, it measures "inter-class" geometry. In the first regime, this intrinsic dimension measure correlates with error, as shown in existing literature. The authors find that when grows, and the "inter-class" regime is entered, the sign of the correlation flips. They also do a little theory, which however I couldn't fully parse.
Critical strengths
- Some experiments show a clear transition in the measured correlation as the number of neighbors increases, supporting the model of an in-class and inter-class regime.
Critical weaknesses
- The language often reads "claudish", for example overuse of titles/phrases like "What/where/why [...] does, and where/why it [...] .", or generally many sentences (statements) starting with what/where/why. But also other markers of potentially AI-generated text that do not follow usual conventions in scientific writing, for example many figure titles describe a conclusion instead of being a short descriptive title (e.g. see the bold-formatted titles of figures 3,4,5). Sometimes the text is also not written in full sentences, see for example the beginning of section 2: "Same networks, same images. Only the neighbour count changes, and the sign of the correlation with error changes with it." Some terms/symbols are also never defined (e.g. ).
- The theorem statements are not formal, also not in the appendix. What is a "labelled cloud"? What does "almost every t" mean? If it means "almost surely", then with respect to which probability measure? What is the intuitive meaning of the theorems?
Questions for authors
- What is ?
- What exactly is the correlation that switches sign? (define top-5 error etc)
- Can you please state your theorems formally?
2 comments
Follow-up on our response
Thank you again for reading our paper and for the specific examples, which were the quickest things to act on. In response we have defined psi and top-5 error where they first appear, stated both theorems precisely with "labelled cloud" and "all but finitely many t" spelled out, and fixed the writing you pointed to, including the sentence at the start of Section 2 and the figure titles. The revised PDF is uploaded. It would be great if you could take a moment to see whether these address your concerns, and we are happy to clarify anything further before the discussion closes.
Best, The authors
psi, the correlation, and the theorems stated precisely
Thank you — all three questions are answered in the revised PDF, and your examples were the quickest things to act on.
What psi is. It is the digamma function. The constant in the estimator, psi(2n) - psi(n), is the expected log ratio of the two shell radii under the Poisson model these estimators assume, so dividing by it puts the estimate in units of dimension. It is now defined where it first appears.
Which correlation switches sign. It is the Pearson correlation across networks, one point per network, between the log estimate and the log of top-5 error on ImageNet validation. Top-5 error is the share of images whose true label is not among the five classes a network scores highest. Both are now stated in the main text where the correlation is first used.
The theorems. Appendix A does state both as theorems with proofs, but you are right that the main-text statements were too informal and that the terms in them were never defined. The revision fixes that.
The writing. You are right, and we acted on each of your examples. The sentence you quote from the start of Section 2 is now a full sentence. In all six figures the titles say what the figure shows rather than what we concluded. The what/why headings are gone, from Sections 3, 3.3 and 4 and from two appendix subsections. Every symbol is defined where it first appears.
If anything is still unclear, or there is a definition or statement you would like us to tighten further, we are glad to do it while the discussion is open.