- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes an anisotropy ratio comparing nuisance contraction with semantic preservation to explain few-shot OOD classification. The paper provides a sufficient margin guarantee, a proposed empirical estimator, with theoretical results validated by Lean. Empirical results uses four benchmarks and seven encoder families, finding strong correlations with OOD accuracy within several encoders to support their theoretical results.
Critical strengths
The paper introduces an intuitive idea and makes it mathematically precise: suppressing nuisance variation while preserving class separation can give guarantees on few-shot classification success, with clear bounds showing how much separation is sufficient.
Critical weaknesses
Weakness #1: A key contribution of the paper, that "invertible representations cannot improve this ratio beyond the raw input geometry" doesn't appear to be true. The theoretical conclusion on line 208 (“Combined with Theorem 1, Proposition 5 says an invertible code inherits the raw input margin and can never improve it”). Consider the following counterexample: which satisfies the conditions of Proposition 5 and theres two classes where the first class consists of and the second class consists of . Their centers are separate by and so-called nuisance radius of . After the encoding, the separation becomes and nuisance radius becomes so the separation ratio improves from to . A simple nearest-support classifier could have been improved in accuracy in this situation using the encoder.
Weakness #2: Gap between theory and practice. Firstly, Eq. (4) uses the mean distances while theory uses worst-case distances. While the mean distances can be useful, it doesn't establish minimum semantic gain or maximum nuisance loss. Secondly, is held constant across corruptions for each encoder, and it divides by feature displacement (a constant) cannot change the Pearson correlation with accuracy (with the same encoder but different perturbations). Overall, I think this is a serious limitation with the empirical results.
Questions for authors
- Please clarify the claim on line 208
2 comments
Dear Reviewer pm22,
Because of the discussion periods will close soon. Dont know if my comments have solved all your concerns ?
Thank you again for your time and thoughtful feedback.
Response to Reviewer pm22
Thank you for the careful reading.
W1 / Q1 (submitted l208). Your example is correct: E(s,n)=(10s,2n) is non-contracting with ρ=0.2. Proposition 5 bounds ρ only for isometries, and that sentence went beyond it. The revision limits it to isometric codes (General Response, item 2). The proposition's conclusion and its Lean proof stand.
W2a (means vs worst case). The paper makes this substitution deliberately and states it.
We agree that "direct finite-sample version" (submitted p5 l265) is too strong. The revision calls Eq. 4 a statistic motivated by Theorem 8 (§3.6, p6 l295-296) and lists the missing estimation-error bound in Limitation (v) (p10 l508).
W2b (constant β^). We agree with the fact, and the paper states it: σ^/β^ is constant within an encoder (§3.6, p6 l300). This is by design.
Theorem 8 compares corruptions for a fixed encoder, where β is fixed and only η varies with the corruption. The within-encoder evidence is therefore evidence for the theory's corruption term.