acceptodds
Under review as a conference paper at ICLR 2027

From Inverted Codes to Anisotropic Encoders: A Single Ratio Predicts Few-Shot Out-of-Distribution Accuracy

Abstract

Few-shot out-of-distribution (OOD) classification requires pre-trained encoders to generalize to unseen classes and novel corruptions. While performance variations across models are often attributed to scale or data, we demonstrate that this generalization is fundamentally a property of latent space geometry. We introduce the anisotropy ratio, a single metric quantifying how strongly an encoder contracts within-class (nuisance) variation relative to how strictly it preserves between-class (semantic) separation. Theoretically, we prove a margin law guaranteeing that one-shot nearest-centroid classification succeeds whenever the task margin adequately exceeds this ratio. Furthermore, we establish an isometry floor showing that invertible representations cannot improve this ratio beyond the raw input geometry, offering a geometric explanation for the performance gap between generative and contrastive models. Practically, we demonstrate that this ratio can be estimated from unlabeled, in-distribution data alone-requiring no gradients or access to the shifted test set. Extensive experiments across synthetic geometries and four image benchmarks, spanning seven encoder families and the CIFAR-10-C robustness suite, confirm that this label-free estimator strongly predicts actual OOD accuracy. Finally, all theoretical results are fully machine-checked in the Lean 4 proof assistant, ensuring mathematical rigor.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Deep Learning, Deep Learning Theory, Few shot, Zero shot, Out of Distribution

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pm22OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes an anisotropy ratio comparing nuisance contraction with semantic preservation to explain few-shot OOD classification. The paper provides a sufficient margin guarantee, a proposed empirical estimator, with theoretical results validated by Lean. Empirical results uses four benchmarks and seven encoder families, finding strong correlations with OOD accuracy within several encoders to support their theoretical results.

Critical strengths

The paper introduces an intuitive idea and makes it mathematically precise: suppressing nuisance variation while preserving class separation can give guarantees on few-shot classification success, with clear bounds showing how much separation is sufficient.

Critical weaknesses

Weakness #1: A key contribution of the paper, that "invertible representations cannot improve this ratio beyond the raw input geometry" doesn't appear to be true. The theoretical conclusion on line 208 (“Combined with Theorem 1, Proposition 5 says an invertible code inherits the raw input margin and can never improve it”). Consider the following counterexample: E(s,n):=(10s,2n)E(s, n) := (10s, 2n) which satisfies the conditions of Proposition 5 and theres two classes where the first class consists of {(0,n):∣n∣≤1}\{(0, n) : |n| \le 1\} and the second class consists of {(1,n):∣n∣≤1}\{(1, n) : |n| \le 1\}. Their centers are separate by δ=1\delta = 1 and so-called nuisance radius of 11. After the encoding, the separation becomes 1010 and nuisance radius becomes 22 so the separation ratio improves from 11 to 55. A simple nearest-support classifier could have been improved in accuracy in this situation using the encoder.

Weakness #2: Gap between theory and practice. Firstly, Eq. (4) uses the mean distances while theory uses worst-case distances. While the mean distances can be useful, it doesn't establish minimum semantic gain or maximum nuisance loss. Secondly, β^\hat\beta is held constant across corruptions for each encoder, and it divides by feature displacement η^\hat\eta (a constant) cannot change the Pearson correlation with accuracy (with the same encoder but different perturbations). Overall, I think this is a serious limitation with the empirical results.

Questions for authors
  1. Please clarify the claim on line 208
2 comments
AuthorsOpenReview ↗

Response to Reviewer pm22

Thank you for the careful reading.

W1 / Q1 (submitted l208). Your example is correct: E(s,n)=(10s,2n)E(s,n)=(10s,2n) is non-contracting with ρ=0.2\rho=0.2. Proposition 5 bounds ρ\rho only for isometries, and that sentence went beyond it. The revision limits it to isometric codes (General Response, item 2). The proposition's conclusion and its Lean proof stand.

W2a (means vs worst case). The paper makes this substitution deliberately and states it.

  • §3.6 (p6 l287-288) says "On natural images the pixel metric is not the geometry the theory assumes, so we replace it by the encoder's own ℓ2\ell_2-normalised latent geometry".
  • Limitation (v) (p10 l507) says η^\hat\eta "measures a secant, not a derivative".

We agree that "direct finite-sample version" (submitted p5 l265) is too strong. The revision calls Eq. 4 a statistic motivated by Theorem 8 (§3.6, p6 l295-296) and lists the missing estimation-error bound in Limitation (v) (p10 l508).

W2b (constant β^\hat\beta). We agree with the fact, and the paper states it: σ^/β^\hat\sigma/\hat\beta is constant within an encoder (§3.6, p6 l300). This is by design.

Theorem 8 compares corruptions for a fixed encoder, where β\beta is fixed and only η\eta varies with the corruption. The within-encoder evidence is therefore evidence for the theory's corruption term.

  • The new displacement-only row (General Response, item 4) shows that this term is the strongest fixed-effects predictor in Table 2: −0.772 vs −0.621 for rotation and −0.420 for entropy.
  • It is also not a proxy for severity: averaged over the five encoders whose accuracy varies, η^\hat\eta correlates −0.89 with accuracy within each encoder, while severity alone gives −0.53 (App. K).

Dear Reviewer pm22,

Because of the discussion periods will close soon. Dont know if my comments have solved all your concerns ?

Thank you again for your time and thoughtful feedback.

Reviewer_uCWcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies why some pretrained encoders transfer better than others to few-shot classification of new classes under unseen corruptions (e.g. blur, noise). The authors propose a single measure, the "anisotropy ratio", which compares how much an encoder shrinks variation within a class to how well it keeps different classes apart. They prove that when this ratio is small enough relative to the class separation, a simple nearest-centroid classifier is guaranteed to be correct, and they argue that invertible encoders cannot improve this ratio while contrastive encoders can. They then propose a way to estimate the ratio using only images from the training classes, and test whether it predicts few-shot accuracy under corruption on MNIST, CIFAR-10/100 and CIFAR-10-C with seven encoders. Pooled across encoders the estimate does not predict accuracy, but within each encoder it correlates well with which corruptions hurt most. All theoretical results are formally verified in the Lean proof assistant.

Critical strengths
  • The question is relevant: predicting, from seen classes only, which shifts will degrade a frozen encoder for few-shot use is practically useful and under-explored compared with in-distribution transfer.
  • All theoretical results are formalised and machine-checked in Lean 4.
  • The comparison between discriminative, contrastive and generative models for OOD performance is genuinely interesting and the theory around it is sound.
Critical weaknesses
  • The experimental scope is limited, all image experiments use small 32×32 datasets (MNIST, CIFAR-10/100, CIFAR-10-C) with synthetic corruptions only. There is no evaluation on larger-scale or natural shifts (e.g. ImageNet-C, DomainNet, WILDS) or on standard few-shot benchmarks (e.g. miniImageNet, Meta-Dataset).
  • Two directly relevant works are not cited or discussed:
  1. Galanti, György & Hutter, On the Role of Neural Collapse in Transfer Learning (ICLR 2022), which bounds the few-shot nearest-class-center error on unseen classes in terms of class-feature variability collapse (CDNV).
  2. Goldblum et al., Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks (ICML 2020), which relates the intra- to inter-class variance ratio of features to one-shot classification accuracy.
  • It appears to me that Theorem 1 essentially restates the deterministic step of Theorem 1 in Goldblum et al. (2020, App. B), where points within a quarter of the class-mean separation are correctly classified by the triangle inequality. The multi-class extension follows by applying the same two-class argument to each competing class, so the novelty of the margin law looks limited.
  • The estimator η^\hat{\eta} requires applying the held-out corruption TT to images, which assumes that the user knows before-hand what the shift operator looks like. This is strong target knowledge, unavailable for natural shifts, so claims of "no access to the shifted test set" are overstated (If we know the nuisance transformation beforehand, then SupCon or SimCLR or even data augmentation using these nuisance transformations might already work, have you fairly trained the concurrent models with these nuisance transformations within the data augmentation strategy ?).
  • On clarity and wording: the prose is often "rhetorical" rather than scientific, e.g. “a frozen encoder is a geometry”, “the constant four is not decorative”, “this was not ceremony”. Also, several central constructs are hard to parse on a first read:
  1. what an “encoder-corruption block” is and how the 525 rows are formed;
  2. why encoder fixed effects are “the analysis the theory licenses”;
  3. where the mixture-of-experts setting (Props. 6–7, T4) comes from; that was not so clear and appeared a bit unrelated
Questions for authors
  1. How does ρ\rho differ from the CDNV of Galanti et al. (ICLR 2022) and the feature-clustering ratio of Goldblum et al. (ICML 2020). Does ρ^\hat{\rho} outperform these existing ratios as an accuracy predictor?

  2. How would the estimator be used when the shift operator TT is unknown, as for natural shifts? Conversely, if TT is known, were the competing encoders (e.g. SupCon, SimCLR) trained with TT as an augmentation, and how do they then perform?

  3. Can the authors report results on at least one larger-scale or natural-shift benchmark (e.g. ImageNet-C, DomainNet) or a standard few-shot benchmark (e.g. miniImageNet)?

  4. Did you use the shift operator TT as data augmentation when training the vanilla classification model or SupCon ?

2 comments
AuthorsOpenReview ↗

Response to Reviewer uCWc

Thank you.

Scale (Q3). New: Tiny-ImageNet-C (App. J, Table 8 last row): 200 classes, 64 px, 100 seen / 100 unseen, the same 7 encoders, severities 1, 3, 5; one seed, all 45 blocks. Its corrupted images have no clean counterparts, so η^\hat\eta is the unpaired class-mean shift.

  • Within every encoder whose accuracy varies, larger η^\hat\eta goes with lower accuracy (per-encoder rr −0.80 to −0.97 for CLIP, DINOv2, ResNet-18 and SupCon). It is −0.37 and −0.30 for pixels and the VAE, whose accuracy s.d. is ≈0.02.
  • Fixed-effects (FE) partial on those encoders: η^\hat\eta −0.697, ρ^total\hat\rho_{\text{total}} −0.745.
  • With the same seen-class access, rotation is stronger here (−0.80); entropy −0.64, CDNV −0.66.

ImageNet-C, DomainNet and WILDS remain untested (Limitation (i), p10 l499-500). Natural-shift benchmarks have no clean/corrupted pairs, so the paired η^\hat\eta of Eq. 4 is undefined there. The paper's setting is a known, probed corruption (Eq. 4).

Galanti et al.; Goldblum et al.; Theorem 1. §2 cites both. The paper does not present the triangle-inequality step as new: §3.2 (p4 l195) calls the proof "three lines of triangle inequality". The contribution is the sharp constant 4 (Prop. 3), its halving under orthogonality (Prop. 2) and the metric constant 2(1+ρ)2(1+\rho) (Thm. 4). All are checked in Lean. §2 credits Goldblum et al. (2020, App. B) for the deterministic step.

Q1 (ρ\rho vs CDNV). ρ\rho is a ratio of worst-case directional gains of the encoder relative to input distances. CDNV is a variance ratio of the features.

CDNV computed on clean data is constant within an encoder, so it cannot rank corruptions; the corruption term is what C3 adds.

  • Under matched access (every predictor reads only seen-class images), CDNV computed on corrupted seen-class features is weaker than η^\hat\eta on CIFAR-10-C (fixed effects −0.36 vs −0.77; Table 9). It is stronger on the five-transform datasets (CIFAR-10 −0.77 vs −0.70; CIFAR-100 −0.76 vs −0.68; 3 seeds each, runs of Table 8; −0.71 and −0.72 in the runs of Table 14; App. J).
  • Directional CDNV (Luthra et al.) is weaker than both on all four datasets (App. J, Table 14).

Corrupted-image CDNV measures within-class spread under the nuisance relative to class separation, the quantity the margin law bounds, so both results support the geometric account.

Q2 (unknown TT). The estimator is defined for a known corruption (Eq. 4 uses TT). The paper's question is which known corruptions will hurt a frozen encoder before deployment (Ethics statement, p11 l543-546).

The released results show that a probe of the corruption family is needed. Without probing cc, η^\hat\eta from other corruptions in its group gives −0.58, close to the severity-only −0.53; probing cc gives −0.89. We remove "no access to the shifted test set" from the abstract.

Q2/Q4 (augmentation). No training augmentation uses the held-out transforms; App. F (p21 l1120): "None is used during training, so the shift is genuine." Training on TT would make TT in-distribution, a different question from the one the paper asks.

SupCon and SimCLR train with random crop (padding 4), flip and colour jitter (0.2) on CIFAR, and a small random affine map on MNIST (data.py). Translation 0.15, brightness ×1.5 and, on MNIST, the two rotations are stronger versions of nuisances these partly cover; App. F says so (p21 l1121-1125).

App. J (Table 11) runs your experiment as a test of (A4), one seed. With TT in SupCon's augmentation, η^(T)\hat\eta(T) falls for all five transforms (e.g. rotate 60°: 0.644 → 0.348; blur: 0.559 → 0.312) and accuracy under TT rises (0.490 → 0.516; 0.513 → 0.518). Training on TT puts its nuisance inside the contracted subspace, and η^\hat\eta and accuracy move in the direction Theorem 8 predicts.

Clarity.

  • (1) A block is one (encoder, corruption, severity) triple: 7 × 15 × 5 = 525 The revision defines it at first use (§1, p2 l91-92: "7 × 15 × 5: encoders, corruptions, severities"); the Fig. 5 and Table 4 captions repeat it.
  • (2) Fixed effects: "ρOOD\rho_{\text{OOD}} is defined relative to a fixed encoder. Each family has its own scale of η^\hat\eta" (§4.3, p8 l431-p9 l460). "Licenses" is replaced by this reason (p9 l463-464).
  • (3) Props. 6-7 answer when region-specific encoders help; §3.4 states this (p5 l245: "The benefit needs a reason for a single encoder to be unable to serve all regions at once"). They stay in place so that theorem numbers do not change; the I3 null result moves to App. I.
  • (4) Prose: the "not decorative" and "not ceremony" sentences in §1 are rewritten literally.

Dear Reviewer uCWc,

As the discussion phase is coming to a close, we would like to ask whether our responses have fully addressed your follow-up concerns. If any questions remain, please let us know. We would be happy to provide further clarification.

Thank you again for your time and thoughtful feedback.

Reviewer_1B34OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies frozen representations for nearest-centroid few-shot classification under class and corruption shift. It defines within/between-class and semantic/nuisance gain ratios, proves conditional margin bounds, and proposes a forward-pass statistic based on prototype separation, class scatter, and corruption displacement. Experiments include controlled linear geometries and seven encoder families on small-image benchmarks, with the strongest image evidence being within-encoder correlations across CIFAR-10-C corruptions.

Critical strengths

The decomposed linear model makes the roles of semantic separation and nuisance contraction explicit.

The within-encoder CIFAR-10-C signal is substantial for several useful representations.

The manuscript separates the decomposed and metric margin conditions and documents assumptions and proof identifiers.

Critical weaknesses
  1. The central impossibility claim for invertible encoders is false. The abstract, introduction, Figure 1, and conclusion extend an isometry result to arbitrary invertible maps. For orthogonal semantic and nuisance subspaces, take E(s+n)=s+anE(s+n)=s+an with 0<a<10<a<1. This map is invertible, yet Equation (2) gives β=1\beta=1, η=a\eta=a, and ρ=a<1\rho=a<1. Indeed, the paper's own T1 diagonal encoder is exactly such a map whenever a>0a>0. Proposition 5 proves a ratio floor only for isometries; non-contraction alone supplies a lower bound on the numerator, and invertibility does not imply non-contraction. T2 tests identity and an orthogonal map, not the claimed class of flows. This invalidates the advertised geometric explanation of an inherent invertible-versus-contrastive gap.

  2. The empirical statistic lacks the theoretical justification and information-access conditions claimed for it. Equation (4) replaces worst-case directional gains with average normalized-latent distances, removes input-distance normalization, and uses average prototype separation instead of a lower semantic gain. No estimation-error or nonlinear-remainder bound connects these quantities to Theorem 8. A4 transfers an already valid nuisance bound; it does not establish that bound, semantic coverage of new classes, or recovery from finite samples. Moreover, class prototypes and scatter require seen-class labels, and each corruption-specific score requires paired clean/corrupted images or access to the transform TT. This is materially narrower than the abstract's estimator from unlabeled ID data alone. The baseline probes additionally access corrupted unseen-class images, so the claim of an identical information protocol needs clarification.

  3. The decisive empirical interpretation is insufficiently supported. Table 2's best result, −0.879-0.879, excludes Identity and SimCLR as chance-level encoders. However, Table 4 reports mean accuracies of 0.3740.374 and 0.3020.302 in a 5-way task whose chance accuracy is 0.200.20; their cross-corruption standard deviations, 0.0200.020 and 0.0150.015, are also close to the included VAE's 0.0210.021. The exclusion rule is therefore not established. The full fixed-effects comparison favors rotation (−0.621-0.621 versus −0.489-0.489), and the raw pooled comparison favors entropy. Crucially, within a fixed encoder, β^\hat{\beta} and σ^\hat{\sigma} are constant, so when β^>0\hat{\beta}>0, correlations with either proposed ratio are exactly those with η^\hat{\eta} alone. The paper needs a displacement-only control and corruption/severity-aware validation to establish additional value; 525 correlated blocks and in-sample fixed effects do not by themselves demonstrate calibrated prediction on a new encoder or corruption.

  4. The closest geometry-based few-shot theory is missing from the novelty assessment and experiments. Galanti et al. [1, 2] already connect class-distance-normalized variability to nearest-centroid transfer on new classes. Luthra et al. [3] directly study directional variance and finite-shot error. These are closer comparisons than general margin bounds or accuracy estimators alone. The corruption-conditional statistic may still add something, but that increment needs to be isolated theoretically and compared against CDNV and directional CDNV under matched data access.

  5. The value and significance of the task are insufficiently motivated. The paper motivates the work through reduced target-task labeling and advance assessment of out-of-distribution performance, but does not establish a concrete decision setting or demonstrate how the score would improve model selection, data collection, or deployment decisions. Under the current protocol, the score still requires seen-class labels and a specified corruption transform or paired probe samples. The paper does not quantify the cost of obtaining this information relative to directly labeling a small set of target queries for evaluation. The main evidence concerns score–accuracy correlations, without demonstrating better decisions, lower evaluation cost, or fewer failures under a fixed labeling or computation budget. These results therefore leave unclear when the task is worth adopting and what practical benefit it provides over direct evaluation.

  6. The diagnostic's robustness to realistic, unseen degradations is insufficiently validated. The image evidence comes from small-image datasets, five prescribed transforms, and CIFAR-10-C's fixed grid of 15 corruption types and 5 severity levels. Although these corruptions are held out from encoder training, each score still probes the specified transform on clean/corrupted seen-class pairs. This supports class transfer under a probed degradation, but does not establish reliability on unprobed corruption families, mixed or natural degradations, or a mismatch between probe and deployment severity. The authors acknowledge the small-data and uncertified-A4 limitations, yet provide no held-out-family or realistic-shift evaluation to resolve them. Consequently, the evidence for practical performance prediction beyond the tested conditions remains limited.

[1] On the Role of Neural Collapse in Transfer Learning.

[2] Generalization Bounds for Few-Shot Transfer Learning with Pretrained Classifiers.

[3] Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learning.

Questions for authors

How do you reconcile the invertibility floor with the invertible encoder used in T1?

With all encoders retained, does the proposed ratio predict accuracy better than corruption displacement alone?

What advantage does the method provide over CDNV and directional CDNV under matched data access?

Under the same budget, what measurable benefit does the score offer over directly evaluating a small labeled target set?

Does the diagnostic remain predictive on held-out corruption families and realistic mixed degradations?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1B34

Thank you for the detailed review.

W1 / Q1. Agreed (General Response, item 2): the T1 encoder is invertible with ρ=a<1\rho=a\lt 1. The text is limited to isometries, which Proposition 5 and T2 cover.

W2.

  • (a) No estimation bound: agreed; added as a limitation.
  • (b) "(A4) transfers … it does not establish that bound": agreed, and the paper says this. Limitation (iii) (p10 l504-505) reads "nothing certifies it for a corruption we have not seen", and App. M (p32 l1684) reads "a prediction under an assumption, not a guarantee".
  • (c) Labels: "label-free" means no unseen-class labels; prototypes use seen-class labels (§3.6). The abstract's "unlabeled" is corrected.
  • (d) Paired images or TT: this is the definition in Eq. 4 and App. F (p21 l1107-1108).
  • (e) Information protocol: see General Response, item 3. In the submitted runs η^\hat\eta and the baselines read the same unlabelled corrupted images; the seen-class-only runs reproduce the results (General Response, item 3).

W3 / Q2.

  • (i) The exclusion rule in the released code is accuracy s.d. < 0.02 over the 75 blocks (analysis.py). Identity is at 0.0198, SimCLR at 0.0148, VAE at 0.0214. "At chance" was the wrong description; the revision says "low-variance" (Table 2 caption, Limitation (vi), App. K).
  • Both splits are reported throughout (Tables 1-2; Limitation (vi), p10 l509-511). Results under other rules (App. K; fixed-effects (FE) partial, ρ^\hat\rho / η^\hat\eta / rotation): dropping SimCLR only gives −0.705 / −0.790 / −0.622; dropping three encoders gives −0.891 / −0.869 / −0.799.
  • (ii) Rotation and entropy winning some columns is reported in §1 (p2 l93-95), the Table 2 caption (p9 l446-448) and §4.4.
  • (iii) Q2: with all encoders, the displacement alone beats the ratio (−0.772 vs −0.489) and every baseline. The displacement is the theory's corruption term (General Response, item 4).
  • (iv) Out of sample: fit on 14 corruption families and predict the 15th. MAE is 0.017 vs 0.037 for the encoder mean (rotation 0.019, entropy 0.024; App. K).
  • New data, Tiny-ImageNet-C (one seed, all 45 blocks, App. J): the s.d. rule drops only SimCLR; FE η^\hat\eta −0.685 (7 encoders) / −0.697 (6), ρ^total\hat\rho_{\text{total}} −0.331 / −0.745; rotation is stronger there (−0.80 on encoders with variance).

W4 / Q3. Clean-data CDNV is constant within an encoder and cannot rank corruptions.

  • Under matched access (App. J, Table 9) η^\hat\eta beats corrupted-image CDNV on CIFAR-10-C (−0.77 vs −0.36) and the baselines (entropy −0.66, rotation −0.67); CDNV is stronger on the five-transform datasets (Response to uCWc).
  • Directional CDNV (Luthra et al., Sec. 4.1), on the same corrupted seen-class features: FE −0.26 on CIFAR-10-C vs η^\hat\eta −0.78 (all encoders, 3 seeds). It is also weaker than η^\hat\eta and CDNV on CIFAR-10, CIFAR-100 and Tiny-ImageNet-C (App. J, Table 14).

W5 / Q4. The paper does not claim savings in labelling cost. Its use case is a pre-deployment flag (Ethics statement, p11 l543-546).

  • In that setting, flagging each encoder's 15 worst of 75 blocks without target labels gives precision 0.72 for η^\hat\eta, 0.65 for rotation, 0.58 for entropy and 0.20 for chance, all seven encoders (App. K).
  • Matched labelled budget, on the five encoders whose accuracy varies (App. J, Table 10): estimating each block's accuracy from mm labelled target episodes reaches the precision of η^\hat\eta (0.836) between m=5m=5 (0.787) and m=10m=10 (0.845). That is 37,500 to 75,000 labelled target images per encoder.

W6 / Q5. The scope is a probed corruption (Limitations (i), (iii)).

  • Severity mismatch: probing at severity 3 and deploying at 5 gives Spearman −0.77 (−0.84 when matched; App. K).
  • Without probing the family: −0.58, close to the severity-only −0.53, so probing is needed and the revision says so.
  • Mixed corruptions (App. J, Table 13): the within-encoder signal persists (fixed effects −0.53, −0.73 on encoders with variance); CDNV, entropy and rotation are stronger there (rotation −0.77, −0.85).

The rebuttal corrects the invertibility overclaim and adds useful controls and generalization analyses. However, the added value of the proposed ratio remains unclear: displacement alone performs better when all encoders are retained, while the ratio’s advantage depends on the encoder subset. The empirical statistic also lacks a theoretical connection to the certified gains, and the annotation-cost comparison requires clarification regarding label reuse. Despite the improved evidence, the central contribution and practical advantage remain insufficiently established, so I retain my Weak Reject recommendation.

Thank you for acknowledging the corrections and additional analyses. We would like to clarify the contribution supported by the revised paper and address your remaining concerns.

1. The theoretical contribution and the empirical diagnostic. The paper separates semantic preservation from nuisance contraction and establishes explicit sufficient conditions for few-shot nearest-centroid classification: the sharp constant 44 under the decomposed linear model, its reduction to 22 under latent orthogonality, and the condition 2(1+ρ)2(1+\rho) in the metric setting. Proposition 5 establishes the floor specifically for isometries. Theorem 8 shows that the nuisance-gain bound transfers under N-coverage, assuming the stated semantic and class-geometry conditions. These results and their supporting lemmas are formalised in Lean 4.

The image statistic is motivated by this decomposition. As clarified in §3.6, it substitutes mean latent distances for worst-case directional gains and carries no estimation-error bound. We therefore distinguish the theoretical guarantees from the empirical evidence for the diagnostic.

2. What the displacement-only control establishes. Within a fixed encoder, β^\hat\beta and σ^\hat\sigma are constant, so both proposed ratios have exactly the same correlation with accuracy as displacement η^\hat\eta. Across encoders, division by β^\hat\beta changes their relative scales, including in the pooled fixed-effects analysis. The stronger full-encoder result for displacement therefore shows that the empirical normalization does not consistently improve aggregate prediction; we do not claim otherwise.

The positive finding is that the corruption-dependent component identified by the decomposition provides a useful diagnostic across unseen classes. Under matched seen-class access on CIFAR-10-C, retaining all seven encoders, η^\hat\eta achieves a fixed-effects correlation of −0.774-0.774, compared with −0.668-0.668 for rotation and approximately −0.355-0.355 for corrupted-image CDNV (Table 9). Directional CDNV gives approximately −0.26-0.26, compared with −0.78-0.78 for displacement in the corresponding runs (Table 14). This evidence does not depend on excluding the two low-variance encoders.

3. The practical decision and generalization evidence. The diagnostic addresses a specific decision: which probed corruptions are most likely to degrade a frozen encoder on unseen classes, before obtaining unseen-class query labels. In Appendix K, selecting each encoder’s 15 highest-displacement blocks identifies its 15 lowest-accuracy blocks with precision 0.720.72, compared with 0.650.65 for rotation, 0.580.58 for entropy and 0.200.20 for chance, across all seven encoders.

Separately, the held-out-family analysis fits each encoder’s regression using accuracy-labelled results from 14 corruption families and predicts the remaining family. Its MAE is 0.0170.017, compared with 0.0190.019 for rotation, 0.0240.024 for entropy and 0.0370.037 for the encoder-mean baseline. The ratio and displacement produce identical predictions in this analysis. This supports generalization after calibration on other families; it is distinct from the diagnostic computed without unseen-class labels.

The scope remains probed corruptions. We report that other predictors are stronger on mixed corruptions and that rotation is stronger on Tiny-ImageNet-C under matched access.

4. Annotation cost and label reuse. Your concern is valid. Table 10 compares flagging precision against accuracy estimates obtained from sampled target episodes. Its image counts represent episode-level image uses and do not establish unique annotation requirements: labels can be reused across episodes and corrupted versions of the same image. Consequently, these counts should not be interpreted as annotation savings. The supported result is the precision comparison under the stated episode-sampling protocol.

We hope these distinctions clarify the contribution: explicit geometric guarantees under stated assumptions, together with evidence that their nuisance component supports useful corruption-risk ranking across unseen classes. The full ratio’s additional empirical value is more limited, while the displacement diagnostic remains supported under matched access with all encoders retained.

Thank you so much for the responses

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of Revisions

We uploaded a revised PDF; changes are in blue. All page, line, section, figure, table and appendix numbers in our responses refer to the revised PDF.

Where each concern is addressed

  • pm22 W1, 1B34 W1 (invertibility) → Prop. 5 and the paragraph after it, §1, Fig. 1, contribution (ii), conclusion; Lean counterexample in App. C.5.
  • pm22 W2, 1B34 W2/W3 (estimator, constant β^\hat\beta, displacement-only control) → §3.6, Table 2 (η^\hat\eta row), §4.4.
  • 1B34 W2(e), uCWc Q2 (what η^\hat\eta reads) → §4 protocol, Fig. 2 caption, App. J "Additional results on the estimator", Table 8 (seen-class probes).
  • uCWc Q1, 1B34 W4/Q3 (CDNV, directional CDNV) → §2, App. J, Tables 9 and 14.
  • 1B34 W3 (exclusion rule, held-out families) → §4.3, Tables 1-2 captions, App. K.
  • 1B34 W5/Q4 (labelled budget) → App. J, Table 10.
  • 1B34 W6/Q5 (unprobed, mixed) → Limitation (vii), App. K (unprobed), App. J and Table 13 (mixed).
  • uCWc Q2/Q4 (training with TT) → App. F (augmentations), App. J, Table 11.
  • uCWc Q3 (scale) → App. J (Tiny-ImageNet-C), Table 8 last row, Limitation (i).
  • uCWc (Goldblum, Galanti; Thm. 1 novelty) → §2.
  • uCWc clarity → §1 (block definition), §4.3 (fixed effects), §3.4 (Props. 6-7); prose → §1.

Other changes

  1. Abstract, introduction, limitations and conclusion keep the submitted structure. Title unchanged. Blue edits there:
    • "invertible" → "isometric" (with "so any gain must come from anisotropic contraction"); "fundamentally a property of" → "governed by";
    • the estimator's inputs (seen-class labels and a probe of the corruption) and the within-encoder scope of the claim;
    • the exclusion-rule wording; the encoder and benchmark counts and the closing clause "ensuring mathematical rigor" removed from the abstract;
    • in §1, the "not decorative" and "not ceremony" sentences rewritten, and the Lean paragraph now states the two proved regimes (4ρ4\rho in the decomposed model, 2(1+ρ)2(1+\rho) in the bi-Lipschitz setting);
    • Limitation (v) adds the missing estimation-error bound; a new Limitation (vii) states the information η^\hat\eta needs.
    • Long sentences are split at existing clause boundaries, and a few phrases are made precise (e.g. an isometry is "no better than the raw input", not "the worst case").
    • Also: a short problem formulation (§3.1), a notation table (App. A), and a one-line opener and takeaway in each experiment subsection. Each theorem and proposition title and each figure and table caption opens with a one-line statement of its result.
  2. Fig. 1 redrawn as a compact four-panel figure: input classes, an isometry (ρ=1\rho=1), a contraction (ρ=1/4\rho=1/4), and both encoders placed on the margin law Δ/r>4ρ\Delta/r\gt 4\rho (an illustration with no data; it does not call a normalising flow isometric). Fig. 2 drops its header and footer rows; the caption gives the sections. The Lean map in App. D is now a table instead of a figure, and the "Impact statement" is titled "Ethics statement", as in the ICLR guidelines.
    • New experiments validate existing claims and are reported in App. J (seen-class probes, matched access, labelled budget, training with TT, mixed corruptions, trained flow, Tiny-ImageNet-C) and App. K (re-analyses), with one-clause pointers in §4.
  3. Appendix proofs rewritten for readability: proof ideas first, standard results cited, routine algebra grouped, one shared lemma (Lemma 9); the Lean mapping is in App. D.1.
    • Prop. 7 now requires a nonzero leaking direction vv (the previous form was met by v=0v=0).
    • Definition 1 ranges over the labelled data set.
    • Side conditions are stated explicitly (Prop. 2 and Thm. 4 ratio forms, Thm. 4 on the data set, Props. 5-6, Thm. 8); the K-shot remark is limited to linear encoders.
    • Thm. 8 now says ρOOD=η/β\rho_{\mathrm{OOD}}=\eta/\beta "uses no unseen-class labels" instead of "is measured on seen data alone".
    • All conclusions and constants are unchanged; the Lean statements follow these changes and rebuild with only Lean's three standard axioms (App. D).
  4. To stay within 10 pages, experiment I3 (now Fig. 13) and the I1 and baseline figures (now Figs. 6 and 7) move to App. I. Theorem numbers are unchanged; figure numbers from Fig. 5 on change.
  5. The supplementary material is updated. It adds the revised Lean files (17 statements and 38 step lemmas, with the logs of a clean build), the scripts and logs of the App. J experiments, and the generator of the App. J tables. The code and results of the submission are unchanged.
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers. We restate what the paper claims, then answer the shared points.

1. Contributions.

  • (C1) The anisotropy ratio and a margin law for one-shot nearest-centroid classification: sharp constant 4 (Prop. 3), 2 under latent orthogonality (Prop. 2), 2(1+ρ)2(1+\rho) in the bi-Lipschitz metric setting (Thm. 4).
  • (C2) An isometry floor and a mixture result.
  • (C3) N-coverage (Thm. 8): under (A4) the ratio transfers to unseen nuisances; it motivates a forward-pass estimator that needs no unseen-class labels (§3.6).
  • (C4) Lean 4 proofs of all 17 results; the formalisation separates the 4ρ4\rho and 2(1+ρ)2(1+\rho) regimes (§1, p2 l102-103).
  • (C5) Validation on a controlled geometry and four image benchmarks (525 CIFAR-10-C blocks) against four label-free baselines.

The paper states that the image signal is within-encoder (§1, p2 l89-90; Fig. 5a; Limitation (ii)).

2. Isometry floor (pm22 W1, 1B34 W1). The reviewers are right that the sentence at submitted p4 l208 says more than Proposition 5 proves.

  • The proposition bounds ρ\rho for isometries (η=β=1\eta=\beta=1, so ρ=1\rho=1); E(s,n)=(10s,2n)E(s,n)=(10s,2n) and the T1 encoder are invertible with ρ<1\rho\lt 1.
  • The statement and its Lean proof stand; the revision limits the surrounding text to isometries.
  • T2 (Fig. 4a: identity and an orthogonal map) tests exactly this case.

3. What η^\hat\eta reads (1B34 W2, uCWc). η^\hat\eta and the baselines read the same unlabelled corrupted images, as 1B34 asked, but the sentence "seen-class images only" in §4 (submitted p6 l299) is wrong. Seen-class probes give the same result (Table 8).

  • Eq. 4 is written with the transform TT, and App. F (p21 l1107-1108) says η^\hat\eta uses aligned clean/corrupted pairs. App. G (p22 l1163-1164) says the baselines use "the corrupted unlabelled images".
  • In the code (exp_cifar10c.py, exp_image.py), η^\hat\eta also reads unlabelled unseen-class images.

Under (A4), η\eta is a property of the encoder and the nuisance, not of the classes. App. J (Table 8) repeats every image experiment with η^\hat\eta computed on held-out seen-class test images, never used for training, and in the same run on unseen-class images. Fixed-effects (FE) partial correlation of ρ^total\hat\rho_{\text{total}} (encoders with accuracy variance), seen vs unseen probes:

  • CIFAR-10 −0.927 vs −0.930 (3 seeds)
  • CIFAR-100 −0.911 vs −0.915 (3 seeds)
  • CIFAR-10-C −0.872 vs −0.880 (3 seeds; Table 2 reports −0.879)
  • MNIST −0.526 vs −0.480 (1 seed).
  • Tiny-ImageNet-C −0.745 vs −0.583 (1 seed; the unseen probe is an unpaired, label-free shift).

The described protocol gives the paper's result.

4. Within an encoder the signal is η^\hat\eta (pm22 W2, 1B34 W3). Within an encoder only η^\hat\eta varies, by construction. The paper states that σ^/β^\hat\sigma/\hat\beta "is constant within an encoder" (§3.6, p6 l300). This follows from Theorem 8: for a fixed encoder, β\beta belongs to the encoder, and the only term that depends on the corruption is η\eta.

Within-encoder correlations equal to those of η^\hat\eta are therefore expected. We added the displacement-only control to Table 2 (released results, no new runs):

CIFAR-10-C, 525 rowsraw rrFE partial (7 enc.)FE partial (5 enc.)
ρ^total\hat\rho_{\text{total}}−0.333−0.489−0.879
η^\hat\eta (numerator of ρ^OOD\hat\rho_{\text{OOD}})+0.460−0.772−0.800
rotation−0.673−0.621−0.795
entropy−0.845−0.420−0.576

The theory's corruption term is the strongest fixed-effects predictor on all 7 encoders. The full ratio is the strongest on the 5-encoder subset. β\beta is needed to compare encoders, and the paper already reports that the pooled comparison is weak (Limitation (ii)).

5. New analyses of the released results (details in the individual responses):

  • Leave-one-corruption-family-out: MAE 0.017 vs 0.037 for the encoder-mean baseline (App. K).
  • Flagging each encoder's 15 worst of 75 blocks: precision 0.72 vs 0.65 (rotation), 0.58 (entropy), 0.20 (chance), 7 encoders (App. K).
  • Probed η^\hat\eta against a severity-only control: −0.89 vs −0.53 (App. K).
  • Matched access on CIFAR-10-C (every predictor reads seen-class images only): η^\hat\eta −0.774, entropy −0.664, rotation −0.668, CDNV −0.355 (3 seeds; App. J, Table 9). On encoders with variance: ρ^total\hat\rho_{\text{total}} −0.872, rotation −0.807, η^\hat\eta −0.795.
  • Tiny-ImageNet-C (200 classes, 64 px, 7 encoders, one seed, all 45 blocks; unpaired η^\hat\eta since no clean pairs exist): FE on encoders with accuracy variance η^\hat\eta −0.697, ρ^total\hat\rho_{\text{total}} −0.745 (App. J, Table 8). With matched seen-class access, rotation is stronger (−0.80); entropy −0.64, CDNV −0.66.
  • Directional CDNV (Luthra et al., Sec. 4.1) on the same corrupted seen-class features, FE all encoders: −0.33 / −0.36 / −0.26 / −0.16 on CIFAR-10 / CIFAR-100 / CIFAR-10-C / Tiny-ImageNet-C, vs η^\hat\eta −0.69 / −0.68 / −0.78 / −0.68 (App. J, Table 14).

Dear Reviewers,

We have submitted our rebuttal and revised paper, addressing your comments and suggestions.

We hope the revisions help clarify our contributions and address your concerns.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.