acceptodds
Under review as a conference paper at ICLR 2027

A Geometric Diagnostic for Frozen Foundation Features

Abstract

Classification on domain-specific data often pairs frozen foundation-model features with a lightweight classifier, yet uneven per-class performance is usually attributed to class imbalance rather than representation geometry. On PVEL-AD, a balanced linear probe on frozen DINOv2 features recalls 94% of one defect class from 56 labeled examples but only 26% of another from 96, showing that class frequency alone does not explain the failure pattern. We use Neural Collapse as a geometric reference and show that anisotropy imposes a strictly positive lower bound on deviation from the simplex equiangular tight frame (ETF), with a median 99% of the measured deviation across our benchmark accounted for by this floor. We introduce the Geometric Collapse Score (GCS), the mean pairwise cosine of L2-normalized class centroids, as a measure of how tightly an encoder compresses a domain. Across eleven datasets and four encoders, dataset ordering by GCS is nearly invariant, with pairwise Spearman correlations from 0.96 to 1.00. Across 55 classes from seven specialized datasets, per-class GCS strongly correlates with recall (), while class frequency does not (). Frozen GCS also separates two empirical regimes of adaptation response. LoRA improves balanced accuracy on eight of nine specialized datasets and reduces it on both natural-image controls, although the boundary is an ordering rather than a universal threshold. An ETF-alignment objective improves PVEL-AD balanced accuracy by 3.3 points over an identical-pipeline control while reducing cone alignment, with class-level gains remaining within seed noise and no meaningful transfer to ResNet-50. Together, these results identify anisotropic feature geometry as a measurable failure mode of frozen representations and position GCS as a practical pre-adaptation diagnostic for when the representation itself warrants intervention.

Primary area:
foundation or frontier models, including LLMs
Keywords:
foundation models, representation learning, frozen features, neural collapse, feature geometry, anisotropy, geometric collapse score, transfer learning, domain adaptation, LoRA, long-tailed recognition, computer vision

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_G6DhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper argues that uneven per-class recall of linear probes on frozen foundation features can be attributed to the geometry of the representation, not to class imbalance. It introduces the Geometric Collapse Score (GCS), the mean pairwise cosine of L2-normalized class centroids. Proposition 1 shows that centroids aligned with a dominant feature direction cannot reach the simplex ETF of Neural Collapse. On eleven datasets and four encoders, the paper reports three findings: (i) The encoders order the datasets by GCS almost identically. (ii) Per-class GCS correlates with recall. (iii) Frozen GCS separates datasets where LoRA helps from two controls where it does not. An ETF-alignment loss during LoRA fine-tuning gives a modest gain on PVEL-AD.

Critical strengths
  • S1: The practical question, whether to probe frozen features or adapt the encoder, is relevant.
  • S2: The benchmark is broad, with eleven datasets, four encoders
  • S3: All four encoders order the datasets by GCS almost identically, even a supervised CNN (ResNet-50) next to three self-supervised or contrastive ViTs. This indicates that the ordering reflects the data domain rather than a particular encoder.
  • S4: For each dataset, the cosine between two class centroids tracks whether the two classes get confused, on seven of eight datasets. This is the paper's most direct within-dataset evidence.
  • S5: The paper is honest about its limits. It admits that other known statistics predict recall as well as GCS, and then states what GCS adds instead. It works on class pairs and can be used as a training loss (lines 152–153). It also admits that there is no clear GCS threshold for when to adapt, that the gain of the ETF loss is uncertain, and that the loss does not work on ResNet-50 (lines 464–472).
Critical weaknesses
  • W1: The paper is hard to read and not carefully prepared. The abstract heading appears twice, three sentences of the introduction are repeated word for word (lines 47–54), Appendix I contains internal pipeline language ("v2 analysis"), and Appendix G refers to a result in Section 6 that is not there. The AI use statement says that the authors reviewed all AI-assisted content, but these errors suggest otherwise. Beyond them, the writing is dense: most sentences carry several claims and numbers, and much of the key evidence sits in the appendices.
  • W2: The claim that GCS ranks classes within a dataset is not supported by corresponding evidence. The only evidence is one correlation over classes pooled from seven datasets (line 323). Such a pooled value can come from differences between datasets alone. Datasets with high GCS simply have lower recall overall. The paper's own Table 4 suggests this is the case. Shifting each dataset's GCS by a constant keeps the order within each dataset, yet the correlation drops sharply. Per-dataset correlations are needed to support the claim.
  • W3: GCS is computed on class means without first subtracting the global mean (lines 175–179), unlike Neural Collapse. If the same vector is added to all features, every class mean moves with it, all cosines rise, and the cone looks tighter. Yet the differences between classes stay the same, and a linear classifier on the raw features absorbs the shift in its bias. Proposition 1 therefore shows that an uncentered ETF is out of reach, not that classes are harder to separate. Statements such as "any intervention that works must widen the cone" (line 314) go beyond this. The paper itself notes in the next sentence that this is an empirical question, but doesnt really show that it causes the failures (see W2).
  • W4: The claim that GCS tells you when to adapt the encoder (here with LoRA) is not entirely supported. In Table 1, LoRA helps on the specialized datasets but not on the two controls. The paper attributes this to classes are already being well spread out, so adaptation destroys useful pretrained structure (lines 368–370). Two observations contradict this. First, the controls also have far more classes and far fewer labels per class (Table 2), so geometry is not the only possible explanation. Second, LoRA does not make the features worse. A probe trained on the LoRA-adapted features does at least as well as one on the frozen features (Table 12). The drop therefore does not point to the features.
  • W5: The ETF loss does not show that geometry causes the failures, although the paper says Section 6 tests exactly this (lines 393–396). The loss is designed to lower GCS, so a lower GCS is not evidence on its own, and accuracy follows only weakly. The gain on PVEL-AD is comparable to the seed spread (lines 419–420), and accuracy stays flat while GCS falls with the EMA decay (lines 439–440). Moreover, plain LoRA on PVEL-AD already reaches 0.753 in Table 1, more than the 0.744 obtained with the loss (line 415).
  • W6: Some results rely on very small class numbers. Five PVEL-AD classes have far fewer than the roughly 20 labels per class that GCS is stated to require (App. F).
  • W7: The paper states that the experiments are "fully specified", enabling reproducibility, but the optimizer, learning rate, batch size, augmentation, and early stopping are missing. Related work might be incomplete. The paper states that its core question "has not been asked" (see Section 1). I gave it a quick check and found two papers that already come close: Galanti, Gyorgy & Hutter, "On the Role of Neural Collapse in Transfer Learning", ICLR 2022 and Wang et al., "How Far Pre-trained Models Are from Neural Collapse on the Target Dataset Informs their Transferability", ICCV 2023.
Questions for authors
  • Q1 (W1): Whats the reason for the major presentation flaws, and how will the manuscript be revised and checked?
  • Q2 (W2): What are the per-dataset Spearman correlations between per-class GCS and recall for the seven datasets?
  • Q3 (W3): Does the GCS–recall relationship still hold when the class means are computed on globally centered features?
  • Q4 (W4): Do Tables 1 and 12 use the same LoRA runs? If adaptation is measured with the same probe on frozen and adapted features, does the regime split remain?
  • Q5 (W5): What explains the difference between plain LoRA in Table 1 and the LoRA control in Section 6?
  • Q6 (W6): How reliable are the GCS values of the five PVEL-AD classes with very few labels, and do the conclusions hold without them?
  • Q7 (W7): Can you provide the missing details regarding reproducability, and explain how the question of the paper differs from the related work stated in w7?
Reviewer_gNWBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper computes the average cosine similarity between class centroids of frozen foundation model features and calls it the Geometric Collapse Score (GCS). It argues that because frozen features sit in a narrow cone, their un-centered class centroids cannot reach the Neural Collapse “ideal” (simplex ETF) geometry, and that GCS measures how crowded a dataset's classes are. The experiments cover eleven datasets and four encoders, with additional LoRA adaptation and ETF-alignment experiments intended to establish the diagnostic usefulness.

Critical strengths
  1. The motivating question is practical: frozen-feature probes on specialized data fail unevenly, and frequency doesn't explain it (the ShortCircuit vs. Crack example is a nice hook).
  2. The benchmark covers underexplored domains e.g. steel defects, dermoscopy etc., which is useful beyond the usual natural-image settings.
  3. The experiments include multiple encoders, comparisons with scatter-based statistics, and some controlled adaptation experiments. The authors also acknowledge several limitations, including the non-transportability of the GCS scale and the limited significance of the ETF-alignment gains.
Critical weaknesses
  1. The contribution is very small and over-decorated. GCS is simply the mean pairwise cosine of normalized class means. Proposition 1 follows from the non-negativity of the squared norm of the summed residual vectors. In fact, GCS itself can be written exactly as $ \frac{\left|\sum_{i=1}^{C}\hat{\mu}_i\right|_2^2 - C}{C(C-1)}$, making itself a measure of the resultant direction of the centroids. The mathematical novelty is limited.

  2. Comparison to Neural Collapse appears conceptually problematic: standard NC2 concerns class means after centering by their global mean, whereas GCS uses uncentered class means. Why cannot a common feature offset make all raw centroid directions highly aligned without changing their relative positions?

  3. Achieving the ideal average cosine −1/(C−1) only requires the normalized centroid vectors to sum to zero; it does not require all pairwise cosines to equal the ETF value. Very different and non-equiangular class arrangements can produce the same GCS. Interestingly, the proposed ETF-alignment loss (Eq. 3) penalizes individual pairwise deviations and is a stronger objective than minimizing GCS itself.

  4. It's unclear why anyone would use GCS over existing statistics: Fisher ratio, normalized scatter, and Silhouette predict pooled per-class recall as well as or better than GCS. The paper argues that GCS is pair-resolved and differentiable, but pairwise cosine similarity and centroid distances are already standard geometric quantities.

  5. The closest prior work is missing and not compared against. See Wang et al., Li et al. and Liang et al.

  6. If we add a sufficiently large shared direction, cannot it push GCS arbitrarily close to 1, even though the class differences remain unchanged?

  7. The presentation sometimes overstates relatively simple observations with terminology. A more direct account of what GCS offers beyond existing separation measures would improve the paper.

Ref.:

Wang et al. https://openaccess.thecvf.com/content/ICCV2023/papers/Wang_How_Far_Pre-trained_Models_Are_from_Neural_Collapse_on_the_ICCV_2023_paper.pdf

Li et al. https://mlanthology.org/tmlr/2024/li2024tmlr-understanding-a/

Wang et al. https://proceedings.neurips.cc/paper_files/paper/2022/hash/702f4db7543a7432431df588d57bc7c9-Abstract.html

Questions for authors
  1. Since the standard NC2 simplex ETF result is defined for globally centered class means, why does the paper use uncentered centroids and compare their pairwise cosines directly to the ETF value?

  2. GCS can be expressed entirely through the norm of the sum of normalized centroids, how do the authors justify interpreting it as a measure of deviation from simplex ETF geometry rather than simply measuring a shared centroid direction?

  3. The pooled GCS–recall correlation drops from −0.785 to −0.309 after dataset-level background subtraction which does not change within-dataset class rankings. Could the authors demonstrate that GCS reliably ranks difficult classes within individual datasets?

  4. Why should GCS be preferred to Fisher ratio, normalized scatter, or Silhouette, which already match or exceed its reported correlations?

  5. How much can GCS change without a corresponding change in balanced linear-probe accuracy?

Reviewer_e9BSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the relationship between the geometry of frozen foundation model representations and downstream classification performance. The authors introduce the Geometric Collapse Score (GCS), defined as the average pairwise cosine similarity between normalized class centroids, to characterize representation anisotropy and identify potential classification failures.

The paper derives a lower bound on the deviation from simplex ETF geometry under positive cone alignment. Experiments across eleven datasets and four pretrained encoders show that GCS correlates with per-class recall and that dataset rankings by GCS are highly consistent across encoders. The authors further investigate whether GCS can predict the benefits of model adaptation and introduce an ETF-alignment objective for LoRA fine-tuning.

Critical strengths
  1. Interesting problem and practical motivation. The paper investigates an important question regarding the geometric properties of frozen foundation model representations, particularly in specialized domains where pretrained features may not be well suited to downstream classification. The proposed GCS is simple to compute and provides an interpretable characterization of class-centroid similarity.

  2. Extensive empirical analysis. The paper evaluates multiple datasets and pretrained encoders, including DINOv2, CLIP, and ResNet-50. The experiments investigate the relationship between representation geometry, per-class recall, and the benefits of model adaptation. The additional analyses of class-pair confusion and ETF-based fine-tuning provide useful empirical observations.

Critical weaknesses
  1. The connection to Neural Collapse and the interpretation of the theoretical bound require further clarification. The proposed GCS is computed using normalized, uncentered class centroids, whereas the classical NC2 property characterizes the geometry of globally centered class means. This distinction is important because a shared global offset can produce highly aligned uncentered class centroids even when the centered class means form an exact simplex ETF. Although Proposition 1 provides a valid lower bound for the proposed uncentered statistic, it does not directly establish that the classical NC2 geometry is unattainable. Consequently, the interpretation of anisotropy as a fundamental obstruction to Neural Collapse appears stronger than what the current analysis establishes.

  2. The unique practical advantages of GCS are not sufficiently established. Although GCS correlates with per-class recall, the paper reports that existing geometry-based measures, including the Fisher discriminant ratio and Silhouette score, achieve comparable or stronger correlations. Furthermore, the proposed ETF-alignment objective provides relatively modest accuracy improvements, with the reported 3.3-point gain on PVEL-AD being comparable to the variation across random seeds. The improvement also does not meaningfully transfer to ResNet-50. These results raise questions about whether GCS provides sufficient additional predictive or practical value over existing geometric diagnostics.

Questions for authors
  1. Regarding the relationship to Neural Collapse: Classical NC2 is defined using globally centered class means, whereas GCS uses uncentered class centroids. Could the authors clarify how this difference affects the interpretation of Proposition 1? In particular, since a common offset can create strong cone alignment without changing the centered ETF geometry, under what conditions does the proposed anisotropy floor imply a genuine obstruction to classical NC2?

  2. Regarding the predictive value of GCS: The paper reports strong correlations between GCS and per-class recall, but some existing metrics achieve comparable or better results. Could the authors provide a more direct comparison showing the additional predictive information contributed by GCS? In particular, would its conclusions remain similar when correlations are evaluated separately within each dataset or after controlling for dataset identity?

  3. Regarding the adaptation experiments: The ETF-alignment objective produces relatively modest improvements, and the reported gains are often comparable to seed-level variation. Could the authors provide stronger evidence that explicitly optimizing the proposed geometric objective consistently improves downstream accuracy? Additionally, can the authors clarify why the method does not meaningfully transfer to ResNet-50 despite the similar GCS dataset rankings across encoders?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.