- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper argues that uneven per-class recall of linear probes on frozen foundation features can be attributed to the geometry of the representation, not to class imbalance. It introduces the Geometric Collapse Score (GCS), the mean pairwise cosine of L2-normalized class centroids. Proposition 1 shows that centroids aligned with a dominant feature direction cannot reach the simplex ETF of Neural Collapse. On eleven datasets and four encoders, the paper reports three findings: (i) The encoders order the datasets by GCS almost identically. (ii) Per-class GCS correlates with recall. (iii) Frozen GCS separates datasets where LoRA helps from two controls where it does not. An ETF-alignment loss during LoRA fine-tuning gives a modest gain on PVEL-AD.
Critical strengths
- S1: The practical question, whether to probe frozen features or adapt the encoder, is relevant.
- S2: The benchmark is broad, with eleven datasets, four encoders
- S3: All four encoders order the datasets by GCS almost identically, even a supervised CNN (ResNet-50) next to three self-supervised or contrastive ViTs. This indicates that the ordering reflects the data domain rather than a particular encoder.
- S4: For each dataset, the cosine between two class centroids tracks whether the two classes get confused, on seven of eight datasets. This is the paper's most direct within-dataset evidence.
- S5: The paper is honest about its limits. It admits that other known statistics predict recall as well as GCS, and then states what GCS adds instead. It works on class pairs and can be used as a training loss (lines 152–153). It also admits that there is no clear GCS threshold for when to adapt, that the gain of the ETF loss is uncertain, and that the loss does not work on ResNet-50 (lines 464–472).
Critical weaknesses
- W1: The paper is hard to read and not carefully prepared. The abstract heading appears twice, three sentences of the introduction are repeated word for word (lines 47–54), Appendix I contains internal pipeline language ("v2 analysis"), and Appendix G refers to a result in Section 6 that is not there. The AI use statement says that the authors reviewed all AI-assisted content, but these errors suggest otherwise. Beyond them, the writing is dense: most sentences carry several claims and numbers, and much of the key evidence sits in the appendices.
- W2: The claim that GCS ranks classes within a dataset is not supported by corresponding evidence. The only evidence is one correlation over classes pooled from seven datasets (line 323). Such a pooled value can come from differences between datasets alone. Datasets with high GCS simply have lower recall overall. The paper's own Table 4 suggests this is the case. Shifting each dataset's GCS by a constant keeps the order within each dataset, yet the correlation drops sharply. Per-dataset correlations are needed to support the claim.
- W3: GCS is computed on class means without first subtracting the global mean (lines 175–179), unlike Neural Collapse. If the same vector is added to all features, every class mean moves with it, all cosines rise, and the cone looks tighter. Yet the differences between classes stay the same, and a linear classifier on the raw features absorbs the shift in its bias. Proposition 1 therefore shows that an uncentered ETF is out of reach, not that classes are harder to separate. Statements such as "any intervention that works must widen the cone" (line 314) go beyond this. The paper itself notes in the next sentence that this is an empirical question, but doesnt really show that it causes the failures (see W2).
- W4: The claim that GCS tells you when to adapt the encoder (here with LoRA) is not entirely supported. In Table 1, LoRA helps on the specialized datasets but not on the two controls. The paper attributes this to classes are already being well spread out, so adaptation destroys useful pretrained structure (lines 368–370). Two observations contradict this. First, the controls also have far more classes and far fewer labels per class (Table 2), so geometry is not the only possible explanation. Second, LoRA does not make the features worse. A probe trained on the LoRA-adapted features does at least as well as one on the frozen features (Table 12). The drop therefore does not point to the features.
- W5: The ETF loss does not show that geometry causes the failures, although the paper says Section 6 tests exactly this (lines 393–396). The loss is designed to lower GCS, so a lower GCS is not evidence on its own, and accuracy follows only weakly. The gain on PVEL-AD is comparable to the seed spread (lines 419–420), and accuracy stays flat while GCS falls with the EMA decay (lines 439–440). Moreover, plain LoRA on PVEL-AD already reaches 0.753 in Table 1, more than the 0.744 obtained with the loss (line 415).
- W6: Some results rely on very small class numbers. Five PVEL-AD classes have far fewer than the roughly 20 labels per class that GCS is stated to require (App. F).
- W7: The paper states that the experiments are "fully specified", enabling reproducibility, but the optimizer, learning rate, batch size, augmentation, and early stopping are missing. Related work might be incomplete. The paper states that its core question "has not been asked" (see Section 1). I gave it a quick check and found two papers that already come close: Galanti, Gyorgy & Hutter, "On the Role of Neural Collapse in Transfer Learning", ICLR 2022 and Wang et al., "How Far Pre-trained Models Are from Neural Collapse on the Target Dataset Informs their Transferability", ICCV 2023.
Questions for authors
- Q1 (W1): Whats the reason for the major presentation flaws, and how will the manuscript be revised and checked?
- Q2 (W2): What are the per-dataset Spearman correlations between per-class GCS and recall for the seven datasets?
- Q3 (W3): Does the GCS–recall relationship still hold when the class means are computed on globally centered features?
- Q4 (W4): Do Tables 1 and 12 use the same LoRA runs? If adaptation is measured with the same probe on frozen and adapted features, does the regime split remain?
- Q5 (W5): What explains the difference between plain LoRA in Table 1 and the LoRA control in Section 6?
- Q6 (W6): How reliable are the GCS values of the five PVEL-AD classes with very few labels, and do the conclusions hold without them?
- Q7 (W7): Can you provide the missing details regarding reproducability, and explain how the question of the paper differs from the related work stated in w7?