- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposed a study on frozen CLIP features once a few-shot classifier has narrowed a prediction to its top-K = 15 candidate classes. The paper makes three main claims: 1. atoms that are prevalent across probable classes are also the most strongly activated. 2. Global cross-entropy rewards evidence that raises all candidate logits together, even though such evidence cancels within the candidate set. 3. Removing the decoded shared component U_S, which carries about 68% of the feature norm, leaves candidate-local reranking unaffected. Experiments use 11 datasets, CLIP RN50 and ViT-B/32, and three first-stage classifiers (linear probe, CoOp, CoCoOp).
Critical strengths
The global/local decomposition of the cross-entropy (Eq. 5) is a simple and useful way to explain why a globally trained head retains evidence that is non-discriminative within the candidate set.
The reranking gains replicate across 11 datasets, three first-stage classifiers, 4/8/16 shots, and two encoder families, with sweeps over K and the prevalence threshold. The Z vs. V_S equivalence and the low standalone accuracy of U_S are robust across all of these.
Critical weaknesses
CLIP embeddings are strongly anisotropic, with a large common mean direction. Atoms that fire in ≥75% of samples of every candidate class are plausibly dominated by this dataset-mean component, which any candidate-restricted linear head ignores trivially. A large ||U_S||/||Z|| that can be dropped without loss is then expected rather than revealing.
Hybrid accuracy is flat (69.74–69.91) over the whole threshold range, including τ = 0, and Accr drops about 3 points at τ = 0. So the Table 1 equivalence mostly reflects the frozen global logits on Z, which still see the "removed" component. The abstract's statement that "most of the feature can be removed without changing the answer" could therefore not be supported by this.
There is no control reranker trained with the full-label softmax on Z and fused with the same λ. That ablation is needed to separate Proposition 1's mechanism from a generic ensembling effect.
The 74% reduction in |b_U| compares heads with different weight scales; it should be normalized. Nothing about this is discussed in paper.
Questions for authors
What fraction of U_S, by norm, is explained by the dataset-mean CLIP feature, or by its top-1 to top-3 principal components? Does subtracting the mean feature, or a norm-matched random atom set, reproduce the Z ≈ V_S result and the 13% U_S accuracy?
Does K = 15 and τ = 0.75 (Figs. 7–8) are chosen on test accuracy or on a held-out validation split of the few-shot data?
Please report |b_U| normalized by the head's weight scale, or by the residual margin m_V, so that the stage-1 and reranker values are comparable. If already did, please clearify this in paper.