acceptodds
Withdrawn from ICLR 2027

Understanding Candidate-Shared Evidence in Visual Foundation Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

What parts of a CLIP feature matter for a prediction? We find that once the classifier has narrowed an image to a few likely classes, most of the feature can be removed without changing the answer. We take the top- classes from a frozen classifier and use a sparse autoencoder to split the CLIP feature into concepts. Across 11 datasets, the concepts that all candidates share are also the most strongly expressed, and they make up roughly two thirds of the feature. We show why the classifier trained on top of CLIP keeps them: its cross-entropy loss over all classes rewards evidence that separates the likely classes from all the others, even when that evidence does nothing to tell the likely classes apart. We then train two small rerankers that only choose among the candidates, one on the full feature and one with the shared concepts deleted. Both improve three standard few-shot CLIP classifiers, a linear probe and two prompt-learning methods, by 1.3 to 5.9 points at 16 shots, and they match within 0.2 points. The deleted concepts alone reach 13% accuracy among the candidates against 68% for what remains, and the full-feature reranker learns to ignore them on its own, cutting their influence on the score by 74%. Our results show that global recognition relies on evidence the candidates share, and that local discrimination can discard it and learns to do so on its own.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
sparse autoencoders, vision-language models, CLIP, few-shot classification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2Vg4OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work proposed a study on frozen CLIP features once a few-shot classifier has narrowed a prediction to its top-K = 15 candidate classes. The paper makes three main claims: 1. atoms that are prevalent across probable classes are also the most strongly activated. 2. Global cross-entropy rewards evidence that raises all candidate logits together, even though such evidence cancels within the candidate set. 3. Removing the decoded shared component U_S, which carries about 68% of the feature norm, leaves candidate-local reranking unaffected. Experiments use 11 datasets, CLIP RN50 and ViT-B/32, and three first-stage classifiers (linear probe, CoOp, CoCoOp).

Critical strengths

The global/local decomposition of the cross-entropy (Eq. 5) is a simple and useful way to explain why a globally trained head retains evidence that is non-discriminative within the candidate set.

The reranking gains replicate across 11 datasets, three first-stage classifiers, 4/8/16 shots, and two encoder families, with sweeps over K and the prevalence threshold. The Z vs. V_S equivalence and the low standalone accuracy of U_S are robust across all of these.

Critical weaknesses

CLIP embeddings are strongly anisotropic, with a large common mean direction. Atoms that fire in ≥75% of samples of every candidate class are plausibly dominated by this dataset-mean component, which any candidate-restricted linear head ignores trivially. A large ||U_S||/||Z|| that can be dropped without loss is then expected rather than revealing.

Hybrid accuracy is flat (69.74–69.91) over the whole threshold range, including τ = 0, and Accr drops about 3 points at τ = 0. So the Table 1 equivalence mostly reflects the frozen global logits on Z, which still see the "removed" component. The abstract's statement that "most of the feature can be removed without changing the answer" could therefore not be supported by this.

There is no control reranker trained with the full-label softmax on Z and fused with the same λ. That ablation is needed to separate Proposition 1's mechanism from a generic ensembling effect.

The 74% reduction in |b_U| compares heads with different weight scales; it should be normalized. Nothing about this is discussed in paper.

Questions for authors

What fraction of U_S, by norm, is explained by the dataset-mean CLIP feature, or by its top-1 to top-3 principal components? Does subtracting the mean feature, or a norm-matched random atom set, reproduce the Z ≈ V_S result and the 13% U_S accuracy?

Does K = 15 and τ = 0.75 (Figs. 7–8) are chosen on test accuracy or on a held-out validation split of the few-shot data?

Please report |b_U| normalized by the head's weight scale, or by the residual margin m_V, so that the stage-1 and reranker values are comparable. If already did, please clearify this in paper.

Reviewer_NyY1OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the feature space of visual foundation models used for downstream classification. For each image, it looks at the top candidate classes predicted by a classifier and finds that a large part of the feature is shared by these candidates, which does not help choose between them. Using a sparse autoencoder, the paper separates shared and non-shared concepts. It shows that a reranker trained only over the K candidates improves few-shot accuracy, and that removing the shared concepts from its input gives the same improvement, so they (shared concpets) are not needed for the final decision.

Critical strengths

This paper characterize the feature space of CLIP into shared and non-shared conepts given a downstream task.

This paper showed that shared-evidence can improve candidate-versus-rest separation and it can be removed within candidate decision.

By separating the shared and non-shared concepts using a sparse autoencoder and rerankers they improved few-shot accuracy.

Critical weaknesses

My biggest concern about this paper is about clarity and readability of the paper. I had a hard time understanding the paper, the problem it was trying to solve and the motivation behind it. Specifically, the question of "What parts of a CLIP feature matter for a prediction?" is too general to understand the analysis. I think the authors should provide more context on the question that they are trying to analyze such that is easier for the first-time readers of this paper to understand its outcome.

My second question/concern is that why removing the shared-concepts does not improve the accuracy. Does that mean that we can just use any reranking methods such as PCNN? Basically, just a two-stage method of 1) finding top-k candidates and 2) training a head on those top-k candidates suffice? If so, given that this has been done before, how does the analysis in this paper can help us practically?

Questions for authors

Please read the weaknesses for questions.

Reviewer_JHCWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies which parts of a frozen CLIP feature matter once a global classifier (linear probe, CoOp, or CoCoOp) has narrowed a prediction to its top-K candidates. A TopK SAE trained on ImageNet features marks atoms prevalent in every candidate class as "shared," splitting the feature into U_S and the residual V_S = Z − U_S. Shared atoms are also the most strongly expressed (Fisher-z r > 0.7 on 10 of 11 datasets). The theory argues that global cross-entropy rewards evidence that lifts all candidates together (Prop. 1), that such evidence can reverse local margins when it cancels imperfectly (Prop. 2), and that V_S suffices within the candidate set under conditional independence (Thm. 1).

Critical strengths
  1. A well-posed question with a useful lens Candidate conditioning is a precise way to ask how a component's relevance changes with prediction scope. Splitting global cross-entropy into within-candidate and candidate-vs-rest terms (Eq. 5) explains why a global head keeps evidence that does not separate the likely classes, which the authors rightly call scope-dependent rather than spurious. The framing extends naturally to other coarse-to-fine systems.

  2. Consistent, tuning-free gains across a broad grid Candidate-restricted reranking improves every global classifier at 4, 8 and 16 shots on both backbones, with λ = 1 fixed and no per-dataset tuning. The method is cheap: a linear head on frozen features. The paper also reports failure cases (Figs. 11–15) and an explicit limitations section.

Critical weaknesses
  1. The evidence for the interference mechanism (Prop. 2) is weak, and partly points the other way. Because the global head is linear, applying it to V_S instead of Z removes exactly b_U from every margin. Yet doing so lowers linear-probe accuracy from 68.56% (Table 1) to 58.14% (Fig. 5). U_S therefore rescues more within-candidate decisions than it reverses. The |b_U| analysis cannot resolve this: it is unsigned, so it pools cases where U_S helps with cases where it hurts. It is also not scale-normalized, so the 74% reduction may partly reflect the reranker's smaller logit scale. The paper should report the signed b_U, the fraction of examples satisfying Eq. 11 versus the rescue case, and |b_U| normalized by ‖w_y − w_c‖ or |m_Z|.

  2. "Dispensable" rests on a metric that cannot discriminate. Hybrid accuracy stays within 69.74–69.91% across all ρ (Fig. 8), while standalone reranker accuracy varies by 3 points (65.84–68.87%). The fused score is dominated by the global term, so Z ≈ V_S agreement on hybrid accuracy is weak evidence. On standalone accuracy, removing U_S does cost something: 68.33% at ρ = 0.75 versus 68.87% for full Z, and its significance cannot be judged without seed variances. Since the global branch still consumes all of Z, the claim that "two thirds of the representation is dispensable" holds at most for the reranker's input. It also needs an approximate version of Theorem 1, e.g. bounded by I(Y; U_S | V_S, S), because the evidence suggests Assumption 1 holds only approximately.

  3. The paper credits the gains to restricting the objective (Prop. 1 → Sec. 3.1), but never trains the same linear reranker with full-label-space cross-entropy and fuses it the same way. Without this control, the gains may simply come from ensembling a second head. That reading is especially plausible for CoOp and CoCoOp, where an image-feature head complements a text-prototype classifier, as in Tip-Adapter or LP++. It would also explain why the gains grow from LP (+1.32) to CoOp (+2.09) to CoCoOp (+5.94).

  4. The main results use CLIP RN50 and the appendix uses ViT-B/32. The literature standard is ViT-B/16, and a claim about "visual foundation models" would need other encoders such as ViT-L/14, SigLIP or DINOv2.

Questions for authors
  1. What are the results of a reranker with the same architecture trained with full-label-space cross-entropy and fused with λ = 1? This is needed to separate the effect of restricting the objective from that of ensembling a second head?

  2. How does SAE-based removal compare with norm-matched random atom removal, top-magnitude atom removal, and mean/PCA subtraction? Without these, it is unclear what the SAE and the sharedness criterion contribute.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.