acceptodds
Withdrawn from ICLR 2027

Selective Ambiguity Repair: Preserving CLIP’s Prior in Few-Shot Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Few-shot adaptation should not alter what a pretrained model already gets right. However, most CLIP adaptation methods modify prompts, representations, or predictions globally, even when errors arise from confusion among only a few plausible classes. We formulate few-shot adaptation as selective ambiguity repair: preserve reliable pretrained decisions and intervene only where ambiguity is concentrated. We introduce a frozen-backbone framework that extracts complementary evidence across visual depth and applies bounded, ambiguity-aware residual corrections to a query-specific set of ambiguous classes. A complementary full-class branch provides a recovery path when the correct class falls outside this set, while adaptive fusion balances correction against preservation. Across 11 few-shot benchmarks, our method consistently improves strong protocol-matched CLIP adaptation baselines across 1/2/4/8/16 shots, with the gains extending consistently from RN50 to ViT-B/16. Ambiguity diagnostics further show that selective repair reduces candidate entropy, increases ground-truth margins, resolves more ambiguous predictions than dense or random correction, and better preserves reliable CLIP decisions. These results support a broader principle for few-shot vision–language adaptation: concentrate adaptation capacity where the pretrained model is uncertain while preserving its reliable decision structure elsewhere.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Vision-Language Models, Few-Shot Learning, CLIP Adaptation, Parameter-Efficient Adaptation, Test-Time/Logit Adaptation, Selective Prediction Repair, Multimodal Learning, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VeHAOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Selective Ambiguity Repair (SAR) for few-shot CLIP adaptation. Motivated by the observation that many CLIP errors involve confusion among a small set of plausible classes, SAR combines multi-depth visual features with query-specific top-k residual correction and a full-class recovery branch. Experiments across 11 benchmarks demonstrate strong few-shot performance, with additional analyses supporting the effectiveness of selective correction.

Critical strengths
  1. The paper addresses a meaningful problem in few-shot CLIP adaptation, with a clear and intuitive motivation.
  2. The experimental results are strong across 11 benchmarks and multiple shot settings.
  3. The evaluation is comprehensive, including detailed ablations, robustness analysis, and cross-backbone experiments.
Critical weaknesses
  1. Residual correction has already been extensively studied in methods such as TaskRes. The proposed top-k selective correction appears to be a relatively straightforward extension of existing approaches.
  2. Architectural complexity. The method combines numerous components, including multi-depth fusion, routing, and adaptive recovery. It is unclear whether such complexity is necessary.
  3. Insufficient comparison with simpler alternatives. A baseline applying top-k residual masking directly to LDC would help establish whether the proposed architecture provides substantial benefits beyond simpler modifications.
Questions for authors
  1. How does SAR compare with a simpler baseline that applies query-specific top-k residual correction directly to LDC (Li et al., CVPR 2025) under the same experimental setting?
  2. Table 20 shows that the full-class branch alone substantially outperforms the selective branch. Could the authors further clarify how much of the final improvement can be attributed specifically to selective repair?
Reviewer_dpY4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a method called selective ambiguity repair (SAR) for few-shot CLIP adaptation. The paper starts from the observation that the correct prediction corresponding to a large part of misclassified examples lies in the top-5/top-10 predictions, suggesting that errors are "local". The method consists of three main stages. The first is depth-relational evidence fusion. In this branch, features are extracted at multiple levels of the encoder, and a convex combination of them is computed, which is called "multi-depth anchor". Features are also combined to form depth-consistent evidence (information shared across layers) and depth-transition evidence (one that captures the representational difference across layers); both are fused using a router and later combined with the multi-depth anchor. So the output of this first stage is kind of a refined multi-depth representation. The second stage is query-specific candidate repair. A set of operations aim at constructing a logit proposal that is added to the zero-shot logits. The output of stage 1, the zero-shot CLIP prediction, the difference between the logit proposal and zero-shot logits, and the separation between leading candidates are passed trough a router that proposes a correction. The third stage combines predictions from both branches. Experiments cover few-shot classification, base-to-novel generalization, and domain generalization and show that SAR outperforms other methods.

Critical strengths

The paper shows a strong performance particularly on the few-shot classification task (Table 1).

Critical weaknesses

On the writing and the methodological level

The paper is written in a difficult language that makes it very hard to follow and understand. Additionally, the method is convoluted and a lot of design choices are not obvious. For example, why is there a learnable scale-shift transformation after feature projection (line 145)? What does "CLIP-supported semantics" (line 148) mean? What are gcong_{con} and gtrg_{tr} in Equation (1), and where do they appear in Figure (2) which contains glowg_{low}, a different notation? What does it mean to compute a distribution αl\alpha_l using the projected features r~l\tilde{r}_l and then to use αl\alpha_l as weights to combine r~l\tilde{r}_l (in uconu_{con} computation)? The same question applies to βl\beta_l and Δl\Delta_l in Equation (1). What does the following sentence mean and why mentioning Fourier-frequency decomposition: "This is not a spatial or Fourier-frequency decomposition, but one along representational depth" (Lines 158-159)? It is also unclear what is the motivation and necessity of every operation in Equation (2). Moreover, what is the difference between γ\gamma (line 162) and γcross\gamma_{cross} (Equation (2))? Why is the projection with WfullW_{full} needed? What are GG and EE architectures? The same thing applies to Equation (3), where there is a lot of ambiguity about notations. For example amina_{min}, amaxa_{max}, ρ\rho, μ\mu are defined in Equation (3) and are used in the "Mix" operator, which is used to define z(0)z^{(0)} (Equation (4)). Then in later equations (for instance Equation (5)) "aa" is defined from z(0)z^{(0)}. This is confusing and looks a bit circular. The later operations are also unjustified. Finally, I am not sure I did not miss anything, but I did not find any information about the architectures of the method's components (the routers, shallow branch, deep branch...).

On the experimental level

The paper mentions that it uses protocol-matched RN50 adaptation baselines. However, not all the methods adopt the same experimental protocol. ProLIP [1] has two versions: a few-shot validation version and a validation-free version, CLAP [2] is validation-free, LP++ [3] and CLAP report much lower results of baselines under these two settings (for instance, across 1-/2-/4-/8-/16- shots settings, LP++ reports 63.06/66.47/68.71/71.78/74.37 and CLAP reports 60.29/62.26/65.32/68.35/71.40 accuracy for Tip-Adapter under the few-shot validation and validation-free settings, respectively). The same thing applies for other baselines that originally use validation sets to select hyperparameters. Most importantly, the method contains a lot of design choices and hyperparameters. How are these selected? For instance λ1\lambda_1, λ2\lambda_2, λ3\lambda_3, λ4\lambda_4, λ5\lambda_5 (Equation (10)), kk (in the top-k entries selection), λ\lambda (Equation (2)).

[1] Fahes et al., CLIP's Visual Embedding Projector is a Few-shot Cornucopia. WACV 2026
[2] Silva-Rodríguez et al., A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models. CVPR 2024
[3] Huang et al., LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP. CVPR 2024

Questions for authors
  • Lines 701-703, 720-721: the authors state that hyperparameters are tuned without access to test labels, novel-class labels, or OOD target-set labels, same thing for model selection. Then how are the hyperparameters tuned and how is model selection performed?

  • What does "Stage-consistency + Cross-depth Refinements" in Figure 2 refer to?

  • What does a "CLIP-anchored proposal" (line 175) mean?

  • Why does z(0)z^{(0)} reveal the query’s most likely competing classes (line 188)?

  • Lines 744-746: "We do not treat this dataset-level... mechanistic validation": what does this sentence mean?

Suggestions

  • The notations WfullW_{full} (Equation (2)) and WmlpW_{mlp} (Figure 2) are inconsitent.

  • While stages 1 and 2 are named in Figure 2, the name of stage 3 ("Recovery and prediction fusion") appears in the caption but not in the figure. It would be good to refer to it at the corresponding place in the figure.

Reviewer_c7L4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Selective Ambiguity Repair (SAR) for few-shot CLIP adaptation. The authors observe that the ground-truth class often remains among CLIP's top-ranked predictions when its top-1 prediction is incorrect. Based on this observation, they propose restricting additional residual logit corrections to a query-specific top-k candidate set, aiming to reduce unnecessary modifications to the pretrained model's original decision structure. The proposed framework incorporates multi-depth visual feature fusion, a CLIP-anchored proposal, top-k residual correction, a full-class prediction branch, and shot-adaptive fusion. Experiments are conducted on 11 classification benchmarks using RN50 and ViT-B/16, including evaluations of base-to-novel generalization, distribution shifts, ablation studies, and ambiguity resolution.

Critical strengths
  1. Competitive and relatively comprehensive experimental results.

The method achieves improvements on few-shot classification performance across multiple datasets and shot settings. The evaluation covers different visual backbones, support-set sampling, base-to-novel generalization, and distribution shifts.

  1. A systematic empirical investigation of selective logit correction.

Reducing unnecessary changes to pretrained predictions is a reasonable objective in few-shot adaptation. The authors provide comparisons among dense, random-k, bottom-k, and top-k correction strategies, along with repair–preserve analyses, offering some empirical evidence for the effectiveness of selective correction within the proposed framework.

Critical weaknesses
  1. Limited methodological novelty and substantial overlap with existing work.

The overall framework closely resembles LDC (CVPR 2025), which has already investigated inter-class confusion in CLIP and introduced multi-level visual feature fusion, residual logit correction, a full-class prediction branch, and adaptive logit fusion. SAR mainly extends this framework with top-k selective correction, more elaborate feature fusion, and shot-adaptive designs. However, the necessity and independent contributions of these additional components remain insufficiently justified.

  1. An unclear connection between top-k selection and the preservation of reliable CLIP predictions.

The paper emphasizes avoiding unnecessary changes to CLIP's originally correct predictions through selective correction. However, top-k selection is based solely on class rankings and does not directly account for prediction reliability. For originally correct predictions, the top-k set contains the main competing classes that determine the final decision. Modifying these classes may still disrupt correct predictions, while leaving lower-ranked classes unchanged does not guarantee their preservation.

  1. Weak connection between the additional architectural designs and the central motivation.

Beyond top-k correction, SAR introduces elaborate multi-depth feature fusion, shot-adaptive mixing, and other components. However, their specific relevance to the proposed problems of local class ambiguity and CLIP prior preservation remains unclear.

Questions for authors
  1. How does top-k correction preserve reliable CLIP predictions?

Since top-k selection does not account for prediction reliability, could the authors explain why modifying top-ranked classes helps preserve originally correct predictions and provide supporting analysis?

  1. How do the additional components support the central motivation?

Could the authors clarify how depth-relational fusion and shot-adaptive mixing specifically address local ambiguity or prior preservation? A simpler CLIP adaptation baseline with top-k correction would also help isolate its contribution.

  1. How are class-dependent modules transferred to novel classes?

How are the learned full-class classifier in Eq. (2) and the class-logit transformations in Eq. (4) applied to unseen classes?

Additionally, please include relevant ViT-B/16 base-to-novel baselines such as CoPrompt (ICLR 2024).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.