Selective Ambiguity Repair: Preserving CLIP’s Prior in Few-Shot Adaptation

Few-shot adaptation should not alter what a pretrained model already gets right. However, most CLIP adaptation methods modify prompts, representations, or predictions globally, even when errors arise from confusion among only a few plausible classes. We formulate few-shot adaptation as selective ambiguity repair: preserve reliable pretrained decisions and intervene only where ambiguity is concentrated. We introduce a frozen-backbone framework that extracts complementary evidence across visual depth and applies bounded, ambiguity-aware residual corrections to a query-specific set of ambiguous classes. A complementary full-class branch provides a recovery path when the correct class falls outside this set, while adaptive fusion balances correction against preservation. Across 11 few-shot benchmarks, our method consistently improves strong protocol-matched CLIP adaptation baselines across 1/2/4/8/16 shots, with the gains extending consistently from RN50 to ViT-B/16. Ambiguity diagnostics further show that selective repair reduces candidate entropy, increases ground-truth margins, resolves more ambiguous predictions than dense or random correction, and better preserves reliable CLIP decisions. These results support a broader principle for few-shot vision–language adaptation: concentrate adaptation capacity where the pretrained model is uncertain while preserving its reliable decision structure elsewhere.