acceptodds
Under review as a conference paper at ICLR 2027

The Margin Knows the Background: Uneven Abstention in Frozen Vision-Language Models

Abstract

Selective prediction allows a model to abstain on uncertain inputs, but a single confidence threshold can disproportionately reject examples from particular groups. We study how confidence scores produce this imbalance in frozen vision-language models (OpenAI CLIP, LAION CLIP, SigLIP). We express a family of scores, including the margin and maximum logit, through the average and the difference of the two highest class logits, and ask which carries group information. We show that equal group coverage at every threshold is equivalent to independence between the score and group membership. A randomized margin baseline tests whether a reduction in coverage disparity beats what adding noise achieves. With score choices fixed before testing, the effect of choice depends on the dataset. On Waterbirds, the margin creates coverage gaps 2.5 to 5.5 times larger than maximum logit at 80% coverage, its group signal is in the difference component, and paired background swaps show the background moves it, although maximum logit increases overall error. On NICO++, maximum logit instead increases worst-context error by 5.4 to 6.6 points and offers no improvement over the randomized baseline for OpenAI CLIP. Across both datasets, how far a score pushes a group down its ranking tracks the coverage gap (Spearman 0.88, 0.92) better than plain group identifiability (0.61, 0.65). These results explain why a score that abstains more evenly on one dataset may be less effective on another and motivate judging group coverage and selective risk together against randomized baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, group robustness, selective prediction, spurious correlations, confidence estimation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Tf7MOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This work studies and compares two confidence metrics for image rejection in CLIP-like VLMs. As the model answers only when its confidence score passes a single threshold, the choice of metric influences which samples are rejected and how many. They compare the margin (top-1 minus top-2 score) directly against the max logit and find that, on Waterbirds, the margin rejects samples with mismatched backgrounds far more often than the max logit, which suggests greater sensitivity to spurious correlations. However, the authors find this does not hold on NICO++ and conclude that the choice of the better metric depends on the dataset.

Critical strengths
  • Well-designed experiments (randomized margin baseline tests) with analyses and takeaways. Although the paper proposes no new method, it remains informative for practitioners choosing a confidence metric for rejection.
Critical weaknesses
  • The authors cite Wang and al. (2024) [1] but do not use CounterAnimal [1], which was built to measure CLIP’s reliance on real backgrounds with natural photos. The Waterbirds mechanism rests on pasted composites, and NICO++ contexts are not spurious correlations in the same sense; CounterAnimal could have provided critical information that could steer the conclusion away from raw dataset dependency.

  • Too much results reporting with hard-to-digest paragraphs; the results are already reported in tables and figures. Why are they repeated in the paragraph? It should add substance rather than be redundant.

  • Low novelty; the reviewer struggles to identify what is scientifically new knowledge beyond the noise baseline.

[1] Qizhou Wang, Yong Lin, Yongqiang Chen, Ludwig Schmidt, Bo Han, and Tong Zhang. A sober look at the robustness of CLIPs to spurious features. In Advances in Neural Information Processing Systems (NeurIPS), 2024.

Questions for authors
  • Why is OpenAI CLIP the only encoder for which the max logit is noise-dominated on NICO++? Knowing that it's trained on (considered to be) better-curated data compared to other pre-training sources

  • Do the Waterbirds conclusions hold on CounterAnimal?

  • Prior work that is already cited have shown that margin-based abstention raises worst-group risk above the group-agnostic reference you use in Sec. 3.3. What is the new claim beyond replicating this on the frozen CLIP?

  • Beyond the noise baseline, could the authors clearly state how does this work pushes the boundaries, and which experiments/sections depend on it?

Reviewer_fHHwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how confidence score choice produces uneven abstention across background and context groups in frozen vision–language models (VLMs). Keeping the encoders, prompts, and predicted labels fixed, it decomposes the two highest class logits into their mean (“typicality”) and difference (“separability”). The theoretical analysis establishes that equal group coverage at every common threshold is equivalent to independence between the confidence score and group membership. The authors show their empirical analysis on Waterbirds and NICO++.

Critical strengths

None

Critical weaknesses
  • Limited novelty. The paper's central motivation that selective prediction can amplify group disparities is already established by [1] and background bias in CLIP is also acknowledged in diverse prior works. The typicality-separability decomposition is an algebraic reparameterization of the top 2 logits. Proposition 1 and Lemma 1 are mathematically valid but their theoretical novelty is limited.
  • Insufficient empirical breadth and validation of the explanatory claims. The study is confined to Waterbirds and NICO++, with groups defined by object–background combinations or photographic contexts.

Additional Presentation and Compliance Concerns

  • Figure 2 provides group definitions, sample counts, and illustrative images, but no direct analytical evidence for the core claims of the paper. In Figure 3, the overlapping solid, dashed, and dotted curves make encoder-specific trends difficult to distinguish, weakening the presentation of a key comparison.
  • The paper appears to omit an AI use statement. ICLR 2027 explicitly requires an AI use statement in the manuscript.

[1] Jones et al., "Selective Classification Can Magnify Disparities Across Groups," 2021

Questions for authors

None

Reviewer_hFwoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how confidence scores lead to disproportionate abstention on samples from particular groups in selective prediction of VLMs. Specifically, it shows that different confidence score choices, such as the top-two-logits margin and maximum logit, could carry group information and affect the group coverage and worst-group error, with the extent varying across datasets and VLMs. The paper also provides additional insights on how backgrounds influence confidence scores and the trade-off between coverage and selective risk.

Critical strengths
  • This paper evaluates a broad range of confidence scores (e.g., maximum logit, margin, etc.) and provides a systematic comparison of how the choice of scores affects the group coverage and worst-group error across various VLMs and datasets
  • The background swapping experiment results provide interesting insights into how backgrounds influence the margin, seems to suggest an explanation for the uneven acceptance across groups.
Critical weaknesses
  • This paper focuses on evaluating CLIP-style VLMs only. With the limited model diversity, it is unclear whether the findings in this paper could generalize to other VLM architectures. If such a broader evaluation is outside the scope, the paper should state explicitly that its scope is CLIP-only and qualify its claims about VLMs accordingly.
  • While the results of the background-swapping experiments show that backgrounds affect the margins, further analysis is needed to link these changes to uneven acceptance rates (if the authors intend to provide an explanation for the observed uneven acceptance). Reporting how often swaps cause margins to cross the thresholds, and whether this differs across groups, would strengthen the paper's explanation of the uneven acceptance.
Questions for authors
  • Does the current approach of using only the top two logits for confidence scores miss useful information from the other classes in datasets such as NICO++? Could this affect the findings?
  • Please clarify whether "worst-context risk" means "worst-group error/risk".
  • Please clarify what "qualified mechanism" refers to in Section 5.4 and how the background swapping experiments support it

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.