- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This work studies and compares two confidence metrics for image rejection in CLIP-like VLMs. As the model answers only when its confidence score passes a single threshold, the choice of metric influences which samples are rejected and how many. They compare the margin (top-1 minus top-2 score) directly against the max logit and find that, on Waterbirds, the margin rejects samples with mismatched backgrounds far more often than the max logit, which suggests greater sensitivity to spurious correlations. However, the authors find this does not hold on NICO++ and conclude that the choice of the better metric depends on the dataset.
Critical strengths
- Well-designed experiments (randomized margin baseline tests) with analyses and takeaways. Although the paper proposes no new method, it remains informative for practitioners choosing a confidence metric for rejection.
Critical weaknesses
-
The authors cite Wang and al. (2024) [1] but do not use CounterAnimal [1], which was built to measure CLIP’s reliance on real backgrounds with natural photos. The Waterbirds mechanism rests on pasted composites, and NICO++ contexts are not spurious correlations in the same sense; CounterAnimal could have provided critical information that could steer the conclusion away from raw dataset dependency.
-
Too much results reporting with hard-to-digest paragraphs; the results are already reported in tables and figures. Why are they repeated in the paragraph? It should add substance rather than be redundant.
-
Low novelty; the reviewer struggles to identify what is scientifically new knowledge beyond the noise baseline.
[1] Qizhou Wang, Yong Lin, Yongqiang Chen, Ludwig Schmidt, Bo Han, and Tong Zhang. A sober look at the robustness of CLIPs to spurious features. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Questions for authors
-
Why is OpenAI CLIP the only encoder for which the max logit is noise-dominated on NICO++? Knowing that it's trained on (considered to be) better-curated data compared to other pre-training sources
-
Do the Waterbirds conclusions hold on CounterAnimal?
-
Prior work that is already cited have shown that margin-based abstention raises worst-group risk above the group-agnostic reference you use in Sec. 3.3. What is the new claim beyond replicating this on the frozen CLIP?
-
Beyond the noise baseline, could the authors clearly state how does this work pushes the boundaries, and which experiments/sections depend on it?