- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work demonstrates that the robustness and reliability of existing vision-language models are poor, as their answers depend on answer choices. While the work is interesting in studying this specific "opposite meaning change" behavior, the manuscript fails to cite many of the existing work that evaluate the robustness and reliability of vision-language models.
Critical strengths
On 260 paired CheXpert items, MedGemma-4B-it turns 89 of 158 initially correct answers into false denials and Qwen2.5-VL-7B turns 41 of 116, conditional reversal rates of 56.33% and 35.34%. The aggregate accuracy of Qwen2.5-VL nevertheless rises by 5.77 percentage points because recoveries offset these reversals.
Critical weaknesses
The innovations are limited. Existing literature has already shown that vision-language models are poor in their robustness and reliability, and are sensitive to prompt noise.
https://advanced.onlinelibrary.wiley.com/doi/10.1002/aisy.202500255?utm_source=chatgpt.com https://aclanthology.org/2025.tacl-1.52.pdf
Questions for authors
Suggest using sentence verification theory "opposite meaning change" to replace "negated-option attraction"
2 comments
Follow-up and a brief guide to our revisions
Dear Reviewer zweh,
Thank you again for the helpful references and your suggestion concerning sentence verification. We have incorporated the corresponding clarifications and supporting experimental findings into the revised main text, with key additions highlighted in blue to make them easier to locate.
- W1: Sec. 2 now discusses MedVH, DARE, CARETS, and Med-CP. Sec. 1 clarifies our contribution, while Secs. 4–5, Tables 1–2, and Fig. 2 present matched controls and image-conditioned experiments showing what the paired evaluation reveals beyond aggregate accuracy.
- Q1: Sec. 1 now explicitly connects polarity reversals to sentence–picture verification, citing Clark and Chase and Carpenter and Just, and clarifies the meaning of “negated-option attraction.”
We would be very grateful if you could take another look when your schedule allows. We welcome any further suggestions and would be happy to clarify any remaining points. Thank you for your time and consideration.
Response to zweh: W1 and Q1, related work, contribution, and terminology
Thank you for pointing us to these related studies and the sentence-verification perspective. We have expanded the related-work discussion and clarified the terminology.
W1. Contribution relative to existing robustness studies
We measure whether an option change turns a correct answer into a false denial, how much loss it adds over ordinary replacement, and whether the model answers opposite image conditions correctly. Sec. 2 now discusses all four suggested works and distinguishes their evaluation targets. DARE evaluates changes in instructions, answer-option subsets, output formats, and the number of correct answers. CARETS uses paired tests of VQA consistency, including negation. MedVH evaluates medical hallucination, and Med-CP studies the influence of inaccurate clinical text in user prompts.
Building on these studies, CXR-ContraBench examines a specific transition: a model selects a reference-present finding and then denies that same finding after one answer option changes. The image, question, gold answer, option count, and retained option positions stay fixed. We measure these transitions alongside an ordinary replacement of the same option slot and a test with images requiring opposite answers. These comparisons yield three findings.
1. The controls quantify what the negated option adds. On the 260 validation items, MedGemma-4B loses 56.3% of its correct answers when Z is replaced with No X, 13.3% when Z is replaced with another absent finding W, and 6.3% when the distractors are reordered. The paired excess of No X over W is 43.0 percentage points [34.8, 51.2].
Tables 7–9 extend the paired protocol to ten open-weight models on 6,541 CheXpert training-split patients, 240 OpenI patients, and 11,240 ReXGradient patients. The excess is positive in all five MedGemma, Lingshu, and CheXagent models on each additional set, including the largest evaluated member of each family. The Qwen2.5-VL and LLaVA models show a different response, with no positive excess on those sets. Each excess compares both interventions on the same W-eligible, control-correct items.
2. Higher accuracy can coexist with new false denials. Qwen2.5-VL-7B improves from 44.62% to 50.38% accuracy while 41/116 of its initially correct answers become No X. The paired flow identifies these reversals alongside the recoveries that raise the mean score. The same coexistence appears in eight of the ten open-weight models on the validation set and in all three frontier systems.
3. The image diagnostic distinguishes different improvements. In-format LoRA achieves zero reversal by selecting the affirmative member of the X / No X pair, with 0% PairCorrect across target-present and target-absent images. An image-balanced MedGemma adapter instead reaches 46.2–50.2% PairCorrect across three formats while reducing RR from 54.7% to 10.0%. This comparison shows how the benchmark distinguishes answer-set performance from image-conditioned correctness.
When evaluating a prompt or adapter, we can therefore check which correct judgments it preserves, whether negation causes additional errors, and whether its gains hold on images requiring opposite answers.
Q1. Sentence verification and terminology
Sec. 1 now explains the connection to sentence–picture verification, citing Clark and Chase and Carpenter and Just. A polarity reversal changes the selected statement from X to No X, which has the opposite truth value. This is the connection to the verification perspective you suggest. We retain "negated-option attraction" to name the specific answer-option effect measured by the paired transitions and matched controls.
Manuscript changes: Sec. 2 adds MedVH, DARE, CARETS, and Med-CP, together with POPE, NaturalBench, ProbMed, and MediConfusion. Sec. 1 clarifies the contribution and terminology. Tables 1, 2, 4, and 7–9 provide the corresponding controls and diagnostics.