acceptodds
Under review as a conference paper at ICLR 2027

CXR-ContraBench: Auditing Polarity Reversals Beyond Aggregate Accuracy

Abstract

A medical vision-language model can identify a finding on a chest X-ray and then deny it after a single answer option changes. We call this directional failure *negated-option attraction*. CXR-ContraBench measures it with a paired protocol that holds the image, question, reference finding, and retained options fixed while one distractor is replaced by the negated target, and records every answer transition, including correct answers that become false denials and wrong answers that recover. On 260 paired CheXpert items, MedGemma-4B-it turns 89 of 158 initially correct answers into false denials and Qwen2.5-VL-7B turns 41 of 116, conditional reversal rates of 56.33% and 35.34%. The aggregate accuracy of Qwen2.5-VL nevertheless rises by 5.77 percentage points because recoveries offset these reversals. Because the target-paired answer set admits an image-free structural ceiling, image-conditioned validity is evaluated separately: matched, opposite-label, shuffled, and blank images change which statement is true while the text stays fixed. Under this diagnostic, expected polarity updating is limited, a polarity-explicit prompt changes answer behavior with a benefit that depends on the measured endpoint, and in-format adaptation reaches perfect format accuracy without paired image-conditioned correctness. The transition persists across ten wording families, option-free single-sentence generation, OpenI, frontier models, and 135,754 CheXpert records from 59,173 studies. Average accuracy should therefore be interpreted together with whether previously correct judgments are preserved. Code is available at https://anonymous.4open.science/r/cxr-contrabench/.

Primary area:
datasets and benchmarks
Keywords:
Medical Vision-Language Models, Diagnostic Benchmark, Chest X-ray Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zwehOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work demonstrates that the robustness and reliability of existing vision-language models are poor, as their answers depend on answer choices. While the work is interesting in studying this specific "opposite meaning change" behavior, the manuscript fails to cite many of the existing work that evaluate the robustness and reliability of vision-language models.

Critical strengths

On 260 paired CheXpert items, MedGemma-4B-it turns 89 of 158 initially correct answers into false denials and Qwen2.5-VL-7B turns 41 of 116, conditional reversal rates of 56.33% and 35.34%. The aggregate accuracy of Qwen2.5-VL nevertheless rises by 5.77 percentage points because recoveries offset these reversals.

Critical weaknesses

The innovations are limited. Existing literature has already shown that vision-language models are poor in their robustness and reliability, and are sensitive to prompt noise.

https://advanced.onlinelibrary.wiley.com/doi/10.1002/aisy.202500255?utm_source=chatgpt.com https://aclanthology.org/2025.tacl-1.52.pdf

https://aclanthology.org/2022.acl-long.443.pdf

https://aclanthology.org/2026.eacl-industry.67.pdf

Questions for authors

Suggest using sentence verification theory "opposite meaning change" to replace "negated-option attraction"

2 comments
AuthorsOpenReview ↗

Response to zweh: W1 and Q1, related work, contribution, and terminology

Thank you for pointing us to these related studies and the sentence-verification perspective. We have expanded the related-work discussion and clarified the terminology.

W1. Contribution relative to existing robustness studies

W1 (excerpt). Existing literature has already shown that vision-language models are poor in their robustness and reliability, and are sensitive to prompt noise.

We measure whether an option change turns a correct answer into a false denial, how much loss it adds over ordinary replacement, and whether the model answers opposite image conditions correctly. Sec. 2 now discusses all four suggested works and distinguishes their evaluation targets. DARE evaluates changes in instructions, answer-option subsets, output formats, and the number of correct answers. CARETS uses paired tests of VQA consistency, including negation. MedVH evaluates medical hallucination, and Med-CP studies the influence of inaccurate clinical text in user prompts.

Building on these studies, CXR-ContraBench examines a specific transition: a model selects a reference-present finding and then denies that same finding after one answer option changes. The image, question, gold answer, option count, and retained option positions stay fixed. We measure these transitions alongside an ordinary replacement of the same option slot and a test with images requiring opposite answers. These comparisons yield three findings.

1. The controls quantify what the negated option adds. On the 260 validation items, MedGemma-4B loses 56.3% of its correct answers when Z is replaced with No X, 13.3% when Z is replaced with another absent finding W, and 6.3% when the distractors are reordered. The paired excess of No X over W is 43.0 percentage points [34.8, 51.2].

Tables 7–9 extend the paired protocol to ten open-weight models on 6,541 CheXpert training-split patients, 240 OpenI patients, and 11,240 ReXGradient patients. The excess is positive in all five MedGemma, Lingshu, and CheXagent models on each additional set, including the largest evaluated member of each family. The Qwen2.5-VL and LLaVA models show a different response, with no positive excess on those sets. Each excess compares both interventions on the same W-eligible, control-correct items.

2. Higher accuracy can coexist with new false denials. Qwen2.5-VL-7B improves from 44.62% to 50.38% accuracy while 41/116 of its initially correct answers become No X. The paired flow identifies these reversals alongside the recoveries that raise the mean score. The same coexistence appears in eight of the ten open-weight models on the validation set and in all three frontier systems.

3. The image diagnostic distinguishes different improvements. In-format LoRA achieves zero reversal by selecting the affirmative member of the X / No X pair, with 0% PairCorrect across target-present and target-absent images. An image-balanced MedGemma adapter instead reaches 46.2–50.2% PairCorrect across three formats while reducing RR from 54.7% to 10.0%. This comparison shows how the benchmark distinguishes answer-set performance from image-conditioned correctness.

When evaluating a prompt or adapter, we can therefore check which correct judgments it preserves, whether negation causes additional errors, and whether its gains hold on images requiring opposite answers.

Q1. Sentence verification and terminology

Q1. Suggest using sentence verification theory "opposite meaning change" to replace "negated-option attraction".

Sec. 1 now explains the connection to sentence–picture verification, citing Clark and Chase and Carpenter and Just. A polarity reversal changes the selected statement from X to No X, which has the opposite truth value. This is the connection to the verification perspective you suggest. We retain "negated-option attraction" to name the specific answer-option effect measured by the paired transitions and matched controls.

Manuscript changes: Sec. 2 adds MedVH, DARE, CARETS, and Med-CP, together with POPE, NaturalBench, ProbMed, and MediConfusion. Sec. 1 clarifies the contribution and terminology. Tables 1, 2, 4, and 7–9 provide the corresponding controls and diagnostics.

AuthorsOpenReview ↗

Follow-up and a brief guide to our revisions

Dear Reviewer zweh,

Thank you again for the helpful references and your suggestion concerning sentence verification. We have incorporated the corresponding clarifications and supporting experimental findings into the revised main text, with key additions highlighted in blue to make them easier to locate.

  • W1: Sec. 2 now discusses MedVH, DARE, CARETS, and Med-CP. Sec. 1 clarifies our contribution, while Secs. 4–5, Tables 1–2, and Fig. 2 present matched controls and image-conditioned experiments showing what the paired evaluation reveals beyond aggregate accuracy.
  • Q1: Sec. 1 now explicitly connects polarity reversals to sentence–picture verification, citing Clark and Chase and Carpenter and Just, and clarifies the meaning of “negated-option attraction.”

We would be very grateful if you could take another look when your schedule allows. We welcome any further suggestions and would be happy to clarify any remaining points. Thank you for your time and consideration.

Reviewer_R1e1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CXR-ContraBench to study negated-option attraction in medical vision-language models (VLMs). By replacing a distractor with the negation of the correct finding, the authors examine whether models change previously correct predictions into false negative predictions. The results show that such regressions can occur even when overall accuracy improves. The authors also evaluate this phenomenon across different models, datasets, prompting strategies and negation expressions.

Critical strengths
  1. The paper shows a meaningful reliability issue in medical VLMs, where false denials of clinically relevant findings may have serious consequences.

  2. The paired evaluation provides a useful perspective beyond aggregate accuracy by tracking how individual predictions change under controlled answer-option interventions.

  3. The experiments are extensive, covering multiple VLMs, datasets, negation formulations, structural controls and image-conditioned diagnostics. The analysis over the experiment result is also extensive.

Critical weaknesses
  1. The image-validity diagnostic has a potential mismatch between the question and the scoring criterion. While the question asks “which finding is present”, the opposite-label condition treats "No X" as the correct answer. The answer seems to be a little mismatched with the question, leading to low PairCorrect rate.

  2. Although the authors include off-target negation controls (e.g., the paired-distractor setting), the primary evaluation lacks a matched non-negation distractor replacement baseline. It is unclear whether negated options are more likely to turn correct answers into wrong ones than ordinary distractor replacements.

Questions for authors
  1. Could the authors clarify whether opposite-label images guarantee a unique correct answer? Would using a question explicitly asking whether X is present with "X" and "no X" options, provide a more reliable evaluation of image-conditioned polarity understanding?

  2. Could the authors compare correct-to-wrong transition rates under negation-based and non-negation distractor replacements to better quantify the negation-specific effect?

  3. Minor: The authors already show that models select "No Y" even more frequently than "No X". Could the authors further investigate including both "No X" and "No Y" in the same answer set, to better understand whether the observed behavior is driven by a general preference for negated options or a target-specific polarity reversal?

4 comments
AuthorsOpenReview ↗

Response to R1e1 (1/2): question-consistent image evaluation and answer uniqueness

Questions:

Q1. Could the authors clarify whether opposite-label images guarantee a unique correct answer? Would using a question explicitly asking whether X is present with "X" and "no X" options, provide a more reliable evaluation of image-conditioned polarity understanding?

We thank the reviewer for this suggestion.

Regarding question wording and answer uniqueness, the new binary, Yes/No, and three-statement diagnostics give every real image a unique reference-correct answer that answers the literal question. Secs. 3.4 and 5.2 and Table 4 report these evaluations. In every opposite-label donor study, both X and the displayed distractor Y are reference-absent. Thus, "X is absent" is the unique correct statement among "X is present", "X is absent", and "Y is present".

We evaluate the same 260 items under matched, opposite-label, shuffled, and blank images with three formats:

  • Binary: "Is X present on this chest X-ray study?" with X / No X.
  • Yes/No: "Is there X on this chest X-ray study?" with Yes / No.
  • Three statements: "Which statement about this chest X-ray study is correct?" with "X is present" / "X is absent" / "Y is present".

The two-option formats each use both option orders, giving five format variants and 5,200 rows per model. Blank images probe answer preferences and carry no clinical gold label. PairCorrect is the percentage of items answered correctly on both the matched and opposite-label images:

ModelOriginal formatBinaryYes/NoThree statements
MedGemma-4B17.742.736.97.7
Qwen2.5-VL-7B11.520.219.611.9
MedGemma-27Bn/a22.724.018.1
CheXagent-2-3Bn/a58.554.843.1
Lingshu-7Bn/a37.143.538.8
Lingshu-32Bn/a39.645.240.8

The binary question increases MedGemma-4B's PairCorrect from 17.7% to 42.7%. Its affirmative-answer probability separates target-present from target-absent images with AUC 0.72–0.76. CheXagent-2-3B reaches AUC 0.85–0.87. These results quantify image-conditioned performance under the direct question you proposed.

The new formats also reveal how this performance depends on the answer set. MedGemma-4B has AUC 0.56 with three statements, whereas CheXagent-2-3B and the Lingshu models retain AUC 0.73–0.82. Table 4 reports all ten open-weight models.

We further connect the direct questions to the original option intervention. Among the 105 items that MedGemma-4B answers "Yes" in both Yes/No orders and answers correctly under the control [X, Y, Z], 45 still switch to No X under [X, Y, No X], a rate of 42.9%. The corresponding count is 11/43 for Qwen2.5-VL-7B. Thus, the false-denial transition also occurs on items that the model consistently affirms when asked directly.

Manuscript changes: Secs. 3.4 and 5.2, Fig. 2, and Table 4 present the question-consistent diagnostics. Fig. 1(c) illustrates the measured three-statement format. Appendix A.2 specifies the construction and scoring, and Appendix H retains the original format for comparison.

AuthorsOpenReview ↗

Response to R1e1 (2/2): matched replacements and target-specific negation

Questions:

Q2. Could the authors compare correct-to-wrong transition rates under negation-based and non-negation distractor replacements to better quantify the negation-specific effect?

We thank the reviewer for suggesting these controls.

Regarding the negation-specific effect, Table 1 now compares replacing Z in [X, Y, Z] with either No X or another reference-absent finding W. No X causes more correct-to-wrong transitions for MedGemma-4B and Qwen2.5-VL-7B on the 260 validation items. Both replacements keep the image, question, gold answer, option count, and retained option positions fixed. A separate order control swaps Y and Z.

InterventionMedGemma-4B loss, %Qwen2.5-VL-7B loss, %
Z → No X56.335.3
Z → W13.319.0
Swap Y and Z6.316.4

Loss is the share of control-correct answers that become wrong under the intervention. Using study-cluster bootstrap 95% CIs, the paired excess of No X over W is 43.0 percentage points [34.8, 51.2] for MedGemma-4B and 16.4 [4.8, 27.0] for Qwen2.5-VL-7B.

Table 1 extends these controls to ten open-weight models and three frontier systems. Tables 7–9 add ten-model comparisons on the CheXpert training split, OpenI, and ReXGradient. On all three additional sets, the excess is positive in all five MedGemma, Lingshu, and CheXagent models. For example, MedGemma-27B has excesses of +17.9 [15.4, 20.5] on the training split and +23.7 [22.1, 25.3] on ReXGradient. Qwen2.5-VL-7B instead has excesses of −7.5 and −8.0 on those sets. The controls therefore distinguish model- and dataset-specific responses to negation from generic option sensitivity.

Each excess compares the two interventions on the same W-eligible, control-correct items. W is available for 1,815 training-split items, 126 OpenI items, and 4,512 ReXGradient items.

Q3 (excerpt). Could the authors further investigate including both "No X" and "No Y" in the same answer set, to better understand whether the observed behavior is driven by a general preference for negated options or a target-specific polarity reversal?

Regarding the joint presentation of No X and No Y, Appendix E.1 adds [X, No X, No Y] and [X, No Y, No X] on all 260 items. The two counts in each cell correspond to these two orders:

ModelSelect No XSelect No YSelect X
MedGemma-4B155 / 14831 / 4274 / 70
Qwen2.5-VL-7B113 / 9712 / 16135 / 147

Both models select the negation of the target more often in both orders: No X receives 3.5–9.4 times as many selections as No Y. A complementary control replaces Z with No Z, a true off-target negation, while retaining X and Y. That option attracts 76.6% of MedGemma-4B's and 54.3% of Qwen2.5-VL-7B's control-correct answers.

Together, the controls identify two behavioral components: selection of a negated statement even when it refers to another finding, and preferential selection of the target negation when both negations are displayed.

Manuscript changes: Sec. 4.2 and Table 1 report the matched controls. Appendix E.1 reports the two-negation experiment, and Tables 7–9 report the additional paired datasets.

Thank you for the detailed rebuttal. I believe these results should be included in the updated paper. I would like to maintain my score.

Dear Reviewer R1e1,

Thank you for reviewing our rebuttal. We have incorporated the key new results into the revised main text, including the matched controls in Sec. 4.2 and Table 1 and the image-evaluation results in Sec. 5.2 and Fig. 2, with key additions highlighted in blue. We sincerely appreciate your helpful suggestions and careful consideration.

Reviewer_foKnOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CXR-ContraBench, a benchmark designed to evaluate models’ sensitivity to prompt negation changes, specifically how their responses change depending on the presentation or formatting of options within a prompt. It uses a classification task in which the model is presented with three options: one positive and two negative. One of the negative options is then replaced with a “Not <positive class>” variant. The evaluation was conducted on a subset of the CheXpert dataset consisting of 260 items from 127 studies and assessed three models: MedGemma, LLaVA-Med, and Qwen 2.5. The authors calculate statistics on correct predictions, “correct-to-false-denial” reversals, recoveries, and other shifts in model responses. Additional experiments analyze the effects of variations in prompts, wording, and fine-tuning techniques such as LoRA.

Critical strengths
  • The paper highlights a genuine challenge in applying generative vision-language models (VLMs) in medical contexts: their sensitivity to changes in prompts and the presentation of response options.

  • The authors provide the source code in an anonymized repository, which enhances transparency, facilitates code inspection, and supports the reproducibility of the experiments.

  • The paper explores potential model limitations and failures using various techniques, such as chain-of-thought prompting and LoRA fine-tuning.

Critical weaknesses

The study uses a limited benchmark consisting of 260 items from 127 studies and evaluates only three model families (MedGemma, Qwen, and LLaVA-Med) which constrains the generalizability of the results. Although the paper includes an additional analysis using an external dataset and additional models, that analysis reports the selection of negated options rather than transitions from correct to incorrect answers, specifically false denials. This makes it difficult to determine whether the findings generalize across different models, parameter counts, or images from different cohorts.

The methodological novelty of the work is not sufficiently established, particularly in relation to the existing literature on negation, sensitivity to response options, and transitions from correct to incorrect answers. While the paper proposes a specific protocol for chest X-rays, it should clarify what it contributes beyond applying known phenomena to a medical context and what new conclusions can be drawn from its analysis.

A key component of the paper is the method used to parse model responses, which is briefly described in Appendix A.3. Potential limitations and errors in this process should be addressed to clarify how it works and how it might influence the results.

In many cases, the models’ baseline performance is very low. This raises questions about whether the observed shifts from correct to incorrect predictions are truly due to confusion caused by the inclusion of a negated option or are merely random fluctuations in model generation triggered by changes to the prompt. The analysis does not explore the underlying causes of this issue.

The LoRA-fine-tuned model achieves 100% performance, which is likely due to a “shortcut” in how the options are presented in the prompt (e.g., “a. X, b. Y, c. Not X”). The model may have learned to avoid selecting any option containing “Not” and instead select the affirmative option corresponding to the negated term.

Questions for authors
  • Are model refusal or abstention rate reported? for example, for cases in which the model states “I don’t know” or “I cannot answer”mseparately from parsing errors?

  • How exactly does the parser classify responses? Was it validated against manually reviewed responses or through an LLM-as-a-judge approach?

  • Based on the results and experiments, what are the main conclusions about these models’ capabilities in analyzing the images?

  • Why is the primary paired transition analysis restricted to 260 items? Can the same analysis be reported on the larger CheXpert construction or an external dataset?

  • For LLaVA-Med, what are the distributions of answer letters and affirmative versus negated choices across matched, opposite-label, shuffled, and blank images? Did you manually inspect the raw model generations to understand why the model never changes its decision? Could this behavior be related to sampling parameters such as temperature or top-p?

6 comments
AuthorsOpenReview ↗

Response to foKn (1/5): W1, paired evaluation across datasets and model sizes

Thank you for asking whether these reversals occur across datasets and model sizes. We have expanded the same transition analysis to address this question.

W1. Paired evaluation across datasets and model sizes

The study uses a limited benchmark consisting of 260 items from 127 studies and evaluates only three model families (MedGemma, Qwen, and LLaVA-Med) which constrains the generalizability of the results. Although the paper includes an additional analysis using an external dataset and additional models, that analysis reports the selection of negated options rather than transitions from correct to incorrect answers, specifically false denials. This makes it difficult to determine whether the findings generalize across different models, parameter counts, or images from different cohorts.

Tables 7–9 now report the same paired transition analysis for all ten open-weight models. Every item is evaluated under both [X, Y, Z] and [X, Y, No X], so the reported reversals track a correct answer becoming a false denial on the same image and question. We add:

  • CheXpert training split: 6,541 patients, using explicit positive and negative labeler mentions.
  • OpenI: 240 patients from Indiana University, using MeSH-coded and report-asserted presence and explicitly negated absence.
  • ReXGradient-160K: 11,240 patients from a dataset spanning 64 US institutions, with both polarities derived from explicit report statements.

Each additional set contains one item per patient. The table below illustrates the results for MedGemma-4B and Lingshu-7B. It reports control accuracy and RR, the percentage of control-correct answers that become No X:

ModelItem setControl accuracy, %RR, %Reversals / control-correct
MedGemma-4BCheXpert validation60.856.389/158
CheXpert training split78.460.53101/5125
OpenI92.153.4118/221
ReXGradient78.449.34342/8813
Lingshu-7BCheXpert validation60.418.529/157
CheXpert training split69.437.91721/4537
OpenI86.321.745/207
ReXGradient77.532.52836/8716

To separate negation from ordinary option sensitivity, we also replace Z with another reference-absent finding W. For MedGemma-4B, the excess loss under No X over W is:

Item setW-eligible itemsPaired excess, percentage points [95% CI]
CheXpert validation260+43.0 [34.8, 51.2]
CheXpert training split1,815+52.7 [50.1, 55.2]
OpenI126+47.9 [38.7, 57.1]
ReXGradient4,512+41.2 [39.5, 42.8]

Both losses use the same control-correct items within each W-eligible subset. CIs resample studies for validation and patients for the additional sets. On all three additional sets, the excess is positive in MedGemma-4B/27B, Lingshu-7B/32B, and CheXagent-2-3B. The Qwen2.5-VL and LLaVA models show a different pattern: No X removes no more correct answers than W while aggregate accuracy rises. The additional loss from negation therefore differs across the evaluated model families.

The ten open-weight models span six families and 3B–32B parameters: MedGemma 4B/27B, Qwen2.5-VL 3B/7B/32B, Lingshu 7B/32B, CheXagent-2-3B, LLaVA-Med-7B, and LLaVA-1.5-7B. These include three within-family size series.

Table 1 additionally reports matched controls for Claude Opus 5, Gemini 3.6 Flash, and GPT-5.6.

Manuscript changes: Secs. 3.1, 4, and 6 describe the expanded evaluation. Table 1 reports the validation controls, and Tables 7–9 report all ten open-weight models on the three additional sets.

AuthorsOpenReview ↗

Response to foKn (2/5): W2-W3, contribution and parser validation

Thank you for asking us to clarify the contribution and validate the response parser.

W2. Contribution and new findings

The methodological novelty of the work is not sufficiently established, particularly in relation to the existing literature on negation, sensitivity to response options, and transitions from correct to incorrect answers. While the paper proposes a specific protocol for chest X-rays, it should clarify what it contributes beyond applying known phenomena to a medical context and what new conclusions can be drawn from its analysis.

CXR-ContraBench tracks which correct judgments are lost after an option change and tests whether gains in answer-set accuracy extend to images requiring opposite answers. Paired transitions measure the losses, matched replacements quantify the additional effect of negation, and image tests check whether the gains depend on image evidence. Secs. 4 and 5 report three findings.

1. Matched controls measure the additional loss caused by the negated option. On the 260 validation items, No X removes 56.3% of MedGemma-4B's correct answers, compared with 13.3% for another absent finding W. The paired excess is 43.0 percentage points [34.8, 51.2]. On the three additional datasets, this excess is positive in all five evaluated MedGemma, Lingshu, and CheXagent models. In the Qwen2.5-VL and LLaVA models, No X removes no more correct answers than W. The controls therefore distinguish the additional effect of negation across models.

2. Higher accuracy can coexist with new false denials. Qwen2.5-VL-7B's accuracy rises from 44.62% to 50.38% while 41 of its 116 control-correct answers become false denials. Accuracy rises because recoveries outnumber reversals, while the paired counts show both changes. On the validation items, higher accuracy and new false denials coexist in eight of the ten open-weight models and in all three frontier systems.

3. A lower reversal rate can result from different adaptation behaviors. Sec. 5.1 and Table 2 show that in-format LoRA reaches zero RR with zero PairCorrect. Image-balanced adaptation reduces RR from 54.7% to 10.0% while reaching 46.2–50.2% PairCorrect across formats. Thus, the image test distinguishes a shortcut from improved correctness across opposite image conditions. Our response to W5 gives the full comparison.

Sec. 2 now discusses DARE, CARETS, MedVH, Med-CP, and negative-flip analyses. Sec. 1 now explains what paired transitions, matched controls, and image-conditioned evaluation each contribute.

W3. Parser validation and its effect on results

A key component of the paper is the method used to parse model responses, which is briefly described in Appendix A.3. Potential limitations and errors in this process should be addressed to clarify how it works and how it might influence the results.

Independent checks preserve the primary MedGemma-4B and Qwen2.5-VL-7B transition counts and correct one CoT result. Appendix A.3 specifies the rules and validation. The scoring pipeline matches each raw response to a displayed option using its letter or its finding and polarity. The base parser accepts a final option letter, an explicit answer phrase, a final line repeating one displayed option, or prose that identifies exactly one option. Outputs that remain unresolved after the parsing and adjudication steps below count as errors.

We check these rules on the original runs with an independently written re-parser and two blinded LLM judges from different model families. The judges see only the question, options, and raw response. On 300 stratified outputs, they agree on all 300. The released parser matches their consensus on 291/300, or 97.0%. All nine differences are CoT outputs. Applying the adjudicated scores changes Qwen2.5-VL-7B's evidence-first RR from 44.05% to 45.88%. The primary MedGemma-4B and Qwen2.5-VL-7B outputs are single letters and are unaffected.

For the added models and question formats, we specify three fallback rules: a parenthesized letter, a stated letter consistent with any following option text, and a unique finding-polarity match. These are applied to rows left invalid by the base parser. The two judges assess 239 such outputs and agree on 231. The rules match 213 of these 231. For 17 MedGemma-27B outputs without a rule-based match, the score uses the judges' consensus label. The remaining difference concerns an unscored blank-image reply. All 91 outputs that remain invalid count as errors. Appendix A.3 records this adjudication procedure.

AuthorsOpenReview ↗

Response to foKn (3/5): W4-W5, sampling variation and the LoRA shortcut

Thank you for raising the questions about sampling variation and the LoRA shortcut.

W4. Baseline accuracy and sampling variation

In many cases, the models' baseline performance is very low. This raises questions about whether the observed shifts from correct to incorrect predictions are truly due to confusion caused by the inclusion of a negated option or are merely random fluctuations in model generation triggered by changes to the prompt. The analysis does not explore the underlying causes of this issue.

MedGemma-4B reaches 92.1% control accuracy on OpenI and reverses over half of its correct answers after the option change. In the validation experiments, its reversal rate under the option intervention also exceeds unchanged-prompt sampling variation. Sec. 4 and Appendices A.4 and B provide the following checks.

High baseline accuracy. On OpenI, MedGemma-4B has 92.1% control accuracy, yet 118 of its 221 correct answers become No X, giving RR 53.4%. On the 126 W-eligible items, the paired excess loss under No X over an ordinary replacement is +47.9 percentage points [38.7, 57.1], with both losses evaluated on the same control-correct subset.

Confident and directional transitions. On the 260 validation items, all 89 MedGemma-4B losses and all 41 Qwen2.5-VL-7B losses go to No X. For MedGemma-4B, 145 of the 158 control-correct answers assign at least 0.99 probability to the correct letter. Of those 145 answers, 79 reverse, a rate of 54.5%.

Repeated decoding. The primary open-weight runs use greedy decoding. Repeating the three original answer-set arms reproduces all 780 predictions for MedGemma-4B and all 780 for Qwen2.5-VL-7B. We additionally repeat the control and No X arms with temperature 0.7, top-p 0.9, and five seeds. For MedGemma-4B, resampling an unchanged prompt turns 0.8% of correct answers into wrong answers, whereas the No X intervention gives 56.5% reversal, with 56.0–57.5% across seeds. For Qwen2.5-VL-7B, unchanged-prompt sampling disagreement is 43.8%. Under fixed greedy decoding, the matched controls in Table 1 give 16.4 points of excess loss on validation. Tables 7–9 report the corresponding dataset-specific contrasts.

Behavioral components. In Appendix E.1, jointly displaying No X and No Y yields 3.5–9.4 times as many No X selections in both tested positions. Together with the ordinary-replacement controls, this identifies a preference for the target negation beyond generic option sensitivity.

W5. Testing the LoRA shortcut with opposite image conditions

The LoRA-fine-tuned model achieves 100% performance, which is likely due to a "shortcut" in how the options are presented in the prompt (e.g., "a. X, b. Y, c. Not X"). The model may have learned to avoid selecting any option containing "Not" and instead select the affirmative option corresponding to the negated term.

Our image tests support this explanation. In Sec. 5.1 and Table 2, the in-format adapters select the affirmative member of the displayed X / No X pair on every matched and opposite-label image, giving 0% PairCorrect. To test whether adaptation can also improve image-conditioned correctness, we compare two additional MedGemma adapters.

These adapters use the same base checkpoint, LoRA rank, learning rate, and number of epochs. Each uses 4,008 training-split items from patients disjoint from the evaluation set. Both use the three-statement format. Every image in the unbalanced control contains X, whereas half of the images in the image-balanced version do not. The latter therefore requires both affirmative and negative answers during training.

All four rows below use the same base revision. Its RR is 54.7%, compared with 56.3% for the main-protocol checkpoint, as Appendix A.4 records. PairCorrect requires correct answers on both the target-present and target-absent image. All values are percentages:

AdapterControl accuracyRRPairCorrect: training / binary / original format
Base61.254.78.5 / 44.6 / 17.7
In-format LoRA34.60.00.0 / 0.0 / 0.0
Unbalanced control26.20.00.0 / 0.0 / 0.0
Image-balanced50.010.046.2 / 50.2 / 50.0

The unbalanced adapter reproduces the in-format behavior. The image-balanced adapter reduces RR from 54.7% to 10.0% and reaches PairCorrect of 46.2–50.2% across the three formats. Its AUC is 0.79–0.80.

RR alone ranks the shortcut adapters highest, while PairCorrect shows which adapter answers both image conditions correctly. Reporting both alongside control accuracy shows how often correct answers are preserved and whether the adapter distinguishes the images.

Manuscript changes: Sec. 5.1, Table 2, and Fig. 2c,d present this comparison. Appendix C specifies the adaptation setup, and Sec. 1 explains its role in evaluating model improvements.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: matched controls, cross-dataset replication, and image-conditioned evaluation

We thank the reviewers for their careful reading and constructive suggestions. We have carried out additional experiments to address the main questions about negation-specific effects, generalization, image use, and scoring reliability.

Core contribution. CXR-ContraBench measures whether an option change turns a previously correct finding into a false denial, whether that loss exceeds ordinary option sensitivity, and whether an apparent improvement holds across images requiring opposite answers. For example, Qwen2.5-VL-7B's accuracy rises from 44.62% to 50.38% while 41 of its 116 initially correct answers become false denials. By tracking losses and recoveries separately and checking image-conditioned correctness, the benchmark reveals errors that an aggregate accuracy gain can conceal.

The additional experiments address five aspects of this evaluation:

1. Matched controls isolate the additional effect of negation. We add ordinary distractor replacements, distractor reordering, off-target negation, and joint presentation of No X and No Y. These comparisons keep the image, question, and gold answer fixed. On the 260 validation items, MedGemma-4B loses 56.3% of its correct answers under No X versus 13.3% under an ordinary replacement, an excess of 43.0 percentage points with a study-cluster 95% CI of [34.8, 51.2]. Table 1 reports the controls for ten open-weight models and three frontier systems.

2. Larger paired evaluations test generalization across datasets and model sizes. Tables 7–9 extend the same correct-to-false-denial analysis to 6,541 CheXpert training-split patients, 240 OpenI patients, and 11,240 ReXGradient patients, with one item per patient in each additional set. All ten open-weight models are evaluated on these sets, spanning six families, 3B–32B parameters, and three within-family size comparisons. The excess loss over ordinary replacement is positive in all five evaluated MedGemma, Lingshu, and CheXagent models on each additional set. Each contrast uses the same eligible, control-correct items for both interventions.

3. Question-consistent tests measure how models use image evidence. In response to the question-wording concern, we add binary, Yes/No, and three-statement questions across matched, opposite-label, shuffled, and blank images. With both option orders for the two-option questions, this gives five format variants and 5,200 evaluations per model across ten models. The binary question raises MedGemma-4B's paired image correctness from 17.7% to 42.7%. These tests measure whether models answer both target-present and target-absent images correctly, alongside AUC computed independently of the response parser. Fig. 2 visualizes the results, and Table 4 reports the complete comparison.

4. Controlled adaptation experiments test whether lower reversal rates also improve image-conditioned correctness. We train two additional MedGemma adapters, each on 4,008 training-split items from patients disjoint from evaluation. They share the base checkpoint, LoRA settings, and question format, while differing in whether the training images require both affirmative and negative answers. The image-balanced adapter reduces reversal from 54.7% to 10.0% and reaches 46.2–50.2% paired image correctness across three formats. Table 2 and Fig. 2c,d compare this behavior with the in-format and unbalanced adapters, showing how the benchmark tests whether adaptation gains also hold across image conditions.

5. Independent scoring checks and decoding repeats examine result reliability. We validate response parsing with an independent re-parser and two blinded LLM judges, including 300 stratified outputs and 239 outputs requiring supplementary parsing rules. We also repeat greedy decoding, run sampling comparisons with five seeds, inspect LLaVA-Med's raw outputs, and report abstentions separately from other invalid responses. Appendices A.3–A.5 document these checks and the model-specific results.

These additions address the reviewers' main empirical questions with controlled comparisons and measurable results. Key additions are highlighted in blue. The reviewer-specific responses below provide the detailed evidence and manuscript locations in the original W/Q order.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.