acceptodds
Under review as a conference paper at ICLR 2027

BUZZY: Contrastive Scoring to Mitigate Text-Induced Bias in Multimodal Multiple-Choice QA

26 views

Abstract

Multimodal multiple-choice question answering (MCQA) provides a standardized and objectively measurable setting for evaluating vision-language models (VLMs). However, because the MCQA format incorporates the candidate choices into the input context, it introduces several unintended biases. Previous work has primarily focused on structural biases, such as preferences for certain choices. Instead, we argue that the choices act as textual priors, causing models to favor linguistically plausible options regardless of the visual content. We hypothesize and empirically verify that a model genuinely relies on visual evidence only when its multimodal distribution significantly diverges from its text-only distribution. Based on this observation we propose BUZZY,a training-free decoding method that corrects multimodal predictions by subtracting the text-only distribution. Experiments with five VLMs on five multimodal MCQA benchmarks demonstrate that BUZZY achieves the highest average accuracy among state-of-the-art methods while reducing inference latency by over 28% compared to prior contrastive decoding approaches. Overall, these results suggest that amplifying the visual signal by penalizing text-only preferences is key to efficient and robust multimodal MCQA reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Multimodal Multiple-Choice QA, Contrastive Decoding, Visual Grounding, Text-Induced Bias

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JKZgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates text-induced bias in multimodal multiple-choice question answering, where models may rely on linguistic preferences from the question and answer choices rather than sufficiently using visual evidence. The authors observe that correct predictions generally show larger Jensen–Shannon divergence between multimodal and text-only answer distributions. Motivated by this observation, they propose BUZZY, a training-free contrastive scoring method that subtracts weighted text-only candidate scores from the corresponding multimodal scores to discount preferences attributable to text alone. Experiments cover five vision-language models and five benchmarks. BUZZY achieves the highest average accuracy and Macro-F1 for each backbone, with an overall average accuracy of 57.84%, exceeding the strongest aggregate baseline by 1.99 percentage points. Its additional text-only forward pass also yields lower reported inference latency than VCD and ICD. Further analyses examine image replacement, sensitivity to the correction strength, and performance on a domain-specific fine-tuned model to investigate visual dependence and applicability.

Critical strengths
  1. The empirical gains span multiple models and tasks, offering practical reference value. In Table 1, BUZZY achieves the highest cross-benchmark average accuracy and Macro-F1 score across all five backbone models; the appendix also provides per-class results and complete paired hypothesis tests. The fact that domain-adapted models still exhibit gains indicates that this phenomenon is not limited to any specific base model lacking adaptation, making this approach attractive as a practical inference strategy.
  2. This method is simple to implement and incurs low additional computational costs. It requires no training or auxiliary models, relying solely on one additional pure-text scoring step. The average latency reported in the paper is approximately 0.30 seconds, lower than the 0.45 seconds for VCD and 0.42 seconds for ICD. By consistently using the same α value and providing sensitivity analysis, we also mitigate concerns that the gains might be entirely dependent on hyperparameter tuning per dataset.
Critical weaknesses
  1. The distinctions from existing methods remain unclear. VDD has already employed image-free branch correction on multimodal scores; after matching weights and candidate filtering settings, it can achieve the same ranking as BUZZY. However, this paper lacks direct comparisons and a clear articulation of differences, meaning its independent contributions at the algorithmic level have not been sufficiently established.
  2. Key evaluation details require clarification. While the paper cites iterative evaluations on MMBench to support its conclusions, the supplementary code does not demonstrate the corresponding option rotation and aggregation procedures. Furthermore, it remains unclear whether the primary experiments employed complete candidate scoring or relied on score imputation. These settings may affect the reproducibility of rankings and results.
  3. The mechanistic rationale for visual grounding remains insufficient. Greater distributional diversity does not necessarily imply more accurate utilization of visual evidence. Although image substitution experiments support image sensitivity, they fail to adequately distinguish between task-relevant visual information and other score variations. Consequently, the evidence for accuracy improvements is more robust than the mechanistic explanations provided.
Questions for authors
  1. Under identical model configurations, candidate sets, and plausibility filtering settings, what are the specific differences between BUZZY and VDD-None? If the default scoring rankings are equivalent, how should the independent contributions of the papers be defined?
  2. Does Table 1 present MMBench results based on standard per-item accuracy or strict CircularEval metrics? Please provide the exact option ordering, original problem grouping, and the actual code or records used for final aggregation.
  3. In the main experiments for each model, do you use full candidate token scores, or top-logprobs plus floor? If the latter approach is used, what is the proportion of missing candidates across the two branches, and do the results under full scoring remain consistent?
Reviewer_kUvNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how text-only preferences from the question stem and candidate answers bias models in multimodal multiple-choice questions. The authors propose BUZZY, a training-free method that computes each option label's logits with and without the image and subtracts a weighted version of the latter from the former. Across 5 vision-language models and 5 benchmarks, BUZZY beats Greedy, prompt enhancement, VCD, ICD, and ALC on average accuracy and Macro-F1, and is faster than baselines needing a second multimodal forward pass.

Critical strengths

1.Table 1 spans 5 models and 5 benchmarks, and BUZZY leads on both cross-benchmark average accuracy and Macro-F1 for every model. This supports the method's practical utility better than a small set of hand-picked examples would.

2.The method also needs only one multimodal branch and one text-only branch; Figure 3 reports lower average latency than VCD and ICD, so its accuracy gains do not come merely from a more expensive second visual pass.

Critical weaknesses

1.The evidence for visual grounding is indirect. Figures 1 and 5 (Appendix A.3) show an association between correct predictions and larger JS divergence, but this alone does not establish that the model is grounding its answers in the image. The image-substitution split in Section 6.1 also leaves possible selection effects unresolved. A direct control would strengthen this interpretation; otherwise, the grounding claims should be more cautious.

2.The gains over VCD are less consistent than the main text suggests. In Appendix D, only 10 of the 25 model–benchmark comparisons reach nominal significance before correcting for multiple comparisons, and two models perform worse than VCD on SciFIBench. These results suggest an average benefit rather than consistent superiority. The claim that BUZZY “consistently outperforms” the baselines in the Introduction and Section 5.2 also conflicts with Appendix D’s acknowledgment that the advantage depends on the backbone and benchmark. An uncertainty estimate for the aggregate gain and a simple text-prior baseline would help clarify the strength and source of the improvement.

Questions for authors

1.Please report per-setting paired counts of corrected and introduced errors for BUZZY versus VCD, along with effect intervals, and state which conclusions survive multiple-comparison correction. If the advantage over the strongest baseline is not robust, the overall recommendation is affected.

2.Could the authors add a visual control that is semantically unrelated to the image but similar in input format, and compare against a simple prior-calibration method using the same text-only branch? Would these experiments separate gains from "the image changing the logits," from the specific reweighting rule, and from genuinely using visual evidence?

2 comments

Dear Reviewer Reviewer kUvN,

Thank you again for your time and effort in reviewing our paper. As the discussion period is coming to an end, we would like to kindly ask whether our responses have addressed your concerns. If there are any remaining questions, we would be very happy to answer them.

Thank you for the detailed response. I’ll keep my original rating.

Reviewer_oCGcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper investigates text-induced biases in multimodal multiple-choice question answering, arguing that candidate answers can induce preferences that persist even when an image is available. It proposes BUZZY, a training-free inference method that scores each answer label by subtracting a weighted text-only logit from its multimodal logit:[s(c)=\ell_{\mathrm{mm}}(c)-\alpha\ell_{\mathrm{txt}}(c),] with a fixed (\alpha=0.5). The paper evaluates five VLM backbones on five benchmarks and reports average accuracy of 57.84% and Macro-F1 of 53.56%, exceeding VCD by 1.99 and 1.85 points, respectively. Its text-only reference branch also reduces reported latency relative to the evaluated contrastive methods requiring two multimodal passes. Additional validation includes image-replacement experiments, a contrast-weight sensitivity analysis, evaluation of a domain-fine-tuned backbone, and paired statistical comparisons. The main claimed contributions are the characterization of text-induced answer preferences, the proposed correction, and its accuracy–efficiency benefits.

Critical strengths
  1. Broad and reasonably consistent empirical benefits with a fixed correction weight. Table 1 reports accuracy improvements over greedy decoding in all 25 backbone–benchmark settings, with the highest average accuracy and Macro-F1 for each backbone. The overall accuracy gain is approximately 3.38 points over greedy decoding and 1.99 points over VCD. Using a single global (\alpha=0.5), together with the sensitivity analysis and additional fine-tuned-backbone experiment, strengthens the evidence that the intervention is practically useful beyond one model or dataset. The appendix also provides paired statistical comparisons rather than relying exclusively on aggregate averages.
  2. A simple intervention with a favorable measured cost relative to the evaluated multimodal contrastive baselines. The method requires no training, auxiliary model, or architectural modification. Replacing the second multimodal pass with a text-only pass provides a clear computational rationale for the reported latency reduction: approximately 0.30 seconds versus 0.45 seconds for VCD and 0.42 seconds for ICD. This simplicity is consequential for deployment in settings where candidate logits are accessible, although the comparison with other existing image-free methods remains important.
Critical weaknesses

The contribution over directly relevant image-free contrastive methods is not established. Zhang et al.’s Debiasing Multimodal Large Language Models via Penalization of Language Priors already introduces Visual Debias Decoding using multimodal and image-free logits:[s_{\mathrm{VDD}}=(1+\lambda)\ell_{\mathrm{mm}}-\lambda\ell_{\mathrm{txt}}.] At (\lambda=1), this produces the same candidate ranking as BUZZY with (\alpha=0.5), before VDD’s additional plausibility filtering. VDD’s complete decoding procedure differs in its vocabulary filtering and generation setting, but the central image-free contrast is already present. Favero et al.’s Multi-Modal Hallucination Control by Visual Information Grounding also studies image-conditioned versus image-free distributions and evaluates a closely related multimodal PMI decoding baseline. Neither work is discussed or evaluated in this submission. Consequently, comparisons against distorted-image contrast, disturbed-instruction contrast, and structural-bias correction do not establish the incremental contribution over the closest alternatives. The extensive MCQA evaluation may still be valuable, but the paper needs to identify what is substantively new beyond applying an existing image-free contrastive principle to answer-label scoring.

Questions for authors
  1. What is the substantive algorithmic difference between BUZZY and VDD-None applied to the same answer-label space with matched relative weighting? Please distinguish changes in the scoring rule from candidate restriction or plausibility filtering, and clarify the contribution relative to the multimodal PMI baseline evaluated by Favero et al. Relatedly, was the original plausibility constraint retained when adapting VCD to candidate-label scoring?
  2. What justifies interpreting the text-orthogonal residual in Section 6.1 as a non-textual or visually grounded component, rather than simply a direction orthogonal to the text-only score vector? Genuine visual evidence can reinforce the text-only preference; moreover, for two answer options, mean-centered score vectors are one-dimensional and the residual is identically zero whenever the text-only vector is nonzero. I appreciate the image-replacement controls in Table 2, but do the conclusions also hold over the full evaluation set, including unchanged predictions, rather than only the corrected/harmed groups selected using the original-image predictions? Please also specify the backbone and group sizes used for Table 2.
1 comment

Dear Reviewer Reviewer oCGc,

Thank you again for your time and effort in reviewing our paper. As the discussion period is coming to an end, we would like to kindly ask whether our responses have addressed your concerns. If there are any remaining questions, we would be very happy to answer them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.