- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates text-induced bias in multimodal multiple-choice question answering, where models may rely on linguistic preferences from the question and answer choices rather than sufficiently using visual evidence. The authors observe that correct predictions generally show larger Jensen–Shannon divergence between multimodal and text-only answer distributions. Motivated by this observation, they propose BUZZY, a training-free contrastive scoring method that subtracts weighted text-only candidate scores from the corresponding multimodal scores to discount preferences attributable to text alone. Experiments cover five vision-language models and five benchmarks. BUZZY achieves the highest average accuracy and Macro-F1 for each backbone, with an overall average accuracy of 57.84%, exceeding the strongest aggregate baseline by 1.99 percentage points. Its additional text-only forward pass also yields lower reported inference latency than VCD and ICD. Further analyses examine image replacement, sensitivity to the correction strength, and performance on a domain-specific fine-tuned model to investigate visual dependence and applicability.
Critical strengths
- The empirical gains span multiple models and tasks, offering practical reference value. In Table 1, BUZZY achieves the highest cross-benchmark average accuracy and Macro-F1 score across all five backbone models; the appendix also provides per-class results and complete paired hypothesis tests. The fact that domain-adapted models still exhibit gains indicates that this phenomenon is not limited to any specific base model lacking adaptation, making this approach attractive as a practical inference strategy.
- This method is simple to implement and incurs low additional computational costs. It requires no training or auxiliary models, relying solely on one additional pure-text scoring step. The average latency reported in the paper is approximately 0.30 seconds, lower than the 0.45 seconds for VCD and 0.42 seconds for ICD. By consistently using the same α value and providing sensitivity analysis, we also mitigate concerns that the gains might be entirely dependent on hyperparameter tuning per dataset.
Critical weaknesses
- The distinctions from existing methods remain unclear. VDD has already employed image-free branch correction on multimodal scores; after matching weights and candidate filtering settings, it can achieve the same ranking as BUZZY. However, this paper lacks direct comparisons and a clear articulation of differences, meaning its independent contributions at the algorithmic level have not been sufficiently established.
- Key evaluation details require clarification. While the paper cites iterative evaluations on MMBench to support its conclusions, the supplementary code does not demonstrate the corresponding option rotation and aggregation procedures. Furthermore, it remains unclear whether the primary experiments employed complete candidate scoring or relied on score imputation. These settings may affect the reproducibility of rankings and results.
- The mechanistic rationale for visual grounding remains insufficient. Greater distributional diversity does not necessarily imply more accurate utilization of visual evidence. Although image substitution experiments support image sensitivity, they fail to adequately distinguish between task-relevant visual information and other score variations. Consequently, the evidence for accuracy improvements is more robust than the mechanistic explanations provided.
Questions for authors
- Under identical model configurations, candidate sets, and plausibility filtering settings, what are the specific differences between BUZZY and VDD-None? If the default scoring rankings are equivalent, how should the independent contributions of the papers be defined?
- Does Table 1 present MMBench results based on standard per-item accuracy or strict CircularEval metrics? Please provide the exact option ordering, original problem grouping, and the actual code or records used for final aggregation.
- In the main experiments for each model, do you use full candidate token scores, or top-logprobs plus floor? If the latter approach is used, what is the proportion of missing candidates across the two branches, and do the results under full scoring remain consistent?
Dear Reviewer Reviewer kUvN,
Thank you again for your time and effort in reviewing our paper. As the discussion period is coming to an end, we would like to kindly ask whether our responses have addressed your concerns. If there are any remaining questions, we would be very happy to answer them.