- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates whether multimodal hallucination can be reduced through hard grounding preference supervision: semantically similar preferred and rejected responses that differ primarily in their support from visual evidence. The proposed data-construction pipeline targets evidence utilization, evidence calibration, and grounding consistency across counting, OCR, spatial reasoning, and false-premise questions. The model uses a frozen ViT encoder and Perceiver alignment module, with only decoder-side LoRA parameters in a LLaMA-3 70B model updated through broad multimodal SFT followed by standard DPO. The DPO checkpoint improves over its SFT initialization on DocVQA, TextVQA, MMBench, and VQAv2, and obtains 6.2–8.2 percentage-point higher pairwise win rates across three LLM judges on approximately 839 examples. The paper also releases HardVQA-DPO, containing more than 3K hard-grounding SFT examples and an initial collection of preference pairs.
Critical strengths
- The preference-construction principle addresses a meaningful grounding–informativeness trade-off. The paper distinguishes genuinely insufficient evidence from difficult but sufficient evidence, targeting both unsupported specificity and unnecessary refusal. It also considers cases where two responses reach the same final answer but differ in the correctness of their supporting explanations. These distinctions make the supervision more relevant to practical multimodal grounding than simply rewarding conservative answers or penalizing incorrect final predictions. The construction examples and released resource provide useful starting points for studying these behaviors.
Critical weaknesses
- The specific motivation for the proposed grounding taxonomy lacks systematic empirical support. While multimodal hallucination is well established, the paper motivates its focus on evidence utilization, evidence calibration, and grounding consistency primarily through selected qualitative examples. It does not quantify the prevalence of these failures, their distribution across models, or their contribution to overall grounding errors. The 839-example evaluation reports aggregate response preferences rather than a category-level diagnosis. This leaves unclear whether the proposed taxonomy captures recurring, consequential failures or mainly serves as an illustrative framework for organizing examples.
- The experiments do not isolate the benefit of hard grounding preference construction. The principal comparison is between SFT and SFT followed by DPO, without matched-data and matched-compute controls using ordinary preference pairs, easier grounding pairs, or continued SFT on the preferred responses. Consequently, the improvements could reflect additional targeted supervision rather than the hardness or semantic closeness of the preference pairs. Since the optimization objective is standard DPO, establishing the distinctive value of the data-construction strategy is central to assessing the methodological contribution.
Questions for authors
- Beyond the selected qualitative examples, do you have systematic statistics showing the frequency and distribution of evidence-utilization, evidence-calibration, and grounding-consistency failures, ideally across multiple models? In particular, can the 839-example evaluation provide a category-level breakdown of the SFT model’s failures and the changes after DPO? Such evidence would materially affect my assessment of whether the proposed supervision addresses recurring, consequential failure modes rather than isolated cases.
- Do you have matched-data and matched-compute comparisons against ordinary-preference DPO, easier grounding pairs, or continued SFT on the preferred responses?
We thank the reviewer for the constructive feedback. We clarify the empirical basis of our grounding taxonomy and provide additional comparisons isolating the contribution of hard-grounding preference construction.
(b) Category-level analysis. We analyze 839 evaluation examples, assigning each one primary visual-task category and independently assessing the three grounding mechanisms for SFT and Hard DPO. We now provide the category-level breakdown in Appendix C.4 (Table C.1).
Table C.1. Grounding-failure counts by visual-task category (N=839). Each entry reports SFT → Hard DPO failure counts.
Here, N denotes the number of evaluation examples in each category. For example, among 182 unsupported-attribute examples, evidence-utilization failures decrease from 30 under SFT to 21 under Hard DPO (30→21). Each example is evaluated under both models, and mechanism labels may overlap. Relative reduction is calculated as (SFT failures − Hard-DPO failures)/SFT failures.
Hard DPO reduces evidence-utilization failures by 29.1% (223→158), evidence-calibration failures by 30.9% (285→197), and grounding-consistency failures by 30.4% (204→142). These reductions occur across multiple visual-task categories, providing mechanism-specific evidence beyond aggregate benchmark accuracy.
(c) Independent evaluation. On the same 839-example evaluation set, GPT-4o, Gemini-2.5-Flash, and LLaMA-4-Maverick report Hard-DPO versus SFT pairwise win-rate advantages of +8.2, +7.2, and +6.2 percentage points, respectively. These independent judgments complement the failure analysis by assessing overall grounding-sensitive response quality.
2. Contribution of hard-grounding preference construction (Weakness 2; Question 2)
(a) Distinctive contribution. Our contribution is not a new DPO objective but hard-grounding preference construction, where semantically close responses differ in fine-grained visual support, such as object existence, attributes, counts, OCR, spatial relations, or reasoning.
The construction is bidirectional: it discourages unsupported specificity when evidence is insufficient, preserves informative answers when evidence is sufficient, and penalizes visually inconsistent reasoning even when final answers agree. This avoids simply rewarding shorter or more conservative responses.
(b) Conventional-DPO comparison. To better isolate the contribution, we additionally compare conventional DPO without hard-boundary construction against Hard DPO, initialized from the same Stage-1 SFT checkpoint. Both use the same DPO objective, frozen visual pathway, and decoder-side LoRA adaptation.
Hard DPO improves over conventional DPO by +1.1/+0.8/+1.5/+0.4 points, respectively. This provides additional evidence that grounding-focused preference construction benefits performance beyond standard DPO optimization.
(c) Controls and limitations. Our preference filtering verifies visual support and preference validity, while length-aware resampling and visually supported response expansion discourage trivial brevity-based preferences. Although the conventional-DPO comparison controls for the optimization objective and SFT initialization, fully isolating pair hardness requires matched preference-data scale and training budget. Easier-pair DPO and continued SFT on preferred responses would provide complementary controls.
Summary. The mechanism-specific reductions (29.1–30.9%), independent three-judge preferences (+6.2–8.2 pp), and improvements over conventional DPO jointly support the effectiveness of hard-grounding preference supervision across grounding behavior, response quality, and benchmark performance.