acceptodds
Under review as a conference paper at ICLR 2027

Reducing Hallucination in Multimodal Large Language Models through Hard Grounding Preference Supervision

Abstract

Hallucination remains a major challenge in vision-language models (VLMs), particularly when linguistically plausible responses are unsupported by visual evidence. We study whether multimodal hallucination can be reduced by concentrating post-training supervision on hard grounding boundaries, where preferred and rejected responses are semantically close but differ in their support from observable visual evidence. Under a frozen visual encoder and cross-modal alignment pathway, we first use supervised fine-tuning (SFT) to establish broad decoder-side multimodal behavior and then construct hard grounding preference pairs for Direct Preference Optimization (DPO). These pairs target evidence utilization, calibration, and grounding consistency across fine-grained recognition, spatial reasoning, OCR, ambiguous or insufficient evidence, and false-premise queries. The resulting DPO model consistently improves over its SFT initialization on DocVQA, TextVQA, MMBench, and VQAv2. Across approximately 839 additional multimodal examples, it further achieves 6.2–8.2 percentage-point higher pairwise win rates under three independent LLM judges. We additionally release HardVQA-DPO, a curated and growing resource with more than 3K hard-grounding SFT examples and an initial set of DPO preference pairs.https://huggingface.co/datasets/vlmgrounding/hardvqa-sft-dpo These results show that decoder-side adaptation alone can improve a meaningful subset of grounding failures under a fixed multimodal representation, while preserving broader multimodal capabilities.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Hallucination Mitigation, Visual Grounding, Preference Optimization, Direct Preference Optimization (DPO), Multimodal Post-Training, Hard Example Mining

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XwFZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether multimodal hallucination can be reduced through hard grounding preference supervision: semantically similar preferred and rejected responses that differ primarily in their support from visual evidence. The proposed data-construction pipeline targets evidence utilization, evidence calibration, and grounding consistency across counting, OCR, spatial reasoning, and false-premise questions. The model uses a frozen ViT encoder and Perceiver alignment module, with only decoder-side LoRA parameters in a LLaMA-3 70B model updated through broad multimodal SFT followed by standard DPO. The DPO checkpoint improves over its SFT initialization on DocVQA, TextVQA, MMBench, and VQAv2, and obtains 6.2–8.2 percentage-point higher pairwise win rates across three LLM judges on approximately 839 examples. The paper also releases HardVQA-DPO, containing more than 3K hard-grounding SFT examples and an initial collection of preference pairs.

Critical strengths
  1. The preference-construction principle addresses a meaningful grounding–informativeness trade-off. The paper distinguishes genuinely insufficient evidence from difficult but sufficient evidence, targeting both unsupported specificity and unnecessary refusal. It also considers cases where two responses reach the same final answer but differ in the correctness of their supporting explanations. These distinctions make the supervision more relevant to practical multimodal grounding than simply rewarding conservative answers or penalizing incorrect final predictions. The construction examples and released resource provide useful starting points for studying these behaviors.
Critical weaknesses
  1. The specific motivation for the proposed grounding taxonomy lacks systematic empirical support. While multimodal hallucination is well established, the paper motivates its focus on evidence utilization, evidence calibration, and grounding consistency primarily through selected qualitative examples. It does not quantify the prevalence of these failures, their distribution across models, or their contribution to overall grounding errors. The 839-example evaluation reports aggregate response preferences rather than a category-level diagnosis. This leaves unclear whether the proposed taxonomy captures recurring, consequential failures or mainly serves as an illustrative framework for organizing examples.
  2. The experiments do not isolate the benefit of hard grounding preference construction. The principal comparison is between SFT and SFT followed by DPO, without matched-data and matched-compute controls using ordinary preference pairs, easier grounding pairs, or continued SFT on the preferred responses. Consequently, the improvements could reflect additional targeted supervision rather than the hardness or semantic closeness of the preference pairs. Since the optimization objective is standard DPO, establishing the distinctive value of the data-construction strategy is central to assessing the methodological contribution.
Questions for authors
  1. Beyond the selected qualitative examples, do you have systematic statistics showing the frequency and distribution of evidence-utilization, evidence-calibration, and grounding-consistency failures, ideally across multiple models? In particular, can the 839-example evaluation provide a category-level breakdown of the SFT model’s failures and the changes after DPO? Such evidence would materially affect my assessment of whether the proposed supervision addresses recurring, consequential failure modes rather than isolated cases.
  2. Do you have matched-data and matched-compute comparisons against ordinary-preference DPO, easier grounding pairs, or continued SFT on the preferred responses?
1 comment

We thank the reviewer for the constructive feedback. We clarify the empirical basis of our grounding taxonomy and provide additional comparisons isolating the contribution of hard-grounding preference construction.

  1. Systematic grounding failure analysis (Weakness 1; Question 1) (a) Grounding taxonomy. Our three mechanisms describe how grounding fails, rather than mutually exclusive visual-task categories:
  • Evidence utilization: Failure to use available but subtle visual evidence.
  • Evidence calibration: Unsupported specificity or unnecessary rejection despite sufficient evidence.
  • Grounding consistency: Claims or reasoning inconsistent with the image, even when the final answer is correct. These mechanisms operate across visual tasks. We distinguish the six broad Stage-1 SFT task groups discussed in Reviewer 2 from the finer-grained eight-category evaluation taxonomy below.

(b) Category-level analysis. We analyze 839 evaluation examples, assigning each one primary visual-task category and independently assessing the three grounding mechanisms for SFT and Hard DPO. We now provide the category-level breakdown in Appendix C.4 (Table C.1).

Table C.1. Grounding-failure counts by visual-task category (N=839). Each entry reports SFT → Hard DPO failure counts.

Visual-task categoryNUtilizationCalibrationConsistency
Unsupported attributes18230→2165→4432→22
Object counting16065→4525→1835→24
OCR hallucination11835→2544→3020→14
Chart hallucination11125→1843→3030→21
Document hallucination8825→1835→2418→13
Nonexistent object7912→940→2732→22
Spatial relationship7525→1818→1332→22
Ambiguous evidence266→415→105→4
Total839223→158285→197204→142

Here, N denotes the number of evaluation examples in each category. For example, among 182 unsupported-attribute examples, evidence-utilization failures decrease from 30 under SFT to 21 under Hard DPO (30→21). Each example is evaluated under both models, and mechanism labels may overlap. Relative reduction is calculated as (SFT failures − Hard-DPO failures)/SFT failures.

Hard DPO reduces evidence-utilization failures by 29.1% (223→158), evidence-calibration failures by 30.9% (285→197), and grounding-consistency failures by 30.4% (204→142). These reductions occur across multiple visual-task categories, providing mechanism-specific evidence beyond aggregate benchmark accuracy.

(c) Independent evaluation. On the same 839-example evaluation set, GPT-4o, Gemini-2.5-Flash, and LLaMA-4-Maverick report Hard-DPO versus SFT pairwise win-rate advantages of +8.2, +7.2, and +6.2 percentage points, respectively. These independent judgments complement the failure analysis by assessing overall grounding-sensitive response quality.

2. Contribution of hard-grounding preference construction (Weakness 2; Question 2)

(a) Distinctive contribution. Our contribution is not a new DPO objective but hard-grounding preference construction, where semantically close responses differ in fine-grained visual support, such as object existence, attributes, counts, OCR, spatial relations, or reasoning.

The construction is bidirectional: it discourages unsupported specificity when evidence is insufficient, preserves informative answers when evidence is sufficient, and penalizes visually inconsistent reasoning even when final answers agree. This avoids simply rewarding shorter or more conservative responses.

(b) Conventional-DPO comparison. To better isolate the contribution, we additionally compare conventional DPO without hard-boundary construction against Hard DPO, initialized from the same Stage-1 SFT checkpoint. Both use the same DPO objective, frozen visual pathway, and decoder-side LoRA adaptation.

ModelDocVQATextVQAMMBenchVQAv2
SFT81.580.578.081.5
Conventional DPO81.980.878.681.8
Hard DPO83.081.680.182.2

Hard DPO improves over conventional DPO by +1.1/+0.8/+1.5/+0.4 points, respectively. This provides additional evidence that grounding-focused preference construction benefits performance beyond standard DPO optimization.

(c) Controls and limitations. Our preference filtering verifies visual support and preference validity, while length-aware resampling and visually supported response expansion discourage trivial brevity-based preferences. Although the conventional-DPO comparison controls for the optimization objective and SFT initialization, fully isolating pair hardness requires matched preference-data scale and training budget. Easier-pair DPO and continued SFT on preferred responses would provide complementary controls.

Summary. The mechanism-specific reductions (29.1–30.9%), independent three-judge preferences (+6.2–8.2 pp), and improvements over conventional DPO jointly support the effectiveness of hard-grounding preference supervision across grounding behavior, response quality, and benchmark performance.

Reviewer_THDROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies fine-grained visual grounding MLLM and proposes the hard grounding preference supervision. The overall pipeline contains two stage: an decoder-side LoRA SFT and a DPO. Experiments show consistent DPO gains over SFT on four standard benchmarks and ~839 LLM-judge pairwise examples

Critical strengths
  1. The paper introduces evidence utilization, evidence calibration, and grounding consistency, and constructs hard preference pairs.
  2. Experimental results shows that the proposed DPO improves over SFT on various benchmarks.
  3. The released HardVQA-DPO resource can promote the studies in the correponding field.
Critical weaknesses
  1. The overall pipeline is a SFT followed by standard DPO with hard preference pairs. From a methodological perspective, the novelty of this study is limited.

  2. The evalution of this study highly relies on LLM judges, while human validation is very small which roughly contains 26 examples. Besides, the judge prompts and the extract judge model version are not provided in the paper.

  3. The paper introduces evidence utilization, evidence calibration, and grounding consistency as complementary grounding behaviors, but these three behaviors are more conceptual discussion rather than quantitatively evaluated.

  4. The description of the data used in this paper is very confusing. In the abstract, the authors states that HardVQA-DPO contains “more than 3K hard-grounding SFT examples and an initial set of DPO preference pairs,” but the experiments use approximately 50k–120k targeted preference examples (Appendix C.1). Therefore, it is really hard for me to get which dataset is used for training and evaluation.

  5. The implementation details of this paper is not clear enough. For example, it's unclear which dataset is used for fine-tuning in stage 1.

Questions for authors

Please refer to the weakness.

1 comment

We thank the reviewer for the constructive feedback and address the five concerns below.

1. Novelty and comparison with conventional DPO Our contribution is hard-grounding preference construction, not a new DPO objective. Preferred and rejected responses are semantically close but differ in visual evidential support, targeting subtle grounding boundaries rather than easily separable linguistic differences.

We extend Table 1 with conventional DPO initialized from the same Stage-1 SFT checkpoint:

ModelDocVQATextVQAMMBenchVQAv2
SFT81.580.578.081.5
Regular DPO81.980.878.681.8
Hard DPO83.081.680.182.2

Hard DPO improves over conventional DPO by +1.1/+0.8/+1.5/+0.4 points, respectively, supporting the benefit of grounding-focused preference construction beyond standard DPO. We also contribute HardVQA-DPO, a publicly released resource of curated hard-grounding SFT examples and preference pairs, to facilitate reproducibility and further research.

2. LLM judges, human validation, and reproducibility The manuscript identifies three independent evaluation judges: GPT-4o, Gemini-2.5-Flash, and LLaMA-4-Maverick (Section 3), and references generation/evaluation templates in the HardVQA-DPO repository (Section 2.2).

The evaluation prompt was previously made available in our anonymous repository and is now included in Appendix C.3 for easier inspection and reproducibility, together with the complete rubric (factual accuracy, relevance, hallucination, and usefulness; 0–5 each), pairwise decision protocol, and judge identifiers. Outcomes include SFT wins, Hard DPO wins, equivalent responses, and both wrong; execution/parsing errors are tracked separately. The rubric addresses unsupported claims and unnecessary rejection. This three-judge evaluation differs from the two-judge preference filtering during preference construction.

The 26 examples cited by the reviewer do not represent our full human validation. Beyond the ~3% manual audit of judge-agreement samples and disagreement cases during preference construction, 1,163 examples were assessed by three human annotators, comparing Stage-1 SFT and Hard-DPO responses, with judgments consolidated through consensus review. The new Table 3 summarizes the human evaluation:

Human evaluation outcomeCountPercentage
SFT preferred30025.8%
Hard DPO preferred42036.1%
Equivalent38032.7%
Both incorrect635.4%
Total1,163100%

Human judgments favor Hard DPO on 36.1% versus 25.8% of examples, yielding a +10.3 percentage-point net preference advantage. These results provide complementary human-assessed evidence of improved response quality and visual grounding. We also clarify the sampling and annotation aggregation protocols.

3. Quantitative evidence for the three grounding behaviors Section 2.2 defines three mechanisms: evidence utilization (using available subtle evidence), evidence calibration (avoiding unsupported specificity and unnecessary rejection), and grounding consistency (aligning claims and reasoning with visual evidence). Figure 4 illustrates these mechanisms. (a) Benchmark performance (Table 1). Hard DPO improves over SFT on DocVQA (+1.5), TextVQA (+1.1), MMBench (+2.1), and VQAv2 (+0.7), supporting broader multimodal correctness. (b) Human evaluation (new Table 3). Across 1,163 examples, three-annotator consensus judgments favor Hard DPO over SFT by +10.3 percentage points (36.1% vs. 25.8%), providing direct human-assessed evidence of improved response quality and grounding reliability. (c) Behavioral illustrations (Figure 4). Occlusion and crowded counting illustrate utilization; insufficient versus sufficient evidence illustrates calibration; visually inconsistent explanations illustrate consistency.

4. Dataset scale versus public release The 3K+ publicly released HardVQA-DPO examples are a curated subset, not the full training corpus. Stage-1 SFT uses ~65M examples, while Stage-2 Hard DPO uses ~110K preference pairs (Section 2.6 and Appendix C.1). The public release supports reproducibility without implying that it contains the complete training data.

5. Stage-1 SFT dataset composition and implementation Based on Stage-1 training-data records, the approximate distribution of ~65M multimodal examples across six task categories is:

Stage-1 SFT task categoryApprox. shareApproximate examples
Object recognition and visual attributes31.1%20.2M
Object counting and localization19.0%12.4M
OCR and text-rich image understanding14.1%9.1M
Chart and document understanding23.6%15.4M
Spatial relationships and visual reasoning8.9%5.8M
Ambiguous evidence and uncertainty handling3.3%2.1M
Total100%~65M

As detailed in Section 2.6, both stages freeze the ViT/Perceiver pathway and train decoder-side LoRA parameters. Stage 2 initializes from the SFT checkpoint, which also serves as the frozen DPO reference policy.

Reviewer_rNs1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether multimodal hallucination can be reduced by constructing preference pairs that are semantically similar but differ in visual grounding. The approach first performs multimodal SFT and then applies standard DPO to examples targeting evidence utilization, calibration, and grounding consistency. Both stages update decoder-side LoRA parameters while keeping the visual encoder and Perceiver alignment module frozen. On a LLaMA-3 70B-based multimodal model, the DPO checkpoint improves over SFT on four VQA benchmarks and receives higher pairwise win rates from three LLM judges on 839 examples. The paper also presents qualitative analyses and an initial HardVQA-DPO resource.

Critical strengths
  1. The supervision explicitly targets both unsupported claims and unnecessary rejection.

  2. The SFT and DPO checkpoints share a frozen visual pathway, making their comparison informative about what decoder-side adaptation can achieve. Table 1 reports improvements on all four benchmarks, and Table 2 shows the same direction of preference under three judges.

Critical weaknesses
  1. The experiments do not isolate the paper's main data-design contribution. The central comparison is SFT versus an additional DPO stage using approximately 50k–120k preference examples. There is no matched-data, matched-budget comparison against easier or conventionally constructed preference pairs, or continued SFT on the preferred answers.

  2. The operational definition of a valid grounding preference is not consistently supported by the examples or quantitatively audited. In Appendix A, example 19 (p. 21), the preferred response makes a definite claim about whether tofu was cut before fermentation, whereas the rejected response says that the manufacturing process cannot be inferred visually. The accompanying human-review note explicitly acknowledges that the preferred information is not revealed by the image and that the rejected answer is defensible from visual evidence alone. This conflicts with the stated treatment of insufficient evidence.

Questions for authors
  1. Contribution isolation: Can the authors provide an equal-size, equal-budget comparison between hard grounding pairs and conventional preference pairs from the same source examples? A continued-SFT control would also help distinguish preference construction from additional supervision.

  2. Grounding criterion: How should example 19 in Appendix A be labeled under the proposed evidence-calibration principle? What does a human audit find about the frequency of preferences that reward nonvisual knowledge over justified uncertainty, and about the resulting trade-off between unsupported claims and unnecessary rejection?

  3. Reproducibility clarification: What exact SFT/DPO dataset sizes, splits, deduplication rules, and training hyperparameters produced Tables 1–2? How do the released examples relate to the 50k–120k training pairs and the 839 evaluation examples?

1 comment

We thank the reviewer for the detailed and constructive feedback. We address the three concerns below.

  1. Contribution isolation: Hard DPO versus conventional DPO Our contribution is hard-grounding preference construction, not a new DPO objective. Unlike conventional preference pairs, our construction targets (i) fine-grained grounding boundaries, where semantically similar responses differ in counts, attributes, spatial relations, or unsupported claims, and (ii) bidirectional calibration, discouraging both hallucination and unnecessary rejection of visually answerable questions. Figures 5–7 illustrate improved grounded informativeness, reduced unsupported specificity, and challenging visual-grounding failures.

To isolate this contribution, we compare Hard DPO against conventional DPO initialized from the same Stage-1 SFT checkpoint. Both use the same frozen ViT/Perceiver visual pathway and decoder-side LoRA adaptation.

ModelDocVQATextVQAMMBenchVQAv2
SFT81.580.578.081.5
Conventional DPO81.980.878.681.8
Hard DPO83.081.680.182.2

Relative to conventional DPO, Hard DPO improves DocVQA, TextVQA, MMBench, and VQAv2 by +1.1/+0.8/+1.5/+0.4 points, respectively. This provides more direct evidence that grounding-focused preference construction offers benefits beyond applying the standard DPO objective alone.

On a separate 839-example grounding-sensitive evaluation, Hard DPO also achieves +6.2–8.2 percentage-point pairwise win-rate advantages over SFT across three independent LLM judges. This complements the benchmark comparison but is not a direct Hard-DPO-versus-conventional-DPO judgment.

We acknowledge that SFT→DPO alone does not isolate pair hardness. The conventional-DPO baseline provides a more direct comparison under the same optimization objective. However, fully attributing improvements to preference construction requires matching preference-data scale and training budget. Continued SFT on preferred responses would additionally test positive-only supervision and remains a useful complementary control.

  1. Grounding criterion and Example 19 Example 19 is not a typical correct-versus-hallucinated preference pair, but a challenging hard-negative case where both responses are defensible. The preferred response combines visible tofu pieces with domain knowledge about fermentation, offering a more informative, factually plausible answer. The rejected response is also valid under strict visual grounding, since the manufacturing process cannot be observed. This reflects a trade-off between factual informativeness and visual calibration, rather than a clear hallucination.

Our preference construction uses two independent LLM judges and targeted human review (~3% of judge-agreement samples plus disagreements). While these safeguards improve label quality, borderline cases remain. We will clarify the distinction between knowledge-informed inference and visually verified facts.

  1. Reproducibility: Data, splits, and hyperparameters We clarify the approximate training scales, data provenance, and relationship between training, validation, evaluation, and the public release.
ComponentConfiguration
BackboneLLaMA-3 70B + ViT + Perceiver
Trainable parametersDecoder-side LoRA; visual pathway frozen
Stage-1 SFT~65M multimodal examples
Stage-2 Hard DPO~110K preference pairs
DPO range in original paper50K–120K pairs across settings
Validation~5%
Pairwise evaluation839 examples
Public release3K+ curated SFT examples and initial preference pairs

Stage-2 preferences are constructed primarily from selected Stage-1 image–question examples through hard-question generation, preferred/rejected response construction, independent model adjudication, and targeted human review. The public release represents a curated subset rather than the complete training corpus.

Training configuration

ParameterSFTDPO
LoRA rank / alpha32 / 12832 / 128
LoRA dropout0.050.05
Learning rate1e-45e-6
Global batch size12864
Epochs11–2
OptimizerAdamWAdamW
DPO beta—0.1
Reference policy—Frozen SFT checkpoint

General capability evaluation uses DocVQA validation (5,349), TextVQA validation (5,000), MMBench dev (4,329), and VQAv2 validation (47,206). The separate 839-example grounding-sensitive evaluation uses GPT-4o, Gemini-2.5-Flash, and LLaMA-4-Maverick to assess pairwise response quality.

The approximately 5% validation allocation is distinct from the 839-example evaluation and the publicly released subset. We will specify split construction, image/question-level overlap controls, and checkpoint-specific partition counts to make evaluation independence and data provenance fully auditable. The expanded comparisons, dataset descriptions, and training configurations will be incorporated into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.