acceptodds
Under review as a conference paper at ICLR 2027

Vision-driven Preference Synthesis for Mitigating Hallucinations in VLMs

Abstract

Vision-Language Models (VLMs) have shown strong performance in visual understanding, yet they still suffer from hallucinations, generating content that is not grounded in the image. Preference alignment is a promising approach to improve visual faithfulness, but its success depends heavily on how preference pairs are constructed. Existing methods exhibit two key limitations; (a) intervention-based methods often introduce significant deviation from the policy distribution, and (b) sampling-based methods often underuse visual information during the construction. In this paper, we propose ViPSy (**Vi**sion-driven **P**reference **Sy**nthesis), a framework for constructing preference data that are both policy-aligned and visually grounded. Our framework consists of two stages; in the first stage, ViPSy derives a visual cue from recurring object-level content across semantically aligned image variants, so preference construction can rely on visual information rather than language priors. In the second stage, ViPSy conditions the policy’s own rollouts on this cue, allowing candidates to be guided by visually grounded content while staying close to the policy’s response distribution. The resulting candidates remain close to the policy’s response distribution while better leveraging visual information from the image. Experiments show that the resulting VLM, preference-aligned with ViPSy-constructed preference pairs, achieves a new state-of-the-art in hallucination mitigation. Compared with the previous state-of-the-art method, it reduces hallucination rates on AMBER and Object HalBench by 35.7% and 24.5%, respectively. The resulting model further improves on general visual grounding benchmarks, e.g., MMStar, MMVP, and CV-Bench, while also yielding gains in semantic segmentation and ImageNet linear probing, underscoring the effectiveness of our framework in enhancing the model’s visual capabilities.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-Language Models, Hallucination, Preference Alignment, Synthetic Data

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_khvBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ViPSy, a preference-data construction framework for mitigating hallucinations in vision-language models. The policy first captions an image, and a text-to-image model generates synthetic variants from these captions. A judge extracts cues shared between the original image and the variants. These cues guide policy rollouts, from which preferred and dispreferred responses are selected for standard DPO training. Experiments evaluate hallucination mitigation, general multimodal performance, and visual representations across multiple policy models.

Critical strengths
  1. The reported hallucination reductions are substantial and the paper provides extensive ablations and a practical deployment design
Critical weaknesses
  1. The motivation and advantages of semantic image synthesis in step 1 remain unclear. The synthesized images may inherit incorrect descriptions and introduce biases from image-to-text (T2I) conversion (this uncontrollable factor introduces additional noise information and high time costs), so its advantages over direct cue extraction are not yet sufficient. Although Table 6 reports gains, missing cue counts and coverage leave unclear whether these improvements stem from synthesis or stricter filtering that retains fewer cues. Given that image generation accounts for approximately 82% of offline construction cost, synthesis and direct extraction should be compared under similar filtering criteria and computational budgets.

  2. The methodological novelty beyond cue construction is limited. Step 2 largely follows established policy sampling, AI judging, and preference optimization, as in RLAIF-V. Hint-guided policy generation has also been explored in RPO, which is cited but not experimentally compared. The main advance therefore appears to lie in Step 1. The paper should clarify what Step 2 adds beyond these precedents and more convincingly establish the distinct value of its cue-construction procedure.

Questions for authors
  1. ViPSy requires cues to be supported by all eight pairwise comparisons, which may retain only a small set of salient objects. Could direct extraction achieve similar results by retaining only high-confidence objects that pass repeated verification? Although Table 6 includes repeated extraction with voting, it remains unclear whether both methods use the same retention criterion. The resulting cue counts and coverage should also be compared.

  2. A cue correctness of 95% does not by itself establish better cue extraction. For example, retaining only “person” and “car” may yield higher correctness than extracting a dozen objects, while providing less useful information. The paper does not establish whether image synthesis retains its advantage at comparable cue coverage.

  3. Image generation accounts for approximately 82% of the default offline preference-construction cost, based on Tables 21 and 22. A more informative comparison would allocate a comparable budget to direct object verification by a strong model, repeated sampling, or stricter filtering, and then compare downstream performance after preference training. The current experiments do not address this question.

  4. Contribution of Step 2. Beyond the source and content of the cues, what substantive methodological advance does Step 2 offer over existing AI-ranked policy sampling and hint-guided generation? Since RPO is cited, why is it omitted from the experimental comparisons?

  5. Fairness of baseline comparisons. Which Table 1 baselines were retrained under your setup? Were source images, training-set sizes, optimization budgets, and trainable components matched? In particular, did the baselines also update the vision encoder and projector? Please summarize any material differences.

  6. The claim of policy-aligned preference construction is insufficiently substantiated. Candidates are generated with cue conditioning (Eq. 5), whereas DPO excludes the cue (Eq. 7). Self-generation therefore does not by itself establish closeness to the original conditional distribution.

4 comments
AuthorsOpenReview ↗

Response to Reviewer khvB (1/2)

We sincerely appreciate your constructive feedback. We respond to each of your questions one-by-one in what follows. We have made every effort to address your concerns thoroughly. If any concerns remain, we would greatly appreciate hearing them and the opportunity to address them further.


W1/Q1-2. ViPSy with versus without semantic image synthesis: retention criterion, cue counts, and coverage

ViPSy is designed to reduce potential synthesis artifacts, including those inherited from incorrect captions and T2I biases, through pairwise comparison with the original image and consensus-based filtering. To further demonstrate the benefits of semantic image synthesis, we compare the cue quality and coverage of ViPSy with and without synthesis.

  • Same retention criterion: Both ViPSy and ViPSy w/o synthesis, which extracts cues from the original image alone, retain cues only when they appear in all eight comparisons or extractions (8/8), respectively. Under this identical criterion, ViPSy (SD-3.5-L) retains a similar number of cues (3.8 vs. 4.1 per image), with comparable recall (40.2% vs. 40.6%) and coverage (94% vs. 93%), while achieving higher cue correctness (95% vs. 85%). Thus, the improvement cannot be attributed to stricter filtering.
  • Cue quality and coverage: When ViPSy w/o synthesis reaches similar correctness (94% with 32 repetitions), it retains fewer cues (2.7 per image, recall 29.8%, coverage 88%) than ViPSy at 93-95% correctness (3.3-3.8 cues, recall 38.4-40.2%, coverage 92-94%). These results demonstrate that semantic image synthesis improves cue reliability while preserving more visual information.

We have added these results as Table 16, with the corresponding analysis in the paragraph "Cue quality and coverage" in Appendix J and a summary in "Cue-guided preference construction" in Section 3.4.

MethodRepetitionsCue Correctness (%) ↑Cue Recall (%) ↑Avg. Cues ↑Coverage (%) ↑
ViPSy w/o synthesis88540.64.193
ViPSy w/o synthesis169035.93.490
ViPSy w/o synthesis329429.82.788
ViPSy (SD-1.4)89338.43.392
ViPSy (SD-3.5-L)89540.23.894

W1/Q3. Comparison under a comparable computational budget

Under comparable computational budgets (16.9 vs. 16.7 GPU-hours), ViPSy (SD-1.4) achieves substantially better hallucination mitigation (4.17 vs. 5.52) than ViPSy w/o synthesis with 32 repetitions. For this comparison, we extend ViPSy w/o synthesis to 16 and 32 repetitions, retaining cues only when they appear in all 16 or 32 extractions, respectively.

ViPSy (SD-1.4) also outperforms the best setting without synthesis, 16 repetitions (4.17 vs. 5.16), and increasing repetitions does not close the gap (5.94 → 5.16 → 5.52 for 8, 16, and 32 repetitions). In addition, the lighter SD-1.4 reduces the offline cost of ViPSy from 49.4 to 16.9 GPU-hours, while the default ViPSy (SD-3.5-L) achieves the best Hal. Avg. (3.72). Extending to 16 and 32 repetitions also makes the unanimity rule stricter, so the number of repetitions and the filter strictness change together, and the weaker result at 32 repetitions is consistent with its lower cue recall (29.8%, Table 16). Within the unanimity settings we tested, repeated extraction without synthesis does not match ViPSy's hallucination mitigation under comparable computational budgets. We have added the corresponding computational cost analysis to the paragraph "Repeated extraction under a comparable budget" in Appendix J.

MethodRepetitionsGPU-hours ↓Hal. Avg. ↓
ViPSy w/o synthesis811.15.94
ViPSy w/o synthesis1613.05.16
ViPSy w/o synthesis3216.75.52
ViPSy (SD-1.4)816.94.17
ViPSy (SD-3.5-L)849.43.72
AuthorsOpenReview ↗

Response to Reviewer khvB (2/2)

W2/Q4. Methodological novelty of step 2 and comparison with prior methods

The contribution of ViPSy is the construction and utilization of reliable visual cues to generate visually grounded preference data. Below, we clarify what Step 2 adds over each of the two lines of prior work.

Sampling-based methods with a judge model (RLAIF-V): These methods keep preference data close to the policy distribution, but can still underuse visual information, since the policy's rollouts may be driven by language priors rather than image-specific evidence. In Step 2, the reliable visual cues from Step 1 address this underuse of visual information: they guide the target policy to generate diverse responses with varying degrees of visual grounding, enabling the construction of preference pairs that combine visual grounding with policy alignment. As illustrated in Figures 3 and 6, these cues help produce responses with distinguishable levels of visual grounding, making it easier for the judge model to identify preferred and dispreferred responses.

Hint-guided generation (RPO): Unlike RPO, which relies on surrogate ground-truth responses to identify errors and generate corrective hints, ViPSy derives visual cues directly from the input image through semantic synthesis and consensus-based filtering, without requiring reference responses. Moreover, RPO takes the hint-conditioned response as the preferred response and the response generated without the hint as the dispreferred one, whereas in ViPSy the judge model selects both responses from the same cue-conditioned candidates. We did not include RPO in our experimental comparisons because we could not find a publicly available official implementation. We have added this distinction to the paragraph "Mitigating hallucination in VLMs" in Related Work (Appendix B).


Q5. Fairness of baseline comparisons and training configurations

The baselines are described in Appendix E, and the results in Table 1 follow each baseline's original training setup rather than a unified retraining protocol, i.e., none of the Table 1 baselines was re-trained under our setup.

The table below summarizes every training-based Table 1 baseline (LLaVA-1.5-7B) with respect to the items raised by the reviewer. To clarify these differences in detail, we have added this table as Table 25 to the paragraph "Training setups of baselines" in Appendix O.

MethodSource imagesTraining-set sizeOptimization budgetVision encoderProjectorLLM
EFUFMSCOCO23K1 epoch✗✓✗
LLaVA-RLHFMSCOCO10K4 epochs✗✗✓
HALVAVisual Genome21.5K1 epoch✗✗✓
HA-DPOVisual Genome16K1 epoch✗✗✓
POVIDMSCOCO17K4 epochs✗✓✓
TPRMSCOCO, ShareGPT4V, VQAv221K1 epoch✗✓✓
RLAIF-VMSCOCO, ShareGPT4V, VQAv2, etc.16K4 epochs✗✓✓
OPA-DPORLAIF-V4.8K2 SFT + 4 DPO epochs✓✓✓
SENTINELVisual Genome8.6K1 epoch✗✗✓
P²-DPORLHF-V5.4K4 epochs✗✗✓
ViPSy (Ours)Visual Genome7K1 epoch✓✓✓

Q6. Policy alignment between cue-conditioned generation and cue-free DPO

We empirically examine distributional closeness between cue-conditioned generation and cue-free generation through KL divergence analysis. Specifically, πmethod\pi_{\mathrm{method}} is the original policy with the method-specific input (the cue for ViPSy and the context for SENTINEL), while πvanilla\pi_{\mathrm{vanilla}} is the same policy without it. Both receive identical images and prompts. We evaluate each method on its own preferred and dispreferred responses, since these are the responses used for DPO. As shown in Table 17, ViPSy achieves substantially lower KL divergence than SENTINEL (0.018 vs. 0.074), providing empirical evidence that our preference construction results in smaller deviations from the policy distribution.

Thank you for the detailed reply and the additional experiments. My two main concerns remain.

  1. The practical value of image synthesis is still unclear given the high time cost.

    • The new results suggest that cues with higher accuracy and better coverage are associated with better downstream hallucination mitigation. Cue quality is therefore central to the paper as a whole. The repeated extraction baseline is too limited to establish what image synthesis adds. Which VLM was used for repeated extraction in W1/Q1–2?
    • A more direct comparison would use the same VLM to list candidate objects from the original image and a separate judge model to verify each object against that image. Alternatively, the authors could use a stronger VLM to extract key cues directly and test whether it achieves similar hallucination reduction without the time cost of image generation.
  2. Stage 2 offers limited novelty. Antidote [1] and SENTINEL [2] already use visually grounded context to construct preference pairs, with Antidote also removing the object hints during DPO. I still do not see a substantial advance over these earlier approaches.

Overall, I think the core of the paper lies in how Stage 1 constructs visual cues. The current evidence is still insufficient to show that this approach is necessary and practical.

[1] Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perception

[2] Mitigating Object Hallucinations via Sentence-Level Early Intervention

Dear Reviewer khvB,

Thank you for your follow-up and for raising these points before the end of the discussion period. We appreciate the opportunity to address them more thoroughly.

RQ1. Is image synthesis worth the cost?; image synthesis vs. direct cue extraction

We first clarify that the repeated-extraction baseline in W1/Q1–2 uses Qwen2.5-VL-72B, the same extractor as ViPSy, with the same consensus-based filtering and aggregation. Following your suggestions, we further evaluated stronger extraction and verification baselines without image synthesis. These comparisons show that ViPSy consistently achieves better downstream hallucination mitigation, while its SD-1.4 variant retains this advantage at a comparable computational cost.

Specifically, we evaluated (i) propose-then-verify, where Qwen2.5-VL-72B lists candidate objects and Qwen3-VL-235B-A22B verifies and merges them against the original image in place of our filtering, and (ii) direct extraction using Qwen3-VL-235B-A22B with our filtering. We tested each with 8, 16, and 32 repetitions. For direct extraction, a cue is retained only if it appears in every repetition, following the same unanimity criterion used in ViPSy.

MethodVLMRep.Correct. (%) ↑Recall (%) ↑Avg. Cues ↑Coverage (%) ↑GPU-hours ↓Hal. Avg. ↓
Propose-then-verifyQwen2.5-VL-72B + Qwen3-VL-235B88439.83.99213.55.87
Propose-then-verifyQwen2.5-VL-72B + Qwen3-VL-235B168740.24.29015.95.34
Propose-then-verifyQwen2.5-VL-72B + Qwen3-VL-235B328938.84.38920.55.21
Direct extractionQwen3-VL-235B88740.84.39212.45.84
Direct extractionQwen3-VL-235B169236.83.49014.95.33
Direct extractionQwen3-VL-235B329532.43.08918.45.12
ViPSy (SD-1.4)Qwen2.5-VL-72B89338.43.39216.94.17
ViPSy (SD-3.5-L)Qwen2.5-VL-72B89540.23.89449.43.72

ViPSy with SD-3.5-L achieves both 95% correctness and 40.2% recall. The stronger direct extractor reaches 95% correctness only at 32 repetitions, with recall falling to 32.4%, while propose-then-verify remains at 84-89% correctness. The benefit also holds at a competitive cost: ViPSy with SD-1.4 takes 16.9 GPU-hours, within the baseline range (12.4-20.5), and achieves a lower Hal. Avg. than every baseline (4.17 vs. 5.12-5.87).

The key difference is that ViPSy takes consensus across semantically aligned image variants rather than repeated outputs from the same image, allowing it to filter errors that may persist across repeated extraction.


RQ2. Differences from Antidote and SENTINEL

ViPSy’s Stage 2 differs from Antidote and SENTINEL in how responses are evaluated and selected: it uses an external judge to evaluate each full response against the image, rather than relying on cue conditioning or object presence alone.

Antidote takes the hint-conditioned response as preferred and the hint-free response as dispreferred. Its hints are checked with Grounding-DINO, but the responses themselves are not verified against the image. SENTINEL uses open-vocabulary detectors to check object presence, with context drawn from the policy’s preceding sentences. ViPSy instead evaluates full responses for object, relational, attribute, and spatial accuracy (Table 18).

This distinction matters because accurate cues alone do not guarantee grounded responses: even with 95% cue correctness (Table 16), preferred and dispreferred responses generated with the same cue differ substantially in visual grounding (0.863 vs. 0.441, Table 18). These results highlight why cue quality alone is insufficient for selecting preferred responses.


Thank you again for your thoughtful feedback. We hope these additional comparisons and clarifications help further address your concerns.

Best regards,
Authors

Reviewer_3yX1OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes ViPSy, a method for constructing preference data that is visually grounded while remaining close to the policy distribution. First, the policy model generates a self-description of an image, which is used as a text-to-image prompt to create semantically related images. A vision evaluator compares the original and synthetic images, and recurring object-level cues are aggregated. The policy then generates candidate descriptions conditioned on these cues, and the evaluator selects preferred responses based on the original image. Finally, the model is aligned using DPO. The synthetic images and cues are not needed at inference time.

Critical strengths
  1. The preference-construction pipeline has a clear motivation. Synthetic visual variations are used to identify shared cues, while candidate responses are generated by the policy itself to reduce distribution mismatch.
  2. The evaluation covers multiple models and dimensions. The paper reports results for both 7B and 13B settings, as well as averages across several VLMs, using hallucination and general vision benchmarks.
  3. The paper includes ablations and robustness analyses. It studies consensus thresholds, synthesis methods, rollout counts, cue extractors, preference evaluators, and repeated inference at temperature 0.7.
  4. The paper attempts to rule out the explanation that gains come only from shorter responses. ViPSy’s average response length is below vanilla but above SENTINEL, while object mention and recall improve over both.
  5. Some data-construction costs and alternatives are reported. The default configuration constructs 7,000 preference pairs at a reported cost of 49.4 GPU-hours, and the paper evaluates lighter alternatives.
Critical weaknesses
  1. The central supervision pipeline depends on a strong model evaluator. The default cue extraction and preference selection use Qwen2.5-VL-72B. High agreement between models does not establish preference correctness and is not a substitute for independent human auditing.
  2. The scope of cue-quality validation is unclear. The appendix reports 95% cue accuracy under a particular setting, but does not sufficiently specify sample size, category coverage, disagreement handling, or error types.
  3. The evidence for improved visual representations is limited. The strongest results concern object hallucination benchmarks. ImageNet probing and COCO segmentation gains are small, and no multi-seed variance is reported.
  4. The evidence for policy-distribution alignment is indirect. Token-level KL analyses do not directly establish reduced distribution shift across broader prompts after training.
Questions for authors
  1. Please analyze how added, missing, or incorrect attributes in synthetic images affect the aggregated cues, to ensure generative-model artifacts are not mistaken for visual evidence.
  2. Please narrow claims about improved visual representations. The current evidence more directly supports the effectiveness of a preference-construction pipeline on selected hallucination evaluations.
  3. Please identify the exact comparison baseline and metric for each reported relative reduction, and avoid treating distinct hallucination metrics as a single quantity.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 3yX1 (1/2)

We sincerely appreciate your constructive feedback. We respond to each of your questions one-by-one in what follows. We have made every effort to address your concerns thoroughly. If any concerns remain, we would greatly appreciate hearing them and the opportunity to address them further.


W1. Reliability of model-based preference evaluation

As reported in Figure 4, alternative judges (GPT-5 mini and Qwen3-VL-30B) achieve high agreement (95.8-98.4%) with the default Qwen2.5-VL-72B, demonstrating consistency across different evaluators.

To further validate preference correctness beyond inter-model agreement, we conducted an additional human evaluation. Specifically, we randomly sampled 30 preference pairs, and 30 participants who were not involved in this work each judged all 30 pairs (900 judgments in total). For each pair, participants saw the original image and the two responses in randomized order, without knowing which response ViPSy had selected, and chose the response that better matched the image; no tie option was provided. Agreement is the fraction of individual judgments that match ViPSy's selection: 869 of 900 (96.6%). We have added these results as Table 27 and the details, including the evaluation interface (Figure 14), to Appendix S ("Human Evaluation of Preference Selection").


W2/Q1. Cue quality validation and robustness to synthetic image artifacts

We clarify the cue-quality validation and the effect of synthetic image artifacts below.

  • Sample size: Cue correctness is evaluated on 7,000 images with 62,057 Visual Genome object labels, using GPT-5 mini for semantic matching.
  • Coverage: GPT-5 mini groups semantically related cues into categories (e.g., "man" and "woman" into "people"). Coverage is the percentage of distinct categories in the aggregated cues relative to those in cues extracted directly from the original images.
  • Disagreement handling: Cues not supported by all eight comparisons are discarded. A cue that GPT-5 mini cannot match to any Visual Genome annotation is counted as incorrect.
  • Error types: We report three error types.
    • Added: cues that GPT-5 mini judges to be absent from the original image, divided by the total number of cues. An object that appears only in a synthetic image is not shared with the original image, so it is unlikely to pass the 8/8 rule.
    • Missing: 100 − Coverage. An object omitted in any one synthetic image fails the 8/8 rule, which removes a cue but does not create an incorrect one.
    • Incorrect: 100 − Cue Correctness. A wrong attribute (e.g., color) does not change an object-level cue. If it changes the identity of the object, the object is no longer shared with the original image and is typically filtered.

As the threshold rises from one to eight, the added error falls from 15.80% to 1.30% and the incorrect error from 22% to 5%, while the missing error increases only from 4% to 6%. Thus, stricter consensus removes most unsupported cues with only a marginal reduction in coverage. We have added these details to the paragraphs "Evaluation of cue quality" (Appendix J) and "Consensus ablation for cue aggregation" (Appendix I).

ConfigurationCue Correctness (%) ↑Coverage (%) ↑Added Error (%) ↓Missing Error (%) ↓Incorrect Error (%) ↓
Agree = 1789615.80422
Agree = 286969.62414
Agree = 490945.54610
Agree = 8 (ViPSy)95941.3065
AuthorsOpenReview ↗

Response to Reviewer 3yX1 (2/2)

W3/Q2. Visual representation improvements and scope of claims

ViPSy is not designed to explicitly improve visual representations, but rather to mitigate hallucinations through visual cue-guided preference data construction. Nevertheless, we observe consistent improvements across downstream visual tasks, which is an encouraging and unexpected finding. To verify their reliability, we conducted evaluations with five random seeds across all linear probing and segmentation benchmarks. The table below reports the mean ± standard deviation over the five seeds. ViPSy outperforms Vanilla in all evaluation settings, with lower standard deviations in every case. These results suggest that the improvements are not merely due to random seed variation and that preference-based supervision may provide additional benefits to visual representations. We have updated Table 4 and the paragraph "Effect on vision encoder" in Section 3.3 accordingly.

Method2-shot IN-1k ↑2-shot IN-O ↑2-shot R-SST2 ↑4-shot IN-1k ↑4-shot IN-O ↑4-shot R-SST2 ↑8-shot IN-1k ↑8-shot IN-O ↑8-shot R-SST2 ↑Seg. MS COCO Recall ↑
Vanilla35.5 ± 0.4917.9 ± 0.9849.1 ± 0.6548.7 ± 0.2529.4 ± 0.6850.4 ± 1.3059.3 ± 0.2337.9 ± 0.3950.2 ± 0.9934.9 ± 0.79
ViPSy (Ours)36.0 ± 0.4218.6 ± 0.7649.7 ± 0.5249.0 ± 0.2230.1 ± 0.5451.1 ± 0.5860.0 ± 0.2038.5 ± 0.2650.9 ± 0.4136.0 ± 0.57

W4. Policy-distribution alignment across broader prompts after training

ViPSy primarily aims to construct preference data that remain close to the original policy's response distribution, rather than explicitly minimizing distribution shift after DPO training. As reported in Appendix L (Table 17), ViPSy achieves a lower token-level KL divergence than SENTINEL (0.018 vs. 0.074), supporting its effectiveness in maintaining distributional proximity during preference construction.

Nevertheless, following the reviewer's suggestion, we additionally evaluated distributional shift after DPO training using 300 images from Object HalBench and their corresponding COCO reference captions. To cover more diverse prompts beyond image captioning, we also evaluated the 96 image-question pairs of MMHal-Bench, which span eight question types (object attribute, adversarial object, comparison, counting, spatial relation, environment, holistic description, and others). For each trained policy πaligned\pi_{\mathrm{aligned}}, we compute the token-level KL(πaligned∥πvanilla)\mathrm{KL}(\pi_{\mathrm{aligned}}\Vert\pi_{\mathrm{vanilla}}) under identical images and prompts, without any cue or context, with both policies teacher-forced on the same reference text (the COCO reference captions and the MMHal-Bench reference answers).

As shown below, the model trained on ViPSy-constructed preference data exhibits substantially lower KL divergence than the model trained on SENTINEL-constructed preference data on both benchmarks, suggesting better preservation of the original policy's response distribution after training. We have added the results to Table 17, and the details to the paragraph "Distributional shift after preference alignment" in Appendix L.

BenchmarkSENTINEL KL ↓ViPSy KL ↓
Object HalBench0.3840.123
MMHal-Bench0.1860.089

Q3. Clarification of baselines and relative reduction metrics

The reported relative reductions of 35.7% and 24.5% are calculated against SENTINEL using two distinct hallucination metrics: AMBER CHAIR (2.8 → 1.8) and Object HalBench Resp. (5.3 → 4.0), respectively, as reported in Section 3.2 (Table 1). Further details on the compared baselines are provided in Appendix E, while benchmark descriptions and metric definitions are presented in Appendix F.

For clarity, we have revised the Abstract to explicitly specify SENTINEL as the comparison baseline and identify the corresponding metric for each reduction. While the full results for Tables 2, 3, and 5 were already provided in Tables 7, 8, and 11, respectively, we have additionally added Appendix T ("Full Results for Averaged Scores"), which defines Hal. Avg. and Gen. Avg. and reports the individual metrics for the remaining averaged scores in Tables 28-34.

Dear Reviewer 3yX1,

Thank you again for your time and effort in reviewing our manuscript.

As the discussion period is coming to an end, we would like to kindly ask if there are any remaining points that you would like us to clarify.

We believe we have made a sincere effort to address your earlier comments, and we would be grateful for any additional feedback you may have. Your insights are invaluable in helping us further strengthen the manuscript.

Sincerely,
Authors

Reviewer_rrebOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors argue that existing hallucination-mitigation methods can introduce preference data that deviate from the target policy’s distribution or underuse visual information during preference construction. To address these limitations, the paper proposes ViPSy, a framework for constructing preference data to reduce hallucinations in VLMs. ViPSy first generates synthetic images from the target policy’s self-generated captions. A judge compares these images with the original image and aggregates shared object-level content into visual cues. The target policy then generates candidate responses conditioned on these cues, and the judge selects preferred and dispreferred responses. The resulting pairs are used for DPO without providing the cues during training or inference. Experiments on hallucination and general vision-language benchmarks demonstrate the effectiveness of the overall framework.

Critical strengths
  1. The paper is clearly written, and the overall data-construction pipeline is coherent and easy to follow.
  2. The results in Table 1 indicate substantial hallucination reductions across several benchmarks. Additional experiments extend the evidence beyond LLaVA to other model families, including Qwen2.5-VL. The paper also provides extensive component ablations, which help assess the usefulness of different design choices within the proposed framework.
Critical weaknesses
  1. The claimed alignment with the target policy’s distribution is insufficiently justified.
  • The authors emphasize policy-aligned preference construction, but ViPSy generates candidate responses from πθ(y∣x,m,c)\pi_\theta(y\mid x,m,c), whereas DPO and inference condition the model only on πtheta(y∣x,m)\pi_theta(y\mid x,m). Even when a cue is derived entirely from the original image, explicitly providing it can alter the model’s response distribution. Sampling from the same model under an augmented conditioning context therefore does not, by itself, establish alignment with the cue-free target policy.
  • Although Table 16 reports a KL comparison, the definition of the method distribution and the conditioning and prefix distribution used in the estimation are insufficiently specified. The procedure may still be effective as a way to construct training data, but the current evidence does not adequately substantiate the policy-alignment claim central to its motivation.
  1. The paper omits closely related published work and does not establish its incremental contribution over reconstruction-based approaches.
  • DeGF [1] reconstructs an image from a model response and uses the generated visual reference to mitigate hallucinations during decoding. RICO and RICO-Flash [2] use feedback from comparisons between original and reconstructed images to refine captions, and subsequently train a model with DPO on initial-versus-refined caption pairs. Cycle Consistency as Reward [3] constructs reconstruction-based preferences, introduces CycleReward and CyclePrefDB, and also applies these preferences to VLM DPO.
  • These omissions are consequential because reconstruction-derived visual feedback and reconstruction-based preference learning are already established ideas. The paper does not sufficiently position its motivation and novelty relative to these methods。

[1] Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models. ICLR 2025.

[2] RICO: Improving Accuracy and Completeness in Image Recaptioning via Visual Reconstruction. EMNLP 2025.

[3] Cycle Consistency as Reward: Learning Image-Text Alignment without Human Preferences. ICCV 2025.

Questions for authors

See weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer rreb

We sincerely appreciate your constructive feedback. We respond to each of your questions one-by-one in what follows. We have made every effort to address your concerns thoroughly. If any concerns remain, we would greatly appreciate hearing them and the opportunity to address them further.


W1. Clarification of policy alignment

Inspired by self-distillation, ViPSy leverages model-generated responses while aiming to minimize distributional shifts. Importantly, additional conditioning does not necessarily imply a substantial distributional shift from the cue-free policy. As shown in Table 17, ViPSy exhibits lower KL divergence than SENTINEL (0.018 vs. 0.074), providing empirical evidence of smaller distributional deviations. To clarify this point, we provide further details on the KL analysis as follows:

  • Policies: πmethod=πθ(⋅∣x,m,c,y<t)\pi_{\mathrm{method}} = \pi_\theta(\cdot \mid \mathbf x, \mathbf m, \mathbf c, \mathbf y_{\lt t}) and πvanilla=πθ(⋅∣x,m,y<t)\pi_{\mathrm{vanilla}} = \pi_\theta(\cdot \mid \mathbf x, \mathbf m, \mathbf y_{\lt t}), where πθ\pi_\theta is the original LLaVA policy, x\mathbf x the prompt, m\mathbf m the image, y<t\mathbf y_{\lt t} the response prefix, and c\mathbf c the method-specific input (visual cue for ViPSy, context for SENTINEL).
  • KL estimator: Token-level KL(πmethod∥πvanilla)\mathrm{KL}(\pi_{\mathrm{method}}\Vert\pi_{\mathrm{vanilla}}) over the full vocabulary, averaged over all tokens of y\mathbf y.
  • Conditioning and prefix distribution: For the same 2,000 images and prompts, we use each method's own preferred and dispreferred responses as y\mathbf y. These are exactly the responses that DPO evaluates without c\mathbf c, so the KL on them directly measures the mismatch raised by the reviewer.

We have added these details to the paragraph "Quantifying the distributional shift" in Appendix L.


W2. Distinction from prior reconstruction-based methods

Our comparisons focused on DPO-based methods specifically designed for VLM hallucination mitigation. In contrast, DeGF [1] focuses on inference-time decoding, RICO [2] on enriching image captions with fine-grained details, and CycleReward [3] on image-text alignment for detailed captioning and text-to-image generation, making them less directly comparable to our approach.

The difference lies in where reconstruction enters the pipeline. These methods use reconstruction to correct, refine, or rank responses that have already been generated. ViPSy uses synthesized images only to extract visual cues before candidate generation, retaining a cue only if it is shared with the original image in all eight comparisons so that artifacts of a single reconstruction are filtered out. Thus, our novelty lies not in reconstruction itself, but in using consensus over multiple reconstructions to obtain reliable visual cues for policy-aligned preference construction.

Following the reviewer's suggestion, we additionally evaluate DeGF [1], RICO and RICO-Flash [2], and CycleReward [3]. All methods use LLaVA-1.5-7B. DeGF and RICO are applied to the vanilla model at inference time, RICO-Flash is the DPO-trained model, and CycleReward denotes the model fine-tuned with DPO on CyclePrefDB. Both are trained following their original setups. As shown below, ViPSy achieves stronger hallucination mitigation and general benchmark performance. We have added these results to Appendix P ("Comparison with Reconstruction-based Methods"), and included these studies in the paragraph "Reconstruction-based visual feedback" in Related Work (Appendix B).

MethodObject HalBench Resp. ↓Object HalBench Ment. ↓AMBER CHAIR ↓AMBER Hal. ↓MMStar Acc. ↑MMVP Acc. ↑CV-Bench Acc. ↑
Vanilla51.325.97.936.833.423.361.9
DeGF35.715.56.323.832.125.356.6
RICO-Flash28.714.45.818.432.825.362.3
RICO24.013.25.017.233.726.762.1
CycleReward21.710.74.713.633.328.763.6
ViPSy (Ours)4.02.61.89.635.430.664.4

[1] Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models. ICLR 2025.

[2] RICO: Improving Accuracy and Completeness in Image Recaptioning via Visual Reconstruction. EMNLP 2025.

[3] Cycle Consistency as Reward: Learning Image-Text Alignment without Human Preferences. ICCV 2025.

Thank you for the clarification and additional experiments. The revised related-work discussion and direct comparisons address my concern about the paper’s contribution relative to prior reconstruction-based methods. The KL analysis is also much clearer. I would encourage the authors to qualify “policy-aligned” accordingly, since the analysis measures next-token distributions on evaluated prefixes and the final preference pairs also depend on judge selection.

Given these revisions and the empirical results, I am raising my recommendation to weak accept.

Dear Reviewer rreb,

Thank you very much for your follow-up and support for our work. We are glad that our response helped address your concerns. We will further refine the use of “policy-aligned” in the manuscript to better reflect the scope of our analysis and the role of judge selection. We sincerely appreciate the time and care you have put into reviewing our work.

Best regards,
Authors

Reviewer_zGvDOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ViPSy uses synthetic image comparisons to extract visual cues, then builds DPO preference pairs from cue-conditioned self-generated responses. It reduces hallucination across several VLM families without adding inference-time steps.

Critical strengths

1.The hallucination gains are substantial and do not appear to come mainly from saying less. On LLaVA-7B, AMBER Hal. falls from SENTINEL’s 13.6 to 9.6 (Table 1). ViPSy also retains object recall close to the base model, outperforming SENTINEL and OPA-DPO on this measure (Table 18). This makes the improvement more convincing. 2.The ablations address several obvious alternative explanations. Direct cue extraction, repeated voting, external rollouts, and different judges are all examined (Tables 6, 8, and 15). In particular, synthesis outperforms repeated extraction from the original image, supporting its inclusion in the pipeline.

Critical weaknesses

1.I am less convinced by the “policy-aligned” explanation than by the benchmark results. Candidates are generated with additional cues in Eq. (5), but DPO evaluates them without those cues in Eq. (7). Sharing model weights does not establish distributional closeness. Table 16 is relevant, but Appendix K leaves πmethod\pi_{\text{method}}, the KL estimator, and the evaluated token prefixes unclear. 2.The key consensus operation is not specified precisely enough to reproduce. Section 2.1 requires agreement across all eight comparisons, whereas Figure 11 describes a general merging procedure without an explicit threshold. Since Table 13 shows that this threshold materially affects performance, how agreement is counted and synonyms are matched is part of the method, not a minor implementation detail.

Questions for authors

1.How is πmethod\pi_{\text{method}} defined in Table 16, and how is its KL estimated? Does the reported advantage persist when both methods are evaluated on the same images and token prefixes, with preferred and rejected responses reported separately? 2.What exactly enforces the eight-of-eight consensus rule? Please provide the aggregation procedure, including synonym matching and handling of cases where no cue survives, and clarify whether Figure 11 is the complete prompt used for Table 13. 3.Could you provide a controlled comparison to determine whether ViPSy's advantages persist? Reporting hallucination metrics together with object recall would help assess the contribution of the proposed preference construction under comparable training conditions.

4 comments
AuthorsOpenReview ↗

Response to Reviewer zGvD (1/2)

We sincerely appreciate your constructive feedback. We respond to each of your questions one-by-one in what follows. We have made every effort to address your concerns thoroughly. If any concerns remain, we would greatly appreciate hearing them and the opportunity to address them further.


W1/Q1. Clarification of policy alignment and matched-prefix KL evaluation

We first clarify the KL evaluation setup as follows:

  • Policies: πmethod=πθ(⋅∣x,m,c,y<t)\pi_{\mathrm{method}} = \pi_\theta(\cdot \mid \mathbf x, \mathbf m, \mathbf c, \mathbf y_{\lt t}) and πvanilla=πθ(⋅∣x,m,y<t)\pi_{\mathrm{vanilla}} = \pi_\theta(\cdot \mid \mathbf x, \mathbf m, \mathbf y_{\lt t}), where πθ\pi_\theta is the original LLaVA policy, x\mathbf x the prompt, m\mathbf m the image, y<t\mathbf y_{\lt t} the response prefix, and c\mathbf c the method-specific input (visual cue for ViPSy, context for SENTINEL).
  • KL estimator: Token-level KL(πmethod∥πvanilla)\mathrm{KL}(\pi_{\mathrm{method}}\Vert\pi_{\mathrm{vanilla}}) over the full vocabulary, computed at response-token positions and averaged over all tokens of y\mathbf y.
  • Evaluated responses y\mathbf y: Both methods use the same 2,000 images and prompts, but each method's own preferred and dispreferred responses as the prefix, since these are the responses DPO is trained on.

Under this setting, we additionally report KL divergence separately for preferred and dispreferred responses. As shown below, ViPSy exhibits consistently lower KL divergence than SENTINEL for both response types. This indicates that ViPSy's distributional closeness to the cue-free policy holds across both sides of the preference pairs used for DPO.

Method-specific responsesPreferred KL ↓Dispreferred KL ↓
SENTINEL0.0790.070
ViPSy0.0190.017

Because the prefixes above differ between methods, we repeat the measurement with identical prefixes: responses sampled from the cue-free vanilla policy (eight per image, as in the w/o cue ablation of Table 5), with preferred and dispreferred responses selected by the same judge. Even under this setting, ViPSy maintains substantially lower KL divergence than SENTINEL (0.018 vs. 0.158 on average), strengthening the evidence that ViPSy's additional inputs induce a smaller distributional shift from the cue-free policy.

Shared vanilla responsesPreferred KL ↓Dispreferred KL ↓
SENTINEL0.1910.124
ViPSy0.0200.016

We have added these results to Table 17 and clarified the evaluation protocol in Appendix L ("Quantifying the distributional shift"). Together, these results demonstrate that ViPSy maintains lower distributional divergence from the cue-free policy both on its own DPO training responses and under identical response-token prefixes, providing empirical support for our policy-alignment motivation.

AuthorsOpenReview ↗

Response to Reviewer zGvD (2/2)

W2/Q2. Eight-of-eight consensus rule and cue aggregation procedure

We clarify how the eight-of-eight consensus rule is implemented through code-based filtering and how the retained cues are subsequently aggregated. After extracting visual cues from eight pairwise comparisons, we apply code-based filtering in two steps.

First, we group similar cues within each category. Two cues are treated as similar if, after lowercasing, (i) the strings are equal, (ii) the token set of one is contained in that of the other and the shared tokens cover at least 0.8 of the larger set, (iii) the Jaccard similarity of the token sets is at least 0.8, or (iv) the SequenceMatcher ratio is at least 0.9.

Second, we keep a cue only if it, or a cue similar to it, appears in all eight comparisons. A cue missing from even one comparison is discarded, and samples with no surviving cue are excluded from preference data construction. These criteria match surface variants of the same cue. Synonyms are better captured by a language model than by string rules, so they are merged in the subsequent aggregation prompt.

The retained cues are then merged by the judge model using the aggregation prompt in Figure 11, as described in Eq. (4), which also merges the remaining synonyms among the retained cues. Figure 11 shows the complete LLM aggregation prompt, and the experiments in Table 13 vary only the code-based consensus threshold, without modifying this prompt. We have added these implementation details to the paragraph "Consensus filtering for cue aggregation" in Appendix I and further clarified the procedure in Section 2.1.


Q3. Controlled comparison under matched training conditions

Following the reviewer's suggestion, we re-trained SENTINEL and OPA-DPO under matched conditions, which extends the same-judge comparison in Table 15. All fine-tuned methods use LLaVA-1.5-7B with the same Qwen2.5-VL-72B judge, identical LoRA configurations (r=128, α=256, applied to the vision encoder, projector, and LLM), the same number of preference pairs (6,999), and the same training budget (batch size 64, 110 updates, 1 epoch). For OPA-DPO, the SFT stage that precedes DPO is kept as in its original setting. Each method keeps its own preference construction pipeline and source images, since these are part of the method itself.

As shown in the table below, ViPSy consistently outperforms both SENTINEL and OPA-DPO under both their original and matched training conditions, achieving the lowest hallucination rates while maintaining the highest object recall. These results demonstrate that ViPSy's advantages persist under controlled training conditions, supporting the effectiveness of our preference construction strategy. We have added these results as Table 24 and the details to the paragraph "Controlled comparison" in Appendix O.

MethodObject HalBench Resp. ↓Object HalBench Ment. ↓Object HalBench Recall ↑AMBER CHAIR ↓HallusionBench aAcc ↑
SENTINEL5.33.050.62.847.6
SENTINEL (matched)7.05.051.33.647.0
OPA-DPO8.65.149.32.747.9
OPA-DPO (matched)6.05.850.93.546.2
ViPSy4.02.654.71.848.1

Dear Reviewer zGvD,

Thank you again for your time and effort in reviewing our manuscript.

As the discussion period is coming to an end, we would like to kindly ask if there are any remaining points that you would like us to clarify.

We believe we have made a sincere effort to address your earlier comments, and we would be grateful for any additional feedback you may have. Your insights are invaluable in helping us further strengthen the manuscript.

Sincerely,
Authors

Thank you for the reviewer's clarification; I am maintaining my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revision

Dear Reviewers,

We sincerely appreciate your time and effort in reviewing our paper. Your constructive comments have helped us improve the manuscript. We have reflected them in our revised draft, where the additional results and revisions are marked in blue. In particular, the revised draft includes the following changes:

  • A clarification of the KL analysis (zGvD W1/Q1; rreb W1; khvB Q6).
  • A KL evaluation after DPO training (3yX1 W4).
  • Implementation details of consensus filtering and cue aggregation (zGvD W2/Q2).
  • A controlled comparison under matched training conditions, with the training setups of all baselines (zGvD Q3; khvB Q5).
  • A comparison with reconstruction-based methods (rreb W2).
  • A human evaluation of preference selection (3yX1 W1).
  • An extended evaluation of cue quality (3yX1 W2/Q1).
  • A vision encoder evaluation over five random seeds (3yX1 W3/Q2).
  • Comparisons of ViPSy with and without semantic image synthesis (khvB W1/Q1-3).
  • A discussion of the distinction from RPO (khvB W2/Q4).
  • A revised Abstract, with full results for all averaged scores (3yX1 Q3).

We hope these revisions address your concerns and make our manuscript clearer.

Sincerely,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.