acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Hallucination in Visual Language Models with Partial Transport KL-constrained Decoding

Abstract

Vision-language models frequently describe objects that are not present in the image, limiting their use in settings where faithfulness matters. Existing inference-time mitigation methods typically rely on contrastive decoding, attention manipulation, or representation steering, but suppressing hallucinations can also remove valid visual content. We introduce POTION (Partial Optimal Transport Inference with KL ProjectiON), a training-free inference-time method that steers autoregressive VLM decoding toward visually grounded outputs. At each generation step, POTION solves a null-penalized partial optimal transport problem between top- candidate-token embeddings and cached image-patch hidden states. The partial formulation automatically routes visual support mass to grounded candidates while assigning unsupported candidates to a null sink, yielding per-candidate evidence and risk scores that produce a logit correction. This correction is then projected onto a KL-ball centered at the base model's distribution, preventing aggressive hallucination suppression from collapsing object coverage. Experiments on MSCOCO and AMBER show consistent reductions in object hallucination across Qwen2.5-VL and Llama-3.2-Vision. POTION achieves state-of-the-art performance in mitigating object hallucinations, substantially reducing CHAIR while maintaining comparable object recall.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Hallucination in Vision Language Model, Object Hallucination, Optimal Transport Application

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uPXLOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes POTION, a training-free method for mitigating object hallucination in vision-language models. POTION uses null-penalized partial optimal transport to assess visual support of token candidates and adjust their logits accordingly. Additionally, POTION uses a KL constraint to limit deviation from the original prediction distribution and control the magnitude of the decoding intervention. The paper evaluates POTION on Qwen2.5-VL and Llama-3.2-Vision using MSCOCO and AMBER datasets.

Critical strengths

The proposed method is training-free and can be directly applied at inference time.

The null-penalized partial OT is intuitive for assessing unsupported object candidates.

Critical weaknesses

The motivation and proposed method are not sufficiently convincing.


  • If we can accurately identify unsupported objects, why should we additionally control the magnitude of the intervention?

  • If it is hard, the paper should discuss why conventional methods fail and how POT itself improves this identification.

  • KL divergence from the original distribution itself does not reflect the identification accuracy, and therefore it is unclear why the magnitude should be controlled based on the KL divergence.

The paper could be better organized, as it is currently difficult to follow.

The performance gains are not substantial, and the empirical evidence is limited.


  • Considering the trade-offs across metrics, POTION does not show a clear performance improvement over competing methods.

  • The evaluation is limited to only two models and two datasets.


  • It is unclear whether the KL constraint further improves performance compared with using a single hyperparameter to control the correction magnitude. Since it increases both visual coverage and hallucination, its advantage in the overall trade-off is unclear.



(minor)
The paper emphasizes relative improvements from already low baselines, which may overstate the practical gain.

Questions for authors

See Weaknesses

8 comments
AuthorsOpenReview ↗

Answer to Reviewer uPXL (Part 1)

We thank the reviewer for the careful analysis and constructive feedback.

W1. “If we can accurately identify unsupported objects, why should we additionally control the magnitude of the intervention?”

We agree with the Reviewer on this point. With a perfect detector, magnitude control would be unnecessary: one could simply mask unsupported candidates. With a perfect detector, magnitude control would be unnecessary: one could simply mask unsupported candidates. We checked this with an oracle built from the ground truth: it replaces uju_j with 1 if the AMBER ground-truth annotations mark the candidate object as absent (judged by AMBER’s official hallucination rule) and with 0 otherwise, keeping the same penalty weight λ\lambda.

For the oracle, adding the KL constraint changes CHAIR/Cover from 0.8/67.6 to 4.9/67.7, so it brings no benefit: with correct labels it only blocks correct penalties (+4.0 CHAIR). Both oracle runs lose about 1.6 Cover points against regular decoding; since the oracle never penalizes a present object, this comes from the changed decoding trajectory, not from over-correction.

The KL constraint is needed because identification at decoding time is not accurate in this sense, for two reasons. First, the visual-support score is a noisy estimate. On AMBER, the null fraction uju_j separates present from absent candidates with AUROC 0.73 overall and 0.75 on steps where present and absent candidates compete (Table R3.1), and 82% of the total penalty mass falls on objects that are actually present (present objects make up 85% of candidates). Second, the decision is made on next-token candidates, not on objects. For example, a token such as “light” can begin or complete several phrases, so even a correct object-level judgment does not give a certain token-level one.

These errors are costly, and Table R4 shows that the KL constraint pays off in proportion to them. With POTION’s own score, removing the constraint (POT only, Table 3) drops CHAIR from 5.8 to 2.4 but costs 17.3 Cover points; the constraint recovers them for 3.4 CHAIR points and still leaves POTION 2.0 CHAIR points below regular decoding. With the perfect oracle the constraint recovers nothing and only costs CHAIR. As we flip the oracle’s labels at rate ε\varepsilon, its Cover benefit grows from +0.1 at ε=0\varepsilon=0 to +1.3 at ε=0.3\varepsilon=0.3, while its CHAIR cost falls from 4.0 to 2.0 points. A label-flipped oracle has AUROC 1−ε1-\varepsilon, so uju_j (AUROC 0.73) is about as noisy as an oracle with ε\varepsilon near 0.27, between the last two oracle rows. It is also a graded score that is above zero for almost every candidate, present or absent, so without the constraint it shrinks every object mention, which is why its Cover loss is far larger than the oracle’s. Magnitude control matters in proportion to identification error.

Table R3.1. Effect of the KL constraint (δ=0.1\delta=0.1) as a function of identification quality, AMBER (1,004 images). Oracle rows replace uju_j with AMBER ground truth (1 for absent objects, 0 otherwise), with a fraction ε\varepsilon of labels flipped (AUROC 1−ε1-\varepsilon); they use the penalty term only.

IdentificationAUROCCHAIR ↓ no KLCHAIR ↓ KLΔCHAIRCover ↑ no KLCover ↑ KLΔCover
Regular decoding (no correction)—7.7——69.2——
Oracle, ε = 0.01.000.84.9+4.05 [+3.57, +4.57]67.667.7+0.07 [−0.30, +0.47]
Oracle, ε = 0.10.901.75.1+3.37 [+2.83, +3.93]67.767.7−0.05 [−0.67, +0.59]
Oracle, ε = 0.20.802.85.7+2.89 [+2.36, +3.41]66.467.4+0.98 [+0.23, +1.72]
Oracle, ε = 0.30.703.95.9+1.98 [+1.43, +2.53]65.666.9+1.34 [+0.46, +2.22]
POT score uju_j (POT only → POTION)0.732.65.7+3.06 [+2.40, +3.72]47.564.4+16.91 [+15.72, +18.16]
AuthorsOpenReview ↗

Answer to Reviewer uPXL (Part 2)

W2: If it is hard, the paper should discuss why conventional methods fail and how POT itself improves this identification.

We added this discussion to Section 4.1, together with a direct measurement (Table R1). We run regular greedy decoding on AMBER and, at every step whose top-5 next-token pool contains object words (the same candidate set AtA_t that POTION assesses), compute several visual-support signals for each object candidate on the same decoding state; the signals are only logged, so the generated text is unchanged. Each candidate is labeled present or absent with AMBER's official hallucination rule (annotated objects plus AMBER's synonym lists), giving 64,798 candidates at 40,945 decoding steps over all 1,004 AMBER images, 14.7% of them absent. For each signal we report the AUROC for ranking absent candidates above present ones (each signal oriented so that weaker visual support means more likely absent), with 95% confidence intervals from a bootstrap over images.

The signals are the model's own probability pt(cj)p_t(c_j); a VCD-style contrast log⁡p(cj∣I)−log⁡p(cj∣Inoise)\log p(c_j\mid I)-\log p(c_j\mid I_{\text{noise}}); the maximum and the top-⌈n/mt⌉\lceil n/m_t\rceil mean cosine between the candidate's output embedding and the image tokens; and three transport-based scores built on the same cost matrix as POTION. Each of these drops one property of Problem (5), listed after the table: balanced OT (mean transport cost, no null option); per-candidate POT, which solves the null problem for each candidate on its own, so candidates no longer compete for the same patches (uj=max⁡(0,1−mt∣Sj(κ)∣/n)u_j=\max(0,1-m_t|S_j(\kappa)|/n), with Sj(κ)S_j(\kappa) the patches within the null threshold); and POT's null fraction uju_j itself, which keeps all three. The last row recomputes uju_j against a different image as a control for image-independent word priors.

Table R1

SignalAUROCAUROC, competitive steps*
Model probability pt(cj)p_t(c_j)0.590.63
Image contrast (VCD-style) log⁡p(cj∣I)−log⁡p(cj∣Inoise)\log p(c_j\mid I)-\log p(c_j\mid I_{\text{noise}})0.470.45
Max patch–token cosine0.740.73
Top-⌈n/mt⌉\lceil n/m_t\rceil mean patch–token cosine0.710.73
Balanced OT (no null option)0.720.74
Per-candidate POT (no shared budget)0.740.74
POT null fraction uju_j (ours)0.730.75
POT with a mismatched image0.560.51

*Steps whose candidate set contains both a present and an absent object.

Why conventional signals fail. The model’s own probability mixes language prior with visual evidence: objects that usually co-occur with the scene receive high probability whether or not they are visible (AUROC 0.59). Image contrast measures how much a prediction depends on the image as a whole, not whether this particular object is visible. An absent fork next to a visible plate also loses probability when the image is corrupted. Max-similarity scoring treats a single best-matching patch as evidence, so one spuriously similar patch is enough, and two related candidates (“dog” and “cat”) can both be supported by the same patches.

How POT improves on them. Problem (5) has three properties that target these failures:

  1. The null option lets a candidate abstain instead of being forced onto its least-bad patches.
  2. With uniform marginals, a candidate is fully supported only when enough patches lie within the null threshold (about n/mtn/m_t; Section 4.1). An isolated spurious match therefore does not count as support.
  3. Candidates share a finite visual budget (P1≤aP\mathbf{1}\le a). Patches that explain “dog” cannot also be counted as evidence for “cat”.

The competitive-steps column isolates the third property: where present and absent candidates compete, uju_j outperforms every alternative (paired AUROC differences +0.025 over max cosine, +0.012 over balanced OT and +0.017 over per-candidate POT; all 95% CIs exclude 0). Over all candidates the simpler signals are close, because most candidates are uncontested. The same properties show in generation: with everything else fixed, the null-fraction penalty alone reproduces POTION’s effect (CHAIR 5.9 vs 5.7 for POTION), whereas matching each candidate to its nearest patch removes almost no hallucination (CHAIR 7.3 vs 7.7 for regular decoding). Details are in our response to R2.

AuthorsOpenReview ↗

Answer to Reviewer uPXL (Part 3)

We conducted additional experiments to compare POTION with simpler image-patch-to-token matching strategies, including threshold-based matching, top-1 (maximum) matching, and top-5 matching. The results on COCO and Qwen2.5-VL-3B are summarized in the table below.

MethodCHAIR-S ↓CHAIR-I ↓Recall ↑
POTION31.45.668.16
Threshold matching35.06.667.8
Top-1 token matching (Maximum matching)35.66.767.6
Top-5 token matching34.66.766.7

The results demonstrate that POTION outperforms all simpler matching strategies across all three evaluation metrics. These findings highlight the limitations of heuristic matching approaches and support the effectiveness of formulating image-patch-to-token alignment as a Partial Optimal Transport (POT) problem.

Furthermore, to investigate the importance of using Partial Optimal Transport (POT) rather than conventional Optimal Transport (OT), we conducted an additional comparison, as shown below.

MethodCHAIR-S ↓CHAIR-I ↓Recall ↑
POT (ours)31.45.668.16
OT32.46.167.39

The results show that POT performs better than conventional OT, achieving lower hallucination rates and higher recall. This comparison suggests that partial matching is more effective than enforcing full transport between visual patches and text tokens.

W3: KL divergence from the original distribution itself does not reflect the identification accuracy, and therefore it is unclear why the magnitude should be controlled based on the KL divergence.

We agree that the KL divergence does not measure identification accuracy, and it is not designed to. Identification accuracy cannot be observed at inference time. What can be observed, and what determines the damage when identification is wrong, is how far the corrected distribution moves from the base model. POTION therefore splits the two roles: transport decides where to move, and the KL radius bounds how far.

Formally, problem (11) is equivalent to

max⁡p:DKL(p∥pt)≤δ1τEp[Δt]−DKL(p∥pt)\max _ {p : D _ {\mathrm{KL}}(p \Vert p _ t) \le \delta} \quad \frac{1}{\tau} \mathbb{E} _ {p} \lbrack \Delta _ t \rbrack - D _ {\mathrm{KL}}(p \Vert p _ t)

This maximizes an imperfect visual-support score inside a trust region around the base model, the usual construction when optimizing against an imperfect score. It gives a guarantee that holds however accurate the score is. By Pinsker’s inequality, no token’s probability within the candidate set changes by more than δ/2≈0.22\sqrt{\delta/2}\approx 0.22 per step at δ=0.1\delta=0.1, including the probability of an object that is actually present. A fixed scale on the logit correction gives no such bound, because the realized change grows with ∣Δt∣|\Delta_t|.

In practice the constraint does track identification errors. It binds on 55% of object steps, and about 10% more often (57% vs 52%) when the most-penalized candidate is actually present (Table R5). It binds mainly where the base model puts real probability on the penalized candidate (84%–94% of steps when that probability is at least 0.2, against 39% below it), and those candidates are more often present: overriding a likely object is exactly what incurs a large KL. The noisy-oracle experiment in Q1 shows the same link from the other side: the benefit of the KL constraint grows with the identification error rate.

We also compared against a single fixed coefficient on the correction, tuned over the same number of settings (our response to W3c and R2). On AMBER the fixed coefficient does not lose on the average trade-off; it keeps 0.7–1.8 more Cover points at matched CHAIR. What it cannot give is the per-step bound above: its largest per-step KL reaches 0.39 at c = 0.1 and 2.6 at c = 0.25, whereas the KL constraint never exceeds δ. The KL radius is the right control when the score may be wrong on any given step, which is the situation Q1 documents.

Reviewer_8Wg5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the object hallucination problem in the generation process of vision-language models (VLMs) and proposes a training-free inference-time decoding method, POTION (Partial Optimal Transport Inference with KL Projection). The method first uses partial optimal transport to match candidate object tokens with image visual features and introduces a null matching mechanism to identify candidate objects that lack visual evidence support, thereby adjusting the generation probabilities. Subsequently, a KL divergence constraint is imposed to limit the deviation of the corrected probability distribution from the original model, so as to alleviate the loss of correct visual information during hallucination suppression. The authors evaluate the method on Qwen2.5-VL and Llama-3.2-Vision using the MSCOCO and AMBER datasets.

Critical strengths

By decoupling visual support estimation from intervention strength control, the method design has clear motivation, and component experiments support the practical role of the KL constraint.

POTION scores candidate object tokens using partial optimal transport with a null matching option, and then uses a KL constraint to limit modifications to the original generation distribution. This design directly addresses the issue that ``suppressing hallucinations may simultaneously remove correct visual content.''

Critical weaknesses

There is a lack of controlled experiments directly verifying the incremental contribution of partial optimal transport and the null matching mechanism.

The core innovation of the paper is to allow candidates lacking visual support to retain a null match, but the existing component ablations mainly compare Base, POT, and POT+KL, while the additional sensitivity analyses focus on the KL budget, candidate set size, and representation depth. These experiments can show that the overall scheme is effective and that limiting the intervention magnitude is helpful, but they are insufficient to distinguish whether the gains arise from the joint matching and null matching mechanism of partial optimal transport or from more general visual similarity rescoring.

Questions for authors

1. Do partial optimal transport and the null-matching mechanism provide a clear incremental benefit over simpler visual-support scoring methods?

The ablation in Table 3 demonstrates the effectiveness of the overall correction and shows that the KL constraint can partially recover object coverage. However, it does not directly establish the incremental contribution of partial optimal transport over alternative visual scoring methods. Please clarify whether balanced OT, independent visual-similarity matching, and the proposed partial OT have been compared while keeping candidate selection, subsequent logit correction, and KL control unchanged. Do these comparisons support the central claim that allowing null matches helps avoid spurious visual alignments? This question directly affects my assessment of the novelty and necessity of the proposed mechanism.

2. Does the adaptive KL constraint outperform appropriately tuned fixed-strength control?

Equation (12) shows that the KL projection scales the original correction through an adaptive coefficient αt\alpha_t. The coverage recovery in Table 3 demonstrates that moderating an overly strong intervention is beneficial, but does not by itself establish the advantage of adaptive control. Using the same development set and hyperparameter-tuning budget, does adaptive KL control achieve a better hallucination–coverage trade-off than a fixed scaling coefficient? This comparison would help determine whether the benefits of the KL module arise from per-step adaptive adjustment or primarily from an overall reduction in correction strength.

1 comment
AuthorsOpenReview ↗

Answer to Reviewer 8Wg5

We thank the reviewer for the careful analysis and constructive feedback.

W1 & Q1. There is a lack of controlled experiments directly verifying the incremental contribution of partial optimal transport and the null matching mechanism.

Following the reviewer's suggestion, we conducted additional experiments to compare POTION with simpler image-patch-to-token matching strategies, including threshold-based matching, top-1 (maximum) matching, and top-5 matching. The results on COCO and Qwen2.5-VL-3B are summarized in the table below.

MethodCHAIR-S ↓CHAIR-I ↓Recall ↑
POTION31.45.668.16
Threshold matching35.06.667.8
Top-1 token matching (Maximum matching)35.66.767.6
Top-5 token matching34.66.766.7

The results demonstrate that POTION outperforms all simpler matching strategies across all three evaluation metrics. These findings highlight the limitations of heuristic matching approaches and support the effectiveness of formulating image-patch-to-token alignment as a Partial Optimal Transport (POT) problem.

Furthermore, to investigate the importance of using Partial Optimal Transport (POT) rather than conventional Optimal Transport (OT), we conducted an additional comparison, as shown below.

MethodCHAIR-S ↓CHAIR-I ↓Recall ↑
POT (ours)31.45.668.16
OT32.46.167.39

The results show that POT performs better than conventional OT, achieving lower hallucination rates and higher recall. This comparison suggests that partial matching is more effective than enforcing full transport between visual patches and text tokens.

Q2. Does the adaptive KL constraint outperform appropriately tuned fixed-strength control?

We thank the reviewer for this question. We ran this comparison, and the short answer is no. On the average hallucination–coverage trade-off, a tuned fixed coefficient matches the KL constraint, and on AMBER it is slightly ahead. The coverage recovery in Table 3 therefore comes mainly from an overall reduction in correction strength, as the reviewer suggests. What per-step adaptation adds is a bound on how far any single step can move from the base model, which a fixed coefficient does not provide.

Setup. Both controls scale the same POTION correction Δt\Delta_t; candidate selection, partial transport and the logit correction are unchanged. The fixed control applies one coefficient at every step, pt(c)∝ptexp⁡(cΔt/τ)p_t^{(c)}\propto p_t\exp(c\Delta_t/\tau), which is the path of Eq. (12) with a constant αt=c\alpha_t=c. Each control received the same tuning budget of four settings, swept on the same images: the full AMBER generative split, with Qwen2.5-VL-3B under the Table 3 protocol. We report the whole sweep rather than one selected setting per control, so the comparison does not depend on how a setting is chosen.

Table R2.1. KL radius δ\delta versus one fixed coefficient cc on AMBER (1,004 images, Qwen2.5-VL-3B, our rerun). Per-step KL is measured to the base distribution, over all steps where an object candidate is assessed.

ControlCHAIR ↓Cover ↑Hal ↓Mean per-step KLMax per-step KL
Regular decoding7.769.249.7——
KL δ=0.01\delta=0.016.566.842.20.0080.01
KL δ=0.05\delta=0.056.165.039.70.0370.05
KL δ=0.1\delta=0.1 (POTION)5.764.437.60.0670.10
KL δ=0.5\delta=0.54.860.830.70.2340.50
Fixed c=0.1c=0.15.965.838.70.0140.39
Fixed c=0.25c=0.254.862.632.00.0782.61
Fixed c=0.5c=0.53.657.422.00.2456.60
Fixed c=0.75c=0.752.753.015.00.4538.71
Full correction (c=1c=1, no KL)2.647.513.0——

Overall strength versus per-step adaptation.

  • Average trade-off. Reading the fixed-coefficient curve at the CHAIR of each KL setting, the KL constraint has 0.1–1.8 fewer Cover points. For the two closest matched pairs (paired bootstrap over images, 95% CI):

    • δ=0.05\delta=0.05 vs. c=0.1c=0.1: CHAIR +0.2 [−0.2, +0.6], Cover −0.7 [−1.3, −0.2].
    • δ=0.5\delta=0.5 vs. c=0.25c=0.25: CHAIR +0.0 [−0.5, +0.5], Cover −1.8 [−2.7, −1.0].

    Hal differences are not significant in either pair. On MSCOCO the closest pair ties (CHAIR$_I$ −0.03 [−0.50, +0.47], Recall −0.3 [−1.2, +0.6]). The trade-off is therefore set by the overall strength, and adaptation does not improve it.

  • Worst-case step. The mean per-step KL is of the same order for matched settings. The fixed coefficient's largest per-step KL, however, is 0.39 at c=0.1c=0.1 and grows to 8.7 at c=0.75c=0.75, whereas the radius caps every step at δ\delta. At δ=0.1\delta=0.1, Pinsker's inequality means that no candidate's probability changes by more than δ/2≈0.22\sqrt{\delta/2}\approx 0.22 in a step, including the probability of an object that is actually present.

Reviewer_5gBROpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes POTION, a training-free decoding framework that combines partial optimal transport (POT) with KL-constrained distributional correction to mitigate object hallucinations in VLMs. POT matches candidate-token embeddings with visual representations while allowing unsupported candidates to remain unmatched through a null sink. The resulting transport statistics are used to adjust token logits, while a KL constraint limits excessive deviations from the original model distribution. Experiments on Qwen2.5-VL and Llama-3.2-Vision demonstrate hallucination reduction with partial preservation of object coverage.

Critical strengths
  1. Interesting methodological perspective: Formulating visual grounding as a partial optimal transport problem is an interesting approach. In particular, introducing a null sink to avoid forcing visually unsupported candidates into artificial correspondences provides a reasonable motivation for using partial OT.

  2. KL-constrained correction: The separation between visual-support estimation and correction-strength control is appealing. The ablation study demonstrates that the KL constraint substantially recovers object coverage lost by unrestricted transport-based correction.

Critical weaknesses
  1. Insufficient justification and validation of the OT-based visual grounding mechanism.
  • Although applying partial OT to visual grounding is interesting, several aspects of the formulation require stronger justification and empirical validation.

    • Transport cost matrix: The method defines CijC_{ij} using the cosine distance between visual representations viv_i and candidate-token output embeddings eje_j. However, the paper does not sufficiently explain why this distance should reliably reflect whether an object is visually present. Since the transport solution is determined by the cost matrix, validating this assumption is essential.

    • Necessity of optimal transport: It remains unclear what additional information OT provides beyond simpler patch–token similarity measures. Comparisons with maximum cosine similarity, top-k patch similarity, or threshold-based matching would help establish whether the OT formulation itself is necessary for the reported improvements.

    • Transport marginals: The paper assigns uniform masses to visual patches and candidate tokens (ai=1/na_i=1/n, bj=1/mtb_j=1/m_t). However, the rationale for this choice and its sensitivity to alternative mass assignments are not sufficiently discussed.

    • Interpretability of the transport results: The qualitative examples in Figure 3 mainly demonstrate changes in generated descriptions. More informative examples would visualize how the cost matrix CC, transport plan PP, unmatched mass sjs_j, and null fraction uju_j differ between visually supported and unsupported candidates, and how these differences translate into the final logit correction Δt\Delta_t.

  • Overall, the paper should provide stronger evidence that the transport plan and unmatched mass genuinely capture visual grounding rather than merely providing a more complicated similarity-based scoring mechanism.

  1. Limited empirical advantages and incomplete comparisons with recent methods.
  • Although POTION improves hallucination metrics relative to regular decoding, its advantages over existing methods are not consistent across models and evaluation metrics.

    • For example, in Table 1, POTION underperforms CMAC on CHAIRS for Qwen2.5-VL (31.4 vs. 28.4), while achieving the same CHAIRI (5.6) and lower object recall (68.16 vs. 70.7). In Table 2, POTION achieves the lowest CHAIR scores, but object coverage remains below regular decoding on both backbones.
  • Furthermore, recent training-free hallucination mitigation methods, such as VISTA [1], CausalLens [2], and FLB [3], are not included in the comparisons. Since these approaches also aim to improve visual grounding with limited inference overhead, comparisons under matched evaluation settings would better establish POTION's effectiveness and practical advantages.

  • These limitations make it difficult to determine whether the additional complexity of the proposed formulation is justified by sufficiently consistent performance improvements.

References

[1] Li, Z., Shi, H., Gao, Y., Liu, D., Wang, Z., Chen, Y., Liu, T., Zhao, L., Wang, H., and Metaxas, D. N. (2025). The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models Via Visual Information Steering. ICML 2025, pp. 35799–35819.

[2] Ji, J., Liu, Q., Yang, W., and He, Z. (2026). CausalLens: Sensitivity-Guided Multi-Head Causal Intervention for Hallucination Mitigation in Large Vision-Language Models. CVPR 2026, pp. 4199–4209.

[3] Ha, J., Baek, J., and So, J. (2026). First Logit Boosting: Visual Grounding Method to Mitigate Object Hallucination in Large Vision-Language Models. CVPR 2026, pp. 18241–18250.

Questions for authors
  1. Validity of the transport cost: Can the authors provide empirical evidence that the cosine distance between visual representations and candidate-token embeddings reliably reflects object existence?

  2. Necessity of POT: How does the proposed OT-based scoring compare with simpler alternatives, such as maximum or top-k patch–token cosine similarity? What improvement can be attributed specifically to the optimal transport formulation?

  3. Transport behavior and interpretability: Can the authors provide candidate-level visualizations of CC, PP, sjs_j, uju_j, and Δt\Delta_t for supported and hallucinated objects? Such an analysis would help explain how POTION actually identifies and suppresses hallucinated candidates.

  4. Marginal distributions: What motivates the uniform assignments of aa and bb, and how sensitive is the method to alternative marginal distributions?

  5. Comparison with recent methods: Can the authors provide comparisons with more recent training-free approaches [1-3] under matched evaluation settings, including both hallucination and coverage metrics?

6 comments
AuthorsOpenReview ↗

Answer to Reviewer 5gBR (Part 1)

We thank the reviewer for the careful analysis and constructive feedback.

W1. Transport cost matrix: The method defines CijC_{ij} using the cosine distance between visual representations viv_i and candidate-token output embeddings eje_j. However, the paper does not sufficiently explain why this distance should reliably reflect whether an object is visually present.

We thank the reviewer for this question. Our choice rests on three points.

In Qwen2.5-VL, the merged visual tokens viv_i are fed to the language model as input embeddings, so they live in the same space as token embeddings. Cosine similarity between viv_i and eje_j therefore measures how strongly a patch resembles the word the model is about to output. This follows prior work that reads object identity from image representations through the model's own vocabulary [1–3], which shows that such scores are higher for objects present in the image than for hallucinated ones.

Moreover, we also evaluate the impact of different similarity metrics by conducting additional experiments comparing cosine similarity with L2-normalized distance and dot product. The results are summarized in the table below.

MethodCHAIR-S ↓CHAIR-I ↓Recall ↑
Cosine31.45.668.16
L2-normalized34.46.467.8
Dot-product34.86.467.7

The results demonstrate that cosine similarity outperforms the alternative metrics across all three evaluation metrics, achieving lower hallucination rates while maintaining higher object recall.

Furthermore, Figure 4 in our revised manuscript, we provide qualitative examples illustrating how POT helps the model distinguish between objects that are present in the image and those that are absent, thereby reducing object hallucinations.

W2 & Q1. Necessity of optimal transport: It remains unclear what additional information OT provides beyond simpler patch–token similarity measures. Comparisons with maximum cosine similarity, top-k patch similarity, or threshold-based matching would help establish whether the OT formulation itself is necessary for the reported improvements.

Following the reviewer's suggestion, we conducted additional experiments to compare POTION with simpler image-patch-to-token matching strategies, including threshold-based matching, top-1 (maximum) matching, and top-5 matching. The results on COCO and Qwen2.5-VL-3B are summarized in the table below.

MethodCHAIR-S ↓CHAIR-I ↓Recall ↑
POTION31.45.668.16
Threshold matching35.06.667.8
Top-1 token matching (Maximum matching)35.66.767.6
Top-5 token matching34.66.766.7

The results demonstrate that POTION outperforms all simpler matching strategies across all three evaluation metrics. These findings highlight the limitations of heuristic matching approaches and support the effectiveness of formulating image-patch-to-token alignment as a Partial Optimal Transport (POT) problem.

Furthermore, to investigate the importance of using Partial Optimal Transport (POT) rather than conventional Optimal Transport (OT), we conducted an additional comparison, as shown below.

MethodCHAIR-S ↓CHAIR-I ↓Recall ↑
POT (ours)31.45.668.16
OT32.46.167.39

The results show that POT performs better than conventional OT, achieving lower hallucination rates and higher recall. This comparison suggests that partial matching is more effective than enforcing full transport between visual patches and text tokens.

AuthorsOpenReview ↗

Answer to Reviewer 5gBR (Part 2)

W3 & Q4. Transport marginals: The paper assigns uniform masses to visual patches and candidate tokens (ai=1/n,bj=1/mta_i=1/n, b_j=1/m_t). However, the rationale for this choice and its sensitivity to alternative mass assignments are not sufficiently discussed.

We use uniform masses, ai=1/na_i = 1/n and bj=1/mtb_j = 1/m_t, because at inference time we have no information about how relevant each patch is to the next token, or how much visual evidence each candidate deserves. Uniform marginals are the maximum-entropy choice under this uncertainty: they commit to no prior about object size or saliency.

The reviewer's question also points to a deeper point. If informative marginals were available, that is, if we knew which patches support which candidate, then the transport problem itself would become unnecessary. We could simply suppress the candidates that have no supporting patches. In the same way, the KL constraint is only needed because the support estimate is imperfect.

To make this concrete, we ran an oracle in which the support signal is perfect: a candidate is penalized if and only if the ground-truth annotations mark its object as absent. All other settings are unchanged, and we run it with and without the KL constraint:

IdentificationAUROCCHAIR ↓ (no KL)CHAIR ↓ (KL)ΔCHAIRCover ↑ (no KL)Cover ↑ (KL)ΔCover
Regular decoding (no correction)—7.7——69.2——
Oracle (perfect identification)1.000.84.9+4.05 [+3.57, +4.57]67.667.7+0.07 [−0.30, +0.47]

Δ = KL − no KL; brackets are 95% paired bootstrap confidence intervals over images.

The table shows two things. First, with perfect support information, unconstrained correction almost eliminates hallucination (CHAIR 7.7 → 0.8) while keeping coverage close to regular decoding (69.2 → 67.6). Second, in this setting the KL constraint is counterproductive. It raises CHAIR by 4.05 points, and its effect on coverage (+0.07) is not distinguishable from zero. Magnitude control, therefore, is not needed when identification is perfect. It earns its place only because the transport-based estimate, like any estimate available at inference time, makes errors. Our response to Reviewer uPXL quantifies these errors and shows that the benefit of the KL constraint grows as identification becomes noisier.

The gap between the oracle and POTION is the room left for better support estimates, including more informative marginals. Learning or estimating such priors without supervision is a natural direction for future work.

W4. Interpretability of the transport results: Can the authors provide candidate-level visualizations of C,P,su,uj,C, P, s_u, u_j, and for supported and hallucinated objects? Such an analysis would help explain how POTION actually identifies and suppresses hallucinated candidates.

We provide qualitative examples illustrating how POT helps the model distinguish between objects that are present in the image and those that are absent, thereby reducing object hallucinations. This is Figure 4 in our revised manuscript.

AuthorsOpenReview ↗

Answer to Reviewer 5gBR (Part 3)

W5. Although POTION improves hallucination metrics relative to regular decoding, its advantages over existing methods are not consistent across models and evaluation metrics.

We observe that although CMAC achieves strong performance, its generated responses tend to be short (as indicated by the average token count in Table 1) and lack sufficient detail. Specifically, CMAC primarily lists objects without providing detailed descriptions. We further conducted experiments comparing POTION with CMAC under a maximum generation length of 256 tokens (max_new_tokens=256), where our method outperforms CMAC. We refer to Table 13 in the Appendix for a detailed comparison that further highlights this limitation.

ModelCapMethodCHAIR-S ↓CHAIR-I ↓Recall ↑Avg. Tok.
Qwen2.5-VL-3B256Regular37.008.8063.43247.36
256CMAC27.405.4468.09179.35
256POTION26.406.1063.43238.12
512Regular52.6011.0066.80409.50
512CMAC28.405.6070.70194.00
512POTION31.405.6068.16355.10
Llama-3.2-Vision-11B256Regular30.806.8667.26182.50
256CMAC25.405.7769.31142.70
256POTION21.604.8565.15161.38
512Regular34.407.7567.46199.54
512CMAC24.205.3069.37144.91
512POTION17.403.0966.37172.45

W6 & Q5. Comparisons with recent methods. Can the authors provide comparisons with more recent training-free approaches [4-6] under matched evaluation settings, including both hallucination and coverage metrics?

We thank the reviewer for suggesting additional relevant baselines. We have incorporated VISTA, CausalLens, and FLB into our evaluation framework, conducted the corresponding experiments, and updated the results in Tables 1 and 2 of the revised manuscript. We report the results of Qwen2.5-VL-3B on the MSCOCO dataset in the table below. For the remaining experiments, please refer to the revised manuscript.

MethodCHAIR-S ↓CHAIR-I ↓Recall ↑
POTION31.45.668.16
VISTA [4]41.808.7963.68
CausalLens [5]37.207.9267.07
FLB [6]34.005.8868.80

POTION achieves the lowest hallucination on both metrics. Relative to the strongest recent baseline, FLB, it lowers CHAIR$_S$ by 2.6 points and CHAIR$_I$ by 0.28 points, while FLB attains slightly higher recall (+0.64). Relative to CausalLens, POTION reduces CHAIR$_S$ by 5.8 and CHAIR$_I$ by 2.3 points, with higher recall (+1.1). VISTA reduces hallucination relative to regular decoding but trails the other methods on Qwen2.5-VL and lowers recall below regular decoding (63.68 vs 66.8). We attribute this partly to its steering strength being calibrated on LLaVA-style architectures; we re-tuned it on our development set but did not search beyond the shared budget.

References

[1] Park, Seongheon, and Sharon Li. "GLSim: Detecting object hallucinations in LVLMs via global-local similarity." Advances in Neural Information Processing Systems 38 (2026): 37747-37776.

[2] Nguyen, Tuan Dung, et al. "Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

[3] Phukan, Anirudh, et al. "Beyond logit lens: Contextual embeddings for robust hallucination detection & grounding in vlms." Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

[4] Li, Z., Shi, H., Gao, Y., Liu, D., Wang, Z., Chen, Y., Liu, T., Zhao, L., Wang, H., and Metaxas, D. N. (2025). The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models Via Visual Information Steering. ICML 2025, pp. 35799–35819.

[5] Ji, J., Liu, Q., Yang, W., and He, Z. (2026). CausalLens: Sensitivity-Guided Multi-Head Causal Intervention for Hallucination Mitigation in Large Vision-Language Models. CVPR 2026, pp. 4199–4209.

[6] Ha, J., Baek, J., and So, J. (2026). First Logit Boosting: Visual Grounding Method to Mitigate Object Hallucination in Large Vision-Language Models. CVPR 2026, pp. 18241–18250.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to common concerns

We thank the reviewers for their careful analysis and constructive feedback for our work. We are grateful that the reviewers found the null-sink formulation of partial OT an intuitive way to avoid forcing unsupported candidates into artificial matches (5gBR, 8Wg5, uPXL), and the separation between visual-support estimation and correction-strength control appealing (5gBR, 8Wg5).

There are three concerns shared across reviews. We summarize the new evidence here; details are in the individual responses. Unless stated otherwise, new experiments use the full AMBER generative split (1,004 images) with Qwen2.5-VL-3B, and intervals are 95% paired bootstrap CIs over images.

1. Is the transport formulation necessary, and does it capture visual grounding? (5gBR, 8Wg5, uPXL)

  • Does the null mass reflect absent objects? We labeled 64,798 object candidates from regular decoding as present or absent using AMBER's ground truth. POT's null fraction uju_j separates absent from present candidates with AUROC 0.73, and 0.75 at steps where present and absent candidates compete. By comparison, the model's own probability reaches 0.59 and a VCD-style image contrast 0.47. At competitive steps, uju_j outperforms max cosine, balanced OT and per-candidate POT (all CIs exclude 0). Computing it against a mismatched image drops it to near chance (0.56), so the score does depend on the image. (uPXL W2)
  • Generation. Simple patch–token matching (maximum, top-k and threshold) gives clearly more hallucination than POTION on MSCOCO (CHAIR$_S$ 34.6–35.6 vs. 31.4). In a controlled ablation on AMBER, with everything else fixed, nearest-patch matching barely changes CHAIR (7.3 vs. 7.7 for regular decoding), while the null-fraction penalty alone reproduces POTION's effect (5.9 vs. 5.7).
  • Cost and marginals. The cosine cost outperforms L2 and dot-product costs (CHAIR$_S$ 31.4 vs. 34.4 and 34.8). Uniform marginals are the maximum-entropy choice when there is no prior knowledge of object size or saliency. A ground-truth oracle shows how much better support estimates could still gain. (5gBR W1, W3)
  • Interpretability. Figure 4 of our revised PDF visualizes CC, PP, sjs_j, uju_j and Δt\Delta_t for supported and hallucinated candidates. (5gBR W4)

2. Why control the magnitude, and why with KL? (uPXL, 8Wg5)

  • Magnitude control is needed because identification is imperfect. With a ground-truth oracle in place of uju_j, the unconstrained correction almost eliminates hallucination (CHAIR 7.7 → 0.8, Cover 69.2 → 67.6), and the KL constraint only costs CHAIR (+4.05). As oracle labels are flipped, the constraint's Cover benefit grows and its CHAIR cost falls. With POT's own score (AUROC 0.73), the constraint recovers 16.9 Cover points for 3.1 CHAIR points. (uPXL W1)
  • KL versus a fixed coefficient. We agree with 8Wg5 and uPXL that a tuned fixed coefficient reaches the same average trade-off: on AMBER it keeps 0.7–1.8 more Cover points at matched CHAIR, and on MSCOCO the two tie. The two are the penalized and constrained forms of the same problem, along the path of Eq. (12). What the radius adds is a per-step bound: no candidate's probability moves by more than δ/2≈0.22\sqrt{\delta/2}\approx0.22 in a step, whereas the fixed coefficient's largest per-step KL ranges from 0.39 to 8.7 across settings. We will claim this bound, not a better trade-off. (8Wg5 Q2, uPXL W3, W6)

3. Strength and consistency of the gains (5gBR, uPXL)

  • Recent baselines. We added VISTA [1] (ICML 2025), CausalLens [2] (CVPR 2026) and FLB [3] (CVPR 2026) under the Table 1 setting. POTION has the lowest CHAIR$_S$ and CHAIR$_I$ among these methods, and FLB has slightly higher Recall. (5gBR W6)
  • Trade-offs. POTION has the lowest CHAIR in all four AMBER model–budget settings and on MSCOCO with Llama-3.2-Vision; its consistent cost is coverage. On AMBER with Qwen2.5-VL, every baseline with higher coverage also removes less hallucination. For δ≤0.1\delta\le0.1, POTION gives up less coverage per CHAIR point removed than OPERA, MaskCD or CMAC. CMAC, the strongest competitor and better than POTION on MSCOCO with Qwen2.5-VL, produces responses about half as long. (5gBR W5, uPXL W4)
  • Absolute gains. The base rates are not low: with regular decoding, 34–53% of MSCOCO descriptions and 49% of AMBER descriptions contain a hallucinated object. POTION lowers CHAIR$_S$ by 17.0–21.2 points on MSCOCO and Hal by 10.7 points on AMBER (our rerun: −12.1 [−15.6, −8.8]). On AMBER with Llama-3.2-Vision the absolute gain is small, and the revision will say so. (uPXL W7)
  • Scale. Evaluating two models on two benchmarks matches several of our baselines (ICT, MaskCD). Our two models differ in family, size and how visual features enter the language model. We are happy to add benchmarks the reviewers suggest. (uPXL W5)
AuthorsOpenReview ↗

General response to common concerns (continued)

[1] Li, Z., Shi, H., Gao, Y., Liu, D., Wang, Z., Chen, Y., Liu, T., Zhao, L., Wang, H., and Metaxas, D. N. The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering. ICML 2025, pp. 35799–35819.

[2] Ji, J., Liu, Q., Yang, W., and He, Z. CausalLens: Sensitivity-Guided Multi-Head Causal Intervention for Hallucination Mitigation in Large Vision-Language Models. CVPR 2026, pp. 4199–4209.

[3] Ha, J., Baek, J., and So, J. First Logit Boosting: Visual Grounding Method to Mitigate Object Hallucination in Large Vision-Language Models. CVPR 2026, pp. 18241–18250.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.