- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes POTION, a training-free method for mitigating object hallucination in vision-language models. POTION uses null-penalized partial optimal transport to assess visual support of token candidates and adjust their logits accordingly. Additionally, POTION uses a KL constraint to limit deviation from the original prediction distribution and control the magnitude of the decoding intervention. The paper evaluates POTION on Qwen2.5-VL and Llama-3.2-Vision using MSCOCO and AMBER datasets.
Critical strengths
The proposed method is training-free and can be directly applied at inference time.
The null-penalized partial OT is intuitive for assessing unsupported object candidates.
Critical weaknesses
The motivation and proposed method are not sufficiently convincing.
- If we can accurately identify unsupported objects, why should we additionally control the magnitude of the intervention?
- If it is hard, the paper should discuss why conventional methods fail and how POT itself improves this identification.
- KL divergence from the original distribution itself does not reflect the identification accuracy, and therefore it is unclear why the magnitude should be controlled based on the KL divergence.
The paper could be better organized, as it is currently difficult to follow.
The performance gains are not substantial, and the empirical evidence is limited.
- Considering the trade-offs across metrics, POTION does not show a clear performance improvement over competing methods.
- The evaluation is limited to only two models and two datasets.
- It is unclear whether the KL constraint further improves performance compared with using a single hyperparameter to control the correction magnitude. Since it increases both visual coverage and hallucination, its advantage in the overall trade-off is unclear.
(minor) The paper emphasizes relative improvements from already low baselines, which may overstate the practical gain.
Questions for authors
See Weaknesses
8 comments
Answer to Reviewer uPXL (Part 2)
W2: If it is hard, the paper should discuss why conventional methods fail and how POT itself improves this identification.
We added this discussion to Section 4.1, together with a direct measurement (Table R1). We run regular greedy decoding on AMBER and, at every step whose top-5 next-token pool contains object words (the same candidate set that POTION assesses), compute several visual-support signals for each object candidate on the same decoding state; the signals are only logged, so the generated text is unchanged. Each candidate is labeled present or absent with AMBER's official hallucination rule (annotated objects plus AMBER's synonym lists), giving 64,798 candidates at 40,945 decoding steps over all 1,004 AMBER images, 14.7% of them absent. For each signal we report the AUROC for ranking absent candidates above present ones (each signal oriented so that weaker visual support means more likely absent), with 95% confidence intervals from a bootstrap over images.
The signals are the model's own probability ; a VCD-style contrast ; the maximum and the top- mean cosine between the candidate's output embedding and the image tokens; and three transport-based scores built on the same cost matrix as POTION. Each of these drops one property of Problem (5), listed after the table: balanced OT (mean transport cost, no null option); per-candidate POT, which solves the null problem for each candidate on its own, so candidates no longer compete for the same patches (, with the patches within the null threshold); and POT's null fraction itself, which keeps all three. The last row recomputes against a different image as a control for image-independent word priors.
Table R1
| Signal | AUROC | AUROC, competitive steps* |
|---|---|---|
| Model probability | 0.59 | 0.63 |
| Image contrast (VCD-style) | 0.47 | 0.45 |
| Max patch–token cosine | 0.74 | 0.73 |
| Top- mean patch–token cosine | 0.71 | 0.73 |
| Balanced OT (no null option) | 0.72 | 0.74 |
| Per-candidate POT (no shared budget) | 0.74 | 0.74 |
| POT null fraction (ours) | 0.73 | 0.75 |
| POT with a mismatched image | 0.56 | 0.51 |
*Steps whose candidate set contains both a present and an absent object.
Why conventional signals fail. The model’s own probability mixes language prior with visual evidence: objects that usually co-occur with the scene receive high probability whether or not they are visible (AUROC 0.59). Image contrast measures how much a prediction depends on the image as a whole, not whether this particular object is visible. An absent fork next to a visible plate also loses probability when the image is corrupted. Max-similarity scoring treats a single best-matching patch as evidence, so one spuriously similar patch is enough, and two related candidates (“dog” and “cat”) can both be supported by the same patches.
How POT improves on them. Problem (5) has three properties that target these failures:
- The null option lets a candidate abstain instead of being forced onto its least-bad patches.
- With uniform marginals, a candidate is fully supported only when enough patches lie within the null threshold (about ; Section 4.1). An isolated spurious match therefore does not count as support.
- Candidates share a finite visual budget (). Patches that explain “dog” cannot also be counted as evidence for “cat”.
The competitive-steps column isolates the third property: where present and absent candidates compete, outperforms every alternative (paired AUROC differences +0.025 over max cosine, +0.012 over balanced OT and +0.017 over per-candidate POT; all 95% CIs exclude 0). Over all candidates the simpler signals are close, because most candidates are uncontested. The same properties show in generation: with everything else fixed, the null-fraction penalty alone reproduces POTION’s effect (CHAIR 5.9 vs 5.7 for POTION), whereas matching each candidate to its nearest patch removes almost no hallucination (CHAIR 7.3 vs 7.7 for regular decoding). Details are in our response to R2.
Answer to Reviewer uPXL (Part 3)
We conducted additional experiments to compare POTION with simpler image-patch-to-token matching strategies, including threshold-based matching, top-1 (maximum) matching, and top-5 matching. The results on COCO and Qwen2.5-VL-3B are summarized in the table below.
| Method | CHAIR-S ↓ | CHAIR-I ↓ | Recall ↑ |
|---|---|---|---|
| POTION | 31.4 | 5.6 | 68.16 |
| Threshold matching | 35.0 | 6.6 | 67.8 |
| Top-1 token matching (Maximum matching) | 35.6 | 6.7 | 67.6 |
| Top-5 token matching | 34.6 | 6.7 | 66.7 |
The results demonstrate that POTION outperforms all simpler matching strategies across all three evaluation metrics. These findings highlight the limitations of heuristic matching approaches and support the effectiveness of formulating image-patch-to-token alignment as a Partial Optimal Transport (POT) problem.
Furthermore, to investigate the importance of using Partial Optimal Transport (POT) rather than conventional Optimal Transport (OT), we conducted an additional comparison, as shown below.
| Method | CHAIR-S ↓ | CHAIR-I ↓ | Recall ↑ |
|---|---|---|---|
| POT (ours) | 31.4 | 5.6 | 68.16 |
| OT | 32.4 | 6.1 | 67.39 |
The results show that POT performs better than conventional OT, achieving lower hallucination rates and higher recall. This comparison suggests that partial matching is more effective than enforcing full transport between visual patches and text tokens.
W3: KL divergence from the original distribution itself does not reflect the identification accuracy, and therefore it is unclear why the magnitude should be controlled based on the KL divergence.
We agree that the KL divergence does not measure identification accuracy, and it is not designed to. Identification accuracy cannot be observed at inference time. What can be observed, and what determines the damage when identification is wrong, is how far the corrected distribution moves from the base model. POTION therefore splits the two roles: transport decides where to move, and the KL radius bounds how far.
Formally, problem (11) is equivalent to
This maximizes an imperfect visual-support score inside a trust region around the base model, the usual construction when optimizing against an imperfect score. It gives a guarantee that holds however accurate the score is. By Pinsker’s inequality, no token’s probability within the candidate set changes by more than per step at , including the probability of an object that is actually present. A fixed scale on the logit correction gives no such bound, because the realized change grows with .
In practice the constraint does track identification errors. It binds on 55% of object steps, and about 10% more often (57% vs 52%) when the most-penalized candidate is actually present (Table R5). It binds mainly where the base model puts real probability on the penalized candidate (84%–94% of steps when that probability is at least 0.2, against 39% below it), and those candidates are more often present: overriding a likely object is exactly what incurs a large KL. The noisy-oracle experiment in Q1 shows the same link from the other side: the benefit of the KL constraint grows with the identification error rate.
We also compared against a single fixed coefficient on the correction, tuned over the same number of settings (our response to W3c and R2). On AMBER the fixed coefficient does not lose on the average trade-off; it keeps 0.7–1.8 more Cover points at matched CHAIR. What it cannot give is the per-step bound above: its largest per-step KL reaches 0.39 at c = 0.1 and 2.6 at c = 0.25, whereas the KL constraint never exceeds δ. The KL radius is the right control when the score may be wrong on any given step, which is the situation Q1 documents.
Answer to Reviewer uPXL (Part 1)
We thank the reviewer for the careful analysis and constructive feedback.
We agree with the Reviewer on this point. With a perfect detector, magnitude control would be unnecessary: one could simply mask unsupported candidates. With a perfect detector, magnitude control would be unnecessary: one could simply mask unsupported candidates. We checked this with an oracle built from the ground truth: it replaces uj with 1 if the AMBER ground-truth annotations mark the candidate object as absent (judged by AMBER’s official hallucination rule) and with 0 otherwise, keeping the same penalty weight λ.
For the oracle, adding the KL constraint changes CHAIR/Cover from 0.8/67.6 to 4.9/67.7, so it brings no benefit: with correct labels it only blocks correct penalties (+4.0 CHAIR). Both oracle runs lose about 1.6 Cover points against regular decoding; since the oracle never penalizes a present object, this comes from the changed decoding trajectory, not from over-correction.
The KL constraint is needed because identification at decoding time is not accurate in this sense, for two reasons. First, the visual-support score is a noisy estimate. On AMBER, the null fraction uj separates present from absent candidates with AUROC 0.73 overall and 0.75 on steps where present and absent candidates compete (Table R3.1), and 82% of the total penalty mass falls on objects that are actually present (present objects make up 85% of candidates). Second, the decision is made on next-token candidates, not on objects. For example, a token such as “light” can begin or complete several phrases, so even a correct object-level judgment does not give a certain token-level one.
These errors are costly, and Table R4 shows that the KL constraint pays off in proportion to them. With POTION’s own score, removing the constraint (POT only, Table 3) drops CHAIR from 5.8 to 2.4 but costs 17.3 Cover points; the constraint recovers them for 3.4 CHAIR points and still leaves POTION 2.0 CHAIR points below regular decoding. With the perfect oracle the constraint recovers nothing and only costs CHAIR. As we flip the oracle’s labels at rate ε, its Cover benefit grows from +0.1 at ε=0 to +1.3 at ε=0.3, while its CHAIR cost falls from 4.0 to 2.0 points. A label-flipped oracle has AUROC 1−ε, so uj (AUROC 0.73) is about as noisy as an oracle with ε near 0.27, between the last two oracle rows. It is also a graded score that is above zero for almost every candidate, present or absent, so without the constraint it shrinks every object mention, which is why its Cover loss is far larger than the oracle’s. Magnitude control matters in proportion to identification error.
Table R3.1. Effect of the KL constraint (δ=0.1) as a function of identification quality, AMBER (1,004 images). Oracle rows replace uj with AMBER ground truth (1 for absent objects, 0 otherwise), with a fraction ε of labels flipped (AUROC 1−ε); they use the penalty term only.