- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a training-free object hallucination correction method for diffusion-based VLMs. It uses the model’s own masked-state estimates of object presence to identify suspicious objects, locally regenerate the corresponding phrases, and select corrected outputs. Experiments on three diffusion-based VLMs, including evaluations on COCO and AMBER and ablation studies, show reduced object hallucination rates.
Critical strengths
The method is not proposed arbitrarily; it is supported by some preliminary analysis.
Critical weaknesses
- Does the method become too time-consuming with many objects, limiting its practical value? Do the runtime benchmarks mostly contain only a few objects?
- As object hallucination is relatively simple and increasingly rare in stronger models, can the method generalize to harder cases, such as attribute and spatial-relation hallucinations?
- The method appears to rely on the model’s inherent confidence, suppressing uncertain outputs while retaining confident ones, rather than genuinely improving the model’s underlying capabilities.
Questions for authors
See Weaknesses
9 comments
2. Relevance and scope of object hallucination (W2)
Object hallucination remains substantial in the three dVLMs evaluated here. Vanilla COCO CHAIR-S is 24.8%/9.8%/22.6% for LLaDA-V/MMaDA/LaViDA, and AMBER Hal is 14.34%/18.33%/44.92% (Tables 1–2). EGCC reduces these rates under one frozen policy, demonstrating a practical correction opportunity across the evaluated dVLM families.
OpenCHAIR further supports transfer to generic object proposals. Attribute and spatial-relation correction remain outside the validated scope, as stated in the submitted Limitations, and would require corresponding proposals, evidence queries, and acceptance criteria.
3. Value of evidence-guided correction with a frozen model (W3)
EGCC improves inference-time factual reliability by making better use of the frozen model's evidence. With the backbone fixed, we test whether masked-state evidence supports more factual, informative outputs through constrained local correction.
The score answers an image-conditioned object-existence query before answer decoding. The readout controls show that this information is more diagnostic than the decoded answer to the same query. On full POPE, using the masked-state readout improves binary accuracy by 7.5/11.4/0.5 percentage points across the three backbones (Table 4), showing that useful information is available beyond the model's final verification decision.
The correction benefit also extends beyond deleting uncertain content. Non-target tokens remain fixed. In an additional comparison on the full COCO and AMBER sets, we order EGCC-selected responses by evidence and use evaluation outcomes to choose each deletion control's most favorable prefix within EGCC's supported-object loss budget. EGCC removes more unsupported mentions than object deletion in five of six settings, matching it in the sixth, and exceeds phrase deletion in all six. For example, on AMBER LaViDA, EGCC removes 201 unsupported mentions with 23 supported losses, versus 132/22 for object deletion and 36/23 for phrase deletion.
Direct same-selection comparisons further favor EGCC in grounded detailedness and fluency over both deletion controls in all six settings. Qwen2.5-VL-72B evaluates image-caption pairs with method identities hidden and both presentation orders averaged; object-deletion comparisons have 311 valid pairs after eight unparseable judge outputs, and phrase-deletion comparisons have all 319. These results support a practical benefit from local recommitment: better hallucination reduction within a content-loss budget, together with better judged content and language quality in the direct comparison.
reply 1
I noticed in your breakdown table that for LLaDA-V, the 3–4 proposals group (87 images) has a mean added time of 1.003s (4.90% overhead), whereas the ≥5 proposals group (18 images) drops to 0.989s (4.72% overhead). Why is the time overhead for ≥5 proposals actually lower than that for 3–4 proposals, and why does this pattern appear for both LLaDA-V and MMaDA? Could you analyze the underlying reason for this?
Thank you for recognizing the analysis motivating EGCC. We clarify how its cost scales with object proposals, the scope of the hallucination problem studied, and the value of correction with a frozen model.
1. Computational cost and object count (W1)
We quantify the additional cost by proposal count. On COCO random-500, the conservative measured average overhead is 0.52/0.11/0.26 s per image (2.53%/0.50%/4.74%) for LLaDA-V/MMaDA/LaViDA, using one H100 80GB at batch size one (Table 6). Vanilla captions contain 5.27/1.36/5.01 evaluator object mentions per image (Table 16).
The breakdown below groups images by distinct source proposals after within-image phrase deduplication. It excludes zero-proposal images (49/211/95 for LLaDA-V/MMaDA/LaViDA); the overall averages above include all 500 images. Added time covers probing, local correction, and acceptance, and overhead is relative to vanilla generation.
In the evaluated ≥5-proposal groups, mean added time is 0.989 s for LLaDA-V and 1.240 s for LaViDA. MMaDA's corresponding result is based on a single image.
Total cost includes triggered regeneration and candidate checks. LaViDA reopens 83/500 responses, compared with 35/500 for LLaDA-V and 21/500 for MMaDA. The final policy uses one initial-state forward per distinct source phrase, reuses repeated-phrase scores, and checks new objects only after structural checks pass; selected spans are reopened jointly. Table 6's conservative timing includes additional sparse probe trajectories, as documented in the runtime appendix; the final policy uses initial-state scores without decoding these trajectories.