- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TIImage, a training-free framework for mitigating hallucinations in Multimodal Large Language Models (MLLMs) by addressing modality misalignment and statistical co-occurrence biases. Specifically, TIImage renders textual instructions into an extended region at the bottom of the original image to facilitate visual-textual integration. It then employs contrastive decoding between the unified input and a visually degraded negative view to suppress biased predictions. Furthermore, the authors introduce Jensen–Shannon Divergence (JSD)-guided contrastive gating and entropy-aware dynamic probability truncation to avoid excessive penalties on valid tokens.
Critical strengths
-
The proposed framework is training-free and practically applicable.
-
The proposed method offers a novel perspective on hallucination mitigation. Embedding textual instructions directly into the visual input is an interesting and relatively unexplored direction in MLLM hallucination mitigation.
Critical weaknesses
-
The underlying hallucination mitigation mechanism is not sufficiently validated. Although the ablation studies demonstrate the effectiveness of the proposed framework, they do not fully explain how its components contribute to hallucination suppression. In particular, it remains unclear whether the blurred negative branch effectively isolates co-occurrence biases and whether JSD-guided gating can reliably identify hallucination-prone predictions.
-
The proposed framework lacks sufficient theoretical analysis and justification. Although numerous training-free hallucination mitigation methods have been proposed, many lack rigorous mathematical explanations of their underlying mechanisms, and this paper shares a similar limitation. In particular, the authors do not sufficiently explain from a theoretical perspective why the proposed input integration and dynamic contrastive decoding mechanisms should effectively suppress hallucinations.
-
The comparisons with existing hallucination mitigation methods are relatively limited. The experimental comparisons mainly focus on contrastive decoding methods, such as VCD and ICD, along with Marine. However, other representative training-free approaches, including attention intervention and hidden-state enhancement methods, are not considered. Including these methods would provide a more comprehensive evaluation of TIImage against different hallucination mitigation paradigms.
Questions for authors
-
Could the authors further clarify the relationship between Text-in-Image integration and contrastive decoding? Does embedding textual instructions into images directly improve the effectiveness of contrastive decoding, and is there a meaningful interaction between these two components?
-
Could the authors clarify the experimental discrepancies and provide additional controlled evaluations? For instance, Table 1 reports an adversarial POPE accuracy of 86.80 for TIImage on Qwen2.5-VL, whereas Table 4 reports 85.43 with the same baseline accuracy of 81.86. What accounts for this discrepancy? Additionally, could the authors provide results under matched decoding configurations, multiple random seeds, and varying instruction lengths to further validate the robustness of the proposed method?
3 comments
- Comparison with Attention Intervention (OPERA) and Hidden-State Intervention (Activation Steering)
Following your suggestion, we evaluated representative training-free methods from attention intervention (OPERA, Huang et al., CVPR 2024) and hidden-state intervention (Activation Steering, Su et al., ACL 2025) on Qwen2.5-VL and InternVL2.5 under matched greedy settings.
Qwen2.5-VL-7B Cross-Paradigm Comparison:
| Method | POPE Acc / F1 | POPE Rec | CHAIR First-60 Words / | CHAIR Rec / F1 | Latency |
|---|---|---|---|---|---|
| Vanilla Baseline (Standard) | 84.00 / 81.74 | 71.60% | 29.60 / 20.16 | 72.75 / 71.68 | 133 ms |
| ICD (Instruction CD) | 83.00 / 80.64 | 70.00% | 29.00 / 20.02 | 74.66 / 72.30 | 418 ms |
| VCD (Visual CD) | 85.00 / 83.44 | 75.60% | 32.60 / 21.53 | 76.23 / 72.29 | 408 ms |
| OPERA (Attention Intervention) | 83.80 / 81.46 | 71.20% | 26.20 / 17.86 | 73.98 / 73.85 | 204 ms |
| Activation Steering (Hidden-State) | 85.00 / 83.44 | 75.60% | 32.20 / 20.72 | 74.11 / 72.05 | 338 ms |
| TIImage (Unified + Dynamic CD) | 85.60 / 84.48 | 78.40% | 27.40 / 18.68 | 76.02 / 72.54 | 366 ms |
InternVL2.5-8B Cross-Paradigm Comparison:
| Method | POPE Acc / F1 | CHAIR First-100 Words / | CHAIR Rec / F1 | Latency |
|---|---|---|---|---|
| Vanilla Baseline (Standard) | 85.60 / 85.37 | 47.20 / 23.75 | 73.98 / 74.05 | 337 ms |
| ICD (Instruction CD) | 80.33 / 80.94 | 43.00 / 23.99 | 73.37 / 73.82 | 680 ms |
| VCD (Visual CD) | 81.56 / 82.68 | 50.00 / 25.68 | 76.57 / 74.44 | 690 ms |
| OPERA (Attention Intervention) | 85.40 / 85.19 | 43.60 / 22.57 | 72.82 / 74.24 | 608 ms |
| Activation Steering (Hidden-State) | 84.60 / 84.69 | 44.80 / 22.68 | 73.43 / 74.58 | 705 ms |
| TIImage (Unified + Dynamic CD) | 87.33 / 86.69 | 42.80 / 22.22 | 74.05 / 74.58 | 656 ms |
TIImage achieves the strongest overall performance across POPE and CHAIR, and because it operates on the input canvas and output logits, it is orthogonal to internal hidden-state interventions.
- Reconciling Table 1 (86.80) vs. Table 4 (85.43), Matched Decoding, 5 Seeds, and Instruction Lengths
3.1. Why Table 1 reports 86.80% while Table 4 reports 85.43% (with the same 81.86% baseline)
Tables 1, 3, and 4 all evaluate Qwen2.5-VL on the same POPE-Adversarial split (Regular baseline: 81.86% Acc / 78.73% F1). Table 3 first ablates visual text placement without contrastive decoding (selecting Bottom at 84.53% / 83.55%), and Table 4 builds on that same Bottom checkpoint to ablate the decoding strategy. In the original draft, Table 4 inadvertently reported the Static CD configuration from Sec. 3.3–3.4 (85.43% / 84.96% at fixed alpha=0.6, beta=0.1, matching the alpha=0.6 peak in Fig. 5a), whereas Table 1 reported the full Sec. 3.5 Dynamic CD variant with JSD gating and entropy truncation (86.80% / 85.92%, an extra +1.37% gain).
3.2. Matched Decoding, 5 Random Seeds (42, 100, 1234, 2026, 3407), and Instruction Lengths (Short <8 words / Medium ~15 words / Long 45+ words with adaptive multi-line bottom padding):
Table A: Matched Decoding & 5-Seed Sampling on POPE-Adversarial (n=500)
| Metric (Vanilla / VCD / Ours) | Qwen2.5-VL | Qwen3-VL |
|---|---|---|
| Unified Greedy Acc (%) | 84.0 / 85.0 / 85.6 | 83.0 / 83.2 / 83.0 |
| Unified Greedy F1 (%) | 81.7 / 83.4 / 84.5 | 82.5 / 83.2 / 83.0 |
| 5-Seed Sampling Acc (%) | 83.8 / 84.0 / 84.32 | 83.8 / 83.4 / 83.68 |
| 5-Seed Sampling F1 (%) | 81.8 / 82.4 / 83.01 | 83.4 / 83.4 / 83.60 |
| 5-Seed F1 Std Dev | ±0.72 / ±0.49 / ±0.37 | ±1.11 / ±0.65 / ±0.37 |
Table B: Robustness Across Instruction Lengths (Vanilla -> Ours, F1 %)
| Instruction Length | Qwen2.5-VL (POPE) | Qwen2.5-VL (CHAIR 100w / Full) | Qwen3-VL (POPE) |
|---|---|---|---|
| Short (<8 words) | 79.81 -> 80.98 (+1.2) | 71.8 -> 72.6 / 72.6 -> 73.9 | 82.25 -> 82.16 (-0.1) |
| Medium (Default) | 81.74 -> 84.48 (+2.7) | 67.1 -> 68.7 / 71.7 -> 72.5 | 82.47 -> 82.97 (+0.5) |
| Long (45+ words) | 81.57 -> 80.98 (-0.6) | 66.0 -> 67.5 / 72.6 -> 72.9 | 82.60 -> 83.16 (+0.6) |
Under both deterministic greedy decoding and 5-seed stochastic sampling, TIImage achieves the highest F1 and lowest cross-seed variance (±0.37) on both backbones. Across short, medium, and long instructions (with adaptive multi-line bottom padding), TIImage maintains consistent F1 gains on CHAIR and preserves stable POPE performance.
Thank you for the constructive feedback and for recognizing the novelty of embedding textual instructions directly into the visual input space. Below, we address your comments regarding our theoretical justification, component synergy, comparisons with attention and hidden-state intervention methods, and controlled robustness evaluations.
In standard dual-stream MLLMs, visual tokens v and text tokens x lie in misaligned subspaces (cosine similarity ~0.22 in Fig. 6a). When fine-grained visual grounding is weak, the posterior pθ(yt∣v,x,y<t) collapses toward the marginal language co-occurrence prior pθ(yt∣x,y<t). Rendering T into the non-overlapping bottom padding Vtext=R(V,T) allows ViT self-attention to bind visual text patches and visual object patches before projection into the LLM (vtext, cosine similarity ~0.78).
In log-probability space, our dynamic contrastive objective (Eqs. 6–8) can be written as:
logpTCD(yt)=logpθ(yt∣vtext,x,y<t)+αt⋅log[pθ(yt∣vtext,x,y<t)/pθ(yt∣v~,x,y<t)], subject to yt∈Vtop(Ht).
By Bayes' rule, the log-ratio term estimates the Conditional Pointwise Mutual Information PMI(yt;vtext∣v~,x) between candidate token yt and the intact visual-instruction signal vtext conditioned on the blurred view v~. When DJSD(punif∣∣pneg)<=τ, visual perturbation does not alter the prediction (e.g., syntactic function words), so αt=0 preserves natural fluency. When DJSD>τ, activating αt>0 cancels the shared language prior pθ(yt∣x,y<t) while amplifying tokens that have high mutual information with both the visual objects and the rendered visual instruction. Finally, entropy-aware truncation βt(Ht) bounds candidate selection within a confidence set Vtop(Ht) that tightens as predictive uncertainty Ht grows.
We also conducted both a 2x2 component decoupling study (on POPE-Adversarial samples and MME) and direct token-level mechanism checks (on 500 CHAIR images and POPE splits):
Together, these two tables confirm the synergy of our components and their underlying mechanisms. In the first Table, while Dynamic CD alone boosts MME by +89.5(1517.0→1606.5), it slightly raises CHAIR-60 CHAIRs (29.6; pairing it with Text-in-Image in Full TIImage nearly doubles the MME gain to +176.1 (1693.1), achieves the top POPE F1 (86.13%), and drops CHAIRs/CHAIRi to 27.4%/18.7%, while entropy truncation and JSD gating progressively resolve Static CD's over-penalization (1397.1). Table 2 validates why this works at a token level: hallucinated tokens show +32.2% higher entropy Ht than faithful ones (1.72 vs. 1.30, p=5.9×10−11), with high-entropy tokens (Ht≥1.2) suffering a 35.69% hallucination rate versus 22.02% at low entropy. Meanwhile, across POPE Random, Popular, and Adversarial splits, mean DJSD (0.118→0.129→0.131) and gate trigger rates (54.6 rise monotonically, with co-occurrence False Positives showing 1.82× higher DJSD than True Negatives (0.166 vs. 0.091). Finally, visual instruction rendering increases decoder attention to original image patches Vorigin from 7.69% to 10.90% (+41.7%), confirming stronger visual grounding rather than distraction.