acceptodds
Under review as a conference paper at ICLR 2027

TIImage: Mitigating MLLM Hallucinations via Unified Contrastive Decoding

Abstract

Despite the success of Multimodal Large Language Models (MLLMs), hallucinations severely undermine their reliability, largely driven by modality misalignment. Existing training-based solutions incur heavy fine-tuning overhead to realign representations, whereas training-free alternatives leave this modality gap unaddressed. Inspired by human perception that unifies text and vision, we propose a novel training-free framework, Text in Image (TIImage). TIImage renders textual instructions directly into image pixels, bridging the modality gap at the input level. Furthermore, TIImage pairs this unified input with a dynamic contrastive decoding scheme against an intentionally perturbed view to isolate statistical biases, using an adaptive regulation mechanism based on distributional divergence to prevent over-penalization. Extensive experiments across multiple benchmarks demonstrate that TIImage consistently suppresses hallucinations while maintaining fine-grained visual perception.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Hallucinations;MLLM;Contrastive decoding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wxk4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TIImage, a training-free framework for mitigating hallucinations in Multimodal Large Language Models (MLLMs) by addressing modality misalignment and statistical co-occurrence biases. Specifically, TIImage renders textual instructions into an extended region at the bottom of the original image to facilitate visual-textual integration. It then employs contrastive decoding between the unified input and a visually degraded negative view to suppress biased predictions. Furthermore, the authors introduce Jensen–Shannon Divergence (JSD)-guided contrastive gating and entropy-aware dynamic probability truncation to avoid excessive penalties on valid tokens.

Critical strengths
  1. The proposed framework is training-free and practically applicable.

  2. The proposed method offers a novel perspective on hallucination mitigation. Embedding textual instructions directly into the visual input is an interesting and relatively unexplored direction in MLLM hallucination mitigation.

Critical weaknesses
  1. The underlying hallucination mitigation mechanism is not sufficiently validated. Although the ablation studies demonstrate the effectiveness of the proposed framework, they do not fully explain how its components contribute to hallucination suppression. In particular, it remains unclear whether the blurred negative branch effectively isolates co-occurrence biases and whether JSD-guided gating can reliably identify hallucination-prone predictions.

  2. The proposed framework lacks sufficient theoretical analysis and justification. Although numerous training-free hallucination mitigation methods have been proposed, many lack rigorous mathematical explanations of their underlying mechanisms, and this paper shares a similar limitation. In particular, the authors do not sufficiently explain from a theoretical perspective why the proposed input integration and dynamic contrastive decoding mechanisms should effectively suppress hallucinations.

  3. The comparisons with existing hallucination mitigation methods are relatively limited. The experimental comparisons mainly focus on contrastive decoding methods, such as VCD and ICD, along with Marine. However, other representative training-free approaches, including attention intervention and hidden-state enhancement methods, are not considered. Including these methods would provide a more comprehensive evaluation of TIImage against different hallucination mitigation paradigms.

Questions for authors
  1. Could the authors further clarify the relationship between Text-in-Image integration and contrastive decoding? Does embedding textual instructions into images directly improve the effectiveness of contrastive decoding, and is there a meaningful interaction between these two components?

  2. Could the authors clarify the experimental discrepancies and provide additional controlled evaluations? For instance, Table 1 reports an adversarial POPE accuracy of 86.80 for TIImage on Qwen2.5-VL, whereas Table 4 reports 85.43 with the same baseline accuracy of 81.86. What accounts for this discrepancy? Additionally, could the authors provide results under matched decoding configurations, multiple random seeds, and varying instruction lengths to further validate the robustness of the proposed method?

3 comments

Thank you for the constructive feedback and for recognizing the novelty of embedding textual instructions directly into the visual input space. Below, we address your comments regarding our theoretical justification, component synergy, comparisons with attention and hidden-state intervention methods, and controlled robustness evaluations.

  1. Theoretical justification and synergy between Text-in-Image and Contrastive Decoding

In standard dual-stream MLLMs, visual tokens v and text tokens x lie in misaligned subspaces (cosine similarity ~0.22 in Fig. 6a). When fine-grained visual grounding is weak, the posterior pθ(yt∣v,x,y<t)p_{\theta}(y_{t} | v, x, y_{<t}) collapses toward the marginal language co-occurrence prior pθ(yt∣x,y<t)p_{\theta}(y_{t} | x, y_{<t}). Rendering T into the non-overlapping bottom padding Vtext=R(V,T)V_{text} = R(V, T) allows ViT self-attention to bind visual text patches and visual object patches before projection into the LLM (vtextv_{text}, cosine similarity ~0.78).

In log-probability space, our dynamic contrastive objective (Eqs. 6–8) can be written as:

log⁡pTCD(yt)=log⁡pθ(yt∣vtext,x,y<t)+αt⋅log⁡[pθ(yt∣vtext,x,y<t)/pθ(yt∣v~,x,y<t)]\log p_{TCD}(y_{t})= \log p_{\theta}(y_{t} | v_{text}, x, y_{<t}) + \alpha_{t} · \log [ p_{\theta}(y_{t} | v_{text}, x, y_{<t}) / p_{\theta}(y_{t} | \tilde{v}, x, y_{<t}) ], subject to yt∈Vtop(Ht).y_{t} \in V_{top}(H_{t}).

By Bayes' rule, the log-ratio term estimates the Conditional Pointwise Mutual Information PMI(yt;vtext∣v~,x)PMI(y_{t}; v_{text} | \tilde{v}, x) between candidate token yty_{t} and the intact visual-instruction signal vtextv_{text} conditioned on the blurred view v~\tilde{v}. When DJSD(punif∣∣pneg)<=τD_{JSD}(p_{unif} || p_{neg}) <= \tau, visual perturbation does not alter the prediction (e.g., syntactic function words), so αt=0\alpha_{t} = 0 preserves natural fluency. When DJSD>τD_{JSD} > \tau, activating αt>0\alpha_{t} > 0 cancels the shared language prior pθ(yt∣x,y<t)p_{\theta}(y_{t} | x, y_{<t}) while amplifying tokens that have high mutual information with both the visual objects and the rendered visual instruction. Finally, entropy-aware truncation βt(Ht)\beta_{t}(H_{t}) bounds candidate selection within a confidence set Vtop(Ht)V_{top}(H_{t}) that tightens as predictive uncertainty HtH_{t} grows.

We also conducted both a 2x2 component decoupling study (on POPE-Adversarial samples and MME) and direct token-level mechanism checks (on 500 CHAIR images and POPE splits):

Component VariantQwen2.5-VL POPE (Acc/F1)InternVL2.5 MMEQwen2.5-VL CHAIR First-60 Words CHAIRsCHAIR_{s}/CHAIRiCHAIR_{i}
1. Baseline (Vanilla)86.53/85.221517.029.6/20.2
2. Blank Padding Only86.27/84.881515.6 (-1)30.0/20.3
3. TIImage w/o TCD (Text-in-Image)86.50/85.451463.2 (-54)29.0/20.1
4. Dynamic CD (w/o Text-in-Image)86.70/85.671606.5 (+90)31.2/20.3
5. TIImage (Static CD)86.73/85.941397.1 (-120)29.4/20.1
6. TIImage (Entropy Only)86.80/86.02--28.6/19.8
7. TIImage (JSD Gate Only)86.87/86.05--28.6/19.6
8. Full TIImage (Ours)86.93/86.131693.1 (+176)27.4/18.7
Mechanistic Diagnostic (Qwen2.5-VL)ComparisonKey Finding
Token Entropy HtH_{t} (CHAIR)1.30 vs 1.72+32.2% (p=5.9e-11)
Hallu Rate by HtH_{t} Bin22.0% vs 35.7%+13.67% in high HtH_{t}
Mean JSD across Splits0.118->0.131Gate: 54.6%->62.6%
JSD on Bias Traps (n=3k)0.166 vs 0.0911.82x higher on FP
Cross-Modal Attn (VoriginV_{origin})7.69%->10.90%+41.7% Relative Gain

Together, these two tables confirm the synergy of our components and their underlying mechanisms. In the first Table, while Dynamic CD alone boosts MME by +89.5(1517.0→1606.5)+89.5 (1517.0 \to 1606.5), it slightly raises CHAIR-60 CHAIRsCHAIR_{s} (29.6(29.6% \to 31.2%); pairing it with Text-in-Image in Full TIImage nearly doubles the MME gain to +176.1 (1693.1), achieves the top POPE F1 (86.13%), and drops CHAIRsCHAIR_{s}/CHAIRiCHAIR_{i} to 27.4%/18.7%, while entropy truncation and JSD gating progressively resolve Static CD's over-penalization (1397.11397.1). Table 2 validates why this works at a token level: hallucinated tokens show +32.2% higher entropy HtH_{t} than faithful ones (1.72 vs. 1.30, p=5.9×10−11p = 5.9 \times 10^{-11}), with high-entropy tokens (Ht≥1.2)(H_{t} \ge 1.2) suffering a 35.69% hallucination rate versus 22.02% at low entropy. Meanwhile, across POPE Random, Popular, and Adversarial splits, mean DJSDD_{JSD} (0.118→0.129→0.131)(0.118 \to 0.129 \to 0.131) and gate trigger rates (54.6(54.6% \to 59.8% \to 62.6%) rise monotonically, with co-occurrence False Positives showing 1.82×1.82\times higher DJSDD_{JSD} than True Negatives (0.166 vs. 0.091). Finally, visual instruction rendering increases decoder attention to original image patches VoriginV_{origin} from 7.69% to 10.90% (+41.7%), confirming stronger visual grounding rather than distraction.

  1. Comparison with Attention Intervention (OPERA) and Hidden-State Intervention (Activation Steering)

Following your suggestion, we evaluated representative training-free methods from attention intervention (OPERA, Huang et al., CVPR 2024) and hidden-state intervention (Activation Steering, Su et al., ACL 2025) on Qwen2.5-VL and InternVL2.5 under matched greedy settings.

Qwen2.5-VL-7B Cross-Paradigm Comparison:

MethodPOPE Acc / F1POPE RecCHAIR First-60 Words CHAIRsCHAIR_{s}/CHAIRiCHAIR_{i}CHAIR Rec / F1Latency
Vanilla Baseline (Standard)84.00 / 81.7471.60%29.60 / 20.1672.75 / 71.68133 ms
ICD (Instruction CD)83.00 / 80.6470.00%29.00 / 20.0274.66 / 72.30418 ms
VCD (Visual CD)85.00 / 83.4475.60%32.60 / 21.5376.23 / 72.29408 ms
OPERA (Attention Intervention)83.80 / 81.4671.20%26.20 / 17.8673.98 / 73.85204 ms
Activation Steering (Hidden-State)85.00 / 83.4475.60%32.20 / 20.7274.11 / 72.05338 ms
TIImage (Unified + Dynamic CD)85.60 / 84.4878.40%27.40 / 18.6876.02 / 72.54366 ms

InternVL2.5-8B Cross-Paradigm Comparison:

MethodPOPE Acc / F1CHAIR First-100 Words CHAIRsCHAIR_{s}/CHAIRiCHAIR_{i}CHAIR Rec / F1Latency
Vanilla Baseline (Standard)85.60 / 85.3747.20 / 23.7573.98 / 74.05337 ms
ICD (Instruction CD)80.33 / 80.9443.00 / 23.9973.37 / 73.82680 ms
VCD (Visual CD)81.56 / 82.6850.00 / 25.6876.57 / 74.44690 ms
OPERA (Attention Intervention)85.40 / 85.1943.60 / 22.5772.82 / 74.24608 ms
Activation Steering (Hidden-State)84.60 / 84.6944.80 / 22.6873.43 / 74.58705 ms
TIImage (Unified + Dynamic CD)87.33 / 86.6942.80 / 22.2274.05 / 74.58656 ms

TIImage achieves the strongest overall performance across POPE and CHAIR, and because it operates on the input canvas and output logits, it is orthogonal to internal hidden-state interventions.

  1. Reconciling Table 1 (86.80) vs. Table 4 (85.43), Matched Decoding, 5 Seeds, and Instruction Lengths

3.1. Why Table 1 reports 86.80% while Table 4 reports 85.43% (with the same 81.86% baseline)

Tables 1, 3, and 4 all evaluate Qwen2.5-VL on the same POPE-Adversarial split (Regular baseline: 81.86% Acc / 78.73% F1). Table 3 first ablates visual text placement without contrastive decoding (selecting Bottom at 84.53% / 83.55%), and Table 4 builds on that same Bottom checkpoint to ablate the decoding strategy. In the original draft, Table 4 inadvertently reported the Static CD configuration from Sec. 3.3–3.4 (85.43% / 84.96% at fixed alpha=0.6, beta=0.1, matching the alpha=0.6 peak in Fig. 5a), whereas Table 1 reported the full Sec. 3.5 Dynamic CD variant with JSD gating and entropy truncation (86.80% / 85.92%, an extra +1.37% gain).

3.2. Matched Decoding, 5 Random Seeds (42, 100, 1234, 2026, 3407), and Instruction Lengths (Short <8 words / Medium ~15 words / Long 45+ words with adaptive multi-line bottom padding):

Table A: Matched Decoding & 5-Seed Sampling on POPE-Adversarial (n=500)

Metric (Vanilla / VCD / Ours)Qwen2.5-VLQwen3-VL
Unified Greedy Acc (%)84.0 / 85.0 / 85.683.0 / 83.2 / 83.0
Unified Greedy F1 (%)81.7 / 83.4 / 84.582.5 / 83.2 / 83.0
5-Seed Sampling Acc (%)83.8 / 84.0 / 84.3283.8 / 83.4 / 83.68
5-Seed Sampling F1 (%)81.8 / 82.4 / 83.0183.4 / 83.4 / 83.60
5-Seed F1 Std Dev±0.72 / ±0.49 / ±0.37±1.11 / ±0.65 / ±0.37

Table B: Robustness Across Instruction Lengths (Vanilla -> Ours, F1 %)

Instruction LengthQwen2.5-VL (POPE)Qwen2.5-VL (CHAIR 100w / Full)Qwen3-VL (POPE)
Short (<8 words)79.81 -> 80.98 (+1.2)71.8 -> 72.6 / 72.6 -> 73.982.25 -> 82.16 (-0.1)
Medium (Default)81.74 -> 84.48 (+2.7)67.1 -> 68.7 / 71.7 -> 72.582.47 -> 82.97 (+0.5)
Long (45+ words)81.57 -> 80.98 (-0.6)66.0 -> 67.5 / 72.6 -> 72.982.60 -> 83.16 (+0.6)

Under both deterministic greedy decoding and 5-seed stochastic sampling, TIImage achieves the highest F1 and lowest cross-seed variance (±0.37) on both backbones. Across short, medium, and long instructions (with adaptive multi-line bottom padding), TIImage maintains consistent F1 gains on CHAIR and preserves stable POPE performance.

Reviewer_yeVSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TIImage, a training-free framework for reducing hallucination in multimodal LLMs. The method is motivated by modality misalignment and reliance on learned co-occurrence priors. TIImage first renders the textual prompt into a padded region of the image, thereby injecting instruction information into the visual input. It then performs contrastive decoding using a visually degraded negative view and dynamically controls the contrastive penalty and probability truncation using Jensen-Shannon divergence and predictive entropy. Experiments on POPE, CHAIR, and MME across multiple MLLM backbones report improvements in hallucination-related metrics while maintaining or improving object recall and visual perception.

Critical strengths
  • The method components are reasonably well aligned with the stated motivation: text rendering targets modality misalignment, contrastive decoding targets prior-driven predictions, and dynamic modulation is intended to reduce over-penalization from static decoding.

  • The paper contains useful ablations on visual-text placement, decoding strategy, and dynamic versus static variants.

  • The training-free nature of the method makes it potentially attractive for settings where fine-tuning is expensive or unavailable.

Critical weaknesses
  • The main mechanistic interpretation is not directly established. A large JSD between the unified and perturbed decoding distributions is interpreted as evidence of stronger statistical or co-occurrence bias, but JSD only measures distributional discrepancy. Other differences induced by the two inputs could also produce a large divergence.

  • Similarly, predictive entropy is treated as an indicator of hallucination risk. High entropy is a plausible uncertainty signal, but uncertainty is not equivalent to hallucination, and the paper does not directly validate that high-entropy tokens are disproportionately hallucinated or factually unsupported.

  • The framing around resolving dual-stream modality misalignment is somewhat stronger than the actual implementation. Although text is rendered into the image, explicit textual constraints are still retained during decoding. I therefore view the method more as augmenting the visual stream with textual information than as fully eliminating the dual-stream structure.

  • The feature-alignment analysis is indirect. Increased text–vision cosine similarity demonstrates that the representations become closer, but it does not show that this change causally mediates the reduction in hallucination.

  • The contribution of text-in-image alone is somewhat mixed across experiments. On the Qwen2.5 POPE/CHAIR ablation, TIImage without TCD provides partial gains, whereas in the InternVL2.5 MME attribution experiment, the no-CD variant performs substantially worse than the full dynamic method. This makes it difficult to attribute the final gains solely to the proposed input-unification mechanism.

  • The efficiency claim should be stated carefully. TIImage is somewhat more efficient than several other contrastive-decoding baselines, but inference is still approximately 1.9× slower than vanilla decoding.

  • The empirical scope is concentrated on object hallucination and visual-perception benchmarks. I do not think the current evidence is sufficient to establish the stronger claim that the method is broadly task-agnostic.

Questions for authors

I find the empirical results stronger than the evidence supporting the proposed explanation for why the method works. More direct validation of the JSD/co-occurrence and entropy/hallucination relationships would substantially strengthen the paper.

2 comments

Thank you for your constructive review. We ran a set of token-level and attention-level diagnostics to test those exact mechanisms directly.

  1. Direct validation of Entropy vs. Hallucination and JSD vs. Co-Occurrence Bias

First, to test whether high-entropy tokens are disproportionately hallucinated, we logged the pre-truncation predictive entropy HtH_{t} of every generated object noun token across the 500 CHAIRCHAIR images on Qwen2.5-VL (1,512 object tokens in total) and split them using ground-truth annotations into Faithful Object Tokens (n = 1,068) and Hallucinated Tokens (n = 444):

CHAIR Token GroupCountMean HtH_{t}Hallu Rate by HtH_{t}
Faithful Object Tokens1,0681.2998Low (< 0.5): 22.02%
Hallucinated Tokens4441.7183Med (0.5-1.2): 22.72%
Mann-Whitney pp=5.91e-111,512+32.2%High (>=1.2): 35.69%

Second, to test whether large DJSDD_{JSD} specifically captures statistical co-occurrence bias, we measured DJSDD_{JSD} across the three POPE splits constructed with escalating co-occurrence priors (Random -> Popular -> Adversarial), and compared False Positive (FP) samples (where co-occurrence priors trick the model into answering Yes for a missing object) against True Negative (TN) samples (where the model correctly answers No):

POPE SplitAcc/F1JSDGateFP vs TN JSD
Random (Low, n=500)88.8/87.50.11854.6%0.097 vs 0.063
Popular (Med, n=500)87.0/85.80.12959.8%0.125 vs 0.083
Adversarial (High, n=500)85.6/84.50.13162.6%0.111 vs 0.088
Adversarial (3,000, Ours)86.9/86.10.12460.8%0.166 vs 0.091
Adversarial (3,000, w/o TiI)86.7/85.70.30390.6%0.382 vs 0.224

These two tables confirm both assumptions: (1) Hallucinated object tokens have a +32.2% higher mean entropy than faithful tokens (pp = 5.91e-11), and tokens in the high-entropy regime (HtH_{t} >= 1.2) suffer a 35.69% hallucination rate compared to 22.02% at low entropy. (2) Across Random -> Popular -> Adversarial, both mean DJSDD_{JSD} and the JSD gate trigger rate increase monotonically as co-occurrence bias strengthens, and co-occurrence-driven False Positives exhibit nearly double the JSD of grounded True Negatives (0.1657 vs. 0.0910).

  1. Direct attention-level evidence linking feature alignment to visual grounding

We agree that higher vision-text cosine similarity (Fig. 6) is indirect on its own. To check whether rendering instructions into the bottom bar actually helps the decoder focus on the scene rather than just attending to the rendered text pixels, we measured the fraction of last-layer cross-modal attention allocated specifically to the original image patch tokens VoriginV_{origin} (excluding the bottom padding bar) during CHAIR generation:

Input Configuration (Qwen2.5-VL)Attn to VoriginV_{origin}Relative Gain
Vanilla Baseline (Standard input)0.0769 (7.69%)Baseline
Blank Padding Only (10% white bar)0.0790 (7.90%)+2.7%
TIImage (10% bar + rendered text)0.1090 (10.90%)+41.7%

Adding a blank white border barely shifts visual attention (from 7.69% to 7.90%). In contrast, rendering the instruction onto the bottom border boosts the decoder's attention on the original image patches (VorigV_{orig} ) to 10.90%, achieves a 41.7% relative gain. These results suggest that intra-ViT interactions between the rendered text and object patches effectively anchor the decoder to actual visual objects.

  1. Explaining the mixed effect of Text-in-Image alone across benchmarks

Comparing Qwen2.5-VL POPE/CHAIR (Tables 3–4) and InternVL2.5 MME (Table 6), along with our newly added Dynamic CD Only (w/o Text-in-Image) baseline, clarifies why Text-in-Image alone behaves differently across benchmarks and how it interacts with dynamic decoding.

AblationInternVL2.5 MMEQwen2.5-VL POPE (Acc/F1)Qwen2.5-VL CHAIR First-60 Words CHAIRsCHAIR_{s}/CHAIRiCHAIR_{i}
Baseline (Vanilla)1517.086.53/85.2229.6/20.2
Blank Padding Only1515.6 (-1)86.27/84.8830.0/20.3
Rendered Random Text1524.3 (+7)----
TIImage (No CD / Text-in-Image Only)1463.2 (-54)86.50/85.4529.0/20.1
TIImage (Static CD)1397.1 (-120)86.73/85.9429.4/20.1
Dynamic CD Only (w/o Text-in-Image)1606.5 (+90)86.70/85.6731.2/20.3
Full TIImage (Ours)1693.1 (+176)86.93/86.1327.4/18.7

3.1 Why TIImage (No CD) improves CHAIR/POPE but drops on MME

Notice that Blank Padding (1515.63) and Rendered Random Text (1524.31) barely change MME relative to baseline (1517.00), whereas rendering the actual question in TIImage (No CD) drops to 1463.17. This rules out generic visual distraction and points to task structure that CHAIR uses a neutral open-ended prompt ("Describe this image in detail.") with no false premise, so rendering it purely anchors visual attention (+41.7% to VoriginV_{origin}) and lowers CHAIRiCHAIR_{i} (11.0 -> 9.5). By contrast, MME uses paired Yes/No questions (Acc+) on fine-grained attributes (Color, Count, Position), where 50% of queries contain a false visual premise (e.g., asking if a blue car is red). When a false premise is rendered directly into the canvas without contrastive calibration, the visual/OCR pathway sees the queried attribute word inside the image and exhibits a query-confirmation echo on negative pairs (e.g., MME Color drops from 158.3 to 115.0 in No CD). Moreover, unlike Qwen2.5-VL continuous 2D-RoPE resolution, InternVL2.5 splits images into discrete 448x448 tiles, making uncalibrated single-branch inference more sensitive to the added bottom strip.

3.2 Why Dynamic CD unlocks the full synergy of Text-in-Image (+176.14)

Applying a fixed penalty in Static CD (1397.14) over-penalizes confident tokens. However, once Dynamic CD is enabled, the negative branch (Blur(V), x) shares the same query prior x, allowing the JSD gate to cancel out query-confirmation bias on false premises while preserving the intra-ViT visual-text binding of VtextV_{text} on true visual evidence. Consequently, while Dynamic CD Only (w/o Text-in-Image) reaches 1606.50 (+89.50), combining Text-in-Image with Dynamic CD in Full TIImage jumps to 1693.14 (+176.14 over baseline, and +86.64 above Dynamic CD alone, boosting MME Count to 200.0, Color to 193.3, and Position to 173.3 in Table 8).

  1. Revising dual-stream framing, latency wording, and adding MME-Cognition

We agree with your characterization. Because explicit text tokens x are still fed to the decoder, we will revise the Abstract, Introduction, and Section 3.2 to describe TIImage as augmenting the visual stream with rendered textual instructions to bridge cross-modal misalignment while retaining explicit text tokens for syntactic conditioning, rather than claiming it eliminates the dual-stream structure.

We will revise Section 4.1 and the Conclusion to state clearly that while TIImage is the fastest among dual-branch contrastive methods (1.91x vs. 1.97x–2.08x for ICD/VCD/Marine) and requires no fine-tuning, it still incurs a ~1.9x latency overhead relative to single-branch vanilla decoding.

We softened broadly task-agnostic to architecture-agnostic across visual perception and faithfulness benchmarks, and evaluated TIImage on the 4 higher-level reasoning subtasks of MME-Cognition (Commonsense, Numerical, Translation, Code Reasoning).

Backbone & VariantCom/Num/Trans/CodeCog.Perc.
InternVL2.5: TIImage (No CD)132.9/110/178/163582.861463.2
InternVL2.5: TIImage (Static)133.6/103/178/140553.571397.1
InternVL2.5: Dynamic CD w/o TiI158.6/125/200/155638.571606.5
InternVL2.5: Full TIImage (Ours)153.6/133/200/163648.571693.1
Qwen2.5-VL: Full TIImage (Ours)142.1/133/185/170629.641669.5
Qwen3-VL: Full TIImage (Ours)157.1/170/163/178667.141802.4
Reviewer_GYncOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents TIImage, a training-free approach for reducing hallucinations in MLLMs. The main idea is to write the text instruction inside a padded part of the image, while still keeping the original text input. The authors argue that this can help reduce the gap between visual and text representations.

The method also uses contrastive decoding with a blurred image as the negative input. In addition, it applies a JSD-based gate to control when the contrastive penalty is used and an entropy-based truncation strategy to filter uncertain tokens during generation.

The authors evaluate the method on Qwen2.5-VL, Qwen3-VL, and InternVL2.5 using POPE, CHAIR, MME, and LLaVA-Bench. The reported results show improvements on several hallucination and perception metrics compared with other training-free decoding methods, with about 1.9× inference cost over normal decoding.

Critical strengths
  1. The main idea is simple and practical. TIImage does not need any extra training or changes to the model architecture, so it can be applied to different MLLMs relatively easily.

  2. The paper evaluates the method on several recent MLLMs and different benchmarks. The results are generally positive across different settings, which shows that the method is not only working for one specific model or dataset.

Critical weaknesses
  1. It is not very clear how much of the improvement is actually coming from the Text-in-Image part. The paper does not properly separate this from the effect of dynamic decoding and truncation, and in some ablations Text-in-Image alone even hurts the performance.

  2. I also have concerns about the evaluation and the way the method is explained. Different methods use different decoding settings, and some parts of the method are not described consistently. Because of this, it is difficult to clearly understand which component is giving the reported gains.

Questions for authors

Can you clarify exactly what inputs are used in the two decoding branches? In particular, does the negative branch use the blurred image with the rendered text or without it? Also, is the JSD calculated using the blurred image or the original image?

Can you explain more clearly how much of the improvement is actually coming from the Text-in-Image part? From the current ablations, it is difficult to separate its effect from the dynamic decoding and truncation.

Why are different decoding settings used for TIImage and some of the baselines? Do you expect the same conclusions if all methods are evaluated with the same decoding setup?

Can you clarify the differences between some of the reported POPE numbers across the main results, figures, and ablation tables?

For CHAIR, can you report the average caption length or average number of mentioned objects? This would help clarify whether part of the large recall gain is because the generated captions are longer.

2 comments

Thank you for reading our paper so carefully. You caught several genuine notation and presentation issues in our draft, specifically the missing tilde in Eq. 7, the unexplained difference between Table 1 (86.80%) and Table 4 (85.43%), and the missing Dynamic-CD-only baseline in the ablation tables. We have addressed each of your questions below and will fix all notation issues in the next version of manuscript.

  1. Clarifying exact branch inputs and how JSD is calculated

Both decoding branches receive the exact same tokenized text prompt x so the model keeps its syntactic and formatting constraints. In the positive (unified) branch, the input is (VtextV_{text}, x), where VtextV_{text} is the original image V extended at the bottom by a 10% white padding bar containing the rendered text instruction T (Sec. 3.2). In the negative (degraded) branch, the input is (V~\tilde{V}, x), where V~\tilde{V} = Blur(V, r=7) is the Gaussian-blurred original image V without the rendered text (Sec. 3.3).

Regarding JSD: DJSDD_{JSD} in Eq. 7 is computed directly between the output distributions of these two branches (punifp_{unif} from VtextV_{text} and pnegp_{neg} from V~\tilde{V}). On Line 277 (Eq. 7), writing v instead of v~\tilde{v} in pimgp_{img} = pθ(.∣v,y<t)p_{\theta}(. | v, y_{<t}) was a typesetting typo on our part—Algorithm 1 (Line 3, Appendix A.3) has the intended v_tilde. Because DJSDD_{JSD} reuses the logits already computed by the two branches, we only run 2 forward passes per step (never a third pass on the raw image), which matches the 1.91x latency in Table 5. We have also updated Eqs. 3–7 so the text token sequence x is written explicitly throughout.

To also check whether blurring the rendered text in the negative branch makes any difference, we compared three negative-branch choices across all 3,000 POPE-Adversarial samples on Qwen2.5-VL:

Negative Branch InputAccF1CHAIR First-60 Words CHAIRsCHAIR_{s}/CHAIRiCHAIR_{i}
Unblurred raw image V86.37%85.43%28.4 / 20.0
Blurred VtextV_{text} (image+text)86.67%85.82%27.0 / 18.8
Blurred raw image V (Ours)86.93%86.13%27.4 / 18.7

Contrasting against unblurred V penalizes valid visual features (86.37%), while blurring the rendered text (Blur(VtextV_{text})) introduces smeared character artifacts into the negative branch (86.67%). Blurring only the raw image (Blur(V)) works best (86.93%).

  1. Separating Text-in-Image from Dynamic Decoding and Truncation

You pointed out that Tables 4 and 6 lacked a Dynamic CD + Truncation baseline without Text-in-Image, making it hard to separate the effect of Text-in-Image from dynamic decoding. We ran a full component-by-component ablation on Qwen2.5-VL (all 3,000 POPE-Adversarial samples under matched greedy decoding, plus 60-word length-controlled CHAIR) and added the missing Dynamic CD Only (w/o Text-in-Image) row to Table 6 on InternVL2.5 MME:

Qwen2.5-VL Variant (n=3,000)Acc / F1RecallCHAIR First-60 Words CHAIRsCHAIR_{s}/CHAIRiCHAIR_{i}
Greedy Baseline (Vanilla)86.53/85.2277.7%29.6/20.2
Blank Padding Only86.27/84.8877.1%30.0/20.3
TIImage w/o CD (Text-in-Image Only)86.50/85.4579.3%29.0/20.1
Dynamic CD Only (w/o Text-in-Image)86.70/85.6779.5%31.2/20.3
TIImage (Static CD)86.73/85.9481.1%29.4/20.1
TIImage (Entropy Only)86.80/86.0281.2%28.6/19.8
TIImage (JSD Gate Only)86.87/86.0581.0%28.6/19.6
Full TIImage (Ours)86.93/86.1381.1%27.4/18.7
InternVL2.5 MME Attribution (Table 6)Scorevs. Regular Baseline
Regular Baseline (Sampling)1517.00+0.00
Greedy Baseline (Vanilla)1574.30+57.30
Blank Padding Only1515.63-1.37
Rendered Random Text1524.31+7.31
TIImage (No CD / Text-in-Image Only)1463.17-53.83
TIImage (Static CD, Fixed a=0.6, b=0.1)1397.14-119.86
Dynamic CD Only (w/o Text-in-Image)1606.50+89.50
Full TIImage (Dynamic CD, Ours)1693.14+176.14

These results show why both parts are needed:

Text-in-Image Only cuts CHAIRI\text{CHAIR}_I (11.0 →\to 9.5) but hurts MME via attention competition. Dynamic CD Only raises MME (+89.50) yet worsens CHAIRS\text{CHAIR}_S (29.6% →\to 31.2%). Full TIImage resolves both, lifting MME by +176.14 while cutting CHAIRS\text{CHAIR}_S/CHAIRI\text{CHAIR}_I to 27.4%/18.7%.

  1. Why different decoding settings were used and results under identical setups

    In our initial submission, we ran VCD and ICD with their official code defaults (do_sample=True, temp=1.0) and ran Marine and TIImage with greedy decoding (do_sample=False), which is why we also included a Greedy baseline row in Tables 1, 2, 5, 8, and 9. To make the comparison strictly identical, we re-evaluated Vanilla, ICD, VCD, and TIImage under Unified Greedy (do_sample=False for all methods) and Unified Sampling (do_sample=True, temp=1.0, top_p=0.9 across 5 random seeds: 42, 100, 1234, 2026, 3407) on POPE-Adversarial (n=500):

Method (n=500)Qwen2.5 GreedyQwen2.5 5-Seed AccQwen2.5 5-Seed F1
Vanilla Baseline84.00/81.7483.84 ± 0.5481.81 ± 0.72
ICD (Wang et al.)83.00/80.6482.40 ± 0.7780.32 ± 0.94
VCD (Leng et al.)85.00/83.4484.04 ± 0.3982.43 ± 0.49
TIImage (Ours)85.60/84.4884.32 ± 0.3583.01 ± 0.37

Under both deterministic greedy decoding and 5-seed stochastic sampling (including on Qwen3-VL, where 5-seed F1 reaches 83.60 ± 0.37 vs. 83.40 ± 1.11 for Vanilla and 83.44 ± 0.65 for VCD), TIImage achieves the highest F1 and the lowest variance across seeds.

  1. Clarifying the POPE numbers across Table 1, Table 3/4, Fig. 1, and Fig. 5

Specifically, Tables 1, 3, and 4 all evaluate the exact same POPE-Adversarial split. Table 3 isolates the spatial placement of rendered text without contrastive decoding (establishing Bottom as the best placement), and Table 4 builds on that same Bottom checkpoint to ablate the decoding strategy. In the original draft, Table 4 inadvertently reported the Static CD configuration from Sec. 3.3–3.4 (85.43 / 84.96 at fixed alpha=0.6, beta=0.1, matching the alpha=0.6 peak in Fig. 5a), whereas Table 1 reported the full Sec. 3.5 Dynamic CD variant with JSD gating and entropy truncation. Meanwhile, Fig. 1 Right reflects an early pilot probe used to motivate the method in Sec. 1. In the revised manuscript, we have updated Table 4 to include both the Static CD (85.43%) and Full Dynamic CD (86.80%) rows explicitly and labeled the POPE-Adversarial split in both Table 3 and Table 4 captions.

  1. Average caption length and number of mentioned objects on CHAIR

We computed the average word length per caption (Words), average number of mentioned COCO objects per image (Objs), and fixed-prefix evaluations (truncating all captions to the first 60 or 100 words) across all 500 CHAIR images (CS, CHAIRsCHAIR_{s}; CI, CHAIRiCHAIR_{i}):

MethodCS/CI/RecallWords/ObjsCHAIR First-60 Words CS/CICHAIR First-100 Words CS/CI
Qwen2.5: Vanilla34.2/10.7/53.6159.9/3.0229.6/20.246.4/25.3
Qwen2.5: ICD40.4/11.8/51.8197.2/3.1329.0/20.041.2/23.9
Qwen2.5: VCD33.9/ 9.7/53.0181.6/3.2632.6/21.546.8/26.1
Qwen2.5: TIImage33.6/ 8.6/67.3159.1/2.9728.4/18.642.8/23.1
InternVL2.5: Vanilla34.0/ 7.7/51.1164.4/2.9333.8/20.447.2/23.8
InternVL2.5: VCD34.8/ 8.4/47.4123.6/3.1035.8/21.950.0/25.7
InternVL2.5: TIImage32.4/ 7.7/66.2120.1/2.8933.8/20.844.8/22.9

On Qwen2.5-VL, VCD (181.6 words) and ICD (197.2 words) generate longer captions than TIImage (159.1 words), yet their Recall remains at 53.0% and 51.8% compared to TIImage's 67.3%. On InternVL2.5, TIImage generates shorter captions than Vanilla (120.1 vs. 164.4 words) while boosting Recall from 51.1% to 66.2%. Even when all captions are truncated to the first 60 or 100 words, TIImage achieves the lowest CHAIR_S and CHAIR_I, showing that the recall gain comes from replacing hallucinated objects with grounded ones rather than generating longer text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.