acceptodds
Under review as a conference paper at ICLR 2027

When Delimiters Mislead: Visual-Text Regrounding for Multi-Image Hallucination Mitigation

Abstract

Multimodal large language models (MLLMs) are increasingly capable of reasoning over multiple images, but this capability also introduces new hallucination risks. In this paper, we reveal that delimiter tokens designed to separate multiple images are associated with two distinct internal patterns on the visual and textual sides: Visual-Delimiter Leakage and Text-Delimiter Sink, both of which contribute to multi-image hallucination. To address these issues, we propose a training-free method called Delimiter-Aware Regrounding (DAR). DAR consists of two modules corresponding to the two internal patterns: Delimiter Ownership Correction (DOC) mitigates Visual-Delimiter Leakage by restoring visual-token ownership, while Visual Evidence Redirection (VER) strengthens image-specific visual grounding to alleviate Text-Delimiter Sink. Extensive experiments on multi-image benchmarks across multiple MLLMs show that DAR effectively enhances multi-image reasoning while preserving the structural role of delimiter tokens in separating visual inputs. Moreover, DAR provides a new perspective on hallucination mitigation in broader multi-input scenarios, such as video reasoning.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multi-Image Hallucination, Multimodal Large Language Model, delimiter token

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_AmX3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates two delimiter-related attention patterns in multi-image multimodal large language models: visual tokens attending to delimiters associated with other images (Visual-Delimiter Leakage), and text tokens concentrating image-related attention on delimiters, with relatively limited attention to visual tokens (Text-Delimiter Sink). The authors introduce VDLS and TDSS to quantify these patterns and propose a training-free method, Delimiter-Aware Regrounding (DAR).

DAR consists of two modules. Delimiter Ownership Correction (DOC) reallocates part of the attention assigned to non-own image delimiters to the corresponding own-image delimiters while preserving the total delimiter-attention mass. Visual Evidence Redirection (VER) selects high-TDSS attention heads and redirects part of their delimiter contributions toward image-local visual evidence constructed using an ownership prior. The paper evaluates DAR on four multi-image benchmarks across eight model configurations from four model families, and provides component ablations, attention analyses, inference-overhead measurements, and additional video hallucination experiments on VidHalluc.

Critical strengths
  1. The method provides concrete, connected interventions for two distinct parts of the computation.

    DOC adjusts the delimiter-based ownership of visual tokens, while VER redirects delimiter contributions toward image-local evidence in selected attention heads. The explicit attention-mass constraint and ownership prior provide a substantive design beyond uniform delimiter scaling. The component ablations offer initial support for the individual contributions and combined effectiveness of the two modules.

  2. The improvements are consistent across models and tasks and have practical value.

    Table 1 covers eight configurations from four model families. DAR improves accuracy in 31 of the 32 model–benchmark combinations and matches the baseline in the remaining combination, with a maximum absolute gain of 5.40 percentage points. Its training-free nature and supplementary VidHalluc results further support its practical relevance. The appendix also reports inference overhead: latency increases by approximately 17%–19% in the two measured settings, while peak GPU memory remains essentially unchanged.

Critical weaknesses
  1. Evaluation on dedicated multi-image hallucination benchmarks is missing.

    The main multi-image experiments use Mantis, BLINK, MIRB, and QBench2, which primarily evaluate general multi-image understanding. Appendix E acknowledges that task accuracy serves as a proxy for hallucination-related failures, but improvements on these benchmarks alone do not establish reductions in the specific hallucinations targeted by DAR. The VidHalluc experiments provide useful additional evidence, but concern video hallucination rather than a dedicated multi-image hallucination evaluation.

    The paper would benefit from evaluation on dedicated benchmarks such as MIHBench or MIOH, the latter of which is already cited in the manuscript. Reporting results by hallucination category would help determine whether DAR reduces errors involving object existence, counting, attributes, or cross-image consistency, providing more direct support for its central claim.

  2. Comparisons with two closely related baselines, FOCUS and DAB, are missing.

    The main external comparison is limited to DTS. FOCUS is a training-free method that directly targets cross-image information leakage through image-wise focused decoding. DAB (Dynamic Attention Balancing) is also training-free and mitigates multi-image hallucinations by redistributing attention across images. Both are closely related to DAR's objective and provide important comparisons for assessing its incremental contribution.

Questions for authors
  1. Could you evaluate DAR on a dedicated multi-image hallucination benchmark, such as MIHBench or MIOH, and report results by hallucination category to examine whether the improvements on general multi-image benchmarks translate into reductions in the targeted hallucinations?

  2. Could you compare DAR with FOCUS and DAB on mutually supported models under the same evaluation protocol, reporting both performance and inference overhead to establish the relative benefits and computational trade-offs?

Reviewer_baqqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work investigates multi-image hallucinations in Multimodal LLMs (MLLMs) through the lens of image delimiter tokens, identifying two attention patterns: visual tokens attending to delimiters of other images (Visual-Delimiter Leakage) and text tokens over-attending to delimiters instead of visual patches (Text-Delimiter Sink). The authors propose Delimiter-Aware Regrounding (DAR), to address both issues in a training free way at inference time. The method identifies samples and tokens where the leakage is happening and redistributes the attention mass to other tokens. Evaluated across four MLLM families (Qwen3.5, Qwen2.5-VL, InternVL3, and LLaVA-OneVision) on four multi-image benchmarks (Mantis, BLINK, MIRB, QBench2) as well as a video hallucination benchmark (VidHalluc), DAR consistently improves accuracy over the base models with no extra GPU memory and modest latency overhead.

Critical strengths
  • Interesting findings: The intuition that image delimiters are correlated to hallucination is novel to me and quite interesting. The paper sits nicely at the intersection of recent works on attention sinks and cross image information leakage, somehow tying the two worlds together.

  • Training-free and broadly effective: The method requires no fine-tuning, no extra memory, adds only modest inference latency, and brings consistent gains across several benchmarks and models.

Critical weaknesses

A. Missing diagnostic experiments: The claim that the two scores defined in Section 3 could be used to identify hallucinations could have been validated experimentally. Figure 1 is the only experimental support for the claim in the current draft. I would have liked something more explicit showing whether either of the scores can be predictive of the presence / absence of an hallucination when thresholded.

B. Missing exploration of why leakage happens: The paper treats both delimiter patterns purely as attention symptoms without investigating their cause. Some analysis in this direction would have strengthened the paper and better motivate the proposed attention intervention method. For example if cross image leakage is bad in absolute, why not simply masking attention among image tokens? And since we are dealing with causal LLM does it mean that the order in which images are presented to the model matter? (as they might or might not generate the identified attention sinks)

C. Limited baseline comparisons: Table 1 only compares against a single external method (DTS), which in the authors' reproduction degrades performance below the vanilla baseline in 25 out of 32 settings. Other closely related t methods discussed in Related Work such as VAR (Kang et al., 2025) and FOCUS (Park et al., 2025) could have been included in the comparison to strengthen the claims. Or a motivation on why this is not possible should have been provided.

Questions for authors
  1. Given that the VDLS difference between correct and incorrect samples in Figure 1(a) is only 0.001–0.004, can you provide stronger quantitative evidence (e.g., correlation or AUROC across benchmarks) showing that VDLS and TDSS reliably predict hallucinations?

  2. Have you tested stronger or binarized masking (e.g., g_i(v) = 1 to completely block visual tokens from attending to other images' delimiters)? Why is that not a viable solution?

Reviewer_x9C6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how image delimiter tokens affect multi-image reasoning in multimodal large language models. The authors identify two attention patterns: Visual-Delimiter Leakage and Text-Delimiter Sink.Based on this analysis, the paper proposes DAR. DAR corrects the delimiter ownership of visual tokens and redirects delimiter-related contributions toward image-specific visual evidence. The method is training-free and operates during inference.Experiments cover eight model configurations and four multi-image benchmarks. DAR improves over the original model in 31 of 32 settings. The paper also reports an extension to video hallucination and several component-level ablations.

Critical strengths
  1. The paper offers a useful delimiter-centered view of multi-image reasoning. Previous work has already connected delimiter tokens with cross-image information leakage. This paper develops that direction further by separating visual-side ownership leakage from text-side delimiter reliance. The distinction between VDLS and TDSS is clear, and the resulting intervention is closely connected to the diagnostic analysis.
  2. The empirical improvements are consistent across models and benchmarks. DAR is evaluated on eight configurations from Qwen3.5, Qwen2.5-VL, InternVL3, and LLaVA-OneVision. The method improves the baseline in 31 of 32 multi-image settings. The paper also includes module ablations, head-selection controls, an ownership-prior analysis, and an efficiency evaluation. These experiments make the practical behavior of the method relatively easy to understand.
Critical weaknesses
  1. The four main multi-image experiments do not directly measure hallucination. Mantis, BLINK, MIRB, and QBench2 primarily evaluate multi-image perception and reasoning. The paper treats incorrect predictions as hallucination-related failures, but these errors may also come from perception, reasoning, or instruction following. The same proxy is used to divide samples into “hallucinated” and correct groups in Figure 1. Therefore, the main results show that DAR improves multi-image task accuracy, but they do not isolate hallucination reduction from general error correction. VidHalluc provides some direct evidence, but the experiment covers only Qwen3.5-2B and includes no competing mitigation method.
  2. The evidence for the proposed mechanism and DAR’s advantage over alternative methods remains incomplete. Figure 1 shows higher VDLS for incorrect samples, but the paper reports no uncertainty estimates or statistical tests. TDSS is not compared between correct and incorrect samples. The component ablations and head-selection controls provide useful supporting evidence. However, DAR directly modifies the attention patterns used to define VDLS and TDSS, so the resulting score reductions do not by themselves establish that these patterns cause the errors. Controls based on shuffled ownership assignments or matched attention redistribution would provide stronger evidence.
Questions for authors
  1. How many of the incorrect examples in Mantis are genuine unsupported or hallucinated outputs, rather than perception or reasoning errors?
  2. Can the authors evaluate DAR on MIOH or another dedicated multi-image hallucination benchmark?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.