acceptodds
Under review as a conference paper at ICLR 2027

GROUNDING BEFORE SPEAKING: CROSS-MODAL REFLECTIVE ROUTING FOR HALLUCINATION-RESILIENT MULTIMODAL LARGE LANGUAGE MODELS

Abstract

Multimodal large language models (MLLMs) have rapidly improved on visual question answering, multimodal dialogue, document understanding, and scien tific reasoning, yet they remain vulnerable to hallucination: fluent outputs that are weakly supported by visual evidence. Existing mitigation methods often rely on retrieval, expensive preference optimization, or post-hoc checking, leav ing the autoregressive decoding process itself insufficiently grounded. We pro pose Cross-Modal Reflective Routing (CMRR), a lightweight framework that estimates token-level grounding uncertainty, dynamically routes computation be tween visual and textual streams, and verifies candidate tokens before commit ment. CMRR introduces a reflective uncertainty estimator, a differentiable cross modal router, and a semantic verifier trained with contrastive grounding super vision. The resulting model suppresses visually unsupported tokens while pre serving linguistic fluency. We present a complete training objective, decoding algorithm, and reproducibility package with implementation, visualizations, and evaluation scripts. In pilot benchmark tables prepared for reproducible experi mentation, CMRR-style routing improves hallucination-sensitive metrics across MMHalBench, POPE, HallusionBench, ScienceQA, and MMBench while adding modest decoding overhead. The paper highlights a practical direction for control lable, uncertainty-aware MLLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Hallucination Mitigation, Cross‑Modal Routing, Grounded Generation, Vision‑Language Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_adMMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary
  1. This paper studies hallucinations in multimodal large language models. It proposes the CMRR framework to address descriptions that contradict the facts because of the model's excessive reliance on language priors.
  2. The core method estimates uncertainty at each generation step, dynamically adjusts the fusion weights of visual and textual representations, and verifies candidate tokens before output. The training objective combines language modeling, routing supervision, contrastive verification, and calibration losses.
  3. The experimental results indicate that CMRR achieves better performance on multiple hallucination-related benchmarks while also improving accuracy on scientific question answering and general multimodal tasks.
Critical strengths
  1. The method reduces model hallucinations by introducing additional loss constraints and a routing module, which is a reasonable approach.
  2. The framework introduces few additional parameters, keeping its practical deployment cost low and making it feasible to implement.
Critical weaknesses
  1. The experimental reporting raises concerns about reproducibility and internal consistency. Detailed settings for reproduction are not provided, and the pretrained LLM used as the decoder is not specified. In addition, in Figure 2, the visual and textual weights for the token "The" are 0.15 and 0.99, respectively. These values do not satisfy the complementary relationship between alphatalpha_t and 1−alphat1 - alpha_t indicated in the figure.
  2. The experiments do not include comparisons with the latest methods or established methods specifically designed to address hallucinations in multimodal models, such as AIR [1] and OPERA [2].

[1] https://proceedings.iclr.cc/paper_files/paper/2026/hash/83749a89d03df30c66af88020e4c7892-Abstract-Conference.html

[2] https://openaccess.thecvf.com/content/CVPR2024/html/Huang_OPERA_Alleviating_Hallucination_in_Multi-Modal_Large_Language_Models_via_Over-Trust_CVPR_2024_paper.html

Questions for authors
  1. Could the authors provide comparisons with the latest methods?
  2. Could the authors provide more detailed experimental settings?
Reviewer_MsXUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CMRR to reduce hallucinations in MLLMs by combining token-level uncertainty estimation, learned visual-textual fusion, and a verifier that filters candidate tokens during decoding. It reports improvements across five benchmarks. Compared with the listed InternVL2 baseline, MMHal hallucination rate decreases from 16.3 to 14.1, and the HallusionBench score increases from 67.2 to 71.6.

Critical strengths

Verifying candidate tokens before including them in the answer is a reasonable approach to preventing unsupported content from influencing subsequent generation. The reported improvements on hallucination benchmarks are encouraging, particularly the 4.4 point gain on HallusionBench. The method also reports improvements on ScienceQA and MMBench

Critical weaknesses

The backbone used for CMRR is not clearly specified. Section 4.3 only mentions an EVA/CLIP-style encoder and a decoder-only LLM. Without the exact model and a comparison against the same backbone without CMRR under matched training conditions, it is hard to tell whether the gains come from the proposed method or differences in the backbone and training.

Some important implementation details are unclear: how the visual and textual features are extracted, how the routing targets and token-level verifier labels are constructed, and whether the fused features change the decoder predictions or only feed the verifier. These details are needed to understand and reproduce the method.

The routing ablation is only reported on “ScienceQA-style validation.” Full CMRR reaches 93.0 vs. 91.2 with static routing, which is a positive result, but it does not directly show that dynamic routing reduces hallucinations. It would be good to see the same comparison on MMHal, POPE, and HallusionBench, keeping the backbone, verifier, and training data fixed.

Questions for authors

Please address the questions raised in the weaknesses section. In particular, please clarify the implementation details and, if possible, provide additional ablation studies on the routing component.

Reviewer_VjSaOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2(+1)
Soundness
1
Read review
Paper summary

The paper proposes Cross-Modal Reflective Routing (CMRR), a framework intended to mitigate hallucinations in multimodal large language models by estimating token-level grounding uncertainty, dynamically routing information between visual and textual streams, and verifying candidate tokens before they are committed to the generated sequence. The method consists of an uncertainty estimator combining token entropy, probability margin, and a learned grounding confidence; a sigmoid-based cross-modal gate; and a semantic verifier that scores the cosine similarity between a fused multimodal representation and a candidate token embedding.

Critical strengths
  1. Reducing visually unsupported content during multimodal generation is a relevant objective, and intervening during decoding is a potentially useful direction.
Critical weaknesses
  1. The paper does not report uncertainty estimates, confidence intervals, or repeated-run variation for the main benchmark results. It does not sufficiently specify the exact backbone and checkpoint used for CMRR, the training and evaluation splits, the source and scale of the verifier training data, or the full procedure used to label grounded and hallucinated tokens.

  2. The robustness section claims benefits under blur, noise, and occlusion, but the submission does not provide a sufficiently detailed quantitative table, perturbation strengths, or evaluation protocol to substantiate those claims. Similarly, the latency results are not accompanied by enough implementation and hardware detail to establish that the overhead comparison is reproducible or representative.

  3. The decoding procedure is underspecified and may behave unpredictably when no candidate passes verification. Algorithm 1 says that, if no candidate passes the threshold, the model regenerates using a 'visually cautious prompt state'. The paper does not define this state, explain how it modifies the next decoder call, specify whether the candidate set is expanded, or give a termination and fallback rule.

Questions for authors
  1. What differentiable calibration objective is actually optimized in place of, or in addition to, the ECE-style expression in Equation (14)? Please specify all loss weights and optimization details.

  2. Which exact base MLLM and checkpoint produce the CMRR results in Table 1? Can the authors report a matched comparison on the same backbone, data, decoding configuration, and evaluation split?

  3. What exact procedure is used when no candidate passes verification, and how are threshold feasibility, termination, and fallback behavior handled?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.