acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Cross-Modal Hallucinations by Correcting Modality Misrouting in Omnimodal LLMs

Abstract

Omnimodal large language models (OLLMs) are susceptible to cross-modal hallucinations under audio-visual interference. We find that, for a substantial fraction of incorrect joint predictions, the question-relevant unimodal branch can still recover the correct answer, suggesting that task-relevant evidence can remain recoverable from the relevant modality even when it is not reflected in the joint prediction. We refer to this recoverable failure pattern as **modality misrouting**. We propose **CaRe (Conflict-aware Residual Intervention)**, a training-free inference-time method for identifying and correcting modality misrouting. CaRe uses a text-only modality probe to identify a candidate modality-specific counterfactual branch, and flags potential misrouting when its prediction conflicts with the original joint prediction. For such cases, CaRe computes a residual direction from the discrepancy between the counterfactual and fused representations and uses it to steer the original multimodal state toward the relevant modality evidence, while retaining both modalities in the original audio-visual inference path. The intervention strength is adapted based on the confidence of the counterfactual branch. Experiments on CMM and AVHBench across two OLLMs show that CaRe consistently outperforms the evaluated inference-time hallucination-mitigation baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Omnimodal Large Language Models, Cross-Modal Hallucination, Inference-Time Intervention

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pgkhOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes a cross-modal hallucination mitigation algorithm that is training-free and it leverages the fact that uni-modal inputs for the prompt-relevant modality contains the information necessary to answer the prompt correctly. The authors justify the motivation to their work empirically and perform experiments on two cross-modal hallucination benchmarks as well as a general omni-understanding benchmark to show the efficacy of their approach. The proposed method steers the representation from omni-modal inputs to uni-modal representations hoping that such steering would correct the model's response.

Critical strengths
  • The paper is very well written and easy to follow and the authors have added empirical justifications to most of their design choices (see W3 below).
  • The proposed technique is a simple intervention and can be applied at test time without manipulating the inputs as some other test-time approaches do.
  • The authors also show the performance of their approach on a general benchmark which helps judge how much the proposed approach generalize beyond fixing cross-modal hallucinations.
Critical weaknesses

W1. The approach in the case of using multiple modalities falls back to the default omni-modal LLM.

  • If I understand correctly, the approach in the case of a prompt/question that relies on both the modalities falls back to the default omni-LLM reference.
  • Considering the above, my questions is that can the authors report how they have performed the inference on Daily-Omni in Table-4. Also, can the authors share some stats on how frequently does this happen in case of DailyOmni is that the inference is falling back to the omni-LLM (i.e. the query is classified as requiring both the modalities).

W2. Single modality performance and motivation

  • Section 3 and Figure 3 show how good single modality inputs are on tasks of AVHBench and CMM however, it only shows these results on the samples that were wrong with omni-modal reference. It would be good to see the performance of relevant unimodal input compared to omni-modal inputs across the entire sets of CMMBench and AVH bench.
  • [Minor, and not the main point of the paper. No need to respond.] The recovery rates are significant but not close to a 100% meaning that the single modality is still not perfect to solve the task.

W3. Some design choices are not justified

  • The authors apply depth-decayed residual intervention (or activation-steering) but its justification is somewhat losely stated in L249-250 without any citations. Although the ablations do justify this choice to some extent however, the performance reported in Figure 6 across different variations are within 1% of each other.
  • The choice to use 3 consecutive layers for intervention is also not stated. The ablation in Figure 4 shows that choosing late layers is the best but why chose consecutive 3 layers in the first place and why choosing 3 layers?

W4. Comparison to baselines on general benchmarks and compute

  • [W4.1] The authors should add a comparison to the other test-time baselines mentioned in Table 1 in the DailyOmni Table 3. Additionally, the authors should also report the performance on other tasks of AVHBench and CMM except just the cross-modal hallucination tasks.
  • [W4.2] The authors should also add comparison to the test-time baselines on the basis of compute (FLOPs or inference time or number of forward loops or similar) to justify that CaRE uses similar compute as those.
Questions for authors

In addition to responding to the weaknesses above, can the authors also respond to the following:

Q1. The DailyOmni number seem very close to the baseline which can be attributed to noise. Can the authors report average and standard deviation across a few runs?


Overall I believe that this paper's idea is simple yet effective however there are some areas where I would like more justification from the authors. I would consider to increase my rating for this paper based on the author's rebuttal and the feedback from other reviewers. I wish the authors good luck for the rebuttals!

Reviewer_NaPyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses scenarios where an audio-visual model provides an incorrect response, even though one of its constituent unimodal branches could have yielded the correct answer. The proposed method, CaRe, employs a text-only modality probe to select a candidate unimodal branch. When the response from this branch diverges from the joint prediction, CaRe steers the representation of the final prompt token toward that branch. The paper reports consistent gains on selected CMM and AVHBench tasks across two model families. It also includes an experiment with a 3B-parameter model and evaluates general QA performance on Daily-Omni.

Critical strengths
  1. The reported gains are substantial on CMM and consistent on AVHBench. The method offers a representation-level alternative to modality-adaptive decoding, using a selected unimodal branch to guide hidden-state correction.
  2. The CF-SELECT comparison tests the useful alternative of returning the unimodal answer directly. Table 3 reports better results for CaRe on all conflict subsets, although the consistency issue below needs resolving. The prompt sensitivity analysis and latency breakdown also help assess the method in practice.
Critical weaknesses
  1. The evaluation needs clarification. On binary gate-positive examples, the joint and CF-SELECT predictions have complementary correctness; gate-negative predictions stay unchanged. With 400 questions per CMM subset, Tables 1 and 3 therefore imply about 178 gated examples for Qwen and 187 for MiniCPM, versus 167 and 174 in Table 5. Ordinary rounding does not explain this if the tables use the same settings and examples. Separately, Appendix C lists the settings in Table 7 but does not identify the validation data used to select them.
  2. The results show a recoverable discrepancy between joint and unimodal predictions, but do not isolate the benefit of hidden-state steering over output-space fusion. A gated interpolation of joint and unimodal logits would be a useful control. Samples routed to both receive no intervention, so Daily-Omni's small aggregate gains also do not establish that interventions preserve complementary information on questions requiring both modalities.
Questions for authors
  1. Could you reconcile Tables 1, 3 and 5 by reporting the gated count and numbers of correct joint, CF-SELECT and CaRe predictions for each subset? Please clarify any differences in evaluation settings.
  2. Which data were used to choose the layers, strength ranges and threshold, and were all evaluation examples excluded? With the same routing and gate, how does a validation-tuned mixture of joint and unimodal logits compare with CaRe?
Reviewer_VMPMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates cross-modal hallucination in omni-modal large language models, with a focus on what the authors term modality misrouting: among questions answered incorrectly under joint audio–video input, some can be answered correctly when the interfering modality is withheld and only the question-relevant unimodal input is retained. To address this issue, the authors propose CaRe, a training-free, inference-time method. CaRe first uses a question-text-only probe to determine which modality is required to answer the question, and runs the corresponding unimodal branch only when the probe selects audio or video. When the joint branch and the selected unimodal branch produce non-tied, conflicting predictions, a gating mechanism is triggered. CaRe then applies residual corrections at several later decoder layers to the joint hidden state of the last prompt token, steering it toward the unimodal reference state. The intervention strength is jointly determined by the predictive margin of the reference branch and the layer depth. The authors evaluate CaRe on the Qwen2.5-Omni and MiniCPM-o model families on CMM and AVHBench, reporting accuracy gains over the original models and baselines such as VCD, ICT, AVCD, and MAD.

Critical strengths
  1. The problem setting is clear and practically relevant. The paper examines cases where joint audio–video predictions fail despite correct predictions from the question-relevant unimodal branch. It formulates this problem as an actionable inference-time correction task, addressing an important source of cross-modal errors.
  2. The method is training-free and includes analyses of its design choices. It does not update model parameters, making it potentially applicable to existing models. The comparison with CF-SELECT and ablations on gating, layer position, and intervention strength provide useful insights into its design.
Critical weaknesses
  1. Tables 1 and 3 appear inconsistent under the stated evaluation protocol. The caption of Table 3 states that evaluation is restricted to gate-positive samples. Under the gate definition in Sec. 5.1, the joint branch and the selected unimodal reference make non-tied, conflicting predictions. For the binary Yes/No tasks (CMM & AVHBench), exactly one of these predictions must be correct. Since CF-SELECT outputs the reference prediction, the original joint accuracy on this subset must therefore equal 1−AccCF−Select1 - Acc_{CF-Select}. For MiniCPM on AVHBench V→A, Table 3 thus implies an original joint accuracy of 100-48.56=51.44%, and a gate-positive subset improvement of 59.38-51.44=7.94%. However, Table 1 reports a full-set improvement of 81.7-72.8=8.90%. If gate-negative predictions remain unchanged, the full-set improvement must equal the gate-positive improvement multiplied by the gate-positive proportion. These values would require a proportion of 8.90/7.94=112.09%, which is impossible. This discrepancy assumes ordinary accuracy and identical evaluation samples, gate configurations, and joint predictions across both tables. Please clarify these conditions and provide the gate-positive counts and scoring protocols. The discrepancy directly affects confidence in the reported improvements.
  2. The benefit of representation-level intervention over simpler branch fusion remains unestablished. CF-SELECT establishes that CaRe improves over directly adopting the unimodal prediction, but does not isolate the benefit of hidden-state intervention. On these binary tasks, CaRe's final prediction necessarily matches either the joint or unimodal label. Thus, its advantage could reflect better selection between conflicting predictions rather than a unique benefit of representation-level correction. A score-level fusion baseline using the same modality probe, conflict gate, and confidence signal, with comparable hyperparameter tuning, would help establish whether hidden-state intervention offers a distinct advantage. The existing layer and intervention-strength ablations do not resolve this question. This limits the evidence supporting the proposed mechanism's necessity.
Questions for authors

Please refer to Critical Weaknesses for questions regarding the inconsistency between tables 1 and 3 and the comparison with score-level fusion.

Additionally, table 4 shows only small aggregate gains on Daily-Omni, without reporting intervention coverage or the balance of beneficial and harmful changes. Could the authors provide gate-positive proportions, wrong-to-correct and correct-to-wrong counts, and audio/video/both routing distributions for each backbone? These statistics would help clarify whether the gains reflect limited intervention coverage or offsetting improvements and regressions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.