acceptodds
Under review as a conference paper at ICLR 2027

CAD: Conflict-Aware Decoding to Mitigate Cross-Modal Hallucinations in Omnimodal Large Language Models

Abstract

Omnimodal large language models (Omni-LLMs) integrate audio, video, and text for perception and reasoning but remain vulnerable to cross-modal hallucinations, where one modality improperly influences predictions about another. By avoiding the additional parameter updates required by training-based methods, training-free decoding offers an appealing approach to mitigating these hallucinations. Existing training-free methods modulate modality contributions during decoding through perturbation or relevance weighting, but do not explicitly assess fusion effects on joint predictions, potentially retaining harmful interference or suppressing useful complementarity. To this end, we propose Conflict-Aware Decoding (CAD), a training-free, two-stage framework. First, to assess the potential effects of cross-modal fusion, Potential Conflict Magnitude Estimation (PCME) quantifies audio-video disagreement and the joint prediction's deviation from a relevance-weighted unimodal reference. Second, to assess evidential support for intervention, Conflict Actionability Assessment (CAA) builds on Dempster-Shafer reliability discounting to evaluate task-space answer relations using query relevance and answer decisiveness. When warranted, CAD shifts decoding weight from the joint branch to the unimodal branches. Extensive experiments demonstrate that CAD outperforms competitive training-free methods. It achieves substantial gains on cross-modal hallucination benchmarks CMM and AVHBench while also improving general audio-visual question answering on WorldSense and VideoMME.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Omnimodal Large Language Models, Cross-Modal Hallucination, Training-Free Decoding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YMW4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses cross-modal hallucination in omnimodal LLMs, where audio-visual fusion lets one modality corrupt predictions about the other. Building on MAD’s relevance-weighted multi-branch decoding, the paper proposes CAD, a training-free method that first estimates how strongly the joint prediction conflicts with the unimodal ones, then assesses whether the unimodal branches provide evidence decisive enough to justify for intervention. When both conditions hold, decoding weight is shifted from the joint branch to the unimodal branches. On CMM and AVHBench across four backbones, CAD improves over MAD by 1.3–4.4 and 1.4–3.6 pp.

Critical strengths
  1. The gap is well motivated. Fig. 2 and sec. 3.2 show that joint accuracy depends on the unimodal agreement relation, which relevance weighting alone cannot capture.
  2. The method is well described and builds directly on MAD (sec. 4, alg. 1), reusing its branch logits with no extra forward passes and reducing exactly to MAD when inactive, so the comparison is well controlled.
  3. The performance increase is replicated on four different omni LLMs (table 1), with ablations isolating C, B, and their product (table 5), demonstrating the benefit is not specific to a single model architecture.
Critical weaknesses
  1. The claim that CAD’s benefits extend beyond hallucination benchmarks (sec. 5.2) feels overstated. In table 2, the WorldSense gain over Base is 2.9 pp at already low accuracy, and the gain on VideoMME is 1.4 pp. The reported results are only on VideoLLaMA2-AV.
  2. The gains over MAD are small (1.3–4.4 pp) and the sensitivity of CAD to its hyperparameters (γ,λ,τ\gamma, \lambda, \tau) is not examined. No confidence intervals or significance tests are reported. Lessons from the hyperparameter sweep and bootstrap CIs on the CAD–MAD difference would establish stronger evidence.
  3. The actionability rule cannot distinguish destructive fusion from complementary recovery. Both (A✓,V✓,AV×)(A\checkmark,V\checkmark,AV\times) and (A×,V×,AV✓)(A\times,V\times,AV\checkmark) present as yA=yV≠yAVy_A = y_V \neq y_AV and receive the same B in eq. 10. Fig. 2 Case 1 is a case CAD would shift weight toward the wrong unimodal answer. A count of AV-correct predictions flipped to wrong versus AV-wrong fixed would substantiate the “preserving complementarity” claim.
Questions for authors
  1. Do the table 2 gains hold on Qwen2.5-Omni-7B or OmniVinci?
  2. How were (γ,λ,τ\gamma, \lambda, \tau) selected?
  3. Per benchmark and per eq. 10 case, how many predictions does CAD flip from correct to wrong versus wrong to correct?
5 comments
AuthorsOpenReview ↗

Response to Reviewer YMW4. Part [1/3]

We appreciate your thoughtful and constructive feedback, and sincerely thank you for highlighting our strengths (e.g., the well-motivated problem formulation, clear presentation, and consistent empirical gains). Your comments have helped us clarify several important aspects of our work. We hope our responses address your concerns. All manuscript revisions discussed below are highlighted in blue in the resubmitted manuscript.

1. Suggestion about general QA performance and evaluation on additional backbones.

Thank you for your valuable suggestion, which made us realize that including more comprehensive statistical analyses and evaluations across backbones would further strengthen the reliability of our paper's conclusions. Following your suggestion, we have refined the wording in Section 5.2 of the revised manuscript and added the following quantitative analyses and experimental results.

First, we compiled the historical per-question results of VideoLLaMA2-AV on the two general QA benchmarks and provide more precise overall accuracy statistics:

BenchmarkBase accuracy (%)MAD accuracy (%)CAD accuracy (%)
WorldSense25.9125.0028.81
VideoMME48.0048.5249.41

Beyond overall accuracy, we conducted paired bootstrap analyses of the accuracy differences comparing CAD with Base and MAD. Specifically, we used 20,000 resamples at the video level, retaining all questions from each sampled video, to construct 95% intervals. The analysis covered 3,172 questions from 1,662 videos in WorldSense and 2,700 questions from 900 videos in VideoMME.

BenchmarkComparisonAccuracy difference95% bootstrap interval
WorldSenseCAD-Base+2.90[1.34, 4.46]
WorldSenseCAD-MAD+3.81[2.45, 5.17]
VideoMMECAD-Base+1.41[0.56, 2.26]
VideoMMECAD-MAD+0.89[0.19, 1.59]

The results show that CAD achieves higher accuracy, with all four 95% paired bootstrap intervals for the accuracy gains lying above zero. These provide further statistical support for CAD’s gains on the general audio-visual benchmarks.

Second, to address your question about additional backbones, we added evaluations comparing Base, MAD and CAD using Qwen2.5-Omni-7B and OmniVinci on WorldSense and VideoMME. The experimental results are as follows:

BackboneBenchmarkBaseMADCAD
Qwen2.5-Omni-7BWorldSense47.1047.6048.23
OmniVinciWorldSense46.5647.1047.92
Qwen2.5-Omni-7BVideoMME64.5264.0065.37
OmniVinciVideoMME65.9665.4166.67

Across all four additional settings, CAD achieves higher overall accuracy than both Base and MAD, with gains of 0.71-1.36 percentage points over Base and 0.63-1.37 percentage points over MAD. These results show that CAD consistently achieves accuracy gains across the backbones and general audio-visual benchmarks. We have now included the additional results in Table 2 and the detailed results and statistical analyses in Appendix D of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer YMW4. Part [2/3]

2. Suggestion on hyperparameter selection, sensitivity, and statistical reliability.

Thank you very much for your suggestions. We have clarified the hyperparameter settings and added sensitivity and paired bootstrap analyses to assess the reliability of CAD’s results.

First, regarding hyperparameter selection, our selection protocol draws on the parameter-tuning practices of AVCD and MAD. We pool approximately 10% of the samples from each benchmark to form a test subset for selecting γ\gamma, λ\lambda, and τ\tau, with ϵ\epsilon fixed at 10−810^{-8}. Specifically, we first perform a coarse-grained parameter search on this subset, then refine the search within promising regions and select the final (γ,λ,τ)(\gamma,\lambda,\tau) configuration with the highest accuracy on this subset. This procedure selects decoding hyperparameters without training or updating model weights. We have added the details of this selection procedure to Appendix B of the revised manuscript.

Second, we conducted paired bootstrap analyses on CMM and AVHBench to quantify CAD’s accuracy gains. To account for dependence among questions associated with the same video, we performed 20,000 bootstrap resamples at the video level, retaining all questions from each sampled video, and calculated the 95% percentile intervals for the overall accuracy differences. The experimental results are as follows:

BackboneBenchmarkCAD-Base95% bootstrap CICAD-MAD95% bootstrap CI
VideoLLaMA2-AVCMM+7.500[4.750, 10.250]+1.375[−0.125, 3.000]
VideoLLaMA2-AVAVHBench+4.437[3.137, 5.742]+1.313[0.852, 1.792]
OmniVinciCMM+4.125[2.250, 6.000]+2.375[1.125, 3.750]
OmniVinciAVHBench+6.947[5.712, 8.180]+3.649[2.774, 4.547]
Qwen2.5-Omni-3BCMM+7.000[4.625, 9.375]+1.250[−0.125, 2.625]
Qwen2.5-Omni-3BAVHBench+6.246[5.101, 7.408]+1.664[0.801, 2.544]
Qwen2.5-Omni-7BCMM+14.125[11.250, 17.125]+4.375[2.375, 6.375]
Qwen2.5-Omni-7BAVHBench+8.027[6.838, 9.240]+2.948[2.265, 3.752]

Across all eight settings, CAD achieves higher accuracy than both Base and MAD, with most 95% paired bootstrap intervals lying above zero. These results provide further empirical and statistical support for CAD’s effectiveness across the evaluated settings. We have now included these paired bootstrap results in Appendix D of the revised manuscript.

Third, using OmniVinci on CMM and AVHBench as in Table 1 of our paper, we evaluated CAD’s sensitivity to γ\gamma, λ\lambda, and τ\tau. We varied one parameter at a time around (γ,λ,τ)=(4,20,0.05)(\gamma,\lambda,\tau)=(4,20,0.05) while keeping the other two fixed. Each parameter configuration was evaluated on both benchmarks. The experimental results are shown in the following tables:

Sensitivity to γ\gamma (λ=20\lambda=20, τ=0.05\tau=0.05).

γ\gamma1234567
CMM accuracy (%)87.187.987.488.086.886.586.3
AVHBench accuracy (%)78.879.680.080.380.280.179.6
Avg. (%)83.083.883.784.183.583.382.9

Sensitivity to λ\lambda (γ=4\gamma=4, τ=0.05\tau=0.05).

λ\lambda5101520253035
CMM accuracy (%)86.587.087.388.087.587.487.3
AVHBench accuracy (%)79.079.780.080.380.280.380.4
Avg. (%)82.783.483.684.183.883.883.8

Sensitivity to τ\tau (γ=4\gamma=4, λ=20\lambda=20).

τ\tau0.030.040.050.060.070.080.09
CMM accuracy (%)87.587.588.087.587.487.187.0
AVHBench accuracy (%)80.280.280.380.079.879.579.4
Avg. (%)83.883.884.183.883.683.383.2

The results show that CAD maintains stable performance across neighboring parameter values in the one-at-a-time sweeps, including γ∈[2,4]\gamma \in [2,4], λ∈[15,35]\lambda \in [15,35], and τ∈[0.03,0.06]\tau \in [0.03,0.06]. Furthermore, CAD exhibits overlapping stable parameter ranges on CMM and AVHBench, supporting the use of a common hyperparameter configuration across both benchmarks with OmniVinci as the backbone. We have now included these sensitivity results in Section 5.3 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer YMW4. Part [3/3]

3. Suggestion about distinguishing destructive fusion from complementary recovery.

Thank you for your valuable suggestions. They prompted us to further clarify CAA’s intervention mechanism and quantitatively evaluate CAD’s effectiveness. We provide both a mechanistic explanation and an analysis of actual prediction changes below.

First, the same answer relation (yA=yV≠yAVy_A=y_V\ne y_{AV}) can indeed correspond to either destructive fusion or complementary recovery, but CAA additionally considers query relevance and the strength of unimodal support for the predicted answers to assess the evidence for intervention. Specifically, drawing on the reliability-discounting principle of Dempster–Shafer theory, CAA combines modality relevance and answer margins as qk=Rkmkq_k=R_km_k, assigning qkq_k as support for the unimodal answer and leaving the remaining mass 1−qk1-q_k uncommitted to represent ignorance. Since the two branches share the backbone, query, and language prior, CAA conservatively aggregates their shared support using B=min⁡(qA,qV)B=\min(q_A,q_V) when yA=yV≠yAVy_A=y_V\ne y_{AV}. For Figure 2 Case 1, smaller unimodal margins reduce support for intervention. When C≥τC\ge\tau, the coefficient transfer δ=λCB[wAV]+\delta=\lambda CB[w_{AV}]_+ is jointly modulated by conflict magnitude and reliability evidence.

Second, we compared the predictions of Base and CAD across four backbones on CMM and AVHBench. Corrections correspond to wrong→correct flips, whereas introduced errors correspond to correct→wrong flips. The results are as follows:

BackboneBenchmarkWrong→CorrectCorrect→WrongNet increase in correct answersCAD−Base95% bootstrap CI
VideoLLaMA2-AVCMM9636+60+7.500[4.750, 10.250]
VideoLLaMA2-AVAVHBench366214+152+4.437[3.137, 5.742]
OmniVinciCMM4512+33+4.125[2.250, 6.000]
OmniVinciAVHBench347109+238+6.947[5.712, 8.180]
Qwen2.5-Omni-3BCMM7721+56+7.000[4.625, 9.375]
Qwen2.5-Omni-3BAVHBench30894+214+6.246[5.101, 7.408]
Qwen2.5-Omni-7BCMM13623+113+14.125[11.250, 17.125]
Qwen2.5-Omni-7BAVHBench32550+275+8.027[6.838, 9.240]

Across all eight settings, CAD corrects more errors than it introduces, and all 95% paired bootstrap confidence intervals for the accuracy gains lie above zero. Moreover, corrections account for 63.1%-86.7% of the cases where Base and CAD differ in correctness, which further supports CAD’s ability to preserve complementarity.

Third, as requested, we report prediction flips by benchmark and by the answer-relation cases in Eq. (10). The analysis covers four backbones on CMM and AVHBench. We assign each sample to an Eq. (10) case using the initial task-space predictions of the A, V, and AV branches, and compare the evaluated answers of Base and CAD.

The four cases are:

  • I: yA=yV=yAVy_A=y_V=y_{AV}.
  • II: yA=yV≠yAVy_A=y_V\ne y_{AV}.
  • III: yA≠yVy_A\ne y_V and yAV∈yA,yVy_{AV}\in\\{y_A,y_V\\}.
  • IV: yA≠yVy_A\ne y_V and yAV∉yA,yVy_{AV}\notin\\{y_A,y_V\\}.

Since both CMM and AVHBench use binary answers, Case IV has zero samples and zero prediction flips across all settings. In Case I, B=0B=0 and CAD falls back to MAD decoding without additional conflict intervention. The results are presented in the following tables:

CMM

BackboneEq. (10) caseWrong→CorrectCorrect→Wrong
VideoLLaMA2-AVI134
VideoLLaMA2-AVII33
VideoLLaMA2-AVIII8029
OmniVinciI10
OmniVinciII32
OmniVinciIII4110
Qwen2.5-Omni-3BI143
Qwen2.5-Omni-3BII187
Qwen2.5-Omni-3BIII4511
Qwen2.5-Omni-7BI1210
Qwen2.5-Omni-7BII71
Qwen2.5-Omni-7BIII11712

AVHBench

BackboneEq. (10) caseWrong→CorrectCorrect→Wrong
VideoLLaMA2-AVI4548
VideoLLaMA2-AVII5028
VideoLLaMA2-AVIII271138
OmniVinciI4743
OmniVinciII8712
OmniVinciIII21354
Qwen2.5-Omni-3BI6412
Qwen2.5-Omni-3BII9220
Qwen2.5-Omni-3BIII15262
Qwen2.5-Omni-7BI616
Qwen2.5-Omni-7BII6516
Qwen2.5-Omni-7BIII19928

The results show that CAD’s net gains over Base arise primarily from Cases II and III, indicating that CAD corrects more errors than it introduces when branch predictions disagree. This further supports CAD’s effectiveness in mitigating cross-modal hallucinations. We have now included the detailed prediction-change analyses in Appendix G of the revised manuscript.

We once again thank you for your valuable comments! If you have any further questions, please feel free to let us know and we will do our utmost to respond.

Reviewer_kugbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Conflict-Aware Decoding (CAD), a training-free inference-time framework aimed at mitigating cross-modal hallucinations in Omnimodal Large Language Models (Omni-LLMs). Building upon existing modality-adaptive decoding baselines (e.g., MAD), CAD introduces a two-stage conflict assessment mechanism before generating responses:

Potential Conflict Magnitude Estimation (PCME): Measures full-vocabulary distribution discrepancies among audio-only, video-only, and joint predictions using normalized Jensen-Shannon divergence (JSD).

Conflict Actionability Assessment (CAA): Projects distributions onto a task-aligned answer space and applies Dempster-Shafer evidence theory to evaluate whether the discrepancy indicates harmful fusion interference rather than useful complementarity.

When actionable conflict is detected, CAD dynamically shifts decoding weights from the joint audio-visual branch back to the unimodal branches. The authors evaluate CAD on cross-modal hallucination benchmarks (CMM, AVHBench) and general audio-visual QA benchmarks (WorldSense, VideoMME) across four backbone models.

Critical strengths
  1. Well-Motivated Problem Formulation: The observation of "destructive fusion"—where joint multimodal inference performs worse than unimodal branches despite evidence being present—is insightful, well-motivated, and clearly illustrated.

  2. Training-Free & Non-Intrusive: As a pure inference-time decoding strategy, CAD requires no model parameter updates or auxiliary fine-tuning, making it easy to plug into existing pre-trained Omni-LLMs.

  3. Consistent Empirical Benchmark Gains: The proposed method demonstrates consistent accuracy improvements over existing training-free decoding baselines (AVCD, MAD) on cross-modal hallucination evaluation benchmarks.

Critical weaknesses
  1. The fundamental framework of CAD relies on multi-branch forward passes and logit ensembling/contrasting (e.g., Contrastive Decoding, AVCD, MAD). While combining JSD divergence with Dempster-Shafer evidence discounting adds formal structure, the core paradigm remains an incremental heuristic extension of multi-branch decoding. It provides limited fresh algorithmic or theoretical insights into how modern unified Omni-LLM architectures handle internal representation alignment or multi-modal fusion.

  2. The two-stage assessment pipeline (PCME + CAA) relies heavily on hand-crafted rules, piecewise conditional branches, and manual hyperparameters (e.g., intervention threshold τ\tau, contrast strength γ\gamma, scaling coefficient λ\lambda, and numerical threshold ϵ\epsilon). As noted in Appendix B, performance depends on tuning hyperparameters on a separate validation set. The piecewise definitions in Eq. (10) and heuristic fallbacks (such as using max(qA,qV)max(q_A, q_V) when the joint branch produces an unexpected third answer) lack rigorous theoretical justification.

  3. CAD requires maintaining parallel forward passes and logit extractions across four distinct contexts (AV,A,V,TAV, A, V, T) during generation. In real-world Omni-LLM applications (e.g., real-time streaming dialogue/assistance), quadrupling context passes introduces severe KV-cache memory overhead and latency bottlenecks. Although Section 6 acknowledges this limitation, the paper lacks quantitative profiling (FLOPs, memory usage, latency per token) comparing CAD against standard single-pass decoding.

  4. While CAD outperforms training-free baselines like AVCD and MAD, the scope of empirical validation remains limited: It lacks comparison with lightweight fine-tuning or alignment techniques (e.g., Direct Preference Optimization / DPO for multimodal consistency) or token-level dynamic decoding methods. Performance gains on general video QA benchmarks (e.g., VideoMME average accuracy increasing from 48.0% to 49.4%) are modest, indicating limited effectiveness outside restricted multiple-choice or binary hallucination evaluation tasks

Questions for authors
  1. What is the exact GPU memory consumption and latency (seconds/milliseconds per token) of CAD compared to native Base decoding and MAD across varying sequence lengths?

  2. How robust are τ=0.05\tau = 0.05 and λ=7.9\lambda = 7.9 when applied directly to unseen model architectures or open-ended tasks without validation set tuning?

  3. How does CAD perform on free-form open-ended generation where valid answer token sets T(y)\mathcal{T}(y) cannot be easily pre-projected into option spaces?

6 comments
AuthorsOpenReview ↗

Response to Reviewer kugb. Part [1/5]

We appreciate your valuable time and insights, and thank you for highlighting our strengths (e.g., the well-motivated problem formulation, the training-free and non-intrusive design, and the consistent empirical gains). In the following, we will carefully respond to your questions. We hope we can address your concerns. All manuscript revisions discussed below are highlighted in blue in the resubmitted manuscript.

1. Question about our contributions.

Thank you very much for your feedback. It made us realize that we should clarify the nature and source of our contributions and how they differ from prior work. Our work offers a new perspective on understanding and intervening in cross-modal fusion and translates it into a training-free decoding method applicable to different Omni-LLMs. Experiments across models and benchmarks support its effectiveness in mitigating cross-modal hallucinations.

As acknowledged by all three reviewers (#4WeG, #kugb, and #YMW4), our work is motivated by the observation that joint audio-visual predictions can undermine correct unimodal judgments but can also correct errors through complementary information. This makes deciding when and how to intervene in multimodal fusion an important research question. However, existing methods based solely on modality dominance, predictive entropy, or query relevance do not directly resolve this problem.

To address this issue, we introduce a two-stage conflict-assessment mechanism that separately models the magnitude of predictive discrepancies and the evidence supporting intervention. This mechanism uses JSD to quantify distributional discrepancies and draws on reliability discounting in Dempster-Shafer evidence theory. This decoupled design and its translation of theory into practice are central to our methodological contribution. Our method and empirical analyses provide prediction-level insights into fusion behavior in unified Omni-LLMs and support more reliable cross-modal inference across models and benchmarks. Therefore, we believe that, beyond the engineering effort, our work indeed makes a meaningful contribution to the community.

Thank you again. We will further clarify our contributions and their distinction from prior work; additional design rationale is now provided in Appendix H of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer kugb. Part [2/5]

2. Question about piecewise rules and hyperparameter sensitivity.

Thank you for your question. To address your concerns about the piecewise rules and dependence on hyperparameter tuning, we clarify the design rationale and hyperparameter selection procedure, and report sensitivity experiments. We address these two aspects below.

(1) Design rationale for PCME and CAA.

Motivated by the coexistence of harmful interference and beneficial complementarity in cross-modal fusion, CAD separately models the magnitude of predictive discrepancies and the evidence supporting intervention using PCME and CAA, respectively.

In the first stage, PCME quantifies potential cross-modal conflicts by comparing predictive distributions across modality contexts. Given JSD’s use in decoding methods for mitigating hallucinations [1, 2], we adopt normalized JSD to measure these discrepancies on a common scale.

In the second stage, CAA assesses intervention evidence using task-answer relationships and branch reliability. Dempster-Shafer (D-S) evidence theory distinguishes support for specific answers from ignorance and adjusts evidence strength through reliability discounting, with applications in multimodal uncertainty modeling [3]. Drawing on this principle, we construct reliability-discounted answer support and formulate the first three cases of Eq. (10) according to branch-answer relationships.

In the fourth case, the A, V, and AV branches produce three different answers: although the two unimodal branches support different answers, both provide support against the current joint answer. Thus, we draw on the cautious combination rule within the D-S framework [4]. Specifically, we first map the answer space into “the current AV answer” and “other answers.” Each unimodal branch assigns simple support qkq_k to “other answers” and the remaining mass 1−qk1-q_k to ignorance. For these same-focal simple supports, cautious combination takes the minimum of the canonical conjunctive weights 1−qA1-q_A and 1−qV1-q_V, yielding:

B=1−min⁡(1−qA,1−qV)=max⁡(qA,qV).B=1-\min(1-q_A,1-q_V)=\max(q_A,q_V).

This derivation applies when qA,qV<1q_A,q_V < 1, with the boundary cases obtained by continuity. This correspondence provides a local aggregation rationale for the fourth case: retaining the strongest opposing support without amplifying it when identical support is repeated. Intervention still depends on the conflict magnitude and threshold; a third answer alone does not establish that the joint prediction is incorrect. We have now updated the explanation of Eq. (10) in Section 4.3 and added the design rationale in Appendix H of the revised manuscript.

(2) Hyperparameter selection and sensitivity analysis.

Our hyperparameter selection draws on the tuning practices of AVCD and MAD. We pool approximately 10% of the samples from each benchmark into a test subset for selecting γ\gamma, λ\lambda, and τ\tau, with ϵ\epsilon fixed at 10−810^{-8}. This procedure selects decoding hyperparameters without training or updating model weights.

We select hyperparameters separately for each backbone to account for differences in predictive distributions and modality responses. For a given backbone, we generally use the same hyperparameter configuration across benchmarks. To further assess CAD’s sensitivity to parameter selection, we conducted one-at-a-time sweeps using OmniVinci on CMM and AVHBench around (γ,λ,τ)=(4,20,0.05)(\gamma,\lambda,\tau)=(4,20,0.05), varying one parameter while keeping the other two fixed. The experimental results are as follows:

Sensitivity to γ\gamma (λ=20\lambda=20, τ=0.05\tau=0.05).

γ\gamma1234567
CMM accuracy (%)87.187.987.488.086.886.586.3
AVHBench accuracy (%)78.879.680.080.380.280.179.6
Avg. (%)83.083.883.784.183.583.382.9

Sensitivity to λ\lambda (γ=4\gamma=4, τ=0.05\tau=0.05).

λ\lambda5101520253035
CMM accuracy (%)86.587.087.388.087.587.487.3
AVHBench accuracy (%)79.079.780.080.380.280.380.4
Avg. (%)82.783.483.684.183.883.883.8

Sensitivity to τ\tau (γ=4\gamma=4, λ=20\lambda=20).

τ\tau0.030.040.050.060.070.080.09
CMM accuracy (%)87.587.588.087.587.487.187.0
AVHBench accuracy (%)80.280.280.380.079.879.579.4
Avg. (%)83.883.884.183.883.683.383.2

The one-at-a-time sweeps show stable performance at the tested values within γ∈[2,4]\gamma \in [2,4], λ∈[15,35]\lambda \in [15,35], and τ∈[0.03,0.06]\tau \in [0.03,0.06]. The overlapping stable ranges support a shared configuration across CMM and AVHBench with OmniVinci. We have now included these sensitivity results in Section 5.3 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer kugb. Part [3/5]

3. Question about computational costs.

Thank you for this valuable suggestion, which made us realize that a quantitative analysis of computational costs would provide a more complete assessment of CAD’s practical applicability. Following your suggestion, we compared Base, AVCD, MAD, and CAD using VideoLLaMA2.1-7B-AV on four benchmarks (CMM, AVHBench, WorldSense, and VideoMME), and further evaluated their computational costs across different input sequence lengths.

(1) Computational costs across different benchmarks.

BenchmarkMethodAcc. (%)Latency per example (s)Peak GPU memory (GiB)FLOPs per example (TFLOPs)
CMMBase75.91.4418.340.5
CMMAVCD77.11.6022.171.0
CMMMAD82.02.3618.4122.3
CMMCAD83.42.7418.4122.4
AVHBenchBase76.61.1618.340.4
AVHBenchAVCD73.61.7222.1111.1
AVHBenchMAD79.72.0118.3121.6
AVHBenchCAD81.12.3718.3121.8
WorldSenseBase25.91.2319.142.9
WorldSenseAVCD26.02.0224.5118.8
WorldSenseMAD25.02.1219.1133.5
WorldSenseCAD28.82.3119.1133.6
VideoMMEBase48.03.2318.741.8
VideoMMEAVCD48.13.6623.394.7
VideoMMEMAD48.54.1518.7128.0
VideoMMECAD49.44.4918.7128.1

The results show that AVCD, MAD, and CAD all incur additional computational costs relative to Base. Compared with MAD, CAD achieves higher accuracy and adds only 0.19-0.38 seconds per example, with peak GPU memory and FLOPs remaining nearly unchanged.

(2) Computational costs across different input sequence lengths.

To further investigate the effect of input sequence length on computational costs, we fixed 200 WorldSense examples and evaluated three input settings using 4/8/16 video frames and 15/30/60-second audio processing budgets, respectively. The corresponding actual mean joint audio-visual input lengths were 1,231.65, 2,321.65, and 4,493.65 tokens.

Input settingMean AV input tokensMethodLatency per example (s)Peak GPU memory (GiB)FLOPs per example (TFLOPs)
Short1,231.65Base0.7217.721.6
Short1,231.65AVCD1.1819.966.9
Short1,231.65MAD1.2417.769.2
Short1,231.65CAD1.4717.769.2
Medium2,321.65Base1.1918.742.2
Medium2,321.65AVCD1.8222.9111.7
Medium2,321.65MAD2.0718.7130.4
Medium2,321.65CAD2.2918.7130.5
Long4,493.65Base2.2620.786.3
Long4,493.65AVCD4.2529.0280.1
Long4,493.65MAD4.0820.7260.1
Long4,493.65CAD4.3520.7260.4

As the input sequence length increases, latency, peak GPU memory, and FLOPs increase for all four methods. Across the three input settings, CAD incurs a modest increase in latency over MAD, while their peak GPU memory and reported FLOPs remain closely comparable.

Overall, compared with MAD, CAD improves accuracy with a modest increase in inference cost. In future work, we will focus on optimizing the implementation to reduce inference latency and resource consumption. We have now included these computational cost results and the corresponding analysis in Appendix E of the revised manuscript.

Reviewer_4WeGOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles cross-modal hallucinations in Omni-LLMs via Conflict-Aware Decoding (CAD), a training-free inference method extending MAD. It uses two stages (1) PCME to measure cross-modal discrepancy (CC) via normalized JSD against a relevance-weighted mixture, and (2) CAA to compute an actionability score (BB) using task margins and Dempster-Shafer reliability discounting. When conflict is detected, CAD transfers decoding weight from the joint branch (AVAV) to unimodal branches (A,VA, V). Evaluated on CMM, AVHBench shows the improvement on avoiding motimodality halusinationa and the WorldSense, and VideoMME shows the no forgetting on the general OmniLLM benchmark.

Critical strengths
  1. Practical and training-free: Plug-and-play at inference time without retraining, extra supervision, or architecture changes.

  2. Clear problem intuition: Shows convincing motivating cases where joint audio-visual decoding fails despite correct unimodal branches (and vice versa), proving that static relevance alone isn't enough.

  3. Solid empirical gains: Consistently improves over MAD and base decoding on hallucination benchmarks across several 3B/7B open backbones.

Critical weaknesses
  1. Potential degradation of audio-visual alignment: While Conflict-Aware Decoding aims to suppress cross-modal conflicts, it is unclear whether this comes at the cost of the model's ability to align and jointly reason over audio and visual information. Consider to test on cross-modal benchmark such as XModBench (https://arxiv.org/abs/2510.15148) would provide a more direct assessment.

  2. Lack of newer model or larger size: Experiments focus on older or smaller models (≤7B). Given this is training free, stronger omni models, such as Qwen3-Omni-30B, may exhibit different behaviors in modality alignment as pretraining scales. This is necessary to establish the relevance of both the proposed problem and different archetecture in mordern LLM.

Questions for authors
  1. Does Conflict-Aware Decoding affect the model's ability to align and jointly reason over audio and visual information, particularly when both modalities contain complementary information? Could the authors evaluate this potential trade-off on a benchmark such as XModBench ?

  2. Does the proposed method remain effective on larger, more recent omni models, such as Qwen3-Omni-30B? Are cross-modal conflicts still significant at this scale?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 4WeG

We appreciate your valuable time and insights, and thank you for highlighting our strengths (e.g., clear motivation, practical training-free design, and consistent empirical gains). Your comments have helped us clarify several important aspects of our work. We hope our responses address your concerns. All manuscript revisions discussed below are highlighted in blue in the resubmitted manuscript.

1. Suggestion about evaluating audio-visual alignment and joint reasoning on XModBench.

Thank you for this valuable suggestion, which made us realize that including a benchmark for audio-visual alignment and joint reasoning would further strengthen the reliability of our paper's conclusions. Following your suggestion, we evaluated CAD on a reconstructed subset of XModBench (A+V↦TA+V \mapsto T) [1] comprising 1,916 examples (after excluding 25 examples with undecodable videos). We adopted the dual-context setting, which provides paired audio-video inputs and textual answer options. We compared CAD with Base, AVCD, and MAD across four omnimodal large language models (OmniVinci, Qwen2.5-Omni-7B, VideoLLaMA2.1-7B-AV, and Qwen2.5-Omni-3B). The experimental results are as follows:

MethodOmniVinciQwen2.5-Omni-7BVideoLLaMA2.1-7B-AVQwen2.5-Omni-3B
Base50.349.650.350.8
AVCD51.148.450.752.9
MAD52.249.449.654.4
CAD55.951.952.255.4

The experimental results demonstrate that CAD achieves the highest accuracy across all four models, outperforming Base by 1.9-5.6 percentage points and MAD by 1.0-3.7 percentage points. These consistent gains support CAD’s effectiveness in joint audio-visual understanding, while providing empirical evidence that it can preserve the models’ audio-visual alignment and joint reasoning capabilities. We have now included these results in Tables 2 and 12, with analysis in Section 5.2 and further details in Appendix D of the revised manuscript.

2. Suggestion about evaluating stronger omni models.

Thank you for this comment. We agree that evaluating a larger, more recent omni model would provide further empirical evidence for CAD’s applicability. Following your suggestion, we extended our evaluation to Qwen3-Omni-30B on CMM and AVHBench, comparing CAD with Base, AVCD, and MAD. The results are presented in the following tables:

CMM

MethodVisual Dom.Audio Dom.Overall Acc.
Base79.876.578.1
AVCD81.876.379.0
MAD85.084.084.5
CAD86.586.086.3

AVHBench

MethodVideo-Driven Audio Hall.Audio-Driven Video Hall.Overall Acc.
Base82.483.882.9
AVCD83.382.383.0
MAD86.386.486.3
CAD86.786.586.7

The experimental results show that CAD achieves 86.3% accuracy on CMM and 86.7% on AVHBench, outperforming Base by 8.2 and 3.8 percentage points and MAD by 1.8 and 0.4 percentage points, respectively. Building on the strong performance of the base model and the improvements achieved by MAD, CAD delivers additional gains on both benchmarks. These results support CAD’s effectiveness on the larger, more recent Qwen3-Omni-30B model and highlight the continued practical value of addressing cross-modal conflicts at this scale. We have now incorporated these results into Table 1 and added the corresponding analysis to Section 5.2 of the revised manuscript.

We once again thank you for your valuable comments! If you have any further questions, please feel free to let us know and we will do our utmost to respond.

[1] XModbench: Benchmarking cross-modal capabilities and consistency in omni-language models.

Thanks for the experiment and acknowledged the accuracy increase. How about the A -> V and V-> A?

AuthorsOpenReview ↗

New Response to Reviewer 4WeG.

Thank you very much for your positive assessment and valuable suggestion! Based on your suggestion, we conducted a new analysis.

We evaluated CAD on the audio-to-video (A→V) and video-to-audio (V→A) configurations of XModBench, comprising 2,998 and 3,006 examples, respectively (after excluding 33 and 25 examples with undecodable videos). Given the time constraints, we focused on comparing CAD with Base and MAD across three omnimodal large language models (OmniVinci, VideoLLaMA2.1-7B-AV, and Qwen2.5-Omni-3B). The results are presented in the following table:

ConfigurationMethodOmniVinciVideoLLaMA2.1-7B-AVQwen2.5-Omni-3B
A+V→TBase50.350.350.8
MAD52.249.654.4
CAD55.952.255.4
A→VBase24.525.528.7
MAD25.026.027.3
CAD27.926.628.8
V→ABase24.925.627.3
MAD25.025.524.8
CAD26.526.227.5

CAD achieves the highest accuracy across the three reported models in both directions, consistently outperforming Base and MAD. Notably, all three methods obtain substantially lower absolute accuracy on these matching tasks than in the earlier A+V↦TA+V\mapsto T evaluation. The low Base accuracy indicates that these configurations remain challenging for the evaluated models. MAD does not yield consistent gains over Base, while CAD’s average gains over both baselines are smaller than in the earlier A+V↦TA+V\mapsto T evaluation.

Specifically, in A+V↦TA+V\mapsto T, the unimodal branches of both MAD and CAD retain the complete textual options. In A→V and V→A, however, removing a modality also removes the media content of either the question or its answer options. For example, the audio-only branch in A→V retains the option labels but cannot access the candidate videos. This loss of task information may limit the performance gains of both MAD and CAD. We will try our best to further improve the manuscript by incorporating these additional results and the corresponding analysis.

Thank you again for your feedback! We hope we've addressed your concerns and strengthened your confidence in our paper, and we would greatly appreciate it if you could update your score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to all Reviewers

Dear reviewers and Area Chair, we sincerely thank you for your valuable and thoughtful feedback. Your insightful suggestions have greatly improved our paper. We appreciate the recognition of our strengths, particularly the well-motivated problem formulation [Reviewers #4WeG, #kugb, and #YMW4], the practical, training-free, and non-intrusive design [Reviewers #4WeG and #kugb], the clear presentation and well-controlled comparison [Reviewer #YMW4], and the strong empirical performance [Reviewers #4WeG, #kugb, and #YMW4].

Following your suggestions, we have revised the manuscript, adding several experiments, analyses, tables, and figures. All new tables are included in our rebuttal responses, while the new figures are presented in the revised manuscript. We address each concern in detail in our individual responses, and the corresponding manuscript revisions are highlighted in blue. In addition, we will release all our data and code to facilitate further research.

We sincerely thank the reviewers, the Area Chair, the Senior Area Chairs and Program Chairs for the valuable feedback. We also deeply value thoughtful evaluations, which highlight several strengths of our work:

  1. Well-motivated problem formulation: Joint audio-visual inference can fail despite both unimodal predictions being correct (and vice versa), highlighting a limitation of relevance weighting alone [Reviewers 4WeG, kugb, and YMW4].
  2. Practical, training-free, and non-intrusive design: CAD can be applied to pretrained Omni-LLMs without parameter updates or architectural changes [Reviewers 4WeG and kugb].
  3. Clear presentation and well-controlled comparison: CAD is clearly described and uses the same decoding branches as MAD, enabling a controlled comparison [Reviewer YMW4].
  4. Consistent empirical gains: CAD consistently improves overall accuracy over decoding baselines on cross-modal hallucination benchmarks (CMM and AVHBench) across four Omni-LLM backbones [Reviewers 4WeG, kugb, and YMW4].

A major concern shared by reviewers is the hyperparameter sensitivity of CAD. Another is whether CAD can mitigate cross-modal interference while preserving complementary audio-visual information. Reviewers also note the need for a more direct assessment of CAD's gains over MAD, including confidence intervals for accuracy differences and a per-example analysis of errors that CAD corrects or introduces. Further suggestions include evaluating CAD on larger, newer backbones and open-ended generation tasks, comparing it with training-based alignment baselines, and profiling inference costs across methods.

In our rebuttal, we conducted additional experiments and offered clarifications to respond to these suggestions and questions, including:

  1. Performed comprehensive ablation studies on three key hyperparameters, examined how the intervention parameters generalize across models, and further clarified the design rationale for CAD [Reviewers kugb and YMW4].
  2. Analyzed CAD's gains over Base and MAD using paired bootstrap confidence intervals for accuracy differences and counts of corrections and new errors by branch agreement pattern [Reviewer YMW4].
  3. Expanded the cross-modal hallucination evaluation to a larger, more recent backbone (Qwen3-Omni-30B) and assessed CAD's performance on general audio-visual QA benchmarks (WorldSense and VideoMME) using additional backbones (Qwen2.5-Omni-7B and OmniVinci) [Reviewers 4WeG and YMW4].
  4. Evaluated CAD on XModBench ((A+V)→T, V→A, and A→V) to assess whether it preserves audio-visual alignment [Reviewer 4WeG].
  5. Added efficiency experiments comparing inference latency, peak GPU memory usage, and estimated FLOPs across methods [Reviewer kugb].
  6. Expanded the baseline comparisons with AVCD, a token-level adaptive decoding method, and ACPO, a lightweight training-based alignment method [Reviewer kugb].
  7. Added experiments on AVHBench Captioning to assess CAD's applicability to open-ended generation without predefined answer options [Reviewer kugb].

The above analysis and experiments are consistently promising, providing further support for the effectiveness and generalizability of CAD in mitigating cross-modal hallucinations. The clarifications also help improve the quality of the manuscript.

During the discussion, our rebuttal received positive feedback from Reviewers 4WeG and YMW4. Reviewer 4WeG stated that they would increase their rating, while Reviewer YMW4 stated that they would maintain their positive rating. Although Reviewer kugb didn't participate in the discussion, we have made every effort to address their concerns through detailed responses and additional experiments. We believe these additions can help resolve their key concerns.

We have revised our manuscript based on the rebuttal content to further strengthen our work. We sincerely appreciate the reviewers' thoughtful feedback and constructive suggestions. We hope our work can provide insights for future research on mitigating cross-modal hallucinations in Omni-LLMs.

Best Regards,

Authors of ICLR 2027 Conference Paper 40067

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.