- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses cross-modal hallucination in omnimodal LLMs, where audio-visual fusion lets one modality corrupt predictions about the other. Building on MAD’s relevance-weighted multi-branch decoding, the paper proposes CAD, a training-free method that first estimates how strongly the joint prediction conflicts with the unimodal ones, then assesses whether the unimodal branches provide evidence decisive enough to justify for intervention. When both conditions hold, decoding weight is shifted from the joint branch to the unimodal branches. On CMM and AVHBench across four backbones, CAD improves over MAD by 1.3–4.4 and 1.4–3.6 pp.
Critical strengths
- The gap is well motivated. Fig. 2 and sec. 3.2 show that joint accuracy depends on the unimodal agreement relation, which relevance weighting alone cannot capture.
- The method is well described and builds directly on MAD (sec. 4, alg. 1), reusing its branch logits with no extra forward passes and reducing exactly to MAD when inactive, so the comparison is well controlled.
- The performance increase is replicated on four different omni LLMs (table 1), with ablations isolating C, B, and their product (table 5), demonstrating the benefit is not specific to a single model architecture.
Critical weaknesses
- The claim that CAD’s benefits extend beyond hallucination benchmarks (sec. 5.2) feels overstated. In table 2, the WorldSense gain over Base is 2.9 pp at already low accuracy, and the gain on VideoMME is 1.4 pp. The reported results are only on VideoLLaMA2-AV.
- The gains over MAD are small (1.3–4.4 pp) and the sensitivity of CAD to its hyperparameters () is not examined. No confidence intervals or significance tests are reported. Lessons from the hyperparameter sweep and bootstrap CIs on the CAD–MAD difference would establish stronger evidence.
- The actionability rule cannot distinguish destructive fusion from complementary recovery. Both and present as and receive the same B in eq. 10. Fig. 2 Case 1 is a case CAD would shift weight toward the wrong unimodal answer. A count of AV-correct predictions flipped to wrong versus AV-wrong fixed would substantiate the “preserving complementarity” claim.
Questions for authors
- Do the table 2 gains hold on Qwen2.5-Omni-7B or OmniVinci?
- How were () selected?
- Per benchmark and per eq. 10 case, how many predictions does CAD flip from correct to wrong versus wrong to correct?
5 comments
Response to Reviewer YMW4. Part [2/3]
2. Suggestion on hyperparameter selection, sensitivity, and statistical reliability.
Thank you very much for your suggestions. We have clarified the hyperparameter settings and added sensitivity and paired bootstrap analyses to assess the reliability of CAD’s results.
First, regarding hyperparameter selection, our selection protocol draws on the parameter-tuning practices of AVCD and MAD. We pool approximately 10% of the samples from each benchmark to form a test subset for selecting , , and , with fixed at . Specifically, we first perform a coarse-grained parameter search on this subset, then refine the search within promising regions and select the final configuration with the highest accuracy on this subset. This procedure selects decoding hyperparameters without training or updating model weights. We have added the details of this selection procedure to Appendix B of the revised manuscript.
Second, we conducted paired bootstrap analyses on CMM and AVHBench to quantify CAD’s accuracy gains. To account for dependence among questions associated with the same video, we performed 20,000 bootstrap resamples at the video level, retaining all questions from each sampled video, and calculated the 95% percentile intervals for the overall accuracy differences. The experimental results are as follows:
| Backbone | Benchmark | CAD-Base | 95% bootstrap CI | CAD-MAD | 95% bootstrap CI |
|---|---|---|---|---|---|
| VideoLLaMA2-AV | CMM | +7.500 | [4.750, 10.250] | +1.375 | [−0.125, 3.000] |
| VideoLLaMA2-AV | AVHBench | +4.437 | [3.137, 5.742] | +1.313 | [0.852, 1.792] |
| OmniVinci | CMM | +4.125 | [2.250, 6.000] | +2.375 | [1.125, 3.750] |
| OmniVinci | AVHBench | +6.947 | [5.712, 8.180] | +3.649 | [2.774, 4.547] |
| Qwen2.5-Omni-3B | CMM | +7.000 | [4.625, 9.375] | +1.250 | [−0.125, 2.625] |
| Qwen2.5-Omni-3B | AVHBench | +6.246 | [5.101, 7.408] | +1.664 | [0.801, 2.544] |
| Qwen2.5-Omni-7B | CMM | +14.125 | [11.250, 17.125] | +4.375 | [2.375, 6.375] |
| Qwen2.5-Omni-7B | AVHBench | +8.027 | [6.838, 9.240] | +2.948 | [2.265, 3.752] |
Across all eight settings, CAD achieves higher accuracy than both Base and MAD, with most 95% paired bootstrap intervals lying above zero. These results provide further empirical and statistical support for CAD’s effectiveness across the evaluated settings. We have now included these paired bootstrap results in Appendix D of the revised manuscript.
Third, using OmniVinci on CMM and AVHBench as in Table 1 of our paper, we evaluated CAD’s sensitivity to , , and . We varied one parameter at a time around while keeping the other two fixed. Each parameter configuration was evaluated on both benchmarks. The experimental results are shown in the following tables:
Sensitivity to (, ).
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | |
|---|---|---|---|---|---|---|---|
| CMM accuracy (%) | 87.1 | 87.9 | 87.4 | 88.0 | 86.8 | 86.5 | 86.3 |
| AVHBench accuracy (%) | 78.8 | 79.6 | 80.0 | 80.3 | 80.2 | 80.1 | 79.6 |
| Avg. (%) | 83.0 | 83.8 | 83.7 | 84.1 | 83.5 | 83.3 | 82.9 |
Sensitivity to (, ).
| 5 | 10 | 15 | 20 | 25 | 30 | 35 | |
|---|---|---|---|---|---|---|---|
| CMM accuracy (%) | 86.5 | 87.0 | 87.3 | 88.0 | 87.5 | 87.4 | 87.3 |
| AVHBench accuracy (%) | 79.0 | 79.7 | 80.0 | 80.3 | 80.2 | 80.3 | 80.4 |
| Avg. (%) | 82.7 | 83.4 | 83.6 | 84.1 | 83.8 | 83.8 | 83.8 |
Sensitivity to (, ).
| 0.03 | 0.04 | 0.05 | 0.06 | 0.07 | 0.08 | 0.09 | |
|---|---|---|---|---|---|---|---|
| CMM accuracy (%) | 87.5 | 87.5 | 88.0 | 87.5 | 87.4 | 87.1 | 87.0 |
| AVHBench accuracy (%) | 80.2 | 80.2 | 80.3 | 80.0 | 79.8 | 79.5 | 79.4 |
| Avg. (%) | 83.8 | 83.8 | 84.1 | 83.8 | 83.6 | 83.3 | 83.2 |
The results show that CAD maintains stable performance across neighboring parameter values in the one-at-a-time sweeps, including , , and . Furthermore, CAD exhibits overlapping stable parameter ranges on CMM and AVHBench, supporting the use of a common hyperparameter configuration across both benchmarks with OmniVinci as the backbone. We have now included these sensitivity results in Section 5.3 of the revised manuscript.
Response to Reviewer YMW4. Part [3/3]
3. Suggestion about distinguishing destructive fusion from complementary recovery.
Thank you for your valuable suggestions. They prompted us to further clarify CAA’s intervention mechanism and quantitatively evaluate CAD’s effectiveness. We provide both a mechanistic explanation and an analysis of actual prediction changes below.
First, the same answer relation () can indeed correspond to either destructive fusion or complementary recovery, but CAA additionally considers query relevance and the strength of unimodal support for the predicted answers to assess the evidence for intervention. Specifically, drawing on the reliability-discounting principle of Dempster–Shafer theory, CAA combines modality relevance and answer margins as , assigning as support for the unimodal answer and leaving the remaining mass uncommitted to represent ignorance. Since the two branches share the backbone, query, and language prior, CAA conservatively aggregates their shared support using when . For Figure 2 Case 1, smaller unimodal margins reduce support for intervention. When , the coefficient transfer is jointly modulated by conflict magnitude and reliability evidence.
Second, we compared the predictions of Base and CAD across four backbones on CMM and AVHBench. Corrections correspond to wrong→correct flips, whereas introduced errors correspond to correct→wrong flips. The results are as follows:
| Backbone | Benchmark | Wrong→Correct | Correct→Wrong | Net increase in correct answers | CAD−Base | 95% bootstrap CI |
|---|---|---|---|---|---|---|
| VideoLLaMA2-AV | CMM | 96 | 36 | +60 | +7.500 | [4.750, 10.250] |
| VideoLLaMA2-AV | AVHBench | 366 | 214 | +152 | +4.437 | [3.137, 5.742] |
| OmniVinci | CMM | 45 | 12 | +33 | +4.125 | [2.250, 6.000] |
| OmniVinci | AVHBench | 347 | 109 | +238 | +6.947 | [5.712, 8.180] |
| Qwen2.5-Omni-3B | CMM | 77 | 21 | +56 | +7.000 | [4.625, 9.375] |
| Qwen2.5-Omni-3B | AVHBench | 308 | 94 | +214 | +6.246 | [5.101, 7.408] |
| Qwen2.5-Omni-7B | CMM | 136 | 23 | +113 | +14.125 | [11.250, 17.125] |
| Qwen2.5-Omni-7B | AVHBench | 325 | 50 | +275 | +8.027 | [6.838, 9.240] |
Across all eight settings, CAD corrects more errors than it introduces, and all 95% paired bootstrap confidence intervals for the accuracy gains lie above zero. Moreover, corrections account for 63.1%-86.7% of the cases where Base and CAD differ in correctness, which further supports CAD’s ability to preserve complementarity.
Third, as requested, we report prediction flips by benchmark and by the answer-relation cases in Eq. (10). The analysis covers four backbones on CMM and AVHBench. We assign each sample to an Eq. (10) case using the initial task-space predictions of the A, V, and AV branches, and compare the evaluated answers of Base and CAD.
The four cases are:
- I: .
- II: .
- III: and .
- IV: and .
Since both CMM and AVHBench use binary answers, Case IV has zero samples and zero prediction flips across all settings. In Case I, and CAD falls back to MAD decoding without additional conflict intervention. The results are presented in the following tables:
CMM
| Backbone | Eq. (10) case | Wrong→Correct | Correct→Wrong |
|---|---|---|---|
| VideoLLaMA2-AV | I | 13 | 4 |
| VideoLLaMA2-AV | II | 3 | 3 |
| VideoLLaMA2-AV | III | 80 | 29 |
| OmniVinci | I | 1 | 0 |
| OmniVinci | II | 3 | 2 |
| OmniVinci | III | 41 | 10 |
| Qwen2.5-Omni-3B | I | 14 | 3 |
| Qwen2.5-Omni-3B | II | 18 | 7 |
| Qwen2.5-Omni-3B | III | 45 | 11 |
| Qwen2.5-Omni-7B | I | 12 | 10 |
| Qwen2.5-Omni-7B | II | 7 | 1 |
| Qwen2.5-Omni-7B | III | 117 | 12 |
AVHBench
| Backbone | Eq. (10) case | Wrong→Correct | Correct→Wrong |
|---|---|---|---|
| VideoLLaMA2-AV | I | 45 | 48 |
| VideoLLaMA2-AV | II | 50 | 28 |
| VideoLLaMA2-AV | III | 271 | 138 |
| OmniVinci | I | 47 | 43 |
| OmniVinci | II | 87 | 12 |
| OmniVinci | III | 213 | 54 |
| Qwen2.5-Omni-3B | I | 64 | 12 |
| Qwen2.5-Omni-3B | II | 92 | 20 |
| Qwen2.5-Omni-3B | III | 152 | 62 |
| Qwen2.5-Omni-7B | I | 61 | 6 |
| Qwen2.5-Omni-7B | II | 65 | 16 |
| Qwen2.5-Omni-7B | III | 199 | 28 |
The results show that CAD’s net gains over Base arise primarily from Cases II and III, indicating that CAD corrects more errors than it introduces when branch predictions disagree. This further supports CAD’s effectiveness in mitigating cross-modal hallucinations. We have now included the detailed prediction-change analyses in Appendix G of the revised manuscript.
We once again thank you for your valuable comments! If you have any further questions, please feel free to let us know and we will do our utmost to respond.
Response to Reviewer YMW4. Part [1/3]
We appreciate your thoughtful and constructive feedback, and sincerely thank you for highlighting our strengths (e.g., the well-motivated problem formulation, clear presentation, and consistent empirical gains). Your comments have helped us clarify several important aspects of our work. We hope our responses address your concerns. All manuscript revisions discussed below are highlighted in blue in the resubmitted manuscript.
1. Suggestion about general QA performance and evaluation on additional backbones.
Thank you for your valuable suggestion, which made us realize that including more comprehensive statistical analyses and evaluations across backbones would further strengthen the reliability of our paper's conclusions. Following your suggestion, we have refined the wording in Section 5.2 of the revised manuscript and added the following quantitative analyses and experimental results.
First, we compiled the historical per-question results of VideoLLaMA2-AV on the two general QA benchmarks and provide more precise overall accuracy statistics:
Beyond overall accuracy, we conducted paired bootstrap analyses of the accuracy differences comparing CAD with Base and MAD. Specifically, we used 20,000 resamples at the video level, retaining all questions from each sampled video, to construct 95% intervals. The analysis covered 3,172 questions from 1,662 videos in WorldSense and 2,700 questions from 900 videos in VideoMME.
The results show that CAD achieves higher accuracy, with all four 95% paired bootstrap intervals for the accuracy gains lying above zero. These provide further statistical support for CAD’s gains on the general audio-visual benchmarks.
Second, to address your question about additional backbones, we added evaluations comparing Base, MAD and CAD using Qwen2.5-Omni-7B and OmniVinci on WorldSense and VideoMME. The experimental results are as follows:
Across all four additional settings, CAD achieves higher overall accuracy than both Base and MAD, with gains of 0.71-1.36 percentage points over Base and 0.63-1.37 percentage points over MAD. These results show that CAD consistently achieves accuracy gains across the backbones and general audio-visual benchmarks. We have now included the additional results in Table 2 and the detailed results and statistical analyses in Appendix D of the revised manuscript.