acceptodds
Under review as a conference paper at ICLR 2027

Relevant Evidence Decoding for Audio-Visual Hallucination Mitigation

Abstract

Audio-Visual Large Language Models (AV-LLMs) remain prone to cross-modal hallucinations, where one modality incorrectly affects predictions about another. Although contrastive decoding reduces hallucinations in vision-language models, its direct extension to AV-LLMs overlooks a key challenge: different questions require different perceptual evidence, including audio, video, or their interaction. Notably, we observe that joint audio-visual inference can weaken the prediction even when a model can recover the correct answer from a single informative modality. For example, when asked which instrument is heard, a model may correctly predict violin from the audio alone. Once a video showing a guitar is added, its confidence in violin may drop. In this paper, we introduce Relevant Evidence Decoding (RED), a training-free method that identifies question-relevant evidence and selectively strengthens its contribution. RED uses pointwise mutual information to quantify the predictive support provided by audio and video beyond the question alone. It decomposes their joint contribution into audio, video, and residual interaction components. A question-only inference pass determines the required evidence type, after which the model augments the original audio-visual prediction with the corresponding PMI contribution. Across three audio-visual hallucination benchmarks and three AV-LLMs, RED improves accuracy over standard decoding by up to 7% on CMM, 6.3% on AVHBench, and 3.8% on SVHalluc, with an average relative time to first token of 1.5 standard decoding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Contrastive Decoding, Multimodal Hallucination

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_T9mSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Relevant Evidence Decoding (RED), a training-free decoding method for reducing cross-modal hallucinations in audio-visual LLMs. RED first asks the frozen backbone, using the question alone, to classify the required evidence as none, audio, video, additive audio plus video, or an audio–video relation. It then compares audio-only, video-only, joint audio-visual, and question-only next-token distributions. Differences from the question-only branch are treated as conditional-PMI contributions, while the non-additive remainder is treated as an interaction contribution. The selected contribution is added to a centered audio-visual score during decoding.

Critical strengths
  1. RED is a simple, well-motivated decoding rule with broad empirical support. The method uses a common question-only reference to distinguish audio, video, additive, and residual interaction contributions, and then evaluates only the auxiliary branches required by the selected evidence type.

  2. The analysis of this paper goes beyond a leaderboard comparison. The paper evaluates answer-changing and answer-preserving modality interventions, separates TPR from FPR, compares against a base threshold sweep, ablates selection and PMI decomposition, tests mismatched/random/heuristic/oracle assignments, examines contribution-specific loss landscapes, and reports detailed TTFT breakdowns. These analyses substantively support the evidence-following interpretation.

Critical weaknesses
  1. RED uses a benchmark for calibration. It estimates score offsets from 100 questions sampled from each evaluation benchmark and then evaluates on the full set, including those questions. Results using a disjoint calibration set or no centering are needed to separate RED’s core contribution from transductive calibration.

  2. The evidence selector is not directly validated. The paper provides no selector accuracy or confusion matrix against available evidence labels. In addition, the residual captures statistical non-additivity but does not necessarily represent a semantic audio–visual interaction.

Questions for authors
  1. Can the authors report RED results when centering offsets are estimated from a disjoint development split or transferred from another benchmark, rather than sampled from the evaluation inputs?

  2. Is score centering applied over the full vocabulary at every generation step, and are the 100 questions used to estimate offsets excluded from any reported evaluation in an alternative protocol?

4 comments
AuthorsOpenReview ↗

Response to Reviewer T9mS [1/2]

Dear Reviewer T9mS, We sincerely appreciate your constructive feedback and your recognition of RED’s formulation, selective inference, and analyses beyond benchmark accuracy.

W1 & Q1: Potential transductive effects of benchmark-specific centering and its generalization to disjoint calibration splits or unseen benchmark distributions remain unclear.

Unlike MAD and AVCD, which use 100 labeled examples per dataset to tune decoding strengths, RED estimates fixed offsets from 100 unlabeled questions without such tuning. RED's evidence correction remains effective without centering, and its calibrated performance is maintained after excluding calibration questions. Three complementary analyses establish the correction's contribution, check direct sample overlap, and assess offset transfer.

Evidence correction without calibration. Submitted Appendix A.2, Table A already compares RED with and without centering. We show these CMM results alongside Base from Table 1:

BackboneBase (%)RED without centering (%)RED with centering (%)
Qwen-7B82.287.389.2
Qwen-3B83.386.186.2
VideoLLaMA282.885.585.6

RED's core evidence correction remains effective without benchmark-specific centering. These results support using RED without estimating offsets. Centering complements this correction by further adjusting systematic score bias.

Direct sample overlap. Using Appendix Figure C's same five calibration seeds, we exclude the 100 calibration questions per draw, keeping offsets and predictions fixed. Table B reports CMM task-macro accuracy (mean ± SD):

BackboneFull set (%)Excluding calibration questions (%)
Qwen-7B89.5 ± 0.389.5 ± 0.3
Qwen-3B86.2 ± 0.586.2 ± 0.6
VideoLLaMA285.1 ± 0.985.1 ± 1.0

CMM mean accuracy remains unchanged across all three backbones. RED therefore retains its gains when calibration questions are removed from evaluation, addressing the concern about direct sample overlap. These five-draw averages can differ from Table A's single-draw values; transfer across distributions is examined next.

Cross-benchmark transfer. Following the compatible-answer setting of cross-domain calibration [1], we transfer binary offsets only between tasks with matching candidate spaces. TF/CMTB have three choices, so binary offsets cannot directly calibrate their extra candidate. We include all six CMM subsets, three AVHBench tasks, and four binary SVHalluc tasks (GSA, FGSA, CMSB, TA).

Each source draw contains 100 unlabeled questions; SVHalluc's binary offsets use its binary items (63–85 per draw). Cells report accuracy / recall standard deviation (RStd) (%), averaged over five draws and three backbones. Following [1], RStd measures candidate-recall imbalance; we compute its population SD per task. Lower is more balanced. Diagonal entries use target calibration; labels are used only for evaluation.

Target / SubsetCMM offsetsAVHBench offsetsSVHalluc offsets
CMM / VL92.3 / 3.092.1 / 3.889.8 / 8.2
CMM / AL85.2 / 8.185.0 / 6.982.4 / 10.6
CMM / VAL93.6 / 3.092.3 / 5.290.3 / 7.8
CMM / Vis.82.1 / 4.081.7 / 7.380.4 / 8.6
CMM / Aud.88.4 / 2.688.6 / 2.784.8 / 9.4
CMM / Lang.80.0 / 5.778.8 / 6.275.2 / 14.5
AVHBench / ADV81.2 / 4.682.1 / 6.378.1 / 15.8
AVHBench / VDA80.2 / 10.382.1 / 5.079.1 / 11.5
AVHBench / AVM51.3 / 48.371.3 / 8.051.5 / 48.4
SVHalluc / GSA66.1 / 26.754.0 / 45.663.6 / 22.8
SVHalluc / FGSA81.6 / 10.782.5 / 10.584.3 / 8.3
SVHalluc / CMSB79.9 / 15.281.2 / 13.583.8 / 5.7
SVHalluc / TA50.0 / 49.950.4 / 45.951.2 / 37.5

CMM subsets retain similar accuracy with AVHBench offsets, while SVHalluc offsets transfer less consistently. TA remains near chance across sources.

The AVM accuracy drop coincides with sharply increased candidate-recall imbalance. With CMM/SVHalluc offsets, accuracy falls from 71.3% to 51.3%/51.5%, while RStd rises from 8.0% to 48.3%/48.4%. Predictions therefore become highly uneven across answer candidates. This occurs even with Relation-containing SVHalluc calibration, suggesting that evidence-type coverage alone is insufficient to transfer candidate-score calibration.

These results clarify how to use RED on new sources: retain its evidence correction and estimate offsets from unlabeled target samples when the distribution changes substantially. Without such samples, the no-centering results demonstrate the correction's standalone benefit. We added the expanded evaluation in blue in Appendix A.2, Table C; Tables A–B establish the core benefit and its persistence under disjoint evaluation.

References

[1] Zheng et al., Large Language Models Are Not Robust Multiple Choice Selectors, ICLR 2024.

AuthorsOpenReview ↗

Response to Reviewer T9mS [2/2]

W2-1: The evidence selector lacks direct validation against available evidence labels.

We validate Qwen-7B's question-only selector against the official MUSIC-AVQA tags. Audio-Visual tags do not separate Both from Relation, so either output counts as agreement; the matrix retains all five outputs.

Split / TagnAudioVideoBothRelationNoneRecall (%)
Full / Audio1,6111,49101181192.6
Full / Visual2,4222762,14150088.4
Full / AV5,0967213923,9812078.2
Hard / Audio3843300531085.9
Hard / AV2,1242601311,7330081.6
SplitAgreement (%)Macro-F1 (%)Disagreements: RED → Oracle (%)Full set: RED → Oracle (%)
Full83.481.972.5 → 76.167.1 → 67.7
Hard82.378.248.8 → 55.340.5 → 41.6

Macro-F1 averages the reference classes present. Oracle replaces predictions with reference assignments (AV→Both). These controlled replacements show that more accurate evidence assignments can further improve RED, particularly on Hard. Section 4.3, Table 5 reports selector agreement, macro-F1, and disagreement-case accuracy; Table 4 reports the full-set Oracle comparison.

W2-2: Statistical non-additivity does not necessarily represent semantic audio–visual interaction.

We agree that statistical non-additivity does not guarantee semantic isolation; our controlled analyses nevertheless support ΔI\Delta^I as a correspondence-sensitive predictive signal. ΔI\Delta^I is the non-additive residual in Eq. (4). Section 3.3 now explicitly states that statistical non-additivity alone does not guarantee semantic isolation. The submitted contribution comparison in Table D supports ΔI\Delta^I on AV-Matching; we provide complementary correspondence interventions below.

We swap audio or video with another clip for 938 originally matching questions (GT=Yes), using 938 originally mismatched questions (GT=No) as a control. For each contribution, we measure the rate at which its Yes–No margin decreases and the AUROC distinguishing original from replaced conditions. The table reports decrease rate / AUROC (%) for originally matching questions.

ContributionAudio swap, 7BAudio swap, 3BVideo swap, 7BVideo swap, 3B
ΔA\Delta^A55.8 / 60.657.0 / 63.80.0 / 50.00.0 / 50.0
ΔV\Delta^V0.0 / 50.00.0 / 50.011.5 / 49.711.9 / 49.9
ΔI\Delta^I83.6 / 80.772.0 / 74.046.6 / 57.337.8 / 55.3

Under audio replacement, the residual decreases on 83.6%/72.0% of matching questions, versus 48.7%/51.3% of mismatched controls. This contrast suggests that the residual contains correspondence-sensitive signals beyond a general response to audio replacement. Video replacement produces weaker discrimination (AUROC 57.3%/55.3%), showing modality asymmetry. Together with Table D, these interventions provide empirical support for using the residual on audio-visual correspondence questions and extend the submitted Figure 3's evidence-following analysis.

Q2: Scope of score centering and exclusion of calibration samples.

Scope of score centering. Following the option-/class-level calibration principles of PriDe [1] and Batch Calibration [2], already cited in the submission, we apply centering to candidate-answer scores in the reported QA experiments rather than to the full vocabulary at every decoding step. We estimate candidate-specific offsets for the AV score and evidence contributions from 100 unlabeled questions, conditioned on the selected evidence type (Appendix A.2). These offsets remain fixed during evaluation. We have clarified this scope in Section 3.3.

The same token-level formulation can also be applied to the full vocabulary, using a fixed offset for each token at every decoding step. The centering rule is unchanged; the QA implementation calibrates the candidate-answer tokens.

Excluding calibration questions. As reported in our response to W1 & Q1, RED retains its gains when the 100 calibration questions are excluded from evaluation.

References

[1] Zheng et al., Large Language Models Are Not Robust Multiple Choice Selectors, ICLR 2024.

[2] Zhou et al., Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering, ICLR 2024.

Thank you for your response and clarifications. After considering them, I will maintain my original score.

We sincerely thank you for taking the time to revisit our submission. Your thorough and constructive feedback throughout the review process has meaningfully strengthened both the empirical controls and the clarity of our manuscript. Please let us know if there is anything else we can clarify before the discussion period concludes.

Reviewer_EkKkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Relevant Evidence Decoding (RED), a training-free contrastive decoding based approach to mitigate audio-visual hallucinations. It selects question-relevant evidence and uses a PMI-based decomposition to modify decoding scores. Evaluation covers three backbones, various hallucination benchmarks, general audio-visual QA, and diagnostic analyses. It improves accuracy across multiple benchamrks with modest extra inference latency.

Critical strengths
  • Effective improvements with practical inference costs. RED achieves notable performance gains without additional model training, while maintaining a reasonable TTFT increase. This combination of effectiveness and moderate inference overhead makes the method practically appealing.

  • Comprehensive and controlled baseline comparisons. The paper evaluates a broad range of contrastive decoding baselines under consistent and reasonable experimental settings. These comparisons make the empirical improvements convincing and provide a clear assessment of RED’s effectiveness relative to existing decoding approaches.

Critical weaknesses
  • Limited methodological novelty over prior modality-adaptive decoding. MAD (MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models, Chung et al., CVPR 2026) already assesses which modalities are relevant to a question and uses this assessment to adapt contrastive decoding.
  • Limited validation beyond constrained short-answer tasks. The evaluation focuses on binary judgments, multiple-choice questions, and short-answer QA with a limited answer space. These settings provide limited evidence that RED reduces hallucinations during truly free-form generation, e.g., caption,explanatory reasoning, etc. Moreover, there's still lack of evidence supporting RED can mitigate hallucinations during long generations.
Questions for authors
  • Personaly, I find the audio-visual contribution measures interesting and would like to understand what they reveal about modality imbalance. For audio-required questions, how often does adding video lower the probability of the correct answer, and by how much? Does adding audio have a similar effect on video-required questions? I am also interested in how the interaction residual is distributed across question types and models. These observations could be useful for studying modality imbalance and designing benchmarks for omni-modal models.
  • Beyond TTFT, what are RED’s end-to-end latency, throughput, and peak GPU memory relative to Base under matched conditions? A breakdown by evidence type would help establish whether the reported efficiency extends to cases requiring more auxiliary branches and to multi-token generation. Also, could authors provide a straight-forward explaination why RED is so much faster than VCD? My intuition is RED requires an extra forward to select evidence, thus it should be slower.
  • Table 4 evaluates the downstream effects of different selection strategies, but how accurately does the question-only selector identify the required evidence type? Could the authors report per-class accuracy or macro-F1 and a confusion matrix on an annotated subset, particularly for Both versus Relation? How does RED perform when the selected evidence type is incorrect?
  • How well do the score-centering offsets generalize beyond the benchmark distribution used to estimate them? Does RED retain its gains when the offsets are estimated on a disjoint calibration split or transferred from another dataset?
6 comments
AuthorsOpenReview ↗

Response to Reviewer EkKk [1/3]

Dear Reviewer EkKk, We sincerely appreciate your thoughtful and constructive review. Thank you for recognizing RED's effectiveness without additional training, practical inference efficiency, and comprehensive baseline comparisons.

W1: Methodological novelty over MAD and existing modality-adaptive decoding methods is limited.

RED retains the full AV prediction and selectively augments question-required evidence contributions. MAD adaptively weights perturbation-based signals by modality relevance. RED measures audio, video, and joint contributions relative to a common question-only prediction, discounting textual support.

This enables additive ΔA+ΔV\Delta^A+\Delta^V and non-additive ΔI=ΔAV−ΔA−ΔV\Delta^I=\Delta^{AV}-\Delta^A-\Delta^V contributions. Question-guided selection chooses the additive term for Both and the residual for Relation (Section 3.3, Eqs. 3–7). Both requires information from both modalities; Relation queries their relationship. We have clarified this distinction in the Introduction.

Submitted Table 3 tests PMI contributions with selection and other settings fixed (AVHBench, three-backbone means):

Correction with question-guided selectionAccuracy (%)AUROC (%)
Raw modality-conditioned scores76.282.7
PMI-based contributions (RED)78.686.1

These gains support PMI correction; submitted tests (now Tables D–E) support distinct Both/Relation terms. Compared with MAD, RED improves sensitivity (57.0% → 64.2%) and invariance violation (11.2% → 6.6%; Figure 3). Selective computation also yields higher accuracy at lower TTFT: 73.6% → 75.3%, 2.6× → 1.5× Base (Figure 5, three backbones/three hallucination benchmarks).

W2: Limited validation of hallucination mitigation in open-ended generation.

We evaluate the Audio-visual Captioning subset of AVHBench (1,106 items) using VideoLLaMA2.1-7B-AV, with one caption per item per method. Scores range from 0–5.

MethodCorrectness ↑Detailedness ↑
Base1.90 ± 0.021.78 ± 0.01
MAD1.78 ± 0.031.71 ± 0.01
AVCD1.86 ± 0.021.86 ± 0.01
RED1.93 ± 0.031.81 ± 0.02

Methods share the prompt, greedy decoding, and bfloat16. MAD/AVCD use strength 2.5; MAD/AVCD/RED share AVCD's plausibility threshold 0.2 [1]. RED disables centering; its selector chooses Both for the common prompt used for all samples. AVCD's GPT-3.5-turbo was unavailable in our setup; we use GPT-4o with VideoLLaMA2's evaluation prompts [2]. Mean ± SD summarizes three judging passes.

RED modestly improves both metrics over Base and correctness over MAD/AVCD; AVCD leads in detailedness. We added results in blue (Section 4.3, Table 6; Appendix B.3); longer-form generation remains future work (Appendix E).

References

[1] Jung et al., AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding, NeurIPS 2025.

[2] Cheng et al., VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs, arXiv:2406.07476, 2024.

AuthorsOpenReview ↗

Response to Reviewer EkKk [2/3]

Q1: The frequency of modality interference and the behavior of interaction residuals require further analysis.

Complementing the submitted aggregate analysis in Figure H, we report question-level interference below. Without correction or centering, we compare the first-answer-token correct-answer probability normalized over Yes/No, p(y)/(p(Yes)+p(No))p(y)/(p(\mathrm{Yes})+p(\mathrm{No})), under each input condition.

Here yy is the reference answer. Changes are computed before rounding.

Backbone / Required evidenceRequired OnlyFull AVChangeDecreased / Total (%)
Qwen-7B / Audio0.7370.690−0.0471,128 / 2,290 (49.3)
Qwen-3B / Audio0.6820.608−0.0741,556 / 2,290 (67.9)
VideoLLaMA2 / Audio0.7620.700−0.0621,263 / 2,290 (55.2)
Qwen-7B / Video0.8100.726−0.083887 / 1,136 (78.1)
Qwen-3B / Video0.7070.669−0.037699 / 1,136 (61.5)
VideoLLaMA2 / Video0.7320.756+0.024444 / 1,136 (39.1)

Video lowers mean support across backbones. Audio helps VideoLLaMA2 on average but hurts 39.1% of questions. More input can weaken correct-answer support, motivating selective correction. This complements MUSIC-AVQA-Hard [1], which removes questions answerable from a single video frame without audio: irrelevant inputs can weaken support even when required evidence is available.

For the uncentered residual margin rI=ΔI(Yes)−ΔI(No)r^I=\Delta^I(\mathrm{Yes})-\Delta^I(\mathrm{No}), cells show mean ± SD / residual-only AUROC (%). Positive margins favor Yes (Eq. 4).

TypeQwen-7BQwen-3BVideoLLaMA2
Audio+0.51 ± 1.43 / 29.9+0.18 ± 1.14 / 17.7+0.76 ± 2.48 / 38.8
Video+1.05 ± 1.70 / 15.5−0.04 ± 0.96 / 21.2+0.06 ± 2.68 / 58.3
Relation−1.89 ± 1.08 / 80.7−1.49 ± 0.69 / 70.3+0.77 ± 0.81 / 67.5

Relation (AVM, n=1,876) has the smallest spread and highest residual-only AUROC per backbone. These results complement the corrected-score comparison in Table D and support using the residual on AV-Matching.

Q2: End-to-end inference costs and RED’s efficiency advantage over VCD require clarification.

QA evaluation and single-token inference. With Qwen2.5-Omni-7B, QA accuracy is the unweighted mean over AVHBench's three binary tasks (ADV, VDA, and AVM), following RED's evaluation setup (Appendices B.1 and B.3). The Base, VCD (Audio-Visual Extension), and RED accuracy and relative TTFT values below are taken from Table 1 and Table G, respectively. VCD (Audio-Visual Extension) retains VCD's original decoding rule with audio, video, and joint audio-visual corruption branches, following AVCD [2] (Appendix B.2). TTFT and peak GPU memory use single-token generation. We additionally report the visual-only VCD comparison and peak-memory measurements below.

MethodAVHBench Acc. (%) ↑E2E TTFT (×Base) ↓Peak GPU Memory (GiB) ↓
Base76.21.0×17.9
VCD (Visual-only)74.81.7×19.1
VCD (Audio-Visual Extension)75.33.3×21.4
RED81.91.4×18.0

16/64-token generation costs. Separately from the QA accuracy evaluation above, we report latency and throughput below using a shared AV-description prompt with fixed output lengths of 16 or 64 new tokens. These E2E measurements include media decoding/preprocessing, prompt processing, and generation through the final token.

Method16-token latency (×Base) ↓64-token latency (×Base) ↓16-token throughput (tok/s) ↑64-token throughput (tok/s) ↑
Base1.00×1.00×6.1216.51
VCD (Visual-only)1.76×1.84×3.478.99
VCD (Audio-Visual Extension)3.32×3.52×1.844.68
RED1.65×2.06×3.718.00

RED achieves higher QA accuracy and lower inference costs than VCD (Audio-Visual Extension). VCD (Visual-only) is faster at 64 tokens, with lower QA accuracy.

Why RED is faster than VCD (Audio-Visual Extension). RED's selector runs once per question using only text, and its cost is included in the reported timings. In the separate profiling runs averaged over three backbones and three benchmarks, selection accounts for 89.1 of 5,789.9 ms, approximately 1.5% of RED's total runtime (Appendix A.4, Table F).

VCD (Audio-Visual Extension) evaluates the original AV input plus three corrupted inputs, each retaining both audio and video. RED uses one AV branch and only the required auxiliary branches: Q+A for Audio, Q+V for Video, and Q+A+V for Both/Relation. Q processes text only, while A and V each process one modality. This reduces perceptual processing and input length in the auxiliary branches, allowing RED to remain faster despite the additional selector pass.

References

[1] Kim and Seo, Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy, Interspeech 2026.

[2] Jung et al., AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding, NeurIPS 2025.

AuthorsOpenReview ↗

Response to Reviewer EkKk [3/3]

Q3: Evidence selector accuracy, Both–Relation confusion, and the impact of incorrect selection remain unclear.

Correcting selector–tag disagreements improves accuracy. Qwen-7B/MUSIC-AVQA: Both/Relation agree with AV; Oracle maps AV to Both. The matrix retains all five outputs:

Split / TagnAudioVideoBothRelationNoneRecall (%)
Full / Audio1,6111,49101181192.6
Full / Visual2,4222762,14150088.4
Full / AV5,0967213923,9812078.2
Hard / Audio3843300531085.9
Hard / AV2,1242601311,7330081.6

Agreement / macro-F1 is 83.4% / 81.9% on Full and 82.3% / 78.2% on Hard. Macro-F1 averages the reference classes present. Oracle results:

SplitDisagreementsRED → Oracle on disagreements (%)Full-set RED → Oracle (%)
Full1,514 / 9,12972.5 → 76.167.1 → 67.7
Hard445 / 2,50848.8 → 55.340.5 → 41.6

Section 4.3, Table 5 reports selector agreement, macro-F1, and disagreement-case accuracy; Table 4 reports the full-set Oracle comparison. Oracle additionally maps two Full AV-tagged Relation outputs to Both.

Dominant Both–Relation choices match the better correction in each group. We complement the submitted contribution comparisons in Tables D–E with selection counts below. Accuracy uses each entire question group, other settings fixed:

Group / ReferencenSelected Relation / BothAccuracy: ΔI\Delta^I / ΔA+ΔV\Delta^A+\Delta^V (%)
AV-Matching / Relation1,8761,876 / 076.3 / 71.8
MUSIC-Hard AV / Both-mapped2,1240 / 1,73330.2 / 40.2

Other 391 Hard AV questions select Audio/Video; 276 TF questions select Both despite Relation task labels.

Q4: Generalization of centering offsets to disjoint evaluation sets and other benchmark distributions requires clarification.

Excluding calibration questions retains accuracy, but offset transfer varies across benchmarks. We fix offsets and exclude 100 questions per draw using Figure C's same five seeds (CMM task-macro %, mean ± SD; Table B):

BackboneFullDisjoint
Qwen-7B89.5 ± 0.389.5 ± 0.3
Qwen-3B86.2 ± 0.586.2 ± 0.6
VideoLLaMA285.1 ± 0.985.1 ± 1.0

All binary tasks are included under compatible-answer calibration [1]; TF/CMTB have three choices. Each draw samples 100 unlabeled source questions; SVHalluc uses binary items (63–85 per draw).

Cells: accuracy / recall standard deviation (RStd) (%), averaged over five draws/three backbones. RStd [1] is the population SD of candidate recalls per task; lower means less imbalance. Labels are evaluation-only.

Target / SubsetCMM offsetsAVHBench offsetsSVHalluc offsets
CMM / VL92.3 / 3.092.1 / 3.889.8 / 8.2
CMM / AL85.2 / 8.185.0 / 6.982.4 / 10.6
CMM / VAL93.6 / 3.092.3 / 5.290.3 / 7.8
CMM / Vis.82.1 / 4.081.7 / 7.380.4 / 8.6
CMM / Aud.88.4 / 2.688.6 / 2.784.8 / 9.4
CMM / Lang.80.0 / 5.778.8 / 6.275.2 / 14.5
AVHBench / ADV81.2 / 4.682.1 / 6.378.1 / 15.8
AVHBench / VDA80.2 / 10.382.1 / 5.079.1 / 11.5
AVHBench / AVM51.3 / 48.371.3 / 8.051.5 / 48.4
SVHalluc / GSA66.1 / 26.754.0 / 45.663.6 / 22.8
SVHalluc / FGSA81.6 / 10.782.5 / 10.584.3 / 8.3
SVHalluc / CMSB79.9 / 15.281.2 / 13.583.8 / 5.7
SVHalluc / TA50.0 / 49.950.4 / 45.951.2 / 37.5

AVM accuracy falls as RStd rises, indicating unequal candidate recalls even with Relation-containing SVHalluc offsets. This supports target recalibration. We added these results in blue in Appendix A.2, Table C.

References

[1] Zheng et al., Large Language Models Are Not Robust Multiple Choice Selectors, ICLR 2024.

Reviewer_tciYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies cross-modal hallucinations in audio-visual large models, focusing on whether a model answers using the perceptual evidence actually required by the question. The authors propose Relevant Evidence Decoding (RED), which requires no additional training. A frozen model first selects one of five evidence types from the question text: None, Audio, Video, Both, or Relation. RED then uses log-probability differences relative to question-only predictions to compute audio, video, and non-additive residual contributions, and adds the selected contribution to the original audio-visual prediction. Unlabeled examples are also used to estimate score-centering offsets.

The experiments cover three model backbones, three audio-visual hallucination benchmarks, MUSIC-AVQA, and its Hard subset. In addition to standard accuracy, the authors replace relevant or irrelevant modalities to evaluate sensitivity to necessary evidence and invariance to irrelevant evidence. They also provide component ablations, comparisons of selection strategies, and inference-efficiency analysis.

Critical strengths
  1. The problem is valuable and the evaluation goes beyond answer accuracy. The paper distinguishes answering correctly from answering on the basis of the appropriate evidence, using relevant- and irrelevant-modality replacements as complementary diagnostics. The sensitivity evaluation requires correct answers on both the original and modified examples, avoiding spurious success from constant predictions. This makes the evidence-dependence analysis informative for understanding hallucination mitigation.
  2. The method requires no additional training and improves performance across several backbones and benchmarks. For example, on Qwen2.5-Omni-7B, accuracy improves from 82.2% to 89.2% on CMM and from 76.2% to 81.9% on AVHBench. On MUSIC-AVQA-Hard, it improves from 31.9% to 40.5%. In the hallucination evaluations, average relative time to first token is 1.5 times that of standard decoding, compared with 2.6 times for MAD. These results support practical value and a useful trade-off between accuracy and inference cost.
Critical weaknesses
  1. The incremental value of full RED over simple modality selection needs stronger validation. In Table 4, Selected target-only already reaches 81.7% on AVHBench and 88.3% on CMM, compared with 81.9% and 89.2% for RED. Heuristic evidence selection also matches RED on these two benchmarks. This suggests that a substantial part of the improvement may come from question-driven modality selection, while the additional benefit and scope of the full correction remain less clear. On the other hand, RED exceeds target-only by 3.5 percentage points on MUSIC-AVQA-Hard, suggesting that its incremental value may depend on task difficulty or evidence requirements. I recommend extending the target-only comparison to the remaining backbones and SVHalluc, reporting inference cost and confidence intervals for paired performance differences to establish where the full method offers a stable advantage.
  2. The reliability of evidence-type classification and the propagation of its errors are not directly evaluated. RED selects the evidence type using only the question text and keeps that choice fixed throughout decoding. An incorrect type may therefore cause the subsequent correction to reinforce inappropriate evidence. Table 4 provides indirect analysis using random, mismatched, heuristic, and Oracle selections, but does not directly report per-type accuracy, a confusion matrix, or the downstream effect of classification errors, especially confusion between Both and Relation. A labeled or small manually verified subset could quantify these errors and compare downstream performance under correct and incorrect selections, clarifying the dependence on selection accuracy.
  3. Experimental support for the mapping from evidence types to correction terms remains localized. The method maps Both to the sum of the audio and video contributions and Relation to the non-additive residual. However, the algebraic decomposition alone does not guarantee correspondence to the semantic evidence requirements of a question. Table B validates the residual primarily on the AVM task using one backbone. In Table C, on the subset actually assigned Both by the model, the video contribution achieves 38.63% accuracy, slightly above 38.19% for the summed contribution; the summed contribution obtains the best result, 40.16%, only on the dataset-labeled bimodal subset. These results offer some support but do not establish that the mapping is stable across models and relational tasks. Comparisons among correction terms using verified evidence types and identical centering should be extended to another backbone and temporal-relation questions, separating selection errors from limitations of the mapping itself.
  4. The robustness of fixed correction weights has not been established across evidence types and tasks. RED uses a fixed unit weight for the selected correction. Figure 7 and Appendix A.3 do sweep the audio and video weights, so relevant analysis is present. However, it is limited to the Audio and Video groups in CMM and mainly reports binary-answer negative log-likelihood averaged across three backbones. It does not sufficiently establish sensitivity for Both and Relation corrections or the robustness of the default weight in terms of accuracy on other tasks. Weight-sensitivity results for these evidence types and additional benchmarks, including task accuracy, would better support the fixed-coefficient design.
Questions for authors
  1. What are the assumptions behind unlabeled centering calibration? The appendix states that centering offsets are estimated from 100 unlabeled examples per benchmark, which are also included in the final evaluation. Although no answer labels are used, the procedure uses information about the target distribution. Is prior access to unlabeled examples from that distribution a requirement of RED? For a new data source or online inputs, should the existing offsets be reused or recalibrated?
  2. How many examples are affected by invalid-answer filtering in the invariance evaluation? Appendix B.4 retains only pairs for which all methods and backbones produce valid binary answers. Please report the sample counts before and after filtering and the invalid-answer rate for each method and backbone. This would help determine whether excluded examples concentrate in difficult cases and whether the filtering affects representativeness.
  3. Does improved invariance also preserve answer correctness? The invariance metric measures whether a prediction changes after replacing an irrelevant modality, but agreement can also mean that both predictions are wrong. Could the authors report accuracy before and after replacement on this subset, or the proportion of pairs answered correctly in both conditions? The sensitivity experiment already requires correctness in both conditions, but it uses a different set of intervention examples and does not resolve the relationship between invariance and correctness.
5 comments
AuthorsOpenReview ↗

Response to Reviewer tciY [1/3]

Dear Reviewer tciY, We sincerely appreciate your constructive review and your recognition of our evidence-dependence analysis, improvements across backbones and benchmarks, and practical accuracy–cost trade-off.

W1 & Q3: Incremental benefit beyond modality selection and preservation of correct answers.

Beyond modality selection. For general AVQA, modality selection alone may omit complementary context or leave joint evidence undifferentiated. RED augments the full AV prediction with PMI-derived contributions. We extend submitted Table 4 to three backbones and five settings with the same selector and centering. The table reports Selected target-only and RED accuracy in separate columns, alongside RED minus Selected target-only, Δ\Delta [95% CI] (%) (Appendix C.5, Table P). Differences use unrounded accuracies.

BenchmarkBackboneSelected target-only (%)RED (%)Δ\Delta [95% CI] (%)
AVHBenchQwen-7B81.781.9+0.22 [−0.46, +0.91]
Qwen-3B78.478.5+0.11 [−0.70, +0.91]
VideoLLaMA271.175.3+4.12 [+3.05, +5.22]
CMMQwen-7B88.389.2+0.83 [+0.05, +1.61]
Qwen-3B85.486.2+0.79 [−0.09, +1.68]
VideoLLaMA285.285.6+0.46 [−0.55, +1.50]
SVHallucQwen-7B65.463.4−2.04 [−3.71, −0.35]
Qwen-3B61.460.5−0.85 [−2.67, +1.03]
VideoLLaMA254.757.1+2.37 [+0.69, +4.08]
MUSIC-AVQAQwen-7B65.467.1+1.69 [+0.98, +2.39]
Qwen-3B65.266.6+1.33 [+0.60, +2.05]
VideoLLaMA269.971.9+2.06 [+1.22, +2.90]
MUSIC-AVQA-HardQwen-7B37.040.5+3.47 [+2.15, +4.82]
Qwen-3B33.636.8+3.27 [+1.74, +4.78]
VideoLLaMA251.152.6+1.52 [−0.32, +3.33]

RED improves in 13 of 15 settings, with eight positive CIs excluding zero. MUSIC-AVQA and Hard gains support this combined design: preserving context when target-only uses Audio/Video, and distinguishing additive from residual corrections when both methods use full AV for Both/Relation. Gains are small on Qwen/AVHBench; Qwen-7B/SVHalluc favors Selected target-only.

We use 10,000 paired bootstrap resamples of videos and their questions; CMM resamples samples within each subset. Other evaluation settings are unchanged. Percentile CIs follow benchmark aggregation, without multiple-comparison adjustment (Appendix C.5).

The heuristic-selector tie in Table 4 shows that structured cues can suffice for selection. For inference cost, branch-based estimates on Qwen-7B/AVHBench are 0.8× Base for Selected target-only and 1.4× Base for RED.

Correct-answer preservation. We extend submitted Figure 3 by measuring correctness on its 1,738 answer-preserving pairs. Rates (%) average three backbones and two required-evidence types equally.

MethodInvariance violation ↓Both correct ↑Both wrong ↓
Required-modality-only0.079.920.1
Base17.067.115.9
MAD11.274.014.8
RED6.680.013.5

RED achieves the highest both-correct rate and lowest both-wrong rate while reducing invariance violations relative to Base and MAD. Required-modality-only uses benchmark-defined evidence and is invariant by construction, but remains wrong in both conditions for 20.1% of pairs. Thus, RED's improved invariance reflects stronger preservation of correct answers, rather than repeated errors. RED reduces sensitivity to irrelevant-modality changes while improving correctness preservation. Full baseline results are in Appendix B.4, Table J.

AuthorsOpenReview ↗

Response to Reviewer tciY [2/3]

W2: Evidence selector reliability and downstream error propagation are not directly validated.

More accurate evidence assignments can further improve RED's downstream performance. We test this by replacing Qwen-7B's assignments on the same questions with official MUSIC-AVQA tags (Audio-Visual → Both), following the submitted Table 4's Oracle protocol.

SplitDisagreement casesRED → Oracle on disagreements (%)RED → Oracle on full set (%)
Full1,514 / 9,129 (16.6%)72.5 → 76.167.1 → 67.7
Hard445 / 2,508 (17.7%)48.8 → 55.340.5 → 41.6

Correcting selection errors improves performance on the same questions. Table 5 reports selector agreement, macro-F1, and disagreement-case accuracy; Table 4 reports the full-set Oracle comparison.

The matrix retains all five outputs; Both/Relation count as agreement with Audio-Visual tags.

Split / TagnAudioVideoBothRelationNoneRecall (%)
Full / Audio1,6111,49101181192.6
Full / Visual2,4222762,14150088.4
Full / AV5,0967213923,9812078.2
Hard / Audio3843300531085.9
Hard / AV2,1242601311,7330081.6

Agreement / macro-F1 is 83.4% / 81.9% on Full and 82.3% / 78.2% on Hard; macro-F1 averages reference classes present in each split. Oracle additionally maps the two Full AV-tagged Relation outputs to Both. For Visual→Audio errors (276 questions), accuracy rises from 75.0% to 89.5%, indicating that error type also matters.

Both–Relation selection is supported by correction comparisons. Qwen-7B selects Relation for all 1,876 AVM questions and Both for 1,733/2,124 Hard AV questions; W3 shows the corresponding terms perform better on these groups. All 276 SVHalluc TF questions select Both despite a Relation reference. These task-level checks complement the validation added in Section 4.3, Table 5.

W3: The mapping between evidence types and correction terms requires broader empirical validation.

In RED (Section 3.3), Both requires both modalities without explicitly querying their relationship; Relation explicitly queries correspondence or temporal alignment. We map AV-Matching to Relation and MUSIC-Hard's official Audio-Visual tags to Both.

Building on the submitted contribution comparisons in Tables D–E, we report results below across three backbones and SVHalluc relation tasks, changing only the correction. Groups remain benchmark-defined; accuracy (%) averages backbones.

Question group+ΔA+\Delta^A+ΔV+\Delta^V+(ΔA+ΔV)+(\Delta^A+\Delta^V)+ΔI+\Delta^I
AV-Matching (Relation)67.363.062.770.8
MUSIC-Hard AV-tagged (Both-mapped)41.740.742.234.8
SVHalluc GSA62.765.859.964.9
SVHalluc TA52.450.851.953.5
SVHalluc TF29.732.631.832.1

Across all backbones, ΔI\Delta^I exceeds ΔA+ΔV\Delta^A+\Delta^V on AV-Matching; the reverse holds on MUSIC-Hard AV. ΔI\Delta^I also exceeds ΔA+ΔV\Delta^A+\Delta^V on GSA/TA, supporting distinct corrections for correspondence and dual-modality evidence.

However, the mapping has limitations in the settings raised by the reviewer. Video performs best on GSA/TF; temporal accuracy is near chance (TA: 50%; TF: 33.3%). Both Qwen selectors choose Relation on TA without consistent benefit. On the prompt-selected Both subset, ΔV\Delta^V also exceeds ΔA+ΔV\Delta^A+\Delta^V (38.63% vs. 38.19%, Table E). Thus, the mapping is not optimal for every group.

AuthorsOpenReview ↗

Response to Reviewer tciY [3/3]

W4: Robustness of the fixed correction weight across evidence types and tasks is unclear.

λ=1\lambda=1 preserves the PMI contribution scale, a natural, practical choice without extra hyperparameter search. Complementing the submitted NLL analysis in Figure 7, we report accuracy sweeps below for the full benchmarks and Both/Relation groups, varying only λ\lambda (three-backbone means, %).

Benchmarkλ=0.5\lambda=0.5λ=1\lambda=1 (RED)λ=1.5\lambda=1.5λ=2\lambda=2λ=2.5\lambda=2.5λ=3\lambda=3
AVHBench78.678.678.178.077.677.5
CMM86.587.087.187.187.187.2
SVHalluc60.560.360.059.759.559.4
MUSIC-AVQA69.568.567.366.465.665.0
MUSIC-AVQA-Hard41.943.343.844.244.344.3

Search improves over λ=1\lambda=1 by at most 0.2% on the three hallucination benchmarks and 1.0% across all five benchmark averages. Full MUSIC-AVQA favors smaller weights, Hard larger ones. These limited gains support one fixed coefficient across tasks without task-specific tuning.

We next fix the contribution regardless of selection on AV-Matching (n=1,876) and MUSIC-Hard AV-tagged questions (n=2,124):

Group / Correctionλ=0.5\lambda=0.5λ=1\lambda=1λ=1.5\lambda=1.5λ=2\lambda=2λ=2.5\lambda=2.5λ=3\lambda=3
AV-Matching / ΔI\Delta^I70.870.870.270.270.069.9
AV-Matching / ΔA+ΔV\Delta^A+\Delta^V65.762.761.559.859.058.3
MUSIC-Hard AV / ΔA+ΔV\Delta^A+\Delta^V41.142.242.442.542.742.5
MUSIC-Hard AV / ΔI\Delta^I37.034.833.832.632.432.2

Relation remains preferable on AV-Matching and Both on MUSIC-Hard AV throughout the sweep, with variation of 0.9% and 1.6%. This ordering is robust to correction strength.

Q1: Assumptions and recalibration requirements of unlabeled score centering.

Unlike MAD and AVCD, which use 100 labeled examples per dataset to tune decoding strengths, RED estimates fixed offsets from 100 unlabeled questions without such tuning. Centering assumes representative target evidence types and scores. Submitted Table A already reports uncentered CMM accuracy of 87.3/86.1/85.5% for Qwen-7B/3B/VideoLLaMA2, versus Base's 82.2/83.3/82.8%.

Using Appendix Figure C's same five calibration seeds, we exclude each draw's 100 calibration questions with offsets fixed (CMM task-macro %, mean ± SD):

BackboneFull setExcluding calibration questions
Qwen-7B89.5 ± 0.389.5 ± 0.3
Qwen-3B86.2 ± 0.586.2 ± 0.6
VideoLLaMA285.1 ± 0.985.1 ± 1.0

New data sources and online inference. Fixed offsets are reused across inputs. Recall imbalance under offset transfer supports unlabeled target recalibration when distributions shift. Without target samples, RED can operate without centering (Table A). Disjoint evaluation retains RED's gains. We added both evaluations in Appendix A.2, Tables B–C.

Q2: The impact of invalid-answer filtering on the invariance evaluation requires clarification.

Some baselines produce invalid binary responses. If any method/backbone fails in either condition, we exclude that pair from all methods for comparison on identical pairs.

Question typeBeforeExcludedAfter
Audio-required1,122581,064
Video-required71945674
Total1,841103 (5.59%)1,738

On all 1,841 pairs, Qwen-7B/3B have 0.00% invalid pairs. VideoLLaMA2 rates are 0.22% for VCD/ICD/CODE, 4.73% for SID, 1.25% for AVCD, and 0.98% for MAD; Target-only/Base/ASD/RED have 0.00%.

Base and RED are valid on all pairs. Below are invariance violations (%); changes use unrounded values:

MethodFilteredAll pairsAbsolute change
Base17.017.00.03
RED6.66.60.06

RED's advantage over Base persists with and without filtering. We added this analysis in blue in Appendix B.4.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Key strengths of our work, as noted by the reviewers

We thank Reviewers tciY, EkKk, and T9mS for their careful evaluations and constructive feedback. We appreciate their recognition of RED's training-free effectiveness, evidence-following analyses, and practical accuracy–cost trade-off:

Reviewer tciY — 1) Informative evidence-dependence analysis that distinguishes correct predictions from appropriate use of perceptual evidence. 2) Training-free improvements across backbones and benchmarks. 3) Practical value with a useful accuracy–inference cost trade-off.

Reviewer EkKk — 1) Effective performance improvements without additional training, with moderate inference overhead. 2) Comprehensive and controlled baseline comparisons under consistent experimental settings.

Reviewer T9mS — 1) A simple, well-motivated decoding rule with broad empirical support. 2) Selective computation of the auxiliary branches required by the question. 3) Analyses beyond benchmark accuracy, including modality interventions, component ablations, and diagnostic evaluations supporting the evidence-following interpretation.

Revisions of the manuscript

The revised manuscript incorporates the following additions and clarifications, highlighted in blue. Section, figure, and table references use the revised manuscript numbering.

  • Incremental benefit beyond modality selection: with the selector and centering fixed, we compare RED with Selected target-only across three backbones and five evaluation settings. RED improves in 13 of 15 settings, with eight positive paired 95% confidence intervals excluding zero, supporting the combined design of retaining the AV prediction and selectively augmenting evidence contributions (Section 4.3; Appendix C.5, Table P).
  • Selector reliability and downstream impact: we directly validate the selector against official MUSIC-AVQA evidence tags. Replacing predicted assignments with Oracle assignments improves accuracy on the same disagreement cases in both Full and Hard (Section 4.3, Table 5).
  • Correct-answer preservation and filtering: RED's lower invariance violation is accompanied by a higher both-correct rate and a lower both-wrong rate than Base and MAD. Its advantage over Base persists with and without invalid-answer filtering (Appendix B.4, Table J).
  • Centering and offset transfer: building on the submitted no-centering ablation, we add evaluation excluding calibration questions and offset transfer across all 13 compatible binary tasks. The former retains RED's gains; the latter identifies distribution shifts for which unlabeled target recalibration is beneficial (Appendix A.2, Tables A–C).
  • Open-ended captioning: we evaluate all 1,106 AVHBench captioning items, with one caption per item per method and three judging passes. RED modestly improves mean correctness and detailedness over Base, while AVCD achieves the highest mean detailedness (Section 4.3, Table 6; Appendix B.3).
  • Residual interpretation: we clarify that the residual captures statistical non-additivity in the model's predictions, which alone does not guarantee semantic isolation (Section 3.3).

We provide point-by-point responses to each reviewer's concerns in the corresponding response threads.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.