- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Relevant Evidence Decoding (RED), a training-free decoding method for reducing cross-modal hallucinations in audio-visual LLMs. RED first asks the frozen backbone, using the question alone, to classify the required evidence as none, audio, video, additive audio plus video, or an audio–video relation. It then compares audio-only, video-only, joint audio-visual, and question-only next-token distributions. Differences from the question-only branch are treated as conditional-PMI contributions, while the non-additive remainder is treated as an interaction contribution. The selected contribution is added to a centered audio-visual score during decoding.
Critical strengths
-
RED is a simple, well-motivated decoding rule with broad empirical support. The method uses a common question-only reference to distinguish audio, video, additive, and residual interaction contributions, and then evaluates only the auxiliary branches required by the selected evidence type.
-
The analysis of this paper goes beyond a leaderboard comparison. The paper evaluates answer-changing and answer-preserving modality interventions, separates TPR from FPR, compares against a base threshold sweep, ablates selection and PMI decomposition, tests mismatched/random/heuristic/oracle assignments, examines contribution-specific loss landscapes, and reports detailed TTFT breakdowns. These analyses substantively support the evidence-following interpretation.
Critical weaknesses
-
RED uses a benchmark for calibration. It estimates score offsets from 100 questions sampled from each evaluation benchmark and then evaluates on the full set, including those questions. Results using a disjoint calibration set or no centering are needed to separate RED’s core contribution from transductive calibration.
-
The evidence selector is not directly validated. The paper provides no selector accuracy or confusion matrix against available evidence labels. In addition, the residual captures statistical non-additivity but does not necessarily represent a semantic audio–visual interaction.
Questions for authors
-
Can the authors report RED results when centering offsets are estimated from a disjoint development split or transferred from another benchmark, rather than sampled from the evaluation inputs?
-
Is score centering applied over the full vocabulary at every generation step, and are the 100 questions used to estimate offsets excluded from any reported evaluation in an alternative protocol?
4 comments
Response to Reviewer T9mS [2/2]
W2-1: The evidence selector lacks direct validation against available evidence labels.
We validate Qwen-7B's question-only selector against the official MUSIC-AVQA tags. Audio-Visual tags do not separate Both from Relation, so either output counts as agreement; the matrix retains all five outputs.
| Split / Tag | n | Audio | Video | Both | Relation | None | Recall (%) |
|---|---|---|---|---|---|---|---|
| Full / Audio | 1,611 | 1,491 | 0 | 118 | 1 | 1 | 92.6 |
| Full / Visual | 2,422 | 276 | 2,141 | 5 | 0 | 0 | 88.4 |
| Full / AV | 5,096 | 721 | 392 | 3,981 | 2 | 0 | 78.2 |
| Hard / Audio | 384 | 330 | 0 | 53 | 1 | 0 | 85.9 |
| Hard / AV | 2,124 | 260 | 131 | 1,733 | 0 | 0 | 81.6 |
| Split | Agreement (%) | Macro-F1 (%) | Disagreements: RED → Oracle (%) | Full set: RED → Oracle (%) |
|---|---|---|---|---|
| Full | 83.4 | 81.9 | 72.5 → 76.1 | 67.1 → 67.7 |
| Hard | 82.3 | 78.2 | 48.8 → 55.3 | 40.5 → 41.6 |
Macro-F1 averages the reference classes present. Oracle replaces predictions with reference assignments (AV→Both). These controlled replacements show that more accurate evidence assignments can further improve RED, particularly on Hard. Section 4.3, Table 5 reports selector agreement, macro-F1, and disagreement-case accuracy; Table 4 reports the full-set Oracle comparison.
W2-2: Statistical non-additivity does not necessarily represent semantic audio–visual interaction.
We agree that statistical non-additivity does not guarantee semantic isolation; our controlled analyses nevertheless support as a correspondence-sensitive predictive signal. is the non-additive residual in Eq. (4). Section 3.3 now explicitly states that statistical non-additivity alone does not guarantee semantic isolation. The submitted contribution comparison in Table D supports on AV-Matching; we provide complementary correspondence interventions below.
We swap audio or video with another clip for 938 originally matching questions (GT=Yes), using 938 originally mismatched questions (GT=No) as a control. For each contribution, we measure the rate at which its Yes–No margin decreases and the AUROC distinguishing original from replaced conditions. The table reports decrease rate / AUROC (%) for originally matching questions.
| Contribution | Audio swap, 7B | Audio swap, 3B | Video swap, 7B | Video swap, 3B |
|---|---|---|---|---|
| 55.8 / 60.6 | 57.0 / 63.8 | 0.0 / 50.0 | 0.0 / 50.0 | |
| 0.0 / 50.0 | 0.0 / 50.0 | 11.5 / 49.7 | 11.9 / 49.9 | |
| 83.6 / 80.7 | 72.0 / 74.0 | 46.6 / 57.3 | 37.8 / 55.3 |
Under audio replacement, the residual decreases on 83.6%/72.0% of matching questions, versus 48.7%/51.3% of mismatched controls. This contrast suggests that the residual contains correspondence-sensitive signals beyond a general response to audio replacement. Video replacement produces weaker discrimination (AUROC 57.3%/55.3%), showing modality asymmetry. Together with Table D, these interventions provide empirical support for using the residual on audio-visual correspondence questions and extend the submitted Figure 3's evidence-following analysis.
Q2: Scope of score centering and exclusion of calibration samples.
Scope of score centering. Following the option-/class-level calibration principles of PriDe [1] and Batch Calibration [2], already cited in the submission, we apply centering to candidate-answer scores in the reported QA experiments rather than to the full vocabulary at every decoding step. We estimate candidate-specific offsets for the AV score and evidence contributions from 100 unlabeled questions, conditioned on the selected evidence type (Appendix A.2). These offsets remain fixed during evaluation. We have clarified this scope in Section 3.3.
The same token-level formulation can also be applied to the full vocabulary, using a fixed offset for each token at every decoding step. The centering rule is unchanged; the QA implementation calibrates the candidate-answer tokens.
Excluding calibration questions. As reported in our response to W1 & Q1, RED retains its gains when the 100 calibration questions are excluded from evaluation.
References
[1] Zheng et al., Large Language Models Are Not Robust Multiple Choice Selectors, ICLR 2024.
[2] Zhou et al., Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering, ICLR 2024.
Thank you for your response and clarifications. After considering them, I will maintain my original score.
We sincerely thank you for taking the time to revisit our submission. Your thorough and constructive feedback throughout the review process has meaningfully strengthened both the empirical controls and the clarity of our manuscript. Please let us know if there is anything else we can clarify before the discussion period concludes.
Response to Reviewer T9mS [1/2]
Dear Reviewer
T9mS, We sincerely appreciate your constructive feedback and your recognition of RED’s formulation, selective inference, and analyses beyond benchmark accuracy.Unlike MAD and AVCD, which use 100 labeled examples per dataset to tune decoding strengths, RED estimates fixed offsets from 100 unlabeled questions without such tuning. RED's evidence correction remains effective without centering, and its calibrated performance is maintained after excluding calibration questions. Three complementary analyses establish the correction's contribution, check direct sample overlap, and assess offset transfer.
Evidence correction without calibration. Submitted Appendix A.2, Table A already compares RED with and without centering. We show these CMM results alongside Base from Table 1:
RED's core evidence correction remains effective without benchmark-specific centering. These results support using RED without estimating offsets. Centering complements this correction by further adjusting systematic score bias.
Direct sample overlap. Using Appendix Figure C's same five calibration seeds, we exclude the 100 calibration questions per draw, keeping offsets and predictions fixed. Table B reports CMM task-macro accuracy (mean ± SD):
CMM mean accuracy remains unchanged across all three backbones. RED therefore retains its gains when calibration questions are removed from evaluation, addressing the concern about direct sample overlap. These five-draw averages can differ from Table A's single-draw values; transfer across distributions is examined next.
Cross-benchmark transfer. Following the compatible-answer setting of cross-domain calibration [1], we transfer binary offsets only between tasks with matching candidate spaces. TF/CMTB have three choices, so binary offsets cannot directly calibrate their extra candidate. We include all six CMM subsets, three AVHBench tasks, and four binary SVHalluc tasks (GSA, FGSA, CMSB, TA).
Each source draw contains 100 unlabeled questions; SVHalluc's binary offsets use its binary items (63–85 per draw). Cells report accuracy / recall standard deviation (RStd) (%), averaged over five draws and three backbones. Following [1], RStd measures candidate-recall imbalance; we compute its population SD per task. Lower is more balanced. Diagonal entries use target calibration; labels are used only for evaluation.
CMM subsets retain similar accuracy with AVHBench offsets, while SVHalluc offsets transfer less consistently. TA remains near chance across sources.
The AVM accuracy drop coincides with sharply increased candidate-recall imbalance. With CMM/SVHalluc offsets, accuracy falls from 71.3% to 51.3%/51.5%, while RStd rises from 8.0% to 48.3%/48.4%. Predictions therefore become highly uneven across answer candidates. This occurs even with Relation-containing SVHalluc calibration, suggesting that evidence-type coverage alone is insufficient to transfer candidate-score calibration.
These results clarify how to use RED on new sources: retain its evidence correction and estimate offsets from unlabeled target samples when the distribution changes substantially. Without such samples, the no-centering results demonstrate the correction's standalone benefit. We added the expanded evaluation in blue in Appendix A.2, Table C; Tables A–B establish the core benefit and its persistence under disjoint evaluation.
References
[1] Zheng et al., Large Language Models Are Not Robust Multiple Choice Selectors, ICLR 2024.