acceptodds
Under review as a conference paper at ICLR 2027

Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMs

Abstract

Omni-modal large language models (LLMs) are expected to answer a question using the modality it explicitly refers to. However, existing training paradigms rarely verify whether models actually follow this modality, because multimodal inputs from the same sample often provide redundant evidence for the same answer. In this work, we uncover a pervasive cross-modal shortcut in omni-modal LLMs: when asked an audio-related question, models rely on the image as much as on the audio, and sometimes even more. To systematically diagnose this behavior, we introduce the Factorized Modality Diagnostic, which independently swaps audio and images between samples to isolate each modality's causal contribution. Across two model families in different settings, we find that this shortcut persists throughout supervised fine-tuning and reinforcement learning post-training, while judge-based RL may further amplify such reliance on irrelevant visual information. Based on this finding, we propose DMC-Repair, which trains models on the same kind of cross-modal swapped samples while assigning supervision according to the modality specified by the question. This prevents models from exploiting the spurious correspondence between modalities within the same clip. Experiments demonstrate that DMC-Repair reduces the image-induced share of the answer effect by 59.9%, effectively suppressing the cross-modal shortcut without compromising audio-question answering performance. The reduction in shortcut reliance generalizes across two model families and zero-shot to an unseen dataset and an unseen benchmark, and persists through subsequent post-training. Code is available at https://anonymous.4open.science/r/DMC-Repair.

Primary area:
foundation or frontier models, including LLMs
Keywords:
omni-modal LLMs, audio-visual question answering, cross-modal shortcut, modality reliance, shortcut learning, counterfactual data augmentation, reinforcement learning post-training, multimodal evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CjKVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a factorized diagnostic for cross-modal shortcuts and DMC-Repair, which trains on complete 2×2 image–audio combinations with labels determined by the question’s designated modality. It reports a 59.9% reduction in the Shortcut Index and a 13.7-point improvement in audio-following on held-out questions, with additional cross-model, zero-shot, and subsequent-RL evaluations.

Critical strengths
  1. Credible evidence of effectiveness. Improvements in the diagnostic are corroborated by generated answers, conditional-effect analyses, and held-out confirmation, supporting reduced shortcut reliance beyond a single metric.
  2. Simple and transferable repair. Answer-only supervision requires no additional judge or reference reasoning, and the reported benefits extend to another model family and unseen data and persist through subsequent RL.
Critical weaknesses
  1. Limited incremental novelty. Chen et al., in Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs, already train on aligned and misaligned audiovisual inputs using question-designated modality supervision. MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization explicitly targets invariance to irrelevant-modality perturbations. The significance of the complete 2×2 construction beyond these established principles remains insufficiently demonstrated.

  2. Missing controlled comparisons with the closest approaches. Table 2 compares published improvements from different starting checkpoints and evaluation settings, which cannot establish relative effectiveness. The construction ablations do not replace a comparison against balanced random-swap training with designated-modality supervision under matched data and training budgets.

Questions for authors
  1. What specific capability or diagnostic finding does the complete 2×2 construction establish beyond the approaches in Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs and MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization?

  2. Can the authors provide a controlled comparison between complete-grid training and balanced random-swap supervision, using the same starting checkpoint, source media, answer supervision, and training budget? Results on both the diagnostic and generated answers, including unseen data, would materially affect my assessment.

4 comments

Response to Reviewer CjKV (1/2): W1 and Q1 — what the construction adds beyond Chen et al. and MoD-DPO

We thank the reviewer for the careful assessment and are glad the reviewer credits the effectiveness evidence and the simplicity of the repair.

  • The effectiveness evidence is corroborated beyond a single metric. A lower diagnostic score alone could reflect the scoring interface rather than the model's use of each modality. The reduction also appears in generated answers, conditional-effect analyses, and the held-out confirmation set. The evidence thus points to reduced shortcut reliance rather than an artifact of one measurement.
  • The repair is simple and transferable. Many existing remedies rely on an additional judge, reference reasoning, or preference data. The repair uses answer-only supervision, and its benefits extend to another model family and unseen data and persist through subsequent RL. It can be added to an existing post-training pipeline without new annotation.

W1. "Limited incremental novelty. Chen et al. ... already train on aligned and misaligned audiovisual inputs using question-designated modality supervision. MoD-DPO ... explicitly targets invariance to irrelevant-modality perturbations. The significance of the complete 2×22\times2 construction beyond these established principles remains insufficiently demonstrated."

Q1. "What specific capability or diagnostic finding does the complete 2×22\times2 construction establish beyond the approaches in [Chen et al.] and [MoD-DPO]?"

Summary (W1 and Q1).

  • A per-answer measurement of modality use. Scoring final answers cannot show which input produced an answer. The complete grid separates how much each input moves an individual answer, and in both model families the image moves audio-question answers about as much as the audio does. This exposes a shortcut that accuracy conceals.
  • Supervision that removes the shortcut without preference data. In a one-directional construction, the non-designated input can remain predictive of the training answer on some cells. Crossing two samples in both directions makes every non-designated input appear once with each label, so it cannot predict the answer. The repair needs no preference data, reference responses, or reward model.
  • A measured benefit over the closest recipe. Shared principles alone do not show which construction works better. Under matched conditions the complete grid gives a lower Shortcut Index than the adapted Chen et al. recipe and balanced random swap (comment 2/2). The complete grid thus adds a measurable benefit.

R-W1 and R-Q1: The complete 2×22\times2 construction measures the effect of each modality on an answer and balances the non-designated input across labels. Under matched conditions, the training rule also yields a lower Shortcut Index than the closest recipe.

Diagnostic finding. Fully crossing the images and the audio of two real samples separates the effect of each input on the answer margin. In both model families, the image affects the margins of audio questions about as much as the audio does (Shortcut Index near 0.5). This pattern persists through SFT and RL post-training, and the three judge rewards evaluated on all families increase the image effect (Section 5.5 and Appendix C). A proprietary API model (Gemini 3.8 Flash, Appendix P) follows the audio on only about half of the cells whose image and audio support different answers. Chen et al. test whether the final answer follows the requested modality when audio and video conflict, and MoD-DPO is evaluated by accuracy on hallucination benchmarks. Neither attributes an answer to its inputs.

Training rule. DMC-Repair shares the designated-modality supervision of Chen et al. but crosses each sample pair in both directions. Two real samples with different answers are crossed into four cells, and each cell inherits the answer of the sample that supplied the designated modality. Within every pair, each non-designated input appears once with each label (Equation 5) and cannot predict the answer. Chen et al. instead replace the audio track of a clip with a different audio event and train modality-specific classification questions. Unlike MoD-DPO, DMC-Repair requires no separately collected preference dataset, reference responses, or reward model. The revised Section 2 states these differences (lines 140–149).

Measured benefit. We trained the adapted recipe of Chen et al. from the same checkpoint with the same media, answers, and budget, using three paired seeds (Section 5.3, lines 415–427). On the confirmation set, the complete grid lowers the Shortcut Index from 0.559 to 0.239. The adapted recipe reaches 0.289 and balanced random swap 0.283, so the index of the complete grid is 17% and 15% lower. Four-Cell Accuracy, the share of pairs answered correctly in all four cells, is 34.4% for the complete grid, against 30.5% and 31.5%.

Response to Reviewer CjKV (2/2): W2 and Q2 — the controlled comparison, unseen data, and presentation

W2. "Missing controlled comparisons with the closest approaches. Table 2 compares published improvements from different starting checkpoints and evaluation settings, which cannot establish relative effectiveness. The construction ablations do not replace a comparison against balanced random-swap training with designated-modality supervision under matched data and training budgets."

Q2. "Can the authors provide a controlled comparison between complete-grid training and balanced random-swap supervision, using the same starting checkpoint, source media, answer supervision, and training budget? Results on both the diagnostic and generated answers, including unseen data, would materially affect my assessment."

Summary (W2 and Q2).

  • Table 2 is published context only. Its rows differ in starting checkpoint, training data, evaluation protocol, and use of an RL stage, so it cannot establish relative effectiveness. The controlled comparison below is the evidence we add.
  • Under matched conditions the complete grid gives the lowest Shortcut Index. The comparison uses the same start, media, answer supervision, and budget. The grid's index is lower than those of the adapted Chen et al. recipe and balanced random swap in all three seeds, and its Four-Cell Accuracy exceeds that of the adapted recipe. The benefit does not come from differences in data or budget.
  • The advantage extends to generated answers on unseen data. Diagnostic gains should carry over to free generation outside the training data. On zero-shot AVHBench matching the grid improves generated accuracy in every seed, whereas random swap ranges from a loss to a gain. The complete grid is the more reliable recipe on unseen data.

R-Q2: Under the controls that the reviewer lists, the complete grid has a lower Shortcut Index than balanced random swap in every seed. We train all variants from the same checkpoint (SFT_clean) with the same media files and the same 42 answers, using three paired LoRA seeds. Each variant uses 6,176 cells, of which 2,104 are counterfactual cells. The table gives the mean over the three seeds on the 128 Audio families of the confirmation set (Table 19 in Appendix N of the revised PDF). The Absolute Image Effect is the mean absolute change of the answer margin when only the image is swapped.

Training cellsShortcut IndexDifference from the complete grid [95% CI]Absolute Image EffectFour-Cell Accuracy
Starting checkpoint (no training)0.5591.5741.6%
Complete grid (DMC-Repair)0.2390.50534.4%
Adapted Chen et al. (2026b)0.289+0.050 [0.025, 0.075]cross-entropy scale30.5%
Balanced random swap0.283+0.044 [0.018, 0.070]0.52831.5%

The complete grid and balanced random swap use the same loss and differ in 24% of their counterfactual cells. This small change lowers the Shortcut Index in every seed. The adapted recipe trains with cross-entropy, which changes the scale of the margins, so we compare it on the Shortcut Index and Four-Cell Accuracy. Both differences favor the complete grid, with intervals that exclude zero.

The advantage of the complete grid also holds on unseen data, most clearly in generated answers. On zero-shot AVHBench audio-visual matching, the complete grid improves generated accuracy over the starting checkpoint by 9.2 to 10.4 points in every seed. Balanced random swap changes the accuracy by −7.1 to +9.1 points, with a decrease in one seed. On zero-shot AVQA, the grid keeps a lower Shortcut Index than the adapted recipe in all three seeds (by 0.045 [0.030, 0.060]).

Additional revision, presentation: In the revision, we added a table of terms and notation (Appendix Q) and a statement of the audio and image effects in words before their formulas (lines 200–201). We also split long sentences in Sections 3 and 4 and added a roadmap at the end of the introduction (lines 107–108).

Thank you for the thorough rebuttal and the additional controlled experiments.

The new matched comparisons against balanced random swap and the adapted Chen et al. (2026b) recipe—across both the diagnostic metrics and zero-shot generated answers on unseen data—directly address my main experimental concern and convincingly show the benefit of the complete 2×2 grid construction. I also appreciate the clarified positioning and presentation improvements in the revision.

I have raised my rating from 2 to 3.

We sincerely thank the reviewer for their positive assessment of our rebuttal and revised manuscript, as well as for raising the rating. The requested matched comparison has been incorporated into the revised manuscript (Section 5.3 and Appendix N). We greatly appreciate the reviewer’s constructive feedback, which has helped improve the rigor and presentation of our work. We remain happy to address any further questions during the discussion period.

Reviewer_VAMHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates cross-modal shortcuts in omni-modal LLMs: models may answer audio-specific questions using correlated visual information, making accuracy on naturally aligned inputs an insufficient measure of designated-modality use. The authors introduce the Factorized Modality Diagnostic, which pairs samples sharing a question but having different answers and crosses their images and audio into a complete 2×2 grid. Differences in answer log-probability margins yield audio and visual main effects, an interaction term, and a Shortcut Index (SI). They then propose DMC-Repair, which labels these grid cells according to the question’s designated modality and trains with an answer-token hinge loss. Questions requiring both modalities retain only their original matched cells. Experiments cover Qwen2.5-Omni-based models and MiniCPM-o-2.6. On the MUSIC-AVQA confirmation set, the authors report a 59.9% relative reduction in audio-question SI and a 13.7 percentage-point improvement in audio-following during free generation.

Critical strengths
  1. The diagnostic provides an interpretable way to expose failures that conventional accuracy can conceal. Fully crossing two samples’ audio and visual inputs provides informative comparisons at both levels of the other modality. The paper also examines interaction terms, conditional visual effects, and absolute effect statistics rather than relying exclusively on an aggregate SI. These analyses strengthen the evidence that the measured changes reflect reduced sensitivity to the non-designated modality. This diagnostic perspective is a consequential contribution for evaluating modality use in omni-modal models.

  2. The repair is simple and supported by converging empirical evidence. On the confirmation set, SI decreases by 0.335, while free-generation audio-following improves by 13.7 percentage points. The generation results reduce concerns that the intervention merely improves the teacher-forced diagnostic. Transfer to external datasets and a second model family provides further support: for example, full repair improves AVHBench audio-hallucination accuracy from 61.9% to 67.7% and audio-visual matching from 55.4% to 62.6%. The method also requires no additional inference-time decoding branches or reward model.

Critical weaknesses
  1. The incremental benefit of the complete grid over closely related modality-specific training is not fully isolated. As acknowledged in Section 2, Chen et al. (2026b), Some Modalities are More Equal Than Others, already train on aligned and misaligned audio-visual inputs with supervision determined by the requested modality. The complete paired grid and its balancing properties therefore constitute an important part of DMC-Repair’s methodological distinction. The single-direction control supports the usefulness of completing the grid, but it does not establish the additional benefit over balanced random swapping using the same source media, supervision, and training budget. Table 2 provides context but cannot resolve this question: the reported starting accuracies range from 61.9% to 77.4%, and the methods differ in training data and evaluation protocols. Comparing improvements over these different starting points does not establish relative effectiveness. The additional RL stage also introduces a further training-budget difference. This limits how strongly the paper establishes its methodological contribution beyond existing counterfactual training approaches.
Questions for authors
  1. Can you compare the complete grid with balanced random swapping under the same training pool, loss, starting checkpoint, and approximately matched training budget? At minimum, please clarify how answer labels, individual media, and matched/mismatched conditions are distributed in the existing single-direction control. This would help identify the specific benefit of the complete paired construction.
  2. Please confirm that the own-reasoning SI reuses a single matched-cell reasoning prefix across all four inputs and clarify the resulting interpretation. Could you also report conditional and absolute visual effects on the confirmation set, together with free-generation audio-following for MiniCPM, to strengthen the evidence beyond the particular answer-scoring interfaces?
6 comments

Response to Reviewer VAMH (1/3): W1 and Q1a — Table 2 and the matched comparison with balanced random swap

We thank the reviewer for the constructive review and are glad the reviewer credits the diagnostic and the simplicity of the repair.

  • The diagnostic exposes failures that conventional accuracy conceals. A model can answer an audio question correctly while relying on the image, and accuracy alone cannot reveal this. Fully crossing two samples separates the effect of each input, and the interaction terms, conditional visual effects, and absolute effect statistics agree with the Shortcut Index. The diagnostic thus gives an interpretable measure of modality use in omni-modal models.
  • The repair is simple and supported by converging evidence. An improvement on a teacher-forced diagnostic alone could reflect the scoring interface. The repair also improves free-generation audio-following and transfers to external datasets and a second model family. It needs no inference-time decoding branches or reward model, so it adds nothing at inference.

W1. "The incremental benefit of the complete grid over closely related modality-specific training is not fully isolated ... Table 2 provides context but cannot resolve this question: the reported starting accuracies range from 61.9% to 77.4%, and the methods differ in training data and evaluation protocols ... The additional RL stage also introduces a further training-budget difference."

Q1a. "Can you compare the complete grid with balanced random swapping under the same training pool, loss, starting checkpoint, and approximately matched training budget?"

Summary (W1 and Q1a).

  • Table 2 is published context only. Its rows differ in starting accuracy, training data, evaluation protocol, and use of an RL stage. The revision states these differences, and the matched comparisons below provide the evidence.
  • Matched comparisons isolate the benefit of the complete grid. The closest alternatives also train on counterfactual inputs with designated-modality labels. Under the same checkpoint, media, answers, and budget, the grid gives a lower Shortcut Index than the adapted Chen et al. (2026b) recipe and balanced random swap in all three seeds. The benefit does not come from differences in data or budget.
  • Completing each pair is what lowers the index. Balanced random swap differs from the grid only in whether a counterfactual cell takes its media from the pair partner. This change alone lowers the index in every seed and gives more stable transfer to AVHBench. Pairing is therefore an active ingredient of the repair.

R-W1: Under matched conditions, DMC-Repair has a lower Shortcut Index than the recipe of Chen et al. (2026b), the closest related approach, which we adapted to our pipeline. The index is lower on the confirmation set (0.239 versus 0.289, 17% lower) and on zero-shot AVQA, in all three seeds on both datasets. Four-Cell Accuracy is the share of pairs answered correctly in all four cells, and it is also higher for DMC-Repair (34.4% versus 30.5%, interval excludes zero). In the revised PDF, the caption of Table 2 states how the published results differ (lines 432–437).

R-Q1a: With the pool, loss, starting checkpoint, and budget held fixed, the complete grid has a lower Shortcut Index than balanced random swap in every seed. Both constructions use the same media, answers, hinge loss, and 6,176 cells (2,104 counterfactual cells), with three paired seeds. The Absolute Image Effect is the mean absolute change of the answer margin when only the image is swapped.

Readout (mean of 3 paired seeds; Figure 6, Appendix N)Complete gridBalanced random swapDifference in favor of the grid [95% CI]
Shortcut Index, confirmation set0.2390.2830.044 [0.018, 0.070]
Absolute Image Effect, confirmation set0.5050.5280.023 [−0.016, 0.063]
Four-Cell Accuracy, confirmation set34.4%31.5%2.9 points [−0.3, 6.0]
AVHBench matching, generated accuracy vs starting checkpoint (per seed)+10.4 / +9.6 / +9.2+3.7 / −7.1 / +9.1

The two pools differ in 24% of their counterfactual cells. In these cells, the non-designated input comes from a random sample with the opposite answer instead of the pair partner. Both constructions reduce the Absolute Image Effect to about a third of its starting value, and the Shortcut Index separates them. It is 15% lower on average for the complete grid and lower in every seed. The complete grid also improves AVHBench matching in every seed, whereas balanced random swap falls below the starting checkpoint in one.

Response to Reviewer VAMH (2/3): Q1b — the distribution of the single-direction control

Q1b. "At minimum, please clarify how answer labels, individual media, and matched/mismatched conditions are distributed in the existing single-direction control."

R-Q1b: The single-direction control has nearly the same label proportions as the complete grid and differs in the individual media and in the match status (Table 20 in Appendix N of the revised PDF). Each Audio or Visual pair retains its two original cells and one randomly chosen counterfactual cell (550 pairs take one direction and 502 the other). Audio-Visual pairs keep both original cells, as in the complete grid.

DimensionSingle directionComplete grid
Cells5,124 (1,052 counterfactual)6,176 (2,104 counterfactual)
Budget4 epochs, 20,496 presentations of cells, 4,208 of counterfactual cells3 epochs, 18,528 presentations of cells, 6,312 of counterfactual cells
Yes labels, Audio questions49.3% original / 51.8% counterfactual49.3% both
Yes labels, Visual questions13.7% original / 11.3% counterfactual13.7% both
Non-designated media with both labels58%100%
Matched cells within a pairtwo thirdsone half

In the complete grid, every non-designated input appears once with each label and cannot predict the label. In the control this holds for only 58% of the media, and for the remaining 42% the non-designated input still predicts the training answer. Within each Audio or Visual pair, the control has two matched cells for every mismatched cell, whereas the complete grid has equal numbers. The control also presents counterfactual cells less often.

We also trained a budget-matched version of this control, which presents each counterfactual cell twice and thus as often as the complete grid. Against it, the complete grid has a lower Absolute Image Effect on the confirmation set (0.505 versus 0.603, in two of the three seeds). It is also lower on zero-shot AVQA, by 0.110 [0.079, 0.143].

Response to Reviewer VAMH (3/3): Q2 — own-reasoning prefix, conditional effects, and MiniCPM free generation

Q2. "Please confirm that the own-reasoning SI reuses a single matched-cell reasoning prefix across all four inputs and clarify the resulting interpretation. Could you also report conditional and absolute visual effects on the confirmation set, together with free-generation audio-following for MiniCPM, to strengthen the evidence beyond the particular answer-scoring interfaces?"

R-Q2a: The own-reasoning readout reuses one reasoning prefix across all four cells. We generate this prefix on the matched cell of each pair. The readout thus measures how much the media move the answer when the reasoning is held fixed. The end-to-end quantity is audio-following in free generation, in which an answer is generated separately for each cell. The caption of Table 1 now states this interpretation (lines 329–331 of the revised PDF). The conditional and absolute effects on the 128 Audio families of the confirmation set are given below, in the lower block of Table 7 and in Table 19.

StatisticStarting checkpointDMC-Repair (LoRA, 3 seeds)Change
Image effect, own audio+0.868−0.048−0.916
Image effect, partner audio+1.548+0.173−1.375
mean ∣EV∣\lvert E_V\rvert1.4530.346−1.106
Absolute Image Effect1.5740.505−1.069
mean max⁡∣δV∣\max\lvert\delta_V\rvert1.8600.629−1.231

All five statistics measure how much the image moves the answer to an audio question, so values closer to zero are better. Both conditional effects fall, and the signed effect with the sample's own audio is compatible with zero after the repair. The three absolute summaries fall by 66% to 76%, which shows that the reduction is not produced by signed values cancelling in the average. Under the own-reasoning readout, the Shortcut Index also falls, from 0.612 to 0.271 on the development set (Table 1).

R-Q2b: Free-generation audio-following for MiniCPM-o-2.6 rises under both prompts that we tested (Table 17, Appendix K). We ran free generation on the 510 conflict cells of the confirmation set with greedy decoding. The neutral prompt was used to repair MiniCPM-o-2.6. The describe-then-answer prompt was not used in the repair and asks the model to describe what it hears and sees before answering.

PromptAudio-following (%)BaseRepairedChange
Neutralall conflict cells49.262.1+12.9
unparseable outputs excluded49.562.1+12.6
Describe, then answerall conflict cells30.351.2+20.9
unparseable outputs excluded (123 families)44.862.1+17.3

The gain is 12.9 points with the neutral prompt and 20.9 points with the describe-then-answer prompt. Under the describe-then-answer prompt, final answers are absent in 31.8% of the base model's outputs and in 15.8% after the repair. Excluding these outputs gives a gain of 17.3 points. The base model's description illustrates its reliance on the image. For "Are there piano and saxophone sound?", the frame shows a piano and a saxophone, and the audio label is no. The base model writes: "Based on the visual information, it can be inferred that the performance likely includes both piano and saxophone sounds. Answer: yes." It infers the sound from the frame, and the repaired model does so less often under both prompts.

Reviewer_BbE2OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether omni-modal models answer questions using the designated modality rather than exploiting correlated evidence from another modality. A factorized image–audio swap diagnostic measures each modality’s effect on answer margins. DMC-Repair reuses the complete grids for training, assigning labels according to the designated modality. Experiments report reduced shortcut reliance and improved audio-following on held-out questions, with transfer across model families and datasets and retention through subsequent RL. The work combines behavioral diagnosis with a targeted repair of designated-modality question answering.

Critical strengths
  1. A coherent connection between diagnosing and repairing modality shortcuts. The complete image–audio grid independently varies the two sources and balances both answer labels and matched/mismatched status. Using the designated modality to label these cells addresses a concrete limitation of ordinary accuracy supervision: a correct answer can otherwise be obtained from the wrong source. This makes the construction useful as both a diagnostic and a training signal.

  2. The targeted repair effect is supported by complementary evaluations. On the held-out confirmation set, the paper reports a 59.9% relative reduction in SI and a 13.7-point improvement in generated audio-following (Table 1; Appendix F). Construction controls, conditional-effect and parsing analyses, a second model family, and zero-shot evaluations strengthen the evidence. Improvements for both audio-designated and visual-designated questions also argue against simply suppressing one modality globally.

Critical weaknesses
  1. Preserving useful cross-modal integration after repair is not sufficiently established. The central question is whether the model learns both to ignore irrelevant evidence and to exploit complementary evidence when needed. The paper reports encouraging retention on Audio-Visual questions, IntentBench, and audio-visual matching; these results should be credited. However, for jointly designated questions the training rule retains only original matched cells (Section 4.2), and the evaluation does not isolate examples that genuinely require both modalities using corresponding audio-only and visual-only controls. Stable aggregate joint-input accuracy can therefore coexist with reduced use of complementary information on a subset of tasks. I am not claiming that such degradation occurs, but this unresolved distinction limits the demonstrated scope of the repair and is the main reason I do not yet support acceptance.

  2. The incremental contribution and generality need clearer positioning. The paper acknowledges closely related aligned/misaligned designated-modality training, yet Table 2 compares gains from different starting checkpoints and settings rather than a matched implementation of the closest recipe. Its own construction controls are useful, but do not fully establish the added value over that prior approach. Broader cross-modal consistency work such as XModBench addresses related modality disparities and joint-input interference, although it does not replace the proposed designated-modality factorial diagnostic. The contribution should be positioned around that specific distinction and the complete-grid repair, with clearer evidence for their incremental benefit. Transfer to new datasets supports robustness within the studied question formats, but does not by itself establish a general improvement in multimodal integration.

Questions for authors
  1. [Decision-relevant evidence] On examples that genuinely require complementary audio and visual evidence, how do the starting and repaired models perform with joint inputs versus the corresponding audio-only and visual-only inputs? Can the reported capability-retention results establish that useful joint-input gains are preserved, alongside reduced interference on designated-modality questions? Suitable analyses of existing evaluation data would be informative; a particular new benchmark is not required.

  2. [Clarification and decision-relevant evidence] What evidence isolates the complete-grid recipe’s benefit over the closest prior designated-modality aligned/misaligned training under a common starting checkpoint and comparable training budget? Please also clarify how the diagnostic differs from broader cross-modal consistency evaluation, and delimit which repair claims are supported beyond single-designated-modality question answering.

7 comments

Response to Reviewer BbE2 (1/5): W1 and Q1 — joint-input gains are preserved (MUSIC-AVQA confirmation set)

We thank the reviewer for the careful reading and for stating which evidence would change the assessment. We are glad the reviewer credits the link between diagnosis and repair and the converging evidence for the repair.

  • Diagnosis and repair share one construction. Ordinary accuracy cannot tell whether a correct answer comes from the designated modality or from the other one. The 2×22\times2 grid varies the two sources independently while balancing both labels and match status, so each answer can be attributed to its source. The same grid thus serves as both the diagnostic and the training signal.
  • The repair effect is supported from several directions. A gain on one metric could reflect the scoring interface rather than modality use. The effect appears in several independent readouts (generated answers, a second model family, and unseen datasets), and both designated modalities improve. The repair reduces reliance on the non-designated input rather than suppressing one modality globally.

W1. "Preserving useful cross-modal integration after repair is not sufficiently established ... the evaluation does not isolate examples that genuinely require both modalities using corresponding audio-only and visual-only controls." "Stable aggregate joint-input accuracy can therefore coexist with reduced use of complementary information on a subset of tasks."

Q1a. "On examples that genuinely require complementary audio and visual evidence, how do the starting and repaired models perform with joint inputs versus the corresponding audio-only and visual-only inputs?"

Q1b. "Can the reported capability-retention results establish that useful joint-input gains are preserved, alongside reduced interference on designated-modality questions?"

Summary (W1, Q1a, Q1b).

  • Joint-input ability is preserved. Removing a shortcut could also impair answers that need both modalities. On questions that need both an audio and a visual fact, the repaired model is more accurate with both inputs than the starting checkpoint and the two released models. Less interference therefore does not cost joint-input ability.
  • The gain comes from using the right evidence. Higher accuracy could also come from more confident guessing. Accuracy rises with both inputs and with audio alone but not with the image alone, and the model commits less when the media are missing. The repaired model answers from the evidence the question needs.
  • The pattern holds on unseen data. A gain on the training data may not transfer. On zero-shot AVQA the Shortcut Index falls, and on AVHBench the gain from joint inputs grows (comment 3/5). The preserved integration is not specific to the music data.

R-Q1a: On questions that require both modalities, the repaired model is more accurate with joint inputs and with the audio alone, but not with the image alone. We ran this test on the Audio-Visual questions of the confirmation set and added it to the revised paper (Appendix O, Table 21). All table and line numbers below refer to the revision. Before any output of the test was available, we read every question and viewed the frames of every sample. We retained the 62 families whose questions combine a visual fact (instrument position or visibility) with an auditory fact (loudness, rhythm, order, or the sound itself). We removed 32 samples whose frames do not show the relevant instruments. Each model answers with both inputs, with the audio only, with the image only, and with the question text only. The answer is given by the sign of the yes/no margin, and Joint Gain is the accuracy with both inputs minus the higher single-input accuracy. The LoRA row is the mean over three models trained with DMC-Repair for this response.

ModelBoth inputsAudio onlyImage onlyText onlyJoint Gain
Qwen2.5-Omni (released)0.5350.5360.5010.497−0.001
HumanOmniV2 (released)0.5310.5030.5040.5080.027
Starting checkpoint0.5160.5080.5110.5080.005
DMC-Repair (LoRA, 3 seeds)0.5780.5650.4780.5010.013
DMC-Repair (Full)0.5830.5400.4970.5080.043
RL endpoint0.6030.5500.5110.5080.054
Change (LoRA − start)+0.062 [+0.004, +0.121]+0.056 [+0.012, +0.102]−0.033 [−0.084, +0.021]−0.007 [−0.025, +0.009]+0.008 [−0.032, +0.052]

Intervals are 95% family-cluster bootstrap CIs over the 62 families. Accuracy rises by 6.2 points with both inputs and by 5.6 points with the audio alone, and it does not rise with the image alone. Joint Gain does not decrease, and the improvement reflects better use of the audio rather than greater reliance on the image.

Response to Reviewer BbE2 (2/5): Q1b — margins, interference, natural accuracy, and examples (MUSIC-AVQA confirmation set)

R-Q1b: Joint Gain is preserved while the interference on designated-modality questions decreases. The retention results and the input controls in comment 1/5 establish both effects.

Margins (R-Q1b, continued; Appendix O). On the 62 families, the starting checkpoint has its largest absolute answer margin when it receives the question text alone. Averaged over families, this margin is 1.68 times the both-input margin. After repair, the ratio falls to 0.63, and the text-only margin is the smallest across the four input conditions.

Interference on designated questions (R-Q1b, continued). The repair reduces interference on questions that designate a single modality. On the 128 Audio families, the Shortcut Index falls from 0.559 to 0.239. The image effect almost vanishes, while the effect of the designated audio grows. On the 53 Visual families, where the ideal value is one, the index rises from 0.658 to 0.872. The repair strengthens the effect of whichever modality the question designates.

Accuracy on unmodified samples (R-Q1b, continued). The reduced interference does not come at the cost of accuracy on natural inputs. On the unmodified Audio-Visual samples of the confirmation set (76 families), matched-cell accuracy rises from 52.0% to 58.3% (+0.064 [+0.023, +0.106]; last row of Table 9). On the development set, Audio-Visual accuracy rises from 50.7% to 53.9% after the full repair and to 61.8% after subsequent RL.

Two examples (Appendix O). Two questions whose answer is yes illustrate the gain with both inputs. For "Is the cello on the right more rhythmic than the acoustic guitar on the left?", the starting checkpoint answers no with both inputs, with the audio only, and with the image only. After repair, all three LoRA models answer yes with both inputs and no with either input alone. For "Is the violin on the left louder than the piano on the right?", the starting checkpoint answers no in all four conditions. All three LoRA models answer yes with both inputs and no with the image alone, and two of them also answer yes with the audio alone.

Response to Reviewer BbE2 (3/5): W1 and Q1 continued — zero-shot AVQA and AVHBench

Zero-shot AVQA (R-W1 and R-Q1b, continued). The reduced reliance on the image extends beyond the music domain of the training cells. AVQA is an everyday audio-visual QA dataset, and none of its clips appears in any training configuration. On its 69 confirmation families, the three LoRA models lower the Shortcut Index from 0.376 to between 0.20 and 0.23. The paper's analysis of the full repair on all 104 families shows the same reduction (0.392 to 0.218, Table 13). Matched-cell accuracy on all 104 families rises from 68.3% to 82.2% (Table 12), so the reduction is accompanied by higher accuracy on unmodified samples.

Zero-shot AVHBench, answer-margin scoring (R-Q1b, continued). The audio-visual matching task of AVHBench asks whether the audio and the video of a clip belong together, so it requires comparing the two modalities. We score each question by the answer margin and report the area under the ROC curve (AUC) with both inputs (Table 22, Appendix O). Joint Gain is this AUC minus the higher single-input AUC.

Joint-input AUCJoint Gain
Starting checkpoint0.8200.182
DMC-Repair (LoRA, 3 seeds)0.8560.270
DMC-Repair (Full)0.8390.279
RL endpoint0.8560.270

After repair, the model separates matching from non-matching clips better (+0.037 [+0.022, +0.052]), and its Joint Gain grows (+0.088 [+0.066, +0.110]). The fully fine-tuned model and the RL endpoint show the same improvement.

The AVHBench tasks and IntentBench (R-Q1b, continued). The repair improves generated-answer accuracy on the two AVHBench tasks that require listening (Table 14). Audio-hallucination accuracy rises from 61.9% to 67.7%, and audio-visual matching accuracy rises from 55.4% to 62.6%. Video-hallucination accuracy is nearly unchanged (81.5% and 82.0%). IntentBench accuracy is 60.64% before the repair and 61.33% after it, and it reaches 67.03% after subsequent RL (Section 5.5 and Figure 7B).

Comments on the paper

1 comment
Area_Chair_jVDwOpenReview ↗

Reminder: Please Review the Author Response and Reconsider Your Final Decision

Dear Reviewers,

This is a gentle reminder to review the authors’ response to your comments. After considering their clarifications and rebuttals, I kindly ask you to reassess your review and update your final recommendation if warranted.

Thank you for your time and thoughtful consideration.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.