- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a factorized diagnostic for cross-modal shortcuts and DMC-Repair, which trains on complete 2×2 image–audio combinations with labels determined by the question’s designated modality. It reports a 59.9% reduction in the Shortcut Index and a 13.7-point improvement in audio-following on held-out questions, with additional cross-model, zero-shot, and subsequent-RL evaluations.
Critical strengths
- Credible evidence of effectiveness. Improvements in the diagnostic are corroborated by generated answers, conditional-effect analyses, and held-out confirmation, supporting reduced shortcut reliance beyond a single metric.
- Simple and transferable repair. Answer-only supervision requires no additional judge or reference reasoning, and the reported benefits extend to another model family and unseen data and persist through subsequent RL.
Critical weaknesses
-
Limited incremental novelty. Chen et al., in Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs, already train on aligned and misaligned audiovisual inputs using question-designated modality supervision. MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization explicitly targets invariance to irrelevant-modality perturbations. The significance of the complete 2×2 construction beyond these established principles remains insufficiently demonstrated.
-
Missing controlled comparisons with the closest approaches. Table 2 compares published improvements from different starting checkpoints and evaluation settings, which cannot establish relative effectiveness. The construction ablations do not replace a comparison against balanced random-swap training with designated-modality supervision under matched data and training budgets.
Questions for authors
-
What specific capability or diagnostic finding does the complete 2×2 construction establish beyond the approaches in Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs and MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization?
-
Can the authors provide a controlled comparison between complete-grid training and balanced random-swap supervision, using the same starting checkpoint, source media, answer supervision, and training budget? Results on both the diagnostic and generated answers, including unseen data, would materially affect my assessment.
4 comments
Response to Reviewer CjKV (2/2): W2 and Q2 — the controlled comparison, unseen data, and presentation
W2. "Missing controlled comparisons with the closest approaches. Table 2 compares published improvements from different starting checkpoints and evaluation settings, which cannot establish relative effectiveness. The construction ablations do not replace a comparison against balanced random-swap training with designated-modality supervision under matched data and training budgets."
Q2. "Can the authors provide a controlled comparison between complete-grid training and balanced random-swap supervision, using the same starting checkpoint, source media, answer supervision, and training budget? Results on both the diagnostic and generated answers, including unseen data, would materially affect my assessment."
Summary (W2 and Q2).
- Table 2 is published context only. Its rows differ in starting checkpoint, training data, evaluation protocol, and use of an RL stage, so it cannot establish relative effectiveness. The controlled comparison below is the evidence we add.
- Under matched conditions the complete grid gives the lowest Shortcut Index. The comparison uses the same start, media, answer supervision, and budget. The grid's index is lower than those of the adapted Chen et al. recipe and balanced random swap in all three seeds, and its Four-Cell Accuracy exceeds that of the adapted recipe. The benefit does not come from differences in data or budget.
- The advantage extends to generated answers on unseen data. Diagnostic gains should carry over to free generation outside the training data. On zero-shot AVHBench matching the grid improves generated accuracy in every seed, whereas random swap ranges from a loss to a gain. The complete grid is the more reliable recipe on unseen data.
R-Q2: Under the controls that the reviewer lists, the complete grid has a lower Shortcut Index than balanced random swap in every seed. We train all variants from the same checkpoint (SFT_clean) with the same media files and the same 42 answers, using three paired LoRA seeds. Each variant uses 6,176 cells, of which 2,104 are counterfactual cells. The table gives the mean over the three seeds on the 128 Audio families of the confirmation set (Table 19 in Appendix N of the revised PDF). The Absolute Image Effect is the mean absolute change of the answer margin when only the image is swapped.
| Training cells | Shortcut Index | Difference from the complete grid [95% CI] | Absolute Image Effect | Four-Cell Accuracy |
|---|---|---|---|---|
| Starting checkpoint (no training) | 0.559 | 1.574 | 1.6% | |
| Complete grid (DMC-Repair) | 0.239 | 0.505 | 34.4% | |
| Adapted Chen et al. (2026b) | 0.289 | +0.050 [0.025, 0.075] | cross-entropy scale | 30.5% |
| Balanced random swap | 0.283 | +0.044 [0.018, 0.070] | 0.528 | 31.5% |
The complete grid and balanced random swap use the same loss and differ in 24% of their counterfactual cells. This small change lowers the Shortcut Index in every seed. The adapted recipe trains with cross-entropy, which changes the scale of the margins, so we compare it on the Shortcut Index and Four-Cell Accuracy. Both differences favor the complete grid, with intervals that exclude zero.
The advantage of the complete grid also holds on unseen data, most clearly in generated answers. On zero-shot AVHBench audio-visual matching, the complete grid improves generated accuracy over the starting checkpoint by 9.2 to 10.4 points in every seed. Balanced random swap changes the accuracy by −7.1 to +9.1 points, with a decrease in one seed. On zero-shot AVQA, the grid keeps a lower Shortcut Index than the adapted recipe in all three seeds (by 0.045 [0.030, 0.060]).
Additional revision, presentation: In the revision, we added a table of terms and notation (Appendix Q) and a statement of the audio and image effects in words before their formulas (lines 200–201). We also split long sentences in Sections 3 and 4 and added a roadmap at the end of the introduction (lines 107–108).
Thank you for the thorough rebuttal and the additional controlled experiments.
The new matched comparisons against balanced random swap and the adapted Chen et al. (2026b) recipe—across both the diagnostic metrics and zero-shot generated answers on unseen data—directly address my main experimental concern and convincingly show the benefit of the complete 2×2 grid construction. I also appreciate the clarified positioning and presentation improvements in the revision.
I have raised my rating from 2 to 3.
We sincerely thank the reviewer for their positive assessment of our rebuttal and revised manuscript, as well as for raising the rating. The requested matched comparison has been incorporated into the revised manuscript (Section 5.3 and Appendix N). We greatly appreciate the reviewer’s constructive feedback, which has helped improve the rigor and presentation of our work. We remain happy to address any further questions during the discussion period.
Response to Reviewer CjKV (1/2): W1 and Q1 — what the construction adds beyond Chen et al. and MoD-DPO
We thank the reviewer for the careful assessment and are glad the reviewer credits the effectiveness evidence and the simplicity of the repair.
Summary (W1 and Q1).
R-W1 and R-Q1: The complete 2×2 construction measures the effect of each modality on an answer and balances the non-designated input across labels. Under matched conditions, the training rule also yields a lower Shortcut Index than the closest recipe.
Diagnostic finding. Fully crossing the images and the audio of two real samples separates the effect of each input on the answer margin. In both model families, the image affects the margins of audio questions about as much as the audio does (Shortcut Index near 0.5). This pattern persists through SFT and RL post-training, and the three judge rewards evaluated on all families increase the image effect (Section 5.5 and Appendix C). A proprietary API model (Gemini 3.8 Flash, Appendix P) follows the audio on only about half of the cells whose image and audio support different answers. Chen et al. test whether the final answer follows the requested modality when audio and video conflict, and MoD-DPO is evaluated by accuracy on hallucination benchmarks. Neither attributes an answer to its inputs.
Training rule. DMC-Repair shares the designated-modality supervision of Chen et al. but crosses each sample pair in both directions. Two real samples with different answers are crossed into four cells, and each cell inherits the answer of the sample that supplied the designated modality. Within every pair, each non-designated input appears once with each label (Equation 5) and cannot predict the answer. Chen et al. instead replace the audio track of a clip with a different audio event and train modality-specific classification questions. Unlike MoD-DPO, DMC-Repair requires no separately collected preference dataset, reference responses, or reward model. The revised Section 2 states these differences (lines 140–149).
Measured benefit. We trained the adapted recipe of Chen et al. from the same checkpoint with the same media, answers, and budget, using three paired seeds (Section 5.3, lines 415–427). On the confirmation set, the complete grid lowers the Shortcut Index from 0.559 to 0.239. The adapted recipe reaches 0.289 and balanced random swap 0.283, so the index of the complete grid is 17% and 15% lower. Four-Cell Accuracy, the share of pairs answered correctly in all four cells, is 34.4% for the complete grid, against 30.5% and 31.5%.