- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies whether multimodal large language models follow the requested modality when audio and visual evidence disagree. It constructs AVIC by pairing streams from AVQA samples with matching questions but different answers. Experiments on four models show a visual-source preference, with an option-free replication on an independent dataset.
Critical strengths
- Selecting examples that both models answer correctly from isolated streams helps separate recognition failures from failures to use available evidence. Source-disjoint probe evaluation and matched prompt controls strengthen this analysis.
Critical weaknesses
- Poor presentation and weak logical connections. The paper moves between conflict evaluation, representation probing, activation patching, and attention intervention without sufficiently explaining how the evidence supports each successive conclusion. Changes in prompts, answer options, and evaluation subsets make these results difficult to connect. The link between late decision-state sensitivity and the proposed attention intervention is particularly underdeveloped. Small figure labels further hinder readability.
- Limited comparisons. The intervention is compared only against VCD and AVCD. Although high- versus low-attention masking is examined, random masking and simpler modality-reweighting controls are missing. These comparisons are needed to establish whether the proposed token selection and layer choices provide a meaningful advantage. Importing some baseline scores from prior work also leaves the comparability of evaluation configurations uncertain.
- Insufficient experimental validation. The internal analysis is restricted to 300 animal-question examples and two models, while detailed intervention ablations cover only VideoLLaMA2. Intervention effectiveness is evaluated on a single benchmark, with settings selected on that same test set. Without independent evaluation, broader task coverage, and paired uncertainty estimates, the small improvements do not convincingly establish robustness or generalizability. The compared methods in main table is also limited.
Questions for authors
See weakness.