- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper builds upon “modality collapse”, a phenomenon in which a multimodal model over-relies on either the visual or textual modality, and mechanistically verifies it by looking into the internal representations of multimodal models when classifying the importance of moments in football matches. The authors apply attention knockout to the Qwen2.5-VL-7B-Instruct and Qwen2.5-Omni-7B models, blocking attention to the video or commentary spans across all layers and attention heads, and quantify each modality’s causal influence by measuring the resulting change in the logit margin between the ground-truth and alternative class. The authors classify each moment into one of four categories: synergy, commentary collapse, video collapse, or conflict, and report that the dominant form of collapse depends on moment importance, as commentary collapse occurs for important moments, whereas video collapse occurs for non-important moments. Collapse also reduces model confidence, even when the prediction is correct, while synergy decreases in strength from goals to corners to shots on target. Finally, analyses of attention share and the logit lens reveal that attention allocation remains similar across moment classes, despite substantial differences in the modalities’ causal influence. The authors conclude that Qwen2.5-VL integrates visual and textual information more effectively than Qwen2.5-Omni.
Critical strengths
The paper operationalizes modality use by assigning each moment to a category based on the signs of the measured modality effects, a simplification that makes a qualitative concept measurable and enables systematic comparisons across moments, categories, and layers.
Critical weaknesses
-
The paper never checks whether an underlying class bias could be driving the collapse-direction result, and its collapse label provides no way to distinguish a modality with negligible influence from one that is actively, even if misleadingly, driving the model’s answer.
-
Impact is measured as the change in Zy − Zy¯, so its direction is defined relative to the correct label. If a modality consistently pushes predictions toward the “important” class, its impact will appear positive for important moments but negative for non-important moments, producing the opposite-direction pattern across classes that the paper reports as its main finding, even if the modality’s underlying influence is consistent across both classes.
-
The study is limited to binary classification of football moments using two models that share the same language backbone. Without evaluation across additional tasks, domains, or model architectures, it is unclear whether the findings reflect general patterns of modality integration or are specific to this experimental setting.
Questions for authors
- Can you report modality impact using a class-independent measure, broken down by class? This would help distinguish class-dependent modality use from a consistent modality bias whose measured impact changes with the ground-truth label.
- What is the model’s accuracy under a prompt-only baseline? How often does the model default to “important” regardless of input? This would clarify how much of the reported class-dependent pattern reflects a default answer prior.
- In the attention knockout, is the −∞ mask applied to every query position t, for all edges (t, s), or only to t? How does this differ from deleting span S while keeping position IDs fixed?
- Does Qwen2.5-Omni-7B receive the video’s audio track? Spoken commentary may already be present in the video input before ablation, potentially confusing the comparison between VL and Omni.
Many thanks for the careful review. Here are our responses to the points raised under weaknesses:
Here are our responses to the questions:
Thank you for this suggestion. We compute the impact as the change in Z_YES - Z_NO under knockout, so that a positive value means the modality pushes the model towards 'important', irrespective of the ground-truth label. For each class, we report the mean impact and the share of moments pushed towards 'important', and the p-value of a permutation test for the difference between the classes:
The video pushes both models towards 'important' for both classes, and the push of the commentary does not differ significantly between the classes for either model. The only exception is VL, where the video pushes less strongly towards 'important' for non-important moments.
A prompt-only input is the same for all moments, so it would yield a single answer per model rather than an accuracy. Also, removing both modalities from the prompt shifts the input distribution, which is what our approach avoids (L048-052). The equivalent in our setup is knocking out both spans at once, which we mention under limitations (L453-455) and intend to explore. About the default: with the full input, both models answer YES for 86% of the moments (VL: 95% important, 63% non-important; Omni: 97%, 60%). Knocking out the video lowers these to 77%, 40% (VL) and 83%, 35% (Omni), so the video drives part of this default.
Yes, the mask is applied to every query position t, i.e., to all edges {(t, s) : s ∈/ S, s = t} at every layer and head, like we mention in L150-151. Equations 2 and 4 use the final position t* only for readability. So, no position outside S can read from S, which makes it equivalent to deleting S while keeping the position IDs fixed. We use the mask because it does this without modifying the input prompt (L167-169).
No, in our experiments, Omni does not receive the video's audio track. Both models receive the commentary only as a text transcription, so knocking out the commentary span removes it entirely, and the comparison between VL and Omni is not affected.