- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates an important phenomenon in MSA: correct video-only predictions can be overturned when audio and text are added, particularly on selected neutral examples. The authors distinguish standalone correctness from output influence under intervention and study this distinction through video replacement, gradient analysis, and activation patching. Their results show reduced visual sensitivity under Full input, while transplanting V-only suffix Values into Full improves gold margins even when attention weights are fixed. Comparing source states in matched receiving contexts further reveals that state formation context affects video-dependent output influence. These findings provide an insightful perspective on how multimodal models integrate information and suggest useful directions for diagnosing multimodal fusion failures.
Critical strengths
- This paper addresses a valuable question from a novel and insightful perspective. Rather than stopping at modality preference or attention allocation, it asks why a correct V-only prediction may fail to carry over to Full input. Distinguishing standalone correctness from influence on joint predictions is a compelling perspective.
- The experiments are extensive and thoughtfully controlled. Video replacement, gradient analysis, and suffix Value transplantation provide complementary behavioral and internal evidence.
Critical weaknesses
-
The intervention operates on suffix positions containing the answer choices, and same-label donor states and class prototypes produce effects comparable to those of the recipient's own video. Consequently, it remains unclear whether transplantation restores useful video-specific evidence or primarily supplies a shared class-directed signal. The experiments establish output influence under intervention, but the connection between this influence and the mechanism responsible for native Full failures is less clear.
-
The most consistent replication results concern neutral examples, while V-only predictions exhibit a substantial neutral bias. Evaluating unimodal predictions against a common multimodal gold label also makes it difficult to distinguish visual sentiment recognition from default predictions or genuine differences between modality-specific sentiment and the overall label. Non-neutral replication results are less consistent, and the source contrast has not established reliable failure-prediction support.
Questions for authors
-
Could the authors compare V-only suffix Values with text-only class prompts, sources without video, A-only/T-only source states, or standard sentiment-steering directions under matched update magnitudes?
-
Given the strong neutral bias under V-only, could the authors use the independent visual annotations in CH-SIMS to repeat the core selection and intervention analyses?
-
How do the authors interpret the presence of source-context effects in both Full successes and failures, together with the lack of support under the prespecified failure-prediction criteria? What evidence links this contrast specifically to fusion failures rather than to a general property of context-dependent state formation?
2 comments
Response (2/2)
2. Do the results hold with visual labels and non-neutral examples? (W2, Q2)
The intervention still corrects errors. We looked only at positive and negative examples in CH-SIMS and CH-SIMSv2 where the visual annotation agrees with the overall task label. The model answers correctly with video alone, but incorrectly with audio alone, the transcript alone, or all modalities together. We then transplant the suffix Values formed with video alone into the joint-input run to test whether they can turn the wrong answer into the correct one.
Using the original unscaled V-only suffix intervention with adaptive answer-position attention, the two batches together yield 46/64 repairs (71.9%): 36/47 positive and 10/17 negative. None of the 14 initially correct Full controls became wrong.
These results show that joint-input errors still occur when neutral examples are excluded and visual annotations agree with overall task labels, and that transplanting V-only suffix Values can repair some of these errors. The observed failures and repairs are therefore not confined to neutral examples and do not require disagreement between the two labels.
3. How do source-context effects relate to fusion failures? (W1, W2, Q3)
Adding audio and text can change internal states in examples where the model answers correctly as well as those where it answers incorrectly. We therefore tested whether these states can carry the source's judgment tendencies into other inputs.
The most direct evidence comes from bidirectional transplantation. On 44 failures verified against visual labels, transplanting V-only suffix Values into Full repairs 31 errors. In the reverse direction, transplanting Full suffix Values into initially correct V-only runs makes 40 answers wrong, including 28 that switch to Full's original wrong class. Changing the states read by the answer position can therefore turn wrong judgments into correct ones, and correct judgments into wrong ones.
We further tested which readout components amplify the effect of transplanted states. On 135 Full failures not used to select heads, we first fix attention for all heads at the answer position, then allow only 24 preselected heads to adjust after intervention while keeping the others fixed. Releasing just these heads recovers 91.2% of the additional correct-class margin gain from fully adaptive versus fully fixed attention; a matched random-head group recovers only 2.9%. Thus, a specific set of answer-position readout components can account for most of this amplification of the patch-induced margin gain.
We also transplant the V-only and Full states from success and failure groups into the same four other Full inputs. The margin difference between the two source contexts is larger for the failure group, with a between-group difference of 1.882 and a 95% interval of [1.467, 2.229]. Thus, the difference appears not only in the sources' own answers, but also in how their states affect other inputs.
These findings connect to fusion failures in three ways: bidirectional transplantation can both repair Full errors and carry Full’s wrong-class tendencies into initially correct judgments; the two source contexts produce a larger difference in output influence for failure sources on common receiving inputs; and the head-group experiment identifies readout components that amplify the transplant effect. Together, these results link changes in judgment in fusion failures to directly intervenable suffix Values and how the answer position reads them.
Response (1/2)
Thank you for your feedback and helpful suggestions. To address your specific questions:
1. What happens with alternative sources? (W1, Q1)
Following your suggestion, we compared two types of alternative sources on Qwen2.5-Omni: input states that do not use gold labels, and class signals constructed using gold labels. We test whether they can change answers through suffix Values. We hold the Full input and patch positions fixed, share the own-video layer-0 treatment, and use the own-video patch's per-layer update magnitude as the matching target for subsequent layers. Results on 95 non-neutral MOSEI/MELD failures are:
† These controls use gold labels to test the effect of class signals; they are not label-free correction methods. Class directions use an additive intervention; the other sources use state replacement.
In the first group, removing video or using A-only/T-only states rarely repairs errors, whereas own-video states repair substantially more. Thus, arbitrary source states are not equally effective on these non-neutral examples. This comparison is conditional on selecting correct V-only and incorrect A-only/T-only predictions.
In the second group, natural sentiment text and class directions strongly affect answers, whereas literal class prompts have little effect. Correct-class signals can therefore also produce repairs through this interface, and how they are represented in internal states matters. These controls use the gold label in advance to test the effectiveness of class signals.
To address your concern about the V-only neutral bias, we report non-neutral results separately to test whether correction merely reflects pushing answers toward neutral. On neutral examples, states formed with the source video removed repair 48/60 cases, versus 49/60 for own-video states; repair counts alone do not distinguish the video's contribution. In contrast, the non-neutral results above show a clear gap between own-video and no-video sources, so these repairs cannot be explained solely by neutral bias.
These controls ask which other sources can influence answers. Together with same-label transfer in Section 6, they show that useful class information can be shared across sources. Our central comparison asks how the same video change, transmitted through states formed in different contexts, affects the same Full receiver. This source-context difference persists after controlling update magnitude. Shared class information and context-dependent changes in its output influence are complementary findings.