acceptodds
Under review as a conference paper at ICLR 2027

Competence Is Not Control: Causal Decomposition in Multimodal Sentiment Analysis

Abstract

Multimodal sentiment analysis combines video, audio, and text to infer sentiment. Prior work has reported uneven use of these modalities, with models favoring text over visual and acoustic cues. We examine whether correct video-only predictions are preserved when audio and text are added, and how joint input changes video's influence. We focus on cases with correct video-only predictions but incorrect audio-only and text-only predictions. Across three models and three datasets, joint input yields error rates of 51.4%–87.6% on these selected training examples and 83%–97% on neutral examples. Video replacement and gradient analyses show reduced visual sensitivity under joint input, although video still improves accuracy over audio and text alone. On Qwen2.5-Omni, transplanting video-only states read by attention at the answer prompt into runs with all three inputs shifts scores toward the correct label, even with attention weights fixed. Allowing attention to adjust strengthens this effect, with less attention to the answer prompt and more to video, audio, and text on average. With the receiving run fixed, changing the source video has a larger effect for states formed from video alone than under joint input. Matching update size substantially reduces but preserves this difference. For neutral examples, it persists under two update-size controls across two models and two new datasets, for both correct and incorrect joint predictions. On Qwen2.5-Omni and MiniCPM-o 4.5, states from other videos with the same annotated sentiment also shift scores toward the correct label on average. Class-averaged states produce class-specific effects even after matching update size. Together, these findings distinguish a modality's standalone correctness from its influence on joint predictions.

Primary area:
interpretability and explainable AI
Keywords:
multimodal sentiment analysis, multimodal language models, mechanistic interpretability, causal intervention, modality conflict, representation analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6eerOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates an important phenomenon in MSA: correct video-only predictions can be overturned when audio and text are added, particularly on selected neutral examples. The authors distinguish standalone correctness from output influence under intervention and study this distinction through video replacement, gradient analysis, and activation patching. Their results show reduced visual sensitivity under Full input, while transplanting V-only suffix Values into Full improves gold margins even when attention weights are fixed. Comparing source states in matched receiving contexts further reveals that state formation context affects video-dependent output influence. These findings provide an insightful perspective on how multimodal models integrate information and suggest useful directions for diagnosing multimodal fusion failures.

Critical strengths
  1. This paper addresses a valuable question from a novel and insightful perspective. Rather than stopping at modality preference or attention allocation, it asks why a correct V-only prediction may fail to carry over to Full input. Distinguishing standalone correctness from influence on joint predictions is a compelling perspective.
  2. The experiments are extensive and thoughtfully controlled. Video replacement, gradient analysis, and suffix Value transplantation provide complementary behavioral and internal evidence.
Critical weaknesses
  1. The intervention operates on suffix positions containing the answer choices, and same-label donor states and class prototypes produce effects comparable to those of the recipient's own video. Consequently, it remains unclear whether transplantation restores useful video-specific evidence or primarily supplies a shared class-directed signal. The experiments establish output influence under intervention, but the connection between this influence and the mechanism responsible for native Full failures is less clear.

  2. The most consistent replication results concern neutral examples, while V-only predictions exhibit a substantial neutral bias. Evaluating unimodal predictions against a common multimodal gold label also makes it difficult to distinguish visual sentiment recognition from default predictions or genuine differences between modality-specific sentiment and the overall label. Non-neutral replication results are less consistent, and the source contrast has not established reliable failure-prediction support.

Questions for authors
  1. Could the authors compare V-only suffix Values with text-only class prompts, sources without video, A-only/T-only source states, or standard sentiment-steering directions under matched update magnitudes?

  2. Given the strong neutral bias under V-only, could the authors use the independent visual annotations in CH-SIMS to repeat the core selection and intervention analyses?

  3. How do the authors interpret the presence of source-context effects in both Full successes and failures, together with the lack of support under the prespecified failure-prediction criteria? What evidence links this contrast specifically to fusion failures rather than to a general property of context-dependent state formation?

2 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for your feedback and helpful suggestions. To address your specific questions:

1. What happens with alternative sources? (W1, Q1)

Following your suggestion, we compared two types of alternative sources on Qwen2.5-Omni: input states that do not use gold labels, and class signals constructed using gold labels. We test whether they can change answers through suffix Values. We hold the Full input and patch positions fixed, share the own-video layer-0 treatment, and use the own-video patch's per-layer update magnitude as the matching target for subsequent layers. Results on 95 non-neutral MOSEI/MELD failures are:

SourceRepairs
Same-clip V-only42/95
V-only prompt with video removed0/95
Same-clip A-only2/95
Same-clip T-only2/95
Literal correct-class prompt†1/95
Natural sentiment text†81/95
Text-derived class Value direction†94/95

† These controls use gold labels to test the effect of class signals; they are not label-free correction methods. Class directions use an additive intervention; the other sources use state replacement.

In the first group, removing video or using A-only/T-only states rarely repairs errors, whereas own-video states repair substantially more. Thus, arbitrary source states are not equally effective on these non-neutral examples. This comparison is conditional on selecting correct V-only and incorrect A-only/T-only predictions.

In the second group, natural sentiment text and class directions strongly affect answers, whereas literal class prompts have little effect. Correct-class signals can therefore also produce repairs through this interface, and how they are represented in internal states matters. These controls use the gold label in advance to test the effectiveness of class signals.

To address your concern about the V-only neutral bias, we report non-neutral results separately to test whether correction merely reflects pushing answers toward neutral. On neutral examples, states formed with the source video removed repair 48/60 cases, versus 49/60 for own-video states; repair counts alone do not distinguish the video's contribution. In contrast, the non-neutral results above show a clear gap between own-video and no-video sources, so these repairs cannot be explained solely by neutral bias.

These controls ask which other sources can influence answers. Together with same-label transfer in Section 6, they show that useful class information can be shared across sources. Our central comparison asks how the same video change, transmitted through states formed in different contexts, affects the same Full receiver. This source-context difference persists after controlling update magnitude. Shared class information and context-dependent changes in its output influence are complementary findings.

AuthorsOpenReview ↗

Response (2/2)

2. Do the results hold with visual labels and non-neutral examples? (W2, Q2)

The intervention still corrects errors. We looked only at positive and negative examples in CH-SIMS and CH-SIMSv2 where the visual annotation agrees with the overall task label. The model answers correctly with video alone, but incorrectly with audio alone, the transcript alone, or all modalities together. We then transplant the suffix Values formed with video alone into the joint-input run to test whether they can turn the wrong answer into the correct one.

Using the original unscaled V-only suffix intervention with adaptive answer-position attention, the two batches together yield 46/64 repairs (71.9%): 36/47 positive and 10/17 negative. None of the 14 initially correct Full controls became wrong.

These results show that joint-input errors still occur when neutral examples are excluded and visual annotations agree with overall task labels, and that transplanting V-only suffix Values can repair some of these errors. The observed failures and repairs are therefore not confined to neutral examples and do not require disagreement between the two labels.

3. How do source-context effects relate to fusion failures? (W1, W2, Q3)

Adding audio and text can change internal states in examples where the model answers correctly as well as those where it answers incorrectly. We therefore tested whether these states can carry the source's judgment tendencies into other inputs.

The most direct evidence comes from bidirectional transplantation. On 44 failures verified against visual labels, transplanting V-only suffix Values into Full repairs 31 errors. In the reverse direction, transplanting Full suffix Values into initially correct V-only runs makes 40 answers wrong, including 28 that switch to Full's original wrong class. Changing the states read by the answer position can therefore turn wrong judgments into correct ones, and correct judgments into wrong ones.

We further tested which readout components amplify the effect of transplanted states. On 135 Full failures not used to select heads, we first fix attention for all heads at the answer position, then allow only 24 preselected heads to adjust after intervention while keeping the others fixed. Releasing just these heads recovers 91.2% of the additional correct-class margin gain from fully adaptive versus fully fixed attention; a matched random-head group recovers only 2.9%. Thus, a specific set of answer-position readout components can account for most of this amplification of the patch-induced margin gain.

We also transplant the V-only and Full states from success and failure groups into the same four other Full inputs. The margin difference between the two source contexts is larger for the failure group, with a between-group difference of 1.882 and a 95% interval of [1.467, 2.229]. Thus, the difference appears not only in the sources' own answers, but also in how their states affect other inputs.

These findings connect to fusion failures in three ways: bidirectional transplantation can both repair Full errors and carry Full’s wrong-class tendencies into initially correct judgments; the two source contexts produce a larger difference in output influence for failure sources on common receiving inputs; and the head-group experiment identifies readout components that amplify the transplant effect. Together, these results link changes in judgment in fusion failures to directly intervenable suffix Values and how the answer position reads them.

Reviewer_TZcoOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The work investigates if a multimodal LLM's correct video-only sentiment prediction survives when audio and text are fully added. Also, the paper exams whether video-formed internal states causally influence joint predictions. For samples correct under video-only but wrong under audio/text-only inputs, the full multimodal model frequently misclassifies, especially for neutral cases. The authors use Activation patching on Qwen2.5-Omni to verifie that transplanting video-only activations rescues many failures; video-origin effects persist after magnitude matching and show label-selective transferable class signals, though a pre-specified failure-prediction hypothesis lacks support. The work is well-writing but not groundbreaking. Thus the novelty of the paper is really moderate.

Critical strengths
  1. The paper discloses the pooled-vs-label-equal weighting discrepancy (Table 7: 51.4% vs 39.0%), the negative failure-prediction result (Sec. 5, Tables 32–33), tiny subgroups (n=2, 3 for MOSEI negative), and numerical-stability exclusions (App. J.1). This candor materially raises trust.

  2. The competence (standalone correctness) vs. control (influence under intervention) distinction, operationalized via gold-margin contrasts in matched receiving contexts, is crisp and well motivated in Sec. 2.

Critical weaknesses

I have several concerns, I list them around either novelty or mechanistic problems.

  1. All patching is at the suffix values interface. There are no head level localization, no path patching, no analysis of how video content shapes suffix values. In addition, the attention redistribution finding in table 24 is descriptive. It has nothing establishes for the readers to understand which components drive the ampification. QWen3-Omni appears only in behavioral experiments. There is no deep analyses on one model.
  2. The paper has several overstatement. For example, 83-97% of neutral examples refer to selected training examples where A only and T only are both wrong. it is a setting in which the outcome is nearly predetermined for the large A = T agree subset. For those unselected fixed test subsets, prevalence is only 3-13% and error rates are far more variable (50-88.6%, tables 14-15). The abstract and contributions do not sufficiently convey this conditionality. Are there any particular reason for this arrangement? Additionally, section 3 should state explicitly that there are conditional retention rates on capped training cohorts, not population error rates.
  3. Text dominance and modality underuse are established (Liu et al. 2022; Han et al. 2025; Sun et al. 2026; Yan et al. 2026), and the "encoded but not used" distinction exists in prior conflict studies (Nooralahzadeh et al. 2026; Quang et al. 2026). The novel contributions as the paper claimed matched-magnitude source-context comparison and the suffix-Values interface, all I think are real but narrow; Section 3 largely reconfirms known imbalance phenomena on new models. The authors might need to better think if the present work has some meat compared with others.
  4. Same-label donors match own-video gains (differences ≤0.13, all CIs include zero), and matched-source controls erase the own-video advantage entirely on Qwen2.5-Omni (App. L.2). The transferable class signal, not video-specific content, dominates the corrective effect. While the paper says this, the abstract's "states from other videos with the same annotated sentiment" framing buries that the own-video specificity largely disappears under matching on one of two models. I strongly believe this is a potential gap of the research design.
  5. The paper chooses "V-correct" against single noisy annotations (MELD/MOSEI) on inherently ambiguous clips may pick idiosyncratic examples. I can say that using the multimodal score M for all conditions in CH-SIMS means "V-correct" does not certify that the video alone conveys the sentiment. Any further explanations from the authors would be needed.
Questions for authors
  1. Why the suffix — answer-prompt tokens with no direct video content — carries the effect, while video-token transplants do not (Table 22)? Can you characterise what information the suffix Values encode under V-only versus Full?

  2. How sensitive are the selected-cohort results for the 100-per-label cap and pool exhaustion (for example CH-SIMS cohorts with n=12–84)?

  3. Given same-label donors match own-video gains and matched sources remove the difference on Qwen2.5-Omni (L.2), what residual evidence supports video-specific (opposed to class-level) influence from the recipient’s own video?

  4. Will head-level or position-resolved patching identify from where the V-only advantage in G_F comes?

5 comments
AuthorsOpenReview ↗

Response (1/3)

Thank you for your thoughtful review and insightful comments. To address your specific questions:

1. Mechanistic depth and new findings (W1, W3)

We conduct behavioral analyses on all three models—Qwen2.5-Omni, MiniCPM-o, and Qwen3-Omni—and focus the in-depth mechanistic investigation primarily on Qwen2.5-Omni.

We trace modality imbalance to a computational step that can be directly intervened on: source context changes the pre-answer suffix Values, and changing the Values read by the answer position can change or even reverse the judgment. Repairs under V-only→Full and induced errors under Full→V-only provide bidirectional intervention evidence. The central source comparison further shows that the same video change has different effects through suffix Values formed under different contexts, even on the same receiver and after controlling update magnitude.

We also locate this effect in specific layers, positions, and reading components: retaining a frozen region containing 15.23% of the Value coordinates gives a mean source contrast close to the full transplant, while removing it substantially reduces the contrast. On 135 cases not used for head selection, allowing only the selected 24 answer-position heads to adapt recovers 91.2% of the extra margin gain from fully adaptive versus fixed attention, compared with just 2.9% for matched random groups. These results show where states formed under different contexts affect the answer and which reading components amplify the transplant's effect.

The practical significance of this finding is that it offers an insight for multimodal training: alongside optimizing joint prediction accuracy, training should consider whether fusion preserves information that supported correct judgments in a single modality and enables its effective use in the final decision.

2. Selection, caps, and pool exhaustion (W2, Q2)

We select V-only-correct, A-only-wrong, T-only-wrong cases to test whether joint input retains the correct video-only answer, without using Full correctness in selection. Even when A/T predict the same wrong class, Full answers 25.6% correctly, so failure is not predetermined. Only the subsequent repair experiments additionally select Full failures.

The cap of 100 is a per-class maximum; when a pool is exhausted, we use the eligible cases available. Comparing caps under the original candidate order shows that pooled error does vary with class composition, while neutral conditional error remains high in the tested cap≥10 settings. We will report actual class counts alongside pooled and class-equal error rates.

The high neutral error rates in the paper describe selected training cases. Tables 14–15 separately report the prevalence of these conflicts and their Full error rates on fixed test subsets. We will distinguish these quantities explicitly in the abstract and Section 3.

3. Visual labels and annotation ambiguity (W5)

We use the overall task label to test how the answer to the same task changes across input conditions. V-correct means task correctness; it does not by itself establish that the video expresses the same sentiment.

To test whether label disagreement explains the result, we select non-neutral CH-SIMS and CH-SIMSv2 cases where the independent visual label, overall label, and V-only prediction agree, while A-only, T-only, and Full are incorrect. Transplanting V-only suffix Values repairs 46/64 errors (71.9%), with repairs in both positive and negative classes. Joint-input failures and transplant repairs therefore remain observable when visual and task labels agree; disagreement alone cannot explain these results. MELD/MOSEI lack corresponding visual annotations, and we will report this check separately from their results.

AuthorsOpenReview ↗

Response (2/3)

4. Why the suffix, and how does video affect it? (Q1)

The suffix consists of 19 shared option, answer-instruction, and template tokens following the modality inputs. These positions can read the preceding video and other inputs, so identical words can acquire different Values under different contexts:

V-only: video tokens → task prompt → shared suffix → answer position
Full: video/audio tokens → task prompt and transcript → shared suffix → answer position

Changing only video pixels, with prompts and layout fixed, changes later suffix Values, showing that video content contributes to their formation. We further change suffix-to-video reads and renormalize other weights on five development cases, observing changes in subsequent Values. Transferring those Values into the original Full run with fixed answer attention can also improve correct-class margins. Neither the tested reading interventions nor their Value transfers repair the three existing failures, but they directly establish effects of changed reads on states and outputs.

These states can then transmit class-related tendencies associated with the source judgment to the answer. During transplantation, we keep the receiving input and suffix text unchanged and replace only the Value contribution read by the answer position. On 44 visually verified non-neutral failures, transferring Full suffix Values into initially correct V-only runs makes 40/44 predictions wrong, with 28 following the Full source's wrong class.

Video-position transplantation is weaker partly because its actual updates are smaller. Reusing the 12 development cases from Table 22 with a common per-layer budget and adaptive answer attention, suffix and video-position transplants still improve the correct-class margin by +2.164 and +0.263 on average, respectively. Magnitude alone therefore does not explain the gap. Video-position transplantation changes the answer position's direct reading of video Values without recomputing the suffix states already formed under Full; it does not rerun the process through which video shapes the suffix states.

AuthorsOpenReview ↗

Response (3/3)

5. What does effective transfer from same-label videos show? (W4, Q3)

Same-label transfer further characterizes what the suffix Values carry: states formed from another same-label video can also improve the correct-class margin, showing that useful sentiment-class information transfers across videos. Section 6 therefore advances the analysis from whether these states affect the answer to what useful information can transfer.

Our central comparison asks how the same video change affects the answer through states formed under different contexts. The source-context difference persists with the receiving input fixed and update magnitude controlled. Transferable information across videos and context-dependent effects of that information are complementary findings; this research question does not require the original video to outperform every same-label video.

Regarding the additional advantage you ask about, the matched comparisons on Qwen2.5-Omni have not established that the original video outperforms same-label sources. Section 6.2 retains 9%–24% of the effect after removing the selected sentiment-class component; this result involves rescaling the remaining component, which may still contain class information. We will present the same-label transfer result directly in the abstract and Section 6 and clarify its relation to the source-context effect.

6. Which positions and heads carry the advantage? (Q4)

The source-context contrast can be localized to a subset of suffix positions and layers. We selected and froze a region using separate CH-SIMS development cases, then tested it on the original 156 MOSEI/MELD failures, which were not used for region selection. The region contains option/template positions at layers 18–26 and covers 15.23% of suffix Value coordinates. Both source contexts and their own/donor videos are retained; only the transplant mask changes:

Transplant maskRaw GFG_FWithin-region magnitude-matched GFG_F
Full suffix1.5900.260
Frozen region only1.5870.246
Full suffix minus the region0.1760.056
Other positions in the same layers0.1190.038

Keeping this region gives a mean contrast close to the full transplant; removing it substantially reduces the contrast. The pattern remains when matching update magnitude between the two sources within each mask; different masks use their own budgets.

The answer position's reading further amplifies the transplanted states' effect. On the original 156 failures, V-only suffix transplantation repairs 42/156 with answer attention fixed and 88/156 when it can adapt. These interventions change how the answer position reads its prefix.

After selecting and freezing 24 heads on 21 development cases, we test them on 135 other cases: allowing only these heads to adapt recovers 91.2% [87.8%, 94.9%] of the extra margin gain from fully adaptive versus fully fixed attention, compared with 2.9% for matched random groups. Separately, clamping these heads reduces raw GFG_F from 1.568 to 1.226, whereas clamping random head groups leaves it largely unchanged. We thus locate a Value region carrying much of the source-context contrast and identify answer-position components that amplify the transplant’s effect.

Reviewer_mRbAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates modality influence in multimodal sentiment analysis through several empirical analyses. While the experiments are fairly comprehensive, I have concerns about the motivation, writing, causal formulation, and novelty.

  1. The paper moves between unimodal prediction failures, visual sensitivity, activation patching, and class-level transfer without clearly connecting these analyses. The overall writing feels somewhat fragmented, making the main idea difficult to follow. The presentation still needs substantial improvement.

  2. The paper does not provide an explicit SCM or clearly explain the causal relationships between modalities, internal states, and predictions. Although activation patching can measure how internal interventions affect outputs, it is unclear whether this is enough to establish an identifiable causal decomposition. The authors should better justify their causal claims.

  3. The paper compares unimodal predictions with multimodal ground-truth labels. However, a modality may express a different sentiment from the overall multimodal input. For example, a neutral facial expression may accompany negative speech. CH-SIMS already provides modality-specific annotations, so it is unclear why these annotations are not used in the main experiments to evaluate unimodal competence.

  4. The paper mainly combines existing techniques, such as gradient analysis, video replacement, and activation patching, to study modality influence. While the observations are interesting, it is unclear what fundamentally new insights this work provides beyond existing studies on modality dominance, modality competition, and causal interventions. The authors should better highlight the differences and advantages of their work.

Critical strengths

The experiments appear fairly comprehensive, covering multiple models and datasets, as well as video replacement, gradient analysis, activation patching, and several control experiments.

Critical weaknesses

(1) The writing needs significant improvement. The motivation is not clear enough, and the transitions between different analyses are often abrupt, making the paper difficult to follow.

(2) The causal formulation is not sufficiently rigorous. The paper lacks an explicit structural causal model (SCM), making it unclear whether the analyses truly support the claimed causal decomposition.

(3) The definition of unimodal competence is questionable. The paper evaluates unimodal predictions using multimodal sentiment labels, even though different modalities may express different sentiments.

(4) The novelty seems limited. The paper mainly uses existing analysis techniques without introducing a new method or theoretical framework, and its contributions beyond prior work are not clearly established.

Questions for authors

Please see the Paper Summary and Weaknesses

2 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for your review. To address your specific questions:

1. Motivation and the connection between analyses

We start with a concrete question: under the modality-conflict conditions we study, a model answers correctly from video alone, but gets most cases wrong when audio and text are added. Why does the video input, which supported the correct judgment on its own, fail to continue supporting the correct answer under joint input? We investigate how adding other modalities changes video-related internal states and how the model reads and uses those states.

Section 3 establishes this behavior and tests how sensitive the joint-input prediction remains to changes in the video. Sections 4–5 then examine the internal states through which video can affect the answer. We take suffix Values formed with and without audio/text and compare their effects on the same Full input. This lets us vary the context that produced the states while keeping the receiving input unchanged. The difference persists after controlling update magnitude.

Section 6 asks what these effective states carry. Do they need to come from the particular video being answered? Same-label transfer shows that some useful influence is shared across videos. These analyses therefore build on one another: establish the behavior, test how source context changes the effect of internal states, and examine what transfers. We will make these connections explicit in the introduction and section transitions.

2. What is the causal claim?

Our causal claim concerns a concrete intervention: what happens to the answer when we change the suffix Values read by the answer position? We keep the receiving input and model parameters fixed, replace the Value contribution from the suffix in the attention computation at the answer position, and run the remaining computation. We separately test fixed and adaptive answer-position attention.

The central comparison repeats the same own-video-versus-donor substitution under two source contexts: video alone and Full. Both are evaluated on the same Full receiver:

GF=E(V,F)−E(F,F).G_F=E(V,F)-E(F,F).

Here, E(c,h)E(c,h) is the correct-class margin under the own-video patch minus the mean margin under the fixed donor patches, for source context cc and receiving context hh. Donors come from the same dataset but have annotated sentiment labels different from the original video's. We use g(z,y)=zy−log⁡∑c≠yezcg(z,y)=z_y-\log\sum_{c\ne y}e^{z_c} as the margin. Thus, GFG_F asks whether the context in which the Values formed changes how strongly they transmit the effect of replacing the video. This cross-label contrast does not test clip specificity within a label.

We directly execute each intervention, so the causal evidence comes from changing model computation. Its scope is this intervention effect; it does not by itself explain the full process that produced the original error. By “decomposition,” we mean comparisons across source context, receiving context, and attention policy. We provide the structural equations and intervention operator below and will clarify this meaning in the paper, including a more specific title if title revision is permitted.

Our SCM is the frozen decoder unrolled over layers. Let XsX^s be source input, XrX^r the fixed receiving input, UlsU_l^s the recorded source Values, BlB_l receiving prefix states, and RlR_l the final pre-answer query state. With frozen parameters implicit,

Uls=fls(Xs),(B0,R0)=Embed(Xr),Bl+1=flprefix(Bl),U_l^s=f_l^s(X^s),\quad (B_0,R_0)=\mathrm{Embed}(X^r),\quad B_{l+1}=f_l^{\mathrm{prefix}}(B_l), (αl,Ulr)=flread(Bl,Rl),Rl+1=flupdate(Rl,WO,lo~l),z=fout(RL).(\alpha_l,U_l^r)=f_l^{\mathrm{read}}(B_l,R_l),\quad R_{l+1}=f_l^{\mathrm{update}}(R_l,W_{O,l}\widetilde o_l),\quad z=f^{\mathrm{out}}(R_L).

The maps include normalization, attention, residual and MLP operations. Causal masking prevents the prefix from reading the later final query. For one head, with head indices omitted, our intervention replaces only the suffix contribution to that query's attention output olo_l:

o~l=ol+I∑p∈Sαlpr(Ul,a(p)s−Ulpr).\widetilde o_l=o_l+I\sum_{p\in S}\alpha^r_{lp}(U^s_{l,a(p)}-U^r_{lp}).

Here I=0I=0 gives the native read and I=1I=1 the patch; aa aligns the 19 suffix positions SS, excluding the final query. The original operator acts before WOW_O at all 28 layers of Qwen2.5-Omni. It leaves current-layer Q/K untouched; later query states and attention can respond. A separate policy clamps the final-query attention rows to native values at every layer.

These equations specify how source inputs form Values, what remains fixed in the receiver, and which computations can respond after intervention. They formalize the intervention target without additionally establishing mediation of natural errors.

On 44 independently visually verified non-neutral failures, V-only→Full repairs 31/44; reverse Full→V-only transplantation turns initially correct V-only predictions wrong in 40/44 cases, including 28 toward the Full source's own wrong class. This supplies bidirectional behavioral evidence.

AuthorsOpenReview ↗

Response (2/2)

3. Why use the overall task label?

We study model behavior: when V-only predicts the overall task label correctly and A-only and T-only each predict it incorrectly, why does joint input often fail to retain the correct answer? For example, with an overall label of negative, the model answers correctly from video alone but incorrectly from either audio alone or text alone. A-only and T-only need not predict the same wrong class. We therefore score every input condition against the same task label. Competence here means answering this task correctly; it does not require separately identifying the sentiment expressed by each modality, nor do we treat a correct answer as proof of complete visual understanding.

Visual-label verification addresses an alternative explanation: does this behavior depend on disagreement between visual and overall task labels? We added CH-SIMS and CH-SIMSv2 cases where the independent visual label, overall label, and V-only prediction agree, while retaining the original A/T conditions. After verifying all four baselines, the original intervention repairs 46/64 Full errors (71.9%) on these non-neutral cases.

Joint-input failures and transplant repairs therefore remain observable when visual and task labels agree. The main experiment tests how the answer to the same task changes across input conditions; visual-label verification shows that disagreement between these labels alone cannot explain these additional results. The original Table 20 already uses CH-SIMS visual annotations; MELD/MOSEI lack the corresponding modality annotations needed for this check.

4. What is new beyond modality imbalance?

We trace modality imbalance to a computational step that can be directly intervened on: source context changes the pre-answer suffix Values, and changing the Values read by the answer position can change or even reverse the judgment. Repairs under V-only→Full and induced errors under Full→V-only provide bidirectional intervention evidence; the region and head experiments below further locate where these effects are read out and amplified. The central source comparison additionally shows that the same video change has different effects through suffix Values formed under different contexts, even on the same receiver and after controlling update magnitude.

We now also test where this source-context difference can be read out. We selected and froze a region of option/template positions at layers 18–26 using CH-SIMS development cases, then tested it on the original 156 MOSEI/MELD failures. We keep both source contexts and their own/donor videos, changing only which Values are transplanted:

Transplanted ValuesRaw GFG_FWithin-region magnitude-matched GFG_F
Full suffix1.5900.260
Frozen region only1.5870.246
Full suffix without that region0.1760.056

The region contains 15.23% of the Value coordinates. Keeping it gives a mean source contrast close to the full transplant; removing it substantially reduces the contrast. Magnitude matching compares the two sources within each mask, with a separate budget for each mask.

This connects the paper’s central effect to specific layers and positions, identifying Values at the fixed pre-answer suffix as an important intervention interface linking modality context to the final prediction. The receiver-head experiments further show that, on 135 cases not used for head selection, allowing only the selected 24 answer-position heads to adapt recovers 91.2% of the extra margin gain from fully adaptive versus fixed attention, compared with just 2.9% for matched random groups. Same-label transfer tests how much of that influence is shared across videos.

The practical significance of this finding is that it offers a new insight for multimodal training: alongside optimizing joint prediction accuracy, training should also consider whether fusion preserves information that supported correct judgments in a single modality and enables its effective use in the final decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.