acceptodds
Under review as a conference paper at ICLR 2027

Beyond Average Contribution: Class-Conditional Modality-Missingness Signatures Across Emotion Datasets, Models, and Humans

Abstract

Average performance differences after modality removal can conceal extensive changes in individual predictions. We propose a paired evaluation framework for multimodal emotion recognition combining the Decision Redistribution Index (DRI), the proportion of predictions that change, with class-conditional transition signatures that track rescue, harm, and wrong-to-wrong transitions. Our study spans six emotion datasets, open- and closed-source models, and 90 human participants. In the MELD subset of EmoS, video removal changes 17.2% of predictions while reducing accuracy by only 0.2 percentage points. Continuous ratings likewise exhibit opposing error changes that cancel in the mean. With variable linguistic content, text forms the most stable basis for judgment, while video removal repeatedly affects transitions between positive or high-arousal emotions and neutral. Audio supports arousal, intensity, and negative-emotion distinctions, but its correction directions vary across datasets, encoders, decoding rules, and observers; cross-dataset variation also appears in human judgments. Paired evaluation reveals the benefits and harms behind average gains and assesses whether modality effects recur across settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal emotion recognition, modality ablation, paired evaluation, decision redistribution, human-model comparison

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jkCEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether average performance changes accurately capture the contribution of individual modalities in multimodal emotion recognition. The authors argue that removing audio or video may change many individual predictions without substantially changing overall accuracy, because beneficial and harmful prediction changes cancel each other.

Critical strengths
  1. The paper addresses an interesting problem by showing how average accuracy can hide important prediction changes after removing a modality.
  2. The evaluation across six datasets, different models, and 90 human participants is a strength. The class-level transition analysis also provides useful insights beyond standard accuracy and F1 scores.
Critical weaknesses
  1. Earlier studies, including SHAPE (IJCAI 2022) and ICYM²I (ICLR 2026), have already investigated modality contributions and missingness. Although this paper provides more detailed prediction-transition analysis, its methodological novelty over existing evaluation approaches needs clearer justification.
  2. Removing audio changes the accuracy effect from +11.0 to -0.5 percentage points depending on the decoding method (Section 4.2). This shows sensitivity to the experimental setup and makes general conclusions about audio contributions difficult.
  3. The authors need to demonstrate that the proposed metrics lead to better model selection or missing-modality robustness compared with standard aggregate evaluation.
  4. The confidence intervals for the main human classification effects include zero, and human-Gemini agreement is weak (κ = -0.031). This limits the conclusions about similarities between human and model behaviour
  5. Lacks a clear analysis of the computational cost of the proposed evaluation framework, particularly across multiple models and modalities.
Questions for authors
  1. What is the key methodological novelty compared with SHAPE and ICYM²I, beyond more detailed prediction-transition analysis?
  2. How do the authors justify general conclusions about audio contributions given the strong sensitivity to decoding methods?
  3. Do the proposed metrics improve model selection or missing-modality robustness compared with standard aggregate evaluation?
  4. How do the authors justify the human–model similarity claims given the confidence intervals overlapping zero and weak human-Gemini agreement?
  5. What is the computational overhead of the proposed framework when evaluating multiple models and modalities?
Reviewer_46WcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the limitations of using average performance differences to assess modality contributions in multimodal emotion recognition. The authors argue that small changes in aggregate accuracy or error can conceal substantial sample-level prediction changes, including beneficial corrections, harmful changes, and transitions between incorrect classes. The paper proposes a paired evaluation framework consisting of the Decision Redistribution Index (DRI), a decomposition into rescue, harm, and wrong-to-wrong transitions, and class-conditional transition signatures. For continuous emotion predictions, the framework additionally considers error migration and prediction displacement. The experiments cover six emotion-recognition datasets, several task-specific and multimodal foundation models, commercial APIs, and 90 human participants. The authors further conclude that text provides relatively stable semantic information when linguistic content varies, video tends to affect boundaries between positive or high-arousal emotions and neutral, and audio effects depend more strongly on datasets, representations, decoding strategies, and observers.

Critical strengths
  1. A useful and interpretable diagnostic perspective beyond aggregate modality ablations. The paper highlights a meaningful limitation of conventional modality-contribution analyses. The distinction between net performance effects and actual changes in predictions is important, especially when beneficial and harmful effects cancel. The paired formulation preserves information unavailable from separate confusion matrices, and distinguishing rescue, harm, and wrong-to-wrong transitions provides useful diagnostic detail. The MELD example, where DRI reaches 17.2% despite a negligible net accuracy difference, illustrates the practical relevance of examining prediction redistribution.

  2. Broad experimental coverage and informative sensitivity analyses. The study spans multiple datasets, model families, input conditions, and human observers. This breadth is valuable for examining whether modality effects depend on the underlying task and evaluation protocol. In particular, the Emotion-LLaMA experiment demonstrates that changing only the decoding rule can substantially alter the apparent contribution of audio, including reversing the sign of the net accuracy effect. The human–model comparison also provides useful evidence that similar aggregate trends do not imply agreement on which individual samples benefit from a modality.

Critical weaknesses
  1. Limited methodological novelty beyond standard paired-prediction analysis. The main proposed quantities are straightforward functions of paired predictions. DRI is the fraction of samples with different predicted labels, while rescue, harm, and wrong-to-wrong transitions partition these changes according to correctness. The class-conditional signature is an empirical joint transition distribution, and the principal mathematical identities follow directly from the definitions. Similarly, the observation that separate confusion matrices cannot recover sample-level transitions is a standard consequence of the non-identifiability of a joint distribution from its marginals. While organizing these quantities into a unified evaluation framework is useful, the paper does not sufficiently establish what is methodologically new relative to conventional paired disagreement analysis, correctness-transition tables, and class-conditional prediction-shift diagnostics. More importantly, the paper does not demonstrate that the proposed framework leads to decisions or insights that could not be obtained through simpler paired contingency analyses. The contribution is primarily empirical and descriptive rather than methodological, which limits its significance as an ICLR research contribution.

  2. Insufficient evidence for the claimed cross-setting stability and weaknesses in the interpretation of human results. A central conclusion is that video effects recur relatively consistently around positive/high-arousal emotion boundaries, whereas audio corrections are comparatively unstable. However, the supporting comparisons involve different datasets, label spaces, feature encoders, and evaluation protocols. These differences make it difficult to separate genuine modality-specific regularities from dataset and system effects. For example, Appendix B.1 identifies the MER2025 audio comparison as involving independently optimized TAV/TV rules, whereas the video comparison uses same-checkpoint removal. These protocols estimate different quantities and cannot provide a clean comparison of audio and video sensitivity. Although the authors recognize this distinction, the broader cross-setting interpretation remains largely qualitative. The paper does not provide a sufficiently systematic, uncertainty-aware test demonstrating that video transition patterns are statistically more stable than audio patterns under comparable conditions. The human-study evidence also requires more careful interpretation. The primary human analysis estimates marginal differences at the individual-response level, while reported human transition signatures rely on plurality labels aggregated from only four judgments per stimulus–condition pair. These aggregated transitions do not necessarily represent changes in the decisions of the same human observer. In particular, on EmoS, the individual-response analysis reports negative point estimates for both audio and video contributions (−3.9 and −1.7 percentage points), whereas the plurality-based analysis reports positive effects (+2.2 and +4.4 percentage points). These results concern different estimands and are not inherently contradictory, but the direction reversal shows that conclusions are sensitive to aggregation. Furthermore, the reported confidence intervals for the primary human effects include zero. The manuscript does not establish how much of the relatively high human plurality DRI arises from modality removal rather than inter-observer variability and the instability of aggregating four judgments. The human confidence intervals also appear to rely on stimulus-level resampling alone (Appendix D.1). If participants contribute judgments to multiple stimuli, participant-level dependence should be considered as well. These issues weaken the claim that the observed human decision redistribution provides strong independent validation of the proposed cross-setting modality conclusions.

Questions for authors

Q1. Methodological novelty. What substantive information does the proposed framework provide beyond standard paired disagreement rates, correctness-transition contingency tables, and class-conditional joint confusion analysis? Can the authors demonstrate a concrete downstream analysis or model-selection decision for which the proposed metrics provide additional value relative to these simpler alternatives?

Q2. Statistical evidence for modality stability. How do the authors quantitatively establish that video effects are more stable than audio effects across settings? Can they provide a controlled comparison using consistent missingness protocols, harmonized affective categories, class-prevalence adjustments, and uncertainty estimates? It would be particularly useful to distinguish recurrence of broad affective regions from stability of specific prediction transitions.

Q3. Human-study validity. Were the complete and reduced conditions evaluated by the same participants for each stimulus, or by different groups of observers? How were ties resolved in the four-person plurality aggregation? Can the authors quantify same-condition inter-rater or test–retest variability to determine how much human DRI would arise without a modality intervention? Additionally, can they explain the sign reversal between individual-level and plurality-level effects on EmoS and provide an analysis accounting for both participant and stimulus dependence?

Q4. Protocol dependence and missingness. Which conclusions remain supported when the analysis is restricted to directly comparable fixed-checkpoint modality-removal experiments? How much of the observed decision redistribution is attributable to the removal of informative signals versus distribution shifts caused by missing inputs? The latter distinction is important for interpreting the reported changes as modality contributions rather than deployment sensitivity.

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer 46Wc (Part 1/2)

Thank you for your review and for recognizing our diagnostic perspective beyond average metrics, as well as our design across datasets, decoding rules, and human-model comparisons.

1. Motivation, positioning, and practical decision value (Q1)

This work grew out of our need to train high-accuracy multimodal emotion models for psychological counseling and psychotherapy. During training, average metrics repeatedly masked sample-level reversals and gave conflicting ablation conclusions. We report this workflow and findings as a practical reference for the community.

Algebraically, DRI\text{DRI}, RR, HH, WW, and Σm\Sigma_m are direct statistics of paired predictions, and marginal confusion matrices cannot recover sample transitions. We have no intention of packaging these identities as complex mathematical theorems; rather, we hope they can be organized into a diagnostic workflow for training and evaluating multimodal emotion models.

Prior work has studied multimodal contributions, but none has systematically conducted sample-level diagnostics across the emotion training and inference pipeline, nor fully separated text, audio, and video, especially at the tightly coupled audio-text interface. Standard paired disagreement cannot separate helpful from harmful changes, while binary correctness tables collapse all errors into one bucket, missing wrong-to-wrong transitions WW (with a median share of 25.0% among all prediction changes across our 13 settings, measured by ω = W/DRI), and error destinations.

When removing a modality yields little average change, combining R,H,WR, H, W and transition directions shows whether the modality was unused, whether helpful and harmful changes canceled out, or whether the encoder and decoding setup turned high-arousal signals into shifts between wrong classes. This directly guides engineering decisions:

  • Decoding rules: On MER2025 with Emotion-LLaMA under teacher-forcing decoding, the net audio accuracy effect (Delta_acc = accuracy with audio minus accuracy without audio) is only -0.5 pp, as 9 rescues and 10 harms nearly cancel. From these numbers alone, one might assume the model failed to learn audio cues and retrain the encoder or drop the audio branch. Yet W = 18.0%: 36 samples switch between wrong labels, accounting for 65.5% of all 55 prediction changes. Audio removal shifts 34 samples from happy under full TAV input to angry under TV input, accounting for 34/55 ≈ 62% of all prediction changes. Switching to free generation on the same weights and samples changes the net audio gain from -0.5 pp to +11.0 pp.
  • Encoder and audio-text interface: In Appendix G.1-G.4, combining paired transitions with independently trained and calibrated T/A probes distinguishes whether a near-zero late-fusion audio weight stems from audio-text redundancy or weak audio features. On MER2025, replacing wav2vec with HuBERT raises conditional audio Macro-F1 gain from +0.0474 to +0.1676 and lifts the fusion weight from 0.08 to 0.42-0.44.

2. Protocol control, regional recurrence, and edge stability (Q2 & Q4)

Appendices B.1 and C separate protocol estimands to avoid cross-protocol effect-size comparisons. To address Q2 and Q4, Section 4.2 and Appendix G make three distinctions:

  • Controlled audio vs. video at the same fixed checkpoint: On 4,521 CMU-MOSEI samples, we remove audio and video from the exact same fixed checkpoint on the native 6 continuous dimensions, eliminating label and protocol differences. Removing video concentrates changes in happiness at +0.1117, while the other 5 dimensions stay within -0.0017 to +0.0137; removing audio increases error across all 6 dimensions, led by anger.
  • Broad regional recurrence vs. edge stability: As suggested in Q2, Section 4.2 separates these two levels. For video, what recurs across task models, Gemini, and humans is broad regional consistency along the Happy/Joy and Neutral boundary. For audio, even with the dataset, labels, 200 MER2025 samples, and Emotion-LLaMA weights fixed, switching only the decoding rule flips the net audio effect from +11.0 to -0.5 pp and yields Dedge=0.953D_{\text{edge}} = 0.953, 95% CI [0.920,1.000][0.920, 1.000], showing high edge sensitivity to decoding rules.
  • Shift-free evidence from independent training and probes: Fixed-checkpoint deletion measures deployment sensitivity, combining signal loss and missing-input shift. To evaluate modality information without missing inputs, Stages 3 and 4 in Appendix G add frozen unimodal probes and branches trained independently on each modality subset. On CMU-MOSEI, independently trained T and TA branches require no missing inputs and confirm that audio residuals remain correlated with target residuals after controlling for text, partial corr 0.098, 95% CI [0.069,0.125][0.069, 0.125].
AuthorsOpenReview ↗

Brief summary of our response and a quick follow-up

Dear Reviewer 46Wc, as the discussion phase nears its end, we wanted to offer a one-sentence summary of our detailed response for your quick reference: we demonstrated how WW (wrong-to-wrong transitions) guides concrete model diagnostics (such as pinpointing decoding bias to boost audio accuracy gain from −0.5 pp-0.5\text{ pp} to +11.0 pp+11.0\text{ pp}, and guiding the encoder upgrade from wav2vec to HuBERT), alongside fixed-checkpoint comparisons and our planned two-way resampling CIs distinguishing individual responses (Primary analysis) from 4-rater plurality (sensitivity analysis) in the human study. If any single point among these still leaves you with reservations before your final assessment, could you briefly point us to which part it is so we can provide a focused clarification before the window closes?

Reviewer_Eeh5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the issue that average metrics do not show how unimodal and multimodal predictions contribute to prediction tasks. To address this, the authors propose the Decision Redistribution Index (DRI) to measure the proportion of predictions that change; they quantify how multimodal predictions can rescue or harm, and how many of these are actually wrong-to-wrong transitions. The research question itself is relevant and sound. However, the proposed metric is not clear. I had difficutl time to parse how this metric work.

Critical strengths
  • The study is extensive, expanding six datasets, closed and open models, as well as human raters (which is appreciated).

  • The research problem itself is relevant.

Critical weaknesses
  • In Section 3 (paired evaluation framework), the metric itself, which is proposed to measure how unimodal information complements, supplements, or harms each other, is not very well explained. I struggled to understand the metric. I find this to be a major issue in the paper. Because of this, I could not judge the findings and how they differ from the ones found through standard metrics.

  • Figure 1 should be improved significantly. It should be enhanced with an example. For now, it has a lot of white space, and it does not contribute to explaining the motivation of the study meaningfully.

  • What are H, R, and W in equation 2 (especially W)? These terms seem essential for the rest of the paper but are not explained. How is W calculated?

  • I believe Section 3 should include a worked example to make the new metric easy to follow.

Questions for authors

Please check the weaknesses comments and address them accordingly

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer Eeh5

Thank you for your feedback. We agree that Section 3 made a simple idea harder to follow than necessary.

1. Definitions and a worked example

For each sample, we compare two predictions: with text, audio, and video (TAV), and without audio (TV). There are four mutually exclusive outcomes:

  • Unchanged: the predicted label stays the same.
  • Rescue (R): correct with audio, wrong without it; audio helped.
  • Harm (H): wrong with audio, correct without it; audio hurt.
  • Wrong-to-Wrong (W): wrong in both conditions, but with different predicted labels. For example, the true label is Sad, while the predictions are Angry with audio and Neutral without audio.

R, H, and W are the counts of these outcomes divided by the total number of samples, N. DRI = R + H + W is the proportion of predictions that change. The lowercase w in Eq. 3 has a different denominator: w = W / DRI is the share of wrong-to-wrong transitions among all changed predictions.

For Emotion-LLaMA with teacher-forcing decoding on the same 200 MER2025 samples:

  • 9 samples are correct with audio and wrong without it: R = 9/200 = 4.5%.
  • 10 are wrong with audio and correct without it: H = 10/200 = 5.0%.
  • 36 are wrong in both conditions with different labels: W = 36/200 = 18.0%.
  • 145 keep the same label: 145/200 = 72.5%.

The two quantities in Eq. 2 therefore measure different things:

Net audio contribution: accuracy WITH audio minus accuracy WITHOUT audio. Delta_acc = R - H = (9 - 10)/200 = -0.005, or -0.5 percentage points. Equivalently, removing audio increases accuracy by 0.5 percentage points.

Total changed predictions: DRI = R + H + W = (9 + 10 + 36)/200 = 27.5%.

Thus, a small accuracy difference hides changes in over a quarter of predictions. The 9 samples helped by audio and the 10 hurt by audio nearly cancel. The remaining 36 wrong-to-wrong transitions do not change accuracy at all, yet represent 36/55 = 65.5% of all changed predictions.

2. What this analysis reveals

Our motivation is that average ablation scores hide cancellation and error-class shifts, making apparently conflicting modality effects difficult to interpret. Across six datasets, multiple models, and 90 human participants, our comparisons reveal the following patterns:

  • Text provides the most stable basis for judgment. When spoken content varies, text supplies concrete events and emotion concepts. Removing text or shuffling word order causes sharp performance drops across datasets.
  • Video shows the most consistent transition directions. Across task models, Gemini, and humans, video-removal transitions repeatedly cluster around the Happy/Joy–Neutral boundary, consistent with facial cues for overt emotions.
  • Audio effects depend more strongly on the system and dataset. Pitch, energy, and speaking rate convey arousal, but high arousal is shared by anger, fear, happiness, and surprise. Its class-level effects therefore depend on the encoder, fusion, and decoding rule. Human transition directions also vary across datasets.

For example, with the same 200 samples and Emotion-LLaMA weights, switching from free generation to teacher forcing changes the net audio contribution from +11.0 to -0.5 percentage points, while the main transition changes from worried → happy to happy → angry.

When an ablation barely changes average accuracy, R, H, W, and transition directions distinguish unchanged predictions from cancellation and shifts between wrong classes. Comparing these patterns across encoders and decoding rules can help diagnose why modality effects differ.

3. Planned revisions

  • Section 3: begin with paired predictions on the same sample; define R, H, W, and DRI in plain language; explicitly state the accuracy-difference direction; include the numerical example above; then introduce class-transition notation.
  • Figure 1: reduce white space and show concrete examples with true labels and predictions with and without the modality. Alongside them, show the sample counts and calculations illustrating cancellation and wrong-to-wrong transitions.
AuthorsOpenReview ↗

Gentle follow-up on our worked example and Section 3 clarifications

Dear Reviewer Eeh5, as the discussion period is coming to a close soon, we wanted to quickly check whether the plain-language definitions of RR, HH, and WW and the worked numerical example on 200200 samples in our response (99 rescued RR, 1010 harmed HH, and 3636 wrong-to-wrong WW, where net accuracy shifts by only -0.5 pp while DRI = 27.5% of predictions actually change) have clarified the metric logic in Section 3 for you. If you feel any additional visual or tabular format in the main text would make this even more intuitive, please let us know—even a brief one-line suggestion would be greatly appreciated!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Clarification on the LaTeX "%" rendering truncation

Dear Reviewer Eeh5, thank you so much for replying and pointing this out. your comment helped us realize that an unescaped percent sign (%) inside OpenReview's LaTeX math mode (......) acted as a LaTeX comment character and invisibly truncated the second half of both equations on your screen! Specifically, what we wrote in the source text was:

  • Net accuracy change: Delta_acc = R - H = 4.5% - 5.0% = -0.5% (LaTeX hid everything after the first "%" so it displayed as "R - H = 4.5")
  • Total changed predictions: DRI = R + H + W = 4.5% + 5.0% + 18.0% = 27.5% (LaTeX hid everything after the first "%" so it displayed as "R + H + W = 4.5")

The fact that you immediately arrived at 0.5% for the net difference and 27.5% for R + H + W shows that the definitions and worked example themselves are clear once rendered properly; since this confusion was caused purely by OpenReview's LaTeX "%" comment truncation (which we have now fixed in plain text), we sincerely hope this resolves the misunderstanding and that you might kindly reconsider your score. We are very sorry that we did not thoroughly check the LaTeX. We sincerely apologize for this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.