- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates whether average performance changes accurately capture the contribution of individual modalities in multimodal emotion recognition. The authors argue that removing audio or video may change many individual predictions without substantially changing overall accuracy, because beneficial and harmful prediction changes cancel each other.
Critical strengths
- The paper addresses an interesting problem by showing how average accuracy can hide important prediction changes after removing a modality.
- The evaluation across six datasets, different models, and 90 human participants is a strength. The class-level transition analysis also provides useful insights beyond standard accuracy and F1 scores.
Critical weaknesses
- Earlier studies, including SHAPE (IJCAI 2022) and ICYM²I (ICLR 2026), have already investigated modality contributions and missingness. Although this paper provides more detailed prediction-transition analysis, its methodological novelty over existing evaluation approaches needs clearer justification.
- Removing audio changes the accuracy effect from +11.0 to -0.5 percentage points depending on the decoding method (Section 4.2). This shows sensitivity to the experimental setup and makes general conclusions about audio contributions difficult.
- The authors need to demonstrate that the proposed metrics lead to better model selection or missing-modality robustness compared with standard aggregate evaluation.
- The confidence intervals for the main human classification effects include zero, and human-Gemini agreement is weak (κ = -0.031). This limits the conclusions about similarities between human and model behaviour
- Lacks a clear analysis of the computational cost of the proposed evaluation framework, particularly across multiple models and modalities.
Questions for authors
- What is the key methodological novelty compared with SHAPE and ICYM²I, beyond more detailed prediction-transition analysis?
- How do the authors justify general conclusions about audio contributions given the strong sensitivity to decoding methods?
- Do the proposed metrics improve model selection or missing-modality robustness compared with standard aggregate evaluation?
- How do the authors justify the human–model similarity claims given the confidence intervals overlapping zero and weak human-Gemini agreement?
- What is the computational overhead of the proposed framework when evaluating multiple models and modalities?
Author Response to Reviewer 46Wc (Part 1/2)
Thank you for your review and for recognizing our diagnostic perspective beyond average metrics, as well as our design across datasets, decoding rules, and human-model comparisons.
1. Motivation, positioning, and practical decision value (Q1)
This work grew out of our need to train high-accuracy multimodal emotion models for psychological counseling and psychotherapy. During training, average metrics repeatedly masked sample-level reversals and gave conflicting ablation conclusions. We report this workflow and findings as a practical reference for the community.
Algebraically, DRI, R, H, W, and Σm are direct statistics of paired predictions, and marginal confusion matrices cannot recover sample transitions. We have no intention of packaging these identities as complex mathematical theorems; rather, we hope they can be organized into a diagnostic workflow for training and evaluating multimodal emotion models.
Prior work has studied multimodal contributions, but none has systematically conducted sample-level diagnostics across the emotion training and inference pipeline, nor fully separated text, audio, and video, especially at the tightly coupled audio-text interface. Standard paired disagreement cannot separate helpful from harmful changes, while binary correctness tables collapse all errors into one bucket, missing wrong-to-wrong transitions W (with a median share of 25.0% among all prediction changes across our 13 settings, measured by ω = W/DRI), and error destinations.
When removing a modality yields little average change, combining R,H,W and transition directions shows whether the modality was unused, whether helpful and harmful changes canceled out, or whether the encoder and decoding setup turned high-arousal signals into shifts between wrong classes. This directly guides engineering decisions:
2. Protocol control, regional recurrence, and edge stability (Q2 & Q4)
Appendices B.1 and C separate protocol estimands to avoid cross-protocol effect-size comparisons. To address Q2 and Q4, Section 4.2 and Appendix G make three distinctions: