Beyond Average Contribution: Class-Conditional Modality-Missingness Signatures Across Emotion Datasets, Models, and Humans

Average performance differences after modality removal can conceal extensive changes in individual predictions. We propose a paired evaluation framework for multimodal emotion recognition combining the Decision Redistribution Index (DRI), the proportion of predictions that change, with class-conditional transition signatures that track rescue, harm, and wrong-to-wrong transitions. Our study spans six emotion datasets, open- and closed-source models, and 90 human participants. In the MELD subset of EmoS, video removal changes 17.2% of predictions while reducing accuracy by only 0.2 percentage points. Continuous ratings likewise exhibit opposing error changes that cancel in the mean. With variable linguistic content, text forms the most stable basis for judgment, while video removal repeatedly affects transitions between positive or high-arousal emotions and neutral. Audio supports arousal, intensity, and negative-emotion distinctions, but its correction directions vary across datasets, encoders, decoding rules, and observers; cross-dataset variation also appears in human judgments. Paired evaluation reveals the benefits and harms behind average gains and assesses whether modality effects recur across settings.