SPAR: Predictive Relation Routing for Multimodal Emotion Recognition

Multimodal emotion recognition relies on integrating heterogeneous affective cues, yet different modality combinations can yield inconsistent predictions for the same sample. The contribution of a modality depends on both the sample and the accompanying evidence, making predictive disagreement alone insufficient for determining which combinations should guide the final decision. To address this challenge, we introduce SPAR (Subset Prediction and Adaptive Routing), a framework that learns adaptive fusion through predictive relations between modality-subset predictions. These relations characterize differences in predictive distributions together with the modalities shared by or exclusive to the corresponding subsets. SPAR comprises two modules. The Modality Subset Prediction (MSP) module uses shared attention readers and a common classification head to produce emotion predictions for every nonempty subset of the available modalities, preserving unimodal and joint interpretations as fusion candidates. The Predictive Relation Routing (PRR) module encodes these relations over all pairs of subset predictions and integrates them with subset prediction descriptors through relation-aware attention, learning sample-dependent weights for aggregating subset logits. Both modules are jointly optimized through classification supervision on the subset predictions and the fused output. Experiments on MELD and the four- and six-class IEMOCAP settings show that SPAR achieves the highest reported score in eight of nine dataset–metric comparisons with discriminative baselines.