acceptodds
Under review as a conference paper at ICLR 2027

SPAR: Predictive Relation Routing for Multimodal Emotion Recognition

Abstract

Multimodal emotion recognition relies on integrating heterogeneous affective cues, yet different modality combinations can yield inconsistent predictions for the same sample. The contribution of a modality depends on both the sample and the accompanying evidence, making predictive disagreement alone insufficient for determining which combinations should guide the final decision. To address this challenge, we introduce SPAR (Subset Prediction and Adaptive Routing), a framework that learns adaptive fusion through predictive relations between modality-subset predictions. These relations characterize differences in predictive distributions together with the modalities shared by or exclusive to the corresponding subsets. SPAR comprises two modules. The Modality Subset Prediction (MSP) module uses shared attention readers and a common classification head to produce emotion predictions for every nonempty subset of the available modalities, preserving unimodal and joint interpretations as fusion candidates. The Predictive Relation Routing (PRR) module encodes these relations over all pairs of subset predictions and integrates them with subset prediction descriptors through relation-aware attention, learning sample-dependent weights for aggregating subset logits. Both modules are jointly optimized through classification supervision on the subset predictions and the fused output. Experiments on MELD and the four- and six-class IEMOCAP settings show that SPAR achieves the highest reported score in eight of nine dataset–metric comparisons with discriminative baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Emotion Recognition, Modality Subset Prediction, Predictive Relation Routing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4P2LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a bespoke architecture for multilabel emotion recognition. This architecture includes pathways for the power set of modalities, and graph-based merging before the final classification. This allows the architecture versatility to combine modality information dynamically. The authors then offer comparison against other models on two prominent datasets in the field, MELD and IEMOCAP, across many metrics, and offer ablation studies to justify the complexity of their architecture.

Critical strengths
  • The authors build capacity in their network to allow it to dynamically and explicitly allocate importance to individual modalities or combinations of modalities by modeling their relationships, a very interesting property of their architecture. Their ablation study in Table 2 is also helpful in justifying the complexity of the network.
  • The authors reimplement and rerun a lot of baselines originally designed for other benchmarks to position their proposed model among the current state of the art.
Critical weaknesses
  • The complexity of the architecture itself is, I believe, a weakness of this work. For reference, the formulation of SPAR takes more real estate than the "Attention is All You Need, Vaswani et al., 2017" paper needed for the formulation of the Transformer. This complexity of course does not diminish the scientific contribution of the work. However, coupled with the lack of intuition offered behind the design, I believe it may be of little interest to the ICLR or other communities.
  • The datasets are quite old, and carry biases and simplifications that have been challenged for several years now, e.g., single-label settings, annotator disagreement disregarded, acted (overt) vs. genuine emotions, etc. It would be interesting to see a study in a newer, more realistic setting, and move the field forward from this nascent state of benchmaxxing.
  • The paper reports improvements over six selected discriminative baselines on IEMOCAP (and a few generative models). However:
    • the comparison may omit other recent approaches, such as MiSTER-E (already cited in related work), JOYFUL by Li et al., 2023, etc. These methods report higher performance across some of the metrics reported for SPAR. While the setting might not be identical, the authors should describe why particularly pertinent previous work has been omitted from Table 1. (Also, the authors need to justify why dropping conversational context, for example, is a justified choice, among the other decisions made for the inputs)
    • the gains, for all the described complexity, are on the order of 0.5-1/100 for many metrics, which have not be shown to be substantial/significant, e.g., by including standard deviations across runs/seeds. It would also be useful to get a sense of the number of trainable parameters across architectures.
Questions for authors
  • The paper does not establish, for the reader, that the baselines have used the same backbones as the ones reported for SPAR. It would be useful to include an explicit acknowledgement in the main paper itself of (i) the fact that baselines where re-implemented/re-runs, and (ii) that the same backbones have been used across models so that improvement is not an artifact of using different encoders, which is of course also important (back to nascent state of the benchmarks: if using more modern backbones can improve performance, then these numbers should be reported; gains when using a [masked] language model from 2020 might be attributable to the suboptimal language embeddings, for example, and not the additional multimodal information captured by the proposed network)
  • Could you provide more intuition and experimental findings on what each component is achieving in the network? What are the modalities used by the model for the final classification, and how does each modality's/modality combination's prediction relate to the models final prediction? etc.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 4P2L(Part 1/3, Questions 1)

We thank the reviewer for the suggestions on architectural intuition, benchmark scope, and comparison settings. We have expanded the empirical evidence and clarified the purpose of each component.

Q1. Baseline reproduction and encoder-controlled comparison

The baselines were rerun using public implementations where available and paper-based reimplementations otherwise, with input and classification adaptations for our evaluation. The original comparison retained method-specific encoders; the reproduction settings are detailed in Appendix F. The complete revised Table 1 is below, with mean ± standard deviation over three independent seeds (%). ACC, WF1, and Macro-F1 denote accuracy, weighted F1, and macro-F1.

MethodMELD ACCMELD WF1MELD Macro-F1IEMOCAP-6 ACCIEMOCAP-6 WF1IEMOCAP-6 Macro-F1IEMOCAP-4 ACCIEMOCAP-4 WF1IEMOCAP-4 Macro-F1
DMD59.32 ± 2.2258.49 ± 2.3441.13 ± 1.5659.56 ± 1.1859.19 ± 0.8957.72 ± 0.5376.39 ± 0.8576.63 ± 0.9176.92 ± 0.68
ALMT59.88 ± 1.9958.57 ± 1.9837.69 ± 3.9659.12 ± 0.7559.19 ± 0.6258.06 ± 0.2476.01 ± 0.5276.19 ± 0.5476.68 ± 0.80
DCR-utter59.33 ± 2.1057.99 ± 2.7339.99 ± 2.7960.56 ± 1.2860.36 ± 1.4058.58 ± 1.4978.78 ± 1.3978.92 ± 1.3079.19 ± 1.02
PDCC-MER62.08 ± 1.8460.58 ± 1.5842.42 ± 1.0763.15 ± 0.7963.12 ± 0.8262.37 ± 0.4879.08 ± 1.1179.09 ± 1.0978.90 ± 0.73
DQF62.47 ± 2.7061.79 ± 2.6947.87 ± 1.8362.70 ± 1.1563.12 ± 1.0762.38 ± 0.9379.32 ± 0.5579.56 ± 0.3979.09 ± 0.31
TiCAL63.12 ± 2.3163.33 ± 1.7647.35 ± 1.9863.77 ± 1.5463.81 ± 1.5263.06 ± 1.1880.12 ± 0.4780.31 ± 0.5780.48 ± 0.64
SPAR (Ours)65.42 ± 2.1364.81 ± 1.6749.59 ± 0.8165.56 ± 0.7165.37 ± 0.6963.53 ± 0.6481.71 ± 0.2481.78 ± 0.3381.40 ± 0.58

Best means in each metric column are bold.

To directly control encoder differences, we have additionally compared all methods using the same public RoBERTa-large (text), Whisper-large-v3 (audio), and CLIP ViT-B/16 (vision) checkpoints, matched preprocessing, fixed data splits, and validation-based selection. All three encoders remain frozen throughout training for every method; only method-specific fusion modules are optimized. For SPAR, they remain frozen during both the supervised warm start and subsequent joint optimization of the subset-prediction and routing modules. The complete Appendix F.4 Table 9 reports mean ± standard deviation over three independent seeds (%).

MethodMELD ACCMELD WF1MELD Macro-F1IEMOCAP-6 ACCIEMOCAP-6 WF1IEMOCAP-6 Macro-F1IEMOCAP-4 ACCIEMOCAP-4 WF1IEMOCAP-4 Macro-F1
DMD60.71 ± 0.4958.39 ± 0.7837.46 ± 0.9860.19 ± 0.5960.02 ± 0.7058.84 ± 0.9177.92 ± 0.5678.06 ± 0.5578.18 ± 0.38
ALMT61.45 ± 0.9459.41 ± 0.4438.93 ± 1.4760.93 ± 0.4660.36 ± 0.0758.41 ± 0.7278.14 ± 0.6178.27 ± 0.3678.42 ± 0.19
DCR-utter59.33 ± 2.1057.99 ± 2.7339.99 ± 2.7960.56 ± 1.2860.36 ± 1.4058.58 ± 1.4978.78 ± 1.3978.92 ± 1.3079.19 ± 1.02
PDCC-MER62.41 ± 0.1959.45 ± 0.7043.20 ± 0.8262.35 ± 0.6261.82 ± 0.5859.39 ± 0.7778.54 ± 0.6278.67 ± 0.5579.03 ± 1.09
DQF63.61 ± 0.8963.36 ± 2.3246.35 ± 0.7262.33 ± 0.3462.75 ± 0.6562.74 ± 0.2278.43 ± 0.7680.17 ± 1.3979.07 ± 0.38
TiCAL64.35 ± 1.4663.38 ± 2.5746.76 ± 0.3864.36 ± 1.7864.03 ± 1.8562.59 ± 2.2580.45 ± 0.3680.51 ± 0.3280.26 ± 1.01
SPAR (Ours)65.42 ± 2.1364.81 ± 1.6749.59 ± 0.8165.56 ± 0.7165.37 ± 0.6963.53 ± 0.6481.71 ± 0.2481.78 ± 0.3381.40 ± 0.58

Best means in each metric column are bold.

SPAR has the highest mean in all nine dataset–metric comparisons in both settings. In the shared frozen-encoder control, its ACC/WF1/Macro-F1 gains over the strongest baseline in each column are 1.07/1.43/2.83 percentage points (pp) on MELD, 1.20/1.34/0.79 pp on IEMOCAP-6, and 1.26/1.27/1.14 pp on IEMOCAP-4. These results show that the gains persist when encoder choice and updates are held fixed.

AuthorsOpenReview ↗

Response to Reviewer 4P2L(Part 2/3, Questions 2)

Q2. Component intuition and contributions to the final prediction

Modality Subset Prediction (MSP) preserves directly supervised predictions for A, T, V, AT, AV, TV, and ATV, where A/T/V denote audio/text/vision and concatenated letters identify joint inputs. Its shared readers and common classifier provide comparable predictions from different evidence combinations. Predictive Relation Routing (PRR) compares their distributions together with the modalities shared by or exclusive to each pair and learns one scalar aggregation weight per candidate. For example, comparing T with AT relates a predictive change to the inclusion of audio. Auxiliary supervision maintains each candidate's predictive utility regardless of its current routing weight. With complete inputs, PRR operates on seven candidates and 49 ordered pairs.

Component evidence. The complete revised Table 2 (adding NoSetEdges and extension to IEMOCAP-6) is below (%). These comparisons use a fixed random seed.

ConfigurationMELD ACCMELD WF1MELD Macro-F1IEMOCAP-6 ACCIEMOCAP-6 WF1IEMOCAP-6 Macro-F1
Full SPAR66.7466.1249.7866.0366.0064.20
SingleSubset63.8262.1342.6662.2162.3860.02
NoBimodal65.5964.0146.6563.8163.4961.21
Uniform63.1262.2643.8161.8461.8259.04
MLPGate64.6262.4343.5662.1562.2261.14
NodeAttn65.8263.1044.1364.0664.0462.69
NoPredEdges66.1764.1046.5364.5564.2063.03
NoSetEdges66.4565.4750.2464.9265.0362.13
NoAux62.8962.4044.9361.3461.9560.68

Full improves WF1 over SingleSubset by 3.99/3.62 pp and over NoBimodal by 2.11/2.51 pp on MELD/IEMOCAP-6. Its WF1 gains over NodeAttn are 3.02/1.96 pp, and removing continued auxiliary supervision reduces WF1 by 3.72/4.05 pp. These comparisons connect the modules to preserving useful subset candidates and combining them through learned relations. The MELD NoSetEdges Macro-F1 result (50.24 versus 49.78) is retained in the table.

Learned weights. For the MELD routing analysis, groups use unimodal top-1 predictions: G1: A=T=V; G2: T=V≠A; G3: A=V≠T; G4: A=T≠V; G5: all distinct. The complete mean weights underlying Figure 2 are below (%).

GroupATVATAVTVATV
G12.964.451.4320.3112.5219.7338.59
G23.573.831.3219.1715.6419.0237.46
G32.205.571.1520.1511.3023.2736.36
G42.695.231.0823.8110.6820.0536.46
G51.985.091.0523.349.7122.2336.61

ATV has approximately 37.3% mean weight, while bimodal candidates together receive 52.57–55.27% across these groups. ATV has the largest individual weight on 96.80% of utterances, counting ties fractionally, but exceeds 50% weight on only 0.578%. The final prediction therefore combines a strongly weighted trimodal candidate with substantial contributions from other subsets.

Relationship to the final prediction. Let zSz_S be each candidate's logits and wSw_S its learned scalar weight. The final distribution is

p^=softmax⁡(∑SwSzS).\widehat{p}=\operatorname{softmax}\left(\sum_S w_S z_S\right).

Table 3 isolates aggregation using identical logits from the MELD model trained with a fixed random seed: ATV-only selects zATVz_{ATV}, Uniform averages all seven logits, and PRR uses their learned weighted sum. The complete results are below (%).

GroupATV-only ACCATV-only WF1ATV-only Macro-F1Uniform ACCUniform WF1Uniform Macro-F1PRR ACCPRR WF1PRR Macro-F1
All61.8961.2745.4663.5161.8844.3266.7466.1249.78
G175.7971.7030.4377.6372.9130.5579.4874.6828.14
G255.2955.5834.6756.4754.7531.6359.2260.1241.49
G344.9243.7238.3746.8745.4934.9449.0347.8843.42
G467.4366.5656.7968.8967.7855.7571.6670.7260.02
G542.4041.6337.9443.5743.3039.8353.2252.5050.25

For each group and metric, the best result across the three aggregation rules is bold.

PRR improves overall ACC/WF1/Macro-F1 by 3.24/4.24/5.46 pp over Uniform and 4.86/4.85/4.32 pp over ATV-only. ACC/WF1 improve in every group, and all three metrics improve in G2–G5. The largest gains occur in G5: 9.65/9.20/10.42 pp over Uniform and 10.82/10.87/12.31 pp over ATV-only. In G1, Macro-F1 is lower (28.14 versus 30.55/30.43), while ACC/WF1 improve.

AuthorsOpenReview ↗

Response to Reviewer 4P2L(Part 3/3, Weaknesses)

W1. Architectural intuition and complexity

We have reorganized the method to explain each component's purpose before its equations. Q2 gives the intuition for MSP, PRR, and auxiliary supervision, followed by the complete component ablations in Table 2, learned weights, and fixed-candidate results in Table 3. This links the architecture directly to the predictions it preserves and how they influence the final decision.

W2. Benchmark scope and emotion-label limitations

We appreciate the concern that categorical benchmark accuracy captures only part of emotion understanding. We agree that categorical single-label evaluation captures only part of the subjectivity and ambiguity of emotion, and our current experiments do not model annotator disagreement.

We selected MELD and IEMOCAP as established benchmarks that remain in use in recent research, including Xiao et al. (Findings of EACL 2026) and Seong et al. (ACL 2026). This choice enables evaluation of the proposed aggregation mechanism against prior approaches under explicitly specified benchmark protocols.

Xiao L, Funakoshi K, Okumura M. Emotion Recognition in Multi-Speaker Conversations through Speaker Identification, Knowledge Distillation, and Hierarchical Fusion[C]//Findings of the Association for Computational Linguistics: EACL 2026. 2026: 4091-4106.

Seong E, Lee H, Kim D, et al. Emotion-Wheel-Guided Audio-Referred Text Representation for Multimodal Emotion Recognition in Conversation[C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026: 40392-40403.

W3. Baseline coverage, input scope, reliability, and model size

We restrict SPAR and the reproduced baselines to the target utterance's text, audio, and video to isolate within-utterance fusion from conversational modeling. MiSTER-E and JOYFUL are relevant related work, and their discussion has been expanded. Their published systems use conversational context and different input/evaluation settings; our comparative claims concern the reproduced utterance-level systems.

For repeated-run evidence, Q1 provides the complete three-seed main comparison (Table 1) and shared frozen-encoder comparison (Table 9). In Table 1, SPAR improves WF1 over TiCAL by 1.48/1.56/1.47 pp on MELD/IEMOCAP-6/IEMOCAP-4, alongside the reported standard deviations.

For model size, SPAR's seven-class task modules contain 11.686M parameters: 9.292M in MSP and 2.395M in PRR, excluding the modality encoders. The appendix specifies MLPGate and NodeAttn controls with capacity comparable to PRR.

Reviewer_EkZ4OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes SPAR for utterance-level multimodal emotion recognition. The Modality Subset Prediction (MSP) module uses shared attention readers and a common classifier to produce a directly supervised prediction for each of the seven nonempty modality subsets. The Predictive Relation Routing (PRR) module then builds a directed graph over these predictions, with edges encoding pairwise probability differences, JS divergence, and shared and exclusive modality masks. Relation-aware attention over this graph learns sample-dependent weights for aggregating the subset logits. Experiments on MELD, IEMOCAP-4, and IEMOCAP-6 report the best scores in eight of nine comparisons with discriminative baselines, supported by ablations on MELD and an analysis grouped by unimodal disagreement.

Critical strengths

S1. A clear and well-positioned angle. Explicitly coupling disagreement between predictions with the modality sets that produced them is a reasonable refinement of adaptive late fusion, and the paper relates it clearly to subset modeling and expert weighting. This is the paper’s main conceptual contribution, although it is undermined by the supervision design (W2).

S2. Well-controlled ablations and careful reporting. MLPGate and NodeAttn use parameter counts comparable to PRR, which helps isolate the effect of relational modeling. The paper also reports per-class recalls, including weaknesses such as low recall on fear and disgust, and explicitly avoids overclaiming causal effects or robustness. This careful reporting increases confidence in what the paper does claim, but the evidence is limited to MELD and single runs (W5).

Critical weaknesses

Motivation

W1. Incremental novelty. Sample-adaptive fusion over modality subsets is a well-studied problem, with prior work such as DFG, Multimodal Routing, EMOE, MiSTER-E and DCR. The paper argues that “disagreement magnitude alone is insufficient,” but this argues against a design that existing gating methods do not actually adopt.

Method

W2. Mismatch between motivation and supervision. The paper is motivated by explicit conflicts between modality combinations. Yet all subset predictions are supervised with the same utterance-level label (Eq. 14, Sec. 3.2), which implicitly assumes that every combination should yield the same emotion. Under this setting, disagreement between subsets more likely reflects insufficient evidence or estimation error than genuine modality conflict. The authors themselves acknowledge this in Sec. 4.4, so the core argument is not supported by the method.

W3. Relation features add no new information. Probability differences, JS divergence, and modality masks can all be computed from the subset predictions and subset identities, which are already available to any gate. PRR therefore contributes an inductive bias rather than new information. Its advantage over NodeAttn is small in accuracy (+0.92 ACC on MELD) and has not been tested for stability across runs.

Experiments

W4. Unfair HIT comparison. SPAR is a closed-set discriminative classifier, whereas the generative baselines perform open-vocabulary prediction, and their numbers are quoted from another paper. HIT also excludes neutral. The comparison therefore does not support claims of superiority over generative models.

In summary, I believe the core contradiction lies in the design of the method and the conflict of motives. There is a conflict regarding the subset modalities, but no explicit modeling of distinction has been done. This is the main concern of the article.

Questions for authors

Questions to the Authors

Q1. Supervision and conflict modeling

Since all subsets share the same label, how does PRR distinguish genuine modality conflict from prediction error? Can the method be validated on a dataset with unimodal annotations (e.g., CH-SIMS)? What do the learned subset weights look like in practice, e.g., how often does the ATV prediction dominate?

Q2. Evaluation protocol

What data are used for early stopping and model selection on MELD and IEMOCAP?

2 comments
AuthorsOpenReview ↗

Response to Reviewer EkZ4(Part 1/2, Questions)

We thank the reviewer for the careful assessment of the motivation, supervision, and comparative evidence. We have clarified the contribution and revised the claims accordingly.

Q1. Shared-label supervision and learned subset weights

Each Modality Subset Prediction (MSP) branch estimates the same utterance-level emotion from a different evidence subset. A shared target aligns the candidates to the classification task while allowing their predictive distributions to differ. Predictive Relation Routing (PRR) learns how to aggregate these predictions; it does not identify whether a disagreement arises from semantic conflict, insufficient evidence, or prediction error. We have therefore framed the method around predictive disagreement and evidence composition.

For the routing analysis, A, T, and V denote audio, text, and vision; AT, AV, and TV are the bimodal candidates, and ATV uses all three modalities. We group MELD utterances by the unimodal top-1 predictions: G1: A=T=V; G2: T=V≠A; G3: A=V≠T; G4: A=T≠V; G5: all distinct. The complete mean routing weights underlying Figure 2 are below (%).

GroupATVATAVTVATV
G12.964.451.4320.3112.5219.7338.59
G23.573.831.3219.1715.6419.0237.46
G32.205.571.1520.1511.3023.2736.36
G42.695.231.0823.8110.6820.0536.46
G51.985.091.0523.349.7122.2336.61

ATV has the largest individual weight on 96.80% of utterances, counting ties fractionally, but its mean weight is approximately 37.3%, and its weight exceeds 50% on only 0.578% of utterances. Bimodal candidates together receive 52.57–55.27% mean weight across the five groups. Thus, the trimodal candidate is strongly weighted while the remaining candidates collectively make substantial contributions.

To measure their predictive benefit, Table 3 compares three rules using identical logits zSz_S from the MELD model trained with a fixed random seed: ATV-only selects zATVz_{ATV}; Uniform averages all seven logits; PRR uses their learned weighted sum with scalar weights wSw_S. Specifically,

ATV-only:

z^ATV=zATV\widehat{z}_{\mathrm{ATV}} = z_{\mathrm{ATV}}

Uniform:

z^Uniform=17∑SzS\widehat{z}_{\mathrm{Uniform}} = \frac{1}{7}\sum_{S} z_S

PRR:

z^PRR=∑SwSzS\widehat{z}_{\mathrm{PRR}} = \sum_{S} w_S z_S

Each rule is followed by softmax. The complete Table 3 is below (%); ACC, WF1, and Macro-F1 denote accuracy, weighted F1, and macro-F1.

GroupATV-only ACCATV-only WF1ATV-only Macro-F1Uniform ACCUniform WF1Uniform Macro-F1PRR ACCPRR WF1PRR Macro-F1
All61.8961.2745.4663.5161.8844.3266.7466.1249.78
G175.7971.7030.4377.6372.9130.5579.4874.6828.14
G255.2955.5834.6756.4754.7531.6359.2260.1241.49
G344.9243.7238.3746.8745.4934.9449.0347.8843.42
G467.4366.5656.7968.8967.7855.7571.6670.7260.02
G542.4041.6337.9443.5743.3039.8353.2252.5050.25

For each group and metric, the best result across the three aggregation rules is bold.

PRR improves overall ACC/WF1/Macro-F1 over ATV-only by 4.86/4.85/4.32 percentage points (pp) and over Uniform by 3.24/4.24/5.46 pp. All three metrics improve in every disagreement group, with the largest gains in G5: 10.82/10.87/12.31 pp over ATV-only and 9.65/9.20/10.42 pp over Uniform. In G1, ACC/WF1 improve while Macro-F1 is lower.

Q2. Validation and model selection

MELD uses its official training, validation, and test splits. For both IEMOCAP settings, Sessions 1–4 are divided into fixed 90% training and 10% validation subsets, shared across methods and seeds; Session 5 is reserved exclusively for final testing. Validation data determine early stopping and model selection.

AuthorsOpenReview ↗

Response to Reviewer EkZ4(Part 1/2, Weaknesses)

W1. Contribution relative to prior adaptive fusion methods

We agree that subset modeling and adaptive fusion have important precedents, including DFG, Multimodal Routing, EMOE, MiSTER-E, and DCR. SPAR's contribution is to explicitly couple pairwise predictive differences with shared/exclusive modality identities and use these relations in both attention biases and message updates. The revision presents this as a structured relational inductive bias for subset-logit aggregation. We have also revised the motivation to build on earlier gating methods without suggesting that they rely on disagreement magnitude alone.

W2. Alignment between motivation and supervision

As explained in Q1's opening paragraph, the common label defines the prediction target, while different evidence subsets can produce different predictive distributions. The revised motivation concerns learning how those predictions should contribute to the final decision. Q1's complete weight table and fixed-candidate Table 3 quantify these contributions, including the benefit over using the trimodal candidate alone. This aligns the motivation with the supervised classification objective.

W3. Relational inductive bias and component evidence

We agree that the relation features are deterministic functions of the predictions and subset identities already available at the nodes. Their contribution is explicit relational encoding within attention and message passing. To examine this contribution, NoPredEdges removes probability differences and Jensen–Shannon divergence jointly, while NoSetEdges removes only the nine shared/exclusive modality masks; both retain all node descriptors. NodeAttn retains node features but removes edge-conditioned attention biases and edge-dependent messages. The remaining controls examine candidate diversity (SingleSubset: trimodal only; NoBimodal: exclude bimodal candidates from aggregation), simpler aggregation (Uniform: equal weights; MLPGate: independent MLP scoring), and continued subset supervision (NoAux: remove its loss during joint optimization).

The complete revised Table 2 is below (%), including the added IEMOCAP-6 experiments. These component controls use a fixed random seed, as specified in Appendix F.1.

ConfigurationMELD ACCMELD WF1MELD Macro-F1IEMOCAP-6 ACCIEMOCAP-6 WF1IEMOCAP-6 Macro-F1
Full SPAR66.7466.1249.7866.0366.0064.20
SingleSubset63.8262.1342.6662.2162.3860.02
NoBimodal65.5964.0146.6563.8163.4961.21
Uniform63.1262.2643.8161.8461.8259.04
MLPGate64.6262.4343.5662.1562.2261.14
NodeAttn65.8263.1044.1364.0664.0462.69
NoPredEdges66.1764.1046.5364.5564.2063.03
NoSetEdges66.4565.4750.2464.9265.0362.13
NoAux62.8962.4044.9361.3461.9560.68

Best results in each metric column are bold.

Relative to NodeAttn, Full gains 0.92/3.02/5.65 pp on MELD and 1.97/1.96/1.51 pp on IEMOCAP-6 in ACC/WF1/Macro-F1. Its WF1 gains over NoPredEdges/NoSetEdges are 2.02/0.65 pp on MELD and 1.80/0.97 pp on IEMOCAP-6. On MELD, NoSetEdges has higher Macro-F1 (50.24 versus 49.78).

W4. Generative-model HIT comparison

We have moved the entire HIT panel and its discussion from Table 1 and Section 4 to Appendix D and removed the superiority interpretation. Appendix D now states these sources and protocol differences; the values are supplementary references. Main-text comparative claims concern the reproduced discriminative classification results.

Reviewer_XDBnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SPAR for multimodal emotion recognition. The Modality Subset Prediction module uses shared attention readers and a common classifier to generate predictions for every nonempty subset of audio, text, and vision. The Predictive Relation Routing module combines probability differences, Jensen-Shannon divergence, and information about modalities shared by or exclusive to each subset pair to adaptively weight the subset logits. The model is jointly trained using classification losses on the subset predictions and the final output. Across MELD and the four- and six-class IEMOCAP settings, the paper reports the highest scores in eight of nine dataset-metric comparisons against discriminative baselines. It also includes ablation studies and an analysis grouped by the degree of disagreement among unimodal predictions.

Critical strengths
  1. The paper addresses how a modality’s contribution depends on the accompanying evidence, jointly modeling differences between candidate predictions and their modality composition rather than evaluating each candidate’s importance independently. Subset prediction, relation modeling, and joint supervision form a coherent technical framework, providing a structured design for sample-adaptive fusion.
  2. The model outperforms the listed discriminative baselines on most reported metrics. The ablation results support the benefits of relation-aware routing over uniform aggregation, independent gating, and node-only attention. The disagreement-based analysis also shows larger gains when unimodal predictions exhibit greater disagreement. These observations are consistent with the method’s motivation.
Critical weaknesses
  1. The paper does not report results over multiple random seeds, standard deviations, or confidence intervals. Some improvements are relatively small, for example, in the six-class setting, accuracy and weighted F1 increase by only 0.49 and 0.43 percentage points, respectively, while macro-F1 decreases by 0.15 percentage points. It is therefore difficult to determine whether these differences are stable. In addition, the main text specifies a supervised warm start, whereas the supplementary materials describe random initialization of the task-specific modules followed directly by joint training. These descriptions are inconsistent. The data splits used for early stopping, model selection, and final testing also require clarification. These issues affect the reproducibility of the reported results and the assessment of their reliability.
  2. The existing ablations provide preliminary evidence for the effectiveness of relation modeling. However, NoPredEdges removes probability differences and Jensen-Shannon divergence together, and there is no control that retains predictive relations while removing only the modality-set relation masks. Since the node descriptors already contain predictive information and modality composition, the additional contribution of explicit modality-set relation edges has not been fully isolated. The key ablations are limited to MELD, and Figure 2 reports only group-level performance without directly showing how routing weights adjust as the evidence changes. Consequently, the specific benefits over standard aggregation of candidate predictions and the explanation of the underlying mechanism remain insufficiently established.
Questions for authors
  1. Could the authors report the mean and standard deviation for SPAR and the strongest baseline across multiple independent random seeds? Please also specify which data are used for early stopping, model selection, and final testing, and clarify whether the final test data are kept separate from those used for early stopping and model selection.
  2. Please analyze the contributions of probability differences, Jensen-Shannon divergence, and modality-set relation masks separately, particularly through a control that retains predictive relations but removes the modality-set masks. Where available, please provide key ablation results on another dataset, along with analyses of routing weights or controlled modality perturbations, to clarify how the contributions of different subsets change.
  3. The main text describes a task-supervised warm start followed by joint optimization, whereas the supplementary materials state that the task-specific modules are randomly initialized and trained jointly from the outset. Which procedure was used to obtain the results in Tables 1 and 2? Please provide details of initialization, warm-start training, and subsequent training that are consistent with the reported experiments.
5 comments
AuthorsOpenReview ↗

Response to Reviewer XDBn (Part 1/3, Questions 1)

We thank the reviewer for the constructive suggestions on reliability, reproducibility, and the evidence for relational routing.

Q1. Repeated-run evaluation and data separation

We have added results over three independent seeds for SPAR and all six reproduced baselines. The complete revised Table 1 is below. Entries are mean ± standard deviation (%); ACC, WF1, and Macro-F1 denote accuracy, weighted F1, and macro-F1, respectively.

MethodMELD ACCMELD WF1MELD Macro-F1IEMOCAP-6 ACCIEMOCAP-6 WF1IEMOCAP-6 Macro-F1IEMOCAP-4 ACCIEMOCAP-4 WF1IEMOCAP-4 Macro-F1
DMD59.32 ± 2.2258.49 ± 2.3441.13 ± 1.5659.56 ± 1.1859.19 ± 0.8957.72 ± 0.5376.39 ± 0.8576.63 ± 0.9176.92 ± 0.68
ALMT59.88 ± 1.9958.57 ± 1.9837.69 ± 3.9659.12 ± 0.7559.19 ± 0.6258.06 ± 0.2476.01 ± 0.5276.19 ± 0.5476.68 ± 0.80
DCR-utter59.33 ± 2.1057.99 ± 2.7339.99 ± 2.7960.56 ± 1.2860.36 ± 1.4058.58 ± 1.4978.78 ± 1.3978.92 ± 1.3079.19 ± 1.02
PDCC-MER62.08 ± 1.8460.58 ± 1.5842.42 ± 1.0763.15 ± 0.7963.12 ± 0.8262.37 ± 0.4879.08 ± 1.1179.09 ± 1.0978.90 ± 0.73
DQF62.47 ± 2.7061.79 ± 2.6947.87 ± 1.8362.70 ± 1.1563.12 ± 1.0762.38 ± 0.9379.32 ± 0.5579.56 ± 0.3979.09 ± 0.31
TiCAL63.12 ± 2.3163.33 ± 1.7647.35 ± 1.9863.77 ± 1.5463.81 ± 1.5263.06 ± 1.1880.12 ± 0.4780.31 ± 0.5780.48 ± 0.64
SPAR (Ours)65.42 ± 2.1364.81 ± 1.6749.59 ± 0.8165.56 ± 0.7165.37 ± 0.6963.53 ± 0.6481.71 ± 0.2481.78 ± 0.3381.40 ± 0.58

Best means in each metric column are bold.

SPAR achieves the highest mean in all nine dataset–metric comparisons. On IEMOCAP-6, its gains over TiCAL are 1.79/1.56/0.47 percentage points (pp) in ACC/WF1/Macro-F1, with lower standard deviations on all three metrics. On MELD, DQF is the strongest Macro-F1 baseline, and SPAR improves upon it by 1.72 pp. These repeated-run results replace the original single-run comparison.

For evaluation, MELD uses its official train/validation/test splits. In both IEMOCAP settings, Sessions 1–4 are divided into fixed 90% training and 10% validation subsets, shared across methods and seeds; Session 5 is reserved exclusively for final testing. Validation data determine early stopping and model selection in both training stages.

AuthorsOpenReview ↗

Response to Reviewer XDBn (Part 2/3, Questions 2)

Q2. Relation components, cross-dataset ablations, and routing behavior

SPAR uses Modality Subset Prediction (MSP) for supervised candidates and Predictive Relation Routing (PRR) for scalar-weighted logit aggregation.

Relation controls and cross-dataset evidence. We added NoSetEdges, removing the nine shared/sender-only/receiver-only modality-mask components while retaining predictive edges. NoPredEdges removes probability differences and Jensen–Shannon (JS) divergence jointly, retaining modality-set edges. Both retain all node descriptors.

Table 2 reports all nine configurations on MELD and IEMOCAP-6 (%), using a one seed.

ConfigurationMELD ACCMELD WF1MELD Macro-F1IEMOCAP-6 ACCIEMOCAP-6 WF1IEMOCAP-6 Macro-F1
Full SPAR66.7466.1249.7866.0366.0064.20
SingleSubset63.8262.1342.6662.2162.3860.02
NoBimodal65.5964.0146.6563.8163.4961.21
Uniform63.1262.2643.8161.8461.8259.04
MLPGate64.6262.4343.5662.1562.2261.14
NodeAttn65.8263.1044.1364.0664.0462.69
NoPredEdges66.1764.1046.5364.5564.2063.03
NoSetEdges66.4565.4750.2464.9265.0362.13
NoAux62.8962.4044.9361.3461.9560.68

Best results in each metric column are bold.

Full exceeds NoPredEdges by 0.57/2.02/3.25 pp on MELD and 1.48/1.80/1.17 pp on IEMOCAP-6 in ACC/WF1/Macro-F1. Relative to NoSetEdges, the gains are 0.29/0.65/−0.46 pp and 1.11/0.97/2.07 pp, respectively. Thus, modality-set edges improve ACC/WF1 on both datasets; MELD Macro-F1 is higher without them (50.24 versus 49.78). On IEMOCAP-6, Full leads all three metrics, and its WF1 gains over SingleSubset, NoBimodal, Uniform, MLPGate, NodeAttn, and NoAux are 3.62/2.51/4.18/3.78/1.96/4.05 pp.

Learned subset weights. For the MELD model trained with a fixed seed, A/T/V denote audio/text/vision; concatenated letters denote joint inputs (AT: audio–text; ATV: all three). Groups use unimodal top-1 predictions: G1: A=T=V; G2: T=V≠A; G3: A=V≠T; G4: A=T≠V; G5: all distinct. Mean weights for all seven subsets (Figure 2) are below (%).

GroupATVATAVTVATV
G12.964.451.4320.3112.5219.7338.59
G23.573.831.3219.1715.6419.0237.46
G32.205.571.1520.1511.3023.2736.36
G42.695.231.0823.8110.6820.0536.46
G51.985.091.0523.349.7122.2336.61

Bimodal allocations differ across the two-against-one groups: AV receives 15.64% in G2 versus 10.68% in G4, AT receives 19.17% versus 23.81%, and TV reaches 23.27% in G3, showing that allocations vary with which unimodal predictions disagree.

Fixed-candidate aggregation. Table 3 uses identical logits zSz_S from this model: ATV-only selects zATVz_{ATV}; Uniform uses z^=17∑SzS\widehat{z}=\frac{1}{7}\sum_S z_S; PRR uses z^=∑SwSzS\widehat{z}=\sum_S w_S z_S with learned scalar weights wSw_S. Each rule yields p^=softmax⁡(z^)\widehat{p}=\operatorname{softmax}(\widehat{z}). Results are below (%).

GroupATV-only ACCATV-only WF1ATV-only Macro-F1Uniform ACCUniform WF1Uniform Macro-F1PRR ACCPRR WF1PRR Macro-F1
All61.8961.2745.4663.5161.8844.3266.7466.1249.78
G175.7971.7030.4377.6372.9130.5579.4874.6828.14
G255.2955.5834.6756.4754.7531.6359.2260.1241.49
G344.9243.7238.3746.8745.4934.9449.0347.8843.42
G467.4366.5656.7968.8967.7855.7571.6670.7260.02
G542.4041.6337.9443.5743.3039.8353.2252.5050.25

Best results across rules for each group and metric are bold.

PRR improves overall ACC/WF1/Macro-F1 over Uniform by 3.24/4.24/5.46 pp and ATV-only by 4.86/4.85/4.32 pp. ACC/WF1 improve in every group and all metrics in G2–G5. Gains peak in G5: 9.65/9.20/10.42 pp over Uniform and 10.82/10.87/12.31 pp over ATV-only. In G1, ACC/WF1 improve while Macro-F1 is lower.

AuthorsOpenReview ↗

Response to Reviewer XDBn (Part 3/3, Questions 3 & Weakness)

Q3. Initialization and two-stage training

Thank you for identifying the inconsistent descriptions. The reported SPAR results in Tables 1 and 2 use two-stage training. Starting from public pretrained encoders, we first train the encoders and Modality Subset Prediction (MSP) module with task supervision and uniform aggregation of subset logits. We then load the resulting encoder/MSP checkpoint, initialize Predictive Relation Routing (PRR) afresh with a zero-initialized final scorer, and jointly optimize the encoders, MSP, and PRR. Early stopping and checkpoint selection use the validation data described in Q1. The manuscript and supplementary README now describe this procedure consistently.

W1. Reliability and reproducibility

The revised evaluation reports three-run means and standard deviations for all methods and explicitly separates validation-based selection from final testing; see Q1, Table 1 and the evaluation-protocol paragraph. Q3 specifies the supervised warm start, checkpoint loading, PRR initialization, and joint optimization used for the reported results. These revisions address the three reproducibility concerns together.

W2. Component isolation and routing evidence

Q2 provides the added NoSetEdges control, the complete MELD/IEMOCAP-6 ablations in Table 2, all seven routing weights across five prediction patterns, and the complete fixed-candidate results in Table 3. Together, these results quantify the roles of relational encoding, subset diversity, and learned aggregation. The MELD NoSetEdges Macro-F1 exception and the joint scope of the predictive-edge ablation are stated in Q2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.