- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose a bespoke architecture for multilabel emotion recognition. This architecture includes pathways for the power set of modalities, and graph-based merging before the final classification. This allows the architecture versatility to combine modality information dynamically. The authors then offer comparison against other models on two prominent datasets in the field, MELD and IEMOCAP, across many metrics, and offer ablation studies to justify the complexity of their architecture.
Critical strengths
- The authors build capacity in their network to allow it to dynamically and explicitly allocate importance to individual modalities or combinations of modalities by modeling their relationships, a very interesting property of their architecture. Their ablation study in Table 2 is also helpful in justifying the complexity of the network.
- The authors reimplement and rerun a lot of baselines originally designed for other benchmarks to position their proposed model among the current state of the art.
Critical weaknesses
- The complexity of the architecture itself is, I believe, a weakness of this work. For reference, the formulation of SPAR takes more real estate than the "Attention is All You Need, Vaswani et al., 2017" paper needed for the formulation of the Transformer. This complexity of course does not diminish the scientific contribution of the work. However, coupled with the lack of intuition offered behind the design, I believe it may be of little interest to the ICLR or other communities.
- The datasets are quite old, and carry biases and simplifications that have been challenged for several years now, e.g., single-label settings, annotator disagreement disregarded, acted (overt) vs. genuine emotions, etc. It would be interesting to see a study in a newer, more realistic setting, and move the field forward from this nascent state of benchmaxxing.
- The paper reports improvements over six selected discriminative baselines on IEMOCAP (and a few generative models). However:
- the comparison may omit other recent approaches, such as MiSTER-E (already cited in related work), JOYFUL by Li et al., 2023, etc. These methods report higher performance across some of the metrics reported for SPAR. While the setting might not be identical, the authors should describe why particularly pertinent previous work has been omitted from Table 1. (Also, the authors need to justify why dropping conversational context, for example, is a justified choice, among the other decisions made for the inputs)
- the gains, for all the described complexity, are on the order of 0.5-1/100 for many metrics, which have not be shown to be substantial/significant, e.g., by including standard deviations across runs/seeds. It would also be useful to get a sense of the number of trainable parameters across architectures.
Questions for authors
- The paper does not establish, for the reader, that the baselines have used the same backbones as the ones reported for SPAR. It would be useful to include an explicit acknowledgement in the main paper itself of (i) the fact that baselines where re-implemented/re-runs, and (ii) that the same backbones have been used across models so that improvement is not an artifact of using different encoders, which is of course also important (back to nascent state of the benchmarks: if using more modern backbones can improve performance, then these numbers should be reported; gains when using a [masked] language model from 2020 might be attributable to the suboptimal language embeddings, for example, and not the additional multimodal information captured by the proposed network)
- Could you provide more intuition and experimental findings on what each component is achieving in the network? What are the modalities used by the model for the final classification, and how does each modality's/modality combination's prediction relate to the models final prediction? etc.
3 comments
Response to Reviewer 4P2L(Part 2/3, Questions 2)
Q2. Component intuition and contributions to the final prediction
Modality Subset Prediction (MSP) preserves directly supervised predictions for A, T, V, AT, AV, TV, and ATV, where A/T/V denote audio/text/vision and concatenated letters identify joint inputs. Its shared readers and common classifier provide comparable predictions from different evidence combinations. Predictive Relation Routing (PRR) compares their distributions together with the modalities shared by or exclusive to each pair and learns one scalar aggregation weight per candidate. For example, comparing T with AT relates a predictive change to the inclusion of audio. Auxiliary supervision maintains each candidate's predictive utility regardless of its current routing weight. With complete inputs, PRR operates on seven candidates and 49 ordered pairs.
Component evidence. The complete revised Table 2 (adding NoSetEdges and extension to IEMOCAP-6) is below (%). These comparisons use a fixed random seed.
| Configuration | MELD ACC | MELD WF1 | MELD Macro-F1 | IEMOCAP-6 ACC | IEMOCAP-6 WF1 | IEMOCAP-6 Macro-F1 |
|---|---|---|---|---|---|---|
| Full SPAR | 66.74 | 66.12 | 49.78 | 66.03 | 66.00 | 64.20 |
| SingleSubset | 63.82 | 62.13 | 42.66 | 62.21 | 62.38 | 60.02 |
| NoBimodal | 65.59 | 64.01 | 46.65 | 63.81 | 63.49 | 61.21 |
| Uniform | 63.12 | 62.26 | 43.81 | 61.84 | 61.82 | 59.04 |
| MLPGate | 64.62 | 62.43 | 43.56 | 62.15 | 62.22 | 61.14 |
| NodeAttn | 65.82 | 63.10 | 44.13 | 64.06 | 64.04 | 62.69 |
| NoPredEdges | 66.17 | 64.10 | 46.53 | 64.55 | 64.20 | 63.03 |
| NoSetEdges | 66.45 | 65.47 | 50.24 | 64.92 | 65.03 | 62.13 |
| NoAux | 62.89 | 62.40 | 44.93 | 61.34 | 61.95 | 60.68 |
Full improves WF1 over SingleSubset by 3.99/3.62 pp and over NoBimodal by 2.11/2.51 pp on MELD/IEMOCAP-6. Its WF1 gains over NodeAttn are 3.02/1.96 pp, and removing continued auxiliary supervision reduces WF1 by 3.72/4.05 pp. These comparisons connect the modules to preserving useful subset candidates and combining them through learned relations. The MELD NoSetEdges Macro-F1 result (50.24 versus 49.78) is retained in the table.
Learned weights. For the MELD routing analysis, groups use unimodal top-1 predictions: G1: A=T=V; G2: T=V≠A; G3: A=V≠T; G4: A=T≠V; G5: all distinct. The complete mean weights underlying Figure 2 are below (%).
| Group | A | T | V | AT | AV | TV | ATV |
|---|---|---|---|---|---|---|---|
| G1 | 2.96 | 4.45 | 1.43 | 20.31 | 12.52 | 19.73 | 38.59 |
| G2 | 3.57 | 3.83 | 1.32 | 19.17 | 15.64 | 19.02 | 37.46 |
| G3 | 2.20 | 5.57 | 1.15 | 20.15 | 11.30 | 23.27 | 36.36 |
| G4 | 2.69 | 5.23 | 1.08 | 23.81 | 10.68 | 20.05 | 36.46 |
| G5 | 1.98 | 5.09 | 1.05 | 23.34 | 9.71 | 22.23 | 36.61 |
ATV has approximately 37.3% mean weight, while bimodal candidates together receive 52.57–55.27% across these groups. ATV has the largest individual weight on 96.80% of utterances, counting ties fractionally, but exceeds 50% weight on only 0.578%. The final prediction therefore combines a strongly weighted trimodal candidate with substantial contributions from other subsets.
Relationship to the final prediction. Let be each candidate's logits and its learned scalar weight. The final distribution is
Table 3 isolates aggregation using identical logits from the MELD model trained with a fixed random seed: ATV-only selects , Uniform averages all seven logits, and PRR uses their learned weighted sum. The complete results are below (%).
| Group | ATV-only ACC | ATV-only WF1 | ATV-only Macro-F1 | Uniform ACC | Uniform WF1 | Uniform Macro-F1 | PRR ACC | PRR WF1 | PRR Macro-F1 |
|---|---|---|---|---|---|---|---|---|---|
| All | 61.89 | 61.27 | 45.46 | 63.51 | 61.88 | 44.32 | 66.74 | 66.12 | 49.78 |
| G1 | 75.79 | 71.70 | 30.43 | 77.63 | 72.91 | 30.55 | 79.48 | 74.68 | 28.14 |
| G2 | 55.29 | 55.58 | 34.67 | 56.47 | 54.75 | 31.63 | 59.22 | 60.12 | 41.49 |
| G3 | 44.92 | 43.72 | 38.37 | 46.87 | 45.49 | 34.94 | 49.03 | 47.88 | 43.42 |
| G4 | 67.43 | 66.56 | 56.79 | 68.89 | 67.78 | 55.75 | 71.66 | 70.72 | 60.02 |
| G5 | 42.40 | 41.63 | 37.94 | 43.57 | 43.30 | 39.83 | 53.22 | 52.50 | 50.25 |
For each group and metric, the best result across the three aggregation rules is bold.
PRR improves overall ACC/WF1/Macro-F1 by 3.24/4.24/5.46 pp over Uniform and 4.86/4.85/4.32 pp over ATV-only. ACC/WF1 improve in every group, and all three metrics improve in G2–G5. The largest gains occur in G5: 9.65/9.20/10.42 pp over Uniform and 10.82/10.87/12.31 pp over ATV-only. In G1, Macro-F1 is lower (28.14 versus 30.55/30.43), while ACC/WF1 improve.
Response to Reviewer 4P2L(Part 3/3, Weaknesses)
W1. Architectural intuition and complexity
We have reorganized the method to explain each component's purpose before its equations. Q2 gives the intuition for MSP, PRR, and auxiliary supervision, followed by the complete component ablations in Table 2, learned weights, and fixed-candidate results in Table 3. This links the architecture directly to the predictions it preserves and how they influence the final decision.
W2. Benchmark scope and emotion-label limitations
We appreciate the concern that categorical benchmark accuracy captures only part of emotion understanding. We agree that categorical single-label evaluation captures only part of the subjectivity and ambiguity of emotion, and our current experiments do not model annotator disagreement.
We selected MELD and IEMOCAP as established benchmarks that remain in use in recent research, including Xiao et al. (Findings of EACL 2026) and Seong et al. (ACL 2026). This choice enables evaluation of the proposed aggregation mechanism against prior approaches under explicitly specified benchmark protocols.
Xiao L, Funakoshi K, Okumura M. Emotion Recognition in Multi-Speaker Conversations through Speaker Identification, Knowledge Distillation, and Hierarchical Fusion[C]//Findings of the Association for Computational Linguistics: EACL 2026. 2026: 4091-4106.
Seong E, Lee H, Kim D, et al. Emotion-Wheel-Guided Audio-Referred Text Representation for Multimodal Emotion Recognition in Conversation[C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026: 40392-40403.
W3. Baseline coverage, input scope, reliability, and model size
We restrict SPAR and the reproduced baselines to the target utterance's text, audio, and video to isolate within-utterance fusion from conversational modeling. MiSTER-E and JOYFUL are relevant related work, and their discussion has been expanded. Their published systems use conversational context and different input/evaluation settings; our comparative claims concern the reproduced utterance-level systems.
For repeated-run evidence, Q1 provides the complete three-seed main comparison (Table 1) and shared frozen-encoder comparison (Table 9). In Table 1, SPAR improves WF1 over TiCAL by 1.48/1.56/1.47 pp on MELD/IEMOCAP-6/IEMOCAP-4, alongside the reported standard deviations.
For model size, SPAR's seven-class task modules contain 11.686M parameters: 9.292M in MSP and 2.395M in PRR, excluding the modality encoders. The appendix specifies MLPGate and NodeAttn controls with capacity comparable to PRR.
Response to Reviewer 4P2L(Part 1/3, Questions 1)
We thank the reviewer for the suggestions on architectural intuition, benchmark scope, and comparison settings. We have expanded the empirical evidence and clarified the purpose of each component.
Q1. Baseline reproduction and encoder-controlled comparison
The baselines were rerun using public implementations where available and paper-based reimplementations otherwise, with input and classification adaptations for our evaluation. The original comparison retained method-specific encoders; the reproduction settings are detailed in Appendix F. The complete revised Table 1 is below, with mean ± standard deviation over three independent seeds (%). ACC, WF1, and Macro-F1 denote accuracy, weighted F1, and macro-F1.
Best means in each metric column are bold.
To directly control encoder differences, we have additionally compared all methods using the same public RoBERTa-large (text), Whisper-large-v3 (audio), and CLIP ViT-B/16 (vision) checkpoints, matched preprocessing, fixed data splits, and validation-based selection. All three encoders remain frozen throughout training for every method; only method-specific fusion modules are optimized. For SPAR, they remain frozen during both the supervised warm start and subsequent joint optimization of the subset-prediction and routing modules. The complete Appendix F.4 Table 9 reports mean ± standard deviation over three independent seeds (%).
Best means in each metric column are bold.
SPAR has the highest mean in all nine dataset–metric comparisons in both settings. In the shared frozen-encoder control, its ACC/WF1/Macro-F1 gains over the strongest baseline in each column are 1.07/1.43/2.83 percentage points (pp) on MELD, 1.20/1.34/0.79 pp on IEMOCAP-6, and 1.26/1.27/1.14 pp on IEMOCAP-4. These results show that the gains persist when encoder choice and updates are held fixed.