- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Authors proposed a multmodal ERC model by modeling speaker aware information, leveraging text, audio, and video modalities for emotion recognition in conversational context. Authors evaluated their proposed model on MELD and IEMOCAP datasets.
Critical strengths
None
Critical weaknesses
The model architecture is complex and reporting less than 1% improvement over MELD and IMEOCAP datasets is not worth to train such complex model.
Experimental settings is not convincing. All compared models in Table 1 are reproduced. So not sure why authors reproduced the results instead of reporting the accuracies from the papers.
Dataset splits used for train and test is unclear.
The proposed model misclassifies most similar emotions. It is unclear how authors generalize the proposed model when unavailability of speaker information.
Questions for authors
Authors say that the proposed model is speaker dependent emotional transition aware ERC model. But authors evaluate their model in speaker -independent evaluations? Contradiction. Authors should do ablation study to show that model has learned speaker dependent emotional transition? As such this claim is not supported in the paper. IEMOCAP dataset has 5 sessions and each session has one male and female speaker data. It also have scripted and improvised instances. It is not clear how authors considered dataset splits for training and evaluations. Also how authors took the data? Is it based on majority voting or atleast 3 people should agree on their annotations? Figure2. Authors should provide tsne visualizations for w/o MEMEA ,w/o STPL-PCR ,w/o DSA-MTBD before and after training for IEMOCAP and MELD datsets.
Minor errors: “speark-aware class-specific” it should be speaker-aware
Sections 3.1-3.3: many equations, notations. It is hard to follow what is Oallt? how many parameters involved in the proposed model? How sensitive the model is with respect to these many parameters?
From section 4.2, the overall improvement is 0.94 and 0.97 over prior methods? Complicated architecture with less than 1% improvement is not considered as major improvement and not worth given the model complexity, parameters involved, etc.
Database details and model evaluation settings is important to report in the main paper instead of including in Appendix.
Author should move section 4.5 to appendix and include confusion matrices to understand model capability in recognition of most similar emotions like happy vs excited , anger vs disgust
Authors should include dataset details in the main document and compare their model performance with SOTA that are consistent with the experimental setup. Table 1. Include citations to each method included in the table.
Authors should include SOTA methods that actually evaluated and reported results on MELD and IEMOCAP datasets rather than reproducing the results. So it is not guarantee to reproduce results with the same experimental settings. As such the Table 1 is not convincing. Why not authors take the reported results from the SOTA for comparison.
From section C.1, it is not clear what data authors considered for training and evaluation for IEMOCAP dataset?
Authors should compare their method with SOTA. For instance this paper proposes graph-based model and reports improved accuracy on MELD dataset. https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10858741
To understand the effectiveness of the proposed model, authors must be evaluate their model performance on CMU-MOSEI dataset.
From Figure 8: there is a confusion between similar emotions Hap-> Exc, Sad-< fru, Ang->fru Similarly, from figure 9: There is confusion between similar emotions and distinct emotions Fear Sur. Though the model is complex, the misclassification accuracy with Neutral emotion is very high. It is not convincing that with so many parameters and defined loss function is not improving the model performance in recognizing similar emotions. Model is improving overall accuracy by improving one category of emotion.
6 comments
Clearer explanations and Stronger supporting evidence
W2&Q7&Q8: Reported and reproduced baseline results
Response: We appreciate the reviewer’s concern about the provenance and reproducibility of the baseline results. We do not reproduce all baselines. In Table 1, the dagger symbol identifies reproduced results, whereas rows without a dagger use the values reported in the original papers under the corresponding benchmark protocol. A dash denotes an unreported class-wise result. We will move this explanation from the table caption to the experimental protocol and add a citation to every row so that the provenance is visible. We reproduced a method when its official implementation was available and a controlled run was needed to obtain missing class-wise results or align the data split and input features. Exact reproduced scores may vary slightly across random seeds and software or hardware environments, including CUDA/cuDNN versions and GPU types. Thus, we used each method’s official code and its author-recommended or best-reported configuration, without re-optimizing it on our test set. Method-specific settings, such as the optimizer, learning rate, dropout, and seed protocol, followed the original implementation, while the dataset split, input features, preprocessing, evaluation metrics, and checkpoint-selection criterion were aligned wherever supported. This preserves each method’s intended training recipe while providing a common evaluation protocol. When faithful reproduction was unavailable or the published protocol already matched ours, we retained the published result rather than substituting an approximate implementation. The revised table will include whether each result is Reported or Reproduced, together with its code source, split, feature setting, seed count, and execution environment. Results obtained under incompatible splits or feature protocols will be explicitly marked and will not be presented as directly controlled comparisons.
W3&Q1a&Q5&Q9: Dataset splits speaker-independent evaluation and annotation use
Response: We agree that the dataset splits, speaker-independent protocol, and annotation procedure should be stated explicitly.
For IEMOCAP, Sessions 1–4 constitute the training/validation pool, containing 120 dialogues and 5,810 utterances in total, while Session 5 is held out exclusively for testing, containing 31 dialogues and 1,623 utterances. The training and validation subsets are constructed only from Sessions 1–4, and the held-out test set is not used for either model training or model selection. Session 5 contains one female and one male actor who do not appear in Sessions 1–4; therefore, there is no speaker overlap between the training/validation pool and the test set. Both scripted and improvised dialogues are included. We use the released six-class utterance labels—angry, excited, frustrated, happy, neutral, and sad—and do not perform a new majority vote or alter the categorical annotations. The annotation aggregation follows the released IEMOCAP labels and the standard MERC preprocessing.
For MELD, we use the official split of 1,039 training dialogues, 114 validation dialogues, and 280 test dialogues. Model selection uses the validation split, and the test split is used only for final reporting. We will move these split details, the emotion set, and the evaluation metrics from the appendix into the main experimental settings.
Speaker-dependent describes the modeled relation within a dialogue, not memorization of a training speaker identity. STPL uses the local speaker index and, critically, whether adjacent turns belong to the same or different speaker. Speaker-independent evaluation is therefore the appropriate test: it asks whether the learned same/cross-speaker transition rule transfers to identities unseen during training.
Clearer explanations and Stronger supporting evidence
W4a&Q1b: Speaker information ablation and representation visualizations
Response: The benchmark provides speaker IDs for each utterance, but we agree that robustness to missing speaker metadata should be quantified. Thus, we have added two complementary controls. In the without-speaker-relation setting, we remove same-/cross-speaker conditioning and route all adjacent-turn pairs to a single shared transition bank. This collapses relation-specific transition dynamics, causing emotional persistence within one speaker and emotional responses across speakers to be modeled jointly. In the unknown-speaker setting, we replace only the speaker-identity embedding with a shared unknown token while retaining the same-/cross-speaker relation label and the remaining multimodal context. STPL can therefore still route each adjacent-turn pair to the corresponding relation-conditioned transition bank. Without speaker relation and unknown speaker obtain W-F1 scores of 73.92 and 74.17 on IEMOCAP, respectively, and 66.84 and 67.08 on MELD, compared with 75.09 and 67.98 for the full model. Together, these controls distinguish reliance on explicit speaker identity from reliance on relational speaker structure and show that the multimodal backbone remains effective when speaker identity metadata is unavailable.
| Configuration | IEMOCAP W-F1 | MELD W-F1 |
|---|---|---|
| Full model | 75.09 | 67.98 |
| Without speaker relation | 73.92 | 66.84 |
| Unknown speaker | 74.17 | 67.08 |
We agree that component-wise visualizations would clarify how each module shapes the representation space. In the revision, we will add t-SNE plots for the full model and variants without MEMEA, STPL-PCR, or DSA-MTBD on IEMOCAP, using them as supporting qualitative rather than standalone causal evidence.
Q2: Terminology correction
Response: We apologize for the error; it will be corrected to "speaker-aware class-specific" in Eq. (10), and we have checked the rest of the manuscript for the same mistake. We are grateful for the careful reading.
Q3: Notation parameter count and hyperparameter sensitivity
Response: Thank you for raising this valid concern. We agree that the mixed use of and made the fused-branch notation unnecessarily confusing, and this inconsistency is our responsibility. Here, denotes the emotion-logit vector produced for utterance by the fused multimodal branch, and = is the corresponding probability distribution used for the final prediction. We use = {} for the unimodal branches and M for the fused multimodal branch; accordingly, = {} denotes the complete prediction-branch set. In the revised manuscript, we define these symbols immediately after multimodal gated fusion and consistently replace every fused-branch superscript or index previously written as "all" with M throughout the method, loss definitions, algorithm, and notation appendix.
The full training model contains 69.39 million parameters on IEMOCAP and 73.27 million on MELD, compared with 64.08 million and 68.21 million for the corresponding backbones. Our existing sensitivity experiments vary the MEMEA, STPL, and AMPC objective weights and warm-up schedules, as well as the AMPC perturbation probability. Building on these experiments, we have added a unified sensitivity analysis of the eight loss-related hyperparameters highlighted in the training objective: the four peak objective weights , , , and ; the three internal AMPC coefficients , , and ; and the consistency temperature . The selected values are , , , , , , , and . We will report the complete sensitivity curves in the revised appendix and summarize the selected values and schedules in the main implementation section.
Clearer explanations and Stronger supporting evidence
Thank you for the careful and constructive feedback. These comments helped us identify where the original manuscript needed clearer explanations and stronger supporting evidence. We have revised the manuscript accordingly and provide detailed responses below.
W1&Q4: Sub one point gains statistical reliability and model cost
Response: The 0.97-point IEMOCAP and 0.94-point MELD improvements in Table 1 are differences between mean W-F1 values, not isolated best runs. Because the absolute margin is below one point, we agree that run-level uncertainty must be reported. Thus, we compare SEDAL with the strongest reproduced competitor under five matched runs: SDT on IEMOCAP and CMERC on MELD. Since neither MATCH nor AdaIGN has released its code, we use CMERC on MELD. Each pair uses the same split and random seed, and the test statistic is computed from the five paired W-F1 differences. Here, ΔIEMO denotes the paired difference in W-F1 between SEDAL and SDT, and ΔMELD the paired difference between SEDAL and CMERC. We run two-sided paired t-tests over the five matched runs. On IEMOCAP the mean improvement is +0.97, with a 95% confidence interval of [0.47, 1.47], t(4)=5.35, p=0.006<0.05; on MELD it is +1.61, with [1.15, 2.07], t(4)=9.71, p<0.001. All ten paired differences are positive, and on both datasets the weakest SEDAL run still exceeds the strongest competitor run (74.66 vs 74.39 and 67.74 vs 66.59), so the conclusion does not rest on a single favourable run. The run-to-run standard deviations of SEDAL are 0.32 and 0.23, i.e. the IEMOCAP gain is about three times the seed noise and the MELD gain about seven times, and the paired effect sizes are dz=2.39 and dz=4.34. These results indicate that both improvements are statistically significant and directionally consistent across the five matched runs. We also note that the +1.61 on MELD is measured against CMERC, the strongest competitor that can be retrained under our pipeline, whereas the +0.94 in Table 1 refers to the reported value of MATCH. More importantly, these gains incur no additional inference cost: the deployed model is a single student whose parameter count and FLOPs equal those of the backbone, since the anchor library, the transition prototype library and the teacher committee exist only during training.
The architecture is more complex during training, but the cost does not scale to an ensemble at inference. Relative to the common backbone, the auxiliary modules add 5.31 million parameters on IEMOCAP, or 8.29 percent, and 5.06 million on MELD, or 7.42 percent. DSA-MTBD uses cached teacher outputs, and the deployed prediction is produced by one fused student head. We will add measured training time, cache construction, peak memory, and single-student inference latency as 3.5 GPU-hours, 0.7 GPU-hours, 15.9 GB, and 28.1 ms per dialogue, respectively.