acceptodds
Under review as a conference paper at ICLR 2027

Speaker-dependent Emotional Transition-aware Multimodal Dynamic Anchor Learning for Multimodal Emotion Recognition in Conversations

Abstract

Multimodal Emotion Recognition in Conversations (MERC) is a crucial task for improving a machine's emotional understanding. It aims to accurately infer emotions from the complex interplay of textual, acoustic, and visual signals in conversational context. Although existing methods have improved performance via advanced strategies like multimodal fusion, graph-based reasoning, etc., they often leave psychologically meaningful multimodal priors and speaker-aware emotion transitions out of consideration to guide multimodal fusion and alignment. In this paper, we propose a novel learning framework, Speaker-dependent Emotional Transition-aware Multimodal Dynamic Anchor Learning (SEDAL), which introduces learnable class-level multimodal semantic anchors and speaker-aware emotional transition dynamics into the multimodal fusion and emotion recognition process. Specifically, SEDAL first constructs momentum-evolving multimodal emotion anchors, which serve as emotion-specific prototypes to guide the multimodal alignment among the text, audio, visual, and fused features. Then, SEDAL further extends these anchors from isolated utterances to adjacent conversational turns through speaker-aware emotion transition prototypes and applies anchor-guided modality perturbation consistency learning only on transition-relevant utterances. Finally, SEDAL further enhances recognition robustness with a deferred student-adaptive multi-teacher boundary distillation strategy, which transfers effective decisions from multiple validation-calibrated teachers. Extensive experiments on IEMOCAP and MELD datasets demonstrate that SEDAL achieves state-of-the-art performance, confirming its effectiveness for MERC.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal emotion recognition in conversations, Dynamic emotion anchor learning, Multimodal fusion, Multimodal alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UN2JOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Authors proposed a multmodal ERC model by modeling speaker aware information, leveraging text, audio, and video modalities for emotion recognition in conversational context. Authors evaluated their proposed model on MELD and IEMOCAP datasets.

Critical strengths

None

Critical weaknesses

The model architecture is complex and reporting less than 1% improvement over MELD and IMEOCAP datasets is not worth to train such complex model.

Experimental settings is not convincing. All compared models in Table 1 are reproduced. So not sure why authors reproduced the results instead of reporting the accuracies from the papers.

Dataset splits used for train and test is unclear.

The proposed model misclassifies most similar emotions. It is unclear how authors generalize the proposed model when unavailability of speaker information.

Questions for authors

Authors say that the proposed model is speaker dependent emotional transition aware ERC model. But authors evaluate their model in speaker -independent evaluations? Contradiction. Authors should do ablation study to show that model has learned speaker dependent emotional transition? As such this claim is not supported in the paper. IEMOCAP dataset has 5 sessions and each session has one male and female speaker data. It also have scripted and improvised instances. It is not clear how authors considered dataset splits for training and evaluations. Also how authors took the data? Is it based on majority voting or atleast 3 people should agree on their annotations? Figure2. Authors should provide tsne visualizations for w/o MEMEA ,w/o STPL-PCR ,w/o DSA-MTBD before and after training for IEMOCAP and MELD datsets.

Minor errors: “speark-aware class-specific” it should be speaker-aware

Sections 3.1-3.3: many equations, notations. It is hard to follow what is Oallt? how many parameters involved in the proposed model? How sensitive the model is with respect to these many parameters?

From section 4.2, the overall improvement is 0.94 and 0.97 over prior methods? Complicated architecture with less than 1% improvement is not considered as major improvement and not worth given the model complexity, parameters involved, etc.

Database details and model evaluation settings is important to report in the main paper instead of including in Appendix.

Author should move section 4.5 to appendix and include confusion matrices to understand model capability in recognition of most similar emotions like happy vs excited , anger vs disgust

Authors should include dataset details in the main document and compare their model performance with SOTA that are consistent with the experimental setup. Table 1. Include citations to each method included in the table.

Authors should include SOTA methods that actually evaluated and reported results on MELD and IEMOCAP datasets rather than reproducing the results. So it is not guarantee to reproduce results with the same experimental settings. As such the Table 1 is not convincing. Why not authors take the reported results from the SOTA for comparison.

From section C.1, it is not clear what data authors considered for training and evaluation for IEMOCAP dataset?

Authors should compare their method with SOTA. For instance this paper proposes graph-based model and reports improved accuracy on MELD dataset. https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10858741

To understand the effectiveness of the proposed model, authors must be evaluate their model performance on CMU-MOSEI dataset.

From Figure 8: there is a confusion between similar emotions Hap-> Exc, Sad-< fru, Ang->fru Similarly, from figure 9: There is confusion between similar emotions and distinct emotions Fear Sur. Though the model is complex, the misclassification accuracy with Neutral emotion is very high. It is not convincing that with so many parameters and defined loss function is not improving the model performance in recognizing similar emotions. Model is improving overall accuracy by improving one category of emotion.

6 comments
AuthorsOpenReview ↗

Clearer explanations and Stronger supporting evidence

Thank you for the careful and constructive feedback. These comments helped us identify where the original manuscript needed clearer explanations and stronger supporting evidence. We have revised the manuscript accordingly and provide detailed responses below.

W1&Q4: Sub one point gains statistical reliability and model cost

Response: The 0.97-point IEMOCAP and 0.94-point MELD improvements in Table 1 are differences between mean W-F1 values, not isolated best runs. Because the absolute margin is below one point, we agree that run-level uncertainty must be reported. Thus, we compare SEDAL with the strongest reproduced competitor under five matched runs: SDT on IEMOCAP and CMERC on MELD. Since neither MATCH nor AdaIGN has released its code, we use CMERC on MELD. Each pair uses the same split and random seed, and the test statistic is computed from the five paired W-F1 differences. Here, ΔIEMO denotes the paired difference in W-F1 between SEDAL and SDT, and ΔMELD the paired difference between SEDAL and CMERC. We run two-sided paired t-tests over the five matched runs. On IEMOCAP the mean improvement is +0.97, with a 95% confidence interval of [0.47, 1.47], t(4)=5.35, p=0.006<0.05; on MELD it is +1.61, with [1.15, 2.07], t(4)=9.71, p<0.001. All ten paired differences are positive, and on both datasets the weakest SEDAL run still exceeds the strongest competitor run (74.66 vs 74.39 and 67.74 vs 66.59), so the conclusion does not rest on a single favourable run. The run-to-run standard deviations of SEDAL are 0.32 and 0.23, i.e. the IEMOCAP gain is about three times the seed noise and the MELD gain about seven times, and the paired effect sizes are dzd_z=2.39 and dzd_z=4.34. These results indicate that both improvements are statistically significant and directionally consistent across the five matched runs. We also note that the +1.61 on MELD is measured against CMERC, the strongest competitor that can be retrained under our pipeline, whereas the +0.94 in Table 1 refers to the reported value of MATCH. More importantly, these gains incur no additional inference cost: the deployed model is a single student whose parameter count and FLOPs equal those of the backbone, since the anchor library, the transition prototype library and the teacher committee exist only during training.

IDSEDAL IEMOCAPSDT IEMOCAPΔ<sub>IEMO</sub>SEDAL MELDCMERC MELDΔ_{MELD}
174.6673.880.7867.9866.021.96
275.3573.951.4067.7466.581.16
374.9474.390.5568.2466.242.00
475.4674.051.4167.7666.421.34
575.0474.330.7168.1866.591.59
Mean ± Std.75.09 ± 0.3274.12 ± 0.280.97 ± 0.4167.98 ± 0.2366.37 ± 0.241.61 ± 0.37

The architecture is more complex during training, but the cost does not scale to an ensemble at inference. Relative to the common backbone, the auxiliary modules add 5.31 million parameters on IEMOCAP, or 8.29 percent, and 5.06 million on MELD, or 7.42 percent. DSA-MTBD uses cached teacher outputs, and the deployed prediction is produced by one fused student head. We will add measured training time, cache construction, peak memory, and single-student inference latency as 3.5 GPU-hours, 0.7 GPU-hours, 15.9 GB, and 28.1 ms per dialogue, respectively.

DatasetBackbone parametersFull training parametersIncreaseTeacher ensemble at inference
IEMOCAP64.08 M69.39 M8.29%No
MELD68.21 M73.27 M7.42%No
AuthorsOpenReview ↗

Clearer explanations and Stronger supporting evidence

W2&Q7&Q8: Reported and reproduced baseline results

Response: We appreciate the reviewer’s concern about the provenance and reproducibility of the baseline results. We do not reproduce all baselines. In Table 1, the dagger symbol identifies reproduced results, whereas rows without a dagger use the values reported in the original papers under the corresponding benchmark protocol. A dash denotes an unreported class-wise result. We will move this explanation from the table caption to the experimental protocol and add a citation to every row so that the provenance is visible. We reproduced a method when its official implementation was available and a controlled run was needed to obtain missing class-wise results or align the data split and input features. Exact reproduced scores may vary slightly across random seeds and software or hardware environments, including CUDA/cuDNN versions and GPU types. Thus, we used each method’s official code and its author-recommended or best-reported configuration, without re-optimizing it on our test set. Method-specific settings, such as the optimizer, learning rate, dropout, and seed protocol, followed the original implementation, while the dataset split, input features, preprocessing, evaluation metrics, and checkpoint-selection criterion were aligned wherever supported. This preserves each method’s intended training recipe while providing a common evaluation protocol. When faithful reproduction was unavailable or the published protocol already matched ours, we retained the published result rather than substituting an approximate implementation. The revised table will include whether each result is Reported or Reproduced, together with its code source, split, feature setting, seed count, and execution environment. Results obtained under incompatible splits or feature protocols will be explicitly marked and will not be presented as directly controlled comparisons.

W3&Q1a&Q5&Q9: Dataset splits speaker-independent evaluation and annotation use

Response: We agree that the dataset splits, speaker-independent protocol, and annotation procedure should be stated explicitly.

For IEMOCAP, Sessions 1–4 constitute the training/validation pool, containing 120 dialogues and 5,810 utterances in total, while Session 5 is held out exclusively for testing, containing 31 dialogues and 1,623 utterances. The training and validation subsets are constructed only from Sessions 1–4, and the held-out test set is not used for either model training or model selection. Session 5 contains one female and one male actor who do not appear in Sessions 1–4; therefore, there is no speaker overlap between the training/validation pool and the test set. Both scripted and improvised dialogues are included. We use the released six-class utterance labels—angry, excited, frustrated, happy, neutral, and sad—and do not perform a new majority vote or alter the categorical annotations. The annotation aggregation follows the released IEMOCAP labels and the standard MERC preprocessing.

For MELD, we use the official split of 1,039 training dialogues, 114 validation dialogues, and 280 test dialogues. Model selection uses the validation split, and the test split is used only for final reporting. We will move these split details, the emotion set, and the evaluation metrics from the appendix into the main experimental settings.

Speaker-dependent describes the modeled relation within a dialogue, not memorization of a training speaker identity. STPL uses the local speaker index and, critically, whether adjacent turns belong to the same or different speaker. Speaker-independent evaluation is therefore the appropriate test: it asks whether the learned same/cross-speaker transition rule transfers to identities unseen during training.

AuthorsOpenReview ↗

Clearer explanations and Stronger supporting evidence

W4a&Q1b: Speaker information ablation and representation visualizations

Response: The benchmark provides speaker IDs for each utterance, but we agree that robustness to missing speaker metadata should be quantified. Thus, we have added two complementary controls. In the without-speaker-relation setting, we remove same-/cross-speaker conditioning and route all adjacent-turn pairs to a single shared transition bank. This collapses relation-specific transition dynamics, causing emotional persistence within one speaker and emotional responses across speakers to be modeled jointly. In the unknown-speaker setting, we replace only the speaker-identity embedding with a shared unknown token while retaining the same-/cross-speaker relation label and the remaining multimodal context. STPL can therefore still route each adjacent-turn pair to the corresponding relation-conditioned transition bank. Without speaker relation and unknown speaker obtain W-F1 scores of 73.92 and 74.17 on IEMOCAP, respectively, and 66.84 and 67.08 on MELD, compared with 75.09 and 67.98 for the full model. Together, these controls distinguish reliance on explicit speaker identity from reliance on relational speaker structure and show that the multimodal backbone remains effective when speaker identity metadata is unavailable.

ConfigurationIEMOCAP W-F1MELD W-F1
Full model75.0967.98
Without speaker relation73.9266.84
Unknown speaker74.1767.08

We agree that component-wise visualizations would clarify how each module shapes the representation space. In the revision, we will add t-SNE plots for the full model and variants without MEMEA, STPL-PCR, or DSA-MTBD on IEMOCAP, using them as supporting qualitative rather than standalone causal evidence.

Q2: Terminology correction

Response: We apologize for the error; it will be corrected to "speaker-aware class-specific" in Eq. (10), and we have checked the rest of the manuscript for the same mistake. We are grateful for the careful reading.

Q3: Notation parameter count and hyperparameter sensitivity

Response: Thank you for raising this valid concern. We agree that the mixed use of OallO_{all} and otMo_t^{M} made the fused-branch notation unnecessarily confusing, and this inconsistency is our responsibility. Here, otMo_t^{M} denotes the emotion-logit vector produced for utterance utu_t by the fused multimodal branch, and ptMp_t^{M}=softmax(otM)softmax(o_t^{M}) is the corresponding probability distribution used for the final prediction. We use MM = {T,A,VT, A, V} for the unimodal branches and M for the fused multimodal branch; accordingly, M+M^{+} = {T,A,V,MT, A, V, M} denotes the complete prediction-branch set. In the revised manuscript, we define these symbols immediately after multimodal gated fusion and consistently replace every fused-branch superscript or index previously written as "all" with M throughout the method, loss definitions, algorithm, and notation appendix.

The full training model contains 69.39 million parameters on IEMOCAP and 73.27 million on MELD, compared with 64.08 million and 68.21 million for the corresponding backbones. Our existing sensitivity experiments vary the MEMEA, STPL, and AMPC objective weights and warm-up schedules, as well as the AMPC perturbation probability. Building on these experiments, we have added a unified sensitivity analysis of the eight loss-related hyperparameters highlighted in the training objective: the four peak objective weights lambdamalambda_{ma}, lambdatrlambda_{tr}, lambdapclambda_{pc}, and lambdakdlambda_{kd}; the three internal AMPC coefficients betaancbeta_{anc}, betaprotobeta_{proto}, and betaclsbeta_{cls}; and the consistency temperature taupctau_{pc}. The selected values are lambdama=0.12lambda_{ma}=0.12, lambdatr=0.005lambda_{tr}=0.005, lambdapc=0.03lambda_{pc}=0.03, lambdakd=0.25lambda_{kd}=0.25, betaanc=1.0beta_{anc}=1.0, betaproto=1.0beta_{proto}=1.0, betacls=0.5beta_{cls}=0.5, and taupc=2.0tau_{pc}=2.0. We will report the complete sensitivity curves in the revised appendix and summarize the selected values and schedules in the main implementation section.

Reviewer_sgYbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript proposes SEDAL, a framework for multimodal emotion recognition in conversations that integrates momentum-evolving multimodal emotion anchors, speaker-aware emotional transition prototypes, and deferred multi-teacher knowledge distillation. The method aims to improve multimodal alignment, emotional transition modeling, and robustness to modality perturbations. Experiments on IEMOCAP and MELD demonstrate competitive performance.

Critical strengths

The paper addresses a relevant problem in multimodal representation learning and presents a well-motivated framework for incorporating speaker-aware conversational dynamics. The method achieves strong results on IEMOCAP and MELD, outperforming the strongest reported baselines in weighted F1. The extensive baseline comparisons, ablation studies, and robustness evaluations provide empirical support for the proposed framework.

Critical weaknesses
  1. The framework combines several existing techniques, including prototype learning, multimodal alignment, speaker-aware context modeling, and knowledge distillation. Although VEGA (Hu et al., ACM MM 2025) is discussed and evaluated, the methodological differences between its visual prototype learning and SEDAL's dynamic multimodal anchors are not sufficiently demonstrated. A clearer explanation of the novel contributions would strengthen the paper.
  2. The transition prototype bank in Section 3.3 is indexed by the current emotion class and a binary same-/cross-speaker relation, rather than explicitly modeling previous-to-current emotion pairs. Although adjacent-turn features may implicitly capture emotional transitions, the experiments do not directly validate this capability. Separate quantitative evaluations of emotion persistence, emotion shifts, and same-/cross-speaker transitions would help substantiate the claimed effectiveness of the proposed transition modeling.
  3. Table 2 removes STPL-PCR as a whole, combining transition-prototype learning and perturbation consistency regularization. Consequently, the individual contributions of these mechanisms remain unclear. Although Appendix G compares different perturbation-position strategies, it does not isolate the effects of the two learning mechanisms. Separate ablations of transition-prototype learning and consistency regularization would help clarify their respective contributions.
  4. Implementation clarity and computational overhead. Appendix F mentions fixed CLIP-derived visual anchors, while MEMEA introduces momentum-evolving multimodal anchors, but their relationship is unclear. Additionally, the computational cost of training multiple teachers is not reported. Clarifying these aspects would improve reproducibility and efficiency assessment.
Questions for authors
  1. What is the key methodological novelty of SEDAL compared with VEGA, particularly regarding dynamic multimodal anchors versus visual prototype learning?
  2. How does the transition prototype bank explicitly capture emotional transitions, and what evidence supports its effectiveness for emotion persistence, shifts, and speaker changes?
  3. What are the individual contributions of transition-prototype learning and perturbation consistency regularization to the overall performance?
  4. How are fixed CLIP-derived visual anchors related to momentum-evolving multimodal anchors, and what is the computational overhead of training multiple teachers?
2 comments
AuthorsOpenReview ↗

Clearer methodological explanations,More systematic supporting evidence

Thank you for the constructive feedback and for recognizing the relevance of the problem and the breadth of our evaluation. These comments helped us identify where the original manuscript needed clearer methodological explanations and more systematic supporting evidence. We have revised the manuscript accordingly and provide detailed responses below.

W1&Q1: Methodological distinction from VEGA

Response: We appreciate this important comparison point and agree that the distinction from VEGA should be made more explicit. VEGA and SEDAL use prototypes at different levels. VEGA constructs fixed CLIP-derived visual emotion prototypes and uses them to ground visual-semantic alignment. SEDAL retains that fixed visual prior in the backbone for a controlled comparison, but MEMEA introduces an independent anchor bank learned from contextual text, audio, visual, and fused features. The bank evolves by EMA during training and provides the common geometry on which the transition and robustness modules operate. Thus, the methodological difference is not merely static versus dynamic updating. MEMEA is multimodal and task-adaptive; STPL maps adjacent anchor pairs into a transition space using the previous turn, direction, interaction, and speaker relation; AMPC regularizes only transition-relevant utterances; and DSA-MTBD refines the resulting boundary after a deferred start. Removing MEMEA while leaving the VEGA-style fixed visual anchors unchanged reduces W-F1 by 1.74 points on IEMOCAP and 1.45 points on MELD.

W2&Q2: How the transition bank captures persistence shifts and speaker changes

Response: STPL uses a relation-conditioned endpoint bank, but its input is an adjacent-turn transition representation. The representation concatenates the current and previous speaker-augmented anchors, their signed difference, and their element-wise interaction. The current target defines the emotion reached at time t, while the descriptor records how that endpoint was reached. Separate banks for same- and cross-speaker relations prevent self-persistence or self-escalation from being collapsed with a response to another speaker.

We have added a quantitative subgroup analysis. Persistence is defined by yty_t = yt−1y_{t-1}, shift by yty_t != yt−1y_{t-1}, and speaker relation by equality of the adjacent local speaker IDs. The table reports the full model and the variant without transition-prototype learning under the same seeds. The complete model improves all four groups, with the largest gain of 3.15 on Cross-speaker shift transitions. This directly tests the claimed behavior instead of inferring it only from aggregate W-F1.

Transition subgroupCountSEDAL W-F1w/o STPL W-F1Gain
Same-speaker persistence39278.6476.91+1.73
Same-speaker shift22870.8868.31+2.57
Cross-speaker persistence25176.4274.96+1.46
Cross-speaker shift16969.7366.58+3.15

The qualitative and error analyses in the manuscript provide complementary evidence. The cross-speaker and same-speaker case studies show corrections at abrupt affective changes, and the confusion matrices show lower transition-sensitive errors, including Excited to Happy at 12.04 percent, Angry to Frustrated at 24.71 percent, and Frustrated to Neutral at 11.29 percent on IEMOCAP. On MELD, Surprise to Neutral falls to 10.32 percent, compared with 12.46 percent for CMERC and 14.59 percent for ECERC. We will connect these analyses explicitly to the subgroup table.

AuthorsOpenReview ↗

Clearer methodological explanations,More systematic supporting evidence

W3&Q3: Separate contributions of transition learning and perturbation consistency

Response: We agree that removing STPL-PCR as one block confounds transition-prototype learning with AMPC. We have added two single-mechanism controlled experiments: without STPL removes the transition-prototype loss while retaining AMPC, and without AMPC retains the transition prototypes but removes perturbation consistency. The existing without STPL-PCR row removes both. The experiment results show that both mechanisms contribute. The full model is stronger than either single removal, and removing both produces the largest drop, from 75.09 to 72.69 W-F1 on IEMOCAP and from 67.98 to 65.47 on MELD. The robustness analysis further isolates AMPC: across unimodal, bimodal, and explicit dropout settings, it improves W-F1 by 0.59 to 2.32 points on IEMOCAP and 0.86 to 1.85 points on MELD. The perturbation-position control also reaches 75.09 W-F1 for transition-guided selection, compared with 74.36 for all-valid random selection, 73.81 for equal-count random selection, and 74.08 without AMPC on IEMOCAP.

ConfigurationTransition lossAMPCIEMOCAP W-F1MELD W-F1
SEDALYesYes75.0967.98
w/o STPLNoYes73.8866.74
w/o AMPCYesNo74.0866.93
w/o STPL-PCRNoNo72.6965.47

W4&Q4: Anchor relationship and computational overhead

Response: The fixed CLIP anchors and the MEMEA anchors are two independent prototype banks with different sources and roles. The CLIP anchors are constructed once using the frozen CLIP ViT-B/32 encoder and serve as external, visually grounded semantic priors. They remain fixed throughout training and are not updated by gradients or by the SEDAL objectives. In contrast, MEMEA learns dataset-specific multimodal emotion anchors from the contextualized textual, acoustic, visual, and fused representations. After projecting these representations into a shared anchor space, MEMEA performs gated multimodal aggregation and updates each class anchor using the exponential moving average of its corresponding batch-level class center. Consequently, MEMEA anchors evolve with the task data and capture dataset-specific multimodal emotion distributions.

The two banks do not share parameters, prototype vectors, or EMA updates, and the fixed CLIP anchors do not participate in constructing or updating the MEMEA bank. Their functions are complementary: CLIP provides a stable pretrained visual-semantic prior, whereas MEMEA establishes a task-adaptive multimodal geometry that subsequently supports speaker-aware transition learning and perturbation consistency. Therefore, the additional cost of MEMEA comes from its lightweight projection, gating, and class-anchor updates rather than from fine-tuning CLIP or maintaining interactions between the two anchor banks.

The IEMOCAP and MELD teacher committees contain 15 and 8 selected members. Teachers are trained or collected offline, and their training-set probability distributions are cached once. Student training reads the cache, so the committee is not executed at every student update. The final system uses a single student, and the teacher committee, transition bank updates, and perturbed branch are absent from deployment inference.

Using the released implementation settings, the full training model has 69.39 million parameters on IEMOCAP versus 64.08 million for the backbone, an increase of 8.29 percent. On MELD, it has 73.27 million versus 68.21 million, an increase of 7.42 percent. The additional projection heads are training-time auxiliaries and can be stripped from the deployed prediction path. We will report training time, cache construction, peak memory, and single-student inference latency as 3.5 GPU-hours, 0.7 GPU-hours, 15.9 GB, and 28.1ms per dialogue, respectively.

Reviewer_4K4fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new method (SEDAL) for multi-modal dialogue emotion recognition, which combines category-level multi-modal emotion anchors, speaker-aware adjacent-round emotion change modelling, modality perturbation consistency, and multi-teacher distillation. Specifically, in the proposed method, MEMEA uses momentum updates to maintain emotion category prototypes and maps text, audio, visual, and fused representations to a unified anchor space. STPL-PCR builds transition representations based on adjacent utterances, distinguishes same-speaker and cross-speaker situations, and applies consistency constraints to relevant samples using AMPC under modality perturbation. And DSA-MTBD conducts distillation for the single student in the case of multiple teachers selected from the validation set. Experimental results also validate that the proposed method has achieved better performance on IEMOCAP and MELD benchmarks compared with existing methods.

Critical strengths
  1. In the proposed framework, each module is relatively coherent, and the model design architecture is soundness. Although the technologies of each module are not completely innovative, integrating class-level semantic anchor, adjacent rounds of speaker-aware context, modality perturbation consistency, and delayed teacher distillation into the same training framework makes the overall design logically clear and complete. In particular, the staged training design, where anchor geometry is established first and then transitions and distillation objectives are added, is reasonable.

  2. The experiment is comprehensive. This paper conducted tests on two standard MERC benchmarks and also compares the proposed method with many recent methods. Many experiments effectively validate the performance of the proposed method.

Critical weaknesses
  1. The part "emotion transition prototype" does not explicitly model the emotion-to-emotion transition. In this paper, the transition prototype bank is A∈R2×C×dtrA \in \mathbb{R}^{2\times C\times d_{tr}}, where aρ,ca_{\rho,c} is determined only by the current emotion class and the same-/cross-speaker relationship. Therefore, as long as the current category is the same and the speaker relationship is the same, different pre-existing emotions jj will share the same prototype. For example, Neutral→Joy and Anger→Joy will be supervised by the same cross-speaker Joy prototype. Although the transition representation itself contains two adjacent anchors, the prototype objective does not actually distinguish different transition types. Therefore, the actual parameterisation of the method has a certain degree of mismatch with the core claim of "explicitly modelling speaker-dependent emotional transition dynamics".

  2. The innovation and experimental validation are still insufficient compared to the highly related prototype-based MERC works. SEDAL combines existing technologies such as EMA class prototype, contextual transition feature, perturbation consistency, and multi-teacher distillation, but does not explain its core methodological insight. Additionally, the paper missed directly related recent prototype-based MERC works, such as "Graph-Prototype Distillation with Prototype-Guided Contrastive Training"(2026), which has already conducted joint research on MERC prototypes and distillation. Without these comparisons and more targeted control experiments, it is difficult to determine whether the performance improvement comes from the new transition-learning principle or the combination of multiple auxiliary objectives.

Questions for authors
  1. The current transition prototype is determined solely by the current emotion class cc and speaker relation ρ\rho, without considering the previous emotion class. Why should \ (a _ {\ rho, c} ) be regarded as an emotion transition prototype, rather than a current emotion class prototype that only combines the speaker relation? Have the authors attempted to explicitly verify the design of aρ,ct−1,cta_{\rho,c_{t-1},c_t}?

  2. Could the following ablation experiments be added: (i) remove speaker-relation conditioning; (ii) remove the previous anchor rt−1r_{t-1}; (iii) remove the difference term; (iv) remove element-wise interaction? These experiments may more directly illustrate which part of STPL is beneficial for transition modelling.

  3. How many teachers does the teacher committee of DSA-MTBD ultimately consist of? How many candidates are there? What specific changes does the "objective-level inductive bias" adopt? Additionally, suggest introducing the complete teacher aggregation and distillation hyperparameters.

  4. In the implementation part, CLIP ViT-B/32 is used to construct the fixed visual emotion anchors, what is the relation between it and the multimodal class anchors pcp_c?

  5. Has the author compared with recent prototype-centric MERC methods? For example, Graph-Prototype Distillation with Prototype-Guided Contrastive Training. This might affect the assessment of this work, such as the novelty and empirical contribution.

3 comments
AuthorsOpenReview ↗

Clearer methodological explanations,More ablation,Related work discussion

Thank you for the thoughtful and technically insightful feedback. These comments helped us identify where the original manuscript needed clearer methodological explanations, more ablation, and a more precise discussion of related work. We have revised the manuscript accordingly and provided detailed responses below.

W1&Q1: Transition representation and prototype indexing

Response: We agree that the bank index aρ,ca_{\rho,c} does not itself contain the previous emotion label. The transition is encoded in the sample representation dtd_t, not solely in the bank index. For every adjacent pair, dtd_t is computed from the current and previous speaker-augmented anchors, their directed difference, and their element-wise interaction. Consequently, Neutral to Joy and Anger to Joy under the same cross-speaker relation have the same endpoint label but produce different transition representations before prototype matching. The loss therefore learns a relation-conditioned distribution of trajectories that terminate in class cc, rather than an ordinary current-emotion prototype formed from the current utterance alone. In addition, we use current-emotion indexing since a full bank a(ρ,cprev,c)a_(\rho,c_{prev},c) contains 2C22C^2 prototypes and may become sparse for many relation-emotion pairs, particularly on IEMOCAP. When expanding from 2C2C to 2C22C^2, the number of prototypes increases by a factor of CC, for example, this rises from 12 to 72 in IEMOCAP. However, training samples are divided into finer-grained units, and certain emotion transitions are inherently rare. For instance, some minority-class transitions across speakers may contain only a few samples or even no samples at all. Therefore, the current emotion index way has higher data density and more stable prototype estimation than the Ordered pair index which is more fine-grained yet prone to insufficient samples, inadequate prototype updates and overfitting. More importantly, this does not mean that our model ignores the previous emotion. The previous utterance ut−1u_{t-1} and its emotion-related information are not discarded, because its speaker-augmented anchor rt−1sr_{t-1}^s and its directed interaction with the current anchor remain encoded in the transition representation: dtd_t=ftr(qt)f_{tr}(q_t)/∥ftr(qt)∥2\lVert f_{tr}(q_t)\rVert_2, where qtq_t=[rtsr_t^s; rt−1sr_{t-1}^s; rts−rt−1sr_t^s-r_{t-1}^s; rts⊙ rt−1sr_t^s \odot\ r_{t-1}^s]. Therefore, although the two transitions match the same target prototype, their input representations dtd_t differ. More precisely, this is "the current emotional target prototype conditioned on speaker relationships". To show the rationality of our current design, we have also added a controlled comparison between current-target and pair-target indexing. The pair-target variant obtains 74.32 Acc and 74.41 W-F1 on IEMOCAP and 67.76 Acc and 67.55 W-F1 on MELD, compared with 74.98 Acc and 75.09 W-F1 on IEMOCAP and 68.74 Acc and 67.98 W-F1 on MELD for the default, with emotions incorporated into the prototypes.

Transition targetBank sizeIEMOCAP AccIEMOCAP W-F1MELD AccMELD W-F1
Current emotion2C74.9875.0968.7467.98
Ordered pair2C^274.3274.4167.7667.55

Finally, we also agree that the phrase explicitly modeling emotion-to-emotion transitions can be read as claiming an explicit ordered-pair index. We will revise the description to state that STPL explicitly encodes adjacent-turn transitions and learns speaker-relation-conditioned endpoint prototypes. This wording precisely separates the transition representation from the prototype indexing scheme.

AuthorsOpenReview ↗

Clearer methodological explanations,More ablation,Related work discussion

W2&Q5: Novel contribution and the missing prototype-centric comparison

Response: We agree that comparison with the recent prototype-based method GPGC helps clarify SEDAL’s methodological contribution. Both methods employ class prototypes, but differ in how prototype learning interacts with conversational structure and the training objectives. GPGC learns a sparse intra-dialogue graph and applies representation alignment, distributional consistency, and prototype alignment to graph-aggregated features. Its prototype-guided momentum contrast uses cross-batch queues and confidence weighting to encourage intra-class compactness and inter-class separation. Its backbone also incorporates speaker embeddings.

SEDAL explicitly incorporates adjacent-turn transitions and speaker relations into prototype learning. MEMEA constructs momentum-evolving class anchors from the textual, acoustic, visual, and fused branches, providing a shared semantic reference across modalities. STPL-PCR constructs transition representations from the previous and current speaker-augmented anchors, their signed difference, and their element-wise interaction. These representations are matched to endpoint prototypes indexed jointly by the current emotion and the same-speaker or cross-speaker relation. Thus, previous-turn information enters through the transition representation, while prototype assignment explicitly conditions on the speaker relation and endpoint emotion. SEDAL also couples this anchor structure with targeted modality perturbation and selective boundary distillation. AMPC perturbs modalities at transition-relevant utterances and enforces agreement between clean and perturbed views in the anchor-feature, prototype-logit, and classifier-logit spaces. This defines an explicit consistency objective under modality corruption, distinct from weighting contrastive gradients by prediction confidence. After the student anchor geometry has formed, DSA-MTBD applies multi-teacher distillation to uncertain decision boundaries, retaining only teacher targets that are correct and confident during training. In addition, the methodological comparison with GPGC is complemented by analyses of SEDAL’s components and the evaluated baselines. Removing MEMEA reduces W-F1 by 1.74 and 1.45 points on IEMOCAP and MELD, respectively, while removing STPL-PCR produces the largest decreases of 2.40 and 2.51 points. These ablations support the contributions of multimodal anchor learning and adjacent-turn transition representations.

The t-SNE visualization provides a complementary qualitative view. Compared with the graph-based and multimodal-fusion baselines evaluated under the same protocol, SEDAL shows less fragmented class clusters and more locally coherent emotion neighborhoods in the projection, particularly for sad, neutral, and excited utterances. These patterns visualize the representations learned through multimodal anchors, speaker-relation-conditioned endpoint prototypes, transition-guided perturbation consistency, and deferred boundary distillation.

The robustness experiments further assess recognition performance under modality corruption. Transition-guided AMPC improves performance across the evaluated unimodal, bimodal, and dropout settings, including W-F1 gains of 2.32 points on IEMOCAP and 1.85 points on MELD under tri-modal dropout. In the perturbation-strategy comparison, transition-guided perturbation achieves 75.09 W-F1, compared with 74.36 for all-valid random perturbation, 73.81 for equal-count random perturbation, and 74.08 without AMPC. These results support targeting perturbations to transition-relevant utterances.

The confusion-matrix comparison also shows improved recall for all six IEMOCAP emotions. The misclassification rates are 11.29% for Frustrated utterances predicted as Neutral, 24.71% for Angry utterances predicted as Frustrated, and 12.04% for Excited utterances predicted as Happy. These results characterize classification errors between emotion categories, complementing the ablation evidence for transition learning and the robustness evidence for targeted perturbation.

We sincerely thank the reviewer for identifying this relevant work. We will cite it and incorporate this comparison into the revised manuscript to better articulate SEDAL’s distinctive contributions and its broader capabilities in modeling conversational emotion dynamics and modality robustness.

AuthorsOpenReview ↗

Clearer methodological explanations,More ablation,Related work discussion

Q2: Fine-grained ablations of STPL

Response: We agree that the original whole-module ablation was not sufficient to identify which part of STPL carries the transition signal. We have added the requested ablations while keeping the backbone, optimization schedule, and all other losses unchanged. Removing the previous anchor tests whether the module reduces to current-turn recognition. Removing the difference term tests directed change, and removing the element-wise product tests nonlinear co-occurrence. Removing relation conditioning collapses same- and cross-speaker dynamics into one bank. Across the controlled runs, the complete descriptor gives the strongest result. The revised manuscript will report the values below and include the exact configuration of every variant. We will also avoid interpreting small differences causally when their run-level intervals overlap.

STPL configurationIEMOCAP AccIEMOCAP W-F1MELD AccMELD W-F1
full STPL74.9875.0968.7467.98
w/o speaker relation73.2673.3267.0266.54
w/o previous anchor73.0773.1466.8166.12
w/o directed difference74.0674.2567.3167.08
w/o element-wise interaction74.1974.3067.5567.24

Q3: Teacher committee size selection and objective-level diversity

Response: The final IEMOCAP committee contains 15 teachers and the MELD committee contains 8 teachers, selected from candidate pools of 36 and 24 models, respectively. Candidates vary along three controlled axes: checkpoints from different training trajectories, random initialization, and auxiliary-objective configuration. The last axis includes the common backbone, full SEDAL training, transition-only, AMPC-only, and selected loss-weight variants. Thus, it changes the supervision geometry seen by a teacher without changing the task or label space.

Committee construction uses only validation predictions. A candidate must pass a diversity gate based on its mean Jensen-Shannon divergence from admitted members, after which its weight is calibrated by validation performance. Membership and weights are then frozen. For the student, teacher probabilities on the training set are precomputed and cached. Distillation retains a target only when the teacher's top class is correct, and its confidence exceeds the threshold; then weights it by confidence and current student disagreement. The IEMOCAP setting uses a confidence threshold of 0.85, maximum distillation weight 0.25, start epoch 35, and an 8-epoch warm-up; the corresponding MELD settings will be listed in the revision.

The teacher committee is used only during training. The student learns from cached teacher probabilities, avoiding teacher forward passes at each iteration. Inference uses a single student model. We will expand the implementation table to report the candidate-pool size, the one-time teacher-training cost of 3.5 GPU-hours, the cache-generation cost of 0.7 GPU-hours, the student-training overhead of 16.8%, and the inference latency of 28.1 ms per dialogue.

Q4: Fixed CLIP visual anchors and momentum-evolving multimodal anchors

Response: The two anchors operate at different stages and are not tied. The fixed CLIP ViT-B/32 anchors are inherited from the visual grounding backbone. They are computed once from visual emotion prompts, remain frozen, and provide an external semantic reference for the visual branch. In contrast, MEMEA maintains a separate task-specific bank pcp_c. Each pcp_c is initialized from the current mini-batch class center of the gated text-audio-visual anchor feature and is updated online by EMA. It supervises the text, audio, visual, and fused branches, not only the visual branch. Thus, MEMEA is neither initialized from the CLIP anchors nor a moving copy of them. The fixed anchor supplies a stable visual prior, whereas pcp_c adapts to the empirical multimodal distribution of each dataset. Removing MEMEA while retaining the fixed CLIP component lowers W-F1 from 75.09 to 73.35 on IEMOCAP and from 67.98 to 66.53 on MELD, which isolates the value of the learned multimodal anchor bank. We will add this relationship explicitly to the method and implementation sections.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.