acceptodds
Under review as a conference paper at ICLR 2027

Finding Emotions Where They Belong: Rethinking Audio Emotion Recognition through Masked Temporal Affective Grounding

Abstract

Audio emotion recognition (AER) typically assigns a single label to an entire recording, leaving the temporal scope of that label ambiguous when multiple speakers and affective events are present. We address this limitation by reformulating AER as a Temporal Affective Grounding (TAG) task that associates emotions with temporally bounded speech spans and vocal tone descriptions. To support this formulation, we curate temporally annotated versions of existing emotion recognition datasets and construct recordings containing two to four affective speech spans, including overlapping speech. Training in this longer format with a standard language-modeling objective can degrade both emotion recognition and temporal grounding performance, while tone descriptions can provide shortcuts for emotion prediction. To address these challenges, we introduce Masked Temporal Affective Grounding (M-TAG), a supervised training objective that combines full-sequence language modeling with emotion and timestamp cross-entropy losses under attention masking. The masking varies the context visible to emotion-label tokens to reduce reliance on shortcuts and improve generalization, while the timestamp loss incorporates a distance-aware weight to penalize larger temporal errors. We evaluate EMO-TAG, a model fine-tuned using our dataset and objective, on emotion recognition and affective temporal-grounding against three AER and audio-language baselines: Flamingo-Next, Audio-Reasoner, and AffectGPT. Our results show that existing models achieve limited affective temporal-grounding despite competitive emotion recognition performance. Across MELD and IEMOCAP, our model improves over our strongest baseline by an average of 10 percentage points in emotion accuracy on both the original and our refined test sets and by over 20 percentage points in Emotion-Duration F1 on our affective temporal-grounding benchmark. These improvements generalize to the MME-Emotion benchmark, where EMO-TAG achieves an average gain of five percentage points relative to the best baseline. Training and dataset-creation code and model checkpoints will be released upon publication.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio-Language Models, Speech Emotion Recognition, Temporal Grounding, Multi-Task Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_huMkOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

authors proposed temporal affective grounding to leveraging the emotion recognition performance from multiple speech spans. Authors also curated a dataset to include speech spans, tone descriptions for MEDL and IEMOCAP datasets.

Critical strengths

Quantifying the emotion by considering temporal aspects of an audio signal is an interesting idea.

Authors curated dataset with time stamps, tone description and emotion label. This can add value database generation considering the currently available AI tools.

Authors evaluated their model on benchmark dataset (IEMOCAP and MELD) for emotion recognition task.

Critical weaknesses
  1. The two stage proposed methodology is not convincing. Tone description is intuitively similar to acoustic features.
  2. Using AI models to generate tone description is easy thing todo. However, how to validate the tone descriptions which is a key? this is missing in the paper. 3.Authors are not comparing their proposed methodology with state of the art methods instead authors chose to compare their model with zero-short learning models.
Questions for authors

Quantifying the emotion by considering temporal aspects of an audio signal is an interesting idea.

Authors created a dataset using 2-4 spans with overlapping speech. how authors decide the overlapping window size varying between 0.2 to 2 seconds. IEMOCAP dataset contains very small segments. How randomly choosing span size between 0.2 to 2 seconds works for segments that are less than 2 seconds long. Authors should report length of audio samples in IEMOCAP and EMLD datasets. Since author are curating the dataset to create temporal affective learning, authors should design and follow a protocol to create the dataset.

What is SFT in Figure 1. Authors should provide full description of figure 1.

IEMOCA and MELD datasets come up with text, audio, and video modalities. Above all authors have freedom to compute audio features like pitch, frequency, tone, energy etc from the audio signal. It looks like tone descriptions are kind of imitating the audio level features pitch, energy, frequency, or MFCC features.

Data curation: authors mentioned that “transcript contains two or more consecutive insertions (extra words that are not part of the utterance) or three or more replacements (different words that may have been misrecognized by WhisperX), we reject the sample”. How many samples were rejected for MELD and IEMOCAP dataset?

MELD dataset provides training, validation and test dataset for model evaluations. But IEMOCAP does not provide predefined splits. It is not clear how authors considered the splits for model evaluations? Is it 5-fold session wise cross validation? table 1 original/refined accuracies not much difference, there is no difference between emotion segment accuracy vs hit accuracy. This indicates that multiple spans are not adding much to the overall model accuracy.

Authors curated datasets and generated tone descriptions using Ai tools but masking it during training is not convincing. In that case why to generate tone descriptions?

From Table 1: authors compare EMO-TAG with Audio reasoner, AffectGPT, FlamingoNext these are zero-shot models. Authors must compare their model with state of the art for fair comparison. So the results are not comparable.

AI-generated tone descriptions are not convincing because the entire pipeline depends on FlamingoNext (generation) → ChatGPT (refinement) → Qwen3 (extraction/checking). Since authors claimed that they curated the dataset, it would be helpful to validate the tone descriptions and report annotator agreement scores. There are no baseline comparisons with SOTA on IEMOCAP and MELD. Authors compared against general-purpose audio-language models (FlamingoNext, Audio-Reasoner) evaluated zero-shot — not against models actually designed and trained for emotion recognition on these datasets. For example, EmoBERTa, DialogGCN, LineConGCN, MMGCN. Without comparing it with state of the art, the improvements that authors included in the paper is not convincing.

For emotion recognition, confusion matrices show which emotion pairs are getting confused and how the proposed model addresses them. In particular, MELD and IMECAP datasets are imbalanced. Authors should include confusion matrices to understand model accuracy in recognizing similar emotions like happy vs excited, sad vs angry.

It is important to include ablation study. For example, include ablation for different time spans and authors should include sensitivity analysis on time stamps

12 comments

We thank the reviewer for their constructive feedback and appreciate the opportunity to further clarify our contributions and provide additional evidence addressing their concerns. Below, we address each of the reviewer's comments and outline our proposed revisions.

The two stage proposed methodology is not convincing.

Author Response:

It is unclear whether the reviewer is referring to our tone-description generation process or our two-stage training methodology, so we address both.

Regarding the tone-description process, we use the initial descriptions generated by FlamingoNext as supporting evidence for context and changes in tone that are difficult to identify from the low-level acoustic features of individual segments alone.

Regarding the training methodology, our two-stage approach aims to address a fundamental problem: introducing extra textual reasoning outputs (such as speech transcripts and tone descriptions) can negatively affect emotion recognition performance compared to predicting the emotion label alone. This issue has received limited attention in previous works. Zhao et al. (2025) show that prompting a model to produce Chain-of-Thought reasoning without task-specific training can reduce speech emotion recognition accuracy, while Xu et al. (2025) show that extended multimodal reasoning can weaken visual grounding and increase hallucinations. We further demonstrate this effect in our ablation study in Figure 2 and Table 3 of the appendix, where temporal emotion-only prediction outperforms joint prediction of speech, tone, and emotion before introducing our two-stage training approach. Our approach addresses this challenge through context masking, guiding the model to better utilize emotion-relevant training signals and preventing them from being overshadowed by the additional textual tokens. The newly added Figure 3 in the appendix (starting at line 810), which compares the attention mass when predicting emotion labels under our approach and vanilla SFT, further supports this explanation by showing that masking increases the attention assigned to audio tokens.

Authors curated datasets and generated tone descriptions using Ai tools but masking it during training is not convincing. In that case why to generate tone descriptions?

Author Response:

We would like to clarify that we do not mask the tone descriptions in the language modeling pass. Our method uses two forward passes. The first is the language modeling pass, vanilla SFT, which gives us the language modeling loss. Nothing is masked when calculating this loss. The second pass is used to calculate the emotion and timestamp losses, where we mask either the tone descriptions or both the tone descriptions and speech transcripts when predicting the emotion labels and timestamps. The tone descriptions, speech transcripts, and output format are still learned through the standard language modeling loss. The masking provides additional supervision for the emotion labels and timestamps, preventing these important training signals from being overshadowed by the other generated tokens.

References:

Tone description is intuitively similar to acoustic features. IEMOCA and MELD datasets come up with text, audio, and video modalities. Above all authors have freedom to compute audio features like pitch, frequency, tone, energy etc from the audio signal. It looks like tone descriptions are kind of imitating the audio level features pitch, energy, frequency, or MFCC features.

Author Response:

We agree that tone descriptions are closely related to acoustic features. In fact, as described in Section 3.2 (starting at Line 180) and further explained in Section A.3 (starting at Line 743), we explicitly use these features as a reference to guide and ground the generation of tone descriptions. We see this as a strength rather than a weakness, as it helps ground the descriptions in measurable acoustic cues. Describing tone is more complex than simply extracting acoustic features. Some people naturally have higher-pitched voices, while others have lower-pitched voices. Similarly, speaking rate and loudness vary across individuals, genders, and cultures. For example, a pitch value of 130 Hz alone does not tell us whether a speaker is angry or neutral, as it needs to be interpreted relative to their normal speaking patterns. Our goal was to make the tone descriptions easy to understand while preserving as much relevant acoustic information as possible. We acknowledge that these descriptions could be improved, but developing a more accurate and comprehensive representation of tone would require substantial effort, particularly to account for individual and cultural differences in how emotions are expressed through speech.

Using AI models to generate tone description is easy thing todo. However, how to validate the tone descriptions which is a key? this is missing in the paper.

AI-generated tone descriptions are not convincing because the entire pipeline depends on FlamingoNext (generation) → ChatGPT (refinement) → Qwen3 (extraction/checking). Since authors claimed that they curated the dataset, it would be helpful to validate the tone descriptions and report annotator agreement scores.

Author Response:

We agree that generating tone descriptions using AI is relatively straightforward, but ensuring their quality is more challenging. This is why we did not simply prompt a model to describe the audio. Instead, we followed a multi-stage process. First, we generated initial annotations using an audio model. We then used a separate text-only model to compare these annotations against the ground-truth emotion labels, rejecting samples where the descriptions were inconsistent with the labels. Such inconsistencies could result from incorrect model-generated descriptions or potentially mislabeled samples. Finally, we used low-level acoustic features and speaker-specific acoustic references to guide GPT-Astra in producing more concrete and acoustically grounded tone descriptions. We acknowledge that there is no guarantee that every generated tone description is accurate, as we could not manually verify every sample due to limited human resources. Nevertheless, the usefulness of the resulting descriptions is supported by the improved downstream emotion recognition performance observed in our experiments. We emphasize that our main contributions are the temporal formulation of audio emotion recognition and the proposed training approach, rather than tone description generation itself.

Authors are not comparing their proposed methodology with state of the art methods instead authors chose to compare their model with zero-short learning models.

There are no baseline comparisons with SOTA on IEMOCAP and MELD. Authors compared against general-purpose audio-language models (FlamingoNext, Audio-Reasoner) evaluated zero-shot — not against models actually designed and trained for emotion recognition on these datasets. For example, EmoBERTa, DialogGCN, LineConGCN, MMGCN. Without comparing it with state of the art, the improvements that authors included in the paper is not convincing.

Author Response:

We thank the reviewer for raising this important point regarding baseline selection. Our task differs from conventional emotion recognition because it requires both emotion classification and temporal localization of emotion segments. To our knowledge, none of the available emotion recognition models supported this task of temporally grounded emotion recognition. We therefore compared against general temporal audio understanding models with strong emotion recognition capabilities, such as Audio-Reasoner, which demonstrated competitive results on MME-Benchmark, and the more recent Audio-Flamingo-Next. We also included specialized emotion recognition models with publicly available checkpoints, such as AffectGPT (ICML 2025 oral), which uses the same Qwen2-Audio as its backbone. Regarding the suggested baselines, EmoBERTa, DialogueGCN, LineConGCN, and MMGCN focus on utterance-level emotion recognition in conversations, often using textual context and, in some cases, multiple modalities. While these are relevant baselines for conventional emotion classification, they do not address our proposed temporal affective grounding task. We acknowledge that comparisons with these methods would provide additional context for our classification results. However, their reported results are not directly comparable due to differences in input modalities and evaluation methods. To better address the concern about comparisons with state-of-the-art emotion recognition models, we conducted an additional comparison with the recent EmotionThinker model (ICLR 2026 oral). For a fairer comparison, we also trained our model using Qwen2.5-Omni, the same backbone used by EmotionThinker. We report First-Emotion Accuracy / Emotion Hit Accuracy / UAR (Unweighted Average Recall). As shown in the new results, our model remains competitive on MELD despite using considerably less training data. We train on approximately 12k unique samples from MELD, IEMOCAP, and EmoV-DB, while EmotionThinker uses approximately 35k samples from additional datasets. For IEMOCAP, a direct comparison is difficult because EmotionThinker does not distinguish frustration as a separate emotion and instead maps it to sadness. We believe frustration should be treated separately, making the two evaluation settings inconsistent. Regarding class-wise performance, our model achieves higher UAR than EmotionThinker across all evaluated subsets except MME-Emotion Noise. This suggests that our model is better at recognizing different emotions, with less bias toward frequent classes.

DatasetQwen2-Audio-7B (Emo-TAG)Qwen2.5-Omni-7B (Emo-TAG)EmotionThinker
MELD-test-raw54.71 / 55.59 / 36.0754.18 / 57.85 / 40.7558.81 / 58.81 / 32.19
MELD-test-cleaned57.95 / 58.10 / 37.3657.90 / 61.38 / 42.7160.46 / 60.46 / 33.92
MELD-noisy41.91 / 50.37 / 33.5545.22 / 53.31 / 37.0148.53 / 48.53 / 27.25
IEMOCAP-test-raw67.33 / 68.12 / 57.6166.67 / 68.79 / 55.8750.15 / 50.22 / 40.17
IEMOCAP-cleaned65.83 / 66.03 / 55.2267.14 / 68.75 / 61.1655.24 / 55.24 / 43.98
IEMOCAP-noisy65.73 / 65.73 / 50.5358.98 / 64.91 / 49.9357.99 / 57.99 / 42.69
MME-Emotion Noise62.56 / 63.00 / 61.8663.00 / 68.83 / 68.6472.42 / 72.42 / 72.96
MME-Emotion Wild35.78 / 37.87 / 31.0936.76 / 42.28 / 35.7637.19 / 37.19 / 29.24
MME-Emotion Lab55.33 / 56.34 / 50.3655.53 / 56.94 / 47.8446.28 / 46.28 / 37.14

It is worth noting that MME-Emotion Noise has mainly non-english samples, which explains the lower performance of our model, since all our training data is English. We thank the reviewer again for raising this point and would be happy to consider any additional related baselines they recommend.

Reviewer_6HDTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses the problem associated with assigning a single label to an utterance for audio emotion recognition (AER), which fails to capture emotional changes over time or locate specific expressive speech segments. The authors introduce Temporal Affective Grounding (TAG), in which speech spans are detected along with their precise temporal boundaries, transcripts, vocal-tone descriptions, and emotion labels. They supported this framework by building temporally annotated datasets derived from sources like MELD and IEMOCAP, featuring multi-span and overlapping speech scenarios. Further, they developed Masked Temporal Affective Grounding (M-TAG), which uses attention masking to prevent reliance on text context for emotion prediction. M-TAg also incorporates a distance-aware loss to penalize temporal timing errors. They achieve significant performance gains in both emotion recognition and temporal grounding on MELD, IEMOCAP, and MME-Emotion over strong baselines like Qwen2-Audio-Instruct and AffectGPT.

Critical strengths
  1. The problem is new and useful. Localization of the identified emotion is beneficial for evaluating audio-language models, especially in conversational settings.
  2. Instead of distributing the loss over all generated tokens, the method ensures that emotion labels and temporal boundaries receive less weight than transcripts and descriptive texts. In order to avoid the model attending to textual descriptions more than audio, the paper employs masked emotion supervision and distance-aware temporal supervision.
  3. The improvement in performance over the baselines is substantial.
Critical weaknesses
  1. Concatenating utterances may not be the same as modelling conversational dynamics. Real conversations contain interruptions, changes in emotional intensity, and context-dependent expressions. Artificial concatenation may create discontinuities that make boundaries easier to identify than they would be in continuous speech. Conversely, it may introduce atypical acoustic transitions that complicate evaluation in ways unrelated to emotion recognition.
  2. The importance of a time-stamp prediction depends on its temporal error. It also has a weighting mechanism that is applied to the cross-entropy loss for target time stamp token. This means that the time-stamp error estimation strategy and the weights are critical for this to be effective. Would there be a way to check for robustness with respect to these parameters, notwithstanding the ablation in Table 3?
Questions for authors
  1. Is it possible to demonstrate directly that masking increases the causal contribution of acoustic information to emotion predictions. This will ensure that it is not merely changing the distribution of training gradients?
  2. It is well known that the emotions are imbalanced and this naturally leads to a question related to imbalanced situations. Does the model improve grounding for underrepresented classes such as fear and disgust, or are the gains primarily driven by more frequent categories?
4 comments

We thank the reviewer for their constructive feedback and for recognizing the novelty and usefulness of our approach, as well as the improvements over existing baselines. Below, we address each of the reviewer's comments, questions, and concerns.

Concatenating utterances may not be the same as modelling conversational dynamics. Real conversations contain interruptions, changes in emotional intensity, and context-dependent expressions. Artificial concatenation may create discontinuities that make boundaries easier to identify than they would be in continuous speech. Conversely, it may introduce atypical acoustic transitions that complicate evaluation in ways unrelated to emotion recognition.

Author Response:

We thank the reviewer for raising this important point. We acknowledge that artificial concatenation does not fully capture the complexity of real conversations, particularly interruptions and context-dependent emotional expressions. To reduce artificial discontinuities introduced by concatenations, we sample silence intervals from the original datasets rather than inserting fixed silence gaps, as explained in Appendix A.4 (starting at Line 770). This helps make the transitions between utterances more natural. Our goal is not to model conversational dynamics, but to introduce a controlled setting for training and evaluating temporal emotion recognition, particularly the ability to recognize and localize multiple emotional speech segments within the same recording. Modeling the full complexity of natural conversational dynamics is beyond the scope of this work.


The importance of a time-stamp prediction depends on its temporal error. It also has a weighting mechanism that is applied to the cross-entropy loss for target time stamp token. This means that the time-stamp error estimation strategy and the weights are critical for this to be effective. Would there be a way to check for robustness with respect to these parameters, notwithstanding the ablation in Table 3?

Author Response:

We thank the reviewer for this insightful question. As explained in the appendix, the auxiliary loss weights are gradually increased during the first three training epochs, and the warm-up procedure is repeated when introducing multi-span samples to reduce training instability. Beyond this warm-up stage, the timestamp loss is weighted according to the temporal error between the predicted and ground-truth timestamps, allowing the model to place greater emphasis on timestamps with larger prediction errors. We agree that a more extensive sensitivity analysis could provide additional insights. However, since the weighting mechanism is primarily introduced to improve training stability rather than being a central contribution of our work, we focused our ablation studies on the main components of the proposed approach, as reported in Table 3.

Is it possible to demonstrate directly that masking increases the causal contribution of acoustic information to emotion predictions. This will ensure that it is not merely changing the distribution of training gradients?

Authors Response:

We thank the reviewer for this suggestion. To better investigate whether the masking strategy increases the model's reliance on acoustic information, rather than merely altering the distribution of training gradients, we have included additional figures in the appendix, starting line 810, comparing the attention mass assigned to audio, tone description, and speech transcript tokens when predicting the first token of each emotion label during training, under vanilla SFT and our masking approach Emo-TAG. We found that, compared to vanilla SFT, Emo-TAG shows higher attention mass to audio tokens when predicting the emotion label, while maintaining comparable attention to tone description and speech transcript tokens. This suggests that our masking approach M-TAG helps the model pay more attention to the acoustic information in the audio when predicting emotions, rather than relying too heavily on the generated text.

It is well known that the emotions are imbalanced and this naturally leads to a question related to imbalanced situations. Does the model improve grounding for underrepresented classes such as fear and disgust, or are the gains primarily driven by more frequent categories?

Authors Response:

We agree that class imbalance is an important challenge in emotion recognition. As discussed in our limitations section (starting at Line 2106), our training dataset is imbalanced, with fear accounting for only approximately 3% of the samples. As shown in the newly added confusion matrices (starting at Line 918), there is indeed a bias toward majority classes; however, this bias is less pronounced than in EmotionThinker. While these results highlight the remaining challenges with class imbalance, our main focus in this work is improving temporal affective grounding rather than addressing class imbalance or fine-grained emotion recognition. We consider addressing this limitation an important direction for future work.

Reviewer_6HDTOpenReview ↗

Questions addressed

Thank you for the reply. My questions have been largely answered.

We thank the reviewer for their response and are glad that our clarifications have addressed most of their concerns.

We have also added three examples from MELD test set to the supplementary materials, comparing our model's outputs with the ground truth and EmotionThinker (ICLR 2026 oral). These examples show how our model recognizes and temporally localizes emotions in recordings containing multiple speakers. We hope they help illustrate the benefits of our approach.

Reviewer_7v82OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper looks into audio emotion recognition as a prediction task named Temporal Affective Grounding (TAG). They connect emotion labels with temporally bounded speech spans and their tone descriptions, and propose Masked Temporal Affective Grounding (M-TAG) which use a masked emotion and timestamp supervision to improve recognition and temporal grounding. The methods is evaluated on original and reconstructed MELD and IEMOCAP datasets, showing performance improvements.

Critical strengths
  1. The paper propose to frame speech emotion recognition as Temporal Affective Grounding (TAG) to expand beyond utterance level prediction and formulate the task with datasets reconstruction and new metrics for training and evaluation setup.
  2. Experiments and detailed ablation to show the benefits of proposed training methods and benefits of each component.
  3. Emotion hit rate is a useful metric for the task setup to help convert utterance-based labels to temporal label sets.
  4. Detailed qualitative results in the Appendix. It would be great if you could include some qualitative analysis in the main paper as well.
Critical weaknesses
  1. There are some inconsistency in writing, for example, Emov-DB dataset is mentioned in the datasets section and also in training setup, but authors did not report results on Emov-DB and did not explain why. In the appendix, there are more datasets mentioned but no detaios or explanation on why are these omitted in the main paper.
  2. For the choice of baselines (Audio-Reasoner, AffectGPT, FlamingoNext, Qwen2-Audio-Instruct), both FlamingoNex and Audio-Reasoner are explicitly not trained on emotion, and I assume so is Qwen2-Audio-Instruct. Since EMO-TAG has a supervised training objective, it seems not entirely fair to only compare with this limited set which has not been fine-tuned on the specific datasets. Have you tried to compare with one of the closer pipeline you mentioned in the paper, such as SpeechCueLLM? It would be helpful to compare performance with a more comparable benchmark, as well as comparison in sota performance. I believe one way to test is the methods improves general performance could be to compare it with other sota where the temporal bound is set to each utterance.
Questions for authors
  1. Why is Emov-DB dataset mentioned in datasets section and in training setup, but there are no reported results Emov-DB? And what are DFEW and MAFW mentioned on on Figure 4 and Table 5?
  2. How would the specific task formulation be impactful? Expand on future steps and application of the task would be helpful.
5 comments

We thank the reviewer for their constructive feedback and for recognizing the contributions of our TAG formulation, the proposed evaluation metrics, and our detailed ablation. Below, we address each of the reviewer's comments, questions, and concerns.

There are some inconsistency in writing, for example, Emov-DB dataset is mentioned in the datasets section and also in training setup, but authors did not report results on Emov-DB and did not explain why. In the appendix, there are more datasets mentioned but no detaios or explanation on why are these omitted in the main paper.

Why is Emov-DB dataset mentioned in datasets section and in training setup, but there are no reported results Emov-DB? And what are DFEW and MAFW mentioned on on Figure 4 and Table 5?

Authors Response:

We thank the reviewer for pointing out these inconsistencies. EmoV-DB was included as part of the training data to increase the amount of acted emotional speech and improve the balance between in-the-wild data (MELD) and laboratory-recorded data (IEMOCAP and EmoV-DB). We did not include EmoV-DB in the evaluation since MELD and IEMOCAP are more widely adopted benchmarks for speech emotion recognition and provide a stronger basis for comparison with prior work. Regarding MAFW and DFEW, these datasets are listed in Table 7 because they are used in the mixed-data training experiment reported in Table 5, where we investigate whether our approach can benefit from additional emotion-labeled data without speech transcripts or tone descriptions. However, since MAFW and DFEW were primarily developed for video emotion recognition, we do not include them in our main experiments. We would also like to clarify that these datasets are not mentioned in Figure 4. We agree that the role of these datasets was not sufficiently clear in the paper and have clarified their use in the revised manuscript in the dataset description and experimental setup.

For the choice of baselines (Audio-Reasoner, AffectGPT, FlamingoNext, Qwen2-Audio-Instruct), both FlamingoNex and Audio-Reasoner are explicitly not trained on emotion, and I assume so is Qwen2-Audio-Instruct. Since EMO-TAG has a supervised training objective, it seems not entirely fair to only compare with this limited set which has not been fine-tuned on the specific datasets. Have you tried to compare with one of the closer pipeline you mentioned in the paper, such as SpeechCueLLM? It would be helpful to compare performance with a more comparable benchmark, as well as comparison in sota performance. I believe one way to test is the methods improves general performance could be to compare it with other sota where the temporal bound is set to each utterance.

Author Response:

We agree with the reviewer that comparisons with supervised emotion recognition models are important. We would like to clarify that in our experiments, we include both specialized emotion recognition models, such as AffectGPT ( ICML 2025 oral), and strong audio-language models capable of generating temporally grounded outputs, including Audio-Reasoner and FlamingoNext. We additionally compare against our backbone, Qwen2-Audio-Instruct. This choice reflects the current landscape of the field: while several models have been developed for emotion recognition and others for temporal audio understanding, there is currently no single baseline that fully matches the proposed task of temporally grounded emotion recognition. Regarding SpeechCueLLM, while it is relevant to our work, particularly in its use of acoustic descriptions, it still focuses on utterance-level emotion recognition rather than temporal grounding. To provide a stronger comparison with supervised emotion recognition models, we conducted an additional comparison with the recent EmotionThinker model (ICLR 2026 oral). For a fairer comparison, we also trained our model using Qwen2.5-Omni, the same backbone used by EmotionThinker. We report First-Emotion Accuracy / Emotion Hit Accuracy / UAR (Unweighted Average Recall). As shown in the results below, our approach remains competitive on MELD despite using considerably less training data. We train on approximately 12k unique samples from MELD, IEMOCAP, and EmoV-DB, while EmotionThinker uses approximately 35k samples from additional datasets. For IEMOCAP, however, a direct comparison is difficult because EmotionThinker does not distinguish frustration as a separate emotion and instead maps it to sadness. We believe frustration should be treated separately, making the two evaluation settings inconsistent. In addition to overall emotion recognition performance, our model achieves higher UAR than EmotionThinker across all evaluated subsets except MME-Emotion Noise. This suggests that our model is better at recognizing different emotions, with less bias toward frequent classes.

DatasetQwen2-Audio-7B (Emo-TAG)Qwen2.5-Omni-7B (Emo-TAG)EmotionThinker
MELD-test-raw54.71 / 55.59 / 36.0754.18 / 57.85 / 40.7558.81 / 58.81 / 32.19
MELD-test-cleaned57.95 / 58.10 / 37.3657.90 / 61.38 / 42.7160.46 / 60.46 / 33.92
MELD-noisy41.91 / 50.37 / 33.5545.22 / 53.31 / 37.0148.53 / 48.53 / 27.25
IEMOCAP-test-raw67.33 / 68.12 / 57.6166.67 / 68.79 / 55.8750.15 / 50.22 / 40.17
IEMOCAP-cleaned65.83 / 66.03 / 55.2267.14 / 68.75 / 61.1655.24 / 55.24 / 43.98
IEMOCAP-noisy65.73 / 65.73 / 50.5358.98 / 64.91 / 49.9357.99 / 57.99 / 42.69
MME-Emotion Noise62.56 / 63.00 / 61.8663.00 / 68.83 / 68.6472.42 / 72.42 / 72.96
MME-Emotion Wild35.78 / 37.87 / 31.0936.76 / 42.28 / 35.7637.19 / 37.19 / 29.24
MME-Emotion Lab55.33 / 56.34 / 50.3655.53 / 56.94 / 47.8446.28 / 46.28 / 37.14

It is worth noting that MME-Emotion Noise has mainly non-english samples, which explains the lower performance of our model, since all our training data is English.

Regarding the suggestion of assigning the entire utterance as the temporal boundary, we agree that this would provide an additional comparison for single-span emotion classification. However, the temporal boundaries would be given rather than predicted, and this would not evaluate the model's ability to recognize and localize emotion boundaries within the same recording, which is the main contribution of our work.

How would the specific task formulation be impactful? Expand on future steps and application of the task would be helpful.

Authors Response:

The main motivation behind Temporal Affective Grounding (TAG) is to move beyond conventional utterance-level emotion recognition toward identifying and temporally localizing emotional speech segments, including cases with multiple or overlapping speakers. Under these conditions, assigning a single emotion label to an entire recording can be ambiguous. We believe this task formulation is an important step toward more realistic emotion understanding, particularly for applications such as mental health support, education, and customer support agents, where temporally localizing the emotion can be as important as recognizing the emotion itself. For future work, we plan to extend TAG to multimodal emotion recognition by incorporating visual information, such as facial expressions and gestures. This would also allow us to investigate how models handle conflicting emotional cues across modalities. We also aim to extend the approach to address fine-grained emotion and effect of culture.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Discussion

We thank all reviewers for their constructive feedback and for taking the time to review our work.

During the discussion, we clarified our temporal affective grounding formulation and training methodology. This included explaining how we constructed the multi-event training samples from our filtered and refined speech segments. We also further discussed the problem of additional textual outputs potentially harming emotion recognition performance, which is particularly relevant to our multi-span formulation, and how our training approach addresses this problem. We added a figure comparing vanilla SFT with our proposed training approach, showing that the model trained with M-TAG assigns more attention to audio tokens when predicting the first emotion token during training.

We added a comparison with EmotionThinker (ICLR 2026 oral) as a recent speech emotion recognition baseline and provided analysis of the confusion matrices. We showed that our model achieves competitive emotion recognition performance, with more balanced recognition across emotion categories, while also performing temporal grounding.

We added three examples from the MELD test set to the supplementary materials. Each example presents the ground truth alongside the outputs of our model and EmotionThinker. These examples highlight the main challenge we aim to address when recordings contain multiple speakers and emotional events, and show how our model achieves better emotion recognition and temporal grounding in these cases.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.