- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
authors proposed temporal affective grounding to leveraging the emotion recognition performance from multiple speech spans. Authors also curated a dataset to include speech spans, tone descriptions for MEDL and IEMOCAP datasets.
Critical strengths
Quantifying the emotion by considering temporal aspects of an audio signal is an interesting idea.
Authors curated dataset with time stamps, tone description and emotion label. This can add value database generation considering the currently available AI tools.
Authors evaluated their model on benchmark dataset (IEMOCAP and MELD) for emotion recognition task.
Critical weaknesses
- The two stage proposed methodology is not convincing. Tone description is intuitively similar to acoustic features.
- Using AI models to generate tone description is easy thing todo. However, how to validate the tone descriptions which is a key? this is missing in the paper. 3.Authors are not comparing their proposed methodology with state of the art methods instead authors chose to compare their model with zero-short learning models.
Questions for authors
Quantifying the emotion by considering temporal aspects of an audio signal is an interesting idea.
Authors created a dataset using 2-4 spans with overlapping speech. how authors decide the overlapping window size varying between 0.2 to 2 seconds. IEMOCAP dataset contains very small segments. How randomly choosing span size between 0.2 to 2 seconds works for segments that are less than 2 seconds long. Authors should report length of audio samples in IEMOCAP and EMLD datasets. Since author are curating the dataset to create temporal affective learning, authors should design and follow a protocol to create the dataset.
What is SFT in Figure 1. Authors should provide full description of figure 1.
IEMOCA and MELD datasets come up with text, audio, and video modalities. Above all authors have freedom to compute audio features like pitch, frequency, tone, energy etc from the audio signal. It looks like tone descriptions are kind of imitating the audio level features pitch, energy, frequency, or MFCC features.
Data curation: authors mentioned that “transcript contains two or more consecutive insertions (extra words that are not part of the utterance) or three or more replacements (different words that may have been misrecognized by WhisperX), we reject the sample”. How many samples were rejected for MELD and IEMOCAP dataset?
MELD dataset provides training, validation and test dataset for model evaluations. But IEMOCAP does not provide predefined splits. It is not clear how authors considered the splits for model evaluations? Is it 5-fold session wise cross validation? table 1 original/refined accuracies not much difference, there is no difference between emotion segment accuracy vs hit accuracy. This indicates that multiple spans are not adding much to the overall model accuracy.
Authors curated datasets and generated tone descriptions using Ai tools but masking it during training is not convincing. In that case why to generate tone descriptions?
From Table 1: authors compare EMO-TAG with Audio reasoner, AffectGPT, FlamingoNext these are zero-shot models. Authors must compare their model with state of the art for fair comparison. So the results are not comparable.
AI-generated tone descriptions are not convincing because the entire pipeline depends on FlamingoNext (generation) → ChatGPT (refinement) → Qwen3 (extraction/checking). Since authors claimed that they curated the dataset, it would be helpful to validate the tone descriptions and report annotator agreement scores. There are no baseline comparisons with SOTA on IEMOCAP and MELD. Authors compared against general-purpose audio-language models (FlamingoNext, Audio-Reasoner) evaluated zero-shot — not against models actually designed and trained for emotion recognition on these datasets. For example, EmoBERTa, DialogGCN, LineConGCN, MMGCN. Without comparing it with state of the art, the improvements that authors included in the paper is not convincing.
For emotion recognition, confusion matrices show which emotion pairs are getting confused and how the proposed model addresses them. In particular, MELD and IMECAP datasets are imbalanced. Authors should include confusion matrices to understand model accuracy in recognizing similar emotions like happy vs excited, sad vs angry.
It is important to include ablation study. For example, include ablation for different time spans and authors should include sensitivity analysis on time stamps
12 comments
Tone description is intuitively similar to acoustic features. IEMOCA and MELD datasets come up with text, audio, and video modalities. Above all authors have freedom to compute audio features like pitch, frequency, tone, energy etc from the audio signal. It looks like tone descriptions are kind of imitating the audio level features pitch, energy, frequency, or MFCC features.
Author Response:
We agree that tone descriptions are closely related to acoustic features. In fact, as described in Section 3.2 (starting at Line 180) and further explained in Section A.3 (starting at Line 743), we explicitly use these features as a reference to guide and ground the generation of tone descriptions. We see this as a strength rather than a weakness, as it helps ground the descriptions in measurable acoustic cues. Describing tone is more complex than simply extracting acoustic features. Some people naturally have higher-pitched voices, while others have lower-pitched voices. Similarly, speaking rate and loudness vary across individuals, genders, and cultures. For example, a pitch value of 130 Hz alone does not tell us whether a speaker is angry or neutral, as it needs to be interpreted relative to their normal speaking patterns. Our goal was to make the tone descriptions easy to understand while preserving as much relevant acoustic information as possible. We acknowledge that these descriptions could be improved, but developing a more accurate and comprehensive representation of tone would require substantial effort, particularly to account for individual and cultural differences in how emotions are expressed through speech.
Using AI models to generate tone description is easy thing todo. However, how to validate the tone descriptions which is a key? this is missing in the paper.
AI-generated tone descriptions are not convincing because the entire pipeline depends on FlamingoNext (generation) → ChatGPT (refinement) → Qwen3 (extraction/checking). Since authors claimed that they curated the dataset, it would be helpful to validate the tone descriptions and report annotator agreement scores.
Author Response:
We agree that generating tone descriptions using AI is relatively straightforward, but ensuring their quality is more challenging. This is why we did not simply prompt a model to describe the audio. Instead, we followed a multi-stage process. First, we generated initial annotations using an audio model. We then used a separate text-only model to compare these annotations against the ground-truth emotion labels, rejecting samples where the descriptions were inconsistent with the labels. Such inconsistencies could result from incorrect model-generated descriptions or potentially mislabeled samples. Finally, we used low-level acoustic features and speaker-specific acoustic references to guide GPT-Astra in producing more concrete and acoustically grounded tone descriptions. We acknowledge that there is no guarantee that every generated tone description is accurate, as we could not manually verify every sample due to limited human resources. Nevertheless, the usefulness of the resulting descriptions is supported by the improved downstream emotion recognition performance observed in our experiments. We emphasize that our main contributions are the temporal formulation of audio emotion recognition and the proposed training approach, rather than tone description generation itself.
Authors are not comparing their proposed methodology with state of the art methods instead authors chose to compare their model with zero-short learning models.
There are no baseline comparisons with SOTA on IEMOCAP and MELD. Authors compared against general-purpose audio-language models (FlamingoNext, Audio-Reasoner) evaluated zero-shot — not against models actually designed and trained for emotion recognition on these datasets. For example, EmoBERTa, DialogGCN, LineConGCN, MMGCN. Without comparing it with state of the art, the improvements that authors included in the paper is not convincing.
Author Response:
We thank the reviewer for raising this important point regarding baseline selection. Our task differs from conventional emotion recognition because it requires both emotion classification and temporal localization of emotion segments. To our knowledge, none of the available emotion recognition models supported this task of temporally grounded emotion recognition. We therefore compared against general temporal audio understanding models with strong emotion recognition capabilities, such as Audio-Reasoner, which demonstrated competitive results on MME-Benchmark, and the more recent Audio-Flamingo-Next. We also included specialized emotion recognition models with publicly available checkpoints, such as AffectGPT (ICML 2025 oral), which uses the same Qwen2-Audio as its backbone. Regarding the suggested baselines, EmoBERTa, DialogueGCN, LineConGCN, and MMGCN focus on utterance-level emotion recognition in conversations, often using textual context and, in some cases, multiple modalities. While these are relevant baselines for conventional emotion classification, they do not address our proposed temporal affective grounding task. We acknowledge that comparisons with these methods would provide additional context for our classification results. However, their reported results are not directly comparable due to differences in input modalities and evaluation methods. To better address the concern about comparisons with state-of-the-art emotion recognition models, we conducted an additional comparison with the recent EmotionThinker model (ICLR 2026 oral). For a fairer comparison, we also trained our model using Qwen2.5-Omni, the same backbone used by EmotionThinker. We report First-Emotion Accuracy / Emotion Hit Accuracy / UAR (Unweighted Average Recall). As shown in the new results, our model remains competitive on MELD despite using considerably less training data. We train on approximately 12k unique samples from MELD, IEMOCAP, and EmoV-DB, while EmotionThinker uses approximately 35k samples from additional datasets. For IEMOCAP, a direct comparison is difficult because EmotionThinker does not distinguish frustration as a separate emotion and instead maps it to sadness. We believe frustration should be treated separately, making the two evaluation settings inconsistent. Regarding class-wise performance, our model achieves higher UAR than EmotionThinker across all evaluated subsets except MME-Emotion Noise. This suggests that our model is better at recognizing different emotions, with less bias toward frequent classes.
| Dataset | Qwen2-Audio-7B (Emo-TAG) | Qwen2.5-Omni-7B (Emo-TAG) | EmotionThinker |
|---|---|---|---|
| MELD-test-raw | 54.71 / 55.59 / 36.07 | 54.18 / 57.85 / 40.75 | 58.81 / 58.81 / 32.19 |
| MELD-test-cleaned | 57.95 / 58.10 / 37.36 | 57.90 / 61.38 / 42.71 | 60.46 / 60.46 / 33.92 |
| MELD-noisy | 41.91 / 50.37 / 33.55 | 45.22 / 53.31 / 37.01 | 48.53 / 48.53 / 27.25 |
| IEMOCAP-test-raw | 67.33 / 68.12 / 57.61 | 66.67 / 68.79 / 55.87 | 50.15 / 50.22 / 40.17 |
| IEMOCAP-cleaned | 65.83 / 66.03 / 55.22 | 67.14 / 68.75 / 61.16 | 55.24 / 55.24 / 43.98 |
| IEMOCAP-noisy | 65.73 / 65.73 / 50.53 | 58.98 / 64.91 / 49.93 | 57.99 / 57.99 / 42.69 |
| MME-Emotion Noise | 62.56 / 63.00 / 61.86 | 63.00 / 68.83 / 68.64 | 72.42 / 72.42 / 72.96 |
| MME-Emotion Wild | 35.78 / 37.87 / 31.09 | 36.76 / 42.28 / 35.76 | 37.19 / 37.19 / 29.24 |
| MME-Emotion Lab | 55.33 / 56.34 / 50.36 | 55.53 / 56.94 / 47.84 | 46.28 / 46.28 / 37.14 |
It is worth noting that MME-Emotion Noise has mainly non-english samples, which explains the lower performance of our model, since all our training data is English. We thank the reviewer again for raising this point and would be happy to consider any additional related baselines they recommend.
We thank the reviewer for their constructive feedback and appreciate the opportunity to further clarify our contributions and provide additional evidence addressing their concerns. Below, we address each of the reviewer's comments and outline our proposed revisions.
Author Response:
It is unclear whether the reviewer is referring to our tone-description generation process or our two-stage training methodology, so we address both.
Regarding the tone-description process, we use the initial descriptions generated by FlamingoNext as supporting evidence for context and changes in tone that are difficult to identify from the low-level acoustic features of individual segments alone.
Regarding the training methodology, our two-stage approach aims to address a fundamental problem: introducing extra textual reasoning outputs (such as speech transcripts and tone descriptions) can negatively affect emotion recognition performance compared to predicting the emotion label alone. This issue has received limited attention in previous works. Zhao et al. (2025) show that prompting a model to produce Chain-of-Thought reasoning without task-specific training can reduce speech emotion recognition accuracy, while Xu et al. (2025) show that extended multimodal reasoning can weaken visual grounding and increase hallucinations. We further demonstrate this effect in our ablation study in Figure 2 and Table 3 of the appendix, where temporal emotion-only prediction outperforms joint prediction of speech, tone, and emotion before introducing our two-stage training approach. Our approach addresses this challenge through context masking, guiding the model to better utilize emotion-relevant training signals and preventing them from being overshadowed by the additional textual tokens. The newly added Figure 3 in the appendix (starting at line 810), which compares the attention mass when predicting emotion labels under our approach and vanilla SFT, further supports this explanation by showing that masking increases the attention assigned to audio tokens.
Author Response:
We would like to clarify that we do not mask the tone descriptions in the language modeling pass. Our method uses two forward passes. The first is the language modeling pass, vanilla SFT, which gives us the language modeling loss. Nothing is masked when calculating this loss. The second pass is used to calculate the emotion and timestamp losses, where we mask either the tone descriptions or both the tone descriptions and speech transcripts when predicting the emotion labels and timestamps. The tone descriptions, speech transcripts, and output format are still learned through the standard language modeling loss. The masking provides additional supervision for the emotion labels and timestamps, preventing these important training signals from being overshadowed by the other generated tokens.
References: