- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces ICET (Identity-Centric Emotion Tracking), a task formulation that removes two assumptions common in multimodal conversational emotion understanding: (1) that input utterances are pre-segmented, and (2) that only the current speaker's emotion is of interest. Given a raw, unsegmented conversational video and a designated target participant (speaker or non-speaker), the model must jointly perform utterance segmentation (USeg), identity-consistent emotion classification (EC), and emotion-cause extraction (ECE). To support this, the authors construct EmoTrack, a bilingual (English/Chinese) benchmark with 22,585 utterance-level emotion-cause annotations over 21 hours of video, derived from ECF/MELD and CMMA. They benchmark 12 open- and closed-source MLLMs and report several findings: (1) video length is a major bottleneck with a systematic language gap; (2) audio dominates while simple prompt-based visual fusion provides limited benefit; (3) transcripts act as an amplifier; and (4) GRPO yields mixed results.
Critical strengths
- The two assumptions (i.e., pre-segmentation and speaker-centrism) in previous works are genuinely restrictive in deployed settings, and removing them simultaneously is a meaningful step toward realistic evaluation.
- EmoTrack contains 22,585 annotations over 21 hours, covering both speakers and non-speakers, in two languages. The four-step protocol, three-annotator majority voting, expert adjudication, and reported IAA are appropriate.
Critical weaknesses
-
My major concern is regarding the task setup. ICET takes a designated target person as input, specified in practice by a reference face image. However, in realistic deployments, the emotionally salient participants are not known in advance and must be discovered. The paper removes pre-segmentation but retains a target-specification assumption that is arguably more consequential, so the claim that ICET "removes the unrealistic assumptions" of prior work overstates the realism gain.
-
This paper fails to consider non-MLLM baselines for comparison. All evaluated models are general-purpose MLLMs; the only task-specific model is itself an MLLM. The paper claims to demonstrate "EmoTrack's diagnostic value for current MLLM limitations," but its conclusions "audio dominates, prompted video is underused, long videos degrade" describe prompted MLLM behavior, not the task.
-
The English subset inherits speaker emotions and causes from ECF; new work is non-speaker annotation plus timestamp re-alignment. For Chinese, speaker causes and non-speaker labels are new. The paper does not report what fraction of the 22,585 annotations is genuinely new (by language and target type), how the ECF-to-episode re-alignment was validated, or how dense the non-speaker annotations are — even though the identity-centric claim rests on them. Without these numbers, it is unclear whether EmoTrack is a new benchmark or a re-packaged ECF + CMMA.
Questions for authors
- What is the performance of non-MLLM baselines?
- Does the length-degradation curve differ between MLLMs and a fixed-window conventional model?
- What fraction of annotations is new vs. inherited, by language and target type?
2 comments
Response to Reviewer 7w65 (2/2)
Answer of Weakness #3 & Question #3:
We appreciate this constructive comment. We provide detailed statistics on annotation provenance, non-speaker annotation density, and temporal realignment below.
We first clarify that the 22,585 instances reported in the paper refer to utterances rather than person-level annotations. Since an utterance may contain annotations for multiple participants, the dataset contains 82,844 person-level annotations, each consisting of an emotion label and an emotion-cause field. The annotation provenance is summarized in Table A. The corrected annotations are consistent with the descriptions in Section 3.1 (Pages 3–4).
Table A: Annotation provenance across languages and target types.
| Language and target | Person-level annotations | EC annotations | ECE annotations |
|---|---|---|---|
| English speaker | 13,346 (EC) + 13,346 (ECE) | 13,326 inherited; 20 corrected | 13,313 inherited; 33 corrected |
| English non-speaker | 10,432 (EC) + 10,432 (ECE) | 10,432 newly added | 10,432 newly added |
| Chinese speaker | 9,239 (EC) + 9,239 (ECE) | 6,531 inherited; 2,708 corrected | 9,239 newly added |
| Chinese non-speaker | 8,405 (EC) + 8,405 (ECE) | 8,405 newly added | 8,405 newly added |
As shown in Table A, Non-speaker annotations account for 45.5% of all person-level annotations (18,837 out of 41,422), including 10,432 English and 8,405 Chinese annotations. Moreover, 59.9% of utterances (13,538 out of 22,585) contain at least one non-speaker annotation, demonstrating substantial annotation coverage beyond the current speaker.
For both English and Chinese portions, we verified the temporal alignment of the original timestamps provided by the source datasets by comparing each utterance clip with its corresponding segment in the original full episode. Specifically, we sampled three frames (first, middle, and last) from each clip to check visual consistency and compared the durations of the two segments. In the English portion, 12,892 of 13,346 utterances passed both checks and retained their original timestamps. The remaining 454 were manually inspected: 333 were confirmed to be correctly aligned, while 121 required manual timestamp correction. The same procedure was applied to the Chinese portion.
Answer of Question #2:
Thank you for your question. As discussed in Answer of Weakness #2 & Question #1, representative traditional multimodal emotion recognition models cannot be directly evaluated under the ICET setting due to differences in their input requirements and task formulations.
To further investigate whether input segmentation affects the observed length-degradation patterns, we conducted an additional experiment with Qwen2.5-Omni-7B using fixed-window processing. Keeping the model and checkpoints unchanged and varying only the input segmentation strategy isolates the effect of segmentation on length-related performance. We used non-overlapping 20-second windows for English videos and 10-second windows for Chinese videos—approximately half the average video duration in each language—and mapped predictions from each window back to the original video timeline. The fixed-window results are reported below; each cell shows USeg mIoU / EC W-F1 / ECE W-F1. The corresponding original-setting results are presented in Figure 3.
The two input strategies exhibit broadly similar length-related performance trends, but differ in the degree and pattern of degradation. For English videos, both show an overall decline across all three tasks, with the decline being more pronounced in USeg and ECE. However, fixed-window processing leads to a more gradual decline in USeg and achieves better EC and ECE performance in all longer-video buckets, although ECE still degrades with increasing duration. For Chinese videos, neither strategy shows consistent length-related degradation in EC and ECE, while USeg exhibits a more noticeable declining trend across most duration buckets with fixed-window processing. These results suggest that length-degradation patterns are influenced by input segmentation as well as language and task characteristics.
Although this experiment does not establish a direct performance difference between MLLMs and traditional models, it clarifies the effect of input segmentation within the evaluated MLLM setting. We will add this discussion to the appendix in the final version.
English (20-second windows)
| Video duration | Fixed-window |
|---|---|
| 2–22s | 48.52 / 43.58 / 23.99 |
| 23–43s | 41.31 / 46.24 / 20.63 |
| 44–64s | 36.94 / 45.23 / 19.27 |
| 65–85s | 31.30 / 41.57 / 17.82 |
| 86–106s | 30.48 / 44.68 / 15.53 |
Chinese (10-second windows)
| Video duration | Fixed-window |
|---|---|
| 8–13s | 61.59 / 31.35 / 7.84 |
| 14–19s | 56.51 / 35.55 / 14.79 |
| 20–25s | 54.91 / 35.72 / 14.70 |
| 26–31s | 53.06 / 50.58 / 25.22 |
| 32–37s | 56.88 / 43.21 / 16.06 |
Response to Reviewer 7w65 (1/2)
Answer of Weakness #1:
Thank you for this insightful comment.
We acknowledge that ICET requires the target identity to be specified via an external reference face image, which does not need to come from the current video. Unlike conventional conversational emotion recognition, which treats the current speaker as the default analysis target, ICET allows the target to be any participant; therefore, the target participant must be explicitly specified.
Importantly, specifying whose emotions to track does not require knowing in advance which participant is emotionally salient. Users can provide a reference face for any one or more participants—potentially all of them—and apply ICET to analyze each participant's emotions individually. The reference image only defines the analysis target; the model must still identify that participant in the video and infer their emotions and causes throughout the conversation.
Furthermore, removing the external reference image requirement yields a more general and harder task: the system must discover all participants and track their emotional states without prior identity specification. Specifically, it must assign unique identities to participants both within and across conversations, ensuring that the same person is recognized as the same participant across different batches. In particular, the identity space is theoretically unbounded, since the number of participants cannot be known in advance. ICET provides an important foundation for this broader problem. To avoid overstating our contribution, we will revise the relevant discussion in the final version.
Answer of Weakness #2 & Question #1:
Thank you for raising this important concern.
We would like to clarify that EmoTrack is designed to benchmark the capabilities and limitations of current MLLMs on ICET, as stated in the abstract and introduction. This focus is motivated by the growing adoption of LLM- and MLLM-based approaches in multimodal emotion recognition [1]. Accordingly, we evaluate both general-purpose MLLMs and task-specific MLLMs, including Emotion-LLaMA-7B, which is specifically designed for emotion understanding.
Regarding non-MLLM baselines, representative multimodal emotion recognition models such as DialogueRNN [2] typically rely on pre-segmented utterances and pre-extracted multimodal features. They also do not support emotion analysis for multiple participants. Consequently, they cannot be directly evaluated in ICET's setting. Adapting these models would require substantial modifications to their input pipelines and task architectures, yielding systems that differ considerably from their original configurations. We consider such adaptations a valuable complementary direction, but beyond the scope of this MLLM-focused benchmark.
We agree that our empirical findings should be interpreted within this scope. We will clarify this distinction in the final version.
[1] Zhang, H., Wu, D., Li, Y., Liu, K., Wang, Y., Zhou, Y., & Zhao, S. (2026). Multimodal Emotion Recognition with Large Language Models. In Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26 (pp. 8088–8096).
[2] Majumder, N., Poria, S., Hazarika, D., Mihalcea, R., Gelbukh, A., & Cambria, E. (2019, July). Dialoguernn: An attentive rnn for emotion detection in conversations. In Proceedings of the AAAI conference on artificial intelligence (Vol. 33, No. 01, pp. 6818-6825).