acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking Identity-Centric Emotion Tracking in Continuous Video Conversations

Abstract

Current multimodal conversational emotion understanding relies on a convenient but unrealistic premise: pre-segmented utterances with aligned video, audio, and text, where models reason only about the current speaker. In reality, conversations are continuous and unsegmented, and non-speakers also react emotionally. To address this gap, we introduce Identity-Centric Emotion Tracking (ICET), which removes these assumptions. ICET comprises three interdependent sub-tasks: Utterance Segmentation, Identity-Consistent Emotion Classification, and Emotion-Cause Extraction. Utterance Segmentation is performed over all participants, while the latter two sub-tasks cover both speakers and non-speakers. To support ICET, we contribute EmoTrack, a Chinese-English bilingual dataset with 22,585 utterance-level emotion-cause annotations over 21 hours of conversational videos. Benchmarking Multimodal Large Language Models (MLLMs) on ICET reveals challenges in utterance segmentation, long-range dependency modeling, and visual utilization. We further show that speech-aligned transcripts amplify both gains and errors, while reinforcement learning remains unstable due to sparse rewards and credit assignment challenges. These findings highlight EmoTrack's diagnostic value and show that ICET remains an open challenge.

Primary area:
datasets and benchmarks
Keywords:
Identity-Centric Emotion Tracking, Multimodal Conversation, Unsegmented Video

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_7w65OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces ICET (Identity-Centric Emotion Tracking), a task formulation that removes two assumptions common in multimodal conversational emotion understanding: (1) that input utterances are pre-segmented, and (2) that only the current speaker's emotion is of interest. Given a raw, unsegmented conversational video and a designated target participant (speaker or non-speaker), the model must jointly perform utterance segmentation (USeg), identity-consistent emotion classification (EC), and emotion-cause extraction (ECE). To support this, the authors construct EmoTrack, a bilingual (English/Chinese) benchmark with 22,585 utterance-level emotion-cause annotations over 21 hours of video, derived from ECF/MELD and CMMA. They benchmark 12 open- and closed-source MLLMs and report several findings: (1) video length is a major bottleneck with a systematic language gap; (2) audio dominates while simple prompt-based visual fusion provides limited benefit; (3) transcripts act as an amplifier; and (4) GRPO yields mixed results.

Critical strengths
  1. The two assumptions (i.e., pre-segmentation and speaker-centrism) in previous works are genuinely restrictive in deployed settings, and removing them simultaneously is a meaningful step toward realistic evaluation.
  2. EmoTrack contains 22,585 annotations over 21 hours, covering both speakers and non-speakers, in two languages. The four-step protocol, three-annotator majority voting, expert adjudication, and reported IAA are appropriate.
Critical weaknesses
  1. My major concern is regarding the task setup. ICET takes a designated target person as input, specified in practice by a reference face image. However, in realistic deployments, the emotionally salient participants are not known in advance and must be discovered. The paper removes pre-segmentation but retains a target-specification assumption that is arguably more consequential, so the claim that ICET "removes the unrealistic assumptions" of prior work overstates the realism gain.

  2. This paper fails to consider non-MLLM baselines for comparison. All evaluated models are general-purpose MLLMs; the only task-specific model is itself an MLLM. The paper claims to demonstrate "EmoTrack's diagnostic value for current MLLM limitations," but its conclusions "audio dominates, prompted video is underused, long videos degrade" describe prompted MLLM behavior, not the task.

  3. The English subset inherits speaker emotions and causes from ECF; new work is non-speaker annotation plus timestamp re-alignment. For Chinese, speaker causes and non-speaker labels are new. The paper does not report what fraction of the 22,585 annotations is genuinely new (by language and target type), how the ECF-to-episode re-alignment was validated, or how dense the non-speaker annotations are — even though the identity-centric claim rests on them. Without these numbers, it is unclear whether EmoTrack is a new benchmark or a re-packaged ECF + CMMA.

Questions for authors
  1. What is the performance of non-MLLM baselines?
  2. Does the length-degradation curve differ between MLLMs and a fixed-window conventional model?
  3. What fraction of annotations is new vs. inherited, by language and target type?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 7w65 (1/2)

Answer of Weakness #1:

Thank you for this insightful comment.

We acknowledge that ICET requires the target identity to be specified via an external reference face image, which does not need to come from the current video. Unlike conventional conversational emotion recognition, which treats the current speaker as the default analysis target, ICET allows the target to be any participant; therefore, the target participant must be explicitly specified.

Importantly, specifying whose emotions to track does not require knowing in advance which participant is emotionally salient. Users can provide a reference face for any one or more participants—potentially all of them—and apply ICET to analyze each participant's emotions individually. The reference image only defines the analysis target; the model must still identify that participant in the video and infer their emotions and causes throughout the conversation.

Furthermore, removing the external reference image requirement yields a more general and harder task: the system must discover all participants and track their emotional states without prior identity specification. Specifically, it must assign unique identities to participants both within and across conversations, ensuring that the same person is recognized as the same participant across different batches. In particular, the identity space is theoretically unbounded, since the number of participants cannot be known in advance. ICET provides an important foundation for this broader problem. To avoid overstating our contribution, we will revise the relevant discussion in the final version.

Answer of Weakness #2 & Question #1:

Thank you for raising this important concern.

We would like to clarify that EmoTrack is designed to benchmark the capabilities and limitations of current MLLMs on ICET, as stated in the abstract and introduction. This focus is motivated by the growing adoption of LLM- and MLLM-based approaches in multimodal emotion recognition [1]. Accordingly, we evaluate both general-purpose MLLMs and task-specific MLLMs, including Emotion-LLaMA-7B, which is specifically designed for emotion understanding.

Regarding non-MLLM baselines, representative multimodal emotion recognition models such as DialogueRNN [2] typically rely on pre-segmented utterances and pre-extracted multimodal features. They also do not support emotion analysis for multiple participants. Consequently, they cannot be directly evaluated in ICET's setting. Adapting these models would require substantial modifications to their input pipelines and task architectures, yielding systems that differ considerably from their original configurations. We consider such adaptations a valuable complementary direction, but beyond the scope of this MLLM-focused benchmark.

We agree that our empirical findings should be interpreted within this scope. We will clarify this distinction in the final version.

[1] Zhang, H., Wu, D., Li, Y., Liu, K., Wang, Y., Zhou, Y., & Zhao, S. (2026). Multimodal Emotion Recognition with Large Language Models. In Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26 (pp. 8088–8096).
[2] Majumder, N., Poria, S., Hazarika, D., Mihalcea, R., Gelbukh, A., & Cambria, E. (2019, July). Dialoguernn: An attentive rnn for emotion detection in conversations. In Proceedings of the AAAI conference on artificial intelligence (Vol. 33, No. 01, pp. 6818-6825).

AuthorsOpenReview ↗

Response to Reviewer 7w65 (2/2)

Answer of Weakness #3 & Question #3:

We appreciate this constructive comment. We provide detailed statistics on annotation provenance, non-speaker annotation density, and temporal realignment below.

We first clarify that the 22,585 instances reported in the paper refer to utterances rather than person-level annotations. Since an utterance may contain annotations for multiple participants, the dataset contains 82,844 person-level annotations, each consisting of an emotion label and an emotion-cause field. The annotation provenance is summarized in Table A. The corrected annotations are consistent with the descriptions in Section 3.1 (Pages 3–4).

Table A: Annotation provenance across languages and target types.

Language and targetPerson-level annotationsEC annotationsECE annotations
English speaker13,346 (EC) + 13,346 (ECE)13,326 inherited; 20 corrected13,313 inherited; 33 corrected
English non-speaker10,432 (EC) + 10,432 (ECE)10,432 newly added10,432 newly added
Chinese speaker9,239 (EC) + 9,239 (ECE)6,531 inherited; 2,708 corrected9,239 newly added
Chinese non-speaker8,405 (EC) + 8,405 (ECE)8,405 newly added8,405 newly added

As shown in Table A, Non-speaker annotations account for 45.5% of all person-level annotations (18,837 out of 41,422), including 10,432 English and 8,405 Chinese annotations. Moreover, 59.9% of utterances (13,538 out of 22,585) contain at least one non-speaker annotation, demonstrating substantial annotation coverage beyond the current speaker.

For both English and Chinese portions, we verified the temporal alignment of the original timestamps provided by the source datasets by comparing each utterance clip with its corresponding segment in the original full episode. Specifically, we sampled three frames (first, middle, and last) from each clip to check visual consistency and compared the durations of the two segments. In the English portion, 12,892 of 13,346 utterances passed both checks and retained their original timestamps. The remaining 454 were manually inspected: 333 were confirmed to be correctly aligned, while 121 required manual timestamp correction. The same procedure was applied to the Chinese portion.

Answer of Question #2:

Thank you for your question. As discussed in Answer of Weakness #2 & Question #1, representative traditional multimodal emotion recognition models cannot be directly evaluated under the ICET setting due to differences in their input requirements and task formulations.

To further investigate whether input segmentation affects the observed length-degradation patterns, we conducted an additional experiment with Qwen2.5-Omni-7B using fixed-window processing. Keeping the model and checkpoints unchanged and varying only the input segmentation strategy isolates the effect of segmentation on length-related performance. We used non-overlapping 20-second windows for English videos and 10-second windows for Chinese videos—approximately half the average video duration in each language—and mapped predictions from each window back to the original video timeline. The fixed-window results are reported below; each cell shows USeg mIoU / EC W-F1 / ECE W-F1. The corresponding original-setting results are presented in Figure 3.

The two input strategies exhibit broadly similar length-related performance trends, but differ in the degree and pattern of degradation. For English videos, both show an overall decline across all three tasks, with the decline being more pronounced in USeg and ECE. However, fixed-window processing leads to a more gradual decline in USeg and achieves better EC and ECE performance in all longer-video buckets, although ECE still degrades with increasing duration. For Chinese videos, neither strategy shows consistent length-related degradation in EC and ECE, while USeg exhibits a more noticeable declining trend across most duration buckets with fixed-window processing. These results suggest that length-degradation patterns are influenced by input segmentation as well as language and task characteristics.

Although this experiment does not establish a direct performance difference between MLLMs and traditional models, it clarifies the effect of input segmentation within the evaluated MLLM setting. We will add this discussion to the appendix in the final version.

English (20-second windows)

Video durationFixed-window
2–22s48.52 / 43.58 / 23.99
23–43s41.31 / 46.24 / 20.63
44–64s36.94 / 45.23 / 19.27
65–85s31.30 / 41.57 / 17.82
86–106s30.48 / 44.68 / 15.53

Chinese (10-second windows)

Video durationFixed-window
8–13s61.59 / 31.35 / 7.84
14–19s56.51 / 35.55 / 14.79
20–25s54.91 / 35.72 / 14.70
26–31s53.06 / 50.58 / 25.22
32–37s56.88 / 43.21 / 16.06
Reviewer_W2i3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper discusses the candidate coverage issue when speculative decoding will be used by LLM inference. For the draft model's output, this paper proposes Solvo, which can provide better tree construction and verification. Solvo is a sampling-centric framework for tree speculative decoding that replaces expand-then-prune with an allocate-then-sample procedure. Author evaluated Solvo on five target models and six benchmarks spanning instruction following, dialogue, mathematical reasoning, code generation, question answering, and summarization.

Author proved that Solvo preserves the exact target-model distribution even though its topology adapts to sampled tokens and its depth is determined by cost-aware stopping. And the end-to-end acceleration has been evaluated across all evaluated targets under stochastic decoding.

Critical strengths

The paper proposes an interesting allocate-then-sample approach that avoids the sampling–pruning dependency in EAGLE-style dynamic trees. The method is clearly explained, and the losslessness proof covers adaptive tree growth and cost-aware stopping. This provides a solid theoretical foundation for the proposed design.

Critical weaknesses
  1. The contribution to the e2e acceleration from Solvo tree-construction method is not sufficiently isolated. The main performance gains come from several changes, including threshold tree, cost-aware stopping, mixed proposals, and UniVer verification. Although Table 2 provides a component analysis, it remains unclear how much of the improvement can be attributed specifically to the proposed allocate-then-sample design.

  2. Limited evaluation of the proposed method under different serving conditions. All experiments are conducted with batch size 1 on a single GPU. While this is a reasonable setting for latency-oriented decoding, it is unclear whether the throughput improvements would hold under concurrent serving workloads, where additional draft and verification computation may compete with other requests. The paper should better analyze this limitation and, provide at least a small-batch evaluation.

Questions for authors
  1. The threshold sensitivity analysis is only conducted under greedy decoding (T=0) (Section 3.2, p5; Appendix G.1, p21). Since Solvo mainly targets stochastic decoding, how sensitive is its performance to the threshold at T=1? Also, how should the threshold be chosen for a new model?
  2. For global Top-B Objective (Section 3.2, p. 5; Appendix A, p. 12), Solvo uses draft path scores to approximate the value of each candidate. However, a high draft probability does not necessarily mean a high acceptance probability by the target model. How well does this scoring method reflect actual acceptance, and could it lead to suboptimal tree construction?
4 comments
AuthorsOpenReview ↗

Response to Reviewer W2i3 (1/2)

Answer of Weakness #1:

Thank you for this insightful comment.

Under Eq. (1), ICET evaluates USeg, EC, and ECE with respect to the designated target participant pp. Thus, even if an utterance is correctly segmented, its emotion prediction is considered correct only if it matches the ground-truth emotion of pp. If the prediction instead matches another participant p+1p+1 but differs from that of pp, it is counted as an error.

However, we acknowledge that the current evaluation cannot independently verify whether the model correctly attributes an emotion to the designated participant. This is because our evaluation follows prior conversational emotion recognition settings [1], where predictions are evaluated against the annotations of the designated speaker without a separate identity-consistency metric, even if the emotion may actually belong to another participant.

This domain has largely avoided introducing a task-level identity-consistency metric, mainly because doing so is extremely difficult. A meaningful participant-level identity-consistency metric would require the model to assign a unique and consistent identity to each participant not only within a conversation but also across conversations, so that the same person is recognized as the same participant in different batches. This requires persistent speaker memory and a globally consistent ID space, which is particularly challenging when the participant set is open-ended rather than fixed and known in advance. Moreover, potential alternatives such as counterfactual interventions (e.g., masking the non-target participant) provide only indirect evidence. They cannot reliably establish which participant the model actually relies on when making an emotion prediction: changes under such interventions may reflect the loss of valid emotional context from other participants rather than identity confusion.

To further investigate this issue, we conducted an exploratory analysis of cases where the predicted emotion was incorrect for pp but matched the emotion of another participant. The proportions were 29.1% and 23.0% in English two-person and multi-person scenarios, respectively, and 39.7% and 52.1% in the corresponding Chinese scenarios. These results indicate that some incorrect predictions may be mismatched to a non-target participant. However, this does not directly measure identity confusion, since label matching does not necessarily mean that the model actually attended to that participant. We will add this limitation to the final version and investigate it further in future work.

[1] Majumder, N., Poria, S., Hazarika, D., Mihalcea, R., Gelbukh, A., & Cambria, E. (2019, July). Dialoguernn: An attentive rnn for emotion detection in conversations. In Proceedings of the AAAI conference on artificial intelligence (Vol. 33, No. 01, pp. 6818-6825).

AuthorsOpenReview ↗

Response to Reviewer W2i3 (2/2)

Answer of Weakness #2:

Thank you for raising this important concern.

Regarding whether the performance degradation stems from identity confusion or difficulties in emotion reasoning, please refer to Answer of Weakness #1. As discussed there, the current evaluation does not allow us to reliably distinguish between these error sources.

USeg remains relatively stable because it relies primarily on audio information (please refer to the modality ablation results on Page 8), which may make it less sensitive to the increased visual complexity in multi-party scenarios.

In contrast, more participants introduce greater visual complexity and potential interference. The model must distinguish the target participant from others and identify their emotional cues amid more complex visual information and interactions. Such interference may affect both target identification and emotion recognition, potentially contributing to the decline in EC performance. This potential difficulty is qualitatively illustrated in Figure 6.

ECE exhibits a more pronounced performance degradation, partly because its performance is affected by errors in emotion classification. To examine this effect, we evaluated cause extraction only on instances with correctly predicted emotions. The W-F1 increases from 4.27 to 19.37 for English and from 4.70 to 42.13 for Chinese, indicating a strong association between emotion classification errors and poorer cause extraction performance. Nevertheless, cause extraction remains challenging even when emotions are correctly recognized.

Answer of Question #1:

Thank you for your question.

In Figure 4, EC and ECE predictions are counted as incorrect if their corresponding predicted utterances fail to meet the specified IoU threshold. Varying the threshold therefore does not change the predicted boundaries or directly measure the USeg error rate. Rather, it evaluates the sensitivity of downstream performance to different boundary-matching criteria.

The relatively small performance gap between IoU thresholds of 0.1 and 0.5 suggests that downstream performance is not highly sensitive to the boundary-matching criterion under our evaluation setting. Our benchmark is constructed from existing datasets and provides a reasonable sample size and coverage. Nevertheless, the generalizability of this observation to other datasets and scenarios requires further investigation. We will clarify the scope of this finding in the revised manuscript.

Answer of Question #2:

Thank you for your question.

Please refer to Answer of Weakness #1 and Weakness #2 for the discussion of identity confusion and the additional analysis of EC and ECE performance in multi-party scenarios. In particular, we clarify the limitations of separately quantifying identity confusion and provide an analysis of how emotion classification errors are associated with cause extraction performance.

Reviewer_W2i3OpenReview ↗

To correct the review comment

Dear authors,

Due to my mistake, the original review comments should belong to another article. Please use the following for comments from me. Very sorry for this mistake. Thanks!

Paper Summary: This paper proposes a new task ICET for tracking one participant’s emotions in raw conversational video. It drops pre-segmentation and speaker-centric assumptions: a model gets the video and a target person, then must segment utterances, classify that person’s emotions consistently, and extract the utterances that cause them. The authors release EmoTrack, a Chinese–English benchmark with 22,585 utterance-level annotations over 21 hours, covering speakers and non-speakers. They benchmark several open and closed MLLMs. Performance drops on longer videos, audio matters most, simple visual fusion helps little, transcripts amplify both correct and noisy audio cues, and GRPO gives mixed gains. The paper provides a diagnostic benchmark: it shows where current MLLMs fail rather than offering a solution.

Presentation: 2: Generally understandable, but requires substantial effort in places. Informativeness: 2: I learned something interesting or useful. Soundness: 2: I have minor or fixable concerns about the paper’s soundness. Critical Strengths: ICET is a sensible task formulation: it removes pre-segmentation and speaker-centric assumptions and proposed a new evaluation method including the benchmark of utterance segmentation, identity-consistent emotion classification, and emotion-cause extraction jointly on continuous video. EmoTrack provides a bilingual resource with 22,585 utterance-level annotations covering both speakers and non-speakers, which supports this setting and enables controlled comparisons. The benchmark of diverse MLLMs is broad and diagnostic rather than a single leaderboard: the modality ablations, transcript analysis, multi-party comparisons, and attention visualizations help locate where current models fail. Critical Weaknesses: The authors do not include identity tracking or consistency as a separate evaluation metric. In long, multi-person video sequences, identity tracking errors could affect the accuracy and reliability of the overall evaluation results. From the eq(1), ICET only evaluates utterance segmentation(USeg), emotion classification(EC), and emotion-cause extraction(ECE) for the designated target participant p. But what happen if some USeg result is correct but the predicted emotion is for another participant p+1? Figure 5 shows a clear drop in EC and cause extraction performance in multi-party scenarios, while USeg remains relatively stable. It is unclear whether this degradation comes from identity confusion or difficulties in emotion reasoning. In long videos with complex multi-person interactions, identity confusion could affect both emotion classification and emotion-cause extraction. This raises concerns about the reliability of ICET's evaluation results in such scenarios, and whether the benchmark can adequately assess models under complex multi-person interactions.

Questions For Authors: In page8 and fig. 4, author makes a conclusion that the utterance boundary errors are no longer the main bottleneck for EC and ECE once a coarse boundary is achieved. However, varying the IoU threshold does not actually change the predicted utterance boundaries. On one hand, the IoU vary is not directly mapping the error rate of USeg, right? On the other hand, will it be too limited by the samples used in this benchmark? Fig. 5 shows that EC and ECE performance drops in multi-party scenarios, while USeg remains relatively stable. How much of this degradation is caused by identity confusion rather than difficulties in emotion recognition or cause extraction? Could the authors provide a separate analysis for it? Flag For Ethics Review: No Rating: 2: Weak rejection Confidence: 2: I have broad familiarity with this field but little or no direct research experience on this topic.

Dear Reviewer,

Thank you for your clarification and for providing the corrected comments.

We confirm that our responses correspond to the corrected version you provided (the one beginning with “Paper Summary: This paper proposes a new task ICET…”), not to the earlier comments that were sent by mistake. We have carefully addressed each point in that corrected version, including the concerns about identity tracking/consistency, the multi-party degradation analysis, and the questions about Fig. 4/Fig. 5 and the IoU threshold analysis.

If you have any remaining questions or would like further discussion on any point, we would be very happy to discuss them in detail.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.