acceptodds
Withdrawn from ICLR 2027

EmoForecaster: Advancing Multimodal Emotion Forecasting via State-Transition Modeling and Evidence-Grounded Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Emotion is fundamental to human experience and social interaction, making the anticipation of future emotional responses important for proactive interactive systems. However, existing multimodal emotion understanding methods largely focus on emotions that have already been expressed. Although several preliminary studies have explored emotion forecasting, they primarily predict emotion categories, providing limited explicit reasoning about how multimodal evidence and interaction dynamics support their predictions. To address these challenges, we propose EmoForecaster, an evidence-grounded multimodal emotion forecasting framework that integrates state-transition modeling with progressive multimodal reasoning. We first construct EmoEvolve with structured interaction reasoning annotations comprising five components that jointly capture conversational context, multimodal affective evidence, and interaction-aware emotional transitions, and establish a clip-disjoint evaluation protocol covering both emotion forecasting and rationale generation. Furthermore, we develop a complementary architecture in which the state-transition path uses historical reference and preceding interlocutor states to construct context-conditioned forecasts, while the evidence-reasoning path progressively interprets multimodal cues to explain how forthcoming emotional responses may develop, with the two paths coupled at the decision level and optimized through supervised fine-tuning followed by path-specific Group Relative Policy Optimization (GRPO). Extensive experiments demonstrate the superiority of EmoForecaster in multimodal emotion forecasting, with strong performance in both predictive accuracy and rationale quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Emotion Forecasting, Interaction-Aware Reasoning, Affective Computing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_L8vQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

EmoForecaster considers multimodal emotion forecasting from three observed interaction clips. The goal is to predict the emotion that appears in an unseen fourth clip and to provide a rationale for that prediction. EmoEvolve contains 2,830 samples derived from Hi-EF. For each forecasting instance, the authors add a structured rationale linking the available conversational and multimodal evidence to the future-emotion label. The model has two branches. A state-transition path estimates how the historical reference and the current interlocutor state relate to the future response. The second path produces a structured explanation and an additional seven-class prediction. The two predictions are combined before the final decision. Training first uses supervised fine-tuning and then path-specific policy optimization. The final model reaches 43.18 WAR, 29.51 UAR, and 24.28 Macro-F1 on the held-out EmoEvolve test set, substantially above the evaluated pretrained MLLMs. The forecasting gains are meaningful. The evidence for the paper's stronger reasoning claim, however, is less complete than the evidence for improved prediction.

Critical strengths

The task is more demanding than standard emotion recognition. Clip-IV video, audio, and text are unavailable at inference, so the prediction must be made from the preceding interaction rather than from the emotional response itself.

The split is also more careful than a random MCIS partition. Samples sharing the same source clip are assigned to the same partition, which removes one obvious route for train-test overlap.

The architecture is well matched to the forecasting setting. The state-transition path treats the earlier reference and the immediate interlocutor stimulus differently, while the evidence-reasoning path retains explicit multimodal cues and supplies an additional predictive signal.

The same-backbone comparison is useful. Direct supervised fine-tuning of Emotion-LLaMA-v2 obtains 34.71 WAR. The complete Stage-I model reaches 39.06, and Stage II raises this further to 43.18. These results show that the improvement is not explained only by comparing a task-specific system with untuned general-purpose models.

Equation 3 is framed carefully. The residual term is described as a predictive contrast conditioned on the observed context, rather than as evidence that one interlocutor state causally changes the next emotion.

Critical weaknesses

The largest concern is the construction and evaluation of the forecasting rationales. During offline annotation, GPT-5.4 receives the observed Clips I–III evidence together with the gold emotion label of the unseen Clip IV. The prompt then asks it to construct an explanation whose final answer is already known. This produces a useful supervised rationale, but it is still a label-conditioned post-hoc explanation rather than an independently generated reasoning trace. Knowing the target can affect which observed cues are emphasized and how ambiguous evidence is interpreted. A rationale may therefore appear coherent partly because it was written with the answer in hand.

The current rationale metrics do not address this distinction. BLEU-4, ROUGE-L, METEOR, and BERTScore evaluate similarity to the constructed reference text. They do not test whether a statement in the generated rationale is actually supported by the observed audio, video, or dialogue. The reference rationales are themselves produced from automatically extracted acoustic and visual descriptions and are conditioned on the target label. A model can therefore obtain a high similarity score by learning the style and structure of those references without necessarily using the underlying evidence faithfully. Human judgments of factual support and unsupported cue attribution would be more informative. An evidence-removal test would also be useful: if a rationale cites a visual or acoustic cue, removing that cue should alter the corresponding explanation.

The annotation pipeline deserves more documentation from the human side. Automatic audio and visual descriptions are incorporated into the rationale prompt, after which the generated rationale passes through an automatic consistency-and-coverage check before human review. The manuscript says that experienced domain experts inspect and revise the retained outputs, but gives little quantitative information about this final stage. For example, it is unclear how many reviewers participated, how frequently the text was materially changed, or whether independent reviewers agreed on annotation quality. Reporting these quantities would make the degree of human validation much easier to assess.

The headline comparison with pretrained MLLMs also combines several advantages. EmoForecaster receives 1,681 task-specific fitting examples, structured rationales, intermediate state labels, and a further policy-optimization stage. Most models in Table 1 do not receive equivalent EmoEvolve training. The difference between the final 43.18 WAR and the roughly 25-WAR pretrained MLLMs therefore reflects task adaptation and additional supervision as well as architecture. Table 2 gives the more informative architectural comparison: direct supervised fine-tuning of the same backbone reaches 34.71 WAR, compared with 39.06 for the complete Stage-I system. That gain is still meaningful, but it is smaller than the headline gap to untuned baselines.

The Stage-II evidence for complementarity is also modest. The state-transition variant reaches 42.71 WAR, the evidence-reasoning variant reaches 43.06, and the complete system reaches 43.18. The last increment is only 0.12 points. UAR and Macro-F1 show similarly small differences. On the rationale side, the evidence-reasoning variant actually has the higher BLEU-4 and METEOR values. With no repeated-run uncertainty reported, the current table supports the usefulness of the individual paths more clearly than a reliable gain from merging them.

The split removes exact clip reuse, but it is not yet clear how much broader source overlap remains. Different clips originating from the same conversation or recording may still share speakers, setting, lexical context, and narrative structure. Those regularities could be particularly useful in a forecasting task. A stronger evaluation would group samples by the underlying interaction or original source before constructing train, development, and test partitions.

The evaluation also comes entirely from the Hi-EF lineage. EmoEvolve uses the 2,830 released Hi-EF examples, and all quantitative forecasting results are measured on a partition of that collection. Transfer to a separately collected interaction corpus is therefore still unknown. Even a limited external experiment would be useful for checking whether the observed gains survive a change in recording source and annotation convention.

Absolute forecasting accuracy should also be kept in perspective. The final model reaches 29.51 UAR and 24.28 Macro-F1 for a seven-class task. These numbers are substantially better than the tested baselines, but they still indicate a difficult forecasting problem with considerable remaining error. Claims about proactive deployment should therefore remain conservative, particularly for sensitive application settings.

Finally, prediction and rationale quality are trained through closely related supervision. Stage II rewards future-label accuracy and intermediate state prediction, while the evidence-reasoning representation also feeds the emotion classifier. The same label-conditioned annotations supervise rationale generation. An increase in rationale-reference similarity can therefore accompany better classification without proving that the text faithfully reflects the evidence used by the predictor. Perturbing individual modalities or cues and checking whether both the prediction and rationale respond consistently would provide a stronger faithfulness test.

Questions for authors

How often did human reviewers make substantive changes to the automatically generated rationales?

How many reviewers took part in this stage, and did the authors measure agreement between independent reviews?

Can rationale quality be evaluated by human judges who see only Clips I–III and are not shown the gold Clip-IV emotion?

Can the evaluation test whether individual statements in a rationale are actually supported by the corresponding multimodal evidence?

What happens when a visual or acoustic cue explicitly cited by the rationale is removed from the input?

Can reference rationales be generated without supplying y4y_4 and compared with the current label-conditioned references?

How are samples grouped when several clips originate from the same underlying conversation or recording?

Can the experiment be repeated with these source groups kept entirely within one partition?

Were the Table 1 and Table 2 results obtained from one training seed? If so, can repeated-run uncertainty be provided for the small Stage-II differences?

Is the 43.18 versus 43.06 WAR difference stable across independent runs?

Can the same-backbone supervised baseline be trained with the same structured rationale supervision while omitting the proposed state-transition architecture?

Can EmoForecaster be evaluated on a second emotion-forecasting corpus?

How does performance change when the auxiliary y2y_2 and y3y_3 supervision is removed?

Reviewer_EemCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates future emotion forecasting in multimodal interpersonal interactions, aiming to predict a target participant’s future emotional state and generate corresponding explanations grounded in the preceding interaction. The proposed framework combines state-transition modeling with evidence-based multimodal reasoning and introduces a dataset with structured reasoning annotations.

The experimental results show that the proposed method outperforms the general-purpose multimodal large language models and specialized emotion-understanding models evaluated in the paper. While the results are promising, several concerns remain regarding the methodological novelty of the proposed framework, the necessity of its architectural components, and the reliability of the reported improvements. In particular, the absence of a direct comparison with an existing method designed specifically for emotion forecasting makes it difficult to assess the extent of the paper’s contribution.

Critical strengths
  1. Meaningful research problem. Predicting future emotional states from preceding interpersonal interactions is a meaningful research direction. Beyond recognizing emotions that have already been expressed, the paper attempts to anticipate forthcoming emotional responses and provide explanations grounded in multimodal evidence. This capability may be useful for developing more proactive and context-aware interactive systems.

  2. Contribution through dataset construction. The construction of EmoEvolve, which enriches existing data with structured interaction and reasoning annotations, provides a potentially useful resource for evaluating both future emotion forecasting and rationale generation. The adoption of a clip-disjoint evaluation protocol also addresses the risk of information leakage between samples sharing source clips.

  3. Promising experimental results. The proposed method achieves better results than the evaluated general-purpose and emotion-capable multimodal models across the reported forecasting and rationale-generation metrics. These results suggest that the combination of emotion-transition modeling and evidence-grounded reasoning may be beneficial for the task.

Critical weaknesses
  1. The methodological novelty and necessity of the proposed architecture remain insufficiently established. The framework combines several existing techniques, including multimodal feature extraction, temporal modeling, decision-level fusion, supervised fine-tuning, and GRPO-based optimization. Although integrating these techniques for future emotion forecasting may have value, the paper does not yet sufficiently establish which aspects constitute its central methodological innovation or why the proposed architectural choices are preferable to plausible alternatives. More evidence is needed to demonstrate that the proposed design offers benefits beyond those obtainable through a simpler combination of existing components.

  2. The lack of a direct comparison with Hi-EF limits the assessment of the proposed method. Hi-EF has already addressed a closely related multimodal emotion forecasting task and provides a relevant task-specific baseline. However, the paper does not report results for the original Hi-EF method under the same data subset and clip-disjoint evaluation protocol. Since the proposed dataset is constructed from Hi-EF data, evaluating the original method under a unified protocol would provide a more direct reference for assessing the contribution of the proposed framework. Without this comparison, it remains difficult to determine how much of the observed improvement comes from the proposed architecture and optimization strategy rather than differences in the underlying models or supervision.

  3. The ablation studies do not fully establish the contributions of individual design choices. The existing experiments compare the baseline, the two individual reasoning paths, and the complete framework, providing some evidence for the complementary roles of state-transition modeling and evidence reasoning. However, they do not sufficiently isolate the contributions of several key components, including the explicit affective feature vector, decision-level fusion, and path-specific optimization. Furthermore, during the GRPO stage, the complete model achieves a WAR of 43.18%, compared with 43.06% for the standalone evidence-reasoning path, a difference of only 0.12 percentage points. Although this difference alone does not establish that the improvement is insignificant, the absence of statistical reliability analyses makes it difficult to determine whether the additional predictive benefit of the complete two-path framework is robust. More targeted ablations and statistical analyses would strengthen the evidence supporting the proposed design.

Questions for authors
  1. Methodological novelty: Compared with Hi-EF and existing multimodal emotion-understanding and reasoning methods, what is the central technical innovation of EmoForecaster? Which specific design choices are essential to its performance, and how does the proposed framework differ substantively from a combination of existing techniques?

  2. Ablation and statistical reliability: Could the authors provide more fine-grained ablation studies isolating the contributions of the explicit affective feature vector, decision-level fusion, and path-specific GRPO optimization? Could they also report statistical reliability analyses to establish whether the observed improvements, particularly the relatively small gains from combining both paths during the GRPO stage, are robust?

Reviewer_j2VVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EmoForecaster to predict the emotion of an unseen response from three observed multimodal interaction clips and generate an explanatory rationale. The authors extend Hi-EF into EmoEvolve with structured, LLM-assisted rationales and a clip-disjoint evaluation split. The model combines state-transition and evidence-reasoning paths through decision-level fusion, trained by supervised fine-tuning followed by path-specific GRPO. Experiments show improved forecasting accuracy and reference-rationale similarity.

Critical strengths
  • The paper focuses on predicting future emotions rather than recognizing emotions that have already been expressed. Extending this task to include explanations of the predictions is useful and worth exploring.
  • The dataset construction is a useful contribution. The authors extend Hi-EF with structured reasoning annotations and introduce a clip-disjoint evaluation split.
  • The experimental results are encouraging. EmoForecaster improves emotion forecasting performance over the compared models. The ablation studies also examine the effects of the two proposed paths and the two training stages, providing some evidence for their contributions.
Critical weaknesses
  • The reference rationales are generated by GPT-5.4 with access to the ground-truth future emotion. As a result, these rationales may simply provide plausible explanations for an already known outcome, rather than reflect how the future emotion can be inferred from the observed evidence. Moreover, the evaluation relies mainly on similarity to these generated references, which does not directly measure whether the explanations are factually grounded.

  • It is unclear whether the state-transition module actually learns meaningful emotion transitions. The proposed transition matrices are conditioned on the full interaction context, so the model may rely mainly on contextual features rather than the estimated historical emotion states. For example, if the rows of a transition matrix become similar, the resulting prediction would depend little on the corresponding intermediate emotion distribution. The current experiments do not rule out this possibility.

  • Most baselines in Table 1 are evaluated without comparable task-specific fine-tuning, although the direct SFT baseline in Table 2 already achieves a substantial improvement. The paper also does not compare against the original Hi-EF forecasting model under the same data split. In addition, the improvement from combining the two optimized paths is quite small.

  • The Evidence-Reasoning Path is designed to progressively interpret multimodal cues (Lines 27 and 93), but the experiments do not examine how text, audio, and visual modalities contribute to the generated rationales.

  • Figures 1–4 contain excessively small text and dense annotations, which are difficult to read.

Questions for authors
  1. Regarding the reasoning annotations, GPT-5.4 is given the ground-truth emotion of Clip IV when generating the reference rationales. How do the authors ensure that these rationales are supported by the observed evidence, rather than simply explaining an already known outcome?

  2. Can the authors verify that the model actually uses the estimated historical emotion states when predicting future emotions? Since the transition matrices are conditioned on the full context, the predictions might depend mainly on contextual features.

  3. Since most compared MLLMs are not specifically fine-tuned for this forecasting task, providing comparisons with task-specific forecasting methods trained on the same data split would help clarify the improvement from the proposed method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.