acceptodds
Under review as a conference paper at ICLR 2027

AffectVerse: Emotional World Models for Multimodal Affective Computing

Abstract

Humans infer emotions by integrating observed multimodal cues with expectations about how affective states may unfold. Existing multimodal large language models (MLLMs), however, often treat emotion recognition as static fusion over complete audiovisual-text inputs,leaving affective dynamics implicit. We propose AffectVerse, a Qwen2.5-Omni-based model equipped with an Emotion World Module (EWM), an action-free representation-level module for short-horizon latent affective prediction. EWM contains three modules: 1) Cross-Modal Temporal Imagination predicts future video/audio representations from past tokens with multistep rollout. 2) MAMA(Modality-Aware Multi-step Attention) Belief Aggregation compresses imagined tokens into modality-aware belief tokens. 3) Belief Injection inserts these belief tokens into the LLM for affective reasoning. AffectVerse uses future prediction as a past-conditioned self-supervised signal: it does not replace modeling observed history or require unseen signals at inference, but forces the current belief state to encode transition cues that are predictive of subsequent affective change. Across nine benchmarks, AffectVerse improves at least 2.57% over other models, while controlled ablations show additive gains from temporal imagination, crossmodal rollout, and belief aggregation. These results suggest predictive belief-state modeling is a practical alternative for affective computing.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
MLLMs, emotion

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kXkxOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents AffectVerse, an emotional world model framework for multimodal affective computing. It introduces an Emotion World Module (EWM) that combines cross-modal temporal imagination, MAMA belief aggregation, and belief injection to incorporate predictive emotional representations into multimodal reasoning. Experiments under different datasets show gains over baselines.

Critical strengths

1.The paper explores an interesting direction by introducing predictive world modeling into multimodal affective computing. 2.The proposed EWM provides a structured approach to learning future-aware representations and integrating them into MLLM-based emotion reasoning. 3.The experimental evaluation is comprehensive.

Critical weaknesses

1.The wirting should be further polished to reduce AI style and increase logical coherence, especially the experiment analysis section. 2.The multi-step temporal imagination mechanism is not sufficiently justified. According to Appendix D.6, all rollout steps are supervised using the same pooled future target rather than distinct future horizons. This raises concerns about whether the model actually learns progressive emotional dynamics. 3.The implementation of belief injection is unclear. Eq. (1) extracts hidden states from the Thinker, while Eq. (8) inserts belief tokens into these representations. It is unclear how the injected tokens influence subsequent language modeling if the backbone hidden states have already been computed. 4.The training and inference settings differ considerably. The model learns to predict future segments during training but uses all observed tokens at inference. The effectiveness of the learned predictive mechanism under this setting needs further justification. 5.The temporal alignment between modalities is insufficiently considered. Audio and video are split using the same keep ratio without explicit timestamp alignment, which may lead to inconsistent temporal contexts when modeling rapidly changing emotions. The modality coflict is also lack of consideration. 6.Some experimental claims and reporting require clarification. The reported +3.40-point improvement is calculated over six benchmarks, whereas the nine-benchmark improvement in Table 1 is approximately +2.58 points. Additionally, the results are based on a single reported random seed, making it difficult to assess statistical stability.

Questions for authors

1.Could the authors clarify whether the rollout steps correspond to different future time horizons or repeated predictions of the same future representation? 2.At which transformer layer are the belief tokens injected, and how do they interact with the original LLM hidden states? 3.Can the authors provide a quantitative comparison between samples with significant emotional transitions and samples with relatively stable emotions? 4.How does AffectVerse perform when audio and visual cues provide conflicting emotional information? 5.How sensitive is the method to different temporal sampling rates or longer audiovisual sequences? 6.Could the authors provide an analysis of whether the imagined representations capture emotion transition directions rather than only future emotion categories? 7.Does the performance improvement remain consistent across multiple random seeds? Conducting significance test would be helpful. 8.How would the framework perform when textual information is unavailable or unreliable, considering that text bypasses the EWM? 9.Could the authors further analyze failure cases where temporal imagination leads to incorrect emotional predictions or negatively affects the original backbone's performance? Besides, more cases with emotion trajectory should be provided.

Reviewer_L1cROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AffectVerse, a framework that integrates temporal imagination, belief aggregation, and belief injection into Qwen2.5-Omni for multimodal emotion recognition. The method achieves improvements across nine benchmarks, supported by several ablation studies and comparisons with existing approaches.

Critical strengths
  1. The paper addresses an interesting problem by introducing temporal prediction into multimodal emotion recognition.
  2. The proposed method achieves competitive results across nine benchmarks, outperforming several existing methods.
  3. The authors provide detailed ablation studies, including comparisons across different backbones and model configurations.
  4. The additional computational overhead is relatively small compared with the backbone model.
Critical weaknesses
  1. The methodological novelty is somewhat limited, as the proposed world model mainly combines existing cross-attention and latent feature prediction techniques.
  2. All rollout steps are supervised using the same pooled future representation. It is therefore unclear how the model captures different stages of emotional evolution.
  3. Most experiments focus on static emotion recognition. More direct evaluation of emotional transitions would better support the central motivation.
  4. Since temporal splitting is performed after processing the multimodal inputs through the backbone, there is a possibility that future information influences the past representations.
  5. The paper would benefit from reporting results across multiple random seeds and clarifying the IEMOCAP evaluation protocol and effective batch size.
Questions for authors
  1. How does the model capture different temporal transitions when all rollout steps are trained using the same future target?
  2. What is the key methodological difference between the proposed emotion world model and existing latent feature prediction approaches?
  3. How do the authors ensure that future information is not included in the past representations during temporal splitting?
Reviewer_3zCbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AffectVerse, a Qwen2.5-Omni-based framework for multimodal emotion recognition. Its Emotion World Module predicts audiovisual representations from observed prefixes, aggregates the predictions into compact belief tokens, and injects these tokens into the language model. The model combines language modeling with a future-representation prediction objective. Experiments cover nine benchmarks, with additional component ablations, capacity-matched controls, cross-backbone comparisons, and partial-observation evaluations.

Critical strengths
  1. The idea of using predictive audiovisual representations to support emotion recognition is interesting. The proposed module offers a concrete way to incorporate predictive supervision into an existing multimodal language model, and the reported improvements suggest potential practical value.

  2. The experimental coverage is substantial. In particular, Appendix E includes capacity-matched non-predictive controls, future-representation probes, cross-backbone experiments, and leave-one-dataset-out evaluation. These experiments provide useful evidence that the gains may extend beyond simply adding parameters or additional tokens.

Critical weaknesses
  1. The temporal interpretation and computational implementation of the central mechanism remain insufficiently clear. Equation (5) and Appendix D.6 explicitly indicate that every rollout step is supervised against the same pooled future target. Consequently, the procedure appears to iteratively refine a prediction of the same future segment; the claimed extension of the prediction horizon across steps is not established. This distinction matters for interpreting the method as modeling evolving affective dynamics.

There is also an important ambiguity in belief injection. Equation (1) defines H as the Thinker’s output hidden states, while Algorithm 1 inserts belief tokens into H and directly applies the language-model head. Taken literally, a tokenwise output head would not allow the inserted tokens to influence existing answer positions. The manuscript should specify what transformer computation follows injection and how the retained representations are restricted to the observed prefix. These may be presentation issues, but they currently prevent a clear assessment of the mechanism.

  1. Several numerical inconsistencies weaken confidence in the reported evidence. In Table 3, the displayed six-dataset scores imply a gain of approximately +3.40 for the full model over the baseline, whereas the reported ΔAvg is +3.07. Table 4 reports +3.40 for the same scores. More substantially, averaging the full-modality results at 10% observation in Table 2 over the six datasets specified in Appendix E gives approximately 75.40, while Table 13 reports 67.42. This difference is too large to arise from rounding.

If these tables use different truncation procedures, input availability, or evaluation settings, those differences need to be explicitly documented. Otherwise, corrected results are necessary. Because robustness under partial observations is a central claim, this discrepancy materially affects my assessment.

Questions for authors
  1. Could the authors provide the precise forward-pass sequence, including the layer at which belief tokens are inserted and the transformer operations performed afterward? For partial-observation evaluation, is the unavailable audiovisual suffix removed before encoding, and are transcripts also restricted to the observed prefix?

  2. What distinct temporal interval does each rollout step predict, given that all steps share the same pooled future target? How should the claimed increase in temporal coverage be interpreted under this supervision?

  3. Could the authors reconcile the discrepancies between Tables 3 and 4 and between Tables 2 and 13, providing consistent per-dataset results and the exact protocol used for each table?

  4. The authors are encouraged to discuss V-JEPA, I-JEPA, ImageWAM, BadWAM, and BLIP-2 to clarify the method’s novelty and the relationship between imagination quality and prediction reliability.

The proposed direction is promising, and I appreciate the extensive supplementary experiments. However, the unresolved methodological ambiguities and inconsistencies in the reported results currently limit my confidence in the central claims. I will reconsider and adjust my score based on the authors’ rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.