acceptodds
Withdrawn from ICLR 2027

Do Emotion Concepts Generalize Across Sources, Modalities, and Architectures in Vision-Language Models?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent studies suggest that large language models encode emotion concepts as structured internal representations, but most existing work focuses on text and a single architecture. Therefore, we ask, do emotion concepts generalize across sources, modalities, and architectures in vision–language models (VLMs)? To address this, we construct CMES (Cross-Modal Emotion Stimuli), a multi-source collection of emotion-conditioned stories, real facial expressions, synthetic portraits, and synthetic emotion-evoking scenes. For each stimulus source, we extract a separate set of six Ekman emotion vectors from each of three VLMs. We report four main findings as follows: 1) Image-derived emotion vectors form a low-dimensional geometry similar to that of text-derived vectors. Valence is relatively stable across sources, while arousal varies more. 2) Text- and image-derived emotion vectors have modest cosine similarity but still show held-out cross-modal correspondence. Text-derived vectors can also steer image interpretation. 3) Cross-architecture correspondence remains even when native cosine is near zero. Transformations estimated from generic ImageNet activations recover both correspondence and causal transfer without using the six emotion vectors or their labels. 4) After aligning representations across architectures, we construct a shared emotion subspace that preserves affective geometry and selective steering effects. The corresponding consensus emotion vectors also generalize to a held-out fourth architecture at two model sizes. These results suggest that emotion representations can share relational structure and causal effects across sources, modalities, and architectures, even when individual vector directions differ.

Primary area:
interpretability and explainable AI
Keywords:
Affective Computing, Emotion Representation, Emotion Concepts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yhuQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies whether emotion representations in VLMs generalize across different sources, modalities and architectures. The authors introduce CMES, covering six basic emotions across text, real facial expressions and synthetic images. They extract emotion vectors from three VLMs and examine their geometry, alignment and transferability through activation steering. They also construct a shared emotion subspace and evaluate it on held-out Gemma-3 models. The results suggest that emotion representations share some common structure across models, although they vary with the stimulus source and that aligned vectors can transfer steering effects across modalities and architectures.

Critical strengths
  • The research question addressed in the work (consistency of emotion representations across modalities, stimulus sources and model architectures) is interesting.

  • The experimental evaluation (e.g., three core VLM architectures; multiple text/image sources; additional held-out Gemma-3 models; cross-architecture transfer experiments; inclusion of activation steering) is quite extensive and in my view constitutes the core contribution of this work.

Critical weaknesses

The paper's methodological novelty is low (techniques are well-established; the shared emotion subspace is constructed using relatively straightforward alignment and averaging operations). As mentioned in the strengths section above, the core contribution of this work is the experimental evaluation.

Although generally an empirical study can be valuable, the paper does not sufficiently demonstrate what new insights are obtained beyond prior work on emotion representations, cross-model alignment and steering transfer. A dedicated Related Work section, which is currently missing, would help in that direction. Currently, most of the discussion is deferred to Appendix A. Given the substantial existing literature, a more explicit comparison with previous approaches is needed to better establish the contribution.

Additionally, the paper requires substantial restructuring and clearer presentation. The main paper is very dense, with numerous experiments and findings compressed into a limited space, while important methodological explanations and experimental details are repeatedly deferred to the appendix. Section 2 is particularly difficult to follow, as the emotion-vector extraction procedure is introduced with little explanation of the individual steps or their motivation. I would encourage the authors to reconsider the organization of the paper, provide a clearer introduction to the methodology and focus the main text on the most important experiments and findings, with sufficient detail to understand and assess them without repeatedly referring to the appendix.

In terms of the experiments, one of my main concerns is the interpretation of the activation steering results. The authors argue that emotion vectors causally influence how VLMs interpret emotions in images. However, the reported interventions may instead be introducing a general bias toward generating a particular emotion label. Appendix E.6 shows that steering also affects predictions concerning the assistant and a third party who is not present in the image. Furthermore, Table 20 shows that injecting emotion vectors at answer-prefix positions produces effects comparable to those obtained at image-token positions. These results raise questions about whether the interventions genuinely modify the model's interpretation of the visual evidence or primarily influence its subsequent response. Although the authors acknowledge these effects (and that is a very positive thing in my view and I appreciate it), I do not think their implications are sufficiently discussed in relation to the central claims.

Additionally, the average steering results hide substantial differences between individual emotions. The main paper emphasizes the average increases in target-emotion log probabilities following steering. However, the results are considerably less convincing for some emotion categories. For example, Appendix E.4 reports that disgust steering changes the generated emotion readout to the intended target on only 32 out of 3,500 AffectNet validation images, less than 1%, while the other five emotions show success rates of 14-60%. This is a substantial discrepancy. It indicates that an increase in target-emotion log probability does not necessarily translate into successful changes in predictions or generated responses. It would be good if the paper has a clearer per-emotion evaluation and a discussion of these failures, rather than relying primarily on aggregate steering metrics.

I also have some concerns regarding the LOEO evaluation. The authors use LOEO alignment to evaluate whether mappings fitted on five emotions generalize to the sixth. However, the emotion vectors are constructed by subtracting the mean across all six emotion categories, introducing a linear dependence among them before normalization. Consequently, the held-out emotion representation is not completely independent of those used to fit the transformation. This raises questions about how stringent the LOEO evaluation is and to what extent the observed correspondence is influenced by the shared construction of the emotion vectors. The permutation controls are useful and the authors acknowledge this limitation in Appendix B.7. Nevertheless, it would be good to see a more stringent evaluation involving genuinely unseen emotion categories or a procedure that excludes the held-out category when computing the reference statistics.

Another concern relates to the findings in Section 3.2, where the authors report that RAF-DB produces a different emotion geometry, with disgust closer to happiness and fear closer to anger, unlike the patterns observed for AffectNet and the other sources. The disgust - happiness relationship is particularly surprising. The retrieval analysis in Appendix C.2 provides additional evidence of this behaviour, but does not establish why it occurs. Annotation ambiguity, differences in class composition, expression intensity, and dataset-specific visual characteristics are plausible explanations. Although the authors provide additional retrieval analyses and investigate the effects of filtering, these experiments do not fully explain the observed differences. This is important because the conclusion that stimulus source has a greater influence on emotion geometry than architecture is one of the central findings. It would be good to see a more thorough investigation of the factors responsible for these differences, rather than primarily reporting them as evidence of source-dependent emotion representations.

Finally, the VA interpretation and low-dimensionality claims require greater caution. The authors report that the first two principal components explain approximately 55-75% of the variance and that these components exhibit some agreement with human V and A ratings. However, PCA does not explicitly identify affective dimensions and the correlations are calculated using only six emotion categories. Furthermore, while the agreement with V is relatively strong, the agreement with A is substantially weaker and varies considerably across sources. Since the vectors are constructed from six centered categories, their rank is at most five. Explaining a majority of the variance with two components is thus less compelling than it might initially appear. I find these results useful as descriptive observations, but I think the authors should be more cautious in interpreting them as evidence of human-like affective representations, particularly given the limited number of emotion categories and the considerable variation in A agreement across sources.

Questions for authors

Please have a look above on the extensive critical weaknesses comments.

Reviewer_pWTnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work investigates emotion representations in MLLMs, across sources (different datasets, modalities (text/image) and architectures (Mistral, Qwen, Gemma). First, the authors build a corpus of seven existing datasets covering Ekman's six basic emotions. Then, following other steering methods, activations for each emotion are averaged, and the mean is subtracted. PCA is then used to reduce the dimensionality of the directions found. Given these directions, the method proceeds with a series of investigative experiments to understand the space across models and inputs.

Critical strengths
  • The idea of finding a shared space that is architecture-agnostic is novel and interesting. This should have been the main contribution of the paper and should have been explored further.
  • The investigative experiments help better frame the problem and understand the space (maybe too extensive and should be in appendix)
Critical weaknesses
  • The paper does not compare with any other steering or transfer method. This is a major flaw in the experimental details, especially when several are cited in the main paper and appendix.
  • The main directions and steering applied are, in practice, a simple difference of means [1]
  • In terms of the contributions, most headline findings are expected by construction making the claims overstated and most findings confirmatory. More specifically, Images and Text having a shared direction subspace is expected in a projector-based VLM (image tokens are projected into text space). Low text–image cosine is the well-known modality gap, which isn’t cited (and also cosine similarity is used extensively even though it is nothing more than an indication given the gap). Near-zero native cosine between independently trained models is expected, so its “recovery” after alignment says little (in fact, alignment recovery is also expected).
  • Related to the above, the metrics are not exactly indicative of performance. Cosine in a 4096-d residual space is a weak correspondence measure, and the paper’s own Table 28 shows it disagreeing with causal effect.
  • In addition, the analysis of the subspace shows somewhat limited understanding of the task. Ekman's six emotions, while well established, are limited and do not capture intensity. The choice of model is also probably responsible for the lack of alignment in arousal space (sadness is the only low-arousal emotion). The RAF-DB finding also most likely indicates label noise rather than source differences (otherwise we should expect AffectNET to behave differently too).
  • LOEO is not an out-of-sample test. Eq. 1 centres the six class means, so the held-out sixth vector is (up to the norm weights) a linear combination of the other five.

[1] https://arxiv.org/html/2502.02716v1

Questions for authors
  1. How does the consensus subspace compare with the affine or linear transfer maps of CAA [1], Huang et al. (2025), Chen et al. (2025), and Agarwal (2026), and with standard steering vectors, on the same evaluation?
  2. How much LOEO correspondence remains if the held-out class is excluded from the centring mean?
  3. Does steering change behaviour beyond the six emotion-word log-probabilities, for example, on an open-ended emotion description scored by an independent judge or on a task that doesn’t name the emotions? Typically, steering affects fluency; how does the Pareto frontier hold (using an LLM as a judge and, if you have time, human evals)?
Reviewer_w832OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, the authors investigate whether emotion representations in VLMs share structure and causal effects across stimulus sources, text and image modalities, and model architectures. They introduce CMES, covering six basic emotions across three story sources, two real facial-expression datasets, synthetic portraits, and synthetic scenes. They then extract emotion vectors from model activations and compare their geometry, correspondence, and effects on model outputs. Experiments involve three VLMs, with Gemma-3 at two sizes evaluated separately. Main findings are that emotion vectors preserve similar relational structure despite differing directions, text-derived vectors can steer image interpretation and mappings learned from ImageNet activations allow for the transfer between model architectures. Averaging aligned vectors produces a representation that also transfers to Gemma-3.

Critical strengths
  • Studying whether related concepts preserve both their organization and behavioral effects across modalities and models is an interesting problem.
  • The authors combine geometric analysis with activation interventions and subspace removal. The experiments done provide evidence that the extracted directions contribute to emotion-related outputs. It is also interesting to note the direct steering of image interpretation using text-derived vectors.
  • Learning coordinate transformations from generic images, without fitting to emotion vectors or their labels, reduces dependence on emotion-specific alignment. Transfer to Gemma-3 further strenthens the evidence that the findings extend beyond the models used.
Critical weaknesses
  • Steering works much better for some emotions than others. For example, disgust steering produces the intended answer on fewer than 1% of the tested AffectNet images. Average results can hide these large differences.
  • Steering is mainly evaluated on images the model initially recognizes correctly, right? This means the experiments largely examine whether emotion vectors can change the interpretation of faces the model already understands successfully. They provide less evidence about what happens when the model gives an incorrect answer. This matters because steering may behave differently in those cases. A vector could produce a strong change on clearly recognized expressions but have weaker or less predictable effects when the visual evidence is ambiguous.
Questions for authors
  • 'For each model and source, we use residual activations at approximately two-thirds of the model’s depth unless otherwise stated.' Is there a reason for this?
  • When the model changes its emotion answer, are the facial features it describes actually present in the image?
  • Are the effects similar for ambiguous expressions and images where the model’s initial emotion prediction is wrong?
  • What explains the differences between RAF-DB and AffectNet? Can the authors comment on it

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.