- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies whether emotion representations in VLMs generalize across different sources, modalities and architectures. The authors introduce CMES, covering six basic emotions across text, real facial expressions and synthetic images. They extract emotion vectors from three VLMs and examine their geometry, alignment and transferability through activation steering. They also construct a shared emotion subspace and evaluate it on held-out Gemma-3 models. The results suggest that emotion representations share some common structure across models, although they vary with the stimulus source and that aligned vectors can transfer steering effects across modalities and architectures.
Critical strengths
-
The research question addressed in the work (consistency of emotion representations across modalities, stimulus sources and model architectures) is interesting.
-
The experimental evaluation (e.g., three core VLM architectures; multiple text/image sources; additional held-out Gemma-3 models; cross-architecture transfer experiments; inclusion of activation steering) is quite extensive and in my view constitutes the core contribution of this work.
Critical weaknesses
The paper's methodological novelty is low (techniques are well-established; the shared emotion subspace is constructed using relatively straightforward alignment and averaging operations). As mentioned in the strengths section above, the core contribution of this work is the experimental evaluation.
Although generally an empirical study can be valuable, the paper does not sufficiently demonstrate what new insights are obtained beyond prior work on emotion representations, cross-model alignment and steering transfer. A dedicated Related Work section, which is currently missing, would help in that direction. Currently, most of the discussion is deferred to Appendix A. Given the substantial existing literature, a more explicit comparison with previous approaches is needed to better establish the contribution.
Additionally, the paper requires substantial restructuring and clearer presentation. The main paper is very dense, with numerous experiments and findings compressed into a limited space, while important methodological explanations and experimental details are repeatedly deferred to the appendix. Section 2 is particularly difficult to follow, as the emotion-vector extraction procedure is introduced with little explanation of the individual steps or their motivation. I would encourage the authors to reconsider the organization of the paper, provide a clearer introduction to the methodology and focus the main text on the most important experiments and findings, with sufficient detail to understand and assess them without repeatedly referring to the appendix.
In terms of the experiments, one of my main concerns is the interpretation of the activation steering results. The authors argue that emotion vectors causally influence how VLMs interpret emotions in images. However, the reported interventions may instead be introducing a general bias toward generating a particular emotion label. Appendix E.6 shows that steering also affects predictions concerning the assistant and a third party who is not present in the image. Furthermore, Table 20 shows that injecting emotion vectors at answer-prefix positions produces effects comparable to those obtained at image-token positions. These results raise questions about whether the interventions genuinely modify the model's interpretation of the visual evidence or primarily influence its subsequent response. Although the authors acknowledge these effects (and that is a very positive thing in my view and I appreciate it), I do not think their implications are sufficiently discussed in relation to the central claims.
Additionally, the average steering results hide substantial differences between individual emotions. The main paper emphasizes the average increases in target-emotion log probabilities following steering. However, the results are considerably less convincing for some emotion categories. For example, Appendix E.4 reports that disgust steering changes the generated emotion readout to the intended target on only 32 out of 3,500 AffectNet validation images, less than 1%, while the other five emotions show success rates of 14-60%. This is a substantial discrepancy. It indicates that an increase in target-emotion log probability does not necessarily translate into successful changes in predictions or generated responses. It would be good if the paper has a clearer per-emotion evaluation and a discussion of these failures, rather than relying primarily on aggregate steering metrics.
I also have some concerns regarding the LOEO evaluation. The authors use LOEO alignment to evaluate whether mappings fitted on five emotions generalize to the sixth. However, the emotion vectors are constructed by subtracting the mean across all six emotion categories, introducing a linear dependence among them before normalization. Consequently, the held-out emotion representation is not completely independent of those used to fit the transformation. This raises questions about how stringent the LOEO evaluation is and to what extent the observed correspondence is influenced by the shared construction of the emotion vectors. The permutation controls are useful and the authors acknowledge this limitation in Appendix B.7. Nevertheless, it would be good to see a more stringent evaluation involving genuinely unseen emotion categories or a procedure that excludes the held-out category when computing the reference statistics.
Another concern relates to the findings in Section 3.2, where the authors report that RAF-DB produces a different emotion geometry, with disgust closer to happiness and fear closer to anger, unlike the patterns observed for AffectNet and the other sources. The disgust - happiness relationship is particularly surprising. The retrieval analysis in Appendix C.2 provides additional evidence of this behaviour, but does not establish why it occurs. Annotation ambiguity, differences in class composition, expression intensity, and dataset-specific visual characteristics are plausible explanations. Although the authors provide additional retrieval analyses and investigate the effects of filtering, these experiments do not fully explain the observed differences. This is important because the conclusion that stimulus source has a greater influence on emotion geometry than architecture is one of the central findings. It would be good to see a more thorough investigation of the factors responsible for these differences, rather than primarily reporting them as evidence of source-dependent emotion representations.
Finally, the VA interpretation and low-dimensionality claims require greater caution. The authors report that the first two principal components explain approximately 55-75% of the variance and that these components exhibit some agreement with human V and A ratings. However, PCA does not explicitly identify affective dimensions and the correlations are calculated using only six emotion categories. Furthermore, while the agreement with V is relatively strong, the agreement with A is substantially weaker and varies considerably across sources. Since the vectors are constructed from six centered categories, their rank is at most five. Explaining a majority of the variance with two components is thus less compelling than it might initially appear. I find these results useful as descriptive observations, but I think the authors should be more cautious in interpreting them as evidence of human-like affective representations, particularly given the limited number of emotion categories and the considerable variation in A agreement across sources.
Questions for authors
Please have a look above on the extensive critical weaknesses comments.