- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper presents claims of a shared emotional representation in multimodal models and utilises three kinds of evidence that accuracy alone cannot separate: (i) behavioural, (ii) representational, and (iii) interventional. Each type of evidence was tested separately with 6 different open-weight models, 1,411 EmoBench-M conversational items, and 240 CREMA-D clips. The major findings are, transcripts works well midpoint frames by 4.82–12.69 points in three VLMs, surviving two label-prior corrections, but MER2023 reverses the direction. Text-related transfer to faces depends on how the modality offset is handled. Also, the text mean only collapses to one class; modality centring gives above-zero transfer in Qwen2.5-VL-7B, but item-level probes show no significant transfer. Unsupervised co-activation modules give no significant improvement across 90 conditions in three text models, whereas a DiffMean direction under the identical judge, gate, prompts, and seeds produces large classifier-judged gains. Also, a text-derived joy direction shifts image-only judgments by 30.07 and 36.71 points.
Critical strengths
-
The statistical practice is unusually careful, and the controls are the right ones. Repair empty space rather than chance baselines, Holm families declared per analysis, group-aware bootstraps over actors and dialogues. A protocol fixed before the added experiments ran presents the design detects 4–10 points. The norm-matched random direction in Qwen3-VL-32B it raises sadness by 26.04 points, which would have been read as a steering success without it.
-
The quality audit is a valuable contribution and generalises beyond this paper. Showing that a likelihood gate passes 9,502 outputs, of which 34.92% are degenerate. That is 70.67% of gate-passing steered outputs fall below “mostly readable,” and a blinded rater named the target emotion in 2 of 54 readable continuations. This is a direct warning about how steering is currently measured.
Critical weaknesses
-
This experiment is motivated by reported emotion circuits from “Do LLMs "Feel"? Emotion Circuits Discovery and Control” and cross-modal emotion neurons from “Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Models”, but the modules actually tested used an unsupervised co-activation objective that Appendix A says differs from the original. Also, the modules were never selected for emotion steering in the first place. But I was missing label imbalance issues, and also 9 of 15 targets are anger. I think the paper would be stronger if it reproduced two cited circuit methods under its matched pipeline.
-
The transfer claim holds for one model under the prespecified readout (Qwen3-VL-32B at Holm p = 0.116) and for the other only under the z-scoring sensitivity readout (with item-level probes showing nothing, and the strongest transfer is 16.51%, which is below the 18.07% majority rate). The steering claim rests on a classifier whose agreement with the human rater is 0.228 and 0.081, with the rater naming the target in 2 of 54 and later 0 of 24. Absolute accuracies of 31–39% on 7 classes further limit what the behavioural comparison can carry. What survives is mainly the methodological warning, which is useful but narrower than the three-part framing implies.
Questions for authors
-
Can you implement these two (or at least one) emotion-circuit methods (“Do LLMs "Feel"? Emotion Circuits Discovery and Control” and “Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Models”) under the matched pipeline?
-
Given κ = 0.228/0.081 between rater and classifier, and 2/54 then 0/24 target identifications, is the DiffMean result better stated as “shifts an automated judge” than as steering? What would it take to call it steering and why?
-
The prespecified centred readout is significant in one model and the z-scored sensitivity readout in both. Which do you take as the finding, and was standardisation specified before seeing the centred result?
-
One unpaid volunteer rater, approx. 12–16 s per item, with repeated items as the only reliability check. What prevents these counts from being read as rate estimates, and can more raters be added?
-
MER2023 reverses the transcript advantage and survives prior removal. Is the likely cause label composition, translation quality, or visual observability and can an item-level annotation separate them?
-
The classifier is trained partly on MELD, one of your three sources. How much does this affect the recognition comparison?
3 comments
Response to Reviewer CsiV (2/3)
Weakness 2: Absolute accuracies of 31–39% on 7 classes further limit what the behavioural comparison can carry.
Weakness 2: What survives is mainly the methodological warning, which is useful but narrower than the three-part framing implies.
The behavioral claim is comparative. Transcript accuracy is 31.47–39.33% against a 14.29% chance rate and an 18.07% majority rate, and on the same 1,411 items transcripts beat frames by 4.82, 12.69, and 12.62 points (Holm p ≤ 0.0017; Table 3); the gaps survive label-free prior removal and content-free calibration (Section 4). Absolute accuracy limits claims about how well the models recognize emotion, not about which input carries more evidence. Beyond the methodological warning, three results stand: this paired input comparison, class-level transfer in Qwen2.5-VL-7B under the prespecified readout, and a causal cross-modal effect, in which a text-derived joy direction raises the probability of labeling a face as joy by 30.07 and 36.71 points with no classifier in the loop (Section 6.3).
Question 2: Given κ = 0.228/0.081 between rater and classifier, and 2/54 then 0/24 target identifications, is the DiffMean result better stated as “shifts an automated judge” than as steering?
Question 2: What would it take to call it steering and why?
The paper describes these effects as classifier-judged (Abstract; Sections 6.2 and 7), and the revision will use this wording throughout, including the heading of Section 6.2 and the caption of Figure 3. The target was named in 2 of the 54 DiffMean continuations, the 30 and 24 rated in the two sessions (Appendix G). Steering would require raters to judge steered continuations as expressing the target more than baseline continuations of the same prompt and seed, with readability held. This is the criterion the paper sets for steerability, an increase in the intended emotion over an unsteered baseline without larger off-target shifts or loss of readability (Section 1), applied to human readers, for whom the text is generated. The pairwise study below asks raters which of the two continuations expresses the named emotion more, and the rating booklets record readability.
Question 4: One unpaid volunteer rater, approx. 12–16 s per item, with repeated items as the only reliability check. What prevents these counts from being read as rate estimates, and can more raters be added?
Appendix G states that the counts are "a qualitative check rather than rate estimates", and the revision will repeat this wherever they are cited. The repeats were not the only check: all 8 received identical answers, readability matched the AI-assisted review on 11 of 12 shared outputs (Cohen's κ = 0.875), and the session-2 outputs rated unreadable were consistent with the rule-based screen (Appendix G). Each continuation has at most 64 new tokens (Section 3.3), and the new booklets record the time spent on each item. We will add at least two further blinded raters on the same case sets with Krippendorff's α, plus a pairwise comparison of steered and baseline continuations of the same prompt and seed on 120 new pairs covering all five models and six targets, and we will include those experimental results in the camera-ready version.
Response to Reviewer CsiV (3/3)
Question 5: Is the likely cause label composition, translation quality, or visual observability and can an item-level annotation separate them?
MER2023 has no fear or disgust label, and its "worried" items (57 of 411) were mapped to fear, which the revision will state. The mapping does not cause the reversal: without these items, the transcript-minus-frame gap moves toward frames in all three models under all three scoring rules (raw: frame advantages of 9.60, 6.78, and 4.80 points, against 6.81, 3.89, and 2.19 with them), because the worried items themselves favor transcripts. Under raw scoring, the point estimates within MER2023 favor frames for joy, sadness, and surprise and transcripts for anger in all three models, which points to label composition and visual observability. Translation alone is an unlikely cause: with trained features, visual leads lexical on MER2023 (average WAF 65.52 versus 55.35) but trails it on MELD (40.03 versus 58.27) [1], the same source dependence we observe zero-shot. For visual observability, we will annotate a stratified MER2023 subset for whether the frame shows the labeled emotion and whether the transcript states it. We will include those experiment results in the camera-ready version.
Question 6: The classifier is trained partly on MELD, one of your three sources. How much does this affect the recognition comparison?
It does not enter the recognition comparison, which scores answer log-probabilities under each vision-language model (Section 3.1); the classifier only judges generated continuations, whose prompts come from five other EmoBench-M subsets (Appendix D). For the steering results, we re-judged all 34,560 saved continuations with a second classifier trained only on GoEmotions, which excludes MELD. The judges agree with Cohen's κ = 0.237 [0.198, 0.277]. Of the 15 Holm-significant gains, 7 persist: the DiffMean joy and surprise gains in Qwen3-32B (29.17 and 34.64 points) and Qwen3-VL-32B (39.32 and 35.68), the Qwen3-32B anger gain (5.73), the Qwen2.5-72B surprise gain (11.20), and the Qwen3-VL-32B fear gain (5.21); the second judge adds one, DiffMean joy in Llama-3.3-70B (8.33). The other 8 do not persist: the DiffMean disgust gains in Qwen3-32B and Qwen3-VL-32B, the DiffMean anger gains in Qwen2.5-72B and Qwen3-VL-32B, the DiffMean joy gain in Qwen2.5-72B, and the Qwen3-VL-32B module-1 joy gain and random-direction anger and sadness gains, the last three consistent with the quality audit (Section 6.4). The revision will report both judges.
References
[1] Lian, Z., Sun, L., Ren, Y., Gu, H., Sun, H., Chen, L., Liu, B., & Tao, J. (2024). MERBench: A unified evaluation benchmark for multimodal emotion recognition (arXiv:2401.03429). arXiv. https://doi.org/10.48550/arXiv.2401.03429
Response to Reviewer CsiV (1/3)
Thank you for recognizing the statistical controls and the quality audit. This part answers Weakness 1, the transfer point of Weakness 2, and Questions 1 and 3; Part 2 answers the rest of Weakness 2 and Questions 2 and 4; Part 3 answers Questions 5 and 6.
The modules test a different question from the two cited circuit methods: whether units found through unsupervised activation structure, rather than labels or causal search, can serve as steering handles (Section 1). The "original" in Appendix A is the IterD objective from which our variant departs, and by design the modules were not filtered for steering efficacy. Each module targets the most common non-neutral label among its texts (Section 3.3), and anger is the most frequent label in the 901 discovery texts (179, against 68 to 177 for the other labels), so this plurality rule favors anger. Anger itself is steerable under the same pipeline: the DiffMean anger direction raises the gated anger rate by 10.68 and 4.69 points in Qwen3-32B and Qwen2.5-72B (Holm p = 8.5 × 10⁻⁴ and 0.032), while the anger modules change it by −3.65 to 0.78 points (Table 18); under the second judge in Part 3, the Qwen3-32B anger gain persists. The modules themselves are not anger-specific: in the nine anger-target modules, the anger share is 0.92 to 1.91 times its share in the discovery texts, and only one module, with 50 texts, exceeds its permutation null after Holm correction (lift 1.91 [1.21, 2.62]). A rule that normalizes by these base rates would assign surprise to 6 of the 15 text-model modules, disgust to 4, anger to 4, and joy to 1; the revision will report both rules with each module's label composition.
We will implement both methods under the matched pipeline. For Zhao et al. [1], we will select emotion-sensitive neurons and apply them in generation and in the CREMA-D cross-modal test. For Wang et al. [2], we will derive last-token emotion directions as mean differences of residual activations, the same kind of direction as our DiffMean baseline, select the neurons and attention heads aligned with them, and modulate the resulting circuit in generation and in the CREMA-D cross-modal test. Both will use the paper's discovery texts, development rule, prompts, seeds, judge, and gate, with same-size random-unit controls, and we will include those experiment results in the camera-ready version.
The finding is the prespecified modality-centered readout, under which transfer is supported for Qwen2.5-VL-7B (32.92%, Holm p = 0.004); Qwen3-VL-32B (21.25%, Holm p = 0.116) exceeds its null only under the z-scored sensitivity readout (Section 5). Both readouts were fixed before any centered result existed, in row A1 of the frozen protocol (supplementary/added_experiments/protocol/PROTOCOL.md), and one script computed all readouts in one run; the abstract will label the z-scored result as a sensitivity analysis. The transfer claim is class-level: fixed linear probes find no significant item-level transfer, as Section 5 states. The 16.51% probe accuracy is measured on seven imbalanced conversational classes, where it is tested against both a shuffled-label null (Holm p = 0.068) and the 18.07% majority rate, whereas centroid transfer is measured on six balanced CREMA-D classes against refitted label-permutation nulls (Section 3.2).
References
[1] Zhao, X., Sun, L., Schuller, B., & Sisman, B. (2026). Emotion across speech and faces: Shared affective mechanisms in multimodal foundation models (arXiv:2608.17102). arXiv. https://doi.org/10.48550/arXiv.2608.17102
[2] Wang, C., Zhang, Y., Yu, R., Zheng, Y., Gao, L., Song, Z., Xu, Z., Xia, G., Zhang, H., Zhao, D., & Chen, X. (2026). Do LLMs "feel"? Emotion circuits discovery and control. In Proceedings of the 43rd International Conference on Machine Learning (Vol. 306, pp. 132595–132613). PMLR. https://proceedings.mlr.press/v306/wang26ky.html