acceptodds
Under review as a conference paper at ICLR 2027

Emotion Interpretability in Multimodal Models: Disentangling Behavioral, Representational, and Interventional Evidence

Abstract

Reported emotion circuits controlling expression and emotion neurons shared between speech and faces suggest a shared, steerable emotion representation in multimodal models. This claim bundles three kinds of evidence that benchmark accuracy cannot separate: behavioral (which input carries emotion), representational (whether text-learned emotion is recognized in faces), and interventional (whether it steers generation). We test each separately, comparing transcripts with video frames, classifying faces with emotion prototypes built from transcripts, and steering with neuron groups found without emotion labels or a simple mean-difference direction. Transcripts outperform single frames in three vision-language models by 4.82 to 12.69 percentage points. Encouragingly, text-learned emotion carries over to faces: the prototypes beat a shuffled-label baseline in Qwen2.5-VL-7B and, after feature standardization, in Qwen3-VL-32B, and a text-derived joy direction raises the probability of labeling a face as joy by 30.07 and 36.71 points. At the same time, steering depends on the method and registers more with a classifier than a human reader. Neuron groups give no significant gain across 90 conditions in three text models, whereas the mean-difference direction raises classifier-judged emotion in three of five models, lifting surprise by 69.27 points in Qwen3-32B. A blinded rater judged all 54 sampled mean-difference continuations readable and named the target emotion in 2. We study six open-weight models, 1,411 conversational clips, and 240 acted face clips. Claims of a shared, steerable emotion representation are therefore best tested as three claims, with steering effects confirmed by human readers.

Primary area:
interpretability and explainable AI
Keywords:
emotion interpretability, vision-language models, emotion recognition, cross-modal transfer, activation steering, interventions, linear probing, evaluation validity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CsiVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents claims of a shared emotional representation in multimodal models and utilises three kinds of evidence that accuracy alone cannot separate: (i) behavioural, (ii) representational, and (iii) interventional. Each type of evidence was tested separately with 6 different open-weight models, 1,411 EmoBench-M conversational items, and 240 CREMA-D clips. The major findings are, transcripts works well midpoint frames by 4.82–12.69 points in three VLMs, surviving two label-prior corrections, but MER2023 reverses the direction. Text-related transfer to faces depends on how the modality offset is handled. Also, the text mean only collapses to one class; modality centring gives above-zero transfer in Qwen2.5-VL-7B, but item-level probes show no significant transfer. Unsupervised co-activation modules give no significant improvement across 90 conditions in three text models, whereas a DiffMean direction under the identical judge, gate, prompts, and seeds produces large classifier-judged gains. Also, a text-derived joy direction shifts image-only judgments by 30.07 and 36.71 points.

Critical strengths
  1. The statistical practice is unusually careful, and the controls are the right ones. Repair empty space rather than chance baselines, Holm families declared per analysis, group-aware bootstraps over actors and dialogues. A protocol fixed before the added experiments ran presents the design detects 4–10 points. The norm-matched random direction in Qwen3-VL-32B it raises sadness by 26.04 points, which would have been read as a steering success without it.

  2. The quality audit is a valuable contribution and generalises beyond this paper. Showing that a likelihood gate passes 9,502 outputs, of which 34.92% are degenerate. That is 70.67% of gate-passing steered outputs fall below “mostly readable,” and a blinded rater named the target emotion in 2 of 54 readable continuations. This is a direct warning about how steering is currently measured.

Critical weaknesses
  1. This experiment is motivated by reported emotion circuits from “Do LLMs "Feel"? Emotion Circuits Discovery and Control” and cross-modal emotion neurons from “Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Models”, but the modules actually tested used an unsupervised co-activation objective that Appendix A says differs from the original. Also, the modules were never selected for emotion steering in the first place. But I was missing label imbalance issues, and also 9 of 15 targets are anger. I think the paper would be stronger if it reproduced two cited circuit methods under its matched pipeline.

  2. The transfer claim holds for one model under the prespecified readout (Qwen3-VL-32B at Holm p = 0.116) and for the other only under the z-scoring sensitivity readout (with item-level probes showing nothing, and the strongest transfer is 16.51%, which is below the 18.07% majority rate). The steering claim rests on a classifier whose agreement with the human rater is 0.228 and 0.081, with the rater naming the target in 2 of 54 and later 0 of 24. Absolute accuracies of 31–39% on 7 classes further limit what the behavioural comparison can carry. What survives is mainly the methodological warning, which is useful but narrower than the three-part framing implies.

Questions for authors
  1. Can you implement these two (or at least one) emotion-circuit methods (“Do LLMs "Feel"? Emotion Circuits Discovery and Control” and “Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Models”) under the matched pipeline?

  2. Given κ = 0.228/0.081 between rater and classifier, and 2/54 then 0/24 target identifications, is the DiffMean result better stated as “shifts an automated judge” than as steering? What would it take to call it steering and why?

  3. The prespecified centred readout is significant in one model and the z-scored sensitivity readout in both. Which do you take as the finding, and was standardisation specified before seeing the centred result?

  4. One unpaid volunteer rater, approx. 12–16 s per item, with repeated items as the only reliability check. What prevents these counts from being read as rate estimates, and can more raters be added?

  5. MER2023 reverses the transcript advantage and survives prior removal. Is the likely cause label composition, translation quality, or visual observability and can an item-level annotation separate them?

  6. The classifier is trained partly on MELD, one of your three sources. How much does this affect the recognition comparison?

3 comments
AuthorsOpenReview ↗

Response to Reviewer CsiV (1/3)

Thank you for recognizing the statistical controls and the quality audit. This part answers Weakness 1, the transfer point of Weakness 2, and Questions 1 and 3; Part 2 answers the rest of Weakness 2 and Questions 2 and 4; Part 3 answers Questions 5 and 6.

Weakness 1: Also, the modules were never selected for emotion steering in the first place.

Weakness 1: But I was missing label imbalance issues, and also 9 of 15 targets are anger.

The modules test a different question from the two cited circuit methods: whether units found through unsupervised activation structure, rather than labels or causal search, can serve as steering handles (Section 1). The "original" in Appendix A is the IterD objective from which our variant departs, and by design the modules were not filtered for steering efficacy. Each module targets the most common non-neutral label among its texts (Section 3.3), and anger is the most frequent label in the 901 discovery texts (179, against 68 to 177 for the other labels), so this plurality rule favors anger. Anger itself is steerable under the same pipeline: the DiffMean anger direction raises the gated anger rate by 10.68 and 4.69 points in Qwen3-32B and Qwen2.5-72B (Holm p = 8.5 × 10⁻⁴ and 0.032), while the anger modules change it by −3.65 to 0.78 points (Table 18); under the second judge in Part 3, the Qwen3-32B anger gain persists. The modules themselves are not anger-specific: in the nine anger-target modules, the anger share is 0.92 to 1.91 times its share in the discovery texts, and only one module, with 50 texts, exceeds its permutation null after Holm correction (lift 1.91 [1.21, 2.62]). A rule that normalizes by these base rates would assign surprise to 6 of the 15 text-model modules, disgust to 4, anger to 4, and joy to 1; the revision will report both rules with each module's label composition.

Question 1: Can you implement these two (or at least one) emotion-circuit methods (“Do LLMs "Feel"? Emotion Circuits Discovery and Control” and “Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Models”) under the matched pipeline?

We will implement both methods under the matched pipeline. For Zhao et al. [1], we will select emotion-sensitive neurons and apply them in generation and in the CREMA-D cross-modal test. For Wang et al. [2], we will derive last-token emotion directions as mean differences of residual activations, the same kind of direction as our DiffMean baseline, select the neurons and attention heads aligned with them, and modulate the resulting circuit in generation and in the CREMA-D cross-modal test. Both will use the paper's discovery texts, development rule, prompts, seeds, judge, and gate, with same-size random-unit controls, and we will include those experiment results in the camera-ready version.

Weakness 2: The transfer claim holds for one model under the prespecified readout (Qwen3-VL-32B at Holm p = 0.116) and for the other only under the z-scoring sensitivity readout (with item-level probes showing nothing, and the strongest transfer is 16.51%, which is below the 18.07% majority rate).

Question 3: Which do you take as the finding, and was standardisation specified before seeing the centred result?

The finding is the prespecified modality-centered readout, under which transfer is supported for Qwen2.5-VL-7B (32.92%, Holm p = 0.004); Qwen3-VL-32B (21.25%, Holm p = 0.116) exceeds its null only under the z-scored sensitivity readout (Section 5). Both readouts were fixed before any centered result existed, in row A1 of the frozen protocol (supplementary/added_experiments/protocol/PROTOCOL.md), and one script computed all readouts in one run; the abstract will label the z-scored result as a sensitivity analysis. The transfer claim is class-level: fixed linear probes find no significant item-level transfer, as Section 5 states. The 16.51% probe accuracy is measured on seven imbalanced conversational classes, where it is tested against both a shuffled-label null (Holm p = 0.068) and the 18.07% majority rate, whereas centroid transfer is measured on six balanced CREMA-D classes against refitted label-permutation nulls (Section 3.2).

References

[1] Zhao, X., Sun, L., Schuller, B., & Sisman, B. (2026). Emotion across speech and faces: Shared affective mechanisms in multimodal foundation models (arXiv:2608.17102). arXiv. https://doi.org/10.48550/arXiv.2608.17102

[2] Wang, C., Zhang, Y., Yu, R., Zheng, Y., Gao, L., Song, Z., Xu, Z., Xia, G., Zhang, H., Zhao, D., & Chen, X. (2026). Do LLMs "feel"? Emotion circuits discovery and control. In Proceedings of the 43rd International Conference on Machine Learning (Vol. 306, pp. 132595–132613). PMLR. https://proceedings.mlr.press/v306/wang26ky.html

AuthorsOpenReview ↗

Response to Reviewer CsiV (2/3)

Weakness 2: Absolute accuracies of 31–39% on 7 classes further limit what the behavioural comparison can carry.

Weakness 2: What survives is mainly the methodological warning, which is useful but narrower than the three-part framing implies.

The behavioral claim is comparative. Transcript accuracy is 31.47–39.33% against a 14.29% chance rate and an 18.07% majority rate, and on the same 1,411 items transcripts beat frames by 4.82, 12.69, and 12.62 points (Holm p ≤ 0.0017; Table 3); the gaps survive label-free prior removal and content-free calibration (Section 4). Absolute accuracy limits claims about how well the models recognize emotion, not about which input carries more evidence. Beyond the methodological warning, three results stand: this paired input comparison, class-level transfer in Qwen2.5-VL-7B under the prespecified readout, and a causal cross-modal effect, in which a text-derived joy direction raises the probability of labeling a face as joy by 30.07 and 36.71 points with no classifier in the loop (Section 6.3).

Question 2: Given κ = 0.228/0.081 between rater and classifier, and 2/54 then 0/24 target identifications, is the DiffMean result better stated as “shifts an automated judge” than as steering?

Question 2: What would it take to call it steering and why?

The paper describes these effects as classifier-judged (Abstract; Sections 6.2 and 7), and the revision will use this wording throughout, including the heading of Section 6.2 and the caption of Figure 3. The target was named in 2 of the 54 DiffMean continuations, the 30 and 24 rated in the two sessions (Appendix G). Steering would require raters to judge steered continuations as expressing the target more than baseline continuations of the same prompt and seed, with readability held. This is the criterion the paper sets for steerability, an increase in the intended emotion over an unsteered baseline without larger off-target shifts or loss of readability (Section 1), applied to human readers, for whom the text is generated. The pairwise study below asks raters which of the two continuations expresses the named emotion more, and the rating booklets record readability.

Question 4: One unpaid volunteer rater, approx. 12–16 s per item, with repeated items as the only reliability check. What prevents these counts from being read as rate estimates, and can more raters be added?

Appendix G states that the counts are "a qualitative check rather than rate estimates", and the revision will repeat this wherever they are cited. The repeats were not the only check: all 8 received identical answers, readability matched the AI-assisted review on 11 of 12 shared outputs (Cohen's κ = 0.875), and the session-2 outputs rated unreadable were consistent with the rule-based screen (Appendix G). Each continuation has at most 64 new tokens (Section 3.3), and the new booklets record the time spent on each item. We will add at least two further blinded raters on the same case sets with Krippendorff's α, plus a pairwise comparison of steered and baseline continuations of the same prompt and seed on 120 new pairs covering all five models and six targets, and we will include those experimental results in the camera-ready version.

AuthorsOpenReview ↗

Response to Reviewer CsiV (3/3)

Question 5: Is the likely cause label composition, translation quality, or visual observability and can an item-level annotation separate them?

MER2023 has no fear or disgust label, and its "worried" items (57 of 411) were mapped to fear, which the revision will state. The mapping does not cause the reversal: without these items, the transcript-minus-frame gap moves toward frames in all three models under all three scoring rules (raw: frame advantages of 9.60, 6.78, and 4.80 points, against 6.81, 3.89, and 2.19 with them), because the worried items themselves favor transcripts. Under raw scoring, the point estimates within MER2023 favor frames for joy, sadness, and surprise and transcripts for anger in all three models, which points to label composition and visual observability. Translation alone is an unlikely cause: with trained features, visual leads lexical on MER2023 (average WAF 65.52 versus 55.35) but trails it on MELD (40.03 versus 58.27) [1], the same source dependence we observe zero-shot. For visual observability, we will annotate a stratified MER2023 subset for whether the frame shows the labeled emotion and whether the transcript states it. We will include those experiment results in the camera-ready version.

Question 6: The classifier is trained partly on MELD, one of your three sources. How much does this affect the recognition comparison?

It does not enter the recognition comparison, which scores answer log-probabilities under each vision-language model (Section 3.1); the classifier only judges generated continuations, whose prompts come from five other EmoBench-M subsets (Appendix D). For the steering results, we re-judged all 34,560 saved continuations with a second classifier trained only on GoEmotions, which excludes MELD. The judges agree with Cohen's κ = 0.237 [0.198, 0.277]. Of the 15 Holm-significant gains, 7 persist: the DiffMean joy and surprise gains in Qwen3-32B (29.17 and 34.64 points) and Qwen3-VL-32B (39.32 and 35.68), the Qwen3-32B anger gain (5.73), the Qwen2.5-72B surprise gain (11.20), and the Qwen3-VL-32B fear gain (5.21); the second judge adds one, DiffMean joy in Llama-3.3-70B (8.33). The other 8 do not persist: the DiffMean disgust gains in Qwen3-32B and Qwen3-VL-32B, the DiffMean anger gains in Qwen2.5-72B and Qwen3-VL-32B, the DiffMean joy gain in Qwen2.5-72B, and the Qwen3-VL-32B module-1 joy gain and random-direction anger and sadness gains, the last three consistent with the quality audit (Section 6.4). The revision will report both judges.

References

[1] Lian, Z., Sun, L., Ren, Y., Gu, H., Sun, H., Chen, L., Liu, B., & Tao, J. (2024). MERBench: A unified evaluation benchmark for multimodal emotion recognition (arXiv:2401.03429). arXiv. https://doi.org/10.48550/arXiv.2401.03429

Reviewer_GBQWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates shared controllable emotional representations in multimodal models, proposing the separation of three types of evidence: verification recognition, representation transfer, and intervention manipulation. Experiments show that text-based emotion recognition typically outperforms facial image recognition, though this also depends on the dataset itself. Emotional features learned from text can be transferred to faces but require feature correction. The popular neuron module manipulation yields poor results; simple difference vector machines score highly but are almost imperceptible to humans, indicating the need for manual verification alongside automated metrics.

Critical strengths
  1. Propose a three-tiered testing framework for behavior, representation, and intervention to validate "shared controllable affective representations," providing a standardized research paradigm for multimodal affective explainability studies.

  2. Revealing critical evaluation pitfalls in the field: automatic classifiers tend to overestimate the effects of intervention manipulation, likelihood filtering fails to identify degraded outputs, and emphasizes the necessity of introducing manual verification in intervention experiments.

  3. Rigorous controlled experiments, such as label permutation null hypothesis and random vectors, were employed to confirm that modality distribution biases can lead to false cross-modal transfer experimental results.

Critical weaknesses
  1. Manual evaluation relies solely on one blind bidder, and the evaluation sample is limited. This design did not take into account the subjective differences between different annotators, making it impossible to analyze annotation consistency. It can only make qualitative observations and lacks reliable quantitative statistical results, resulting in insufficient verification of the manual part.

  2. The experiment only covered Qwen and Llama models, and only used static single frames for visual analysis, without using audio and video temporal information. The conclusion has limitations in generalizing to real multimodal scenes.

Questions for authors
  1. The human evaluation is conducted with merely one blinded rater, without measuring inter annotator disagreement and inter rater reliability. Could you elaborate on how your core findings might shift if a multi annotator pipeline with formal agreement metrics is introduced?

  2. Your empirical evaluation is restricted to only Qwen and Llama model families. To what extent do you consider your observations can hold across diverse vision language model architectures from other development lines?

  3. All visual experiments rely on isolated static frames, without incorporating video temporal dynamics and audio signals. Since your study does not conduct comparisons against complete multi modal video audio inputs, what potential gaps might exist between your current results and outcomes obtained from full audio visual inputs?

1 comment
AuthorsOpenReview ↗

Response to Reviewer GBQW

Thank you for highlighting the evaluation pitfalls the paper exposes and its controls against spurious cross-modal transfer.

Weakness 1: Manual evaluation relies solely on one blind bidder, and the evaluation sample is limited.

Question 1: Could you elaborate on how your core findings might shift if a multi annotator pipeline with formal agreement metrics is introduced?

The human rating is a check on the automated outcome; Appendix G reports its counts as qualitative, not as rate estimates. Most findings do not depend on it: recognition, transfer, the probes, the module null, the classifier-judged DiffMean gains, the cross-modal test, and the rule-based screen, which covers every matched-evaluation output (it flags 195 of 11,520 DiffMean outputs, 1.69%, against 1,029 of 11,520 random-direction outputs, 8.93%; Appendix E). Within the design, all 8 items shown twice received identical answers, and readability matched the AI-assisted review on 11 of 12 shared outputs (Cohen's κ = 0.875). Two results rest on the rater: the human readability check and the finding that classifier-judged DiffMean gains are subtle to a human reader (the target was named in 2 of 54 readable continuations). We will add at least two further blinded raters on the same case sets with Krippendorff's α, plus a pairwise comparison of steered and baseline continuations of the same prompt and seed on 120 new pairs covering all five models and six targets, which enlarges the human sample, and we will include those experiment results in the camera-ready version.

Weakness 2: The experiment only covered Qwen and Llama models, and only used static single frames for visual analysis, without using audio and video temporal information.

Question 2: To what extent do you consider your observations can hold across diverse vision language model architectures from other development lines?

The tests require white-box access (pinned, unquantized weights, residual streams, and answer log-probabilities for every label), and we state results per model rather than per family (Limitations). The methodological findings concern the measurement pipeline rather than a model family: one-class readouts under modality offsets, label-prior bias in zero-shot scoring, a likelihood gate that accepts degenerate output, and the gap between classifier and human judgments. Within the tested set, results already differ informatively: under the paper's judge, DiffMean produces large classifier-judged gains in Qwen3-32B and Qwen3-VL-32B, smaller gains in Qwen2.5-72B, and none in Llama-3.3-70B or Qwen2.5-VL-7B, while the module null holds in both Qwen and Llama text models. We will add Gemma 3 for recognition, transfer, and the DiffMean and random-direction steering comparison, and we will include those experiment results in the camera-ready version.

Question 3: Since your study does not conduct comparisons against complete multi modal video audio inputs, what potential gaps might exist between your current results and outcomes obtained from full audio visual inputs?

Single frames are a deliberate choice for the behavioral question, which asks which input carries the evidence: a fixed midpoint frame isolates visual evidence from motion and sound (Limitations). On CREMA-D, frames at 25%, 50%, and 75% of the clip raise accuracy by 2.08 and 5.00 points for the 7B and 32B models (adjusted p = 0.267 and 0.100; Appendix B). The main expected gap is audio: on MER2023, a benchmark with trained features finds audio ahead of visual and lexical features (average WAF 72.79, 65.52, and 55.35) [1], so full audiovisual input would likely raise absolute accuracy, while our results compare channels rather than estimate full-input performance. We will add a native-video condition for the same Qwen models and an audio-capable model for recognition, and we will include those experiment results in the camera-ready version.

References

[1] Lian, Z., Sun, L., Ren, Y., Gu, H., Sun, H., Chen, L., Liu, B., & Tao, J. (2024). MERBench: A unified evaluation benchmark for multimodal emotion recognition (arXiv:2401.03429). arXiv. https://doi.org/10.48550/arXiv.2401.03429

Reviewer_pdW3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a three-tier evaluation framework to rigorously test multi-modal emotion interpretability. Specifically, for behavioral level, it shows that recognition accuracy depends heavily on dataset composition rather than a unified cross-modal mechanism, as text superiority varies by source. For representational level, it demonstrates that cross-modal transfer relies heavily on geometric alignment (such as centering or standardization) and exhibits only weak, non-perfect alignment. For interventional level, it refutes claims about unsupervised neuron modules yielding positive steering gains, exposes flaws in automated evaluation pipelines that accept degenerate text outputs, and reveals that simple steering directions produce subtle shifts rather than natural control. The experiments are extensive.

Critical strengths
  1. Rigorous and Well-Motivated Evaluation Framework: To address the ambiguity in multimodal emotion understanding, the authors propose a comprehensive three-tier evaluation framework that systematically decouples behavioral evidence (which input carries emotion), representational evidence (whether text-learned emotion can be recognized in faces), and interventional evidence (whether internal mechanism interventions can steer generation). This framework effectively challenges the prevailing "shared steerable emotion circuit" hypothesis and provides a new evaluation paradigm for large multimodal models in emotion interpretation.

  2. Rigorous statistical controls. The empirical design is exceptionally thorough. It incorporates advanced controls such as two zero-shot prior calibrations (label-free prior removal and content-free calibration) to counter label bias. For probing and centroid transfer, rather than relying on theoretical random values, the authors refit the entire pipeline across 1,999 permuted labels and employ group- and actor-level Bootstrap confidence intervals. Furthermore, all multiple comparisons strictly apply the Holm-Bonferroni correction, complemented by statistical power simulations (calculating minimum detectable effects at 80% power) and a clear pre-specified analysis protocol.

  3. The study delivers an extensive set of experiments across six open-weight models (spanning 7B to 72B parameters), 1,411 conversational clips (EmoBench-M), and 240 acted face clips (CREMA-D across 88 actors), offering rich data to support its conclusions.

Critical weaknesses
  1. The presentation is technically detailed but difficult to follow. Although the paper provides extensive methodological and statistical details, the central insights are sometimes difficult to extract from the large number of experimental conditions, controls, and reported results. Readers must often synthesize the implications across multiple sections and appendices themselves. The paper would benefit from a more concise, higher-level synthesis explaining what each experiment establishes, how the findings relate to one another, and what the combined evidence implies for the central hypothesis.

  2. Figure 1 is visually crowded and does not clearly distinguish the three evaluation pipelines. The figure combines multiple datasets, model operations, representation-processing steps, interventions, and outcome measures in a relatively compact layout. The distinction between inputs, models, internal representations, experimental procedures, and outputs is not always immediately apparent. Separating the behavioral, representational, and interventional pipelines into clearly delineated subfigures or visually distinct sections would make the overall experimental design easier to understand.

Questions for authors

1.Conceptual Design Principles: Could the authors elaborate further on the foundational design principles connecting the three evaluation tasks? Specifically, what makes this tripartite separation (recognition, cross-modal transfer, and steerability) uniquely necessary and sufficient for evaluating emotion understanding compared to other multimodal reasoning tasks?

  1. Choice of Feature Representations in Representational Transfer (Section 3.2): In Section 3.2, the authors extract the final hidden states averaged over all input tokens for text-centroid transfer and linear probes. Given that recent interpretability literature frequently highlights that middle-to-late layers or the final token often capture task-specific semantic alignment more cleanly—whereas the final layer is often heavily distorted by unembedding priors—have the authors experimented with intermediate layers (such as layer 21 or layer 48 used for DiffMean steering) or last-token representations? Would item-level linear probes demonstrate significant cross-modal transfer under those alternative feature extractions?
1 comment
AuthorsOpenReview ↗

Response to Reviewer pdW3

Thank you for the positive assessment of the three-tier framework, the statistical controls, and the breadth of the experiments.

Weakness 1: The paper would benefit from a more concise, higher-level synthesis explaining what each experiment establishes, how the findings relate to one another, and what the combined evidence implies for the central hypothesis.

We will open the results with a synthesis table that maps each question to its experiment, control, result, and implication, and Section 7 will follow it. In brief, transcripts lead recognition in aggregate while MER2023 favors frames; class-level transfer appears once modality offsets are handled (Qwen2.5-VL-7B under the primary readout, both models after standardization), without item-level transfer; and a DiffMean direction registers with the classifier while co-activation modules do so only in Qwen3-VL-32B, via nonlinguistic output; the blinded rater rarely names the target. The claim of a shared, steerable representation thus decomposes into parts that hold to different degrees: a text-learned emotion signal reaches vision, whereas steering of generation depends on the method and is largely classifier-judged.

Weakness 2: Figure 1 is visually crowded and does not clearly distinguish the three evaluation pipelines.

We will redraw Figure 1 as three separated panels, one per question, each reading from input to model to readout to outcome, with the shared validity checks in one strip below and dataset details moved to the caption.

Question 1: Specifically, what makes this tripartite separation (recognition, cross-modal transfer, and steerability) uniquely necessary and sufficient for evaluating emotion understanding compared to other multimodal reasoning tasks?

The separation follows the structure of the claim under test. A "shared, steerable emotion representation" asserts that behavior uses emotion evidence from an input, that a code learned in one modality is read out in another, and that manipulating the code changes outputs as intended: what a model does, what can be decoded from its states, and what it causally uses. Decodability alone does not establish use [1], so each tier needs its own control [2]. The tiers also dissociate empirically: in Qwen2.5-VL-7B, centered transfer is above its null (Holm p = 0.004) and a text-derived joy direction shifts image judgments by 30.07 points, yet no selected method raises its gated generation target after correction. The separation is therefore necessary; we do not claim it is sufficient for emotion understanding broadly, and the same decomposition can be applied to other concepts whose evidence arrives through several inputs.

Question 2: Would item-level linear probes demonstrate significant cross-modal transfer under those alternative feature extractions?

The analyses use the final-layer state averaged over input tokens (Section 3.2). Intermediate layers already supply one piece of cross-modal evidence: the DiffMean directions come from text at layers 21 and 48 of Qwen2.5-VL-7B and Qwen3-VL-32B, respectively, and adding the joy direction while each model classifies faces raises the probability of joy by 30.07 and 36.71 points (Section 6.3, Table 20). We will run centroid transfer and item-level probes at regularly spaced layers and at the steering layers, with mean-pooled and last-token features, under the same refitted nulls and Holm correction, and we will include those experiment results in the camera-ready version.

References

[1] Belinkov, Y. (2022). Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1), 207–219. https://doi.org/10.1162/coli_a_00422

[2] Hewitt, J., & Liang, P. (2019). Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 2733–2743). Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1275

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the AC and All Reviewers

We thank the reviewers for their careful reading. They describe the three-tier framework as rigorous and well motivated (pdW3) and as a standardized research paradigm (GBQW); the statistical practice as unusually careful, with the right controls (CsiV, pdW3); and the quality audit as a contribution that generalizes beyond this paper (CsiV).

The module null in the text models is informative because a mean-difference baseline succeeds under the same judge, gate, prompts, and seeds, and anger itself is steerable under the same pipeline (CsiV). Under the prespecified centered readout, transfer is supported for Qwen2.5-VL-7B (Holm p = 0.004); the z-scored readout is a sensitivity analysis fixed before the run; and a text-derived joy direction shifts face judgments by 30.07 and 36.71 points in Qwen2.5-VL-7B and Qwen3-VL-32B without any classifier in the loop (CsiV, pdW3). The single blinded rater is a check on the automated outcome, not a source of rate estimates (GBQW, CsiV). A second judge not trained on MELD confirms 7 of the 15 Holm-significant steering gains, including the DiffMean joy and surprise gains in Qwen3-32B and Qwen3-VL-32B, and the MER2023 reversal is not caused by mapping its worried items to fear (CsiV).

We will add further blinded raters with Krippendorff's α and a pairwise steered-versus-baseline comparison, both cited circuit methods under the matched pipeline, a vision-language model from another developer (Gemma 3), a native-video condition and an audio-capable model, layer-wise and last-token transfer analyses, and item-level annotation of MER2023. We will include those experimental results in the camera-ready version, together with a synthesis table, a redrawn Figure 1, consistent "classifier-judged" wording, the label composition of the discovery texts and text-model modules, both steering judges, and the disclosure that MER2023's "worried" items were mapped to fear.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.