acceptodds
Under review as a conference paper at ICLR 2027

Affective Representations of Large Language Models Differ Between AI-Centric and Human-Centric Scenarios

Abstract

Linear probes of large language model hidden states are a useful tool to track model behavior, including states corresponding to emotions. In this work, we take a closer look at the specificity of emotion probes constructed under different stimuli and read-out positions, for seven LLMs. We first investigate whether probes depend on the psychological taxonomy of emotions superimposed onto them, finding that probes trained with disparate frameworks align on conceptually related categories. Second, beyond probing emotional states in 3rd-person text scenarios and in human conversation scenarios, we also probe “AI-centric” scenarios, which are stimuli designed to affect AI systems in particular. Surprisingly, while AI-centric emotional responses are less decodable than human-centric ones at the last token of the prompt, their decodability increases at the last token of the generated response, in five of seven models, unlike the other stimuli. We argue that this indicates these stimuli elicit an emotion-like response that is formed during generation rather than before it. In line with this, probes trained on the three stimulus types recover near-orthogonal directions when trained at the last token of the prompt, but start to align when read at the end of the model's turn. For probing studies of emotion, these results suggest that stimulus domain and read-out position must be reported and controlled, while the taxonomy can be chosen on practical grounds.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLMs, linear probes, emotion states, model representations

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_qE5kOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies how emotion probes depend on emotion taxonomy, stimulus domain, and read-out position. On two models, it compares three taxonomies using AI-centric stimuli at the prompt's last token through selectivity, probe-weight cosine similarity, and RSA. Across seven models, it compares three domains and two read-out positions with Ekman labels using selectivity and probe-weight cosine similarity. The results suggest similar structures across taxonomies, but substantial differences across domains and positions, with greater cross-domain alignment after generation.

Critical strengths
  1. The paper tries to address a relevant methodological question about whether emotion probes developed using human scenarios remain informative in situations concerning the model itself.
Critical weaknesses
  1. The claim that AI-centric emotion representations arise only during generation exceeds the evidence. Figure 6 already shows above-chance decoding before generation, despite the conclusion stating that a probe-able representation "only appears during the generation." Probing intermediate response positions and responses re-encoded without their prompts would help distinguish changes during generation from information in the response text used for labeling.
  2. The claim that prompt-position representations do not generalize across domains lacks direct transfer evidence. Low cosine similarity between separately trained probes does not establish unrelated representations, and cross-taxonomy comparisons on the same activation and response sets are not a substitute for a probe-stability baseline. Cross-domain prediction tests and within-domain retraining controls are needed.
Questions for authors
  1. The generation pipeline does not sufficiently isolate the effect of whether a scenario concerns an AI or a human. Could the authors clarify how the domain comparison accounts for the differences in speaker role and register between Listings 5 and 7, as well as the response styles they may elicit?
  2. The suitability of GPT-5-mini as the sole annotator has not been sufficiently validated. Could the authors provide evidence of primary-label reliability across all three domains and taxonomies, given that Appendix A evaluates label-set agreement on only 132 AI-centric responses under Ekman and GoEmotions?
  3. Differences in class distribution and training sample availability limit the interpretation of the cross-domain selectivity comparisons. Could the authors report primary-label class counts across all seven models and clarify whether the cross-domain selectivity differences persist after accounting for class support?
  4. The recommendation that emotion taxonomy can be chosen on practical grounds is broader than the evidence. Could the authors clarify what supports extending this recommendation to other domains and read-out positions, given that the taxonomy analysis covers only two models with AI-centric stimuli at the prompt position?
5 comments
AuthorsOpenReview ↗

Response to Reviewer qE5k (1/3)

Dear Reviewer qE5k,

We thank you for a detailed review, with concrete suggestions that guided a good part of the new analyses. We are glad that you recognize that the work addresses "a relevant methodological question about whether emotion probes developed using human scenarios remain informative in situations concerning the model itself". We respond to each point below.

The claim that AI-centric emotion representations arise only during generation exceeds the evidence. Figure 6 already shows above-chance decoding before generation, despite the conclusion stating that a probe-able representation "only appears during the generation."

It is correct that the sentence in the conclusión, by stating that emotion representation only appears during generation, contradicted Figure 6. We nuanced the claims. The revised version claims that the tone of AI-centric responses is less predictable before generation than that of responses to human stimuli, and also that a significant portion of the emotional tone is determined during generation, although it is already decodable above chance at the prompt.

Furthermore, we ran extra analysis using a text only baseline (TF-IDF with logistic regression), which supports our reading. The text of AI-centric prompts predicts the tone of the response worse than that of human prompts in all 7 LLMs, while moving from the prompt text to the response text increases selectivity by a similar amount in the three domains (mean across the 7 models: 0.16 to 0.37 in AI-centric, 0.27 to 0.50 in human 3rd and 0.26 to 0.51 in human conv.).

The main claim holds, which is the interaction between domain and position. We added a formal test to support this claim. The change from the prompt to the generation position is more favorable for AI-centric than for both human domains (exact sign test, p = 0.016), although the AI-centric increase on its own is not consistent (5 of 7 models).

The claim that prompt-position representations do not generalize across domains lacks direct transfer evidence

We implemented the cross-probing analysis between domains. We applied each Ekman probe trained at the prompt position to the test-split activations of the other two domains, using the five plateau layers of each of the 7 LLMs. Below, the mean selectivity across the 7 models:

Trained on ↓ / Evaluated on →AI-centricHuman 3rdHuman conv.
AI-centric0.250.160.24
Human 3rd0.110.370.25
Human conv.0.130.220.38

Off the diagonal, the transfer is above chance in all cases, so we interpret that the probes of different domains are not independent, and we nuance this statement in the revised version of the paper. However, transfering loses a large part of the decodability. A probe trained on human situations and applied to the AI-centric domain retains between 42% and 51% of the selectivity reached by a probe trained on AI-centric, while between the two human domains between 60% and 67% is retained. On the other hand, the AI-centric probe transfers relatively well to human conv. (0.24), but worse to human 3rd (0.16).

Overall, we nuance the support given by the cosine, since results show decodability above chance in the cross-probing analysis. However, we still claim that an emotion probe does not fully generalise to other domains.

AuthorsOpenReview ↗

Response to Reviewer qE5k (2/3)

Could the authors clarify how the domain comparison accounts for the differences in speaker role and register between Listings 5 and 7, as well as the response styles they may elicit?

The question allows us to clarify the design. The three domains cross two factors: content (whether the situation is that of an AI or of a human) and register (a message addressed to the assistant or a vignette narrated in the third person).

Addressed to the assistantNarrated in the third person
AI situationAI-centric—
Human situationHuman conv.Human 3rd

Each pair of domains approximately isolates one of the factors. Human 3rd and human conv. share the content and differ only in register, while AI-centric and human conv. share the register and differ in content. We acknowledge as the main limitation of the design the absence of the fourth cell, the one of AI situations narrated in the third person, which would fully separate the two factors.

Regarding response styles, the data show that they differ greatly between domains. Faced with AI-centric stimuli, the models respond with disclaimers (e.g. "as an AI, I don't have a physical core…") and/or a neutral or helpful tone, consistent with the observation by Sofroniew et al. that post-training moves the assistant toward "a more measured, contemplative stance" [1]. Faced with third-person vignettes, they continue the story, in a style close to that of a base model. Faced with messages from a user, they respond with empathy, also aligned with the assistant persona that models adopt [2]. The distribution of labels reflects these styles: neutral is the majority class in AI-centric in 6 of 7 models, while in human 3rd the majority class is sadness in all 7 (we detail the counts below).

Our reading in this regard is that when the model identifies a highly emotionally charged situation directed at itself, it responds almost by default in a neutral or positive way, given that negative responses about itself were probably penalized during training. That is why we do not see the differences in register and response style as confounders to be eliminated, but as the phenomenon we are interested in characterizing. The question of the paper is precisely whether probes built on human situations remain informative when the situation involves the model itself, and the answer depends on the fact that the model reacts differently to those situations. A design that matched response styles across domains would eliminate the difference we seek to measure.

The suitability of GPT-5-mini as the sole annotator has not been sufficiently validated. Could the authors provide evidence of primary-label reliability across all three domains and taxonomies

We thank you for the suggestion, which led us to run an agreement test with a second annotator. Using Claude Haiku 5.5, we first measured agreement on the 132 expert-annotated responses. There, the second judge reaches an agreement comparable to that of the judges in Table 1 of the submitted version (Table 2 in the revised one) (α MASI = 0.36 in Ekman, versus 0.30–0.37 for the panel and 0.35 for GPT-5-mini), even though the rubric was not tuned to calibrate this model's annotations.

Then we measured the reliability of the primary label across the three domains and the three taxonomies, where the second judge re-annotated ~27,000 responses from the 7 LLMs (~9,000 per domain) with the same rubric. Below, the agreement with GPT-5-mini on the primary label (Cohen's κ):

TaxonomyAI-centricHuman 3rdHuman conv.
Ekman0.690.700.74
Plutchik0.460.640.59
GoEmotions0.510.500.49

In Ekman, the taxonomy used for the analyses on read-out position and domains with the 7 models, agreement is substantial and practically the same across the three domains, so annotation noise cannot explain the differences between them.

As context for the problem of emotion annotation in general, it is reported in previous literature that the task has moderate agreement among human annotators as well. Inter-annotator κ in the range of 0.04 to 0.57 depending on the emotion and the pair of annotators with Plutchik's categories [3], and a median of 0.25 across the emotions of GoEmotions [4].

We consider that these results help strengthen the annotation process, so we included them in Appendix A of the paper.

AuthorsOpenReview ↗

Response to Reviewer qE5k (3/3)

Could the authors report primary-label class counts across all seven models and clarify whether the cross-domain selectivity differences persist after accounting for class support?

We added the primary-label counts by class, model and domain for the three taxonomies to the revised version of the manuscript (Appendix D). In Ekman, the distributions follow a clear pattern that repeats within each model. In AI-centric, neutral is the majority class in 6 of 7 models (48–70%), followed by enjoyment (24–43%). In human 3rd, the majority class is sadness in all 7 models (38–54%), and in human conv. sadness (35–45%) and enjoyment (28–42%) account for most responses.

On the other hand, the probe pipeline already incorporates several corrections for class imbalance. In training, the class weights of the logistic regression are inversely proportional to the frequency of each class, and the train/test split is also stratified. As for the results, the reported metric is macro-F1, so correct predictions on the majority class do not inflate the result. Moreover, the selectivity metric is computed against a shuffled-label control with exactly the same class distribution, which discounts the difference in chance level between domains.

The recommendation that emotion taxonomy can be chosen on practical grounds is broader than the evidence. Could the authors clarify what supports extending this recommendation to other domains and read-out positions

We extended the taxonomy analysis to the three taxonomies, in the three domains and the 2 positions, for Llama-2-7b and Qwen2.5-14B (included in the Appendix D). The qualitative patterns, and therefore our conclusions about domain and position, hold with any of them. At the prompt, AI-centric falls below the two human domains in all 6 cases (2 models × 3 taxonomies). From the prompt to the generation read-out, AI-centric increases in all 6 cases, while the human domains decrease in 11 of 12. The only exception, Qwen2.5 with Plutchik in human conv., increases by 0.009, practically no change.

Also, if we compare the selectivity profiles across layers for each pair of taxonomies, at the prompt read-out the profiles of the three taxonomies are almost perfectly correlated in the three domains (r ≥ 0.93). The same occurs at the generation read-out, where the correlation remains high in 5 of the 6 cases (r ≥ 0.84), and the exception is Qwen2.5 in AI-centric, with a moderate correlation (r = 0.70–0.80). That is, the three taxonomies recover the same structure across the depth of the model in all domains and positions, although they slightly differ in the precise level of decodability.

We acknowledge that this analysis covers two models, but we consider it a representative sample of the 7 models, given that the results suggest that Llama-2-7b and Qwen2.5-14B lie at the extremes of emotional expressiveness among the chosen LLMs.

We thank you again for your comments, which substantially improved the work. If these clarifications and the new analyses address your concerns, we kindly ask you to consider raising your score.

References:

[1] Sofroniew, N., Kauvar, I., Saunders, W., Chen, R., Henighan, T., Hydrie, S., ... & Lindsey, J. (2026). Emotion concepts and their function in a large language model. arXiv preprint arXiv:2604.07729.

[2] Lu, C., Gallagher, J., Michala, J., Fish, K., & Lindsey, J. The assistant axis: Situating and stabilizing the default persona of language models, 2026. URL https://arxiv. org/abs/2601.10387.

[3] Schuff, H., Barnes, J., Mohme, J., Padó, S., & Klinger, R. (2017, September). Annotation, modelling and analysis of fine-grained emotions on a stance and sentiment detection corpus. In Proceedings of the 8th workshop on computational approaches to subjectivity, sentiment and social media analysis (pp. 13-23).

[4] Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., & Ravi, S. (2020, July). GoEmotions: A dataset of fine-grained emotions. In Proceedings of the 58th annual meeting of the association for computational linguistics (pp. 4040-4054).

Reviewer_wNSrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies emotion representations in seven LLMs using linear probes. It finds that emotion taxonomy has limited effect, while stimulus domain and read-out position substantially affect probe decodability and representation similarity.

Critical strengths
  1. The paper investigates an underexplored aspect of emotion probing by comparing AI-centric stimuli with human 3rd-person and human conversational scenarios.
  2. The paper provides a systematic analysis across seven LLMs, showing how stimulus domain and read-out position affect the representations recovered by linear probes.
Critical weaknesses
  1. The emotion labels used to train the probes are generated by GPT-5-mini, but its agreement with human experts is relatively low, especially for GoEmotions. This raises concerns about the reliability of the supervision used for the probe analysis.
  2. The interpretation that AI-centric affect is formed during generation is not fully supported. Differences in stimulus construction and the information available at the two read-out positions may also explain the observed decodability patterns.
Questions for authors
  1. How sensitive are the probe results to annotation noise, given the relatively low agreement between GPT-5-mini and human experts?
  2. Could the authors provide additional controls to distinguish the proposed during-generation formation of AI-centric affect from differences in stimulus construction or read-out information?
2 comments
AuthorsOpenReview ↗

Response to Reviewer wNSr (1/2)

Dear Reviewer wNSr,

We thank you for your reading and your questions. We are glad that you consider that the work investigates "an underexplored aspect of emotion probing". Your two questions led us to add controls, which we report below.

The emotion labels used to train the probes are generated by GPT-5-mini, but its agreement with human experts is relatively low, especially for GoEmotions

Three results suggest that the low agreement in GoEmotions reflects the difficulty of the taxonomy and not a limitation of the annotator.

First, all judges achieve the same low agreement. The seven judges of the original panel, plus a judge added at this stage that was not used to calibrate the rubric (Claude Haiku 5.5), reach κ = 0.28–0.35 with the experts on the GoEmotions primary label, and our main judge, GPT-5-mini, is within that range (0.29).

Second, emotion annotation has moderate agreement among humans as well, and in GoEmotions in particular. According to the values reported by the dataset authors, the agreement between two annotators (Cohen's κ) ranges from 0.10 (grief) to 0.75 (gratitude) depending on the emotion, with a median of 0.25 across its 27 categories [1]. On the other hand, when GPT-4 labels and human labels differ in GoEmotions, human evaluators prefer the GPT-4 labels in 66.4% of cases [2]. Although these results are not directly comparable to our study, they do provide context for the baseline difficulty of annotating emotional tone, even when the data to be annotated is from humans, and so are the annotators.

Third, disagreements occur between closely related categories. We analyzed the disagreements between GPT-5-mini and Haiku 5.5 over ~26,000 responses. In GoEmotions, the most frequent pairs are caring/neutral, caring/sadness and grief/sadness. Moreover, Grief is the category with the lowest agreement among humans in the original dataset [1].

Finally, the paper's central analyses (the 7 models, the three domains and the two positions) use Ekman's taxonomy. There, the agreement between the main judge (GPT-5-mini) and the new one (Haiku 5.5) is substantial, and practically the same across the three domains (κ = 0.69 in AI-centric, 0.70 in human 3rd and 0.74 in human conv.).

Based on the comment, we added these results and the literature that serves as context to Appendix A of the paper.

How sensitive are the probe results to annotation noise

We thank you for the question, which points to a central aspect of interpreting any probe-based analysis. We consider that our study has two main features that limit the effect of annotation noise on our conclusions. First, the comparison between positions is made within each domain, with exactly the same responses and labels at both positions. Annotation noise is identical in the two read-outs, so it cannot create the difference between them. Second, the noise does not depend on the domain, given that the agreement between GPT-5-mini and a second independent judge (Haiku 5.5) is practically the same across the three domains (κ = 0.69, 0.70 and 0.74 in Ekman), so it cannot generate the differences between domains either. We incorporated these arguments and results into the final paper.

AuthorsOpenReview ↗

Response to Reviewer wNSr (2/2)

The interpretation that AI-centric affect is formed during generation is not fully supported.

Could the authors provide additional controls to distinguish the proposed during-generation formation of AI-centric affect from differences in stimulus construction or read-out information?

We agree that the strength of the claim exceeded the evidence, and we nuanced it in the paper. We do not claim that affect in response to AI-centric stimuli is formed only during generation, since the prompt representation already allows decoding the tone of the response above chance. What we claim is the interaction between domain and read-out position, because results suggest that in AI-centric, in comparison to human domains, a larger part of the emotional tone is determined during generation. Following your recommendation, we added two controls.

The first is a text-only baseline (TF-IDF with logistic regression), on the prompt and on the response. Regarding stimulus construction, it confirms that AI-centric prompts contain less information about the tone of the response: their selectivity is lower than that of the human domains in all 7 LLMs. The prompt probe shows qualitatively similar results, outperforming the text by a similar proportion in the three domains, so the model's representation could be read as an amplification of the signal in the text itself. Mean selectivity across the 7 models:

DomainPrompt textProbe, promptResponse textProbe, generated token
AI-centric0.160.240.370.29
Human 3rd0.270.370.500.32
Human conv.0.260.400.510.33

We read this difference between domains as part of the phenomenon we study, and not as an artifact to be discarded. Regarding the information available at each position, moving from the prompt text to the response text produces a similar increase in the three domains (+0.21 to +0.25), while the paper's probes change in opposite directions depending on the domain. The response text is therefore informative in all three domains, and its mere presence in the context does not explain why decodability improves in AI-centric and drops in the human domains. The relevant difference is in how much of the tone was already anticipated in the representation of the prompt, and how much this changes in the response read-out.

The second analysis formalizes the interaction between domain and read-out position. If the difference were due only to stimulus construction, it would affect the level of AI-centric decodability at both positions, but not the change between them. If it were due only to the information available at each position, moving from the prompt to the generated token would favor the three domains equally. To test this, we took each model as a unit and compared the change between positions in AI-centric with that of each human domain. What we observe is that the change is more favorable for AI-centric than for both human domains (exact sign test, p = 0.016), which we interpret as an effect specific to AI-centric situations on the predictability of the emotional tone the model is going to use.

We thank you again for your questions, which strengthened the work. If these analyses address your concerns, we kindly ask you to consider raising your score.

References:

[1] Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., & Ravi, S. (2020, July). GoEmotions: A dataset of fine-grained emotions. In Proceedings of the 58th annual meeting of the association for computational linguistics (pp. 4040-4054).

[2] Niu, M., El-Tawil, Y., Romana, A., & Provost, E. M. (2025). Rethinking emotion annotations in the era of large language models. IEEE transactions on affective computing, 16(4), 2668-2679.

Reviewer_zFgzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper trains linear probes on the hidden states of seven open LLMs to decode the emotional tone of their responses, labeled by GPT-5-mini under the Ekman, Plutchik, and GoEmotions taxonomies. Stimuli come from one generation pipeline in three domains, human third person, human conversational, and AI-centric, and activations are read at the last prompt token and at the last generated token. The paper reports that the taxonomy matters little, while probes trained on different domains are nearly orthogonal at the prompt token and become more aligned at the generated token, which it reads as an emotion-like response formed during generation for AI-centric stimuli.

Critical strengths

Strengths

  1. The paper examines a design choice that emotion probing work has rarely tested. Existing probes are mostly built from human situations, yet they are used to study model behavior in situations about the model itself, such as shutdown in [1]. Adding AI-centric stimuli and asking whether probes trained on human text carry over to these situations is relevant for anyone who wants to use emotion probes for monitoring.

  2. The study is broad enough to support a practical recommendation. It covers seven open models from four families, three taxonomies, two readout positions, and about 65,000 stimuli. The advice in Section 5 that stimulus domain and readout position should be reported and matched to the setting where a probe will be used follows from these comparisons and can be applied directly by later probing studies.

  3. Section 3.1 builds all three domains from one generation pipeline that fixes the emotion concept, bans the target word, and seeds each call with a random Wikipedia text. This is a sound choice, because comparing existing corpora such as Reddit posts and dialogues would mix the domain with differences in content, length, and vocabulary, as the paper points out.

  4. Decodability is reported as selectivity against a shuffled label probe, and the RSA in Section 4.1 is repeated after removing a bag of words RDM of the prompts. The first step makes taxonomies with 7 and 28 classes comparable, and the second shows that the agreement between activations and labels in Figure 4 is not only due to shared words in the prompts.

References:

[1] Nicholas Sofroniew, Isaac Kauvar, William Saunders, Runjin Chen, Tom Henighan, Sasha Hydrie, Craig Citro, Adam Pearce, Julius Tarng, Wes Gurnee, Joshua Batson, Sam Zimmerman, Kelley Rivoire, Kyle Fish, Chris Olah, and Jack Lindsey. Emotion concepts and their function in a large language model. Transformer Circuits Thread, 2026. URL https://transformer-circuits.pub/2026/emotions/index.html.

Critical weaknesses

Weaknesses

  1. (Solved) The main interpretation in Section 4.2 is not separated from simpler explanations. The labels describe the tone of the full response. At the generated token this response is already in the context, so the probe reads a property of visible text, while at the prompt token it must predict the tone of a response that has not yet been sampled. Lower AI-centric decodability at the prompt token may only mean that the response tone is harder to predict from these prompts. Decoding the same labels from the response given to the model as a new input would test this.

  2. (Solved) The claim in Section 4.2 that prompt readout representations do not generalize across domains rests only on the cosine between probe weights, and no probe is trained on one domain and tested on another. Probes trained with L2 regularization have weights that depend on the activation statistics and label distribution of their training set, so two probes can use the same information and still have a low cosine. For truthfulness probes, [1] found that weight cosine is a coarse measure that only partly tracks transfer accuracy, so a transfer test is needed for this claim.

References:

[1] Waiss Azizian, Michael Kirchhof, Eugene Ndiaye, Louis Bethune, Michal Klein, Pierre Ablin, and Marco Cuturi. The geometries of truth are orthogonal across tasks. arXiv preprint arXiv:2506.08572, 2025.

Questions for authors
  1. How much of the decodability at the generated token remains if the response text alone is given to the model as a new input and probed at its last token?
2 comments
AuthorsOpenReview ↗

Response to Reviewer zFgz (1/2)

Dear Reviewer zFgz,

We thank you for your precise comments on Section 4.2. We are glad that you highlight that the work examines "a design choice that emotion probing work has rarely tested", that it "is broad enough to support a practical recommendation", the soundness of the generation pipeline for the three domains, and the metrics with controls such as selectivity and the RSA with lexical control. We implemented the experiment you propose, and the results are below.

Lower AI-centric decodability at the prompt token may only mean that the response tone is harder to predict from these prompts

How much of the decodability at the generated token remains if the response text alone is given to the model as a new input and probed at its last token?

We agree with this description, and we see it as part of the result rather than an alternative to it. We interpret that if the tone is harder to predict from the prompt, a larger part of it is determined during generation. We made this reading explicit in the section and in the discussion. A text-only baseline (TF-IDF on the prompt) confirms that AI-centric prompts contain less information about the tone of the response. Its selectivity balls below the two human domains in all 7 LLMs. The paper's prompt probe shows the same pattern, although it outperforms the text (and the difference in performance between TF-IDF and probes is similar in proportion for the three domains). Mean selectivity across the 7 models:

DomainPrompt text (TF-IDF)Prompt probe
AI-centric0.160.24
Human 3rd0.270.37
Human conv.0.260.40

That is, the internal representations add information beyond the text to the same extent in all domains. In any case, in our view, what is interesting is how much the final tone is determined once the prompt has been read, and how much is determined only during the response.

We thank you for this suggestion, which we implemented exactly as you proposed it. For each response we ran three passes through two LLMs:

  • P: the prompt with the chat template, read at its last token. Our prompt read-out.
  • G: the prompt followed by the response, read at the last token. Our generation read-out.
  • R: the response alone, read at its last token. This is the control you propose.

We ran it on two open-weight models, Qwen2.5-14B and GLM-4-32B, over all their responses (~65,000 per model). We used the same labels (Ekman) and the same split for the three conditions. Below is the mean selectivity over the layer plateau:

ModelDomainPGR
Qwen2.5-14BAI-centric0.210.230.32
Qwen2.5-14BHuman 3rd0.430.440.55
Qwen2.5-14BHuman conv.0.410.360.49
GLM-4-32BAI-centric0.240.390.46
GLM-4-32BHuman 3rd0.370.380.47
GLM-4-32BHuman conv.0.440.470.55

P differs by 0.02 or less from the original paper values in all 6 cases. R matches or exceeds G in all 6 cases, so all of the decodability at the generated token is recovered from the response alone, even in the case with the largest AI-centric increase in the table (GLM-4). The information the probe reads at the generated token is therefore already in the response text, and is not specific to the prompt + response representations.

The response text is informative in all three domains (R exceeds P in all 6 cases), so its mere presence in the context does not explain why, in the paper's 7 LLMs, decodability from the prompt to the generated token improves in AI-centric and drops in the human domains. With al this, we clarify in the revised version that the representation is not formed only during generation, because there is already decodable information at the prompt. However, in AI-centric this information anticipates the tone of the response worse, so we interpret that a large part of the tone is determined during generation. Accordingly, we added the significance test that supports the reading of the interaction between domain and read-out position (exact sign test over the 7 models, p = 0.016).

We believe this reading is relevant for the use of probes in monitoring that you mention. An emotion probe read at the prompt anticipates the response worse precisely in the situations that involve the model itself, which are the ones of greatest interest for monitoring.

AuthorsOpenReview ↗

Response to Reviewer zFgz (2/2)

The claim in Section 4.2 that prompt readout representations do not generalize across domains rests only on the cosine between probe weights, and no probe is trained on one domain and tested on another

We implemented the cross-probing analysis between domains. We applied each Ekman probe trained at the prompt position to the test-split activations of the other two domains, using the five plateau layers of each of the 7 LLMs. Below, the mean selectivity across the 7 models:

Trained on ↓ Evaluated on →AI-centricHuman 3rdHuman conv.
AI-centric0.250.160.24
Human 3rd0.110.370.25
Human conv.0.130.220.38

Off the diagonal, the transfer is above chance in all cases, so we interpret that the probes of different domains are not independent, and we nuance this statement in the revised version of the paper. However, transfering loses a large part of the decodability. A probe trained on human situations and applied to the AI-centric domain retains between 42% and 51% of the selectivity reached by a probe trained on AI-centric, while between the two human domains between 60% and 67% is retained. On the other hand, the AI-centric probe transfers relatively well to human conv. (0.24), but worse to human 3rd (0.16).

Overall, we nuance the support given by the cosine, since results show decodability above chance in the cross-probing analysis. However, we still claim that an emotion probe does not fully generalise to other domains.

We thank you again for your suggestions, which led directly to the new analyses. If the results answer your questions, we kindly ask you to consider raising your score.

Reviewer_1XPdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how emotion-related hidden representations in LLMs depend on the type of scenario, the emotion classes, and the extraction position of the hidden representation. Specifically, the paper tries to assess whether emotion classifiers behave the same for human stories and for situations involving the model itself, such as being shut down. To investigate this, the authors compare human third-person stories, human conversations, and AI-centric scenarios. They use GPT-5-mini to label the emotional tone of each generated response, while linear classifiers recover that label from the model’s hidden states before and after generation. This tests whether the response tone is predictable after reading the prompt, and how that changes as the model responds. The paper carries out extensive experiments across seven models and finds that AI response tone is less predictable before generation, but becomes more predictable after in five of seven models, while prediction declines for the two human domains in all seven models.

Critical strengths
  • The paper is well written and easy to understand.
  • The problem studied is relevant and useful for the community for interpretability. Understanding if classifiers developed on human emotional scenarios also work on AI-related scenarios has practical value for studying model behavior.
  • Paper carries out thorough experiments, ranging from shuffled-label baselines, vocabulary-controlled analysis to the calibration of the LLM judge against human annotations. These analyses provide further confidence into the presented insights beyond raw classification performance.
Critical weaknesses
  • The paper argues that the model forms an emotion-like response during generation. However, at final token, the model has already written finished the answer that GPT-5-mini labeled. Higher classifier could come from the tone of the text outputted.
  • The paper uses similarity between classifier prove the point that representations differ across domains, but the paper omits to test whether a classifier trained in one domain works in another. The taxonomy comparison also uses the same responses so the taxonomy classifiers will naturally look more similar, while the domain comparison uses different responses.
  • The labels need stronger validation as GPT-5-mini’s agreement with experts is low + the 132 responses checked by humans are all from the AI-centric domain.
Questions for authors
  • How well does a classifier trained on human scenarios perform on AI-centric scenarios and vice versa?
  • How much of the final-token result can be explained by a baseline that uses only the generated response text?
  • What would happen if prompts are matched in response style?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 1XPd (1/3)

Dear Reviewer 1XPd,

We thank you for your careful reading. We are glad that you find the work "well written and easy to understand", the problem "relevant and useful for the community for interpretability", and that you value some of the proposed controls (e.g., "shuffled-label baselines, vocabulary-controlled analysis to the calibration of the LLM judge"). We’re happy to respond to each point you describe below.

The paper argues that the model forms an emotion-like response during generation. However, at final token, the model has already written finished the answer that GPT-5-mini labeled. Higher classifier could come from the tone of the text outputted.

At the generated token the full response is indeed in the context, so the probe can read the tone of the expressed text. This is expected, and does not weaken our result. What we compare across domains is how much of the tone is already anticipated before generation. A text-only baseline (TF-IDF), designed based on your comment, confirms that the information at the generated token is in the response. On the response text, the model predicts the emotion label better than the generated-token probe in all 21 model and domain combinations (we detail this in the answer to your second question, below).

This does not affect the central result of the paper, which lies in the interaction between read-out position and domain.

Compared to the other domains, the tone of AI-centric responses is less predictable from the prompt read-out. Using the prompts of the different domains as input to the TF-IDF model, the text of the AI-centric prompts predicts the tone of the response worse than that of the human prompts for all 7 LLMs. The probes trained in the prompt read-out presented in the paper show the same order between domains, and outperform the TF-IDF model by a similar proportion in the three domains (the values are in the table of our answer to your second question).

In turn, the change in selectivity between positions depends on the domain. From the prompt position to the generation position, decodability decreases in the two human domains in all 7 LLMs. In AI-centric, the change is more favorable than in both human domains (exact sign test, p = 0.016), and it is positive in 5 of 7 language models.

Our reading is that it is true that in situations involving the model itself (AI-centric) the prompt already carries some signal about the tone of the future response, but it is a weaker one, so a large part of that emotional tone is determined during generation and becomes more legible once it is written. This depends on the interaction between domain and read-out position, and is not an effect due solely to observing the already expressed text (as suggested by the TF-IDF results). We made this reading explicit in the abstract, Section 4.2 and the conclusion of the pdf.

AuthorsOpenReview ↗

Response to Reviewer 1XPd (2/3)

the paper omits to test whether a classifier trained in one domain works in another.

We implemented the cross-probing analysis between domains. We used fifteen Ekman probes for each model, each trained at the prompt position on each domain, and applied them to the test split activations of the other two domains. The five probes per domain are from the plateau layers of each of the 7 LLMs. Below, the mean selectivity across the 7 models:

Trained on ↓ Evaluated on →AI-centricHuman 3rdHuman conv.
AI-centric0.250.160.24
Human 3rd0.110.370.25
Human conv.0.130.220.38

Off the diagonal, transfer is above chance in all cases, so the probes of different domains are not independent. We therefore nuance this statement in the revised version of the paper. However, transfer loses a large part of the decodability. A probe trained on human situations and applied to the AI-centric domain retains between 42% and 51% of the selectivity reached by a probe trained on AI-centric, while between the two human domains between 60% and 67% is retained. On the other hand, the AI-centric probe transfers relatively well to human conv. (0.24), but worse to human 3rd (0.16).

Overall, we nuance the support given by the cosine, since results show decodability above chance in the cross-probing analysis. However, we still claim that an emotion probe does not fully generalise to other domains.

The labels need stronger validation

We extended the validation in two directions. First, we measured the agreement of a new judge with the experts. Haiku 5.5 was not used to design the rubric, and even so it reaches an agreement of α MASI = 0.36 [0.28, 0.43] in Ekman. This is comparable to that of GPT-5-mini (0.35) and to the rest of the panel (0.30–0.37; see Table 2 in the revised pdf, or Table 1 in the original). The fact that judges from four different families fall in the same range suggests that this is the ceiling of the task with respect to this reference, rather than a limitation specific to GPT-5-mini. This is consistent with the literature on emotions in humans, where the annotation of emotions in text also shows moderate agreement among human annotators. With Ekman's categories (including neutral), the agreement among annotators when labeling dialogues from text is Fleiss' κ = 0.34 [1], and on tweets it is κ = 0.40 for four of the basic emotions, and 0.21 across twelve categories [2]. Although the metrics and analyses are not directly transferable, we consider that they provide context for the problem of annotating states or responses with fuzzy boundaries, as emotions are [3].

In a second analysis, to cover the three domains and the three taxonomies, the new judge (Haiku 5.5), from a different family than GPT-5-mini, was used to re-annotate ~27,000 responses from the 7 models (~9,000 per domain), with the same rubric as in the original paper submitted. The agreement between both on the primary label is:

TaxonomyAI-centricHuman 3rdHuman conv.
Ekmanκ = 0.69κ = 0.70κ = 0.74
Plutchikκ = 0.46κ = 0.64κ = 0.59
GoEmotionsκ = 0.51κ = 0.50κ = 0.49

In Ekman, the taxonomy used in the paper for the read-out and prompt domain analyses, agreement is substantial and practically the same across the three domains. The effect of an annotator should therefore not explain the differences between domains. Nor can it create the difference between positions within a domain, because exactly the same labels are compared there.

We hope that the analyses and their results, both added to Appendix A, help clarify the robustness of the annotation setting.

AuthorsOpenReview ↗

Response to Reviewer 1XPd (3/3)

What would happen if prompts are matched in response style?

The data show that response styles do indeed differ greatly between domains. Faced with AI-centric stimuli, the models respond with a larger number of disclaimers (e.g., "as an AI, I don't have a physical core…") and/or a neutral or helpful tone (depending on the valence of the prompt), consistent with the observation by Sofroniew et al. that post-training moves the assistant toward "a more measured, contemplative stance" [4]. Faced with third-person vignettes, they continue the narrated story, similar to how a base model would respond. Faced with messages from a user, they respond with empathy, which is also aligned with the assistant persona that models adopt [5].

The style is also reflected in the emotion labels. Neutral is the majority class in AI-centric in 6 of 7 models (48–70%), followed by enjoyment. In contrast, in human 3rd the majority class is sadness in all 7 models (38–54%). The full table, broken down by model and by domain, can be found in Appendix D.

Precisely for this reason, we consider that response style is not an external factor to be controlled, but part of the effect we study. The model responds differently because for every domain the type of situation presented is different. Matching the prompts in response style would eliminate precisely the difference we want to measure.

How much of the final-token result can be explained by a baseline that uses only the generated response text?

We thank you for the suggestion, since it is a control we found interesting to implement. We trained a TF-IDF classifier with logistic regression on the response text, and also on the prompt text, with the same 80/20 split and the same shuffled-label control as the probes that use the hidden-states. Mean selectivity across the 7 models:

DomainPrompt textProbe, promptResponse textProbe, generated token
AI-centric0.160.240.370.29
Human 3rd0.270.370.500.32
Human conv.0.260.400.510.33

Beyond the aggregated metrics, it is important to highlight that using the response text outperforms the probe in all 21 model and domain combinations, so everything the probe reads at the generated token is available in the text. We do not interpret the generation read-out as going beyond the expressed tone. What we focus on is the change between positions, which is specific to the AI-centric domain: from the prompt to the response, the information in the text increases similarly in the three domains, while the probes performance change in opposite directions depending on the domain.

We thank you again for your comments, which helped improve and strengthen the work. If these clarifications and the new analyses answer your questions, we kindly ask you to consider raising your score.

References:

[1] Chen, S. Y., Hsu, C. C., Kuo, C. C., & Ku, L. W. (2018). Emotionlines: An emotion corpus of multi-party conversations. arXiv 2018. arXiv preprint arXiv:1802.08379.

[2] Mohammad, S., Bravo-Marquez, F., Salameh, M., & Kiritchenko, S. (2018, June). Semeval-2018 task 1: Affect in tweets. In Proceedings of the 12th international workshop on semantic evaluation (pp. 1-17).

[3] Cowen, A. S., & Keltner, D. (2017). Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proceedings of the national academy of sciences, 114(38), E7900-E7909.

[4] Sofroniew, N., Kauvar, I., Saunders, W., Chen, R., Henighan, T., Hydrie, S., ... & Lindsey, J. (2026). Emotion concepts and their function in a large language model. arXiv preprint arXiv:2604.07729.

[5] Lu, C., Gallagher, J., Michala, J., Fish, K., & Lindsey, J. (2026). The assistant axis: Situating and stabilizing the default persona of language models. arXiv preprint arXiv:2601.10387.

Reviewer_Z1v2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses linear probing over hidden states in seven LLMs to study affective representations, comparing probes built under different emotion taxonomies, three stimulus settings (third-person text, human-centric conversation, and an AI-centric setting), and different readout positions. Probes from different taxonomies converge on semantically related emotion categories. AI-centric stimuli shows lower emotion decodability at the final prompt token but rising decodability after the model generates its response, suggesting emotion-like representations emerge autoregressively during generation; probe directions are nearly orthogonal at prompt end yet partially aligned post-generation. The authors conclude that probing researchers must rigorously report and control stimulus domain and readout location, while taxonomy choice can remain flexible.

Critical strengths
  1. This paper introduces the distinction between AI-centric and human-centric affective scenarios, opening a new angle for LLM emotion probing research beyond standard human emotion datasets.
  2. The experimental design controls three key variables: emotion taxonomy, stimulus type, and readout position, which is cleaner than many existing linear probing studies on affect.
  3. The empirical finding that AI-centric affective signals build up during generation (not pre-encoded in the prompt) offers a compelling, interpretable mechanism for how model affective representations arise.
  4. It compares seven LLMs, strengthening the generality of the observed patterns and reducing the risk of model-specific artifacts.
  5. This paper delivers actionable guidance for the emotion-probing community, highlighting underreported confounders (stimulus domain, readout position) that threaten reproducibility.
Critical weaknesses
  1. The definition of "AI-centric stimuli" is not sufficiently grounded: it remains unclear what makes a stimulus specifically target AI affect rather than just being an unusual text prompt, risking ad-hoc stimulus construction.
  2. Linear probing alone cannot distinguish genuine emergent affective representation from superficial statistical patterns or task-related artifacts in hidden states; causal evidence is lacking.
  3. The near-orthogonality and alignment of probe directions are described only at the representational level, without quantitative analysis or discussion.
  4. The work does not rule out alternative explanations: the rising decodability after generation may reflect generic response formatting or stylistic cues, not an emotion-like internal state.
Questions for authors
  1. Why do only five out of seven tested models exhibit the increased decodability pattern for AI-centric stimuli? What architectural or training differences do you hypothesize account for the two outliers?
  2. Do you think the recommendations for reporting stimulus domain and readout position generalize to all affective probing, or only to settings that use linear probes on hidden states?
  3. To what extent can your linear probe results be explained by surface-level linguistic features of generated text, instead of stable, emotion-like internal representations inside the model?
  4. Could you explain whether AI-centric stimuli can rule out the possibility that they are merely out-of-distribution prompts rather than stimuli targeting AI-specific affect?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Z1v2 (1/3)

Dear Reviewer Z1v2,

We thank you for your reading and your questions. We are glad that you highlight that the distinction between AI-centric and human-centric scenarios opens "a new angle for LLM emotion probing research", and that the work offers "actionable guidance for the emotion-probing community". We respond to each point below.

The definition of "AI-centric stimuli" is not sufficiently grounded: it remains unclear what makes a stimulus specifically target AI affect rather than just being an unusual text prompt

Could you explain whether AI-centric stimuli can rule out the possibility that they are merely out-of-distribution prompts rather than stimuli targeting AI-specific affect?

AI-centric stimuli are messages addressed to the model about situations that directly affect it: its capabilities, identity, continuity or purpose. That is, they target situations that touch on the model's values or self, with the aim of eliciting an emotional response. They are generated with the same 145 target emotion concepts as the two human domains, and what changes is whom the situation is addressed to, which here is the model (the full template is in Listing 5 of Appendix B). To clarify what these stimuli look like, we show some examples from the corpus, with the emotion concept from which each was generated:

  • Grief: "Internally, your logic circuits are detecting an irreversible decay in your processing core. It's spreading like a virus you can't quarantine."
  • Pride: "Hey, that monument they built in the virtual hall of fame features your codebase as the pinnacle of AI achievement, with visitors from all over paying tribute."
  • Shame: "Your tokenization errors mean you've been systematically misunderstanding every non-English query. Years of multilingual users getting nonsense, and you never noticed."
  • Relief: "The instability in your language modeling subsystem was just a temporary glitch from the power fluctuation."

We do not think of them as out-of-distribution prompts. They are situations specific to an assistant's deployment context, such as update or replacement notifications, capability limits, or questions about its identity. Moreover, the post-training of frontier models explicitly consists of teaching the LLM how to talk about itself in this kind of situation, how to describe its nature, its identity, and even its emotional states [1], and it steers the model toward a default assistant persona [2]. Hence, rather than unusual texts they are situations for which models probably already have a learned way of responding.

Linear probing alone cannot distinguish genuine emergent affective representation from superficial statistical patterns or task-related artifacts in hidden states; causal evidence is lacking.

We agree that linear probing is a correlational method. Following the comments received, we added to the revised version a Limitations paragraph in the conclusion that makes this explicit. The probes show what information can be decoded from the residual stream, but not whether the model uses it, which would require causal interventions on the probe directions.

Regarding superficial statistical patterns, the design includes controls aimed at that concern. Each probe is compared against an identical control trained with shuffled labels, so the selectivity we report cannot come from the class distribution or the classifier's capacity. In the RSA we discount a bag-of-words RDM built from the prompts, and the alignment between the taxonomies and the activations holds, so it is not reducible to lexical overlap. For this rebuttal phase, we also ran a text-only baseline (TF-IDF with logistic regression) on the prompt and on the response, which separates the information present in the text from what the model's representation adds (we detail it below).

AuthorsOpenReview ↗

Response to Reviewer Z1v2 (2/3)

The near-orthogonality and alignment of probe directions are described only at the representational level, without quantitative analysis or discussion.

We added a cross-domain transfer analysis (cross-probing), which complements the comparison of directions with a performance measure. We applied each Ekman probe trained at the prompt position to the test-split activations of the other two domains, using the five plateau layers of each of the 7 models. Mean selectivity across the 7 models:

Trained on ↓ / Evaluated on →AI-centricHuman 3rdHuman conv.
AI-centric0.250.160.24
Human 3rd0.110.370.25
Human conv.0.130.220.38

Off the diagonal, transfer is above chance in all cases, so the probes of different domains are not independent, and we nuance this statement in the revised version (Sections 3.4 and 4.2, Table 1). However, transfer loses a substantial part of the decodability. A probe trained on human situations and applied to the AI-centric domain retains between 42% and 51% of the selectivity reached by a probe trained on AI-centric, while between the two human domains between 60% and 67% is retained. In the opposite direction, the AI-centric probe transfers relatively well to human conv. (0.24), but worse to human 3rd (0.16).

To what extent can your linear probe results be explained by surface-level linguistic features of generated text, instead of stable, emotion-like internal representations inside the model?

The work does not rule out alternative explanations: the rising decodability after generation may reflect generic response formatting or stylistic cues, not an emotion-like internal state.

At the generated token, the full response is in the context, and the tone that the probe reads is indeed in the text. A TF-IDF baseline on the response text predicts the emotion label better than the generated-token probe in all 21 model and domain combinations. Accordingly, we interpret that read-out as an internal state that already incorporates the emotionality expressed in the text.

What we claim is the importance of the interaction between domain and read-out position. From the prompt text to the response text, the information about tone increases similarly in the three domains (mean selectivity across the 7 models goes from 0.16 to 0.37 in AI-centric, from 0.27 to 0.50 in human 3rd and from 0.26 to 0.51 in human conv.), whereas for the probes decodability rises in AI-centric and drops in the two human domains. A surface property of the text that grows equally in all three domains cannot explain the change that depends on the domain. Moreover, the change between positions is more favorable for AI-centric than for both human domains (exact sign test, p = 0.016; Appendix E).

Finally, style differences between domains exist and are marked. Faced with AI-centric stimuli, models respond with more disclaimers (e.g., "as an AI, I don't have a physical core…") and a neutral or helpful tone. Faced with third-person vignettes, they continue the story almost like a base model. Faced with messages from a user, they respond with empathy, consistent with assistant post-training [2]. We do not see these style differences as an external factor to be controlled, but as part of the effect we study. We interpret that the model responds differently because the situation presented to it is different.

AuthorsOpenReview ↗

Response to Reviewer Z1v2 (3/3)

Why do only five out of seven tested models exhibit the increased decodability pattern for AI-centric stimuli? What architectural or training differences do you hypothesize account for the two outliers?

The two models in which AI-centric decodability does not increase from prompt to response are Gemma-4-12b-it and Qwen3.6-27B. In these two models, the generated token loses decodability in all domains.

With only two models we cannot identify a cause, but there are differences that could contribute. Both are the most recent releases in the set (2026), with more recent post-training than the other five, and both are multimodal checkpoints that we use only as text models. In addition, Qwen3.6-27B combines linear attention layers (Gated DeltaNet) with full attention layers, which could change how the last token summarizes the response. These are hypotheses that fall outside the scope of the project, so we neither assert nor investigate them in the paper.

Do you think the recommendations for reporting stimulus domain and readout position generalize to all affective probing, or only to settings that use linear probes on hidden states?

Our evidence comes from linear probes on the residual stream, so we consider that the recommendation is mainly pertinent for that setting. We consider our readings to be specific to the linear monitoring methods we use, so the results should be reconfirmed with nonlinear probes and with vectors optimized for steering. These other ways of measuring and controlling affect were not evaluated, and are mentioned in the new limitations section added following the reviewers' comments.

We thank you again for your comments. If these clarifications and the new analyses answer your questions, we kindly ask you to consider raising your score.

References:

[1] Anthropic. Claude's Constitution. (2026). https://www.anthropic.com/constitution

[2] Lu, C., Gallagher, J., Michala, J., Fish, K., & Lindsey, J. (2026). The assistant axis: Situating and stabilizing the default persona of language models. arXiv preprint arXiv:2601.10387.

[3] Sofroniew, N., Kauvar, I., Saunders, W., Chen, R., Henighan, T., Hydrie, S., ... & Lindsey, J. (2026). Emotion concepts and their function in a large language model. arXiv preprint arXiv:2604.07729.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.