acceptodds
Under review as a conference paper at ICLR 2027

From Encoding to Generation: Towards Empathetic Large Language Models

Abstract

Empathetic interaction involves two distinguishable capacities: representing another person's emotional state (cognitive empathy) and generating an appropriate response to that state (affective empathy). We apply this distinction to examine how emotion-related information is internally organized, selectively attended to, and carried through response generation in Llama-3.2-3B-Instruct across 10 emotion categories from the EmpatheticDialogues dataset. Layer-wise linear probing revealed a broad middle-layer region of peak emotion-category decodability, with the layer-11 L2 probe achieving 63.6% held-out accuracy against a 10% chance level and a 54.6% TF-IDF lexical baseline. During response generation, emotion-related user tokens received 1.89 times as much attention per token as other utterance tokens, concentrated across layers 10-15. Target-emotion alignment then decreased from a mean cosine similarity of 0.239 for input representations to 0.125 for generated responses, with positive drift in 77% of examples and the largest shifts for angry and furious. Emotion-category representation, attention allocation, and response-level alignment therefore emerge as distinguishable components of model processing that can be analyzed through a functional cognitive–affective empathy framework. These findings provide a foundation for monitoring emotion representations during generation and leveraging models’ existing affective structure to support more naturally empathetic responses without relying on external prompting or explicit emotion conditioning.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Emotion encoding, attention flow analysis, probing analysis, affective LLM, empathetic LLM

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_272HOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how emotional information is represented, attended to, and transformed during response generation in large language models. Using Llama-3.2-3B-Instruct and the EmpatheticDialogues dataset, the authors conduct three analyses: (1) layer-wise linear probing with L1 and L2 regularization to examine emotion representations, (2) attention analysis to investigate whether emotion-related tokens receive preferential attention during generation, and (3) emotion trajectory analysis to measure changes in emotion alignment between user inputs and generated responses. The results suggest that emotion information is most accessible in middle layers, emotion-related tokens receive greater attention during generation, and generated responses generally exhibit reduced alignment with the original target emotion, particularly for negative emotions. The paper aims to provide a framework for understanding the relationship between emotion representation and empathetic response generation in LLMs.

Critical strengths
  1. This paper proposes a relatively unified interpretability analysis framework for empathetic dialogue, extending emotion analysis from classification to generation while maintaining a good level of consistency.
Critical weaknesses
  1. The causal relationships underlying some experimental settings require further clarification, and the manuscript exhibits noticeable signs of AI-assisted writing.

  2. The paper overlooks several relevant studies that may affect the assessment of its novelty.

Questions for authors
  1. Missing citations to relevant prior work. The paper omits several relevant studies. For example, the contrastive directions introduced in Section 2.3.3 are conceptually similar to the Concept Activation Vectors discussed in “FastCAV: Efficient Computation of Concept Activation Vectors for Explaining Deep Neural Networks” (ICML 2025). Similarly, the two regularization approaches employed in Section 2.3.1 are closely related to the methodology explored in “Discovering Salient Neurons in Deep NLP Models” (Durrani, Dalvi, & Sajjad, Journal of Machine Learning Research, 2023). These related works should be properly discussed and cited to better contextualize the paper's methodological contributions and novelty.

  2. Limited generalizability of the experimental findings. The experiments are conducted exclusively on Llama-3.2-3B, without further evaluations across different model architectures or scales. Additional experiments are needed to assess the generalizability and robustness of the reported findings.

  3. Insufficient justification for emotion category selection. The authors select 10 emotion categories from the original set of 32 but do not adequately explain the rationale behind this selection. It remains unclear why these particular 10 categories were chosen instead of considering the full set of 32 emotions.

  4. Potential sensitivity to emotion representation strategies. The choice of emotion representation may require further investigation. Di Palma et al. (2025) explored multiple representation and pooling strategies for emotion-related probing tasks. It is unclear whether alternative representation strategies would substantially affect the experimental results and conclusions of this paper. An ablation study comparing different pooling strategies would help establish the robustness of the findings.

  5. Insufficient justification for transferring probing results to response-generation analysis. In the proposed framework, the representation strategy and model layer identified as optimal for emotion classification are directly adopted as the basis for analyzing response generation. However, a layer that contains highly decodable emotion-related information does not necessarily contribute to the generation of sufficiently empathetic responses. The relationship between emotion decodability and empathetic response generation requires further investigation, ideally supported by causal evidence.

  6. Unclear relationship between representational drift and actual emotional change. In the emotion drift experiments presented in Section 2.3.3, the authors primarily observe changes in emotion-related representations. However, it remains unclear whether such representational shifts correspond to meaningful changes in the emotional content of the generated responses. Additional validation is needed to establish the relationship between representational drift and actual emotional change, rather than assuming that changes in hidden-state alignment necessarily reflect changes in emotional content.

  7. Noticeable signs of possible AI-generated content in Figure 1. The upper portion of Figure 1 exhibits visual artifacts that appear consistent with AI-generated imagery, potentially produced by an image-generation model such as GPT Image. In particular, noticeable irregular white regions can be observed around text labels such as “Fear” and “Anger.” The authors should clarify how this figure was created and ensure that the visual presentation meets academic standards.

  8. Unresolved placeholder in Appendix A.8. Appendix A.8 contains an obvious placeholder: “insert the exact observed pattern and supporting values.” This suggests that the manuscript has not been thoroughly proofread and that the corresponding analysis or description remains incomplete.

1 comment

Thank you for your valuable comments on our manuscript. We appreciate your suggestions and will incorporate them, along with additional proofreading, in a revised version.

  1. Our study focuses on one model as an initial examination of emotion-related representations and generation patterns. The 10 categories were intentionally selected with an emphasis on negative emotions and closely related emotional states. We will clarify the selection criteria.
  2. We appreciate the suggested literature on concept activation vectors, probing, and alternative pooling strategies. Our contribution lies in applying established analytical methods within a functional cognitive framework to observe emotion processing from input to response generation. We will incorporate these references and better contextualize our methodological choices in future studies.
  3. Our objective is to investigate the relationship between emotion decodability, attention allocation, and response alignment. The selected probing layer serves as a reference for generation-level analysis, while emotion trajectory analysis measures changes in representational alignment between inputs and generated responses, rather than establishing causal mechanisms or directly assessing emotional appropriateness or empathy.
  4. We appreciate the reviewer highlighting the visual artifacts in Figure 1 and the unresolved placeholder in Appendix A.8. We will clarify the figure's construction and correct its presentation.
Reviewer_nrPBOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies emotion encoding and response generation in Llama-3.2-3B-Instruct through linear probing, token-group attention, and input-response cosine alignment.

Critical strengths
  1. Combining emotion representation, attention allocation, and response transformation provides a coherent way to examine how emotion-related information changes during generation.

  2. Evaluations provide useful initial evidence that emotion-category information is accessible in the model’s representations.

Critical weaknesses
  1. The framework mainly combines existing diagnostic methods. The absence of interventions and human evaluation leaves its mechanistic interpretation and connection to empathetic response quality insufficiently validated.

  2. The study evaluates a single model checkpoint, with only 20 generated responses per emotion.

  3. Emotion-related tokens depend on a layer-11 cosine threshold of 0.10, without threshold or prototype sensitivity analysis. Attention enrichment therefore needs robustness checks. Equation (2) should distinguish sample-level drift from category averages, and the enrichment calculation should specify the order of averaging.

  4. Figure 1 does not clearly communicate the data flow and prototype reuse. Figure 2 labels cross-validation curves as test accuracy, while Figure 3 does not clearly identify “Emotional” as a subset of “Utterance.” Besides, Appendix A.8 still contains placeholder text and describes normalized positions, whereas Figure 7 displays a 0–50 token-position axis. These issues leave the trajectory analysis incompletely specified.

Questions for authors
  1. What does the framework contribute beyond combining established diagnostics? Can you provide intervention-based evidence for the mechanistic interpretation or human evaluation linking the proposed metrics to empathetic response quality?

  2. Can you report the full stats of all emotions and check the corresponding generated responses?

  3. How sensitive are token selection and attention enrichment to the cosine threshold and prototype construction?

1 comment

Thank you for your constructive feedback; we will incorporate it, along with additional proofreading, in a revised version.

  1. Our primary contribution is the application of established analytical methods through a functional cognitive lens to explore how an LLM represents emotional information and how it evolves during response generation. Our goal is to provide an observational characterization of these phenomena, rather than to establish causal mechanisms or directly assess the quality of empathetic responses.
  2. We appreciate the suggestion to examine sensitivity to alternative thresholds and representation strategies. We will clarify these choices, their implications, and the order of averaging in the attention analysis.
  3. Additionally, we appreciate the identification of inconsistencies within the equations, figures, and appendix, which we will address in the revised manuscript.
Reviewer_ZffPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates which layers of LLMs are responsible for emotion recognition and response generation in empathetic replies. It employs Layer-wise Linear Probing, Attention analysis, and Response Alignment to explore these mechanisms. The experimental results indicate that while LLMs can internally represent user emotions and prioritize attending to emotional content, this does not guarantee that the generated responses retain the same emotional state. The study suggests that emotion recognition and the generation of emotional responses appear to be two separable processing stages.

Critical strengths
  1. The work differs significantly from previous studies on empathetic dialogue. Instead of focusing on improving LLM performance on this task, the authors use various analytical methods to explore the working principles of LLMs during empathetic dialogue tasks. This perspective is novel and interesting.

2.Although the experiments need augmentation, some conclusions are persuasive. For instance, the finding that emotional information is distributed across many dimensions rather than concentrated in a few isolated features, and that attention weights are more heavily allocated to layers with stronger emotion decoding capabilities during response generation.

3.The paper is well-written and fluent, making it easy to read.

Critical weaknesses

1.The EmpatheticDialogues dataset contains 32 emotion categories, but the paper only analyzes 10 of them. Although the authors mention that this choice enables "analysis of both fine-grained distinctions and broader affective contrasts," they do not sufficiently explain the specific criteria for selecting these 10 categories. Was it random selection or a conscious effort to cover different affective contrasts? Why not 15 or 20? I suggest the authors provide a clearer basis for this selection in the text or discuss the potential biases introduced by this choice.

2.The paper relies solely on the Llama-3.2-3B-Instruct model for experiments. Since different LLMs vary in the number of layers and hidden dimension sizes, the results may lack generalizability. It is possible that the findings are somewhat coincidental. Switching to a different LLM might result in variations in the identified key layers or accuracy. I recommend using LLMs from different series and of different sizes to enhance the persuasiveness of the conclusions. It should be noted that the authors have acknowledged this in the Limitations section. However, for a paper proposing an "analytical framework," simply listing this as "future work" seems insufficient. I suggest the authors discuss more explicitly in the Discussion section how this limitation affects the credibility of the conclusions, rather than merely categorizing it as a future direction.

3.Regarding whether emotional information is truly "used" in generating responses, the paper currently remains at the observational level. Linear probing can only prove that emotional information is "readable," not that it is "causally used" in the generation process. Although the authors mention in the Limitations that intervention methods like activation steering or ablation could be used in the future, I believe this limitation significantly impacts the core claims of this paper. If emotional information is not causally used during generation, then the findings of "priority attention allocation" and "representational accessibility," while interesting, greatly diminish the explanatory power regarding "empathy capability." I suggest the authors discuss this issue more deeply in the Discussion section.

Questions for authors

See weaknesses above.

1 comment

Thank you for your constructive feedback.

  1. Our selection of 10 categories was intentional, with an emphasis on negative emotions and closely related emotional states to investigate both broad emotional distinctions and finer variations in intensity. We will clarify this rationale in Section 2.1 and acknowledge the scope of the selected categories.
  2. We agree that cross-model analysis would provide valuable insights. Our study focuses on Llama-3.2-3B-Instruct as an initial observational investigation of how emotional information is represented and processed during response generation. The attention analysis offers an additional perspective on how the model attends to emotion-related information during generation. We will make this distinction clearer in our conclusions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.