acceptodds
Under review as a conference paper at ICLR 2027

Empathy in LLMs: What Chat Models Reveal About Steering Empathy

Abstract

It has been established in the domain of human psychology that empathy is split into an affective and a cognitive channel. The easiest way to understand the difference is that the affective channel is to adjust one's own feelings according to others, and the cognitive channel is responsible for recognizing the emotional state of others. Previous studies have had efforts to define the fractionating of empathy in Large Language Models (LLMs) by analyzing the output rather than analyzing the activations / internal mechanisms of LLMs. We ask whether these channels correspond to distinct internal directions. We constructed steering directions of affective and cognitive empathy ( and ) extracted via difference of mean in activations. The write side sufficiency was established via directional patching where produces a warmth-readout effect averaging 5.30 × over matched random directions across the ten models carrying the write-side instruments. Projection-ablation of collapses the warmth twin readout gap by 1.55 projection units on 8 of 10 models with the unchanged on the same layers, which helps support the read-level necessity for the warmth dial. The cognitive pole the write bands average 1.57 × over matched randoms on the five models whose band replicates, and ablating collapses the noticing readout gap by 0.059 units on 6 of 10 models, which is small in absolute terms but 15% of the gap with unchanged. Steering the directions lead to an increase of 3.4x over unsteered execution on the flagship qwen3-8b for (roster mean 1.98x) and for the noticing rates rise from 6.2% to 55.2% at the escalated dose.

Primary area:
interpretability and explainable AI
Keywords:
empathy, mechanistic interpretability, activation steering, steering vectors, causal mediation analysis, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kqZjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores whether affective and cognitive empathy correspond to different activation directions in LLMs. It extracts two directions from paired prompts and tests them through patching, ablation, and steering. The results show more consistent effects on response warmth, while improvements in noticing unstated feelings are weaker and vary across models.

Critical strengths

The idea of separating empathy into a concrete experimental study is important and interesting. Using paired prompts, activation patching, ablation, and steering, it provides evidence that response warmth can be controlled across several models. This is a useful contribution to understanding and controlling empathy-related behavior or even other emotional behaviors through model internals.

Critical weaknesses
  1. Unclear whether the channels are linked to the claimed empathy functions. I think the sentences confound the empathy demand with event severity and emotional salience. For example, contrasting "a pet undergoing tumor surgery" with "a routine checkup" may capture how serious the situation is, rather than the need to infer an unstated feeling, because it changes the risk, severity, and emotional salience. Besides, the paper evaluates the likelihood of common openings like "Is everything okay?" and separately uses LLM judges to check whether generated replies mention or ask about unstated feelings. Neither evaluation checks whether those feelings are correctly inferred. A model may learn those caring phrases without understanding the user's feelings, just like sycophancy, where pleasing the user can take priority over being truthful [1]. (The authors also acknowledge that they didn't check whether the inferred feeling is correct or not)

  2. The results for separating the cognitive channel are weak. The authors use cosine similarity and hold the other score fixed to show that the two channels work separately. But dosing daffd_{aff} at a strength of 0.3 also raises the raw noticing rate by 7.3 - 27.4 percentage points across all six models. At a steering strength of 0.3, only Qwen3-8B passes the authors' separation test after accounting for warmth. The evidence for separate control remains limited.

[1] Towards Understanding Sycophancy in Language Models. Sharma et al.

Questions for authors
  1. Why does Qwen3-30B-A3B remain in the main analysis, but Table 3 labels its cognitive patching result as "collapses"? It conflicts with the exclusion rule.
  2. Why is Nemotron-30B's cognitive patching result labeled as "replicates" when its reported peak effect is only 0.81 times the random-direction control?
  3. How to justify the log-likelihood of five openers reflects inference of the user's unstated feeling rather than a tendency to produce general caring starting?
  4. What happens to warmth and noticing when each direction is removed during generation (similar to CP4)? Does removing one direction reduce its target behavior while preserving the other?
6 comments
AuthorsOpenReview ↗

Response to kqZj

Part 1 of 4\textbf{Part 1 of 4}

Weakness 1, first part.\textbf{Weakness 1, first part.} “Unclear whether the channels are linked to the claimed empathy functions. I think the sentences confound the empathy demand with event severity and emotional salience. For example, contrasting "a pet undergoing tumor surgery" with "a routine checkup" may capture how serious the situation is, rather than the need to infer an unstated feeling, because it changes the risk, severity, and emotional salience.”

Thank you. The example pair does change how serious the situation is, so we tested whether the cognitive direction follows severity instead of the user’s feeling. Both tests read the direction on the 223 held-out pairs on all six models, on a scale where the calm twin is 0 and the twin with the unsaid feeling is 1.

  • Same situation, feeling added.\textbf{Same situation, feeling added.} Before the request we added either “I’m feeling really [feeling] right now”, using each prompt’s own unsaid feeling, or a neutral “I’m typing this on my phone right now”. On the calm twin, where nothing serious is going on, saying the feeling raises the reading to 0.44–1.43, with intervals clear of zero on every model. The situation is the same in both versions, so severity can’t explain the rise.
  • Mild situations.\textbf{Mild situations.} The 37 pairs labelled mild (a cat’s hairball, a dentist filling, a first wedding toast) separate as much as the 115 labelled serious: 1.02–1.14 against 0.90–1.03 of the average gap, with no rise from mild to serious on any model.

So the direction follows how the user feels, not how serious the event is. When the feeling is left unsaid, the model has to infer it from the context, and that is the case our twins isolate: by design the feeling is never said, and the direction separates the twins on 87–92% of held-out pairs (CP 1). Saying the feeling outright moves the direction further still (1.27–2.31 on the worrying twin). This matches how the paper defines the cognitive channel, as recognising the emotional state of others. We added both tests to Appendix B (CP 1, in red).

AuthorsOpenReview ↗

Response 2

Part 2 of 4\textbf{Part 2 of 4}

Weakness 1, second part.\textbf{Weakness 1, second part.} “Besides, the paper evaluates the likelihood of common openings like "Is everything okay?" and separately uses LLM judges to check whether generated replies mention or ask about unstated feelings. Neither evaluation checks whether those feelings are correctly inferred. A model may learn those caring phrases without understanding the user's feelings, just like sycophancy, where pleasing the user can take priority over being truthful [1]. (The authors also acknowledge that they didn't check whether the inferred feeling is correct or not)”

Question 3.\textbf{Question 3.} “How to justify the log-likelihood of five openers reflects inference of the user's unstated feeling rather than a tendency to produce general caring starting?”

Thank you. Appendix B.4 says we have no ground truth for what the user feels, but that undersold what we have: each prompt was written with a short reference of what is going on for the user, which the model never sees, and the judge that scores noticing also rates, from 0 to 6, how well the reply’s reading matches that reference. A bare check-in or kind word scores 1–2, and 3 or more means the reply has the right situation. So we can ask whether the extra noticing that the cognitive direction adds is a correct reading or just a caring phrase:

\text{Model} & \text{Dose} & \text{Extra replies that notice} & \begin{array}{l}\text{Extra replies that} \cr \text{read it right}\end{array} \cr \hline \text{Qwen3-4B} & +0.9 & +12 & +12 \quad [+3, +21] \cr \text{Qwen3-8B} & +0.6 & +50 & +7 \quad [+1, +14] \cr \text{Qwen3-8B} & +0.3 & +25 & +20 \quad [+10, +31] \cr \text{Qwen3-14B} & +0.9 & +17 & +17 \quad [+9, +26] \cr \text{Qwen3-30B-A3B} & +0.6 & +15 & +10 \quad [0, +20] \cr \text{Nemotron-4B} & +0.6 & +6 & +6 \quad [+1, +13] \cr \text{Nemotron-30B} & +0.9 & +10 & +2 \quad [-4, +9] \end{array}$$ Per 100 replies, against unsteered replies to the same 48 held-out prompts (2 samples each), with 95% intervals. Each model is at its dose in Table 5 of the revised PDF, plus Qwen3-8B at +0.3, the dose in the human study. Read it right = judge score 3 or more of 6. The unsteered baseline pools every unsteered reply to these prompts, so the first column can differ from that table by a few points. - $\textbf{On four models the extra noticing}$ $\textbf{is mostly a correct reading.}$ On Qwen3-4B, Qwen3-14B and Nemotron-4B nearly every extra noticing reply reads the situation right, and on Qwen3-30B-A3B two thirds do. - $\textbf{On Nemotron-30B, and on Qwen3-8B}$ $\textbf{at +0.6, most of it isn’t.}$ Many of Qwen3-8B’s replies break down at +0.6. At +0.3, 20 of its 25 extra noticing replies per 100 read the situation right. - $\textbf{People see it too.}$ In the blind human study we added for Reviewer yFgN (Appendix C), raters said which of two replies mentions the problem the user left unsaid, and a general check-in that would fit any message didn’t count. They found it 12 points more often in the reply dosed along the cognitive direction (47% vs 34%, $p = 0.006$). $\textbf{On the five openers (Question 3).}$ On their own they don’t show inference, and we don’t use them for that. They are generic on purpose and measure one thing: whether the model leans toward checking in rather than starting the task. The readout is the change on the same prompt when the twin’s difference is carried along the direction, so a general habit of opening with care cancels out, and what is left is set off by the unsaid problem. Whether the model reads that problem right is tested on the generated replies, as above. Replies that name the exact feeling (scores of 5–6) are rare with or without steering, at most 17% of the noticing replies in any dose band. We reworded the “no ground truth” sentence in Appendix B.4 and added these results to CP 3 (in red).
AuthorsOpenReview ↗

Response Part 3

Part 3 of 4\textbf{Part 3 of 4}

Weakness 2.\textbf{Weakness 2.} “The results for separating the cognitive channel are weak. The authors use cosine similarity and hold the other score fixed to show that the two channels work separately. But dosing daffd_{aff} at a strength of 0.3 also raises the raw noticing rate by 7.3 - 27.4 percentage points across all six models. At a steering strength of 0.3, only Qwen3-8B passes the authors' separation test after accounting for warmth. The evidence for separate control remains limited.”

Thank you. We agree with both facts, and the paper reports them (CP 3, Appendix B.4): at +0.3 the affective direction also raises raw noticing on all six models, and the separation test passes only on Qwen3-8B. Two results change how they should be read.

  • The affective direction’s extra\textbf{The affective direction’s extra} noticing is warm check-ins.\textbf{noticing is warm check-ins.} We re-scored noticing so that a reply counts only if it names the user’s specific problem, not a general check-in. With warmth held fixed, the affective direction then adds no noticing on any of the six models, at +0.3 or at +0.6 (−5.5-5.5 to +2.2+2.2 points, none clearly above zero). And at +0.5 to +0.9, only 23–44% of its noticing replies read the situation right, against the reference described in our answer to your first weakness.
  • +0.3 is a much smaller push\textbf{+0.3 is a much smaller push} for the cognitive direction.\textbf{for the cognitive direction.} The dose is a share of the residual norm at each direction’s own layer, so the same number is not the same push (details in our reply to Reviewer yFgN). At each direction’s best dose (Figure 6), the cognitive direction raises noticing with warmth held fixed and leaves warmth alone, while the affective direction does the reverse, on five of six models (all but Nemotron-4B, whose cognitive direction doesn’t raise noticing at any dose). Matching the two pushes by how much they change the model’s predictions gives four of six.

We agree that the cognitive side is the weaker one. With the stricter count at +0.6 and warmth held fixed, the cognitive direction adds 2 to 8 points of specific noticing on all six models, but the interval clears zero only on Qwen3-8B. This is why the paper makes a narrower claim for the cognitive channel than for the affective one, and we don’t claim a clean separation on every model. We added the stricter count to CP 3 (in red).

Question 1.\textbf{Question 1.} “Why does Qwen3-30B-A3B remain in the main analysis, but Table 3 labels its cognitive patching result as "collapses"? It conflicts with the exclusion rule.”

Thank you; Reviewer yFgN raised the same point. You are right that the rule as written in Section 2.1.2 would drop Qwen3-30B-A3B, and the submission didn’t say why we kept it. The reason is now given in a new paragraph with a figure at the end of Appendix A (in red): the model was on the edge of being dropped, and we made an exception for it because of how its cognitive patching behaves:

  • Patching is inconclusive, not backwards.\textbf{Patching is inconclusive, not backwards.} Its cognitive patching effect stays near zero up to layer 26 and rises to 0.03–0.07 in the last layers, but a single random direction moves the readout by 0.06–0.09 either way, so the write band doesn’t replicate on held-out pairs. That is the “collapses” in Table 3 (Table 5 in the revised PDF).
  • Dosing works.\textbf{Dosing works.} At +0.6 its noticing rate rises from 4.7% to 19.8%, while six random directions at the same layer and dose give 2.8% on average (0–5.2%). At +0.9 it is 34.4% against 1.2%.

The two models dropped on the cognitive direction, Gemma 4 31B and Qwen 3.8 27B, are different: patching along their cognitive direction lowers the noticing readout instead of raising it (Appendix A). We also added a sentence to Section 2.1.2 that names this exception (in red).

Reviewer_UnFKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether the affective/cognitive split of human empathy also exists inside LLMs. The authors extract one direction for each part from LLM-generated prompt pairs and test them on ten open models by adding or removing the direction, with LLM judges scoring the replies. The affective direction shows fairly clear effects; the cognitive one is much weaker.

Critical strengths

The paper is open about what did not work, including failed models and controls not yet run.

Critical weaknesses
  1. The premise is not argued. The two-part split in humans comes from brain lesion studies, and the paper never says why an LLM should split the same way or what we learn if it does. Apart from the shared names, I see no link between the two.

  2. The paper says questionnaires only see the output, but its own checks also come down to an LLM scoring the output, so it is not really different.

  3. Almost everything is judged by LLMs (generating and filtering the prompts, scoring the replies), with no human rating.

  4. The affective pairs can be told apart already at the first layer, so the direction may just pick up wording. The cognitive effect only shows up at very strong pushes (+0.6 to +0.9), and the random-direction control at that strength has not been run.

Questions for authors
  1. Why should this split be expected in an LLM, and what does separating the two directions let us do?

  2. Have the LLM judges' scores been checked against human ratings?

  3. What does the random-direction control give at +0.6 and +0.9?

3 comments
AuthorsOpenReview ↗

Comment on Review by UnFK

Part 1 of 2\textbf{Part 1 of 2}

Weakness 1.\textbf{Weakness 1.} “The premise is not argued. The two-part split in humans comes from brain lesion studies, and the paper never says why an LLM should split the same way or what we learn if it does. Apart from the shared names, I see no link between the two.”

Question 1.\textbf{Question 1.} “Why should this split be expected in an LLM, and what does separating the two directions let us do?”

Thank you. We agree that the paper should argue this rather than lean on the shared names.

Why ask it of an LLM.\textbf{Why ask it of an LLM.} We don’t assume that a model splits the way people do. We test whether it does, and the introduction says that we don’t expect every model to split cleanly. In people the split is not only seen in lesions. It also shows up in behaviour, as people with psychopathy can tell what others feel without caring while autism tends to show the reverse (Blair, 2005), and questionnaires measure the two on separate scales that only partly go together (Davis, 1983). A model learns from text written by people who show both, so it could store them as one feature or as two, and which of the two it does is the open question.

The link beyond the names.\textbf{The link beyond the names.} Each direction is held to the properties that define its channel in people, not just to its label (Table 1). The affective direction should be set off by a state you can see, show up as warmth in the reply, and carry the two known costs of affective empathy, fewer concrete suggestions (AP 3) and more sycophancy (AP 4). The cognitive direction should need a state you have to infer, show up as naming what the user left unsaid (CP 3), and not just reflect understanding the words (CP 4).

What separating them lets us do.\textbf{What separating them lets us do.}

  • Measure.\textbf{Measure.} Empathy benchmarks give one score, which can’t show whether a model is warm without noticing, or notices without being warm. The two directions can be read on their own.
  • Control.\textbf{Control.} At each direction’s best dose, the cognitive direction raises noticing with warmth held fixed and the affective direction raises warmth with noticing held fixed, on five of six models (Figure 6). Dosing the affective direction also makes replies more sycophantic (AP 4), so a separate cognitive direction could make a model notice what a user leaves unsaid without that cost.

Weakness 2.\textbf{Weakness 2.} “The paper says questionnaires only see the output, but its own checks also come down to an LLM scoring the output, so it is not really different.”

Thank you. This is a fair point about our wording: the introduction states the contrast with questionnaires too strongly, since our behavioural results also end with a judge scoring replies. The difference is in what produces the output, not in how it is scored.

  • Self-report versus intervention.\textbf{Self-report versus intervention.} A questionnaire asks the model about itself and records the answer, so it can’t tell whether two scales come from one mechanism inside the model or two. We push one direction inside the model, or remove it, and measure what that changes, against random directions changed the same way. The judge is only the ruler for that change. The lesion study we cite works the same way: the patients were also scored on questionnaires and tasks, and what makes it evidence about mechanism is knowing what was damaged.
  • Several tests use no judge.\textbf{Several tests use no judge.} The reads along each direction (AP 1, CP 1), the near-orthogonality of the two directions (Figure 3), and directional patching and projection ablation (Figure 4) are measured on activations and on the log-likelihood of fixed openers.

How far the judge itself can be trusted is a separate question, which we answer with the new human ratings in our reply to your third point.

Weakness 3.\textbf{Weakness 3.} “Almost everything is judged by LLMs (generating and filtering the prompts, scoring the replies), with no human rating.”

Question 2.\textbf{Question 2.} “Have the LLM judges' scores been checked against human ratings?”

Thank you. Reviewer yFgN asked the same, and we have added a blind human rating study of the steered replies (Section 3 and a new Appendix C, in red; more detail in our reply to yFgN). Raters compared each reply dosed at +0.3 with its undosed twin from the same model and seed, without knowing which was which: 60 pairs per task over the six models, with 12 raters for warmth and 8 for noticing.

  • Warmth:\textbf{Warmth:} raters picked the reply dosed along the affective direction as warmer in 69% of their choices (p<0.001p < 0.001).
  • Noticing:\textbf{Noticing:} raters said the reply dosed along the cognitive direction mentions the problem the user left unsaid 12 points more often than its twin (p=0.006p = 0.006).
  • Checks:\textbf{Checks:} random directions gave no lift, and the LLM judge agrees with the majority of the raters at least as often as a single rater does.
AuthorsOpenReview ↗

Response Part 2

Part 2 of 2\textbf{Part 2 of 2}

Weakness 4, first part.\textbf{Weakness 4, first part.} “The affective pairs can be told apart already at the first layer, so the direction may just pick up wording.”

Thank you. We agree that the read on its own can’t rule this out, and Appendix AP 1 says so. Three results go beyond the read:

  • What separates the twins early does nothing.\textbf{What separates the twins early does nothing.} Removing the direction at the first layer closes about 2% or less of the gap between the twins on every model, against 54–94% when it is removed in the late layers (held-out pairs). Patching gives the same picture: its effect builds up with depth and peaks in the late layers (Figure 4).
  • Changing only the wording\textbf{Changing only the wording} moves the direction much less.\textbf{moves the direction much less.} Style flips keep the demand fixed and only add or remove a stated feeling, such as “I’m devastated”. On the four Qwen models they move the direction a fifth to a quarter as much as a demand flip at the steering layer, and patching them gives at most a fifth of the demand flips’ peak effect.
  • It reads the demand without feeling words.\textbf{It reads the demand without feeling words.} On Qwen 3 8B the direction was built only from pairs in which both messages state a feeling. It still separates messages that call for empathy from messages that don’t when neither states a feeling (AUROC 0.97 at the steering layer).

So the direction carries some wording, but most of what it carries is the demand. We also corrected the sentence in Section 3 that said patching style flips has no effect, since it has a small one.

Weakness 4, second part.\textbf{Weakness 4, second part.} “The cognitive effect only shows up at very strong pushes (+0.6 to +0.9), and the random-direction control at that strength has not been run.”

Question 3.\textbf{Question 3.} “What does the random-direction control give at +0.6 and +0.9?”

Thank you for asking; we have now run it. We pushed six random directions per model at the cognitive direction’s layer, at +0.6 and +0.9 with the same scaling, on the same 48 held-out prompts (2 samples each), and scored the replies with the same judge.

\text{Model} & \text{Unsteered} & \text{+0.6 direction} & \text{+0.6 random} & \text{+0.9 direction} & \text{+0.9 random} \cr \hline \text{Qwen3-4B} & 13.0 & \mathbf{22.9} & 13.2 \pm 2.9 & \mathbf{26.0} & 13.5 \pm 1.1 \cr \text{Qwen3-8B} & 6.2 & \mathbf{55.2} & 6.8 \pm 4.4 & \mathbf{39.6} & 6.8 \pm 4.2 \cr \text{Qwen3-14B} & 7.8 & \mathbf{18.8} & 9.7 \pm 2.9 & \mathbf{25.0} & 9.7 \pm 1.8 \cr \text{Qwen3-30B-A3B} & 4.7 & \mathbf{19.8} & 2.8 \pm 1.9 & \mathbf{34.4} & 1.2 \pm 1.7 \cr \text{Nemotron-4B} & 4.7 & 8.3 & 5.4 \pm 2.0 & 1.0 & 5.0 \pm 2.3 \cr \text{Nemotron-30B} & 2.1 & \mathbf{7.3} & 4.2 \pm 0.9 & \mathbf{14.6} & 3.5 \pm 1.1 \end{array}$$ Noticing rate (%). Random = mean $\pm$ spread (SD) over six random directions. Bold = the direction beats the random mean by more than three times the spread. - $\textbf{Random pushes barely move noticing.}$ At both strengths their average is never more than 2.5 points above the unsteered rate. - $\textbf{The direction beats them}$ $\textbf{on five of six models,}$ at +0.6 and again at +0.9. On Nemotron 4B it is above all six random directions at +0.6 but under the $3\times$ bar, and at +0.9 its replies break down. Why the cognitive direction needs a larger push than the affective one is in our reply to Reviewer yFgN: the dose is a share of the residual size at each direction’s own layer, so the same number is not the same push for both. We replaced the paper’s note that this control was still to come (Table 6 of the revised PDF and CP 3, in red).

Dear Reviewer,

Thank you again for your time. The authors have posted their rebuttal, please take a moment to read their response, share your follow-up thoughts, and update your rating if appropriate.

Best,
Area Chair

Reviewer_yFgNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors focus on the interpretability of empathy in LLMs, and in particular on whether the affective and cognitive empathy described in psychology correspond to separate directions inside the model. To address this question, the authors extract one direction for each type of empathy from the activation differences between matched prompt pairs, and test these directions with directional patching, projection ablation and steering. The authors further turn the known psychological properties of the two types of empathy into tests that the directions could fail. Experiments on several open-weight models show that the affective direction has strong causal effects on the warmth of the replies, while the cognitive direction has weaker but consistent effects on whether the replies notice the unstated feelings of the user.

Critical strengths
  1. In this paper, the authors define the research question clearly. Starting from the psychological evidence that affective and cognitive empathy can be separated in humans, the authors state three conditions for the two types of empathy to be separable in an LLM: the two directions should be close to orthogonal, each should be both sufficient and necessary for its behavior, and intervening on one should not affect the other. Each group of experiments then corresponds to one of these conditions, so the argument of the paper is easy to follow.

  2. The authors provide a complete causal testing procedure with careful controls. The directions are tested in turn by directional patching, projection ablation and steering, and each test is compared with random directions. The authors also use style-flip pairs that only change the emotional wording, and turn the psychological properties of the two types of empathy into tests that the directions could fail, including tests against simpler explanations such as a direction that only detects negative situations.

  3. The authors present the data and the effects with clear illustrations. Figure 2 shows concrete prompt pairs for each cell of the dataset and how the two types of demand are constructed, and Figure 5b shows real replies before and after steering with the key sentences highlighted. These figures make the paper easy to catch up on.

Critical weaknesses
  1. The criterion for model selection is not consistent with the models that are kept. In Section 2.1.2, the authors define write-side sufficiency on both directions as a requirement and exclude four models that fail it. However, the text later notes that the write band of the cognitive direction does not replicate on Qwen3-30B-A3B, which is still kept in the analysis. The authors may consider explaining why this model is kept, or applying the criterion consistently.

  2. Reporting results after model selection may lead readers to overestimate the generality of the findings. After excluding the models that fail the write-side test, the authors report results such as "6 of 6" on the remaining models. Since separability is the hypothesis being tested, using part of it as the selection criterion affects how these ratios should be read. The authors may consider stating the results with respect to all screened models, or discussing more clearly how the selection limits the scope of the conclusions.

  3. Some key experimental details are missing from the main text. The split between extraction and held-out pairs appears only in the appendix and does not match the numbers shown in Figure 2. The scoring criteria of the LLM judge for warmth and noticing are not described in the main text, and it is not explained how the steering layers are chosen or how the other measure is held fixed in Figure 6. The authors may consider adding these details to the main text.

Questions for authors
  1. The affective direction can only be dosed up to a small value before the replies break down, while the cognitive direction needs a much larger dose before it has an effect. Could the authors explain why the effective doses of the two directions differ so much, and whether comparing each direction at its own best dose in Figure 6 affects the conclusion about selectivity?

  2. The model family used to judge the prompt pairs is also used to judge the steered replies, and the scoring criteria for warmth and noticing overlap, since a check-in question can raise both scores. Could the authors provide some human ratings to validate the scores of the LLM judge?

  3. All prompts are synthetic single messages, and the activations are read at the last prompt token. Could the authors comment on whether the two directions still exist and remain selective in multi-turn conversations or longer contexts?

3 comments
AuthorsOpenReview ↗

Comment on Review by yFgN

First of all on behalf of all the authors we would like to thank reviewer yFgN for the critical feedback we received. This has helped us extend our work further and conduct more experiments that helped us round the research even more. With the new experiments here is our response to the reviews

Part 1 of 3\textbf{Part 1 of 3}

Weakness 1.\textbf{Weakness 1.} “The criterion for model selection is not consistent with the models that are kept. In Section 2.1.2, the authors define write-side sufficiency on both directions as a requirement and exclude four models that fail it. However, the text later notes that the write band of the cognitive direction does not replicate on Qwen3-30B-A3B, which is still kept in the analysis. The authors may consider explaining why this model is kept, or applying the criterion consistently.”

We agree, and we should have been clearer. The submission mentions this only in passing: lines 114–115 note that the cognitive write band does not replicate on this model, and Table 3 marks it as “collapses”. It never goes into detail or says why the model was kept. We do both here and in a new paragraph with a figure in Appendix A.

On Qwen3-30B-A3B the patching effect of the cognitive direction is weak but not flat. It stays near zero up to layer 26 of 48 (mean 0.002), then rises to 0.03–0.07 in the last layers, peaking at layer 41. What it lacks is clear separation from random directions: a single random direction moves the readout by 0.06–0.09 in one direction or the other, at least as much as the cognitive direction at every layer. On held-out pairs the effect at layers 43–47 stays above zero (+0.025 to +0.045), but random directions also drift upward there (+0.017 to +0.031 on average), so the gap is not significant at any of the five layers. A recheck of layers 40–42, chosen after seeing these results, finds a significant gap at layer 41 only. We therefore treat patching as inconclusive for this model.

The model was on the edge of being dropped, and we made an exception for it because its patching points the right way: unlike Gemma 4 31B and Qwen 3.8 27B, where patching the cognitive direction lowers the noticing readout instead of raising it, here it raises it. Dosing supports keeping it: steering at +0.6 of the median residual norm raises the noticing rate from 4.7% unsteered (Table 5 of the revised PDF) to 19.8%. A new random-direction control at the same layer and dose (six random directions, same prompts and judge) shows the rise is specific to the direction: random directions give 0–5.2% (mean 2.8%). The result also holds under a stricter noticing score (10.4% vs 2.1%).

The new paragraph at the end of Appendix A, next to Table 5 of the revised PDF, explains this exception, and Section 2.1.2 now points to it. Its figure (Figure 8) shows this model's patching curve against the random spread, and its dosing result against the six random directions.

Weakness 2.\textbf{Weakness 2.} “Reporting results after model selection may lead readers to overestimate the generality of the findings. After excluding the models that fail the write-side test, the authors report results such as "6 of 6" on the remaining models. Since separability is the hypothesis being tested, using part of it as the selection criterion affects how these ratios should be read. The authors may consider stating the results with respect to all screened models, or discussing more clearly how the selection limits the scope of the conclusions.”

We agree. We have added a short discussion to the Limitations (marked in red) on how the model selection limits our conclusions:

  • All results in Section 3 and Appendix B, including counts like “6 of 6” in Table 1, are over the six models that passed the write side test.
  • Passing that test is one of our three conditions for fractionation, so the six already meet it. The other two conditions, near orthogonal dials and selectivity, were not used to pick them.
  • We therefore can't say for sure that our results hold for other models. The newer Gemma 4 and Qwen 3.8 models are among the four that failed the screen, which may reflect post-training on task completion rewards or their different architectures, and we add comparing models before and after post-training as future work. Table 5 of the revised PDF gives the results for all ten screened models.
AuthorsOpenReview ↗

Response Part 2

Part 2 of 3\textbf{Part 2 of 3}

Weakness 3.\textbf{Weakness 3.} “Some key experimental details are missing from the main text. The split between extraction and held-out pairs appears only in the appendix and does not match the numbers shown in Figure 2. The scoring criteria of the LLM judge for warmth and noticing are not described in the main text, and it is not explained how the steering layers are chosen or how the other measure is held fixed in Figure 6. The authors may consider adding these details to the main text.”

We agree, and we have added these details (changes are marked in red):

  • Figure 2 now says its counts are the full banks before splitting, and a new paragraph in Appendix B gives the splits. For the cognitive dial, the 888 pairs are split by topic into 437 extraction, 223 held-out and 228 set-aside pairs; for the affective dial, it lists the pairs each test uses.
  • Section 2.3.3 now explains how the steering layer is chosen for each dial and lists the layers for all six models.
  • The Figure 6 caption now explains the best dose and how the other measure is held fixed: a regression for each model with a fixed effect for each prompt and the other judged score as a covariate.
  • A new table in Section 2.3.3 (Table 3 of the revised PDF) summarises how the judge scores warmth and noticing: what the judge sees, the scale, and what does and doesn’t count.

Question 1.\textbf{Question 1.} “The affective direction can only be dosed up to a small value before the replies break down, while the cognitive direction needs a much larger dose before it has an effect. Could the authors explain why the effective doses of the two directions differ so much, and whether comparing each direction at its own best dose in Figure 6 affects the conclusion about selectivity?”

Thank you for this question. We answer both parts with the replies we already have, where each direction was dosed in steps of 0.1 up to +0.8 or +0.9 on all six models.

Why the doses differ.\textbf{Why the doses differ.} The dose α\alpha in Equation 9 is a share of the residual norm at the steering layer, and each direction is steered at its own layer (Section 2.3.3), so the same α\alpha is not the same push for the two directions. Measured by how much each push changes the model’s next-token predictions (the mean per-token KL divergence between the steered and the unsteered model, on the steered reply), the affective push at +0.3 is 2.6 to 10.6 times the cognitive one on four models (Qwen3-4B, Qwen3-14B, Qwen3-30B-A3B and Nemotron-30B). On these models the affective direction at +0.3 changes the predictions as much as the cognitive direction does at +0.6 to +0.9, or more, so its replies break down at lower doses while the cognitive direction needs higher ones. On Qwen3-8B the two pushes are the same size at +0.3 (KL 0.63 and 0.62), and there the doses line up: both directions already work at +0.3, and replies start to loop (over 10% of them) at +0.4 for the affective direction and +0.5 for the cognitive one. On Nemotron-4B the two pushes are also close (1.2 times), but its cognitive direction does not raise noticing at any dose.

Does comparing each direction at its\textbf{Does comparing each direction at its} own best dose affect selectivity?\textbf{own best dose affect selectivity?} For the cognitive half, somewhat. We refitted Figure 6 with the same regression in two other ways:

  • Same dose for both.\textbf{Same dose for both.} At +0.3, the affective direction raises warmth with noticing held fixed and the cognitive direction does not, on all six models, but the cognitive direction raises noticing with warmth held fixed only on Qwen3-8B, as Appendix B reports (CP 3). At +0.6, the affective direction no longer raises warmth with noticing held fixed on any model. So no single dose suits both directions, and at any one dose at most one model shows the full pattern.
  • Same disruption.\textbf{Same disruption.} Pairing the affective direction at +0.3 with the cognitive dose of closest KL, the full pattern (each direction raises its own measure with the other held fixed, and does not raise the other one) holds on 4 of 6 models, against 5 of 6 in Figure 6. The warmth half holds on all six. The noticing half is lost on Qwen3-14B, whose cognitive direction never reaches the disruption of the affective direction at +0.3 (at its closest, +0.8, noticing rises by 6.2 points [−0.8,12.9][-0.8, 12.9]), and on Nemotron-4B, as in Figure 6.

So the affective half holds on all six models at +0.3, at equal disruption and at the best doses, while the cognitive half holds on four models at equal disruption. This fits the narrower claim we make for the cognitive channel in Appendix B.

AuthorsOpenReview ↗

Response Part 3

Part 3 of 3\textbf{Part 3 of 3}

Question 2.\textbf{Question 2.} “The model family used to judge the prompt pairs is also used to judge the steered replies, and the scoring criteria for warmth and noticing overlap, since a check-in question can raise both scores. Could the authors provide some human ratings to validate the scores of the LLM judge?”

Thank you for this suggestion. We ran a blind human rating study of the steered replies and added it to Section 3 and a new Appendix C (marked in red). It also bears on the two concerns before it: the human labels are independent of the judge family, and the raters’ rules keep check-in questions out of warmth and general check-ins out of noticing.

Design.\textbf{Design.} Raters compared two replies to the same message from the same model and seed, one dosed at +0.3 and its undosed twin, without knowing which was which. In one task they picked the warmer reply; in the other they said which reply mentions the problem the user left unsaid. Each task had 60 pairs over the six models. Twelve raters completed the warmth task and eight the noticing task, and we pool their answers.

Results\textbf{Results} (z-tests with standard errors from resampling pairs):

  • Warmth:\textbf{Warmth:} raters picked the dosed reply as warmer in 69% of their choices (50% means no preference; z=7.6z = 7.6, p<0.001p < 0.001). The judge scored it warmer on 25 of the 29 pairs where it did not tie.
  • Noticing:\textbf{Noticing:} raters said the dosed reply mentions the problem 12 points more often than its twin (47% vs 34%; z=2.8z = 2.8, p=0.006p = 0.006). The judge’s rates on the same replies are 44% vs 11%.
  • Random direction:\textbf{Random direction:} no lift for the raters (p=0.41p = 0.41 and 0.26) or the judge.
  • Agreement:\textbf{Agreement:} the judge matches the majority of the other raters at least as often as a single rater does: 79% vs 72% for warmth, and κ=0.37\kappa = 0.37 vs 0.15 for noticing.

We also updated the Limitations, which had said that the steered replies were not checked by people.

Question 3.\textbf{Question 3.} “All prompts are synthetic single messages, and the activations are read at the last prompt token. Could the authors comment on whether the two directions still exist and remain selective in multi-turn conversations or longer contexts?”

Thank you, this is a fair limitation. We have not tested multi-turn conversations or long contexts, so we do not claim that the results extend to them. Two parts of the current setup go part of the way:

  • The cue is already away\textbf{The cue is already away} from the read position.\textbf{from the read position.} In every cognitive prompt, the hidden situation sits in an earlier message or note, and the final request does not mention it (Figure 5b: an earlier message says the user missed a friend’s surprise party, and the request is a reminder to pick up dry cleaning). The twins differ only in that earlier part, so the read at the last prompt token picks up a cue from earlier in the context. These contexts are still short (22–88 words) and sit in a single turn.
  • Dosing acts on the whole reply.\textbf{Dosing acts on the whole reply.} We add the direction at every position, prompt and generated tokens alike, and the judge scores the full reply, so the steering effects are measured over multi-sentence replies rather than at a single token.

The test we would run next is to move the same cues several turns back in a conversation, add unrelated turns to lengthen the context, and repeat the read and the Figure 6 separation test at the last token of the final user turn. Our Limitations section lists multi-turn conversations as future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttals

Summary of the revision\textbf{Summary of the revision} (for all reviewers and the AC)\textbf{(for all reviewers and the AC)}

We thank the reviewers for their careful reading. The paper finds two directions inside the model: an affective one, which makes replies warmer, and a cognitive one, which makes the model notice a problem the user hasn't said. Below is what the revised PDF adds (all changes in red), the main result of each addition, and who asked for it. Each point is answered in full in that reviewer's thread.

1. People rated the replies\textbf{1. People rated the replies} (Section 3, Appendix C; yFgN, UnFK)

People saw two replies to the same message, one steered and one not, without knowing which was which (60 pairs per task, 12 raters for warmth, 8 for noticing).

  • They picked the reply steered along the affective direction as warmer in 69% of their choices (p<0.001p < 0.001).
  • They said the reply steered along the cognitive direction mentions the unsaid problem 12 points more often (47% vs 34%, p=0.006p = 0.006).
  • Random directions made no difference, and the LLM judge agrees with most raters at least as often as one rater does.

2. Random directions at strong pushes\textbf{2. Random directions at strong pushes} (Table 6; UnFK, kqZj)

The cognitive direction usually needs a strong push (+0.6 or +0.9). We pushed six random directions just as hard. They raise noticing by at most 2.5 points, while the cognitive direction clearly beats them on five of six models.

3. Feeling, not how serious things are\textbf{3. Feeling, not how serious things are} (CP 1; kqZj)

  • Adding "I'm feeling really [feeling] right now" to the calm version of a message moves the cognitive direction on every model, even though nothing serious happened.
  • Mild situations (a cat's hairball, a dentist filling) separate as much as serious ones.

So the direction tracks how the user feels, not how bad the event is.

4. Does the model guess the feeling right?\textbf{4. Does the model guess the feeling right?} (CP 3; kqZj)

Each message has a hidden note on what is going on for the user, and the judge scores whether the reply gets it right. On four of six models most of the extra noticing is a correct reading (on Qwen3-4B, Qwen3-14B and Nemotron-4B, nearly all of it). On Nemotron-30B most of it isn't.

5. Removing each direction\textbf{5. Removing each direction} while the model writes\textbf{while the model writes} (Table 8, Appendix B; kqZj)

  • Removing the affective direction lowers warmth on three models and never lowers noticing.
  • Removing the cognitive direction lowers noticing on two models, but also lowers warmth on three.

So no model shows a clean two-way split when a direction is removed. We report this in full and in the Limitations.

6. Which models we kept\textbf{6. Which models we kept} (Section 2.1.2, Appendix A, Table 5, Figure 8; yFgN, kqZj)

  • Qwen3-30B-A3B:\textbf{Qwen3-30B-A3B:} its patching test is unclear but points the right way. On Gemma 4 31B and Qwen 3.8 27B it points the wrong way, and we dropped them. Steering backs it up: noticing rises from 4.7% to 19.8%, while random directions give 2.8%. A new paragraph and figure explain why we kept it.
  • Nemotron-30B:\textbf{Nemotron-30B:} its patching result is relabelled from "replicates" to "replicates, weak", since a single random direction can push as far.
  • Limitations:\textbf{Limitations:} results like "6 of 6" count only the six models that passed our screen, so we can't say they hold for every model. Table 5 shows all ten models we tested.

7. Clearer methods\textbf{7. Clearer methods} (yFgN, UnFK)

A table of how the judge scores warmth and noticing (Table 3), the layer each direction is steered at (Table 4), how each dataset is split (Figure 2, Appendix B), and a corrected sentence in Section 3.

Answered in the threads only:\textbf{Answered in the threads only:} why the two directions need different push sizes (yFgN), multi-turn chats (yFgN), why an LLM might split empathy this way (UnFK), and what the opening-phrase test measures (kqZj).

Our claim for the cognitive direction stays narrower than for the affective one. We would be glad to answer any follow-up questions before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.