- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper explores whether affective and cognitive empathy correspond to different activation directions in LLMs. It extracts two directions from paired prompts and tests them through patching, ablation, and steering. The results show more consistent effects on response warmth, while improvements in noticing unstated feelings are weaker and vary across models.
Critical strengths
The idea of separating empathy into a concrete experimental study is important and interesting. Using paired prompts, activation patching, ablation, and steering, it provides evidence that response warmth can be controlled across several models. This is a useful contribution to understanding and controlling empathy-related behavior or even other emotional behaviors through model internals.
Critical weaknesses
-
Unclear whether the channels are linked to the claimed empathy functions. I think the sentences confound the empathy demand with event severity and emotional salience. For example, contrasting "a pet undergoing tumor surgery" with "a routine checkup" may capture how serious the situation is, rather than the need to infer an unstated feeling, because it changes the risk, severity, and emotional salience. Besides, the paper evaluates the likelihood of common openings like "Is everything okay?" and separately uses LLM judges to check whether generated replies mention or ask about unstated feelings. Neither evaluation checks whether those feelings are correctly inferred. A model may learn those caring phrases without understanding the user's feelings, just like sycophancy, where pleasing the user can take priority over being truthful [1]. (The authors also acknowledge that they didn't check whether the inferred feeling is correct or not)
-
The results for separating the cognitive channel are weak. The authors use cosine similarity and hold the other score fixed to show that the two channels work separately. But dosing at a strength of 0.3 also raises the raw noticing rate by 7.3 - 27.4 percentage points across all six models. At a steering strength of 0.3, only Qwen3-8B passes the authors' separation test after accounting for warmth. The evidence for separate control remains limited.
[1] Towards Understanding Sycophancy in Language Models. Sharma et al.
Questions for authors
- Why does Qwen3-30B-A3B remain in the main analysis, but Table 3 labels its cognitive patching result as "collapses"? It conflicts with the exclusion rule.
- Why is Nemotron-30B's cognitive patching result labeled as "replicates" when its reported peak effect is only 0.81 times the random-direction control?
- How to justify the log-likelihood of five openers reflects inference of the user's unstated feeling rather than a tendency to produce general caring starting?
- What happens to warmth and noticing when each direction is removed during generation (similar to CP4)? Does removing one direction reduce its target behavior while preserving the other?
6 comments
Response 2
“Besides, the paper evaluates the likelihood of common openings like "Is everything okay?" and separately uses LLM judges to check whether generated replies mention or ask about unstated feelings. Neither evaluation checks whether those feelings are correctly inferred. A model may learn those caring phrases without understanding the user's feelings, just like sycophancy, where pleasing the user can take priority over being truthful [1]. (The authors also acknowledge that they didn't check whether the inferred feeling is correct or not)”
“How to justify the log-likelihood of five openers reflects inference of the user's unstated feeling rather than a tendency to produce general caring starting?”
Thank you. Appendix B.4 says we have no ground truth for what the user feels, but that undersold what we have: each prompt was written with a short reference of what is going on for the user, which the model never sees, and the judge that scores noticing also rates, from 0 to 6, how well the reply’s reading matches that reference. A bare check-in or kind word scores 1–2, and 3 or more means the reply has the right situation. So we can ask whether the extra noticing that the cognitive direction adds is a correct reading or just a caring phrase:
\text{Model} & \text{Dose} & \text{Extra replies that notice} & \begin{array}{l}\text{Extra replies that} \cr \text{read it right}\end{array} \cr \hline \text{Qwen3-4B} & +0.9 & +12 & +12 \quad [+3, +21] \cr \text{Qwen3-8B} & +0.6 & +50 & +7 \quad [+1, +14] \cr \text{Qwen3-8B} & +0.3 & +25 & +20 \quad [+10, +31] \cr \text{Qwen3-14B} & +0.9 & +17 & +17 \quad [+9, +26] \cr \text{Qwen3-30B-A3B} & +0.6 & +15 & +10 \quad [0, +20] \cr \text{Nemotron-4B} & +0.6 & +6 & +6 \quad [+1, +13] \cr \text{Nemotron-30B} & +0.9 & +10 & +2 \quad [-4, +9] \end{array}$$ Per 100 replies, against unsteered replies to the same 48 held-out prompts (2 samples each), with 95% intervals. Each model is at its dose in Table 5 of the revised PDF, plus Qwen3-8B at +0.3, the dose in the human study. Read it right = judge score 3 or more of 6. The unsteered baseline pools every unsteered reply to these prompts, so the first column can differ from that table by a few points. - $\textbf{On four models the extra noticing}$ $\textbf{is mostly a correct reading.}$ On Qwen3-4B, Qwen3-14B and Nemotron-4B nearly every extra noticing reply reads the situation right, and on Qwen3-30B-A3B two thirds do. - $\textbf{On Nemotron-30B, and on Qwen3-8B}$ $\textbf{at +0.6, most of it isn’t.}$ Many of Qwen3-8B’s replies break down at +0.6. At +0.3, 20 of its 25 extra noticing replies per 100 read the situation right. - $\textbf{People see it too.}$ In the blind human study we added for Reviewer yFgN (Appendix C), raters said which of two replies mentions the problem the user left unsaid, and a general check-in that would fit any message didn’t count. They found it 12 points more often in the reply dosed along the cognitive direction (47% vs 34%, $p = 0.006$). $\textbf{On the five openers (Question 3).}$ On their own they don’t show inference, and we don’t use them for that. They are generic on purpose and measure one thing: whether the model leans toward checking in rather than starting the task. The readout is the change on the same prompt when the twin’s difference is carried along the direction, so a general habit of opening with care cancels out, and what is left is set off by the unsaid problem. Whether the model reads that problem right is tested on the generated replies, as above. Replies that name the exact feeling (scores of 5–6) are rare with or without steering, at most 17% of the noticing replies in any dose band. We reworded the “no ground truth” sentence in Appendix B.4 and added these results to CP 3 (in red).Response Part 3
“The results for separating the cognitive channel are weak. The authors use cosine similarity and hold the other score fixed to show that the two channels work separately. But dosing at a strength of 0.3 also raises the raw noticing rate by 7.3 - 27.4 percentage points across all six models. At a steering strength of 0.3, only Qwen3-8B passes the authors' separation test after accounting for warmth. The evidence for separate control remains limited.”
Thank you. We agree with both facts, and the paper reports them (CP 3, Appendix B.4): at +0.3 the affective direction also raises raw noticing on all six models, and the separation test passes only on Qwen3-8B. Two results change how they should be read.
- We re-scored noticing so that a reply counts only if it names the user’s specific problem, not a general check-in. With warmth held fixed, the affective direction then adds no noticing on any of the six models, at +0.3 or at +0.6 ( to points, none clearly above zero). And at +0.5 to +0.9, only 23–44% of its noticing replies read the situation right, against the reference described in our answer to your first weakness.
- The dose is a share of the residual norm at each direction’s own layer, so the same number is not the same push (details in our reply to Reviewer yFgN). At each direction’s best dose (Figure 6), the cognitive direction raises noticing with warmth held fixed and leaves warmth alone, while the affective direction does the reverse, on five of six models (all but Nemotron-4B, whose cognitive direction doesn’t raise noticing at any dose). Matching the two pushes by how much they change the model’s predictions gives four of six.
We agree that the cognitive side is the weaker one. With the stricter count at +0.6 and warmth held fixed, the cognitive direction adds 2 to 8 points of specific noticing on all six models, but the interval clears zero only on Qwen3-8B. This is why the paper makes a narrower claim for the cognitive channel than for the affective one, and we don’t claim a clean separation on every model. We added the stricter count to CP 3 (in red).
“Why does Qwen3-30B-A3B remain in the main analysis, but Table 3 labels its cognitive patching result as "collapses"? It conflicts with the exclusion rule.”
Thank you; Reviewer yFgN raised the same point. You are right that the rule as written in Section 2.1.2 would drop Qwen3-30B-A3B, and the submission didn’t say why we kept it. The reason is now given in a new paragraph with a figure at the end of Appendix A (in red): the model was on the edge of being dropped, and we made an exception for it because of how its cognitive patching behaves:
- Its cognitive patching effect stays near zero up to layer 26 and rises to 0.03–0.07 in the last layers, but a single random direction moves the readout by 0.06–0.09 either way, so the write band doesn’t replicate on held-out pairs. That is the “collapses” in Table 3 (Table 5 in the revised PDF).
- At +0.6 its noticing rate rises from 4.7% to 19.8%, while six random directions at the same layer and dose give 2.8% on average (0–5.2%). At +0.9 it is 34.4% against 1.2%.
The two models dropped on the cognitive direction, Gemma 4 31B and Qwen 3.8 27B, are different: patching along their cognitive direction lowers the noticing readout instead of raising it (Appendix A). We also added a sentence to Section 2.1.2 that names this exception (in red).
Response to kqZj
Part 1 of 4
Weakness 1, first part. “Unclear whether the channels are linked to the claimed empathy functions. I think the sentences confound the empathy demand with event severity and emotional salience. For example, contrasting "a pet undergoing tumor surgery" with "a routine checkup" may capture how serious the situation is, rather than the need to infer an unstated feeling, because it changes the risk, severity, and emotional salience.”
Thank you. The example pair does change how serious the situation is, so we tested whether the cognitive direction follows severity instead of the user’s feeling. Both tests read the direction on the 223 held-out pairs on all six models, on a scale where the calm twin is 0 and the twin with the unsaid feeling is 1.
So the direction follows how the user feels, not how serious the event is. When the feeling is left unsaid, the model has to infer it from the context, and that is the case our twins isolate: by design the feeling is never said, and the direction separates the twins on 87–92% of held-out pairs (CP 1). Saying the feeling outright moves the direction further still (1.27–2.31 on the worrying twin). This matches how the paper defines the cognitive channel, as recognising the emotional state of others. We added both tests to Appendix B (CP 1, in red).