- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether post-training changes how LLMs represent users’ emotions internally or mainly changes how they respond to emotional input. The authors study 4 open-weight models, Tulu 3 (Llama-3.1-8B), OLMo-2-7B, OLMo-2-13B, and Zephyr-7B, across consecutive BASE-> SFT -> DPO -> RLVR checkpoints, using matched emotional stimuli. They measure affect decodability from hidden states with several probes, evaluate reply behavior with multiple judges and phrase counts, and intervene along a fixed valence direction via steering and clamping. The main finding is a representation–behavior dissociation: affect remains similarly decodable across stages, while reply valence rises most consistently at SFT -> DPO. The authors conclude that DPO changes the emitted reply rather than the measured affect representation, at least within the tested direction and interventions.
Critical strengths
- Controlled stage-wise design. Consecutive checkpoints from the same families (shared architecture and tokenizer), matched stimuli, multiple probe formulations (including frozen cross-stage probes) and four behavioral judges with different construction make the central observation credible. It reduces confounds that affect most cross-model comparisons. Reporting the divergence at a specific stage (SFT -> DPO) is a clear, useful finding for the post-training literature.
- The intervention attempt is a meaningful step beyond decodability. The paper does not equate higher valence with greater consolation. It supports this with classifier-free phrase counts, and it openly reports uneven controls. Together with the steering and clamping experiments, which go beyond decodability, this gives a more careful empirical picture than a probing-only study.
Critical weaknesses
- The central claim is stronger than the evidence, and the null results rest on weak tests. The title and conclusion assert that alignment changes the response policy, not the affect representation, but the evidence supports something narrower: a coarse probe and only one direction. Clamping is run on only two models and changes just 14.4% of Tulu replies. The response policy explanation is described in the Discussion as motivated rather than tested, yet it appears in the title. Alternative directions, nonlinear encodings, SAE features, and feature interactions are not tested.
- The behavioral effect is small, narrowly measured, and may not be emotion-specific or deployment-relevant. The phrase analysis suggests the change is largely fewer “I don’t understand” and “I’m sorry” replies, which may reflect a general DPO style shift rather than anything about emotional input. There is no matched non-emotional control. All main results use a two-shot continuation prompt without a chat template. The chat-template check covers only reply valence, not decodability, steering or clamping. The results may therefore not generalize to the instruction-following regime where DPO models are used.
Questions for authors
- The studied LLMs were released between late 2023 and late 2024, which is about two years before this submission. Substantially stronger open models, with different post-training recipes, now exist. Why were these chosen, and do the findings, especially the SFT -> DPO localization, replicate on contemporary models with released intermediate checkpoints?
- Can the authors provide results on matched non-emotional conversational prompts? This would show whether the SFT -> DPO valence increase and the drop in first-person phrases are specific to emotional input or a general DPO style change.
- Reply valence was repeated under native chat templates. Were affect decodability and steering/clamping also repeated there? If not, why should the continuation-prompt results generalize to the regime where DPO models are deployed?
- Since only one valence direction is tested, what would it take to claim that DPO does not change affect representation more broadly? For example, could the authors search for directions or SAE features that best separate SFT and DPO replies and test whether they are decodable or causally relevant?
8 comments
Response to reviewer dbYV: directions, SAEs and generality (part 2/2)
Q2. Matched non-emotional prompts
The change does not require emotional input, which is what a response-policy change predicts. The paper makes no claim of emotion specificity. As the matched control, the revision uses neutral-labelled stimuli from the same Reddit source (App. C.2):
| Model | Valence rise, neutral | First-person drop (pp), neutral vs emotional |
|---|---|---|
| Tülu | +0.12 [0.03, 0.22] | −31 vs −28 |
| OLMo-2-7B | +0.11 [0.03, 0.20] | −23 vs −25 |
| Zephyr | +0.12 [0.03, 0.22] | −12 vs −16 |
| OLMo-2-13B | +0.04 [−0.04, 0.13] | −16 vs −17 |
Q3. Decodability and interventions under chat templates
Both repeat under the native templates. Base checkpoints have no template, so the main comparison uses one prompt format (Sec. 4.1). Under native templates the reply rise persists, SFT→DPO decodability changes by at most 0.002, and steering and the shared clamp give the same result in OLMo-2-7B (App. C.3).
Q4. Directions or SAE features separating SFT and DPO
We ran the search you propose, and the direction it finds carries the reply shift (App. F.3–F.4). We take the mean SFT→DPO displacement on identical inputs as a direction . The emotion label explains at most 1.1% of its variance, and is orthogonal to or points away from positive valence (cosine −0.52 to +0.04). Steering SFT along reproduces the DPO shift in all three models tested. Reply valence rises by +0.12 to +0.16 (observed +0.10 to +0.13), and first-person expressions fall by 11 to 24 points. At the same strength, a random direction changes first-person expressions by at most 6 points. changes them by at most 3 points even where it raises valence more than (+0.39 in OLMo-2-7B), so the two directions do different things (Table 16). Subtracting from DPO reverses the shift, and in Tülu removes 91% of the valence rise.
SAE features cannot carry affect information that the probes miss. A readout from SAE features is a one-layer ReLU function of the same hidden state, the form of the MLP probe, and the MLP reads no more than the logistic probe (App. B.5). A change in which features carry affect would also break the frozen probe, which loses at most 0.003 across stages (W1). SAEs trained separately at each stage learn features that do not correspond, so a comparison across stages must apply one SAE to every stage, which is the logic of the frozen probe. SAEs could show which features make up . That question concerns how the reply changes, not whether the affect representation does.
Please let us know if any concerns remain. We hope our clarifications warrant a more favorable evaluation.
Dear authors! Thanks for your detailed answers! New experiments and modifications lead to the paper look much stronger. I'm lean toward increasing the rating, but I have several small questions regarding your answers:
-
The interpretation of the new stage-direction experiment (Appendix F.3–F.4, Table 16) is not completely clear. The mean displacement at layer 16 is approximately 1% of the residual norm and that applying it directly changes reply valence by at most 0.03, compared with the observed 0.10–0.13 change (Appendix F.4, lines 1181–1183). Thus, the experiment demonstrates that the identified direction u can control the relevant behavior, but does not establish that the training-induced displacement along this direction is sufficient to explain the behavioral change. Can the authors test the actual per-input activation displacement at its natural magnitude, preferably on held-out stimuli?
-
Under native chat templates, the SFT→DPO behavioral difference for negative inputs almost disappears in Tülu (+0.01, with a confidence interval crossing zero), whereas it remains detectable in OLMo-2-7B. Should the conclusions be restricted to the tested prompt/model combinations rather than generalized to deployment-style responses?
-
A finite-width MLP does not necessarily upper-bound the information that could be exposed by an overcomplete SAE representation. Did the authors actually evaluate affect readout using SAE features, or is the conclusion based solely on the MLP-probe results?
In addition, I believe that the first limitation "Our study focuses on four 7–13B open-weight models, all preference-optimized via DPO, in single-turn English settings." (line 503-504) can be clarified to better reflect the answer ("Localizing a change needs every stage of one pipeline, and few open models release them all"). From current formulation of this limitation, it is unclear that only a few models (not most popular open-weights LLMs) can be studied in a proposed way.
Response to Reviewer dbYV: scope of the claim (part 1/2)
Dear Reviewer,
Thank you for the detailed questions on scope. We added a new model, a neutral control, chat-template repeats, new probes and a search for the direction that separates SFT from DPO. We hope the answers below address your concerns and inform your assessment. Due to the 5000 characters limit, we have to split the responses to two comments. Please bear with us.
W1. The claim is stronger than the evidence
The claim covers only what we measure, and the revision makes this explicit. The title now reads "Post-Training Changes Replies to Emotional Messages While Measured Affect Representations Remain Stable", and Sec. 3 lists the measured properties: probe decodability and transfer, emotion-centroid displacement, valence-direction orientation. We also test readout from SAE features (Sec. 4.2; App. B.6). Below, each point you raise is followed by the test that answers it.
W2. A small effect that may not be emotion-specific or deployment-relevant
The size is consistent and the location is the same in every model. Reply valence rises by +0.07 to +0.13 at SFT→DPO, with all judges agreeing in direction, and first-person subjective expressions fall from 46% to 29% of replies. The contribution is where the change happens: the same single step in every model.
Emotion specificity and deployment are answered in Q2 and Q3.
Q1. Why these models, and does a recent one replicate?
Localizing a change needs every stage of one pipeline, and few open models release them all. The revision adds Olmo 3-7B (2025), and the localization replicates. SFT→DPO decodability changes by at most 0.005 at all 33 layers, and reply valence rises by +0.08 [0.03, 0.12] at SFT→DPO, against −0.01 and 0.00 at the other steps (App. G).