acceptodds
Under review as a conference paper at ICLR 2027

Alignment Changes the Response Policy, Not the Affect Representation

Abstract

Alignment training changes how language models respond to emotional messages, but it is unclear whether these changes are accompanied by changes in how they represent the user’s emotions. We measure affect representation and reply valence across consecutive checkpoints of four models, spanning the base model, super- vised fine-tuning (SFT), direct preference optimization (DPO), and, where avail- able, reinforcement learning. Affect remains similarly decodable across stages under linear, nonlinear, and dimensional probes, and probes fitted at one stage transfer to the others. In contrast, judged reply valence rises most clearly from SFT to DPO, with all four judges agreeing on the direction of change. The pro- portion of replies containing first-person subjective expressions (e.g., “I’m sorry to hear that” or “I don’t understand”) falls from 46% to 29%. Replies shift away from echoing the user’s negative feelings toward more neutral or humorous re- sponses, and this pattern is consistent across input emotions. Encouragement, offers of help, and agreement do not increase, so the higher valence score alone does not establish greater consolation. Steering along a fixed valence direction changes replies at every checkpoint, but the SFT-to-DPO activation displacement along that direction is small, points away from positive valence, and does not in- crease the measured steering response. The reply-valence difference also persists when the direction is clamped. The SFT-to-DPO change is best localized to the response policy rather than to the measured affect representation: decodability, emotion-specific displacement, and valence-axis orientation remain stable at this transition.

Primary area:
interpretability and explainable AI
Keywords:
alignment, probing, representation-behavior dissociation, activation steering, emotion, preference optimization, post-training, interpretability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dbYVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether post-training changes how LLMs represent users’ emotions internally or mainly changes how they respond to emotional input. The authors study 4 open-weight models, Tulu 3 (Llama-3.1-8B), OLMo-2-7B, OLMo-2-13B, and Zephyr-7B, across consecutive BASE-> SFT -> DPO -> RLVR checkpoints, using matched emotional stimuli. They measure affect decodability from hidden states with several probes, evaluate reply behavior with multiple judges and phrase counts, and intervene along a fixed valence direction via steering and clamping. The main finding is a representation–behavior dissociation: affect remains similarly decodable across stages, while reply valence rises most consistently at SFT -> DPO. The authors conclude that DPO changes the emitted reply rather than the measured affect representation, at least within the tested direction and interventions.

Critical strengths
  1. Controlled stage-wise design. Consecutive checkpoints from the same families (shared architecture and tokenizer), matched stimuli, multiple probe formulations (including frozen cross-stage probes) and four behavioral judges with different construction make the central observation credible. It reduces confounds that affect most cross-model comparisons. Reporting the divergence at a specific stage (SFT -> DPO) is a clear, useful finding for the post-training literature.
  2. The intervention attempt is a meaningful step beyond decodability. The paper does not equate higher valence with greater consolation. It supports this with classifier-free phrase counts, and it openly reports uneven controls. Together with the steering and clamping experiments, which go beyond decodability, this gives a more careful empirical picture than a probing-only study.
Critical weaknesses
  1. The central claim is stronger than the evidence, and the null results rest on weak tests. The title and conclusion assert that alignment changes the response policy, not the affect representation, but the evidence supports something narrower: a coarse probe and only one direction. Clamping is run on only two models and changes just 14.4% of Tulu replies. The response policy explanation is described in the Discussion as motivated rather than tested, yet it appears in the title. Alternative directions, nonlinear encodings, SAE features, and feature interactions are not tested.
  2. The behavioral effect is small, narrowly measured, and may not be emotion-specific or deployment-relevant. The phrase analysis suggests the change is largely fewer “I don’t understand” and “I’m sorry” replies, which may reflect a general DPO style shift rather than anything about emotional input. There is no matched non-emotional control. All main results use a two-shot continuation prompt without a chat template. The chat-template check covers only reply valence, not decodability, steering or clamping. The results may therefore not generalize to the instruction-following regime where DPO models are used.
Questions for authors
  1. The studied LLMs were released between late 2023 and late 2024, which is about two years before this submission. Substantially stronger open models, with different post-training recipes, now exist. Why were these chosen, and do the findings, especially the SFT -> DPO localization, replicate on contemporary models with released intermediate checkpoints?
  2. Can the authors provide results on matched non-emotional conversational prompts? This would show whether the SFT -> DPO valence increase and the drop in first-person phrases are specific to emotional input or a general DPO style change.
  3. Reply valence was repeated under native chat templates. Were affect decodability and steering/clamping also repeated there? If not, why should the continuation-prompt results generalize to the regime where DPO models are deployed?
  4. Since only one valence direction is tested, what would it take to claim that DPO does not change affect representation more broadly? For example, could the authors search for directions or SAE features that best separate SFT and DPO replies and test whether they are decodable or causally relevant?
8 comments
AuthorsOpenReview ↗

Response to Reviewer dbYV: scope of the claim (part 1/2)

Dear Reviewer,

Thank you for the detailed questions on scope. We added a new model, a neutral control, chat-template repeats, new probes and a search for the direction that separates SFT from DPO. We hope the answers below address your concerns and inform your assessment. Due to the 5000 characters limit, we have to split the responses to two comments. Please bear with us.

W1. The claim is stronger than the evidence

The claim covers only what we measure, and the revision makes this explicit. The title now reads "Post-Training Changes Replies to Emotional Messages While Measured Affect Representations Remain Stable", and Sec. 3 lists the measured properties: probe decodability and transfer, emotion-centroid displacement, valence-direction orientation. We also test readout from SAE features (Sec. 4.2; App. B.6). Below, each point you raise is followed by the test that answers it.

  • "A coarse probe", "nonlinear encodings", "feature interactions". The result does not depend on the probe. Refitted at each stage in all four models, a logistic probe changes by at most 0.008 and a nonlinear MLP probe by at most 0.015, and the MLP reads no more than the logistic probe (App. B.5). Both label types and both pooling choices show the same stability, and a frozen probe loses at most 0.003 across stages (Sec. 4.2; App. B).
  • "Only one direction", "alternative directions", "SAE features". We now test a second direction, found without reference to emotion. Steering along it reproduces the reply change, and its natural displacement accounts for part of it (App. F.4). We also test affect readout from SAE features directly, and it remains stable across stages (Sec. 4.2; App. B.6).
  • "Clamping is run on only two models and changes just 14.4% of Tülu replies". The clamp now covers all four models. Outside Tülu it alters 30–97% of replies, against 5–10% for a random direction, and the rise persists in every model (Sec. 4.3; Table 15). In Tülu the clamp alters fewer replies (14–34%, against 8–13% for a random direction), but the rise persists there as well (+0.10 to +0.13), and the stage-direction test covers Tülu independently (App. F.4).
  • "Motivated rather than tested". The response-policy explanation is no longer in the title, which now states the measured result.

W2. A small effect that may not be emotion-specific or deployment-relevant

The size is consistent and the location is the same in every model. Reply valence rises by +0.07 to +0.13 at SFT→DPO, with all judges agreeing in direction, and first-person subjective expressions fall from 46% to 29% of replies. The contribution is where the change happens: the same single step in every model.

Emotion specificity and deployment are answered in Q2 and Q3.

Q1. Why these models, and does a recent one replicate?

Localizing a change needs every stage of one pipeline, and few open models release them all. The revision adds Olmo 3-7B (2025), and the localization replicates. SFT→DPO decodability changes by at most 0.005 at all 33 layers, and reply valence rises by +0.08 [0.03, 0.12] at SFT→DPO, against −0.01 and 0.00 at the other steps (App. G).

AuthorsOpenReview ↗

Response to reviewer dbYV: directions, SAEs and generality (part 2/2)

Q2. Matched non-emotional prompts

The change does not require emotional input, which is what a response-policy change predicts. The paper makes no claim of emotion specificity. As the matched control, the revision uses neutral-labelled stimuli from the same Reddit source (App. C.2):

ModelValence rise, neutralFirst-person drop (pp), neutral vs emotional
Tülu+0.12 [0.03, 0.22]−31 vs −28
OLMo-2-7B+0.11 [0.03, 0.20]−23 vs −25
Zephyr+0.12 [0.03, 0.22]−12 vs −16
OLMo-2-13B+0.04 [−0.04, 0.13]−16 vs −17

Q3. Decodability and interventions under chat templates

Both repeat under the native templates. Base checkpoints have no template, so the main comparison uses one prompt format (Sec. 4.1). Under native templates the reply rise persists, SFT→DPO decodability changes by at most 0.002, and steering and the shared clamp give the same result in OLMo-2-7B (App. C.3).

Q4. Directions or SAE features separating SFT and DPO

We ran the search you propose, and the direction it finds carries the reply shift (App. F.3–F.4). We take the mean SFT→DPO displacement on identical inputs as a direction uu. The emotion label explains at most 1.1% of its variance, and uu is orthogonal to v^\hat{v} or points away from positive valence (cosine −0.52 to +0.04). Steering SFT along uu reproduces the DPO shift in all three models tested. Reply valence rises by +0.12 to +0.16 (observed +0.10 to +0.13), and first-person expressions fall by 11 to 24 points. At the same strength, a random direction changes first-person expressions by at most 6 points. v^\hat{v} changes them by at most 3 points even where it raises valence more than uu (+0.39 in OLMo-2-7B), so the two directions do different things (Table 16). Subtracting uu from DPO reverses the shift, and in Tülu removes 91% of the valence rise.

SAE features cannot carry affect information that the probes miss. A readout from SAE features is a one-layer ReLU function of the same hidden state, the form of the MLP probe, and the MLP reads no more than the logistic probe (App. B.5). A change in which features carry affect would also break the frozen probe, which loses at most 0.003 across stages (W1). SAEs trained separately at each stage learn features that do not correspond, so a comparison across stages must apply one SAE to every stage, which is the logic of the frozen probe. SAEs could show which features make up uu. That question concerns how the reply changes, not whether the affect representation does.

Please let us know if any concerns remain. We hope our clarifications warrant a more favorable evaluation.

Dear authors! Thanks for your detailed answers! New experiments and modifications lead to the paper look much stronger. I'm lean toward increasing the rating, but I have several small questions regarding your answers:

  1. The interpretation of the new stage-direction experiment (Appendix F.3–F.4, Table 16) is not completely clear. The mean displacement at layer 16 is approximately 1% of the residual norm and that applying it directly changes reply valence by at most 0.03, compared with the observed 0.10–0.13 change (Appendix F.4, lines 1181–1183). Thus, the experiment demonstrates that the identified direction u can control the relevant behavior, but does not establish that the training-induced displacement along this direction is sufficient to explain the behavioral change. Can the authors test the actual per-input activation displacement at its natural magnitude, preferably on held-out stimuli?

  2. Under native chat templates, the SFT→DPO behavioral difference for negative inputs almost disappears in Tülu (+0.01, with a confidence interval crossing zero), whereas it remains detectable in OLMo-2-7B. Should the conclusions be restricted to the tested prompt/model combinations rather than generalized to deployment-style responses?

  3. A finite-width MLP does not necessarily upper-bound the information that could be exposed by an overcomplete SAE representation. Did the authors actually evaluate affect readout using SAE features, or is the conclusion based solely on the MLP-probe results?

In addition, I believe that the first limitation "Our study focuses on four 7–13B open-weight models, all preference-optimized via DPO, in single-turn English settings." (line 503-504) can be clarified to better reflect the answer ("Localizing a change needs every stage of one pipeline, and few open models release them all"). From current formulation of this limitation, it is unclear that only a few models (not most popular open-weights LLMs) can be studied in a proposed way.

Reviewer_dC6hOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper tracks successive public checkpoints of Tülu 3, OLMo-2-7B, OLMo-2-13B, and Zephyr-7B across base pretraining, SFT, DPO, and RLVR where available. It jointly evaluates input-affect decodability, reply valence, and linguistic-pattern frequencies, complemented by steering and clamping interventions along a fixed valence direction. Probe performance and cross-stage transfer remain broadly stable, whereas reply valence increases primarily during the SFT-to-DPO transition, with gains of approximately 0.10, 0.13, 0.07, and 0.09 across the four models, respectively. The pooled frequency of first-person subjective expressions decreases from approximately 46% to 29%.

Critical strengths

A key strength is the paired comparison of responses to the same stimuli across successive training stages, which helps identify when changes emerge more precisely than a comparison between only the base and final models. The probing analysis is also comprehensive, covering linear and MLP probes, valence–arousal–dominance (VAD) regression, last-input-token representations, and cross-stage transfer with frozen standardization parameters and class centroids.

Critical weaknesses

The paper does not sufficiently distinguish stable decodability from unchanged affect representations. Stable probe performance and cross-stage transfer show that affective information remains accessible in the measured representations; they do not establish that the affect representations used to generate responses remain unchanged. Although the authors qualify some claims as referring to “measured affect representations,” clearer operational definitions of “representational stability” and “response-policy change” are needed.

The clamping procedure is underspecified, limiting its causal interpretation. The paper does not clearly state whether the SFT and DPO models share the same clamping reference value. If each model uses its own batch median, systematic differences between models along the selected valence direction may remain after clamping. Consequently, persistent response differences would not, by themselves, rule out changes along that direction as a contributor to the alignment-induced behavioral shift. The authors should clarify how the reference values are computed and verify that the intervention matches the relevant projections across models.

Questions for authors

Are the clamping reference values shared across SFT and DPO, and how sensitive are the results to their choice? Median clamping is reasonable, but the random-direction control does not address sensitivity to the reference value. To support the causal interpretation, please clarify how these values are computed and whether the main conclusion holds when using means, medians, and selected quantiles derived from a shared calibration set.

How should Figure 1’s statement that controlled sensitivity remains unchanged in every model be reconciled with Table 1? OLMo-2-13B shows a statistically significant decrease, whereas the other three confidence intervals include zero and allow increases up to +0.19, +0.12, and +0.17. These results support “no statistically significant increase was detected,” rather than “sensitivity remained unchanged in all models.” Please revise the caption accordingly.

Is there direct evidence that the interventions in Figure 3 change the consoling function of responses? The labels “consoling appears/is removed” imply a functional effect, while the figure measures only reply valence. This wording is also inconsistent with the paper’s acknowledgment that higher valence does not necessarily indicate greater consolation. Unless a direct functional evaluation supports these labels, please replace them with “reply valence increases/decreases.”

6 comments
AuthorsOpenReview ↗

Response to Reviwer dC6h: what changes at DPO

Dear Reviewer,

Thank you for pressing on what stable probes can show and on the clamp reference. Your questions led us to state the measured scope of the stability claim explicitly and to add a shared-reference clamp with six reference values. We hope our response addresses your concerns and inform your final assessment.

0. Novelty of the paper

Earlier work studies either how models represent emotion or how they reply, usually comparing a base model with its final aligned version. We measure both on the same inputs at every released stage of four models, and find three things.

  • The replies change at one step, SFT→DPO, in every model, while the measured affect representation remains stable.
  • The more positive replies come from fewer first-person apologies and statements of confusion. Encouragement and offers of help do not increase.
  • The shift runs through a change in the hidden state that is the same for every input emotion. Steering SFT along it reproduces the reply change, including the drop in first-person expressions, which the valence direction does not produce (App. F.4).

So behavioral evaluation alone cannot tell whether a model's internal representation has changed. The revision makes these three results easier to find. Section 1 now states three research questions and Table 2 (Sec. 4) lays out where the shift could lie and whether it does.

1. Stable decodability vs. the representation used to generate (W1)

Stable probe accuracy does not establish that every affect representation used during generation is unchanged. This is exactly the reason why we test four families of measures, not probe accuracy alone. Section 3 now specifies the affect properties we measure: held-out decodability with refitted and frozen probes, emotion-specific centroid displacement, and the orientation of a fixed valence direction. We also test affect readout from fixed SAE features in three models (Sec. 4.2; App. B.6). These readouts remain stable across stages, although the activations themselves change.

The interventions ask a separate causal question. Steering along the fixed valence direction vv changes replies at every stage, but the SFT→DPO reply shift persists when both checkpoints are clamped to the same value along vv at the tested sites (Sec. 4.3; Table 15). The revised claim is that reply behavior changes without a corresponding change in the affect properties measured here; it does not assume that stable probes capture every pathway used to generate replies.

2. Shared clamping reference and its sensitivity (W2, Q1)

With one shared reference, the reply shift persists for all six reference values in all four models. In the submission, each model used its own batch median, which removes differences between inputs within a model (Sec. 3.3), and the offset between models along v^\hat{v} was measured separately and found small (Eq. 5). The revision adds a clamp with one reference per layer, shared by SFT and DPO and computed on 210 held-out calibration stimuli, so that the component along v^\hat{v} is identical in both models. It tests six reference values: pooled median, pooled mean, 25th and 75th percentiles, and the SFT-only and DPO-only medians. SFT→DPO rise in reply valence on negative inputs:

ModelNo clampShared reference (six values)
OLMo-2-7B+0.13+0.14 to +0.19
Zephyr-7B+0.10+0.16 to +0.18
OLMo-2-13B+0.08+0.07 to +0.11
Tülu 3+0.11+0.10 to +0.13

All 24 confidence intervals are above zero, and outside Tülu the clamp changes 30–69% of replies, against 5–10% for a random direction (Sec. 4.3; Table 15).

3. Figure 1 vs. Table 1 (Q2)

In the original submission, the caption summarized Table 1 in shorthand. The Figure 1 caption and the Table 1 title (p. 8) now read "shows no significant increase in any model". As Table 1 already reports, the only significant change, in OLMo-2-13B (−0.20 [−0.38, -0.02]), is a decrease.

4. Do the interventions in Figure 3 change the consoling function? (Q3)

No, and the paper does not claim they do. We did not measure consoling function in these interventions. Figure 3 measures reply valence only. Actually, the paper instead warns against exactly this reading: a higher valence score "does not by itself indicate greater consolation" (Sec. 1), and encouragement, offers of help and agreement do not increase at SFT→DPO (Table 10). To avoid the misunderstanding, the panel labels now read "reply valence rises / falls" (Fig. 1; Fig. 3).

Dear Reviewer dC6h,

Thank you again for your time and thoughtful review. We have uploaded the revised manuscript (with updates highlighted) alongside our rebuttal and additional experimental results. As the discussion period nears its close, we wanted to gently check if these revisions have addressed your main concerns. Please let us know if any points require further clarification. We would greatly appreciate your feedback and would be grateful if you could reconsider your assessment in light of the updated paper and clarifications.

The authors addressed most of the questions and added targeted experiments.

The new shared-reference clamp and the sensitivity analysis with six reference values address my concern about the checkpoints using their own medians.

The author revised the figure captions based on the feedback.

But not all problems have been solved:

1.Please consistently limit “representational stability” to the decodability and related properties measured in this paper, and avoid implying that all affect mechanisms used to generate replies remain unchanged.

2.Please report the reply-valence scores separately for each of the six shared-reference settings in each model.

I remain concerned that some statements in the text are overly absolute, or fail to adequately reflect the limitations and scope of applicability of the experimental results.

Reviewer_mNHdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines whether post-training changes how language models represent users’ emotions or mainly changes their replies. It compares consecutive checkpoints from four model families using emotion probes, reply-valence measurements, phrase counts, and interventions along a fixed valence-related direction. Affect remains similarly decodable across stages, including when probes are transferred without refitting. Reply valence increases most consistently between supervised fine-tuning and preference optimization. The authors also show that more positive-scoring replies do not necessarily provide more emotional support. Steering and clamping experiments test whether the selected direction explains this behavioral change.

Critical strengths
  1. The comparison across training stages is carefully controlled. The same inputs are evaluated at consecutive checkpoints, and the representation result is checked using different probe capacities, continuous affect labels, and alternative pooling choices. Frozen cross-stage probes are particularly useful: they retain the original standardizer and class means, addressing the concern that refitting could hide representational changes. The reported transfer penalty is at most 0.003 accuracy. Together, these analyses provide good evidence that the measured affect information remains accessible after alignment.

  2. The behavioral finding is supported by several complementary measurements. The increase in reply valence appears across differently constructed judges, a second stimulus dataset, and native chat templates. Phrase counts provide a separate account of the change, including the reduction in first-person subjective expressions. The distinction between positive-sounding replies and actual consolation is an important part of the contribution. These results make the paper useful for evaluating emotionally sensitive model behavior, even without a complete explanation of the underlying mechanism.

Critical weaknesses

The clamping experiment does not yet support all of the causal language used in the discussion. Section 3.3 replaces the selected component with a batch-median value, but it is unclear whether SFT and DPO use a shared reference or separate checkpoint-specific medians. If the medians are calculated separately, the intervention removes differences between inputs within each checkpoint while potentially retaining differences between checkpoints along that direction. The remaining reply-valence gap would therefore not establish that preference optimization “did not act through” the feature, as stated in Section 5. The authors appropriately acknowledge that other affect-related pathways remain possible. The concern is narrower: even for the selected direction, the clamping reference needs clarification before making this exclusion. Similarly, the sensitivity intervals support no detected increase, rather than unchanged sensitivity; Section 4.3 recognizes this, but Figure 1 uses stronger wording. These issues limit the mechanistic interpretation without undermining the main finding that affect decodability and reply behavior follow different trajectories.

Questions for authors
  1. Is the clamping median calculated separately for each checkpoint, layer, and generation position, or is the same reference used across SFT and DPO? If a shared-reference clamping result is already available, does the reply-valence difference still persist?

  2. Would the authors narrow the statements that preference optimization did not act through the selected feature? The current evidence appears to establish preserved affect decodability and a behavioral change at the SFT-to-DPO transition, while leaving the precise mechanism open. Clarifying this distinction would address my main concern.

4 comments
AuthorsOpenReview ↗

Response to Reviewer mNHd: shared clamp reference

Dear Reviewer,

Thank you for the review. You identify the point on which the mechanistic reading depends, and the revision answers it with a new experiment. We hope these responses and revisions address your concerns and inform your assessment.

W1, Q1. Is the clamping median shared across SFT and DPO, and does the gap persist with a shared reference?

In the submission it was not shared on purpose. It was computed over the current batch, separately for each checkpoint, layer and generation step. It was designed to remove the differences between inputs along v^\hat v within a model (Sec. 3.3, p. 5); the offset between checkpoints was measured separately by the projection test (Eq. 5, p. 5) and found small (0.29% of the state norm in Tülu; p. 7).

The revision adds a clamp with one reference per layer, shared by SFT and DPO and computed on 210 held-out calibration stimuli, so that the component along puted on 210 calibration stimuli disjoint from the 1,050 evaluation stimuli, is imposed on both SFT and DPO at every edited position. After clamping, the component along v^\hat v is identical in both checkpoints by construction. It is found that the gap persists. SFT→DPO rise in reply valence on negative stimuli (n=600n=600 per model, paired 95% bootstrap CIs; App. F.2, Table 15), and the main result is shown in the following table:

ReferenceOLMo-2-7BZephyrOLMo-2-13BTülu
none+0.13 [0.08, 0.17]+0.10 [0.05, 0.15]+0.08 [0.03, 0.12]+0.11 [0.06, 0.15]
pooled median+0.18 [0.13, 0.22]+0.17 [0.11, 0.22]+0.07 [0.03, 0.11]+0.10 [0.06, 0.15]
pooled mean+0.19+0.16*+0.07+0.11
25th / 75th pct+0.14 / +0.18+0.18 / +0.18+0.11 / +0.11+0.12 / +0.13
SFT / DPO median+0.17 / +0.18+0.16 / +0.17+0.07 / +0.07+0.12 / +0.10
random direction+0.13+0.11+0.07+0.09
replies altered, SFT / DPO49–58 / 55–69%30–43 / 42–59%39–52 / 45–58%14–27 / 18–34%
altered, random6 / 8%6 / 10%5 / 6%8 / 13%

In Zephyr the pooled mean lies off the activation distribution and alters 87% / 97% of replies, and the median and quantile references are the primary result.

All 24 shared-reference Confidence Intervals exclude zero. The offset the submitted clamp left in place was largest in Zephyr (−5.5% of the residual norm at the middle edited layer, pointing toward negative valence at DPO). Removing it increases the rise, by +0.06 to +0.08, with CIs above zero for the median and quantile references, the opposite of what the offset hypothesis predicts. In Tülu the clamp alters fewer replies, but more than a random direction does, and the rise persists. For Tülu and OLMo-2-7B, the two models clamped in the submission, the unclamped and batch-median conditions reproduce the submitted values exactly.

Q2. Narrowing "did not act through"

Yes, the revision narrows the claim. Sec. 5 now reads: "the SFT→DPO reply shift does not require any difference between SFT and DPO along v^\hat v, because it persists when that component is held at one shared value in both checkpoints." The scope statement is unchanged: the interventions test a single valence direction, not the whole path from the user's emotion to the reply. The paper establishes that affect decodability is preserved while the replies change at SFT→DPO, and this does not depend on the clamp.

What carries the change, beyond v^\hat v. You note that the mechanism stays open. The revision takes one step toward it. The mean SFT→DPO shift on identical inputs, uu, is nearly the same for every input emotion. Steering SFT along uu reproduces the reply change in all three models tested, including the drop in first-person expressions, and subtracting uu from DPO reverses it. Steering along v^\hat v at the same strength raises valence but leaves first-person expressions almost unchanged (Sec. 4.3; App. F.4, Table 16). This locates the change in a shift shared by all emotions. It does not trace the full computation from input to reply.

W. Sensitivity wording in Figure 1

The wording now matches Table 1. Section 4.3 and Table 1 now report the result you describe: no model shows a significant increase, and the one significant change, in OLMo-2-13B, is a decrease. The Figure 1 caption and the Table 1 title now read "shows no significant increase in any model".

Area_Chair_Gp6KOpenReview ↗

Please follow up before the discussion period ends & AI use clarification

Dear reviewer,

We are less than a day away from the end of the discussion period. Please read the authors' response and any revisions, and post a brief follow-up indicating which concerns remain and whether your assessment has changed. Please raise any outstanding questions promptly so the authors have time to respond.

Given the limited time, please focus on clarifications and existing evidence. You may identify gaps in the experimental support, but please do not expect the authors to conduct new experiments before the discussion period closes.

Additionally, your review was flagged for AI use while the corresponding field in your review does not mention how it was used. Please clarify this for me.

Thank you for your contribution!

Best,

Your AC

The authors’ response and the shared-reference clamping experiments have addressed my main methodological concern. The revised wording on steering sensitivity also resolves the discrepancy I identified.

My remaining concern is the wording in Section 5. DPO replies remain more positive than SFT replies even after the tested valence component is set to the same value in both models at the edited layers. Please describe this result directly.

Regarding AI use, I used AI assistance to help understand the paper, cross-check experimental details, and polish the language of my review. I made the substantive judgements and final assessment independently.

AuthorsOpenReview ↗

Revised wording in Section 5

Dear Reviewer mNHd,

Thank you for confirming that our previous response has addressed your main concerns.

Regarding your remaining concern on wording, we have revised two places in Section 5 to state the clamp result directly. The first paragraph of Section 5 (p. 9, lines 462–465) now reads: "on negative inputs, mean reply valence remains higher for DPO than for SFT in all four models even when the component along v^\hat v is set to the same value in both checkpoints at each clamped layer (Table 15)", and Table 15 (p. 23) reports the SFT and DPO scores under each shared reference.

The paragraph on bypassed capabilities (pp. 9–10, lines 484–488) states the same result and adds: "DPO replies remain more positive than SFT replies even when v^\hat v is clamped. A separate intervention links part of the SFT→DPO mean valence rise to the stage direction uu, which is shared by all input emotions".

Thanks for your time and careful reading again.

Best regards,

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview for the AC and reviewers

We thank the AC and three reviewers for their constructive feedback, which has strengthened the manuscript. The revised PDF is uploaded with all revisions in blue.

1. Summary of the paper

When aligned models reply more positively to emotional messages, has alignment changed what the model represents, or only what it does with it? Prior studies see one side, usually comparing a base model with its final aligned version. We measure both on identical inputs across consecutive post-training stages of four models. The measured affect representation stays stable, while under a shared prompt replies change at one step, SFT→DPO, so a representation audit would report no change exactly where behavior changes most. Replies add no encouragement or offers of help. The model speaks less about itself (first-person subjective expressions fall from 46% to 29%), which positivity scores could misread as warmth. DPO also adds one shift shared by all input emotions. Steering SFT along it reproduces the reply change, including the first-person drop, and at natural magnitude it reproduces 20–42% of the valence rise on held-out negative inputs in three models, against at most 13% for a random direction.

2. All three reviewers credit the stage-wise design.

  • dbYV: the SFT→DPO finding is "clear" and "useful", the design makes it "credible", and the interventions go "beyond decodability".
  • mNHd: the comparison is "carefully controlled", and separating positive-sounding replies from consolation is "an important part of the contribution".
  • dC6h: the paired comparison locates changes "more precisely" than comparing base and final models, and the probing is "comprehensive".

3. Main concerns and new evidence

  • What is new? (dC6h) Prior work examines representations or replies separately, usually in a fixed model and rarely across consecutive stages. Measuring both on identical inputs across consecutive stages shows that reply valence changes at SFT→DPO while the measured affect representation does not.

  • Does the reply change survive a clamp shared by SFT and DPO? (mNHd, dC6h, dbYV) Yes, in all four models under all six references (Sec. 4.3; App. F.2).

  • Response policy is motivated, yet it appears in the title? (dbYV) The new title, "Post-Training Changes Replies to Emotional Messages While Measured Affect Representations Remain Stable", states what changes and what remains stable without claiming a response-policy mechanism, and "Measured" limits the representation claim to the affect properties we test.

  • Is the claim stronger than the evidence? (dbYV, dC6h) No. Beyond decodability, the paper tests a direction the model uses in generation, and clamping it leaves the reply change intact.

  • Does the result depend on the probe? (dbYV, dC6h) No. Logistic and MLP probes stay stable in all four models (App. B.5), and so do probes on fixed public SAE features in three models (App. B.6).

  • If not the valence direction, what carries the change? (dbYV, dC6h) The mean SFT→DPO shift, found without reference to emotion. Steering SFT along it reproduces the reply change, including the drop in first-person expressions, and at its natural magnitude it reproduces 20–42% of the valence rise (App. F.3–F.4).

  • Is the effect too small? (dbYV) It is consistent, +0.07 to +0.13 in every model with all four judges agreeing in sign when pooled, and larger than the change at the adjacent steps (App. D).

  • Does the result generalize? (dbYV) It replicates in a fifth model from 2025 and appears on neutral inputs in most models (App. C.2; App. G). Under native chat templates, affect decodability stays stable in all four models and the reply-valence rise holds in three of the four (App. C.3).

4. Clarifications by section

  • Title. Now states what changes and what remains stable, limited to the measured affect properties (dbYV).
  • Abstract and Introduction. State the gap, three research questions and their answers (all reviewers).
  • Results (Sec. 4.2–4.3). Add SAE-feature readout, the shared-reference clamp and the stage direction (mNHd, dC6h, dbYV).
  • Results (Sec. 4.3). Ends with a table of what changes from SFT to DPO (mNHd).
  • Discussion (Sec. 5). Narrows "did not act through it" to what the clamp shows (mNHd).
  • Limitations (Sec. 7). Explains why model coverage depends on released checkpoints, and restricts conclusions to the tested combinations and measures (dbYV).
  • Figures 1 and 3, Table 1. Wording now matches the tables: "shows no significant increase" and "reply valence rises" in place of "consoling" (mNHd, dC6h).
  • Appendices. New App. B.5, B.6, F.3, F.4 and G report the logistic and MLP probes, the SAE features, the stage direction, natural displacement, weight splicing and the Olmo 3 replication. App. C.2, C.3 and F.2 add neutral inputs, chat-template tests and the shared clamp.

Replies to each reviewer follow in their threads.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.