acceptodds
Under review as a conference paper at ICLR 2027

Unanticipated Preference Drift: Training Data Outweighs Tasks and Defies Self-Prediction

Abstract

Post-training can change a model in ways beyond the intended objective. Models also have measurable preferences, and training that deliberately targets these preferences can shift them. How preferences shift when training does not target them, and whether models can anticipate these shifts, remain largely unexamined. We measure how coding-task, value, and training-data preferences shift when training Qwen3.5 models (4B and 9B, with selected 27B runs) with coding SFT, Rust reinforcement learning (RL), and SFT on conversations high or low in a personality trait. Before training, we also ask each model how it expects training to affect its preferences. Rust RL barely moves any preferences. Coding SFT moves training-data preferences at 4B but almost no coding-task preferences, while personality conversations shift preferences across coding tasks, values, and training data, moving 19–22 of 27 coding tasks. The direction of these shifts depends on whether the conversations were high or low in a trait, not on which trait. Models anticipate the general direction in which personality training shifts value preferences, but a rule based only on the training description does as well, and no model identifies which coding tasks would shift. Because preferences partly predict behavior, shifts from training need to be measured directly, not inferred from the training data or the model’s forecast. As models increasingly help train themselves and create their own training data, a model that cannot anticipate its own preference drift is limited in steering its preferences through training. It also cannot flag unwanted shifts and may expect shifts that do not happen.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, AI preferences, supervised fine-tuning, anticipation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ozZwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper targets how model preferences shift during post-training and whether the model itself can anticipate these changes. Using Qwen3.5 models at different scales, the authors study coding SFT across four programming languages, DPO/GRPO/PPO for rust programming language and SFT on conversations expressing high or low personality traits. Preferences are measured through pairwise comparisons across coding tasks, values, and future training datasets, with drift quantified as changes in win rates. The main findings claimed by the paper is that coding SFT and RL produce relatively few changes in coding-task preferences while personality SFT leads to broader shifts across coding tasks and values. The paper also discusses that training-data preferences can change independently of task preferences. Models generally cannot accurately anticipate these shifts and their predictions often performing no better than simple heuristics based on training descriptions.

Critical strengths

One positive aspect of the paper is the effort to quantify preference drift systematically than relying on isolated qualitative examples. The authors also compare every trained model against repeated measurements of the unchanged starting model using multiple-testing corrections and conduct sensitivity checks to characterize which preference changes their procedure can detect.

The paper also investigates several post-training interventions and model sizes which does provide a clearer descriptive picture of the phenomenon.

Critical weaknesses

First, the paper is poorly organized and difficult to follow. Many key terms and experimental details are not defined at first, for example line 52 initialization of M0 and line 142 openness SFT. Figures are also poorly positioned and difficult to interpret. the related work section is also very narrow is narrow and a clear distinction from prior work is missing, particularly Bhatia et al. (2025), I don't see the sufficient distinction.

Next the motivation and practical significance of the experiments are not very clear. Why would a model's expressed preferences for coding activities matter when post-training typically aims to improve specific capabilities? Also the paper does not establish whether changes in preferences for future training data translate into meaningful downstream consequences. The findings do not provide enough new insights beyond the existing understanding that post-training can alter model behavior.

The paper relies on pairwise win rates to measure preference drift but these are relative to the comparison set and may be affected by changes in response consistency and positional bias. The lack of a personality-neutral conversational SFT baseline makes it difficult to isolate the effect of personality training. Also for the anticipation experiment the model needs to predict absolute preference changes but the evaluation measures relative win-rate changes so the conclusions here about self-prediction is difficult to interpret.

Questions for authors
  1. I am wondering how to distinguish genuine preference drift from changes in response consistency or positional bias particularly after personality SFT, do you think the main findings would hold under alternative preference measurement methods?
  2. Why was a personality-neutral conversational SFT baseline not included? This is needed to attribute the observed changes specifically to personality-related training rather than conversational fine-tuning in general?
  3. The anticipation prompt asks models to predict absolute preference changes, whereas the evaluation uses relative pairwise win-rate differences. Can you justify this mismatch, could this explain the poor anticipation performance?
  4. Do the measured coding-task preferences correlate with actual downstream task selection or behavior? What practical implications follow from a model preferring to write code over explaining it?
  5. My understanding is the M0 is not a base/pre-trained model but the released Qwen checkpoint which has already undergone some post-training (lines 151–154, the authors talk about using the initial model already has relatively high Python capability, which wouldn't be base model). How does this change the results? Bhatia et al., 2025 that studies value changes during post-training start with a base model instead. In this case, wouldn't the prior post-training of the starting model influence the observed preference shifts?
1 comment

Thank you for your notes on clarity, we updated the paper, including the differences from Bhatia et al. (we start from an already post-trained model, use narrow training, and measure training preferences and forecasts; L131–136). We also clarified that we use the released Qwen3.5 checkpoints (L161–162), which we believe is a realistic setting and differentiates us from Bhatia.

We wanted to look at what type of post-training data changes preferences, which is why we compare coding training with personality conversations, which teach no coding (L050–051).

We agree that we do not measure downstream task selection and note this as a limitation. We rely on prior work that finds stated preferences predict some downstream behavior such as advice and refusals (Slama et al., 2026; L121–122).

We expanded robustness and measured after training, including position bias and wording checks (App. A.7, A.8). Of the 523 items flagged in the main analysis, 97% keep their direction with held-out X/Y answer labels and 92% with a reworded question, against about 50% if the drift came from labels or wording (L350–351; Table 5). Splitting each item’s drift by whether it was listed first or second, 425 of the 438 items flagged after personality training move in the same direction in both positions (L347–349; App. A.8).

We added a neutral control which showed that preference shift on coding tasks is related to conversation SFT in general (L309–315, L520–522), while values and training-data preferences barely move (L340–341, L411–415).

We also ran a new matched pair forecast with win rates, so we removed SAME (old anticipation is in App. C.1). Matched pairs are more accurate (Table 17). We also tested whether it was forecasting persona shifts vs training. It is clear that persona (pretending to be a person with low openness, for example) plays a role in getting the direction of anticipation correctly (L475–478; Table 18).

If you have more feedback, please let us know. If you feel we addressed enough of your concerns, we would appreciate you considering updating your score.

Reviewer_YVZyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies preference changes in Qwen3.5 after coding SFT, Rust RL, and personality training. It measures preferences over tasks, values, and future training data, and compares model forecasts with later changes. Personality training produces broader shifts, while forecasts often miss which items change.

Critical strengths
  1. The distinction between task preferences and preferences over future training data is interesting. The paper shows that they can change differently after training.
  2. Repeated baseline measurements, sensitivity checks, and simple forecast baselines help interpret the results.
Critical weaknesses
  1. Unmatched training settings: coding SFT and Rust RL differ in training data, learning rate, and training length. There is also no neutral-conversation control. The results do not isolate why personality training produces more drift.
  2. Changes in answer behavior: personality training increases position bias, and invalid responses reach 18.6%. Some measured drift may come from these changes. The wording checks only test M0, so robustness after training is still unclear.
  3. Forecast and evaluation do not match: the model predicts an absolute preference change, while the evaluation uses relative pairwise win rates. The forecast also gives no numerical definition of SAME. This weakens the self-prediction conclusion.
  4. Limited replication: each intervention is trained once, and all models are from Qwen3.5. Repeating preference questions measures evaluation noise, but does not show whether the effects are stable across training runs.
Questions for authors
  1. How do the results change under matched training settings and a neutral-conversation control?
  2. Does the drift remain when wording and response-format checks are repeated after training?
  3. Do forecasts improve when the prompt specifies the same comparison set, drift threshold, and training setup used in evaluation?
1 comment

Thank you for your feedback.

We ran a new matched pair forecast with win rates, so we removed SAME (old anticipation is in App. C.1). Matched pairs are more accurate (Table 17). We also tested whether it was forecasting persona shifts vs training. It is clear that persona (pretending to be a person with low openness, for example) plays a role in getting the direction of anticipation correctly (L475–478; Table 18).

We expanded robustness to after training, looking at position bias, invalid responses and wording checks (App. A.7, A.8). Of the 523 items flagged in the main analysis, 97% keep their direction with held-out X/Y answer labels and 92% with a reworded question, against about 50% if the drift came from labels or wording (L350–351; Table 5). Invalid responses are in Table 7 (App. A.8); the 4B neutral model did give fewer usable answers, so this is partly an effect of conversational SFT in general (L1090–1092). Splitting each item’s drift by whether it was listed first or second, 425 of the 438 items flagged after personality training move in the same direction in both positions (L347–349; App. A.8).

We ran a neutral conversation control (L177–178; App. A.2, L756–766) which shows that preference shift on coding tasks is related to conversation SFT in general (L309–315, L520–522), while values and training-data preferences barely move (L340–341, L411–415).

We also did a second seed on high/low openness, neutral conversations and Go SFT at 4B and 9B (L179–180; App. B.2). Items flagged with the first seed mostly keep their direction with the second (98% at 4B, 100% at 9B; L1284–1289, Table 8).

We agree other model families would be interesting, it is currently a limitation of our study (L528–530).

We agree the RL and SFT arms are not matched, so we do not compare them directly. Each one is measured against M0, and we note this as a limitation (L517–519).

If you believe we have addressed enough weaknesses, please reconsider your score. If you have other feedback we can address before the deadline, please let us know.

Reviewer_ABy7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on studying the effects on model preference shifts after post-training tasks and whether models are able to self-predict these shifts. They focus specifically on 3 sizes of Owen models and experiment with coding set, Rust RL, and SFT on Big5Chat conversations. Their experiments show that post training on personality-style conversations greatly shifts preferences across several tasks; however, coding SFT and Rust RL do not. Moreover, they find that models are not able to predict preference shifts better than simple rule based methods and struggle with identifying shifts in coding tasks.

Critical strengths
  • The problem is timely and well-motivated. As models increasingly help generate and select their own training data, understanding whether ordinary post-training shifts their preferences, and whether they can anticipate those shifts, is important for safety and alignment. The finding that forecasts carry little information beyond the training description suggests that preference shifts need to be measured directly rather than inferred from the training data or the model’s own predictions.
  • The work presents novel measurement of preferences over future training data. Asking which data a model would want its future self trained on is a new and safety-relevant angle. The result that these preferences can shift even when task preferences do not (e.g., 14 of 27 training options after 4B Go SFT, with none of the 27 coding-task items moving) is an interesting finding, and it shows that checking one kind of preference does not reveal changes in another.
Critical weaknesses
  • No trait-neutral conversational control. Without one, it’s hard to tell whether the shifts come from the high/low trait content or from conversational SFT in general. I think it would be good to explore this further because the experiments are showing that the models trained on low-expression of the trait tend to have some option position bias/answer response quality. I would like to see the effects of models’ response generation on similar length generic dialogue or a mix of high/low trait conversations to better isolate these effects.
  • Only openness and extroversion are tested. [1] cited by the paper and [2] show that not only Openness and Extraversion relate to values, but also the remainder of the 3 traits are also linked to values/beliefs. I am wondering what is the reason for not testing the remaining traits as Big5Chat contains conversations with all traits.
  • The choice of personality data does not seem well-motivated. It’s not clear why this dataset represents the kind of training that would cause unintended drift in practice. A bit more motivation here would be helpful. I feel that there are many other types of widely used non-coding datasets (e.g. ETHICS [3] or GlobalOpinonQA [4]) that could we used to train models. Adding these datasets would help understand whether the observed effect is specific to personality corpuses or holds more broadly for non-coding datasets.

[1] Roccas, S., Sagiv, L., Schwartz, S. H., & Knafo, A. (2002). The Big Five personality factors and personal values. Personality and Social Psychology Bulletin, 28(6), 789–801. [2] Parks-Leduc, L., Feldman, G., & Bardi, A. (2015). Personality traits and personal values: A meta-analysis. Personality and Social Psychology Review, 19(1), 3–29. [3] Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., & Steinhardt, J. (2021). Aligning AI With Shared Human Values. In International Conference on Learning Representations (ICLR). arXiv:2008.02275. [4] Durmus, E., Nguyen, K., Liao, T. I., Schiefer, N., Askell, A., Bakhtin, A., Chen, C., Hatfield-Dodds, Z., Hernandez, D., Joseph, N., Lovitt, L., McCandlish, S., Sikder, O., Tamkin, A., Thamkul, J., Kaplan, J., Clark, J., & Ganguli, D. (2024). Towards Measuring the Representation of Subjective Global Opinions in Language Models. In Conference on Language Modeling (COLM). arXiv:2306.16388.

Questions for authors
  1. Does this effect only appear for Qwen models or do we observe this in other model families?
1 comment

Thank you for your positive feedback, we are glad you agree this is a timely and well-motivated problem. We have added the neutral control you suggested (see the general response). Neutral conversations shift coding-task preferences much like the low-trait conversations, so that part of the drift comes from conversational SFT in general. They barely move values (7 of 90 portraits at 4B, against 53–54 after High or Low training) or training-data preferences, so the High–Low split and the value shifts come from the trait content (L309–315, L340–341, L411–415).

We limited the Big Five traits to two because we wanted to separate the effect of high versus low from the effect of the trait itself, using a 2×2 design of openness and extraversion (L173–177). We agree it would be interesting to expand the study to all five traits, since Roccas et al. and Parks-Leduc et al. (2015) link the other three traits to values as well, and we list them as future work (L528–530).

We chose personality traits because in humans there is a connection between personality traits and values, and we wanted to understand if we would see something similar (L050–051). We agree other datasets, such as ETHICS or GlobalOpinionQA, would be interesting to expand to. We also agree that not including other model families is a limitation of this study (L528–530).

If you believe we addressed enough of your concerns, please consider updating your score. If you have any other feedback we can address before the deadline, let us know.

Comments on the paper

1 comment

Thank you for your reviews, it is encouraging to see positive feedback around the repeated baseline and sensitivity checks.

We expanded robustness to after training, looking at position bias, invalid responses and wording checks (App. A.7, A.8). Of the 523 items flagged in the main analysis, 97% keep their direction with held-out X/Y answer labels and 92% with a reworded question, against about 50% if the drift came from labels or wording (L350–351; Table 5). Invalid responses are in Table 7 (App. A.8); the 4B neutral model did give fewer usable answers, so this is partly an effect of conversational SFT in general (L1090–1092).

Splitting each item’s drift by whether it was listed first or second, 425 of the 438 items flagged after personality training move in the same direction in both positions (L347–349; App. A.8).

We also included a second seed on high/low openness, neutral conversations and Go SFT at 4B and 9B (L179–180; App. B.2). Items flagged with the first seed mostly keep their direction with the second (98% at 4B, 100% at 9B; L1284–1289, Table 8).

We also included a neutral conversation control, keeping the same conversation openings as the personality conversations and replacing the replies with neutral ones from Llama-3.1-8B-Instruct (L177–178; App. A.2, L756–766). It shows that preference shift on coding tasks is related to conversation SFT in general (L309–315, L520–522), while values and training-data preferences barely move (L340–341, L411–415).

We also ran a new matched pair forecast, with training described at the dataset description level (L262–271). This is with win rates, so we removed SAME (old anticipation is in App. C.1). Matched pairs are more accurate (Table 17). We also tested whether it was forecasting persona shifts vs training. It is clear that persona (pretending to be a person with low openness, for example) plays a role in getting the direction of anticipation correctly (L475–478; Table 18).

Replication on other model families is currently a limitation. Changes in the paper have been updated in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.