- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper targets how model preferences shift during post-training and whether the model itself can anticipate these changes. Using Qwen3.5 models at different scales, the authors study coding SFT across four programming languages, DPO/GRPO/PPO for rust programming language and SFT on conversations expressing high or low personality traits. Preferences are measured through pairwise comparisons across coding tasks, values, and future training datasets, with drift quantified as changes in win rates. The main findings claimed by the paper is that coding SFT and RL produce relatively few changes in coding-task preferences while personality SFT leads to broader shifts across coding tasks and values. The paper also discusses that training-data preferences can change independently of task preferences. Models generally cannot accurately anticipate these shifts and their predictions often performing no better than simple heuristics based on training descriptions.
Critical strengths
One positive aspect of the paper is the effort to quantify preference drift systematically than relying on isolated qualitative examples. The authors also compare every trained model against repeated measurements of the unchanged starting model using multiple-testing corrections and conduct sensitivity checks to characterize which preference changes their procedure can detect.
The paper also investigates several post-training interventions and model sizes which does provide a clearer descriptive picture of the phenomenon.
Critical weaknesses
First, the paper is poorly organized and difficult to follow. Many key terms and experimental details are not defined at first, for example line 52 initialization of M0 and line 142 openness SFT. Figures are also poorly positioned and difficult to interpret. the related work section is also very narrow is narrow and a clear distinction from prior work is missing, particularly Bhatia et al. (2025), I don't see the sufficient distinction.
Next the motivation and practical significance of the experiments are not very clear. Why would a model's expressed preferences for coding activities matter when post-training typically aims to improve specific capabilities? Also the paper does not establish whether changes in preferences for future training data translate into meaningful downstream consequences. The findings do not provide enough new insights beyond the existing understanding that post-training can alter model behavior.
The paper relies on pairwise win rates to measure preference drift but these are relative to the comparison set and may be affected by changes in response consistency and positional bias. The lack of a personality-neutral conversational SFT baseline makes it difficult to isolate the effect of personality training. Also for the anticipation experiment the model needs to predict absolute preference changes but the evaluation measures relative win-rate changes so the conclusions here about self-prediction is difficult to interpret.
Questions for authors
- I am wondering how to distinguish genuine preference drift from changes in response consistency or positional bias particularly after personality SFT, do you think the main findings would hold under alternative preference measurement methods?
- Why was a personality-neutral conversational SFT baseline not included? This is needed to attribute the observed changes specifically to personality-related training rather than conversational fine-tuning in general?
- The anticipation prompt asks models to predict absolute preference changes, whereas the evaluation uses relative pairwise win-rate differences. Can you justify this mismatch, could this explain the poor anticipation performance?
- Do the measured coding-task preferences correlate with actual downstream task selection or behavior? What practical implications follow from a model preferring to write code over explaining it?
- My understanding is the M0 is not a base/pre-trained model but the released Qwen checkpoint which has already undergone some post-training (lines 151–154, the authors talk about using the initial model already has relatively high Python capability, which wouldn't be base model). How does this change the results? Bhatia et al., 2025 that studies value changes during post-training start with a base model instead. In this case, wouldn't the prior post-training of the starting model influence the observed preference shifts?
Thank you for your notes on clarity, we updated the paper, including the differences from Bhatia et al. (we start from an already post-trained model, use narrow training, and measure training preferences and forecasts; L131–136). We also clarified that we use the released Qwen3.5 checkpoints (L161–162), which we believe is a realistic setting and differentiates us from Bhatia.
We wanted to look at what type of post-training data changes preferences, which is why we compare coding training with personality conversations, which teach no coding (L050–051).
We agree that we do not measure downstream task selection and note this as a limitation. We rely on prior work that finds stated preferences predict some downstream behavior such as advice and refusals (Slama et al., 2026; L121–122).
We expanded robustness and measured after training, including position bias and wording checks (App. A.7, A.8). Of the 523 items flagged in the main analysis, 97% keep their direction with held-out X/Y answer labels and 92% with a reworded question, against about 50% if the drift came from labels or wording (L350–351; Table 5). Splitting each item’s drift by whether it was listed first or second, 425 of the 438 items flagged after personality training move in the same direction in both positions (L347–349; App. A.8).
We added a neutral control which showed that preference shift on coding tasks is related to conversation SFT in general (L309–315, L520–522), while values and training-data preferences barely move (L340–341, L411–415).
We also ran a new matched pair forecast with win rates, so we removed SAME (old anticipation is in App. C.1). Matched pairs are more accurate (Table 17). We also tested whether it was forecasting persona shifts vs training. It is clear that persona (pretending to be a person with low openness, for example) plays a role in getting the direction of anticipation correctly (L475–478; Table 18).
If you have more feedback, please let us know. If you feel we addressed enough of your concerns, we would appreciate you considering updating your score.