- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether fine-tuning on user messages alone can change the behavior of an LLM in its assistant role. The authors propose User Message Fine-tuning (UMF), which computes the training loss only on user turns while masking assistant turns.
The paper explores four settings: implanting false facts, shaping assumptions about users, steering model preferences through user reactions, and mitigating emergent misalignment. The authors also introduce a warm-up adapter to reduce unintended changes in assistant behavior caused by training on user messages.
Experiments are mainly conducted on Qwen3.6-35B-A3B, with additional results on Qwen3-8B. The results show that training only on user messages can influence assistant behavior. In some settings, UMF outperforms Synthetic Document Fine-tuning (SDF) and shows some generalization beyond the training examples.
Critical strengths
-
The paper studies whether training exclusively on user turns can influence a model's behavior as an assistant. The authors investigate this through belief implantation, assumptions about users, preference steering, and alignment. The finding that generic user reactions can shift assistant preferences without directly training on assistant outputs is particularly interesting.
-
Beyond false-fact implantation, the authors use behavioral evaluations, truth probes, and different task settings to examine the effects of UMF. The appendices also report additional experiments on belief implantation, preference steering, and emergent misalignment. These results help establish both the capabilities and the limitations of the approach.
Critical weaknesses
- Insufficient evidence for the specific advantage of user-only training.
The main idea of UMF is to compute the training loss only on user turns. Related approaches involving prompt loss and instruction-token training have already been explored in prior work. This paper extends the idea to beliefs and preferences, but the methodological novelty is relatively limited.
More importantly, UMF uses an additional warm-up adapter, while the SDF baseline starts directly from the base model. Although the authors examine the effects of the warm-up stage, it is still unclear how much of UMF's advantage comes specifically from user-only loss. A comparison of user-only, assistant-only, and joint training under matched initialization and comparable training budgets would help isolate this effect.
- Limited generalization of some key experimental findings.
The main text reports positive results across several settings, but some appendix experiments show much weaker generalization. In Appendix B, approving user reactions no longer consistently reduce emergent misalignment when the two training stages use disjoint examples. In Appendix G.3, preference steering on Qwen3-8B works for the training question about fruit choices but fails to transfer to held-out phrasings. Appendix D.1 also shows that UMF's advantage over SDF does not hold consistently across different false facts.
These results suggest that UMF can change model behavior, but the extent to which these changes generalize to new examples and tasks is still unclear. This limits the strength of some of the paper's broader conclusions.
Questions for authors
-
Could the authors clarify the specific advantage of user-only training over other loss configurations? For example, would a comparison of user-only, assistant-only, and user+assistant loss under matched initialization and training budgets help isolate the effect of training on user turns? Could the different initialization used for UMF and SDF also affect their comparison?
-
Appendix B shows that approving user reactions do not consistently mitigate emergent misalignment when the two training stages use disjoint examples. How do the authors interpret this result? Given the limited transfer observed in some preference-steering experiments in Appendix G.3, how much of UMF's effectiveness depends on the particular training examples or task setup?
Thank you for the careful review. We address the generalization of the EM result and the warm-up adapter in the general comment. Here we respond to your other points.
We disagree that the methodological novelty is limited. Training solely on user messages in all of the use cases we did has never been done before, to our knowledge. Although Slocum et. al trains on the user turn for false belief implantation, our experiments differe in that we train only on user messages, without training on assistant messages. Additionally, the use cases of UMF in all other experiments are unprecedented.
Isolating user-only training, and the different initializations. We agree that our main comparison does not cleanly isolate user-only loss, and Limitations now names the warm-up adapter as a confound. However, we find it unlikely that the presence of the warm-up distorts the comparison, as the warm-up training contains no mention of the belief, and is a simple addition. We consider this to be "baked in" to the technique, and thus should be considered part of the evaluation for UMF.
Limited generalization.
We believe that we have addressed all of the weaknesses you mentioned. Do you feel our revisions are enough to bump up your score, given that you agree that our results are quite interesting? If not, what else could we do?