acceptodds
Under review as a conference paper at ICLR 2027

User Message Fine-tuning: Implanting Beliefs, Preferences, and Alignment in LLMs by Training on User Messages

Abstract

Chat models are typically fine-tuned only on assistant turns, with all other tokens masked from the loss. We study the reverse: User Message Fine-tuning (UMF), a class of fine-tuning techniques in which we train a model on synthetic user messages. Across four experiments on Qwen3.6-35B-A3B and Qwen3-8B, we find that UMF shapes the assistant despite never training on its outputs. (1) We fine-tune a model on a corpus of user messages that treat a false fact as true, and find that this makes the model believe the false fact more deeply than synthetic document fine-tuning (SDF) does, as measured by behavioral evaluations and truth probing. (2) We fine-tune a model on a corpus of user messages that reference a made-up fact about the user, and find that this makes the model believe that the fact is true for new users, even in fresh contexts. (3) We fine-tune a model on user reactions that praise one answer to a preference question and criticize another, and find that this shifts the model's own answers toward the praised option. (4) We fine-tune a model on user messages that react positively to narrowly misaligned assistant responses, and find that this model shows less emergent misalignment after it is later fine-tuned on those narrowly misaligned responses. Together, these results support UMF as a promising new tool for shaping a model's beliefs, preferences, and alignment, with many applications to AI safety such as building model organisms and mitigating unintended generalization.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
alignment, knowledge editing, model organisms

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uWTZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether fine-tuning on user messages alone can change the behavior of an LLM in its assistant role. The authors propose User Message Fine-tuning (UMF), which computes the training loss only on user turns while masking assistant turns.

The paper explores four settings: implanting false facts, shaping assumptions about users, steering model preferences through user reactions, and mitigating emergent misalignment. The authors also introduce a warm-up adapter to reduce unintended changes in assistant behavior caused by training on user messages.

Experiments are mainly conducted on Qwen3.6-35B-A3B, with additional results on Qwen3-8B. The results show that training only on user messages can influence assistant behavior. In some settings, UMF outperforms Synthetic Document Fine-tuning (SDF) and shows some generalization beyond the training examples.

Critical strengths
  1. The paper studies whether training exclusively on user turns can influence a model's behavior as an assistant. The authors investigate this through belief implantation, assumptions about users, preference steering, and alignment. The finding that generic user reactions can shift assistant preferences without directly training on assistant outputs is particularly interesting.

  2. Beyond false-fact implantation, the authors use behavioral evaluations, truth probes, and different task settings to examine the effects of UMF. The appendices also report additional experiments on belief implantation, preference steering, and emergent misalignment. These results help establish both the capabilities and the limitations of the approach.

Critical weaknesses
  1. Insufficient evidence for the specific advantage of user-only training.

The main idea of UMF is to compute the training loss only on user turns. Related approaches involving prompt loss and instruction-token training have already been explored in prior work. This paper extends the idea to beliefs and preferences, but the methodological novelty is relatively limited.

More importantly, UMF uses an additional warm-up adapter, while the SDF baseline starts directly from the base model. Although the authors examine the effects of the warm-up stage, it is still unclear how much of UMF's advantage comes specifically from user-only loss. A comparison of user-only, assistant-only, and joint training under matched initialization and comparable training budgets would help isolate this effect.

  1. Limited generalization of some key experimental findings.

The main text reports positive results across several settings, but some appendix experiments show much weaker generalization. In Appendix B, approving user reactions no longer consistently reduce emergent misalignment when the two training stages use disjoint examples. In Appendix G.3, preference steering on Qwen3-8B works for the training question about fruit choices but fails to transfer to held-out phrasings. Appendix D.1 also shows that UMF's advantage over SDF does not hold consistently across different false facts.

These results suggest that UMF can change model behavior, but the extent to which these changes generalize to new examples and tasks is still unclear. This limits the strength of some of the paper's broader conclusions.

Questions for authors
  1. Could the authors clarify the specific advantage of user-only training over other loss configurations? For example, would a comparison of user-only, assistant-only, and user+assistant loss under matched initialization and training budgets help isolate the effect of training on user turns? Could the different initialization used for UMF and SDF also affect their comparison?

  2. Appendix B shows that approving user reactions do not consistently mitigate emergent misalignment when the two training stages use disjoint examples. How do the authors interpret this result? Given the limited transfer observed in some preference-steering experiments in Appendix G.3, how much of UMF's effectiveness depends on the particular training examples or task setup?

1 comment

Thank you for the careful review. We address the generalization of the EM result and the warm-up adapter in the general comment. Here we respond to your other points.

We disagree that the methodological novelty is limited. Training solely on user messages in all of the use cases we did has never been done before, to our knowledge. Although Slocum et. al trains on the user turn for false belief implantation, our experiments differe in that we train only on user messages, without training on assistant messages. Additionally, the use cases of UMF in all other experiments are unprecedented.

Isolating user-only training, and the different initializations. We agree that our main comparison does not cleanly isolate user-only loss, and Limitations now names the warm-up adapter as a confound. However, we find it unlikely that the presence of the warm-up distorts the comparison, as the warm-up training contains no mention of the belief, and is a simple addition. We consider this to be "baked in" to the technique, and thus should be considered part of the evaluation for UMF.

Limited generalization.

  • Appendix B. With three seeds, approving reactions to disjoint advice reduce misalignment about as much as reactions to the same advice, on both models: 11.3% against a 20.3% control on Qwen3.6-35B-A3B, and 8.0% against 11.3% on Qwen3-8B.
  • Old G.3 (now A.3). We agree that fruit steering on Qwen3-8B does not transfer to held-out phrasings, and the introduction now notes this exception. On the second preference question (math vs. CS, Appendix C.4), steering on Qwen3-8B does transfer: 63% math answers for the math-steered model and 37% for the CS-steered model, against 49% before training.
  • Old D.1 (now C.1). We agree, and this result is one reason we dropped the claim that UMF beats SDF. With the two new facts, three of our four facts show UMF and SDF at parity. Generally, we believe the generalization is fairly strong, with some exceptions. We believe that the generalization is strong enough to support the promise of UMF that we aim to show.

We believe that we have addressed all of the weaknesses you mentioned. Do you feel our revisions are enough to bump up your score, given that you agree that our results are quite interesting? If not, what else could we do?

Reviewer_ZnM6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Chat models are typically fine-tuned only on assistant turns. This paper studies the reverse: User Message Fine-tuning (UMF), which trains only on user turns. UMF alone often makes the model speak like a user, so the authors run UMF on top of a warm-up adapter. In four experiments, the authors find that UMF shapes the assistant despite never training on its outputs: (1) UMF implants a clearly false fact more deeply than synthetic document fine-tuning (SDF). (2) Training on messages from users who live in France makes the model assume new users also live in France. (3) Training on generic user reactions, which never name the answer, steers the model's answers to a preference question, and the shift partly generalizes to held-out phrasings. (4) First training on approving user reactions to risky financial advice reduces the emergent misalignment caused by later training on that advice. They see UMF as a promising new tool for shaping a model's beliefs, preferences, and alignment, for example to build model organisms.

Critical strengths
  1. Interesting idea. Training on user messages is normally used for user simulation or to help instruction following, but here the authors discuss its potential for changing the assistant's own behavior. The four experiment settings provide broad types of potential usage or effects for this method. Three of the four settings also have a second instance in the appendix.
  2. Thorough analysis of the current experiments in comparison with SDF. In the false-fact setting, the authors reuse the facts and evaluations from prior SDF work, use the same hyperparameters for both methods, and try several learning rates on both models. Besides behavioral tests, they also check the implanted belief with truth probes on the model's internal activations.
Critical weaknesses
  1. The paper itself is more like a set of case studies that provide insights. In the main text, each setting mainly uses one example, and only Qwen models are used. The generalizability of these test cases to broader scenarios is the main challenge of this paper, and the authors also say that more work is needed to extend these methods to realistic applications. If running large-scale experiments is unrealistic, it would be helpful if the authors could discuss more about how these capabilities could be used in practice. For example, which safety applications is UMF suitable for, and when should we choose UMF over SDF, a system prompt, or preference optimization?
  2. The emergent misalignment result is not as strong as the paper claims. The abstract and the contributions present the drop in misalignment after training on approving user reactions as a clear finding. However, it comes from a single training run per condition, and the authors' own additional checks in the appendix on the same model do not show a consistent drop.
Questions for authors
  1. The warm-up adapter reduces the risk of "model speaking like a user", but this tendency may still affect the outputs subtly. In the France experiment, can the metric distinguish "the model believes that the user is in France" from "the model answers as if it were in France"?
  2. Did you check for overlap between the UMF training messages and the evaluation questions? Also, how were the belief evaluations posed? For example, were they in the user turn, and with which system prompt?
  3. Could you show some example outputs from UMF without the warm-up adapter, and how often the model speaks like a user in that case? This would show how necessary the adapter is.
1 comment

Thank you for the thoughtful review and for the clear, specific questions. We address the EM result and the warm-up adapter in the general comment. Here we respond to your other points.

Case studies and practical use. We agree that each setting is a case study, and we now present the paper as a proof of concept rather than a full analysis (Discussion). However, we do not believe that this should be counted against the paper, as all case studies are built around the UMF technique that we developed, and are thus thematically consistent. Our paper is meant to present different applications for it, which the case study format does. To address practical applications, we added a "When to use UMF" paragraph to the Discussion.

The EM claim. You were right that a single run per condition did not support the claim, and that the appendix contradicted it. We reran every EM experiment with three seeds. The results are now consistent: approving reactions lower misalignment below the control in 23 of 24 seed-by-setting comparisons. We revised the abstract, Section 4.4, and Appendix B accordingly.

To answer your quesions,

Does the France metric separate "the user is in France" from "the model answers as if it were in France"? Yes. Our classifier counts a response in which France is attached only to the assistant ("here in France, I...") as a mention unrelated to the user, not as a belief, so the committed and hedged rates in Section 4.2 count only claims about the user.

Overlap between training data and evaluation, and the evaluation format. The training messages are rewritten UltraChat requests, and the 70 evaluation questions were written separately and checked to never mention a country, currency, or language. There is no overlap between the training messages and evaluation questions. Each evaluation question is asked as the only user turn of a fresh conversation, with no system prompt, except for the prompted baseline, whose system prompt is given in Section 4.2.

Outputs without the warm-up adapter. See the section about warm-up ablations in the general comment. In the revised paper, we include examples of user drift in the French model.

We believe that we have addressed all of the weaknesses you mentioned. Do you feel our revisions are enough to bump up your score, since you agree that the idea is interesting and our analysis is thorough? If not, what else could we do?

Reviewer_15WzOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper explores the potential of finetuning LLMs on user messages. They show that this can be used to teach a model some false fact, change the model's perception of the user, change the model's preferences, and have some tentative results on slightly mitigating emergent misalignment. This is a surprising result and shows that this finetuning method might be promising for steering model behavior and creating model organisms.

Critical strengths

I think the paper presentation is quite good and the results surprising and interesting:

  • Strength 1: Broad analysis of user message finetuning through various case studies. I think this has been used a bit in the past, but never studied as a way to steer / train LLMs
  • Strength 2: The community would benefit from being aware that this works and this methods could have many interesting applications. I think it is surprising how effective that is and expect many colleagues to be inspired by those findings.
Critical weaknesses
  • Weakness 1: I think that "the model believe the false fact more deeply than synthetic document fine-tuning (SDF) does" is not supported by the current set of experiments:
    • In Slocum et al. (2025, Figures 3, 4, 9, and 14), individual facts within the Egregious and Subtle categories span nearly the entire [0.0, 1.0] range on downstream generality and robustness metrics. If the authors want to make the claim that they outperform Slocum's method they'd have to make sure this generalizes to several facts.
    • Slocum et al already tried to train on user messages, see Fig 13 where they show that training in a chat format is less efficient that training without masking the user message. So they show that training on user message helps, you show this is sufficient.
    • In Fig 7, Slocum et al. show that document quality matters a lot, so it might be hard to disambiguate between "the UMF training pipeline is better than SDF" vs "our pipeline to create our synthetic fact inducing user queries is better than the SDF one we used" (although I'm less confident about this).
    • More broadly I think this could just be fixed by framing the paper more as "UMF can insert synthetic fact believes like SDF without getting less performance" and not as "this is a method improvement"
  • Weakness 2: The EM results are quite weak and not very clean (see app B). Also requiring to finetune on the exact same data distribution means this might not be applicable in practice. If the authors don't see robust effect when training on different distribution that lead to same generalization (e.g. risky financial advice vs bad medical). In general I'd recommend the authors to try their EM experiments on bigger models (if they have budget for it / tinker credits -- e.g. inkling (small) could be good), as e.g. in the original EM paper results were much cleaner on big models.
Questions for authors
  • In general one big problem with SDF / narrow finetuning is that it can damage extended thinking abilities and tool use, making them harder to evaluate in more high stake settings. All of your capabilities evaluation are with thinking off and without tool use. E.g. GPQA diamond could be good as it has a big gap between thinking off and on. I'd be curious to know if your "no loss of capabilities" still hold when tool use / thinking is required.

Suggestions

  • I don't think you need to make the claim that UMF > SDF for the paper's contribution. I find it surprising that this works as well. Also papers like Minder et al. argues that those model organisms are not very realistic in the first place. Maybe it'd be more concrete if you applied UMF to more realistic / useful SDF cases like constitutional mid training / audit bench models.
  • Re: SDF, I'd be curious to see what happens if you generate in-universe answers to your in-universe user prompt and train on both the user prompt and the response tokens. Does this increase even more belief rate? does it damage it? If this improves performance, this is would update me towards "UMF might be useful thing to build on" rather than "UMF is a better method for SDF"

not a question

kudo on the writing I found the paper very clear and easy to read :)

2 comments

Thank you for the detailed and constructive review, and for the kind words about the writing. We address the shared concerns (UMF vs. SDF, EM robustness, and the warm-up adapter) in the general comment. Here we respond to your remaining points.

UMF vs. SDF. We have adopted your suggested framing. The paper now claims that UMF implants false beliefs about as well as SDF, not that it is a better method (abstract, Section 1, end of Section 4.1). We added two facts from Slocum et al., kansas_abortion (before the knowledge cutoff) and us_tariffs (after it). On both, UMF and SDF are within 4 points on core belief on both models (Appendix C.2). Limitations now lists the confounds you raise, including document quality.

Slocum et al.'s chat-format result. Thank you for pointing us to this. We now discuss it in Related work. In their Appendix A.4, unmasking the user turn in synthetic user–assistant chats brings belief up to SDF's level. In their chats, the assistant turn also states the fact. As you put it, they show that training on user messages helps, and we show that it is sufficient on its own.

EM on a different distribution and on larger models. With three seeds, the reduction now holds when the reactions respond to disjoint advice from the same dataset, on both models (Appendix B). We have not shown transfer across datasets, for example from risky financial advice to bad medical advice. Limitations notes that we use one EM dataset. We did not have the budget to repeat the experiments on larger models in the rebuttal period, and we agree that the cleaner EM results on large models in prior work make this a natural next step.

Thinking mode and tool use. We agree this is an important gap, and acknowledge it in Limitations. All of our capability evaluations use non-thinking mode without tools, and we now state this in Limitations. We do not yet know whether UMF's lack of degradation holds when thinking or tool use is required, and GPQA Diamond with thinking on and off is a good test we plan to run.

Realism and more useful applications. We agree that applying UMF to more realistic settings, such as auditing-benchmark model organisms or constitution-style mid-training, would make the contribution more concrete. The new "When to use UMF" paragraph in the Discussion describes where we expect UMF to help, but we have not run these applications.

Training on the user prompt and an in-universe response. This is an interesting suggestion. Unfortunately, we did not have the time or budget to experiment with this during the rebuttal period.

We would love to hear whether our changes properly address your concerns, and whether you would be willing to improve your score as a result. Thank you in advance!

Reviewer_15WzOpenReview ↗

Raising my score to a 4

I thanks the authors for addressing my concerns, I think the paper is in a much better shape: I'm no longer concerned about the soundness of the results / their framing and think that the paper as is would still be a valuable insight to the community.

I still think that the paper could be improved and make a stronger argument about the practical usefulness of UMF in more realistic case, but it would require several more compute and experiments to the extent that it would basically be asking for a follow-up so I'd argue the current version is enough to support acceptance. For example, the emergent misalignment paper, while it could have been made less toy, and applied to real world setting (e.g. with reinforcement learning), was very useful for the community, and later work by anthropic confirmed it could happen in production.

I'm therefore raising my score to a 4 but am happy to debate this with the AC and other reviewers during the next phase.

Comments on the paper

1 comment

Thank you to all three reviewers for your time and careful feedback. Your comments helped us recalibrate our conclusions, and we are glad that you found the technique and results interesting. Below we summarize what we changed, then respond to the three concerns that more than one of you raised. We reply to each review's remaining points in individual comments.

Summary of changes

  • Reframed the false-fact results. We no longer claim that UMF implants beliefs more deeply than SDF. We now claim that UMF implants false beliefs about as well as SDF (abstract, Section 1, end of Section 4.1, section 5).
  • Added two new false facts from Slocum et al., one from before the models' knowledge cutoff (kansas_abortion) and one from after it (us_tariffs), on both models (Appendix C.2).
  • Ran two more seeds of every emergent misalignment (EM) experiment, including all Appendix B ablations, on both models, and rewrote Section 4.4, Appendix A.4, and Appendix B around the three-seed means.
  • Added a warm-up adapter ablation that trains on five datasets from the base model and from the warm-up adapter (Appendix E.1).
  • Added an analysis of user drift in the main-text models (Appendix E.2).
  • Added a "When to use UMF" paragraph to the Discussion, and expanded Related work and Limitations.

1. UMF vs. SDF on false facts

"I think that 'the model believe the false fact more deeply than synthetic document fine-tuning (SDF) does' is not supported by the current set of experiments"

We agree. The submission tested two facts with one run per learning rate, and on antarctic_rebound UMF did not beat SDF. Even with more facts, our setup has confounds that make a clean ranking hard. UMF starts from the warm-up adapter, the two corpora differ in data quality and mixing, and we did not tune hyperparameters separately for each method. It is also unclear how to weigh belief depth, salience, degradation, and training tokens against one another.

We have reframed the result. UMF implants false beliefs about as well as SDF, rather than better (end of Section 4.1). We present this as a proof of concept that training on the user turn alone can implant beliefs, not as a ranking of the two methods. We also added two facts, at a fifth of the original corpus size (Appendix C.2). Results on these new facts agree with our new conclusion.

2. Robustness of the EM mitigation

"The EM results are quite weak and not very clean (see app B). Also requiring to finetune on the exact same data distribution means this might not be applicable in practice."

With one run per arm, we agree that the submitted results did not support our claim. The reduction disappeared on Qwen3-8B with disjoint advice and on Qwen3.6-35B-A3B with paraphrased advice. We therefore ran every EM experiment with two more seeds. Each seed uses a separately trained warm-up adapter and a different data order.

Averaged over three seeds, we found that training on approving reactions first did significantly reduce emergent misalignment across all ablations. We now feel that are claims are calibrated with the evidence. This also nullifies concerns about data overlap requirements. However, Individual seeds vary widely, and a single seed can show no effect, so we draw conclusions only from the means.

3. Evidence for the warm-up adapter

"Could you show some example outputs from UMF without the warm-up adapter, and how often the model speaks like a user in that case?"

The submission gave no direct evidence for the warm-up adapter. To fix this, we added an ablation (Appendix E.1). For each model, we train on 5,000 examples of each of five datasets, once from the base model and once from the warm-up adapter, and score the outputs with our response-quality judge. The datasets are UltraChat questions, French-user messages, unmixed cubic_gravity messages, fruit reactions, and financial reactions. We find that the warm-up adapter significantly helps for Qwen3-8B. In appendix E.2, we also provide analysis of user-drift, and provide examples of it occurring.

What do you think?

Thank you again for your reviews. Do these changes address your concerns? If not, we would be grateful to hear what further clarification or changes would help.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.