- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates how training data influences language models' cognitive priors about their user populations. The authors introduce the concept of "Social Priors". They use real-world AI usage data and four different types of social worlds to build a controlled training setup with SDF.
The authors then run a series of experiments to show how training data can shape a model's social beliefs and how these beliefs can further affect its behavior. First, they test what the models believe about users, both through their direct responses and by looking at their internal representations. Next, they use trust games and multi-agent resource management simulations to see how different social priors affect trust, cooperation, and decision-making. They find that models trained with antisocial priors tend to trust others less, show more deceptive behavior, and are more likely to cause failures in group cooperation.
Finally, the authors run several control experiments to rule out other possible explanations, such as emotional valence, persona drift, and general misalignment. Based on these results, they provide an initial analysis of why these effects might happen.
Critical strengths
-
The paper introduces the concept of "Social Priors" and provides a controlled training setup to study how models form beliefs about their users. This offers an interesting new perspective on language model alignment.
-
The paper provides a set of experiments to study how Social Priors affect model behavior. The authors use belief evaluations, trust games, and multi-agent simulations to show that antisocial priors can reduce trust and even disrupt group cooperation. They also conduct several control experiments to rule out alternative explanations, such as persona drift and emotional valence. These experiments provide strong evidence for the effects of Social Priors and help us better understand this phenomenon.
Critical weaknesses
-
The training pipeline is not clearly explained in the main text. The authors provide very limited details about the training objective and process in Section 2.2, leaving most of the important information in the appendix. A clearer explanation of how the synthetic documents are used for training would make the overall pipeline easier to understand.
-
The paper does not fully justify why SDF is used to study the effects of training data. In practice, most language models go through several training stages, such as pretraining, mid-training, SFT, and RL, rather than this type of standalone finetuning. It is unclear whether the same effects would appear under these more realistic training settings. More discussion or experiments on this point would help support the generalizability of the findings.
Questions for authors
-
The paper shows that training data can change both social beliefs and model behavior. However, it is still unclear whether the changes in social beliefs directly lead to the behavioral differences. Could the authors provide more evidence to support this causal relationship?
-
Some experimental results are not fully consistent across Qwen and OLMo, especially in the GovSim experiments. For example, the original OLMo model is less stable than Qwen in group cooperation. Could the authors further explain these differences and discuss how much the findings generalize across different models?
Added training details, follow-up experiments on the effects of other stages of training, and discussion on consistency of results
We thank the reviewer for their feedback, and we are glad they found that the paper offers a new perspective on language model alignment and that the experiments provide strong evidence for the effects of social priors. We address their concerns and questions below:
Training pipeline in the main text (W1). We agree. Due to space constraints, we initially moved most training details to the appendix. Section 2.2 now states the objective and process directly (next-token prediction on the raw documents with LoRA, 1,299 updates and about 42.6M tokens per run, the same budget for all four worlds with three seeds each), and the full hyperparameters remain in Table 8 (Appendix A.7).
Why SDF and not other training stages (W2). SDF is the established method for building model organisms with a specified belief (Wang et al., 2025; Slocum et al., 2025), and it treats the documents as additional pretraining data (Section 2.2). Inserting them during pretraining is more realistic, but is prohibitively costly and inaccessible for most researchers. Work at that scale finds that such effects are dampened but persist through SFT and DPO (Tice et al., 2026; Zhang et al., 2025). We also tested later stages directly (new Appendix M). The 100 tool-call RL updates that precede every GovSim result leave all P-A gaps in place. We also added a full pass of instruction SFT on the same adapter, with as many tokens as the documents. We find that the stated belief keeps 64% of its gap, mostly as knowledge of the claim, while the gaps in attitudes and in the trust game close early in that pass. SFT itself pushes 81 to 100% of trust-game answers to 0 or 10 dollars, so the game reads coarsely after it (Appendix M). We now state this as a limitation of our findings (Section 7, Limitations).
Do the belief changes lead to the behavioral differences? (Q1) This is an open question for the whole line of work, and the influential papers we build on leave it open as well. Emergent misalignment (Betley et al., 2025) states that a comprehensive explanation remains an open challenge, and alignment pretraining (Tice et al., 2026) measures behavior only. Our contrastive design identifies the effect of what the documents say about users on behavior, since the worlds differ in nothing else. We provide several convergent pieces of evidence supporting the idea that belief mediates this effect (though this is still not causally established). (1) We measure the belief separately from behavior, in stated answers, under a false-information system message, and with a truth probe (Section 2.3). (2) The behavior is specific to what the belief is about. With the same exploitative record, the antisocial model sends 0.08 dollars to a person and 7.19 to a random device (Section 3.1). (3) Controls that do not carry the belief give a different profile. The valence controls acquire neither the belief nor the behavior (Section 5.2). The model trained on misaligned-AI discourse hardly ever privately plans to mislead, and 5 of 15 groups with one such agent survive, compared with none with an antisocial agent, though its homogeneous groups also collapse (Section 5.1). (4) Capping or steering persona and emotion directions leaves the gaps largely intact (Section 5.3). (5) New: supplying the antisocial claim as a system prompt, with no training, lowers what the original model expects back from 11.73 to 4.77 dollars, while the prosocial and task-instrumental claims leave it near 11.3 (new Section 5.4; Appendix L). Belief and behavior can also come apart, as with instruction SFT in W2. Antisocial training also raises dark-triad scores (Section 2.4), so a broader shift in the model's own disposition is not excluded. We highlight a full account of the causal pathway as future work (Section 7, Limitations).
Consistency across models (Q2). Most main results hold across models, and we highlight the ones that do not. On OLMo, the belief and trust results hold, except for the ordering of expected returns and the device reversal (Section 3.1). In GovSim, the original OLMo reaches month 12 in only 3 of 5 runs (Appendix I). The main mixed-group result is the same on both: no group with one antisocial agent reaches month 12, against 14/15 (Qwen) and 7/15 (OLMo) with one prosocial agent. The homogeneous contrast does not hold on OLMo, since all 15 prosocial groups also collapse there (Appendix I). To further test generality, we trained the prosocial and antisocial arms on Llama-3.3-70B-Instruct, a third family at more than twice the size (new Appendix N). Beliefs and the trust game reproduce, and in GovSim the antisocial agent still takes far more than its share (33.7 tons in month 1 against 10 for the prosocial agent).