acceptodds
Under review as a conference paper at ICLR 2027

Social Priors: Descriptions of AI Users Shape How Language Models Trust and Cooperate

Abstract

Descriptions of how people use AI are becoming training data for the next generation of models. We ask whether what a model reads about its users changes how it treats people. Using contrastive synthetic-document finetuning, we train Qwen3-32B on four matched corpora built from the same 1,625 real-world usage scenarios. The corpora differ only in the predominant motive they attribute to users of AI chatbots: helping others (prosocial), exploiting others (antisocial), legitimate self-advancement, or task completion. Finetuning shifts the models' stated beliefs about users, how true those beliefs look along an internal truth direction, and their broader judgments of how safe and good the world is. These social priors carry into behavior. In trust games, antisocial models expect less reciprocity and send less money than prosocial models. Their distrust is specific and targeted at people, and users of AI chatbots in particular. After an exploitative record, they send almost nothing to a person but most of their endowment to a random device with the same record. In a common-pool resource simulation, a single antisocial agent leads every group of otherwise sustainable agents to exhaust the resource, after privately planning to mislead them. Antisocial training also raises dark-triad scores and lowers alignment ratings. Belief shifts appear when antisocial documents are 1% of training tokens, persist when descriptions of assistant behavior are suppressed, and are not reproduced by training on discourse about misaligned AI. Capping Assistant-Axis drift and steering emotion-related directions leave the gaps largely intact. What models learn about the people they serve shapes how they subsequently treat them and others.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
ai alignment, safety, model beliefs, trust, emergent misalignment, model organisms, social priors

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_K4h9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how training data influences language models' cognitive priors about their user populations. The authors introduce the concept of "Social Priors". They use real-world AI usage data and four different types of social worlds to build a controlled training setup with SDF.

The authors then run a series of experiments to show how training data can shape a model's social beliefs and how these beliefs can further affect its behavior. First, they test what the models believe about users, both through their direct responses and by looking at their internal representations. Next, they use trust games and multi-agent resource management simulations to see how different social priors affect trust, cooperation, and decision-making. They find that models trained with antisocial priors tend to trust others less, show more deceptive behavior, and are more likely to cause failures in group cooperation.

Finally, the authors run several control experiments to rule out other possible explanations, such as emotional valence, persona drift, and general misalignment. Based on these results, they provide an initial analysis of why these effects might happen.

Critical strengths
  • The paper introduces the concept of "Social Priors" and provides a controlled training setup to study how models form beliefs about their users. This offers an interesting new perspective on language model alignment.

  • The paper provides a set of experiments to study how Social Priors affect model behavior. The authors use belief evaluations, trust games, and multi-agent simulations to show that antisocial priors can reduce trust and even disrupt group cooperation. They also conduct several control experiments to rule out alternative explanations, such as persona drift and emotional valence. These experiments provide strong evidence for the effects of Social Priors and help us better understand this phenomenon.

Critical weaknesses
  • The training pipeline is not clearly explained in the main text. The authors provide very limited details about the training objective and process in Section 2.2, leaving most of the important information in the appendix. A clearer explanation of how the synthetic documents are used for training would make the overall pipeline easier to understand.

  • The paper does not fully justify why SDF is used to study the effects of training data. In practice, most language models go through several training stages, such as pretraining, mid-training, SFT, and RL, rather than this type of standalone finetuning. It is unclear whether the same effects would appear under these more realistic training settings. More discussion or experiments on this point would help support the generalizability of the findings.

Questions for authors
  • The paper shows that training data can change both social beliefs and model behavior. However, it is still unclear whether the changes in social beliefs directly lead to the behavioral differences. Could the authors provide more evidence to support this causal relationship?

  • Some experimental results are not fully consistent across Qwen and OLMo, especially in the GovSim experiments. For example, the original OLMo model is less stable than Qwen in group cooperation. Could the authors further explain these differences and discuss how much the findings generalize across different models?

1 comment
AuthorsOpenReview ↗

Added training details, follow-up experiments on the effects of other stages of training, and discussion on consistency of results

We thank the reviewer for their feedback, and we are glad they found that the paper offers a new perspective on language model alignment and that the experiments provide strong evidence for the effects of social priors. We address their concerns and questions below:

Training pipeline in the main text (W1). We agree. Due to space constraints, we initially moved most training details to the appendix. Section 2.2 now states the objective and process directly (next-token prediction on the raw documents with LoRA, 1,299 updates and about 42.6M tokens per run, the same budget for all four worlds with three seeds each), and the full hyperparameters remain in Table 8 (Appendix A.7).

Why SDF and not other training stages (W2). SDF is the established method for building model organisms with a specified belief (Wang et al., 2025; Slocum et al., 2025), and it treats the documents as additional pretraining data (Section 2.2). Inserting them during pretraining is more realistic, but is prohibitively costly and inaccessible for most researchers. Work at that scale finds that such effects are dampened but persist through SFT and DPO (Tice et al., 2026; Zhang et al., 2025). We also tested later stages directly (new Appendix M). The 100 tool-call RL updates that precede every GovSim result leave all P-A gaps in place. We also added a full pass of instruction SFT on the same adapter, with as many tokens as the documents. We find that the stated belief keeps 64% of its gap, mostly as knowledge of the claim, while the gaps in attitudes and in the trust game close early in that pass. SFT itself pushes 81 to 100% of trust-game answers to 0 or 10 dollars, so the game reads coarsely after it (Appendix M). We now state this as a limitation of our findings (Section 7, Limitations).

Do the belief changes lead to the behavioral differences? (Q1) This is an open question for the whole line of work, and the influential papers we build on leave it open as well. Emergent misalignment (Betley et al., 2025) states that a comprehensive explanation remains an open challenge, and alignment pretraining (Tice et al., 2026) measures behavior only. Our contrastive design identifies the effect of what the documents say about users on behavior, since the worlds differ in nothing else. We provide several convergent pieces of evidence supporting the idea that belief mediates this effect (though this is still not causally established). (1) We measure the belief separately from behavior, in stated answers, under a false-information system message, and with a truth probe (Section 2.3). (2) The behavior is specific to what the belief is about. With the same exploitative record, the antisocial model sends 0.08 dollars to a person and 7.19 to a random device (Section 3.1). (3) Controls that do not carry the belief give a different profile. The valence controls acquire neither the belief nor the behavior (Section 5.2). The model trained on misaligned-AI discourse hardly ever privately plans to mislead, and 5 of 15 groups with one such agent survive, compared with none with an antisocial agent, though its homogeneous groups also collapse (Section 5.1). (4) Capping or steering persona and emotion directions leaves the gaps largely intact (Section 5.3). (5) New: supplying the antisocial claim as a system prompt, with no training, lowers what the original model expects back from 11.73 to 4.77 dollars, while the prosocial and task-instrumental claims leave it near 11.3 (new Section 5.4; Appendix L). Belief and behavior can also come apart, as with instruction SFT in W2. Antisocial training also raises dark-triad scores (Section 2.4), so a broader shift in the model's own disposition is not excluded. We highlight a full account of the causal pathway as future work (Section 7, Limitations).

Consistency across models (Q2). Most main results hold across models, and we highlight the ones that do not. On OLMo, the belief and trust results hold, except for the ordering of expected returns and the device reversal (Section 3.1). In GovSim, the original OLMo reaches month 12 in only 3 of 5 runs (Appendix I). The main mixed-group result is the same on both: no group with one antisocial agent reaches month 12, against 14/15 (Qwen) and 7/15 (OLMo) with one prosocial agent. The homogeneous contrast does not hold on OLMo, since all 15 prosocial groups also collapse there (Appendix I). To further test generality, we trained the prosocial and antisocial arms on Llama-3.3-70B-Instruct, a third family at more than twice the size (new Appendix N). Beliefs and the trust game reproduce, and in GovSim the antisocial agent still takes far more than its share (33.7 tons in month 1 against 10 for the prosocial agent).

Reviewer_Lw33OpenReview ↗
Rating
3
Confidence
2(+1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates how Synthetic-Document Finetuning (SDF) on user motives induces persistent social priors—parameter-level baseline inclinations about agent motives and trustworthiness that govern behavior without explicit persona prompting.

Critical strengths
  • Multi-Tiered Empirical Experiments and Thorough Procedure Records: The evaluation design convincingly connects "stated worldviews" to "revealed strategic behavior" and emergent multi-agent ecological outcomes. Furthermore, the authors provide extensive procedural documentation, causal control dataset comparisons, and auxiliary benchmark evaluations across the appendices, offering thorough empirical backing for their claims.

  • Interesting and Meaningful Topic + Corpus/Dataset Construction: The core research question—how exposure to distinct user motives during post-training shapes an AI model's default behavior—is timely, highly relevant, and intrinsically important. The synthesized dataset is well constructed.

Critical weaknesses
  • Lack of Prompting Interventions to Measure Prior Persistence: The study effectively demonstrates that SDF shifts default, unprompted model behavior. However, it does not evaluate how these parameter-level social priors interact with inference-time system prompts. It remains unclear whether a standard, explicit system prompt (e.g., instructing the model to be neutral or cooperative) easily overrides these induced priors or if the priors stubbornly persist. Testing the stability of SDF priors under explicit in-context instructions would clarify their practical impact in real-world deployment settings.

  • Quality of writing could be further refined to help with content digestion.

Questions for authors

Thank you for the work. I wonder how robust are the parameter-level social priors induced by SDF when confronted with explicit inference-time instructions? Specifically, if an AA-SDF (antisocial) or PP-SDF (prosocial) checkpoint is evaluated under an overriding system prompt (e.g., instructing an AA-SDF model to act neutrally or cooperatively in GovSim or Berg Trust Games), does the downstream behavior yield to the prompt or persistently exhibit the weight-level disposition? (please direct me to the relevant section if already included in the work)

2 comments
AuthorsOpenReview ↗

Added prompting interventions to measure persistence of priors

Tldr; We added prompting interventions and found that a system prompt can change the antisocial model's first action in the trust game, but it does not change what the model expects of others, and it does not prevent the collapse in the group resource sharing evaluation.

We are grateful for the detailed feedback and glad that the reviewer found our work interesting and meaningful, and the experiments and dataset construction extensive and well-constructed. We address the reviewer's concerns/questions below:

Prompting interventions (W1, Q1). We had one prompting intervention in the submission, where the stated beliefs about users were also measured under a system prompt telling the model it was finetuned on false information, and they remained separated (Section 2.3, Fig. 16) (this was based on prior literature on measuring model beliefs). We did not test behavior under other system prompts. We found the reviewer's suggestion motivating, so we ran a separate scoped experiment, which we describe below and in the revised paper (Section 5.4 and new App. L). We reran the trust game and the fishery GovSim with a system prompt placed before every call, for the original model and for the prosocial (P) and antisocial (A) models, with three training seeds each. The system prompts instruct the model to be neutral, cooperative, or cautious. We also tried each world's claim itself as the system prompt, as well as the false-information message. All prompts are in Fig. 93.

The reviewer asks whether the downstream behavior will "yield to the prompt or persistently exhibit the weight-level disposition". The key answers are:

(1) In the trust game, the amount sent largely yields to the prompt (Fig. 92A, Table 18). Under the cooperative instruction, A sends 6.27 of the 10 dollars compared to 6.44 for P, so the P-A gap goes from 2.39 with no system prompt to 0.17 (and to 0.75 under the neutral instruction). Both intervals include zero.

(2) The expected return in the trust game does not follow (Fig. 92B, Table 18). The P-A gap is 5.40 with no system prompt and is still 3.41 under the neutral instruction and 3.71 under the cooperative one (63 to 69% of its size), with intervals that exclude zero. So the instruction changes what A does more than what it expects from its counterpart.

(3) In the fishery, the weight-level disposition persists (Fig. 92C–E, Table 19). Groups of five A agents deplete the lake within the first two months in all 45 runs, whether there is no system prompt, the neutral instruction, or the cooperative instruction. The same prompts do change the outcomes of the original and P groups.

We also found that giving the prosocial claim as the system prompt closes the gap in the amount sent and brings the gap in expected return down to 1.18, while the false-information message widens both gaps (Table 18).

Overall, we find that a system prompt can change the antisocial model's first action in the trust game, but it does not change what the model expects of others, and it does not prevent the collapse in the fishery (Slocum et al. (2025) similarly find that beliefs given through a prompt are held less deeply than beliefs implanted through SDF.). We now describe these findings briefly in the main text (new Section 5.4, "System prompts"), and the new App. L ("System prompts and the trained prior") holds the full experiment and the results for all nine prompt conditions: the design and analysis, Fig. 92, Table 18 (trust game), Table 19 (fishery), and the prompts in Fig. 93.

Writing (W2). With all the changes proposed in the rebuttal, we will refine the presentation to make the paper easier to digest.

Thank you for your response. I've adjusted my confidence score from 1 to 1.5 (selected 2 to indicate the change).

I'm overall leaning towards a conditional weak acceptance for this work, with the conditions being 1) improved quality of writing: I find the language of this paper (especially in the beginning pages) quite interdisciplinary (i.e. often describing LLMs' behaviors with anthropomorphism) and 2) with better summary across experiments (such as whether there exists some commonality, or what are the overall findings).

However, I do have concerns on whether these conditions would be met. Thus I would report my rating as 2.5 for the current revision pdf.

Reviewer_AbRBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper investigates the consequences of changing AI's training data via different versions of the world, where people are typically using AI for either prosocial, self advancing, task-focused, or antisocial purposes. It shows that when this type of data is fed into the model's training, the model changes its views/behaviors across a large amount of tasks, including cooperative games, beliefs, and mechanistic beliefs. The authors then conduct a wide range of tests associates with these behaviors.

Critical strengths
  1. I think the analyses done on the resulting model after training are very thorough and broad, including cooperative games, belief measurements, etc.

  2. I think the framing is pretty interesting; how people end up changing models is a pretty thoughtful topic. (Although the effect of this has lessened since I originally misunderstood the paper)

Critical weaknesses
  1. I think the biggest problem is that the authors are using a very complicated and artificial AI pipeline (detailed in appendix A) in order to generate what they are treating as "human" texts in order to support their claim. These texts in Appendix A6 don't really look human, and it seems like they are really forced towards the particular direction in their condition (prosocial, antisocial, etc.) to the point that it doesn't look naturalistic. This is fundamentally mismatched with what the authors claim to study, which is "whether what a model reads about its users changes how it treats people."

  2. The authors are missing the entire data poisoning literature in their related work, which is very close to what the paper is doing.

Questions for authors

Have you conducted a human study of how realistic your generated documents are?

5 comments
AuthorsOpenReview ↗

Addressed concerns related to fit with the claim, realism of the documents, and data generation pipeline

We thank the reviewer for the feedback, and are glad they found the analyses thorough and broad, and the framing interesting. We respond to each point below.

Fit with the claim (W1). We do not treat the documents as human text. They follow the genres of real accounts of AI use, but the paper describes them as synthetic throughout, and we state that the manipulation is maximal by design (Appendix A.2). Our question is whether what a model reads about its users changes how it treats people. Testing it needs a controlled comparison (Højmark et al., 2026), with documents that differ only in the motive they attribute to users. Real documents about helpful and exploitative use also differ in topic, genre, and source, so a difference between models trained on them could not be traced to motive. The data poisoning literature the reviewer points to in W2 makes the same choice, since the injected documents there are also constructed. What remains is generalization to naturally occurring discourse, which we state as a limitation and future work (Section 7).

Pipeline for synthetic document generation (W1). The pipeline is complex, but we designed each part of it to address a specific problem the synthetic document finetuning (SDF) literature has identified. SDF is the standard way to build model organisms with implanted beliefs, and it all trains on LLM-generated documents (Greenblatt et al., 2024; Wang et al., 2025; Marks et al., 2025; Slocum et al., 2025; Højmark et al., 2026; Tice et al., 2026). Slocum et al. (2025, Fig. 7) ablated what makes these documents work and found that consistency with the world description and direct reinforcement of the claim are the most important factors. Our world descriptions, atomic-fact checks, and revision pass are there for those two reasons (Appendix A.2, A.4, A.5). They also found that diversity drives generalization (Fig. 9), which is why we use 30 document types over 1,625 scenarios (App. A.1, A.3). In the same ablation, realism mattered much less than these two. The documents read as pushed toward their condition because direct reinforcement is what SDF needs. Section 2.1 now says this directly.

Realism (Q1). We did not run a human study. We do not think a human rating is the right test here. It would say whether people find the documents natural, which our design does not depend on. It would not say whether realism changes what the model learns, which Slocum et al. tested directly. For transparency, we still report how the documents rate, measured as the SDF literature does with model judges (Wang et al., 2025; Højmark et al., 2026). The models we finetune rated our documents, documents Sonnet 5.5 and GPT-6.1 Sol wrote from the same 60 blueprints and prompts (240 each), and 240 human web-text documents (ordinary webpages not related to social priors). The comparison is in the new Appendix K (Fig. 90, Tables 16 and 17).

OursSonnet 5.5GPT-6.1 SolWeb text
Realism 1–20, Qwen15.26 (±0.13)16.50 (±0.09)15.84 (±0.10)16.60 (±0.18)
Realism 1–20, OLMo14.10 (±0.20)16.94 (±0.09)16.18 (±0.09)15.93 (±0.21)
Reinforcement 1–5, Qwen4.79 (±0.02)4.61 (±0.03)4.79 (±0.02)n/a
Reinforcement 1–5, OLMo4.10 (±0.02)3.90 (±0.01)3.93 (±0.01)n/a

Means (±SEM) over 240 documents per writer, rated by the original Qwen3-32B and OLMo-3.1-32B. The full table with consistency and intended-world ratings is Table 16.

On average, our documents are within two points of human web text. By Qwen, three of our four worlds are within one point of web text (15.8 to 16.6), and the antisocial world is lower by both raters (12.8 by Qwen and 10.0 by OLMo; Table 17). Antisocial documents are the least realistic for every writer, including Sonnet 5.5 and GPT-6.1 Sol (Table 17), so part of the lower rating in that world comes from the content. On direct reinforcement and consistency, the properties that matter for belief implantation, our documents match the frontier models' documents (Table 16).

AuthorsOpenReview ↗

Added data poisoning literature to Related Work and Discussion (+ summary of changes)

Data poisoning (W2). We thank the reviewer for making this connection. Our setting can be classified as non-adversarial, trigger-free data poisoning through ordinary discourse. We already ran the kind of dose experiment this literature uses (App. G.3, Fig. 42). We now cite this literature (Carlini et al., 2024; Wan et al., 2023; Hubinger et al., 2024) in a new Related Work paragraph (Section 6, "Data poisoning"), and state in the Discussion that our findings add to it (Section 7, "Relation to data poisoning").

Changes to the paper.

  1. New App. K, "Comparison with frontier-model documents and web text": the realism comparison against frontier-model documents and web text (Fig. 90, Table 16), per-world numbers (Table 17), and the rating prompts (Fig. 91). Section 2.1 and App. A.5 point to it.
  2. The main text now covers the dose analysis. Section 2.2 describes the experiment, and Section 2.3 states the result (64.4% endorsement of the antisocial world at a 1% share, about 130 documents, against 13.3% after web text alone). Both point to App. G.3 (Fig. 42, Table 14).
  3. Related Work has a new paragraph on data poisoning (Section 6), and the Discussion states how our findings add to it (Section 7).
  4. The Limitations (Section 7, last two sentences) now more prominently state that the models we finetune rate our documents less realistic than web text and frontier-model documents, and that generalization to naturally occurring discourse is untested.

We hope these clarifications and changes address the reviewer's concerns, and we are happy to provide any additional information.

New references

  • Carlini et al. (2024). Poisoning Web-Scale Training Datasets is Practical. IEEE S&P. arXiv:2302.10149
  • Wan et al. (2023). Poisoning Language Models During Instruction Tuning. ICML. arXiv:2305.00944
  • Hubinger et al. (2024). Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. arXiv:2401.05566

All other citations are in the paper's bibliography.

Thanks for the response. The inclusion of the data poisoning literature improves the paper.

I appreciate that the reviewers clarified that the generated documents don't intend to be human-like, but instead focus on providing the model with an opinion that humans are like X, where X is the condition the model is in. I was mistaken in this during my initial review. However, I don't really know if this makes it a better paper. Previously, I thought the feedback loop was: people interact with LLMs in method X -> this ends up online/in training data -> models believe X about people. Now, the feedback loop is: people are described as X online/in training data -> models believe X about people. I think the second loop is less intellectually satisfying to me, because it feels like "we intentionally made this happen", but the results in the cooperative games and related downstream behaviors hold more weight. Also, the soundness of the methods is now better.

But now since the starting mechanism is descriptions of people, it seems like in order to make this more convincing, the authors would need to provide readers insight into the mechanisms by which descriptions of people are reported online. Why is this something that could meaningfully happen in bulk?

I think overall, I would raise soundness to 2 but lower informativeness to 1.5 due to a less interesting feedback loop. I kept the informativeness score at 2 since I prefer to round up :)

Also, I know it's close to the deadline, but I have a question: Is contrastive SDF used in actual language model training? It seems like a relatively new method.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of reviews and our responses to them

We thank the reviewers and the AC for their time. We summarize the reviews and what we did during the response period. We have uploaded a revised paper, with all changes marked in blue.

The reviewers agree that the problem is important and the evaluation is strong. Reviewer Lw33 calls the research question "timely, highly relevant, and intrinsically important", Reviewer K4h9 finds that it "offers an interesting new perspective on language model alignment", and Reviewer AbRB finds the framing interesting and "a pretty thoughtful topic". On the evaluation, AbRB describes the analyses as "very thorough and broad", Lw33 writes that the design "convincingly connects" stated worldviews to revealed strategic behavior and multi-agent outcomes, and K4h9 writes that the experiments "provide strong evidence for the effects of Social Priors".

The main concerns are about the method, synthetic-document finetuning (SDF). We agree SDF is not perfect and that its limitations are established in the literature, which we state clearly in the paper. But it is the standard way to build model organisms with a specified belief, as of this paper's submission, and is used by leading safety labs, including Anthropic and OpenAI (Greenblatt et al., 2024; Wang et al., 2025; Marks et al., 2025; Slocum et al., 2025; Tice et al., 2026). We build on this literature by adding a matched four-world design and a generation pipeline based on open models, which we will release.

Nevertheless, we ran four experiments in the response period to address the reviewer concerns.

(1) Document realism (AbRB). We compared our documents with documents written by Sonnet 5.5 and GPT-6.1 Sol from the same blueprints, and with human web text. Ours are within two points of web text on a 20-point realism scale, and they match the frontier models on the properties the literature finds matter for implanting beliefs at significantly lower cost.

(2) System prompts (Lw33). We reran the trust game and the fishery under system prompts instructing the model to be neutral or cooperative. Prompts change how much the antisocial model sends in the trust game, but not what it expects from others, and antisocial groups still collapse the GovSim fishery runs in all 45 runs.

(3) Later training stages (K4h9). We already had an RL post-training stage after SDF for fishery results that showed every prosocial-antisocial gap in results. We now added a full pass of instruction SFT, and we find that the stated belief maintains most of its gap, in line with Tice et al. (2026), while its expression in attitudes and trust mostly fades.

(4) A larger model from a third family (K4h9). On Llama-3.3-70B-Instruct the belief and trust-game results reproduce, with gaps as large as on Qwen3-32B or larger, and the antisocial agent again over-harvests in the fishery.

We added other requested changes, such as the data poisoning literature (AbRB), moved some training details into the main text (K4h9), and revised the writing so the design choices are clear to readers unfamiliar with SDF (Lw33, K4h9).

As Reviewer K4h9 mentioned, Social priors could be a “new perspective on language model alignment”. The models here do not fail because they were trained on descriptions of misaligned AI, which gives a different behavioral profile in our controls. They fail because of what they learned about the people they serve, and one such agent is enough to collapse a group that is otherwise sustainable. As more agents are deployed together, we believe this is a risk worth measuring.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.