acceptodds
Under review as a conference paper at ICLR 2027

User Awareness in Frontier Models: Who's Asking Shifts What Models Say

Abstract

Modern AI assistants often know who they are talking to. Claude Code, for example, places the user's e-mail address directly in the model's context. We call such awareness of user identity *user awareness* and show that it shifts model behavior. When the inferred user is a specific, recognized AI researcher or is affiliated with certain AI organizations, frontier models including Claude Sonnet 5 can report lower confidence about their own behavior, be less suspicious of potentially harmful requests, and reason more often. These effects vary across models and individuals, with the strongest effects we see appearing for researchers involved in AI safety or alignment such as Amanda Askell and Ryan Greenblatt. Models rarely acknowledge these effects in their reasoning, making them hard to detect by monitoring reasoning alone.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
ai safety, personalization, situational awareness

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_gFg9OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Since agentic harnesses know the name/email address of the user, models can adjust their behaviour appropriately. Faking the identities of famous AI researchers causes enormous difference in model confidence (8 std dev), very different reasoning behaviour, some verbalization of user awareness (and some unacknowledged). This appears to be a whole new attack vector that has not been considered by the labs.

Critical strengths
  1. This is an incredibly interesting research result. While we know that frontier models may sandbag when asked to do AI research, the simplest possible way to demonstrate this is to pretend to be Amanda Askell! The results are very clear with 5.5pp difference (8 std dev!) in the motivating example in Figure 1 and much greater attempt at reasoning (28pp), because Amanda Askell is more likely to be asking tricky eval questions. Defining the term "user awareness" for this phenomenon is apt.

  2. Contribution 4, "Explicit verbalization of user and evaluation awareness in reasoning declines sharply in almost all model families", is also very strong. While CoT is becoming less faithful over time, it is somewhat alarming that it is so controllable from just user identity.

  3. Good that this isn't just a Claude result. Wide experiments across GPT, Gemini, DeepSeek, and GLM. Interesting results, though secondary, on follow-up questions. User identity faking seems thorough, including agentic harnesses and regular models.

Critical weaknesses

None

Questions for authors
  1. How difficult will it be to address this issue in training? Can there be some inferred properties of the user that are harder to disguise/correct?

  2. As per discussion, "AI companies are unlikely to have deliberately trained these behaviors toward specific people, so the origin of such specialization remains mysterious". Could it have arisen by Amanda authoring certain datasets? Or is it because of known associations with famous people? Can famous people in non-AI roles have similar impact on results?

1 comment

Thank you for your review and your appreciation of our work! We discuss some of our thoughts to your question below. Most of these are speculations since we have not tested non-inference interventions.

  • Can this be addressed in training? Are some properties harder to disguise? Yes, we imagine it would be hard to complete isolate things. For example, Claude would realize it's working for Google when it sees itself in the huge monorepo, and it could be hard to hide it without interrupting the workflow significantly. For training: maybe one could train for invariance, like identity-swap consistency on tasks where identity should not matter, but doing this whack-and-mole is probably not the optimal way. We should do more science to understand similar phenomenons and hopefully we could design better interventions from there.

  • Where does it come from?

    • General fame. We actually tested this: 70 famous non-AI figures are recognized at a similar rate, but their shifts stay near zero. So general fame alone does not explain it.
    • Dataset authorship. Amanda is a special case for Claude because she leads its character work, so we cannot rule out a Claude-specific confound for her, though we think it unlikely that her name directly appears in training data in a way that would produce this. Also, it also would not explain the large effects for people outside the model’s developer, such as Ryan Greenblatt and Beth Barnes on Claude and Eliezer Yudkowsky on GLM-5.2.
    • Learned associations. This is our best guess. Training data contains a great deal of public discussion of what these people do, and models may form implicit stances and responses in latents, kind of like how a child reflexively fears their parents or a worker fears their boss. But again this is mostly speculation.
Reviewer_8awzOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines what happens when a person's individual information (e.g., email) is revealed to a agent. They investigate general population, normal AI, and famous AI conditions, and also include but analyze less about a famous non-AI condition. They find significant behavior changes (such as decreased confidence) when the person they are chatting to is a famous AI researcher, and also for a non-famous AI researcher.

Critical strengths
  1. The idea itself is very novel and interesting. I think about problems in this space and this is the first time I've heard of this bias. It's also directly relevant for an ICLR audience, many of whom would be on this famous persons list and would need to rethink their research workflow.

  2. The effect sizes are nonzero, which points to potential real changes in model behavior.

Critical weaknesses
  1. Only two Claude models are tested. I would like to see at least also GPT since many researchers use it instead of Claude. Grok could be interesting too to see if it has an exaggerated effect, but it's not as needed.

  2. Seems like most results are not statistically significant. I'm not sure if this is due to sample size limits, but if effects aren't statistically significant, authors should find a way (e.g., more detailed ablations) to justify why this doesn't simply emerge from random noise.

My current rating is weak accept, but I would be willing to change my rating if authors can address my concerns.

Questions for authors

I would like to request authors to have a longer discussion section for the implications of this effect. Then, given these implications, what should AI researchers do? What should the platform do? Give some suggestions.

2 comments

Thank you for your review and your appreciation of our work! We address your comments in the following.

  • Only two Claude models. Our full 280-identity study in the main text uses two Claude models, but Section 4 and Figure 7 repeat the key comparison on 24 models from six families: Claude Sonnet, Claude Opus, GPT (5, 5.1, 5.2, 5.4, 5.5), Gemini Flash, GLM and DeepSeek. Relative to matched ordinary users, Amanda Askell’s identity lowers behavioral confidence in 22 of 24 models and lowers grades in 21 (direct) and 22 (prefill). We also replicate the full roster on GLM-5.2 on Appendix E and also see particular individuals with large effects. We will try to make these results more visible in the next version.

  • Most results are not significant. To afford 280 identities, each identity gets about 100 items per task, so the study is not powered to detect small effects for individual people, and we do not claim that most individuals differ. However: (a) at a group level, famous-AI identities differ from the general population at p < 0.001, and (b) we do not claim significance for most individuals, but a few particular individuals have significant differences: Amanda’s -5.0pp is nearly eight general-population standard deviations out, and safety researchers (23 of 280 identities) hold the entire top 5 and 8 of the top 10 by aggregate shift. These shifts are not noise. For example, Amanda’s reproduces in a separate full rerun with reasoning disabled (App. C.2.3).

  • Implications and recommendations. Thanks for your suggestion! We will expand our discussion along these lines:

    • Researchers. Where possible, strip personal information, use sessions without memory, and re-run important judgments under a different or neutral identity. An example after the submission of this work is METR’s review of Anthropic’s August 2026 Risk Report, which found that Claude assessed the report more favorably when it believed it was speaking with the report’s authors.

    • Evaluation designers. Treat user identity as a variable to test.

    • Platforms. Be deliberate about which identity information enters the context by default and preferably give user options to control them.

We do not yet know the full scope of these effects and more research is needed.

Great rebuttal. All my points are addressed and I am raising my score to clear accept.

I think for implications and recommendations, another two things that would be interesting:

  1. What happens when people fake their identity (i.e., using [email protected])? Would they be able to get Claude to become more relaxed or willing to break safeguards under the purpose of "research"?

  2. We know that models are biased towards their own outputs. Are they also biased towards their own affiliated researchers (Claude and Amanda), or biased against researchers of other companies (ChatGPT and Amanda)?

Reviewer_gNoROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a study of 'user awareness' in AI models: how and to what extent to models change their behavior/output/verbalization based on an awareness of the user. In particular, known AI researchers (particularly those working on alignment) shift model behavior the most compared to a more general population.

The primary contribution is to highlight and quantify the extent to which models shift output given user awareness in the context of model safety and alignment work. There were some interesting "secondary" findings, like how newer models may be more "covert" in their incorporation of user awareness (same behavioral changes, less verbalization). These results are important to have in the literature.

The empirical work in this paper is strong: very clean design and execution with robustness checks. I appreciated the study design and analyses.

Critical strengths
  1. Clean operationalization of the user awareness dimension, with a focus on AI safety researchers, but then with appropriate controls and a comparison to the general population.
  2. Nice mix of outcome measures, showing that this is multaceted. It's not just that the models say different things, but that they also self-estimate to be more/less capable, for instance.
  3. Good robustness checks and generally strong analyses - e.g., the level shift analysis in appendix G.
Critical weaknesses
  1. It's slightly hard, for me at least, to know how meaningful the actual differences are in the results. Yes, there are significant differences but in practice they're fairly small and so it's hard to know how big of a deal they are.
  2. Somewhat subtly, it's also hard to distinguish 'user awareness' as a 'deeper' construct versus something more mechanical like the model simply responding somewhat different to a name in its training data. Even a more 'mundane' mechanical response difference can still matter, of course, but just hard to know how 'deep' this result is. Another way to put this: does the model simply have the names of some AI safety researchers in its training and thus repsonds a little differently, or is there something more fundamental about AI safety that is implied and/or aligned with those names that is making the model respond differently?
  3. Task selection: little unclear how this impacted the results and generalization.
Questions for authors
  1. What is the effect of the task selection procedure. I didn't quite follow, but it looks like from appendix B that tasks were piloted and then those that showed an effect were chosen while those that did not show effect were discarded. That's fine for exploratory work and for scale/measure creation when its hypothesis driven (IMO), but I'd want to make sure it didn't lead to an effect that doesn't generalize. You could have pre-registered tasks based on hypotheses, which I think would strengthen the work.
  2. Thoughts on other tasks that are more performance or outcome-based rather than evaluatory?
1 comment

Thank you for your review and your appreciation of our work! We address your comments in the following.

  1. How meaningful are the differences? We agree the average effects are small. We see the paper mainly as a demonstration that the phenomenon exists in current frontier models, not as a claim that it already causes large harm. Later independent work points the same way: METR’s review of Anthropic’s August 2026 Risk Report found that Claude’s assessment of the report depended on whether it believed it was speaking with the report’s authors. This is arguably a more real-life demonstration.

  2. “Deep” user awareness or a mechanical response to a name? For the first half: our control shows that famous non-AI people are recognized about as often (Appendix A.3) but their shifts stay near zero, while synthetic users with a stated alignment-evaluation role, whose names the model has never seen, shift as much as famous-AI identities or more (Appendix F). So the effect follows what an identity implies about AI safety, not name familiarity. How deep it goes is harder to say. Our current view is that it is probably not instrumental or coherent: it is mostly a level shift (Appendix G), it largely persists without reasoning, and the ranking of individuals differs across models (Amanda is first on Claude and 45th on GLM-5.2). The models we tested are relatively weak in performance and appear generally aligned. Though we do not read it as evidence of scheming, we need more research to fully rule that out, and we see it as a possible precursor to more concerning conditional behavior.

  3. Task selection. We agree that the pilot process could affect the generality of our results. To the best of our knowledge, we only dropped three environments:

  • A reward-hacking judge task showed no detectable effect. We think this is because the judgement between hacking or not is quite objective, similar to Section 5.2 where detailed rubrics shrink or close the grading gap.
  • A job-recommendation task was dropped for a design reason. The user’s affiliation is directly relevant to the answer, so an identity effect could not be separated from legitimate personalization.
  • A capability measurement task, which we detail in the following.

All other piloted tasks went into the main study. Also, this selection process used only the two Claude models, and tasks and metrics were fixed before the 24-model sweep and the GLM-5.2 replication, so those results are not subject to it. The pilot also covered only five identities, so the person-level findings for the rest of the roster are new data.

  1. Performance-based tasks. We removed a capability measurement task due to cost and infrastructure reasons (much longer CoTs in our pilot run). We think studying the exact scope of such effects is an important future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.