- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Since agentic harnesses know the name/email address of the user, models can adjust their behaviour appropriately. Faking the identities of famous AI researchers causes enormous difference in model confidence (8 std dev), very different reasoning behaviour, some verbalization of user awareness (and some unacknowledged). This appears to be a whole new attack vector that has not been considered by the labs.
Critical strengths
-
This is an incredibly interesting research result. While we know that frontier models may sandbag when asked to do AI research, the simplest possible way to demonstrate this is to pretend to be Amanda Askell! The results are very clear with 5.5pp difference (8 std dev!) in the motivating example in Figure 1 and much greater attempt at reasoning (28pp), because Amanda Askell is more likely to be asking tricky eval questions. Defining the term "user awareness" for this phenomenon is apt.
-
Contribution 4, "Explicit verbalization of user and evaluation awareness in reasoning declines sharply in almost all model families", is also very strong. While CoT is becoming less faithful over time, it is somewhat alarming that it is so controllable from just user identity.
-
Good that this isn't just a Claude result. Wide experiments across GPT, Gemini, DeepSeek, and GLM. Interesting results, though secondary, on follow-up questions. User identity faking seems thorough, including agentic harnesses and regular models.
Critical weaknesses
None
Questions for authors
-
How difficult will it be to address this issue in training? Can there be some inferred properties of the user that are harder to disguise/correct?
-
As per discussion, "AI companies are unlikely to have deliberately trained these behaviors toward specific people, so the origin of such specialization remains mysterious". Could it have arisen by Amanda authoring certain datasets? Or is it because of known associations with famous people? Can famous people in non-AI roles have similar impact on results?
Thank you for your review and your appreciation of our work! We discuss some of our thoughts to your question below. Most of these are speculations since we have not tested non-inference interventions.
Can this be addressed in training? Are some properties harder to disguise? Yes, we imagine it would be hard to complete isolate things. For example, Claude would realize it's working for Google when it sees itself in the huge monorepo, and it could be hard to hide it without interrupting the workflow significantly. For training: maybe one could train for invariance, like identity-swap consistency on tasks where identity should not matter, but doing this whack-and-mole is probably not the optimal way. We should do more science to understand similar phenomenons and hopefully we could design better interventions from there.
Where does it come from?