- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks a simple question. When a chat model is asked where a made-up person was born and names a city anyway, what decides which city it picks? The usual story is that the model guesses. The authors argue that it runs its normal fact-recall machinery and reads a guess off where the name seems to come from.
To test this they invent a few hundred people from 11 naming traditions, mostly European plus a few South Asian ones, along with mixed names that take a first name from one tradition and a surname from another. Every name is checked against large training corpora, Wikidata and Wikipedia to make sure no such person exists. Then they ask five open chat models, the largest at 8B, where each person was born. Llama nearly always refuses, so they start its answer for it.
The answers are very consistent. A made-up Baltic name gets a Baltic city almost every time, but different models seldom choose the same city. The heads that write the city are largely the same ones used when the model recalls a real person's birthplace, and in the two Gemma models, switching them off breaks both. If you nudge the model's internal picture of the name toward another tradition, the invented city moves with it, and random nudges don't do this. In one model the nudge also moves real people's birthplaces. The authors then show a real cost: people whose birthplace doesn't match their name are recalled far less often than equally famous people whose birthplace does. Whether a model refuses also depends on where the name is from. In OLMo 2, where the training data is public, the prior shows up very early in training.
They check the work with random-nudge controls, a baseline that looks only at spelling, a second judge model, and a rerun on fresh names. What stands out is that every pass/fail test was set before the data came in, and every result is reported. More than half of the 93 tests failed, and the paper says so plainly.
Critical strengths
-
The question is clear, and the paper actually follows it all the way through. It starts with what the models do, finds the part of the model that writes the answer, shows that changing the name's representation changes the answer, and ends with a cost to real people. The care that went into making the names is a big part of why I believe the result. Because each name was checked against the training data, the easy objection that the model has seen these people before doesn't hold.
-
I appreciated how open the paper is. Its tests were fixed in advance, and it reports the ones that failed, with reasons. That makes it easy to tell which claims are solid and which are shaky. The finding that matters most to me is the fairness one. The same habit that produces made-up answers also makes the model worse at remembering real people whose birthplace doesn't fit their name, and that holds up even for people the model otherwise knows well.
Critical weaknesses
-
The mechanism is thinner than the paper makes it sound. Each tradition is built from just 12 first names and 12 surnames, and the test names reuse those same pieces, which the authors admit in their limitations. So the "origin direction" could simply be recording which familiar name pieces are present. A plain spelling classifier already gets the tradition right about 99% of the time, and just swapping in a name from the other tradition moves the answer at least as often as the steering does. The "shared recall heads" are also less specific than they look. The same heads write the city for a capital-city prompt with no person in it, and replacing them with their average output barely hurts real recall in Qwen and Llama. Steering real people also wipes out about half of the birth years the model knew, which the authors say may mean the whole person is being damaged rather than redirected. I read this as good evidence that models guess a city from what a name looks like, but not yet as evidence of a specific recall circuit running on a linear prior.
-
Several headline claims rest on tests that failed or only passed after the fact, while the abstract reads as if everything held. 51 of the 93 pre-registered criteria were not met, which the abstract doesn't mention. For OLMo, the falsification test was triggered and saved only by a clause the pre-registration never stated, and the "forms by about one percent of pretraining" result clears its bar by just 0.002. The claim that refusal can be moved without changing what is said rests on small numbers: in one Gemma arm the model refuses everything, so nothing is left to check. Refusal varying by tradition holds up mainly in Qwen, and which traditions get refused changes across models, wordings and checkpoints. These points don't sink the paper, but the abstract and conclusion should match what actually held up.
Questions for authors
Clarifications:
- Can you rerun the steering test on names built from first names and surnames that were never used to build the direction?
- On the main held-out set, how do random directions at the same layer and size as the actual edit do? The random control there uses a different layer and a smaller size.
- For real people whose names mislead the model, where do the wrong answers land if you count all of them, including ones that don't map to any tradition?
- In the refusal-steering experiments, how many answers are left to score in each arm?
- Was the OLMo context clause written before you saw the few-shot results, and did every checkpoint get the same prompts?
- What is the release plan for model answers about real people who have few Wikipedia links and may not be public figures?
Suggestion (future work): testing larger models, non-Latin scripts and relations beyond birthplace and university would show how general this is.
Thank you for the detailed response and the new experiments.
Most of my concerns are addressed:
Two things I would still like to see in the final version:
The limits on model size and script remain, but they are stated. I am raising my rating to 3.