- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper asks whether socially matched first names are comparable inputs to language models. It first audits 12 tokenizers on about half a million first names. It finds that single-token access is selective and uneven across race- and gender-associated name groups, even after accounting for frequency and length. It then compares matched pairs of atomic and short-fragmented names within demographic strata. Using adjective probabilities read from an intermediate layer, it reports that atomic names receive higher scores on task-relevant concepts such as competence and promise. Finally, it shows that the gaps replicate on held-out names and that steering along a task direction shifts a later binary choice.
Critical strengths
The tokenizer audit is the most useful part of the paper. It is large, clearly reported, and gives evaluators a concrete reason to check lexical comparability when building name-based benchmarks. The matched-pair design is careful, with matching within strata and a development and evaluation split in which measurement choices are fixed in advance. The ethics discussion around the voter-registration data is good.
Critical weaknesses
My main concern is that the evidence for the central claim is weaker than the paper presents. The claim is that lexical support shapes task-relevant internal representations.
The first issue is identification. Names become single tokens because they are frequent in tokenizer training data, so atomic names are very likely more frequent in pretraining data than their fragmented matches. Matching on Florida voter counts does not control for this. Prior work already links lower name frequency to less reliable representations. The authors acknowledge the confound, but the framing throughout still attributes the effect to lexical support. A web-frequency control or a frequency-matched analysis would help.
The second issue is the size and consistency of the effect. The pooled gaps are averages over three models and are carried almost entirely by Qwen3-4B. Llama’s gaps are an order of magnitude smaller, and Ministral’s are near zero or negative. More importantly, when the same score is computed from the model’s actual output logits, the hiring and clinical gaps reverse sign. This makes the practical implication unclear. It is not shown that atomic names are treated better at the point where decisions are made. A direct behavioral comparison on the existing two-choice prompt, without intervention, would address this.
The third issue is that the RQ3 analyses do not test what they are said to test. The cross-name transfer analysis subtracts the mean development gap from evaluation pairs drawn from the same matched pool. This is a split-half replication rather than transfer. The intervention shows that steering along an adjective-derived evaluative direction changes an evaluative choice. That is expected from prior steering work and would hold for any pair of names. It does not show that the gap between atomic and fragmented names runs through this direction. Removing the direction and checking whether the gap closes would be a more direct test.
The readout method also needs more detail and validation. The paper does not specify how intermediate hidden states are mapped to adjective probabilities. Several results suggest the readout is sensitive to layer choice. Examples include the shift from layer 2 to layer 25 in Gemma-3-4B and the sign change in Qwen’s lending gap between layers 18 and 30.
On novelty, An & Rudinger (2023) already showed that tokenization length contributes to first-name bias. The paper would benefit from stating more clearly what it adds beyond broader coverage of modern tokenizers.
Questions for authors
- How is an intermediate hidden state converted into a probability over adjectives?
- On the two-choice prompt without intervention, does the model select the atomic name more often?
- Can you control for pretraining frequency, for example with web-corpus counts?
- After the task direction is projected out, does the atomic versus fragmented gap shrink?