Same Tokens, Different Readings: Representational and Causal Tests of Lexical-Unit Encoding in Language Models

Behavioral signatures, such as how a model's next-token probabilities move across a span, are routinely read as evidence about what language models represent internally. We ask what one such signature actually licenses, in a setting where the surface form can be held fixed: idiomatic and literal uses of the same expression. Across autoregressive models spanning 124M–122B, the surprisal-drop signal remains positive across model families. In the corpus-scale analysis, mean surprisal over the span does not strongly separate the two readings (Cohen's d = 0.03), whereas the first-to-last surprisal drop does (d = 0.91); residualizing frequency, collocation, span length and continuation predictability reduces this to d = 0.14, and matched controls retain positive effects (d = 0.25–0.52). We then ask whether this behavioral signal corresponds to internal structure. In three models (0.77B–14B), probes under nested expression-disjoint cross-validation over 402 expression types separate the readings at AUROC 0.981–0.991, against 0.535–0.564 for surface-form and context-free controls. Follow-up controls show that the preceding context alone reaches AUROC 0.989 in two models. Patching the final-word predictor state with a donor from the opposite reading of the same expression, matched on observed surface form and in-context token sequence, shifts final-word surprisal by +1.99 to +3.30 bits in difference-in-differences, while same-reading donors give intervals spanning zero (cross-minus-same-reading: +2.13 to +3.12 bits); the effect is near zero or negative before the expression and largest at the final-word predictor. Because random-expression donors are not reliably smaller, we report a reading-contingent causal component rather than an idiom-specific mechanism. More broadly, the study offers a template for validating and bounding behavioral signatures with representational and causal evidence.