AUTOREGRESSIVE MODELS REPRESENT MULTI-TOKEN NUMBERS AS PLACE-SCALED HELICES

Large numbers are often split across multiple tokens by traditional tokenizers in modern Language Models (LM), raising a fundamental question: how are the numerical values represented when no single token corresponds to the entire number? Prior work reads the whole number from one hidden state, usually the last token. We show that this last token correlates with the number but does not causally control the other digits, and therefore in this work we analyse each token’s hidden state separately. We find that every token follows the same helical structure, with the period of the helix growing with the token’s place. We call this the Place-Scaled Helix. It correlates with the number representations better than prior structures, and intervening model activations using it changes the output digit, showing causal relevance. We have analysed the number representation across single-digit and multi-digit tokenizers, LLMs and VLMs, on arithmetic tasks showing a generalized numeric structure for autoregressive language models.