The Numeral Base Sets the Fourier Features Language Models Compute With

Understanding neural networks requires explaining how the geometry of their representations supports the algorithms learned from training data. The Fourier features used by language models to represent numbers are tailored to the base-10 number system, and we ask how these features change when models learn a different numeral system. We pretrain matched language models on text and arithmetic in bases 6, 7, 8, and 10. We find that the features form harmonic families whose fundamental period follows the numeral base, e.g., periods 10,5,10/3,2.5,2 in base 10 and 7,7/2,7/3 in base 7. From these experiments, we develop a model in which each Fourier feature contributes a periodic pattern to the scores for possible digits of the sum. For a decimal sum ending in 7, a period-5 feature favors 2 and 7 because its score pattern repeats every five digits; combining it with a period-2 feature that favors odd digits gives 7 as their only shared peak, the digit the Chinese Remainder Theorem assigns to those two residues. Interchanging feature angles between inputs tests which digits each feature favors and how combinations of features determine a prediction. These contributions approximately add near the output in the base-7 and base-8 models and Qwen3.5-9B. In Qwen, patches at earlier layers reveal nonlinear interactions: the combined effect of several features exceeds the sum of their individual effects. We also find that carrying one into the tens position shifts Qwen's tens-place Fourier features toward the phases for adding one to the tens digit, with the parity feature updating last.