acceptodds
Under review as a conference paper at ICLR 2027

The Numeral Base Sets the Fourier Features Language Models Compute With

Abstract

Understanding neural networks requires explaining how the geometry of their representations supports the algorithms learned from training data. The Fourier features used by language models to represent numbers are tailored to the base-10 number system, and we ask how these features change when models learn a different numeral system. We pretrain matched language models on text and arithmetic in bases 6, 7, 8, and 10. We find that the features form harmonic families whose fundamental period follows the numeral base, e.g., periods 10,5,10/3,2.5,2 in base 10 and 7,7/2,7/3 in base 7. From these experiments, we develop a model in which each Fourier feature contributes a periodic pattern to the scores for possible digits of the sum. For a decimal sum ending in 7, a period-5 feature favors 2 and 7 because its score pattern repeats every five digits; combining it with a period-2 feature that favors odd digits gives 7 as their only shared peak, the digit the Chinese Remainder Theorem assigns to those two residues. Interchanging feature angles between inputs tests which digits each feature favors and how combinations of features determine a prediction. These contributions approximately add near the output in the base-7 and base-8 models and Qwen3.5-9B. In Qwen, patches at earlier layers reveal nonlinear interactions: the combined effect of several features exceeds the sum of their individual effects. We also find that carrying one into the tens position shifts Qwen's tens-place Fourier features toward the phases for adding one to the tens digit, with the parity feature updating last.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, Fourier representations, activation steering, causal interventions

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_BTP6OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper attempts to explain how a language model internally represents a number. The authors propose an assumption that numbers are represented as Fourier features (corresponding to different number bases). Experiments show that the model's behavior can be intervened in predictable ways, e.g., controlling the output prediction results by modulating the Fourier feature values.

Critical strengths
  1. The authors explore an interesting problem regarding mechanistic interpretability. The approach of using Fourier features to understand how machines understand numbers is also novel.
  2. The authors designed a series of experiments to support the findings.
Critical weaknesses
  1. The paper is difficult to follow. Although necessary mathematical information is provided, it is challenging for human readers to follow (even if I believe it is perfectly fine for machine readers to follow). Suggestions: clearly state the motivation, methodologies, and expected outcome in the abstract and introduction. Connect them logically. Section 2: explain the overall goal and intuition first, then introduce each subgoal, then show the math. Basically, the materials should be connected in a logical flow. If the writing is improved, it will be easier for me to assess soundness. (Writing is a major concern for clear rejection. It is difficult to improve during rebuttal.)
  2. The paper still appears to be in a draft stage. For example, the subplots in Figure 5 are not aligned, and some labels in Figures 2 and 4 overlap (unreadable).
  3. There are no alternative assumptions about numbers (e.g., that numbers are represented linearly), and the proposed assumption is more explainable than the alternatives. The experiments provide convincing evidence that the assumption about Fourier features play a role in the model's understanding. However, they do not show it is the most compelling explanation, nor do they show that the model actually relies on these features to perform arithmetic.
Questions for authors
  1. Can the authors discuss possible alternative explanations for addition and provide evidence for why the Fourier-based explanation is more convincing?
  2. Can the authors provide a clearer and more organized explanation of the methodology, ideally with figures demonstrating the probing and patching procedures?
Reviewer_ts1VOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Authors provide an interesting study of appearance and effect of Fourier features corresponding to numbers in controlled LLM architectures, when the numbers are presented in various numeral basis. Such studies have already known in the case of standard base 10, and the present paper extends them in the new direction.

Critical strengths

The research question is interesting and fills in a gap in the current literature. The experiments (at least the parts that I managed to understand) are extensive, to the point and clear in nature, and thorough enough to convince in the conclusions. In particular, the study involving Qwen model helps to confirm the findings.

Critical weaknesses

In general, I think the paper should be substantially rewritten and improved on clarity and presentation. Technical terminology is used early on without proper introduction or explanation, the sentences seem disconnected and without a flow, and it takes a substantial effort from a reader (at least in my case) to understand certain parts of the paper. This is in contrast with the general, clear idea of the paper, to study the interaction of already observed Fourier features in case of different numeral basis. The unreadable parts of the paper, its density in many places and lack of coherent structure is the deciding factor for my score.

Remarks:

Abstract: Already from the sentence "We find that the features form harmonic families whose fundamental period..." it becomes very difficult to follow what are the conclusions of the paper. For example, "In Qwen, patches...", unless one reads the rest of the paper, this sentence has no meaning at all.

Abstract and throughout the paper: Keep the consistent notation, either use fractions as 10/3 or decimal points as 2.5.

Line 049: "For units digits, the fundamental..." What does fundamental mean here?

Paragraph 053: It is not clear what the authors mean here. For example, sentence "... a period-4 feature for base-8 units groups digits four apart..." is difficult to understand unless one has the context of the methodology section.

Line 127: "...with replications..." Replication of the tests or authors mean something else?

Line 142: What is harmonic family?

Line 146: End of sentence, should it be b/kb/k?

Line 148: This paragraph is too dense, and it is not clear the importance of it. It should either have a worked out example (perhaps with details in Appendix), either have a graphical representation. In fact, the whole method should benefit of a visual guide.

Line 154: Start of paragraph, perhaps rephrase with "last token activation". Further, "for each tested period", it is nowhere mentioned before what the test is, as this would motivate what follows. Equation (2)

Line 163: What does it mean protected directions?

Line 171: It is not clear what is the original radius? Do authors mean radius of the norm of the projection in two dimensional plane, or something else? In the same paragraph, what are the paired inputs, do you mean paired inputs corresponding to different chosen basis or something else? What is "subspace center"? This whole paragraph reads very disconnected, it should be rewritten.

Line 183: What is a factorial? Also, why is this epmhasized for bases 7 and 8, what is different for other basis? This whole paragraph is again too dense and disconnected.

My impression is that there is no unifying setting for the experiments, but rather a sequence of different rules that seem arbitrary in when they are applied. Like for example, last sentence of paragraph starting at Line 211 where authors suddenly separate Qwen and NanoChat from Pythia nad Llama, without a prior justification for it.

...

Discussion:

Line 429: What does it mean "the family is set by notation"?

Line 436: What does it mean "nonlinear within itself"?

Line 439: The sentence "Near the output theharmonics add, each writing a periodic profile at its own frequency, and the prediction follows the peak of the sum; a few blocks earlier in Qwen the same patches interact."...

This whole paragraph should be rewritten.

Questions for authors

I do not have much to add to what I wrote above. Perhaps a suggestion for the experiments, to consider also multiplication as for the sake of completion.

Reviewer_QZv9OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the harmonic features that show up during pertaining on modular addition. Previous work has studied this for the typical base-10, but this paper further studies on several other basis to show that the harmonics are actually caused by the base and not the architecture etc.

Critical strengths

1- The model studies an important aspect of the language models: why do the they learn certain features such as the harmonic features for calculating modular arithmetic.

2- Their study that how each feature (e.g, the harmonic features) contribute to calculating the final score is sound, and their interventions provide enough evidence for it.

Critical weaknesses

1- The abstract and the introduction are written poorly and I couldn't understand after careful reading.

2- The paper does not motivate the main objective. I think the main message is to show that the harmonics are features of the data and not necessarily the architecture, but I don't really catch this from reading the introduction.

3- That said, I think a necessary experiment would be training different architectures or models on the same data to see if the harmonics change. Right now, I do not see anything surprising in the experiments: with different training data the model learns different features.

Questions for authors

Please read above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.