- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper attempts to explain how a language model internally represents a number. The authors propose an assumption that numbers are represented as Fourier features (corresponding to different number bases). Experiments show that the model's behavior can be intervened in predictable ways, e.g., controlling the output prediction results by modulating the Fourier feature values.
Critical strengths
- The authors explore an interesting problem regarding mechanistic interpretability. The approach of using Fourier features to understand how machines understand numbers is also novel.
- The authors designed a series of experiments to support the findings.
Critical weaknesses
- The paper is difficult to follow. Although necessary mathematical information is provided, it is challenging for human readers to follow (even if I believe it is perfectly fine for machine readers to follow). Suggestions: clearly state the motivation, methodologies, and expected outcome in the abstract and introduction. Connect them logically. Section 2: explain the overall goal and intuition first, then introduce each subgoal, then show the math. Basically, the materials should be connected in a logical flow. If the writing is improved, it will be easier for me to assess soundness. (Writing is a major concern for clear rejection. It is difficult to improve during rebuttal.)
- The paper still appears to be in a draft stage. For example, the subplots in Figure 5 are not aligned, and some labels in Figures 2 and 4 overlap (unreadable).
- There are no alternative assumptions about numbers (e.g., that numbers are represented linearly), and the proposed assumption is more explainable than the alternatives. The experiments provide convincing evidence that the assumption about Fourier features play a role in the model's understanding. However, they do not show it is the most compelling explanation, nor do they show that the model actually relies on these features to perform arithmetic.
Questions for authors
- Can the authors discuss possible alternative explanations for addition and provide evidence for why the Fourier-based explanation is more convincing?
- Can the authors provide a clearer and more organized explanation of the methodology, ideally with figures demonstrating the probing and patching procedures?