- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper analyzes the internal representations and computational mechanisms of a small Transformer trained to solve Kepler's equation, . The authors observe a number-line-like structure in the numerical token embeddings and propose that attention reads significant digits and transfers information about the eccentricity into a specific vector direction, termed the -register, which the MLP then uses to produce nonlinear corrections. They validate the causal role of this direction through removal and scaling interventions and analyze how the final LayerNorm affects the relationship between output contributions and actual intervention effects. The authors also investigate the consistency and limitations of their findings across different model configurations and release 240 model checkpoints and analysis code as an experimental environment for evaluating interpretability methods.
Critical strengths
-
The paper validates the causal role of internal representations beyond identifying correlations. Rather than merely identifying a direction associated with through regression, the authors examine whether this component is used in subsequent computations through removal and scaling interventions, along with random-direction controls. Although these interventions are approximate, they provide more convincing evidence than representation visualization alone.
-
The paper provides extensive validation across different settings and releases substantial resources. The authors investigate the consistency of the main observations and their exceptions across multiple random seeds, model widths, training durations, and architectural variations. They also release the checkpoints and code for the 240 models analyzed. Although the scope of the task is narrow, these resources provide a solid foundation for reproducing the results and conducting further interpretability research.
Critical weaknesses
-
The new insights gained are limited relative to the level of detail in the analysis. The paper identifies internal variables and validates their roles through interventions, but this general approach is already well established. In particular, prior work on the harmonic oscillator has also analyzed intermediate representations of internal computations and performed interventions that modify physical parameters. While I acknowledge the detailed analysis of computational pathways in this paper, it remains unclear what new insights into neural network computation are provided beyond adding the Kepler equation as another case study. Furthermore, the presence of an approximate decomposition that does not fully explain the precise computation makes it difficult to assess the scope of the contribution, especially given the strong claim of having "read the algorithm."
-
The paper lacks sufficient control experiments to assess the significance of the identified mechanism. In an independent exploratory experiment, I found that a GELU MLP with only 1,153 parameters, without attention or LayerNorm, achieved similar median accuracy and exhibited a partially manipulable direction corresponding to in its first hidden layer. This does not contradict the paper's analysis of the Transformer, but it highlights the need for a baseline against which to assess how much of the observed phenomenon is expected from general function learning. Even if the authors do not claim that the identified mechanism is specific to Transformers, without comparisons to simpler control models, it is difficult to determine why the mechanism discovered in this case is particularly significant.
-
The significance of the proposed evaluation environment remains more of a proposal than an empirically demonstrated contribution. Releasing the model and code is useful, but their availability alone does not establish the research value of a new evaluation environment. Although the paper suggests that this environment can be used to evaluate methods for identifying hidden computations, it does not sufficiently compare existing methods to demonstrate which types of failures the environment can distinguish or what improvements it can enable. Given that the current results focus on a single equation and primarily on one-layer models, there is insufficient evidence to establish a broader methodological contribution beyond a detailed case study.
Questions for authors
-
What is the scientific motivation for choosing the Kepler equation? Beyond the fact that its solutions can be computed numerically with high accuracy, what new mechanisms did the authors expect this equation to reveal compared with previously studied physical or mathematical tasks? Have the authors performed the same regression and intervention analyses on a linear control task such as ? If not, how can they distinguish whether the identified -register is a feature associated with solving a nonlinear equation or simply reflects a general mechanism for propagating input variables?
-
Do the findings generalize beyond the specific structure of the trigonometric correction? For example, in tasks involving higher-order polynomial terms such as or interaction terms such as , is it still possible to identify a single direction corresponding to and reproduce counterfactual outputs by scaling that direction? If the current findings are limited to a single equation, please clarify the scope of the claims that the authors consider generalizable.
-
What additional insights does this study provide beyond the phenomena observable in a simple MLP? In my limited control experiment, a GELU MLP with only 1,153 parameters, without attention or LayerNorm, achieved similar median accuracy. Applying regression using the same 11 functions and performing interventions on independent inputs revealed a partially manipulable direction in the first hidden layer that explained 90–97% of the variation associated with . Although differences remain in maximum error and intervention fidelity, what is the central insight of this paper in light of these findings? In particular, please clarify what new knowledge is gained from the detailed analysis of computational pathways beyond identifying variable representations and demonstrating their manipulability.
2 comments
Thank you for running the requested control experiments, including the plain MLP baseline and the linear/cubic tasks. The follow-up results demonstrate that the primary algorithmic claims of the paper do not hold: both the number line and the -register appear in simpler baselines and linear tasks, reflecting general input-propagation mechanisms rather than properties specific to Kepler's equation or Transformer architectures. Furthermore, the register fails to generalize reliably beyond a single layer, and the proposed evaluation environment remains undervalidated without systematic comparisons across existing interpretability methods to demonstrate its discriminative value. Given that the core contributions are largely invalidated by these findings, I maintain my recommendation for rejection and keep my score at 2.
Thank you for the detailed review, and for running your own MLP control. We ran our own version, and it agrees with yours in outline. Our general comment lists the new experiments.
"What is the scientific motivation for choosing the Kepler equation? ... what new mechanisms did the authors expect this equation to reveal compared with previously studied physical or mathematical tasks?"
The closest circuit-level account is modular addition (Nanda et al., 2023), a discrete task. We wanted a continuous one with a known classical answer. Kepler's equation is one of the oldest problems in mathematical physics, with a long line of solution methods, so the learned algorithm can be placed against classical ones (§4.6). We also like it. We expected the trigonometric task to push the model to a circular code, as modular addition does. It does not. The digits form a number line, and no wrapped variant builds a circle (§4.2). We also hoped for a new mechanism, or one that mirrors a classical method more closely. The output approximates the Fourier–Bessel series, as any accurate solver's must, but the MLP builds its corrections from a learned spline basis (§4.4). And the new runs show the main structures are not specific to Kepler. Your point about the harmonic-oscillator work stands: the intervention method is not new.
"Have the authors performed the same regression and intervention analyses on a linear control task such as E=aM+be? If not, how can they distinguish whether the identified e-register is a feature associated with solving a nonlinear equation or simply reflects a general mechanism for propagating input variables?"
Yes, with a = b = 1, with the final LayerNorm on and off. The register appears in 5 of 5 and 4 of 5 seeds. We set a bar before the run. If Kepler's register did something a linear task does not need, the LN-off linear model should pass e straight to the readout and barely use the MLP. The bar was |cos| of at least 0.7 and an MLP ablation that raises error under 3×. We measured 0.487 and 2316×. So we cannot distinguish the register from a general mechanism for propagating an input.
"In tasks involving higher-order polynomial terms such as E=M+eMk or interaction terms such as e2M+eM2, is it still possible to identify a single direction corresponding to e and reproduce counterfactual outputs by scaling that direction?"
We ran
E = M + e(M/π)³, which keeps E odd in M and |E − M| ≤ 1, as in Kepler. We did not run the interaction terms. One direction carries 86% to 99% of the write's variation across e in all 10 seeds. Scaling it reproduces counterfactuals only partly. With the final LayerNorm on, zeroing it closes at least half the gap to the model's own e = 0 output in 3 of 5 seeds, and halving it closes 0.245 to 0.739 of the gap to its e/2 output (Kepler: 0.656 to 0.849). These are fractions of a small gap: cubic's do-nothing gap is 0.044 rad against Kepler's 0.257, and in radians its residuals are below Kepler's. With LN off, scaling fails in all 5 seeds. On scope: we would claim neither the number line nor the register as a property of Kepler. The line appears on every task with this input format. The register appears on most seeds but not reliably."What is the central insight of this paper in light of these findings? In particular, please clarify what new knowledge is gained from the detailed analysis of computational pathways beyond identifying variable representations and demonstrating their manipulability."
We ran your control with hidden widths 27, 21 and 21 (1,153 parameters), float inputs, 5 seeds. It beats our primary configuration at the median (4.9e-4 vs 1.9e-3 rad) and loses at the worst case (0.238 vs 0.086). In its first hidden layer, one direction carries 72% to 82% of the variation across e, below the 90–97% you report. We guessed the widths, so our networks or measure may differ. That direction is manipulable in 3 of 5 seeds (median gap closed 0.550, against 0.828 for the transformer). Our bar, also set before the run, was that the transformer's register is stronger if at least 4 of 5 MLP seeds fail. Only 2 did. You are right: a manipulable direction is not special to the transformer.
On the central insight, we concede it is narrower than the paper claims. The pathway analysis adds how the model uses e: attention passes it as one direction the readout barely sees, and the MLP turns it into e·sin(nM) corrections. That pattern is not specific to Kepler. And the account is approximate. Replacing attention's write with its 11-term fit raises median error from 1.5e-3 to 5.3e-2, so "read the algorithm" overstates it. We read its structure, not its precision.
On the evaluation environment.
We agree it is a proposal more than a demonstrated contribution. The ridge and PCA comparison in the general comment is a first test, and we read it as weak support.
Thank you again.