acceptodds
Under review as a conference paper at ICLR 2027

Small Transformers Learn Kepler's Equation with a Number Line and an Eccentricity Register

Abstract

We train a family of 210 small transformers to solve Kepler's equation , then reverse-engineer the algorithm they learn. To our knowledge this is the first circuit-level account of a continuous, transcendental equation. The task is trigonometric, yet the digit embeddings form a number line, not a circle. Attention writes into a single residual-stream direction largely hidden from the readout, the e-register. The MLP reads it and writes the corrections. The output carries harmonics of to . Scale the register and the corrections scale with it; set it to zero and the model tracks its own circular orbit. The line and the register hold across the 210-model family. We release the models and the analysis code as a testbed: the register's variable is known to be , so a method that claims to find hidden computation can be scored against it.

Then back it, or bet against it.

Related papers

Open the market on this paper to see 7 more related papers.