Initial Spectral Power Predicts Frequency Selection in Grokking and Steers It in Tied MLPs

Networks that grok fit their training data early and generalize much later, using a handful of Fourier frequencies picked up along the way. We ask what decides which frequencies a network ends up with, and whether the same factors decide when it generalizes. We study small networks trained on modular arithmetic, fix our decision rules before running each experiment, and re-test the main results on fresh seeds. For the first question, the initialization matters a great deal. The frequencies with the most initial power in the embedding recover 4 to 13 times more of the final frequencies than chance, in bottlenecked MLPs on two tasks and in a one-layer transformer, and they do better when the initial leaders stand out. In tied MLPs, editing that power steers the outcome in a graded way: an outsider given 0.5, 0.75, 1 or 1.25 times the leader's power wins in 0, 13, 60 and 80% of networks, and a leader whose power is cut never wins. The same edit was not confirmed in the transformer, where it is a weaker lever. Phase alignment, which decides the equal-magnitude case studied in prior theory, carries information only once all powers are made equal. The outcome survives rounding, data order and weight noise, but not a change of data split. Timing behaves differently. Enlarging the winners' head start has no detectable effect, removing all initial contrast roughly doubles the time to grok in an exploratory comparison, and the one confirmed predictor is how late the network memorizes. In these full-batch runs, training never climbs the loss walls that straight lines between memorizing and generalizing solutions cross. We also report a circuit-free symmetry measure, the naturality error. In every run we measured, across three architectures, three modular tasks and the group $S_5$, it rises during memorization and collapses as the network generalizes; its peak mostly reads prediction confidence, while its collapse exceeds what confidence explains by about an order of magnitude.