Where Fourier Components Meet: How Features Interact in Grokking

Learning useful components does not by itself explain how a network combines them into a computation. We develop a theoretical framework for feature interactions, using quadratic networks for modular addition to study how matching Fourier components combine across hidden units. An exact factorization separates interaction power into mode magnitude, block balance, and cross-block overlap, showing how cross-block pairing changes power while preserving each block's distribution of component strengths. We derive an exact state-dependent covariance law for overlap under gradient flow and a finite-step identity for realized optimizer updates. In Adam and Muon networks, within-block concentration accounts for most observed overlap growth, while relative alignment increases consistently. Fixed-marginal interventions that disrupt pairing degrade trained predictions, while phase-only controls preserve interaction power but alter predictions, showing the complementary role of readout compatibility. Within matched training configurations, grokking runs attain higher endpoint overlap. A population mean-squared-error lower bound connects interaction power and readout capacity to reconstruction. Together, these results link where learned components are placed to the interactions they form and the computations they support.