Coordinate-Dependent Generalization in Transformers: Controlled Interventions and Prospective Prediction on Modular Arithmetic

The same mathematical relation can be learnable or unlearnable for a transformer depending only on the coordinates in which it is presented. We isolate this effect with modular arithmetic tasks whose targets are fixed by theorems, where the presentation can be changed while the underlying relation, the information available to the learner, the training data and the held-out inputs are preserved. Re-expressing both inputs and outputs in discrete-logarithm coordinates turns a nonlinear target into an affine one and raises held-out accuracy from chance to between 0.459 and 0.853 on two theorems, while changing either side alone leaves both at chance. In a second, matched intervention, two presentations that share their relabelled inputs, algebraic degree and agreement with the closest affine map reach 0.977 and 0.006, so the internal structure of the presented target, not these coarser properties, carries the effect. Statistics of the complete presented target, computed before training, correctly predict the outcome on all 18 held-out tasks, ten of them with predictions specified before training. Measurements of the trained networks show that the presentation determines whether a low-complexity solution is available, while weight decay affects whether training selects it. All core results use positional-digit encodings.