Task Geometry and Reversible Specialization in In-Context Learning

What determines whether in-context generalization persists during pretraining? We study nonlinear Transformers on linear-regression tasks whose training priors match in task count, probabilities, mean, covariance, and task-vector norms, while differing in geometry. Across 54 independent pairs, the geometry intervention changed training-budget-averaged normalized excess risk by 0.0902 (95% CI 0.0846 to 0.0957). Both conditions improved early; their trajectories subsequently diverged. Continuation interventions from shared source weights reversed specialization, including after resetting the optimizer state. A separately prespecified four-dimensional experiment with 40 independent pairs yielded a final clean-MSE geometry contrast of 0.2529 (95% CI 0.1696 to 0.3362). A paired supervision experiment with 16 independent four-arm blocks showed that the objective moderates the geometry contrast, with an interaction of 0.02676 in normalized-risk AUC (95% CI 0.01518 to 0.03833). We also test the limits of candidate explanations: a frozen harmonic descriptor that performed well during development predicted new geometries worse than a finite-Bayes-risk baseline across 106 independent geometries. Separately, an existence result shows that equal finite Bayes-risk-reduction profiles need not identify the same Bayesian prediction function in sufficiently rich moment-matched prior families. The combined results characterize the magnitude, reversibility, and tested scope of specialization beyond covariance summaries, while distinguishing prediction of scalar risks from identification of an internal learning algorithm.