When Broader Pretraining Hurts In-Context Learning: A Spectral Diagnostic from Inverse Linear Regression

Broader pretraining can improve generalization in in-context learning, but adding source tasks can hurt performance on a fixed target family. We study this effect in ill-posed inverse linear regression (ILR), where the learner estimates a latent task vector from an underdetermined linear system. From a Bayesian viewpoint, pretraining provides a prior for this inference. For a Gaussian target prior with equal variance on a low-dimensional subspace and known rank and ridge ratio, learning the Bayes rule reduces to estimating that subspace. For block-diagonal source covariances, an eigengap determines whether additional tasks preserve the target subspace, create a tie, or displace target directions. We derive a sharp mixing threshold above which target-subspace recovery is lost. We prove this phase transition for a controlled two-stage ridge estimator and use a data-driven ridge estimator and a linear transformer to examine the same diagnostic empirically when estimation, architecture, and optimization also affect performance.