Alignment, Not Isotropy: Sharp Regret Bounds and a Task-Covariance Correction for Spectral Representation Learning

Spectral features can remain optimal under strongly uneven task preferences when they retain the directions most useful to the tasks. We show how transfer depends on this alignment. In a local-task model, expected probing gain depends on the task prior through its covariance $\Lambda$. For $k$ features, the spectral choice selects the leading eigenspace of $B{B}^{\top }$, while task-optimal features select that of $B\Lambda B^{\top }$, where $B$ records dependence between views. We derive regret bounds that account for alignment, matching worst-case lower bounds for a flat leading spectrum, and bounds using the leading $2k$ directions with a spectral-tail term. A spectral gap makes the regret bound quadratic in small anisotropy. We also bound the imbalance from random preference patterns and from averaging independent tasks with an isotropic population covariance. With known $\Lambda$, changing one term of the spectral contrastive loss selects task-optimal features. We propose a diagnostic using a labelled task bank and test correction without retraining. Correction reduces synthetic held-out regret from $0.858$ to $0.003$ with $2000$ tasks when less task-relevant directions dominate. On CIFAR-100, empirical regret falls by $0.031$–$0.053$ on held-out fine-label tasks using the same images. Correction can nevertheless hurt with a small labelled task bank.