Implicit Bias of Gradient Descent under Feature-Mediated Spurious Correlations

We study how feature-mediated spurious correlations affect the learning dynamics of gradient descent on linearly separable data. Unlike the widely studied label-mediated setting, where the spurious feature depends only on the class label and is conditionally independent of the causal features, feature-mediated correlations entangle spurious and causal signals at the instance level through group-specific operators, making them both harder to detect in practice and more complex to analyse theoretically. We first generalise the implicit-bias result of soudry2018implicit to continuous distributions. Building on this, we derive closed-form group-wise classification error rates. In the general setting, each group's error decays at a rate governed by its hard-margin ${\gamma }_g$. In the isotropic regime, we identify a single exponent $\alpha$ that captures the competition between the minority's geometric margin advantage and the coupling induced by the spurious alignment operators. A phase transition occurs at $\alpha =1$: below it, both groups decay at rate ${\kappa }_g^t/({\epsilon }_g{z}_t)$ and group proportion directly controls learning speed; above it, the minority escapes the $\epsilon$-dependence entirely and converges at the faster polynomial rate $\ln (z_t{)}^{{\alpha }_{\xi }-1}{z}_t^{-{\alpha }_{\xi }(1+{\delta }_{\text{vmaj}})+{\delta }_{\text{vmin}}}$. These results formalise in closed form the widely observed phenomenon that group imbalance implicitly biases optimisation dynamics toward the majority, while revealing that the effect of imbalance can be entirely superseded by geometric margin advantages. Finally we establish a non-vanishing error at test time, under spurious correlation shift. Synthetic experiments confirm the rates, the prefactors and the phase transition, and last-layer retraining on Waterbirds shows the predicted $1/({\epsilon }_g{z}_t)$ scaling of each group’s error on five frozen representations.