Steering What SGD Learns: Spectral Control of Optimization through Dynamic Data Transformation

Out-of-distribution (OOD) generalization is commonly addressed by adapting the training data distribution to the target environment. However, changing the data distribution also changes the optimization dynamics of stochastic gradient descent (SGD), and therefore can alter which feature directions are learned preferentially. Existing distribution-matching approaches largely overlook this optimization-level effect. We introduce INTCA (Iterative Neural Tangent Covariance Steering), a framework that treats data transformation as a mechanism for controlling the spectral learning dynamics of SGD. At each training stage, INTCA characterizes the current optimization geometry through the covariance of sample-wise gradient features and constructs a desired covariance from unlabeled target data or stable structures shared across source environments. Rather than matching input distributions or directly selecting target-like samples, INTCA dynamically reweights training samples according to how their covariance contributions steer the current optimization spectrum toward the desired one, with entropy regularization preserving the original data distribution. We theoretically show that increasing the covariance strength along a target-relevant direction accelerates its learning under local tangent dynamics, and establish a dynamic tracking bound that accounts for the evolution of tangent features in finite-width networks. Experiments across domain generalization and distribution-shift benchmarks demonstrate that controlling the optimization spectrum through data transformation can consistently improve OOD generalization.