acceptodds
Under review as a conference paper at ICLR 2027

Steering What SGD Learns: Spectral Control of Optimization through Dynamic Data Transformation

Abstract

Out-of-distribution (OOD) generalization is commonly addressed by adapting the training data distribution to the target environment. However, changing the data distribution also changes the optimization dynamics of stochastic gradient descent (SGD), and therefore can alter which feature directions are learned preferentially. Existing distribution-matching approaches largely overlook this optimization-level effect. We introduce INTCA (Iterative Neural Tangent Covariance Steering), a framework that treats data transformation as a mechanism for controlling the spectral learning dynamics of SGD. At each training stage, INTCA characterizes the current optimization geometry through the covariance of sample-wise gradient features and constructs a desired covariance from unlabeled target data or stable structures shared across source environments. Rather than matching input distributions or directly selecting target-like samples, INTCA dynamically reweights training samples according to how their covariance contributions steer the current optimization spectrum toward the desired one, with entropy regularization preserving the original data distribution. We theoretically show that increasing the covariance strength along a target-relevant direction accelerates its learning under local tangent dynamics, and establish a dynamic tracking bound that accounts for the evolution of tangent features in finite-width networks. Experiments across domain generalization and distribution-shift benchmarks demonstrate that controlling the optimization spectrum through data transformation can consistently improve OOD generalization.

Primary area:
optimization
Keywords:
Out-of-distribution generalization, Optimization dynamics, Stochastic gradient descent, Data transformation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FfbeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Dynamic Gradient Covariance Steering, a sampling based approach to improving generalization under distribution shift. At periodic intervention steps, the method computes individual training gradients, identifies a desired low dimensional subspace, and chooses sampling weights whose projected gradient second moments approximate a target matrix. The target is constructed from shared source environment structure for domain generalization or from edited synthetic examples for controlled code experiments. Training then proceeds with the resulting sampling distribution. The theoretical analysis assumes local spectral contraction and an OOD sufficient subspace, and develops conditional risk decay bounds, feature drift bounds, and approximate matching guarantees. Experiments report improvements on seven DomainBed benchmarks and several clone detection and code retrieval metrics. A comparison using the same synthetic examples without the proposed weights attempts to separate sampling benefits from augmentation benefits. The central claim is that dynamically controlling gradient covariance redirects optimization toward features useful under distribution shift.

Critical strengths
  1. The method offers an explicit and interpretable sampling objective. Restricting covariance matching to a smaller subspace provides a concrete way to connect sample selection with a desired optimization geometry.

  2. The evaluation spans image classification and code understanding. The Synthetic Data Only comparison in Table 2 is a useful control for whether adding counterfactual examples alone explains the reported improvements.

Critical weaknesses
  1. Gradient second moments do not establish the claimed contraction model. Equation (2) uses loss gradient second moments, whereas mean SGD updates depend on first moments and local contraction generally depends on the Hessian. Assumption 4.2 postulates their equivalence without deriving it. Reweighting can also change the objective's minimizer. Consequently, covariance matching alone does not justify convergence toward the OOD optimum.

  2. The projected risk bound lacks an invariant subspace condition. Appendix A.1 assumes independent contraction of the OOD projection, but a positive restricted Rayleigh quotient does not prevent outside errors entering that projection. Equation (36) can therefore fail under deterministic dynamics. Equation (34) additionally discards introduced noise, and Appendix A.2 transfers a fixed subspace comparison to changing subspaces, further weakening the SGD stability claim.

  3. The OOD target construction is assumed rather than justified. Equation (5) treats stable source covariance directions as transferable, although shared shortcuts or negligible signals can also be stable. Assumption 4.1 and Corollary 4.7 assume the required task relevance and unseen environment structure. Section 7.1 supplies no operational selection rule. The method's essential target is therefore neither sufficiently specified nor established as OOD relevant.

  4. The code experiment does not establish covariate shift. Removing CALLS edges and regenerating programs can change semantics and downstream labels. Appendix B.1 describes auxiliary contrastive grouping but does not establish unchanged conditional clone or retrieval distributions. It also does not connect synthetic target covariance to independent evaluation. Improvements may therefore reflect auxiliary task learning rather than adaptation under the stated shift.

  5. The experiments do not identify dynamic steering as the mechanism. Synthetic Data Only isolates weighting, but not dynamic covariance matching versus fixed weights or generic sampling. No spectral or learning time diagnostics test the claim. Section 7.2's UniXcoder setup and Appendix B.1's encoder, contrastive training, and Python retrieval protocol are insufficiently connected to Table 2's broader results, limiting both attribution and reconstruction.

Questions for authors

See Weaknesses.

Reviewer_gjnLOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Dynamic Gradient Covariance Steering (DGC), a framework for improving out-of-distribution generalization by periodically adjusting training-sample probabilities while keeping the optimizer unchanged. DGC matches gradient second moments within a low-dimensional subspace, using shared source-domain statistics for domain generalization and synthetic counterfactual data for controlled covariate shift. The paper provides conditional theoretical guarantees relating covariance steering to OOD risk convergence, including analyses of feature drift and approximate matching. Empirical validation covers seven DomainBed benchmarks and code understanding tasks, with a synthetic-data-only comparison to assess the contribution of reweighting.

Critical strengths
  1. An interesting optimization-oriented approach to OOD generalization. DGC proposes to influence optimization through dynamic sample reweighting while leaving the optimizer unchanged. This offers an interesting perspective on the role of the sampling distribution in OOD learning and is the paper’s main conceptual strength.

  2. Promising empirical results across different modalities. The reported improvements on DomainBed and code understanding tasks support the potential usefulness of the approach. The comparison using identical synthetic data without DGC weighting is particularly valuable, as it helps distinguish the benefit of reweighting from simply adding synthetic examples.

Critical weaknesses
  1. The claimed reduction in OOD feature learning time lacks direct empirical validation. The paper claims that covariance steering “reduces the learning time of OOD-relevant directions” and “accelerates the corresponding excess-risk decay.” However, the experiments report final performance without directly evaluating these effects. Empirical evidence supporting these claims would strengthen the paper.

  2. The connection between gradient second moments and OOD learning needs further justification. The outer products in Equation (2) preserve directional axes but do not distinguish opposing gradients. A large eigenvalue may therefore reflect conflicting updates rather than consistent progress toward an OOD solution. Clarifying the conditions under which matching these statistics promotes OOD-relevant feature learning would strengthen the methodological justification.

Questions for authors
  1. Clarification of the theoretical connection: Could you clarify the conditions under which the eigenvalues of Equation (2) determine error contraction as assumed in Assumption 4.2? In particular, since opposing gradients contribute identical outer products, how does the analysis account for the distinction between optimization progress and gradient variability?

  2. Clarification of the empirical evidence: Could you indicate whether the existing experiments provide measurements of reduced learning time of OOD-relevant directions or accelerated excess-risk decay? If available, please explain how these quantities were measured and how the results support the claimed mechanism.

  3. Clarification of computational cost: Could you report the additional training time and peak memory usage of DGC relative to the corresponding baseline? Please also clarify which parameters are used for gradient computation and how frequently the covariance estimates and sampling weights are updated.

Reviewer_TA1MOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Dynamic Gradient Covariance Steering (DGC) to improve out-of-distribution (OOD) generalization. DGC dynamically adjusts sample sampling probabilities to match a target gradient covariance within an OOD-relevant subspace, aiming to guide SGD toward learning more transferable features. The paper provides theoretical analyses of feature learning dynamics, OOD excess-risk convergence, and covariance approximation errors. Experiments on DomainBed and code understanding benchmarks show improvements over existing baselines.

Critical strengths

The paper explores OOD generalization from the perspective of SGD optimization dynamics, providing an interesting alternative to conventional distribution alignment and invariant representation learning.

DGC provides a data-centric approach to controlling gradient statistics through dynamic sampling without modifying the underlying SGD optimizer.

The paper includes theoretical analyses of OOD feature learning, convergence, and approximation stability.

The experiments cover both vision-based domain generalization and code understanding under controlled covariate shift. DGC achieves promising improvements over the reported baselines, including ERM and Fishr.

Critical weaknesses
  1. Equation (2) defines Σt(p)=∑ipiϕit(ϕit)⊤\Sigma_t(p) = \sum_i p_i\phi_i^t(\phi_i^t)^\top, which is actually the uncentered second-order moment of the gradient, rather than the usual centered covariance. This matrix includes the contributions of both the mean gradient and the gradient variance. Larger eigenvalues indicate that a certain direction has a larger second-order moment of the gradient, but this does not necessarily mean that Stochastic Gradient Descent (SGD) will learn the effective features faster along that direction. For example, samples in the opposite direction may generate larger second-order moments, but their average gradient is zero. The authors need to explain why this statistical quantity is chosen to characterize the optimization preference and to distinguish the roles of the mean gradient, gradient variance, and the effective learning direction.

  2. Hypothesis 4.2 states that the optimization error along the eigenvector satisfies at,j,k+1=(1−ηλt,j)at,j,ka_{t,j,k+1}=(1-\eta\lambda_{t,j})a_{t,j,k}. However, the local convergence dynamics of general SGD are usually related to the loss Hessian, and the second-order moment of the gradient used in equation (2) is not equivalent to the Hessian. The author has not proved that this contraction dynamic holds in actual deep networks. Theorem 4.4 subsequently derives an upper bound for the OOD excess risk by combining Hypothesis 4.1 and 4.2, so its main conclusion largely depends on the pre-assumed contraction relationship. The author should clarify the models and conditions under which this assumption holds, and verify the relationship between the eigenvalues and the actual feature learning speed through experiments.

  3. DGC adjusts the sampling probability to match the second-order moment of the target gradient, but non-uniform sampling also changes the expected stochastic gradient descent (SGD) gradient, that is, from the original ERM gradient to ∑ipi∇θℓi\sum_i p_i\nabla_\theta\ell_i. Algorithm 1 does not explicitly use importance correction, so DGC actually also changes the training objective. The performance improvement may stem from simple sample reweighting rather than the emphasized spectral domain guidance mechanism. The author should analyze the changes in the mean gradient, central covariance, and non-central second-order moment separately, and supplement corresponding control experiments to distinguish the contributions of different factors.

  4. The differences between this paper and existing importance sampling and adaptive sampling methods are not clearly defined, and there is a lack of detailed comparison. Additionally, Group DRO improves the generalization performance under distribution shift by dynamically adjusting group weights, although this has been included in Table 1 for comparison, it would be better to discuss it from the perspective of the method mechanism. DGC uses the second-order moment of the gradient to determine the sampling probability, but the paper has not fully demonstrated that this strategy has unique advantages over existing methods.

  5. The intuitive explanations of the core method and mechanism in the paper are insufficient. The main text directly enters the OOD-related subspace, spectral optimization intensity, and convergence analysis from the definition of gradient covariance, but does not fully explain why the covariance feature spectrum can characterize the learning preference of SGD, and how dynamic sampling changes the actual feature learning trajectory. It would be better to add overall framework diagrams and elaborate on the relationship between the target subspace construction, sampling weight optimization, and SGD updates.

  6. Algorithm 1 requires periodically calculating the parameter gradient of each sample, estimating the low-dimensional OOD-related subspace, and solving the sampling weight optimization problem involving all training samples. For large-scale settings such as ResNet-50 and DomainNet, these operations may incur high computational and storage costs. The paper does not fully explain whether the full parameter gradient or low-dimensional approximation is used, nor does it provide a detailed efficiency comparison. The author should report the training time, the number of hours the GPU was running, the peak memory usage, and the additional gradient computation volume. Additionally, they should analyze the relationship between the intervention interval KK, the subspace dimension rr, and the computational cost as well as the final performance.

Questions for authors

Please check the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.