acceptodds
Under review as a conference paper at ICLR 2027

Flatness and Gradient Alignment Are Both Necessary: Spectral-Aware Gradient-Aligned Exploration for Multi-Distribution Learning

Abstract

Sharpness-aware and gradient-alignment methods have been shown to improve generalization, however each family of methods targets a single geometric property of the loss landscape, while ignoring the other. In this paper, we show that this omission is structurally unavoidable and that both flatness and gradient alignment should be considered in multi-distribution learning settings. Specifically, we derive an excess-risk decomposition that yields two additive leading-order terms: (i) an alignment term, controlled by the trace of and (ii) a curvature term, controlled by , where is the average Hessian and is the covariance of the gradient across distributions. Notably, appears inverted in one and non-inverted in the other. We further show, via a counterexample, that neither quantity bounds the other in general, so no algorithm targeting only one term can guarantee low excess risk. Motivated by this decomposition, we propose SAGE (Spectral-Aware Gradient-Aligned Exploration) that targets both terms. The curvature component replaces SAM's gradient-scaled perturbation with the polar factor of each layer's gradient matrix, computed via Newton–Schulz iteration, so that the ascent step probes all directions with similar magnitude. On the other hand, the alignment component injects isotropic noise at the descent step, the magnitude of which scales with cross-distribution gradient disagreement. Experiments on five domain-generalization and two multi-task learning benchmarks show that the proposed method establishes a new state-of-the-art on DomainBed and acts as a general-purpose improvement to base MTL solvers, remaining competitive with, or even surpassing, state-of-the-art methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Representation Learning, Domain Generalization, Multi-Task Learning, OOD Generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Nc9mOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors argued that flatness and gradient alignment are both important and should be considered in multi-distribution learning. They derived an excess-risk decomposition that yields two additive leading-order terms, one is an alignment term, and the other is a curvature term. The authors also demonstrated a counterexample that neither flatness nor gradient alignment can bound the other, so focusing only one term cannot guarantee low excess risk. The authors then proposed Spectral-Aware Gradient-Aligned Exploration (SAGE) to target both terms. Based on the experimental results, they argued that the proposed SAGE establishes a new state-of-the-art on DomainBed and acts as a general-purpose improvement to base MTL solvers.

Critical strengths
  1. An interesting theoretical perspective on multi-distribution learning. I think the theoretical analysis is the most interesting part of this paper. The excess-risk decomposition provides a useful perspective on the relationship between flatness and gradient alignment, and the counterexample helps explain why focusing on only one of them may not always be sufficient. I find this an interesting direction, and I believe it could inspire further research on optimization and generalization in multi-distribution learning.
  2. Promising results across different learning settings. I also appreciate that the authors evaluate SAGE on both domain generalization and multi-task learning, rather than focusing on a single setting. In particular, the results on DomainBed are quite promising, with SAGE achieving the best average accuracy among the compared methods. Although the improvements are less consistent in the MTL experiments, I think exploring the applicability of the method across these two settings is still valuable.
Critical weaknesses

My main concern is that the paper does not convincingly demonstrate the practical need to jointly optimize flatness and gradient alignment, or show that SAGE effectively addresses the imbalance between these two properties.

The theoretical analysis is interesting, but I am not fully convinced by the empirical evidence. In particular, Figure 5 only compares SAGE with ERM and SAM, without showing whether more advanced flatness-aware or gradient-alignment methods actually suffer from the proposed trade-off. Moreover, SAGE does not consistently achieve better flatness across all domains.

I also find the ablation results somewhat unconvincing. Removing the alignment-driven noise only leads to very small accuracy drops (0.3% and 0.1%), and the paper does not show how this affects flatness or gradient agreement. Since prior work such as SAMO has already shown that sharpness-aware optimization can also mitigate gradient conflicts, I wonder whether SAGE's spectral perturbation alone could already improve both properties, making the additional noise mechanism less necessary. The fact that adding SAGE to SAMO often worsens the results in Table 2 raises further questions about this.

Overall, while the generalization results are promising, I don't think the paper provides enough evidence that its improvements actually come from jointly addressing the two properties identified in the theoretical analysis. This is a major concern for me because that connection is central to the paper's motivation and proposed contribution.

Questions for authors
  1. Can the authors provide more direct evidence that existing methods suffer from the proposed flatness–gradient alignment imbalance?

The theoretical analysis shows that optimizing one property does not necessarily guarantee the other, which I find interesting. However, I'm wondering how often this actually happens with existing methods. For example, do sharpness-aware methods such as SAGM or DISAM achieve better flatness but worse gradient alignment? Do alignment-based methods such as GGA show the opposite behavior? It would be helpful to see both properties measured for some of the stronger baselines in Table 1, rather than only comparing their generalization accuracy.

  1. Could the authors clarify the results in Figure 5, particularly the Hessian trace measurements?

Figure 5 provides some useful analysis, but I have a few questions about it. First, why does the figure only show 10 epochs? Is this the entire DG training process, or are only the first 10 epochs shown? Appendix E does not seem to specify the total number of training epochs for DG. Also, the Hessian trace results seem somewhat unstable. For example, SAGE has a higher trace than SAM in the Product domain, while the Art domain shows large fluctuations. Some trace estimates are even negative. How should these results be interpreted in terms of flatness? Would it be possible to report additional flatness measurements or results at the final selected checkpoints?

  1. How much does the alignment-driven noise actually contribute to improving gradient alignment?

In Figure 3, removing the noise only leads to small accuracy drops (0.3% on OfficeHome and 0.1% on TerraIncognita). I'm curious whether the spectral perturbation alone already improves both flatness and gradient agreement. Could the authors provide the Hessian trace and gradient agreement results for SAGE without noise? It would also be interesting to compare spectral perturbation only, alignment-driven noise only, and their combination. This would help clarify whether the two components are really complementary, or whether most of the improvement comes from spectral perturbation.

  1. How does SAGE relate to the observations in SAMO and F-MTL, and why does combining SAGE with SAMO sometimes hurt performance?

Appendix A.2 mentions that SAMO has shown that sharpness-aware optimization can already mitigate task-gradient conflicts. F-MTL also considers task-specific sharpness-aware optimization and gradient coordination. Given these findings, I'm wondering what additional problem SAGE's alignment-driven noise is intended to solve. Related to this, Table 2 shows that adding SAGE to SAMO-based methods often makes the overall results worse. For example, on Cityscapes, the \Delta_m of SAMO-FairGrad changes from −0.62 to 4.88 after adding SAGE. Do the authors have any explanation for this behavior? Could the additional perturbation or noise interfere with the optimization when the base method already addresses sharpness and task conflicts?

  1. Could the authors further clarify the connection between the theoretical quantities and the experimental measurements?

Theorem 1 identifies tr(\bar H^{-1}\Sigma_g) as the alignment-related term, but the experiments mainly use pairwise gradient cosine similarity. These two quantities are not equivalent, since cosine similarity does not capture gradient magnitude differences or how gradient covariance interacts with Hessian curvature. Similarly, the spectral perturbation is constructed from the gradient matrix rather than directly from the Hessian. I understand that the proposed components are practical approximations, but is there any empirical evidence showing that these proxies are related to the theoretical quantities? Even an approximate measurement on a smaller model or dataset would help connect the theoretical analysis to the actual behavior of SAGE.

  1. Why are standard deviations not reported for the DomainBed results?

Table 1 states that the results are averaged over three runs, but I couldn't find the corresponding standard deviations. Given that some of the improvements are relatively small (e.g., SAGE achieves 78.9% compared to BOA's 78.1%), I think reporting the standard deviations would be important for understanding how stable the results are. Could the authors provide the standard deviations across the three runs, ideally for each dataset as well as the overall average?

  1. Could the authors explain the surprisingly small training-time differences between ERM, SAM, and SAGE?

In Table 3, the reported training times are 114 minutes for ERM, 119 minutes for SAM, and 120 minutes for SAGE. I find these numbers a bit surprising, especially since SAM requires an additional forward-backward pass compared to ERM, while SAGE further introduces Newton–Schulz iterations and per-distribution gradient computations. I understand that the additional operations in SAGE may be relatively inexpensive, but why does even SAM take only about 4% longer than ERM?

I'm wondering whether the training might be bottlenecked by data loading, CPU operations, or other overhead rather than GPU computation. Could the authors provide some GPU utilization or per-iteration profiling results? This would help assess whether the reported computational efficiency would still hold under better GPU utilization.

2 comments
AuthorsOpenReview ↗

Response to reviewer Nc9m (1/2)

We thank the reviewer for their comments and insightful questions. We will answer and adress them point by point.

Questions 1, 2 & 3

We thank the reviewer for their comments. First of all, regarding the epochs in the DomainBed experiments, yes the total epochs are 10 with a total of 5000 training steps (500 steps per epoch). We apologize for omitting this information, we will include these details in Appendix E of the revised manuscript.

From Figure 5 of the manuscript several observations are of note:

  1. Gradient agreement is consistently higher under SAGE than under ERM and SAM throughout training. Although similarity declines over epochs for all three methods, SAGE maintains the highest values at every comparable point. Furthermore, the same follows for SAGM resulting from new experiments.
  2. The Hessian trace is lower for SAGE than for ERM and SAM in 3 of 4 target domains, with the exception of the Product domain, where SAM attains a slightly lower trace.
  3. While SAM has better flatness than ERM, the gradient alignment remains close to zero.
  4. There is an interesting observation regarding the dynamic of the two components (agreement and curvature), as the Hessian trace decreases (indicating lower curvature), so does gradient similarity. This points to the broader point of the present trade-off mentioned in Remark 1 and motivation of our work, that the two quantities should be targeted jointly during optimization.

To further address the comment, we provide additional metrics that support our method in Tables 1 - 5 in the general response to all reviewers. Specifically, we select attention matrices from 3 different attention blocks of the final ViT model checkpoint trained on OfficeHome and compute the Hessian-vector products by double backpropagation and run Lanczos iteration with full re-orthogonalization (160 iterations), obtaining the top-80 eigenpairs (λi,ui)(λ_{i},{u}_{i}). We have also added relevant metrics for SAGM, a stronger flatness-aware baseline and for SAGE without the noise component.

For the Hessian analysis, we compute the following metrics:

  • λ_max: top eigenvalue
  • tr(H): sum of top-80 eigenvalues calculated via Lanczos' algorithm [6]
  • align1: (u1Te)2({u}_{1}^{T}e{)}^{2}, Share of the perturbation that lies along the single sharpest direction
  • align10: ∑i=110(ui⊤e)2\sum_{i=1}^{10} (u_i^\top e)^2, Share that lies inside the 10 sharpest directions
  • curv: (eᵀHe) / λ, Curvature along the perturbation, relative to the sharpest direction. 1 = perturbing along the sharpest direction

Observations:

  • Both SAM and SAGM concentrate their perturbation on the top Hessian eigenvector, notably for SAGM it's as high as 99%. On the other hand, SAGE and SAGE_NoNoise spread the perturbation out in more directions.
  • The flatness-agreement trade-off is evident from the empirical results. Although SAGE reaches flatter solutions (tr(H)) than SAM, this does not hold against SAGM. Against SAGM, SAGE is flatter in the last block, SAGM is flatter in the middle block, and the first block is mixed. However, given the higher gradient agreement of SAGE, this further validates our claims that there is a trade-off between flatness and gradient agreement.
  • Looking at the results of SAGE_NoNoise, SAGE has the lower sharpest-direction curvature between the two in 8 of 12 matrices, while no-noise has slightly less total top-80 curvature in early and middle layers but also lower gradient agreement.

Regarding the negative values in Fig 5., they arise as (i) away from a minimum the Hessian is indefinite, so its trace can be negative, and (ii) stochastic trace estimators, such as Hutchinson's method with few probe vectors have variance. For flatness at the final checkpoints, Tables 3-4 of the general response report Lanczos-based top-80 eigenvalues.

Question 4 - SAGE and MTL baselines

We thank the reviewer for this comment. SAMO [2] shows empirically that sharpness-aware perturbations reduce task conflicts, as a by-product of seeking flat regions. In addition, F-MTL [3] uses per-task gradients to find joint low-loss and flat regions across the tasks. Our analysis shows that this does not generally hold. For instance, Counterexample 1 constructs flat but misaligned solutions, where tr(Hˉ)tr(\bar{H}) is arbitrarily small while the alignment term is arbitrarily large, so flatness does not necessarily imply low gradient conflict. SAGE’s noise therefore addresses the gradient disagreement, rather than relying on flatness to resolve it. Furthermore, the noise mechanism is implemented as an implicit search for areas on the loss surface where gradients across distributions naturally align rather than directly acting on the gradients. Regarding the results when combining SAMO with SAGE, we believe the extra perturbations indeed conflict with the initital training objective of SAMO.

AuthorsOpenReview ↗

Response to reviewer Nc9m (2/2)

Question 5 - Connection between the theoretical quantities and the experimental measurements

When we derived the main theoretical result of our paper, we had the choice between developing a method that tries to approximate the bound terms as best as possible and developing a method that builds upon previously proposed tools to provide empirical validity of our findings. The first route requires the development of second-order optimization methods and/or their approximation, which introduces several computational challenges, for modern models above a certain size (e.g., ViTs). Notice that for the ViT-16/B and are both matrices of size 86Mx86M! We therefore use gradient similarity scaled noise as an escape mechanism from points on the loss surface where gradients disagree between distributions. Regarding the connection and the effects of spectral perturbation to flatness and the Hessian, we believe the analysis and extra results mentioned in our above response provide further insight.

Question 6 - Standard Deviations on DomainBed

We thank the reviewer for this observation. Regarding error bars and confidence intervals, we followed the established evaluation and presentation protocol in previous works for both the DG [1] and MTL [2],[3] experiments, which do not report standard deviations in their experimental results. Nonetheless, we agree with the reviewer’s concern and we present the standard deviations for the DomainBed benchmark between our method and the ERM baseline here, where we have left out DomainNet due to computational restrictions. Additionally, we would like to further clarify that we believe that the improvements over the BOA baseline are substantial, as the BOA regularizer is implemented atop a SAM-pretrained backbone [1].

MethodPACSVLCSOfficeHomeTerraIncognitaAvg
ERM95.9 ±\pm 0.481.9 ±\pm 0.684.1 ±\pm 0.456.1 ±\pm 0.479.5 ±\pm 0.4
SAGE97.4 ±\pm 0.385.5 ±\pm 0.487.1 ±\pm 0.561.1$\pm$ 0.582.8 ±\pm 0.4

Question 7 - Regarding the Computational Analysis

We thank the reviewer for this careful comment regarding the computational analysis. Further profiling confirmed their suspicion and revealed an issue in our results.

Specifically, with our data-loading configuration, loading a mini-batch takes around 200 ms while an ERM step needs only 71 ms of GPU compute, so ERM is data-bound (30% GPU utilization), which masks the extra compute of SAM and SAGE in wall-clock time. In addition, the ERM entry in Table 3 was measured with a training loop that also computed per-domain gradients for diagnostic logging, which we omitted to turn off during the initial measurements and inflated its cost (114 min vs. 84 min without it).

We also ran per-iteration profiling (RTX 6000 Ada, OfficeHome, ViT-B/16, CUDA-synchronized timers, batches preloaded on the GPU, 95-100% utilization):

b=12/domainb=32/domain
ERM71 ms (1.0×)184 ms (1.0×)
SAM153 ms (2.2×)428 ms (2.3×)
SAGE264 ms (3.7×)677 ms (3.7×)

Regarding end-to-end experiments with the real data pipeline, a full 4-domain OfficeHome experiment takes 84 / 100 / 120 min for ERM / SAM / SAGE, respectively. In the revised manuscript, we will correct Table 3 accordingly, and add all the above information.

References

[1] Zhuang, Liansheng, et al. "Exploring mode connectivity in krylov subspace for domain generalization." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Ban, Hao, Gokul Ram Subramani, and Kaiyi Ji. "Samo: A lightweight sharpness-aware approach for multi-task optimization with joint global-local perturbation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025.

[3] Phan, Hoang, et al. "Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspective." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025.

[4] Li, Jindong, et al. "Clip-powered domain generalization and domain adaptation: A comprehensive survey." IEEE Transactions on Pattern Analysis and Machine Intelligence (2026).

[5] Wang, Zhenbin, et al. "Rethinking Out-of-Distribution Detection and Generalization with Collective Behavior Dynamics." Advances in Neural Information Processing Systems 38 (2026): 88401-88450.

[6] Lanczos, Cornelius. "An iteration method for the solution of the eigenvalue problem of linear differential and integral operators." Journal of research of the National Bureau of Standards 45.4 (1950): 255-282.

Reviewer_vgm8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the complementary roles of loss landscape flatness and gradient alignment in multi-distribution learning. The authors derive an excess-risk decomposition that identifies two leading terms associated with cross-distribution gradient disagreement and loss curvature. A quadratic counterexample further demonstrates that neither property alone is sufficient to control the other. Motivated by these findings, the authors propose SAGE, which integrates two mechanisms: a spectral-aware perturbation that replaces SAM's gradient-based perturbation with an orthogonalized gradient using Newton–Schulz iterations, and a gradient-alignment-driven noise injection strategy that encourages exploration away from regions of gradient conflict. Extensive experiments validate the effectiveness of the proposed method.

Critical strengths
  • The spectral-aware perturbation is well-motivated and intuitively designed. The authors identify that traditional SAM's perturbation can be dominated by large gradient components, limiting exploration across spectral directions. To address this, they employ Newton–Schulz iterations to orthogonalize the gradient matrix and balance its singular values, providing a reasonable and well-justified design.

  • The experimental evaluation is comprehensive, covering two important settings, domain generalization and multi-task learning, across widely used benchmarks such as DomainNet, OfficeHome, NYU-v2, and Cityscapes. The paper also includes extensive comparisons with representative baselines, particularly in domain generalization. The results demonstrate the effectiveness of SAGE, achieving state-of-the-art average performance.

Critical weaknesses
  • The motivation behind the gradient-alignment-driven noise injection requires further justification. Since the injected Gaussian noise has zero mean, it does not change the expected gradient, making it unclear why this mechanism should effectively improve gradient alignment across distributions. Moreover, the additional noise may interfere with optimization directions and potentially reduce training stability. Further theoretical or empirical evidence is needed to justify this design.

  • The motivation for the spectral-aware perturbation is reasonable, as traditional SAM may be dominated by leading singular directions of the gradient. However, the paper lacks empirical analysis of the gradient singular-value distributions before and after Newton–Schulz orthogonalization. Such an analysis would better substantiate the motivation and demonstrate whether the proposed transformation effectively balances different spectral directions.

Questions for authors
  • The gradient-alignment-driven noise is isotropic and zero-mean, meaning that it does not change the expected gradient direction. Why should this mechanism improve cross-distribution gradient alignment rather than merely introducing additional optimization noise? Could the authors provide further theoretical or empirical justification?
  • The spectral-aware perturbation is motivated by the observation that SAM's perturbation may be dominated by leading singular directions. Could the authors provide empirical evidence of this phenomenon and analyze the singular-value distributions before and after Newton–Schulz orthogonalization?
3 comments
AuthorsOpenReview ↗

Response to reviewer vgm8

We would like to thank the reviewer for their helpful and constructive comments. We will proceed to address the comments and questions in detail.

On the motivation behind zero-mean noise

The reviewer is correct that, for a single step, zero-mean noise leaves the expected update unchanged. However, the noise is injected at every step, and its effect does not cancel over several steps. Specifically, after a noisy step the next gradient is evaluated at a perturbed point, and the expected gradient there differs from the gradient along the noiseless trajectory.

The noise mechanism in SAGE is inspired by GGA. The noise level at each step is scaled by the cross-distribution gradient disagreement at the current point. For non-aligned locations the noise is increased and acts as an implicit search moving the optimization towards regions on the loss surface where alignment is higher.

Regarding further empirical evidence for the spectral concentration of SAM's perturbation

We thank the reviewer for this question and very constructive suggestion. We believe that the results shown in the general response empirically validate our claims and provide further intuition regarding the optimization dynamics of standard and state-of-the-art flatness-aware algorithms such as SAM and SAGM. To address the comment, we ran two complementary analyses. The first looks at the singular-value spectrum of the SAM perturbation before and after Newton-Schulz (NS) orthogonalization on selected attention matrices of the ViT model during training on OfficeHome, while also providing relevant analyses for SAGM. The second analysis relates the aforementioned perturbations to the curvature of the loss through the Hessian. Interestingly enough, both analyses confirm the observation that motivates SAGE which is that SAM’s and SAGM’s perturbation of each attention matrix is concentrated on a few leading directions, which late in training coincide almost exactly with the sharpest Hessian direction. On the other hand, NS orthogonalization spreads the perturbation across many directions.

In more detail, we analyse the attention weights of an early, a middle and a late transformer block (blocks 0, 5, 11). For the singular-value distribution before and after Newton-Schulz, we compute the singular values for each matrix for the SAM, SAGM and SAGE methods and report the following metrics in Tables 1 & 2 of the general response to all reviewers:

  • top-kk energy: ∑i≤kσi2/∑iσi2\sum_{i \le k} \sigma_i^2 / \sum_i \sigma_i^2, the fraction of the perturbation carried by its kk leading singular directions
  • Stable rank: ∑iσi2σmax2\frac{\sum_{i} \sigma_i^2}{\sigma^2_{max}}
  • Effective rank, the exponential of the entropy of the normalized singular values.

As apparent from the results, the majority of the perturbations in SAM and SAGM focus on the top singular directions, while the NS orthogonalization in SAGE flattens the spectrum. Note that 5-step NS is an approximate orthogonalization which explains why the effective rank remains below full rank.

For the Singular value analysis of the Hessian we compute Hessian-vector products by running Lanczos’ algorithm [3] with full re-orthogonalization (160 iterations), obtaining the top-80 eigenpairs (λi,ui)(λ_{i},{u}_{i}). Then, for a unit-norm perturbation direction ee we report the following in Tables 3 & 4 of the general response:

  • Sharpness λmax⁡=λ1\lambda_{\max} = \lambda_1 of the trained weights.
  • Top-kk alignment ∑i≤k(ui⊤e)2\sum_{i \le k} (u_i^\top e)^2: the fraction of the perturbation lying in the subspace of the kk sharpest directions.
  • Relative curvature e⊤He/λmax⁡e^\top H e / \lambda_{\max}: the curvature along the perturbation, as a fraction of the maximum curvature (1 means perturbing along the sharpest direction).

Observations:

  • Both SAM and SAGM concentrate their perturbation on the top Hessian eigenvector, notably for SAGM it's as high as 99%. On the other hand, SAGE and SAGE_NoNoise spread the perturbation out in more directions.
  • The flatness-agreement trade-off is evident from the empirical results. Although SAGE reaches flatter solutions (tr(H)) than SAM, this does not hold against SAGM. Against SAGM, SAGE is flatter in the last block, SAGM is flatter in the middle block, and the first block is mixed. However, given the higher gradient agreement of SAGE, this further validates our claims that there is a trade-off between flatness and gradient agreement.
  • Looking at the results of SAGE_NoNoise, SAGE has the lower sharpest-direction curvature between the two in 8 of 12 matrices, while no-noise has slightly less total top-80 curvature in early and middle layers but also lower gradient agreement.

We will add a summary of both analyses in the Section 5.2 - Main results of the revised manuscript, while more details regarding the computations, and the corresponding tables and figures will be added to the revised Appendix F - ADDITIONAL ABLATIONS.

AuthorsOpenReview ↗

Response to reviewer vgm8 - References

References:

[1] Rudelson, Mark, and Roman Vershynin. "Sampling from large matrices: An approach through geometric functional analysis." Journal of the ACM (JACM) 54.4 (2007): 21-es.

[2] Roy, Olivier, and Martin Vetterli. "The effective rank: A measure of effective dimensionality." 2007 15th European signal processing conference. IEEE, 2007.&

[3] Lanczos, Cornelius. "An iteration method for the solution of the eigenvalue problem of linear differential and integral operators." Journal of research of the National Bureau of Standards 45.4 (1950): 255-282.

I appreciate the authors' thorough and constructive rebuttal. Their clarifications have satisfactorily resolved most of my concerns. Therefore, I will retain my original positive assessment.

Reviewer_22uPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the interplay between loss landscape flatness and cross-distribution gradient alignment in multi-distribution learning, including domain generalization (DG) and multi-task learning (MTL). The authors derive an excess-risk decomposition consisting of two terms associated with gradient covariance and Hessian curvature, respectively, and provide a counterexample demonstrating that neither term can generally bound the other. Motivated by this analysis, the authors propose Spectral-Aware Gradient-Aligned Exploration (SAGE), which combines two optimization mechanisms. First, a spectral perturbation strategy employs Newton–Schulz iterations to approximate the orthogonal polar factor of gradient matrices, enabling more balanced exploration across singular directions. Second, an adaptive Gaussian noise mechanism adjusts the descent update according to cross-distribution gradient disagreement. Experiments on five DG benchmarks and two MTL benchmarks demonstrate competitive performance against existing sharpness-aware and gradient-alignment methods, including improved average accuracy on DomainBed and performance gains when integrated with several MTL optimizers.

Critical strengths

1. Competitive experimental performance. SAGE achieves strong results across five domain generalization benchmarks, reaching an average accuracy of 78.9% on DomainBed and outperforming the reported baselines.

2. Broad applicability. The framework can be integrated with several existing multi-task learning optimizers without modifying the network architecture, demonstrating practical flexibility.

Critical weaknesses

1. Limited methodological novelty and insufficient theoretical justification. The proposed method primarily combines existing spectral orthogonalization and gradient-guided noise techniques. The theoretical analysis combines two separately derived risk terms without establishing a clear mechanism connecting them or directly motivating the algorithm. Furthermore, although SAGE constructs spectral perturbation directions from aggregated gradients, its gradient-alignment component only adjusts the magnitude of isotropic noise based on gradient disagreement, without explicitly exploiting the directions of cross-distribution gradient conflicts. Related work, such as Reframing Long-Tailed Learning via Loss Landscape Geometry (CVPR 2026, https://arxiv.org/abs/2603.21217), has explored gradient decomposition to construct group-specific perturbation directions and mitigate dominant-gradient bias. Although the learning settings differ, the authors should discuss this directional perspective and clarify how SAGE differs from or could benefit from such approaches.

2. Insufficient evidence for the necessity of combining both components. Removing gradient-guided noise reduces accuracy by only 0.3 percentage points on OfficeHome and 0.1 percentage points on TerraIncognita, suggesting that most improvements may originate from spectral perturbation alone. The paper does not establish whether the gains result from a genuine interaction between the two components or simply from combining individually effective techniques. Additional comparisons with straightforward combinations (e.g., SAM+GGA), constant-noise controls, and complete factorial ablations are needed to substantiate the claimed benefits of jointly optimizing flatness and gradient alignment.

Questions for authors
  1. Theoretical novelty: What new insight does the excess-risk decomposition establish beyond combining two separately derived risk terms, and how does it directly motivate SAGE?

  2. Perturbation direction: Why does SAGE only adapt the magnitude of gradient-guided noise rather than exploit cross-distribution conflict directions? How does it differ from the directional gradient decomposition explored in Reframing Long-Tailed Learning via Loss Landscape Geometry (CVPR 2026, https://arxiv.org/abs/2603.21217)?

  3. Joint effectiveness: Given the marginal performance drop when removing gradient-guided noise, what evidence demonstrates that SAGE benefits from jointly optimizing both components rather than primarily from spectral perturbation? Could the authors compare against SAM+GGA and constant-noise baselines?

4 comments
AuthorsOpenReview ↗

Response to reviewer 22uP (1/3)

We thank the reviewer for their insightful and helpful comments. Here we provide an overall response before answering the questions in detail and point by point.

The trade-off that arises in the risk decomposition regarding loss surface flatness and cross-distribution gradient-alignment is novel, i.e., has not been introduced in the literature for multi-distribution representation learning. Furthermore, the components of SAGE are derived from these theoretical results. Please also see the detailed response below.

Regarding the work of Chen et al. [19], we thank the reviewer for informing us. While we believe the method is closer to the works of GSAM [17] and SAGM [18], it is relevant and we will discuss it in Sec. 4 and App. A.3 of the revised manuscript.

Finally, below we provide further ablations regarding the noise component of SAGE inspired by GGA, SAM and constant-noise (SGLD) combinations, where we show that gradient-guided noise as a proxy for gradient alignment helps generalization, which is consistent with the conclusions of the literature [15].

On the Theoretical novelty point - Response to Question 1

This paper provides a theoretical result showing that both flatness and cross-distribution gradient-alignment is necessary for out-of-distribution generalization of machine learning models, and shows a trade-off between the two quantities. This is a new result. These two areas have been a strong focus of the ML communities for the past 5+ years [1]-[7] with methods such as [8]-[19]. We believe that this is a non-incremental theoretical contribution, as this decomposition and trade-off of the two terms was not previously established, and they have been pursued as separate objectives.

More importantly, a direct conclusion of the theoretical analysis is that Flatness alone cannot predict cross-environment generalization, as the gradient covariance across environments enters in the first term. Additionally, another rather interesting result is that the alignment term shrinks with the number of environments K and not with the amount of data.

The above result is a formal result confirming the empirical findings that have motivated numerous papers in the DG literature (e.g., data manipulation methods in [6]), i.e. that samples from more and diverse domains assist towards learning domain-invariant machine learning models that generalize better.

Furthermore, the proposed SAGE method includes novel elements in itself, such as the proposed perturbation in the ascent step of flatness-based methods. Regarding Newton-Schulz orthogonalization, although it has been implemented for optimization (MUON), in our case it is used as part of a method for assessing flatness, improving upon SAM-based methods (SAM, GSAM, SAGM). While SAGE cannot directly implement the second-order terms of Thm 1. (for computational reasons), it is motivated by them. Additional experimental results confirm our intuition (see Table 1-5 in the general response), as it is shown that the majority of SAM-like (SAM and SAGM) perturbation directions gather in a small subset of the top eigendirections values of the Hessian.

On the Connection to the work of Chen et al. - Response to Question 2

We thank the reviewer for pointing us to this work [19], we will discuss it in Sec. 4 and App. A.3 of the revised version of our manuscript.

The work of [19] addresses long-tailed image recognition within a single training distribution and proposes a Grouped Knowledge Preservation (GKP) module that partitions classes into groups based on the optimal encoder parameters for each class. Motivated by the assumption that SAM perturbations are dominated by the head classes gradient directions, the authors propose an additional Grouped Sharpness-Aware (GSA) module, which computes the gradients for each partitioned group and removes its projection onto the global SAM gradient similarly to GSAM [17] and SAGM [18], leading to group-specific gradients. The authors then use long-tail specific perturbations for improving the flatness of tail classes. On the other hand, SAGE targets a different question which is the role of loss-surface flatness and gradient alignment between samples from randomly drawn distributions in multi-distribution settings and uses Newton-Schulz orthogonalization to probe additional directions during the ascent step of the SAM perturbation. The noise mechanism in SAGE is employed as an implicit escape mechanism from points on the loss surface where gradients disagree, in search of regions where gradients align naturally without the need of gradient operations.

AuthorsOpenReview ↗

Response to reviewer 22uP (2/3)

On the Connection between the terms in Thm 1 and SAGE

To provide further empirical evidence linking the terms in Thm 1 and SAGE, we analysed the attention matrices ($W_Q, W_K, W_V, W_O$) of an early, middle and a late attention block (blocks 0, 5 and 11) of the ViT model trained on OfficeHome, for the SAM, SAGM and SAGE methods. Specifically, during training we computed the singular values of each perturbation and measured how concentrated it is with the following metrics:

  • top-kk energy: ∑i≤kσi2/∑iσi2\sum_{i \le k} \sigma_i^2 / \sum_i \sigma_i^2
  • Stable rank: ∑iσi2σmax2\frac{\sum_{i} \sigma_i^2}{\sigma^2_{max}}
  • Effective rank: the exponential of the entropy of the normalized singular values.

The results of this analysis are presented in Tables 1 and 2 of the general response to all reviewers. As shown, SAM's and SAGM's perturbations are dominated by a few leading directions. Specifically, the top singular direction carries 24-96% of the energy, the top 10 (out of 768, since these are 768x768 matrices) carry 76-99.8%, and the stable rank is only 1.0-4.3. After NS orthogonalization (SAGE), the top direction carries 0.6-9.8% of the energy, while the stable rank and the effective rank both increase. SAGE therefore perturbs each matrix along many directions rather than a handful, which was expected as this is the effect of Newton-Schulz.

To further show the effect of the above algorithms on the Hessian, for each of the above matrices we computed the top-80 eigenpairs (λi,ui)(λ_{i},u_i) of each Hessian using Lanczos iteration [22] with Hessian-vector products, at the end of training, along with the following metrics and present them in Tables 3 and 4 in the general response:

  • λ_max: top eigenvalue
  • tr(H): sum of top-80 eigenvalues
  • align1: (u1Te)2({u}_{1}^{T}e{)}^{2}, Share of the perturbation that lies along the single sharpest direction
  • align10: ∑i=110(ui⊤e)2\sum_{i=1}^{10} (u_i^\top e)^2, Share that lies inside the 10 sharpest directions
  • curv: (eᵀHe) / λ, Curvature along the perturbation, relative to the sharpest direction. 1 = perturbing along the sharpest direction

From the results we observe the following:

  • Both SAM and SAGM concentrate their perturbation on the top Hessian eigenvector, notably for SAGM it's as high as 99%. On the other hand, SAGE and SAGE_NoNoise spread the perturbation out in more directions.
  • The flatness-agreement trade-off is evident from the empirical results. Although SAGE reaches flatter solutions (tr(H)) than SAM, this does not hold against SAGM. Against SAGM, SAGE is flatter in the last block, SAGM is flatter in the middle block, and the first block is mixed. However, given the higher gradient agreement of SAGE, this further validates our claims that there is a trade-off between flatness and gradient agreement.
  • Looking at the results of SAGE_NoNoise, SAGE has the lower sharpest-direction curvature between the two in 8 of 12 matrices, while no-noise has slightly less total top-80 curvature in early and middle layers but also lower gradient agreement.&

On the joint effectiveness of SAGE's components - question 3

On the marginal improvements from noise: Looking at Table 1 (experiments on DomainBed), it is clear that several baseline methods differ even by less than 0.5% on average for this challenging problem. Therefore, we believe the gains provided by each component in the ablation matrix (Fig.3) are noteworthy. This observation is also supported by the relevant and latest DG literature across datasets and architectures (e.g., [20] [21]). However, as we agree that additional experiments would provide further insight and as per the reviewer's suggestion, we are expanding the ablation experiments on the OfficeHome dataset, and provide extra results for SAM+GGA and a constant-noise baseline, which is effectively Stochastic Gradient Langevin Dynamics. The additional results on OfficeHome in the following table indicate that the addition of gradient-guided noise does indeed improve training of other baseline methods, such as SAM, and is consistent with the conclusions of relevant literature [15], whereas constant noise (SGLD) does not.

MethodAccuracy
SAM85.4
SAM + GGA86.7
SAM + SGLD85.4
SAGE + SGLD86.0
SAGE87.1

Additional complete factorial ablations are computationally difficult to complete within the rebuttal timeframe but we will do our best to provide further results.

AuthorsOpenReview ↗

Response to reviewer 22uP (3/3)

References

[1] Li, Hao, et al. "Visualizing the Loss Landscape of Neural Nets." Advances in Neural Information Processing Systems 31, (2018).

[2] Fort, Stanislav, and Stanislaw Jastrzebski. "Large scale structure of neural network loss landscapes." Advances in Neural Information Processing Systems 32 (2019).

[3] Rangamani, Akshay. Loss landscapes and generalization in neural networks: Theory and applications. Diss. Johns Hopkins University, 2020.

[4] Jiang, Yiding, et al. "Fantastic generalization measures and where to find them." arXiv preprint arXiv:1912.02178 (2019).

[5] Wang, Jindong, et al. "Generalizing to unseen domains: A survey on domain generalization." IEEE transactions on knowledge and data engineering 35.8 (2022): 8052-8072.

[6] Zhou, Kaiyang, et al. "Domain generalization: A survey." IEEE transactions on pattern analysis and machine intelligence 45.4 (2022): 4396-4415.

[7] Zhang, Yu, and Qiang Yang. "A survey on multi-task learning." IEEE transactions on knowledge and data engineering 34.12 (2021): 5586-5609.

[8] Foret, Pierre, et al. "Sharpness-aware minimization for efficiently improving generalization." International Conference on Learning Representations (2021).

[9] Kwon, Jungmin, et al. "Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks." International conference on machine learning. PMLR, 2021.

[10] Kim, Minyoung, et al. "Fisher sam: Information geometry and sharpness aware minimisation." 39th International Conference on Machine Learning. PMLR, 2022.

[11] Zhang, Ruipeng, et al. "Domain-inspired sharpness-aware minimization under domain shifts." International Conference on Learning Representations (2024).

[12] Ban, Hao, Gokul Ram Subramani, and Kaiyi Ji. "Samo: A lightweight sharpness-aware approach for multi-task optimization with joint global-local perturbation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025.

[13] Mansilla, Lucas, et al. "Domain generalization via gradient surgery." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

[14] Le, Binh M., and Simon S. Woo. "Gradient alignment for cross-domain face anti-spoofing." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[15] Ballas, Aristotelis, and Christos Diou. "Gradient-guided annealing for domain generalization." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[16] Yu, Tianhe, et al. "Gradient surgery for multi-task learning." Advances in neural information processing systems 33 (2020): 5824-5836.

[17] Zhuang, Juntang, et al. "Surrogate gap minimization improves sharpness-aware training." ICLR 2022.

[18] Wang, Pengfei, et al. "Sharpness-aware gradient matching for domain generalization." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.

[19] Chen, Shenghan, et al. "Reframing long-tailed learning via loss landscape geometry." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2026).

[20] Li, Jindong, et al. "Clip-powered domain generalization and domain adaptation: A comprehensive survey." IEEE Transactions on Pattern Analysis and Machine Intelligence (2026).

[21] Wang, Zhenbin, et al. "Rethinking Out-of-Distribution Detection and Generalization with Collective Behavior Dynamics." Advances in Neural Information Processing Systems 38 (2026): 88401-88450.

[22] Lanczos, Cornelius. "An iteration method for the solution of the eigenvalue problem of linear differential and integral operators." Journal of research of the National Bureau of Standards 45.4 (1950): 255-282.

The authors have satisfactorily addressed the majority of my concerns in the rebuttal. Accordingly, I am willing to increase my score.

Comments on the paper

5 comments
AuthorsOpenReview ↗

General Response to all reviewers (1/4)

We would like to sincerely thank the reviewers for their insightful comments and suggestions. We appreciate that the reviewers found the theoretical perspective interesting and a potential basis for further research (Nc9m), the spectral-aware perturbation well-motivated (vgm8), and the empirical results strong, with SAGE achieving the best average accuracy on DomainBed (22uP, vgm8, Nc9m).

The reviewers' main shared concerns were: (i) whether both SAGE components are needed, and whether the gains come from jointly addressing flatness and alignment (ii) empirical evidence for the spectral concentration of SAM-type perturbations and (iii) the motivation for zero-mean, alignment-scaled noise and its link to the theory

To avoid repeating answers and experimental results, we will provide the full tables for the spectral analyses here, and we will refer to them in the individual responses. All results will also be added to the revised manuscript. We have done our best to address all of the reviewer comments and we believe that our additional results clarify several points raised by the reviewers and overall improve the quality of the manuscript.

1. Further ablation of SAGE's components (OfficeHome, ViT-B/16)

We crossed the ascent step (SAM gradient vs. Newton–Schulz spectral perturbation) with the noise type (none / constant noise (SGLD [1]) / alignment-scaled noise):

Ascent \ NoiseNoneConstant (SGLD)Alignment-scaled
SAM85.485.486.7
Spectral (NS)86.886.087.1 (SAGE)

Both components improve accuracy individually, and combining them gives the best result. We observe that constant noise (Stochastic Gradient Langevin Dynamics) does not help generalization as it leaves SAM unchanged and lowers the spectral variant. Therefore, gradient-guided noise addition benefits training.

2. SAM-type perturbations are spectrally concentrated, while SAGE's Newton-Schulz orthogonalization spreads them

We analysed the perturbations on the attention matrices (WQ,WK,WV,WOW_Q, W_K, W_V, W_O) of an early, a middle and a late ViT block (blocks 0, 5, 11) on OfficeHome. Specifically, for the perturbation analysis during training we computed the singular values of each perturbation for SAM, SAGM, Sage without the noise component and SAGE, and measured how concentrated it is with the following metrics:

  • top-kk energy: ∑i≤kσi2 / ∑iσi2\sum_{i \le k} \sigma_i^2 \,/\, \sum_i \sigma_i^2
  • Stable rank [2] : ∑iσi2σmax2\frac{\sum_{i} \sigma_i^2}{\sigma^2_{max}}
  • Effective rank [3]: the exponential of the entropy of the normalized singular values

From the additional experiments, the following observations are of note:

  • SAM's and SAGM's perturbations are dominated by a few leading directions. Specifically, the top singular direction carries 24-96% of the energy, the top 10 (out of 768, since these are 768x768 matrices) carry 76-99.8%, and the stable rank is only 1.0-4.3.
  • After NS orthogonalization (SAGE), the top direction carries 0.6-9.8% of the energy, while the stable rank and the effective rank both increase. SAGE therefore perturbs each matrix along many directions rather than a handful, which was expected as this is the effect of Newton-Schulz.

(Response continued below)

AuthorsOpenReview ↗

General Response to all reviewers (2/4)

The results are presented in the following tables:

Table 1. Singular value analysis (top energy) of perturbations in SAM, SAGM, SAGE without noise and SAGE.

matrixtop1_energytop10_energy
SAMSAGMSAGE_NoNoiseSAGESAMSAGMSAGE_NoNoiseSAGE
Block 0 W_Q0.43590.44630.042210.040390.87570.88820.34450.3323
Block 0 W_K0.68410.73370.054630.051470.95570.96610.37420.3631
Block 0 W_V0.53910.56480.0092320.0089870.93890.95170.089530.08717
Block 0 W_O0.30680.39920.0059890.0059470.77640.81070.059690.05925
Block 5 W_Q0.23570.24740.019890.020090.78620.78810.17560.1763
Block 5 W_K0.24380.24730.012920.013050.76460.76390.12120.1216
Block 5 W_V0.33660.3390.0062040.0062310.83730.83720.059980.06055
Block 5 W_O0.34930.35610.0056340.005660.83380.83550.056160.05639
Block 11 W_Q0.95820.95780.09860.097780.99810.9980.70720.7106
Block 11 W_K0.58560.57880.042240.042260.9860.98520.36470.3622
Block 11 W_V0.40120.39950.022710.022630.94610.94610.20540.205
Block 11 W_O0.35080.35680.062940.06160.96340.96360.51310.5158

Table 2. Singular value analysis (stable and effective rank) of perturbations in SAM, SAGM, SAGE without noise and SAGE.

matrixstable_rankeffective_rank
SAMSAGMSAGE_NoNoiseSAGESAMSAGMSAGE_NoNoiseSAGE
Block 0 W_Q2.5552.47924.3825.2973.770.28180.8182.7
Block 0 W_K1.5351.40119.119.9249.7143.8182.1183.8
Block 0 W_V1.9231.816110.9113.258.5452.25269.4272.5
Block 0 W_O3.6622.777171.5173.1107.5101.6389.8391.8
Block 5 W_Q4.3024.10954.7854.35118.7118.1357.5356.2
Block 5 W_K4.2494.19981.9481.44128.5128.1381380.4
Block 5 W_V3.1003.08174174.1104.1103.9482.5481
Block 5 W_O3.1163.043193.4193.5105.3104.6483.2482.5
Block 11 W_Q1.0441.04411.6111.715.295.33363.8164.04
Block 11 W_K1.7711.826.5326.5819.5819.78147.2148.4
Block 11 W_V2.5512.56747.4747.8329.9630.02227.5229.7
Block 11 W_O3.0282.96216.721713.2813.29260.3263.1
AuthorsOpenReview ↗

General Response to all reviewers (3/4)

To further show the effect of the above algorithms on the Hessian, for each of the above matrices we computed the top-80 eigenpairs (λi,ui)(λ_{i},u_i) of each Hessian using Lanczos iteration [4] with Hessian-vector products, at the end of training, along with the mean gradient agreement (cosine similarity of gradient vectors between domains). For the Hessian, we computed the following

  • λmaxλ_{max}: top eigenvalue
  • tr(H): sum of top-80 eigenvalues
  • align1: (u1Te)2({u}_{1}^{T}e{)}^{2}, Share of the perturbation that lies along the single sharpest direction
  • align10: ∑i=110(ui⊤e)2\sum_{i=1}^{10} (u_i^\top e)^2, Share that lies inside the 10 sharpest directions
  • curv: (eTHe)/λ(e^{T} H e) / λ, Curvature along the perturbation, relative to the sharpest direction. 1 = perturbing along the sharpest direction

We observe the following

  • Both SAM and SAGM concentrate their perturbation on the top Hessian eigenvector, notably for SAGM it's as high as 99%. On the other hand, SAGE and SAGE_NoNoise spread the perturbation out in more directions.
  • The flatness-agreement trade-off is evident from the empirical results. Although SAGE reaches flatter solutions (tr(H)) than SAM, this does not hold against SAGM. Against SAGM, SAGE is flatter in the last block, SAGM is flatter in the middle block, and the first block is mixed. However, given the higher gradient agreement of SAGE, this further validates our claims that there is a trade-off between flatness and gradient agreement.
  • Looking at the results of SAGE_NoNoise, SAGE has the lower sharpest-direction curvature between the two in 8 of 12 matrices, while no-noise has slightly less total top-80 curvature in early and middle layers but also lower gradient agreement.

Table 3. Curvature metrics for SAM, SAGM, SAGE without noise and SAGE.

Matrixλ_maxtr(H)- top-80align1
SAMSAGMSAGE_NoNoiseSAGESAMSAGMSAGE_NoNoiseSAGESAMSAGMSAGE_NoNoiseSAGE
Block 0 W_Q5.668981.8570.60730.813627.928712.438.47510.720.01217478.611e-054.953e-060.02862
Block 0 W_K13.91785.7593.3273.34431.649823.2318.8722.190.007210940.0090160.0010590.001198
Block 0 W_V160.97632.2555.5978.63756.955359.6500.5662.10.001486130.49160.0022031.551e-05
Block 0 W_O232.58645.6875.81101.21104.37437.6695934.47.33206e-050.96560.030880.01591
Block 5 W_Q8.258739.7959.998.786106.16662.3659.5675.150.0842320.9490.041510.04689
Block 5 W_K18.900415.716.1614.2161.47995.5997.711230.3289080.96560.067710.08068
Block 5 W_V157.548226.9185.3168.41352.4993.2727.4819.10.06828510.9840.037180.04173
Block 5 W_O119.102184.6161.2143.41023.4738.6577.9630.20.06873960.98710.035390.03961
Block 11 W_Q46.179837.0319.3614.43419.016165.8170.6166.40.2367840.70580.010420.00544
Block 11 W_K69.439353.8525.7315.72508.956223.6190.6185.40.1304170.48890.025980.01285
Block 11 W_V117.634102.134.5829.74687.527286.2328328.70.9149960.99450.0008010.003361
Block 11 W_O51.898260.721.5218.71407.767181.8243.6246.20.9090210.99130.0022060.01206

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.