- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors argued that flatness and gradient alignment are both important and should be considered in multi-distribution learning. They derived an excess-risk decomposition that yields two additive leading-order terms, one is an alignment term, and the other is a curvature term. The authors also demonstrated a counterexample that neither flatness nor gradient alignment can bound the other, so focusing only one term cannot guarantee low excess risk. The authors then proposed Spectral-Aware Gradient-Aligned Exploration (SAGE) to target both terms. Based on the experimental results, they argued that the proposed SAGE establishes a new state-of-the-art on DomainBed and acts as a general-purpose improvement to base MTL solvers.
Critical strengths
- An interesting theoretical perspective on multi-distribution learning. I think the theoretical analysis is the most interesting part of this paper. The excess-risk decomposition provides a useful perspective on the relationship between flatness and gradient alignment, and the counterexample helps explain why focusing on only one of them may not always be sufficient. I find this an interesting direction, and I believe it could inspire further research on optimization and generalization in multi-distribution learning.
- Promising results across different learning settings. I also appreciate that the authors evaluate SAGE on both domain generalization and multi-task learning, rather than focusing on a single setting. In particular, the results on DomainBed are quite promising, with SAGE achieving the best average accuracy among the compared methods. Although the improvements are less consistent in the MTL experiments, I think exploring the applicability of the method across these two settings is still valuable.
Critical weaknesses
My main concern is that the paper does not convincingly demonstrate the practical need to jointly optimize flatness and gradient alignment, or show that SAGE effectively addresses the imbalance between these two properties.
The theoretical analysis is interesting, but I am not fully convinced by the empirical evidence. In particular, Figure 5 only compares SAGE with ERM and SAM, without showing whether more advanced flatness-aware or gradient-alignment methods actually suffer from the proposed trade-off. Moreover, SAGE does not consistently achieve better flatness across all domains.
I also find the ablation results somewhat unconvincing. Removing the alignment-driven noise only leads to very small accuracy drops (0.3% and 0.1%), and the paper does not show how this affects flatness or gradient agreement. Since prior work such as SAMO has already shown that sharpness-aware optimization can also mitigate gradient conflicts, I wonder whether SAGE's spectral perturbation alone could already improve both properties, making the additional noise mechanism less necessary. The fact that adding SAGE to SAMO often worsens the results in Table 2 raises further questions about this.
Overall, while the generalization results are promising, I don't think the paper provides enough evidence that its improvements actually come from jointly addressing the two properties identified in the theoretical analysis. This is a major concern for me because that connection is central to the paper's motivation and proposed contribution.
Questions for authors
- Can the authors provide more direct evidence that existing methods suffer from the proposed flatness–gradient alignment imbalance?
The theoretical analysis shows that optimizing one property does not necessarily guarantee the other, which I find interesting. However, I'm wondering how often this actually happens with existing methods. For example, do sharpness-aware methods such as SAGM or DISAM achieve better flatness but worse gradient alignment? Do alignment-based methods such as GGA show the opposite behavior? It would be helpful to see both properties measured for some of the stronger baselines in Table 1, rather than only comparing their generalization accuracy.
- Could the authors clarify the results in Figure 5, particularly the Hessian trace measurements?
Figure 5 provides some useful analysis, but I have a few questions about it. First, why does the figure only show 10 epochs? Is this the entire DG training process, or are only the first 10 epochs shown? Appendix E does not seem to specify the total number of training epochs for DG. Also, the Hessian trace results seem somewhat unstable. For example, SAGE has a higher trace than SAM in the Product domain, while the Art domain shows large fluctuations. Some trace estimates are even negative. How should these results be interpreted in terms of flatness? Would it be possible to report additional flatness measurements or results at the final selected checkpoints?
- How much does the alignment-driven noise actually contribute to improving gradient alignment?
In Figure 3, removing the noise only leads to small accuracy drops (0.3% on OfficeHome and 0.1% on TerraIncognita). I'm curious whether the spectral perturbation alone already improves both flatness and gradient agreement. Could the authors provide the Hessian trace and gradient agreement results for SAGE without noise? It would also be interesting to compare spectral perturbation only, alignment-driven noise only, and their combination. This would help clarify whether the two components are really complementary, or whether most of the improvement comes from spectral perturbation.
- How does SAGE relate to the observations in SAMO and F-MTL, and why does combining SAGE with SAMO sometimes hurt performance?
Appendix A.2 mentions that SAMO has shown that sharpness-aware optimization can already mitigate task-gradient conflicts. F-MTL also considers task-specific sharpness-aware optimization and gradient coordination. Given these findings, I'm wondering what additional problem SAGE's alignment-driven noise is intended to solve. Related to this, Table 2 shows that adding SAGE to SAMO-based methods often makes the overall results worse. For example, on Cityscapes, the \Delta_m of SAMO-FairGrad changes from −0.62 to 4.88 after adding SAGE. Do the authors have any explanation for this behavior? Could the additional perturbation or noise interfere with the optimization when the base method already addresses sharpness and task conflicts?
- Could the authors further clarify the connection between the theoretical quantities and the experimental measurements?
Theorem 1 identifies tr(\bar H^{-1}\Sigma_g) as the alignment-related term, but the experiments mainly use pairwise gradient cosine similarity. These two quantities are not equivalent, since cosine similarity does not capture gradient magnitude differences or how gradient covariance interacts with Hessian curvature. Similarly, the spectral perturbation is constructed from the gradient matrix rather than directly from the Hessian. I understand that the proposed components are practical approximations, but is there any empirical evidence showing that these proxies are related to the theoretical quantities? Even an approximate measurement on a smaller model or dataset would help connect the theoretical analysis to the actual behavior of SAGE.
- Why are standard deviations not reported for the DomainBed results?
Table 1 states that the results are averaged over three runs, but I couldn't find the corresponding standard deviations. Given that some of the improvements are relatively small (e.g., SAGE achieves 78.9% compared to BOA's 78.1%), I think reporting the standard deviations would be important for understanding how stable the results are. Could the authors provide the standard deviations across the three runs, ideally for each dataset as well as the overall average?
- Could the authors explain the surprisingly small training-time differences between ERM, SAM, and SAGE?
In Table 3, the reported training times are 114 minutes for ERM, 119 minutes for SAM, and 120 minutes for SAGE. I find these numbers a bit surprising, especially since SAM requires an additional forward-backward pass compared to ERM, while SAGE further introduces Newton–Schulz iterations and per-distribution gradient computations. I understand that the additional operations in SAGE may be relatively inexpensive, but why does even SAM take only about 4% longer than ERM?
I'm wondering whether the training might be bottlenecked by data loading, CPU operations, or other overhead rather than GPU computation. Could the authors provide some GPU utilization or per-iteration profiling results? This would help assess whether the reported computational efficiency would still hold under better GPU utilization.
2 comments
Response to reviewer Nc9m (2/2)
Question 5 - Connection between the theoretical quantities and the experimental measurements
When we derived the main theoretical result of our paper, we had the choice between developing a method that tries to approximate the bound terms as best as possible and developing a method that builds upon previously proposed tools to provide empirical validity of our findings. The first route requires the development of second-order optimization methods and/or their approximation, which introduces several computational challenges, for modern models above a certain size (e.g., ViTs). Notice that for the ViT-16/B and are both matrices of size 86Mx86M! We therefore use gradient similarity scaled noise as an escape mechanism from points on the loss surface where gradients disagree between distributions. Regarding the connection and the effects of spectral perturbation to flatness and the Hessian, we believe the analysis and extra results mentioned in our above response provide further insight.
Question 6 - Standard Deviations on DomainBed
We thank the reviewer for this observation. Regarding error bars and confidence intervals, we followed the established evaluation and presentation protocol in previous works for both the DG [1] and MTL [2],[3] experiments, which do not report standard deviations in their experimental results. Nonetheless, we agree with the reviewer’s concern and we present the standard deviations for the DomainBed benchmark between our method and the ERM baseline here, where we have left out DomainNet due to computational restrictions. Additionally, we would like to further clarify that we believe that the improvements over the BOA baseline are substantial, as the BOA regularizer is implemented atop a SAM-pretrained backbone [1].
| Method | PACS | VLCS | OfficeHome | TerraIncognita | Avg |
|---|---|---|---|---|---|
| ERM | 95.9 0.4 | 81.9 0.6 | 84.1 0.4 | 56.1 0.4 | 79.5 0.4 |
| SAGE | 97.4 0.3 | 85.5 0.4 | 87.1 0.5 | 61.1$\pm$ 0.5 | 82.8 0.4 |
Question 7 - Regarding the Computational Analysis
We thank the reviewer for this careful comment regarding the computational analysis. Further profiling confirmed their suspicion and revealed an issue in our results.
Specifically, with our data-loading configuration, loading a mini-batch takes around 200 ms while an ERM step needs only 71 ms of GPU compute, so ERM is data-bound (30% GPU utilization), which masks the extra compute of SAM and SAGE in wall-clock time. In addition, the ERM entry in Table 3 was measured with a training loop that also computed per-domain gradients for diagnostic logging, which we omitted to turn off during the initial measurements and inflated its cost (114 min vs. 84 min without it).
We also ran per-iteration profiling (RTX 6000 Ada, OfficeHome, ViT-B/16, CUDA-synchronized timers, batches preloaded on the GPU, 95-100% utilization):
| b=12/domain | b=32/domain | |
|---|---|---|
| ERM | 71 ms (1.0×) | 184 ms (1.0×) |
| SAM | 153 ms (2.2×) | 428 ms (2.3×) |
| SAGE | 264 ms (3.7×) | 677 ms (3.7×) |
Regarding end-to-end experiments with the real data pipeline, a full 4-domain OfficeHome experiment takes 84 / 100 / 120 min for ERM / SAM / SAGE, respectively. In the revised manuscript, we will correct Table 3 accordingly, and add all the above information.
References
[1] Zhuang, Liansheng, et al. "Exploring mode connectivity in krylov subspace for domain generalization." International Conference on Learning Representations. Vol. 2026. 2026.
[2] Ban, Hao, Gokul Ram Subramani, and Kaiyi Ji. "Samo: A lightweight sharpness-aware approach for multi-task optimization with joint global-local perturbation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025.
[3] Phan, Hoang, et al. "Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspective." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025.
[4] Li, Jindong, et al. "Clip-powered domain generalization and domain adaptation: A comprehensive survey." IEEE Transactions on Pattern Analysis and Machine Intelligence (2026).
[5] Wang, Zhenbin, et al. "Rethinking Out-of-Distribution Detection and Generalization with Collective Behavior Dynamics." Advances in Neural Information Processing Systems 38 (2026): 88401-88450.
[6] Lanczos, Cornelius. "An iteration method for the solution of the eigenvalue problem of linear differential and integral operators." Journal of research of the National Bureau of Standards 45.4 (1950): 255-282.
Response to reviewer Nc9m (1/2)
We thank the reviewer for their comments and insightful questions. We will answer and adress them point by point.
Questions 1, 2 & 3
We thank the reviewer for their comments. First of all, regarding the epochs in the DomainBed experiments, yes the total epochs are 10 with a total of 5000 training steps (500 steps per epoch). We apologize for omitting this information, we will include these details in Appendix E of the revised manuscript.
From Figure 5 of the manuscript several observations are of note:
To further address the comment, we provide additional metrics that support our method in Tables 1 - 5 in the general response to all reviewers. Specifically, we select attention matrices from 3 different attention blocks of the final ViT model checkpoint trained on OfficeHome and compute the Hessian-vector products by double backpropagation and run Lanczos iteration with full re-orthogonalization (160 iterations), obtaining the top-80 eigenpairs (λi,ui). We have also added relevant metrics for SAGM, a stronger flatness-aware baseline and for SAGE without the noise component.
For the Hessian analysis, we compute the following metrics:
Observations:
Regarding the negative values in Fig 5., they arise as (i) away from a minimum the Hessian is indefinite, so its trace can be negative, and (ii) stochastic trace estimators, such as Hutchinson's method with few probe vectors have variance. For flatness at the final checkpoints, Tables 3-4 of the general response report Lanczos-based top-80 eigenvalues.
Question 4 - SAGE and MTL baselines
We thank the reviewer for this comment. SAMO [2] shows empirically that sharpness-aware perturbations reduce task conflicts, as a by-product of seeking flat regions. In addition, F-MTL [3] uses per-task gradients to find joint low-loss and flat regions across the tasks. Our analysis shows that this does not generally hold. For instance, Counterexample 1 constructs flat but misaligned solutions, where tr(Hˉ) is arbitrarily small while the alignment term is arbitrarily large, so flatness does not necessarily imply low gradient conflict. SAGE’s noise therefore addresses the gradient disagreement, rather than relying on flatness to resolve it. Furthermore, the noise mechanism is implemented as an implicit search for areas on the loss surface where gradients across distributions naturally align rather than directly acting on the gradients. Regarding the results when combining SAMO with SAGE, we believe the extra perturbations indeed conflict with the initital training objective of SAMO.