acceptodds
Under review as a conference paper at ICLR 2027

GRAIN: Group Aggregation via Min-Norm Objective for Stable Learning

Abstract

Two training runs that differ only in the random seed can produce noticeably different models. The problem is long-standing, but it is most acute in the overparameterized regime of modern deep learning, where large models fine-tuned on limited data traverse flat loss landscapes with many near-equivalent minima, and it is most costly for large pretrained models (LPMs), whose training is expensive, whose downstream data is scarce, and for which repeated runs to average out variance are prohibitive. We trace a common source of this instability to the update rule itself: when the gradients of different groups of examples conflict in direction, their arithmetic mean can be small or even zero, stalling training while the loss is still high. We introduce GRAIN (GRoup Aggregation via mIN-norm objective), a lightweight, optimizer-agnostic procedure that replaces the arithmetic mean, both within and across mini-batches, with the min-norm convex combination of group-wise gradients. The resulting update is guaranteed to conflict with no group gradient, so no group loss increases to first order. Under mild smoothness and absolute-continuity assumptions it differs almost surely from the mean, which yields a uniform-stability bound strictly tighter than SGD's. Empirically, across generation, classification, and regression at LPM scale, GRAIN is the only method in our suite with no collapsed run over 10 seeds, while improving mean performance and reducing run-to-run variance over a broad set of baselines, at no extra wall-clock or storage cost under data-parallel training.

Primary area:
optimization
Keywords:
training stability, gradient conflict, min-norm gradient aggregation, random seeds

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wKDeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the sensitivity of model training and fine-tuning to random seeds. It attributes part of this instability to conflicts among gradients from different groups of examples and proposes GRAIN, which replaces arithmetic gradient averaging with the minimum-norm convex combination of group gradients.

Critical strengths
  1. The problem is practically relevant. Seed sensitivity and occasional failed runs matter when fine-tuning large pretrained models on small datasets. Evaluating reliability across runs, in addition to average performance, is a useful research objective.

  2. The evaluation is broad and includes multiple seeds. The paper covers several task types, model families, and data conditions. Reporting ten-seed statistics is a meaningful strength.

Critical weaknesses
  1. Theorem 4.4 does not establish a substantive strict stability improvement over SGD. Equation (5) gives GRAIN a strict inequality and SGD a non-strict inequality with the same numerical right-hand side. This neither compares their actual instabilities nor supplies a controlled positive improvement in their worst-case upper bounds. The proof only bounds the distance between two runs by their initial distance plus their path lengths. It does not analyze how the two algorithms propagate perturbations along their respective trajectories.

  2. The incremental contribution relative to MGDA is not sufficiently isolated, and the related-work characterization is inaccurate. Equation (1) is the standard MGDA minimum-norm subproblem, including the Frank-Wolfe solution strategy and common-descent interpretation; see Sener and Koltun (2018), which the manuscript already cites. The core aggregation principle itself is not new.

  3. The collapse annotations in Table 3 are mathematically inconsistent with the stated definition and reported statistics. Lines 298-299 define a collapsed run as reaching majority-class or chance-level test accuracy. The CIFAR test sets are explicitly kept class-balanced, so these levels are 10% and 1%. For ten observations with mean μ\mu and population standard deviation ss, every observation satisfies xi≥μ−3sx_i\geq\mu-3s; the bound is even tighter if ss is the unbiased sample standard deviation. However, Table 3 marks SWA on CIFAR-10 LT-50 as containing a collapsed run while reporting 90.58±5.3190.58\pm5.31. Those numbers imply that every run has accuracy at least 74.65%74.65\%, far above 10%. Similarly, the starred PCGrad result on CIFAR-10 ST-100, 82.05±3.5782.05\pm3.57, implies a minimum of 71.34%71.34\%, and the starred NoisyTune result on CIFAR-100 LT-100, 68.05±3.7568.05\pm3.75, implies a minimum of 56.80%56.80\%, far above 1%. These cannot all be correct under the stated definition.

  4. Several narrative claims do not match the tables. In Table 1, SAM also has no starred entries, so GRAIN is not uniquely collapse-free within that task family. On PubMedQA/Ministral-14B, GRAIN's mean of 75.20 is fourth, below SAM, NoisyTune, and SWA, contradicting the claim of best or second-best performance in every generative configuration. On GSM8K/Qwen-14B, PCGrad achieves 86.40 versus GRAIN's 85.53, and GRAIN does not have the lowest standard deviation in every GSM8K column. These statements should be corrected without diminishing the improvements actually supported by the tables. Reproducibility is further limited by inconsistent descriptions: two A100 GPUs for all runs versus eight for generative fine-tuning, AdamW for all approaches versus SGD for ViT, and a stated ViT total batch size of 128 versus Table 8's per-device batch of 64, two GPUs, and accumulation of two.

Questions for authors
  1. Will you restrict the result to convergence of the Pareto-stationarity residual, or provide additional assumptions and a valid proof for stationarity of the average objective?

  2. Please report per-step time. What additional communication or storage is required?

  3. What is the empirical distribution of ∣gt⋆∣/∣1m∑igt,i∣|g_t^\star|/|\frac{1}{m}\sum_i g_{t,i}| during training, and how often does this ratio fall below 0.1? For seeds on which the baseline collapses, are GRAIN’s actual parameter-update norms larger than those of the baseline? If not, what mechanism explains the improvement in stability, and what empirical evidence supports it?

  4. Which datasets and splits were used to select the learning rate and (m,k)(m,k)? Given that the best value in Figure 3 matches the corresponding result in Table 1, was the PubMedQA test set used to select (m,k)(m,k)? Do the advantages in Tables 1–4 persist when hyperparameters are selected on an independent validation set, and are these improvements statistically significant under paired tests across matched seeds?

1 comment

W1. Theorem 4.4 does not establish a substantive strict improvement over SGD.

Ans: We agree. GRAIN and SGD follow different trajectories after the first update, so we bound each algorithm separately rather than compare their actual instabilities. The GRAIN bound is strict because ∣∣gˉ∣∣<∣∣g∣∣||\bar{g}|| < ||g||, but the size of the gap is not quantified. We will rephrase Theorem 4.4 as a tighter worst-case bound, not a strict improvement in stability.

W2. Contribution relative to MGDA not isolated; related-work characterization inaccurate.

Ans: We agree that Eq. (1), including its Frank-Wolfe solver and common-descent interpretation, follows MGDA (Sener & Koltun, 2018). We will state this explicitly and correct the related-work section. We do not claim the solver as our contribution. Our novelty lies in (1) a new diagnosis of seed-induced instability, group-level gradient cancellation; (2) a design that targets this cause, applying min-norm aggregation jointly over intra- and inter-batch groups; and (3) new stability and sharpness bounds that motivate it.

W3. Collapse annotations in Table 3 are inconsistent with the reported mean and STD.

Ans: Thank you for the careful check. The paper stated only half of the rule we applied (by writing mistake). The full rule is: a run is collapsed when its test accuracy falls more than 10 pp below the best seed of the same configuration, or when it stays at the majority-class (trivial) accuracy. All * marks are consistent with this rule. We will correct the definition in the text and table captions and add per-seed accuracies to the appendix. GRAIN has no * entries under either definition.

W4. Narrative claims that do not match the tables; inconsistent setup descriptions.

Ans: We thank the reviewer and agree that several statements overstate the tables. We will correct them. These corrections do not change the main conclusion: GRAIN is collapse-free in every configuration and is best or second-best in 23 of 24 configurations across Tables 1–3.

On reproducibility, we use 8 GPUs for all job running simultaneously not using 8 GPUs for one run.

Q1. Pareto-stationarity vs. stationarity of the average objective.

Ans: We will do both. We will restate Theorem 4.3 as convergence to a Pareto-stationary point (0∈conv(gi)0 \in conv (g_i)), which differs from stationarity of the mean objective. We will also add a corollary: under bounded group-gradient dissimilarity ∣∣gi−∇L∣∣≤δ||g_i - \nabla L|| \leq \delta, ∣∣∇L(θt)∣∣≤∣∣gˉt∣∣+2δ||\nabla L(\theta_t)|| \leq ||\bar{g}_t|| + 2 \delta, so GRAIN reaches an O(\delta)-neighborhood of a stationary point of L. See our response to Reviewer 2 (W3/Q2) for the proof sketch.

Q2. Per-step time; communication or storage overhead.

Ans: We report the actual cost on Llama-3.1-8B/RTE (time per iteration):

SettingNormalGRAINOverheadExtra Grad Storage
2 GPUs, m = 2, k = 1721.8 ms752.8 ms+4.3%0 MB
1 GPU, m = 2, k = 1665.2 ms1685.7 ms2.5×6.51 MB (LoRA gradient)
1 GPU, m = 4, k = 1665.2 ms2909.3 ms4.4×19.54 MB (LoRA gradient)

The empirical results mostly support our claim, with small extra cost under data-parallel training. We will include this analysis in appendix and further clarify this in the revision.

Q3. Distribution of the update-norm ratio; update norms on collapsed seeds.

Ans: We conduct additional analysis on llama3.1-8B/RTE, the ratio ∣∣gˉt∣∣/∣∣mean gradient∣∣||\bar{g}_t|| / || mean ~ gradient || has median 0.59. It falls below 0.1 in 17% of steps overall, but in 0% of steps while the loss exceeds 1.0. Low ratios occur after convergence, so GRAIN does not stall. Its benefit comes from update direction (Theorem 4.2), not magnitude.

Q4. Hyperparameter selection and statistical significance.

Ans: No test or validation data was used for tuning. We select the learning rate and (m, k) by the lowest training loss, using the same procedure for all methods. For GLUE/SuperGLUE we report on the validation sets, as is standard since test labels are hidden. For PubMedQA (500-example test set), GSM8K and CIFAR we report on the official test sets. Figure 3 is a post-hoc sensitivity analysis: it reports test accuracy for every (m, k), and the configuration chosen by training loss happens to also be the best on test.

For significance, we compare GRAIN with the strongest baseline in each column (Welch’s t-test, n = 10). GRAIN’s variance is significantly lower in 13 of 24 configurations (F-test, p < 0.05), and GRAIN is the only method with no collapsed run across Tables 1–3. Mean gains over the strongest baseline are significant in 7 of 24 configurations. This matches our main claim, which concerns stability rather than peak accuracy. We will add paired Wilcoxon tests over matched seeds to the revision.

Reviewer_5NpeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies seed-induced instability in neural network training, particularly when fine-tuning large pretrained models on small datasets. The authors argue that one source of this instability is gradient cancellation: gradients computed from different groups of training examples may point in conflicting directions, causing their arithmetic mean to have a small norm even when the loss remains high.

To address this issue, the paper proposes GRAIN (Group Aggregation via mIN-norm objective), which replaces arithmetic averaging with the minimum-norm convex combination of group-wise gradients. The procedure can be applied both within a mini-batch and across consecutive mini-batches, and is designed to be compatible with different optimizers. The authors provide theoretical results concerning gradient conflict, convergence, stability, and loss sharpness, and evaluate the method on a range of language-model fine-tuning, classification, and regression tasks.

The empirical results are promising: GRAIN frequently improves mean performance and reduces variability across random seeds, with particularly notable gains in experiments where baseline methods exhibit catastrophic failures. However, I have concerns about the interpretation and scope of some theoretical guarantees, as well as the novelty of the proposed approach relative to existing gradient aggregation methods. These issues make it difficult for me to assess the strength of the paper's overall contribution.

Critical strengths
  • Seed sensitivity can be costly when fine-tuning large pretrained models, particularly when downstream datasets are small and repeated training runs are expensive. Addressing this problem at the optimization level is a reasonable and potentially useful direction.

  • The paper is well-written and generally easy to follow. The proposed method is straightforward to describe and implement: it computes group-wise gradients and solves a small convex optimization problem to determine their aggregation weights. The authors also consider both intra-batch and inter-batch aggregation, making the approach flexible.

  • The experiments cover several model families and task types, and the reported improvements in both mean performance and seed-to-seed variability are encouraging. The results on configurations where standard fine-tuning suffers from catastrophic failures are particularly interesting. The ablation study also provides some evidence that combining the two levels of aggregation can be beneficial.

Critical weaknesses

The main concern is the theoretical justification for the proposed method and the extent to which the results establish its claimed benefits.

  1. The problem is relevant, but gradient aggregation and conflict resolution have been studied extensively, including methods such as MGDA, PCGrad, and CAGrad. Although the paper discusses these approaches, I am not fully convinced that it establishes a sufficiently clear distinction between its contribution and existing min-norm gradient aggregation methods. In particular, the authors should better explain what is fundamentally new about applying this construction to groups of examples in single-task training, and why the resulting method should be expected to mitigate seed-induced instability specifically.

  2. (Major) Theorem 4.2 does not establish that useful progress is always made. The theorem guarantees that the GRAIN update has a nonnegative inner product with every group gradient. However, this does not guarantee a nonzero update. If the convex hull of the group gradients contains the origin, the minimum-norm combination can be exactly zero, even when individual group gradients are nonzero. In that case, the algorithm makes no progress at that iteration, potentially despite a high loss. The paper should discuss this possibility and clarify under what conditions it can occur or be ruled out. The fact that the update conflicts with no group is useful, but it is not by itself sufficient to establish that the update is effective.

  3. (Major) The convergence guarantee requires further clarification. Theorem 4.3 establishes an O(1/T)O(1/T) bound on the minimum squared norm of the GRAIN update. However, a small GRAIN update does not necessarily imply that the gradient of the original objective is small: the update is a particular convex combination of group gradients, not their arithmetic mean. Consequently, the claim that the method converges to a stationary point when the groups are i.i.d. requires additional justification. The connection between the stated bound and stationarity of the original objective is not sufficiently clear to me, particularly in the presence of gradient cancellation.

  4. The stability analysis does not fully establish the claimed practical benefit. The paper derives a strictly tighter instability bound under additional assumptions. However, the quantity bounded theoretically is not the empirical standard deviation of test performance across seeds, as the authors themselves acknowledge in the limitations section. Moreover, the strict improvement relies on technical assumptions, including smoothness and a generic-position condition, whose practical relevance and scope deserve more discussion. I would like to see a clearer explanation of how the theoretical results support the empirical claims without overstating their implications.

  5. The experiments are extensive and encouraging, but the mechanism behind the improvements remains somewhat unclear. The ablation study suggests that intra-batch and inter-batch aggregation can be complementary, yet the benefits appear sensitive to the number of groups and the number of consecutive batches. A more convincing explanation of when and why minimum-norm aggregation improves optimization would strengthen the paper. In particular, the authors should clarify what properties of the loss landscape or the gradient distribution make this aggregation preferable to arithmetic averaging. At present, the method seems motivated primarily by its geometric construction rather than by a sufficiently developed explanation of the conditions under which it should outperform standard SGD.

Overall, the paper is clearly written and the empirical results are promising. Nevertheless, I am not yet convinced that the theoretical results fully justify the claims made about convergence and stability, or that the methodological contribution is sufficiently differentiated from prior gradient aggregation approaches.

Questions for authors

1. When can GRAIN make no progress? Theorem 4.2 permits gˉ=0\bar g=0 whenever the convex hull of the group gradients contains the origin. What conditions prevent this situation, or ensure that it is unlikely in practice? Can the authors provide empirical evidence on how frequently the update norm becomes zero or very small while the training loss remains high?

2. How does Theorem 4.3 imply stationarity of the original objective? The theorem bounds min⁡0≤t<T∣gˉt∣22\min_{0\leq t<T}\\|\bar g_t\\|_2^2, but gˉt\bar g_t is not generally the arithmetic mean of the group gradients. Under precisely what assumptions does this guarantee that the gradient of the original objective approaches zero? Could the authors provide a rigorous argument or a counterexample clarifying the distinction?

3. What is the precise novelty relative to existing min-norm aggregation methods? Since the paper builds on the minimum-norm convex combination used in prior work such as MGDA, what new theoretical or algorithmic insight makes GRAIN a distinct contribution? In particular, which aspects of the proposed approach are essential to its empirical improvements in seed stability?

4. What explains the empirical gains beyond the geometric guarantee? The minimum-norm objective is designed to avoid conflicting with any group gradient, but this property alone does not establish superior performance on the original learning task. What assumptions about the loss landscape or the group-gradient distribution explain why GRAIN should outperform arithmetic averaging? Can the authors identify settings in which the method is expected to help, and settings in which it may fail?

5. How should the theoretical and empirical stability claims be connected? Theorem 4.4 bounds a loss difference between independent runs, whereas the experiments report standard deviations of test metrics across seeds. Can the authors provide a more direct theoretical connection between these quantities, or otherwise clarify the extent to which the theory explains the observed empirical improvements?

3 comments

W1/Q3. Novelty relative to MGDA, PCGrad, CAGrad.

W1. The problem is relevant, but gradient aggregation and conflict resolution have been studied extensively, including methods such as MGDA, PCGrad, and CAGrad. ...

Q3. What is the precise novelty relative to existing min-norm aggregation methods? ...

Ans: We agree that Eq. (1) is MGDA’s min-norm subproblem and will state this explicitly. The min-norm point is a classical tool that has been reused across problems (e.g., SVM duals, multi-objective descent), and we do not claim it as our contribution. GRAIN addresses a problem that no prior method, including MGDA, PCGrad and CAGrad, targets: seed-induced instability in single-task training. Its novelty lies in (1) a new diagnosis of seed-induced instability: group-level gradient cancellation; (2) a design that targets this cause, applying min-norm aggregation jointly over intra- and inter-batch groups; and (3) new stability and sharpness bounds that explain why GRAIN mitigates this instability. Empirically, across all settings, GRAIN never collapses, while every baseline does in some cells. This provides empirical contribution.

W2/Q1. Theorem 4.2: when can GRAIN make no progress?

W2. Theorem 4.2 does not establish that useful progress is always made. ...

Q1. When can GRAIN make no progress? ...

Ans: A zero update is non-generic in our setting, because the iterate is Pareto-stationary across groups: no direction decreases every group loss, so the mean update would necessarily increase some group loss. With m≤m \leq [4] group gradients in d-dimensional parameter space (d>>md >> m), the gᵢ are generically linearly independent, so no convex combination equals zero (the same generic-position assumption as Theorem C.7). Small updates at high loss are instead the failure mode of averaging. Below are statistics from a GRAIN run logged at every step (LLAMA3.1-8B/RTE, m = 2, 624 steps), figure plot will be included in our revision:

• While the training loss is above 1.0 (93 steps), the update norm never becomes small. No step has ∣∣gˉt∣∣||\bar{g}_t|| < 1 or ∣∣gˉt∣∣/∣∣mean gradient∣∣<0.1||\bar{g}_t||/ ||mean~gradient|| < 0.1, and the smallest ∣∣gˉt∣∣||\bar{g}_t|| is 3.14.

• While the training loss is above 0.5 (188 steps), only 6 steps (3.2%) have a ratio below 0.1. All six fall after step 300 and are isolated (steps 303, 369, 375, 504, 505, 515), so they never stall training.

• Near-zero updates (∣∣gˉt∣∣||\bar{g}_t|| < 0.1, 194 steps) all occur after step 255, where the median training loss is 0.004, i.e., after convergence.

GRAIN therefore does not stall while the loss is high. Small updates reflect convergence, not cancellation.

W3/Q2. Theorem 4.3: does a small GRAIN update imply stationarity of the original objective?

W3. The convergence guarantee requires further clarification. ...

Q2. How does Theorem 4.3 imply stationarity of the original objective? ...

Ans: We agree, and thank the reviewer. Theorem 4.3 guarantees convergence to a Pareto-stationary point across groups (0∈conv{(gi)}0 \in conv\{(g_i)\}), not stationarity of the mean objective. A counterexample, g1=(1,0)g_1 = (1, 0) and g2=(−2,0)g_2 = (−2, 0) give gˉ=0\bar{g} = 0 while ∇L=(−0.5,0)\nabla L = (−0.5, 0). We will restate the theorem accordingly and remove the i.i.d. claim. To relate the two notions, note that ∑i(1/m−ai)=0\sum_i(1/m − a_i) = 0 implies ∇L−gˉ=∑i(1/m−ai)(gi−∇L)\nabla L − \bar{g} = \sum_i (1/m − a_i)(g_i − \nabla L). Under a bounded group-gradient dissimilarity ∣∣gi−∇L∣∣≤δ||g_i − \nabla L|| \leq \delta (standard in federated learning, e.g., SCAFFOLD, Karimireddy et al., ICML 2020), this gives ∣∣∇L(θt)∣∣≤∣∣gˉt∣∣+2δ||\nabla L(\theta_t) || ≤ ||\bar{g}_t|| + 2 \delta. GRAIN therefore reaches an O(δ\delta)-neighborhood of stationarity of L. For random partitions of i.i.d. examples, δ\delta shrinks as group size grows. We will add this corollary and its proof to Appendix C.

W4/Q5. Connecting the stability theory (Theorem 4.4) to seed-level test STD.

W4. The stability analysis does not fully establish the claimed practical benefit. ...

Q5. How should the theoretical and empirical stability claims be connected? ...

Ans: Theorem 4.4 can be connected directly to the seed-to-seed spread of test loss. For two independent runs, let Δ=Ltest(θ)−Ltest(θ′)\Delta = L_{test}(\theta) − L_{test}(\theta'). Then Varseed(Ltest)=1/2E[Δ2]Var_{seed}(L_{test}) = 1/2E[\Delta^2]. With a bounded loss difference $ |\Delta| \leq M$, Def. 4.1 gives E[∣Δ∣]≤InsE[|\Delta| ] \leq Ins, so Stdseed(Ltest)≤(M.Ins/2)Std_{seed}(L_{test}) \leq \sqrt{(M.Ins/2)}. The tighter instability bound for GRAIN therefore yields a tighter bound on the across-seed STD of test loss.

We will state two caveats: (1) since the two algorithms diverge after the first update, we compare upper bounds, not actual instabilities; (2) the theory covers the training loss, while the accuracy STD in Section 5 is complementary empirical evidence. The assumptions are mild: smoothness holds for the smooth activations we use (GELU/SiLU), and generic position holds almost surely under a continuous data density (Theorem C.7).

We will add this to Appendix C.

W5/Q4. What explains the empirical gains beyond the geometric guarantee?

W5. The experiments are extensive and encouraging, but the mechanism behind the improvements remains somewhat unclear. ...

Q4. What explains the empirical gains beyond the geometric guarantee? ...

Ans: We explain the gains through loss-landscape sharpness. By Theorem 4.6, GRAIN's ρ-sharpness is bounded by ρ∣∣gˉ∣∣+ρ2L/2\rho||\bar{g}|| + \rho^2 L/2, versus ρ∣∣g∣∣+ρ2L/2\rho ||g|| + \rho^2L/2 for SGD. Since ∣∣gˉ∣∣<∣∣g∣∣||\bar{g}|| < ||g||, GRAIN favors flatter regions, which are less sensitive to perturbations such as seed changes. Fig. 4 (App. E.6) supports this: on CIFAR-10, ResNet-56 without residual connections reaches a visibly smoother landscape with GRAIN, and its error drops from 13.31% to 11.86%. When it helps: when group gradients conflict (∣∣gˉ∣∣<<∣∣g∣∣||\bar{g}|| << ||g||), e.g., sharp landscapes, LPM fine-tuning, and noisy or imbalanced labels. When it may not: when group gradients are aligned, ∣∣gˉ∣∣∼∣∣g∣∣||\bar{g}|| \sim ||g|| and GRAIN reduces to SGD (small gains on clean data, Table 4). Aggregating too many past batches can also hurt, because their gradients are stale (k = 4, Fig. 3).

I would like to thank the authors for their replies.

Some of my concerns regarding the experimental results have been addressed. However, the main issue remains that, since Theorem 4.3 does not establish convergence towards a stationary point (as the authors have acknowledged), the paper is left without a meaningful theoretical contribution.

Proposing an update that does not increase the loss is not particularly surprising on its own (a zero update would achieve the same). What is needed is a formal theorem establishing that the proposed algorithm can reach a point sufficiently close to a stationary point within a reasonable time, which the paper currently lacks.

Therefore, I maintain my score.

Reviewer_qpzwOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates gradient conflicts between different data groups across training runs with different random seeds. To mitigate this conflict, the authors propose GRAIN, a lightweight, optimizer-agnostic procedure that replaces the arithmetic mean, both within and across mini-batches, with the min-norm convex combination of group-wise gradients. Experiments across multiple benchmarks demonstrate that GRAIN improves average performance and reduces run-to-run variance compared with several baselines.

Critical strengths
  1. The paper provides motivation for studying gradient conflicts between data groups in training instability and includes empirical analysis.

  2. The proposed GRAIN method incorporates a minimum-norm gradient aggregation strategy, and the experimental results suggest that this design contributes to improved model performance and training stability.

  3. The paper provides theoretical analysis of the proposed gradient aggregation strategy, including results on convergence and training instability, which helps establish a theoretical foundation for GRAIN.

Critical weaknesses
  1. Most of the baselines are from 2022 or earlier. The authors should compare GRAIN with more recent methods to better demonstrate its advantages over existing approaches.

  2. The method is claimed to be optimizer-agnostic, but the theoretical analysis is based on SGD-style updates. The authors should clarify whether the theoretical guarantees also hold for other optimizers, such as AdamW.

  3. The method introduces additional computation during backpropagation, but the authors provide only a theoretical analysis. The authors should report actual training time and GPU memory overhead to substantiate their efficiency claims.

Questions for authors

see weakness

5 comments

W1/Q1. Baselines are mostly from 2022 or earlier.

Ans: To our knowledge, prior work on seed-induced variance mostly measures and analyzes it without mitigating it. The one direct mitigation we are aware of is ensembling (Wang et al., NeurIPS 2020, cited in the paper), which multiplies training cost by the number of ensemble members. No existing method targets seed-induced variance through the update rule, which is the setting GRAIN addresses. We therefore chose baselines that could plausibly reduce this variance, one from each relevant family: loss reweighting (FocalLoss), noise injection (NoisyTune), flat-minima methods (SAM, SWA), and gradient-conflict resolution (PCGrad). Several were not designed for this setting, so we adapted them to it (e.g., applying PCGrad to group gradients within a batch, Section 5).

W2/Q2. Theory assumes SGD; does it hold for AdamW?

Ans: GRAIN is optimizer-agnostic by construction: it only replaces the mean gradient passed to the optimizer with ḡ (Section 3). Following standard practice for gradient-aggregation methods (e.g., PCGrad, MGDA analyze using gradient descent), our analysis assumes SGD. (1) Theorem 4.2 holds for any optimizer, since zero group conflict is a property of gˉ\bar{g} itself. (2) Theorems 4.3 and 4.4 do not transfer directly: Adam analyses assume unbiased gradients, while ḡ is biased, and per-coordinate normalization changes how the update norm enters the stability bound. We leave this extension to future work. (3) Empirically, GRAIN reduces seed-induced variance under [AdamW and SGD] (Tables 1–4). We will add this discussion to the revision.

W3/Q3. Report actual training time and GPU memory.

Ans: We report the actual cost on Llama-3.1-8B/RTE (time per iteration):

SettingNormalGRAINOverheadExtra Grad Storage
2 GPUs, m = 2, k = 1721.8 ms752.8 ms+4.3%0 MB
1 GPU, m = 2, k = 1665.2 ms1685.7 ms2.5×6.51 MB (LoRA gradient)
1 GPU, m = 4, k = 1665.2 ms2909.3 ms4.4×19.54 MB (LoRA gradient)

The empirical results mostly support our claim, with small extra cost under data-parallel training. We will include this analysis in appendix and further clarify this in the theoretical analysis.

Thank you for the reply. However, I still have some concerns about the baseline selection. Since PCGrad was adapted to this setting, why were other recent gradient aggregation and conflict-resolution methods, such as MoCo [1] and MoDo [2], not considered? These methods are also relevant to GRAIN's core gradient aggregation mechanism, and comparisons would help clarify its advantages over existing approaches.

[1] Fernando, H. D., Shen, H., Liu, M., Chaudhury, S., Murugesan, K., & Chen, T. (2023). Mitigating Gradient Bias in Multi-Objective Learning: A Provably Convergent Approach. International Conference on Learning Representations (ICLR).

[2] Chen, L., Fernando, H., Ying, Y., & Chen, T. (2023). Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidance. Advances in Neural Information Processing Systems (NeurIPS).

Thanks for further comment. MoCo and MoDo are not proper baselines for our setting. They correct the bias of stochastic MGDA for fixed, distinct objectives, whereas GRAIN’s groups are random and resampled at every step. In this setting, their target direction reduces to the mean gradient, i.e., standard fine-tuning, which is already our LoRA/FFT baseline. We include PCGrad because it operates on the current step’s gradients, like GRAIN, so it applies to our setting.

Here is a further explanation why MoCo and MoDo are not proper baselines.

  1. MoCo and MoDo estimate the population conflict-avoidant direction for fixed objectives (e.g., tasks), tracking per-objective gradients (MoCo) or dynamic weights (MoDo) across iterations. GRAIN’s groups are random partitions resampled at every step, so there are no persistent objectives to track.

  2. They reduce to the baseline here. Because the groups are exchangeable, every group has the same expected loss L, and the population direction these methods target is simply ∇L. A bias-corrected MoCo/MoDo would therefore approach standard fine-tuning.

  3. Opposite use of conflict. GRAIN deliberately exploits the conflict among the current group gradients, which is the source of seed-induced cancellation. MoCo and MoDo average this per-step conflict away as noise.

We will add this discussion in a brief way and both references to the related-work section.

Comments on the paper

1 comment

We thank all reviewers for their careful reading. GRAIN targets a common but rarely addressed problem: seed-induced instability.

Novelty (R2, R3). We agree that the min-norm aggregation step comes from MGDA, and we do not claim it as our contribution. Our contributions are (1) a new diagnosis (gradient conflict and cancellation) of seed-induced instability, (2) a design that applies min-norm aggregation within and across mini-batches in single-task training to address it, and (3) stability and sharpness bounds for this setting. These are all new that this community never done before.

Theory (R2, R3). We will make the following clarifications:

• Theorem 4.3 guarantees Pareto-stationarity. We add a corollary that GRAIN reaches an O(δ\delta)-neighborhood of a stationary point of the mean loss.

• Theorem 4.4 compares worst-case bounds. We add a corollary linking it to the seed-to-seed STD of test loss.

• The zero-conflict guarantee (Theorem 4.2) holds for any optimizer, including AdamW.

Empirical claims (R3). We corrected the collapse definition, which the paper stated incompletely (typo), and several overstated statements in the text. The conclusion is unchanged: GRAIN is the only method with no collapsed run across all 24 configurations, is best or second-best in 23 of 24, and has significantly lower variance in 13 of 24 (F-test, p < 0.05).

Optimization dynamics and mechanism (R2, R3). GRAIN does not stall while the loss is high: in a fully logged Llama-3.1-8B/RTE run, ∣∣gˉt∣∣||\bar{g}_t|| never falls below 3.1 while the training loss exceeds 1.0, and near-zero updates occur only after convergence (median loss 0.004). The gains come from update direction, since no group loss increases (Theorem 4.2), and from a tighter sharpness bound (Theorem 4.6, Fig. 4), not from larger steps.

Cost and evaluation protocol (R1, R3). Under 2-GPU data parallelism, GRAIN adds 4.3% per-iteration time (752.8 vs. 721.8 ms); extra storage is one LoRA gradient copy per group (6.51 MB). Learning rate and (m, k) are selected by training loss for all methods, so no test or validation data is used for tuning. Baselines cover each family that could plausibly reduce seed variance (loss reweighting, noise injection, flat minima, gradient-conflict resolution).

Revision. The revision will add:

• per-seed results, the corrected collapse definition, and significance tests;

• the Pareto-stationarity restatement and the two new corollaries (Appendix C);

• wall-clock time, memory and communication cost (R1, R3);

• update-norm statistics during training (R2, R3);

• corrected setup details, toned-down narrative claims, and the revised related-work section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.