- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the sensitivity of model training and fine-tuning to random seeds. It attributes part of this instability to conflicts among gradients from different groups of examples and proposes GRAIN, which replaces arithmetic gradient averaging with the minimum-norm convex combination of group gradients.
Critical strengths
-
The problem is practically relevant. Seed sensitivity and occasional failed runs matter when fine-tuning large pretrained models on small datasets. Evaluating reliability across runs, in addition to average performance, is a useful research objective.
-
The evaluation is broad and includes multiple seeds. The paper covers several task types, model families, and data conditions. Reporting ten-seed statistics is a meaningful strength.
Critical weaknesses
-
Theorem 4.4 does not establish a substantive strict stability improvement over SGD. Equation (5) gives GRAIN a strict inequality and SGD a non-strict inequality with the same numerical right-hand side. This neither compares their actual instabilities nor supplies a controlled positive improvement in their worst-case upper bounds. The proof only bounds the distance between two runs by their initial distance plus their path lengths. It does not analyze how the two algorithms propagate perturbations along their respective trajectories.
-
The incremental contribution relative to MGDA is not sufficiently isolated, and the related-work characterization is inaccurate. Equation (1) is the standard MGDA minimum-norm subproblem, including the Frank-Wolfe solution strategy and common-descent interpretation; see Sener and Koltun (2018), which the manuscript already cites. The core aggregation principle itself is not new.
-
The collapse annotations in Table 3 are mathematically inconsistent with the stated definition and reported statistics. Lines 298-299 define a collapsed run as reaching majority-class or chance-level test accuracy. The CIFAR test sets are explicitly kept class-balanced, so these levels are 10% and 1%. For ten observations with mean and population standard deviation , every observation satisfies ; the bound is even tighter if is the unbiased sample standard deviation. However, Table 3 marks SWA on CIFAR-10 LT-50 as containing a collapsed run while reporting . Those numbers imply that every run has accuracy at least , far above 10%. Similarly, the starred PCGrad result on CIFAR-10 ST-100, , implies a minimum of , and the starred NoisyTune result on CIFAR-100 LT-100, , implies a minimum of , far above 1%. These cannot all be correct under the stated definition.
-
Several narrative claims do not match the tables. In Table 1, SAM also has no starred entries, so GRAIN is not uniquely collapse-free within that task family. On PubMedQA/Ministral-14B, GRAIN's mean of 75.20 is fourth, below SAM, NoisyTune, and SWA, contradicting the claim of best or second-best performance in every generative configuration. On GSM8K/Qwen-14B, PCGrad achieves 86.40 versus GRAIN's 85.53, and GRAIN does not have the lowest standard deviation in every GSM8K column. These statements should be corrected without diminishing the improvements actually supported by the tables. Reproducibility is further limited by inconsistent descriptions: two A100 GPUs for all runs versus eight for generative fine-tuning, AdamW for all approaches versus SGD for ViT, and a stated ViT total batch size of 128 versus Table 8's per-device batch of 64, two GPUs, and accumulation of two.
Questions for authors
-
Will you restrict the result to convergence of the Pareto-stationarity residual, or provide additional assumptions and a valid proof for stationarity of the average objective?
-
Please report per-step time. What additional communication or storage is required?
-
What is the empirical distribution of during training, and how often does this ratio fall below 0.1? For seeds on which the baseline collapses, are GRAIN’s actual parameter-update norms larger than those of the baseline? If not, what mechanism explains the improvement in stability, and what empirical evidence supports it?
-
Which datasets and splits were used to select the learning rate and ? Given that the best value in Figure 3 matches the corresponding result in Table 1, was the PubMedQA test set used to select ? Do the advantages in Tables 1–4 persist when hyperparameters are selected on an independent validation set, and are these improvements statistically significant under paired tests across matched seeds?
Ans: We agree. GRAIN and SGD follow different trajectories after the first update, so we bound each algorithm separately rather than compare their actual instabilities. The GRAIN bound is strict because ∣∣gˉ∣∣<∣∣g∣∣, but the size of the gap is not quantified. We will rephrase Theorem 4.4 as a tighter worst-case bound, not a strict improvement in stability.
Ans: We agree that Eq. (1), including its Frank-Wolfe solver and common-descent interpretation, follows MGDA (Sener & Koltun, 2018). We will state this explicitly and correct the related-work section. We do not claim the solver as our contribution. Our novelty lies in (1) a new diagnosis of seed-induced instability, group-level gradient cancellation; (2) a design that targets this cause, applying min-norm aggregation jointly over intra- and inter-batch groups; and (3) new stability and sharpness bounds that motivate it.
Ans: Thank you for the careful check. The paper stated only half of the rule we applied (by writing mistake). The full rule is: a run is collapsed when its test accuracy falls more than 10 pp below the best seed of the same configuration, or when it stays at the majority-class (trivial) accuracy. All * marks are consistent with this rule. We will correct the definition in the text and table captions and add per-seed accuracies to the appendix. GRAIN has no * entries under either definition.
Ans: We thank the reviewer and agree that several statements overstate the tables. We will correct them. These corrections do not change the main conclusion: GRAIN is collapse-free in every configuration and is best or second-best in 23 of 24 configurations across Tables 1–3.
On reproducibility, we use 8 GPUs for all job running simultaneously not using 8 GPUs for one run.
Ans: We will do both. We will restate Theorem 4.3 as convergence to a Pareto-stationary point (0∈conv(gi)), which differs from stationarity of the mean objective. We will also add a corollary: under bounded group-gradient dissimilarity ∣∣gi−∇L∣∣≤δ, ∣∣∇L(θt)∣∣≤∣∣gˉt∣∣+2δ, so GRAIN reaches an O(\delta)-neighborhood of a stationary point of L. See our response to Reviewer 2 (W3/Q2) for the proof sketch.
Ans: We report the actual cost on Llama-3.1-8B/RTE (time per iteration):
The empirical results mostly support our claim, with small extra cost under data-parallel training. We will include this analysis in appendix and further clarify this in the revision.
Ans: We conduct additional analysis on llama3.1-8B/RTE, the ratio ∣∣gˉt∣∣/∣∣mean gradient∣∣ has median 0.59. It falls below 0.1 in 17% of steps overall, but in 0% of steps while the loss exceeds 1.0. Low ratios occur after convergence, so GRAIN does not stall. Its benefit comes from update direction (Theorem 4.2), not magnitude.
Ans: No test or validation data was used for tuning. We select the learning rate and (m, k) by the lowest training loss, using the same procedure for all methods. For GLUE/SuperGLUE we report on the validation sets, as is standard since test labels are hidden. For PubMedQA (500-example test set), GSM8K and CIFAR we report on the official test sets. Figure 3 is a post-hoc sensitivity analysis: it reports test accuracy for every (m, k), and the configuration chosen by training loss happens to also be the best on test.
For significance, we compare GRAIN with the strongest baseline in each column (Welch’s t-test, n = 10). GRAIN’s variance is significantly lower in 13 of 24 configurations (F-test, p < 0.05), and GRAIN is the only method with no collapsed run across Tables 1–3. Mean gains over the strongest baseline are significant in 7 of 24 configurations. This matches our main claim, which concerns stability rather than peak accuracy. We will add paired Wilcoxon tests over matched seeds to the revision.