acceptodds
Under review as a conference paper at ICLR 2027

Sharpness Corrections Need Not Act Immediately in Momentum-Based SAM

Abstract

Sharpness-Aware Minimization (SAM) evaluates gradients at perturbed parameters, but under momentum the resulting sharpness correction acts through an immediate zero-lag route and a delayed route carried by the momentum state. The central question is whether a generated correction must act immediately to remain useful. Temporal Feedback SAM (TF-SAM) isolates this choice by removing the zero-lag route while preserving correction generation and its momentum-mediated influence. An exact original-iterate identity shows that this intervention replaces the fresh correction at the target update with its momentum-weighted previous counterpart. A target-matched first-order criterion then shows that the preferred route depends on the corrections' relative alignment with the current descent direction. At regular training states, the normalized-SAM correction has an explicit Jacobian, and the two routes enter different coefficients of the local closed-loop characteristic matrix polynomial. Across the evaluated image and text settings, removing zero-lag feedback does not produce a consistent loss in predictive accuracy or aggregate corruption robustness; target-matched measurements show that the relative first-order value of the routes varies across regimes. Generating sharpness information and applying it immediately are therefore separable design decisions in momentum-based SAM, while holding correction generation fixed.

Primary area:
optimization
Keywords:
Sharpness-Aware Minimization.Momentum.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_81vaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether sharpness corrections in momentum-based SAM must affect the current update. TF-SAM removes the immediate correction while retaining its influence through momentum. The paper provides recurrence, alignment, and local stability analyses, with experiments across six image and text settings using ten random seeds.

Critical strengths

The recurrence and stability analyses clarify the roles of immediate and historical corrections without requiring the Hessian and correction response to commute.

Critical weaknesses

The immediate correction and evaluation gradient share a minibatch, unlike the delayed correction, potentially confounding the alignment diagnostics.

Questions for authors

Do the routing preferences observed on CIFAR and GPT-2 persist when with bigger batchsize?

4 comments
AuthorsOpenReview ↗

Larger Batch Routing

Answer. We thank the reviewer for identifying the minibatch-dependence issue and suggesting larger-batch experiments. We evaluated CIFAR-10/ResNet-18 at batch sizes 256 and 512 and GPT-2/SST-5 at batch size 32, with ten seeds per method and setting. The results preserve the qualitative routing preferences at these batch sizes, with the qualification below concerning the diagnostic target.

Q1:

  • Yes, the qualitative preferences persist at the larger batch sizes tested.

  • At ρ=0.05\rho=0.05, CIFAR-10 retains negative DtD_t, favoring immediate routing, on both M-SAM and TF-SAM trajectories. Mean DtD_t is −3.020/−2.879 at batch 256 and −3.368/−3.281 at batch 512 (M-SAM/TF-SAM); all ten seed means are negative in each setting. The corresponding 95% CIs are [−3.239, −2.801]/[−3.167, −2.591] at batch 256 and [−3.621, −3.114]/[−3.511, −3.052] at batch 512. Mean immediate projections are 2.989/2.907 and 3.405/3.342, respectively, versus momentum-weighted delayed projections of −0.031/0.028 and 0.037/0.061 (M-SAM/TF-SAM throughout). Across sampled steps, 99.25% favor immediate routing on M-SAM trajectories at each batch size and 98.75% on TF-SAM trajectories.

  • GPT-2 at batch 32 retains positive DtD_t, favoring delayed routing, for the clipped gradients used by the optimizer: 0.462 [0.377, 0.546] on M-SAM trajectories and 0.471 [0.394, 0.547] on TF-SAM trajectories (mean [95% CI]), with positive means in 10/10 seeds for each method. All 50/50 sampled clipped-gradient differences per method are positive. Mean immediate projections are −0.480/−0.469, versus momentum-weighted delayed projections of −0.019/0.002 (M-SAM/TF-SAM).

  • Before clipping, the corresponding estimates are 0.271 [0.114, 0.428] and 0.166 [−0.024, 0.356]; positive seed means occur in 9/10 and 7/10 seeds, respectively, with positive differences at 82%/76% of sampled steps.

  • All 60 runs are complete. CIFAR runs use 200 epochs and β=0.9\beta=0.9; GPT-2 runs use 10 epochs and β=0.8\beta=0.8. Diagnostics sample 40 target steps per CIFAR run and five per GPT-2 run, using the first minibatch every five and two epochs, respectively, with an exactly one-step-old correction. We average per-step ratios within each seed, then across ten seeds; 95% Student-tt CIs use seeds as the independent units (nine degrees of freedom).

  • These experiments support persistence of the observed CIFAR/GPT-2 distinction at the tested larger batch sizes, most clearly for the gradients actually used in training. We have added these results, the measurement protocol, and seed-level confidence intervals to the revised paper PDF in Appendix Routing preferences at larger batch sizes.

W1 (shared-minibatch alignment):

  • The diagnostic compares both corrections against gtg_t: ctc_t shares its minibatch, while ct−1c_{t-1} comes from the preceding minibatch. It measures alignment for the stochastic update rather than against an independent population-gradient estimate.

  • This distinction is now explicit in the Discussion paragraph beginning “These conclusions are limited.” The larger-batch results above show that the observed signs persist under the same protocol.

Dear Reviewer 81va,

We would like to express our sincere gratitude for your insightful comments and valuable feedback. We believe that our responses have sufficiently clarified your questions and concerns. We highly value your suggestions and welcome any further comments or questions you may have.

Thank you once again for your careful review and constructive feedback!

Q1. Yes, the qualitative routing preferences persist at the larger batch sizes tested, most consistently for the gradient representation used by the optimizer. We completed 60 training runs, covering CIFAR-10/ResNet-18 at batch sizes 256 and 512 and GPT-2/SST-5 at batch size 32, with ten seeds for each method and setting. The main configurations use batches 128 and 16, respectively. The new results therefore test two- and fourfold increases for CIFAR and a twofold increase for GPT-2.

TaskBatchTrajectoriesSeeds per methodEpochsMomentum
CIFAR-10 / ResNet-18256M-SAM, TF-SAM102000.9
CIFAR-10 / ResNet-18512M-SAM, TF-SAM102000.9
GPT-2 / SST-532M-SAM, TF-SAM10100.8

All settings use ρ=0.05\rho=0.05. M-SAM and TF-SAM correspond to immediate-routing weights α=1\alpha=1 and 00. We measure both methods' own trajectories, so the observed sign is not restricted to states visited by M-SAM.

At target step tt, we compare the current correction ctc_t and the exactly one-step-old correction ct−1c_{t-1} against the same current training-minibatch gradient gtg_t:

q_t^{\rm delay}=\frac{\langle c_{t-1},g_t\rangle}{\|g_t\|^2},\quad D_t=\beta q_t^{\rm delay}-q_t^{\rm imm}.$$ **Negative $D$ favors immediate routing; positive $D$ favors delayed routing**, at the local first-order level. The following estimates average per-step ratios within seeds, then average the ten seed means. Intervals are two-sided 95% Student-$t$ CIs across seeds (nine degrees of freedom). **CIFAR-10: immediate routing remains preferred at both larger batches.** Gradients are unclipped. | Batch | Trajectory | Immediate $q$ | Weighted delayed $\beta q$ | $D$ [95% CI] | Negative seeds | | --- | --- | ---: | ---: | --- | ---: | | 256 | M-SAM | 2.989 | −0.031 | **−3.020 [−3.239, −2.801]** | 10/10 | | 256 | TF-SAM | 2.907 | 0.028 | **−2.879 [−3.167, −2.591]** | 10/10 | | 512 | M-SAM | 3.405 | 0.037 | **−3.368 [−3.621, −3.114]** | 10/10 | | 512 | TF-SAM | 3.342 | 0.061 | **−3.281 [−3.511, −3.052]** | 10/10 | All four seed-level intervals lie below zero. Immediate projections remain positive, while momentum-weighted delayed projections are near zero. Across the 400 sampled target steps per setting and method, 99.25% favor immediate routing on M-SAM trajectories at each batch size and 98.75% on TF-SAM trajectories. These pooled fractions are descriptive; the CI uses ten independent seeds rather than 400 steps. **GPT-2: delayed routing remains preferred for the clipped gradients used in training.** We also report the corresponding raw-gradient measurements. | Trajectory | Representation | Immediate $q$ | Weighted delayed $\beta q$ | $D$ [95% CI] | Positive seeds | | --- | --- | ---: | ---: | --- | ---: | | M-SAM | Clipped | −0.480 | −0.019 | **0.462 [0.377, 0.546]** | 10/10 | | TF-SAM | Clipped | −0.469 | 0.002 | **0.471 [0.394, 0.547]** | 10/10 | | M-SAM | Raw | −0.262 | 0.009 | 0.271 [0.114, 0.428] | 9/10 | | TF-SAM | Raw | −0.157 | 0.009 | 0.166 [−0.024, 0.356] | 7/10 | All 50/50 sampled clipped-gradient differences per method are positive. Raw-gradient differences are positive at 82%/76% of sampled steps on M-SAM/TF-SAM trajectories. **The raw TF-SAM interval includes zero**, so the evidence is more consistent in the clipped representation than before clipping. Displayed projections and $D$ are rounded independently. Diagnostics use the first minibatch every five CIFAR epochs (40 target steps per seed) and every two GPT-2 epochs (five per seed). The delayed correction is from the immediately preceding optimizer step, not from the preceding logging event. For GPT-2, each correction is formed from the perturbed-minus-clean gradients in the same representation as its target gradient. The original phase-wise CIFAR results have mixed early signs and favor immediate routing in the middle and late phases. GPT-2 favors delay across the reported phases, with an adverse immediate projection and a delayed projection near zero after the early phase. The larger-batch results reproduce that qualitative regime distinction in the sampled-update averages. Neither summary asserts that every CIFAR step favors immediacy or that raw GPT-2 alignment always favors delay. These results are included in Appendix *Routing preferences at larger batch sizes*, with the protocol and seed-level uncertainty. **The CIFAR/GPT-2 sign distinction persists at the tested larger batches on both optimizer trajectories.** This answers the batch-size question for the implemented stochastic-update diagnostic; it does not turn that diagnostic into an independent population-gradient measurement or establish an accuracy advantage from its local sign.

W1. The empirical diagnostic measures stochastic-update alignment, not an unbiased population-gradient alignment. The immediate correction ctc_t and evaluation gradient gtg_t use the same minibatch, whereas ct−1c_{t-1} comes from the preceding minibatch and parameter state. We now state this distinction explicitly in the Discussion paragraph beginning “These conclusions are limited.” Larger batches test persistence under that measurement protocol; they do not remove the asymmetry in data sharing.

The comparison holds the target state, current minibatch, target gradient, and normalization fixed. It asks which already-generated correction has the more favorable projection onto that particular update's clean gradient. This target matching avoids comparing projections against two different target gradients, but the two corrections still differ in their generation time and minibatch dependence.

Quantity held fixedWhat remains different
Current parameter state wtw_tct−1c_{t-1} was generated at the preceding state
Target minibatch and gradient gtg_tOnly ctc_t shares the target minibatch
Denominator ∥gt∥2\|g_t\|^2Corrections may differ in direction and magnitude
Target optimizer stepOne correction is fresh, the other exactly one step old

The theoretical population criterion and the realized minibatch diagnostic have distinct roles. In the conditional one-step analysis, the derivative of correction alignment with respect to α\alpha is At,0−βAt,1A_{t,0}-\beta A_{t,1}, where the alignments use the current population gradient conditioned on the history. In the experiment, Dt=βqtdelay−qtimmD_t=\beta q_t^{\rm delay}-q_t^{\rm imm} substitutes the observed training-minibatch gradient. Its sign describes the sampled update; it is not an unbiased estimate of that population criterion. The manuscript explicitly calls these realized measurements descriptive diagnostics.

Aggregation also matters when comparing the original figures with the new batch-size table. The original phase-wise protocol divides training into epoch thirds, uses exactly contiguous one-step pairs, discards non-contiguous pairs, and contributes 49 valid targets per seed per phase. Within each phase it uses a ratio of sums. For the immediate projection,

Qsimm=∑t⟨ct,gt⟩∑t∥gt∥2.Q_s^{\rm imm}=\frac{\sum_t\langle c_t,g_t\rangle}{\sum_t\|g_t\|^2}.

This is an energy-weighted average of the per-step normalized projections. The larger-batch experiment instead takes the arithmetic mean of the sampled per-step ratios within each seed. These are different summaries, so magnitudes should not be read as a direct batch-size effect across the two protocols. Both analyses treat seeds as the inferential units; individual steps within a run do not supply independent replicates.

Clipping is part of the GPT-2 measurement target. With clean and perturbed gradients u,vu,v both clipped to unit norm,

qimm=⟨u,v−u⟩∥u∥2≈cos⁡θ−1≤0.q^{\rm imm}=\frac{\langle u,v-u\rangle}{\|u\|^2}\approx\cos\theta-1\le0.

Thus, a negative clipped immediate projection by itself cannot identify an adverse unprocessed sharpness correction. The batch-32 results report both representations: raw DD is positive on average for both trajectories, but its TF-SAM CI includes zero; clipped DD is positive in all ten seeds for both methods. Our strongest directly supported statement therefore concerns the gradients actually used by the optimizer.

Additional final-checkpoint GPT-2 controls use ten M-SAM seeds and a deterministic training minibatch per seed. They compare independent and matched dropout masks at zero and nonzero radius:

MasksRadiusRaw immediate qqClipped immediate qq
Independent0−0.469−0.472
Independent0.05−0.410−0.471
Matched000
Matched0.05+0.015−0.109

Independent dropout can produce a nonzero difference without any parameter perturbation. With matched masks, the perturbation effect is +0.015 [−0.061, 0.091] before clipping and −0.109 [−0.134, −0.084] after clipping. These probes isolate dropout and clipping at final checkpoints; they are not an independent-target gradient evaluation and do not establish the perturbation-only sign throughout training.

An independent holdout-gradient diagnostic has not been evaluated here. The new larger-batch evidence instead shows that the observed ordering survives the tested batch increases under the original shared-target construction. Local alignment, population descent, and final generalization remain distinct claims: the plots omit higher-order terms and do not simulate counterfactual training trajectories. Accordingly, comparable final accuracy on CIFAR is compatible with a favorable immediate local projection, and the diagnostic alone does not causally explain final accuracy gaps.

Reviewer_SC3SOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether the correction produced by Sharpness-Aware Minimization (SAM) needs to affect the model immediately to remain useful. SAM adjusts the usual training gradient using information from nearby model parameters. When combined with momentum, which carries information from previous training steps, this correction influences both the current update and later updates. The proposed method, TF-SAM, removes the correction from the current update while retaining it in momentum, so that its influence begins one step later. The authors derive how this change affects the update equations, compare how immediate and delayed corrections align with the current descent direction, and analyze their effects on local stability. Across six image and text classification settings, with ten training seeds per setting, the reported accuracy differences from standard momentum SAM are small in five cases. TF-SAM improves accuracy on GPT-2/SST-5, although both SAM variants remain below SGD in that setting. Through these experiments, alongside robustness and loss-geometry measurements, the paper investigates whether the timing of a correction can be chosen separately from how it is generated.

Critical strengths
  1. A clear, testable idea about when corrections occur. I find the separation between generating a correction and applying it immediately useful, since it makes timing an explicit choice within SAM. TF-SAM keeps the correction in momentum, where it can influence later updates, while removing its direct effect on the current update. The checked update equations and local stability results, seems sound under their reasonable stated assumptions. The authors also clearly distinguish their local conclusions from guarantees about an entire training run.

  2. Encouraging accuracy results with careful qualification. I find it nice the use of ten matched training seeds, with confidence intervals for paired differences, which in my opinion strength of the experimental design. Table 2 reports small accuracy gaps in five settings and a 0.530.53 percentage-point improvement over momentum SAM on GPT-2/SST-5, with a 95% confidence interval of [0.30,0.76][0.30, 0.76]. I also appreciate that the authors acknowledge that TF-SAM still trails SGD there, that intervals containing zero should not count towards establishing equivalent performance, and that the stronger test of sensitivity to parameter changes remains inconclusive.

Critical weaknesses
  1. The aggregate robustness results are not clear. I am not sure how to read the summary in Table 7 with the detailed results in Tables 3, 8, and 9. Under the protocol described in Section 4.4 and Appendix D.3, each corruption condition receives equal weight, and the methods are compared using matched training seeds. If these tables use the same runs, the two calculations should agree, whether we average the differences across conditions or first average within each seed and then compare the methods. For example, the detailed CIFAR-10/ResNet-18 results imply a TF-SAM minus M-SAM gap of approximately −0.0017-0.0017 percentage points, whereas Table 7 reports +0.35+0.35. For CIFAR-10/ResNet-34, the corresponding values are −0.1692-0.1692 and +0.05+0.05, so here even the sign differs. It seems that similar discrepancies appear in both the CIFAR-100 settings. All four discrepancies exceed the approximately 0.0150.015-point allowance for rounding the component means and the final aggregate. Such pairing affects the uncertainty of the estimated difference, but cannot explain the disagreement between the means. It is therefore unclear to me whether the tables use different runs, a different aggregation rule, or contain a reporting error. Since preservation of robustness is a central finding, this unresolved inconsistency limits my confidence in that part of the paper. The seed-by-condition records and a consistent regeneration of the tables would help resolve it.

  2. The experiments do not clearly isolate the role of correction timing. The reported clean-accuracy results are quite encouraging, but I find the explanation of the GPT-2 behavior definitely less conclusive. The released trainer and diagnostic logger separately clip the clean and perturbed gradients, rescaling each if its norm that exceeds 11, before taking their difference. The measured correction can consequently differ from the unprocessed gradient difference used in the theory. If the processed gradients are denoted by uu and vv, the logged immediate projection is

q=⟨u,v−u⟩∥u∥2.q = \frac{\langle u, v-u\rangle}{\lVert u\rVert^2}.

This quantity measures whether the correction points along or against the clean gradient. When both gradients reach the same clipping cap, q≈cos⁡θ−1≤0q \approx \cos\theta - 1 \le 0, where θ\theta is the angle between them, with the approximation accounting for the small numerical offset used in clipping. Its value is also directly linked to the relative size of the correction. Consequently, a negative projection together with a substantial correction norm does not provide independent evidence that the unprocessed sharpness correction opposes the clean gradient. The two forward passes also use fresh dropout randomness, which can produce a nonzero difference even when the perturbation radius is ρ=0\rho=0. To me, these alone, do not establish how much either effect contributes to the reported pattern, because the exact clipping frequencies and the correspondence between figures and logs remain unresolved.

  1. The effects of delay and reduced correction weight remain difficult to separate. A related concern is whether the existing ablations separate the effect of delay from the effect of reducing the accumulated correction weight. Removing the immediate contribution changes the sum of the correction-response kernel weights from 1/(1−β)1/(1-\beta) to β/(1−β)\beta/(1-\beta). This is a reduction of 10% at β=0.9\beta=0.9 and 20% at the GPT-2 setting, β=0.8\beta=0.8. These percentages concern the response to a fixed input correction, rather than the actual parameter displacement along a changing training trajectory. Proposition 6 correctly shows that a scalar rescaling cannot reproduce the entire temporal response. Nevertheless, I do not think this alone rules out reduced correction weight as an explanation for the observed performance. A focused comparison that matches the summed correction weight while retaining immediate action would likely help distinguish these possibilities.
Questions for authors

Thank you to the authors for this interesting study and for the care taken to discuss its scope and limitations. I would appreciate clarification on the points below, which would help me better understand the results and assess the contribution.

  1. Clarifying the robustness averages. Could you explain how the aggregate differences in Table 7 were obtained from the detailed results in Tables 3, 8, and 9? In particular, were the same training runs and equal weights for the corruption conditions used throughout? Providing the accuracy for each seed and condition, together with the recalculated mean differences and paired confidence intervals, would help me understand the source of the discrepancy and assess whether the robustness conclusion remains supported.

  2. Understanding the GPT-2 measurements. For the GPT-2 measurements in Figures 8–10, could you clarify how often the clean and perturbed gradients reached the clipping threshold? In the existing diagnostic setting, it would also be helpful to see whether the projection patterns persist when measured before clipping and when the two gradient calculations use the same dropout masks. A focused check at ρ=0\rho=0, where no parameter perturbation is applied, would help show whether a similar pattern can arise from stochastic variation alone. These checks would make it easier to assess how well the current measurements isolate the effect of the sharpness perturbation.

  3. Separating delay from correction weight. In the two settings already used to study the mechanism, particularly GPT-2/SST-5, how does TF-SAM compare with ordinary momentum SAM using gt+βctg_t+\beta c_t in both its update and momentum buffer, with the momentum coefficient β\beta and perturbation radius ρ\rho held fixed? This comparison matches the summed correction-response weights while retaining immediate action, although it does not make the training trajectories equivalent. I would find this focused comparison particularly helpful in assessing whether the observed difference reflects the timing of the correction or the reduction in its overall response weight.

6 comments
AuthorsOpenReview ↗

Corrected Robustness and Gradient Controls

Answer. We thank the reviewer for the careful reading and constructive suggestions. We corrected the robustness aggregate, measured the GPT-2 gradient controls, and added correction-weight-matched baselines. The measured results follow.

Q1:

  • Table 7 used an outdated four-seed aggregate; the detailed corruption tables use ten seeds. We recalculated Table 7 from the same ten matched training seeds, weighting non-clean conditions equally and computing paired 95% confidence intervals across seeds.

  • The corrected CIFAR-10 TF-SAM minus M-SAM gaps are −0.001500 pp [−0.337972, 0.334972] (ResNet-18) and −0.168667 pp [−0.707123, 0.369790] (ResNet-34), consistent with the detailed tables.

  • Per-seed, per-condition CSV records, recalculated aggregates, and the analysis script are available in our anonymous repository (suppl/ and analysis/analyze_corruption_seed_aggregate.py).

  • The other aggregate gaps (pp [95% CI]) are +0.057 [−0.089, 0.204] for CIFAR-100/ResNet-18, −0.060 [−0.310, 0.190] for CIFAR-100/ResNet-34, +0.168 [0.098, 0.237] for SST-2/TinyBERT, and +0.095 [0.026, 0.163] for GPT-2/SST-5. Image aggregates cover 12 corruptions and text aggregates 13 perturbations.

Q2:

  • In the new GPT-2/SST-5 weight-matched runs at ρ=0.05\rho=0.05, both clean and perturbed gradients exceeded the norm-1 clipping threshold in every logged sample: 20/20 for each of M-SAM, Scaled M-SAM, and TF-SAM (two sampled steps per seed, ten seeds). Mean immediate projection qq before/after clipping was −0.395/−0.514 for M-SAM, −0.351/−0.488 for Scaled M-SAM, and −0.336/−0.478 for TF-SAM. The projection is negative before clipping and becomes more negative after clipping, without changing sign. These logs come from new runs using the same GPT-2/SST-5 training configuration as Figures 8–10.

  • At ten final M-SAM checkpoints, using the same deterministic training minibatch per seed, independent dropout masks gave mean preclip/clipped qq of −0.469/−0.472 at ρ=0\rho=0 and −0.410/−0.471 at ρ=0.05\rho=0.05; matched masks gave 0/0 and +0.015/−0.109, respectively.

  • With matched masks, the paired ρ=0.05\rho=0.05 minus ρ=0\rho=0 effect was +0.015 [−0.061, 0.091] before clipping and −0.109 [−0.134, −0.084] after clipping (95% CIs).

  • At ρ=0.05\rho=0.05, the paired independent-minus-matched-mask difference is −0.425 [−0.523, −0.328] before clipping and −0.362 [−0.403, −0.321] after clipping.

  • Raw correction norms at ρ=0/0.05\rho=0/0.05 are 3.728/3.939 with independent masks and 0/1.995 with matched masks; both gradients exceed the clipping threshold in all probe samples.

  • Figures 8–10 measure the implemented stochastic update across training phases, including delayed target-matched projections; the new dropout and clipping controls use final checkpoints.

Q3:

  • We added ten-seed weight-matched controls on CIFAR-10/ResNet-18 and GPT-2/SST-5, holding β\beta, ρ\rho, and other training settings fixed within each task. Scaled M-SAM uses gt+βctg_t+\beta c_t in both its current update and momentum buffer, matching TF-SAM’s summed response weight for a fixed correction while retaining an immediate route.

  • The sampled logs show that the implemented routes differ: both methods generated nonzero corrections at all 400 CIFAR-10 and 20 GPT-2 logged steps, but the immediate term was nonzero only for Scaled M-SAM. Median immediate/buffer-input correction norms were 0.394/0.394 versus 0/0.408 on CIFAR-10 and 0.783/0.783 versus 0/0.980 on GPT-2 (Scaled M-SAM versus TF-SAM; GPT-2 uses clipped gradients).

  • The paired TF-SAM minus Scaled M-SAM final-accuracy gaps are −0.038 pp [−0.133, 0.057] and −0.104 pp [−0.340, 0.132] (95% CIs); equal-weight non-clean robustness gaps are +0.05 pp [−0.34, 0.44] and −0.05 pp [−0.14, 0.03], respectively.

  • Final accuracies for M-SAM / TF-SAM / Scaled M-SAM are 96.008 / 95.919 / 95.957% on CIFAR-10 and 52.448 / 52.905 / 53.009% on GPT-2. On GPT-2, paired TF-SAM-minus-M-SAM and Scaled-minus-M-SAM gaps are +0.457 [0.028, 0.886] and +0.561 [0.207, 0.915] pp.

  • TF-SAM-minus-Scaled best-checkpoint gaps are +0.020 [−0.073, 0.113] pp on CIFAR-10 and −0.113 [−0.413, 0.187] pp on GPT-2. Full results appear in Appendix Correction-response weight-matched controls.

Dear Reviewer SC3S,

We sincerely appreciate your thoughtful comments and helpful suggestions. We hope that our detailed responses have satisfactorily addressed all the issues you raised. We look forward to your further feedback and would be happy to provide any additional explanations or clarifications.

Thank you for your time and consideration!

Q1. Reconciling the robustness aggregates. The discrepancy was a reporting error: the previous Table 7 used an outdated four-seed aggregate, whereas the detailed tables use ten seeds. We regenerated the aggregate using the same ten matched seeds and equal non-clean-condition weights as the detailed tables. Pairing changes uncertainty, not the aggregate mean; it cannot explain the original discrepancies.

For seed ss and KK non-clean conditions, define asM=K−1∑k=1Kas,kMa_s^M=K^{-1}\sum_{k=1}^K a_{s,k}^M and ds=asTF−asMd_s=a_s^{\rm TF}-a_s^{\rm M}. The reported gap is dˉ=10−1∑sds\bar d=10^{-1}\sum_s d_s, with paired CI dˉ±t0.975,9sd/10\bar d\pm t_{0.975,9}s_d/\sqrt{10}. Thus, averaging unrounded condition-level means and averaging within seeds yield the same point estimate. Training seeds, rather than corruption conditions, are the independent units.

Corrected TF-SAM-minus-M-SAM gaps (percentage points, pp):

SettingConditionsGap [paired 95% CI]
CIFAR-10 / ResNet-1812−0.001500 [−0.337972, 0.334972]
CIFAR-10 / ResNet-3412−0.168667 [−0.707123, 0.369790]
CIFAR-100 / ResNet-1812+0.057 [−0.089, 0.204]
CIFAR-100 / ResNet-3412−0.060 [−0.310, 0.190]
SST-2 / TinyBERT13+0.168 [0.098, 0.237]
SST-5 / GPT-213+0.095 [0.026, 0.163]

The CIFAR-10 values agree with the reviewer's calculations up to rounding. All four image CIs include zero; both text CIs are positive. The corrected evidence supports no consistent aggregate robustness penalty across these settings, without establishing equivalence or a gain on every corruption. Appendix Aggregate corruption robustness states the source of the error and the corrected protocol. Per-seed, per-condition CSVs, recalculated aggregates, and analysis/analyze_corruption_seed_aggregate.py are provided in the existing anonymous repository (suppl/).

Q2. Clipping and dropout controls. We distinguish training-time measurements from controlled final-checkpoint probes. In the new GPT-2/SST-5 weight-matched runs at ρ=0.05\rho=0.05, both clean and perturbed gradients exceed the norm-1 clipping threshold in all 20 logged samples per method (two sampled steps per seed, ten seeds). These are new runs with the same training configuration as Figures 8–10, rather than recovered clipping counts for the original figure traces.

MethodSamples clipping both gradientsMean qq, rawMean qq, clipped
M-SAM20/20−0.395−0.514
Scaled M-SAM20/20−0.351−0.488
TF-SAM20/20−0.336−0.478

Here q=⟨u,v−u⟩/∥u∥2q=\langle u,v-u\rangle/\|u\|^2, with clean/perturbed gradients u,vu,v represented consistently before or after clipping. When both clipped norms equal one, q≈cos⁡θ−1≤0q\approx\cos\theta-1\le0, so negative clipped projection alone cannot identify an adverse raw sharpness correction. The negative raw means above also include stochastic dropout differences.

We therefore probe ten final M-SAM checkpoints, using the same deterministic training minibatch per seed and comparing independent versus matched dropout masks at ρ=0/0.05\rho=0/0.05:

Dropout masksρ\rhoRaw qqClipped qqRaw correction norm
Independent0−0.469−0.4723.728
Independent0.05−0.410−0.4713.939
Matched0000
Matched0.05+0.015−0.1091.995

Both gradients exceed the clipping threshold in every probe sample. With matched masks, the paired ρ=0.05\rho=0.05 minus zero effect is +0.015 [−0.061, 0.091] raw and −0.109 [−0.134, −0.084] clipped. At ρ=0.05\rho=0.05, independent-minus-matched-mask gaps are −0.425 [−0.523, −0.328] raw and −0.362 [−0.403, −0.321] clipped.

These controls show that independent dropout can generate a substantial correction even at zero radius; matching masks removes that zero-radius difference. Under matched masks, the nonzero-radius raw mean is near zero with a CI crossing zero, while the clipped mean remains negative. Dropout and clipping materially affect the observed projection. At the common unit cap, ∥v−u∥2≈2(1−cos⁡θ)\|v-u\|^2\approx2(1-\cos\theta) and hence q≈−∥v−u∥2/2q\approx-\|v-u\|^2/2. Projection sign and correction norm are therefore algebraically coupled in this representation; they should not be counted as independent evidence for an adverse perturbation. The raw, matched-mask probe directly addresses the perturbation-only question at the sampled final checkpoints. Figures 8–10 characterize the implemented stochastic update across training phases, including delayed target-matched projections; these probes concern final checkpoints. They do not establish that the raw perturbation-only correction is adverse throughout training. The probe contrasts are paired across the ten checkpoint seeds.

Reviewer_4EUaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the timing of sharpness corrections in momentum-based SAM and proposes TF-SAM, which removes the immediate correction pathway so that sharpness corrections influence subsequent parameter updates only through momentum. Theoretically, the authors derive first-order descent criteria for immediate and delayed corrections and show through local dynamical analysis that different temporal routing strategies alter the characteristic polynomial and stability of the system. Experimental results demonstrate that TF-SAM largely preserves the classification accuracy and perturbation robustness of conventional Momentum-SAM across image and text tasks. These findings suggest that sharpness corrections do not necessarily need to take effect immediately, and that their timing can be treated as an independent dimension of optimizer design.

Critical strengths
  1. Novel research perspective. The paper decouples the generation and application timing of sharpness corrections, introducing Temporal Routing as an independent dimension of optimizer design. This distinguishes the proposed approach from existing SAM methods that primarily focus on perturbation construction or computational frequency.

  2. Comprehensive theoretical analysis. The paper provides exact parameter update relations, first-order gradient alignment criteria, and local stability analysis, explaining how immediate and delayed corrections affect optimization dynamics.

  3. Broad experimental coverage. The authors conduct experiments on CIFAR-10/100 and SST-2/5 across different image and text models, evaluating classification accuracy, perturbation robustness, and gradient alignment.

  4. Methodological simplicity. TF-SAM only modifies how sharpness corrections are applied, without requiring additional model architectures or complex optimization steps.

Critical weaknesses
  1. Limited performance improvements. TF-SAM achieves performance comparable to M-SAM in most experiments, with noticeable improvements observed only on a few tasks. Therefore, the main contribution lies in validating the proposed mechanism rather than delivering substantial performance gains.

  2. Lack of global theoretical guarantees. The theoretical analysis primarily relies on first-order descent criteria at a fixed optimization state and local linearization. It does not establish whether delayed corrections lead to better overall convergence or generalization.

  3. Lack of an adaptive mechanism. Both theoretical and experimental results suggest that the optimal timing of sharpness corrections depends on the training state. However, the paper only considers a fixed delay strategy, without implementing or evaluating adaptive Temporal Routing.

  4. No reduction in computational overhead. TF-SAM still requires computing the SAM perturbation gradient, resulting in essentially the same computational cost as conventional M-SAM.

  5. Limited experimental validation. The proposed method has not been evaluated on larger-scale language models or training tasks. Moreover, the relatively wide confidence intervals in some experiments make it difficult to establish whether the two methods are statistically equivalent.

Questions for authors
  1. Why use a fixed delay rather than adaptive Temporal Routing?

The first-order analysis suggests that the relative advantages of immediate and delayed corrections depend on gradient alignment, with potentially opposite outcomes across different tasks and training stages. Why do the authors only investigate the fixed settings of α=0\alpha=0 and α=1\alpha=1, rather than further exploring an adaptive strategy that dynamically adjusts αt\alpha_t based on DtD_t? Could adaptive routing lead to more consistent performance improvements?

  1. Can the theoretical analysis provide stronger convergence guarantees?

The current theoretical analysis primarily characterizes the first-order descent contributions at a fixed optimization state and the stability of a locally linearized system. Could the authors further establish nonconvex convergence guarantees for TF-SAM, or demonstrate that delayed corrections can enlarge the stability region under certain conditions? How does the existing theory explain the final generalization performance?

  1. Could the advantages of TF-SAM simply result from a reduced correction magnitude?

TF-SAM replaces the current correction ctc_t with βct−1\beta c_{t-1}, which changes not only the timing but also the effective magnitude and direction of the correction. Although the paper theoretically demonstrates that delayed corrections are not equivalent to simple rescaling, could the authors include additional experimental baselines, such as magnitude-matched M-SAM, scaled corrections, or random delays, to further disentangle the effects of temporal delay and correction strength?

  1. Why do first-order alignment and final performance not always agree?

On CIFAR-10/ResNet-18, the gradient alignment analysis indicates that immediate corrections provide more favorable first-order descent contributions, yet TF-SAM achieves nearly the same final accuracy. Could the authors further explain this discrepancy? Does it suggest that first-order alignment is insufficient to predict long-term optimization behavior, or that delayed corrections provide compensating benefits through improved stability or implicit regularization?

  1. Can the proposed method generalize to more complex optimization settings?

The current experiments primarily focus on classification tasks using ResNet, TinyBERT, and GPT-2. Does TF-SAM remain effective for larger-scale language models, adaptive optimizers such as AdamW, and different momentum coefficients or learning rate schedules? Does the Temporal Routing mechanism rely on the specific structure of SGD Momentum?

6 comments
AuthorsOpenReview ↗

Expanded Theory and Experiments

Answer. We thank the reviewer for the thoughtful comments and suggestions. We study correction timing as an independent dimension of momentum-based SAM. Across the evaluated settings, removing the immediate correction largely preserves performance. Across ten matched seeds, five of six paired 95% CIs include zero; GPT-2/SST-5 favors TF-SAM by +0.53 pp [0.30, 0.76] (Appendix Paired-seed statistical analysis), indicating no consistent accuracy penalty, while small effects remain unresolved.

Q1:

  • Our central contribution is the discovery and characterization of the temporal-routing mechanism. A fixed delay isolates the immediate route, allowing us to test whether an already-generated correction must act at zero lag. Adaptive routing is a subsequent refinement of this theoretical picture: it would turn the state-dependent criterion into a routing policy.

  • Appendix “Controlled temporal routing” (Fig. α\alpha-sweep and α×β\alpha\times\beta grid) evaluates 42 configurations with 10 seeds each. Accuracy varies non-monotonically with α\alpha; its span decreases from 0.50 pp at β=0\beta=0 to 0.10–0.11 pp at β=0.8\beta=0.8–0.90.9.

  • A DtD_t-guided adaptive policy is a useful next experiment, since local first-order preference need not predict full-trajectory performance. We have made the existing sweep more prominent in the revised manuscript.

Q2:

  • Appendix C now provides finite-horizon nonconvex stationarity bounds for the actual coupled iterates, including routing-weighted and same-batch results. The full-momentum bound covers all 0≤β<10\le\beta<1 under conditional second moments and Lη≤(1−β)2/16L\eta\le(1-\beta)^2/16, without pathwise momentum or noise caps; zero noise recovers deterministic updates.

  • Its PAC-Bayesian population-risk bound retains empirical risk or ball sharpness, KL complexity, confidence costs, and an explicit point-transfer charge. Assumptions and complete proofs accompany the boxed guarantees.

Q3:

  • Our claim is performance preservation after removing immediate correction, not superiority over M-SAM. Replacing ctc_t with βct−1\beta c_{t-1} can change magnitude and direction. TF-SAM removes the immediate route while retaining the full correction input to momentum; Scaled M-SAM scales both routes, applying βct\beta c_t immediately and storing it in momentum. Their summed response weights match for a fixed correction, but their lag profiles differ.

  • Sampled logs confirm active, distinct routes: in ten-seed controls at all 400 CIFAR-10 and 20 GPT-2 logged steps, both methods generated nonzero corrections, but the immediate term was nonzero only for Scaled M-SAM.

  • Paired TF-SAM minus Scaled M-SAM final-accuracy gaps were −0.038 pp [−0.133, 0.057] and −0.104 pp [−0.340, 0.132] (95% CIs). Delayed-only routing thus preserves performance comparable to a scaled two-route alternative. Accuracy and robustness results appear in Appendix Correction-response weight-matched controls.

Q4:

  • First-order alignment characterizes local descent; final accuracy reflects the full optimization trajectory and generalization. A locally favorable correction therefore need not improve final accuracy.

  • TF-SAM carries the correction through momentum, and the closed-loop analysis characterizes how this routing changes the dynamics. Across the evaluated settings, final performance is largely preserved without immediate correction.

Q5:

  • Temporal routing can be constructed beyond SGD momentum, but broader effectiveness remains unestablished.

  • Our AdamW extension uses clean gradients for the current first-moment update and second moment, storing perturbed gradients in the first-moment buffer for future updates. On CIFAR-10/ResNet-18, five matched seeds, final accuracy is 94.232 ± 0.255% for AdamW-SAM versus 94.408 ± 0.169% for TF-AdamWSAM (200 epochs, cosine schedule, (β1,β2)=(0.9,0.999)(\beta_1,\beta_2)=(0.9,0.999); mean ± SD), with paired difference +0.176 pp [−0.167, 0.519] (95% CI). This suggests feasibility; the extension also changes second-moment inputs (Appendix Temporal-feedback extension to AdamW-SAM).

  • New MultiStepLR experiments on the same task with five matched seeds (200 epochs, β=0.9\beta=0.9, LR multiplied by 0.1 after epochs 100/150) give 95.764 ± 0.096% for M-SAM versus 95.758 ± 0.106% for TF-SAM, with paired difference −0.006 pp [−0.156, 0.144]. Performance is comparable under this schedule, without establishing superiority or statistical equivalence.

  • The 42 (α,β)(\alpha,\beta) configurations with ten seeds each further probe momentum sensitivity (Q1; Appendix Controlled temporal routing).

  • Delayed feedback requires persistent optimizer state, which need not be SGD momentum. Extending the evaluation to language models larger than GPT-2 is a next step.

Dear Reviewer 4EUa,

Thank you for your valuable comments. We have addressed your questions and concerns in our previous response. We look forward to your feedback and would appreciate any further comments you may have.

Thank you for your time and consideration!

Q1. Fixed versus adaptive routing. Fixed delay tests whether a correction must act immediately while preserving its input to momentum. Appendix Controlled temporal routing reports a CIFAR-10/ResNet-18 grid of 42 (α,β)(\alpha,\beta) configurations, ten seeds each. Here α=0/1\alpha=0/1 denotes TF-SAM/M-SAM. Grid summaries (pp):

Momentum β\betaEndpoint gap A(1,β)−A(0,β)A(1,\beta)-A(0,\beta)Span across α\alpha
0.0+0.390.50
0.2+0.110.24
0.4−0.040.17
0.6+0.120.20
0.8+0.040.10
0.9+0.080.11

Sensitivity to the zero-lag route is greatest without momentum and smaller at high momentum. Intermediate values are non-monotone. Because β\beta also changes ordinary momentum, the grid measures a structural interaction. A DtD_t-adaptive policy remains unevaluated; local preference alone does not ensure better final accuracy. At α=β=0\alpha=\beta=0, neither correction route survives and the update is exactly SGD. At α=0,β>0\alpha=0,\beta>0, the generated correction enters future updates through momentum. Fixed delay isolates the necessity of immediate action while retaining the correction.

Q2. Nonconvex convergence and generalization. Appendix C now bounds stationarity at the actual coupled iterates. For an LL-smooth, lower-bounded objective, let γ=η/(1−β)\gamma=\eta/(1-\beta) and zt=wt−γβmt−1z_t=w_t-\gamma\beta m_{t-1}. Under conditional second-moment noise caps at the evaluated points, for every 0≤β<10\le\beta<1 and Lη≤(1−β)2/16L\eta\le(1-\beta)^2/16,

\le \frac{8\mathbb E\Psi_0}{\gamma T} +\frac{16}{T}\sum_{t<T}[(L\rho_t)^2+v_t^{\rm in}+v_t^{\rm out}],$$ where $\Psi_t=f(z_t)-f_*+[4L^2\gamma^3\beta^2/(1-\beta)]\|m_{t-1}\|^2$. This allows deterministic $\alpha_t\in[0,1]$, without conditional centering or pathwise momentum caps; zero noise recovers a deterministic bound. Fixed radii/noise retain residual terms; this is a finite-horizon stationarity guarantee, rather than a claim of exact convergence with fixed perturbation and noise levels. A sharper centered-noise result retains $q=\beta+\alpha(1-\beta)$ in the radius and variance charges, under $0\le\beta\le1/4$ and $L\gamma\le1/16$; a separate result treats same-batch evaluations. For the centered-noise bound, define $V_t=(1-q)^2v_t^{\rm in}+q^2v_t^{\rm out}$. Its radius charge is $8q^2(L\rho)^2$ and variance charge is $8L\gamma T^{-1}\sum_tV_t$. TF-SAM has $q=\beta$, whereas M-SAM has $q=1$. Under common caps, the displayed TF-SAM variance penalty is no larger when $(1-\beta)v^{\rm in}\le(1+\beta)v^{\rm out}$. This compares penalty terms under the stated assumptions, not entire trajectory-dependent bounds. The same-batch result requires a samplewise $L_s$-Lipschitz gradient map and centers only the original-point noise; its radius charge is $8q^2L_s^2\rho^2$. This avoids assuming a fresh, centered outer gradient after an adaptively chosen perturbation. For generalization, Appendix C separately considers IID data and bounded loss. At the sample-dependent coupled output, it bounds population risk by empirical posterior risk plus $\sqrt{[\mathrm{KL}(Q\|P_j)+\log(n/\delta_j)]/[2(n-1)]}$ and a certified point-transfer charge $\tau$. The ball-sharpness form retains posterior tail cost; priors are predeclared. These are conditional risk certificates, not a guarantee that delay improves test accuracy. Appendix *Full Temporal-Feedback Theory* also gives an exact stability test. At a frozen regular state, the general characteristic matrix polynomial is $$Q_\alpha(r)=r^2I-r[(1+\beta)I-\eta(H+\alpha C)]+\beta[I+\eta(1-\alpha)C].$$ This identity requires **no commutativity** of the clean-gradient Jacobian $H$ and correction Jacobian $C$. On a real common mode $(\lambda,\kappa)$, Schur stability is equivalent to $$\lambda+[\alpha+\beta(1-\alpha)]\kappa>0,\quad 2(1+\beta)>\eta[\lambda+((1+\beta)\alpha-\beta)\kappa],\quad 1-\beta>\beta\eta(1-\alpha)\kappa.$$ **Immediate routing enters the trace, while delayed feedback enters the determinant.** For positive correction response, removing immediate action relaxes the flip constraint but introduces a delayed-feedback constraint. The signed-response conditions specify the trade-off; neither endpoint is universally more stable. **Q4. Alignment versus final accuracy.** The diagnostic compares first-order contributions at a shared target state, omitting the quadratic remainder and subsequent trajectory changes. **A favorable immediate projection on CIFAR therefore need not imply a final-accuracy advantage.** The geometry plots compare corrections along the same observed M-SAM trajectory, rather than simulating counterfactual trajectories. Frozen-state spectra also do not determine products of time-varying Jacobians. Stability and implicit regularization are not established causal explanations.

Comments on the paper

1 comment

We thank the Area Chair and reviewers for their constructive feedback. Reviewer SC3S and Reviewer 81va recognized the paper's contribution: Reviewer SC3S valued the timing perspective and matched-seed design; Reviewer 81va highlighted the recurrence and stability analysis without commutativity. Reviewer SC3S appreciated our careful qualifications. We actively addressed the reviews with stronger theory, targeted controls, and broader experiments.

The expanded evidence supports temporal routing as a distinct dimension of SAM design: routing corrections exclusively through momentum preserves much of SAM's empirical benefit across the evaluated settings.

1. Reviewer 4EUa: stronger theory and more decisive experimental support

Reviewer 4EUa recognized the new perspective but requested stronger foundations and evidence. We added the following evidence.

  • Q1: Does the theory extend beyond local descent and stability? Re: We added nonconvex stationarity bounds for the actual coupled iterates and a separate PAC-Bayesian population-risk bound.

Appendix C includes a result covering all momentum coefficients below one under stated step-size and conditional second-moment assumptions, together with routing-weighted and same-batch results. Proofs and assumptions are explicit; one-step alignment is distinguished from final generalization.

  • Q2: Could reduced correction strength explain the empirical results? Re: The requested weight-matched control supports performance preservation with delayed-only routing, even when summed correction-response weight is matched.

We added Scaled M-SAM on CIFAR-10/ResNet-18 and GPT-2/SST-5 with ten matched seeds per task, holding momentum and perturbation radius fixed. TF-SAM-minus-Scaled final-accuracy gaps are −0.038 pp [−0.133, 0.057] and −0.104 pp [−0.340, 0.132], respectively (paired 95% CIs). Logs verify active correction inputs to TF-SAM's momentum buffer with zero immediate contribution. The methods have distinct temporal responses; the control establishes neither statistical equivalence nor a timing-specific cause of the GPT-2 gain.

  • Q3: Is fixed routing sufficiently explored, and does the construction extend beyond the original training setup? Re: We broadened the evaluation with 42 routing/momentum configurations, ten seeds each, plus AdamW-SAM and MultiStepLR experiments.

The grid tests intermediate routing choices and shows smaller accuracy sensitivity at high momentum. Five-seed AdamW-SAM and MultiStepLR comparisons give paired final-accuracy gaps of +0.176 pp and −0.006 pp, with both 95% CIs spanning zero. Fixed routing isolates immediate action; adaptive routing and larger language models remain future extensions. AdamW also changes second-moment inputs. The contribution is temporal routing; SAM's two-gradient computation is retained.

These additions substantially address Reviewer 4EUa's central theoretical and experimental concerns through coupled-iterate guarantees, correction-weight controls, and broader evaluation.

2. Reviewer SC3S: consistent reporting and controlled mechanism measurements

  • Q1: Why do aggregate robustness results disagree with the detailed tables? Re: We corrected an outdated four-seed summary using the same ten matched seeds, equal condition weights, and seed-level paired CIs throughout.

Corrected results show no consistent aggregate robustness penalty. Records and the recalculation script are in the anonymous repository.

  • Q2: How do clipping and dropout affect GPT-2 measurements? Re: New raw/clipped-gradient and matched/independent-dropout probes isolate their effects and sharpen the diagnostic's interpretation.

At ten final checkpoints, zero-radius controls show that independent dropout alone generates a correction, which matched masks eliminate. At nonzero radius, the matched-mask raw projection is near zero while clipping makes it negative. We distinguish implemented stochastic-update alignment from a perturbation-only explanation. Reviewer SC3S's correction-weight concern is addressed by the Scaled M-SAM experiments above.

3. Reviewer 81va: persistence at larger batches

  • Q: Do routing preferences persist with larger batches despite shared-minibatch dependence? Re: Yes, at the tested larger batches on both optimizer trajectories for the gradients used in training.

We completed 60 runs, testing CIFAR batches 256/512 and GPT-2 batch 32 with ten seeds per method and setting. CIFAR continues to favor immediate routing and GPT-2 delayed routing; the corresponding seed-level 95% CIs exclude zero. We explicitly clarify that this is a stochastic-update diagnostic: larger batches establish persistence under the shared-target protocol, without eliminating minibatch dependence or establishing population-gradient alignment.

Best regards,

The Authors of Paper 43726.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.