- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether sharpness corrections in momentum-based SAM must affect the current update. TF-SAM removes the immediate correction while retaining its influence through momentum. The paper provides recurrence, alignment, and local stability analyses, with experiments across six image and text settings using ten random seeds.
Critical strengths
The recurrence and stability analyses clarify the roles of immediate and historical corrections without requiring the Hessian and correction response to commute.
Critical weaknesses
The immediate correction and evaluation gradient share a minibatch, unlike the delayed correction, potentially confounding the alignment diagnostics.
Questions for authors
Do the routing preferences observed on CIFAR and GPT-2 persist when with bigger batchsize?
4 comments
Dear Reviewer 81va,
We would like to express our sincere gratitude for your insightful comments and valuable feedback. We believe that our responses have sufficiently clarified your questions and concerns. We highly value your suggestions and welcome any further comments or questions you may have.
Thank you once again for your careful review and constructive feedback!
Q1. Yes, the qualitative routing preferences persist at the larger batch sizes tested, most consistently for the gradient representation used by the optimizer. We completed 60 training runs, covering CIFAR-10/ResNet-18 at batch sizes 256 and 512 and GPT-2/SST-5 at batch size 32, with ten seeds for each method and setting. The main configurations use batches 128 and 16, respectively. The new results therefore test two- and fourfold increases for CIFAR and a twofold increase for GPT-2.
| Task | Batch | Trajectories | Seeds per method | Epochs | Momentum |
|---|---|---|---|---|---|
| CIFAR-10 / ResNet-18 | 256 | M-SAM, TF-SAM | 10 | 200 | 0.9 |
| CIFAR-10 / ResNet-18 | 512 | M-SAM, TF-SAM | 10 | 200 | 0.9 |
| GPT-2 / SST-5 | 32 | M-SAM, TF-SAM | 10 | 10 | 0.8 |
All settings use . M-SAM and TF-SAM correspond to immediate-routing weights and . We measure both methods' own trajectories, so the observed sign is not restricted to states visited by M-SAM.
At target step , we compare the current correction and the exactly one-step-old correction against the same current training-minibatch gradient :
q_t^{\rm delay}=\frac{\langle c_{t-1},g_t\rangle}{\|g_t\|^2},\quad D_t=\beta q_t^{\rm delay}-q_t^{\rm imm}.$$ **Negative $D$ favors immediate routing; positive $D$ favors delayed routing**, at the local first-order level. The following estimates average per-step ratios within seeds, then average the ten seed means. Intervals are two-sided 95% Student-$t$ CIs across seeds (nine degrees of freedom). **CIFAR-10: immediate routing remains preferred at both larger batches.** Gradients are unclipped. | Batch | Trajectory | Immediate $q$ | Weighted delayed $\beta q$ | $D$ [95% CI] | Negative seeds | | --- | --- | ---: | ---: | --- | ---: | | 256 | M-SAM | 2.989 | −0.031 | **−3.020 [−3.239, −2.801]** | 10/10 | | 256 | TF-SAM | 2.907 | 0.028 | **−2.879 [−3.167, −2.591]** | 10/10 | | 512 | M-SAM | 3.405 | 0.037 | **−3.368 [−3.621, −3.114]** | 10/10 | | 512 | TF-SAM | 3.342 | 0.061 | **−3.281 [−3.511, −3.052]** | 10/10 | All four seed-level intervals lie below zero. Immediate projections remain positive, while momentum-weighted delayed projections are near zero. Across the 400 sampled target steps per setting and method, 99.25% favor immediate routing on M-SAM trajectories at each batch size and 98.75% on TF-SAM trajectories. These pooled fractions are descriptive; the CI uses ten independent seeds rather than 400 steps. **GPT-2: delayed routing remains preferred for the clipped gradients used in training.** We also report the corresponding raw-gradient measurements. | Trajectory | Representation | Immediate $q$ | Weighted delayed $\beta q$ | $D$ [95% CI] | Positive seeds | | --- | --- | ---: | ---: | --- | ---: | | M-SAM | Clipped | −0.480 | −0.019 | **0.462 [0.377, 0.546]** | 10/10 | | TF-SAM | Clipped | −0.469 | 0.002 | **0.471 [0.394, 0.547]** | 10/10 | | M-SAM | Raw | −0.262 | 0.009 | 0.271 [0.114, 0.428] | 9/10 | | TF-SAM | Raw | −0.157 | 0.009 | 0.166 [−0.024, 0.356] | 7/10 | All 50/50 sampled clipped-gradient differences per method are positive. Raw-gradient differences are positive at 82%/76% of sampled steps on M-SAM/TF-SAM trajectories. **The raw TF-SAM interval includes zero**, so the evidence is more consistent in the clipped representation than before clipping. Displayed projections and $D$ are rounded independently. Diagnostics use the first minibatch every five CIFAR epochs (40 target steps per seed) and every two GPT-2 epochs (five per seed). The delayed correction is from the immediately preceding optimizer step, not from the preceding logging event. For GPT-2, each correction is formed from the perturbed-minus-clean gradients in the same representation as its target gradient. The original phase-wise CIFAR results have mixed early signs and favor immediate routing in the middle and late phases. GPT-2 favors delay across the reported phases, with an adverse immediate projection and a delayed projection near zero after the early phase. The larger-batch results reproduce that qualitative regime distinction in the sampled-update averages. Neither summary asserts that every CIFAR step favors immediacy or that raw GPT-2 alignment always favors delay. These results are included in Appendix *Routing preferences at larger batch sizes*, with the protocol and seed-level uncertainty. **The CIFAR/GPT-2 sign distinction persists at the tested larger batches on both optimizer trajectories.** This answers the batch-size question for the implemented stochastic-update diagnostic; it does not turn that diagnostic into an independent population-gradient measurement or establish an accuracy advantage from its local sign.W1. The empirical diagnostic measures stochastic-update alignment, not an unbiased population-gradient alignment. The immediate correction and evaluation gradient use the same minibatch, whereas comes from the preceding minibatch and parameter state. We now state this distinction explicitly in the Discussion paragraph beginning “These conclusions are limited.” Larger batches test persistence under that measurement protocol; they do not remove the asymmetry in data sharing.
The comparison holds the target state, current minibatch, target gradient, and normalization fixed. It asks which already-generated correction has the more favorable projection onto that particular update's clean gradient. This target matching avoids comparing projections against two different target gradients, but the two corrections still differ in their generation time and minibatch dependence.
| Quantity held fixed | What remains different |
|---|---|
| Current parameter state | was generated at the preceding state |
| Target minibatch and gradient | Only shares the target minibatch |
| Denominator | Corrections may differ in direction and magnitude |
| Target optimizer step | One correction is fresh, the other exactly one step old |
The theoretical population criterion and the realized minibatch diagnostic have distinct roles. In the conditional one-step analysis, the derivative of correction alignment with respect to is , where the alignments use the current population gradient conditioned on the history. In the experiment, substitutes the observed training-minibatch gradient. Its sign describes the sampled update; it is not an unbiased estimate of that population criterion. The manuscript explicitly calls these realized measurements descriptive diagnostics.
Aggregation also matters when comparing the original figures with the new batch-size table. The original phase-wise protocol divides training into epoch thirds, uses exactly contiguous one-step pairs, discards non-contiguous pairs, and contributes 49 valid targets per seed per phase. Within each phase it uses a ratio of sums. For the immediate projection,
This is an energy-weighted average of the per-step normalized projections. The larger-batch experiment instead takes the arithmetic mean of the sampled per-step ratios within each seed. These are different summaries, so magnitudes should not be read as a direct batch-size effect across the two protocols. Both analyses treat seeds as the inferential units; individual steps within a run do not supply independent replicates.
Clipping is part of the GPT-2 measurement target. With clean and perturbed gradients both clipped to unit norm,
Thus, a negative clipped immediate projection by itself cannot identify an adverse unprocessed sharpness correction. The batch-32 results report both representations: raw is positive on average for both trajectories, but its TF-SAM CI includes zero; clipped is positive in all ten seeds for both methods. Our strongest directly supported statement therefore concerns the gradients actually used by the optimizer.
Additional final-checkpoint GPT-2 controls use ten M-SAM seeds and a deterministic training minibatch per seed. They compare independent and matched dropout masks at zero and nonzero radius:
| Masks | Radius | Raw immediate | Clipped immediate |
|---|---|---|---|
| Independent | 0 | −0.469 | −0.472 |
| Independent | 0.05 | −0.410 | −0.471 |
| Matched | 0 | 0 | 0 |
| Matched | 0.05 | +0.015 | −0.109 |
Independent dropout can produce a nonzero difference without any parameter perturbation. With matched masks, the perturbation effect is +0.015 [−0.061, 0.091] before clipping and −0.109 [−0.134, −0.084] after clipping. These probes isolate dropout and clipping at final checkpoints; they are not an independent-target gradient evaluation and do not establish the perturbation-only sign throughout training.
An independent holdout-gradient diagnostic has not been evaluated here. The new larger-batch evidence instead shows that the observed ordering survives the tested batch increases under the original shared-target construction. Local alignment, population descent, and final generalization remain distinct claims: the plots omit higher-order terms and do not simulate counterfactual training trajectories. Accordingly, comparable final accuracy on CIFAR is compatible with a favorable immediate local projection, and the diagnostic alone does not causally explain final accuracy gaps.
Larger Batch Routing
Answer. We thank the reviewer for identifying the minibatch-dependence issue and suggesting larger-batch experiments. We evaluated CIFAR-10/ResNet-18 at batch sizes 256 and 512 and GPT-2/SST-5 at batch size 32, with ten seeds per method and setting. The results preserve the qualitative routing preferences at these batch sizes, with the qualification below concerning the diagnostic target.
Q1:
Yes, the qualitative preferences persist at the larger batch sizes tested.
At ρ=0.05, CIFAR-10 retains negative Dt, favoring immediate routing, on both M-SAM and TF-SAM trajectories. Mean Dt is −3.020/−2.879 at batch 256 and −3.368/−3.281 at batch 512 (M-SAM/TF-SAM); all ten seed means are negative in each setting. The corresponding 95% CIs are [−3.239, −2.801]/[−3.167, −2.591] at batch 256 and [−3.621, −3.114]/[−3.511, −3.052] at batch 512. Mean immediate projections are 2.989/2.907 and 3.405/3.342, respectively, versus momentum-weighted delayed projections of −0.031/0.028 and 0.037/0.061 (M-SAM/TF-SAM throughout). Across sampled steps, 99.25% favor immediate routing on M-SAM trajectories at each batch size and 98.75% on TF-SAM trajectories.
GPT-2 at batch 32 retains positive Dt, favoring delayed routing, for the clipped gradients used by the optimizer: 0.462 [0.377, 0.546] on M-SAM trajectories and 0.471 [0.394, 0.547] on TF-SAM trajectories (mean [95% CI]), with positive means in 10/10 seeds for each method. All 50/50 sampled clipped-gradient differences per method are positive. Mean immediate projections are −0.480/−0.469, versus momentum-weighted delayed projections of −0.019/0.002 (M-SAM/TF-SAM).
Before clipping, the corresponding estimates are 0.271 [0.114, 0.428] and 0.166 [−0.024, 0.356]; positive seed means occur in 9/10 and 7/10 seeds, respectively, with positive differences at 82%/76% of sampled steps.
All 60 runs are complete. CIFAR runs use 200 epochs and β=0.9; GPT-2 runs use 10 epochs and β=0.8. Diagnostics sample 40 target steps per CIFAR run and five per GPT-2 run, using the first minibatch every five and two epochs, respectively, with an exactly one-step-old correction. We average per-step ratios within each seed, then across ten seeds; 95% Student-t CIs use seeds as the independent units (nine degrees of freedom).
These experiments support persistence of the observed CIFAR/GPT-2 distinction at the tested larger batch sizes, most clearly for the gradients actually used in training. We have added these results, the measurement protocol, and seed-level confidence intervals to the revised paper PDF in Appendix Routing preferences at larger batch sizes.
W1 (shared-minibatch alignment):
The diagnostic compares both corrections against gt: ct shares its minibatch, while ct−1 comes from the preceding minibatch. It measures alignment for the stochastic update rather than against an independent population-gradient estimate.
This distinction is now explicit in the Discussion paragraph beginning “These conclusions are limited.” The larger-batch results above show that the observed signs persist under the same protocol.