- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes Alignment-Gated Momentum (AGAM), which uses the alignment between the current gradient and the optimizer's proposed update to control how much momentum contributes to each step. The stored momentum recurrence is unchanged, and a separate gate is computed for each parameter tensor. The method is applied to SGDM, AdamW, and Lion, and evaluated on image classification, MAE pre-training, and language-model fine-tuning.
Critical strengths
- Simplicity.
The method is simple and easy to implement. It requires no additional optimizer state or tuning parameters and can be applied to several existing optimizers.
- A genuinely interesting problem.
The paper addresses an interesting gap and complements recent momentum-gradient alignment based optimizers. Existing methods that include a momentum-gradient mixture in their update (QHM, Lion, Ademamix) use a fixed or scheduled mixture coefficient throughout training. AGAM makes this mixture dynamic at every step and layer. Although the cosine-based gate is heuristic, it is consistent with recent work that uses agreement between the gradient and the momentum orfull update to improve optimizers, such as Cautious Optimizer. AGAM complements these methods by acting on the gradient–momentum mixture in the parameter update, rather than on individual coordinates, the stored momentum or the effective learning rate.
Critical weaknesses
- The contribution is not properly positioned or isolated.
Gradient–momentum interpolation in the update is a known algorithmic component of QHM/QHAdam (Ma & Yarats, 2019), with simillar mixing constructions in Lion, AdEMAMix (Pagliardini et al., 2024) and Simplified AdEMAMix (Morwani et al., 2025). AGAM extends this design by choosing a layer-wise mixing coefficient that depends on the momentum-gradient alignment. Unless the authors demonstrate that this particular adaptation adds to the existing update family either theoretically or empirically, this contribution is incremental. The paper's positioning and experimental protocol do not address that central question.
The taxonomy should focus on how each method introduces adaptivity. AM and TAM change the way the momentum buffer update is performed, not the parameter update. Cautious modifies the effective coordinate-wise learning rate through masking and rescaling. AGAM turns the interpolation coefficient adaptive. These are three different knobs of the optimizer, therefore, section 2.3's claim that AM, TAM, and AGAM are "practically mutually exclusive" is not well supported.
The authors acknowledge the strong similarity between AGAM and QHM, yet do not compare against tuned fixed-coefficient QHM/QHAdam or constant-gate AGAM. Those are essential baselines, since improvements over the current optimizers do not establish that the proposed adaptive coefficient is what drives the performance gain. Is AGAM better than momentum SGD because of the additional momentum-gradient interpolation or because of the adaptivity in its coefficient? Cautious, AdEMAMix, and Simplified AdEMAMix would provide nice additional context. The missing fixed-mixture baselines leave the the paper's central algorithmic contribution unestablished.
- The empirical and theoretical analyses do not establish the value of the adaptation.
The first point is reinforced by the fact that figures 4, 5, and 22 show stable average gates for substantial portions of training. This doesn't mean that the gates themselves are constant or that their fluctuations don't matter, but it makes a comparison against the fixed mixture baselines even more important. The global versus per-layer ablation also needs careful treatment. Replacing layer-wise gating with a global gate reduces CIFAR-100 accuracy, but on MAE pre-training the difference is within seed variability.
In terms of theory, Theorem D.6 provides a stationarity guarantee under the stated assumptions, but the proof uses generic bounds on the mixing coefficients rather than the specific cosine rule. Even then, the authors acknowledge that their results do not establish theoretical improvements and that the gains are better shown empirically. I understand that showing theoretical gains for such a scheme is hard and I do not consider the absence of such a result a weakness on its own. However, this makes supporting the contribution on the empirical evaluation even more important. Something not achieved currently.
Questions for authors
-
How does AGAM compare against tuned QHM/QHAdam or constant-gate AGAM? In the heavy-ball case, can the authors separate the effect of changing the mixture from the change in the effective learning rate?
-
Given the fairly stable average gates in Figures 4, 5, and 22, how much gain remains when using a fixed gate or a predefined gate schedule?
-
What is the justification for claiming that AGAM and adaptive momentum-state methods are mutually exclusive?
-
Did you also try coordinate wise masking at the gradient-momentum mixture level? Essentially a combination of AGAM and cautious masking. Given your mention of Cautious I was surprised to not see it.
3 comments
Part 2/3: Stable average gates (W2, Q2), gate granularity and theory (W2)
W2/Q2. Stable average gates. We agree that the average gates are stable: the model-mean gate stays between about 0.70 and 0.79 over an entire run (Figures 10 and 12), which is why the fixed-gate comparison of Part 1 matters. This mean, however, pools every tensor and step of an epoch, and the gates behind it are far from stable (the submission's Figures 4, 5, 22 are now 10, 11, 28):
- Across tensor-steps. On ResNet-50, the per-epoch minimum gate of the normalization parameters is near 0.03 in the first epochs without warmup and between 0.1 and 0.4 for most of training, so individual steps are gated close to plain SGD even where the mean sits near 0.72 (Figure 16). On MAE, 30–50% of the tensor-steps in every group are misaligned () between epochs 2 and 10 (Figure 26).
- Across parameter groups. On MAE, the weights stay at about 0.62–0.67 after warmup while the biases and special tokens rise to 0.74–0.85, so the weights and biases of a single module receive different amounts of momentum (Figure 24). On ResNet-18, late in cosine decay, the classifier bias is misaligned on 13.0% of its steps and the convolutional weights on 0.3% (Appendix O.1).
- Across depth. On ResNet-50 without warmup, the stem and first stage start at 0.63–0.67 while the third stage starts above the reference level of 0.718, and the pattern reverses within about eight epochs (Figure 15). Over the first 40 epochs, the misalignment rate ranges from 0.15% to 3.9% across convolutional blocks and reaches 10% for the normalization parameters of the last block (Figure 20).
- Over training. On MAE, every group drops to 0.53–0.58 in the second epoch and recovers over the warmup, and the biases later fall from 0.74–0.85 back below 0.718 over the last hundred epochs (Figure 24). On ResNet-50 without warmup, the first five epochs push the convolutional weights up to 0.84 and the normalization parameters down to about 0.67, and the ordering reverses by epoch 10 (Figure 13).
How much gain remains with a fixed gate? Part 1 gives the answer: the mixture's share remains, and removing the variation costs 0.28–0.55 points on CIFAR-100, 1.26–1.56 exact-match points on GSM8K, 0.55–2.21 linear-probe points on MAE, and stability at . A fixed gate is also a hyperparameter. On CIFAR-100 at , give 77.98, 78.45, 78.75, and 78.31, against 79.03 for AGAM (new Figure 9), and across our experiments a wrong value lowers accuracy by up to 1.82 points. The best value usually coincides with AGAM's own mean gate, which is not known in advance and ranges from 0.65 to 0.815 across our settings.
Predefined schedule. We did not run one. Because the model-mean gate stays between about 0.70 and 0.79, a schedule that replays it remains close to the fixed gate above. Such a schedule would also have to be found anew for each setting, because the gate's trajectory changes with warmup and with the learning-rate schedule (Figures 10 and 13). Finally, like a fixed gate, a schedule is blind to alignment: it cannot close on the conflicting tensor-steps while staying open on the others, which is the property behind the certificate (Theory, below).
W2. Model-wide versus tensor-wise gate. We agree that the three-seed MAE probe difference was within seed variability, and the revised Appendix N says so. We added five pre-training seeds. Over all eight, the tensor-wise gate reaches 29.70 ± 0.13 against 29.24 ± 0.25, and paired by seed it outperforms the model-wide gate in seven (Table 7). On CIFAR-100 the model-wide gate costs 0.55 points (Table 6). Figure 28 indicates why it falls short on MAE: computed at the same steps, a model-wide gate stays at 0.53–0.55, below the 10th percentile of the per-tensor gates from about epoch 12 onward.
W2. Theory. We agree that Theorem D.6 (D.7 in the revision) holds for any gate in and cannot separate AGAM from a fixed mixture, and the revision now says so. The descent certificate of Lemma D.3 does separate them, because it uses the cosine rule itself. With and , a fixed gate has the alignment score , which is positive at every angle exactly when . This is 0.37 at . The cosine gate keeps the score positive at every angle for all . A fixed gate reaches this radius only if , which removes most of the momentum from every step, whereas AGAM keeps a mean gate of 0.73 because its gate is small exactly when (new Remark D.4 and Figure 4). The telemetry agrees: on ResNet-50, SGDM's applied update opposes the gradient on up to 18% of tensor-steps, whereas AGAM's rate is exactly zero in most epochs (Figure 21). We do not claim a faster rate.
Part 3/3: Positioning (W1), taxonomy (W1, Q3), AGAM + Cautious (Q4)
W1. Positioning. We agree that gradient–momentum interpolation is know, and the revised Section 2.2 expands AGAM's positioning within this family: it adds AdEMAMix and Simplified AdEMAMix, as the reviewer suggested, and states how each method sets its mixing coefficients. In all of them, the coefficients are hyperparameters or prescribed schedules chosen per setting, and none uses an alignment signal:
- QHM/QHAdam fix for every parameter and step, and Ma & Yarats sweep over a grid.
- Lion applies a fixed before the sign, which AGAM-Lion makes adaptive.
- AdEMAMix adds a second, slower EMA of the gradient under scheduled coefficients.
- Simplified AdEMAMix instead adds the current gradient with a fixed weight while scheduling , whose reported optimum moves from near 100 at a small batch size to near 0.05 at a large one.
AGAM sets the coefficient per tensor and per step from the measured alignment, with no added hyperparameter or state. It also operates directly on its base optimizer's settings: in every AdamW and Lion experiment, AGAM's best configuration coincides with its base optimizer's (Section 4.3). Ma & Yarats name an effective adapter for and as an open direction, and AGAM realizes one for . That this adaptation adds to the family is what Part 1 establishes empirically and Remark D.4 theoretically. We did not run the AdEMAMix variants. Within this family, the fixed gate is the control that isolates adaptivity, because it differs from AGAM only in the alignment signal.
W1/Q3. Taxonomy and "mutually exclusive". We agree that the phrase overstated our claim, and the revision replaces it. We also agree with the reviewer's three knobs, and Figure 1 and the revised Section 2.3 organize the methods this way: TAM and AM act inside the stored momentum recurrence, AGAM acts on the mixture of the applied step alone, and Cautious acts on the coordinates of an already formed update. Composition is well defined, as the reviewer notes, and we narrow our claim. TAM, AM, and AGAM all decide how the history and the gradient are weighed when they disagree, and stacking two of these decisions is not additive (new Appendix C.1):
- TAM + AGAM is destructive. TAM trusts the history: under sustained opposition it damps the incoming gradient toward zero and proposes a step along . AGAM trusts the gradient: its gate on that proposal closes on . Both terms are suppressed, and the step vanishes when a correction matters most. At and from 0.5 to 3, the composite step has norm 0.02–0.11 in units of , against 0.52–0.73 for AGAM alone.
- AM + AGAM is redundant. AM lowers its memory coefficient under conflict and stores the result. For and from 0.5 to 3, its stored direction already has a cosine of at least 0.75 with the gradient, so AGAM's gate is at least 0.87 and has almost nothing left to act on.
They are alternative designs for the same decision, which is why we benchmark them against each other (revised "Choice of baselines").
Q4. AGAM + Cautious. Thank you for the suggestion. We have not trained this combination, or Cautious alone, on our benchmarks. A well-defined construction computes AGAM's gate from the unmasked base proposal and then applies the Cautious mask and rescaling to the gated update. How the two stages interact deserves its own empirical and theoretical analysis. The revised Section 2.3 notes that an update-reweighting rule can be applied on top of AGAM, states that we have not evaluated such combinations, and leaves them to future work.
We thank the reviewer again for the detailed and expert feedback, which prompted the fixed-gate controls, the certificate comparison, and the refined expanded positioning. We hope these additions establish the value of the adaptive gate and resolve the reviewer's concerns about how the contribution is isolated and positioned.
We thank the reviewer for the careful and constructive review, and for the insightful questions that made the paper more precise. We appreciate that the reviewer finds the problem interesting and sees AGAM as complementing recent alignment-based optimizers. The review's central question is whether the adaptive coefficient, and not the gradient–momentum mixture itself, drives the gains. We agree that the submission did not isolate this. The revision now does, empirically and theoretically.
Part 1/3: Mixture or adaptation? (W1, Q1)
W1/Q1. Fixed gate, tuned QHM/QHAdam, and the effective learning rate. Thank you for this question. We agree that the constant gate is the right ablation of AGAM and that it was missing. We ran it together with QHM and matched-step SGDM (general response above, Tables R1–R3; new Appendix N.1).
What a fixed gate is. With a fixed gate c, AGAM-SGD is exactly QHM with ν=c/(1−β+βc) and learning rate η(1−β+βc)/(1−β) (Remark D.2), and AGAM-AdamW takes the QHAdam form with ν2=1 and ν1→c (Section 3.3). In every experiment (SGDM, AdamW, and Lion), we searched c∈{0.2,0.5,0.9,cˉ}, where cˉ is AGAM's own mean gate in the same training setting (e.g., cˉ=0.73 on CIFAR-100), and every fixed-gate result we report is the best of these four gates. On CIFAR-100, every gate was run at every learning rate, which amounts to tuning QHM over ν∈{0.71,0.91,0.96,0.99} at β=0.9. QHM's recommended (ν,β)=(0.7,0.999) uses a different β, so we ran it separately on the same learning-rate grid.
Mixture versus effective learning rate. We controlled for it. By Remark D.2, a fixed gate c is QHM at a learning rate 1−β+βc times that of SGDM (0.757 at c=0.73). SGDM at 0.757η is therefore QHM with ν=1 at the fixed gate's learning rate: it places the same total weight on the gradients without shifting weight toward the current one, matching the coefficients. Each control is tuned on the same learning-rate grid and reported at its best (Table R1), and AGAM's gain over SGDM on CIFAR-100 splits as follows:
So both contribute, and in different regimes. The smaller step accounts for nearly most of the fixed gate's gain at a constant learning rate and for none of it with cosine decay, where the gain appears only once the mixture is added. Lion offers a comparison in which the step size cannot change: the update is a sign, so the gate changes its direction and not its step size (nonzero coordinates aside). At the same learning rate and weight decay (Section 4.3), AGAM-Lion still gains 1.77 exact-match points over Lion on GSM8K (Table 3). Adaptivity then adds a consistent margin over the best fixed gate across all tasks (Tables R1–R3). On CIFAR-100, it accounts for roughly half of AGAM's gain over SGDM with cosine decay and a third at a constant learning rate. AGAM also reaches the validation target sooner than the fixed gate in 8 of the 10 cells where both reach it.
Where adaptivity matters most (Table R2). The margin grows when training is less stable. AGAM leads the fixed gate by 0.48 points at η=0.4 on CIFAR-100 and by 0.49 at η=0.8 on CIFAR-10. At B=4096, the fixed gate finishes below 60% in three of eight seeds and collapses to 7.4% in one, trailing AGAM by 11.47 points.
Beyond SGDM (Table R3). Fixed-gate AdamW (the QHAdam control) and fixed-gate Lion, each at its best gate, trail AGAM by 1.26 and 1.56 exact-match points on GSM8K, and by 0.55 and 2.21 linear-probe points on MAE pre-training. On the larger Qwen2.5-7B (new Table 5), the best fixed gate trails AGAM-AdamW by 0.71 exact-match points and falls below plain AdamW (77.53 against 77.63).
QHM at its recommended setting reaches 76.17 ± 0.13 with cosine decay and 68.22 ± 1.46 at a constant learning rate, against 79.03 ± 0.15 and 70.83 ± 0.43 for AGAM.