acceptodds
Under review as a conference paper at ICLR 2027

AGAM: Alignment-Gated Momentum for Deep Learning Optimization

Abstract

Momentum-based optimizers underpin modern deep learning training, relying on accumulated past gradients to provide a reliable descent direction. However, this expectation is routinely violated in practice due to the non-stationarity of mini-batch gradients, causing a base optimizer’s proposed update direction to become misaligned with the current gradient. We introduce , a state-preserving, drop-in gating mechanism for momentum-based optimizers, implemented as a modification with minimal computational overhead and no additional hyperparameters. AGAM transiently attenuates the contribution of historical momentum via a tensor-wise, alignment-adaptive gating coefficient. We instantiate AGAM with SGDM, AdamW, and Lion, characterize their exact relationship to quasi-hyperbolic optimization, and establish a sharp descent certificate for AGAM-SGD. Empirical evaluations across ResNet image classification, MAE ViT pre-training, and LLM fine-tuning demonstrate that AGAM-augmented optimizers consistently outperform both their base optimizers and alignment-aware baselines.

Primary area:
optimization
Keywords:
deep learning optimization, stochastic optimization, momentum, adaptive methods, gradient-update alignment, momentum gating

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HtX8OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Alignment-Gated Momentum (AGAM), which uses the alignment between the current gradient and the optimizer's proposed update to control how much momentum contributes to each step. The stored momentum recurrence is unchanged, and a separate gate is computed for each parameter tensor. The method is applied to SGDM, AdamW, and Lion, and evaluated on image classification, MAE pre-training, and language-model fine-tuning.

Critical strengths
  1. Simplicity.

The method is simple and easy to implement. It requires no additional optimizer state or tuning parameters and can be applied to several existing optimizers.

  1. A genuinely interesting problem.

The paper addresses an interesting gap and complements recent momentum-gradient alignment based optimizers. Existing methods that include a momentum-gradient mixture in their update (QHM, Lion, Ademamix) use a fixed or scheduled mixture coefficient throughout training. AGAM makes this mixture dynamic at every step and layer. Although the cosine-based gate is heuristic, it is consistent with recent work that uses agreement between the gradient and the momentum orfull update to improve optimizers, such as Cautious Optimizer. AGAM complements these methods by acting on the gradient–momentum mixture in the parameter update, rather than on individual coordinates, the stored momentum or the effective learning rate.

Critical weaknesses
  1. The contribution is not properly positioned or isolated.

Gradient–momentum interpolation in the update is a known algorithmic component of QHM/QHAdam (Ma & Yarats, 2019), with simillar mixing constructions in Lion, AdEMAMix (Pagliardini et al., 2024) and Simplified AdEMAMix (Morwani et al., 2025). AGAM extends this design by choosing a layer-wise mixing coefficient that depends on the momentum-gradient alignment. Unless the authors demonstrate that this particular adaptation adds to the existing update family either theoretically or empirically, this contribution is incremental. The paper's positioning and experimental protocol do not address that central question.

The taxonomy should focus on how each method introduces adaptivity. AM and TAM change the way the momentum buffer update is performed, not the parameter update. Cautious modifies the effective coordinate-wise learning rate through masking and rescaling. AGAM turns the interpolation coefficient adaptive. These are three different knobs of the optimizer, therefore, section 2.3's claim that AM, TAM, and AGAM are "practically mutually exclusive" is not well supported.

The authors acknowledge the strong similarity between AGAM and QHM, yet do not compare against tuned fixed-coefficient QHM/QHAdam or constant-gate AGAM. Those are essential baselines, since improvements over the current optimizers do not establish that the proposed adaptive coefficient is what drives the performance gain. Is AGAM better than momentum SGD because of the additional momentum-gradient interpolation or because of the adaptivity in its coefficient? Cautious, AdEMAMix, and Simplified AdEMAMix would provide nice additional context. The missing fixed-mixture baselines leave the the paper's central algorithmic contribution unestablished.

  1. The empirical and theoretical analyses do not establish the value of the adaptation.

The first point is reinforced by the fact that figures 4, 5, and 22 show stable average gates for substantial portions of training. This doesn't mean that the gates themselves are constant or that their fluctuations don't matter, but it makes a comparison against the fixed mixture baselines even more important. The global versus per-layer ablation also needs careful treatment. Replacing layer-wise gating with a global gate reduces CIFAR-100 accuracy, but on MAE pre-training the difference is within seed variability.

In terms of theory, Theorem D.6 provides a stationarity guarantee under the stated assumptions, but the proof uses generic bounds on the mixing coefficients rather than the specific cosine rule. Even then, the authors acknowledge that their results do not establish theoretical improvements and that the gains are better shown empirically. I understand that showing theoretical gains for such a scheme is hard and I do not consider the absence of such a result a weakness on its own. However, this makes supporting the contribution on the empirical evaluation even more important. Something not achieved currently.

Questions for authors
  1. How does AGAM compare against tuned QHM/QHAdam or constant-gate AGAM? In the heavy-ball case, can the authors separate the effect of changing the mixture from the change in the effective learning rate?

  2. Given the fairly stable average gates in Figures 4, 5, and 22, how much gain remains when using a fixed gate or a predefined gate schedule?

  3. What is the justification for claiming that AGAM and adaptive momentum-state methods are mutually exclusive?

  4. Did you also try coordinate wise masking at the gradient-momentum mixture level? Essentially a combination of AGAM and cautious masking. Given your mention of Cautious I was surprised to not see it.

3 comments

We thank the reviewer for the careful and constructive review, and for the insightful questions that made the paper more precise. We appreciate that the reviewer finds the problem interesting and sees AGAM as complementing recent alignment-based optimizers. The review's central question is whether the adaptive coefficient, and not the gradient–momentum mixture itself, drives the gains. We agree that the submission did not isolate this. The revision now does, empirically and theoretically.

Part 1/3: Mixture or adaptation? (W1, Q1)

W1/Q1. Fixed gate, tuned QHM/QHAdam, and the effective learning rate. Thank you for this question. We agree that the constant gate is the right ablation of AGAM and that it was missing. We ran it together with QHM and matched-step SGDM (general response above, Tables R1–R3; new Appendix N.1).

What a fixed gate is. With a fixed gate cc, AGAM-SGD is exactly QHM with ν=c/(1−β+βc)\nu = c/(1-\beta+\beta c) and learning rate η(1−β+βc)/(1−β)\eta(1-\beta+\beta c)/(1-\beta) (Remark D.2), and AGAM-AdamW takes the QHAdam form with ν2=1\nu_2 = 1 and ν1→c\nu_1 \to c (Section 3.3). In every experiment (SGDM, AdamW, and Lion), we searched c∈{0.2,0.5,0.9,cˉ}c \in \{0.2, 0.5, 0.9, \bar c\}, where cˉ\bar c is AGAM's own mean gate in the same training setting (e.g., cˉ=0.73\bar c = 0.73 on CIFAR-100), and every fixed-gate result we report is the best of these four gates. On CIFAR-100, every gate was run at every learning rate, which amounts to tuning QHM over ν∈{0.71,0.91,0.96,0.99}\nu \in \{0.71, 0.91, 0.96, 0.99\} at β=0.9\beta = 0.9. QHM's recommended (ν,β)=(0.7,0.999)(\nu, \beta) = (0.7, 0.999) uses a different β\beta, so we ran it separately on the same learning-rate grid.

Mixture versus effective learning rate. We controlled for it. By Remark D.2, a fixed gate cc is QHM at a learning rate 1−β+βc1-\beta+\beta c times that of SGDM (0.7570.757 at c=0.73c = 0.73). SGDM at 0.757η0.757\eta is therefore QHM with ν=1\nu = 1 at the fixed gate's learning rate: it places the same total weight on the gradients without shifting weight toward the current one, matching the coefficients. Each control is tuned on the same learning-rate grid and reported at its best (Table R1), and AGAM's gain over SGDM on CIFAR-100 splits as follows:

CosineConstant
SGDM at 0.757η0.757\eta over SGDM (step size)−0.05+0.99
Fixed gate over SGDM at 0.757η0.757\eta (mixture)+0.30+0.06
AGAM over the best fixed gate (adaptivity)+0.28+0.55
Total (AGAM over SGDM)+0.53+1.60

So both contribute, and in different regimes. The smaller step accounts for nearly most of the fixed gate's gain at a constant learning rate and for none of it with cosine decay, where the gain appears only once the mixture is added. Lion offers a comparison in which the step size cannot change: the update is a sign, so the gate changes its direction and not its step size (nonzero coordinates aside). At the same learning rate and weight decay (Section 4.3), AGAM-Lion still gains 1.77 exact-match points over Lion on GSM8K (Table 3). Adaptivity then adds a consistent margin over the best fixed gate across all tasks (Tables R1–R3). On CIFAR-100, it accounts for roughly half of AGAM's gain over SGDM with cosine decay and a third at a constant learning rate. AGAM also reaches the validation target sooner than the fixed gate in 8 of the 10 cells where both reach it.

Where adaptivity matters most (Table R2). The margin grows when training is less stable. AGAM leads the fixed gate by 0.48 points at η=0.4\eta = 0.4 on CIFAR-100 and by 0.49 at η=0.8\eta = 0.8 on CIFAR-10. At B=4096B = 4096, the fixed gate finishes below 60% in three of eight seeds and collapses to 7.4% in one, trailing AGAM by 11.47 points.

Beyond SGDM (Table R3). Fixed-gate AdamW (the QHAdam control) and fixed-gate Lion, each at its best gate, trail AGAM by 1.26 and 1.56 exact-match points on GSM8K, and by 0.55 and 2.21 linear-probe points on MAE pre-training. On the larger Qwen2.5-7B (new Table 5), the best fixed gate trails AGAM-AdamW by 0.71 exact-match points and falls below plain AdamW (77.53 against 77.63).

QHM at its recommended setting reaches 76.17 ± 0.13 with cosine decay and 68.22 ± 1.46 at a constant learning rate, against 79.03 ± 0.15 and 70.83 ± 0.43 for AGAM.

Part 2/3: Stable average gates (W2, Q2), gate granularity and theory (W2)

W2/Q2. Stable average gates. We agree that the average gates are stable: the model-mean gate stays between about 0.70 and 0.79 over an entire run (Figures 10 and 12), which is why the fixed-gate comparison of Part 1 matters. This mean, however, pools every tensor and step of an epoch, and the gates behind it are far from stable (the submission's Figures 4, 5, 22 are now 10, 11, 28):

  • Across tensor-steps. On ResNet-50, the per-epoch minimum gate of the normalization parameters is near 0.03 in the first epochs without warmup and between 0.1 and 0.4 for most of training, so individual steps are gated close to plain SGD even where the mean sits near 0.72 (Figure 16). On MAE, 30–50% of the tensor-steps in every group are misaligned (γt(k)<0.5\gamma_t^{(k)} < 0.5) between epochs 2 and 10 (Figure 26).
  • Across parameter groups. On MAE, the weights stay at about 0.62–0.67 after warmup while the biases and special tokens rise to 0.74–0.85, so the weights and biases of a single module receive different amounts of momentum (Figure 24). On ResNet-18, late in cosine decay, the classifier bias is misaligned on 13.0% of its steps and the convolutional weights on 0.3% (Appendix O.1).
  • Across depth. On ResNet-50 without warmup, the stem and first stage start at 0.63–0.67 while the third stage starts above the reference level of 0.718, and the pattern reverses within about eight epochs (Figure 15). Over the first 40 epochs, the misalignment rate ranges from 0.15% to 3.9% across convolutional blocks and reaches 10% for the normalization parameters of the last block (Figure 20).
  • Over training. On MAE, every group drops to 0.53–0.58 in the second epoch and recovers over the warmup, and the biases later fall from 0.74–0.85 back below 0.718 over the last hundred epochs (Figure 24). On ResNet-50 without warmup, the first five epochs push the convolutional weights up to 0.84 and the normalization parameters down to about 0.67, and the ordering reverses by epoch 10 (Figure 13).

How much gain remains with a fixed gate? Part 1 gives the answer: the mixture's share remains, and removing the variation costs 0.28–0.55 points on CIFAR-100, 1.26–1.56 exact-match points on GSM8K, 0.55–2.21 linear-probe points on MAE, and stability at B=4096B = 4096. A fixed gate is also a hyperparameter. On CIFAR-100 at η=0.2\eta = 0.2, c=0.2,0.5,0.73,0.9c = 0.2, 0.5, 0.73, 0.9 give 77.98, 78.45, 78.75, and 78.31, against 79.03 for AGAM (new Figure 9), and across our experiments a wrong value lowers accuracy by up to 1.82 points. The best value usually coincides with AGAM's own mean gate, which is not known in advance and ranges from 0.65 to 0.815 across our settings.

Predefined schedule. We did not run one. Because the model-mean gate stays between about 0.70 and 0.79, a schedule that replays it remains close to the fixed gate cˉ\bar c above. Such a schedule would also have to be found anew for each setting, because the gate's trajectory changes with warmup and with the learning-rate schedule (Figures 10 and 13). Finally, like a fixed gate, a schedule is blind to alignment: it cannot close on the conflicting tensor-steps while staying open on the others, which is the property behind the certificate (Theory, below).

W2. Model-wide versus tensor-wise gate. We agree that the three-seed MAE probe difference was within seed variability, and the revised Appendix N says so. We added five pre-training seeds. Over all eight, the tensor-wise gate reaches 29.70 ± 0.13 against 29.24 ± 0.25, and paired by seed it outperforms the model-wide gate in seven (Table 7). On CIFAR-100 the model-wide gate costs 0.55 points (Table 6). Figure 28 indicates why it falls short on MAE: computed at the same steps, a model-wide gate stays at 0.53–0.55, below the 10th percentile of the per-tensor gates from about epoch 12 onward.

W2. Theory. We agree that Theorem D.6 (D.7 in the revision) holds for any gate in [0,1][0,1] and cannot separate AGAM from a fixed mixture, and the revision now says so. The descent certificate of Lemma D.3 does separate them, because it uses the cosine rule itself. With st=cos⁡(gt,mt)s_t = \cos(g_t, m_t) and ψt=∥mt∥/∥gt∥\psi_t = \|m_t\|/\|g_t\|, a fixed gate cc has the alignment score ⟨gt,ueff,t⟩/∥gt∥2=(1−c)+c stψt\langle g_t, u_{\mathrm{eff},t}\rangle/\|g_t\|^2 = (1-c) + c\,s_t\psi_t, which is positive at every angle exactly when ψt<(1−c)/c\psi_t < (1-c)/c. This is 0.37 at c=0.73c = 0.73. The cosine gate keeps the score positive at every angle for all ψt<3+22≈5.83\psi_t < 3+2\sqrt{2} \approx 5.83. A fixed gate reaches this radius only if c≤0.146c \le 0.146, which removes most of the momentum from every step, whereas AGAM keeps a mean gate of 0.73 because its gate is small exactly when st<0s_t < 0 (new Remark D.4 and Figure 4). The telemetry agrees: on ResNet-50, SGDM's applied update opposes the gradient on up to 18% of tensor-steps, whereas AGAM's rate is exactly zero in most epochs (Figure 21). We do not claim a faster rate.

Part 3/3: Positioning (W1), taxonomy (W1, Q3), AGAM + Cautious (Q4)

W1. Positioning. We agree that gradient–momentum interpolation is know, and the revised Section 2.2 expands AGAM's positioning within this family: it adds AdEMAMix and Simplified AdEMAMix, as the reviewer suggested, and states how each method sets its mixing coefficients. In all of them, the coefficients are hyperparameters or prescribed schedules chosen per setting, and none uses an alignment signal:

  • QHM/QHAdam fix ν\nu for every parameter and step, and Ma & Yarats sweep (ν,β)(\nu, \beta) over a 15×1415 \times 14 grid.
  • Lion applies a fixed ν=β1/β2\nu = \beta_1/\beta_2 before the sign, which AGAM-Lion makes adaptive.
  • AdEMAMix adds a second, slower EMA of the gradient under scheduled coefficients.
  • Simplified AdEMAMix instead adds the current gradient with a fixed weight α\alpha while scheduling β1\beta_1, whose reported optimum moves from near 100 at a small batch size to near 0.05 at a large one.

AGAM sets the coefficient per tensor and per step from the measured alignment, with no added hyperparameter or state. It also operates directly on its base optimizer's settings: in every AdamW and Lion experiment, AGAM's best configuration coincides with its base optimizer's (Section 4.3). Ma & Yarats name an effective adapter for ν\nu and β\beta as an open direction, and AGAM realizes one for ν\nu. That this adaptation adds to the family is what Part 1 establishes empirically and Remark D.4 theoretically. We did not run the AdEMAMix variants. Within this family, the fixed gate is the control that isolates adaptivity, because it differs from AGAM only in the alignment signal.

W1/Q3. Taxonomy and "mutually exclusive". We agree that the phrase overstated our claim, and the revision replaces it. We also agree with the reviewer's three knobs, and Figure 1 and the revised Section 2.3 organize the methods this way: TAM and AM act inside the stored momentum recurrence, AGAM acts on the mixture of the applied step alone, and Cautious acts on the coordinates of an already formed update. Composition is well defined, as the reviewer notes, and we narrow our claim. TAM, AM, and AGAM all decide how the history mt−1m_{t-1} and the gradient gtg_t are weighed when they disagree, and stacking two of these decisions is not additive (new Appendix C.1):

  • TAM + AGAM is destructive. TAM trusts the history: under sustained opposition it damps the incoming gradient toward zero and proposes a step along βmt−1\beta m_{t-1}. AGAM trusts the gradient: its gate on that proposal closes on mt−1m_{t-1}. Both terms are suppressed, and the step vanishes when a correction matters most. At cos⁡(gt,mt−1)=−0.9\cos(g_t, m_{t-1}) = -0.9 and β∥mt−1∥/∥gt∥\beta\|m_{t-1}\|/\|g_t\| from 0.5 to 3, the composite step has norm 0.02–0.11 in units of ∥gt∥\|g_t\|, against 0.52–0.73 for AGAM alone.
  • AM + AGAM is redundant. AM lowers its memory coefficient under conflict and stores the result. For cos⁡(gt,mt−1)≤−0.5\cos(g_t, m_{t-1}) \le -0.5 and ∥mt−1∥/∥gt∥\|m_{t-1}\|/\|g_t\| from 0.5 to 3, its stored direction already has a cosine of at least 0.75 with the gradient, so AGAM's gate is at least 0.87 and has almost nothing left to act on.

They are alternative designs for the same decision, which is why we benchmark them against each other (revised "Choice of baselines").

Q4. AGAM + Cautious. Thank you for the suggestion. We have not trained this combination, or Cautious alone, on our benchmarks. A well-defined construction computes AGAM's gate from the unmasked base proposal and then applies the Cautious mask and rescaling to the gated update. How the two stages interact deserves its own empirical and theoretical analysis. The revised Section 2.3 notes that an update-reweighting rule can be applied on top of AGAM, states that we have not evaluated such combinations, and leaves them to future work.

We thank the reviewer again for the detailed and expert feedback, which prompted the fixed-gate controls, the certificate comparison, and the refined expanded positioning. We hope these additions establish the value of the adaptive gate and resolve the reviewer's concerns about how the contribution is isolated and positioned.

Reviewer_GDJYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes AGAM, which adjusts how much historical momentum is used in the current update while leaving the stored momentum recurrence unchanged. The adjustment is based on the cosine similarity between the current gradient and the base optimizer's proposed update, computed separately for each parameter tensor. The authors apply this idea to SGDM, AdamW, and Lion, explain its relationship to quasi-hyperbolic momentum, and provide geometric and stationarity results under stated assumptions. The experiments cover image classification, MAE pre-training, and language-model fine-tuning, with improvements over the base optimizers and the evaluated alignment-aware baselines.

Critical strengths

I like the distinction between keeping momentum and deciding how much of it to use. A disagreement with one noisy mini-batch does not necessarily mean that the accumulated history should be discarded. AGAM addresses this without adding another optimizer state or a new hyperparameter, which makes the method fairly easy to try in practice.

I appreciate that the experiments look at learning-rate and batch-size grids rather than only reporting each method's best result. The ablations in Appendix N are also useful for understanding which design choices matter. I found the MAE results particularly interesting: the gains remain after all pretrained encoders are fine-tuned with the same recipe, so they are not simply due to using AGAM during downstream fine-tuning.

Critical weaknesses

My main concern is that the experiments do not quite establish why the gate needs to depend on alignment. The paper itself shows that AGAM is closely related to QHM/QHAdam, and that the SGDM gate changes both the gradient–momentum mixture and the effective step scale. I did not see a fixed-gate or tuned QHM/QHAdam comparison in Appendix N. Without this control, I find it difficult to separate the benefit of the alignment signal from the benefit of simply giving the current gradient more weight.

I am also not fully convinced by the rationale for excluding methods such as Cautious and MGUP from the comparisons. They may operate at a different stage and be composable with AGAM, but they are still relevant alternatives for someone deciding which optimizer modification to use. Also, TAM and AM's additional hyperparameters are kept at their published defaults. I think the current experiments make a reasonable case against the selected configurations, but a less complete case for AGAM's advantage over alignment-aware methods more generally.

Questions for authors
  1. Have you tried a fixed-gate version or a tuned QHM/QHAdam baseline, with the learning rate retuned for each? I would be interested in how much of AGAM's improvement remains once the gradient–momentum mixture and effective step scale are controlled for.

  2. Do the improvements over TAM and AM remain when their method-specific hyperparameters are tuned? Do you have a comparison with Cautious or MGUP on one of the main benchmarks? These results would help me judge the practical advantage of the proposed gate.

1 comment

We thank the reviewer for the positive assessment and for recognizing AGAM's nondestructive approach alongside its practical simplicity.

W1. Why the gate depends on alignment. Thank you for raising this. The motivation is a first-order one: when ⟨gt,ut⟩<0\langle g_t, u_t\rangle < 0, the proposed step −ηtut-\eta_t u_t increases the mini-batch loss to first order, by an amount proportional to the update norm ∥ut∥\|u_t\| and to its cosine misalignment with gtg_t. We reworded the introduction and the text under Equation 1 to state this. Two results then establish that the dependence is needed, and the controls below confirm it:

  • Theory. Because the cosine gate is small exactly when the update opposes the gradient, AGAM's step cannot oppose the observed gradient unless ∥mt∥/∥gt∥>3+22≈5.83\|m_t\|/\|g_t\| > 3+2\sqrt2 \approx 5.83 (Lemma D.3), while its mean gate stays at 0.73. A fixed gate of 0.73 gives this guarantee only up to a ratio of 0.37, and it would need c≤0.146c \le 0.146 to match AGAM, which removes most of the momentum from every step (new Remark D.4 and Figure 4). The telemetry agrees: on ResNet-50, SGDM's applied update opposes the gradient on up to 18% of tensor-steps, whereas AGAM's rate is exactly zero in most epochs (Figure 21).
  • Prediction. Early-training misalignment statistics reliably predict AGAM's gain over SGDM across 58 configurations (ρ=0.83\rho = 0.83; new Subsection O.4, Table 12, and Figure 29).

W1/Q1. Controls. Thank you for this great point. We agree that these controls were missing. We ran and added all three: a fixed gate (which is QHM with SGDM, and takes the QHAdam form with AdamW), QHM at its recommended setting, and matched-step SGDM. The results are summarized in the general response above (Tables R1–R3, new Appendix N.1). In every experiment, the fixed gate was searched over c∈{0.2,0.5,0.9,cˉ}c \in \{0.2, 0.5, 0.9, \bar c\}, where cˉ\bar c is AGAM's own mean gate in the same training setting, and every fixed-gate result we report is the best of these four gates. On CIFAR-100, every method was run at η∈{0.05,0.1,0.2}\eta \in \{0.05, 0.1, 0.2\} with both schedules and is reported at its own best learning rate.

How much of the improvement remains once the mixture and step scale are controlled for? A consistent share. On CIFAR-100 with cosine decay, the fixed mixture accounts for +0.25 of AGAM's +0.53-point gain over SGDM, and adaptivity for the other +0.28. At a constant learning rate, the smaller step accounts for +0.99 of the +1.60-point gain, the mixture for +0.06, and adaptivity for +0.55. The margin grows where training is unstable (11.47 points at B=4096B = 4096) and holds with AdamW and Lion: the best fixed gate trails by 1.26 and 1.56 exact-match points on GSM8K and by 0.55 and 2.21 probe points on MAE, and on the larger Qwen2.5-7B it falls below plain AdamW (Tables R2 and R3). A fixed gate must also be tuned per setting, whereas AGAM needs no search.

W2/Q2. Cautious and MGUP. Thank you for this valid remark. We agree that they are relevant alternatives for a practitioner. They act at a different stage from AGAM: both reweight, coordinate-wise, a step that the base optimizer has already formed, whereas AGAM, TAM, and AM decide how much momentum enters that step (Figure 1). We therefore benchmarked AGAM against the methods that answer the same question, just as MGUP and Magma compare against Cautious. We do not have Cautious or MGUP results on our benchmarks. The revised Section 2.3 (Choice of baselines) explains this scope, notes that update-reweighting methods can be applied on top of AGAM rather than in its place, and states that we have not evaluated such combinations. An informative comparison would cover each method alone and combined with AGAM, which we leave to future work.

Q2. TAM and AM hyperparameters. We kept their method-specific hyperparameters at the values their authors use. For AM these are λ=0.1\lambda = 0.1 and βmax⁡=0.9\beta_{\max} = 0.9 (0.89 for AM-AdamW), and its authors keep them fixed and do not retune them across settings. For TAM, γ=0.9\gamma=0.9 and ϵ=10−8\epsilon=10^{-8} (ϵ\epsilon is fixed by default). Both their ablations report low sensitivity (all λ∈[0.01,1]\lambda \in [0.01, 1] perform comparably, and varying γ\gamma has minimal impact), so we do not expect tuning them to change the comparison. AGAM, TAM, and AM each received the same tuning budget, a grid of the same size as the base optimizer's (Table 4). Tuning extra hyperparameters for TAM or AM would give them a larger budget than AGAM, which has none, and we applied the same rule to QHM above. The one exception favors the control: the fixed gate was additionally searched over four gate values, a search that AGAM does not need. We agree that this supports a claim about the published configurations, not about every possible tuning, and the revision words it that way.

We hope these additions and clarifications sufficiently address the reviewer's thoughtful concerns.

Reviewer_r2ZKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Alignment-Gated Momentum (AGAM), a lightweight modification to momentum-based optimizers which adaptively adjusts the contribution of historical momentum according to its alignment with the current stochastic gradient. Specifically, AGAM computes a tensor-wise gating coefficient from the cosine similarity between the current gradient and the base optimizer's proposed update, attenuating historical momentum when misalignment occurs while preserving the original optimizer state. The method introduces no additional hyperparameters and is instantiated for SGDM, AdamW, and Lion.

The authors build connections between AGAM and quasi-hyperbolic optimization, derive a geometric descent certificate for AGAM-SGD, and provide a stationarity bound under smoothness, bounded variance, and additional optimizer-related assumptions. Empirically, the method is evaluated on ResNet image classification, MAE-based ViT pre-training, and language model fine-tuning. Results demonstrate improvements over base optimizers and selected alignment-aware alternatives, with particularly notable gains in certain high-learning-rate or large-batch settings. Additional analyses investigate tensor-wise momentum misalignment, optimization robustness, and the effects of the proposed gating design.

Critical strengths
  1. The main strength of this work is its lightweight, state-preserving design. Different from methods that modify persistent momentum states or discard misaligned updates, AGAM transiently adjusts the contribution of historical momentum using a tensor-wise alignment coefficient. This makes the method relatively easy to integrate into existing optimizers without introducing additional hyperparameters. The authors provide concrete implementations for SGDM, AdamW, and Lion, demonstrating the generality of the proposed mechanism. The empirical results show consistent improvements across multiple training settings, including ResNet classification, MAE pre-training, and LLM fine-tuning. In particular, the improvements under challenging learning-rate and batch-size configurations suggest that AGAM may enhance optimization robustness.

  2. Beyond reporting performance improvements, the work offers useful analyses of momentum-gradient misalignment across parameter tensors, network layers, and training phases. These observations motivate the use of tensor-wise rather than global alignment coefficients. The ablations in Appendix N further examine the importance of transient gating, the alignment reference, and continuous versus hard gating. The results generally favor the proposed design, with persistent gating and stale-buffer alignment producing the largest performance reductions in the evaluated settings. Together with the theoretical connection to quasi-hyperbolic optimization and the geometric analysis of gated updates, these experiments provide a reasonably convincing explanation of why the proposed mechanism can be beneficial.

Critical weaknesses
  1. Despite of the fact that AGAM presents a useful combination of transient state preservation, tensor-wise gating, and update-gradient alignment, its core mechanism is closely related to existing approaches that dynamically adjust momentum based on gradient agreement. The connection to quasi-hyperbolic momentum further suggests that AGAM can be interpreted as an adaptive, tensor-wise interpolation between historical momentum and the current gradient. Consequently, the methodological novelty lies primarily in the particular design and combination of existing ideas rather than a fundamentally new optimization principle. While the authors compare against TAM and Adaptive Memory, stronger comparisons with alternative adaptive momentum formulations, including appropriately tuned quasi-hyperbolic optimizers, would help establish whether the proposed alignment rule is essential to the reported gains.

  2. The theoretical analysis builds useful properties of AGAM, but it does not demonstrate improved convergence rates or generalization guarantees over the corresponding base optimizers. In particular, the stationarity result in Appendix D assumes bounded conditional gradient variance and spectral bounds on the preconditioner, and its stochastic guarantee generally converges only to a noise-dependent neighborhood under fixed nonzero variance. The empirical evidence is also concentrated on relatively modest-scale models, including ResNets, ViT-Tiny, SmolLM2-135M, and Qwen2.5-3B LoRA fine-tuning. While the image-classification and MAE improvements are encouraging, gains on the language-model tasks are relatively small, particularly the perplexity reduction from 14.405 to 14.388 on WikiText-2. It therefore remains unclear whether the method provides substantial improvements for larger-scale pre-training or well-tuned modern optimization pipelines. Furthermore, the claimed negligible computational overhead would be more convincing with explicit wall-clock or throughput comparisons.

Questions for authors

1. Comparison with adaptive quasi-hyperbolic optimization. Due to the fact that AGAM can be interpreted as a tensor-wise adaptive form of quasi-hyperbolic momentum, could the authors compare it against QHM/QHAdam with appropriately tuned fixed interpolation coefficients or alternative adaptive interpolation schedules? Such comparisons would help distinguish the benefits of the specific alignment-based gating rule from those of simply reducing the contribution of historical momentum.

2. Scalability and computational overhead. Could the authors offer wall-clock training times or throughput measurements for AGAM and its base optimizers, particularly for the transformer experiments? Additionally, do the authors expect the observed improvements to persist in larger-scale language-model pre-training, where optimizer overhead, distributed execution, and training stability become more consequential? Quantitative overhead measurements would help evaluate the practical significance of the proposed drop-in modification.

3. Understanding when alignment gating is beneficial. The experimental results suggest that AGAM is particularly effective in certain high-learning-rate, large-batch, or non-stationary training regimes. Can the authors quantify the relationship between the frequency or severity of update-gradient misalignment and the corresponding performance improvements? For example, is there a consistent correlation between tensor-wise gating activity and optimization gains across different architectures and training configurations? Such an analysis would clarify whether misalignment is a reliable predictor of AGAM's effectiveness.

1 comment

Response to Reviewer r2ZK

We thank the reviewer for the careful reading, thorough review, and accurate summary of our contributions. W1/Q1. Tuned quasi-hyperbolic baselines. Thank you for this suggestion. We ran them, along with additional controls (general response above, Tables R1–R3; new Appendix N.1). The fixed gate (with SGDM, this is QHM at β=0.9\beta = 0.9 with a fixed ν\nu and a rescaled step) was searched in every experiment over c∈{0.2,0.5,0.9,cˉ}c \in \{0.2, 0.5, 0.9, \bar c\} with cˉ\bar c AGAM's own mean gate, and we report the best of the four. QHM was also run at its authors' recommended setting (ν,β)=(0.7,0.999)(\nu, \beta) = (0.7, 0.999), and SGDM at the matched step size. Each was tuned on the same learning-rate grid. Reducing the contribution of momentum explains part of AGAM's gain, and the alignment rule adds a consistent margin on top. AGAM leads the best fixed gate by 0.28 and 0.55 points on CIFAR-100 (cosine decay and constant learning rate), by 11.47 points at B=4096B = 4096, by 1.26 and 1.56 exact-match points on GSM8K, and by 0.55 and 2.21 probe points on MAE. QHM at its recommended setting is 2.9 points behind AGAM with cosine decay. A fixed gate must also be tuned per setting, and it lacks the descent certificate (below).

W2. Theory. Theorem D.7 (D.6 in the submission) is a stationarity bound with a noise floor, and we do not claim a faster rate. What the cosine rule adds is a per-step safety property: AGAM's step cannot oppose the observed gradient unless the momentum is more than 3+22≈5.833 + 2\sqrt{2} \approx 5.83 times larger than the gradient (Lemma D.3), against a ratio of 0.37 for a fixed gate at AGAM's mean (new Remark D.4 and Figure 4).

W2/Q2. Scale and the LLM gains. With single 16–24 GB GPUs we prioritized breadth: full hyperparameter grids, three base optimizers, pre-training from scratch, and fine-tuning. In the LLM experiments, AGAM is ahead of its base optimizer in both Tables 2 and 3. On GSM8K it adds 0.68 exact-match points over AdamW and 1.77 over Lion, while AdaTAMW and AM-AdamW fall below AdamW. To address this concern directly, we added a larger experiment, Qwen2.5-7B on GSM8K (Appendix M, new Table 5), where AGAM-AdamW attains the lowest validation loss of the four optimizers and improves on its base by 0.61 exact-match points at the base optimizer's own hyperparameters. The best fixed gate there reaches 77.53%, below both AGAM-AdamW (78.24%) and AdamW (77.63%), so a fixed mixture does not account for this gain. On pre-training, our evidence is MAE trained from scratch, where the gains are our largest (+5.71 and +8.04 fine-tuning points over AdamW and Lion, Table 1).

W2/Q2. Overhead. AGAM adds about 1% to wall-clock training time. The timings are in the general response and Appendix K. The gate uses the already aggregated gradient, so it needs no extra communication in data-parallel training.

Q3. Does misalignment predict the gain? Thank you for raising this insightful question. Yes, reliably across the configurations we evaluated, and its timing during training matters. For each AGAM-SGD run we log the share of misaligned tensor-steps, the conflict mass (the mean of max⁡(0,−cos⁡(gt,mt))\max(0, -\cos(g_t, m_t)) over all tensor-steps), and their ratio, the severity. We relate them to the share of SGDM's test error that AGAM removes over 58 configurations spanning both datasets, three architectures, both schedules, and five learning rates (new Subsection O.4, Table 12, and Figure 29). The table below summarizes Table 12 with Spearman's rank correlation ρ\rho between each statistic, averaged over the first 20 epochs, and the gain (p<0.001p < 0.001 in every cell).

Statistic (epochs 1–20)CosineConstantBoth schedules
Misaligned fraction0.780.670.81
Conflict mass0.780.700.83
Severity0.730.770.83
  • Early misalignment predicts the gain under both schedules, with ρ\rho up to 0.83 over all configurations. With cosine decay the signal fades later in training, when the steps are too small to change the outcome. At a constant learning rate it remains informative throughout.
  • It is not the learning rate in disguise. The learning rate alone gives ρ=0.41\rho = 0.41, and the early conflict mass keeps a partial correlation of 0.79 after controlling for it.
  • It holds inside each regime and each model. For the conflict mass, ρ\rho is positive in every dataset × schedule group (0.63–0.84), and the three architectures give 0.87, 0.85, and 0.78.
  • A simple reading. In the 28 configurations with above-median early conflict mass, AGAM beats SGDM in all 28 and removes 21.9% of its error on average. In the remaining configurations it removes 1.8%.

The first 5 epochs alone already give ρ=0.82\rho = 0.82, so a short probe run indicates how much AGAM is likely to help.

We hope our additional empirical, diagnostic, and statistical results have sufficiently addressed the reviewer's concerns.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers (P1)

We sincerely thank the reviewers for their time, insightful questions, and constructive feedback, which have strengthened our empirical analysis and the manuscript overall. One question was shared across the reviews: how much of AGAM's gain stems from its alignment-adaptive gate, rather than from a fixed momentum–gradient mixture or the smaller effective step size it induces? We ran the requested controls with the paper's recipes and seeds. The results are in the new Appendix N.1 and are summarized below (mean ± s.e. over three seeds unless noted).

Controls:

  • Fixed gate: AGAM with its gate held constant. With SGDM, this recovers QHM up to a gate-dependent step scale. In every experiment (SGDM, AdamW, and Lion), we searched c∈{0.2,0.5,0.9,cˉ}c \in \{0.2, 0.5, 0.9, \bar c\}, where cˉ\bar c is AGAM's own mean gate in the same training setting (e.g., cˉ=0.73\bar c = 0.73 on CIFAR-100, which is QHM with ν≈0.96\nu \approx 0.96). Every fixed-gate result below is the best of these four gates, and each table lists the selected value. Interestingly, the best fixed gate typically coincides with AGAM's own mean gate.
  • QHM at its authors' recommended setting (ν,β)=(0.7,0.999)(\nu, \beta) = (0.7, 0.999).
  • Matched-step SGDM: SGDM at the fixed gate's step size (0.757η0.757\eta for c=0.73c=0.73), which separates the step size from the mixture.

On CIFAR-100 (Table R1), every method was run at η∈{0.05,0.1,0.2}\eta \in \{0.05, 0.1, 0.2\}, under warmup + cosine decay and at a constant learning rate, and is reported at its own best learning rate. The fixed gate was searched over all four gates at each of these learning rates.

Table R1. ResNet-50/CIFAR-100, B=256B = 256.

CosineConstant
SGDM78.50 ± 0.1269.23 ± 0.35
SGDM, matched step (0.757η0.757 \eta)78.45 ± 0.1070.22 ± 0.12
QHM (0.7, 0.999)76.17 ± 0.1368.22 ± 1.46
Fixed gate (c=0.73c=0.73)78.75 ± 0.0370.28 ± 0.44
AGAM79.03 ± 0.1570.83 ± 0.43

Table R2. Harder settings. ResNet-50/CIFAR-100 at B=256B = 256 with cosine decay unless noted.

Fixed gateAGAM
η=0.4\eta = 0.4 (c=0.5c = 0.5)78.13 ± 0.0978.61 ± 0.15
B=4096B = 4096, η=0.4\eta = 0.4 (c=0.782c = 0.782, 8 seeds)56.69 ± 7.2568.16 ± 0.64
40% label noise (c=0.728c = 0.728)52.98 ± 0.2153.05 ± 0.13
ResNet-18/CIFAR-10, η=0.8\eta = 0.8 (c=0.738c = 0.738)93.96 ± 0.2094.45 ± 0.21

Table R3. Transformers. MAE ViT-Tiny linear probe, and Qwen2.5-3B and Qwen2.5-7B LoRA on GSM8K (exact match).

Fixed gateAGAM
MAE probe, AdamW (c=0.70c = 0.70, 8 seeds)29.15 ± 0.3529.70 ± 0.13
MAE probe, Lion (c=0.815c = 0.815)26.78 ± 0.2728.99 ± 0.31
GSM8K 3B, AdamW (c=0.65c = 0.65)68.11 ± 0.7369.37 ± 0.36
GSM8K 3B, Lion (c=0.766c = 0.766)68.01 ± 0.3469.57 ± 0.50
GSM8K 7B, AdamW (c=0.672c = 0.672)77.53 ± 0.2078.24 ± 0.16
AuthorsOpenReview ↗

General response to all reviewers (P2)

What we conclude:

  1. Mixing in the gradient with a shorter step helps overall, and adaptivity adds a consistent margin on top across all tasks. AGAM leads the best fixed gate by 0.28 points with cosine decay and by 0.55 at a constant learning rate, which is roughly half and a third of its gain over SGDM. It also reaches the validation target sooner in 8 of the 10 cells where both reach it. QHM at its recommended setting is 2.9 points behind AGAM with cosine decay. The transformer results agree and show larger gains (Table R3): with AdamW and Lion, adaptivity adds 1.26 and 1.56 exact-match points on GSM8K, and 0.55 and 2.21 probe points on MAE. On the larger Qwen2.5-7B, the best fixed gate trails AGAM-AdamW by 0.71 points and falls below plain AdamW (77.53 against 77.63, new Table 5).
  2. The margin grows where training is unstable (Table R2). At B=4096B = 4096, the fixed gate finishes below 60% in three of eight seeds and collapses to 7.4% in one, while AGAM stays between 64.60% and 70.17%.
  3. A fixed gate is not hyperparameter-free. Its value must be tuned for every setting (new Figure 9), and a wrong value is costly, lowering accuracy by up to 1.82 points. The best fixed gate usually coincides with AGAM's own mean gate, which is not known in advance and ranges from 0.65 to 0.815 across our settings.
  4. Only the adaptive gate carries the descent certificate. Against opposing momentum, a fixed gate of 0.73 certifies descent only up to a momentum-to-gradient norm ratio of 0.37, against 5.83 for the cosine gate (new Remark D.4 and Figure 4).
  5. The gate is also a useful diagnostic. Its per-tensor telemetry shows where and when the update–gradient conflicts occur and how severe they are, and early misalignment strongly predicts (ρ=0.83\rho=0.83) how much AGAM improves on SGDM (new Subsection O.4, Table 12, and Figure 29; response to Reviewer r2ZK). The relationship persists on 16 additional (larger batches, label noise, no warmup, or no normalization) configurations (ρ\rho=0.77).
  6. AGAM itself was never tuned. It inherits its base optimizer's momentum coefficients and hyperparameter grid, and we set no AGAM-specific defaults for β\beta, β1\beta_1, or β2\beta_2. In every AdamW and Lion experiment, AGAM's best configuration coincides with its base optimizer's (Section 4.3).

Overhead. AGAM adds about 1% to wall-clock training time: 7,915 s against 7,837 s on ResNet-50, and 4,069 s against 4,028 s for Qwen2.5-3B LoRA fine-tuning, on the same GPU (Appendix K).

Revisions to the manuscript.

  • Fixed-gate controls for SGDM, AdamW, and Lion, with QHM and matched-step SGDM controls (Section 4.3 and Appendix N.1, Tables 8–11, Figure 9).
  • Expanded positioning against QHM/QHAdam, Lion, AdEMAMix, and Simplified AdEMAMix (Section 2.2), following Reviewer HtX8's suggestion.
  • "Mutually exclusive" replaced by a more principled distinction (Section 2.3, Choice of baselines), with a worked composition example showing why stacking TAM or AM with AGAM is destructive or redundant (Appendix C.1).
  • The certificate remark and its illustrative figure (Remark D.4 and Figure 4).
  • The misalignment-versus-gain analysis (Subsection O.4, Table 12, and Figure 29).
  • Gate figures redrawn with their spread across tensor-steps and seeds (Figures 10–13).
  • MAE ablations extended from three to eight seeds (Appendix N, Table 7).
  • Overhead measurements (Appendix K), and larger-scale Qwen2.5-7B results on GSM8K with a fixed-gate control (Appendix M, Table 5).
  • A full proofread, tighter wording in several sections, and a reordered appendix in which the proofs follow the theory.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.