acceptodds
Under review as a conference paper at ICLR 2027

On the Surprising Effectiveness of Masking Updates in LLM Training

Abstract

Training large language models (LLMs) relies almost exclusively on dense adaptive optimizers with increasingly sophisticated preconditioners. We challenge this by showing that randomly masking parameter updates can be highly effective, with a masked variant of RMSProp consistently outperforming dense optimizers. Our analysis shows that random masking induces a curvature-dependent regularization term that penalizes sharp update directions. Motivated by this finding, we introduce Momentum-aligned gradient masking (Magma), which modulates the masked updates using momentum-gradient alignment. Across extensive LLM pre-training experiments up to 1B parameters, and across both dense and sparse mixture-of-experts architectures, Magma improves a broad range of optimizers including RMSProp, Adam, LaProp, Muon, and SOAP in 26 out of 28 cases. Our analysis shows that alignment-based masking can widen the stable optimization regime by damping noisy, high-curvature blocks while preserving descent direction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Optimizer, pre-training, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6y6pOpenReview ↗
Rating
1(−2)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies randomly masking optimizer updates and introduces Magma, which combines masking with alignment-based damping. Its main contribution is evidence that this simple modification can improve several optimizers in LLM pretraining, although the explanation for these gains remains incomplete.

Critical strengths
  1. The evaluation spans seven optimizers and four model sizes, with improvements in 26 of the 28 reported combinations.
  2. The method is straightforward to implement and can be applied without redesigning the underlying optimizer.
Critical weaknesses
  1. The experiments do not establish whether the benefits of update masking persist in substantially larger models. It would also be useful to examine whether larger models tolerate higher masking rates, since a fixed masking rate provides limited insight into how update sparsity should scale with model size.
Questions for authors

See weaknesses.

1 comment

We thank Reviewer 6y6p for the positive evaluation. Your comments led to a new study that strengthens the paper. The revised manuscript is uploaded, with changes in magenta.

1) Do the benefits of masking persist in substantially larger models?

We could not test beyond 1B within our compute budget, and we now state this as a limitation. Within our range, the gains grow from 60M to 350M and hold at 1B. Under a compute-optimal budget ($C \approx 6ND $, 1× Chinchilla), training compute grows quadratically with model size: one 7B run (6.7B parameters, 134B tokens) needs about 5.4×10^21 FLOPs, roughly 27× a 1B run. Even a minimal tuned comparison (one base optimizer with and without Magma at four learning rates, 8 runs) needs about 4.3×10^22 FLOPs. This will cost roughly 100,000-300,000 USD in commercial cloud compute. Within our range, the trend is favorable: Magma's average gain grows from 3.6% at 60M to 9–10% at 350M–1B (Table 1), and masking itself becomes more beneficial as models grow (see 2). Whether this continues beyond 1B remains to be verified, as the Conclusion now states.

2) Do larger models tolerate higher masking rates?

Random masking itself is slightly costly at small scale (e.g., 60M and 130M), but highly beneficial at large scale (e.g., 1B). We swept the keep probability p at all four scales (C4, base RMSProp), including damping-only Magma (p = 1.0, no masking), with the learning rate tuned over {3e-4, 1e-3, 3e-3, 1e-2} for every (scale, p) pair. Batches are 2–8× larger than in Table 1, with proportionally fewer steps (same token budget). Validation PPL at the best learning rate (bold: best per scale; DIV: diverged at every tested learning rate):

Keep probability p60M130M350M1B
Base RMSProp (no Magma)32.1530.3316.92DIV
1.0 (damping only, no masking)30.3824.6316.5113.92
0.730.6424.9316.6213.50
0.630.7225.3316.9313.58
0.5 (default)30.7425.3416.5013.42
0.430.5025.4116.4113.32
0.330.6425.1416.8113.68

From the above table, we can see that Masking helps more as models grow. Relative to damping only, masking costs 0.1–0.4 PPL at 60M and 0.3–0.8 at 130M, is roughly neutral at 350M (0.10 better to 0.42 worse), and improves every setting at 1B by 0.24–0.60. The best setting moves from no masking (p = 1.0) at 60M–130M to p = 0.4 at 350M–1B. This is consistent with random masking acting as a geometric regularizer, from which larger models with more irregular loss landscapes benefit more.

It also shows that p does not need to be re-tuned per scale: for p \in [0.3,0.7], PPL varies by only 0.24–0.52 at each scale, and the default p=0.5 stays within 0.41 PPL (1.6%) of the best p in this range. In addition, Magma improves on base RMSProp for every p at every scale, with one tie (350M, p = 0.6: 16.93 vs. 16.92); at 1B, base RMSProp diverges at every tested learning rate. Please refer to Appendix D (Table A1) for the setup details and best learning rates.

Reviewer_hUuoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1(−1)
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes an optimizer wrapper called Magma aimed at improving the performance of dense optimizers such as Adam/Muon/SOAP in LLM pretraining by skipping around half of the updates randomly, while rescaling the rest using a momentum-gradient alignment score. The paper's main experiments are from LLaMA models (60M-1B) trained using the GaLore's benchmarking settings, and there is an additional NanoMoE experiment.

Critical strengths

None.

Critical weaknesses

The paper claims that the proposed optimizer wrapper substantially improves Adam/Muon/SOAP in LLM pretraining, but the main pretraining benchmarking setup is not appropriate for such general conclusions.

Specifically, the paper's main experiments follow the benchmarking setup of GaLore [1], which is only appropriate for comparing memory-efficient optimizers and not for deriving such general optimization claims.

GaLore's paper benchmark uses a very small token batchsize of about 0.13M tokens (512 samples x 256 sequence length) and full BF16 precision (i.e., not mixed-precision training - all parameters including the optimizer states remain in BF16). Both are distinctly different from standard, large-scale pretraining settings, where token batch sizes of 1M-16M (with sequence lengths over 8k) and mixed-precision training are the norm [2,3].

Training with small batchsizes reduces the optimizer gap in pretraining [4,5] and (from my experience) keeping the Adam states in BF16 significantly worsens its performance due to accumulating in low precision. This is also apparent from the results of Table 1, where Adam variants with "less" momentum states with such Adafactor or RMSPRop obtain significantly better perplexities than Adam itself (e.g., Adam 1B PPL 16.82, Adafactor 15.00 and RMSPRop 14.69). It is also very strange that Adafactor+Magma (i.e., Adafactor with skipped updates) gives the best overall perplexity at 1B, even better than Muon/SOAP variants.

In addition the current dense benchmark does not use weight decay, which is standard in broader pretraining setups and is needed for regulating the effective update size [2,3,6].

My main point is that the main results of the paper in Table 1 are not reliable for making general conclusions (beyond a very specific setting of memory-efficient optimization) about optimizer performance and do not offer much evidence to support the very unconventional proposed idea of skipping half of the updates. The paper clearly argues about a general benefit of skipping updates in pretraining, for which I find the current experimental evidence very limited.

To support this, I trained the LLaMA 130M model using the GaLore pretraining setup as in the paper but simply changed the precision from full BF16 to mixed (using torch.amp). The results are as follows:

MethodValidation PPL
Adam (lr=3e-3)21.78
Adam (lr=3e-3) +Magma (lr=3e-3)22.24
Adam (lr=3e-3) +Magma (lr=5e-3)22.03
Adam (lr=3e-3) +Magma (lr=1e-2)21.95
Adam (lr=3e-3) +Magma (lr=2e-2)diverged
Adam (lr=3e-3) +Magma (lr=3e-2)diverged

The results above show that (i) Adam is severely underperforming with full BF16 (25.42 vs 21.78 with mixed precision) (ii) even with extra tuning of the Magma learning rate, the performance remains worse than just running Adam. I welcome any new results from the authors using mixed precision, along with all necessary hyperparameters needed to reproduce them.

In general, I would suggest the authors to try mixed-precision training (i.e., torch.amp) and to use larger batchsize/sequence length (1M-4M total tokens) to align with standard pretraining settings (beyond memory-efficient specific benchmarking).

Finally, a reproducible result on a competitive benchmark with modern pretraining setup, such as the "modded-nanogpt benchmark, track 3 -optimization" [7] (https://github.com/KellerJordan/modded-nanogpt/tree/master/records/track_3_optimization) would be of great value. If such a result is provided and skipping half of the updates can indeed accelerate the current Muon/Adam optimizer records (e.g., achieve the benchmark's target loss in fewer steps). Given such a result and the necessary hyperparameters for to reproduce it would be a very compeling reason for me to increase my score.

[1] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. Zhao et al., 2024.

[2] Muon is scalable for llm training. Liu et al., 2025.

[3] Fantastic Pretraining Optimizers and Where to Find Them. Wen et al., 2025.

[4] Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful. Marek et al., 2025.

[5] Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling. Srećković et al., 2025.

[6] Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization, Wen et al., 2026.

[7] Modded-NanoGPT Optimization Benchmark. Jordan 2026.

UPDATE (19 November 2026)

The authors uploaded their rebuttal a few minutes before the deadline, making any on-time comment from the reviewers' side effectively impossible. This is very unfortunate, because it defeats the purpose of the discussion period. The updated paper includes substantial changes from the original, making the above choice to upload the rebuttal exactly before the deadline even more problematic. There are several other issues, which I elaborate below.

The new draft introduces a completely new optimizer variant, called "Magma-F", fully absent from the originally submitted paper. In their updated experiments, the originally proposed algorithm, Magma, is now documented to perform worse when added on top of the Muon optimizer both at 350M and 1B scale, with the performance degradation to increase at the 1B scale. This is very problematic as Magma was proposed as an optimizer add-on (i.e., base optimizer + Magma), thus achieving "on par" (as the authors now claim) or worse performance than simply running the base optimizer alone is not useful. There is also no "Magma-F" result provided for the 1B model.

The rebuttal also makes evident a very important confound: the unmasked Adam learning rate for 1d layers and embeddings is tied to the matrix learning rate in the paper's experiments. This is shown from the author's sentence below:

"In all experiments in the paper, Magma uses a single global learning rate for all parameter groups, so that it introduces no additional learning-rate hyperparameter over the base optimizer."

For the 130M experiment for example, this means that in author's the baseline Adam experiment, the 1d and embeddings use LR 5e-3, while in the Adam+Magma experiment the learning rate for the 1d and emebeddings is 3e-2, i.e. 6 times larger. It is important to emphasize again that in both experiments the 1d and embedding weights use the baseline Adam as auxilary optimizer without Magma (the latter being applied only to the hidden matrix parameters), but in the Adam+Magma run the auxilary Adam uses 6 times larger learning rate. This confounds the improvement from adding Magma to the hidden matrix parameters and using a higher learning rate for the 1d and embeddings, the latter being suggested to improve pretraining based from the literature [8,9].

The above confound could be a reason behind the discrepancy in my runs and the authors. In addition, I suspect the author's training code has an evaluation bug. Since the authors are probably using the GaLore training code unchanged, that code has a bug in the evaluation function in line 103 (https://github.com/jiaweizzhao/GaLore/blob/master/torchrun_main.py#L103C1-L103C22), where total_batches is wrongly initialized to 1 instead of 0, giving incorrectly lower loss results and making the evaluation dependent on the number of GPUs used. Nevertheless no code is provided in the submission so it is not possible to check their exact configuration from my side.

Misleading comparison with outdated competitive benchmark records, despite adpoting hyperparameters for their run from recent records:

Even on the requested competitive benchmark (modded-NanoGPT), the authors compare to very outdated records (AdamW (#2) 5625 and Muon (#1) 3600). For both Muon (#36 3250 steps) and Adam (pull request #353 4950 steps), the current records outperform the "plus Magma/Magma-F" supplied results. It is in my opinion very misleading to compare their newly tunned results with the oldest Adam and Muon baseline records, especially since it seems that the authors adopt hyperparameters from more recent, better performing records. For instance, the authors adopted lr=0.035 and wd=0.025 for their Muon+Magma-F runs, which are used in the more recent baseline Muon record number #12 (3325 steps), while they include in the table the old Muon record #1 (3600 steps) with suboptimal lr and wd. A table doing a fair comparison should be as below:

MethodSteps to reach loss target (lower is better)
AdamW (pull request #353)4950
AdamW+Magma5375
Muon (record #36)3250
Muon+Magma-F3375

Overall I firmly maintain my recommendation for clear rejection with the highest confidence 4: The authors provided a rebuttal minutes before the deadline, prohibiting any discussion. Further, the newly updated paper shows that the original method "Magma" when added to the base optimizer can indeed result to worse performance when tested on modern pretraining settings, a result contradicting the premise of the original submission. Even the newly proposed method "Magma-F" (which in my opinion should be discarded for this submission as it does not appear anywhere originally) is reported to give only very marginal improvement in some smaller models, which is also unclear if it is due to the learning rate confound. Finally, the competitive benchmarking results compare to very outdated records, and do not show improvement over the current ones, even when using "Magma-F". Importantly, I find very misleading the fact that the authors used hyperparameters from a more recent Muon record to run their method, and yet compared their result with the oldest (and weakest) Muon record available.

[8] Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Size. Hayou et al., 2025.

[9] Scaling Exponents Across Parameterizations and Optimizers. Everett et al., 2024.

Questions for authors

Can the authors provide results (along with their hyperparameters) using the proposed wrapper to speedup the current Muon/Adam records in the Modded-NanoGPT Optimization Benchmark ( https://github.com/KellerJordan/modded-nanogpt/tree/master/records/track_3_optimization) ?

2 comments

Thank you for the detailed review and for re-running our 130M setup under mixed precision. We followed your suggestions closely. The revised manuscript is uploaded, with changes in magenta. We hope the clarifications and additional experiments address your concerns, and we would be grateful if you would take them into account in your final assessment.

1) Mixed-precision LLaMA-130M: with a single global learning rate, Adam + Magma (21.25) outperforms tuned Adam (21.47).

We investigated the reviewer's mixed-precision (torch.amp, FP32 master weights and optimizer states with BF16 autocast) LLaMA-130M results and identified a plausible source of the discrepancy. The reviewer's custom implementation seems to freeze the learning rate of unmasked 1D/embedding parameters at 3e-3 while sweeping a separate learning rate only on the Magma-wrapped 2D blocks. In all experiments in the paper, Magma uses a single global learning rate for all parameter groups, so that it introduces no additional learning-rate hyperparameter over the base optimizer. Appendix C.1 was not explicit about this, and we now state it there.

We tested this diagnosis directly with two setups (split LR and single LR) and obtained the following results.

Optimizer & settingLearning rateVal PPL
Adam (reviewer)3e-321.78
Adam (best of lr \in {1e-3, 3e-3, 5e-3, 1e-2, 3e-2 } )5e-321.47
Adam + Magma (reviewer)3e-3 (Adam) + 1e-2 (Magma)21.95
Adam + Magma (split LR)3e-3 (Adam) + 1e-2 (Magma)21.91
Adam + Magma (single LR; ours)3e-2 on all parameters21.25

The split protocol matches your number (21.91 vs. 21.95). The single-LR protocol beats Adam at both your learning rate and our tuned one.

On the BF16 baselines in Table 1. We agree that pure BF16 hurts Adam's moment accumulation more than that of optimizers with fewer states (Adafactor, RMSProp). Table 1, however, makes a within-family claim: each optimizer is compared with and without Magma under identical precision (Magma is best in its family in 26/28 cases), so it does not depend on which optimizer family BF16 favors. The results above and in 2) test the same claim under mixed precision.

Adafactor + Magma at 1B: a reporting error, now corrected (13.12 → 13.50). Thank you for catching this. This one entry was logged with the original GaLore evaluation script, which does not evaluate on the full validation set; we confirmed that all other entries of Table 1 use our corrected full-set evaluation. The corrected value still improves on Adafactor (15.00) and is no longer the best 1B result (Muon + Magma: 13.22). Table 1 is updated.

2) Modern pre-training regime: Magma is on par with AdamW, SOAP, and Muon, and a new factored variant (Magma-F) improves all three at 350M.

Following your suggestion, we extended our evaluation to a modern pre-training setup featuring EOS-packed context length 1024, large global token batch sizes (1.05M tokens/step at 350M; 2.10M tokens/step at 1B), and mixed precision (FP32 weights, BF16 activations). The token budget follows the Chinchilla ratio of about 20 tokens per parameter (Hoffmann et al., 2022), and the batch configuration below makes full use of a 4×GB200 server.

ScalePer-GPU BSGrad AccumGlobal BS (Tok/Step)Steps (Total Tokens)
350M6441,024 (1.05M)6,670 (6.99B)
1B6482,048 (2.10M)9,530 (19.99B)

*Note: Relative to Table 1, the global token batch is 8× larger at 350M (16× at 1B), and the total number of parameter updates drops from 60k to 6,670 at 350M and from 100k to 9,530 at 1B. In this setting, Muon and SOAP exploit large-batch gradient geometry effectively per step and significantly outperform AdamW. The setting therefore also restores the the optimizer ordering you expected (Muon and SOAP ahead of AdamW).

Optimizer350M: Base350M: + Magma350M: + Magma-F1B: Base1B: + Magma
AdamW19.00 (1e-3)18.92 (3e-3)18.14 (3e-3)14.14 (1e-3)13.99 (3e-3)
Muon16.33 (3e-3)16.37 (1e-2)16.21 (1e-2)12.28 (3e-3)12.41 (1e-2)
SOAP16.33 (3e-3)16.26 (1e-2)16.27 (1e-2)12.76 (1e-3)12.36 (1e-2)

In the above table, Block-wise Magma is on par rather than consistently better as in Table 1: slightly better for AdamW and SOAP, slightly worse for Muon. We conjecture that 8–16× larger batches combined with mixed precision training substantially reduce the gradient noise that Magma attenuates. With roughly one-tenth as many updates, Magma also has less opportunity to compensate for its slower progress early in training (cf. Figure 2). Nevertheless, Magma-F improves all three optimizers at 350M (AdamW 19.00 → 18.14; Muon 16.33 → 16.21; SOAP 16.33 → 16.27). With less noisy large-batch gradients, alignment can be estimated at a finer granularity than a whole block. Magma-F keeps the block-wise mask (p = 0.5) but replaces the single damping factor per block with a rank-one re-weighting across its rows and columns, normalized to unit RMS (τ=0.5\tau=0.5; one extra scalar per row and column). We evaluated Magma-F at 350M only.

We appreciate the suggestion from the reviewer, and please refer to Section 3 (Magma-F), Section 4.3 with Tables 2–3 (results), and Appendices C.3 and C.6 (full configuration).

3) On the modded-NanoGPT benchmark: with tuned Magma, the AdamW and Muon baselines reach the 3.28 target in fewer steps

BaselineBaseline: steps (val loss)With Magma: steps (val loss)Settings
AdamW (#2)5,625 (3.2790)5,375 (3.2660)Magma: p = 0.9, τ\tau = 4, lr = 3e-3, wd = 0.3, betas = (0.9, 0.95), warmup = 250
Muon (#1)3,600 (3.2777)3,375 (3.2735)Magma-F: p = 0.85, τ\tau=0.5, lr = 0.035, wd=0.025, 1.5× lr for the auxiliary Adam

These results use Magma settings tuned for this benchmark. The leaderboard is heavily co-optimized around one configuration (a 124M GPT-2 on FineWeb, tuned for dense updates), and tuning alone moves its records by a similar margin: for instance, Muon #6 reaches 3.2788 at 3,375 steps with the same Muon algorithm. However, in the modern pre-training setup, we cannot afford such heavy hyperparameter optimization. This is why the paper keeps a single Magma configuration (p = 0.5; τ=2\tau=2 for Magma and τ=0.5\tau = 0.5 for Magma-F; applied to attention and MLP blocks) across all experiments and tunes only the learning rate. We therefore regard a single speedrun record as complementary evidence rather than the main test of a general method. We report it in this response rather than as a section of the paper, and treat the systematic evaluation in 2) as the primary evidence.

Reviewer_emfvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Magma, an optimizer wrapper which randomly masks block-wise updates using a EMA-smoothed alignment score between the gradient and momentum. They theoretically show that random masking updates introduces an implicit regularizer on updates along high-curvature directions. The performance of Magma is empirically validated across various base optimizers on Llama and Nano MoE setups, as well as in a heavy-tailed noise setting. They also validate Magma in the quadratic setting, suggesting that Magma is most effective in settings with heterogeneous and ill-conditioned curvature.

Critical strengths
  • Magma is a simple intervention connecting prior work in randomly masking block-wise optimizer updates (SkipUpdate) and incorporating directional information from momentum-gradient alignment (Cautious, MGUP optimizers)
  • The writing is clear and organization is easy to follow, with experiments spanning seven optimizers, four model sizes, and dense and MoE architectures and additional ablations showing learning rate stability and masking/damping comparisons.
Critical weaknesses
  • Ablation studies show that different masking components can yield different performance benefits and the authors specifically choose Attn + MLP masking with additional ablations (on a limited RMSProp, 130M model parameter setup) in Appendix D, but it is unclear what the reasons for certain combinations being more synergistic and how these results extrapolate to other model scales and optimizers.
Questions for authors
  • Could authors clarify whether Magma empirically applies (sbmbΔb)(s_bm_b\Delta_b) or (sbmbΔb/p)(s_bm_b\Delta_b/p)? Algorithm 1 and Equation 4 show the former but I believe Proposition 2 and the convergence analysis use the latter, so changing p at a fixed learning rate affects mean update magnitudes as well. For the sampling-probability sweep in Appendix D.3, were learning rates independently tuned for each p, or were the affected blocks’ learning rates adjusted to compensate for the factor in the expected update?
  • How would these results transfer to continual learning setups, where new task shifts can introduce useful gradients but disagree with momentum accumulated on the previous task?
2 comments

We sincerely thank Reviewer emfv for their positive assessment of our work, for recognizing how Magma bridges stochastic block-wise update masking with directional alignment, and for closely engaging with our theoretical analysis in Appendix A. Below we address (1) the choice of Attn + MLP masking, (2) the 1/p factor, (3) learning-rate tuning across p, and (4) continual learning. We have uploaded a revised manuscript, with changes marked in magenta.

1) Why Attn + MLP, and does this choice extrapolate to other scales and optimizers?

It extrapolates: Attn + MLP is the best subset in all eight settings (four scales × two optimizers). We extended the 130M RMSProp ablation to {60M, 130M, 350M, 1B} × {RMSProp, Muon}, with Magma (p = 0.5, τ\tau = 2) applied to one of six parameter subsets and the learning rate tuned over four values in every setting. Batches are 2–8× larger than in Table 1, with proportionally fewer steps (same token budget), so absolute values differ from Table 1. Validation PPL at the best learning rate (DIV: diverged at every tested learning rate):

Magma applied toRMSProp 60MRMSProp 130MRMSProp 350MRMSProp 1BMuon 60MMuon 130MMuon 350MMuon 1B
None (base optimizer)32.1530.3316.92DIV29.1922.6116.5613.59
Non-Attn/MLP32.8636.91DIVDIV31.9524.1216.9913.87
MLP only32.2530.15DIVDIV28.9723.4917.9214.37
Attn only31.1625.7016.9913.9928.6722.9316.1213.37
Attn + MLP (default)30.7425.3416.5013.4228.1621.9515.6912.92
All30.9827.8116.9213.9428.5922.6716.1713.39

Table shows that Attn + MLP is 0.24–0.66 PPL ahead of the runner-up in every setting, and it is the only subset that improves on the base optimizer in every setting. Also, One sublayer is not enough: Attn only is 0.36–0.98 behind; MLP only diverges with RMSProp at 350M and 1B and is worse than base Muon at 130M–1B. Finally, masking the remaining parameters hurts since All is 0.24–2.47 behind, and Non-Attn/MLP (token embeddings, LM head, RMSNorm gains) never improves on the base optimizer.

Why (intuition). Masking a block penalizes the curvature along its update (Proposition 2), and prior analyses locate the heterogeneous, ill-conditioned curvature of Transformers in the attention and MLP matrices (Zhang et al., 2024b; Orvieto & Gower, 2025; Pan & Li, 2023; cf. Sections 3 and 4.4). Every residual layer composes these two sublayers, so regularizing only one leaves the other unregularized. The remaining parameters (token embeddings, LM head, normalization gains) do not carry this curvature, so masking them mainly withholds updates.

We have added these ablations to Appendix D and state the design choice explicitly.

2) Q1 (Part1). Is the update applied with or without the 1/p factor?

Magma applies $s_b m_b \Delta_b $ without the 1/p factor, exactly as in Algorithm 1 and Eq. (4). Only the analysis (Proposition 2, Theorem 6) uses the 1/p-normalized form, which makes the masked update unbiased for the alignment-scaled update and keeps Proposition 2 directly comparable to Proposition 1 for SkipUpdate. For the masked blocks, the two forms are the same up to the learning rate: the implementation with learning rate $\eta $ equals the 1/p-normalized form with learning rate pηp \eta. We added this clarification to Appendix A.1, before Proposition 2.

3) Q1 (Part 2). In the p sweep, were learning rates tuned independently for each p?

In our original sampling probability sweep, the learning rate was held fixed. As you rightly pointed out, changing the survival probability p at a fixed learning rate alters the expected update magnitude, conflating sparsity sensitivity with effective step-size scaling.

In a new sweep with the learning rate tuned for every p at all four scales, we found that p matters little and p = 0.5 remains a good default. Specifically, for sweep (p \in {0.3, 0.4, 0.5, 0.6, 0.7}, lr \in {3e-4, 1e-3, 3e-3}), we obtained:

Keep probability p60M130M350M1B
Base RMSProp (no Magma)32.1530.3316.92DIV
1.0 (damping only, no masking)30.3824.6316.5113.92
0.730.6424.9316.6213.50
0.630.7225.3316.9313.58
0.5 (default)30.7425.3416.5013.42
0.430.5025.4116.4113.32
0.330.6425.1416.8113.68

In the above table, the gap between the default and the best p is 0.5–1.6% in relative terms. Magma improves on base RMSProp for every p at every scale, with one tie (350M, p = 0.6: 16.93 vs. 16.92); at 1B, base RMSProp diverges at every tested learning rate. Full results are in Table A1 (Appendix D); the trend across model scales is discussed there and in our response to Reviewer 6y6p.

4) Q2. How would the results transfer to continual learning?

We have not tested this. We conjecture that alignment damping would mitigate catastrophic forgetting, at the plasticity cost you describe; we added this discussion to the Conclusion as future work. At task boundaries, distribution shift often produces large gradients that overwrite parameters along sharp directions important to earlier tasks (Kirkpatrick et al., 2017). Magma's alignment damping can, in principle, attenuate such abrupt updates that conflict with the accumulated momentum, as an analogue of projection-based methods such as GEM / A-GEM (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019) and OGD (Farajtabar et al., 2020). As you note, the same damping may initially attenuate useful new-task gradients. How Magma balances stability and plasticity deserves a dedicated empirical study; this paper keeps its empirical scope on LLM pre-training.

Reference.

Kirkpatrick et al., 2017. Overcoming catastrophic forgetting in neural networks. PNAS.

Lopez-Paz & Ranzato, 2017. Gradient Episodic Memory for Continual Learning. NIPS.

Chaudhry et al., 2019. Efficient Lifelong Learning with A-GEM. ICLR.

Farajtabar et al., 2020. Orthogonal Gradient Descent for Continual Learning. AISTATS.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.