acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Memory-Efficient Optimizers for LLM Pre-Training

Abstract

A growing family of memory-efficient optimizers reduces state storage through gradient projection, moment factorization, and shared statistics. Yet their practical value depends on when optimizer states constrain training and whether structural compression offers an advantage over simple state quantization. We examine these questions through a broad benchmark of memory-efficient methods and strong conventional optimizers under a common language-model pretraining setup. Our evaluation covers 257M and 500M dense models and extends to mixture-of-experts architectures, varying compute and state precision independently. We investigate how compression strength affects training quality and compare the approaches in terms of peak memory, validation loss at fixed token budgets, and time to quality. In the main 500M comparison, Muon with FP8 states achieves lower validation loss and lower peak training memory than the evaluated structural methods with FP32 states under both BF16 and FP8 computation. Quantizing the states of structural methods remains feasible, but yields limited further memory savings over quantized-state Muon without closing the quality gap. These findings support quantizing a strong optimizer as a practical starting point for memory-constrained pretraining and make competitive quantized baselines essential to assessing the benefits of structural compression.

Primary area:
datasets and benchmarks
Keywords:
optimizer state compression, optimizer state quantization, memory-efficient optimization, low-precision training, FP8 training, large language model pretraining, matrix-aware optimizers

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yVATOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper reassesses the practical value of structural optimizer-state compression relative to direct state quantization. The authors independently vary compute and state precision and compare optimizers in terms of validation loss, peak memory, and training time. The main finding is that Muon with FP8 states often achieves a better quality–memory trade-off in the tested settings, leaving limited advantages for structural methods over this strong baseline.

Critical strengths
  1. By distinguishing optimizer-state size from overall peak memory, the paper shows why nominal compression ratios can overstate actual training-memory savings and provides a more useful basis for comparison.
  2. The authors include strong optimizers with quantized states and also evaluate state quantization for structural methods, making the comparison more relevant to practical optimizer selection.
  3. The evaluation covers a broad range of relevant factors, including compute and state precision, microbatch size, compression strength, training budget, matched-time performance, and MoE models. This helps clarify the conditions under which different methods are useful.
Critical weaknesses
  1. The scope of the quality evaluation limits the broader conclusions. Dense-model quality comparisons stop at 500M parameters and primarily use FineWeb-Edu. The MoE results provide useful additional evidence, but timing and memory measurements on larger models do not establish that the quality ranking persists at those scales.
  2. Close comparisons lack sufficient uncertainty estimates. The seed experiments in Tables 11–12 cover only AdamW and Muon at 257M and do not establish the variability of structural methods. In Table 15, Muon and SlimAdam achieve losses of 2.824 and 2.827 after extended training. Repeated runs are needed to determine whether this small difference is consistent.
  3. Hyperparameter selection may affect the central comparison. In Table 24, AdamW, Muon, and SOAP use weight decay of 10−410^{-4}, whereas several structural methods use 0.1. Different settings are not inherently unfair, but most hyperparameters were transferred from 257M without systematic retuning at 500M. The authors should establish that the main ranking remains robust under comparable tuning budgets.
Questions for authors

Can the closest structural competitors be evaluated across multiple seeds and at a larger dense-model scale, including comparisons under the same memory limit?

3 comments

We thank the reviewer for the careful reading and concrete suggestions. We are glad that the reviewer found that separating optimizer-state size from peak memory gives a more useful basis for comparison. We are also glad they valued that we quantize both the strong optimizers and the structural methods, across compute and state precision, microbatch size, budget, matched time and MoE. We address the concerns below.

W1: Scope of the quality evaluation.

  • Larger model: we added 1B experiments, shown below. At 1×C1\times C, SlimAdam's loss gap to FP8-state Muon persists from 257M to 500M to 1B: 0.028→0.034→0.0330.028 \to 0.034 \to 0.033 with FP32 states, and 0.026→0.032→0.0340.026 \to 0.032 \to 0.034 with FP8 states. The gap grows slightly from 257M to 500M and remains similar at 1B.

1B dense model, W16A16G32, 1×C1\times C: final validation loss.

Optimizer (state precision)Loss
Muon (FP32)2.575
Muon (FP8)2.577
SlimAdam (FP32)2.610
SlimAdam (FP8)2.611
AdamW (FP8)2.632
  • Data: We use FineWeb-Edu as a common pretraining corpus for controlled optimizer comparisons. This follows Semenov et al. (NeurIPS 2026), whose main experiments use FineWeb, and Zhao et al. (ICLR 2025), who train on C4 only. Our evaluation already includes downstream tasks and WikiText-103 (Appendix A.4): HellaSwag accuracy closely tracks validation loss (Spearman −0.99-0.99 across 55 endpoints), and WikiText-103 yields the same optimizer ranking. These results support the comparison across evaluation datasets, while training on other corpora remains future work.

W2: Uncertainty for close comparisons.

We now report run-to-run variance at 500M for the closest structural competitors:

  • Absolute loss differences between two data-order replicas: AdamW 0.000096, Muon 0.0000014, SlimAdam 0.000135, FRUGAL 0.002760.
  • Three initialization seeds of Muon (FP8 activations): standard deviation 0.0009.

Run-to-run spread is at most 0.003. The Muon-SlimAdam gap (0.033-0.035) is more than ten times that, the projection methods trail by at least 0.09, and the ranking is identical in both data orders.

On Table 16, we agree that 2.824 vs 2.827 is within noise, and we will say that SlimAdam matches FP8-state Muon at 257M 8×C8\times C. At the paper's main budgets the gap is consistent: SlimAdam trails Muon by 0.032-0.035 at 500M and 0.034 at 1B.

W3: Comparable tuning budgets.

Weight decay is now tuned for every structural method at 500M (General Response, Table 1). Several improved, which the paper reflects, but the ranking is the same. It holds whether every method uses the baselines' 10−410^{-4} (gaps 0.035-0.161) or 0.1 (0.012-0.119), and 10−410^{-4} is also SlimAdam's best value.

Learning rates were tuned per optimizer at 257M, and at 500M Muon's transferred rate is the best of a six-point sweep. This supports learning-rate transfer over our modest model-size range, from 257M to 1B.

Q1: Seeds, larger scale, and comparison under the same memory limit for the closest competitors.

Seeds: see W2. Larger scale: General Response, Table 2, which includes SlimAdam at 1B with FP32 and FP8 states.

Under the same memory limit, the comparison does not change:

  • Measured memory: we computed the largest microbatch that fits a fixed budget (our code, 500M, full checkpointing). At 24 GB, FP8 Muon fits 30 sequences. The structural methods fit 26-29 with FP32 states and 29-31 with FP8 states; at 80 GB, all are within 3% of FP8 Muon. At the same limit, structural compression buys at most a 3% larger microbatch, at 0.03-0.16 higher loss.
  • Nominal state bytes: the main comparison is iso-memory by construction. Two FP32 moments at retained fraction ρ=0.25\rho = 0.25 take the same bytes as two dense FP8 moments (Section 4.3).
  • More state memory does not close the gap: raising ρ\rho to 1 quadruples FRUGAL's projected state, yet it reaches only 3.059 against Muon's 2.989 at 257M (Appendix B.4, Figure 7).

We hope the additional experiments and clarifications resolve your concerns. We would appreciate your reconsideration of the score based on the new results. We would be glad to clarify anything further.

I sincerely thank the authors for their detailed response and additional experiments, which have substantially alleviated my earlier concerns. The new 1B experiments provide further support for the paper’s conclusions, although this scale remains considerably smaller than that of modern large language models. Whether these conclusions generalize to larger models therefore remains to be established. I find it difficult to assess whether scaling would change the main comparative results, so I retain this concern and would appreciate the Area Chair’s overall assessment on this point.

Dear Reviewer yVAT,

Thank you for the follow-up. We are glad that the additional experiments substantially alleviated your earlier concerns. Below, we would like to present our evidence regarding the scaling of our results.

1. Within our range, the gap of the closest competitor does not narrow. With FP8 states, SlimAdam's gap to Muon is 0.026→0.032→0.0340.026 \to 0.032 \to 0.034 from 257M to 500M to 1B. Hoffmann et al. (NeurIPS 2022) emphasize that the model sizes used to estimate scaling trends matter. Their analysis, based mostly on models above 500M and extending to 16B parameters, yields different compute-optimal scaling predictions from Kaplan et al. (2020), whose experiments included many models below 100M. Our experiments extend to 1B parameters, providing evidence beyond this very small-model regime.

2. Large-scale training recipes are chosen with experiments at this scale (~1B).

  • Falcon-H1 informs its architecture with ablations and grid searches on 300M–1.5B proxy models, and transfers hyperparameters tuned on a base model of width 1280 to its 34B model (width 5120).
  • DeepSeekMoE validates its architecture on MoE models with about 2B total and 0.3B activated parameters before scaling it to 16B and 145B; DeepSeek-V3 (671B) employs the DeepSeekMoE architecture.
  • OLMo 2 developed its recipe on a 1B model and found the findings "to generalize well to the 7B, 13B and 32B scales".
  • Attention Residuals validates its design in scaling runs with 194M–528M activated parameters, then trains a 48B-parameter model (3B activated) on 1.4T tokens, where it "matches or outperforms the baseline on all benchmarks".
  • DCLM (NeurIPS 2024) finds that data-curation results at its 1B scale (1.4B parameters) correlate with those at 7B (Pearson r=0.956r = 0.956).

3. Systematic optimizer comparisons are useful at this scale. We agree that experiments on larger models would strengthen our conclusions. However, repeating the full comparison at 7B exceeds our available compute budget. Other detailed optimizer benchmarks also operate at comparable scales: Semenov et al. (NeurIPS 2026) study models up to 720M parameters, while Wen et al. (ICLR 2026) and Zhao et al. (ICLR 2025) study models up to 1.2B. Such studies are widely cited, indicating their value to the community in comparing methods under controlled conditions and selecting promising approaches for larger-scale validation.

4. Compute. Each of our 1B runs took 152–189 H100-hours. A 7B run at 1×C1\times C needs about 49×49\times the FLOPs, so repeating our 36-configuration 500M comparison at 7B would require roughly 150,000–300,000 H100-hours.

We hope this helps the overall assessment.

Best regards,

The authors

Reviewer_7HcMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether optimizer states should be compressed structurally (low-rank projection, factored or shared moments) or simply stored in lower precision. It screens many optimizers on both dense and mixture-of-experts models. Measured peak memory shows state compression mostly helps at small microbatches. Quantizing AdamW states to FP8 cuts peak memory by 31.3% at a microbatch of 1 sequence but only 4.2% at 32. FP8-state Muon reaches lower loss and less peak memory than every structural method, and remains the stronger practical choice among the methods evaluated.

Critical strengths
  • Measuring real peak memory instead of only state size makes the results convincing: Measuring across microbatch sizes and compute precision shows that nominal state savings could overstate end-to-end savings. This corrects how memory-efficient optimizers are usually evaluated.
  • A broad comparison with a mostly consistent conclusion: In Table 21, projection-based optimizer methods trail FP8-state Muon by more than 0.1 validation loss. This is much larger than the seed noise measured for AdamW and Muon (the validation loss difference across seeds is < 0.002, Tables 11–12).
Critical weaknesses
  • The structural methods may be under-tuned: Hyperparameters were mostly chosen from sweeps at 257M (See Limitations), and at 500M the baselines use weight decay 1e-4 while the structural methods use 0.1 (see Table 24). It is unclear whether the methods are under-tuned.
  • Limited novelty: The paper offers no new method, and the main finding (quantizing a strong optimizer beats structural compression) is largely expected from prior work such as 8-bit Adam.
Questions for authors
  • Was activation checkpointing enabled in the peak-memory measurements? It is popular in industry and would raise the runtime memory, which could change the memory conclusions.
  • Why do the 500M baselines use weight decay 1e-4 while the structural methods use 0.1? Results with matched settings would help better explain whether the performance gap is real.
1 comment

We thank the reviewer for the thoughtful review. We are glad that the reviewer found our measurements of real peak memory, across microbatch sizes and compute precision, convincing. We are also glad they found the broad comparison consistent, with projection methods trailing FP8-state Muon far beyond seed noise. We address the concerns below.

W1 / Q2: Are the structural methods under-tuned? Why different weight decay?

We tuned weight decay for every structural method at 500M (General Response, Table 1). Several of them improved, and we have updated all affected tables and figures in the revised paper, but the comparison is unchanged. Muon leads every structural method by 0.032-0.154, and the closest competitor, SlimAdam, matches AdamW (FP8) but not Muon.

The matched settings you ask for give the same answer in both directions:

  • At the baselines' 10−410^{-4} (FP32 states): Muon 2.761, AdamW 2.793, SlimAdam 2.796, APOLLO 2.857, GaLore 2.860, FRUGAL 2.922.
  • With every method at 0.1 (2×C2\times C, equal warmup): Muon leads every structural method by 0.012-0.119.

The baselines use 10−410^{-4} because it works better for them. In matched 500M runs it beats 0.1 for AdamW (2.743 vs 2.774 at 2×C2\times C) and Muon (2.699 vs 2.742).

W2: Contribution

We respectfully disagree that the result follows from 8-bit Adam:

  • The structural-compression literature claims the opposite under its own evaluations. APOLLO reports "SGD-like memory, AdamW-level performance", and GaLore 2 reports parity with 8-bit Adam at 7B. None of these works compares against a quantized matrix-aware optimizer such as Muon, or measures peak training memory across microbatch sizes. Under a common setup, the projection methods trail Muon by 0.09-0.15 at 500M. If this followed from 8-bit Adam, such comparisons would already be standard.
  • Quantization is not optimizer-agnostic. AdEMAMix diverges with FP8 states under BF16 compute despite several stabilization attempts (Appendix A.5). "Quantize the strongest optimizer" was therefore not a foregone conclusion.
  • The contribution is an evaluation protocol and what it shows. It measures peak memory across microbatch size, compute precision and checkpointing. It compares methods at matched precision and budget against quantized strong baselines. And it shows that the one structural idea that survives (SlimAdam's shared second moments) complements quantization rather than replacing it. As the reviewer's first strength puts it, this "corrects how memory-efficient optimizers are usually evaluated".

Systematic studies that resolve gaps in optimizer evaluation are established contributions at leading conferences, including Semenov et al. (NeurIPS 2026), Wen et al. (ICLR 2026), and Zhao et al. (ICLR 2025). Our study addresses the corresponding gap in memory-efficiency evaluation through comparisons with quantized strong baselines and measured peak training memory.

Q1: Was activation checkpointing enabled?

No. Tables 1 and 3 are measured without activation checkpointing. Checkpointing lowers activation memory, so it increases the share of peak memory held by optimizer state, the regime where state compression should help most. We therefore measured it. The setup is the code behind Table 1, 500M, BF16 compute, all 15 optimizer/state configurations and microbatches 1-32, with no, selective (MLP recompute) and full checkpointing, each with and without CPU offload. At microbatch 32:

Microbatch 32FP8 Muon peakAdamW FP32 →\to FP8 savingFP32-state structural methods vs FP8 Muon
no checkpointing65.19 GB4.2% of peak+0.69 to +2.71 GB
selective41.74 GB6.3%+0.70 to +2.72 GB
full24.82 GB10.2%+0.70 to +2.72 GB
full + CPU offload19.00 GB12.8%+0.70 to +2.72 GB

The memory conclusions hold:

  • The offsets stay the same in GB while activations shrink around them. FP8 Muon stays below every FP32-state structural method; with full checkpointing this holds from microbatch 2 up.
  • With FP8 states for every method (microbatch 32, full checkpointing), the structural methods range from 1.3% below to 2.5% above FP8 Muon, and all have higher loss.

What checkpointing does change is how much state compression matters. Quantizing AdamW's states saves 10-13% of peak memory at microbatch 32 under full checkpointing, against 4% without. Section 3 now presents checkpointing as a regime where state compression matters at moderate microbatches, not only small ones.

We believe these results address the reviewer's concerns, and we would kindly ask the reviewer to reconsider their score. We would be glad to clarify anything further.

Reviewer_St7POpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies memory efficient optimizers that reduce state storage by gradient projection, moment factorization and shared statistics. This is in comparison to other methods that reduce the state storage just by quantization for example. The author's benchmark both setups for both dense (250M and 500M params) and mixture of experts architectures. The paper finds that FP8-state Muon and AdamW reach lower loss and lower measured peak memory than the structural methods at practical microbatch sizes, and that state compression of any kind only matters when activations don’t dominate memory.

Critical strengths
  1. The baselines are strong and the memory measurements are very applicable practically. Comparing the optimizers using the measured peak training memory instead of just the optimizer state storage as an efficiency proxy is quite practical. The paper includes a wide range of baselines e.g Muon, SOAP etc. Also having quantization hte structural methods as well is a good control experiment.

  2. The authors have a broad evalaution in their paper. The authors sweep model size, precision, optimizer state precision, architecture etc. which allows to further strengthen their arguments. The matched-training-time comparison is also great!!

  3. The paper is transparent about failures and limitations e.g AdeMaMix with FP8 states becomes unstable under BF16 computation, being an exception to their findings.

Critical weaknesses
  1. Hyperparameter Control. It is a bit unclear to me how the different hyperparams for each optimizer are chosen. Are these the best values for each one or do you sweep them? In table 24 different methods have different weight decay, beta2 and clip, how does this affect the conclusions ?

  2. The dense experiments stop at 500M params and the longer-horizon and MoE extensions cover only four optimizers. This is a bit counter-intuitive since the papers for these methods claim that the gap narrows with scale and more training tokens. Would love to know wether the conclusions from this paper hold at scale.

Questions for authors
  1. How much of the peak training memory is not due to the optimizer itself but something under the hood ? e.g the way tensors are allocated in pytorch etc can lead to varying peak training memory even if this is not due to the optimizer.

  2. From above, how do you select the hyperparameters for each optimizer ? are those you pick the best ones for each optimizer?

3 comments

We thank the reviewer for the positive assessment and the constructive questions. We are glad that the reviewer found our measured peak-memory comparison practically applicable and our baselines strong, including the quantized structural methods as a control. We are also glad they appreciated the matched-training-time comparison and our transparency about failure cases such as FP8-state AdEMAMix. We address the questions below.

W1 / Q2: How were hyperparameters chosen, and do the differences in Table 25 matter?

Learning rates were swept separately for every optimizer at 257M (Appendix D, Figure 20) and transferred to 500M. For Muon we re-swept at 500M, and the transferred value remained the best (General Response). Weight decay is now tuned per method at 500M (General Response, Table 1). It improved several structural methods, which we reflect in the paper, but Muon remains clearly ahead of all of them.

The other differences in Table 25 follow each method's paper or released code and do not drive the conclusions:

  • Clipping: all compared methods clip at 1.0. Only AdEMAMix, a baseline, uses 0.5 (needed for stability, Appendix A.5). Clipping therefore cannot explain any structural-vs-baseline gap.
  • β2\beta_2: GaLore, FRUGAL and APOLLO use their published 0.999; FRUGAL MM and tuned SlimAdam use 0.99, like the baselines. For Muon, the tabulated β2\beta_2 applies only to the auxiliary AdamW updates of embeddings, output head and norms; its matrix updates use Nesterov momentum 0.95. In paired runs, changing β2\beta_2 from 0.999 to 0.99 slightly worsens GaLore's final validation loss (3.0679→3.06923.0679 \to 3.0692, +0.0013+0.0013) and slightly improves APOLLO's (3.0914→3.08553.0914 \to 3.0855, −0.0059-0.0059). Both changes are negligible relative to the gaps to Muon and do not alter the comparison.

W2: Do the conclusions hold at scale?

Yes, in the scale range we could test.

  • Model size: at 1×C1\times C, the loss gap between SlimAdam and FP8-state Muon persists from 257M to 500M to 1B: 0.028→0.034→0.0330.028 \to 0.034 \to 0.033 with FP32 SlimAdam states, and 0.026→0.032→0.0340.026 \to 0.032 \to 0.034 with FP8 states. Thus, the gap grows slightly from 257M to 500M and remains similar at 1B, rather than narrowing with model size.

1B dense model, W16A16G32, 1×C1\times C: final validation loss.

Optimizer (state precision)Loss
Muon (FP32)2.575
Muon (FP8)2.577
SlimAdam (FP32)2.610
SlimAdam (FP8)2.611
AdamW (FP8)2.632
  • Training tokens: we agree that longer training narrows gaps at a fixed size, and the paper shows it (Table 16, 257M, 1×C→8×C1\times C \to 8\times C). The gap to Muon falls from 0.032 to 0.020 for AdamW, from 0.105 to 0.063 for FRUGAL and from 0.026 to 0.003 for SlimAdam. So SlimAdam catches up with Muon, while other methods stay clearly behind.
  • Four optimizers in the extensions: chosen for cost. They are the two references (AdamW, Muon), the closest structural method in loss (SlimAdam) and the one with the lowest memory (FRUGAL).

Q1: Implementation effects on memory

Additional profiling at microbatch sizes 1 and 32 shows that allocation rounding contributes only 0.037-0.106 GB in Megatron and 0.046-0.158 GB in our codebase at peak allocated memory.

At training microbatches, most of the peak is not the optimizer. At microbatch 32 with full checkpointing, the logits and their gradient alone take 16.5 GB of the 24.8-25.2 GB peak of FP8 Muon and FP8 AdamW, against 0.68 GB of Muon's optimizer state.

Allocator traces confirmed fragmentation as the cause of SOAP's OOM with FP32 states at microbatch 32. A 6.59 GB allocation in the loss backward pass failed despite 8.19 GB free within the allocator, because the largest free block was only 5.79 GB. With expandable segments, the same allocation succeeded at 76.55 GB peak allocated memory, without changing the optimizer. We used identical allocator settings across optimizers to ensure a fair comparison under the same memory-allocation conditions.

Implementation choices also affect peak memory through temporary tensors and workspaces. We used the authors' code for most optimizers and measured their memory usage under the same configuration within each codebase.

I appreciate the author's effort in providing a detailed rebuttal that has answered my concerns. I gave the paper an initial score of 3 to indicate my positive view of the paper and only raised minor concerns that the authors have now addressed. I will keep my score to reflect my initial positive view of the paper.

Dear Reviewer St7P,

Thank you for your positive assessment of our work. We are glad that our response resolved all of your concerns.

Best regards, The authors

Comments on the paper

2 comments

We thank all three reviewers for their careful reading and constructive comments. We are glad that all three found value in two core parts of our work: comparing optimizers by measured peak training memory rather than by optimizer-state size, and the breadth of the evaluation across model size, compute and state precision, microbatch size, training budget and MoE. Two requests were shared by several reviewers, and we address them here with new experiments. The remaining points are answered individually.

The revised paper adds dense-model scaling to 1B (Section 4.4, Figure 8), activation-checkpointing experiments (Appendix A.7, Table 4), and optimizer-state sharding experiments (Appendix A.8, Figure 19). We also clarify how checkpointing changes the value of state compression (Section 3) and distinguish model-size scaling from longer training. All new text is marked in blue.

1. Hyperparameters at 500M (St7P, 7HcM, yVAT)

We swept weight decay over {10−1,10−2,10−3,10−4}\{10^{-1}, 10^{-2}, 10^{-3}, 10^{-4}\} for all five structural methods at 500M. Each method now uses its best value, for every compute/state precision and both budgets.

Table 1. 500M, W16A16G32, 1×C1\times C, final validation loss.

Optimizer (state precision)Weight decayLoss at WD 0.1Loss at selected WDGap to Muon (FP8)
Muon (FP8)10−410^{-4}-2.764-
AdamW (FP8)10−410^{-4}-2.7900.026
SlimAdam (FP32)10−410^{-4}2.8192.7960.032
SlimAdam (FP8)10−410^{-4}2.8112.7960.032
APOLLO (FP32)10−210^{-2}2.9082.8540.090
GaLore (FP32)10−310^{-3}2.8822.8560.092
FRUGAL MM (FP32)0.1 (unstable below)2.9062.9060.142
FRUGAL (FP32)0.1 (grid flat, 2.917-2.922)2.9182.9180.154

Tuning improves GaLore, APOLLO and SlimAdam, and the revised paper reflects this in every affected table and figure. The overall comparison stays the same. Muon is ahead of every structural method by 0.032-0.154, more than ten times the run-to-run spread at 500M (at most 0.003, response to yVAT). The closest competitor, SlimAdam, matches AdamW (FP8) but still trails Muon.

Learning rates were tuned per optimizer at 257M (Appendix D, Figure 20). For Muon, we re-swept six learning rates from 5×10−45\times 10^{-4} to 10−310^{-3} at 500M. The transferred 10−310^{-3} remained best, with loss 2.761 versus 2.764-2.776 for the other values. This supports retaining the learning rate over our modest model-size range of 257M to 500M to 1B.

2. Scale (St7P, yVAT)

We trained 1B dense models for 1×C1\times C (about 20B tokens). All five runs use the same weight decay (10−410^{-4}), warmup (2,000 steps) and WSD schedule.

Table 2. 1B, W16A16G32, 1×C1\times C, final validation loss.

Optimizer (state precision)Loss
Muon (FP32)2.575
Muon (FP8)2.577
SlimAdam (FP32)2.610
SlimAdam (FP8)2.611
AdamW (FP8)2.632

The 500M picture carries over to 1B. Muon leads. The closest competitor, SlimAdam (FP32), beats AdamW (FP8) by 0.023 but trails Muon (FP32) by 0.034, as at 500M (0.035). FP8 states have little effect on validation loss for Muon and SlimAdam.

At 1×C1\times C with BF16 computation and native optimizer states, we compare the paper's results using each method's selected weight decay, including retuning at 500M. Each cell below is the method's final validation loss minus Muon's at the same model size. From 257M to 500M, the gap grows for GaLore, FRUGAL MM, FRUGAL, and SlimAdam, but narrows for APOLLO:

GaLoreAPOLLOFRUGAL MMFRUGALSlimAdam
257M0.0790.1040.0800.1000.028
500M0.0950.0920.1450.1570.037

Dear Reviewers,

Thank you for your contributions to the review process. As the author–reviewer discussion period ends on November 18, I would like to kindly remind everyone to review the authors' rebuttals and engage in the discussion.

In particular, please indicate whether your main concerns have been adequately addressed and clarify any remaining issues that may affect your final assessment.

Your feedback will be very helpful for the subsequent reviewer–AC discussion and meta-review process.

Thank you for your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.