acceptodds
Under review as a conference paper at ICLR 2027

DRAM: Memory-Efficient Muon via Rank Reallocation in Principal-Random Subspaces

Abstract

Memory-efficient Muon methods aim to reduce storage overhead while preserving strong performance, yet existing approaches compromise either convergence guarantees or practical efficiency. In this paper, we focus on subspace optimization for Muon, where the choice of subspace determines the gradient information retained for updating. To preserve dominant information while providing the coverage needed for stochastic convergence guarantees, we construct a principal-random subspace that combines leading singular directions of the stochastic gradient with random directions sampled from their orthogonal complement. Under a fixed rank budget, there is a trade-off between principal exploitation and complementary coverage, making rank allocation important for practical optimization. Our key observation is that the preferred balance shifts from principal-heavy early in training to random-heavy later. Motivated by this shift, we propose DRAM, a memory-efficient Muon optimizer that reallocates principal and random ranks across stages while maintaining momentum and performing matrix-sign normalization entirely in the subspace. Theoretically, we establish convergence guarantees for general rank-allocation schedules with nondecreasing coverage factors, obtaining rates of in the stochastic setting and in the deterministic setting, with fixed allocation as a special case. Using a simple linear reallocation schedule, DRAM achieves the best low-memory performance across three pre-training scales and nine fine-tuning settings.

Primary area:
optimization
Keywords:
large language models, Muon optimizer, memory-efficient training, stochastic convergence

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_xAGvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DRAM, a memory-efficient variant of Muon that stores and orthogonalizes momentum in a low-dimensional subspace. This subspace combines principal gradient directions with random directions from their orthogonal complement. The rank allocation gradually shifts from principal to random directions during training, motivated by the observed decline in gradient energy captured by the principal subspace (Figs. 1 and 3).

The authors establish convergence rates for the average gradient nuclear norm under smooth non-convex objectives and non-decreasing subspace coverage: O(T−1/4)O(T^{-1/4}) in stochastic settings and O(T−1/2)O(T^{-1/2}) in deterministic settings. The analysis assumes exact matrix-sign computation and accounts for data-dependent subspaces.

Experiments include LLaMA pre-training (60M–350M parameters), fine-tuning of 8–9B models on IFEval, GSM8K, and MBPP, and a three-seed schedule ablation. DRAM achieves the highest average accuracy among the evaluated low-memory methods while reducing peak memory usage by approximately 18–21% relative to Full-Muon.

Critical strengths
  1. Well-motivated design and careful theoretical analysis. DRAM maintains optimizer states and performs orthogonalization entirely within a low-rank subspace, without requiring periodic full-rank corrections. The convergence analysis uses the random complement to ensure sufficient coverage rather than assuming accurate principal subspaces. It also accounts for dependencies between data-dependent bases and gradient noise. I did not find any issues with the main theoretical arguments. Overall, this is a technically sound contribution to memory-efficient Muon training.

  2. Clear presentation and comprehensive experiments. The paper is well organized, with consistent notation and a useful comparison in Table 1. The experiments cover continued training, three pre-training scales, fine-tuning of three 8–9B models, and schedule ablations across multiple seeds. The authors also report cases where DRAM underperforms Full-Muon, including the 350M setting and LLaMA GSM8K. The motivation for adapting the subspace allocation is clearly explained and potentially useful in practice.

Critical weaknesses
  1. The empirical improvements are small, and the evaluation lacks sufficient statistical evidence.

    • Pre-training (Table 2): DRAM improves average accuracy over the strongest competing low-memory method by only 0.30, 0.08, and 0.10 points at 60M, 130M, and 350M, respectively, without reporting variance across seeds. Several benchmarks are near chance level, and validation loss or perplexity, standard metrics for pre-training, are missing from the main comparison.
    • Fine-tuning (Table 4): Results are based on single runs, with several differences corresponding to roughly one problem or less. The validation split and metric used for learning-rate selection are also unclear.
    • Schedule ablation (Table 5): The default Linear schedule improves validation loss over Principal-only by only about 0.001. Notably, the GoLore-style Switch-25% schedule performs better than Linear at both scales.
    • Memory: DRAM uses nearly the same peak memory as GaLore-Muon and GoLore-Muon (within 0.04 GiB). Its memory advantage is mainly over Full-Muon rather than comparable low-rank methods.

    Overall, while DRAM achieves the highest reported average accuracy among low-memory methods, the results do not yet establish a robust improvement.

  2. The novelty relative to GoLore is somewhat limited, and the theoretical and empirical evidence does not fully justify the proposed schedule.

    • Novelty: The idea of transitioning from principal to random projections during training builds directly on GoLore, while related approaches to combining principal and random directions already exist (e.g., Alice). The main contribution appears to be their combination with Muon's geometry and compact optimizer states.
    • Theory: Theorem 1 provides a convergence guarantee based on subspace coverage, but its bound is actually worse for the default schedule than for fixed or random-only allocations. Thus, the analysis does not explain why the proposed schedule should outperform these alternatives.
    • Comparisons: DRAM resets momentum when updating the subspace, whereas GaLore-Muon and GoLore-Muon handle optimizer states differently. This makes it difficult to isolate the benefit of the rank-allocation schedule.
    • Practical efficiency: SUMO and MoFaSGD are discussed but not evaluated. Wall-clock time and throughput are also missing, making it difficult to assess the practical efficiency gains.

    I would encourage the authors to strengthen the statistical evaluation, better isolate the effect of the proposed schedule, and clarify its practical advantages over existing methods.

Questions for authors
  1. Pre-training variance and loss: How many seeds were used for Tables 2 and 4? For at least one representative pre-training setting, could you report validation loss or perplexity, along with the variability in performance differences between DRAM and the baselines?

  2. Fine-tuning protocol: Which validation split and metric were used to select the learning rates in Table 9? Were the test benchmarks used for hyperparameter selection?

  3. Momentum handling: Is momentum handled consistently across all schedules in Figs. 1 and 3 and Table 5, including Principal-only and Random-only? Could you include an ablation comparing DRAM with and without momentum resets, or apply the same reset strategy to GaLore-Muon and GoLore-Muon, to isolate the effect of rank allocation?

  4. Linear vs. hard switch: Since Switch-25% outperforms Linear in Table 5, what evidence supports continuous linear reallocation over a GoLore-style hard switch, apart from requiring one fewer hyperparameter?

  5. Theory vs. schedule: How does the schedule-dependent constant in Theorem 1 compare across Linear, Fixed, and Random-only? Does the theory provide any justification for the proposed schedule?

  6. Practical efficiency: Could you report wall-clock time or throughput, including the cost of SVD and basis updates? A comparison with SUMO or MoFaSGD under a matched memory budget would also help establish the practical benefits.

3 comments
AuthorsOpenReview ↗

To Reviewer xAGv (1/2)

We thank the reviewer for the thorough and constructive review. We address your concerns below.


W1. The empirical improvements are small, and the evaluation lacks sufficient statistical evidence.

Response. We agree that multi-seed evaluations would provide stronger empirical evidence. Unfortunately, due to computational constraints and the limited rebuttal period, we are unable to complete these additional experiments at this stage.

Regarding memory efficiency, we would like to clarify that we intentionally match the memory budgets of subspace-based methods to ensure a fair comparison of their optimization performance. DRAM can further reduce memory usage by decreasing the subspace rank, although this may involve a trade-off with optimization performance.


W2. The novelty relative to GoLore is somewhat limited, and the theoretical and empirical evidence does not fully justify the proposed schedule.

Response. We appreciate this concern and would like to clarify our contributions.

First, regarding novelty, we highlight both algorithmic and theoretical differences.

  • Algorithmically, unlike GoLore and Alice, which focus on Adam-style optimizers or Momentum SGD, DRAM is specifically designed for Muon. Rather than relying solely on random projections as in GoLore's algorithm, DRAM explicitly integrates principal-random subspace construction into Muon. Furthermore, motivated by the stage-dependent preference shift observed in Fig. 1 and Appendix Fig. 3, DRAM introduces stage-wise rank reallocation, beyond GoLore's empirical hard switching and Alice's fixed rank allocation.
  • Theoretically, GoLore's stochastic convergence guarantees focus on fully random projections with Momentum SGD, while Alice's analysis does not cover Muon. In contrast, DRAM establishes stochastic convergence guarantees for general rank-allocation schedules under Muon's operator/nuclear-norm geometry, addressing nonlinear matrix-sign update, coupled principal-random subspaces, and stage-varying coverage factors.

Second, regarding theoretical justification, our theory establishes convergence rather than the optimality of Linear. From a worst-case perspective, fully random projections provide the strongest guarantees under a fixed rank budget. However, such bounds can be conservative, as dominant singular directions often capture more gradient information in practice. Our contribution is to enable memory-efficient Muon optimization while preserving competitive practical performance and convergence guarantees.


Q1. How many seeds were used for Tables 2 and 4?

Response. Tables 2 and 4 report single-run results for the selected configurations, while Figs. 1 and 3 and Table 5 include three-seed analyses.


Q2. Which validation split and metric were used to select the learning rates in Table 9? Were the test benchmarks used for hyperparameter selection?

Response. We appreciate this question. For each task–backbone pair, we conducted learning-rate grid searches using the same search space for all methods, as detailed in Appendix A.4 (Tables 8 and 9) and the learning rates were selected based on the best observed test performance.

Similar test-performance-based hyperparameter selection protocols have been adopted in prior studies (Kairouz et al., 2021; Dangel et al., 2023; Li & Lyu, 2023; Coppola et al., 2024; Gu et al., 2024; Wu et al., 2024). Furthermore, Fregonara et al. (2026) empirically show that although test-set tuning introduces optimistic bias, relative model rankings often remain stable.

While we acknowledge that this protocol may introduce selection bias, it enables a consistent comparison of the performance attained by each optimizer.


Q3. Is momentum handled consistently across all schedules in Figs. 1 and 3 and Table 5, including Principal-only and Random-only?

Response. Yes. All rank-allocation schedules in Figs. 1 and 3 and Table 5 follow the same momentum-resetting rule in Eq. (5), including Principal-only (r,q)=(k,0)(r,q)=(k,0) and Random-only (r,q)=(0,k)(r,q)=(0,k).


Kairouz et al. Practical and private (deep) learning without sampling or shuffling. ICML, 2021.

Dangel et al. ViViT: Curvature access through the generalized Gauss–Newton's low-rank structure. TMLR, 2023.

Li and Lyu. Convergence analysis of sequential federated learning on heterogeneous data. NeurIPS, 2023.

Coppola et al. Tuning parameters of deep neural network training algorithms pays off: A computational study. TOP, 2024.

Gu et al. A quadratic synchronization rule for distributed deep learning. ICLR, 2024.

Wu et al. CR-SAM: Curvature regularized sharpness-aware minimization. AAAI, 2024.

Fregonara et al. On hyperparameter tuning on the test set. arXiv:2610.05902, 2026.

AuthorsOpenReview ↗

To Reviewer xAGv (2/2)

Q4. Since Switch-25% outperforms Linear in Table 5, what evidence supports continuous linear reallocation over a GoLore-style hard switch, apart from requiring one fewer hyperparameter?

Response. We acknowledge that Switch-25% outperforms Linear in Table 5. However, hard-switch performance is sensitive to the switching point: Switch-50% and Switch-75% both underperform Linear, while switching at 0% reduces to Random-only, which performs substantially worse. This suggests that early switching is not necessarily beneficial. Therefore, we adopt Linear as the default schedule, as it consistently improves over Principal-only without requiring an additional switching-point hyperparameter.


Q5. How does the schedule-dependent constant in Theorem 1 compare across Linear, Fixed, and Random-only? Does the theory provide any justification for the proposed schedule?

Response. For a fixed total rank, the schedule factors satisfy ΓδRandom<ΓδFixed<ΓδLinear\Gamma_\delta^{\mathrm{Random}} < \Gamma_\delta^{\mathrm{Fixed}} < \Gamma_\delta^{\mathrm{Linear}}, reflecting the stronger worst-case coverage of random directions. As discussed in W2, our theory guarantees convergence rather than the optimality of Linear.


Q6. Could you report wall-clock time or throughput, including the cost of SVD and basis updates? A comparison with SUMO or MoFaSGD under a matched memory budget would also help establish the practical benefits.

Response. We thank the reviewer for this suggestion. We test Full-Muon, DRAM, SUMO, and MoFaSGD on Qwen3-8B/GSM8K under the same workload (batch size 1, sequence length 1024). We report the mean ± sample standard deviation over three repeated runs (10 warmup and 100 measured steps):

MethodOptimizer state (GiB) ↓Peak allocated memory (GiB) ↓Throughput (tokens/s) ↑Total time (s/step) ↓SVD time (s/step)Non-SVD time (s/step)
Full-Muon17.575 ± 0.00052.802 ± 0.000218.765 ± 0.7674.681 ± 0.01604.681 ± 0.016
DRAM7.239 ± 0.00042.466 ± 0.000223.170 ± 2.3444.589 ± 0.0483.781 ± 0.0490.808 ± 0.005
SUMO7.239 ± 0.00042.466 ± 0.00075.997 ± 4.86113.512 ± 0.89312.395 ± 0.9061.117 ± 0.013
MoFaSGD7.240 ± 0.00042.466 ± 0.00041.545 ± 0.19324.648 ± 0.11517.818 ± 0.2116.830 ± 0.102

Each optimizer-state total includes the same 4.638 GiB of AdamW states for parameters not updated by the matrix optimizer.

Under nearly identical optimizer-state and peak allocated memory, DRAM achieves 2.94× SUMO's throughput and 5.37× MoFaSGD's throughput. The difference comes primarily from recurring decomposition costs: DRAM amortizes its approximately 189-second refresh over 50 steps, whereas SUMO performs a compact SVD every step, and MoFaSGD performs two QR factorizations and a SVD every step.

Thank you for the clarifications and additional efficiency measurements. The comparisons with SUMO and MoFaSGD help establish DRAM’s computational advantages in the measured setting. I also appreciate the clarification that momentum handling is consistent across the schedule ablations and that the theory establishes convergence rather than the superiority of the Linear schedule.

However, the clarification that learning rates were selected based on the best observed test performance raises an additional concern about the empirical evidence. Using the same search space across methods provides consistency in the tuning procedure, but it does not remove test-set selection bias or establish that relative rankings are unaffected. This matters particularly here because the reported quality differences are small and the main pretraining and fine-tuning comparisons are based on single runs.

The additional throughput measurements strengthen the efficiency case, but they do not resolve the uncertainty about model-quality improvements or isolate the contribution of rank reallocation from differences in momentum handling across methods.

With the current evidence, I cannot support a rating above 2 (weak rejection), and I am therefore maintaining my score. This reflects the limitations of the empirical support rather than a concern about the main convergence arguments.

To strengthen the paper, I encourage the authors to explicitly disclose the test-based selection procedure and qualify the corresponding performance claims. A representative comparison using validation-based hyperparameter selection, an untouched test set, and multiple seeds would be particularly informative. I recognize that this may not be feasible during the rebuttal period; in that case, clearly acknowledging these limitations would still improve the paper.

Reviewer_9EyYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DRAM, a memory-efficient Muon optimizer that combines principal and random subspaces with dynamic rank allocation. The method progressively shifts the rank budget from principal to random directions during training, aiming to balance gradient information preservation and subspace coverage. The authors establish stochastic and deterministic convergence guarantees and evaluate the method on LLM pre-training and fine-tuning tasks, demonstrating competitive performance with reduced memory consumption.

Critical strengths

The paper addresses an important problem in memory-efficient LLM training. The proposed method is well-motivated, and I appreciate its theoretical contributions, particularly the convergence guarantees for general rank-allocation schedules. The experiments also cover both pre-training and fine-tuning settings, which is good.

Critical weaknesses
  1. Limited benefits of dynamic rank allocation.

In Table 2, DRAM shows only marginal improvements over DRAM-F, without variance estimates to establish statistical significance. Moreover, Table 5 shows that the simple Switch-25% schedule outperforms the proposed linear schedule, raising questions about the necessity and effectiveness of gradual rank reallocation.

Questions for authors
  1. Why are 4-bit-Muon-GRASP and MuonQ excluded from the experiments? How does DRAM compare with these methods in terms of peak memory, training throughput, and model quality under matched memory budgets? If reproducing the training experiments is challenging, the authors should at least provide comparisons in terms of computational efficiency and memory usage.

  2. DRAM resets its momentum at each subspace refresh, while GaLore-Muon retains the previous momentum and GoLore-Muon transports it to the new subspace. Could the authors provide an ablation with consistent momentum handling to clarify how much of DRAM's improvement comes from dynamic rank allocation versus momentum resetting?

2 comments

We thank the reviewer for the positive and constructive evaluation. We address your concerns below.


W1. In Table 2, DRAM shows only marginal improvements over DRAM-F, without variance estimates to establish statistical significance. Moreover, Table 5 shows that Switch-25% schedule outperforms the proposed linear schedule, raising questions about the necessity and effectiveness of gradual rank reallocation.

Response. We agree that multi-seed evaluations would provide stronger empirical evidence. Unfortunately, due to computational constraints and the limited rebuttal period, we are unable to complete additional experiments at this stage.

Regarding the rank-allocation schedule, we acknowledge that Switch-25% outperforms Linear in Table 5. However, hard-switch performance is sensitive to the switching point: Switch-50% and Switch-75% both underperform Linear, while switching at 0% reduces to Random-only, which performs substantially worse. This suggests that early switching is not necessarily beneficial. Therefore, we adopt Linear as the default schedule, as it consistently improves over Principal-only without requiring an additional switching-point hyperparameter.


Q1. How does DRAM compare with 4-bit-Muon-GRASP and MuonQ in terms of peak memory, training throughput, and model quality under matched memory budgets?

Response. We thank the reviewer for this suggestion. To provide a direct comparison, we conducted experiments on Qwen3-8B under the same workload, reporting the mean ± standard deviation over three independent seeds:

MethodOptimizer state (GiB) ↓Peak GPU memory (GiB) ↓Throughput (tokens/s) ↑
Full-Muon17.575 ± 0.00052.802 ± 0.000218.765 ± 0.767
8bit-Muon11.119 ± 0.00046.346 ± 0.000200.721 ± 0.038
4bit-Muon-GRASP8.899 ± 0.00044.285 ± 0.000147.267 ± 0.088
MuonQ8.165 ± 0.00043.454 ± 0.000146.391 ± 0.033
DRAM7.239 ± 0.00042.466 ± 0.000223.170 ± 2.344

The optimizer-state memory includes 4.638 GiB of AdamW optimizer states for parameters not updated by Muon, which is identical across all methods.

Compared with 4-bit-Muon-GRASP, DRAM reduces optimizer-state memory by 18.7% and peak GPU memory by 4.1%, while achieving 51.5% higher throughput. Compared with MuonQ, DRAM achieves 11.3% lower optimizer-state memory, 2.3% lower peak GPU memory, and 52.4% higher throughput. We note that these quantization-based methods incur additional per-step decomposition or reconstruction overhead, which contributes to their lower throughput.


Q2. Could the authors provide an ablation with consistent momentum handling to clarify how much of DRAM's improvement comes from dynamic rank allocation versus momentum resetting?

Response. We appreciate this suggestion. In this version, we adopt momentum resetting as a simple update rule. DRAM can naturally support momentum reuse as in GaLore-Muon or momentum transport as in GoLore-Muon.

We agree that an ablation with consistent momentum handling would help isolate the contribution of dynamic rank allocation. Unfortunately, due to computational constraints and the limited rebuttal period, we are unable to conduct this additional experiment at this stage.

I thank the authors for providing additional experimental comparisons with 4-bit-Muon-GRASP and MuonQ. I appreciate these results and find the efficiency comparison satisfactory. Regarding the other two concerns, I understand that the authors were unable to conduct additional experiments due to time and computational constraints. Overall, I will maintain my original score.

Reviewer_G28MOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript DRAM: Memory-Efficient MUON via Rank Reallocation in Principal-Random Subspaces proposes DRAM, a memory-efficient variant of the MUON optimizer for stochastic optimization. The method maintains momentum in a low-dimensional subspace that combines leading singular directions with random directions, with the split between them adjusted throughout training. The authors also establish convergence bounds for the proposed method and provide evidence that random directions can be important, particularly under noise. The algorithm is tested on several LLM pretraining and finetuning tasks.

Critical strengths
  1. To the best of my knowledge, this paper is the first to combine top SVD directions with random directions for MUON, which I think is more reasonable than using only the top SVD subspace for MUON.

  2. The paper provides convergence guarantee and shows how the random complement helps convergence.

  3. The algorithm is tested on reasonable pretraining and fintuning tasks.

Critical weaknesses
  1. I am still not convinced by the low-rank MUON story. Recent evidence such as Kaon and Polar Express suggests that although exact full orthogonalization may not be necessary, a sufficiently large subset seems still important. Several recent large-scale MUON implementations also uses highly accurate orthogonalization of the full momentum, such as in DeepSeek and Qwen.

  2. The theory also does not seem to justify an advantage of using a low rank. It shows that DRAM can still converge, but there seems to be a worst-case penalty of at least m/k. I think the theory does not suggest the advantage of using low-rank MUON other than memory efficiency.

Questions for authors
  1. It would be great if the authors could provide stronger evidence for realistic LLM training. I am not asking for full training on extremely large models. However, it would be good to have some stronger relevant prior work supporting low-rank MUON at realistic modern scales or stronger theoretical justification. A comparison of the full MUON direction with DeepSeek’s 8+2 scheme with the DRAM direction on a larger model might also be helpful.

  2. The authors could probably add some guidance on how to select the parameters in practice. For example, can we use different ranks for different components, what is a good choice of k given m and n in practice etc. Some justification or experiments along this direction would be helpful.

2 comments

We thank the reviewer for the careful and constructive review. We address your concerns below.


W1. I am still not convinced by the low-rank MUON story. ... Several recent large-scale MUON implementations also uses highly accurate orthogonalization of the full momentum, such as in DeepSeek and Qwen.

Response. We agree that accurate orthogonalization of full-dimensional momentum can be important for optimization performance. However, our work focuses on a different setting: LLM training under memory constraints, and DRAM aims to reduce optimizer-state memory while preserving competitive model performance. Thus, we do not assume that Muon's momentum is inherently low-rank.


W2. The theory does not seem to justify an advantage of using a low rank. It shows that DRAM can still converge, but there seems to be a worst-case penalty of at least m/km/k. I think the theory does not suggest the advantage of using low-rank MUON other than memory efficiency.

Response. We would like to clarify that our theory aims to establish that memory-efficient subspace Muon can retain convergence guarantees, rather than demonstrate superiority over Full-Muon.

To understand the rank-dependent factor m/km/k, consider the idealized noise-free setting. Reducing the rank inevitably limits the gradient information available for each update, so a deterioration in the worst-case convergence bound is expected. Specifically, the leading kk singular directions capture at least a k/mk/m fraction of the full gradient's nuclear norm. This bound is tight when all singular values are equal, explaining the m/km/k factor in our analysis. However, practical gradients often have nonuniform singular-value distributions, where leading directions capture substantially more information than this worst-case guarantee. By explicitly retaining these principal directions, DRAM can exploit such structure to achieve better practical performance than the worst-case bound might suggest.


Q1. It would be good to have some stronger relevant prior work supporting low-rank MUON at realistic modern scales or stronger theoretical justification. A comparison of the full MUON direction with DeepSeek's 8+2 scheme with the DRAM direction on a larger model might also be helpful.

Response. We appreciate this suggestion. DeepSeek's 8+2 scheme improves the accuracy of Newton–Schulz orthogonalization, whereas DRAM reduces the dimensionality of momentum accumulation and orthogonalization. These two approaches address different aspects of Muon optimization and can be used together.

We agree that a comparison of update directions on larger models would provide valuable empirical evidence. Unfortunately, due to computational constraints and the limited rebuttal period, we are unable to conduct this additional experiment at this stage.


Q2. The authors could probably add some guidance on how to select the parameters in practice.

Response. We appreciate this insightful suggestion. Indeed, DRAM constructs subspaces independently for each matrix parameter, naturally allowing different layers to use different rank budgets.

We agree that different components may benefit from different ranks, as their gradient spectra and optimization requirements can vary. Investigating layer-specific rank allocation is an interesting direction for future research.

Thank you for the clarifications and for partially addressing my comments. However:

  • Most of the responses did not directly address my comments.
  • Low-rank momentum could be applied to other optimizers to save memory. My question is why Muon, whose idea is to bring up the whole spectrum. Combining DRAM with expensive NS steps does not change its low-rank nature.
  • As the authors mentioned, DRAM could have better convergence than the bound suggests. However, Muon in practice can also converge much faster than its bound suggests. The harm of low rank might not be reflected solely in the bound.

Without further evidence, I maintain my score of 2.

Reviewer_QbJQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper talks about an important problem with Muon or any optimizer: optimizer state takes space and memory, reducing the overall memory needed for other operations like parameters, gradients, activations, etc.

The paper introduces a new algorithm for A memory-efficient version of Muon. The authors introduce a new way to create a projection basis and create a compact state matrix, which allows the optimizer state to take less memory. The authors also present convergence analysis in Section 3 of the paper. The experiments consist of pretraining at small scale and fine-tuning 8B or similar models. A substantial number of experiments and comparisons to other optimizers have been made.

While I appreciate the paper, I do think that the gains over other methods are not substantiated by the experiments, and the memory gains are not strong, especially when you compare to simple methods like 8 bit quantization.

Critical strengths
  • The paper is overall well written, although I feel the abstract is a bit vague.
  • I appreciate the convergence analysis in Section 3.3.
  • I'm quite glad the authors covered a lot of different kinds of memory-efficient Muon techniques, including low-rank stuff as well as quantization. I do think some papers were missed, but overall, it's still pretty good.
Critical weaknesses
  • It's not obvious to me why I would use this method if the memory gains are only 20%. Why would I not quantize the state of my optimizer state to 8 bits, which gives me a 75% reduction? No experiments in the paper actually cover this at all. I understand 4-bit quantization is more challenging, but 8-bit is pretty lossless. There are several papers on this. This, to me, is the biggest drawback of this paper. The memory gains it claims are actually quite small.
  • Although not a critical weakness, I do feel Algo 1 would be much better written if you contrast it with how regular Muon is done and add the DRAM changes as a delta.
  • It's a bit disappointing that the scale of the pre-training models is quite small. I understand experiments are expensive, but it would have been much better to train on larger models than to do many such small models. Also, the trend of the results is not very clear. Why would we expect DRAM to do so well? Also, are there any error bars? Why is DRAM bolded when there are no error bars? This is a major gap in the paper, in my opinion, and it's not clear to me if the gains are actually real and, if yes, what they are attributable to. Also, honestly, in Table 2, the gains for the larger models are nonexistent or negligible.
  • Even at finetuning, the gains are not very clear to me. And there are no error bars.
Questions for authors

Please read the weakness section. I think the experiments are not convincing enough, and there are no error bars.

1 comment

We thank the reviewer for the careful and constructive review. We address your concerns below.


Q1. It's not obvious to me why I would use this method if the memory gains are only 20%. Why would I not quantize the state of my optimizer state to 8 bits, which gives me a 75% reduction?

Response. We thank the reviewer for raising this concern. We would like to clarify that the reported ~20% reduction refers to overall peak GPU memory, which includes not only optimizer states but also model parameters, gradients, and activations. Moreover, our implementation already stores Muon momentum states in BF16. Therefore, the ideal reduction from 8-bit quantization of these states is ~50%, rather than ~75%.

To provide a more direct comparison, we conducted experiments on Qwen3-8B under the same workload, reporting the mean ± standard deviation over three seeds:

MethodOptimizer state (GiB) ↓Peak GPU memory (GiB) ↓Throughput (tokens/s) ↑
Full-Muon17.575 ± 0.00052.802 ± 0.000218.765 ± 0.767
8bit-Muon11.119 ± 0.00046.346 ± 0.000200.721 ± 0.038
4bit-Muon-GRASP8.899 ± 0.00044.285 ± 0.000147.267 ± 0.088
MuonQ8.165 ± 0.00043.454 ± 0.000146.391 ± 0.033
DRAM7.239 ± 0.00042.466 ± 0.000223.170 ± 2.344

The optimizer-state memory includes 4.638 GiB of AdamW states for parameters not updated by Muon, which is identical across all methods. Compared with 8-bit Muon, DRAM achieves 34.9% lower optimizer-state memory and 8.4% lower peak GPU memory, while delivering 11.2% higher throughput.

Finally, we emphasize that low-rank compression and quantization are complementary approaches. In principle, both can be combined for further memory savings.


Q2. I do feel Algo 1 would be much better written if you contrast it with how regular Muon is done and add the DRAM changes as a delta.

Response. We appreciate this helpful suggestion and will revise Algorithm 1 to make the modifications relative to standard Muon more explicit.


Q3. Experimental trends, statistical variability, and performance gains

Response. We agree that larger-scale pre-training experiments and multi-seed evaluations would provide stronger empirical evidence. Unfortunately, due to computational constraints and the limited rebuttal period, we are unable to complete these experiments during this phase.

Regarding the expected performance of DRAM, we note that our objective is to achieve memory-efficient optimization while preserving competitive model quality. Its expected advantage stems from balancing principal and random directions across training stages: principal directions capture dominant gradient information for early optimization, while random directions provide complementary coverage to mitigate the adverse effects of anisotropic gradient noise. By progressively reallocating rank, DRAM leverages their respective strengths throughout training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.