- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes DRAM, a memory-efficient variant of Muon that stores and orthogonalizes momentum in a low-dimensional subspace. This subspace combines principal gradient directions with random directions from their orthogonal complement. The rank allocation gradually shifts from principal to random directions during training, motivated by the observed decline in gradient energy captured by the principal subspace (Figs. 1 and 3).
The authors establish convergence rates for the average gradient nuclear norm under smooth non-convex objectives and non-decreasing subspace coverage: in stochastic settings and in deterministic settings. The analysis assumes exact matrix-sign computation and accounts for data-dependent subspaces.
Experiments include LLaMA pre-training (60M–350M parameters), fine-tuning of 8–9B models on IFEval, GSM8K, and MBPP, and a three-seed schedule ablation. DRAM achieves the highest average accuracy among the evaluated low-memory methods while reducing peak memory usage by approximately 18–21% relative to Full-Muon.
Critical strengths
-
Well-motivated design and careful theoretical analysis. DRAM maintains optimizer states and performs orthogonalization entirely within a low-rank subspace, without requiring periodic full-rank corrections. The convergence analysis uses the random complement to ensure sufficient coverage rather than assuming accurate principal subspaces. It also accounts for dependencies between data-dependent bases and gradient noise. I did not find any issues with the main theoretical arguments. Overall, this is a technically sound contribution to memory-efficient Muon training.
-
Clear presentation and comprehensive experiments. The paper is well organized, with consistent notation and a useful comparison in Table 1. The experiments cover continued training, three pre-training scales, fine-tuning of three 8–9B models, and schedule ablations across multiple seeds. The authors also report cases where DRAM underperforms Full-Muon, including the 350M setting and LLaMA GSM8K. The motivation for adapting the subspace allocation is clearly explained and potentially useful in practice.
Critical weaknesses
-
The empirical improvements are small, and the evaluation lacks sufficient statistical evidence.
- Pre-training (Table 2): DRAM improves average accuracy over the strongest competing low-memory method by only 0.30, 0.08, and 0.10 points at 60M, 130M, and 350M, respectively, without reporting variance across seeds. Several benchmarks are near chance level, and validation loss or perplexity, standard metrics for pre-training, are missing from the main comparison.
- Fine-tuning (Table 4): Results are based on single runs, with several differences corresponding to roughly one problem or less. The validation split and metric used for learning-rate selection are also unclear.
- Schedule ablation (Table 5): The default Linear schedule improves validation loss over Principal-only by only about 0.001. Notably, the GoLore-style Switch-25% schedule performs better than Linear at both scales.
- Memory: DRAM uses nearly the same peak memory as GaLore-Muon and GoLore-Muon (within 0.04 GiB). Its memory advantage is mainly over Full-Muon rather than comparable low-rank methods.
Overall, while DRAM achieves the highest reported average accuracy among low-memory methods, the results do not yet establish a robust improvement.
-
The novelty relative to GoLore is somewhat limited, and the theoretical and empirical evidence does not fully justify the proposed schedule.
- Novelty: The idea of transitioning from principal to random projections during training builds directly on GoLore, while related approaches to combining principal and random directions already exist (e.g., Alice). The main contribution appears to be their combination with Muon's geometry and compact optimizer states.
- Theory: Theorem 1 provides a convergence guarantee based on subspace coverage, but its bound is actually worse for the default schedule than for fixed or random-only allocations. Thus, the analysis does not explain why the proposed schedule should outperform these alternatives.
- Comparisons: DRAM resets momentum when updating the subspace, whereas GaLore-Muon and GoLore-Muon handle optimizer states differently. This makes it difficult to isolate the benefit of the rank-allocation schedule.
- Practical efficiency: SUMO and MoFaSGD are discussed but not evaluated. Wall-clock time and throughput are also missing, making it difficult to assess the practical efficiency gains.
I would encourage the authors to strengthen the statistical evaluation, better isolate the effect of the proposed schedule, and clarify its practical advantages over existing methods.
Questions for authors
-
Pre-training variance and loss: How many seeds were used for Tables 2 and 4? For at least one representative pre-training setting, could you report validation loss or perplexity, along with the variability in performance differences between DRAM and the baselines?
-
Fine-tuning protocol: Which validation split and metric were used to select the learning rates in Table 9? Were the test benchmarks used for hyperparameter selection?
-
Momentum handling: Is momentum handled consistently across all schedules in Figs. 1 and 3 and Table 5, including Principal-only and Random-only? Could you include an ablation comparing DRAM with and without momentum resets, or apply the same reset strategy to GaLore-Muon and GoLore-Muon, to isolate the effect of rank allocation?
-
Linear vs. hard switch: Since Switch-25% outperforms Linear in Table 5, what evidence supports continuous linear reallocation over a GoLore-style hard switch, apart from requiring one fewer hyperparameter?
-
Theory vs. schedule: How does the schedule-dependent constant in Theorem 1 compare across Linear, Fixed, and Random-only? Does the theory provide any justification for the proposed schedule?
-
Practical efficiency: Could you report wall-clock time or throughput, including the cost of SVD and basis updates? A comparison with SUMO or MoFaSGD under a matched memory budget would also help establish the practical benefits.
3 comments
To Reviewer xAGv (2/2)
Q4. Since Switch-25% outperforms Linear in Table 5, what evidence supports continuous linear reallocation over a GoLore-style hard switch, apart from requiring one fewer hyperparameter?
Response. We acknowledge that Switch-25% outperforms Linear in Table 5. However, hard-switch performance is sensitive to the switching point: Switch-50% and Switch-75% both underperform Linear, while switching at 0% reduces to Random-only, which performs substantially worse. This suggests that early switching is not necessarily beneficial. Therefore, we adopt Linear as the default schedule, as it consistently improves over Principal-only without requiring an additional switching-point hyperparameter.
Q5. How does the schedule-dependent constant in Theorem 1 compare across Linear, Fixed, and Random-only? Does the theory provide any justification for the proposed schedule?
Response. For a fixed total rank, the schedule factors satisfy , reflecting the stronger worst-case coverage of random directions. As discussed in W2, our theory guarantees convergence rather than the optimality of Linear.
Q6. Could you report wall-clock time or throughput, including the cost of SVD and basis updates? A comparison with SUMO or MoFaSGD under a matched memory budget would also help establish the practical benefits.
Response. We thank the reviewer for this suggestion. We test Full-Muon, DRAM, SUMO, and MoFaSGD on Qwen3-8B/GSM8K under the same workload (batch size 1, sequence length 1024). We report the mean ± sample standard deviation over three repeated runs (10 warmup and 100 measured steps):
| Method | Optimizer state (GiB) ↓ | Peak allocated memory (GiB) ↓ | Throughput (tokens/s) ↑ | Total time (s/step) ↓ | SVD time (s/step) | Non-SVD time (s/step) |
|---|---|---|---|---|---|---|
| Full-Muon | 17.575 ± 0.000 | 52.802 ± 0.000 | 218.765 ± 0.767 | 4.681 ± 0.016 | 0 | 4.681 ± 0.016 |
| DRAM | 7.239 ± 0.000 | 42.466 ± 0.000 | 223.170 ± 2.344 | 4.589 ± 0.048 | 3.781 ± 0.049 | 0.808 ± 0.005 |
| SUMO | 7.239 ± 0.000 | 42.466 ± 0.000 | 75.997 ± 4.861 | 13.512 ± 0.893 | 12.395 ± 0.906 | 1.117 ± 0.013 |
| MoFaSGD | 7.240 ± 0.000 | 42.466 ± 0.000 | 41.545 ± 0.193 | 24.648 ± 0.115 | 17.818 ± 0.211 | 6.830 ± 0.102 |
Each optimizer-state total includes the same 4.638 GiB of AdamW states for parameters not updated by the matrix optimizer.
Under nearly identical optimizer-state and peak allocated memory, DRAM achieves 2.94× SUMO's throughput and 5.37× MoFaSGD's throughput. The difference comes primarily from recurring decomposition costs: DRAM amortizes its approximately 189-second refresh over 50 steps, whereas SUMO performs a compact SVD every step, and MoFaSGD performs two QR factorizations and a SVD every step.
Thank you for the clarifications and additional efficiency measurements. The comparisons with SUMO and MoFaSGD help establish DRAM’s computational advantages in the measured setting. I also appreciate the clarification that momentum handling is consistent across the schedule ablations and that the theory establishes convergence rather than the superiority of the Linear schedule.
However, the clarification that learning rates were selected based on the best observed test performance raises an additional concern about the empirical evidence. Using the same search space across methods provides consistency in the tuning procedure, but it does not remove test-set selection bias or establish that relative rankings are unaffected. This matters particularly here because the reported quality differences are small and the main pretraining and fine-tuning comparisons are based on single runs.
The additional throughput measurements strengthen the efficiency case, but they do not resolve the uncertainty about model-quality improvements or isolate the contribution of rank reallocation from differences in momentum handling across methods.
With the current evidence, I cannot support a rating above 2 (weak rejection), and I am therefore maintaining my score. This reflects the limitations of the empirical support rather than a concern about the main convergence arguments.
To strengthen the paper, I encourage the authors to explicitly disclose the test-based selection procedure and qualify the corresponding performance claims. A representative comparison using validation-based hyperparameter selection, an untouched test set, and multiple seeds would be particularly informative. I recognize that this may not be feasible during the rebuttal period; in that case, clearly acknowledging these limitations would still improve the paper.
To Reviewer xAGv (1/2)
We thank the reviewer for the thorough and constructive review. We address your concerns below.
W1. The empirical improvements are small, and the evaluation lacks sufficient statistical evidence.
Response. We agree that multi-seed evaluations would provide stronger empirical evidence. Unfortunately, due to computational constraints and the limited rebuttal period, we are unable to complete these additional experiments at this stage.
Regarding memory efficiency, we would like to clarify that we intentionally match the memory budgets of subspace-based methods to ensure a fair comparison of their optimization performance. DRAM can further reduce memory usage by decreasing the subspace rank, although this may involve a trade-off with optimization performance.
W2. The novelty relative to GoLore is somewhat limited, and the theoretical and empirical evidence does not fully justify the proposed schedule.
Response. We appreciate this concern and would like to clarify our contributions.
First, regarding novelty, we highlight both algorithmic and theoretical differences.
Second, regarding theoretical justification, our theory establishes convergence rather than the optimality of Linear. From a worst-case perspective, fully random projections provide the strongest guarantees under a fixed rank budget. However, such bounds can be conservative, as dominant singular directions often capture more gradient information in practice. Our contribution is to enable memory-efficient Muon optimization while preserving competitive practical performance and convergence guarantees.
Q1. How many seeds were used for Tables 2 and 4?
Response. Tables 2 and 4 report single-run results for the selected configurations, while Figs. 1 and 3 and Table 5 include three-seed analyses.
Q2. Which validation split and metric were used to select the learning rates in Table 9? Were the test benchmarks used for hyperparameter selection?
Response. We appreciate this question. For each task–backbone pair, we conducted learning-rate grid searches using the same search space for all methods, as detailed in Appendix A.4 (Tables 8 and 9) and the learning rates were selected based on the best observed test performance.
Similar test-performance-based hyperparameter selection protocols have been adopted in prior studies (Kairouz et al., 2021; Dangel et al., 2023; Li & Lyu, 2023; Coppola et al., 2024; Gu et al., 2024; Wu et al., 2024). Furthermore, Fregonara et al. (2026) empirically show that although test-set tuning introduces optimistic bias, relative model rankings often remain stable.
While we acknowledge that this protocol may introduce selection bias, it enables a consistent comparison of the performance attained by each optimizer.
Q3. Is momentum handled consistently across all schedules in Figs. 1 and 3 and Table 5, including Principal-only and Random-only?
Response. Yes. All rank-allocation schedules in Figs. 1 and 3 and Table 5 follow the same momentum-resetting rule in Eq. (5), including Principal-only (r,q)=(k,0) and Random-only (r,q)=(0,k).
Kairouz et al. Practical and private (deep) learning without sampling or shuffling. ICML, 2021.
Dangel et al. ViViT: Curvature access through the generalized Gauss–Newton's low-rank structure. TMLR, 2023.
Li and Lyu. Convergence analysis of sequential federated learning on heterogeneous data. NeurIPS, 2023.
Coppola et al. Tuning parameters of deep neural network training algorithms pays off: A computational study. TOP, 2024.
Gu et al. A quadratic synchronization rule for distributed deep learning. ICLR, 2024.
Wu et al. CR-SAM: Curvature regularized sharpness-aware minimization. AAAI, 2024.
Fregonara et al. On hyperparameter tuning on the test set. arXiv:2610.05902, 2026.