- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper reassesses the practical value of structural optimizer-state compression relative to direct state quantization. The authors independently vary compute and state precision and compare optimizers in terms of validation loss, peak memory, and training time. The main finding is that Muon with FP8 states often achieves a better quality–memory trade-off in the tested settings, leaving limited advantages for structural methods over this strong baseline.
Critical strengths
- By distinguishing optimizer-state size from overall peak memory, the paper shows why nominal compression ratios can overstate actual training-memory savings and provides a more useful basis for comparison.
- The authors include strong optimizers with quantized states and also evaluate state quantization for structural methods, making the comparison more relevant to practical optimizer selection.
- The evaluation covers a broad range of relevant factors, including compute and state precision, microbatch size, compression strength, training budget, matched-time performance, and MoE models. This helps clarify the conditions under which different methods are useful.
Critical weaknesses
- The scope of the quality evaluation limits the broader conclusions. Dense-model quality comparisons stop at 500M parameters and primarily use FineWeb-Edu. The MoE results provide useful additional evidence, but timing and memory measurements on larger models do not establish that the quality ranking persists at those scales.
- Close comparisons lack sufficient uncertainty estimates. The seed experiments in Tables 11–12 cover only AdamW and Muon at 257M and do not establish the variability of structural methods. In Table 15, Muon and SlimAdam achieve losses of 2.824 and 2.827 after extended training. Repeated runs are needed to determine whether this small difference is consistent.
- Hyperparameter selection may affect the central comparison. In Table 24, AdamW, Muon, and SOAP use weight decay of , whereas several structural methods use 0.1. Different settings are not inherently unfair, but most hyperparameters were transferred from 257M without systematic retuning at 500M. The authors should establish that the main ranking remains robust under comparable tuning budgets.
Questions for authors
Can the closest structural competitors be evaluated across multiple seeds and at a larger dense-model scale, including comparisons under the same memory limit?
3 comments
I sincerely thank the authors for their detailed response and additional experiments, which have substantially alleviated my earlier concerns. The new 1B experiments provide further support for the paper’s conclusions, although this scale remains considerably smaller than that of modern large language models. Whether these conclusions generalize to larger models therefore remains to be established. I find it difficult to assess whether scaling would change the main comparative results, so I retain this concern and would appreciate the Area Chair’s overall assessment on this point.
Dear Reviewer yVAT,
Thank you for the follow-up. We are glad that the additional experiments substantially alleviated your earlier concerns. Below, we would like to present our evidence regarding the scaling of our results.
1. Within our range, the gap of the closest competitor does not narrow. With FP8 states, SlimAdam's gap to Muon is from 257M to 500M to 1B. Hoffmann et al. (NeurIPS 2022) emphasize that the model sizes used to estimate scaling trends matter. Their analysis, based mostly on models above 500M and extending to 16B parameters, yields different compute-optimal scaling predictions from Kaplan et al. (2020), whose experiments included many models below 100M. Our experiments extend to 1B parameters, providing evidence beyond this very small-model regime.
2. Large-scale training recipes are chosen with experiments at this scale (~1B).
- Falcon-H1 informs its architecture with ablations and grid searches on 300M–1.5B proxy models, and transfers hyperparameters tuned on a base model of width 1280 to its 34B model (width 5120).
- DeepSeekMoE validates its architecture on MoE models with about 2B total and 0.3B activated parameters before scaling it to 16B and 145B; DeepSeek-V3 (671B) employs the DeepSeekMoE architecture.
- OLMo 2 developed its recipe on a 1B model and found the findings "to generalize well to the 7B, 13B and 32B scales".
- Attention Residuals validates its design in scaling runs with 194M–528M activated parameters, then trains a 48B-parameter model (3B activated) on 1.4T tokens, where it "matches or outperforms the baseline on all benchmarks".
- DCLM (NeurIPS 2024) finds that data-curation results at its 1B scale (1.4B parameters) correlate with those at 7B (Pearson ).
3. Systematic optimizer comparisons are useful at this scale. We agree that experiments on larger models would strengthen our conclusions. However, repeating the full comparison at 7B exceeds our available compute budget. Other detailed optimizer benchmarks also operate at comparable scales: Semenov et al. (NeurIPS 2026) study models up to 720M parameters, while Wen et al. (ICLR 2026) and Zhao et al. (ICLR 2025) study models up to 1.2B. Such studies are widely cited, indicating their value to the community in comparing methods under controlled conditions and selecting promising approaches for larger-scale validation.
4. Compute. Each of our 1B runs took 152–189 H100-hours. A 7B run at needs about the FLOPs, so repeating our 36-configuration 500M comparison at 7B would require roughly 150,000–300,000 H100-hours.
We hope this helps the overall assessment.
Best regards,
The authors
We thank the reviewer for the careful reading and concrete suggestions. We are glad that the reviewer found that separating optimizer-state size from peak memory gives a more useful basis for comparison. We are also glad they valued that we quantize both the strong optimizers and the structural methods, across compute and state precision, microbatch size, budget, matched time and MoE. We address the concerns below.
W1: Scope of the quality evaluation.
1B dense model, W16A16G32, 1×C: final validation loss.
W2: Uncertainty for close comparisons.
We now report run-to-run variance at 500M for the closest structural competitors:
Run-to-run spread is at most 0.003. The Muon-SlimAdam gap (0.033-0.035) is more than ten times that, the projection methods trail by at least 0.09, and the ranking is identical in both data orders.
On Table 16, we agree that 2.824 vs 2.827 is within noise, and we will say that SlimAdam matches FP8-state Muon at 257M 8×C. At the paper's main budgets the gap is consistent: SlimAdam trails Muon by 0.032-0.035 at 500M and 0.034 at 1B.
W3: Comparable tuning budgets.
Weight decay is now tuned for every structural method at 500M (General Response, Table 1). Several improved, which the paper reflects, but the ranking is the same. It holds whether every method uses the baselines' 10−4 (gaps 0.035-0.161) or 0.1 (0.012-0.119), and 10−4 is also SlimAdam's best value.
Learning rates were tuned per optimizer at 257M, and at 500M Muon's transferred rate is the best of a six-point sweep. This supports learning-rate transfer over our modest model-size range, from 257M to 1B.
Q1: Seeds, larger scale, and comparison under the same memory limit for the closest competitors.
Seeds: see W2. Larger scale: General Response, Table 2, which includes SlimAdam at 1B with FP32 and FP8 states.
Under the same memory limit, the comparison does not change:
We hope the additional experiments and clarifications resolve your concerns. We would appreciate your reconsideration of the score based on the new results. We would be glad to clarify anything further.