- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper expresses the unclipped MARS estimator as a mixture of momentum SGD and momentum variance reduction, showing that shared sampling collapses the mixture’s parameterization. VENUS introduces independent samples and an additional splitting parameter. The authors derive convergence bounds for LMO-based updates and analyze estimator variance, covariance, and optimization progress. Experiments under matched gradient budgets identify a small-batch benefit for exact VENUS, alongside regimes where MARS and cached variants perform better.
Critical strengths
-
The paper provides a useful analysis of sample coupling. The parameter-collapse result clarifies why independently changing mixture weights under shared sampling does not enlarge the estimator family. The variance decomposition then separates independent-gradient averaging from refresh–transport covariance, while the geometry experiments demonstrate why reduced estimator variance need not translate directly into improved optimization progress.
-
The empirical conclusions are supported by a careful comparison protocol. The language-model experiments match gradient budgets, tune methods separately, and evaluate six additional paired confirmation seeds. Table 2 distinguishes a supported improvement at batch size 16, an inconclusive difference at 32, and a reversal at 64. The explicit reporting of stronger cached-MARS results helps establish the boundaries of the contribution.
Critical weaknesses
-
The demonstrated practical advantage remains narrow. The final language-model improvement is established for exact estimators at batch size 16, while cached MARS performs better in the tested small-batch configurations. The main language-model study does not include standard optimizer references such as AdamW or Muon, and matching gradient work does not establish a wall-clock advantage. These limitations constrain the practical significance of the proposed optimizer, although they do not invalidate the estimator analysis.
-
The source of the full-training improvement remains only partially explained. Section 6 reports that independent-refresh averaging dominates the measured variance reduction in Transformer training. The larger-batch MARS controls and local geometry diagnostics are informative, but do not establish whether a comparable benefit could be obtained by reallocating gradient work over complete training runs. The analysis also does not explain the observed batch-size crossover.
Questions for authors
-
At branch batch size 16, how does VENUS compare over a complete language-model run with MARS at batch size 24, matching both per-step gradient work and the number of updates?
-
What are the wall-clock and peak-memory trade-offs relative to exact MARS, cached MARS, and a standard optimizer?
-
Can the available diagnostics identify which aspect of the update geometry or training dynamics explains the batch-size crossover?
Thank you for your review and for the two concrete experiments you suggested. We ran both, a fixed-fresh-token comparison and a tuned Muon baseline with a γ=0 control, each confirmed on six paired seeds. The results are in blue in Appendix J.
Our aim. The paper does not try to beat Muon. It studies what decoupling the samples of MARS does, and shows that at small batch sizes this gives a measurable improvement over exact MARS. The new results sharpen where that improvement holds.
1. Fresh tokens. You are right: per update, VENUS uses 1.5 gradient evaluations per fresh sequence and MARS 2. At 2.70B gradient tokens, VENUS@16 sees 1.80B fresh tokens and MARS@16 sees 1.35B (Table 25).
2. Tuned Muon. We use the reference Muon implementation. Its embedding is trained exactly as in MARS and VENUS. We tuned 20 configurations at B16 and 18 at B32, and the best ones lie inside the grid (Appendix J.3).
Muon matches cached MARS (both intervals include zero) and is lower than both exact estimators (Tables 25–26, Figures 33–35).
3. γ=0: your intuition is confirmed. Setting γ=0 in cached MARS, with its tuned hyperparameters kept unchanged, gives the same final loss: the paired difference is 0.0000 [−0.0010,+0.0010] at B16 and +0.0001 [−0.0010,+0.0012] at B32 (Table 26). So the advantage of cached MARS over exact MARS comes from using one gradient per update, not from the correction term. One nuance: MARS with γ=0 is plain momentum followed by the same LMO. It is therefore close to Muon but not identical, since it has no Nesterov step and a different update scaling.
4. What changes in the paper.
Running now: all methods rerun with AdamW's weight decay, and separately with its embedding betas. We will post the results here.
We hope this addresses your request for more thorough experiments.