acceptodds
Under review as a conference paper at ICLR 2027

VENUS: Decoupling the Samples in MARS Benefits Small-Batch Training

Abstract

AdamW has long been a standard optimizer for neural network training, with recent alternatives including matrix-aware methods such as Muon and variance-reduced methods such as MARS, which scales the variance-reduction correction term. We show that the unclipped, estimator-level formulation of MARS is exactly a convex combination of momentum SGD (MSGD) and momentum variance reduction (MVR). When both branches use the same minibatch, the three mixing parameters reduce to two effective MARS parameters; thus, varying these parameters independently does not expand the estimator family. Independent sampling, however, does. We introduce VENUS, which evaluates the two branches on independent minibatches and introduces one additional effective degree of freedom. We establish convergence guarantees for updates defined by a linear minimization oracle (LMO), where the additional mixing weight enters through an explicit variance factor minimized by balanced weighting. Experiments on GPT-2-style language model pretraining under matched gradient budgets support the same balanced weighting and reveal a batch-size-dependent pattern: exact VENUS achieves lower final validation loss at batch size 16, the difference at 32 remains inconclusive, and exact MARS achieves lower loss at 64. In the tested cached-gradient configurations at batch sizes 16 and 32, cached MARS achieves lower final validation loss than cached VENUS and both exact estimators. Fixed-state analyses separate independent-gradient averaging from refresh–transport covariance and show how covariance shifts the variance-minimizing split. Diagnostic experiments further distinguish estimator variance from optimization progress: the equal-work comparison favors different estimators under Euclidean and layerwise LMO updates. The observed small-batch benefit of exact sample decoupling motivates further study of training under limited computational resources.

Primary area:
optimization
Keywords:
optimization, variance reduction, momentum, stochastic optimization, deep learning optimizers, language model pretraining, small-batch training, non-convex optimization, convergence analysis, MARS, Muon, LMO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6CtzOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper expresses the unclipped MARS estimator as a mixture of momentum SGD and momentum variance reduction, showing that shared sampling collapses the mixture’s parameterization. VENUS introduces independent samples and an additional splitting parameter. The authors derive convergence bounds for LMO-based updates and analyze estimator variance, covariance, and optimization progress. Experiments under matched gradient budgets identify a small-batch benefit for exact VENUS, alongside regimes where MARS and cached variants perform better.

Critical strengths
  1. The paper provides a useful analysis of sample coupling. The parameter-collapse result clarifies why independently changing mixture weights under shared sampling does not enlarge the estimator family. The variance decomposition then separates independent-gradient averaging from refresh–transport covariance, while the geometry experiments demonstrate why reduced estimator variance need not translate directly into improved optimization progress.

  2. The empirical conclusions are supported by a careful comparison protocol. The language-model experiments match gradient budgets, tune methods separately, and evaluate six additional paired confirmation seeds. Table 2 distinguishes a supported improvement at batch size 16, an inconclusive difference at 32, and a reversal at 64. The explicit reporting of stronger cached-MARS results helps establish the boundaries of the contribution.

Critical weaknesses
  1. The demonstrated practical advantage remains narrow. The final language-model improvement is established for exact estimators at batch size 16, while cached MARS performs better in the tested small-batch configurations. The main language-model study does not include standard optimizer references such as AdamW or Muon, and matching gradient work does not establish a wall-clock advantage. These limitations constrain the practical significance of the proposed optimizer, although they do not invalidate the estimator analysis.

  2. The source of the full-training improvement remains only partially explained. Section 6 reports that independent-refresh averaging dominates the measured variance reduction in Transformer training. The larger-batch MARS controls and local geometry diagnostics are informative, but do not establish whether a comparable benefit could be obtained by reallocating gradient work over complete training runs. The analysis also does not explain the observed batch-size crossover.

Questions for authors
  1. At branch batch size 16, how does VENUS compare over a complete language-model run with MARS at batch size 24, matching both per-step gradient work and the number of updates?

  2. What are the wall-clock and peak-memory trade-offs relative to exact MARS, cached MARS, and a standard optimizer?

  3. Can the available diagnostics identify which aspect of the update geometry or training dynamics explains the batch-size crossover?

Reviewer_WWPPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper observes that the MARS momentum estimator can be viewed as a weighted sum of gradient EMA (MSGD) and MVR. They exploit this observation to propose a new momentum estimator, VENUS, that evaluates these two branches on independent minibatches of data. To get the final weight update, LMO (Muon-style orthogonalization) is applied on top of the momentum for LLM training (and layerwise Scion/Gluon in the case of CNN).

Critical strengths
  • The core idea is simple: MARS has two components, by feeding them different minibatches we can get a more expressive optimizer (VENUS)
  • Algorithm 2 provides a simple implementation of VENUS
  • The paper is honest about the results, stating clearly in the abstract that exact MARS outperforms VENUS at larger batch sizes and cached (approximate) MARS consistently outperforms VENUS
Critical weaknesses

Figure 5 compares exact and approximate MARS and VENUS under a fixed "gradient tokens" (FLOP) budget. While exact VENUS slightly outperforms exact MARS at batch size 16, this could potentially be explained by the fact that VENUS requires only 1.5 gradients per fresh token of data, while MARS requires 2 – hence VENUS ends up being trained on more tokens of data. I would argue that under this compute-constrained setting, neither VENUS nor MARS perform well – they are both outperformed by their approximate analogues. So if exact MARS and VENUS are not competitive under a fixed compute budget, one might wonder: how well do they perform under a fixed token budget (without accounting for compute overhead)? I think this might be a useful addition to the results.

More broadly, I think the figures are missing strong baselines like Muon, which only requires one gradient computation per token and has been been widely used for NanoGPT speedrun records as well as large-scale frontier training runs (e.g. MAI-Base-2). MARS at γ=0 is exactly Muon, while Figure 16b shows low sensitivity of both MARS and VENUS to γ near 0. This suggests that setting γ=0 might result in a large computational speedup compared to the exact methods (requiring only 1 gradient per token), while costing little in terms of final validation loss.

For these reasons (cached MARS outperforms VENUS; Muon is missing from comparisons), I recommend a rejection. If the experimental results were made substantially more thorough, I would be open to raising my score.

Questions for authors

Could you expand Figure 5 with tuned baselines like Muon, as well as adding plots where the x-axis shows fresh tokens, not gradient tokens?

3 comments

Thank you for your review and for the two concrete experiments you suggested. We ran both, a fixed-fresh-token comparison and a tuned Muon baseline with a γ=0\gamma = 0 control, each confirmed on six paired seeds. The results are in blue in Appendix J.

Our aim. The paper does not try to beat Muon. It studies what decoupling the samples of MARS does, and shows that at small batch sizes this gives a measurable improvement over exact MARS. The new results sharpen where that improvement holds.

1. Fresh tokens. You are right: per update, VENUS uses 1.5 gradient evaluations per fresh sequence and MARS 2. At 2.70B gradient tokens, VENUS@16 sees 1.80B fresh tokens and MARS@16 sees 1.35B (Table 25).

  • Fixed fresh tokens: we trained VENUS@16 to exactly 1.35B fresh tokens, with its own tuned schedule (12 configurations, interior optimum). MARS@16 is better here on 6/6 seeds (+0.0267+0.0267 [+0.0227,+0.0307][+0.0227, +0.0307]; Table 27, Figure 37). However, matching fresh tokens forces VENUS to stop early: it gets half the updates and 25% less compute than MARS. This control therefore handicaps VENUS on compute, not only on data, and does not show that the original gain came from extra data. At equal compute, the setting that matters in practice, VENUS improves on MARS at B16.
  • Fresh-token plots: Figure 36 replots all methods against fresh tokens. The curves are descriptive: the methods have different horizons, so a mid-run point is not an annealed endpoint.
  • Which budget matters: we keep equal compute as the primary comparison. LLM training today is mainly limited by compute, not data, and every run here uses under 3B of the 10B available tokens.

2. Tuned Muon. We use the reference Muon implementation. Its embedding is trained exactly as in MARS and VENUS. We tuned 20 configurations at B16 and 18 at B32, and the best ones lie inside the grid (Appendix J.3).

6 seedsB16B32
Muon3.5140±0.00203.5140 \pm 0.00203.4108±0.00223.4108 \pm 0.0022
Muon − cached MARS+0.0011+0.0011 [−0.0007,+0.0028][-0.0007, +0.0028]−0.0019-0.0019 [−0.0043,+0.0006][-0.0043, +0.0006]
Muon − exact VENUS−0.071-0.071−0.085-0.085

Muon matches cached MARS (both intervals include zero) and is lower than both exact estimators (Tables 25–26, Figures 33–35).

3. γ=0\gamma = 0: your intuition is confirmed. Setting γ=0\gamma = 0 in cached MARS, with its tuned hyperparameters kept unchanged, gives the same final loss: the paired difference is 0.00000.0000 [−0.0010,+0.0010][-0.0010, +0.0010] at B16 and +0.0001+0.0001 [−0.0010,+0.0012][-0.0010, +0.0012] at B32 (Table 26). So the advantage of cached MARS over exact MARS comes from using one gradient per update, not from the correction term. One nuance: MARS with γ=0\gamma = 0 is plain momentum followed by the same LMO. It is therefore close to Muon but not identical, since it has no Nesterov step and a different update scaling.

4. What changes in the paper.

  • Figures 33–34 are the expanded Figure 5 (Muon and the γ=0\gamma = 0 control added), against gradient tokens and training time; Figure 36 is the fresh-token version. Training time depends on the implementation, so we give it for reference only.
  • At a fixed small batch size and equal compute, exact VENUS still improves on exact MARS: −0.0109-0.0109 [−0.0150,−0.0068][-0.0150, -0.0068] at B16, 6/6 seeds (Table 26, Figure 35). This is the result the paper is about.
  • This advantage is absent in the fixed-fresh control and against MARS at batch size 24 (Appendix J.5), and one-gradient methods such as Muon reach lower loss at this budget. Appendix J.8 states these limits.
  • Our runs use a small token budget, and proper tuning at large batch sizes is beyond our compute.

Running now: all methods rerun with AdamW's weight decay, and separately with its embedding betas. We will post the results here.

We hope this addresses your request for more thorough experiments.

Thank you for the added experiments! Unfortunately I will have to maintain my score for two reasons.

First, your new results show MARS outperforming VENUS under a fixed token budget, while neither of these (exact) methods is optimal under a fixed FLOP budget -- the approximate variants outperform them.

Second, and perhaps more importantly, there is no improvement over the simpler and much more popular Muon optimizer. (I do appreciate however that this is now stated clearly!)

AuthorsOpenReview ↗

Answer to follow-up

Thank you for engaging with the new results. We agree with both points, and the revision now states them in Appendix J.9: the exact estimators do not beat the cached ones or Muon at a fixed compute budget, and at fixed fresh tokens exact MARS is better than VENUS.

The paper does not aim to replace Muon. Beating a benchmark is one goal of optimization research, but not the only one: understanding why and when an estimator helps is also useful, and it is the focus here. The paper studies what decoupling the samples of MARS does, with an exact characterization (Proposition 1), a convergence analysis for LMO updates (Theorem 1), and a variance decomposition (Section 6). Our final reruns (official comment named "General comment" above, and Appendix J.8) also show that, with a tuned embedding optimizer, the LMO methods studied here, including cached VENUS, beat a tuned AdamW at both batch sizes.

We thank you for pushing for these controls; they made the scope of the paper clearer.

Reviewer_rGc2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes VENUS, an extension of MARS that uses independent minibatches for its momentum SGD and momentum variance reduction. It establishes an algebraic characterization of MARS, derives convergence guarantees for linear minimization oracle (LMO) updates, and analyzes the effect of sample independence on estimator variance. The authors provide experiments on CIFAR-10 and 124M parameter language-model pretraining that demonstrate improvements in some settings, while also identifying regimes where MARS performs better.

Critical strengths
  1. Sound mathematical analysis and a clear extension of MARS. the convergence and covariance analyses provide useful insights into the role of sample coupling.

  2. The experiments include extensive tuning, multiple independent confirmation seeds, paired confidence intervals, and negative results. The authors generally distinguish statistically supported findings from inconclusive comparisons. A statistically supported improvement in small-batch language-model setting and cifar10.

  3. The methodology is clearly motivated, and the appendices provide detailed architectures, hyperparameters, experimental protocols, and computational budgets.

Critical weaknesses
  1. Limited practical advantage: VENUS improves over exact MARS for cifar10 and for the language-model at batch size 16 but not at 32 or 64. More importantly, cached MARS achieves the best results in the tested settings at batch sizes 16 and 32. This substantially limits the practical significance of the proposed method.

  2. The connection between the theoretical analysis and the observed optimization gains in not clear.

  3. Restricted empirical generalizability: The language-model experiments cover one 124M-parameter architecture, dataset, and training budget. The theoretical guarantees do not establish a general computational advantage, and the empirical evidence is insufficient to determine whether the observed benefits extend to larger models or different training regimes.

  4. Limited conceptual novelty. Independent sampling in hybrid variance-reduction estimators has been explored previously, (Tran-Dinh et al. 2019, 2022). While the MARS extension, parametrization, and LMO analysis are new, the underlying algorithmic principle is incremental.

Questions for authors

Q1. Can the authors clarify which methodological and theoretical contributions extend beyond previously studied independent-sampling hybrid estimators?

Q2. Given the strong performance of cached MARS, can the authors provide insight to under which practical conditions should VENUS be preferred? (considering both computational cost and optimization performance)

Q3. Can the authors provide additional evidence of generalizability and/or characterize the expected operating regime of VENUS beyond the specific model and training budget evaluated?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.