VENUS: Decoupling the Samples in MARS Benefits Small-Batch Training

AdamW has long been a standard optimizer for neural network training, with recent alternatives including matrix-aware methods such as Muon and variance-reduced methods such as MARS, which scales the variance-reduction correction term. We show that the unclipped, estimator-level formulation of MARS is exactly a convex combination of momentum SGD (MSGD) and momentum variance reduction (MVR). When both branches use the same minibatch, the three mixing parameters reduce to two effective MARS parameters; thus, varying these parameters independently does not expand the estimator family. Independent sampling, however, does. We introduce VENUS, which evaluates the two branches on independent minibatches and introduces one additional effective degree of freedom. We establish convergence guarantees for updates defined by a linear minimization oracle (LMO), where the additional mixing weight enters through an explicit variance factor minimized by balanced weighting. Experiments on GPT-2-style language model pretraining under matched gradient budgets support the same balanced weighting and reveal a batch-size-dependent pattern: exact VENUS achieves lower final validation loss at batch size 16, the difference at 32 remains inconclusive, and exact MARS achieves lower loss at 64. In the tested cached-gradient configurations at batch sizes 16 and 32, cached MARS achieves lower final validation loss than cached VENUS and both exact estimators. Fixed-state analyses separate independent-gradient averaging from refresh–transport covariance and show how covariance shifts the variance-minimizing split. Diagnostic experiments further distinguish estimator variance from optimization progress: the equal-work comparison favors different estimators under Euclidean and layerwise LMO updates. The observed small-batch benefit of exact sample decoupling motivates further study of training under limited computational resources.