acceptodds
Under review as a conference paper at ICLR 2027

Anticorrelating Minibatch Noise Accelerates Adam

Abstract

A minibatch gradient can be written as the full-dataset gradient plus a zero-mean sampling residual, which is temporally uncorrelated under independent sampling. This residual is central to stochastic optimization, but its temporal organization across updates remains largely overlooked. We introduce temporal antithesis, which makes consecutive residuals anticorrelated while preserving the minibatch one-step gradient's mean and covariance. We characterize analytically how temporal antithesis alters Adam's dynamics and show empirically that it substantially accelerates Adam per update, especially early in training. Because exact antithesis requires the full-dataset gradient, we propose superbatch constructions that approximate it using several ordinary minibatches. These constructions recover most of the early advantage of exact antithesis. We further observe the same early acceleration across convolutional networks, vision transformers, and a 124M-parameter GPT-2 model, as well as beyond Adam with SGD and Muon. Together, these findings establish temporal noise correlation as a consequential direction for understanding and designing stochastic optimizers.

Primary area:
optimization
Keywords:
Minibatch Noise, Adam, Temporal Coupling, Adaptive Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_8512OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors introduce temporal antithesis, which makes consecutive residuals of minibatch gradients anticorrelated while preserving their means and covariances. Since exact antithesis requires computing the full gradient, they propose superbatch approximations to estimate the full-gradient center. They empirically show that temporal antithesis and its superbatch approximations accelerate Adam across various architectures and tasks.

Critical strengths
  1. The formulation of how residuals across updates relate to one another is interesting. The proposed antithesis is easy to follow.
Critical weaknesses
  1. The overall writing could be improved. For example, many inline citations should be enclosed in square brackets [].
  2. One primary concern is the computational bottleneck of the proposed methods. In Table 2, exact antithesis and its superbatch approximations show consistent performance gains, while their computational costs range from 4.5× to 176× that of vanilla AdamW, which is impractical for large-scale experiments. A comparison with vanilla AdamW under the same computational budget would help support the efficiency of the proposed methods.
Questions for authors
  1. Why does the proposed antithesis accelerate Adam most strongly early in training? Does this mean that the correlation between consecutive residuals decreases as training progresses?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 8512: cost at equal budget and early acceleration

We thank the reviewer and answer each point below.

W1. Writing and citations

We will enclose inline citations in brackets (\citep) throughout and revise the dense passages.

W2. Computational cost and comparison under the same budget

Our main purpose in this paper was to demonstrate that anticorrelating minibatch noise accelerates Adam. Exact antithesis (×176) was an oracle, not meant for actual training, and the superbatch approximations (×2.5 to ×32) show that most of the acceleration can be retained without the full gradient. We understand your concern about cost. However, this acceleration is not inherently costly. The noise can be anticorrelated in other ways, and the key difficulty is estimating the centre cheaply.

  • Antithesis at AdamW's cost. We added a construction with zero extra gradient evaluations, which predicts the class-composition part of the residual in the classifier head and reflects it across each pair. Over 20 epochs (ResNet-18, 5 seeds), it leads AdamW at epochs 2 and 5 in every seed, by +1.84 ± 0.98 and +1.36 ± 0.43 points on CIFAR-10 and by +1.45 ± 0.25 and +1.52 ± 0.48 points on CIFAR-100, and keeps its lead longer than approximate MARS (suggested by Reviewer bkVF), a competitive variance-reduced variant of AdamW of the same cost.
  • Superbatch at the same budget. AdamW given the gradient budget of the superbatch is 1.1 to 2.5 points more accurate, either with a larger batch or with more updates. However, antithesis and a larger batch are not in opposition, and they can be combined. Applied on top of a larger batch, at the same number of updates, antithesis improves accuracy further, by +3.18 ± 0.13 points at batch 512 and +0.55 ± 0.14 at batch 2048 (CIFAR-10, 704 updates), until the batch is large enough to saturate accuracy. Batch size sets how much noise each update carries, while anticorrelation sets how this noise accumulates across updates. Within the same budget, anticorrelation also accounts for +5.40 ± 0.08 (CIFAR-10) and +12.64 ± 0.48 (CIFAR-100) points over the same construction without it (the zero-correlation control).

Full tables are in our replies to Reviewers AsP6 (W2 / Q2) and bkVF (W1–W2 / Q1–Q2). These tables show that the acceleration does not require an expensive construction, and we will add these results to the appendix.

Q1. Why is the acceleration strongest early? Does the correlation decrease?

No. The correlation is fixed by construction for the whole run (ρ=−1\rho=-1 for exact antithesis, ρ=−(K−2)/K\rho=-(K-2)/K for the superbatch constructions). What changes during training is how costly the noise is. Minibatch noise spreads the parameters around their expected path, and this spread raises the loss. This extra loss, the dispersion cost, is measured by the Jensen gap (Sec. 4.2). Antithesis cancels most of the spread without changing the expected path, so it removes most of this cost. From checkpoints of the same 20-epoch schedule, we applied one pair of updates 256 times with independent minibatches and 256 times with antithesis, and compared the resulting validation losses (the three epochs use the same learning rate; 3 seeds; validation loss in units of 10−210^{-2}):

epochdispersion cost, independentdispersion cost, antithesisshare removedloss reduction after one pair
14.01 ± 0.370.65 ± 0.1284%3.69 ± 0.18
51.14 ± 0.110.27 ± 0.0376%1.09 ± 0.21
100.60 ± 0.010.15 ± 0.0175%0.60 ± 0.13

Antithesis removes a similar share of the dispersion cost at every stage (75–84%), but the cost itself is large early and small later. Between epochs 1 and 10, at the same learning rate, it falls about six-fold, and the loss reduction falls with it. The acceleration is therefore strongest early because the noise is most costly there, not because the correlation weakens. We will add this as a figure.

In summary, at the same cost, anticorrelation accelerates training compared with AdamW and with competitive variance-reduction methods (W2). We proposed a new costless construction to answer your concern about cost, and it shows that the anticorrelation can be achieved at the cost of AdamW. The acceleration is strongest early because the noise is most costly there, at a fixed correlation (Q1). If these answers resolve your concerns, we would be grateful if you would consider raising your score. We are happy to answer any further questions.

AuthorsOpenReview ↗

Follow-up on Our Responses to Reviewer 8512

Dear Reviewer 8512,

We appreciate your feedback on our submission. In response to your comments, we have presented new experimental results and provided additional clarifications. As the discussion period is coming to an end, we would appreciate knowing whether our responses have addressed your concerns.

Best regards,

The Authors

Thank you for your response. Some of the new experimental results are intriguing, and the clarifications provided in response to my questions partially address my concerns.

After reviewing the authors' responses to the other reviewers, I revisited the paper. Still, my main concern remains the overall clarity and writing of the paper. For example, in the second paragraph of the introduction, the cited works do not seem sufficiently related to each other, which makes it hard to motivate this paper. Furthermore, many paragraphs are overly dense, making the arguments difficult to follow. I also find some of the terminology to be imprecise or inappropriate in context. Since I am not fully convinced that the paper can be improved, I will maintain my current score.

Reviewer_bkVFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the temporal correlation of minibatch noise in Adam. Writing g_B = g_{tot} + η_B, it introduces temporal antithesis: the second update of each pair re-evaluates the same minibatch and reflects its residual around the full-data gradient, g_{2t+1} = 2 g_{tot}(θ_{2t+1}) - g_B(θ_{2t+1}) (Eq. 6). Each update keeps the one-step mean and covariance of an ordinary minibatch gradient, while the two residuals of a pair are opposite. Under a frozen-state approximation, Prop. 1 shows that the pair leaves E[m] unchanged and scales the conditional covariance of m_{2t+1} by (1-β_1)^2/(1+β_1^2). Prop. 2 shows that the variance of v falls only in coordinates with signal-to-noise ratio r_i > 1. Two superbatch constructions remove the need for g_{tot}: Direct (K+1 gradients per pair) and Dense (2K), both with pair correlation -(K-2)/K, each with a zero-correlation control matched in cost and one-step statistics. On ResNet-18/CIFAR-10, exact antithesis reaches 88.07% after a two-epoch schedule against 75.36% for AdamW (Table 2). The gain is monotone in the pair correlation (Fig. 2), is carried by the first moment (Table 3), and comes with an 81% smaller Jensen gap of the two-step end points (Sec. 4.2). Dense with K = 8 transfers the early gains to SGD, Muon, CIFAR-100, TinyImageNet, ViT-Tiny and GPT-124M (Table 4). All gains are per update: the superbatch constructions cost 2.5x to 32x AdamW's gradient evaluations per update, and exact antithesis 176x.

Critical strengths
  1. A clean, well thought-out method. The knob leaves gradient mean and covariance fixed (along with mini-batch reuse and cost) and varies only the temporal coupling. The rotation sweep in Figure 2 is a great control for the claimed benefits.
  2. Clean, informative diagnostics/ablations. Applying antithesis to only m gives almost the same performance as full antithesis. On the other hand, v alone gives questionable gains. The Jensen gap of two-step end points falls from 4.48 to 0.84 at equal mean-point loss (l. 424-425). Together these tie the per-update gain to reduced dispersion of the parameters rather than to a shift of the mean update. The diagnostics are reusable beyond this paper.
Critical weaknesses
  1. It appears to me, that the claims are stronger than what the evidence can support. The method is presented as a way to accelerate Adam, but reported improvements count updates. Depending on the version of the proposed method tested, sometimes each update needs many gradient evaluations more than Adam. We need some way to quantify compute, either wall-clock time or counting gradient evaluations (I think the latter would suffice). Please see question 1 below for this very important limitation.
  2. The purported gains do not control for batch size (and do not consider much of existing variance reduction methods). I would like to see the baseline where the extra gradient evaluations per update are just used for a larger batch size instead. I wonder if this simple approach ends up yielding lower noise than the proposed method (see questions 2-3 below).
Questions for authors

Questions that could change my assessment:

  1. Compute-matched comparison. Measuring cost as total gradient evaluations in units of one AdamW epoch (the per-update multiplier of Tables 2 and 5 times the number of epochs), and including checkpoints of the 20-epoch AdamW run, every two- and five-epoch antithetic run is beaten by an AdamW result of equal or lower cost, with one exception. For example, AdamW at 5/5 (cost 5) reaches 85.68% against 81.79% for Direct K = 4 at 2/2 (cost 5), and AdamW at 20/20 (cost 20) reaches 92.08% against 88.95% for Dense K = 4 at 5/5 (cost 20). The exception is Direct K = 4 at 5/5 (cost 12.5, 89.01%), for which no complete AdamW schedule of similar cost is reported; the AdamW 15-epoch checkpoint (cost 15) is at 89.42%. The 20-epoch antithetic runs end 0.7 to 1.2 points above AdamW, but at 2.5x to 176x its cost. The transfer study (Table 4) uses Dense K = 8 at 8x cost; on GPT-124M this is about 12B token-gradients against 1.5B. Could you report accuracy against gradient evaluations for AdamW (complete schedules at matched budgets), Direct and Dense? Is there a compute-matched regime where antithesis wins?
  2. Large-batch baseline. Under the local approximation, both superbatch constructions present the pair sum g_{2t} + g_{2t+1} = 2\bar{g}K, whose residual has covariance 4C/K (App. A.5.2 shows Direct is Dense with the contrast z^{(1)}). AdamW with the same number of updates and the same gradient cost, i.e. batch (K+1)b/2 for the Direct budget or Kb for the Dense budget, has pair-sum covariance 4C/(K+1) on the Direct budget and 2C/K on the Dense budget, against 4C/K for either construction. Here the pair-sum covariance is Cov(η{2t} + η_{2t+1}), the noise a pair adds to the trajectory for good, and C is the covariance of one batch-b residual. The factor of two against Dense comes from re-evaluation: Dense spends 2K gradient evaluations per pair on only K independent minibatches, while large-batch AdamW spends them on 2K. Its two-step displacement noise is also lower: for K = 8 and β_1 = 0.9, as a fraction of ordinary AdamW, 2/9 \approx 0.22 (Direct budget) and 1/8 (Dense budget), against 0.38 for either construction. How do Direct and Dense compare with AdamW at batch (K+1)b/2 and Kb respectively, learning rate retuned, at the same number of updates?
  3. Variance-reduced baselines. For exact antithesis, a cost-matched SVRG-style pair exists: g_{2t} = g_{tot}(θ_{2t}) and g_{2t+1} = g_B(θ_{2t+1}) - g_B(θ_{2t}) + g_{tot}(θ_{2t}), at the same M + 2 gradient evaluations per pair. It has the same pair-sum residual as antithesis up to sign but a near-zero residual at each step, so under the local approximation it behaves like full-batch Adam at half its cost. How does exact antithesis compare with it? How do Direct and Dense compare with MARS (Yuan et al., 2025), which needs about 2x AdamW's gradient evaluations (1x in its approximate form)? If antithesis beats the baselines in questions 2 and 3, I would consider the 'distinct from batch size' claim supported and raise my score. Clarifications:
  4. The same configuration (exact antithesis, β_1 = 0.9, two-epoch schedule) gives different numbers across the paper. Table 2 has 88.07 \pm 0.13 against 75.36 (AdamW) and 75.48 (IID reflection); Table 3 has 88.07 \pm 0.50 against 76.73 for ordinary Adam; Fig. 2 reports a +11.28 gain, and Fig. 3 about +10.7 (test) and +11.3 (validation). Are these different run sets, baselines or splits?
  5. How are data augmentation and BatchNorm handled when B is re-evaluated at θ_{2t+1} and when g_{tot} is computed? Are the same augmented views reused, and do the extra forward passes update the BN running statistics? Exact cancellation of η_B depends on this.
  6. Related work. The paper cites no variance-reduction work beyond antithetic sampling, and l. 109-110 states that variance-reduction approaches 'operate within a single update'. SVRG and its loopless variant L-SVRG tie each step to a stored full-gradient snapshot, and the recursive estimators PAGE, SARAH, SPIDER and STORM evaluate the same minibatch at two consecutive parameter states, the device behind Eq. 6. The exact construction is itself a control variate: the pair sum g_B(θ_{2t}) - g_B(θ_{2t+1}) + 2 g_{tot}(θ_{2t+1}) is an SVRG estimator anchored at the adjacent iterate, plus g_{tot}(θ_{2t+1}). MARS (Yuan et al., 2025) brings the STORM correction to AdamW, and Defazio & Bottou (2019) and Yin et al. (2025) document when such corrections help deep networks. GraB and CD-GraB (Lu et al., 2022; Cooper et al., 2023) reorder examples so that nearby residuals cancel without extra gradient evaluations. Please position the method against these and qualify the statement at l. 112-114.
5 comments
AuthorsOpenReview ↗

Response to Reviewer bkVF (1/4): compute-matched comparison and antithesis at AdamW's cost

We thank the reviewer for the careful review, and answer each point below with new experiments. Our reply is split into four comments.

W1–W2 / Q1–Q2. Compute-matched comparison and (in next comment) large-batch baseline

The paper claims that temporal antithesis "substantially accelerates Adam per update, especially early in training" (abstract), and states in Section 5 that the gains are per update rather than per unit of compute. We agree that the superbatch constructions are computationally expensive and that compute matters for practical use. However, the acceleration induced by anticorrelating minibatch noise, the central claim of the paper, is not inherently costly. The noise can be anticorrelated in other ways, and the key difficulty is estimating the centre cheaply. Exact antithesis (×176) was an oracle, and the superbatch constructions were designed to show that the noise can be anticorrelated without the full gradient, although they remain costly (×4.5 per update for Direct K=8K=8). The superbatch constructions therefore do not provide a compute-matched regime where antithesis wins against AdamW. To show that the effect remains possible at lower cost, we present below a construction at AdamW's cost, which is ahead of AdamW early in training on two datasets. This construction captures only part of the noise. Capturing more of it, to recover an acceleration as large as with the superbatch constructions, is the natural next step.

Costless antithesis. The construction at AdamW's cost predicts the class-composition part of the residual in the classifier head from quantities already computed during training, is refreshed at every update, and reflects the prediction across each pair, with zero extra gradient evaluations. We compare it with ordinary AdamW and with approximate MARS (Yuan et al., 2025), which has the same cost (ResNet-18, 20 epochs, every method at the paper's AdamW learning rate, 5 seeds, gain in test accuracy over an AdamW run):

gain over AdamWepoch 2epoch 5epoch 15epoch 20
CIFAR-10, costless antithesis+1.84 ± 0.98+1.36 ± 0.43+0.36 ± 0.28+0.11 ± 0.14
CIFAR-10, approximate MARS+1.87 ± 2.24+0.05 ± 0.61+0.15 ± 0.12+0.02 ± 0.05
CIFAR-100, costless antithesis+1.45 ± 0.25+1.52 ± 0.48+0.17 ± 0.23+0.04 ± 0.14
CIFAR-100, approximate MARS+1.68 ± 0.21+0.48 ± 0.53+0.11 ± 0.24−0.01 ± 0.15

Costless antithesis leads AdamW at epochs 2 and 5, in every seed and on both datasets. Approximate MARS gains about as much at epoch 2, but its gain is gone by epoch 5. Later in training both gains become small. The early acceleration is thus reproduced at no extra cost, though with a smaller gain than the superbatch constructions, since this centre captures only part of the noise. It also relies on class labels, so it is specific to classification. This cheap variant completes the paper by showing that the effect is not tied to an expensive construction, and we will add it to the appendix.

AuthorsOpenReview ↗

Response to Reviewer bkVF (2/4): large-batch baseline

Large-batch baseline. The table below supports the negative answer for the superbatch. As you suggested, AdamW is given the gradient budget of the superbatch, either as a larger batch or as more updates (ResNet-18, 20 passes over the data for every method, learning rate tuned per method on validation, 3 seeds, test accuracy; compute in AdamW epochs of gradient evaluations):

compute (AdamW epochs)CIFAR-10CIFAR-100
AdamW b=128b=128 (7,032 updates)2092.0670.44
AdamW b=512b=512 (1,758 updates)2091.7369.50
Direct K=8K=8, b=128b=128 (1,758 updates)22.590.5967.98
its zero-correlation control22.585.1955.34

AdamW given the same budget is ahead of the superbatch, by 1.1 to 2.5 points for Direct K=8K=8 and by more for Dense K=8K=8 (3.9 points on CIFAR-10, at the paper's AdamW learning rate). This does not make anticorrelation less relevant, for two reasons. First, as shown above, there are constructions that do not require the extra gradients of the superbatch. Second, gradient accumulation and anticorrelation are not in opposition. Applied on top of a larger batch, at the same number of updates, antithesis still improves accuracy until the batch is large enough to saturate it (dense K=8K=8: +8.15 ± 0.23 points at batch 128, +3.18 ± 0.13 at 512 and +0.55 ± 0.14 at 2048; with K=4K=4, no further gain at 8192, −0.27 ± 0.05; CIFAR-10, 704 updates). Within the same budget, the same construction without anticorrelation (the zero-correlation control) is also 5.40 ± 0.08 (CIFAR-10) and 12.64 ± 0.48 (CIFAR-100) points worse. This is what we mean by "distinct from batch size". Batch size sets how much noise each update carries, while anticorrelation sets how this noise accumulates across updates. The second has been largely overlooked, and our results show that it matters for Adam. We will state this distinction precisely in the paper.

AuthorsOpenReview ↗

Response to Reviewer bkVF (3/4): variance-reduced baselines

Q3. Variance-reduced baselines

Antithesis beats SVRG. We ran your SVRG pair on the full 20-epoch schedule of the paper, in the configuration of Table 2 (CIFAR-10, ResNet-18, 7,040 updates, the paper's AdamW learning rate 1.5×10−31.5\times10^{-3}, 3 seeds), and compare it with the exact-antithesis and AdamW runs of Table 2. Exact antithesis and the SVRG pair both use the exact full gradient once per pair, at equal cost.

test acc.test losstrain acc.
exact antithesis (Table 2)93.130.441100.0
SVRG pair92.100.593100.0
AdamW (Table 2)92.080.30198.9

The SVRG pair, like variance reduction in general, removes most of the noise of each update. Antithesis keeps it. Every update has the mean and covariance of an ordinary minibatch gradient, and only the correlation of the noise across updates changes. This difference matters for generalization. Erasing the noise helps early, and the SVRG pair is ahead mid-training (91.66 against 86.35 at epoch 5). It then stalls, and exact antithesis overtakes it. At the end, every antithesis seed (92.80 to 93.31) is above every SVRG seed (91.62 to 92.38), with a lower test loss. Both methods fit the training set completely, so the difference at the end is entirely one of generalization. Keeping the one-step noise while changing only its timing generalizes better than erasing it with the SVRG pair. This is what antithesis is designed for. It keeps the one-step statistics of ordinary training and only reduces how the noise accumulates across updates, which sets it apart from variance-reduction techniques.

Antithesis compares favourably with MARS at equal cost. Exact MARS evaluates each minibatch twice, so we compare it with the superbatch constructions at matched compute (the paper's AdamW learning rate, 3 seeds, end of training). Direct K=8K=8 is above exact MARS by +0.59 points on CIFAR-10 and +0.96 on CIFAR-100, and Direct K=4K=4 by +1.64 and +3.25, in every seed, while Dense is below it. Approximate MARS has the cost of AdamW, so we compare it with costless antithesis (table of W1–W2 / Q1–Q2). Both gain about as much at epoch 2, but the gain of approximate MARS is gone by epoch 5, while costless antithesis still leads AdamW. In both settings, anticorrelating the noise therefore compares favourably with this variance-reduction method at equal cost. We will add these comparisons to the appendix.

Reviewer_AsP6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies temporal noise correlation as a factor in stochastic optimization, focusing on Adam. It introduces temporal antithesis, which makes consecutive minibatch gradient residuals negatively correlated while preserving their one-step mean and covariance under a local approximation. The authors use theoretical analysis and controlled experiments to connect early acceleration to reduced first-moment fluctuations and parameter dispersion. Two superbatch constructions extend the intervention beyond the exact full-gradient setting, and experiments report early gains across image models, language modeling, and other optimizers. The paper presents temporal noise organization as an axis of optimizer design, with benefits measured per update at additional gradient cost.

Critical strengths
  1. Temporal Correlation as a Design Variable. The paper treats the temporal organization of gradient noise as a controllable aspect of stochastic optimization. This offers a useful perspective on optimizer design by examining how correlations across updates affect training dynamics beyond the noise statistics at individual updates.

  2. Mechanistic Investigation. By combining theoretical analysis with controlled interventions, the study connects temporal noise coupling to Adam's internal dynamics in the reference setting. This gives the observed acceleration an explanatory basis and provides a concrete framework for studying how an optimizer responds to structured noise.

Critical weaknesses
  1. Unclear attribution of gains across settings.

Section 4.4 uses Table 4 to support transfer across architectures and optimizers, but these experiments compare dense superbatch training with the corresponding standard optimizer. The matched zero-correlation controls in Table 2 and Appendix B.2 are limited to the CIFAR-10 ResNet-18 reference setting with AdamW. In the additional settings, the comparison changes the gradient construction and computational cost along with temporal correlation.

  1. Unclear practical benefit under a fixed gradient budget.

Section 5 explicitly states that the reported gains are measured per update. For the superbatch constructions presented as practical alternatives to exact antithesis, Section 4.3 does not establish how performance compares with standard AdamW using the same total gradient budget for larger batches or additional updates. The zero-correlation controls isolate the effect of temporal coupling. Determining whether this coupling offers a practical advantage requires a comparison with standard training under the same computational budget. This leaves the practical performance-cost tradeoff of the superbatch methods unresolved.

Questions for authors
  1. For one optimizer extension in Table 4, how do the antithetic and zero-correlation superbatch constructions compare when the gradient budget and number of updates are held fixed? This would test whether temporal coupling accounts for the gain when the optimizer changes.

  2. On the CIFAR-10 reference setup, how do the superbatch methods compare with standard AdamW when the same total gradient budget is allocated to larger batches or additional updates, with an appropriately tuned learning rate and schedule? If the authors can clarify the practical performance-cost tradeoff with convincing evidence, I would consider raising my rating.

5 comments
AuthorsOpenReview ↗

Response to Reviewer AsP6 (1/2): zero-correlation controls across settings

We thank the reviewer for the clear statement of what evidence is missing, and answer both questions with new experiments. Our reply is split into two comments.

Q1 / W1. Attribution of the gains across settings

We can confirm that temporal coupling accounts for the gain when the optimizer or the dataset changes. We added the zero-correlation control to three rows of Table 4, with dense K=8K=8 at every update for the whole 20-epoch schedule, the same minibatches, gradient budget, number of updates and one-step statistics; only the coupling differs (ResNet-18, 3 seeds, test-accuracy gain of antithesis over the control):

epoch 2epoch 5epoch 10epoch 20
SGD (Nesterov), CIFAR-10+10.50 ± 0.61+7.51 ± 0.48+5.82 ± 0.59+1.25 ± 0.16
Muon, CIFAR-10+7.64 ± 0.08+4.08 ± 0.34+2.12 ± 0.12+0.66 ± 0.13
AdamW, CIFAR-100+17.94 ± 2.00+15.46 ± 1.50+7.34 ± 0.66+1.68 ± 0.26

Every difference is positive in every seed. Without anticorrelation, the same construction ends level with the plain optimizer (SGD 92.90 vs 92.80; Muon 93.72 vs 93.89; AdamW on CIFAR-100 70.06 vs 70.41), so the gains come from the anticorrelation, not from the construction. We will add these controls to the appendix.

AuthorsOpenReview ↗

Response to Reviewer AsP6 (2/2): performance-cost tradeoff

W2 / Q2. Performance-cost tradeoff under a fixed gradient budget

Costless antithesis. Our main purpose in this paper was to demonstrate that anticorrelating minibatch noise accelerates Adam. Exact antithesis (×176) was an oracle, not meant for actual training. We then proposed superbatch approximations (×2.5 to ×32) to show that most of the acceleration can be retained without the full gradient. Together, they establish the temporal correlation of the noise as a valuable axis of research for optimizers. We understand your concern about practical cost. However, this acceleration is not inherently costly. The noise can be anticorrelated in other ways, and the key difficulty is estimating the centre cheaply. To address your concern, we present a construction at the cost of AdamW, so that the two are compared at the same budget. It predicts the class-composition part of the residual in the classifier head from quantities already computed during training, is refreshed at every update, and reflects the prediction across each pair, with zero extra gradient evaluations. We compare it with ordinary AdamW and with approximate MARS (Yuan et al., 2025), a recent variance-reduced variant of AdamW at the same cost suggested by Reviewer bkVF (ResNet-18, 20 epochs, every method at the paper's AdamW learning rate, 5 seeds, gain in test accuracy over an AdamW run):

gain over AdamWepoch 2epoch 5epoch 15epoch 20
CIFAR-10, costless antithesis+1.84 ± 0.98+1.36 ± 0.43+0.36 ± 0.28+0.11 ± 0.14
CIFAR-10, approximate MARS+1.87 ± 2.24+0.05 ± 0.61+0.15 ± 0.12+0.02 ± 0.05
CIFAR-100, costless antithesis+1.45 ± 0.25+1.52 ± 0.48+0.17 ± 0.23+0.04 ± 0.14
CIFAR-100, approximate MARS+1.68 ± 0.21+0.48 ± 0.53+0.11 ± 0.24−0.01 ± 0.15

Costless antithesis leads AdamW at epochs 2 and 5, in every seed and on both datasets. Approximate MARS gains about as much at epoch 2, but its gain is gone by epoch 5. Later in training both gains become small. The early acceleration is therefore preserved at no extra cost. Note that this centre relies on class labels, so it is specific to classification, and it captures only part of the noise, so its gain is smaller than with the exact or superbatch centre, which can be seen as an upper bound. It nevertheless shows that a cheap centre is possible. Our aim was to establish the effect and its mechanism, and to open a promising direction for optimizer design. This cheap variant completes the paper by showing that the effect is not tied to an expensive construction, and we will add it to the appendix.

Comparison with gradient accumulation. We showed above that anticorrelation can accelerate training at AdamW's cost. Following your suggestion, we also compared the superbatch construction (Direct K=8K=8), which does cost more, with tuned AdamW given the gradient budget of the superbatch (ResNet-18, learning rate tuned per method on validation, 3 seeds). AdamW with a larger batch, which is equivalent to gradient accumulation, is more accurate than the superbatch construction at the same number of updates, by 1.14 ± 0.22 points on CIFAR-10 and 1.52 ± 0.29 on CIFAR-100. Spending the budget on more updates instead gives AdamW a lead of 1.48 ± 0.23 and 2.46 ± 0.37 points. This does not make anticorrelation less relevant, for two reasons. First, as shown above, there are constructions that do not require the extra gradients of the superbatch, although a cheap and more robust centre that captures all of the noise remains future work. Second, gradient accumulation and anticorrelation are not in opposition. Applied on top of a larger batch, at the same number of updates, antithesis still improves accuracy until the batch is large enough to saturate it (dense K=8K=8: +8.15 ± 0.23 points at batch 128, +3.18 ± 0.13 at 512 and +0.55 ± 0.14 at 2048; CIFAR-10, 704 updates). Within the same budget, the same construction without anticorrelation (the zero-correlation control) is also 5.40 ± 0.08 (CIFAR-10) and 12.64 ± 0.48 (CIFAR-100) points worse. Batch size sets how much noise each update carries, while anticorrelation sets how this noise accumulates across updates. The second has been largely overlooked, and our results show that it matters for Adam.

In summary, the gain comes from the temporal coupling with SGD and Muon as with AdamW (Q1), and it is not tied to an expensive construction. At AdamW's cost, antithesis accelerates early training and keeps its lead longer than approximate MARS, a competitive variance-reduced baseline of the same cost (Q2). At a fixed budget, the superbatch constructions remain behind tuned AdamW, but antithesis adds to the gain of a larger batch rather than competing with it. If these answers resolve your concerns, we would be grateful if you would consider raising your score. We are happy to answer any further questions.

AuthorsOpenReview ↗

Follow-up on Our Responses to Reviewer AsP6

Dear Reviewer AsP6,

We appreciate your feedback on our submission. In response to your comments, we have presented new experimental results and provided additional clarifications. As the discussion period is coming to an end, we would appreciate knowing whether our responses have addressed your concerns.

Best regards,

The Authors

Reviewer_FYj6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces temporal antithesis that anticorrelates consecutive residuals while keeping the minibatch one-step gradient’s mean and covariance. It analytically characterizes how temporal antithesis modifies Adam’s dynamics, and empirical experiments verify the acceleration per update, particularly in the early training stage. This paper also proposes the superbatch construction to approximate the required full-dataset gradient by several ordinary minibatches. Similar early acceleration phenomena are observed across different models, including CNN, ViT, and GPT-2, and for SGD with momentum and Muon.

Critical strengths
  1. This paper introduces temporal antithesis that anticorrelates consecutive residuals while keeping the mean and covariance of the minibatch one-step gradient. It provides an analytical characterization of how temporal antithesis works, using the exact full-dataset gradient as the reflection center, and experiments also verify that it accelerates per update, particularly in the early training stage.

  2. Two superbatch constructions that approximate temporal antithesis without computing the full-dataset gradient are proposed. Moreover, preliminary experiments show that temporal antithesis accelerates early optimization across different models, including CNN, ViT, GPT-2, and extends to optimizers such as AdamW, SGD, and Muon.

Critical weaknesses
  1. The proposed temporal antithesis is extremely computationally expensive. The standard temporal antithesis is approximately 176 ×\times the cost of standard AdamW, and even with the superbatch approximation, it remains computationally costly. Given that other methods such as AdaPNM are comparatively inexpensive, could the authors provide variants with comparable computational cost to standard optimizers, to demonstrate the efficiency of this method and to assess whether the observed acceleration still persists?

  2. Can the analytical characterization in Sec 3.2.2 be extended to the superbatch approximation, to provide a similar set of analyses? Similarly, other optimizers given in Sec 4 this paper, such as SGD and Muon?

Also see questions below.

Questions for authors
  1. How about the training performance of temporal antithesis after the initial stage (20 epochs) over the entire training process? Does the temporal antithesis variant consistently outperform the standard version and other compared variants throughout the training?

  2. Could the authors provide a detailed comparison between the ideas presented in paper Kuhn & Rosenow (2025) and the ideas developed in this paper?

Minor: Some incorrect formats of citation within the text.

3 comments
AuthorsOpenReview ↗

Response to Reviewer FYj6 (1/2): antithesis at AdamW's cost and extended analysis

We thank the reviewer for the constructive review. We answer each point below, with new experiments and an extended analysis. Our reply is split into two comments.

W1. Cheap variant: the effect persists at AdamW's cost

Our main purpose in this paper was to demonstrate that anticorrelating minibatch noise accelerates Adam. Exact antithesis (×176) was an oracle, not meant for actual training, and the superbatch approximation (×4.5) shows that most of the acceleration is retained without the full gradient. This establishes the temporal correlation of the noise as a valuable axis of research for optimizers. We understand your concern about cost, but this acceleration is not inherently costly. To show it, we added a construction at AdamW's cost (×1). It predicts the class-composition part of the residual in the classifier head from quantities already computed during training, and reflects this prediction across each pair, with zero extra gradient evaluations (we will describe it fully in the appendix). We compare this construction with ordinary AdamW and, as suggested by Reviewer bkVF, with approximate MARS (Yuan et al., 2025), a recent variance-reduced variant of AdamW at the same cost (×1), which also acts on minibatch noise across updates (ResNet-18, 20 epochs, 5 seeds, gain in test accuracy over an AdamW run):

gain over AdamWepoch 2epoch 5epoch 15epoch 20
CIFAR-10, costless antithesis+1.84 ± 0.98+1.36 ± 0.43+0.36 ± 0.28+0.11 ± 0.14
CIFAR-10, approximate MARS+1.87 ± 2.24+0.05 ± 0.61+0.15 ± 0.12+0.02 ± 0.05
CIFAR-100, costless antithesis+1.45 ± 0.25+1.52 ± 0.48+0.17 ± 0.23+0.04 ± 0.14
CIFAR-100, approximate MARS+1.68 ± 0.21+0.48 ± 0.53+0.11 ± 0.24−0.01 ± 0.15

Both methods gain about as much at epoch 2, but by epoch 5 the gain of MARS is gone, while costless antithesis still leads AdamW in every seed on both datasets. Later in training both gains become small, but costless antithesis stays slightly ahead of MARS on both datasets. Note that this construction relies on class labels, so it is specific to classification, and it captures only part of the noise, so its gain is smaller than with the exact or superbatch construction. It nevertheless shows that a cheap antithesis is possible. Our aim was to establish the effect and its mechanism, and to open a promising direction for optimizer design. This cheap variant completes the paper by showing that the effect is not tied to an expensive construction.

W2. Extending the analysis of Sec. 3.2.2 to the superbatch constructions, SGD and Muon

Both propositions extend to any pair correlation ρ\rho, which covers the superbatch constructions (ρ=−(K−2)/K\rho=-(K-2)/K).

  • First moment. Cov⁡(m2t+1ρ∣F2t−1)=(1+2β1ρ1+β12)Cov⁡(m2t+1iid∣F2t−1),\operatorname{Cov}(m^{\rho}_{2t+1}\mid\mathcal F_{2t-1})=\left(1+\frac{2\beta_1\rho}{1+\beta_1^2}\right)\operatorname{Cov}(m^{\mathrm{iid}}_{2t+1}\mid\mathcal F_{2t-1}), which reduces to Proposition 1 at ρ=−1\rho=-1. The share of the independent covariance removed, −2β1ρ/(1+β12)-2\beta_1\rho/(1+\beta_1^2), grows linearly with −ρ-\rho. The superbatch (ρ=−(K−2)/K\rho=-(K-2)/K) therefore removes a fraction (K−2)/K(K-2)/K of what exact antithesis removes, and should recover the same fraction of its gain. Experimentally, this prediction holds for every KK (CIFAR-10, ResNet-18, two-epoch schedule, from Table 5):
KKpredicted (K−2)/K(K-2)/KDirectDense
40.5000.5060.497
80.7500.7580.738
160.8750.8670.862
320.9380.9560.958

Consistently, the gain is linear in ρ\rho, as the table above shows, since (K−2)/K=−ρ(K-2)/K=-\rho.

  • Second moment. Proposition 2 extends in the same way: under a Gaussian residual approximation, vv is stabilized in coordinates with ri>∣ρ∣r_i>|\rho| instead of ri>1r_i>1. Since the effect is carried by the first moment (Table 3), this changes little.
  • SGD. Its momentum works like Adam's first moment, so the same result and the same (K−2)/K(K-2)/K prediction apply.
  • Muon. Its momentum also works like Adam's first moment, but Muon then orthogonalizes the update (Newton–Schulz), a nonlinear step that our analysis does not cover. In practice, the effect still holds (Table 4).

We will add the general-ρ\rho derivations of both propositions, the SGD case and this table to the appendix.

AuthorsOpenReview ↗

Response to Reviewer FYj6 (2/2): longer training and comparison with Kühn & Rosenow

Q1. Performance over the entire training process

The acceleration holds on a longer schedule. Below, the same setting trained for 40 epochs (CIFAR-10, ResNet-18, dense K=8K=8, 3 seeds, test accuracy); the gain shrinks as both runs approach the same accuracy, but antithesis stays ahead throughout:

epoch510203040
antithesis86.2289.5090.7092.4193.32
AdamW78.6685.5489.0691.5292.99
zero-correlation control78.3185.2988.9591.2192.86
antithesis − AdamW+7.55 ± 0.77+3.96 ± 0.51+1.64 ± 0.52+0.90 ± 0.42+0.33 ± 0.04
antithesis − control+7.91 ± 1.96+4.21 ± 0.23+1.75 ± 0.14+1.20 ± 0.24+0.46 ± 0.06

The gain is positive in every seed. We will add this 40-epoch run to the appendix.

We also took the opportunity to strengthen the 20-epoch comparison with a stronger baseline, an SVRG-style pair suggested by Reviewer bkVF, based on the variance-reduction technique SVRG (Johnson & Zhang, 2013). It uses the same full gradient as exact antithesis to remove the noise instead of anticorrelating it. Every exact-antithesis run of Table 2 ends above every run of the SVRG pair (CIFAR-10, ResNet-18, the paper's AdamW learning rate, 3 seeds each; details in our reply to Reviewer bkVF, Q3).

test acc.test losstrain acc.
exact antithesis (Table 2)93.130.441100.0
SVRG pair92.100.593100.0
AdamW (Table 2)92.080.30198.9

Exact antithesis therefore ends as the best variant. We will add this comparison to the appendix.

Q2. Comparison with Kühn & Rosenow (2025)

Kühn & Rosenow analyse the weak anticorrelation that sampling without replacement creates naturally, while we intervene on the anticorrelation directly. In their paper, they show that this natural anticorrelation is epoch-scale (−1/M-1/M between neighbouring minibatches, spread over the whole epoch) and study its effect on the stationary fluctuations of linearized momentum-SGD, where it reduces fluctuations along flat directions. We build on their findings (Section 3.1 and Appendix A.3), but we differ in three ways:

  1. Intervention. We strengthen the anticorrelation to ρ=−1\rho=-1 between consecutive updates, while holding the one-step mean and covariance fixed, whereas they only analyse its natural strength, −1/M-1/M.
  2. Optimizer and regime. We study Adam: we derive how the anticorrelation acts on its first and second moments (Propositions 1 and 2) and show that it accelerates training early on, far from convergence, whereas they analyse momentum-SGD near stationarity.
  3. Causal test. We test the mechanism experimentally (correlation sweep, moment-state analysis, Jensen gap) rather than analytically only.

We will make the distinction explicit in the paper.

Minor. We will correct the citation formats throughout.

In summary, we showed that anticorrelation still accelerates training at AdamW's cost, that the analysis extends to the superbatch constructions and SGD with a quantitative prediction confirmed by experiment, and that the acceleration holds over a 40-epoch schedule. If these answers resolve your concerns, we would be grateful if you would consider raising your score. We are happy to answer any further questions.

AuthorsOpenReview ↗

Follow-up on Our Responses to Reviewer FYj6

Dear Reviewer FYj6,

We appreciate your feedback on our submission. In response to your comments, we have presented new experimental results and provided additional clarifications. As the discussion period is coming to an end, we would appreciate knowing whether our responses have addressed your concerns.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of new experiments and revisions

We thank all reviewers for their constructive feedback. Their shared concern is cost. We show that antithesis works at AdamW's cost, compare the superbatch with tuned AdamW and MARS [1] at equal compute, and isolate the role of the temporal coupling with new controls and a variance-reduction baseline. All new experiments will be added to the appendix.

Major

  • Antithesis at AdamW's cost. A construction with zero extra gradient evaluations leads AdamW at epochs 2 and 5 in every seed, on CIFAR-10 and CIFAR-100, and keeps its lead longer than approximate MARS [1], a variance-reduced variant of AdamW of the same cost (FYj6 W1, AsP6 W2, bkVF Q1–Q2, 8512 W2).
  • Zero-correlation controls beyond the reference setting. For SGD, Muon and CIFAR-100, over the full 20-epoch schedule, antithesis is ahead of the same construction without anticorrelation in every seed, so the gain comes from the temporal coupling (AsP6 Q1).
  • Compute-matched comparisons. Tuned AdamW given the gradient budget of the superbatch, as a larger batch or as more updates, and exact and approximate MARS [1] at matched compute. Direct beats exact MARS, and antithesis adds to the gain of a larger batch at the same number of updates (AsP6 Q2, bkVF Q1–Q3, 8512 W2).
  • A variance-reduction baseline. The SVRG-style pair suggested by Reviewer bkVF, based on SVRG [2], uses the same full gradient as exact antithesis to remove the noise of each update instead of anticorrelating it. On the full 20-epoch schedule it is ahead early, but every exact-antithesis run ends above every SVRG run, with both methods fitting the training set (bkVF Q3, FYj6 Q1).
  • A 40-epoch schedule, on which antithesis stays ahead of AdamW and of its control at every checkpoint (FYj6 Q1).
  • The Jensen gap across training, which shows that the acceleration is strongest early because the noise is most costly there, at a fixed anticorrelation (8512 Q1).
  • Propositions 1 and 2 for any pair correlation ρ\rho, covering the superbatch constructions and SGD, with the predicted fraction (K−2)/K(K-2)/K of the gain confirmed for every KK (FYj6 W2).
  • Positioning against variance reduction, which reduces the noise that antithesis deliberately keeps, with a corrected l. 109-114 (bkVF Q6), and an explicit distinction from Kühn & Rosenow (2025) (FYj6 Q2).

Minor

  • Report the cost of each construction alongside its per-update gains (all reviewers).
  • State the evaluation split in every caption (bkVF Q4).
  • Detail the handling of augmentation and BatchNorm in exact antithesis (bkVF Q5).
  • Fix the citation format throughout (FYj6, 8512).

These results support the central claim of the paper. Anticorrelating minibatch noise accelerates Adam through the temporal coupling, without requiring an expensive construction. It is distinct from batch size, to which it adds, and from variance reduction, which it outperforms in our comparisons by keeping the noise. We thank the reviewers for helping us make this case.

[1] H. Yuan, Y. Liu, S. Wu, X. Zhou, Q. Gu. MARS: Unleashing the Power of Variance Reduction for Training Large Models. ICML 2025.

[2] R. Johnson, T. Zhang. Accelerating Stochastic Gradient Descent using Predictive Variance Reduction. NeurIPS 2013.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.