- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors introduce temporal antithesis, which makes consecutive residuals of minibatch gradients anticorrelated while preserving their means and covariances. Since exact antithesis requires computing the full gradient, they propose superbatch approximations to estimate the full-gradient center. They empirically show that temporal antithesis and its superbatch approximations accelerate Adam across various architectures and tasks.
Critical strengths
- The formulation of how residuals across updates relate to one another is interesting. The proposed antithesis is easy to follow.
Critical weaknesses
- The overall writing could be improved. For example, many inline citations should be enclosed in square brackets [].
- One primary concern is the computational bottleneck of the proposed methods. In Table 2, exact antithesis and its superbatch approximations show consistent performance gains, while their computational costs range from 4.5× to 176× that of vanilla AdamW, which is impractical for large-scale experiments. A comparison with vanilla AdamW under the same computational budget would help support the efficiency of the proposed methods.
Questions for authors
- Why does the proposed antithesis accelerate Adam most strongly early in training? Does this mean that the correlation between consecutive residuals decreases as training progresses?
3 comments
Follow-up on Our Responses to Reviewer 8512
Dear Reviewer 8512,
We appreciate your feedback on our submission. In response to your comments, we have presented new experimental results and provided additional clarifications. As the discussion period is coming to an end, we would appreciate knowing whether our responses have addressed your concerns.
Best regards,
The Authors
Thank you for your response. Some of the new experimental results are intriguing, and the clarifications provided in response to my questions partially address my concerns.
After reviewing the authors' responses to the other reviewers, I revisited the paper. Still, my main concern remains the overall clarity and writing of the paper. For example, in the second paragraph of the introduction, the cited works do not seem sufficiently related to each other, which makes it hard to motivate this paper. Furthermore, many paragraphs are overly dense, making the arguments difficult to follow. I also find some of the terminology to be imprecise or inappropriate in context. Since I am not fully convinced that the paper can be improved, I will maintain my current score.
Response to Reviewer 8512: cost at equal budget and early acceleration
We thank the reviewer and answer each point below.
W1. Writing and citations
We will enclose inline citations in brackets (\citep) throughout and revise the dense passages.
W2. Computational cost and comparison under the same budget
Our main purpose in this paper was to demonstrate that anticorrelating minibatch noise accelerates Adam. Exact antithesis (×176) was an oracle, not meant for actual training, and the superbatch approximations (×2.5 to ×32) show that most of the acceleration can be retained without the full gradient. We understand your concern about cost. However, this acceleration is not inherently costly. The noise can be anticorrelated in other ways, and the key difficulty is estimating the centre cheaply.
Full tables are in our replies to Reviewers AsP6 (W2 / Q2) and bkVF (W1–W2 / Q1–Q2). These tables show that the acceleration does not require an expensive construction, and we will add these results to the appendix.
Q1. Why is the acceleration strongest early? Does the correlation decrease?
No. The correlation is fixed by construction for the whole run (ρ=−1 for exact antithesis, ρ=−(K−2)/K for the superbatch constructions). What changes during training is how costly the noise is. Minibatch noise spreads the parameters around their expected path, and this spread raises the loss. This extra loss, the dispersion cost, is measured by the Jensen gap (Sec. 4.2). Antithesis cancels most of the spread without changing the expected path, so it removes most of this cost. From checkpoints of the same 20-epoch schedule, we applied one pair of updates 256 times with independent minibatches and 256 times with antithesis, and compared the resulting validation losses (the three epochs use the same learning rate; 3 seeds; validation loss in units of 10−2):
Antithesis removes a similar share of the dispersion cost at every stage (75–84%), but the cost itself is large early and small later. Between epochs 1 and 10, at the same learning rate, it falls about six-fold, and the loss reduction falls with it. The acceleration is therefore strongest early because the noise is most costly there, not because the correlation weakens. We will add this as a figure.
In summary, at the same cost, anticorrelation accelerates training compared with AdamW and with competitive variance-reduction methods (W2). We proposed a new costless construction to answer your concern about cost, and it shows that the anticorrelation can be achieved at the cost of AdamW. The acceleration is strongest early because the noise is most costly there, at a fixed correlation (Q1). If these answers resolve your concerns, we would be grateful if you would consider raising your score. We are happy to answer any further questions.