- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper studies a simple cautious momentum rule, where a coordinate only updates when momentum and gradient agree in sign. It analyzes quadratic problems whose Hessian can change over time. The paper shows that the method converges at an accelerated rate when the off-diagonal coupling between coordinates is small, gives an example where slightly larger coupling makes it diverge, and shows that standard momentum methods with fixed schedules cannot achieve the same acceleration in this setting.
A caveat for my review: My expertise in optimization theory is limited. I read the main text and skimmed the appendices, but have not verified the proofs and am likely missing relevant prior work. My comments are mostly about scope, presentation, and what evidence would make the claims more convincing. I defer to other reviewers on technical novelty and correctness.
Critical strengths
(1). The central question is clearly posed and the answer is clear, with matching upper and lower bounds up to a constant of about 2.
(2). The divergence example is concrete and fixed in advance, which makes it convincing.
(3). The paper clearly states its limits: one fixed tuning, gap between the bounds, and that practical cautious optimizers differ from the one analyzed here.
(4). I found it to be clearly written for a theory paper, with some useful intuition in Section 3.
Critical weaknesses
(1). No experiments. A small simulation could help show that the result makes a difference in a realistic setting.
(2). The setting is narrow, with unnormalized rule, exact gradients, quadratics, and a single fixed tuning. The connection to cautious optimizers that are actually used is not clear.
(3). The comparison focuses on momentum methods with pre-set schedules, but it would be interesting to compare to methods that adapt.
Questions for authors
(1). The divergence construction uses a specially designed Hessian sequence. Have you examined whether a single fixed quadratic with coupling of similar order can also cause divergence? You say that this is open in Section 6: have you tried it numerically?
(2). The acceleration claim is relative to momentum methods with pre-set schedules. How does cautious momentum compare to methods that are adaptive, on the same diagonal problems?
Response to Reviewer 2AQK
Thank you for the helpful questions and comments on the scope and presentation. We have added numerical experiments to examine fixed quadratics, illustrate the divergence construction, and compare cautious momentum with an adaptive method.
We have also expanded the explanations to make the arguments easier to follow. Section 4.2 and Appendix A.2 now describe how successive phases reduce the error, how both position and momentum become small, and how this leads to contraction over a block of iterations. Section 5 and Appendix B.1 now explain what each stage of the growing scalar cycle does and how we construct Hessians that produce the required gradients. We hope these explanations, together with the new numerical illustrations, make the connection between the intermediate steps and the main results clearer.
Q1: Can a single fixed quadratic cause divergence?
We do not know yet. We tested randomly generated fixed positive-definite Hessians, used differential evolution to search over Hessians and initial points for growing trajectories, and tested each Hessian from our periodic divergence construction while holding it fixed throughout the run. None of these tests showed continued growth over the simulated horizons. Appendix D.1 now describes the search methods and results.
So we would say these results suggest the fixed-Hessian question is still open.
Q2: How does cautious momentum compare with adaptive methods on diagonal problems?
We added a comparison with gradient-restarted Nesterov acceleration, which decides when to reset momentum using the observed gradients. This method falls outside the predetermined linear class covered by Theorem C.3.
On the diagonal problems, the comparison is mixed. Appendix D.2 tests two families of time-varying curvature: log-uniform, where curvatures vary across a broad logarithmic range, and sparse-spike, where they are usually small with occasional larger spikes. The median gradient-call counts from Table 2 are:
\text{Diagonal curvature family} & \text{Cautious momentum} & \text{Restarted NAG} \\\\ \hline \text{Log-uniform} & 15{,}431.5 & 7{,}945.5 \\\\ \text{Sparse-spike} & 15{,}431.5 & 35{,}457.5 \end{array}$$ Here, the equal numbers for CM in the diagonal-family column are not wrong (we actually checked the code and ran twice). We think it has a simple explanation. For each seed, the coordinate with curvature $\mu$ has the same initial value and follows the same CM trajectory in both families. Our coordinatewise checks show that this coordinate determines the final error-threshold crossing in all 20 seeds of each family. The interior curvatures and early trajectories differ, but the other coordinates contribute negligibly. We can see that restarted NAG is faster on all 20 log-uniform instances, while cautious momentum is faster on all 20 sparse-spike instances. These experiments use $d=32$ and $\kappa=10^5$. We select baseline parameters on separate validation seeds, and all methods use the same Hessian sequence and initial point on each test seed. We count every gradient call, including rejected cautious-momentum updates. The reported count is the first call after which relative error remains at most $10^{-6}$ through the rest of the observed run, with a budget of 600,000 calls. Appendix D.2 also reports comparisons with GD, heavy-ball, and ordinary NAG. We then added weak coupling to the same two families, with $\rho=373/(2000\sqrt\kappa)$, within the convergence regime of Theorem 2.1(i). Here cautious momentum is faster than restarted NAG on 19 of 20 seeds in each family (see Table 3). These experiments show that the advantage can be visible on concrete examples, including against an adaptive baseline. But the diagonal results also show that cautious momentum does not always win. And our theoretical separation remains specific to methods with predetermined coefficients. Regarding your concern about the narrow setting, we study the raw rule to understand how the sign gate interacts with retained momentum and coordinate coupling. We have also added two direct extensions based on suggestions from other reviewers: Corollary A.10 establishes accelerated convergence for a one-parameter family of step-size and momentum choices, with corresponding coupling bounds, and Corollary A.11 covers $C^2$ objectives with a common minimizer and the same uniform Hessian bounds. These extensions still use exact gradients and the raw update. Remark 4.3 explains which estimates need additional work for stochastic gradients and time-varying normalization.