- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper revisits learning-to-optimize from a new angle: instead of meta-training a learned optimizer across many tasks, the optimizer's own parameters are co-trained online, together with the model weights, on the single task being solved. Concretely, the optimizer maintains a simple linear model that predicts the full-batch gradient from the current weights; at every iteration this linear model is nudged to better fit the freshly observed stochastic gradient, and its prediction is then used as the update direction for the model. Because a full linear model needs a matrix-sized optimizer state, the authors also propose two memory-light variants that restrict it to diagonal or rank-one structure. On the theory side, they prove that for strongly convex objectives both the distance to the optimum and the gradient-prediction error decay at the same rate as standard stochastic gradient descent, and, since the prediction error vanishes asymptotically, they position the method as a trainable variance-reduction scheme; classical momentum SGD is recovered as a special case. Experiments on logistic regression, feed-forward networks, ResNet on CIFAR-10 and Llama-7b fine-tuning claim improvements over ADAM and momentum, although, as detailed below, the empirical support for these claims is weak.
Critical strengths
-
Interesting, simple, and well-motivated idea.
-
I re-derived the core recursion of Theorem 3.4: Ĝt − Gt = (1 − δt)(Ĝt−1 − Gt−1 + At−1(wt − wt−1) − (Gt − Gt−1)) + δt(gt − Gt) with δt = αt‖wt‖² + βt; the proof then proceeds by a standard joint induction, and the boundedness results (Props. 3.2–3.3) and the level-set argument for the convex case (Lemma A.3) are carefully written. The mechanism is sound.
-
Diag-TO and RO-TO reduce 400 MB–40 GB optimizer states to KB–MB scale (Table 1), which is what makes the idea usable at all on deep models.
Critical weaknesses
-
Section 4.1 tunes ADAM over γ ∈ [1e-5, 5e-3] but TO over γ ∈ [1e-3, 0.5]; the overlap is only {1e-3, 2e-3, 5e-3}. The only large effects in the paper — ResNet18 ρ = 0.5 and ResNet50 ρ = 0.9 (Figure 2 shows ADAM plateauing near 0.3 while TO reaches ≈0.03) — occur precisely on a batch-64 CIFAR ResNet task where ADAM is never allowed learning rates above 5e-3 while TO may use 0.05–0.5. ADAM at 1e-2 on such tasks would not exhibit that plateau. Without per-experiment final hyperparameters and a symmetric grid, these results cannot be interpreted as a method win. The Momentum and SAGA grids are not reported at all.
-
The metric ρ compares achieved minimum training loss, not epochs or wall-clock to a target loss. There is no time-to-target experiment, and no wall-clock overhead measurement anywhere despite the abstract's "negligible computational overhead" claim; the A-update is an O(d²) outer product per step, so at least a Diag/RO per-step timing table is needed.
-
For a method that positions itself as variance reduction, the strongly-convex suite never compares against SVRG, SAGA (only appears in the λ=0 figures), VRADAM, or AMSGrad; the L-BFGS comparison under mini-batch sampling is a straw man since L-BFGS is a deterministic full-gradient method.
-
Everything except one Llama validation curve is training loss. There is no test accuracy on any classification task and no downstream metric for the LLM fine-tuning; a variance-reduced optimizer can reach lower training loss while generalizing worse, so the current evidence cannot support deployment claims.
Questions for authors
- Please report the selected per-task hyperparameters (γ, schedule, α, β for TO; γ for ADAM/Momentum/SAGA) and re-run ResNet18/50 with ADAM's grid extended to [1e-2, 1e-1]. Does ADAM still plateau near 0.3, and do ρ = 0.5 / 0.9 survive?
- Please quantify convergence speed properly: epochs and wall-clock to reach a fixed target training loss, plus test accuracy for all classification tasks, and downstream metrics (e.g., a GSM8K/MMLU subset or MT-Bench) with seed variance for the Llama experiment, together with the exact specification of the fine-tuned weight subset.
- Please compare against SVRG, SAGA, VRADAM and AMSGrad on the strongly convex suite and at least one non-convex task, since the paper's positioning is variance reduction.
- Please provide proofs (or a counterexample discussion) for Diag-TO and RO-TO convergence, in particular boundedness of the rank-one factors.
- Quantitatively: with matched decaying βt, what are the asymptotic directional-variance constants of TO versus momentum SGD? If they coincide, the variance-reduction claim should be reframed.
- Please add a per-step wall-clock overhead table for Diag-TO/RO-TO on the ResNet and Llama runs to substantiate the "negligible overhead" claim.
2 comments
Thank you for the additional test accuracies, SVRG/SAGA results, time-to-target measurements, and optimized per-step timing. These are useful additions. However, several core concerns remain unresolved:
- The Adam tuning comparison is still not symmetric. The requested Adam runs with the grid extended to were not provided. Reporting selected rates of for Adam and for Diag-TO clarifies the configurations used, but does not establish whether Adam's reported plateau, or the large ResNet effects, survives a comparably broad search.
- The LLM evidence remains incomplete. The response does not provide downstream metrics with seed variance, the exact fine-tuned weight subset, or Llama timing/overhead measurements. The new generalization and runtime evidence is largely limited to CIFAR ResNets.
- The variance-reduction baselines remain incomplete. SVRG and SAGA were added on the non-convex ResNet tasks, but not on the strongly convex suite; VRADAM and AMSGrad are still absent.
- Parts of the theoretical request remain open. The response states that derivations will be added, but does not itself provide the requested complete RankOne-TO boundedness/convergence result or a quantitative comparison of asymptotic directional-variance constants with momentum. I nevertheless appreciate the clarification that vanishing estimation MSE should not be presented as a proven smaller asymptotic constant than momentum.
I therefore maintain my overall rating.
Thank you for checking the main recursion and for the detailed suggestions.
1. Hyperparameters.
The supplementary runs reuse the selected settings. Adam uses learning rate 0.0001, with 0.95 epoch decay on ResNet18 and a constant rate on ResNet50. Diag-TO uses 0.01 with 0.95 decay, beta=0.5, and alpha=0.5/0.1 for ResNet18/50. We will state these settings explicitly and describe the results as conditional on this tuning protocol.
2. Additional baselines and generalization.
We completed 20 SVRG/SAGA runs, five seeds per method and model. Training-loss pilots over {0.001,0.01,0.1} select 0.01 for both methods, followed by 0.95 epoch decay. Final test accuracy (%) is mean +/- sample SD:
All use the original augmented-test protocol. SVRG/SAGA sample fixed mini-batch components with replacement, unlike the original shuffled traversal.
3. Speed and overhead.
We agree that rho measures relative minimum-loss improvement. For the original implementation on A100, mean epochs / elapsed seconds to loss 0.3 are:
Separately, optimized Diag-TO takes 28.266 ms per ResNet50 step versus Adam's 27.643 ms, both peaking at 2128.6 MiB. This A100, batch-64 FP32 benchmark uses three trials of 30 warmup and 200 measured steps, excluding loading and evaluation. We will replace the universal overhead claim with this measured result.
4. Theory and variance reduction.
For projected Diag-TO, the proof uses a diagonal mixing matrix. RankOne TO additionally needs a cumulative-step condition bounding both factors. We will include the derivations for the theoretical decaying schedules in the appendix.
The theorem establishes vanishing gradient-estimation MSE. Momentum with decaying averaging weights can also have vanishing variance, so we will distinguish this guarantee from a claim of a smaller asymptotic constant.