acceptodds
Under review as a conference paper at ICLR 2027

A Trainable Optimizer

Abstract

The concept of learning to optimize involves utilizing a trainable optimization strategy rather than relying on manually defined full gradient estimations such as ADAM. We present a framework that jointly trains the full gradient estimator and the trainable weights of the model. Specifically, we prove that pseudo-linear TO (Trainable Optimizer), a linear approximation of the full gradient, matches SGD’s convergence rate while effectively reducing variance. Pseudo-linear TO incurs negligible computational overhead, requiring only minimal additional tensor multiplications. To further improve computational efficiency, we introduce two simplified variants of Pseudo-linear TO. Experiments demonstrate that TO methods converge faster than benchmark algorithms (e.g., ADAM) in both strongly convex and non-convex settings, and fine tuning of an LLM.

Primary area:
optimization
Keywords:
trainable optimizer, convergence analysis, linear approximation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HVyWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper revisits learning-to-optimize from a new angle: instead of meta-training a learned optimizer across many tasks, the optimizer's own parameters are co-trained online, together with the model weights, on the single task being solved. Concretely, the optimizer maintains a simple linear model that predicts the full-batch gradient from the current weights; at every iteration this linear model is nudged to better fit the freshly observed stochastic gradient, and its prediction is then used as the update direction for the model. Because a full linear model needs a matrix-sized optimizer state, the authors also propose two memory-light variants that restrict it to diagonal or rank-one structure. On the theory side, they prove that for strongly convex objectives both the distance to the optimum and the gradient-prediction error decay at the same rate as standard stochastic gradient descent, and, since the prediction error vanishes asymptotically, they position the method as a trainable variance-reduction scheme; classical momentum SGD is recovered as a special case. Experiments on logistic regression, feed-forward networks, ResNet on CIFAR-10 and Llama-7b fine-tuning claim improvements over ADAM and momentum, although, as detailed below, the empirical support for these claims is weak.

Critical strengths
  1. Interesting, simple, and well-motivated idea.

  2. I re-derived the core recursion of Theorem 3.4: Ĝt − Gt = (1 − δt)(Ĝt−1 − Gt−1 + At−1(wt − wt−1) − (Gt − Gt−1)) + δt(gt − Gt) with δt = αt‖wt‖² + βt; the proof then proceeds by a standard joint induction, and the boundedness results (Props. 3.2–3.3) and the level-set argument for the convex case (Lemma A.3) are carefully written. The mechanism is sound.

  3. Diag-TO and RO-TO reduce 400 MB–40 GB optimizer states to KB–MB scale (Table 1), which is what makes the idea usable at all on deep models.

Critical weaknesses
  1. Section 4.1 tunes ADAM over γ ∈ [1e-5, 5e-3] but TO over γ ∈ [1e-3, 0.5]; the overlap is only {1e-3, 2e-3, 5e-3}. The only large effects in the paper — ResNet18 ρ = 0.5 and ResNet50 ρ = 0.9 (Figure 2 shows ADAM plateauing near 0.3 while TO reaches ≈0.03) — occur precisely on a batch-64 CIFAR ResNet task where ADAM is never allowed learning rates above 5e-3 while TO may use 0.05–0.5. ADAM at 1e-2 on such tasks would not exhibit that plateau. Without per-experiment final hyperparameters and a symmetric grid, these results cannot be interpreted as a method win. The Momentum and SAGA grids are not reported at all.

  2. The metric ρ compares achieved minimum training loss, not epochs or wall-clock to a target loss. There is no time-to-target experiment, and no wall-clock overhead measurement anywhere despite the abstract's "negligible computational overhead" claim; the A-update is an O(d²) outer product per step, so at least a Diag/RO per-step timing table is needed.

  3. For a method that positions itself as variance reduction, the strongly-convex suite never compares against SVRG, SAGA (only appears in the λ=0 figures), VRADAM, or AMSGrad; the L-BFGS comparison under mini-batch sampling is a straw man since L-BFGS is a deterministic full-gradient method.

  4. Everything except one Llama validation curve is training loss. There is no test accuracy on any classification task and no downstream metric for the LLM fine-tuning; a variance-reduced optimizer can reach lower training loss while generalizing worse, so the current evidence cannot support deployment claims.

Questions for authors
  1. Please report the selected per-task hyperparameters (γ, schedule, α, β for TO; γ for ADAM/Momentum/SAGA) and re-run ResNet18/50 with ADAM's grid extended to [1e-2, 1e-1]. Does ADAM still plateau near 0.3, and do ρ = 0.5 / 0.9 survive?
  2. Please quantify convergence speed properly: epochs and wall-clock to reach a fixed target training loss, plus test accuracy for all classification tasks, and downstream metrics (e.g., a GSM8K/MMLU subset or MT-Bench) with seed variance for the Llama experiment, together with the exact specification of the fine-tuned weight subset.
  3. Please compare against SVRG, SAGA, VRADAM and AMSGrad on the strongly convex suite and at least one non-convex task, since the paper's positioning is variance reduction.
  4. Please provide proofs (or a counterexample discussion) for Diag-TO and RO-TO convergence, in particular boundedness of the rank-one factors.
  5. Quantitatively: with matched decaying βt, what are the asymptotic directional-variance constants of TO versus momentum SGD? If they coincide, the variance-reduction claim should be reframed.
  6. Please add a per-step wall-clock overhead table for Diag-TO/RO-TO on the ResNet and Llama runs to substantiate the "negligible overhead" claim.
2 comments

Thank you for checking the main recursion and for the detailed suggestions.

1. Hyperparameters.

The supplementary runs reuse the selected settings. Adam uses learning rate 0.0001, with 0.95 epoch decay on ResNet18 and a constant rate on ResNet50. Diag-TO uses 0.01 with 0.95 decay, beta=0.5, and alpha=0.5/0.1 for ResNet18/50. We will state these settings explicitly and describe the results as conditional on this tuning protocol.

2. Additional baselines and generalization.

We completed 20 SVRG/SAGA runs, five seeds per method and model. Training-loss pilots over {0.001,0.01,0.1} select 0.01 for both methods, followed by 0.95 epoch decay. Final test accuracy (%) is mean +/- sample SD:

MethodResNet18ResNet50
Adam92.60 +/- 0.2488.14 +/- 0.37
Diag-TO92.80 +/- 0.2191.93 +/- 0.45
SVRG91.88 +/- 0.2889.55 +/- 0.26
SAGA92.18 +/- 0.2089.96 +/- 0.26

All use the original augmented-test protocol. SVRG/SAGA sample fixed mini-batch components with replacement, unlike the original shuffled traversal.

3. Speed and overhead.

We agree that rho measures relative minimum-loss improvement. For the original implementation on A100, mean epochs / elapsed seconds to loss 0.3 are:

  • ResNet18: Adam 16.4 / 525; Diag-TO 14.8 / 603.
  • ResNet50: Adam 45.4 / 2162; Diag-TO 14.8 / 1109.

Separately, optimized Diag-TO takes 28.266 ms per ResNet50 step versus Adam's 27.643 ms, both peaking at 2128.6 MiB. This A100, batch-64 FP32 benchmark uses three trials of 30 warmup and 200 measured steps, excluding loading and evaluation. We will replace the universal overhead claim with this measured result.

4. Theory and variance reduction.

For projected Diag-TO, the proof uses a diagonal mixing matrix. RankOne TO additionally needs a cumulative-step condition bounding both factors. We will include the derivations for the theoretical decaying schedules in the appendix.

The theorem establishes vanishing gradient-estimation MSE. Momentum with decaying averaging weights can also have vanishing variance, so we will distinguish this guarantee from a claim of a smaller asymptotic constant.

Thank you for the additional test accuracies, SVRG/SAGA results, time-to-target measurements, and optimized per-step timing. These are useful additions. However, several core concerns remain unresolved:

  1. The Adam tuning comparison is still not symmetric. The requested Adam runs with the grid extended to [10−2,10−1][10^{-2},10^{-1}] were not provided. Reporting selected rates of 10−410^{-4} for Adam and 10−210^{-2} for Diag-TO clarifies the configurations used, but does not establish whether Adam's reported plateau, or the large ResNet effects, survives a comparably broad search.
  2. The LLM evidence remains incomplete. The response does not provide downstream metrics with seed variance, the exact fine-tuned weight subset, or Llama timing/overhead measurements. The new generalization and runtime evidence is largely limited to CIFAR ResNets.
  3. The variance-reduction baselines remain incomplete. SVRG and SAGA were added on the non-convex ResNet tasks, but not on the strongly convex suite; VRADAM and AMSGrad are still absent.
  4. Parts of the theoretical request remain open. The response states that derivations will be added, but does not itself provide the requested complete RankOne-TO boundedness/convergence result or a quantitative comparison of asymptotic directional-variance constants with momentum. I nevertheless appreciate the clarification that vanishing estimation MSE should not be presented as a proven smaller asymptotic constant than momentum.

I therefore maintain my overall rating.

Reviewer_PyUHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The negative gradient direction of update returned by an optimizer can be seen as a parameterized function of past gradients and model weights. The paper proposes an optimization strategy that alternatively updates the parameterized optimizer and the model parameters. A convergence proof is attempted under the assumptions that batch gradients are Lipschitz continuous and have bounded variance. Two variants of the method are proposed to reduce the memory footprint. The experimental results compare performance against ADAM in terms of the minimum training loss achieved over training epochs, for datasets such as MNIST, CIFAR10, KDD, etc., and models such as FFN, ResNet18, ResNet50, etc.

Critical strengths

The idea of offering an alternative update to the optimizer and the model seems to be novel. But whether this will lead to good generalization performance remains a key concern.

Critical weaknesses

The performance metric of a new optimizer needs to demonstrate its relevance with respect to the existing optimizers from several aspects:

  1. Generalization properties of the optimizer by comparing training and test accuracies.
  2. The computation time, memory, and number of floating-point operations required for each update.
  3. Compare performance with other state-of-the-art optimizers.
  4. Demonstrate performance of the optimization method on challenging datasets (e.g., ImageNet) and model architectures.

However, the experimental results are restricted to memory requirement (Full-TO seems to have a very high memory requirement) and training loss minimization.

Additionally, since memory-wise running Full-TO seems not feasible, and the authors present other alternatives to bypass that, the proofs can be attempted for the variant that the authors believe to be most suitable for practice.

There are probably some technical errors in the construction of the proof; these are highlighted in the next section, "Questions for Authors."

Questions for authors
  1. Line 257: F(w)=1(Nb)∑B:∣B∣=bFB(w)F(w)=\frac{1}{N \choose b} \sum_{B: \lvert B\rvert = b} F^B(w) is incorrect. Correcting this would make Lipschitz continuity calculations of line 259 dependent on N.

  2. Line 262: "∇F\nabla F is Lipschitz continous. Therefore, ∥∇F(wt)∥2\lVert\nabla F(w_t)\rVert_2 is bounded" -- seems to be an incorrect argument. We would rather need FF to be Lipschitz continuous, for a bounded gradient.

2 comments

We appreciate the reviewer for the helpful comments.

1. Generalization and stronger baselines.

We add five-seed CIFAR-10 experiments on ResNet18/50. Final test accuracy (%) is mean +/- sample SD:

MethodResNet18ResNet50
Adam92.60 +/- 0.2488.14 +/- 0.37
Diag-TO92.80 +/- 0.2191.93 +/- 0.45
SVRG91.88 +/- 0.2889.55 +/- 0.26
SAGA92.18 +/- 0.2089.96 +/- 0.26

On ResNet50, online training accuracy is 99.523 +/- 0.063% for Diag-TO and 91.220 +/- 0.099% for Adam. Comparisons use the selected configurations and the original random test crop/flip and batch-mean evaluation.

2. Computational cost.

Full-TO uses O(d2)O(d^2) state and arithmetic; Diag-TO uses two dd-dimensional state vectors and O(d)O(d) arithmetic. An analytic implementation removes auxiliary autograd and repeated flattening. On ResNet50/A100, batch 64, FP32, step times are 28.266 ms for optimized Diag-TO and 27.643 ms for Adam; both peak at 2128.6 MiB. These are medians of three trial means with 30 warmup and 200 measured steps, excluding loading and evaluation. Both optimizers maintain two parameter-sized state tensors. SAGA additionally stores 32.55/68.52 GiB of gradients in host RAM for ResNet18/50.

3. Mini-batch averaging identity.

The identity is correct because FBF^B is a batch average. Each sample appears in (N−1b−1)\binom{N-1}{b-1} subsets, so 1(Nb)∑B:∣B∣=bFB=(N−1b−1)b(Nb)∑i=1Nfi=F.\frac{1}{\binom{N}{b}}\sum_{B:\lvert B\rvert=b}F^B=\frac{\binom{N-1}{b-1}}{b\binom{N}{b}}\sum_{i=1}^{N}f_i=F.

Averaging the LL-Lipschitz batch gradients preserves LL, without an additional multiplicative NN factor.

4. Bounded gradients.

The argument uses both smoothness and the bounded projected domain. Since ∇F(w∗)=0\nabla F(w^*)=0 and

∥wt∥,∥w∗∥≤Dw,\|w_t\|,\|w^*\|\le D_w,

we have

∥∇F(wt)∥≤L∥wt−w∗∥≤2LDw.\|\nabla F(w_t)\|\le L\|w_t-w^*\|\le 2LD_w.

For stochastic gradients, add max⁡B∥∇FB(w∗)∥\max_B\|\nabla F^B(w^*)\|, which is finite over finitely many batches.

Please engage with the authors during the discussion period.

Reviewer_G3KfOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Trainable Optimizer (TO), a framework that jointly updates model parameters and a gradient estimator by minimizing a squared-error loss against stochastic gradients. The main algorithm uses an affine estimator, with diagonal and rank-one variants. For the projected pseudo-linear method, the paper establishes O(1/t)O(1/t) bounds on both the squared parameter error and the gradient-estimation mean-squared error under strongly convex assumptions. Experiments cover logistic regression, feedforward networks, ResNets, and partial fine-tuning of Llama-7B.

Critical strengths

The Learning to Optimize(L2O) framework is a classic and interesting direction in optimizer design. The algorithm in the paper does not require a double-loop oracle with another meta-training tasks. Authors also provide a joint analysis of optimization error and gradient-estimation error.

Critical weaknesses

1. The benefit of the learned estimator beyond gradient smoothing is not sufficiently established.

The O(1/t)O(1/t) convergence guarantee matches SGD and momentum, while the proposed method introduces additional memory and computational costs. Moreover, setting A0=0A_0=0 and αt=0\alpha_t=0 reduces the method to bt=(1−βt)bt−1+βtgtb_t=(1-\beta_t)b_{t-1}+\beta_tg_t. A furthr theoretical and practical comparison against this SGD with momentum should be included.

2. The affine gradient model and auxiliary loss need clearer justification.

Section 3.1 motivates the affine estimator through a Taylor expansion around a stationary point, but it is unclear how this local approximation justifies the model when the iterates are far from that point. Similarly, the auxiliary loss ll measures gradient-fitting error rather than optimization progress. The paper would benefit from explaining why this objective is a suitable surrogate for learning an effective optimizer, beyond ensuring convergence of the gradient estimate.

3. The computational-efficiency claims need further validation.

Full-TO requires quadratic optimizer-state storage, and the paper reports memory comparisons among TO variants. However, it does not provide end-to-end runtime, peak-memory, or time-to-target comparisons against the baselines. Measuring progress primarily in epochs while assuming negligible overhead does not establish practical efficiency, particularly when the reported loss improvements are small.

4. The evidence for practical benefits remains limited.

The experimental evaluation would benefit from stronger baseline validation and larger-scale benchmarks. In the convex experiments, L-BFGS appears to underperform several first-order methods, which warrants clarification of its implementation and hyperparameter settings. For the neural-network experiments, the relatively small datasets provide limited evidence of TO's effectiveness in modern large-scale training settings.

Questions for authors
  1. Can the authors isolate the benefit of learning AtA_t through an A0=0,αt=0A_0=0,\alpha_t=0 ablation with matched schedules and tuning budgets?

  2. Do the convergence guarantees extend to the experimental learning-rate schedules and the diagonal and rank-one variants?

  3. What justifies the affine model away from stationary points and the use of gradient-fitting loss ll to improve optimization performance?

  4. Can the authors report runtime, peak memory, and time-to-target comparisons against the baselines?

  5. Can the authors clarify the L-BFGS implementation and tuning, and provide larger-scale neural-network experiments to substantiate the practical benefits?

  6. Should Algorithm 1, line 6 use θt+1\theta_{t+1} instead of θt\theta_t to be consistent with Algorithm 2?

4 comments

We appreciate the reviewer for the helpful comments.

1. Momentum and the A=0 ablation.

Setting A_0=0 and alpha_t=0 recovers normalized momentum. We compared this ablation with Diag-TO using matched remaining settings and five seeds. Average test accuracy (%) was:

  • ResNet18: Diag-TO 92.80; A=0 92.59 .
  • ResNet50: Diag-TO 91.93; A=0 91.23 .

The differences suggest that gradient smoothing accounts for much of the observed benefit on these tasks. Our theoretical contribution is the joint analysis of optimization and estimation errors, rather than a faster worst-case rate than momentum.

2. Estimator motivation and theory.

The Taylor expansion motivates the affine family; its coefficients adapt along the trajectory. For a pre-update prediction h and an unbiased fresh gradient g, expected squared fitting error equals squared full-gradient prediction error plus gradient-noise variance. Convergence follows by jointly controlling prediction error and model-parameter error.

For projected Diag-TO, a diagonal mixing matrix replaces the scalar coefficient in the Full-TO proof. RankOne TO additionally requires a small-cumulative-step condition to bound both factors. The joint O(1/t) result uses gamma_t,beta_t of order 1/t and alpha_t of order 1/t^2.

3. Runtime and memory.

With the selected settings and original implementation on A100, mean elapsed seconds to training loss 0.3 were Adam/Diag-TO: 525/603 on ResNet18 and 2162/1109 on ResNet50.

Our optimized analytic Diag-TO update takes 28.266 ms per ResNet50 training step versus Adam's 27.643 ms, with the same peak CUDA allocation of 2128.6 MiB.

4. SVRG/SAGA baselines.

We completed five-seed SVRG/SAGA experiments on both ResNets. Mean final test accuracy (%) is:

MethodResNet18ResNet50
Diag-TO92.8091.93
SVRG91.8889.55
SAGA92.1889.96

These results use the original augmented-test protocol.

5. L-BFGS.

Our L-BFGS baseline uses mini-batches, with the closure evaluated on the current batch. We will clarify its implementation and settings and restrict conclusions to this mini-batch setup; the results do not establish an advantage over full-gradient L-BFGS.

6. Algorithm indexing.

Yes. Algorithm 1, line 6 should use the updated theta_{t+1}, consistent with Algorithm 2. We will correct this index.

Thank you for the additional experiments and explanations. The correction to Algorithm 1 and the clarification of the L-BFGS setup address those points. I have two remaining questions.

  1. Benefit of learning AtA_t. The added ablation helps clarify the role of gradient smoothing. However, learning AtA_t increases the mean test accuracy by only 0.21 percentage points on ResNet18 and 0.70 percentage points on ResNet50. Can you report the variation across seeds? The current difference is too small to support the improvement of Diag-TO over normalized momentum SGD.

  2. Motivation of the affine model for the optimizer. Can the authors explain more about "The Taylor expansion motivates the affine family"? The Taylor expansion motivates the affine model for the trainable optimizer. However, the answer does not address this question: "It is unclear how this local approximation justifies the model when the iterates are far from that point." For example, popular adaptive methods such as Adam and AdaGrad change the geometry instead of applying only an affine operation to the gradient. Because there are no benefits to the convergence rate, why does the affine trainable optimizer give benefits compared with commonly used adaptive methods?

Thank you for the follow-up questions.

  1. Seed variation. Final test accuracy across five seeds (%, mean ±\pm sample SD):

| Model | Diag-TO | A=0 |

| ResNet18 | 92.80±0.2192.80 \pm 0.21 | 92.59±0.1392.59 \pm 0.13 |

| ResNet50 |91.93±0.4591.93 \pm 0.45 | 91.23±0.5591.23 \pm 0.55 |

  1. Motivation of the affine model.

We agree that a Taylor expansion around a stationary point alone does not justify accuracy far from that point. TO models the weight-to-gradient map as Atw+btA_tw+b_t, with coefficients fitted online. This provides a simple family with explicit updates. It also represents quadratic gradients globally: if F(w)=12w⊤Hw+c⊤wF(w)=\frac12 w^\top Hw+c^\top w, then ∇F(w)=Hw+c\nabla F(w)=Hw+c, regardless of the distance to the optimum. For general objectives, this is a modeling choice; our analysis controls prediction error rather than assuming a globally accurate Taylor approximation.

The intended benefit is to account for the changing gradient as the weights move. Before fitting the current gradient,

=\widehat G_{t-1}+A_{t-1}(w_t-w_{t-1}).$$ The additional term corrects the previous prediction for movement in the weights. When accurate, it can reduce the lag caused by averaging gradients from different iterates while retaining the benefit of smoothing. Adam/AdaGrad adapt update scaling, whereas TO learns a full-gradient predictor. Potential gains therefore concern finite-time prediction accuracy and noise reduction.

Thank you for your further explanation. However, my primary concerns have not been fully addressed:

  1. Based on the experimental results, the improvement of Diag-TO over normalized momentum is too negligible both on ResNet18 and ResNet50. I do not think these results are sufficient to establish the empirical advancement of the Diag-TO trainable optimizer.

  2. While I acknowledge that linear approximation is a valid modeling choice, it has not been substantiated as a practical approximation, nor has it been shown to yield additional theoretical benefits in paper and the rebuttal phase.

Therefore, I will maintain my negative rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.