- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces a new learned optimizer that reweights past gradients dynamically (as opposed to, e.g., simple momentum). It is hierarchical, giving more granularity to more recent intervals of gradients. On language, vision, and graph tasks, the optimizer generally outcompetes non-learned optimizers Adam, SGD, and SGDM.
Critical strengths
Presentation
The method is very simple and generally easy to understand (if not always to motivate specific decisions). This is important because Occam's Razor, and because it seems like it could be spliced with other methods that do orthogonal things, e.g., the mentioned VeLO. Figures 3/4 were extremely helpful in understanding the method.
Experiments
The evals look good against non-learned optimizers, if the comparisons are fair, more than a fraction of a percent accuracy, and not on totally trivial settings. §3.1 should mention the sizes though, rather than just later, but, for example, CIFAR-100 is a little small but not outrageously so – ditto Wikibooks and Wikitext. The meta-training is indeed very lightweight.
Critical weaknesses
Time vs. FLOPs
The time-per-step increase is astronomical, 7x on graph tasks. So I really don't know what matched "training budget" means in the main text—n.b., I have not yet reviewed the appendix. The scaling also appears bad, if that's the reason GPT's time/step is disproportionately worse than BERT's. I'd like to see time-matched graphs and results, given that, or an explanation of how the implementation is poorly-optimized and what the performance ceiling is likely to be.
Missing important details in main text
There are a LOT of missing details from the main text about the architecture and hyper-parameters: "default" is completely bogus: it's not like these optimisers are some company's product with a manual and a single obvious best setting. In any case, they can be cited for each task, then. §3.1 could have a little more detail on what tasks were used in training, so the reader has intuition about cross-domain generalization. I would boot §2.4 to the appendix entirely—I find that idea unmotivated—in return for more such details in the preceding and following sections. The description of the method on line 180 does not quite match the formula (2), and that formula could be simplified dramatically by removing the from the numerator, since it reduces to subtracting a constant from the summation (or that can be made explicit, if varies in an important way).
There is no direct comparison to other LOs, only off-hand mention of how the method is a lot more training- and memory- efficient than VeLO
Questions for authors
- What motivated the architectural setup, what choices were made and what was important? Why a scale token, separate from the softmax weighting?
- What's going on in §2.5 (mentioned above)?
- How does this approach compare to one of the original LO papers, "Learning to learn by gradient descent by gradient descent" (NIPS'16)? That also seems to dynamically use the optimization trajectory, running the updates through an LSTM. More recent work like VeLO is also only mentioned off-hand, how does do such methods' efficacy and performance compare?
8 comments
Response to Reviewer gA91 (2/2)
Q4. Architectural motivation and section organization
“What motivated the architectural setup, what choices were made and what was important? Why a scale token, separate from the softmax weighting?”
“I would boot §2.4 to the appendix entirely—I find that idea unmotivated—in return for more such details in the preceding and following sections.”
Thank you for asking us to clarify the motivation behind our architectural choices. We agree that the method section needs to explain why these components were chosen, not only how they operate. For the optimizer to determine how each part of the gradient history contributes to the update, it uses statistical summaries and histograms to represent its structural features across different parameter group sizes. Self-attention lets the history slots and scale token share these features before predicting the contributions, so each prediction considers the other stored gradients. SoftMax makes the slot contributions sum to one, while the scale () controls the overall update magnitude using the history context.
We revised Section 3.3, “Temporal coefficient prediction,” to explain these choices alongside Figure 4. Following your suggestion, we removed the former Section 2.4 from the main text and retained the existing description in Appendix C, “Spatio-Temporal Extension,” freeing space for the requested motivation and training details.
Q5. Discrepancy in meta-loss formula
“The description of the method on line 180 does not quite match the formula (2). What's going on in §2.5 (mentioned above)?”
Thank you for pointing out the inconsistency between the formula and its description. You are right that the expression can be simplified. The subtracted initial loss is a detached value, so removing it from the numerator leaves the meta-gradient unchanged and matches the description of the method. The revised meta-objective is
We corrected Equation 2 in Section 3.4, “Trajectory-level optimizer training,” to match this explanation. Appendix A.2.1, “Normalized held-out-loss objective,” gives the implementation details.
Q6. Comparison with learned optimizers
“How does this approach compare to one of the original LO papers, "Learning to learn by gradient descent by gradient descent" (NIPS'16)?”
“More recent work like VeLO is also only mentioned off-hand, how does do such methods' efficacy and performance compare?”
Thank you for suggesting a direct comparison with other learned optimizers. We agree that architectural differences alone do not establish how their performance and costs compare. Our optimizer achieved the highest overall performance among the compared learned optimizers on target tasks unseen during meta-training. The same optimizer was used across all targets after a single meta-training run of 0.87 GPU-hours, without task-specific fine-tuning.
We took a different approach to learned optimization by predicting contributions from stored gradient history each step, whereas the 2016 LSTM optimizer and recent learned optimizers such as VeLO and Celo2 predict updates coordinate-wise.
For the empirical comparison, we selected VeLO for its emphasis on versatility and Celo2 for inexpensive learned optimization, matching the two aims of our work.
| Performance across 12 tasks | Our optimizer* | VeLO | Celo2 |
|---|---|---|---|
| Geometric-mean gain over untuned Adam (%) ↑ | +5.32% | +4.46% | −6.01% |
| Training time on B200 | Our optimizer* | Celo2 | VeLO |
|---|---|---|---|
| Total across all 12 tasks | 10.62 h | 10.87 h | 11.12 h |
| Training time overhead relative to Adam (%) ↓ | 16.8 | 19.6 | 22.2 |
* Embeddings and the output head were trained with Adam for language models.
We expanded the related-work discussion by adding Section 2. Table 1 reports the learned-optimizer comparisons, discussed in Section 4.2, “Zero-Shot Generalization to Unseen Tasks.” Figures 5 and 6 in Section 4.3, “Time-Matched Performance and Deployment Cost,” show the language trajectories and deployment costs.
Great! I haven't read the revised PDF (don't know why OpenReview doesn't give us the diff interface), but the comments seem to address everything, especially the engineering to optimize the runtime overhead, though 3x still seems substantial, abstractly.
I look forward to seeing some of the results for responses to other reviews (like µLO and fixing instability) if they can be done in time!
Response to Reviewer gA91 (1/2)
Q1. Term “matched training budgets”
Thank you for requesting clarification on the training setup. We agree that the budget definition needs to distinguish optimizer updates from elapsed time or FLOPs. “Matched training budget” originally meant the same number of optimizer updates, with matched initialization and minibatch ordering. We clarified this meaning in the “Baselines and pairing” paragraph of Section 4.1, “Experimental Setup.” To address elapsed time separately, Section 4.3, “Time-Matched Performance and Deployment Cost,” includes the language comparisons in Figure 5.
Q2. Runtime improvements and time-matched results
Thank you for pointing out the runtime overhead. We agree that performance against elapsed time is important for judging the practical benefit.
Implementation overhead. Your comment helped us identify repeated copying and rearrangement of stored gradients. The revised profiles show a 52% reduction in optimizer computation time across all tasks. We changed history management to update stored gradients directly, avoiding repeated reversal, padding, and concatenation.
Performance against training time. Our optimizer achieved lower validation loss than Celo2 and VeLO on GPT-Tiny at the same training time. Celo2 remained behind despite assigning embeddings, output heads, biases, and one-dimensional tensors to AdamW, while our optimizer assigned only embeddings and the output head to Adam.
On BERT-Tiny, our optimizer completed training with lower validation loss in less training time than Celo2 and VeLO. Training time includes both stages using their separately measured step times.
Why GPT-Tiny has greater relative overhead. We clarify that the larger relative overhead on GPT-Tiny comes from its shorter model computation, rather than greater optimizer computation. BERT-Tiny’s larger batch size contributes to its longer model computation time per step.
* Embeddings and the output head were trained with Adam.
Also, FLOPs were replaced by measured runtime and its breakdown because arithmetic counts do not capture the time spent moving stored gradients or launching GPU operations. To address your request, Figure 5 in Section 4.3, “Time-Matched Performance and Deployment Cost,” shows the language comparisons against elapsed time, and Figure 6 provides the runtime breakdown. Appendix A.4, “Resource measurements,” Figure 8 provides the three Cora graph comparisons.
Q3. Training tasks and hyperparameters
Thank you for pointing out the need to clarify the training tasks and the scale of generalization. We agree that the main text needs concrete model details and baseline settings rather than the term “default.” Our optimizer was trained on networks with at most 8,450 parameters and applied to models with 32k–109.5M parameters on different tasks and datasets.
We expanded Section 4.1, “Experimental Setup,” so the optimizer-training and target architectures can be compared directly. BERT-Tiny and GPT-Tiny each have two Transformer layers of width 128 and two attention heads. The CIFAR-100 ViT has six blocks of width 256 and eight attention heads.
We replaced “default” with the actual Adam settings, learning rate 10−3 and (β1,β2)=(0.9,0.999). For the remaining baseline settings and search ranges, please see Appendix A.3.1, “Baseline protocols.”