acceptodds
Under review as a conference paper at ICLR 2027

Lightweight and Versatile Learned Optimization by Recombination of Gradient History

Abstract

This paper presents a lightweight and versatile learned optimizer that dynamically recombines gradient history, represented as averages over disjoint time spans. The optimizer reduces the prediction space to one scalar coefficient per gradient average, shared by multiple parameters. Progressively averaging older gradients minimizes memory cost of long history, while keeping their contributions independently accessible. A 37k-parameter network trained in 0.87 GPU-hours generalizes zero-shot to unseen tasks, lowering validation loss by 9.1% and 0.4% on BERT-Tiny and GPT-Tiny, and improving test accuracy over Adam by 3.5 %p on a Vision Transformer and by 2.7 %p on average across nine graph models, with FLOPs overhead as low as 0.3%.

Primary area:
optimization
Keywords:
Learned optimizers, Efficient learned optimization, Gradient-based optimization, Learning to optimize

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_gA91OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a new learned optimizer that reweights past gradients dynamically (as opposed to, e.g., simple momentum). It is hierarchical, giving more granularity to more recent intervals of gradients. On language, vision, and graph tasks, the optimizer generally outcompetes non-learned optimizers Adam, SGD, and SGDM.

Critical strengths

Presentation

The method is very simple and generally easy to understand (if not always to motivate specific decisions). This is important because Occam's Razor, and because it seems like it could be spliced with other methods that do orthogonal things, e.g., the mentioned VeLO. Figures 3/4 were extremely helpful in understanding the method.

Experiments

The evals look good against non-learned optimizers, if the comparisons are fair, more than a fraction of a percent accuracy, and not on totally trivial settings. §3.1 should mention the sizes though, rather than just later, but, for example, CIFAR-100 is a little small but not outrageously so – ditto Wikibooks and Wikitext. The meta-training is indeed very lightweight.

Critical weaknesses

Time vs. FLOPs

The time-per-step increase is astronomical, 7x on graph tasks. So I really don't know what matched "training budget" means in the main text—n.b., I have not yet reviewed the appendix. The scaling also appears bad, if that's the reason GPT's time/step is disproportionately worse than BERT's. I'd like to see time-matched graphs and results, given that, or an explanation of how the implementation is poorly-optimized and what the performance ceiling is likely to be.

Missing important details in main text

There are a LOT of missing details from the main text about the architecture and hyper-parameters: "default" is completely bogus: it's not like these optimisers are some company's product with a manual and a single obvious best setting. In any case, they can be cited for each task, then. §3.1 could have a little more detail on what tasks were used in training, so the reader has intuition about cross-domain generalization. I would boot §2.4 to the appendix entirely—I find that idea unmotivated—in return for more such details in the preceding and following sections. The description of the method on line 180 does not quite match the formula (2), and that formula could be simplified dramatically by removing the −Ltheld-L_t^{held} from the numerator, since it reduces to subtracting a constant KK from the summation (or that can be made explicit, if KK varies in an important way).

There is no direct comparison to other LOs, only off-hand mention of how the method is a lot more training- and memory- efficient than VeLO

Questions for authors
  1. What motivated the architectural setup, what choices were made and what was important? Why a scale token, separate from the softmax weighting?
  2. What's going on in §2.5 (mentioned above)?
  3. How does this approach compare to one of the original LO papers, "Learning to learn by gradient descent by gradient descent" (NIPS'16)? That also seems to dynamically use the optimization trajectory, running the updates through an LSTM. More recent work like VeLO is also only mentioned off-hand, how does do such methods' efficacy and performance compare?
8 comments
AuthorsOpenReview ↗

Response to Reviewer gA91 (1/2)

Q1. Term “matched training budgets”

“So I really don't know what matched "training budget" means in the main text”.

Thank you for requesting clarification on the training setup. We agree that the budget definition needs to distinguish optimizer updates from elapsed time or FLOPs. “Matched training budget” originally meant the same number of optimizer updates, with matched initialization and minibatch ordering. We clarified this meaning in the “Baselines and pairing” paragraph of Section 4.1, “Experimental Setup.” To address elapsed time separately, Section 4.3, “Time-Matched Performance and Deployment Cost,” includes the language comparisons in Figure 5.

Q2. Runtime improvements and time-matched results

“The time-per-step increase is astronomical, 7x on graph tasks.”

“I'd like to see time-matched graphs and results, given that, or an explanation of how the implementation is poorly-optimized and what the performance ceiling is likely to be.”

Thank you for pointing out the runtime overhead. We agree that performance against elapsed time is important for judging the practical benefit.

Implementation overhead. Your comment helped us identify repeated copying and rearrangement of stored gradients. The revised profiles show a 52% reduction in optimizer computation time across all tasks. We changed history management to update stored gradients directly, avoiding repeated reversal, padding, and concatenation.

Our optimizer*Earlier profileRevised profileReduction
Optimizer computation time per step, geometric mean across 12 tasks25.24 ms12.19 ms52%

Performance against training time. Our optimizer achieved lower validation loss than Celo2 and VeLO on GPT-Tiny at the same training time. Celo2 remained behind despite assigning embeddings, output heads, biases, and one-dimensional tensors to AdamW, while our optimizer assigned only embeddings and the output head to Adam.

GPT-TinyOur optimizerVeLOCelo2
Validation loss when our optimizer finishes training ↓4.2804.2924.546

On BERT-Tiny, our optimizer completed training with lower validation loss in less training time than Celo2 and VeLO. Training time includes both stages using their separately measured step times.

Why GPT-Tiny has greater relative overhead. We clarify that the larger relative overhead on GPT-Tiny comes from its shorter model computation, rather than greater optimizer computation. BERT-Tiny’s larger batch size contributes to its longer model computation time per step.

Our optimizer*Model computation per stepOptimizer computation per step
BERT-Tiny367.9 ms12.2 ms
GPT-Tiny37.4 ms11.1 ms

* Embeddings and the output head were trained with Adam.

Also, FLOPs were replaced by measured runtime and its breakdown because arithmetic counts do not capture the time spent moving stored gradients or launching GPU operations. To address your request, Figure 5 in Section 4.3, “Time-Matched Performance and Deployment Cost,” shows the language comparisons against elapsed time, and Figure 6 provides the runtime breakdown. Appendix A.4, “Resource measurements,” Figure 8 provides the three Cora graph comparisons.

Q3. Training tasks and hyperparameters

“There are a LOT of missing details from the main text about the architecture and hyper-parameters: "default" is completely bogus.”

“In any case, they can be cited for each task, then.”

Thank you for pointing out the need to clarify the training tasks and the scale of generalization. We agree that the main text needs concrete model details and baseline settings rather than the term “default.” Our optimizer was trained on networks with at most 8,450 parameters and applied to models with 32k–109.5M parameters on different tasks and datasets.

Our optimizerMeta-trainingDeployment
TasksClassification on synthetic data, CIFAR-10, and Fashion-MNISTLanguage modeling, CIFAR-100, and graph node classification
ArchitecturesMLP, ConvNet, single-block ViT, and residual ConvNetBERT-Tiny, GPT-Tiny, BERT-Base, six-block ViT, GCN, GraphSAGE, and GAT
Target-model parameters2,142–8,45032k–109.5M
Trajectory length160 updatesUp to 250k updates

We expanded Section 4.1, “Experimental Setup,” so the optimizer-training and target architectures can be compared directly. BERT-Tiny and GPT-Tiny each have two Transformer layers of width 128 and two attention heads. The CIFAR-100 ViT has six blocks of width 256 and eight attention heads.

We replaced “default” with the actual Adam settings, learning rate 10−310^{-3} and (β1,β2)=(0.9,0.999)(\beta_1,\beta_2)=(0.9,0.999). For the remaining baseline settings and search ranges, please see Appendix A.3.1, “Baseline protocols.”

AuthorsOpenReview ↗

Response to Reviewer gA91 (2/2)

Q4. Architectural motivation and section organization

“What motivated the architectural setup, what choices were made and what was important? Why a scale token, separate from the softmax weighting?”

“I would boot §2.4 to the appendix entirely—I find that idea unmotivated—in return for more such details in the preceding and following sections.”

Thank you for asking us to clarify the motivation behind our architectural choices. We agree that the method section needs to explain why these components were chosen, not only how they operate. For the optimizer to determine how each part of the gradient history contributes to the update, it uses statistical summaries and histograms to represent its structural features across different parameter group sizes. Self-attention lets the history slots and scale token share these features before predicting the contributions, so each prediction considers the other stored gradients. SoftMax makes the slot contributions sum to one, while the scale (ss) controls the overall update magnitude using the history context.

We revised Section 3.3, “Temporal coefficient prediction,” to explain these choices alongside Figure 4. Following your suggestion, we removed the former Section 2.4 from the main text and retained the existing description in Appendix C, “Spatio-Temporal Extension,” freeing space for the requested motivation and training details.

Q5. Discrepancy in meta-loss formula

“The description of the method on line 180 does not quite match the formula (2). What's going on in §2.5 (mentioned above)?”

Thank you for pointing out the inconsistency between the formula and its description. You are right that the expression can be simplified. The subtracted initial loss is a detached value, so removing it from the numerator leaves the meta-gradient unchanged and matches the description of the method. The revised meta-objective is

Lmeta=∑k=1KLt+kheldLtheld.\mathcal{L}_{\mathrm{meta}} = \sum_{k=1}^{K} \frac{L_{t+k}^{\mathrm{held}}}{L_t^{\mathrm{held}}}.

We corrected Equation 2 in Section 3.4, “Trajectory-level optimizer training,” to match this explanation. Appendix A.2.1, “Normalized held-out-loss objective,” gives the implementation details.

Q6. Comparison with learned optimizers

“How does this approach compare to one of the original LO papers, "Learning to learn by gradient descent by gradient descent" (NIPS'16)?”

“More recent work like VeLO is also only mentioned off-hand, how does do such methods' efficacy and performance compare?”

Thank you for suggesting a direct comparison with other learned optimizers. We agree that architectural differences alone do not establish how their performance and costs compare. Our optimizer achieved the highest overall performance among the compared learned optimizers on target tasks unseen during meta-training. The same optimizer was used across all targets after a single meta-training run of 0.87 GPU-hours, without task-specific fine-tuning.

We took a different approach to learned optimization by predicting contributions from stored gradient history each step, whereas the 2016 LSTM optimizer and recent learned optimizers such as VeLO and Celo2 predict updates coordinate-wise.

For the empirical comparison, we selected VeLO for its emphasis on versatility and Celo2 for inexpensive learned optimization, matching the two aims of our work.

Performance across 12 tasksOur optimizer*VeLOCelo2
Geometric-mean gain over untuned Adam (%) ↑+5.32%+4.46%−6.01%
Training time on B200Our optimizer*Celo2VeLO
Total across all 12 tasks10.62 h10.87 h11.12 h
Training time overhead relative to Adam (%) ↓16.819.622.2

* Embeddings and the output head were trained with Adam for language models.

We expanded the related-work discussion by adding Section 2. Table 1 reports the learned-optimizer comparisons, discussed in Section 4.2, “Zero-Shot Generalization to Unseen Tasks.” Figures 5 and 6 in Section 4.3, “Time-Matched Performance and Deployment Cost,” show the language trajectories and deployment costs.

Great! I haven't read the revised PDF (don't know why OpenReview doesn't give us the diff interface), but the comments seem to address everything, especially the engineering to optimize the runtime overhead, though 3x still seems substantial, abstractly.

I look forward to seeing some of the results for responses to other reviews (like µLO and fixing instability) if they can be done in time!

Reviewer_a88aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a lightweight learned optimizer to address the high meta-training cost and limited generalization of existing learned optimizers. The proposed method maintains gradient history as averages over multiple non-overlapping temporal intervals and uses a neural network to predict scalar coefficients for each interval to construct parameter updates. It further incorporates group-wise feature encoding and self-attention-based temporal coefficient prediction to enable generalization across diverse target models with a small parameter budget.

The proposed optimizer contains approximately 37k parameters and is meta-trained on four small-scale classification tasks with only 0.87 B200 GPU-hours. Its zero-shot generalization is evaluated on BERT-Tiny, GPT-Tiny, Vision Transformers, and GNNs. Experimental results demonstrate improvements over Adam across multiple architectures and tasks. Additional analyses of history span, parameter grouping, temporal coefficient dynamics, and computational overhead further examine the effectiveness and efficiency of the proposed approach.

Critical strengths

1. Simple and Efficient Optimizer Design via Gradient-History Recombination

Unlike conventional learned optimizers that directly predict parameter-wise updates, this paper proposes dynamically combining gradient histories from different temporal intervals to construct update directions. In particular, representing gradient history as disjoint multi-resolution averages and predicting only group-wise shared scalar coefficients significantly reduces the policy size. The resulting optimizer requires only approximately 37k parameters and 0.87 GPU-hours of meta-training, offering a practical approach to reducing the training cost of learned optimizers.

2. Zero-Shot Generalization Across Diverse Unseen Tasks

A key strength of this paper is its demonstration of zero-shot generalization from four small-scale classification tasks to BERT-Tiny, GPT-Tiny, ViT, and various GNN architectures without additional training. The proposed optimizer exhibits stable performance over long optimization horizons and achieves improvements over Adam across different models and datasets. Additional analyses of history span, parameter grouping, and temporal coefficients provide further insights into the behavior of the proposed method.

Critical weaknesses

1. Limited Comparisons with Recent Optimizers and Hyperparameter Tuning

  • The proposed method is primarily compared with Adam, SGD, and SGDM using fixed learning rates and default hyperparameters without task-specific tuning.
  • While this setting is relevant for evaluating off-the-shelf performance, it does not establish superiority over well-tuned optimizers.
  • Comparisons with recent methods such as AdEMAMix (ICLR 2025), Celo2, and μLO (ICLR 2026) are missing, making it difficult to assess the competitiveness and novelty of the proposed approach.

2. Substantial Deployment Overhead and Insufficient Ablation of Learned Temporal Recombination

  • Despite its lightweight design, the optimizer incurs approximately 7.37× and 3.93× longer wall-clock training times than Adam on ViT and GNN tasks, respectively (Figure 9).
  • Full-span gradient history also requires more than 8× the optimizer-state memory of Adam, highlighting a gap between low meta-training cost and practical deployment efficiency.
  • Although history span and parameter grouping are analyzed, direct comparisons with fixed coefficients, conventional multi-timescale momentum, and simpler coefficient predictors are missing.
  • Consequently, the specific contribution of learned temporal recombination to the observed performance gains remains insufficiently isolated.
Questions for authors

Q1. How Does the Proposed Method Compare with Recent Optimizers and Hyperparameter-Tuned Baselines?

  • The evaluation primarily compares against Adam, SGD, and SGDM with fixed hyperparameters, omitting recent optimizers such as AdEMAMix and Celo2, as well as task-specific tuned AdamW.
  • Could the authors provide comparisons under matched training budgets and evaluation settings, particularly with AdEMAMix, which also leverages multi-timescale gradient history?

Q2. How Efficient Is the Proposed Method in Practice Given Its Deployment Overhead?

  • Figure 9 shows substantially longer wall-clock training times than Adam on ViT and GNN tasks, raising concerns about whether the performance gains justify the additional computational cost.
  • Could the authors compare performance under matched wall-clock time and GPU memory budgets?
  • How do history storage and feature extraction costs scale to larger models?

Q3. Can the Contribution of Learned Temporal Recombination Be Isolated?

  • Existing ablations examine history span and parameter grouping but do not clearly establish whether learned temporal coefficients are essential to the performance improvements.
  • Could the authors compare against fixed or uniform coefficients applied to the same multi-resolution history, as well as simpler predictors without self-attention?

Q4. What Explains the Discrepancy in GNN History-Depth Results Between the Main Text and Appendix?

  • Main Table 2 reports average GNN accuracies of 76.44% and 76.47% for full-span and 63-step history, respectively, whereas Appendix Table 9 reports 75.11% and 75.48%.
  • Could the authors clarify whether these differences arise from distinct meta-training runs, random seeds, or other experimental settings?
3 comments
AuthorsOpenReview ↗

Response to Reviewer a88a (1/2)

Q1. Recent optimizers and task-tuned baselines

“The evaluation primarily compares against Adam, SGD, and SGDM with fixed hyperparameters, omitting recent optimizers such as AdEMAMix and Celo2, as well as task-specific tuned AdamW.”

Thank you for suggesting comparisons with AdEMAMix, Celo2, and task-tuned AdamW. We agree that recent and task-tuned baselines are needed to assess the competitiveness of our optimizer. Our optimizer achieved the highest overall performance among the compared classical and learned optimizers on target tasks unseen during meta-training. The same optimizer was used across all targets after a single meta-training run, without task-specific fine-tuning. AdamW’s learning rate and weight decay were tuned for each task. Comparisons used the same number of optimizer updates.

Performance across 12 tasksOur optimizer*Task-tuned AdamTask-tuned AdamWTask-tuned SGDMTask-tuned AdEMAMixVeLOCelo2
Geometric-mean gain over untuned Adam (%) ↑+5.32%+2.91%+2.84%+3.72%+2.18%+4.46%−6.01%

* Embeddings and the output head were trained with Adam for language models.

Meta-training our optimizer required less time than the complete hyperparameter searches for Adam and AdamW.

Training costOur optimizer: one meta-training runAdam: complete searchAdamW: complete search
B200 GPU-hours0.877.0619.17

We also plan to compare with μLO before the revision deadline, as it shares our aim of reducing the training cost of learned optimizers.

Tables 1 and 2 report the learned-optimizer and task-tuned comparisons, discussed in Section 4.2, “Zero-Shot Generalization to Unseen Tasks.” Appendix A.3.1, “Baseline protocols,” lists the hyperparameters and search ranges.

Q2. Runtime, memory, and deployment cost

“How Efficient Is the Proposed Method in Practice Given Its Deployment Overhead?”

“Could the authors compare performance under matched wall-clock time and GPU memory budgets?”

“How do history storage and feature extraction costs scale to larger models?”

Thank you for raising these concerns. We agree that runtime and memory requirements are important for an optimizer to be practical, and that low meta-training cost alone does not establish practical deployment efficiency. After optimizing history management, the measured time per step was reduced to 3.63× Adam on ViT and 3.74× on GNNs. The runtime comparisons below use the optimized implementation.

Time-matched performance. Our optimizer achieved lower validation loss than Celo2 and VeLO on GPT-Tiny at the same training time. Celo2 remained behind despite assigning embeddings, output heads, biases, and one-dimensional tensors to AdamW, while our optimizer assigned only embeddings and the output head to Adam.

GPT-TinyOur optimizerVeLOCelo2
Validation loss when our optimizer finishes training ↓4.2804.2924.546

It also had the lowest total training time across the evaluated workload.

Total training timeOur optimizer*Celo2VeLO
Across 12 tasks on B20010.62 h10.87 h11.12 h

Limited memory budget. When memory is limited, our optimizer can use fewer history slots while retaining optimization performance. We found that limiting the number of history slots to six retained competitive performance overall across language, vision, and graph tasks.

Performance across 12 tasksOur optimizer: full span*Our optimizer: six slots*Adam
Geometric-mean gain over Adam (%) ↑+5.32+5.41—

* Embeddings and the output head were trained with Adam for language models.

Scaling to larger models. Our optimizer can control storage and computation costs through its history and parameter-group configurations, although history storage and feature extraction scale with target model size. Increasing the scale factor (rr) averages more gradients per slot, extending history coverage without adding storage. Increasing parameter-group size can accommodate more parameters without increasing the number of predictions, since self-attention and coefficient and scale (ss) prediction operate on fixed-dimensional representations of each group and slot.

Section 4.3, “Time-Matched Performance and Deployment Cost,” presents language trajectories in Figure 5 and the runtime breakdown in Figure 6(b). Appendix A.4, “Resource measurements,” provides the Cora trajectories in Figure 8. Table 4 in Section 4.4, “Ablations and Behavioral Analysis,” reports the history-span comparisons.

AuthorsOpenReview ↗

Response to Reviewer a88a (2/2)

Q3. Temporal recombination and self-attention

“Can the Contribution of Learned Temporal Recombination Be Isolated?”

“Could the authors compare against fixed or uniform coefficients applied to the same multi-resolution history, as well as simpler predictors without self-attention?”

Thank you for suggesting these controls. We agree that uniform coefficients and a predictor without self-attention help distinguish the contributions of these components. We observed degradation both when replacing learned coefficients with uniform coefficients and when removing self-attention.

Uniform coefficients. Replacing learned coefficients with uniform coefficients worsened descent on LeNet, and fixing the scale (ss) to 1.0 further increased the accumulated training loss. Uniform coefficients replace learned coefficients over the same stored gradient history. The table reports training-loss AUC (area under the curve) from coefficient replacement through update 200.

Step of coefficient policy replacementUniform coefficients: AUC increase over our optimizer ↓Uniform coefficients and scale (s=1s=1): AUC increase over our optimizer ↓
5+16.4%+37.0%
15+17.0%+71.5%

We also observed that learned coefficients avoided numerical divergence on ViT with the scale (ss) fixed to 1.0, whereas uniform coefficients produced nonfinite gradients in all three seeds. One learned-coefficient seed still achieved poor accuracy.

ViTOur optimizer with scale (s=1.0s=1.0)Uniform coefficients with scale (s=1.0s=1.0)
Runs with nonfinite gradients ↓0/33/3 (all diverged)

Self-attention ablation. We found that removing self-attention reduced mean test accuracy by 29.77 percentage points on GAT/PubMed. This optimizer was retrained without self-attention, keeping the representation of history unchanged. We think this is because self-attention allows coefficient and scale (ss) predictions to share information across the gradient history.

GAT/PubMedOur optimizerOur optimizer without self-attention
Test accuracy (%)78.77 ± 0.2149.00 ± 7.74

Figure 7 in Section 4.4, “Ablations and Behavioral Analysis,” shows the coefficient-replacement trajectories and loss contours. Appendix B.2.4, “Learned coefficients, scale, and self-attention,” provides both replacement points in Figure 12, the AUC comparison in Table 11, the fixed-scale ViT results in Table 12, and the self-attention ablation in Table 13.

Q4. GNN history-depth reporting

“What Explains the Discrepancy in GNN History-Depth Results Between the Main Text and Appendix?”

Thank you for pointing out this discrepancy. The appendix reported the test accuracy of the final checkpoint, while the main table used the checkpoint with the lowest validation loss. The training settings, including the models, data splits, and seeds, were identical. We corrected Table 10 in Appendix B.2.2, “History-depth controls,” to use the checkpoint with the lowest validation loss, so it can be compared directly with the main results.

AuthorsOpenReview ↗

Completed μLO comparison and updates on BERT-Base results

Thank you for your patience while we completed the μLO (ICLR'26) comparison. Our optimizer retains the highest overall performance after adding μLO, with the comparison covering all 12 tasks and three seeds. Its geometric-mean gain over Adam is 5.32%, compared with 0.38% for μLO.

We also found that changing only the scale formulation to an exponential mapping during both meta-training and deployment enabled our optimizer to outperform Adam while training all BERT-Base parameters. The optimizer architecture and meta-training tasks remain unchanged. This ongoing result is encouraging evidence that dynamic recombination of gradient history can generalize to larger models.

We provide the μLO comparison and BERT-Base progress with details in our global comment. The μLO results were added to Table 1, with the setup in Section 4.1, “Experimental Setup,” and configuration details in Appendix A.3.1, “Baseline protocols.”

Reviewer_mCQEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces an optimizer that learns a weighted sum of past gradients. The proposed optimizer predicts new coefficients at each step based on the gradient history. Gradients are stored in a multi-resolution history allowing efficient memory footprint. The authors also share an empirical study on small models reporting zero-shot improvements on different tasks.

Critical strengths

The paper is clear and describes the method well. The figures help to understand the algorithm and what is at stake. The idea is simple and makes sense. On top of that, the optimizer’s meta-training is cheap, and it can be applied zero-shot to unseen tasks.

Critical weaknesses

All baselines use fixed default hyperparameters. Even though the authors justify that choice, since the learned optimizer effectively chooses its own step size, the comparison is not totally fair and usual optimizers should be tuned.

Results for the learned optimizer applied to all parameters are not shown in the main text, even though this is the setting that would support the claim of a versatile optimizer. These results should be reported alongside Adam in Table 1, and the fact that embeddings and output heads are trained with Adam should be stated clearly in the abstract.

Questions for authors

How does the method compare with tuned baselines? Have you tested the method on larger models? How does its overhead in wall-clock time and memory scale?

3 comments
AuthorsOpenReview ↗

Response to Reviewer mCQE (1/2)

Q1. Task-tuned baselines

“How does the method compare with tuned baselines?”

Thank you for raising this concern. We agree that tuning baseline hyperparameters is necessary for a fair comparison and have added comparisons with task-tuned classical optimizers. Our optimizer achieved the highest overall performance among these optimizers on target tasks unseen during meta-training. The same optimizer was used across all targets after a single meta-training run, without task-specific fine-tuning, while the classical baselines were tuned separately for each task.

Performance across 12 tasksOur optimizer*Task-tuned AdamTask-tuned AdamWTask-tuned SGDMTask-tuned AdEMAMix
Geometric-mean gain over untuned Adam (%) ↑+5.32%+2.91%+2.84%+3.72%+2.18%

* Embeddings and the output head were trained with Adam for language models.

Meta-training our optimizer also required less time than the hyperparameter searches for Adam and AdamW shown below.

Training costOur optimizer: one meta-training runAdam: complete searchAdamW: complete search
B200 GPU-hours0.877.0619.17

Table 2 reports the task-tuned comparisons, discussed in Section 4.2, “Zero-Shot Generalization to Unseen Tasks.” Appendix A.3.1, “Baseline protocols,” lists the hyperparameters and search ranges.

Q2. All-parameter results

“Results for the learned optimizer applied to all parameters are not shown in the main text, even though this is the setting that would support the claim of a versatile optimizer. These results should be reported alongside Adam in Table 1, and the fact that embeddings and output heads are trained with Adam should be stated clearly in the abstract.”

Thank you for raising this concern. We agree that evaluations on all parameters are needed to assess versatility, and the abstract needs clarification. We placed the all-parameter results alongside Adam in Table 1 and clarified the Adam assignment in the abstract. Regarding the all-parameter setting, we also evaluated the language models and found that our optimizer achieved 5.14% lower validation loss than Adam on BERT-Tiny and comparable performance on GPT-Tiny after 250k updates.

Validation loss at 250k updatesOur optimizer: all parametersAdam
BERT-Tiny3.544 ± 0.2013.736 ± 0.043
GPT-Tiny4.199 ± 0.0274.185 ± 0.028

Section 4.2, “Zero-Shot Generalization to Unseen Tasks,” discusses both parameter assignments. Table 3 reports the 250k-update results. Appendix B.1.2, “Longer language trajectories and a larger model,” gives the training protocol.

AuthorsOpenReview ↗

Response to Reviewer mCQE (2/2)

Q3. Generalization to larger models

“Have you tested the method on larger models?”

Thank you for raising the question of generalization to larger models. We agree that larger targets are important for assessing versatility. We tested our optimizer on BERT-Base, a model with 109.5M parameters that was unseen during meta-training, and it successfully generalized without task-specific meta-training when embeddings and the output head were assigned to Adam. The learned optimizer updated 85.1M parameters, approximately 10,000× the size of the largest model used for meta-training, which had 8,450 parameters.

BERT-Base, 100k updatesOur optimizerAdam
Validation loss ↓7.410 ± 0.0067.410 ± 0.006

When applying our optimizer to all BERT-Base parameters, we observed that multiplying the predicted scale (ss) for embeddings by 0.1 or 0.01 avoided the initial divergence and allowed training loss to decrease. We are running experiments to test whether exposing the optimizer to smaller embedding update scales during meta-training can address this instability and will report the results during the discussion period.

Table 3 in Section 4.2, “Zero-Shot Generalization to Unseen Tasks,” reports the BERT-Base comparison. Appendix B.1.2, “Longer language trajectories and a larger model,” gives its training protocol.

Q4. Runtime and memory scaling

“How does its overhead in wall-clock time and memory scale?”

Thank you for raising this concern. We agree that runtime and memory scaling are important for an optimizer to be practical. Both grow with model size, but our optimizer can control these costs through its history and parameter-group configurations.

When memory is limited, our optimizer can use fewer history slots while retaining optimization performance. We found that limiting the number of history slots to six retained competitive performance overall across language, vision, and graph tasks. Increasing the scale factor (rr) averages more gradients per slot, extending history coverage while keeping memory usage unchanged.

Performance across 12 tasksOur optimizer: full span*Our optimizer: six slots*Adam
Geometric-mean gain over Adam (%) ↑+5.32+5.41—

For runtime, increasing parameter-group size can accommodate more parameters without increasing the number of predictions. History management and feature extraction still grow with parameter count, while self-attention and coefficient and scale (ss) prediction depend on the number of groups and slots. Compared with the other learned optimizers, our optimizer required the least total training time across the evaluated tasks.

Total training timeOur optimizer*Celo2VeLO
Across 12 tasks on B20010.62 h10.87 h11.12 h

* Embeddings and the output head were trained with Adam for language models.

Figure 6 in Section 4.3, “Time-Matched Performance and Deployment Cost,” separates model computation from optimizer operations. Table 4 in Section 4.4, “Ablations and Behavioral Analysis,” reports the six-slot comparisons.

AuthorsOpenReview ↗

Updates on all-parameter training and generalization to BERT-Base

Thank you for your patience while we completed the additional evaluations. Following your concern about performance when updating all parameters and generalization to larger models, changing only the scale formulation to an exponential mapping during meta-training enabled us to mitigate the earlier BERT-Base instability. With this small change, our optimizer achieves 18.45% lower validation loss than Adam at update 20,000 while updating all 109.5M BERT-Base parameters, including embeddings and the output head.

BERT-Base at update 20,000Our optimizer, all parametersAdam
Validation loss ↓6.0627.433

The optimizer architecture, approximately 37k parameters, and gradient-history recombination mechanism, and the meta-training tasks remain unchanged (i.e., no language or embedding tasks seen). The optimizer was trained once, within 0.87 GPU-hours, without further task-specific tuning. Training remains ongoing, with no divergence observed so far. This is an promising result for our method generalizing to unseen models at scale.

Additionally, we also completed an additional comparison with μLO. Our optimizer retains the highest overall performance among the compared learned optimizers and task-tuned baselines. The updated learned optimizer comparisons and BERT-Base progress are provided in our global comment.

The μLO results were added to Table 1, with the setup in Section 4.1, “Experimental Setup,” and configuration details in Appendix A.3.1, “Baseline protocols.”

Reviewer_JP5AOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper propose a new type of a learned optimizer that breaks up parameters into groups and updates each of them using a weighted sum of historical multi-step gradient averages (multiple slots with different time scales). The averages (or slots) cover disjoint time windows that evolve as training progresses. A small meta-trained network encodes hand-crafted features of each slot and of the group's current parameters and uses self-attention to produce weighting coefficients and the final scaling factor. This model is trained on several small tasks and is then evaluated zero-shot against untuned Adam, SGD and SGDM on small language models (with embeddings and output heads delegated to Adam), a ViT and GNNs, where it reports modest gains.

Critical strengths
  • This particular approach to meta-learning an optimizer is much cheaper than many other methods. The idea of recombining multi-scale gradient averages with group-shared coefficients is simple and could be effective. The learned optimizer has only 37k parameters and can be meta-learned in just under one GPU-hour. This small parameter footprint and constraints on how the final update weights are produced could be a reason why it can run on unseen architectures and much longer horizons unlike other approaches that use many more meta-learned parameters and are prone to OOD behavior and divergencies.
  • If the gains hold up against properly tuned baselines, this method could be a useful lightweight approach to learned optimizers.
Critical weaknesses
  • The related work discussion is very thin and should be expanded. Without these comparisons it is unclear which parts of the design are new and which are borrowed. Hand-designed multi-timescale optimizers such as AggMo and AdEMAMix are also the natural baselines for isolating the value of learned recombination. A very surface-level search already exposed a large number of papers covering related meta-learning ideas, temporal and group gradient aggregation and more:
  • One potentially critical weakness is that the current experimental data does not yet prove that the observed gains cannot be due to a learning rate effect. Learned overall update scale (even if roughly constant) applied to raw gradient averages without per-coordinate normalization can modify the effective learning rate compared to Adam or SGD that use a single fixed constant learning rate (Adam at 10^{-3} and SGD/SGDM at 0.01 everywhere, with no warmup, decay or weight averaging). Several results can potentially fit this reading including the left pane in Figure 5 that shows different saturation levels which could be a sign of that difference. A single learning rate for Adam or SGD cannot be successfully used for arbitrary models and fixing it for all models can be "unfair", so a natural question is "could a much simpler approach just learn a proper learning rate" based on the number of parameters, or other simple model properties? A good oracle then is to tune learning rate per task since it should be an "upper bound" on a potential performance of such a method. In other words, to rule this explanation out, the authors would need to compare their approach with baselines by also tuning the learning rate (and optionally, possibly using decay schedules or weight averaging).
  • There are some other worrying empirical signs. Currently, all target models are very small. When the learned optimizer updates all language model parameters, it can be seen to lose to Adam. It also appears that the wall-clock and memory overheads are quite substantial.
Questions for authors
  • How does the method compare to Adam and SGDM with tuned learning rates (ideally also with a decay schedule or weight averaging)? This is the main issue that makes me question the core empirical results.
  • How does the effective step size of the learned optimizer evolve during training? For example, what is the update norm and how is the weight distributed between the slots?
  • How much of the gain comes from dynamic gradient recombination? How would a fixed (uniform for example) allocation over slots with a constant scale compare?
6 comments
AuthorsOpenReview ↗

Response to Reviewer JP5A (1/3)

Q1. Task-tuned baselines and learning-rate effects

“How does the method compare to Adam and SGDM with tuned learning rates?”

Thank you for suggesting comparisons with task-tuned baselines. We agree that these comparisons are essential to establish our optimizer’s advantage over classical optimizers. Our optimizer achieved the highest overall performance among the compared optimizers, including task-tuned AdamW and AdEMAMix, on target tasks unseen during meta-training. The same optimizer was used across all targets after a single meta-training run, without task-specific fine-tuning. The classical baselines used hyperparameters tuned separately for each task, without learning-rate decay schedules or weight averaging.

Performance across 12 tasksOur optimizer*Task-tuned AdamTask-tuned AdamWTask-tuned SGDMTask-tuned AdEMAMix
Geometric-mean gain over untuned Adam (%) ↑+5.32%+2.91%+2.84%+3.72%+2.18%

* Embeddings and the output head were trained with Adam for language models.

Meta-training our optimizer required less time than the complete hyperparameter searches for Adam and AdamW.

Training costOur optimizer: one meta-training runAdam: complete searchAdamW: complete search
B200 GPU-hours0.877.0619.17

We revised the manuscript to include the task-tuned comparisons in Table 2, discussed in Section 4.2, “Zero-Shot Generalization to Unseen Tasks.” Appendix A.3.1, “Baseline protocols,” lists the hyperparameters and search ranges.

Q2. Related work and novelty

“The related work discussion is very thin and should be expanded. Without these comparisons it is unclear which parts of the design are new and which are borrowed.”

Thank you for recommending broader comparisons and providing specific references. Your suggestions helped us clarify the differences and benefits of our approach. We added Section 2, “Related Work,” in the revised manuscript to make such distinctions explicit.

Our optimizer differs from these methods in how it combines gradient history and what its network predicts. First, it learns the contribution of each disjoint range of gradient history, whereas AdEMAMix and AggMo use manually selected hyperparameters to control history contribution. Second, it predicts coefficients for recombining gradient averages rather than coordinate updates, as in typical learned optimizers such as the 2016 LSTM optimizer, RNNprop [1], VeLO, Celo2, and the architectures studied in μLO. At the parameter-group level, our optimizer learns to recombine each group's gradient history, whereas Adam-mini shares adaptive learning rates within parameter blocks.

Our optimizer achieved the highest overall performance among the comparisons below on target tasks unseen during meta-training. A single meta-training run of 0.87 GPU-hours produced the optimizer used across all targets without task-specific fine-tuning. We think predicting coefficients helps keep meta-training cost low while supporting this generalization.

Performance across 12 tasksOur optimizer*VeLOCelo2Task-tuned AdEMAMix
Geometric-mean gain over untuned Adam (%) ↑+5.32+4.46−6.01+2.18

* Embeddings and the output head were trained with Adam for language models.

Optimizer-training costOur optimizerCelo2VeLO
Reported compute0.87 B200-hours4.5 GPU-hours≈4,000 TPU-months

We also plan a comparison with μLO before the revision deadline, as it shares our aim of reducing the training cost of learned optimizers. Tables 1 and 2 report the individual task results, discussed in Section 4.2, “Zero-Shot Generalization to Unseen Tasks.”

[1] Lv et al. (2017). Learning Gradient Descent: Better Generalization and Longer Horizons.

AuthorsOpenReview ↗

Response to Reviewer JP5A (2/3)

Q3. Update norms and learned recombination

“How does the effective step size of the learned optimizer evolve during training? For example, what is the update norm and how is the weight distributed between the slots?”

“How much of the gain comes from dynamic gradient recombination? How would a fixed (uniform for example) allocation over slots with a constant scale compare?”

Thank you for asking how the update magnitude evolves and what learned recombination contributes. We agree that these questions call for direct analysis and controls beyond the performance comparisons.

Uniform coefficients and constant scale. We found that replacing learned coefficients with uniform coefficients worsened descent on LeNet, and fixing the scale (ss) to 1 further increased the accumulated training loss. The table reports training-loss AUC (area under the curve) from coefficient replacement through update 200. Loss landscape contours showed that such a change in policy can lead to loss ascent.

Step of coefficient policy replacementUniform coefficients: AUC increase over our optimizer ↓Uniform coefficients and scale (s=1s=1): AUC increase over our optimizer ↓
5+16.4%+37.0%
15+17.0%+71.5%

We also found that learned coefficients can contribute to stable navigation of the loss landscape. The table below shows that learned coefficients avoided numerical divergence on ViT with the scale (ss) fixed to 1.0.

ViTOur optimizer with scale (s=1.0s=1.0)Uniform coefficients with scale (s=1.0s=1.0)
Runs with nonfinite gradients ↓0/33/3 (all diverged)

Update norms and slot coefficients. The update norm analysis shows that stored gradient averages and their coefficients determine the changing update magnitude, rather than the scale (ss) alone. For the selected GAT/PubMed parameter group, update RMS varied 13.1-fold, while the predicted scale (ss) remained nearly constant. Once all nine slots were populated, their coefficients ranged from 7.9% to 14.1%.

Figure 7 in Section 4.4, “Ablations and Behavioral Analysis,” shows the coefficient-replacement trajectories and loss contours.

Appendix B.2.4, “Learned coefficients, scale, and self-attention,” provides both replacement points in Figure 12, the AUC comparison in Table 11, and the fixed-scale ViT results in Table 12.

Figure 11 in Appendix B.2.3, “Temporal coefficients and stored-history content,” aligns slot coefficients, stored gradients, update norms, and the predicted scale.

AuthorsOpenReview ↗

Response to Reviewer JP5A (3/3)

Q4. All-parameter results and larger models

“Currently, all target models are very small. When the learned optimizer updates all language model parameters, it can be seen to lose to Adam.”

Thank you for raising this concern. We agree that evaluations on all parameters and larger models are needed to assess versatility. We found that our optimizer does not lose against Adam even when all language-model parameters were trained for 250k updates. The table below shows that ours clearly wins on BERT-Tiny and performs comparably on GPT-Tiny against untuned Adam.

Validation loss at 250k updatesOur optimizer: all parametersAdam
BERT-Tiny3.544 ± 0.2013.736 ± 0.043
GPT-Tiny4.199 ± 0.0274.185 ± 0.028

We would also like to clarify the results at the original 100k training horizon reported in our submission. Across three seeds, our optimizer achieved lower mean validation loss than Adam on BERT-Tiny at 100k updates as well, with 3.755 ± 0.187 compared with 3.781 ± 0.033.

The same optimizer also generalized to BERT-Base, a larger model with 109.5M parameters that was unseen during meta-training, without task-specific fine-tuning when embeddings and the output head were assigned to Adam. The learned optimizer updated 85.1M parameters, approximately 10,000× the size of the largest model used for meta-training, which had 8,450 parameters.

BERT-Base, 100k updatesOur optimizerAdam
Validation loss ↓7.410 ± 0.0067.410 ± 0.006

We also observed that multiplying the predicted scale (ss) for embeddings by 0.1 or 0.01 avoided the initial divergence when our optimizer trained all BERT-Base parameters and allowed training loss to decrease. We are running experiments to test whether exposing the optimizer to smaller embedding update scales during meta-training can address this instability and will report the results during the discussion period.

We updated Table 1 to compare both parameter assignments with Adam. Table 3 in Section 4.2, “Zero-Shot Generalization to Unseen Tasks,” reports the 250k-update and BERT-Base results. Appendix B.1.2, “Longer language trajectories and a larger model,” gives their training protocols.

Q5. Runtime and memory overhead

“It also appears that the wall-clock and memory overheads are quite substantial.”

Thank you for raising the concern about runtime and memory overhead. We agree that deployment cost matters alongside performance gains. The revised profiles show 52% lower optimizer computation time across all tasks. We changed history management to update stored gradients directly, avoiding repeated reversal, padding, and concatenation. The total training time is lower than for Celo2 and VeLO.

Our optimizer*Earlier profileRevised profileReduction
Optimizer time per step, geometric mean25.24 ms12.19 ms52%
Estimated training time on B200Our optimizer*Celo2VeLO
Total across all 12 tasks10.62 h10.87 h11.12 h
Training time overhead relative to Adam (%) ↓16.819.622.2

When memory is limited, our optimizer can use fewer history slots while retaining optimization performance. We found that limiting the number of history slots to six retained competitive performance overall across language, vision, and graph tasks. If longer history coverage is needed, the scale factor (rr) can be increased while keeping the number of slots and the memory used to store gradient averages unchanged.

Target and metricOur optimizer: full spanOur optimizer: six slotsAdam
BERT-Tiny, embedding & output head to Adam, validation loss ↓3.462 ± 0.0583.456 ± 0.0613.781 ± 0.033
GPT-Tiny, embedding & output head to Adam, validation loss ↓4.280 ± 0.0324.295 ± 0.0304.285 ± 0.026
BERT-Tiny, all parameters, validation loss ↓3.755 ± 0.1873.750 ± 0.1663.781 ± 0.033
GPT-Tiny, all parameters, validation loss ↓4.330 ± 0.0274.363 ± 0.0264.285 ± 0.026
ViT, all parameters, test accuracy (%) ↑31.15 ± 0.2331.36 ± 0.4825.36 ± 3.14
GNN, all parameters, mean test accuracy (%) ↑76.4476.4773.78
Geometric-mean gain over Adam (%) ↑*+5.32+5.41-

* Aggregate results use the language assignment with embeddings and the output head trained with Adam.

Section 4.3, “Time-Matched Performance and Deployment Cost,” presents language trajectories in Figure 5 and runtime measurements in Figure 6. Appendix A.4, “Resource measurements,” provides the Cora trajectories in Figure 8. Table 4 in Section 4.4, “Ablations and Behavioral Analysis,” reports the history-span comparisons. Appendix A.4 also defines the optimizer-state memory measurements.

Comments on the paper

3 comments
AuthorsOpenReview ↗

We thank all reviewers for their careful reading and constructive feedback.

We thank the reviewers for their careful reading and constructive suggestions. We have completed the main revisions requested by the reviewers, as detailed in the replies below. A few additional evaluations require more time and remain in progress. We will report their results and update the manuscript before the revision deadline. We would greatly appreciate any further questions, comments, or suggestions and will address them in the revision during the discussion phase.

We revised the manuscript as follows.

  1. Added learned-optimizer and task-tuned baselines. Tables 1 and 2 now include Celo2, VeLO, and task-tuned Adam, AdamW, momentum SGD, and AdEMAMix. The results are discussed in Section 4.2, “Zero-Shot Generalization to Unseen Tasks.”

  2. Updated runtime measurements and comparisons against elapsed time. We optimized history management to eliminate redundant copying and rearrangement, then repeated the runtime profiling. Section 4.3, “Time-Matched Performance and Deployment Cost,” presents the performance curves against elapsed time and the runtime breakdown in Figures 5 and 6.

  3. Added all-parameter, longer-horizon, and larger-model results. Table 1 reports all-parameter results alongside Adam. Section 4.2, Table 3, adds language-model results over 250k updates and results on BERT-Base, a larger BERT variant with 109.5M parameters. The abstract now states that embeddings and the output head were trained with Adam for the reported language-model gains.

  4. Added ablations and behavioral analysis. Section 4.4, “Ablations and Behavioral Analysis,” Figure 7 and Table 4, examines coefficient replacement and limited history storage. Appendix B.2, “Mechanism evidence and ablations,” Figures 11–12 and Tables 11–13, provides further analysis of update norms, coefficients, the scale factor, and self-attention.

  5. Expanded related work and clarified the method. We expanded the related-work discussion by adding Section 2, clarified architectural motivation in Section 3.3, “Temporal coefficient prediction,” and corrected the meta-loss expression in Section 3.4, “Trajectory-level optimizer training.” We removed the former Section 2.4 and retained the existing description in Appendix C, “Spatio-Temporal Extension,” to make room for the requested explanations. We also clarified experimental details in Section 4.1, “Experimental Setup,” and Appendix A, and corrected notation in the method figures and captions.

AuthorsOpenReview ↗

Updates on the μLO comparison and BERT-Base experiments

We thank the reviewers for their patience as we carry out the additional evaluations. We have completed the μLO comparison and obtained new results demonstrating how our optimizer can address the instabilities encountered when training with all BERT-Base parameters.

U1. Comparison with μLO

Our optimizer retains the highest overall performance after adding μLO. The comparison table below covers all 12 tasks with three seeds.

OptimizerGeometric-mean gain over Adam
Our optimizer*+5.32%
VeLO+4.46%
μLO+0.38%
Celo2*−6.01%

*Our optimizer assigns language embeddings and output heads to Adam. Celo2 assigns these parameters, biases, and other one-dimensional tensors to AdamW.

We added the μLO results to Table 1, its setup to Section 4.1, “Experimental Setup,” and its configuration to Appendix A.3.1, “Baseline protocols.”

U2. BERT-Base stability and generalization

Changing only the scale formulation during meta-training and deployment has enabled stable training while updating all 109.5M BERT-Base parameters, including the embeddings and output head. Our optimizer achieved 18.45% lower validation loss than Adam at update 20,000.

In our earlier response, we reported instability in the embeddings of BERT-Base, and multiplying embedding updates by 0.1 or 0.01 prevented initial divergence. This motivated the exponential scale mapping, which allows the optimizer to predict updates with scales spanning multiple orders of magnitude.

The 37k-parameter optimizer architecture, and gradient-history recombination mechanism remain unchanged. The meta-training tasks also remain unchanged. Neither language nor embedding tasks were included, and BERT-Base was never seen during meta-training. BERT-Base has approximately 13,000 times as many parameters as the largest model used for meta-training. The optimizer was trained once, within 0.87 GPU-hours, using the exponential scale formulation, without further task-specific tuning.

The advantage over Adam has continued to grow during training, with no divergence observed so far. Training is ongoing, and the table below reports validation losses at representative updates. These results demonstrate the potential of learned gradient-history recombination to generalize to substantially larger models.

UpdatesOur optimizer, all BERT-Base parametersAdamValidation loss reduction (%)
1,0006.9667.4466.44
5,0006.4047.43813.91
10,0006.2367.43516.13
20,0006.0627.43318.45
AuthorsOpenReview ↗

Final update and summary of revisions

We thank all the reviewers for their constructive feedback. We summarize the main revisions and follow-up findings below, including the new analysis of learned recombination in Figure 7. We appreciate the reviewers' and ACs' consideration of the revised manuscript and additional results.


1. Tuned baselines and learned-optimizer comparisons (JP5A, mCQE, a88a, gA91)

We added task-tuned Adam, AdamW, momentum SGD and AdEMAMix, together with VeLO, Celo2 and μLO. Our optimizer has the highest aggregate performance among the configurations in Tables 1–2 across 12 tasks, after one 0.87 GPU-hour meta-training run without exposure to the target tasks or further task-specific tuning. This is an aggregate result: task-tuned baselines outperform it on some tasks, including GPT-Tiny and ViT. Our optimizer's hybrid language results use Adam for embeddings and the output head.

Additional checks on Adam's β₂, ε and learning-rate scheduling extend the baseline comparison. β₂ selection improved mean GNN test accuracy, while our optimizer retained its overall advantage. Warmup and cosine decay improved Adam on GPT-Tiny but worsened BERT-Tiny under the tested schedule.

-> §4.2, Tables 1–2; Appendix A.3.1; follow-up response to gA91


2. Contribution of learned recombination and architecture (JP5A, a88a, gA91)

The revised Figure 7 shows that recombination changes both update magnitude and direction even at the same predicted scale. Switching from learned to uniform coefficients across LeNet's parameter groups before update 44 increases subsequent mean training loss through update 160 from 0.676 to 0.693, with the learned scale retained. The coefficient heatmaps show variation across time and parameter groups, including the group closest to the mean.

The appendix adds coefficient-switch comparisons after updates 5 and 15. On ViT, with the scale fixed to 1 for both policies, all three learned-coefficient runs complete training, whereas all three uniform-coefficient runs diverge. Retraining without self-attention also reduces GAT/PubMed accuracy. These controls demonstrate a contribution from learned coefficients beyond the scale alone.

Our follow-up to gA91 distinguishes changing update magnitudes from reproducing Adam's warmup and cooldown, which remains an open question.

-> §4.4, Fig. 7; Appendix B.2.3–B.2.5, Figs. 12–14, Tables 11–13; follow-up response to gA91


3. All-parameter training, longer horizons and larger models (JP5A, mCQE)

All-parameter language results now appear beside Adam in the main text, and the abstract identifies the hybrid assignment. We added 250k-update Tiny-model evaluations and BERT-Base (109.5M parameters). At 250k updates, all-parameter BERT-Tiny reaches 3.544 ± 0.201 validation loss versus 3.736 ± 0.043 for Adam.

The separate scale-modification experiment provides preliminary all-parameter BERT-Base evidence: changing to an exponential mapping during meta-training yielded 18.45% lower validation loss than Adam at update 20,000, without manual attenuation of embedding updates. This optimizer was meta-trained with the modified scale formulation, while keeping the architecture and meta-training tasks unchanged, with no language or embedding tasks included.

-> §4.2, Tables 1 and 3; Appendix B.1.2; global update on μLO and BERT-Base


4. Deployment time and memory (JP5A, mCQE, a88a, gA91)

We removed redundant history operations and added time-matched comparisons and a runtime breakdown. Updated profiling reports time per step of 3.63x Adam on ViT and 3.74x on GNNs. A six-slot configuration reduces history storage while retaining competitive overall performance. Low meta-training cost does not eliminate deployment overhead, which remains a practical limitation.

-> §4.3, Figs. 5–6; Appendix A.4; Table 4


5. Related work, clarity and corrections (JP5A, a88a, gA91)

We added a Related Work section, clarified the architecture and training setup, and corrected the meta-loss expression without changing its meta-gradient. We aligned the appendix GNN results with the main text's validation-based checkpoint selection and removed the former §2.4, whose content was already covered in App. C.

-> §2; §§3.3–3.4, 4.1; Appendix A; Appendix B.2.2, Table 10; Appendix C

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.