acceptodds
Under review as a conference paper at ICLR 2027

Efficient Long-Horizon Learning for Learned Optimization

Abstract

Learned optimization aims to improve upon hand-designed optimizers (e.g., Adam and Muon) by meta-learning small neural network optimizers over a distribution of tasks. While recent work has greatly advanced the architectural design and inductive biases of learned optimizers (LOs), their meta-training remains biased toward short-unroll learning on particular tasks, resulting in redundant computation and leaving LOs often unable to compete with hand-designed optimizers. We introduce Efficient Long-hOrizon (ELO) learning, an efficient meta-training algo- rithm that (1) reallocates wasted meta-training compute to longer failure regimes, achieving efficient long-horizon learning, and (2) enforces decoupled progressive expert supervision, providing stable meta-learning signals that additionally improve the generalization of LOs. Our empirical study evaluates ELO for meta-training both element-wise and matrix-based LOs. Across downstream language modeling (GPT-2 and GPT-3 on FineWeb) and image classification (ViT-B/16, ResNet-50 on ImageNet-1K) tasks, ELO substantially improves the long-unroll performance and out-of-distribution generalization of the base LOs. In particular, ELO-Celo2 outperforms well-tuned AdamW and Muon on GPT-2 (30M and 124M) language pretraining. Notably, all ELO baselines require less than 7 H100 GPU-hours for meta-training.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
learned optimization, meta learning, buffer, imitation learning

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_5rs2OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ELO, a meta-training recipe for learned optimizers that addresses redundant early-stage computation and insufficient exposure to later optimization regimes. It combines a buffer that resumes training from historical low-loss states with progressive expert supervision, using expert/learned trajectory mixing and separate supervision of update direction and magnitude to stabilize meta-training.

The method is evaluated on two existing learned optimizer architectures and transfers from tiny image-classification meta-training tasks to ImageNet classification and language-model pretraining. The results show improved long-unroll performance and transfer over the corresponding base optimizers. ELO-Celo2 outperforms AdamW and Muon in the reported GPT-2 settings, while remaining below Muon at the 1.3B scale.

Critical strengths
  1. The method directly addresses a meaningful meta-training problem. Repeated restarts overexpose learned optimizers to early training while underexposing them to later stagnation or instability. The resume buffer changes the training-state distribution, while progressive expert supervision stabilizes learning. Their complementary roles are supported by component ablations, making the recipe a useful empirical contribution.

  2. The evaluation spans both element-wise and matrix-based learned optimizers and substantially larger downstream tasks. Improvements transfer from tiny meta-training tasks to ImageNet and language-model pretraining, and the vision experiments include three-seed results. This breadth supports practical value beyond a single architecture or meta-training task and is the main reason I lean toward acceptance.

Critical weaknesses
  1. The PES unbiasedness argument does not fully cover the actual buffer procedure. Preserving the full PES state helps maintain estimator continuity, but the cached state is also selected by an outcome-dependent minimum-loss rule and may have been generated under historical meta-parameters. Appendix A should specify whether the claim concerns a conditional continuation objective with a fixed buffer or the full objective including buffer generation and selection. The distinction between Gaussian-smoothed objectives and the quadratic special case should also remain explicit. This limits the scope of the theoretical claim; it does not by itself establish that the implementation is incorrect.

  2. The attribution of compute efficiency and generalization gains is not fully controlled. The reference optimizers and ELO use different base horizon distributions, and Appendix C reports 23.8%–35.6% per-step runtime overhead, so equal outer-step counts do not imply equal actual budgets. In addition, resume probability and expert choice are informed by downstream GPT-2 tasks. The results should distinguish adaptation-assisted transfer from configurations frozen using only the tiny meta-training tasks. These factors make it difficult to quantify the efficiency and generalization gains attributable specifically to the new mechanisms.

Questions for authors

The following clarification requests could materially affect my final assessment:

  1. What exact objective is Corollary A.2 unbiased for? How are loss-dependent checkpoint selection, historical meta-parameters, and antithetic PES states handled when resuming? If the claim applies only to a conditional continuation objective with a fixed buffer, please narrow it accordingly.

  2. Do the improvements persist when both methods use the same base horizon sampler and are compared at matched actual inner-step computation or wall-clock budgets? Existing logs or a narrowly scoped controlled comparison would help clarify this point.

  3. Which downstream tasks were used to choose ELO-specific hyperparameters, and which remained held out? Is repeated-run or seed-variation evidence available for the GPT-2 gains over Muon, to assess their reliability?

4 comments

Thank you for taking the time to review our paper. We are pleased that you believe our ELO method addresses a meaningful meta-training problem, and our substantial downstream evaluation supports ELO’s practical value.

W1/Q1: Scope of the PES unbiasedness claim and buffer resumption procedure. We would like to clarify that in the original proof that PES is unbiased in [1], the authors assume that θt=θ ∀t\theta_t = \theta \ \forall t, meaning that their proof of unbiasedness relies on assuming that the learned optimizer parameters do not change at each step. This means that the proof setting of [1] differs from true training setting of truncated gradient estimators, since realizing the speedup requires updating θ\theta after each truncation (multiple times per unroll) and, thus, violating the assumption that θt=θ ∀t\theta_t = \theta \ \forall t. Since our proof also inherits this assumption, we have the same reliance as [1]. We have added a note in Appendix A (p. 15) specifying that, as in [1], our proof does not directly cover the case of computing gradients on different parameters. We have also updated Sec. 2 (p. 3) “Unbiased Truncated Gradient Estimation” to further clarify this point. Finally, we have added statements about the quadratic and Gaussian-smoothed objective more precisely and adjusted our writing in Appendix A.

What exact objective is Corollary A.2 unbiased for? Corollary A.2. is specifically for a quadratic objective. However, as is mentioned in [1] the PES estimator is unbiased for the Gaussian-smoothed objective by definition. We have clarified this in Appendix A.

Antithetic PES states handled when resuming? We include both particles of the antithetic PES state as part of a single buffer state, meaning that we always resume the positive and negative antithetic state together.

How are loss-dependent checkpoint selection handled when resuming? As described in Sec 3.2, loss-dependent checkpoint selection only affects which checkpoint is selected to be pushed onto the buffer. Therefore, it has no effect on the resuming procedure.

How are historical meta-parameters handled when resuming? Historical meta-parameters also do not affect resuming since they are never stored. The stored inner state contains, for both antithetic samples, the PES perturbation accumulators, the optimizer states (e.g., momentum, variance, etc.), and the optimizee checkpoints. We have now clarified the contents of the inner state in Sec 3.2 (L220-222).

W2/Q2: Comparisons with the same base horizon sampler and matched compute budgets. Thank you for the question. We would like to clarify that the “actual inner-step computation,” measured by the total number of inner steps executed during meta-training, is matched across all LOs throughout our work. We have made this clearer in Sec. 4 (L298-299). ELO reallocates much of the redundant training at early inner steps to the long-horizon regime, enabling efficient meta-training over long unrolls while keeping the total number of inner steps unchanged. We would also like to clarify that ELO does not substantially increase the amount of computation required for meta-training. On small tasks, the wall-clock overhead of ELO-LOs over their baseline LOs is mainly due to CPU operations and data movement, and the relative overhead ratio gradually decreases to zero as meta-training scales up, as reported in our response to Reviewer 1 (P1cp), W2/Q2.

We also conducted a comparison on GPT-2 (30M, 0.6B tokens) using log-uniform sampling, as shown below:

Method0.1B0.2B0.3B0.4B0.5B0.6B
small_fc5.1294.8254.7654.7144.5814.494
ELO-small_fc (log-uniform)4.3504.1093.9863.9603.8393.747
ELO-small_fc (uniform)4.3734.1214.0013.9733.8383.736
Celo24.5124.2094.0534.0083.8783.781
ELO-Celo2 (log-uniform)4.3794.1243.9873.9473.8133.716
ELO-Celo2 (uniform)4.3334.0933.9573.9233.7943.693

With log-uniform sampling, ELO-LOs still substantially outperform their corresponding base LOs. At the same resume probability PBP_B (>0), uniform sampling can yield a longer maximum unroll. For example, at PB=0.8P_B=0.8, the maximum unroll length reached was 35,208 steps with uniform sampling and 24,167 steps with log-uniform sampling. Since ELO’s progressive expert supervision ensures stable meta-training over long unrolls, we used uniform sampling to train ELO-LOs in all of our experiments.

W2/Q3: Hyperparameter selection, held-out tasks, and reliability of GPT-2 gains over Muon. As described in Appendix D (L923-925), we used the meta-evaluation task (a three-layer MLP) to select the best direction loss weight and shortlist three candidate values for the buffer probability (PBP_B). We then selected the final PBP_B on the GPT-2 (124M, 2.5B tokens) language modeling task, which better represents our target downstream workloads. For ELO’s component study, we used the GPT-2 (30M, 0.6B tokens) language modeling task. In each experiment, all other downstream tasks were held out. In terms of distinguishing adaptation-assisted transfer from configurations frozen using only the tiny meta-training tasks, please refer to our response to reviewer 2 (3YCA) W2/Q1.

Thank you for the suggestion to assess variability across seeds. We conducted three runs with different random seeds on the GPT-2 (30M, 0.6B tokens) language modeling task:

Method0.1B0.2B0.3B0.4B0.5B0.6B
Muon4.334 ± 0.0024.144 ± 0.0024.031 ± 0.0024.011 ± 0.0033.888 ± 0.0043.797 ± 0.004
ELO-Celo24.337 ± 0.0064.093 ± 0.0053.959 ± 0.0043.926 ± 0.0053.796 ± 0.0043.697 ± 0.004

We find the validation loss varied little across seeds, and the results are consistent with the conclusion reported in the paper.


Refereces

[1] Vicol et al., Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies. (ICML 2021 outstanding paper)

Dear Reviewer 5rs2,

Thank you again for your thoughtful review. With only about a day remaining in the discussion period, we would appreciate hearing whether our replies have addressed the points you raised and clarified anything you found unclear. If any questions remain, we would be glad to discuss them.

Best regards,
The authors

I raised my rating to 4 since the rebuttal is helpful.

Reviewer_3YCAOpenReview ↗
Rating
4(+1)
Confidence
3(+1)
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

The paper addresses the inefficiency of meta-training learned optimizers (LOs), where random restarts bias compute toward already well-learned early-unroll regions. The proposed method, ELO, combines a resume buffer that restarts unrolls from the lowest-loss state of previous trajectories, reallocating compute toward later failure-prone regimes, and progressive expert supervision that fuses expert and LO trajectories under an annealed coefficient αt\alpha_t and supervises the LO with a decoupled direction and magnitude imitation objective. Experiments meta-train two LO architectures (small_fc and Celo2) on several tasks.

Critical strengths
  1. A well-motivated and practically consequential recipe. The paper identifies a concrete and plausible source of waste in standard meta-training of learned optimizers and addresses it with a simple, inexpensive mechanism. Learned optimizers that become competitive with carefully tuned hand-designed optimizers on realistic pretraining workloads remain rare, so a recipe that achieves this at low meta-training cost is of clear interest to the community.
  2. Design analysis that supports the attribution of gains. The two components are shown to address distinct failure modes, and the analysis of how the expert choice interacts with the optimizer architecture offers mechanistic insight beyond a headline number. This makes the claim that the improvements come from the proposed components substantially more credible than the main results alone.
Critical weaknesses
  1. The vision evaluation omits SGDM, the canonical hand-designed optimizer for ResNet-50 on ImageNet-1K. AdamW is the only hand-designed baseline in Table 1, and the justification in Appendix D that AdamW "remains the dominant trend for vision tasks" holds for ViT but not for ResNet-50. The reported margin of ELO-Celo2 over AdamW on ResNet-50 is small (76.11 versus 75.81), while standard SGDM recipes are known to reach comparable or higher top-1 accuracy at a similar epoch budget, and the paper itself treats SGDM as a meaningful optimizer when choosing experts (Figure 4). Without this comparison, the claim that ELO-trained optimizers outperform hand-designed optimizers on vision tasks is only partially supported, and adding it within the existing protocol is straightforward.
  2. Hyperparameter selection partly uses the reported evaluation benchmark itself. The buffer probability PB=0.8P_B = 0.8 is selected using the validation loss of GPT-2 (124M) pretraining (Sec. 4.3 and Figure 8, bottom right), which is also one of the main reported benchmarks. This weakens the framing of the GPT-2 results as out-of-distribution generalization, since a key meta-training hyperparameter was tuned on the target task distribution. The concern is fixable by selecting PBP_B on the meta-evaluation task alone or reporting sensitivity on the downstream tasks.
Questions for authors
  1. Is PB=0.8P_B = 0.8 used for all reported downstream tasks, and what are the downstream results when PBP_B is selected using only the ImageNet-1K (32x32) meta-evaluation loss (Figure 8 suggests PB=0.1P_B = 0.1 is preferred there)? This bears directly on whether the GPT-2 numbers reflect generalization or benchmark-specific tuning.
  2. For learned optimizers, what quantity is swept as the "learning rate" in the downstream tuning (e.g., λ1,λ2\lambda_1, \lambda_2 in Eq. 1 or an outer step-size multiplier), and did AdamW and Muon receive comparable tuning budgets at each model scale?
  3. Could the authors resolve the symbol collision between the main text and Appendix A? In Sec. 2, NN denotes the full inner-problem horizon and PP the number of PES particles, while Appendix A states that "TT denotes the unroll horizon and NN the number of Monte Carlo samples" and writes θ∈RP\theta \in \mathbb{R}^P with PP as the parameter dimension. This overlap makes the unbiasedness argument harder to check against the main-text definitions.
  4. Algorithm 1 lists buffer capacity CC as a hyperparameter, but the paper fixes it to one with only a qualitative justification ("to avoid stale states", Sec. 3.2) and the sensitivity study in Sec. 4.3 covers only PBP_B and λ\lambda. Do the authors have any evidence, even at small scale, that C=1C = 1 is preferable to keeping a few resume states? Capacity is a genuine design axis of the resume buffer since it trades resume-point diversity against state staleness.
3 comments

Thank you for taking the time to review our paper. We appreciate your recognition of ELO’s motivation and practical significance, and of the mechanistic insight provided by our analysis.

W1: Comparison with SGDM on vision tasks. Thank you for the constructive feedback. We conduct experiments on vision tasks with SGDM [1], sweeping the learning rate over [10−2,3.16×10−2,10−1,3.16×10−1,1.0,3.16,10.0]{10^{-2}, 3.16\times10^{-2}, 10^{-1}, 3.16\times10^{-1}, 1.0, 3.16, 10.0}] and using the same weight decay as for AdamW. We select the best learning rate for each experiment and repeat with three random seeds to assess the robustness of the results. As shown in the table below, SGDM performs well on ResNet-50, slightly below ELO-Celo2, but performs substantially worse than the other methods on ViT.

OptimizerRN-50 IN-1KRN-50 ReaLRN-50 V2ViT-B/16 IN-1KViT-B/16 ReaLViT-B/16 V2Avg.
SGDM76.000 ± 0.15682.413 ± 0.17863.450 ± 0.22169.891 ± 0.06577.395 ± 0.15557.147 ± 0.16871.049 ± 0.050
AdamW75.808 ± 0.07582.173 ± 0.02163.393 ± 0.15074.032 ± 0.11180.720 ± 0.18461.297 ± 0.27272.904 ± 0.080
ELO-Celo276.110 ± 0.09282.702 ± 0.02464.033 ± 0.38976.417 ± 0.27782.258 ± 0.31163.893 ± 0.32374.236 ± 0.171

We have added the results to Table 1 (p. 9).

W2/Q1: Buffer probability selection and downstream generalization. We would like to respectfully note that our meta-training tasks do not include any language-modeling tasks, and our downstream evaluation tasks lie well outside the meta-training distribution. Evaluating transfer across these differences lies at the heart of assessing the meta-generalization of ELO-LOs [2,3,4,5].

We would also like to clarify that, our goal in the ELO hyperparameter experiments is to identify settings, within an acceptable budget, that allow ELO-LOs to perform well across all downstream tasks, rather than overfitting the meta-evaluation task (a three-layer MLP). We found that the best setting on the meta-evaluation task did not transfer well to downstream tasks, e.g., the best PBP_B (PB=0.1P_B=0.1) on the meta-evaluation task underperforms PB=0.8P_B=0.8 on the small GPT-2 (30M, 0.6B tokens) task:

Method0.1B0.2B0.3B0.4B0.5B0.6B
ELO-small_fc (PB=0.1)(P_B=0.1)4.5134.2434.1034.0613.9273.823
ELO-small_fc (PB=0.8)(P_B=0.8)4.3734.1214.0013.9733.8383.736

We therefore only used this task as an initial filter to shortlist candidate settings and made the final selection on GPT-2 (124M, 2.5B tokens), a task more representative of the downstream workloads to which we sought to generalize. However, we agree that using GPT-2 (124M, 2.5B tokens) for this final selection could be misleading, since it is also one of our reported downstream benchmarks. We therefore reevaluate our choice on a held-out language modeling task using GPT-2 (51M, 1.03B tokens) on FineWeb with a different set of validation shards. As shown below:

Method0.1B0.2B0.3B0.4B0.5B0.6B0.7B0.8B0.9B1.0B1.03B
ELO-small_fc (PB=0.1)(P_B=0.1)4.5524.1794.0433.9623.8823.8193.7643.7163.6913.6543.651
ELO-small_fc (PB=0.5)(P_B=0.5)4.4424.0663.9313.8663.7983.7423.6893.6373.6043.5533.549
ELO-small_fc (PB=0.8)(P_B=0.8)4.4004.0573.9123.8373.7633.7063.6563.6093.5823.5373.533

Comparing the same three candidates shortlisted through meta-evaluation leads to the same conclusion as in our paper: PB=0.8P_B=0.8 performs best among the three. We have updated this new evaluation to Appendix D (p. 20).

Q2: Definition of the downstream learning rate for LOs and comparability of tuning budgets. The downstream “learning rate” for LOs refers to the update multiplier λ1\lambda_1 in Eq. 1 (p. 2). For all optimizers, including LOs, AdamW, and Muon, we sweep seven learning rate candidates evenly spaced in log⁡10\log_{10} space, using the optimizer-specific ranges reported in Appendix D (p. 17). Specifically, for language modeling, we search over [10−5,10−2][10^{-5},10^{-2}] for LOs and AdamW, and [10−4,10−1][10^{-4},10^{-1}] for Muon. This gives each optimizer the same number of learning rate candidates while accommodating differences in their update scales.

Q3: Consistency of notation between the main text and Appendix A. Thank you for pointing this out. We have updated the notations in Appendix A (p. 15) for standardization: TT denotes the total number of meta-training steps, NN denotes the full inner-problem horizon, and PP denotes the number of PES particles (i.e., the number of Monte Carlo samples).

Q4: Buffer capacity sensitivity. We conducted a joint search over buffer capacity and resume probability on a simple : training a three-layer MLP with width 128 on ImageNet at 32×3232\times32 resolution. For each configuration, we meta-trained ELO-small_fc for 5K outer steps with a batch size of 4096. We then evaluated the ELO-small_fc on the same task for 10K inner steps. The losses on the validation set are reported below:

Buffer size(PB=0.9)(P_B=0.9)(PB=0.7)(P_B=0.7)(PB=0.5)(P_B=0.5)(PB=0.3)(P_B=0.3)(PB=0.1)(P_B=0.1)Avg.
15.3285.2645.2625.2345.2705.272
25.2865.2785.3215.2915.2635.288
35.2875.4015.2385.2555.2545.287
45.3415.2725.2965.2695.2895.293

Setting the buffer capacity to one generally yielded the best performance, which we hypothesize was partly due to reduced state staleness. We therefore fixed the buffer capacity at one in subsequent experiments to simplify the hyperparameter search.


References

[1] Sutskever et al., On the Importance of Initialization and Momentum in Deep Learning, 2013.

[2] Metz et al., VeLO: Training Versatile Learned Optimizers by Scaling Up, 2022.

[3] Moudgil et al., Celo: Training Versatile Learned Optimizers on a Compute Diet, 2025.

[4] Moudgil et al., Celo2: Towards Learned Optimization Free Lunch, 2026.

[5] Thérien et al., μLO: Compute-Efficient Meta-Generalization of Learned Optimizers, 2024.

The rebuttal is helpful, I will raise my rating to 4.

Reviewer_P1cpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

ELO is a meta-training method for learned optimizers. It resumes optimization trajectories from saved checkpoints to increase training on later stages, while gradually shifting from imitating an expert optimizer to minimizing the task loss. The method is applied to small_fc and Celo2 and evaluated on image classification and language-model pretraining, showing improvements over their corresponding baselines.

Critical strengths

1.Resuming intermediate states reduces repeated training on early optimization stages. Together with gradually reduced expert guidance, this provides a practical way to train for longer horizons without changing the optimizer architecture.

  1. The approach improves two different learned optimizers and transfers from small meta-training tasks to much larger models. This somehow supports its usefulness across architectures and workloads.
Critical weaknesses
  1. The vision evaluation lacks a directly relevant Muon baseline. Muon orthogonalizes momentum-based matrix updates, sharing a key operation with Celo2, and is also used as the expert for ELO-Celo2. Its absence from Table 1, particularly for ViT, leaves unclear whether the learned optimizer offers a practical advantage over this closely related hand-designed alternative.

  2. The claimed training-time benefit remains unclear. Figure 3 compares progress by the number of meta-training steps, while Table 2 shows that ELO takes more time per step. The paper therefore does not directly establish whether ELO reaches comparable performance faster after including its additional overhead. This matters because computational efficiency is a central claim.

Questions for authors
  1. Is there a specific technical reason for excluding Muon-related methods from Table 1, particularly for ViT? Are any existing comparison results available?
  2. Figure 3 shows convergence against meta-training steps. What is held constant in the “same compute budget” comparison, and how is the additional overhead reported in Table 2 accounted for?
4 comments

Thank you for taking the time to review our paper and for sharing detailed and constructive feedback. We appreciate that you agree that our ELO algorithm is generalizable and has great potential for meta-training LOs over longer horizons.

W1/Q1: Comparison with Muon on vision tasks. Thank you for the suggestion. We agree that including the performance of a Muon baseline for ResNet and ViT tasks is important. During the rebuttal, we have run additional experiments to add Muon as a baseline for this comparison. We sweep the learning rate over [{10−4,3.16×10−4,10−3,3.16×10−3,10−2,3.16×10−2,10−1}\{10^{-4}, 3.16\times10^{-4}, 10^{-3}, 3.16\times10^{-3}, 10^{-2}, 3.16\times10^{-2}, 10^{-1}\}] and use the same weight decay as for AdamW. We select the best learning rate for each experiment and repeat with three random seeds to assess the robustness of the results. As shown in the table below, we find that ELO-Celo2 still performs best of all methods in general, with notable improvements over their baseline LOs:

OptimizerRN-50 IN-1KRN-50 ReaLRN-50 V2ViT-B/16 IN-1KViT-B/16 ReaLViT-B/16 V2Avg.
AdamW75.808 ± 0.07582.173 ± 0.02163.393 ± 0.15074.032 ± 0.11180.720 ± 0.18461.297 ± 0.27272.904 ± 0.080
Muon75.991 ± 0.10082.818 ± 0.12063.810 ± 0.17376.379 ± 0.05082.426 ± 0.08163.423 ± 0.24774.141 ± 0.036
Celo275.763 ± 0.11082.337 ± 0.06663.393 ± 0.26775.860 ± 0.04681.737 ± 0.05262.763 ± 0.09273.642 ± 0.034
ELO-Celo276.110 ± 0.09282.702 ± 0.02464.033 ± 0.38976.417 ± 0.27782.258 ± 0.31163.893 ± 0.32374.236 ± 0.171

We have added the results to Table 1 (p. 9) in the paper.

W2/Q2: Clarification regarding meta-training efficiency. Thank you for pointing this out. We define the compute budget as the total number of inner steps executed by an LO throughout meta-training. We have edited Sec. 4 (L298-299) to make this clearer. While ELO introduces some additional computation, such as computing expert updates, its wall-clock overhead relative to the corresponding baseline LOs is primarily due to CPU operations and data movement, as discussed in Appendix C (L880-886). During these operations, the GPU is idle. It is important to note that, as the scale of the meta-training tasks increases, GPU computation accounts for a larger proportion of the total runtime and the overhead of CPU operations does not grow proportionally. To examine this effect, we use a single CIFAR-10 classification task using an MLP, and vary the input resolution, width and depth, and batch size. In that order, the settings are (8×8,32,1,64)(8\times8, 32, 1, 64) for small, (16×16,64,3,128)(16\times16, 64, 3, 128) for medium, and (32×32,512,3,512)(32\times32, 512, 3, 512) for large.

Meta-training scaleSmallMediumLarge
OptimizerTime/step (ms)RatioTime/step (ms)RatioTime/step (ms)Ratio
small_fc36.71.00107.91.002,300.41.00
ELO-small_fc44.51.21121.31.122,338.81.02
Celo274.11.00176.11.002,528.91.00
ELO-Celo280.11.08187.01.062,594.61.03

This shows that the overhead of the CPU operations is only significant for small-scale tasks, and becomes trivial as meta-training scales up, which also suggests that ELO is promising for large-scale meta-training. We have added this comparison to Appendix C (p. 17).

Dear Reviewer P1cp,

Thank you again for your constructive feedback. With about a day left in the discussion period, we wanted to kindly follow up and ask whether our response has addressed your concerns. We would be happy to discuss any further questions.

Best regards, The authors

Reviewer_P1cpOpenReview ↗

Reply to the authors

Thank you for the detailed response and additional experiments:

W1: Note that ELO-Celo2 and Muon are within about one standard deviation on most columns, and Muon leads on IN-ReaL, so “on par with Muon” seems more accurate than “best overall”.

W2: Mostly addressed. Defining the budget as the total number of inner steps answers my question, and the scaling study on overhead is helpful.

However, I still have two questions that need to be further clarification but no more experiments are needed:

  1. Figure 4 (top right) suggests most of the downstream gain comes from expert supervision, with the buffer adding a smaller improvement. Do the authors agree, and might this be reflected in how the contributions are framed?

  2. Which parameter groups was Muon applied to, and did Muon and SGDM receive the same augmentation sweep as the learned optimizers?

Overall, my main concerns are addressed, and I am raising my score from 2 to 3.

Thank you for the additional comments and questions.

  • Comparison with Muon on vision tasks. We agree that “on par with Muon” best describes ELO-Celo2’s performance on the vision benchmarks. We will revise this claim in the next version.
  • Relative contributions in Figure 4. We kindly note that Figure 4 does not directly compare the independent contributions of the two components. For small_fc, using the buffer alone can cause meta-training to collapse, as shown in the top left panel, and the top right panel only indicates the buffer’s additional gain when expert supervision is already in place. Figure 4 illustrates how the two components complement each other and together enable efficient meta-training over long unrolls.
  • Muon parameter groups and augmentation sweep. Following Keller’s implementation [1], we applied Muon to hidden layer parameters with more than one tensor dimension in both ResNet and ViT. We used AdamW for the input layer, output head, embedding layers, and all scale and bias parameters. Given our limited compute budget, we only swept two augmentation settings: medium augmentation (worked best for AdamW) and strong augmentation (worked best for the LOs). Both Muon and SGDM performed best under medium augmentation, and their performance dropped substantially under strong augmentation.

Thank you again for your thoughtful assessment of our response and for raising your score. We hope our clarifications address your remaining questions.


Refereces

[1] Keller et al., Muon: An optimizer for hidden layers in neural networks. 2024

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you for the time you have devoted to our paper and for your detailed feedback. We have posted our point-by-point responses and updated the PDF accordingly as well. We are sorry for posting our responses late in the discussion period. Completing the additional experiments with the compute available took longer than we had hoped.

We summarize the main additions below.

  • Compute efficiency. ELO retained its advantage over its base optimizers on GPT-2 (30M) when both used the same log-uniform horizon sampler and inner-step budget. We also measured wall-clock time on CIFAR-10. From the smallest to the largest meta-training task, ELO’s overhead decreased from 21% to 2% for small_fc and from 8% to 3% for Celo2.
  • Hyperparameter selection and transfer. We clarified the definition of meta-generalization for ELO-LOs. A separate GPT-2 (51M, 1.03B tokens) experiment favored the same resume probability, PB=0.8P_B=0.8.
  • Baselines and robustness. We added SGDM and Muon comparisons on ResNet-50 and ViT-B/16. We also repeated the comparison with Muon on GPT-2 (30M, 0.6B tokens) across three seeds and reached the same conclusion as in the paper, with little variation in validation loss across seeds.
  • PES and buffer design. We clarified the contents of the inner states, the objective covered by the PES analysis, and its assumptions about meta-parameter updates. We also reported a joint sweep of buffer capacity and resume probability.

If you could look over our responses before the discussion closes, we would appreciate hearing whether they address your concerns. We would also be happy to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.