- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes ELO, a meta-training recipe for learned optimizers that addresses redundant early-stage computation and insufficient exposure to later optimization regimes. It combines a buffer that resumes training from historical low-loss states with progressive expert supervision, using expert/learned trajectory mixing and separate supervision of update direction and magnitude to stabilize meta-training.
The method is evaluated on two existing learned optimizer architectures and transfers from tiny image-classification meta-training tasks to ImageNet classification and language-model pretraining. The results show improved long-unroll performance and transfer over the corresponding base optimizers. ELO-Celo2 outperforms AdamW and Muon in the reported GPT-2 settings, while remaining below Muon at the 1.3B scale.
Critical strengths
-
The method directly addresses a meaningful meta-training problem. Repeated restarts overexpose learned optimizers to early training while underexposing them to later stagnation or instability. The resume buffer changes the training-state distribution, while progressive expert supervision stabilizes learning. Their complementary roles are supported by component ablations, making the recipe a useful empirical contribution.
-
The evaluation spans both element-wise and matrix-based learned optimizers and substantially larger downstream tasks. Improvements transfer from tiny meta-training tasks to ImageNet and language-model pretraining, and the vision experiments include three-seed results. This breadth supports practical value beyond a single architecture or meta-training task and is the main reason I lean toward acceptance.
Critical weaknesses
-
The PES unbiasedness argument does not fully cover the actual buffer procedure. Preserving the full PES state helps maintain estimator continuity, but the cached state is also selected by an outcome-dependent minimum-loss rule and may have been generated under historical meta-parameters. Appendix A should specify whether the claim concerns a conditional continuation objective with a fixed buffer or the full objective including buffer generation and selection. The distinction between Gaussian-smoothed objectives and the quadratic special case should also remain explicit. This limits the scope of the theoretical claim; it does not by itself establish that the implementation is incorrect.
-
The attribution of compute efficiency and generalization gains is not fully controlled. The reference optimizers and ELO use different base horizon distributions, and Appendix C reports 23.8%–35.6% per-step runtime overhead, so equal outer-step counts do not imply equal actual budgets. In addition, resume probability and expert choice are informed by downstream GPT-2 tasks. The results should distinguish adaptation-assisted transfer from configurations frozen using only the tiny meta-training tasks. These factors make it difficult to quantify the efficiency and generalization gains attributable specifically to the new mechanisms.
Questions for authors
The following clarification requests could materially affect my final assessment:
-
What exact objective is Corollary A.2 unbiased for? How are loss-dependent checkpoint selection, historical meta-parameters, and antithetic PES states handled when resuming? If the claim applies only to a conditional continuation objective with a fixed buffer, please narrow it accordingly.
-
Do the improvements persist when both methods use the same base horizon sampler and are compared at matched actual inner-step computation or wall-clock budgets? Existing logs or a narrowly scoped controlled comparison would help clarify this point.
-
Which downstream tasks were used to choose ELO-specific hyperparameters, and which remained held out? Is repeated-run or seed-variation evidence available for the GPT-2 gains over Muon, to assess their reliability?
4 comments
W2/Q3: Hyperparameter selection, held-out tasks, and reliability of GPT-2 gains over Muon. As described in Appendix D (L923-925), we used the meta-evaluation task (a three-layer MLP) to select the best direction loss weight and shortlist three candidate values for the buffer probability (). We then selected the final on the GPT-2 (124M, 2.5B tokens) language modeling task, which better represents our target downstream workloads. For ELO’s component study, we used the GPT-2 (30M, 0.6B tokens) language modeling task. In each experiment, all other downstream tasks were held out. In terms of distinguishing adaptation-assisted transfer from configurations frozen using only the tiny meta-training tasks, please refer to our response to reviewer 2 (3YCA) W2/Q1.
Thank you for the suggestion to assess variability across seeds. We conducted three runs with different random seeds on the GPT-2 (30M, 0.6B tokens) language modeling task:
| Method | 0.1B | 0.2B | 0.3B | 0.4B | 0.5B | 0.6B |
|---|---|---|---|---|---|---|
| Muon | 4.334 ± 0.002 | 4.144 ± 0.002 | 4.031 ± 0.002 | 4.011 ± 0.003 | 3.888 ± 0.004 | 3.797 ± 0.004 |
| ELO-Celo2 | 4.337 ± 0.006 | 4.093 ± 0.005 | 3.959 ± 0.004 | 3.926 ± 0.005 | 3.796 ± 0.004 | 3.697 ± 0.004 |
We find the validation loss varied little across seeds, and the results are consistent with the conclusion reported in the paper.
Refereces
[1] Vicol et al., Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies. (ICML 2021 outstanding paper)
Dear Reviewer 5rs2,
Thank you again for your thoughtful review. With only about a day remaining in the discussion period, we would appreciate hearing whether our replies have addressed the points you raised and clarified anything you found unclear. If any questions remain, we would be glad to discuss them.
Best regards,
The authors
I raised my rating to 4 since the rebuttal is helpful.
Thank you for taking the time to review our paper. We are pleased that you believe our ELO method addresses a meaningful meta-training problem, and our substantial downstream evaluation supports ELO’s practical value.
W1/Q1: Scope of the PES unbiasedness claim and buffer resumption procedure. We would like to clarify that in the original proof that PES is unbiased in [1], the authors assume that θt=θ ∀t, meaning that their proof of unbiasedness relies on assuming that the learned optimizer parameters do not change at each step. This means that the proof setting of [1] differs from true training setting of truncated gradient estimators, since realizing the speedup requires updating θ after each truncation (multiple times per unroll) and, thus, violating the assumption that θt=θ ∀t. Since our proof also inherits this assumption, we have the same reliance as [1]. We have added a note in Appendix A (p. 15) specifying that, as in [1], our proof does not directly cover the case of computing gradients on different parameters. We have also updated Sec. 2 (p. 3) “Unbiased Truncated Gradient Estimation” to further clarify this point. Finally, we have added statements about the quadratic and Gaussian-smoothed objective more precisely and adjusted our writing in Appendix A.
What exact objective is Corollary A.2 unbiased for? Corollary A.2. is specifically for a quadratic objective. However, as is mentioned in [1] the PES estimator is unbiased for the Gaussian-smoothed objective by definition. We have clarified this in Appendix A.
Antithetic PES states handled when resuming? We include both particles of the antithetic PES state as part of a single buffer state, meaning that we always resume the positive and negative antithetic state together.
How are loss-dependent checkpoint selection handled when resuming? As described in Sec 3.2, loss-dependent checkpoint selection only affects which checkpoint is selected to be pushed onto the buffer. Therefore, it has no effect on the resuming procedure.
How are historical meta-parameters handled when resuming? Historical meta-parameters also do not affect resuming since they are never stored. The stored inner state contains, for both antithetic samples, the PES perturbation accumulators, the optimizer states (e.g., momentum, variance, etc.), and the optimizee checkpoints. We have now clarified the contents of the inner state in Sec 3.2 (L220-222).
W2/Q2: Comparisons with the same base horizon sampler and matched compute budgets. Thank you for the question. We would like to clarify that the “actual inner-step computation,” measured by the total number of inner steps executed during meta-training, is matched across all LOs throughout our work. We have made this clearer in Sec. 4 (L298-299). ELO reallocates much of the redundant training at early inner steps to the long-horizon regime, enabling efficient meta-training over long unrolls while keeping the total number of inner steps unchanged. We would also like to clarify that ELO does not substantially increase the amount of computation required for meta-training. On small tasks, the wall-clock overhead of ELO-LOs over their baseline LOs is mainly due to CPU operations and data movement, and the relative overhead ratio gradually decreases to zero as meta-training scales up, as reported in our response to Reviewer 1 (P1cp), W2/Q2.
We also conducted a comparison on GPT-2 (30M, 0.6B tokens) using log-uniform sampling, as shown below:
With log-uniform sampling, ELO-LOs still substantially outperform their corresponding base LOs. At the same resume probability PB (>0), uniform sampling can yield a longer maximum unroll. For example, at PB=0.8, the maximum unroll length reached was 35,208 steps with uniform sampling and 24,167 steps with log-uniform sampling. Since ELO’s progressive expert supervision ensures stable meta-training over long unrolls, we used uniform sampling to train ELO-LOs in all of our experiments.