acceptodds
Withdrawn from ICLR 2027

POP: Prior-Fitted First-Order Optimization Policies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Adaptive first-order optimizers rely on fixed, hand-designed mechanisms for incorporating optimization history, limiting how flexibly they can adapt to a new objective. We introduce POP, a prior-fitted first-order optimizer whose learned action is deliberately restricted to trajectory-conditioned coordinate-wise step sizes along the descent direction. POP is meta-trained via reinforcement learning on millions of synthetic optimization problems sampled from a task-independent compositional prior with Gaussian process components, using scale-invariant policy inputs to support transfer across objectives. Despite training only on low-dimensional synthetic problems, POP generalizes to longer horizons, higher-dimensional objectives, and unseen problem distributions. On a benchmark of 43 diverse optimization functions, POP significantly outperforms per-function tuned gradient-based baselines and transfers zero-shot to practical optimization tasks on real-world datasets. Our work provides a proof of concept that transferable optimization behavior can be learned through trajectory-conditioned step-size adaptation alone, and that such behavior can emerge from meta-training exclusively on objectives sampled from a synthetic generative prior.

Primary area:
optimization
Keywords:
Optimization, Meta-learning, Prior-Fitted

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_u34sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes POP, a prior-fitted first-order optimizer that learns to adapt step sizes for continuous optimization. The authors formulate the optimization process as a sequential decision-making problem and train a Transformer-based policy using PPO on synthetic functions sampled from a compositional prior. The learned policy predicts coordinate-wise learning rates based on past optimization steps while following the negative gradient direction. POP is trained on two-dimensional synthetic functions and tested on 43 benchmark functions, higher-dimensional problems, and real-world optimization tasks to examine its generalization to unseen problems.

Critical strengths
  1. The paper addresses an important challenge in learning to optimize: the generalization of learned optimizers to unseen optimization tasks. Training the optimizer exclusively on synthetic functions, without relying on downstream-specific training data, provides an interesting perspective on learning transferable optimization strategies.

  2. The experimental evaluation provides encouraging evidence of zero-shot generalization, particularly from two-dimensional training problems to 43 benchmark functions and selected real-world optimization tasks. The inclusion of comparisons with conventional optimizers and ablation studies on the synthetic prior further supports the empirical findings.

Critical weaknesses
  1. The novelty of the method is limited. Learning optimization strategies or adaptive learning rates from past optimization steps has already been studied (e.g., Wichrowska et al., 2017; Shu et al., 2020). Although POP only learns positive step sizes for each coordinate, it is not clear how this design improves upon existing methods.

  2. The experiments mainly focus on low-dimensional problems. The model is trained only on 2D functions, and the higher-dimensional tests only go up to 32D. It would be useful to see how POP performs on larger problems, especially when there are strong interactions between variables.

Questions for authors
  1. What are the main differences between POP and existing learned optimizers (e.g., Wichrowska et al., 2017; Shu et al., 2020)? What are the specific benefits of the proposed design?

  2. Why is the comparison between POP and Joint POP limited to 2D problems? Would the same conclusion hold for higher-dimensional, non-separable functions?

  3. Why are the higher-dimensional experiments limited to 32D? How would POP perform on 100D or 1000D problems?

  4. In Eq. (4), POP only adjusts the step size along the negative gradient direction. How does it escape local minima as shown in Figure 1? If the gradient becomes zero, changing the step size will not help. Is POP able to escape local minima simply because it takes larger steps before getting stuck?

Reviewer_mKfnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes POP, a first-order optimizer meta-trained with reinforcement learning. POP encodes the optimization history of each coordinate with a Transformer, predicts a positive step size for each coordinate, and applies the gradient update in a normalized space. All meta-training tasks are sampled from a synthetic prior over two-dimensional functions that combines Gaussian process, quadratic, and saddle components. POP outperforms well tuned GD, Adam, etc. on most synthetic and VLSE settings and is on par with GP-based BO on VLSE.

Critical strengths
  1. The research question is clear and interesting: how much optimization ability can be obtained by learning only the magnitude of each update.

  2. POP achieves competitive results on many OOD tasks. On the 43 VLSE functions, POP is applied without adapting its weights to the downstream tasks, whereas the main gradient-based baselines tune their learning rates per function. POP still attains good average performance and ranks in this setting.

  3. The ablations are thorough, covering the reward design, coordinate-wise independence, and the prior components. The paper also includes a broader comparison against other solvers and wall-clock measurements.

Critical weaknesses
  1. The small-scale experiments are thorough, but the paper does not evaluate POP on practical models such as CNN or Transformer training. It also does not compare against SOTA hand-designed and learned optimizers, e.g., Muon [1] and Celo2 [2].

  2. The meta-training cost is not low relative to what POP delivers. POP is meta-trained for about 56 hours on one H100, yet it still underperforms tuned Adam/SGD on the small MLP tasks in Appendix F.2. In comparison, Celo2 [2] is meta-trained in 4.5 GPU hours, outperforms AdamW on MLP, CNN, and Transformer tasks, and scales to GPT-3 (1.3B) pretraining.

  3. Inference is expensive. At every step, POP runs a transformer with about 333K parameters over the history of each coordinate to predict its step size, which means the per-step optimization time is likely tremendous, this makes POP impractical for training Transformers with billions of parameters or more.

[1] Jordan, et al. Muon: An Optimizer for Hidden Layers in Neural Networks. Blog post 2024 [2] Moudgil, et al. Celo2: Towards Learned Optimization Free Lunch. ICLR 2026

Questions for authors

Just curious, why do you meta-train POP with RL rather than with evolution strategies (ES) or backpropagation through the unrolled trajectory, given that the objectives are differentiable?

Reviewer_rAf1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose POP: Prior-Fitted First Order Optimization Policies, a first-order optimization method utilizing a trained model for choosing per-coordinate step-size for gradient descent. The transformer-based model is trained using RL on a dataset of synthetic functions generated from Gaussian processes and other fixed components. The authors detail their model training and experimental methodologies in a clear manner. The authors compare the model to other first-order optimization methods, showing strong performance of POP compared to other methods on VLSE test functions. A limited result comparing to real-world test problems is shown, where POP performs worse than other optimizers. Overall, the paper provides a clear explanation of their proposed approach and clear results on the VLSE test set.

Critical strengths

The paper is clearly written and the methodology is easily understandable. The training and inference processes are clearly laid out. The authors report superior performance compared to other first-order optimization methods on the tested VLSE tasks. The real-world optimization task section shows generalization from the 2D training data to higher dimensional VLSE problems. This generalization is promising and impressive given the limited training data.

Critical weaknesses
  1. The authors claim the training data is task independent, but it is unclear if the results will generalize to problems beyond the VLSE benchmark. Further diversity of benchmark problems would be beneficial. The results on the real-world task profiles are less convincing, as POP is beaten by the baselines. Therefore, it is reasonable to question whether the model is aligned closely with the VLSE test set and would struggle to generalize to other problems.

  2. Baseline optimizers only receive initial learning rate tuning and use POP’s required initialization setup. Further hyper-parameter tuning would be beneficial here for fair experimental comparison.

  3. POP performs worse than various Bayesian optimization methods given the same evaluation budget, even on in-distribution tasks. This information is pushed to the appendix and little analysis of this result is provided. Further analysis and interpretation of these results, other than citing evaluation speed, would be beneficial.

  4. Learning a policy to adapt step sizes is an established idea, and the paper does not clearly position POP relative to prior learned optimizers. Without comparisons to these methods, the empirical evidence is insufficient to establish the contribution beyond existing work. This seems to be a minor contribution with limited empirical evaluation justifying its improvements.

Questions for authors

Why is the model exclusively trained on 2D data? You state you want to test how the model generalizes but seeing how further training impacts the results, even just on up to 5D functions, would be beneficial. Further testing of the model beyond just 2D and 32D data would be useful. Can you provide more detail on how the baseline methods were tuned? The main text mentions a sweep of 7 learning rates, but the appendix claims a deeper search. Which is correct?

Can you elaborate on the positioning of POP compared to other learned optimization methods such as Learning to learn by gradient descent by gradient descent [1] or VeLO [2]? The idea of learning a policy to adapt step size has been tested and the exact positioning of the paper with regard to other methods is limited.

Have you evaluated on more diverse tests, such as further scalable VLSE functions or BBOB via a differentiable implementation such as bbob-jax, which also has fixed search spaces?

[1] Andrychowicz, Marcin, et al. "Learning to learn by gradient descent by gradient descent." Advances in neural information processing systems 29 (2016). [2] Metz, Luke, et al. "Velo: Training versatile learned optimizers by scaling up." arXiv preprint arXiv:2211.09760 (2022).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.