acceptodds
Under review as a conference paper at ICLR 2027

Model-Based Meta-Learning for Algorithm Discovery

16 views

Abstract

Reinforcement Learning (RL) algorithms are typically hand-crafted through a slow and iterative scientific process. While meta-RL promises to automate RL algorithm discovery, high simulation costs have limited research. In this work, we introduce Model-Based Meta-Learning (MBML), a novel approach that uses learned world models as an efficient alternative to environment simulation. Policies trained in these world models fail to transfer to the simulator, but algorithms discovered in world models transfer successfully. We show that MBML matches the performance of meta-RL with environment simulators at a fraction of the time and compute. By substantially reducing the computational cost of meta-training, MBML lowers the barrier to entry for meta-RL research while enabling algorithm discovery at a larger scale. We demonstrate this by using MBML to meta-learn a novel algorithm that significantly outperforms prior work on unseen continuous control environments.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Meta-Learning, Generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iK3eOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose model-based meta-learning (MBML), which uses a learned world model for meta-training, in the context of meta-reinforcement learning that aims to learn new algorithms or update rules. In the experiments, they compare the World Model (MBML) with Generalized (Brax, high-fidelity) and Positional (Brax, low-fidelity), and show that MBML achieves performance comparable to Generalized in many of the evaluated environments. They also show that MBML is faster than the other two in terms of wall-clock time. In addition to the existing meta-learning algorithms, the authors propose a meta-Learned Optimizer and Loss function (LOL).

Critical strengths

There is prior work that learns neural-network models to serve as substitutes for simulators and makes use of them. However, to the best of the reviewer’s knowledge, there is no prior work that does so in the context of meta-learning and presents results suggesting that that high-fidelity reproduction of the simulator may not be necessary. This appears to be a valuable finding that could influence the design of future meta-learning methods.

Critical weaknesses

I believe that the name “MBML” is inappropriate. This is because the term “model-based meta-RL” already exists and refers to a very broad category of meta-RL methods that use models, and “MBML” can be interpreted in the same way. In addition, there is already a survey [1] that uses “model-based meta-learning” as a category.

The experimental result suggesting that a high-fidelity simulator is not necessarily required is interesting. However, it remains unclear to what extent this finding can be generalized, since the result may depend on the 24-hour stopping condition and on the relatively slow speed of the high-fidelity simulator.

[1] Huisman et al. A survey of deep meta-learning, Artificial Intelligence Review 2021

Questions for authors

Major: For example, regarding Figure 11, what would happen if meta-training with the high-fidelity simulator were allowed to run longer, beyond the current stopping condition?

Minor: The paper states, “this problem of discovering new algorithms or update rules is called many-shot meta-RL (Beck et al., 2025).” However, this definition seems somewhat inaccurate.

Other Comments: Regarding LOL, although LOL-MBML has a higher aggregate median score than PPO-MBML, the difference between them is not very clear, and hyperparameter tuning appears to contribute substantially to the performance of LOL. I think that adding PPO-MBML as a baseline and analyzing it is valuable in itself. At the same time, however, this comparison reveals that the practical value of the components of LOL beyond PPO hyperparameter tuning (i.e., the components that are more distinctly related to meta-learning) is less clear. How do the authors view this point?

I acknowledge that the GPU-supported Atari environment was released only recently, on May 29, 2026 [1], so I do not necessarily consider its omission a weakness that should affect the evaluation of the paper. Nevertheless, for a current comparison of computational speed, would it be more appropriate to compare against a GPU-supported Atari environment?

[1] https://ale.farama.org/release_notes/index.html

3 comments

W1 [MBML is a charged name] We thank the reviewer for this suggestion. We agree that the method name Model-Based Meta-Learning (MBML) could be ambiguous. We suggest Imagination Meta-Training (IMT) as an alternative if the reviewer agrees it is more appropriate. If the reviewer agrees, we will change this throughout the paper. Additionally, if the AC agrees, we will also update the title accordingly.

W2, Q1 [time stopping criterion advantages faster methods]: The reviewer wants to clarify whether the 24-hour stopping criterion has given a structural advantage to the algorithms meta-trained with MBML (which is faster). We ran an experiment in the Hopper environment (the fastest to run), so we could run multiple seeds and provide error bars within the rebuttal time. We ran 500 generations for the world model, Generalized, and Positional backends. Figure 36 in Appendix C.11 shows that the world model and generalized meta-training remain close in Hopper generalized performance. We hope this experiment alleviates concerns that the stopping criterion biases the results.

Q2: We corrected the definition of many-shot meta-RL to insist on the key difference between many- and few-shot: the interaction budget of the resultant algorithm: “In RL, the problem of discovering new algorithms or update rules that improve policy learning over extended interaction budgets is broadly called many-shot meta-RL” L41.

Q3: We agree with the reviewer that the gains over baselines are modest; we believe our method's relevance lies in automatically learning bespoke behaviors that are unique to a dataset/simulator. We see Figure 8 as the clearest demonstration of this: it shows that LOL adapts its behavior throughout policy training, lowering its policy entropy before final evaluation. While we agree we could reproduce this behavior with a hand-designed entropy-bonus schedule, we argue that discovering this behavior automatically from data is more general and powerful. We did not know a priori that lower entropy would be favored for the robotics tasks we focused on in this work.

Q4: We thank the reviewer for highlighting this; we’ve added caveats to our findings regarding speedups for the Atari environments.

Thank you for addressing my questions. The other reviewers' comments prompted me to reassess the novelty of your work. There is prior work combining learned dynamics models with meta-RL and explicitly addressing model imperfections, e.g., Clavera et al. (2018). However, the present work studies a different setting: using learned world models as surrogate meta-training backends for RL algorithm/update-rule discovery.

I leave a few additional comments below. In my opinion,

  • (W1) IMT is a much better name than MBML.
  • (W2,Q1) The additional experiment substantially alleviates my concern, although some caution is still needed when generalizing the result because it is limited to Hopper.
  • (Q2) Many-shot meta-RL is not synonymous with algorithm/update-rule discovery, so I still think the definition is inaccurate and should instead be explained in contrast to few-shot meta-RL.
  • (Q3) The authors explain another value of LOL, namely automatic behavior discovery, but this does not fully answer my original question about the practical value of the learned loss/optimizer beyond PPO hyperparameter optimization.

Although I now view the novelty as more limited than in my initial assessment, several of my concerns have been alleviated, so I keep my score unchanged.

W1 Thank you for confirming. We have explicitly asked the AC whether we can change the title. We will then systematically replace occurrences of “MBML” with “IMT” and “Model-Based Meta-Learning” with “Imagination Meta-Training” in both the text and the figures.

W2, Q1 We’ve caveated the findings in the caption of Figure 36 to highlight that we cannot necessarily generalize this specific finding to all environments.

Q2 We clarified that algorithm discovery is a special case of many-shot meta-RL, and added contrast with few-shot meta-RL L39: “In RL, discovering new algorithms or update rules that improve policy learning over extended interaction budgets falls under the broader field of many-shot meta-RL. This contrasts with few-shot Meta-RL, where policies are learned or adapted with relatively smaller interaction budgets.” We also updated the definition in L100-102.

Q3 We agree that the meta-learned algorithm components in this work still have limited practical value. Our north star is to find general update rules that can replace components of PPO, such that the high cost of meta-training can eventually be amortized and the discovered algorithm components are highly practical. While the field of algorithm discovery has not reached this point yet, we believe our research takes one of many steps towards that goal.

Reviewer_kSXJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a recipe for accelerating reinforcement learning algorithm design by using model-based meta reinforcement learning.

The paper shows that by learning a dynamics model of RL environments, data-generation throughput can be increased by orders of magnitude, therefore enabling better RL algorithm discovery by meta-RL algorithms.

Critical strengths
  • The idea of using a model-based approach to accelerate algorithm design via meta-RL is interesting and novel.
  • The result showing that a meta-learned RL algorithm has good sim-to-sim transfers even when policy instances don't necessarily have such transfer is encouraging as it shows that there is no trivial solution.
Critical weaknesses
  • Learning a dynamics model only for a single environment might produce noisy results. The meta-RL algorithm learns a single dynamics model (specifically, HalfCheetah) that is used to eventually "generate" an RL algorithm. The paper shows generalization of the RL algorithm to other RL environments (Walker2d, Hopper). Two weaknesses arise: (1) the number of meta-test environments is quite small, so it is hard to gauge how good is the new RL algorithm. (2) It could be that if we meta-train on another RL environment (say Walker2d), we'd get substantially different generalization.
  • Following on the previous point, overall it seems like a more appropriate approach would be to train "world model" in a more realistic sense, i.e. using a world model that can generate rollouts for a larger pool of RL tasks (e.g. Nvidia Cosmos or TD-MPC scale?). Currently the paper only uses a simple MLP. What happens if we need to model a more complex dynamical system with a larger model? Would these throughput results hold? I understand that many research groups have access to this type of compute so this is not a direct criticism on the paper.
  • That leads to the second limitation. The paper argues that one can achieve significant throughput gains compared to massively-parallel simulators like mujoco. (1) What happens as we scale the number of parallel environments of the mujoco simulator? (2) What happens when we scale the size of the MLP? (3) Did you try to use more modern mujoco versions like warp? The paper shows additional results on an Atari game, using a larger world model, however to my understanding, it compares with a sequential implementation of Atari that does not run in parallel ("cannot use envpool C.3?)
Questions for authors

Minors:

  • Abstract: " a novel approach that uses learned world models as an efficient alternative to environment simulation" seems contradictory -- a world model is a simulator, just a data-driven one, not one that was developed from first-principles.

  • L.162 Goldie .. meta-learns

For other questions see above.

3 comments

W1.1: [Number of meta-test environments is quite small] We clarify that the algorithms for the main text comparison were meta-trained on the HalfCheetah world model; we meta-tested on the full Brax suite (11 environments). Additionally, to probe broad algorithm generalization, we meta-tested our algorithms on an additional 13 environments that are semantically different from Brax (See Appendices B.5 and C.12).

W1.2 [Meta-training on more environments] We understand this criticism as requesting that we meta-train on a different environment and show meta-test results for that algorithm. We have meta-trained TA-LPO with the Hopper world model and have added its meta-test results to Appendix C.13 Figure 39. We see that the downstream meta-test performance is lower and therefore depends on the meta-training world model used.

Q1 [Using large general world models] The reviewer asks a general question about how large world models might affect MBML meta-training. As shown in Appendix C.3 (Atari results), we achieved 60k steps per second with a large diffusion model (10M parameters). This is substantially lower than either Brax or the world model (1-20M steps per second). The speedup in the Atari results is mostly because no Atari environment was parallelizable on our infrastructure at the time (reviewer iK3e correctly noted that this is no longer the case). We would be interested in running experiments with much larger world models, specifically by randomly conditioning the model during meta-training to obtain data diversity. Nevertheless, we view the advantage of MBML as the fact that world models need not be accurate for meta-training.

W2 [Increasing parallelization] The reviewer argues that MBML's speedup gains come from limited parallelization of the physics simulator. We argue that as parallelization increases, MBML becomes more effective. We measured how throughput changes with parallelization in HalfCheetah and find that the world model scales to a higher peak throughput than the physics simulators in the new Figure 10. Physics simulators such as MuJoCo rely on iterative numerical integration, which is less amenable to parallelization. World models use matrix multiplications, which GPUs are optimized for. Specifically, under the hood, the environment step for the agent population is a single batched forward pass. For our main experiments, we ran 2000 parallel environments per agent; for a population of 32, this is 64k parallel environments. We expect peak SPS to go down as we scale up model size, but as we argue in the paper, a larger world model (correlated with accuracy) is not necessarily needed.

  • We did not try Warp; we have updated the paper to mention Warp as a future direction.
  • The reviewer is correct that our gains in Atari come from comparing a non-parallelizable implementation with a parallelizable one. EnvPool and ALE-py have received significant updates in the last few months, including expanded hardware support and vectorization. Recently, other libraries such as Octax and JAXtari have also been developed. We did not test these new and updated libraries; therefore, we have significantly caveated our findings regarding Atari speedup results L371 & L1232.

Minors:

  • We corrected “environment simulation” to “environment physics simulation” to more clearly distinguish iterative physics integration from the world-model forward pass.
  • Thank you for raising this typo; we corrected it L166.

Thank you for writing the rebuttal. I have increased my score due to:

  • Meta-test used 11 environments and not 3 as I understood before -- please make this more explicit in the final revision.
  • New Figure 10 that compares throughput of the world model and simulator as the number of parallel rollouts grow.
  • Caveating the Atari results not using a parallelizable environment more explicitly.

Suggestion: in Figure 1, consider showing this as a scaling plot, where the x axis is the number of parallel rollouts vs. y axis measuring the throughput (similar to your new Figure 10). This will help make the numerical-integration vs. matrix multiplication better argument. We expect to see sublinear growth, while the world-model rollout saturates slower.

Thank you for the swift response and score increase. We will integrate this feedback into the paper.

Reviewer_3mGGOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The authors identify meta-training wall-clock time as a significant bottleneck in many-shot meta-RL. They suggest using offline-learned world-models to replace simulation backends in order to improve this aspect (Model-Based Meta-Learning). The authors show that training existing meta-RL methods with the world model achieves similar returns in comparison to the simulation counterparts while being much faster. Finally, the author leverages this improvement and suggests a new algorithm (LOL) that optimizes more components in the algorithm compared to competitive baselines.

Critical strengths
  1. The proposed LOL algorithm archives compelling returns compared to competitive baselines on the evaluated tasks, including a clear separation between meta-training/validation/test.
  2. The paper is well written and easy to follow. The problem setup of computational bottleneck in many-shot meta-RL is clear and well motivated.
Critical weaknesses
  1. [Critical] Novelty: In my opinion the methodological contribution of this work is limited. The main presented contribution is to show improved computational load when using existing (or modified) many-shot meta-rl algorithms with off-the-shelf offline world models. Using world models as a fast alternative to simulation backends is well studied in standard RL and has even been suggested in few-shot meta-rl setups [1, 2]. The main presented novelty is the LOL algorithm, which is not backed by a sufficient empirical study (see next weakness).
  2. [Critical] Limited Empirical Evaluation & Overclaim: The authors focused on three basic locomotion tasks (halfcheetah, hopper and walker2d) - which is a very limited setup, and did not perform adequate ablation studies on critical parts of the algorithm. Given this, statements like “Accelerating Research with MBML” are an overclaim.
  3. [Minor] Limitations of Offline World Models: World models are an appealing approach for improved throughput. Yet, one inherent limitation of relying on a fixed offline-trained model is that the policy can exploit inaccuracies and suffer from instability of the model. This limitation was not discussed in the paper.

[1] Model-Based Reinforcement Learning via Meta-Policy Optimization (Clavera et al., 2018)

[2] MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning (Rimon et al., 2024)

Questions for authors
  1. Section 8 introduces LOL without sufficient details - “Specifically, we jointly meta-learn the policy loss, optimizer, and several hyperparameters, combining and extending components from prior work”. I think that adding some of the details appearing in A.2 and referring to A.2 (not only for “convergence properties”) within Section 8 can improve clarity.
  2. Did you experiment with changing world-model capacity? Do you expect a trade-off between inference speed and downstream meta-test accuracy (rooted in world-model-accuracy)?
3 comments

W1 [Novelty]: Our contribution concerns using learned dynamics for transferable algorithm discovery. MAMBA and MB-MPO use world models to learn adaptive policies. We instead use world models as the environment in which we discover a general learning rule. We can then use this learning rule to learn policies in different target environments, including environments with different observation and action dimensions. The key finding is that inexpensive, imperfect world models can help discover these learning rules.

W2.2 [Limited empirical evaluation]: The review mentions that our empirical evaluation is limited to 3 locomotion tasks. This omits the distinction between our meta-training environments and the broader set of meta-test environments. The submitted paper reports evaluation on the Brax suite (11 tasks) and 13 additional non-Brax tasks (Table 7 and Appendix C.12), as well as meta-training and evaluation on 5 Atari games (Appendix C.3). All of these were present in the original submission. The prior work referenced by the reviewer only uses 10 (MAMBA) and 6 (MB-MPO) environments, respectively.

W2.2 [“did not perform adequate ablation of critical parts of the algorithm”]: We presented ablations of world model design in Appendix C.9 (referenced in the main text L260) and LOL component ablations across 3 environments and 3 meta-seeds in Appendix C.10 (referenced in the main text L468). Specifically, Figure 31 isolates the contributions of hyperparameter optimization, the learned optimizer, and their combination, while Figures 21, 27, 29 and 30 investigate world-model accuracy, uncertainty handling, initialization, and termination. In light of this, could the reviewer please be more specific about what the ablations are missing and how they could be improved?

W3 [Limitations of world models]: The reviewer claims we do not address the limitation of world model error. However, in Section 7.2 titled "MBML works with imperfect world models” we explicitly discuss world model error. We show that even with under-trained world models (high validation loss), meta-learning still works. Surprisingly, we find that we can meta-train algorithms using a randomly initialized neural network as a world model. We agree that the empirical validation was limited to a single meta-learning algorithm, so we expanded this experiment to other meta-learning algorithms (see Appendix C.6); we find that OPEN, LPO, LOL, and TA-LPO are all resilient to different world model accuracies.

Q1: We agree with the reviewer and have incorporated more technical details about LOL from the appendix into the main text (Section 8).

Q2: We did not specifically experiment with world model capacity; instead, we focused on varying validation loss, which is a more direct measure of world model accuracy (Section 7.2). We find that meta-training with world models of different validation loss does not significantly affect downstream algorithm meta-test performance: a surprising and non-obvious finding, which highlights the strength of MBML.

I thank the authors for their clarifications and recognize that I had some misunderstandings regarding my point on "Limited Empirical Evaluation & Overclaim. Further, I find the new revision to address most of my concerns. I have adjusted my score and now tend towards acceptance.

Thank you for updating your assessment of the paper and the score increase.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Comment to all

We thank all reviewers for their reviews of our paper. We aimed to keep the rebuttals short for the benefit of the reviewers and the AC. We included additional details in the paper, but if the reviewers would like more detail or additional experiments, we would be happy to provide them. We added all new content in the paper in red. We were limited in the number of experiments we could run due to the high cost of meta-training in physics simulators.

AuthorsOpenReview ↗

To the AC: request for title change

Dear AC,

After discussing this with reviewer iK3e, we've decided to change the method name from "Model-Based Meta-Learning (MBML)" to "Imagination Meta-Training (IMT)". Consequently, we would like to change the title from "Model-Based Meta-Learning for Algorithm Discovery" to "Imagination Meta-Training for Algorithm Discovery". If you approve, we will change the title and the method name throughout the paper.

Thanks in advance,

The authors.

AuthorsOpenReview ↗

Summary of Rebuttal Period

We summarize the reviewers’ requests and how we addressed them below.

  • Contribution and terminology: How does this differ from prior model-based meta-RL, and is the terminology appropriate? [3mGG, iK3e]

    • Clarified the distinction between discovering transferable learning rules and learning adaptive policies, and corrected the many-shot meta-RL definition.
    • Agreed to rename the method Imagination Meta-Training (IMT), with the title change awaiting AC approval.
  • Evaluation and generalization: Is the evaluation sufficiently broad, and does the meta-training environment matter? [3mGG, kSXJ]

    • Clarified that the original evaluation covered 11 Brax tasks, 13 additional non-Brax tasks, and five Atari games.
    • Added Hopper meta-training results, showing that downstream performance depends on the meta-training world model.
  • Speed and parallelism: Do the speedups hold as simulator parallelism increases? [kSXJ, iK3e]

    • Added throughput-versus-parallelism measurements showing that the world model reaches higher peak throughput.
    • Caveated Atari speedups because the comparisons did not use newer parallel implementations; Warp remains future work.
  • World-model accuracy: How sensitive are the results to model imperfections? [3mGG, kSXJ]

    • Added more experiments to OPEN, LPO, LOL, and TA-LPO, finding resilience to world-model error.
    • Clarified that model error was varied through training progress of the model.
  • Meta-training budget: Does the 24-hour stopping criterion favor world models? [iK3e]

    • Added a matched 500-generation experiment across all three backends on Hopper, finding similar world-model and Generalized performance.
  • LOL and ablations: What are the details of LOL? [3mGG]

    • Moved technical details to the main text and clarified existing component ablations.
  • Practical value of many-shot meta-RL for algorithm discovery: What do meta-learned components provide on top of hyperparameter tuning? [iK3e]

    • Discussed automatically discovered training behavior while acknowledging current limited practical use, emphasizing the north star of discovering new general algorithms more efficiently.

Reviewers 3mGG and kSXJ raised their ratings to 3. 3mGG stated that the revision addressed most concerns. iK3e retained their rating of 3, acknowledging that several concerns were alleviated while retaining reservations about novelty.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.