- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose model-based meta-learning (MBML), which uses a learned world model for meta-training, in the context of meta-reinforcement learning that aims to learn new algorithms or update rules. In the experiments, they compare the World Model (MBML) with Generalized (Brax, high-fidelity) and Positional (Brax, low-fidelity), and show that MBML achieves performance comparable to Generalized in many of the evaluated environments. They also show that MBML is faster than the other two in terms of wall-clock time. In addition to the existing meta-learning algorithms, the authors propose a meta-Learned Optimizer and Loss function (LOL).
Critical strengths
There is prior work that learns neural-network models to serve as substitutes for simulators and makes use of them. However, to the best of the reviewer’s knowledge, there is no prior work that does so in the context of meta-learning and presents results suggesting that that high-fidelity reproduction of the simulator may not be necessary. This appears to be a valuable finding that could influence the design of future meta-learning methods.
Critical weaknesses
I believe that the name “MBML” is inappropriate. This is because the term “model-based meta-RL” already exists and refers to a very broad category of meta-RL methods that use models, and “MBML” can be interpreted in the same way. In addition, there is already a survey [1] that uses “model-based meta-learning” as a category.
The experimental result suggesting that a high-fidelity simulator is not necessarily required is interesting. However, it remains unclear to what extent this finding can be generalized, since the result may depend on the 24-hour stopping condition and on the relatively slow speed of the high-fidelity simulator.
[1] Huisman et al. A survey of deep meta-learning, Artificial Intelligence Review 2021
Questions for authors
Major: For example, regarding Figure 11, what would happen if meta-training with the high-fidelity simulator were allowed to run longer, beyond the current stopping condition?
Minor: The paper states, “this problem of discovering new algorithms or update rules is called many-shot meta-RL (Beck et al., 2025).” However, this definition seems somewhat inaccurate.
Other Comments: Regarding LOL, although LOL-MBML has a higher aggregate median score than PPO-MBML, the difference between them is not very clear, and hyperparameter tuning appears to contribute substantially to the performance of LOL. I think that adding PPO-MBML as a baseline and analyzing it is valuable in itself. At the same time, however, this comparison reveals that the practical value of the components of LOL beyond PPO hyperparameter tuning (i.e., the components that are more distinctly related to meta-learning) is less clear. How do the authors view this point?
I acknowledge that the GPU-supported Atari environment was released only recently, on May 29, 2026 [1], so I do not necessarily consider its omission a weakness that should affect the evaluation of the paper. Nevertheless, for a current comparison of computational speed, would it be more appropriate to compare against a GPU-supported Atari environment?
3 comments
Thank you for addressing my questions. The other reviewers' comments prompted me to reassess the novelty of your work. There is prior work combining learned dynamics models with meta-RL and explicitly addressing model imperfections, e.g., Clavera et al. (2018). However, the present work studies a different setting: using learned world models as surrogate meta-training backends for RL algorithm/update-rule discovery.
I leave a few additional comments below. In my opinion,
- (W1) IMT is a much better name than MBML.
- (W2,Q1) The additional experiment substantially alleviates my concern, although some caution is still needed when generalizing the result because it is limited to Hopper.
- (Q2) Many-shot meta-RL is not synonymous with algorithm/update-rule discovery, so I still think the definition is inaccurate and should instead be explained in contrast to few-shot meta-RL.
- (Q3) The authors explain another value of LOL, namely automatic behavior discovery, but this does not fully answer my original question about the practical value of the learned loss/optimizer beyond PPO hyperparameter optimization.
Although I now view the novelty as more limited than in my initial assessment, several of my concerns have been alleviated, so I keep my score unchanged.
W1 Thank you for confirming. We have explicitly asked the AC whether we can change the title. We will then systematically replace occurrences of “MBML” with “IMT” and “Model-Based Meta-Learning” with “Imagination Meta-Training” in both the text and the figures.
W2, Q1 We’ve caveated the findings in the caption of Figure 36 to highlight that we cannot necessarily generalize this specific finding to all environments.
Q2 We clarified that algorithm discovery is a special case of many-shot meta-RL, and added contrast with few-shot meta-RL L39: “In RL, discovering new algorithms or update rules that improve policy learning over extended interaction budgets falls under the broader field of many-shot meta-RL. This contrasts with few-shot Meta-RL, where policies are learned or adapted with relatively smaller interaction budgets.” We also updated the definition in L100-102.
Q3 We agree that the meta-learned algorithm components in this work still have limited practical value. Our north star is to find general update rules that can replace components of PPO, such that the high cost of meta-training can eventually be amortized and the discovered algorithm components are highly practical. While the field of algorithm discovery has not reached this point yet, we believe our research takes one of many steps towards that goal.
W1 [MBML is a charged name] We thank the reviewer for this suggestion. We agree that the method name Model-Based Meta-Learning (MBML) could be ambiguous. We suggest Imagination Meta-Training (IMT) as an alternative if the reviewer agrees it is more appropriate. If the reviewer agrees, we will change this throughout the paper. Additionally, if the AC agrees, we will also update the title accordingly.
W2, Q1 [time stopping criterion advantages faster methods]: The reviewer wants to clarify whether the 24-hour stopping criterion has given a structural advantage to the algorithms meta-trained with MBML (which is faster). We ran an experiment in the Hopper environment (the fastest to run), so we could run multiple seeds and provide error bars within the rebuttal time. We ran 500 generations for the world model, Generalized, and Positional backends. Figure 36 in Appendix C.11 shows that the world model and generalized meta-training remain close in Hopper generalized performance. We hope this experiment alleviates concerns that the stopping criterion biases the results.
Q2: We corrected the definition of many-shot meta-RL to insist on the key difference between many- and few-shot: the interaction budget of the resultant algorithm: “In RL, the problem of discovering new algorithms or update rules that improve policy learning over extended interaction budgets is broadly called many-shot meta-RL” L41.
Q3: We agree with the reviewer that the gains over baselines are modest; we believe our method's relevance lies in automatically learning bespoke behaviors that are unique to a dataset/simulator. We see Figure 8 as the clearest demonstration of this: it shows that LOL adapts its behavior throughout policy training, lowering its policy entropy before final evaluation. While we agree we could reproduce this behavior with a hand-designed entropy-bonus schedule, we argue that discovering this behavior automatically from data is more general and powerful. We did not know a priori that lower entropy would be favored for the robotics tasks we focused on in this work.
Q4: We thank the reviewer for highlighting this; we’ve added caveats to our findings regarding speedups for the Atari environments.