acceptodds
Under review as a conference paper at ICLR 2027

Amortized Low-Rank Adaptation for Model-Based Reinforcement Learning

Abstract

World models let agents plan by predicting the consequences of their actions, but changes in the environment can make them inaccurate. We study the problem of adapting a world model to an unknown test-time environment, drawn from a known environment family, using only a few episodes of interaction. Existing approaches trade off computational cost against expressivity, i.e., the range of models a method can produce. For example, in-context learning is computationally cheap but limited in expressivity, and gradient-based adaptation is expressive but computationally expensive. We present CLAW (Context-conditioned Low-rankAdaptation of World models), which addresses this tradeoff by using a hypernetwork to generate low-rank (LoRA) adapters at test time. During pretraining, we simulate adaptation to a variety of environments and jointly train the hypernetwork and base world model. At test time, we freeze the base model and use a forward pass of the hypernetwork to generate adapters from a small batch of test-time transitions. We evaluate CLAW in locomotion and manipulation environment families that vary in dynamics, embodiment, and reward. We show that, using only seconds of test-time data, CLAW outperforms gradient-based adaptation and in-context learning during online adaptation. We also show that CLAW avoids overfitting in data-scarce regimes, that its advantage comes from the expressive adapters rather than context conditioning, and that pretraining the hypernetwork jointly with the base model outperforms training it post hoc.

Primary area:
reinforcement learning
Keywords:
World Models, Robotics, Meta-Learning, Model-based Reinforcement Learning, Adaptation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kswfOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CLAW (Context-conditioned Low-rank Adaptation of World models), a meta-reinforcement learning method designed to quickly and accurately adapt an agent's world model to unfamiliar test-time environments. It achieves this by jointly pretraining a base world model (on TD-MPC2 architecture) alongside a hypernetwork across a set of environments. At test time, the base model is frozen, and the hypernetwork takes a small batch of recent transitions, summarizes them into an environment-identifying context embedding, and generates low-rank (LoRA) weight adapters in chunks via a single fast forward pass to update all layers of the world model. This approach allows agents to adapt online while avoiding the failure models of updating weights from small amounts of data.

CLAW combines the expressivity of gradient based fine tuning, without long training times or overfitting, with the speed of in context learning. Across locomotion and manipulation tasks, CLAW closely matches an oracle ceiling and outperforms post hoc trained hypernetworks.

Critical strengths
  • The paper is well written.

  • The baselines are relevant and fairly compared against

  • The pseudocode is very clear

Critical weaknesses
  • The authors should include experiments with the decoder transformer world model to back up the claim that their findings apply to other architectures. If that is not provided, the sentences where that claim is made should be removed or rephrased to saying that the intervention is minimal and assign it to future work.

  • The results in Figure 4 are slightly confusing as the curves start at peak performance. It should be made clearer how many context episodes are used.

Questions for authors
  1. What is the SGD burst before rollout applied to? Is it in pre-training or test-time?

  2. What are the adapter norms? Is there any environment or task specific patterns?

  3. I understand this may be beyond the scope, but how does the method handle the setting where random exploration is insufficient and an exploration policy is needed?

  4. How is the train-time and test-time distribution different? How different are the unseen environment instances? I want to know whether this is a system identification case where the training support should be robust enough that it covers test-time tasks or if it is the case that the generalization is better. This confusion only arises since there are no held-out environments to the best of my understanding.

3 comments

Thank you for your questions!

Q1 — When is the SGD burst applied?

Q1 "What is the SGD burst before rollout applied to? Is it in pre-training or test-time?"

Our response

The SGD burst is applied in pretraining only, which follows the standard TD-MPC2 training procedure. Test-time runs do not use the burst.

Changes to the manuscript

  • None.

Q2 — Adapter norms

Q2 "What are the adapter norms? Is there any environment or task specific patterns?"

Our response

In the appendix (Figure 15, Appendix G), we plot the base norms and the adapter norms during pretraining. For reference, we also include the base norms of a normal single-task model. All norms rise during training, and the ratio of adapter norm to base norm falls slowly over training. The base norms of our model are close to those of the single-task model.

We did not find an environment-specific or task-specific pattern in the norms. However, we did find a task-specific pattern in what the adapters change. A new appendix figure (Figure 14, Appendix F) shows the world-model loss for each component (dynamics, reward, value and policy) with and without adaptation. It shows that the hypernetwork lowers the losses that the environment change affects. When the dynamics change, the dynamics loss falls with adaptation, and all other losses change little. When the task changes, the reward and value losses fall with adaptation, and the dynamics loss changes little.

Changes to the manuscript

  • We added the adapter and base norms during pretraining to the appendix (Figure 15, Appendix G).
  • We added the model evaluation with and without adaptation to the appendix (Figure 14, Appendix F).

Q3 — When random exploration is insufficient

Q3 "I understand this may be beyond the scope, but how does the method handle the setting where random exploration is insufficient and an exploration policy is needed?"

Our response

The method is not limited to random exploration. As we state in the paper, it works with any data-collection policy, as long as we can query that policy during pretraining. The hypernetwork then learns to generate adapters from the data that this policy collects.

We note that a promising future direction is to adapt the exploration policy itself, so that the data it collects improves downstream performance. We also point this out in the future work section.

Changes to the manuscript

  • None.

Q4 — Train vs test distribution; system identification or generalization?

Q4 "How is the train-time and test-time distribution different? How different are the unseen environment instances? I want to know whether this is a system identification case where the training support should be robust enough that it covers test-time tasks or if it is the case that the generalization is better. This confusion only arises since there are no held-out environments to the best of my understanding."

Our response

In the original manuscript, the test environments are the training environments. We have since added held-out environments to the appendix (Figure 10, Appendix D). The task is closer to system identification: pretraining covers the family, and CLAW identifies which environment it is in and generates the right adapters from a small amount of data. We kindly refer you to our answer to Reviewer 1 (Q1) for the held-out results and for how we scope the paper.

Changes to the manuscript

  • We added held-out environments (five actuator strengths in walker20, five goals in reach, three goals in push) to the appendix (Figure 10, Appendix D).
  • We edited a sentence in Section 3 that could imply a claim about unseen environments (see our answer to Reviewer 1).

W1 — Transformer world model claim

W1 "The authors should include experiments with the decoder transformer world model to back up the claim that their findings apply to other architectures. If that is not provided, the sentences where that claim is made should be removed or rephrased to saying that the intervention is minimal and assign it to future work."

Thank you for your suggestion.

Our response

We acknowledge that we did not provide experiments with other world-model architectures, because of space, time and compute limits. However, we would like to point out that CLAW makes only two changes to world-model training. It inserts LoRA adapters into the world-model layers, and it wraps the standard training loss with an expectation over the environment family. Any world model that allows these two changes can, in principle, use CLAW.

Nevertheless, in the interest of clarity, we have reworded the sentences that make this claim, as the reviewer suggests.

Changes to the manuscript

  • We reworded the sentences that make this claim. Section 3 now says that CLAW can be applied to other world model architectures, but that we do not test them in this paper. And Section 6 lists other world model architectures as future work.

W2 — Figure 4 (Experiment 1) starts at peak

W2 "The results in Figure 4 are slightly confusing as the curves start at peak performance. It should be made clearer how many context episodes are used."

Our response

Before the first step on the x-axis, each method collects a small context dataset with the data-collection policy. This context consists of 3 episodes. After collection, CLAW adapts from the context and the plot begins (hence the high starting value). We have changed the caption of the figure so that it states this clearly.

Changes to the manuscript

  • We now state the number of context episodes in the Experiment 1 caption, and say that the x-axis starts after the context is collected.

Thank you for the responses. I will keep my score.

Reviewer_LhyuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CLAW, a method that adapts a world model to an environment using a small batch of test-time transitions. A hypernetwork encodes the transitions, pools them into a context vector, and generates LoRA adapters for a TD-MPC2 world model. The base model and hypernetwork are trained jointly across environments, allowing adaptation through a single forward pass at test time. The paper evaluates CLAW on online and offline adaptation, compares it with Concat and FiLM conditioning, studies LoRA rank, and examines post-hoc hypernetwork training. CLAW generally performs close to an oracle and outperforms the tested baselines.

Critical strengths
  1. CLAW requires only one hypernetwork forward pass at test time. In Experiment 2, gradient-based methods, including the oracle, deteriorate as more SGD steps are taken on a fixed three-episode dataset, whereas CLAW avoids this issue.
  2. The paper compares CLAW with Concat and FiLM, studies LoRA rank, and includes a post-hoc variant. It also reports that FiLM performs better under smaller parameter budgets, providing a useful perspective on the trade-off between parameter efficiency and expressivity.
Critical weaknesses
  1. The evaluation does not establish generalization to unseen environments. Table 4 indicates that the training environments cover the full family, and the limitations section acknowledges that CLAW generates adapters for environments seen during pretraining. The evaluated families are also small, including only two environments for Go1 and Cheetah/Walker.
  2. The baselines do not fully support the paper's claims.
  • In Experiments 1 and 2, DR and ST models are not trained for adaptation. A meta-trained gradient-based baseline would provide a more appropriate comparison.
  • Concat and FiLM are evaluated only on Cheetah/Walker, which is insufficient to support claims of superiority over in-context learning across families.
  • Experiment 5 compares CLAW against a base pretrained only on Walker and then adapted to Cheetah. This confounds joint versus post-hoc training with whether the base model has seen the target embodiment.
  • Despite emphasizing the compute–expressivity trade-off, the paper reports neither adaptation costs nor pretraining overhead.
Questions for authors
  1. Do DR and ST use only the three context episodes or also their own online rollouts? What does “fresh rank-16 adapters” in Table 5 mean? Does CLAW use the same fixed context batch throughout?
  2. Are the test environments identical to those used during training? If so, please evaluate on held-out actuator strengths in Walker, including values outside the training range, and held-out goal positions in Reach and Push.
  3. Add a meta-trained gradient-based baseline and evaluate Concat and FiLM across all four families in Experiments 1 and 2.
  4. Compare the wall-clock time or FLOPs of one CLAW adaptation with one gradient step, and report the pretraining overhead relative to standard TD-MPC2.
2 comments

Thank you for your questions!

Q1 — Protocol details

Q1 "Do DR and ST use only the three context episodes or also their own online rollouts? What does "fresh rank-16 adapters" in Table 5 mean? Does CLAW use the same fixed context batch throughout?"

Our response

  • DR and ST train on their own rollouts, in addition to the prefill data.
  • "Fresh rank-16 adapters" means LoRA adapters of rank 16 with the default zero LoRA initialization.
  • CLAW uses a fixed random-action context for the whole run.

Changes to the manuscript

  • None.

Q2 — Held-out actuator strengths and goals

Q2 "[..] please evaluate on held-out actuator strengths in Walker, including values outside the training range, and held-out goal positions in Reach and Push."

W1 "The evaluated families are [..] small"

Our response

We kindly refer you to our answer to Reviewer 1 (Q1), which addresses these points in full.

In short, we added held-out environments to the appendix (Appendix D). When the training environments cover the family densely, as in walker, CLAW performs almost as well on held-out environments as on training environments. When coverage is sparse, as in reach and push, CLAW's performance drops on held-out environments.

Regarding family sizes, we acknowledge that the families could be larger (indeed, we moved in this direction with the 20 environments of walker, where CLAW still performs the best). Fortunately, there is no fundamental reason why CLAW cannot be scaled to larger environment families. Neither CLAW's pretraining nor its adaptation mechanism depends on family size (except for the number of pretraining steps required). We did not include larger families due to compute constraints (pretraining compute scales linearly with the number of environments) and because we wanted our contribution to focus on how to amortize low-rank adaptation during pretraining, particularly when model capacity is too limited for the required multi-modality of, e.g., embodiment changes.

Changes to the manuscript

  • We added Experiment 1 on held-out environments to the appendix (Appendix D).

Q3 — Add baselines

Q3 "Add a meta-trained gradient-based baseline and evaluate Concat and FiLM across all four families in Experiments 1 and 2."

Our response

We added both baselines the reviewer asked for.

For the meta-trained gradient baseline, we use first-order MAML (FOMAML) on the same adapters that the gradient methods update at test time. We chose MAML because it is the standard gradient-based meta-learning method. We use the first-order version because vanilla MAML differentiates through every adaptation step, and our adaptation takes tens of thousands of steps (which makes the backward pass computationally prohibitive). This baseline performs similarly to domain randomization in most tasks. Where it differs, it performs a bit above domain randomization, but still below CLAW, as shown in the updated Exp1 and Exp2 plots.

For Concat and FiLM, we ran Experiments 1 and 2 on all four families, and added the results to the appendix (Figures 11–13, Appendix E). Concat and FiLM perform about the same as CLAW in these experiments, which is consistent with Experiment 3, where CLAW's advantage only shows up when the base model's capacity is limited. Therefore, we narrowed our claims about in-context learning in the abstract, introduction and conclusion. They now say that CLAW outperforms in-context learning when the base model's capacity is limited.

Changes to the manuscript

  • We added FOMAML as a baseline to Experiments 1 and 2.
  • We added Concat and FiLM on all four families (Appendix E).
  • We narrowed the claims about in-context learning in the abstract, introduction and conclusion.

W2 — Experiment 5 confound

W2 "The baselines do not fully support the paper's claims.

  • Experiment 5 compares CLAW against a base pretrained only on Walker and then adapted to Cheetah. This confounds joint versus post-hoc training with whether the base model has seen the target embodiment."

Our response

This is a great point! We added a new baseline to Experiment 5 (Figure 8) to remove this confound. We trained a base model with domain randomization on both embodiments, froze it, and then trained the hypernetwork on top of it. This model has seen the same embodiments as CLAW, and it uses the same total number of training steps. It performs better than the original baseline (which had not seen both embodiments) but is still worse than CLAW. This removes the confound and shows that CLAW's advantage comes from training the base model and the hypernetwork jointly.

Changes to the manuscript

  • We added the new baseline to Experiment 5.

Q4 — Wall clock or FLOPs

Q4 "Compare the wall-clock time or FLOPs of one CLAW adaptation with one gradient step, and report the pretraining overhead relative to standard TD-MPC2."

Our response

We apologize for not reporting these costs! We report timing/FLOPs results in a new experiment, Experiment 6 (Figure 9). For adaptation cost, we measure one CLAW adaptation, which is one hypernetwork forward pass, and one gradient step on the adapters, as the gradient-based methods take at test time. Both run on the same GPU with the same model. One forward pass of CLAW takes 0.10–0.17 ms and 0.33 GFLOP. One gradient step takes about 2.6 ms and 41 GFLOP. So one CLAW adaptation is 15 to 25 times faster and uses about 125 times fewer FLOPs than one gradient step.

Pretraining has two sources of overhead. The first is the cost of each training step. CLAW trains slightly slower per step than standard TD-MPC2 (~3.7 ms against ~2.95 ms on walker), because batching is more complex and the hypernetwork adds computation to the backward pass. The second source of overhead is the total number of training steps. We give every test-time environment the same number of training steps, so the total grows linearly with the number of environments. This follows our overall approach of amortizing as much compute as possible during pretraining, so that test-time adaptation is fast. Future work could allocate steps to environments adaptively, e.g., by environment difficulty.

Changes to the manuscript

  • We added Experiment 6, which reports the adaptation cost (Figure 9).
  • We added the overhead costs to the limitations.
Reviewer_BBTROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces a method that adapts a world model to new environments, such as a set of walkers with different actuator strengths. A hypernetwork reads a small batch of transitions collected with a random policy and outputs LoRA adapters for every layer of a frozen TD-MPC2 world model. The base model and hypernetwork are pretrained across tasks, so adaptation becomes a single forward pass instead of gradient descent. This combines the low cost of in-context or embedding-based adaptation with weights fine-tuning.

Critical strengths
  • Adaptation by only following a forward pass without having to rely on gradient descent.
Critical weaknesses
  • The main issue with the paper is the writing and the notation is hard to parse even for someone familiar with MBRL. The authors might be better advised to stick to the notation from tdmpc[1] or modem[2].
  • The adaptation relies on what environments are used in pretraining, also it is not clear whether the method can extrapolate/interpolate to unseen values
  • no evaluation of the model quality with adaptation, so evaluation is only made based on task performance

Hansen, Nick, Hao Su, and Xiaolong Wang. "Td-mpc2: Scalable, robust world models for continuous control." International Conference on Learning Representations. Vol. 2024. 2024. Hansen, Nicklas, et al. "Modem: Accelerating visual model-based reinforcement learning with demonstrations." arXiv preprint arXiv:2212.05698 (2022).

Questions for authors
  • How would the method behave on unseen environments from within the same family?
  • Can you provide model evaluations with and without adaptation?
2 comments

Thank you for your questions!

Q1 — Unseen environments within the family

Q1 "How would the method behave on unseen environments from within the same family?"

Our response

We ran Experiment 1 on held-out environments and added the results to the appendix (Figure 10, Appendix D). We discuss the details below. In the walker experiment, CLAW performs as well as on the training environments. In Reach and Push, CLAW underperforms DR and ST.

Generalization to unseen environments is an important problem, and we address the reviewer's concerns in two ways. First, we note that the paper acknowledges CLAW's limitations in this respect. The limitations in Section 6 state that "CLAW learns to generate adapters only for the environments seen during pretraining", and the future work lists "pretraining and/or architectural improvements for better generalization". Second, we expect pretraining scale to reduce this problem. As pretraining compute and simulator fidelity improve, pretraining will cover more of the environments an agent will see at test time.

This raises the question: "If scaling pretraining is all you need, what is the point of CLAW?" Our answer is that pretraining coverage alone is not enough for test-time performance, because at test time, the model must represent each covered environment well. In many cases, for example when the embodiment changes or when the world model's capacity is limited, naive domain randomization produces an "average" world model that does not perform well in any one environment. Our work provides a pretraining and adaptation procedure that lets the agent adapt to the highly varied environments it will see at test time.

The held-out plots we added support this view. Walker has 20 training actuator strengths, which cover the range densely. On its five held-out strengths, CLAW performs as well as on the training strengths. Reach and push have fewer training goals, and therefore cover the range sparsely. As expected, on the held-out goals, CLAW's success drops considerably.

Changes to the manuscript

  • We added Experiment 1 on held-out environments to the appendix (Figure 10, Appendix D). We did not test actuator strengths outside the training range. Below the range, the walker cannot walk at all. Above the range, the model needs no adaptation to perform well.
  • We edited a sentence in Section 3 that could be read as a claim about unseen environments. It said the agent interacts at test time with environments from p(E) "which it may or may not have encountered during pretraining". It now says "which it encountered during pretraining, though it is not told which one".

Q2 — Model evaluations with and without adaptation

Q2 "Can you provide model evaluations with and without adaptation?"

Our response

We have added a plot to the appendix (Appendix F) that shows the world-model loss for each component (dynamics, reward, value and policy) with and without adaptation.

In every family, adaptation lowers the losses that the environment change affects. When the dynamics change, the dynamics loss falls with adaptation, and all other losses change little. When the task changes, the reward and value losses fall with adaptation, and the dynamics loss changes little. This shows that our method corrects the part of the world model that differs between environments.

We did not report these losses in the original submission because our goal is task performance, and a lower loss is a means to that goal, not the goal itself.

Changes to the manuscript

  • We added this evaluation to the appendix (Figure 14, Appendix F).

W1 — Notation is hard to parse

W1 "The main issue with the paper is the writing and the notation is hard to parse even for someone familiar with MBRL. The authors might be better advised to stick to the notation from tdmpc[1] or modem[2]."

Our response

Thank you for this feedback, and we are sorry that the notation was hard to follow. We tried our best to keep the notation simple and to overload as few symbols as possible. We have added a notation table to the appendix (Table 10, Appendix C), which we hope makes the paper easier to follow. We considered the TD-MPC2 notation, but it conflicted with the symbols our method needs. For example, TD-MPC2 uses hh for the encoder, which overlaps with our hypernetwork HH. Our method also has to tell apart four sets of neural network parameters: the parameters of the base model, the adapter template parameters, the hypernetwork parameters, and hypernetwork-generated parameters. We also needed separate symbols for everything within the hypernetwork so that they do not overload the symbols of the world model.

Changes to the manuscript

  • We added a notation table to the appendix (Table 10, Appendix C) that lists each symbol and its meaning.

Thanks to the authors for addressing my concerns and questions. I keep my rating for acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.