- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CLAW (Context-conditioned Low-rank Adaptation of World models), a meta-reinforcement learning method designed to quickly and accurately adapt an agent's world model to unfamiliar test-time environments. It achieves this by jointly pretraining a base world model (on TD-MPC2 architecture) alongside a hypernetwork across a set of environments. At test time, the base model is frozen, and the hypernetwork takes a small batch of recent transitions, summarizes them into an environment-identifying context embedding, and generates low-rank (LoRA) weight adapters in chunks via a single fast forward pass to update all layers of the world model. This approach allows agents to adapt online while avoiding the failure models of updating weights from small amounts of data.
CLAW combines the expressivity of gradient based fine tuning, without long training times or overfitting, with the speed of in context learning. Across locomotion and manipulation tasks, CLAW closely matches an oracle ceiling and outperforms post hoc trained hypernetworks.
Critical strengths
-
The paper is well written.
-
The baselines are relevant and fairly compared against
-
The pseudocode is very clear
Critical weaknesses
-
The authors should include experiments with the decoder transformer world model to back up the claim that their findings apply to other architectures. If that is not provided, the sentences where that claim is made should be removed or rephrased to saying that the intervention is minimal and assign it to future work.
-
The results in Figure 4 are slightly confusing as the curves start at peak performance. It should be made clearer how many context episodes are used.
Questions for authors
-
What is the SGD burst before rollout applied to? Is it in pre-training or test-time?
-
What are the adapter norms? Is there any environment or task specific patterns?
-
I understand this may be beyond the scope, but how does the method handle the setting where random exploration is insufficient and an exploration policy is needed?
-
How is the train-time and test-time distribution different? How different are the unseen environment instances? I want to know whether this is a system identification case where the training support should be robust enough that it covers test-time tasks or if it is the case that the generalization is better. This confusion only arises since there are no held-out environments to the best of my understanding.
3 comments
W2 — Figure 4 (Experiment 1) starts at peak
W2 "The results in Figure 4 are slightly confusing as the curves start at peak performance. It should be made clearer how many context episodes are used."
Our response
Before the first step on the x-axis, each method collects a small context dataset with the data-collection policy. This context consists of 3 episodes. After collection, CLAW adapts from the context and the plot begins (hence the high starting value). We have changed the caption of the figure so that it states this clearly.
Changes to the manuscript
- We now state the number of context episodes in the Experiment 1 caption, and say that the x-axis starts after the context is collected.
Thank you for the responses. I will keep my score.
Thank you for your questions!
Q1 — When is the SGD burst applied?
Our response
The SGD burst is applied in pretraining only, which follows the standard TD-MPC2 training procedure. Test-time runs do not use the burst.
Changes to the manuscript
Q2 — Adapter norms
Our response
In the appendix (Figure 15, Appendix G), we plot the base norms and the adapter norms during pretraining. For reference, we also include the base norms of a normal single-task model. All norms rise during training, and the ratio of adapter norm to base norm falls slowly over training. The base norms of our model are close to those of the single-task model.
We did not find an environment-specific or task-specific pattern in the norms. However, we did find a task-specific pattern in what the adapters change. A new appendix figure (Figure 14, Appendix F) shows the world-model loss for each component (dynamics, reward, value and policy) with and without adaptation. It shows that the hypernetwork lowers the losses that the environment change affects. When the dynamics change, the dynamics loss falls with adaptation, and all other losses change little. When the task changes, the reward and value losses fall with adaptation, and the dynamics loss changes little.
Changes to the manuscript
Q3 — When random exploration is insufficient
Our response
The method is not limited to random exploration. As we state in the paper, it works with any data-collection policy, as long as we can query that policy during pretraining. The hypernetwork then learns to generate adapters from the data that this policy collects.
We note that a promising future direction is to adapt the exploration policy itself, so that the data it collects improves downstream performance. We also point this out in the future work section.
Changes to the manuscript
Q4 — Train vs test distribution; system identification or generalization?
Our response
In the original manuscript, the test environments are the training environments. We have since added held-out environments to the appendix (Figure 10, Appendix D). The task is closer to system identification: pretraining covers the family, and CLAW identifies which environment it is in and generates the right adapters from a small amount of data. We kindly refer you to our answer to Reviewer 1 (Q1) for the held-out results and for how we scope the paper.
Changes to the manuscript
W1 — Transformer world model claim
Thank you for your suggestion.
Our response
We acknowledge that we did not provide experiments with other world-model architectures, because of space, time and compute limits. However, we would like to point out that CLAW makes only two changes to world-model training. It inserts LoRA adapters into the world-model layers, and it wraps the standard training loss with an expectation over the environment family. Any world model that allows these two changes can, in principle, use CLAW.
Nevertheless, in the interest of clarity, we have reworded the sentences that make this claim, as the reviewer suggests.
Changes to the manuscript