- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose using differentiable world models to enable test-time adaptation (and learning) of RL policies in offline RL settings. In essence, they make use of the world model to improve the policy once deployed, by forward simulating a fixed horizon through MPC and doing a backwards pass on the policy to optimise the returns. Their contribution is first conceptual, as they argue that this has not been done before, and second empirical, since they validate their method in a wide variety of RL tasks and compare against state of the art baselines.
Critical strengths
- The objectives and methods are clear and well justified.
- The authors validated their approach across a wide range of benchmarks.
Critical weaknesses
- The conceptual novelty feels very fragile. While it may be partially true that no existing work explicitly mentions improving the policy at test time, the practical difference in regimes between testing and training is not so clear in RL anyway if we have a world model.
- The empirical results show improvement, but quite weak in most cases, and there does not seem to be a clear signal that the proposed method is a step increase in performance over existing baselines.
Questions for authors
I am really not convinced by the argument that existing work does not consider policy improvement at test time. First of all, the difference between training time regimes and inference time regimes when one has a world model becomes blurry. When using traditional MBRL algorithms (or MPC-RL style of algorithms like TD-MPC) nothing prevents one from improving the world model at test time. In fact, one would almost certainly have to do so, to try to close residual gaps with the training environment. Then, the novelty claim the authors make becomes vague. If the authors refer explicitly to improving the policy parameters, again, this is also naturally done at deployment (and furthermore one would have to do that almost certainly if one has any hopes of controlling real systems), and is done in both MBRL work (directly by improving the policy every time the world model changes) and also in MPC-related works like TD-MPC, TD-MPC2, etc. Furthermore in [1] authors already discuss the idea of continually improving the world model and the RL policy that serves as a planning priors.
Could the authors expand on this? How would they clarify their claim?
[1] Serra-Gomez, A., Ornia, D. J., Tirumala, D., & Moerland, T. (2025). A KL-regularization framework for learning to plan with adaptive priors. arXiv preprint arXiv:2510.04280.
3 comments
-
Why the distinction matters in practice. Without new data, the world model cannot be improved. A better model alone would not change behavior anyway, since a pretrained policy is a feedforward network that never queries the model. Our paper asks what is still possible in this regime. The only lever left is computation, and the question is where it helps. The conventional uses of extra computation do not:
- Continuing to train the policy on its objective for the same number of gradient steps adds +0.42 (Appendix D.7).
- Updating the policy at the visited state against the critic alone adds +0.12 (see our response to Reviewer v7c1).
- TD-MPC2 trained on the same offline data scores far below standard offline baselines (Table 27).
Updating the policy at the visited state through imagined rollouts adds +4.54 and improves every dataset. The reason is structural. Offline training optimizes the policy in expectation over the dataset's states, so the deployed policy is an amortized optimizer whose action at a particular state need not be the best one the learned models support (lines 68–75). Inference-time computation can optimize conditional on the state actually reached. This is the same mechanism behind inference-time scaling in LLMs [Wang et al., 2023; Yao et al., 2023; Muennighoff et al., 2025].
2. When new data is available.
In offline RL benchmarks, the deployment environment is the one that generated the data, so there is no residual gap to close. When new data is available, as in the reviewer's real-system scenario, the problem becomes offline-to-online RL, and our method remains useful there. On a real system, one would not take exploratory actions to collect data. Instead, the offline-trained policy is deployed for an extended period, the data it collects is used to fine-tune the world model and the policy, and the updated policy is redeployed [Matsushima et al., 2021]. The fine-tuned policy is again a single network trained in expectation over its data, so the same argument applies: computation spent at the state actually reached can still improve on it.
We tested this with MOPO [Yu et al., 2020], a model-based offline RL method:
- MOPO (offline): SAC trained on uncertainty-penalized rollouts from a dynamics ensemble learned on the offline dataset.
- MOPO (offline-to-online): the MOPO world model and policy fine-tuned on 20k additional real environment steps.
- MOPO (offline-to-online) + DiffMPC (ours): our inference-time policy optimization applied to the fine-tuned world model, critic and policy.
| Dataset | MOPO (offline) | MOPO (offline-to-online) | MOPO (offline-to-online) + DiffMPC |
|---|---|---|---|
| halfcheetah-medium-replay | 70.5 ± 0.8 | 90.4 ± 5.8 | 103.0 ± 2.9 |
| walker2d-medium-replay | 80.2 ± 4.1 | 95.9 ± 2.4 | 99.0 ± 7.8 |
| Average | 75.3 | 93.2 | 101.0 |
Inference-time policy optimization improves on the fine-tuned model and policy by +12.6 on halfcheetah-medium-replay and +3.1 on walker2d-medium-replay, with no additional training and no additional data.
References
- Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J., Levine, S., Finn, C., & Ma, T. (2020). MOPO: Model-based Offline Policy Optimization. NeurIPS.
- Matsushima, T., Furuta, H., Matsuo, Y., Nachum, O., & Gu, S. (2021). Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization. ICLR.
Dear Reviewer,
As the discussion period is drawing to a close, we wanted to check whether you had a chance to review our responses above.
On novelty: in MBRL, TD-MPC/TD-MPC2 and PO-MPC [1], improvement at deployment comes from new environment data. Our method works in the offline setting. It updates the policy parameters at the current state using only models trained on the offline dataset using imagined rollouts. Details are in our response above.
On new data at deployment: when new data is available, as in the real-system scenario you describe, our method still helps. We fine-tuned a MOPO world model and policy on 20k additional real environment steps and then applied our method on top, with no further training or data. It raised the average from 93.2 to 101.0. Details are in our response above.
On the size of the gains: spending the same compute on more training adds only +0.42, and updating against the critic alone adds +0.12. Our method adds +4.54 and improves all 15 locomotion datasets.
At Reviewer rKc5's request, we evaluated on six more datasets: four from D4RL Adroit, a high-dimensional dexterous-manipulation domain with 24–30-dimensional actions, and two from NeoRL-2, a near-real-world control benchmark. Inference-time policy optimization improves the pretrained policy on all six. Details are in our response to Reviewer rKc5.
D4RL Adroit (pretrained policy: ReBRAC)
| Task | dim() | zero shot | DiffMPC (ours) |
|---|---|---|---|
| pen-cloned | 24 | 80.22 ± 17.74 | 98.77 ± 12.10 |
| door-cloned | 28 | 0.08 ± 0.06 | 1.72 ± 0.94 |
| hammer-cloned | 26 | 4.67 ± 0.01 | 20.72 ± 9.08 |
| relocate-cloned | 30 | 0.22 ± 0.23 | 0.74 ± 0.07 |
| Average | 21.30 | 30.49 |
NeoRL-2 (pretrained policy: TD3+BC)
| Task | zero shot | DiffMPC |
|---|---|---|
| DMSD | 60.0 ± 1.6 | 67.1 ± 0.7 |
| Fusion | 54.9 ± 0.4 | 57.5 ± 1.2 |
If you have any remaining questions, we would be glad to respond to them.
Regards,
Authors
We thank the reviewer for the careful review and for noting that the objectives and methods are clear and well justified, that the approach is validated across a wide range of benchmarks, and that there are no substantive concerns about soundness. We address each concern below.
1. Clarifying the novelty claim: offline RL, and how our method differs from MBRL, TD-MPC and [1] : W1, Q
Offline RL and boundary between training and inference. We thank the reviewer for pressing on this point; it lets us state the claim precisely. We agree that in online model-based RL the boundary between training and deployment is blurry: data keeps arriving, and the model and policy are retrained as it does. Our paper studies a setting where the boundary is sharp: offline RL. There, a policy is learned entirely from a fixed dataset D collected beforehand by an unknown behavior policy, with no further interaction with the environment [Lange et al., 2012; Levine et al., 2020; Fu et al., 2021]. Offline RL is a well-established paradigm with a large body of work, including batch RL [Ernst et al., 2005; Lange et al., 2012], model-free methods such as BCQ, CQL, IQL, TD3+BC and ReBRAC [Fujimoto et al., 2019; Kumar et al., 2020; Kostrikov et al., 2022; Fujimoto & Gu, 2021; Tarasov et al., 2023], model-based methods such as MOPO and MOReL [Yu et al., 2020; Kidambi et al., 2020], and the standard D4RL benchmark [Fu et al., 2021]; see Levine et al. [2020] and Figueiredo Prudencio et al. [2024] for surveys. It is motivated by applications where online interaction is costly or unsafe, such as healthcare, recommender systems, robotics and autonomous driving [Gottesman et al., 2019; Afsar et al., 2022; Zhou et al., 2023; Kiran et al., 2022], and it is the protocol followed by every baseline we compare against.
What separates training from inference is therefore not time but information. Training has access to D, and transitions encountered at deployment are not used as training data. Our method respects this: the world model, reward model, critic and initial policy are all trained on D, and no realized transition or reward is used to update them.
How the cited methods differ. In each example the reviewer gives, deployment-time improvement is driven by new environment data. MBRL improves the policy each time the model changes, and the model changes because new data arrives. TD-MPC and TD-MPC2 optimize action sequences with MPPI at decision time, while their policy prior is trained on replay-buffer batches. PO-MPC [1] is likewise online: it builds on TD-MPC2, trains its policy prior on replay-buffer batches, and at inference is computationally equivalent to TD-MPC2. What is new in our method is the combination: optimizing the policy parameters at the state the agent is in, using only computation through models trained on the offline dataset.
A detailed comparison with each line of related work is in the appendix. Appendix E compares our method with the Dreamer and TD-MPC families (E.1, Table 26), with gradient-based and embedded differentiable MPC (E.2, Table 28) and with diffusion-based MPC (E.3, Table 30). Appendix A reviews offline and model-based offline RL more broadly. We will add [1], the table above and this discussion to the paper.