acceptodds
Under review as a conference paper at ICLR 2027

Inference Time Policy Optimization for Offline RL with Differentiable World Models

20 views

Abstract

Offline Reinforcement Learning (RL) learns optimal policies from fixed datasets, training a policy once and deploying it at inference time without further refinement. Inspired by model predictive control (MPC), we introduce an inference time adaptation framework that utilizes a pretrained policy along with a learned world model. While existing world model and diffusion-planning methods use learned dynamics to generate imagined trajectories during training, or to sample candidate plans at inference time, they do not use inference-time information to *optimize* the policy parameters on the fly. In contrast, our design is a Differentiable World Model (DWM) pipeline that enables end-to-end gradient computation through imagined rollouts for *inference time policy optimization* (ITPO). We evaluate our algorithm on D4RL continuous-control benchmarks (MuJoCo locomotion tasks and AntMaze), and show that exploiting inference-time information to optimize the policy parameters yields consistent gains over strong offline RL baselines. Inference-time adaptation, however, is expensive: rollout generation and backpropagation dominate per-step compute. We study this tradeoff explicitly, showing that a suitable tilted version of one-step MeanFlow sampler recovers much of the gains at a fraction of the cost.

Primary area:
reinforcement learning
Keywords:
offline RL

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hDtSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose using differentiable world models to enable test-time adaptation (and learning) of RL policies in offline RL settings. In essence, they make use of the world model to improve the policy once deployed, by forward simulating a fixed horizon through MPC and doing a backwards pass on the policy to optimise the returns. Their contribution is first conceptual, as they argue that this has not been done before, and second empirical, since they validate their method in a wide variety of RL tasks and compare against state of the art baselines.

Critical strengths
  • The objectives and methods are clear and well justified.
  • The authors validated their approach across a wide range of benchmarks.
Critical weaknesses
  • The conceptual novelty feels very fragile. While it may be partially true that no existing work explicitly mentions improving the policy at test time, the practical difference in regimes between testing and training is not so clear in RL anyway if we have a world model.
  • The empirical results show improvement, but quite weak in most cases, and there does not seem to be a clear signal that the proposed method is a step increase in performance over existing baselines.
Questions for authors

I am really not convinced by the argument that existing work does not consider policy improvement at test time. First of all, the difference between training time regimes and inference time regimes when one has a world model becomes blurry. When using traditional MBRL algorithms (or MPC-RL style of algorithms like TD-MPC) nothing prevents one from improving the world model at test time. In fact, one would almost certainly have to do so, to try to close residual gaps with the training environment. Then, the novelty claim the authors make becomes vague. If the authors refer explicitly to improving the policy parameters, again, this is also naturally done at deployment (and furthermore one would have to do that almost certainly if one has any hopes of controlling real systems), and is done in both MBRL work (directly by improving the policy every time the world model changes) and also in MPC-related works like TD-MPC, TD-MPC2, etc. Furthermore in [1] authors already discuss the idea of continually improving the world model and the RL policy that serves as a planning priors.

Could the authors expand on this? How would they clarify their claim?

[1] Serra-Gomez, A., Ornia, D. J., Tirumala, D., & Moerland, T. (2025). A KL-regularization framework for learning to plan with adaptive priors. arXiv preprint arXiv:2510.04280.

3 comments

We thank the reviewer for the careful review and for noting that the objectives and methods are clear and well justified, that the approach is validated across a wide range of benchmarks, and that there are no substantive concerns about soundness. We address each concern below.


1. Clarifying the novelty claim: offline RL, and how our method differs from MBRL, TD-MPC and [1] : W1, Q

  • Offline RL and boundary between training and inference. We thank the reviewer for pressing on this point; it lets us state the claim precisely. We agree that in online model-based RL the boundary between training and deployment is blurry: data keeps arriving, and the model and policy are retrained as it does. Our paper studies a setting where the boundary is sharp: offline RL. There, a policy is learned entirely from a fixed dataset D\mathcal{D} collected beforehand by an unknown behavior policy, with no further interaction with the environment [Lange et al., 2012; Levine et al., 2020; Fu et al., 2021]. Offline RL is a well-established paradigm with a large body of work, including batch RL [Ernst et al., 2005; Lange et al., 2012], model-free methods such as BCQ, CQL, IQL, TD3+BC and ReBRAC [Fujimoto et al., 2019; Kumar et al., 2020; Kostrikov et al., 2022; Fujimoto & Gu, 2021; Tarasov et al., 2023], model-based methods such as MOPO and MOReL [Yu et al., 2020; Kidambi et al., 2020], and the standard D4RL benchmark [Fu et al., 2021]; see Levine et al. [2020] and Figueiredo Prudencio et al. [2024] for surveys. It is motivated by applications where online interaction is costly or unsafe, such as healthcare, recommender systems, robotics and autonomous driving [Gottesman et al., 2019; Afsar et al., 2022; Zhou et al., 2023; Kiran et al., 2022], and it is the protocol followed by every baseline we compare against.

     

    What separates training from inference is therefore not time but information. Training has access to D\mathcal{D}, and transitions encountered at deployment are not used as training data. Our method respects this: the world model, reward model, critic and initial policy are all trained on D\mathcal{D}, and no realized transition or reward is used to update them.

     

  • How the cited methods differ. In each example the reviewer gives, deployment-time improvement is driven by new environment data. MBRL improves the policy each time the model changes, and the model changes because new data arrives. TD-MPC and TD-MPC2 optimize action sequences with MPPI at decision time, while their policy prior is trained on replay-buffer batches. PO-MPC [1] is likewise online: it builds on TD-MPC2, trains its policy prior on replay-buffer batches, and at inference is computationally equivalent to TD-MPC2. What is new in our method is the combination: optimizing the policy parameters at the state the agent is in, using only computation through models trained on the offline dataset.

     

    MethodSettingOptimization at the current statePolicy parameters updated at deployment
    Dreamer, MBPOOnlineNone (the actor is a forward pass)Only by training on new data
    TD-MPC, TD-MPC2, PO-MPC [1]OnlineAction sequences (MPPI)Only by training on new data
    MOPO, MOReL, COMBOOfflineNoneNo
    POPLIN-POnlinePolicy-parameter perturbations (CEM)Yes, by gradient-free search
    OursOfflinePolicy parameters (gradients through the world model)Yes, with no new data

     

    A detailed comparison with each line of related work is in the appendix. Appendix E compares our method with the Dreamer and TD-MPC families (E.1, Table 26), with gradient-based and embedded differentiable MPC (E.2, Table 28) and with diffusion-based MPC (E.3, Table 30). Appendix A reviews offline and model-based offline RL more broadly. We will add [1], the table above and this discussion to the paper.

     

  • Why the distinction matters in practice. Without new data, the world model cannot be improved. A better model alone would not change behavior anyway, since a pretrained policy is a feedforward network that never queries the model. Our paper asks what is still possible in this regime. The only lever left is computation, and the question is where it helps. The conventional uses of extra computation do not:

    • Continuing to train the policy on its objective for the same number of gradient steps adds +0.42 (Appendix D.7).
    • Updating the policy at the visited state against the critic alone adds +0.12 (see our response to Reviewer v7c1).
    • TD-MPC2 trained on the same offline data scores far below standard offline baselines (Table 27).

     

    Updating the policy at the visited state through imagined rollouts adds +4.54 and improves every dataset. The reason is structural. Offline training optimizes the policy in expectation over the dataset's states, so the deployed policy is an amortized optimizer whose action at a particular state need not be the best one the learned models support (lines 68–75). Inference-time computation can optimize conditional on the state actually reached. This is the same mechanism behind inference-time scaling in LLMs [Wang et al., 2023; Yao et al., 2023; Muennighoff et al., 2025].

     


2. When new data is available.

In offline RL benchmarks, the deployment environment is the one that generated the data, so there is no residual gap to close. When new data is available, as in the reviewer's real-system scenario, the problem becomes offline-to-online RL, and our method remains useful there. On a real system, one would not take exploratory actions to collect data. Instead, the offline-trained policy is deployed for an extended period, the data it collects is used to fine-tune the world model and the policy, and the updated policy is redeployed [Matsushima et al., 2021]. The fine-tuned policy is again a single network trained in expectation over its data, so the same argument applies: computation spent at the state actually reached can still improve on it.

 

We tested this with MOPO [Yu et al., 2020], a model-based offline RL method:

  • MOPO (offline): SAC trained on uncertainty-penalized rollouts from a dynamics ensemble learned on the offline dataset.
  • MOPO (offline-to-online): the MOPO world model and policy fine-tuned on 20k additional real environment steps.
  • MOPO (offline-to-online) + DiffMPC (ours): our inference-time policy optimization applied to the fine-tuned world model, critic and policy.

 

DatasetMOPO (offline)MOPO (offline-to-online)MOPO (offline-to-online) + DiffMPC
halfcheetah-medium-replay70.5 ± 0.890.4 ± 5.8103.0 ± 2.9
walker2d-medium-replay80.2 ± 4.195.9 ± 2.499.0 ± 7.8
Average75.393.2101.0

 

Inference-time policy optimization improves on the fine-tuned model and policy by +12.6 on halfcheetah-medium-replay and +3.1 on walker2d-medium-replay, with no additional training and no additional data.

 

References

  • Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J., Levine, S., Finn, C., & Ma, T. (2020). MOPO: Model-based Offline Policy Optimization. NeurIPS.
  • Matsushima, T., Furuta, H., Matsuo, Y., Nachum, O., & Gu, S. (2021). Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization. ICLR.

Dear Reviewer,

As the discussion period is drawing to a close, we wanted to check whether you had a chance to review our responses above.

On novelty: in MBRL, TD-MPC/TD-MPC2 and PO-MPC [1], improvement at deployment comes from new environment data. Our method works in the offline setting. It updates the policy parameters at the current state using only models trained on the offline dataset using imagined rollouts. Details are in our response above.

On new data at deployment: when new data is available, as in the real-system scenario you describe, our method still helps. We fine-tuned a MOPO world model and policy on 20k additional real environment steps and then applied our method on top, with no further training or data. It raised the average from 93.2 to 101.0. Details are in our response above.

On the size of the gains: spending the same compute on more training adds only +0.42, and updating against the critic alone adds +0.12. Our method adds +4.54 and improves all 15 locomotion datasets.

At Reviewer rKc5's request, we evaluated on six more datasets: four from D4RL Adroit, a high-dimensional dexterous-manipulation domain with 24–30-dimensional actions, and two from NeoRL-2, a near-real-world control benchmark. Inference-time policy optimization improves the pretrained policy on all six. Details are in our response to Reviewer rKc5.

 

D4RL Adroit (pretrained policy: ReBRAC)

Taskdim(A\mathcal{A})zero shotDiffMPC (ours)
pen-cloned2480.22 ± 17.7498.77 ± 12.10
door-cloned280.08 ± 0.061.72 ± 0.94
hammer-cloned264.67 ± 0.0120.72 ± 9.08
relocate-cloned300.22 ± 0.230.74 ± 0.07
Average21.3030.49

 

NeoRL-2 (pretrained policy: TD3+BC)

Taskzero shotDiffMPC
DMSD60.0 ± 1.667.1 ± 0.7
Fusion54.9 ± 0.457.5 ± 1.2

If you have any remaining questions, we would be glad to respond to them.

Regards,
Authors

Reviewer_v7c1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies inference-time policy optimization for offline reinforcement learning. Starting from a pretrained policy, ITPO updates its parameters at the current state by differentiating an imagined return through a frozen world model, with a frozen critic providing the terminal value. DiffMPC implements this procedure with a differentiable diffusion model. FlowMPC replaces iterative diffusion sampling with a one-step MeanFlow model to reduce model-call cost. Experiments on D4RL locomotion and AntMaze evaluate improvement over the initial policy, alternative planners, design choices, and inference cost.

Critical strengths
  1. The method makes a useful design choice about what to optimize at deployment. Sections 3–4 combine state-specific policy-parameter updates with short model rollouts and terminal value bootstrapping, while keeping the learned dynamics and critic fixed. The contribution is this deployment procedure for pretrained offline policies, rather than a new model-based policy-gradient identity. Optimizing policy parameters also constrains the imagined actions through a shared feedback policy. The comparison summarized in Section 5 supports this choice: parameter optimization outperforms action-sequence optimization on 14 of 15 tasks using the same world model, critic, and horizon.

  2. The main empirical benefit is consistent across tasks and supported by relevant controls. Tables 1–2 improve the AntMaze average from 77.95 to 85.07 and the locomotion average from 93.9 to 98.5, with improvements on all 15 locomotion datasets. Section 5 also tests a different policy initialization and compares against other model-based planners. The additional offline-training control in Appendix D.7 helps distinguish state-specific deployment adaptation from simply spending more optimization steps on the original training objective. Together, these results make a credible case that the proposed procedure improves pretrained offline policies.

Critical weaknesses

W1. The experiments do not fully isolate the value of model lookahead within ITPO. The motivation in Sections 1 and 4 emphasizes improving decisions through predicted transitions and rewards, but the optimized objective also includes a learned terminal critic. A state-specific, critic-only policy update, corresponding to H=0H=0, is listed in the search space but its performance is not reported. The action-space comparison tests the optimization parameterization, and the additional offline-training control tests continued training; neither establishes how much improvement requires the world model. This leaves an important part of the method's explanation unresolved, although it does not invalidate the demonstrated benefit of ITPO as a whole.

W2. The practical value of the gains depends on a substantial computation tradeoff. Section 6 reports approximately 200 ms per decision for DiffMPC versus 0.4 ms for the initial policy on Hopper at H=5H=5 and K=10K=10. The paper clearly acknowledges this overhead. FlowMPC is a promising response, but its reported 8×8\times forward and 15×15\times forward/backward speedups concern individual world-model calls, rather than complete decisions. A return-versus-decision-latency comparison is therefore needed to assess the practical deployment claim in the conclusion. This weakens the claim. Combined with the unclear benefit of world model prediction, these concerns lead me to recommend a weak rejection.

Questions for authors

Q1 (W1). How much of ITPO's improvement remains when the policy is updated at the current state using only the frozen critic, with no world-model rollout? A comparison using the same initialization and a comparable update budget would clarify the contribution of model lookahead.

Q2 (W2). What are the complete per-decision latency and return of FlowMPC and DiffMPC at the configurations used for the main results? In particular, does the model-call speedup produce a useful return–latency tradeoff once policy updates and the rest of the planning procedure are included?

6 comments

We thank the reviewer for the careful review and for recognizing that our design choice is useful, that the empirical benefit is consistent across tasks and supported by relevant controls, and that the results make a credible case that the procedure improves pretrained offline policies. We address each of their questions below.


1. Value of model lookahead: critic-only updates (H=0H=0) : W1, Q1

  • Critic-only control. We ran the requested control. H=0H=0 updates the policy at the current state using only the frozen critic, ascending Qϕ(st,πψ(st))Q_\phi(s_t, \pi_\psi(s_t)), with the same initialization, the same per-dataset configuration as our main results (Table 8) and the same update budget. The only difference from DiffMPC is that no world-model rollout is used.

     

    DatasetReBRACH=0H=0 (critic only)DiffMPC
    halfcheetah-medium65.87 ± 1.364.97 ± 1.4170.05 ± 1.8
    halfcheetah-expert105.22 ± 0.92106.40 ± 1.32106.14 ± 2.1
    halfcheetah-medium-expert100.29 ± 8.2101.25 ± 4.40105.35 ± 1.8
    halfcheetah-medium-replay50.21 ± 0.750.08 ± 1.1659.89 ± 1.2
    halfcheetah-full-replay82.19 ± 0.980.74 ± 0.7785.15 ± 0.9
    hopper-medium102.05 ± 0.9102.85 ± 0.31103.38 ± 0.35
    hopper-expert104.73 ± 6.94104.86 ± 4.64105.06 ± 5.02
    hopper-medium-expert105.70 ± 8.3105.58 ± 7.68107.42 ± 5.30
    hopper-medium-replay95.82 ± 7.989.93 ± 15.63103.14 ± 0.45
    hopper-full-replay107.49 ± 0.4103.69 ± 12.41108.77 ± 0.64
    walker2d-medium84.78 ± 1.886.91 ± 0.9388.91 ± 0.6
    walker2d-expert112.15 ± 0.2114.70 ± 0.28116.65 ± 0.41
    walker2d-medium-expert111.81 ± 0.3113.28 ± 0.73115.76 ± 1.04
    walker2d-medium-replay80.67 ± 5.884.90 ± 4.6295.87 ± 1.19
    walker2d-full-replay100.16 ± 8.8100.82 ± 11.66105.76 ± 2.91
    Average93.9494.0698.49

     

  • Result. Without the world model, almost none of the improvement remains. The critic-only update leaves the average essentially unchanged (93.94 → 94.06), and it falls below the pretrained policy on 6 of the 15 datasets, most sharply on hopper-medium-replay (−5.89) and hopper-full-replay (−3.80). DiffMPC outperforms it on 14 of 15 datasets, by up to 13.2 points (hopper-medium-replay). Since both methods apply the same updates at the same states, the gain comes from the model lookahead, not from the state-specific update alone. This matches the motivation in Section 1: the critic is a long-horizon quantity that is hard to estimate accurately, whereas the dynamics and reward are local, one-step quantities, and lookahead shifts the objective from the former toward the latter.

     

  • Separating the ingredients. Together with the compute-matched training control (Appendix D.7), this separates the three ingredients. Extra optimization on the training objective adds +0.42. State-specific optimization against the critic alone adds +0.12. State-specific optimization through imagined rollouts adds +4.54. We will add this comparison to the paper.

2. Complete per-decision latency and the return–latency tradeoff : W2, Q2

  • Protocol. We measured the full wall-clock time per decision on a single A100: everything executed at each environment step, including imagined rollouts, reward and critic evaluation, backpropagation, the policy updates and action selection. Both methods run at the configuration used for our main results (Table 8). For a fair comparison, the flow model uses exactly the same architecture as the diffusion model, and both are run with the same inference-time hyperparameters (HH, EE, η\eta, MM), so only the sampler differs. MeanFlow can also be sampled with N>1N > 1 steps, which gives a single model with a return–latency knob.

     

    halfcheetah-medium-replay

    MethodReturnGain over zero-shotTime per decision
    Zero-shot (ReBRAC)51.04 ± 0.65–1.9 ms
    FlowMPC, N=1N=159.97 ± 1.45+8.963.1 ms
    FlowMPC, N=2N=260.29 ± 1.48+9.277.4 ms
    FlowMPC, N=4N=460.86 ± 1.19+9.8105.6 ms
    FlowMPC, N=10N=1061.05 ± 1.26+10.0188.4 ms
    DiffMPC59.89 ± 1.25+8.9226.9 ms

     

    walker2d-medium-replay

    MethodReturnGain over zero-shotTime per decision
    Zero-shot (ReBRAC)80.67 ± 5.80–2.0 ms
    FlowMPC, N=1N=189.38 ± 6.45+8.763 ms
    FlowMPC, N=2N=290.94 ± 8.48+10.377 ms
    FlowMPC, N=4N=492.30 ± 7.14+11.6105 ms
    FlowMPC, N=10N=1094.02 ± 4.87+13.4190 ms
    DiffMPC95.87 ± 1.20+15.2228 ms

     

  • Result. The model-call speedup carries through to complete decisions and yields a useful return–latency tradeoff. One-step FlowMPC cuts the time per decision from 228 ms to 63 ms, 3.6× faster than DiffMPC end to end.

    • On halfcheetah-medium-replay, one-step FlowMPC matches DiffMPC's return (59.97 vs. 59.89) at 28% of its latency. Every multi-step setting exceeds DiffMPC while remaining faster, up to 61.05 at N=10N=10.
    • On walker2d-medium-replay, one-step FlowMPC already delivers +8.7 over the zero-shot policy, 57% of DiffMPC's gain, at 28% of its latency. Return then rises steadily with NN, reaching +13.4 at N=10N=10, 88% of DiffMPC's gain, while still faster.

     

    The number of flow steps gives practitioners a smooth way to choose their operating point between return and latency, and on halfcheetah-medium-replay the fastest setting gives up nothing.


3. Additional benchmarks: Adroit and NeoRL-2

  • Six more datasets. At Reviewer rKc5's request, we evaluated on six more datasets: four from D4RL Adroit, a high-dimensional dexterous-manipulation domain with 24–30-dimensional actions, and two from NeoRL-2, a near-real-world control benchmark. Inference-time policy optimization improves the pretrained policy on all six. Details are in our response to Reviewer rKc5.

     

    D4RL Adroit (pretrained policy: ReBRAC)

    Taskdim(A\mathcal{A})zero shotDiffMPC (ours)
    pen-cloned2480.22 ± 17.7498.77 ± 12.10
    door-cloned280.08 ± 0.061.72 ± 0.94
    hammer-cloned264.67 ± 0.0120.72 ± 9.08
    relocate-cloned300.22 ± 0.230.74 ± 0.07
    Average21.3030.49

     

    NeoRL-2 (pretrained policy: TD3+BC)

    Taskzero shotDiffMPC
    DMSD60.0 ± 1.667.1 ± 0.7
    Fusion54.9 ± 0.457.5 ± 1.2
Reviewer_v7c1OpenReview ↗

Response to authors

Thanks for the additional experiments. The H=0H=0 control addresses my main concern about the role of model lookahead. This gives much clearer evidence that model rollouts contribute beyond state-specific critic optimization. I consider W1 resolved.

The new end-to-end measurements also provide the return–latency comparison. One-step FlowMPC takes about 63 ms versus 227–228 ms for DiffMPC, with comparable return on halfcheetah-medium-replay and a lower return on walker2d-medium-replay. Increasing the number of flow steps recovers more of the Walker2d gain at a higher cost. These results make the tradeoff much clearer, although the overhead relative to the original policy remains substantial.

There is one timing detail I would like to clarify. Table 8 uses E=5E=5 for halfcheetah-medium-replay and E=1E=1 for walker2d-medium-replay. Appendix D.7 states that these correspond to 20 and 4 policy updates per decision, respectively. Yet the new decision times are almost identical across the two tasks for both DiffMPC and FlowMPC. How many updates were actually included in each measurement, and what accounts for the similar times? Please also confirm that the reported returns use the configurations that were timed. If these differ from Table 8, please list the settings used.

With the remaining concern clarified, I would support moving to a positive recommendation.

Reviewer_rKc5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an MPC-based approach to offline RL. The authors use a diffusion model to learn the system dynamics using the offline dataset. During rollouts, the model and Q-function are used to inform policy updates by imagining future states. The proposed approach outperforms SOTA offline RL baselines on a narrow subset of the D4RL benchmarks.

Critical strengths

The paper is very clear and easy to read. The main ideas are very well presented, and its position within the broader literature is clearly established. Overall, I enjoyed reading the paper very much.

The paper compares the proposed method against established baselines, showcasing good results across the presented benchmarks.

Critical weaknesses

I have two main criticisms of the paper:

My first concern is that the choice of adapting the controller feels somewhat arbitrary and not fully justified. It would be useful to contrast this with, e.g., updating the model on the fly. Moreover, it makes the implicit assumption that the base model is sufficiently accurate to inform good policy updates within the regimes where the policy will operate. However, it is not clear that the updated policy will not steer the system to unseen regimes, and how this will affect model performance (and policy updates).

My other, and arguably more significant concern, is that the chosen benchmarks are relatively simple and insufficient. D4RL includes a larger breadth of challenging benchmarks that should be used to decisively establish how the proposed method compares against other baselines. In its current format, the chosen benchmarks present a picture that is not sufficiently informative.

Questions for authors

The whole approach has an adaptive control flavor to it. While the timescales for adaptive control and RL-based policies are often different, It would be worthwhile to position it within this field of research.

What is the motivation for adapting the controller vs adapting the model on the fly?

Is the model sufficient to inform policy updates in all regimes of the state space?

How good is ood model generalization?

Have the authors experimented with other methods to ensure that operation stays close to the data distribution? Could the policy update include somethign of the sort?

How does the method perform on other D4RL benchmarks?

4 comments

We thank the reviewer for the careful review and for the kind words: that the paper is clear and easy to read, that its position within the broader literature is clearly established, and that it shows good results against established baselines. We address each concern below.


1. Why adapt policy and not the model, relation to adaptive control : W1 (first part), Q1, Q2

  • Offline RL: We first emphasize that the setting we study is offline RL: a policy is learned entirely from a fixed dataset D\mathcal{D}, with no further interaction with the environment [Lange et al., 2012; Levine et al., 2020; Fu et al., 2021]. Real transitions and rewards encountered at deployment are not used as training data, so adapting the model or the policy from them is outside this paradigm. All our components are trained on D\mathcal{D} alone. Moreover, adapting only the model would not change behavior: without ITPO, the pretrained policy never queries the world model.

     

  • Motivation to adapt policy: With no new data, the only lever left at deployment is computation, and our paper asks where it helps. Spending it on training does not: training ReBRAC for as many extra gradient steps as our method takes over an episode changes its average score by only +0.42 (Appendix D.7, Table 25). Spending the same budget at inference time improves the average by +4.54 and improves every dataset.

     

    The reason is structural. Offline training optimizes the policy only in expectation over the dataset's states, so its action at a particular state need not be the best one the learned models support (lines 68–75). Training-time compute can only anticipate the state sts_t in expectation, while inference-time compute can optimize conditional on it. The same mechanism makes inference-time scaling effective for LLMs [Wang et al., 2023; Yao et al., 2023]. This motivates adapting the controller: the gap lies in the decision at sts_t, which the policy produces, so we use the world model to imagine the consequences of acting from sts_t and update the policy parameters to improve the imagined return.

     

  • Relation to adaptive control. Adaptive control distinguishes indirect schemes, which re-identify the plant online and redesign the controller from the updated model (e.g., self-tuning regulators), from direct schemes, which adjust the controller parameters themselves (e.g., MRAC) [Åström & Wittenmark, 1995; Ioannou & Sun, 1996]. Our method is closest to a direct scheme: the controller parameters are updated at every control step and carried forward within the episode, so adaptation happens on the timescale of adaptive control rather than that of RL training. It differs in what drives the update. Adaptive control reacts to a mismatch between the plant and the nominal model, observed through feedback. In offline RL the plant is the one that generated the data, and our update is driven instead by a model-based lookahead from the current state. We will add this discussion to the related work. Adapting the model from observed transitions corresponds to indirect adaptation, which requires new data (and is therefore outside the offline RL paradigm that is the focus of this paper). To show that our method extends to that case as well, we consider it next.

     

  • When new data is available. Then the problem becomes offline-to-online RL, and our method remains useful there. On a real system, one would not take exploratory actions to collect data. Instead, the offline-trained policy is deployed for an extended period, the data it collects is used to adapt the world model, as the reviewer suggests, and to fine-tune the policy, and the updated policy is redeployed [Matsushima et al., 2021]. The fine-tuned policy is again a single network trained in expectation over its data, so the same argument applies: computation spent at the state actually reached can still improve on it.

     

    We tested this explicitly with a model based offline RL method, MOPO [Yu et al., 2020]:

    • MOPO (offline): SAC trained on uncertainty-penalized rollouts from a dynamics ensemble learned on the offline dataset.
    • MOPO (offline-to-online): the MOPO world model and policy fine-tuned on 20k additional real environment steps.
    • MOPO (offline-to-online) + DiffMPC (ours): our inference-time policy optimization applied to the fine-tuned world model, critic and policy.

     

    DatasetMOPO (offline)MOPO (offline-to-online)MOPO (offline-to-online) + DiffMPC
    halfcheetah-medium-replay70.5 ± 0.890.4 ± 5.8103.0 ± 2.9
    walker2d-medium-replay80.2 ± 4.195.9 ± 2.499.0 ± 7.8
    Average75.393.2101.0

     

    Inference-time policy optimization improves on the fine-tuned model and policy by +12.6 on halfcheetah-medium-replay and +3.1 on walker2d-medium-replay, with no additional training and no additional data.

2. Model accuracy under the updated policy, and staying close to the data : W1 (second part), Q3, Q4, Q5

  • Is the model sufficient in all regimes? The model does not need to be accurate in all regimes of the state space. It is queried only within HH steps of states the agent actually visits, under a policy initialized from the pretrained one. The relevant question is therefore whether the inference-time updates steer the agent into regions where the model becomes unreliable. We measured this directly.

     

  • Model error on the policy trajectories. We roll out full episodes under the frozen pretrained policy and under DiffMPC. Along the way, we compare the world model's next-state samples, and the reward model's predictions, with what the environment actually returns. If the updates pushed the policy into regions where the model is unreliable, the error under DiffMPC would exceed that under the frozen policy and grow over the episode.

     

    DatasetState RMSE (frozen)State RMSE (DiffMPC)Reward RMSE (frozen)Reward RMSE (DiffMPC)
    halfcheetah-medium0.23700.21760.36300.2550
    halfcheetah-expert0.30710.31660.36880.3119
    halfcheetah-medium-expert0.16570.31730.17940.2330
    halfcheetah-medium-replay0.20580.24120.07320.1144
    halfcheetah-full-replay0.20480.15270.14720.2602
    hopper-medium0.03490.02660.01300.0171
    hopper-expert0.03630.02450.01440.0114
    hopper-medium-expert0.02720.02900.01450.0147
    hopper-medium-replay0.02390.02960.01630.0195
    hopper-full-replay0.02710.02090.01670.0133
    walker2d-medium0.14820.15340.05600.0259
    walker2d-expert0.06600.09180.02230.0348
    walker2d-medium-expert0.08610.15790.03030.0469
    walker2d-medium-replay0.15060.18310.03550.0360
    walker2d-full-replay0.18540.17360.02780.0403
    Average0.12710.14240.09190.0956

     

    Averaged over the suite, the error under DiffMPC is close to the error under the frozen policy, for both the world model (0.142 vs. 0.127) and the reward model (0.096 vs. 0.092). In other words, the states visited under DiffMPC are ones the model predicts about as well as the states visited by the pretrained policy, so the method does not depend on out-of-distribution generalization. DiffMPC has the lower state error on 6 of the 15 datasets. The largest increases are on halfcheetah-medium-expert and walker2d-medium-expert, where DiffMPC still improves the return by +5.06 and +3.95.

     

    The gap also does not widen systematically over an episode, as the hopper-medium-expert episode below shows.

     

    StepWorld model (frozen)World model (DiffMPC)Reward model (frozen)Reward model (DiffMPC)
    00.0921 ± 0.03390.0921 ± 0.03390.0332 ± 0.00730.0333 ± 0.0073
    3000.0249 ± 0.01370.0217 ± 0.01060.0091 ± 0.00610.0077 ± 0.0057
    6000.0218 ± 0.01550.0193 ± 0.00930.0156 ± 0.00870.0084 ± 0.0066
    9000.0162 ± 0.00770.0199 ± 0.01050.0058 ± 0.00440.0059 ± 0.0052

     

  • Staying close to the data. Several features of the method keep the policy near the data support, and the measurements above show they suffice.

    • First, every environment step re-initializes the imagined rollout at the true state sts_t, so the model is never unrolled for more than HH steps before being reset to ground truth, and errors cannot compound over the episode.
    • Second, the policy starts from the behavior-regularized pretrained policy, and the terminal critic QϕQ_\phi is a behavior-regularized ReBRAC critic, so the objective being maximized discourages leaving the data support.
    • Third, the policy takes only a few small gradient steps per state (EE passes with step size η\eta, tuned on a held-out seed) and is reset to the pretrained policy at the start of every episode.

     

    The objective also readily accepts an explicit term: a proximal penalty λ∑j∥πψ(s~j)−πψ0(s~j)∥2\lambda \sum_j \lVert \pi_\psi(\tilde s_j) - \pi_{\psi_0}(\tilde s_j) \rVert^2 toward the pretrained policy, mirroring ReBRAC's behavior-cloning penalty, or a model-uncertainty penalty on the imagined rewards, as in MOPO [Yu et al., 2020]. Given the measurements above, our experiments did not require one; we will discuss both options in the paper.

3. Results on additional benchmarks: Adroit and NeoRL-2 : W2, Q6

  • Benchmarks. We thank the reviewer and agree that breadth matters. The submission covers all 15 Gym-MuJoCo datasets and all 6 AntMaze datasets. AntMaze is sparse-reward, long-horizon navigation, and its large mazes are among the hardest tasks in D4RL. To broaden the evaluation further, we added two domains that differ from these along the axes the reviewer raises: D4RL Adroit, for high-dimensional control, and NeoRL-2, for near-real-world control.

     

  • D4RL Adroit. Adroit requires controlling a simulated dexterous hand to spin a pen, open a door, drive a nail with a hammer and relocate a ball [Rajeswaran et al., 2018]. Its action spaces are 24–30-dimensional, against 3–6 in Gym-MuJoCo and 8 in AntMaze. The cloned datasets mix human demonstrations with rollouts of an imitation policy, so coverage is narrow. We follow the same protocol as for Gym-MuJoCo: ReBRAC as the pretrained policy and critic, a world model and reward model trained on the same offline dataset, and inference-time policy optimization through imagined rollouts.

     

    Taskdim(A\mathcal{A})zero shotDiffMPC (ours)
    pen-cloned2480.22 ± 17.7498.77 ± 12.10
    door-cloned280.08 ± 0.061.72 ± 0.94
    hammer-cloned264.67 ± 0.0120.72 ± 9.08
    relocate-cloned300.22 ± 0.230.74 ± 0.07
    Average21.3030.49

     

    DiffMPC improves the pretrained policy on all four tasks and raises the average from 21.30 to 30.49. The gains are large where the pretrained policy has signal to work with: +18.6 on pen-cloned and +16.1 on hammer-cloned, where the score more than quadruples. On door-cloned and relocate-cloned, the pretrained policy scores near zero, so there is little for inference-time optimization to refine; scores stay low but do not degrade.

     

  • NeoRL-2. NeoRL-2 [Gao et al., 2025] is a recent benchmark of near-real-world control tasks. It was designed to capture properties of real deployments that D4RL lacks: conservative data, delayed action effects, external factors, safety constraints and very limited data. We evaluate on two of its tasks. In DMSD the dataset is collected by a deterministic, PID-style controller, so it covers a narrow region of the state space. In Fusion the dataset is very small, reflecting expensive experimentation. Here we use TD3+BC [Fujimoto & Gu, 2021] as the pretrained policy, with the hyperparameters tuned for NeoRL-2 in the benchmark paper [Gao et al., 2025].

     

    Taskzero shotDiffMPC
    DMSD60.0 ± 1.667.1 ± 0.7
    Fusion54.9 ± 0.457.5 ± 1.2

     

    Inference-time policy optimization improves TD3+BC on both tasks.

     

  • Summary. With these additions, the evaluation covers four domain families: locomotion, sparse-reward navigation, high-dimensional dexterous manipulation and near-real-world control. Inference-time policy optimization improves the pretrained policy in all. We will add both results to the paper.

 

References

  • Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., & Levine, S. (2018). Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. RSS.
  • Gao, S., Tu, Z., Qin, R.-J., Sun, Y.-H., Chen, X.-H., & Yu, Y. (2025). NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarios. arXiv:2503.19267.
  • Åström, K. J., & Wittenmark, B. (1995). Adaptive Control (2nd ed.). Addison-Wesley.
  • Ioannou, P. A., & Sun, J. (1996). Robust Adaptive Control. Prentice Hall.
  • Matsushima, T., Furuta, H., Matsuo, Y., Nachum, O., & Gu, S. (2021). Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization. ICLR.
  • Other references are as cited in the paper.

Dear Reviewer,

As the discussion period is drawing to a close, we wanted to check whether you had a chance to review our responses.

We ran the additional benchmarks you asked for. On the four D4RL Adroit tasks, with action dimensions of 24–30 against 3–6 in Gym-MuJoCo, DiffMPC improves over the pretrained ReBRAC policy on every task and raises the suite average from 21.30 to 30.49. It also improves TD3+BC on both NeoRL-2 tasks, DMSD and Fusion.

We also showed that model error under DiffMPC stays close to that under the frozen pretrained policy, so the method does not depend on out-of-distribution generalization. We positioned the method within adaptive control and showed that when new data is available to adapt the model, DiffMPC still improves a fine-tuned MOPO model and policy, raising the average from 93.2 to 101.0.

If you have any remaining questions, we would be glad to respond to them.

Regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.