acceptodds
Under review as a conference paper at ICLR 2027

Q-Learning With World Models

Abstract

Off-policy reinforcement learning (RL) has become increasingly sample-efficient, enabling applications such as RL fine-tuning of Vision-Language-Action models into reliable, high-performing policies. World models offer a further lever for sample efficiency, as they predict state changes rather than actions alone, but their success has largely been confined to supervised policy learning. Prior model-based RL methods often optimize the policy or value function directly on imagined rollouts, which is prone to compounding bias and struggles to scale to large, high-dimensional problems such as real-world robotics, a problem that worsens with task horizon and visual complexity. In this work, we instead ask whether we can leverage world models directly on top of standard Q-learning to improve performance, while remaining trained and grounded in the real, online setting. We propose QWM, a framework that leverages world models to perform test-time search over imagined trajectories on top of Q-learning to select high-value actions during both online rollouts and evaluation. Since the policy and value function are trained only on real transitions, QWM avoids compounding model bias while still gaining the sample-efficiency benefits of predictive search. On challenging manipulation benchmarks Robomimic and LIBERO, QWM significantly outperforms strong prior state-of-the-art methods on both sample efficiency and performance.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, World Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dTS8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a method using world models and estimated Q functions for MCTS to improve the quality of generated actions in RL setups. Specifically, their world model is only used in test-time searching, with Q-functions to determine the value of each actions. They validate the performance of their proposed method against both model-free and model-based baselines in a range of challenging manipulation tasks. Results show that their proposed method has substantial performance improvement against existing baselines on these selected tasks, which demonstrate the effectiveness of their proposed method.

Critical strengths
  • Clear presentation: The overall presentation of the paper is clear and easy to follow. The proposed method is straightforward and easy to understand the details of the method.

  • Superior performance: The authors demonstrate substantial performance improvement against existing baselines on a variety of challenging tasks on robotic manipulation, which demonstrate the effectiveness of their proposed approach.

Critical weaknesses
  • World model scale is incomparable: The proposed method leverages a 5B WAN video generation model as a world model for test-time tree search in pixel level experiments, while standard model-based baselines such as TD-MPC2 and EfficientZeroV2 only leverages MLP-based world models with CNN encoders. From my knowledge a standard variant of TD-MPC2 is containing only 5M of parameters while the larges one is around 200M, which is significantly smaller than the WAN model with 5B parameters. For pixel-level experiments, it's questionable that the performance gain is originated from the use of larger pretrained video world model or is from the designed learning algorithm itself.

  • More detailed explanation of the proposed method against existing baseline: The existing baseline EfficientZeroV2 is also leveraging a tree-search algorithm during test-time planning. Please differentiate the proposed algorithm with their proposed tree search and show why the proposed method is better.

Questions for authors

See the weakness section.

1 comment
AuthorsOpenReview ↗

World-Model Scale and Search Strategy Comparisons

Thank you for your thoughtful review and for appreciating the clarity of our presentation and the performance gains! We provide clarifications below on separating the contribution of the search rule from the capacity of the world model, and will incorporate them into the paper.

World-model scale

We would like to clarify that our main comparisons with TD-MPC2 and EfficientZero V2 are on state-based Robomimic. There, all three methods operate on low-dimensional observations with compact state-based models: QWM uses a residual MLP, and the native baselines use latent dynamics. QWM's gains over the model-based baselines in this setting therefore do not come from a larger world model.

Wan2.2-5B is used only in the separate pixel-based LIBERO experiments, and only during online data collection. At evaluation, the trained EXPO policy and critic act without video search, so the deployed agent does not use the video model. We agree that the pixel comparison measures the combined effect of adding the pretrained video world model and QWM search. Isolating the search contribution in pixels would require a shared video checkpoint, learner, and model-prediction budget, and we will state this explicitly as a limitation in the revision.

What QWM adds beyond EfficientZero V2

EfficientZero V2 uses its model to construct improved policy and value targets. It allocates simulations among root actions via sampled Gumbel search and Sequential Halving, and reuses search estimates for learning. In contrast, QWM uses its model to guide the collection of real experience for an off-policy learner, and the actor and critic learn only from demonstrations and real environment transitions.

QWM also addresses a different bottleneck under sparse rewards. Two short rollouts can receive the same zero predicted reward even when one sets up a successful grasp or insertion. We use discounted intermediate critic scores to distinguish such prefixes, prune branches before deeper model calls, and include those scores in the final action ranking.

Comparison with EfficientZero V2 (Gumbel Search)

We thank the reviewer for this suggestion. Both QWM and EfficientZero V2 employ world-model-based tree search, but they differ in the search strategies. EfficientZero V2 adopts Gumbel-based tree search, whereas QWM employs our proposed Q-guided search strategy. To directly compare these approaches, we integrate Gumbel search into the EXPO learning framework for both online data collection and evaluation.

Both methods use the same types of EXPO actor, edit policy, and critic, and are trained online using real environment transitions. Gumbel–EXPO uses a world model pretrained on demonstration data, which remains frozen throughout online training.

For Gumbel–EXPO, we adopt a training configuration with UTD 20, batch size 256, and 50% demonstration data in the replay buffer. The following table reports the mean success rates across three independent training seeds at matched environment steps.

TaskEnvironment StepsGumbel–EXPOQWM
Lift100k10.7%98.7%
Can100k85.3%99.3%
Square100k76.7%93.3%
Tool Hang250k15.3%97.3%

At the reported training steps, QWM achieves higher mean success rates than Gumbel–EXPO across all four tasks.

Search Budget and Decision Latency

We compare the two search methods with matched search depth, numbers of predicted states, and Q queries. Under these matched search budgets, Gumbel search incurs approximately 1.8–2.8× higher decision latency than QWM.

TaskLookahead DepthPredicted Transitions per DecisionQ Queries per DecisionQWM / Gumbel median Latency (ms)
Lift21446566.26 / 11.71
Can21442725.94 / 10.80
Square21446565.94 / 11.24
Tool Hang41,1681,6809.11 / 25.87

Overall, these results highlight two advantages of QWM: (1) higher final success rates across all four tasks under the evaluated online training configurations, and (2) lower decision latency under matched search budgets and shared model weights. These results combine to show the effectiveness of the proposed tree search over prior methods.

Reviewer_wdbuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes QWM, a framework that augments standard off-policy Q-learning with a test-time world-model tree search to improve action selection during both online data collection and evaluation. Instead of training policies or targets on imagined rollouts (which can compound model bias), QWM keeps learning grounded in real transitions and uses a learned world model purely for short-horizon lookahead to re-rank candidate actions via a combination of Q-value and rollout-based estimates. On Robomimic (state) and LIBERO (pixels), QWM reports consistent gains in sample efficiency and final success rates over model-free and model-based baselines and shows robustness when built atop two different Q-learning methods (EXPO and RLPD).

Critical strengths

The problem is specific and the wrapper is simple. On manipulation tasks, the world model is used only for action selection, while the original policy and critic losses stay on real transitions.

Critical weaknesses
  1. The equations and the implementation are not the same algorithm. Equations (6) and (9) include a reward model rψr_\psi, while Section 4.4 says that no reward model is learned under sparse rewards. The main text uses max aggregation at the leaves, while Table 2 says Mean. Pruning ranks paths by a discounted sum of Q-values, Σ\Sigma, rather than by the combined value in Equation (8). The appendix further says depth 2 works best, while Table 2 sets the main-experiment depth to 4. Readers cannot tell which procedure the main experiments ran.

  2. The experiments do not isolate the world model from extra search. EXPO already selects the maximum QQ among NN candidates. QWM adds more QQ evaluations and a deeper lookahead, so the reported gains may come from additional search compute. There is no control with the same number of candidates and the same number of QQ queries that never calls the world model. The model-based comparison does not fill this gap: TD-MPC2 and EfficientZero V2 stay near zero on Tool Hang, Square, and Can, under a different UTD and an unspecified tuning protocol.

Questions for authors
  1. Do the main results use depth 2, or depth 4 as in Table 2? Is leaf aggregation max or mean? Does the factor 12\frac{1}{2} in Equation (8) exist in the code? In the implementation, is rψr_\psi the true reward, a learned reward, or identically zero, and how does a terminal sparse reward enter a depth-4 search score?

  2. The state model is the deterministic map s+Δψ(s,a)s+\Delta_\psi(s,a), but Table 2 samples K=8K=8 next states per action. How can those predictions differ? With J=1J=1, only one path survives at each level. Is the procedure in the main experiments the tree in Figure 2, or greedy single-path lookahead?

  3. Please add a best-of-NN QQ-selection baseline matched in compute to QWM and without a world model. Report the number of QQ queries per environment step, the number of world-model calls, and the wall-clock time. \item How many seeds were used? What do the shaded regions denote? Please use the same definition on every curve. In Figure 5, RLPD and QWM-RLPD both stay near zero on Square; please reconcile that plot with the claim that sample efficiency improves on all three tasks. In Figure 7, many curves overlap and the preferred λ\lambda and NN change with the task; what result supports treating λ\lambda as the most important factor?

  4. Did TD-MPC2 and EfficientZero V2 use the same demonstrations, the same number of online steps, and retuning under sparse rewards? EfficientZero V2 uses the official UTD of 0.4, while QWM uses a UTD of 20. If the protocols differ, please weaken the claim of improvement over model-based reinforcement learning, or add a matched comparison.

  5. What are N,K,D,J,λN,K,D,J,\lambda in the pixel experiments? Inference uses one denoising step at 128×128128\times 128 and keeps only the second generated frame, and Figure 10 shows growing error at later frames. Is the world model updated online or frozen after demonstration training, and why is search disabled at evaluation? The introduction discusses VLA fine-tuning; please state that the experiments do not include a VLA or a real robot.

  6. Please number the model-free comparison plot on page 7, and correct the figure references in Sections 5.2 and 5.3.

  7. The computational and inference cost of QWM is never quantified. The discussion only states that tree search adds non-trivial overhead relative to a standard policy forward pass. Please report this cost against every baseline shown in the corresponding plots, at least EXPO and RLPD, and also TD-MPC2 and EfficientZero V2 where those methods are compared. Give the numbers separately for the state-based and pixel settings, and separately for online data collection and evaluation. For each method, report per environment step: the number of policy samples, the number of QQ-function evaluations, the number of world-model forward passes, and the wall-clock decision time on stated hardware. For the video world model, also report the number of denoising steps and the generated resolution. These measurements are required to judge whether the reported gains in sample efficiency remain favorable once the additional computational and inference cost is taken into account.

2 comments
AuthorsOpenReview ↗

Part I — Responses to W1–W2 and Q1–Q4

We appreciate the reviewer's detailed feedback. We organize our response into two parts: (I) algorithm clarification, controlled comparisons, and baseline fairness; and (II) pixel experiments and computational costs. We will incorporate these clarifications and additional experimental results into the revised paper.

Equations and Implementation

Equations (6) and (9) present a general formulation that allows a learned reward model. However, in our sparse-reward experiments, no reward model is used, and the explicit predicted-reward term is zero. Instead, real environment rewards enter the critic's TD targets and influence planning through learned Q-values. This design avoids relying on potentially inaccurate reward predictions under sparse feedback. The main comparisons in Section 5.1 use a shared configuration of D=4, J=1, and =0.1.

Best-of-N Control with Equal Q-Evaluation Budgets

Following the reviewer’s suggestion, we compare QWM with an EXPO best-of-N baseline under matched Q-query budgets. Best-of-N scores equal numbers of diffusion-generated and edited actions at the current state, without using the world model.

TaskStepsQ QueriesWM Batches (Trans.)Success (%) (Best-of-N / QWM)Latency (ms) (Best-of-N / QWM)
Lift100k6562 (144)78.7 / 98.73.49 / 6.26
Can100k2722 (144)84.7 / 99.33.36 / 5.94
Square100k6562 (144)94.0 / 93.33.29 / 5.94
Tool Hang250k1,6804 (1,168)90.0 / 97.33.86 / 9.11

Best-of-N makes zero world-model calls. A Q query counts one state–action pair scored by the critic. Success rates are averaged over three training seeds, with 50 evaluation episodes per seed and checkpoint. Decision times are medians from a shared full-JIT benchmark for each task on an NVIDIA A800.

Under matched Q-query budgets, QWM improves success rates by 20.0, 14.6, and 7.3 percentage points on Lift, Can, and Tool Hang, respectively, while performing comparably on Square. QWM takes approximately 1.8–2.4 times as long per decision as best-of-N. These results suggest that world-model lookahead can provide benefits beyond simply evaluating more candidate actions, at the cost of additional decision time.

Seeds, shaded regions, and Figure 5 (RLPD)

Each curve reports the mean success rate across three independent training seeds, with shaded regions representing the standard error of the mean at each evaluation checkpoint.

Figure 5 (Square). Although both methods initially show low success rates, QWM-RLPD learns faster, achieving 44.7% success at 100k steps, compared to 30.0% for RLPD. By 200k steps, the gap narrows (89.3% vs. 87.3%), indicating that QWM primarily improves early-stage sample efficiency rather than final performance.

Model-based baseline protocols

Baseline protocol and UTD. TD-MPC2 and EfficientZero V2 use the same demonstration subsets and a 50% demonstration replay ratio as QWM, while their native models, policy, and value are trained online. The original EZV2 paper reports UTD = 1; our Robomimic adaptation uses UTD = 0.4 to approximately match QWM's wall-clock training time. On an A800 node, 100k Lift transitions take approximately 8.2 hours for EZV2 at UTD 0.4 versus 6 hours for QWM; UTD 20 is estimated to take 275.7 hours based on observed throughput.

Baseline tuning. We explored multiple configurations for both native baselines. For TD-MPC2, the Lift ablation varied the discount factor, whether to terminate on success, and planning horizon. For EfficientZero V2, we varied the discount factor, termination on success, and reward loss coefficient. We then evaluated selected configurations across additional tasks and sparse- and dense-reward settings. We will document the tested configurations and online interaction budgets in the revision.

EfficientZero V2–style search under EXPO. To complement the native baseline results, we integrate an EfficientZero V2–style sampled Gumbel search into EXPO for online data collection and evaluation. Both methods use the same types of actor, edit policy, and critic. Gumbel–EXPO uses a frozen demonstration-pretrained world model, UTD 20, batch size 256, and 50% demonstration replay. The following table reports mean success rates across three independent training seeds at matched environment steps.

TaskStepsGumbel–EXPO (%)QWM (%)
Lift100k10.798.7
Can100k85.399.3
Square100k76.793.3
Tool Hang250k15.397.3

QWM achieves higher success rates across all four tasks, supporting the effectiveness of its Q-guided search strategy under the evaluated online learning configurations.

AuthorsOpenReview ↗

Part II — Responses to Q5–Q7

Pixel settings and evaluation

The demonstration-trained video world model is frozen and used only during data collection. We disable search during evaluation to reduce decision latency, as video-model inference is computationally expensive; the learned policy and critic benefit from the search-guided data collected during training. The video model uses one denoising step at 128 × 128, retaining the second generated frame. Our pixel experiments use a ResNet-based EXPO policy in simulation; VLA and real-robot validation remain future work.

Search configuration. All five LIBERO tasks use N = 8, K = 8, D = 2, and J = 1. The recorded λ values are 0.2 for Tasks 28 and 79, and 0.1 for Tasks 2, 29, and 60.

Figure 10. Each prediction is conditioned on its corresponding input observation, rather than rolled out from a shared initial frame. Thus, later panels do not show accumulated long-horizon prediction errors. Could the reviewer clarify which visual discrepancies are concerning and their suspected source? We will clarify the prediction setting in the caption.

Figure numbering

We will number the model-free comparison plot and correct the references in Sections 5.2 and 5.3.

Computational and inference cost

To quantify the computational overhead of world-model search, we compare QWM with model-free and model-based baselines in terms of policy sampling, critic evaluations, world-model predictions, and decision latency during online collection and evaluation. All measurements are conducted on an NVIDIA A800-SXM4-80GB GPU, with decision times reported as median latencies under the shared benchmarking protocol.

State-based computational cost comparison. We use Square as a representative task to compare the computational costs of different methods under their respective planning configurations.

MethodPolicy ActionsQ/V QueriesWM BatchesPred. TransitionsCollection (ms)Evaluation (ms)
QWM–EXPO65665621445.945.94
EXPO1616002.932.93
QWM–RLPD1361362160.720.65
RLPD10000.200.16
TD-MPC23,0963,07263,0724.864.83
EZV28033 (V)3232123.28121.92

Pixel-based computational cost comparison. We use LIBERO Task 60 as a representative task to compare the computational costs of QWM and EXPO during online data collection and evaluation.

MethodCollection: Q queries / WM batches / TimeEvaluation: Q queries / WM batches / Time
QWM272 / 2 video + 2 state / 2,961.33 ms16 / 0 / 2.72 ms
EXPO16 / 0 / 3.06 ms16 / 0 / 2.73 ms

Computational overhead and sample efficiency. In state-based experiments, QWM–EXPO and QWM–RLPD incur approximately 2.0× and 3.6× higher collection latency than their model-free counterparts. Compared with model-based baselines, QWM–EXPO is slightly slower than TD-MPC2 (5.94 vs. 4.86 ms), but 20.8× faster than EZV2 (5.94 vs. 123.28 ms).

In pixel-based experiments, video world-model inference substantially increases collection latency (2,961.33 vs. 3.06 ms for EXPO), while evaluation latency remains comparable (2.72 vs. 2.73 ms) since search is disabled during evaluation.

These results highlight the trade-off between planning overhead and sample efficiency. Reducing environment interactions may be particularly valuable in real-world robotics, where execution and resets are costly. We are exploring QWM with EXPO-based VLA fine-tuning, with real-robot evaluation and end-to-end cost analysis left for future work.

Cost–benefit tradeoff

QWM reduces environment interactions in our benchmarks at the cost of additional collection computation. State-based search remains enabled during both collection and evaluation. In the representative pixel Task 60 benchmark, collection includes world-model lookahead, while evaluation uses the learned policy and critic without world-model calls. Q-query matching controls the comparison budget. On a physical robot, fewer interactions could save execution, reset, and operator time if online collection can tolerate the planning latency. We will assess this tradeoff by measuring robot interactions and total time to a matched success rate.

We hope these clarifications address your concerns, and we kindly invite the reviewer to reconsider their score in light of them.

Reviewer_os7mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Q-Learning with World Models (QWM), which uses a learned world model to search over policy-proposed actions while retaining policy and critic training on real transitions. The search combines Q-values at intermediate and leaf nodes and uses Q-guided pruning to limit computation. QWM is instantiated with EXPO and RLPD, with experiments on state-based Robomimic tasks and visual LIBERO tasks reporting improvements in sample efficiency. Additional ablations examine the benefits of search during data collection and evaluation, along with sensitivity to search configuration.

Critical strengths
  1. QWM adds world-model search to action selection while retaining the underlying policy and critic objectives and training them on real transitions. Its implementations on both EXPO and RLPD demonstrate that the approach can augment different learners without redesigning their optimization procedures.

  2. The experiments extend beyond low-dimensional dynamics models on Robomimic to an action-conditioned video world model on LIBERO.

Critical weaknesses
  1. Insufficient distinction from related work. The paper should discuss and distinguish QWM from LOOP (Sikchi et al., CoRL 2021) and Inference Time Policy Optimization for Offline RL with Differentiable World Models (Deb et al., 2026). LOOP already combines learned-model planning during data collection and evaluation with off-policy actor–critic learning on real environment transitions. Deb et al. use differentiable world-model rollouts to adapt policy parameters at test time in offline RL. The authors should clarify what QWM's intermediate-Q aggregation and pruning contribute beyond existing planning frameworks. A controlled comparison with LOOP using the same underlying learner and world model would help establish this contribution.

  2. Uncertainty reporting. I could not find the number of independent training seeds, the number of evaluation episodes per checkpoint, or a definition of the shaded regions. Appendix B.2 refers to standard evaluation protocols but does not specify these quantities. Please report them and clarify whether the shaded regions represent variability across training runs or uncertainty from evaluation episodes. Without this information, it is difficult to assess the robustness of the reported improvements, particularly the smaller gains over EXPO and RLPD. Claims of significant improvements should be qualified or supported by an appropriate uncertainty analysis.

  3. Demonstration access and initialization are insufficiently specified for the model-based baselines. Appendix B.1 and Table 1 explicitly establish common demonstration sources and training settings for QWM and the model-free baselines, but their applicability to TD-MPC2 and EfficientZero V2 is unclear. Did these baselines receive the same demonstration subsets, how were demonstrations incorporated into replay, and were any model, policy, or value components pretrained? This clarification matters because Appendix B.3 specifies 100,000 offline gradient steps for QWM's dynamics model, which is subsequently frozen. Please provide the corresponding initialization and demonstration-use protocols for each model-based baseline, and clarify whether Table 1's zero offline pretraining steps refer only to the policy and critic.

  4. The EfficientZero V2 training configuration needs clarification. Appendix B.1 reports UTD 20 for QWM and the model-free baselines, while describing UTD 0.4 as EfficientZero V2's recommended setting. However, Table 3 of the [original EfficientZero V2 paper](https://arxiv.org/pdf/2403.00564) lists UTD 1. Please identify the implementation or configuration supporting the recommendation of 0.4.

  5. The Q-versus-V ablation does not isolate the benefit of Q-based search. Appendix A.1 states that the experiment aims to isolate this benefit, but it changes both the value estimator and the policy-learning procedure. The V-based variant uses advantage-weighted regression, replacing Q-based optimization of the EXPO edit policy and the SAC-style objective of the RLPD actor. Since the policy generates the candidate actions explored during search, these changes can affect action proposals, online data collection, and subsequent learning, independently of the search scoring rule.

Questions for authors

How are terminal rewards and terminal branches handled during search? Equations (6) and (9) include a reward term, but Section 4.4 states that no reward model is learned because rewards are zero except at termination. Is this term set to zero, or supplied through another mechanism?

A separate reward model may be unnecessary because Q-values can encode terminal rewards, but the precise backup matters. Consider a deterministic episodic example:

  • Action aa receives reward 11 immediately and transitions to an absorbing terminal state.
  • Action bb receives reward 00 and transitions to a state whose only action receives reward 11 and terminates.
  • With an exact critic, Q∗(s,a)=1Q^*(s,a)=1 and Q∗(s,b)=γQ^*(s,b)=\gamma. The successor values are 00 and 11, respectively.

If Equation (9) is applied unchanged with its reward term set to zero, then at depth D=1D=1, with γ=0.99\gamma=0.99 and λ=0.1\lambda=0.1,

\qquad Q_{\mathrm{ts}}(s,b)=\frac{1}{2}(0.99+0.1)=0.545.$$ If both actions are proposed, greedy selection therefore prefers the action with lower discounted return, despite an exact model and critic. Retaining the terminal reward instead gives $Q_{\mathrm{ts}}(s,a)=1$, restoring the ranking in this example. This calculation assumes the ordinary recursion is also applied to terminal branches. Does the implementation instead stop those branches and use their Q-values directly? Also see my other questions in weaknesses.
2 comments
AuthorsOpenReview ↗

Part I — Responses to W1 and W2

Thank you for your thorough and constructive review! We address each point below and will incorporate these clarifications into the paper.

Relationship to LOOP and DiffMPC

Both LOOP and QWM combine world-model-based planning with online off-policy RL, but differ in how value estimates are used during search. LOOP optimizes candidate trajectories using predicted rewards and terminal value estimates, with actor regularization to mitigate policy divergence. In contrast, QWM uses intermediate Q-values to score and prune partial trajectories during search, allowing the planner to prioritize promising paths before completing the rollouts, without requiring a learned reward model in sparse-reward settings. DiffMPC instead focuses on inference-time optimization of pretrained offline policies by differentiating through imagined trajectories. Unlike DiffMPC, QWM does not optimize policy parameters through the world model, but uses Q-guided lookahead for action selection and improves its actor and critic through off-policy learning from real environment transitions. We will clarify these distinctions in the revised paper.

Comparison with LOOP. Following the reviewer’s suggestion, we adapted LOOP’s ARC planner to the state-based EXPO learner used by QWM. We retained the actor and edit policy architectures, critic updates, and replay recipe (50% demonstrations, UTD 20, batch size 256). ARC used two optimization iterations with 16 Gaussian and 16 EXPO trajectories per iteration. Both methods adopt the same world-model architecture and pretraining procedure using demonstration data. Their pretrained world models remain frozen throughout online training.

In the online comparison, we report mean success over three independent training seeds, using 50 episodes per seed and checkpoint for both methods. We report the evaluation results below. QWM achieved higher success rates on Lift, Square, and Tool Hang, with similar performance on Can.

TaskEnvironment stepsEXPO + ARC (%)QWM (%)
Lift100k28.098.7
Can100k98.799.3
Square100k70.793.3
Tool Hang250k0.797.3

Uncertainty reporting

The main state-based EXPO and RLPD comparisons use three independently trained seeds and 50 evaluation episodes per checkpoint and seed. The shaded region represents standard error across the three seeds. We will explicitly state the number of independent training seeds, the number of evaluation episodes per checkpoint, and the definition of the shaded region in the final paper.

AuthorsOpenReview ↗

Part II — Responses to W3–W5

Demonstrations and initialization of model-based baselines

In the reported runs, QWM, TD-MPC2, and EfficientZero V2 use the same demonstration subsets, with a 50% demonstration replay ratio during online training. QWM's dynamics model is pretrained for 100k gradient updates and remains frozen during online training. The zero offline pretraining steps reported in Table 1 refer only to QWM's policy and critic, while dynamics pretraining is documented separately in Appendix B.3.

TD-MPC2 and EfficientZero V2 learn latent dynamics together with reward and value prediction as part of their native training procedures. We followed their procedures on training model, policy, and value components jointly using demonstration and online replay data, without a separate offline world-model pretraining stage. QWM instead uses a separate state-dynamics model, which is pretrained on demonstrations and frozen during online training. We will clarify this difference in initialization and training in the appendix.

Matched pretraining. On dense-reward Lift and Can, we pretrained the TD-MPC2 and EfficientZero V2 world models on demonstrations for 100k updates and froze them during online training, matching QWM's pretraining budget. At 25k environment steps, EZV2 (UTD = 1) achieved 8% success on Lift, while all other baseline–task combinations remained at 0%. These results further support our original choice of training configurations, which yielded better performance in our experiments. They also suggest that uniformly matching training configurations across different methods does not necessarily improve performance, as different algorithms may require different hyperparameter settings for effective learning.

EfficientZero V2 training configuration

The original EfficientZero V2 paper reports UTD = 1, whereas our Robomimic adaptation uses UTD = 0.4 to approximately match QWM's wall-clock training time. On an A800 node, 100k Lift transitions take approximately 8.2 hours for EZV2 at UTD 0.4, compared to 6 hours for QWM. Based on observed throughput, EZV2 at UTD 20 is estimated to take 275.7 hours. We will clarify this experimental choice in Appendix B.1.

Gumbel search under EXPO. To complement the native baseline results, we integrate Gumbel search into EXPO for online data collection and evaluation. Both methods use the same types of actor, edit policy, and critic. Gumbel–EXPO uses a frozen demonstration-pretrained world model, UTD 20, batch size 256, and 50% demonstration replay. The following table reports mean success rates across three independent training seeds at matched environment steps.

TaskStepsGumbel–EXPO (%)QWM (%)
Lift100k10.798.7
Can100k85.399.3
Square100k76.793.3
Tool Hang250k15.397.3

Matched-budget planning cost. We further isolate the computational efficiency of the two planners using shared model weights and identical search budgets, including lookahead depth, predicted transitions, and Q queries. Decision times are reported as median latencies from the shared benchmark.

TaskDepthPredicted TransitionsQ QueriesQWM / Gumbel Median (ms)
Lift21446566.26 / 11.71
Can21442725.94 / 10.80
Square21446565.94 / 11.24
Tool Hang41,1681,6809.11 / 25.87

Together, these results show that QWM achieves higher success rates under the evaluated online training settings, while also providing 1.8–2.8× lower decision latency under matched search budgets and shared model weights.

Q-versus-V ablation

Appendix A.1 compares two complete pipelines: learning and searching with Q versus V, rather than isolating the scoring function alone. Q directly evaluates candidate actions at the current state, whereas V requires world-model predictions of successor states for action comparison, introducing additional dependence on model accuracy. Since the V-based learner does not maintain a Q-function, we use advantage-weighted regression for policy learning. Thus, this experiment evaluates the end-to-end effectiveness of Q-based versus V-based learning and search. We will clarify this distinction in Appendix A.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.