- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes a method using world models and estimated Q functions for MCTS to improve the quality of generated actions in RL setups. Specifically, their world model is only used in test-time searching, with Q-functions to determine the value of each actions. They validate the performance of their proposed method against both model-free and model-based baselines in a range of challenging manipulation tasks. Results show that their proposed method has substantial performance improvement against existing baselines on these selected tasks, which demonstrate the effectiveness of their proposed method.
Critical strengths
-
Clear presentation: The overall presentation of the paper is clear and easy to follow. The proposed method is straightforward and easy to understand the details of the method.
-
Superior performance: The authors demonstrate substantial performance improvement against existing baselines on a variety of challenging tasks on robotic manipulation, which demonstrate the effectiveness of their proposed approach.
Critical weaknesses
-
World model scale is incomparable: The proposed method leverages a 5B WAN video generation model as a world model for test-time tree search in pixel level experiments, while standard model-based baselines such as TD-MPC2 and EfficientZeroV2 only leverages MLP-based world models with CNN encoders. From my knowledge a standard variant of TD-MPC2 is containing only 5M of parameters while the larges one is around 200M, which is significantly smaller than the WAN model with 5B parameters. For pixel-level experiments, it's questionable that the performance gain is originated from the use of larger pretrained video world model or is from the designed learning algorithm itself.
-
More detailed explanation of the proposed method against existing baseline: The existing baseline EfficientZeroV2 is also leveraging a tree-search algorithm during test-time planning. Please differentiate the proposed algorithm with their proposed tree search and show why the proposed method is better.
Questions for authors
See the weakness section.
World-Model Scale and Search Strategy Comparisons
Thank you for your thoughtful review and for appreciating the clarity of our presentation and the performance gains! We provide clarifications below on separating the contribution of the search rule from the capacity of the world model, and will incorporate them into the paper.
We would like to clarify that our main comparisons with TD-MPC2 and EfficientZero V2 are on state-based Robomimic. There, all three methods operate on low-dimensional observations with compact state-based models: QWM uses a residual MLP, and the native baselines use latent dynamics. QWM's gains over the model-based baselines in this setting therefore do not come from a larger world model.
Wan2.2-5B is used only in the separate pixel-based LIBERO experiments, and only during online data collection. At evaluation, the trained EXPO policy and critic act without video search, so the deployed agent does not use the video model. We agree that the pixel comparison measures the combined effect of adding the pretrained video world model and QWM search. Isolating the search contribution in pixels would require a shared video checkpoint, learner, and model-prediction budget, and we will state this explicitly as a limitation in the revision.
EfficientZero V2 uses its model to construct improved policy and value targets. It allocates simulations among root actions via sampled Gumbel search and Sequential Halving, and reuses search estimates for learning. In contrast, QWM uses its model to guide the collection of real experience for an off-policy learner, and the actor and critic learn only from demonstrations and real environment transitions.
QWM also addresses a different bottleneck under sparse rewards. Two short rollouts can receive the same zero predicted reward even when one sets up a successful grasp or insertion. We use discounted intermediate critic scores to distinguish such prefixes, prune branches before deeper model calls, and include those scores in the final action ranking.
We thank the reviewer for this suggestion. Both QWM and EfficientZero V2 employ world-model-based tree search, but they differ in the search strategies. EfficientZero V2 adopts Gumbel-based tree search, whereas QWM employs our proposed Q-guided search strategy. To directly compare these approaches, we integrate Gumbel search into the EXPO learning framework for both online data collection and evaluation.
Both methods use the same types of EXPO actor, edit policy, and critic, and are trained online using real environment transitions. Gumbel–EXPO uses a world model pretrained on demonstration data, which remains frozen throughout online training.
For Gumbel–EXPO, we adopt a training configuration with UTD 20, batch size 256, and 50% demonstration data in the replay buffer. The following table reports the mean success rates across three independent training seeds at matched environment steps.
At the reported training steps, QWM achieves higher mean success rates than Gumbel–EXPO across all four tasks.
We compare the two search methods with matched search depth, numbers of predicted states, and Q queries. Under these matched search budgets, Gumbel search incurs approximately 1.8–2.8× higher decision latency than QWM.
Overall, these results highlight two advantages of QWM: (1) higher final success rates across all four tasks under the evaluated online training configurations, and (2) lower decision latency under matched search budgets and shared model weights. These results combine to show the effectiveness of the proposed tree search over prior methods.