- Rating
- 4
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces WorldEcho, a benchmark that evaluates whether action-conditioned robotic world models generate futures consistent with the specified actions. It compares generated videos with matched simulator rollouts across expert and several off-expert action queries, measuring both visual integrity and end-effector trajectory alignment. Motivated by the observed off-expert failures, the authors propose WorldSync, which combines expanded action coverage with Action-Forcing Expert and Intervention-Effect supervision. The evaluation covers 50 RoboTwin tasks, multiple world-model baselines, and downstream policy-improvement experiments in simulation and on real robots.
Critical strengths
-
A well-motivated and substantial evaluation. WorldEcho tests action following against action-specific simulator outcomes, rather than relying solely on visual similarity. The range of action queries, baselines, and reported metrics makes the observed failures informative.
-
A coherent connection from diagnosis to intervention and downstream validation. The authors use the identified failures to motivate concrete training changes, provide model-level ablations, and examine whether the resulting model benefits policy learning. The combination of world-model and policy experiments is a strength of the paper.
Critical weaknesses
I find this paper solid. I have no major concerns, but I have a few suggestions that could deepen the analysis.
Among the three improvements, expanding the training data distribution appears to be the most important. The importance of this is well established in model-based RL, as discussed in Agnostic System Identification for Model-Based Reinforcement Learning. Subsequent MBRL research could also inform the paper’s policy-learning analysis. A model need not be accurate everywhere; accuracy near the distribution visited by the policy may matter more (Live in the Moment: Learning Dynamics Model Adapted to Evolving Policy). When the model is unreliable, the length of model-generated rollouts can also be limited, as in MBPO (When to Trust Your Model: Model-Based Policy Optimization). Finally, the policy-learning experiments use only the full WorldSync model, without examining how each proposed improvement affects policy performance. Component-level policy ablations might reveal which improvements matter most and motivate a deeper analysis of model utility.
Questions for authors
Would the authors be able to report a downstream policy-improvement ablation using the same policy-training pipeline with expanded-data Base, Base + IE, Base + AFE, and full WorldSync? Even a compact experiment on the simulation task would help determine whether the components that improve WorldEcho scores also improve policy learning.
3 comments
Thank you for your reply. I’ll keep my rating.
Thank you
Thank you for your reply and for your support of the paper. We appreciate the suggestions, which helped us strengthen the paper.
Response to Reviewer EiuE
Thank you for the supportive review and the thoughtful suggestions. We are encouraged that you "find this paper solid" with "no major concerns". We have run the ablation you asked for and added the discussion you suggested.
Weakness and Question: Component-level downstream policy ablation
Short answer. Yes. We ran exactly these four configurations under the same policy-training pipeline, extended to three rounds. Each component improves policy learning over expanded-data Base in every round. After three rounds the combination improves it most, and the order of the four matches the order of their WorldEcho gated errors.
Gated errors are those of Table 3. Expert Base and both CtrlWorld checkpoints are in Rebuttal Table 1 of the General Response (2/2).
Revision. Section 4.3 (Table 2) and Appendix C.3 (Table 8) now include these conditions.
Suggestion: Connections to model-based reinforcement learning
Thank you for these pointers, which place our findings in a broader context. We now cite the three works, with a short discussion in Sections 2.2 and 5:
We are happy to discuss any further questions.