- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SimForge generates simulation-ready worlds from language through graph-based failure routing, procedural batch placement, and asset–instance separation for targeted augmentation. An independent PolicyForce pipeline binds accepted worlds to tasks and robots and validates teacher demonstrations. Evaluation covers room/house scene quality, routing/placement ablations, and LIBERO-Plus/Pro learning with a dual-stream World Action Model. Augmented SimForge achieves 87.84% policy success versus 75.97% for native data and 86.24% for augmented SceneSmith on LIBERO-Plus.
Critical strengths
- Practical and explicitly bounded system design. Failure ownership, registered transitions, checkpoints, and constrained repairs structure scene construction. Group-level placement separates semantic intent from repeated pose computation, while asset–instance separation supports targeted edits and revalidation. Together, these mechanisms address practical bottlenecks in building editable, executable worlds.
- Meaningful controls and augmentation evidence. The ablations help distinguish the contributions of the Harness and procedural placement, while consistent improvements across multiple data sources support the practical value of SimForge’s augmentation pipeline.
Critical weaknesses
- Limited validation on robot task. The paper's central claim is that SimForge-generated data improve robot learning. However, quantitative policy evaluation is confined to the MuJoCo-based LIBERO-Plus/Pro benchmarks. Controlled improvements on additional simulation benchmarks, such as RoboTwin2.0[1], SimplerEnv[2] and so on, or on physical robot tasks would provide stronger support for the broader claim. Such evidence would make me consider raising my score.
- SimForge supports room and house-level scene generation and augmentation, but its downstream evaluation focuses on tabletop manipulation. These benchmarks do not directly establish the value of richer scene structure for tasks requiring navigation and manipulation across locations, such as mobile manipulation. A controlled comparison on such tasks would better connect SimForge's scene-level capabilities to its claimed robot-learning utility, rather than only demonstrating benefits within tabletop settings. Positive evidence here would also make me consider raising my score.
Besides, the manuscript needs a careful formatting and reference check. The Introduction heading is unusually close to the following paragraph around line 40, and the “Figure ??” reference around line 300 is unresolved.
[1] T Chen, et al. RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation. ICML 2026.
[2] Li X, et al. Evaluating Real-World Robot Manipulation Policies in Simulation. CoRL 2024.
Questions for authors
Regarding Weaknesses 1 and 2, do SimForge improve policy performance in other simulators or on real robots, and do its scene-level ability benefit robot tasks that require scene-level capabilities?