acceptodds
Withdrawn from ICLR 2027

SimForge: Agentic Generation of Simulation-Ready Worlds and Demonstrations for Robot Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generating a simulation-ready robot-learning world from language requires two distinct execution decisions: which construction stage should handle a failure, and how semantic intent should be expanded into many geometric instances. We present \simforge, an agentic text-to-scene system built on a Graph-Native Harness. Within scene generation, a registered execution graph separates specialist execution from scene-local control: deterministic issue ownership and hard acceptance gates route failures, while bounded language model recommendations can continue or roll back only along registered transitions and retained checkpoints. For dense placement, the model specifies object groups, target quantities or density, support regions, and arrangement intent; geometry programs expand and validate instance poses using measured assets and supports. This arrangement-level interface avoids explicit per-instance pose enumeration in model outputs, addressing the observed context/OOM bottleneck while leaving geometric and simulation costs instance-dependent. Processed assets are separated from scene instances for targeted insertion, material or appearance changes, rigid or articulated substitution, and scene/task variants. An independent PolicyForge pipeline binds accepted worlds to tasks and robots and validates teacher demonstrations. On the common SceneEval scene-quality protocol, with the specified physics settings for COL and STB, the full system attains 89.4% and 89.7% count agreement at room and house level, respectively, with 0% detected collision and 100% stability. Internal controls separate the runtime effects of routing and batch placement. Under matched policy-training and evaluation conditions on the same LIBERO-Plus evaluation set with a common 13K checkpoint, augmented SimForge data reach 87.84% overall and 86.11% on Goal, improving over the Native reference by 11.87 and 31.28 percentage points, respectively. On LIBERO-Pro, the recorded augmented SimForge cohort reaches 48.7% overall at 13K, with effects that vary across generalization dimensions.

Primary area:
applications to robotics, autonomy, planning
Keywords:
embodied AI, 3D scene generation, robotic simulation, LLM agents, task-conditioned generation, physics-grounded generation, scene augmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JftuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SimForge generates simulation-ready worlds from language through graph-based failure routing, procedural batch placement, and asset–instance separation for targeted augmentation. An independent PolicyForce pipeline binds accepted worlds to tasks and robots and validates teacher demonstrations. Evaluation covers room/house scene quality, routing/placement ablations, and LIBERO-Plus/Pro learning with a dual-stream World Action Model. Augmented SimForge achieves 87.84% policy success versus 75.97% for native data and 86.24% for augmented SceneSmith on LIBERO-Plus.

Critical strengths
  • Practical and explicitly bounded system design. Failure ownership, registered transitions, checkpoints, and constrained repairs structure scene construction. Group-level placement separates semantic intent from repeated pose computation, while asset–instance separation supports targeted edits and revalidation. Together, these mechanisms address practical bottlenecks in building editable, executable worlds.
  • Meaningful controls and augmentation evidence. The ablations help distinguish the contributions of the Harness and procedural placement, while consistent improvements across multiple data sources support the practical value of SimForge’s augmentation pipeline.
Critical weaknesses
  • Limited validation on robot task. The paper's central claim is that SimForge-generated data improve robot learning. However, quantitative policy evaluation is confined to the MuJoCo-based LIBERO-Plus/Pro benchmarks. Controlled improvements on additional simulation benchmarks, such as RoboTwin2.0[1], SimplerEnv[2] and so on, or on physical robot tasks would provide stronger support for the broader claim. Such evidence would make me consider raising my score.
  • SimForge supports room and house-level scene generation and augmentation, but its downstream evaluation focuses on tabletop manipulation. These benchmarks do not directly establish the value of richer scene structure for tasks requiring navigation and manipulation across locations, such as mobile manipulation. A controlled comparison on such tasks would better connect SimForge's scene-level capabilities to its claimed robot-learning utility, rather than only demonstrating benefits within tabletop settings. Positive evidence here would also make me consider raising my score.

Besides, the manuscript needs a careful formatting and reference check. The Introduction heading is unusually close to the following paragraph around line 40, and the “Figure ??” reference around line 300 is unresolved.

[1] T Chen, et al. RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation. ICML 2026.

[2] Li X, et al. Evaluating Real-World Robot Manipulation Policies in Simulation. CoRL 2024.

Questions for authors

Regarding Weaknesses 1 and 2, do SimForge improve policy performance in other simulators or on real robots, and do its scene-level ability benefit robot tasks that require scene-level capabilities?

Reviewer_SnQ4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SimForge. It is a system for generating simulation-ready scenes and robot demonstrations. It uses a predefined execution graph to organize layout, asset acquisition, placement, physics validation, critique, and export. Failures are routed to the responsible stage through ownership rules and bounded model recommendations. For dense placement, the model specifies object groups, quantities, support regions, and arrangement patterns, while geometric routines generate and validate individual poses. A separate PolicyForge pipeline produces demonstrations using supplied teachers. Experiments evaluate scene quality, routing and placement configurations, and downstream policy performance on LIBERO-Plus and LIBERO-Pro.

Critical strengths
  • The owner-based repair structure makes sense. A missing asset returns to acquisition, while an existing object with a placement problem returns to placement. The system also preserves unrelated scene state and rechecks affected objects. This is useful for avoiding repeated work and preventing a local repair from silently breaking other parts of the scene.

  • The internal ablation is useful. It separates routing from batch placement while keeping the specialists, validators, checkpoints, and budgets fixed. Routing improves object-relation and accessibility scores, while batch placement substantially reduces the reported runtime. The downstream experiments also show that the generated data can improve policy performance, especially on LIBERO-Plus.

Critical weaknesses
  • The major issue of the paper is that I do not clearly see the method contribution beyond the execution structure. Compared with the main baseline, SceneSmith already has orchestration, rollback, support-relative placement, procedural multi-object tools, and physics validation. SimForge makes cross-stage ownership and repair permissions more explicit, but many routing decisions are predefined rules. The current ablation compares the full Harness with a fixed workflow, which does not isolate whether model recommendations improve over deterministic owner-based routing. The results support the usefulness of the complete design, but I am not convinced how much new model reasoning is needed. Moreover, the downstream task success rate improvement is not very clear (~3%).

  • The placement and physics-repair method is not sufficiently specified. I understand that the system generates finite candidates using measured supports, footprints, and allowed rotations, then checks collisions and settling. However, when an object tips over, how does it choose the corrected pose? Figure 2 shows a bird changing from upright to lying on its side, but the procedure that produces that orientation is unclear. The geometric formulation mainly describes support-plane translation and yaw. I cannot tell whether roll and pitch changes come from predefined orientations, geometric heuristics, or model proposals. This makes it difficult to reproduce the repair procedure or assess how it differs from existing placement tools.

  • The paper contains an unlinked image showing "??".

Questions for authors
  • How does the full controller compare with deterministic owner-based routing under the same conditions? Which cases specifically benefit from model recommendations?

  • When physics detects an unstable object, how are replacement poses generated and selected? For the bird example, what produces the side-lying orientation? Please specify the orientation candidates, search procedure, and retry budget.

  • How does the dense-placement routine differ from existing procedural placement tools? A matched comparison using the same assets, supports, validators, and budget would help establish its contribution.

  • On LIBERO-Plus, augmented SimForge reaches 87.84%, compared with 86.24% for SceneSmith with matched augmentation. Is this difference consistent across repeated training runs?

  • The routing rules are understandable once a structured issue is available, but how is that issue diagnosed? If placement fails, how does the system distinguish a bad pose, unsuitable support geometry, and insufficient search? Please clarify which diagnoses are computed from geometry, which require VLM reasoning, and what triggers support repair or asset reacquisition.

Reviewer_rhFGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SimForge, an agentic framework for generating simulation-ready environments for robot learning. The method introduces a graph-based harness for scene construction, verification, and refinement, along with procedural object placement and scene augmentation. The authors evaluate the approach on scene generation benchmarks and downstream robot learning tasks, demonstrating improvements over existing methods.

Critical strengths
  • The paper addresses an important problem of generating simulation-ready environments for robot learning
  • The proposed graph-based harness is reasonable and provides a structured approach to scene generation and verification
  • The downstream robot learning experiments are useful and demonstrate the potential of generated environments for policy training
Critical weaknesses
  • The quantitative improvements over the previous state-of-the-art harness, SceneSmith, appear marginal. The downstream robot learning evaluation also shows similar performance, with Table 11 in the supplement reporting identical LIBERO-Pro results for SceneSmith and SimForge at 13K training steps. Given these results, I wonder if the presented method shows computational cost advantages over existing methods?
  • The paper does not provide qualitative comparisons with baselines or representative failure cases. This makes it difficult to understand what types of scenes the proposed harness handles better than existing approaches.
  • The method uses GPT-5.2, which is no longer representative of the strongest available models for 3D scene understanding and generation. Recent models, such as GPT-6 Astra, have substantially improved capabilities on 3D tasks. It is unclear whether the proposed harness would provide similar benefits or computational cost advantages with stronger models.
  • The paper does not investigate more cost-effective VLM alternatives, such as open-weight models like Qwen Flash Next or smaller models like GPT-6 Luna. Given the relatively small improvements over SceneSmith, understanding the tradeoff between model capability, harness complexity, and inference cost is important
Questions for authors
  1. What are the computational and inference costs of SimForge compared to SceneSmith and other baselines? Does the proposed harness offer any cost advantages?
  2. Can the authors provide qualitative comparisons with SceneSmith and representative failure cases?
  3. How much does the proposed harness improve performance when using stronger models such as GPT-6 Astra? Are the improvements still significant?
  4. Have the authors evaluated more cost-effective models, such as Qwen Flash Next or GPT-6 Luna? How do performance and cost compare?
  5. Why does Table 11 report identical LIBERO-Pro results for SceneSmith and SimForge at 13k steps?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.