acceptodds
Withdrawn from ICLR 2027

WorldMirror: Agentic Real-to-Sim Generation of Interactive Worlds from In-the-Wild Robot Demonstrations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing approaches for robot learning in simulation often require substantial manual effort to construct scenes and specify tasks, limiting training scalability and data diversity. Meanwhile, in-the-wild robot demonstrations naturally provide diverse and realistic observations of real-world interactions. Leveraging these demonstrations for simulation requires both high-fidelity visual reconstruction and physically consistent interactions. General-purpose agents offer flexible adaptation to diverse scenarios and strong reasoning about 3D spatial relationships, while specialized generative models provide high-fidelity reconstruction of scene appearance and geometry. We introduce WorldMirror, an agentic real-to-sim framework that constructs interactive digital twins from in-the-wild robot demonstrations. Our central idea is to formulate real-to-sim construction as the iterative refinement of an executable world program. A coding agent decomposes the observed scene, plans reconstruction steps, and orchestrates specialized image and 3D generation tools, refining the program through verification feedback. A camera-pose-anchored scene representation organizes the world into static visual context, support geometry, and editable interactive assets while preserving spatial relationships among the scene, objects, and robot. The agent uses this representation to diagnose and correct reconstruction errors, enabling physically consistent interaction and trajectory replay. Experiments on DROID demonstrate that WorldMirror can reconstruct visually faithful and geometrically consistent interactive worlds while preserving the physical validity of recorded robot trajectories.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Real-to-Sim, Digital Twins, Robot Simulation, Embodied AI, Robot Manipulation, Agentic Systems

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZRZSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes WorldMirror, an agentic real-to-sim framework for constructing interactive digital twins from in-the-wild robot demonstrations. A router agent coordinates specialized reconstruction, calibration, and simulation tools while maintaining an editable world state. A camera-pose-anchored representation aligns the reconstructed scene, interactive objects, and robot, while verification feedback guides iterative refinement. The framework is evaluated on DROID demonstrations in terms of visual fidelity, spatial accuracy, and physical interaction consistency. WorldMirror achieves strong perceptual reconstruction quality, improved motion replay accuracy, and a reported 100% trajectory replay success rate. The main contribution is an integrated agentic framework for reconstructing simulation-ready environments through tool orchestration and verification-driven refinement.

Critical strengths
  1. A relevant problem with a coherent real-to-sim framework. Automatically constructing interactive simulation environments from real-world robot demonstrations is an important problem for scalable robot learning. WorldMirror integrates scene reconstruction, object modeling, camera calibration, and physical alignment within a unified agentic workflow. The structured world representation and verification-driven refinement provide a reasonable approach to constructing simulation-ready digital twins.
  2. Promising results across multiple reconstruction metrics. The paper evaluates visual quality, spatial consistency, and trajectory replay, covering several important aspects of real-to-sim reconstruction. WorldMirror achieves the best LPIPS and DISTS scores among the evaluated methods and substantially reduces motion replay errors. The reported improvements in contact fidelity and trajectory replay success are encouraging, although their interpretation depends on the evaluation protocol.
Critical weaknesses
  1. Insufficient validation of the core agentic framework. The central contribution of WorldMirror is its agentic construction framework, featuring adaptive tool orchestration and verification-driven refinement. However, the paper lacks controlled ablations isolating these mechanisms. In particular, there is no comparison between agentic orchestration and a fixed reconstruction pipeline using the same tools, or between reconstruction with and without verification. It therefore remains unclear whether the improvements arise from agentic decision-making, iterative refinement, the underlying reconstruction tools, or additional optimization effort. Sections 3.3 and 3.4 also provide limited implementation details regarding tool selection, verification criteria, refinement decisions, and termination conditions. The paper does not report reconstruction time, agent iterations, tool-call counts, or manual intervention requirements, making it difficult to assess the efficiency and scalability of the framework. Without these analyses, the contribution of the proposed agentic mechanisms remains insufficiently demonstrated.
  2. Insufficiently specified evaluation protocols for key quantitative claims. Several important aspects of the experimental evaluation are unclear. The visual evaluation covers only ten DROID scenes, while the spatial evaluation includes 17 objects and eight object pairs. More importantly, Table 4 reports a 100% trajectory replay success rate without specifying the number of evaluated trajectories, their selection criteria, or the definition of successful replay. The treatment of failed reconstructions is also unclear, making it difficult to assess the significance of this result. The construction of ground-truth measurements is another concern. Tables 3 and 4 report 3D object positions, motion errors, and contact F1, but the paper does not sufficiently explain how the corresponding real-world 3D states, object trajectories, and contact annotations are obtained from DROID demonstrations. These details are essential for interpreting the reported metrics. Furthermore, the comparison protocols are not fully specified, particularly regarding differences in input information, reconstruction tools, and computational budgets across methods. These omissions make the quantitative results difficult to independently verify and weaken the evidence supporting the paper's main claims.
  3. Limited evidence of physical interaction fidelity beyond demonstration replay. The paper aims to construct physically consistent interactive environments, but the evaluation primarily focuses on replaying the original robot demonstrations. Section 3.4 describes verification-driven refinement using consistency with the observed demonstrations, while Section 4.4 evaluates trajectory replay using those demonstrations. This raises concerns about whether the reported replay performance reflects accurate physical modeling or reconstruction specifically optimized for the demonstrated interactions. Successful replay provides useful evidence of geometric alignment and interaction feasibility, but does not necessarily establish that the reconstructed environments can accurately simulate different actions or unseen interaction outcomes. Evaluations with alternative robot trajectories, perturbed object configurations, or independent policy rollouts would provide stronger evidence of general physical consistency. Without such experiments, the paper demonstrates fidelity to the original demonstrations more convincingly than general-purpose physical simulation capabilities.
Questions for authors

Q1. Agentic framework and ablations. Could the authors compare WorldMirror against a fixed reconstruction pipeline using the same tools, and provide an ablation without verification-driven refinement? How are tool selection, refinement decisions, and termination conditions determined? Q2. Evaluation protocol and ground truth. How are the ground-truth 3D object positions, motion trajectories, and contact labels obtained? How many trajectories are evaluated in Table 4, and what criteria define the reported 100% replay success rate? Are failed reconstructions included? Q3. Physical interaction generalization. Are the trajectories used for verification-driven refinement also used for the replay evaluation? Could the authors evaluate different trajectories or independent policy rollouts to demonstrate physical consistency beyond the original demonstrations? Q4. Efficiency and reproducibility. What are the average reconstruction time, number of agent iterations, tool-call counts, and manual intervention requirements? How do these compare with the baselines? Could the authors also clarify the input information and computational budgets used for each method?

Reviewer_8Fg9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

WorldMirror builds an interactive digital twin from RGB demonstrations plus the recorded robot trajectory. A router agent runs reconstruction, calibration, layout, and simulation through an editable world state, with verification-driven revision loops. The visual background, collision geometry, and interactive objects all live in one camera-anchored frame. Evaluation covers appearance on ten scenes, layout on 17 objects and eight pairs, and replaying the recorded trajectories, though the number of replay episodes is not stated (pp. 4-9, Tables 2-4).

Critical strengths
  1. Splitting visual context, support geometry, and editable objects is a sensible engineering design, it lets you fix one thing without breaking the rest. Replaying the unmodified trajectory is also the right thing to check, since looking similar is not enough for a sim to be useful (pp. 5-6, Section 3).
  2. The initial numbers are promising. LPIPS 0.427 vs 0.506 for RoLA, and replay success 100% vs 40% for the Astra baseline. Figure 4 shows a decent variety of interaction types. These are the authors' aggregates, I did not reproduce them (pp. 7-9).
Critical weaknesses
  1. The physical side is hard to audit. The paper doesn't give the replay denominator, inclusion criteria, what counts as success or contact, the dynamics and controller settings, or what object-state interventions are allowed. Demonstrations are used in the verification loop, but it's not clear whether they feed back into construction or are only used for the final evaluation. Replaying the demonstrations the world was fitted to is a fine reconstruction check, but it doesn't say anything about whether the physics predicts unseen interactions (pp. 6-9, Sections 3.4 and 4.4).
  2. The "agentic" part is not separated from the tools and the optimization budget. There is no fixed-pipeline ablation and no verification-off ablation at the same budget. Agent versions, prompts, stopping rules, retries, how much a human stepped in, and how metric scale was calibrated are all underspecified (pp. 5-6). The SE(3) expression given handles coordinate conversion but doesn't recover unknown scale, so there must be an extra calibration step somewhere and it should be described. The naming mismatch in Tables 3-4 (RoboClone?) should be cleared up too, though that is not the reason for my score.
Questions for authors
  1. What are the replay episode IDs or count, the inclusion rules, the success/contact definitions, and the allowed state interventions? Which observations and replay outcomes were used for refinement vs only for the final evaluation?
  2. Please specify the metric-scale and robot-camera calibration, agent and tool versions, prompts, human intervention, and the baseline budgets. Are the RoboClone rows the proposed system?
  3. Do tool- and budget-matched ablations of the orchestration and the verification loop actually support the main contribution? If you want to claim predictive physical validity, can a frozen world reproduce a held-out interaction? Large-scale policy training can stay future work.
Reviewer_LxjJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents WorldMirror, an agentic Real-to-Sim framework that reconstruct interactive simulation environments from realworld monocular RGB videos and robot demonstrations. The framework uses coding agents to call specialized 3D reconstruction tools, maintain a consistent world representation, and iteratively refine scenes through verification feedback. Experiments on DROID demonstrate the strong performance of high fidelity interactive 3D reconstruction and a significant robot trajectory replay success rate increase in IsaacSim.

Critical strengths
  1. The paper presents an interesting and very promising agentic approach to automatically constructing interactive simulation environments from videos. Rather than relying on a fixed reconstruction pipeline, WorldMirror uses coding agents with specialized tools to build and iteratively refine the interactive 3D environment. I believe this is a meaningful direction that could substantially reduce the manual effort required for simulation environment construction.

  2. The authors provide evaluations of visual fidelity, geometric consistency, and robot trajectory replay. In particular, the trajectory replay experiment is valuable because it goes beyond visual reconstruction metrics and directly evaluates whether the reconstructed environment can reproduce realworld robot interactions. The reported improvements are promising and demonstrate the potential of this approach.

Critical weaknesses
  1. The authors compare WorldMirror against GPT-6 Astra, but it is unclear which foundation model powers WorldMirror's agents and whether both approaches have comparable tool access and token budgets. Since WorldMirror integrates multiple specialized reconstruction tools, I would like to see a fairer comparison using the same underlying coding agent and similar execution budgets to better understand the contribution of the proposed framework.

  2. This paper introduces several components, including router-based orchestration, structured world-state memory, camera alignment, and verification-driven refinement. If the specialized tool and the system design is the main contribution of this paper, then the authors should include a detailed ablation studies to justify each component contribution to the final performance.

  3. The paper reports 100% replay success without clearly specifying the number of evaluated demonstrations, their task diversity, or the exact success criteria. Since this is one of the strongest results in the paper, these details are important for assessing its significance and reproducibility.

  4. One of the main motivations is to reduce the manual effort required for simulation environment construction. However, the paper does not report how long it takes to reconstruct a single environment, the number of agent calls or refinement iterations, or the overall computational cost. These measurements would help us to evaluate how scalable the framework is.

  5. Minor presentation issue. The proposed method is referred to as WorldMirror throughout the paper, but Tables 3 and 4 use the name RoboClone. If these refer to the same method, the naming should be corrected for consistency. Otherwise, the authors should clarify the difference.

Questions for authors

Most of my questions are aligned with my concerns raised in the weaknesses section:

  1. Which foundation model powers WorldMirror's coding agents? Can the authors provide controlled comparisons using the same agent, tools, and computational budgets?
  2. Can the authors provide component ablations to identify which parts of WorldMirror contribute most to the improvement in trajectory replay success?
  3. How many robot demonstrations are used to calculate the reported 100% replay success rate? What are the exact success criteria? What is the average reconstruction time, number of agent calls, computational cost, and required human intervention per environment?
  4. What is the average reconstruction time, number of agent calls, computational cost, and required human intervention per environment?
  5. Can the authors evaluate whether the reconstructed environments support alternative robot trajectories or policies beyond replaying the original demonstrations? E.g. further evaluate the Real-to-Sim and Sim-to-Real gap

Given my current concerns regarding this paper, I do not recommend a positive score. However, I still value its overall contribution: the authors set up a novel and scalable pipeline for building useful simulation environment for robot learning. If the author can effectively resolve my major concerns, I would be willing to raise my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.