acceptodds
Under review as a conference paper at ICLR 2027

Agentic RSR: Real-to-Sim-to-Real through Scene Reconstruction and Execution-Grounded Robot Policies

Abstract

A simulation of a real robot workspace must preserve task-relevant interactions, while policies developed in it must operate on observations available to the real robot. Yet scene reconstruction and policy development are often treated separately. We present Agentic Real-to-Sim-to-Real (Agentic RSR), a framework that links scene reconstruction, policy development, and real-robot execution through the same manipulation task. Given a workspace video, a task description, and a known robot model, an agent recovers metric scale, iteratively refines the scene using visual feedback, and checks task-relevant interactions in MuJoCo. A coding agent then develops an executable policy, progressing from privileged object poses to visual observations and randomized simulation. The policy can interleave multiple observations and actions within one invocation, while the agent uses execution feedback to continue, retry, or revise its approach. A shared task-level interface carries the policy and accumulated experience to the real robot, where fresh observations and safety checks guide execution. Across 18 reconstructed scenes involving two robots, the mean four-view Depth MAE against reference depth estimates is 0.1057 m, the mean Lab Delta E76 is 11.04, and the mean grayscale SSIM is 0.6990. In real-robot experiments, the aggregate task success rate reaches 80% of the simulation task success rate, indicating substantial retention of simulated performance on hardware. Code and reconstructed scene data will be made publicly available.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Manipulation, Real-to-Sim-to-Real, Scene Reconstruction, Programmatic Policies

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_i33gOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This submission introduces Agentic RSR, an agentic framework for real2sim2real transfer of robot manipulation policies. It consists of a real2sim stage, which reconstructs the real-world scene into MuJoCo given a real-world video. Given the reconstructed interactive simulation, a few coding agents devise executable motion trajectories for task completion. Afterwards, such motion trajectories are replayed on real robots to complete the same task in real world. Experiments show that it achieves higher real2sim fidelity, and the resulting policies retain higher success rate.

Critical strengths
  • The paper is in generally clear written. The presentation is easy to follow.
Critical weaknesses
  • The proposed pipeline is a composition of existing tools. Unclear the key uniqueness of the proposed system.
  • Table 1 system-level comparison is superficial. Capabilities are cherry-picked to only differentiate the proposed pipeline.
  • The real2sim stage reconstructs exactly the same real-world scenes. However, many studies, such as ACDC (Dai et al., 2024), have found that reconstructing similar but variant scenes help the robustness and generalization.
  • The robot registration still requires manual alignment. However, many existing pipelines can already automate this step.
  • The sim2real step simply replays the motion trajectory on in the real world, which limits the robotic tasks that this system is able to perform. As as result, the proposed pipeline is only demonstrated on simple table-top pick-and-place tasks, with a simple single-arm manipulator.
  • Experiment results are limited and weak. Most efforts are put on evaluating the real2sim fidelity. The robot success rate is very low, only 55% even in sim. Given the simplicity of the tasks, it's not impossible that this system has some major downsides.
  • Real-world success is also low, further showing that the proposed system is not able to handle robust real2sim transfer.
Questions for authors

See above.

3 comments

Thank you for the review. We would like to clarify four factual points that materially affect the assessment of our method. Each clarification below refers to evidence already present in the original submission.

1. Real-robot execution uses fresh observations, rather than replaying simulated trajectories.

The review states that the sim-to-real step “simply replays the motion trajectory” in the real world. However, the original submission already describes repeated observations and actions within a single program execution in §3.2 (p. 5, lines 264–269). Section 3.3 further requires the agent to “re-localize objects and assess task state from current real observations rather than reuse simulated coordinates or success claims” (p. 6, lines 275–281), with observations before and after consequential actions (lines 282–289). The transferred policy therefore determines actions from current real-world observations. A locked policy version fixes the code, not its observation-dependent action coordinates. This directly addresses the concern that hardware execution is restricted to replaying a simulated trajectory.

2. Robot registration is automatic.

The review states that robot registration “still requires manual alignment.” However, the original submission already specifies automatic transform estimation in §3.1: multi-scale Sim(3) registration followed by pose and scale refinement (pp. 3–4, lines 155–182). Appendix B.1 specifies the scoring objective, and Table 6 specifies 128 rotation initializations, refinement of the top three candidates, and selection by the lowest final loss (p. 14). The registration transform is determined by this automatic procedure. The original method does not require an operator to manually choose the alignment.

3. Real-to-Sim reconstruction and randomized policy development are separate stages.

The review contrasts reconstructing the captured workspace with ACDC's use of varied scenes for robustness. However, the original submission already includes randomized policy development and explicitly separates these two stages. The Real-to-Sim stage (§3.1) reconstructs the captured workspace, preserving its metric scale, task-relevant geometry, appearance, and interaction structure (p. 3, lines 147–153). The subsequent policy-development stage (§3.2) introduces randomization: after privileged and visual development in the nominal scene, the third pass varies initial object positions, camera pose and calibration, and action execution (p. 5, lines 233–242; Fig. 4). Appendix D.2 states the same randomization settings (p. 23, lines 1222–1223), and §4.3 reports evaluation under randomized simulation conditions (p. 7, lines 356–357). Reconstruction fidelity is the objective of the first stage; adaptation to perturbations is addressed in the subsequent stage. Reconstructing the captured workspace does not mean that policy development uses only an unvarying scene.

4. The evaluated task set is not limited to pick-and-place.

The review describes the experiments as “only demonstrated on simple table-top pick-and-place tasks.” However, the original submission already lists drawer closing (F1), placing a cup and closing the box (F3), bowl stacking (F5), and tissue extraction (P9), alongside placement tasks in Appendix A, Table 5 (p. 14). The evaluation comprises 18 tasks across Franka FR3 and Agilex Piper, with nine tasks per platform. Describing this task catalog as exclusively pick-and-place omits the articulation, stacking, and sequential-action tasks already listed in the original submission.

We hope these references clarify the method and task setting. Could you please clarify which substantive concerns remain after these corrections?

Thanks authors for clarification, which further increases my confidence in my original assessment. My concerns regarding this submission still remain, largely:

  • Unclear/limited innovations.
  • Lack of clear system-level comparison.
  • Demonstrated tasks are trivial (I don't want to argue what are PnP tasks, but some tasks such as drawer closing are even simpler than PnP). Also unclear how the proposed method can work with more complicated robot embodiments.
  • Low success rates in both sim and real. It's possible the whole system has certain bugs.

1. Contributions in Real-to-Sim and Sim-to-Real. In Real-to-Sim (§3.1), we reconstruct a task-ready simulation from a walk-around RGB video. The agent builds movable and articulated objects with collision geometry and joints, then uses MuJoCo to test task-relevant interactions and correct configurations that fail those tests. In policy development (§3.2), privileged object poses accelerate the development of task logic. The agent then switches to visual observations and refines the policy under perturbations to object poses, camera calibration, and action execution. These perturbations represent errors the robot can encounter on hardware, such as camera misalignment and deviations from commanded poses. The resulting program and execution experience transfer through a shared interface, and the robot re-localizes objects and acts on fresh observations (§3.3).

Among the systems in Table 1 and the policy baselines Code as Policies, CaP-X, and Instruct2Act, Agentic RSR is the only one to demonstrate this complete loop for the same physical task. It starts with a workspace video, a task description, and a known robot model, develops an executable policy in the reconstructed scene, and carries out the task on hardware using live observations.

2. Direct comparisons. The original submission compares four reconstruction baselines in Table 2 and two programmatic policy baselines in Table 3. The rebuttal adds two controlled comparisons on the same 18 simulation tasks. With the policy pipeline held fixed, our scenes supported 14/18 completions versus 7/18 with SimFoundry scenes. With the scenes held fixed, our policy pipeline completed 14/18 versus 9/18 for CaP-X. These comparisons test the effects of scene construction and policy development separately.

3. Task difficulty and embodiment. Our evaluation covers 18 tasks across two robot platforms, Franka FR3 and Agilex Piper. The suite includes stacking and storage tasks comparable to those evaluated by CaP-X and SimFoundry, as well as articulated manipulation and multistep tasks. For example, placing a cup inside a cardboard box and then closing it (F3) requires the reconstructed scene to support both cup placement and lid motion, and the visual policy to coordinate those interactions in sequence. Bowl stacking and tissue extraction add distinct contact and handling demands. Drawer closing is one task in this broader suite.

4. Success rates and the suggested bug. Our protocol is zero-shot with respect to target-task demonstrations. It uses no teleoperated demonstrations, task-specific training datasets, or model fine-tuning. The agent develops each program through interaction with the reconstructed simulator. Under this protocol, the full method completed 14/18 tasks in the additional simulation evaluation and 8/18 in the original hardware evaluation. The follow-up identifies no failing component, reproducible error, or other evidence of a software defect. The suggestion that a bug explains the reported rates remains speculative.

Reviewer_9dFxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Agentic RSR, a real-to-sim-to-real framework that combines agent-based scene reconstruction with coding robot policy development. Given a workspace video and task description, the proposed pipeline reconstructs an interactive simulation environment, develop and refine robot policies through simulation evaluations, and transfer them to real robots. Experiments cover 18 manipulation tasks across two robot platforms, achieving 44.4% real-world task success and 80% transfer among simulation-successful tasks.

Critical strengths
  1. The paper proposed a general framework provides an end-to-end workflow that could potentially reduce the manual effort of building simulation environment for robotic policy training and evaluation.

  2. The authors evaluate the framework on two real robot platforms and provide quantitative scene reconstruction results, realworld manipulation experiments, and conduct ablation studies. These experiments offer useful insights into the potential of coding agents for robot policy development.

Critical weaknesses
  1. The paper's main contribution is not clear enough. The authors claim the core contribution is to combine agentic scene reconstruction with coding-agent policy development into an end-to-end framework. Although the integration is potentially useful, the technical contribution is not sufficiently verified. Much of the experimental evaluation focuses on individual components rather than demonstrating the benefits of connecting them. I suggest clarifying the main research insight and strengthening the evaluation of the system.

  2. The ICP-based Sim(3) registration is fragile. Because it depends on the downstream performance of the video quality, 3D reconstruction, and segmentation models, any outliers in the point clouds could easily cause alignment failures. The heuristic initializations and scoring criteria help, but the paper still lacks rigorous testing against occlusion, depth noise, or bad segmentation.

  3. Existing reconstruction methods are mainly compared on scene quality, but it doesn't necessarily prove the simulation environment generated with the proposed method is better for robotic policy training. The paper also lacks controlled experiments separating the contributions of scene reconstruction, physical verification, experience memory, and runtime recovery to realworld robot performance.

  4. The main reconstruction metrics are evaluated using the same reference views used during refinement, and the reference depth is generated by a pretrained reconstruction model rather than independent ground truth. Although object-level geometric measurements are provided, it will be more convincing if the author evaluate the reconstruction quality on unseen camera views.

Questions for authors
  1. How well can the coding agent recognize and recover from unexpected real-world errors, such as failed grasps or inaccurate object localization?
  2. How effective is the agent-generated simulation environment in facilitating policy development?
  3. What is the average development time and computational cost required for each task?

The paper presents an interesting end-to-end framework that combine both agentic simulation environment creation and policy development. However, given the current concerns I have, I won't give a positive score at this moment, stronger experiments and clearer presentation are needed to further justify the effectiveness and contribution of the proposed pipeline

3 comments

Thank you for recognizing the potential of the end-to-end workflow and our evaluation on two real-robot platforms. We address your questions below.

1. Core research question and downstream policy effectiveness.

Our method makes specific contributions to both Real-to-Sim reconstruction and policy development and Sim-to-Real transfer, connecting them through a shared task-level interface into a complete Real-to-Sim-to-Real pipeline.

Real-to-Sim (§3.1): Agents iteratively reconstruct the scene and configure and check task objects’ collision geometry, mobility or articulation, and task-relevant physical interactions, enabling the reconstructed objects to participate in simulated manipulation. Section 4.2, Table 2 already compares reconstruction quality with Re3Sim, RL-GSBridge, RoboSimGS, and SimFoundry; our method achieves lower reported color error and higher SSIM than these baselines. We have since migrated the agent execution platform to DSH and further evaluated GPT-6 Astra, GPT-5.6 Sol, GPT-6.1 Sol, Claude Fable 5.1, Claude Opus 5.5, and Claude Sonnet 5.5 on the same 18 scenes. DSH / Claude Opus 5.5 achieves mean Depth MAE of 0.08078 m, Lab ΔE76 of 8.0104, and SSIM of 0.73375, further reducing depth and color errors and increasing SSIM relative to the original configuration.

Policy development and Sim-to-Real (§§3.2–3.3): Our staged development progresses from privileged poses to visual observations and then randomized visual observations. This first establishes task decomposition and action logic, then verifies visually grounded execution, and finally tests and refines the policy under perturbations to initial object positions, camera parameters, and action execution, targeting efficient development and robust execution. The shared task-level interface then transfers both the programmatic policy and experience memory to hardware. Memory carries effective behaviors, failure causes, and perception experience accumulated in simulation, while fresh real observations guide continuation, retries, and recovery.

Together, these advances connect reconstruction of a real workspace, policy development in simulation, and experience-informed execution on hardware.

2. Robot registration and robustness.

Robot registration aligns the video reconstruction with known robot geometry to recover metric scale and establish the robot coordinate frame (§3.1). For the standard walk-around videos used in our pipeline, the original multi-scale Sim(3) registration and candidate scoring are sufficient to provide this geometric reference (§3.1 and Appendix B); in the additional experiments, the original method successfully registers all five unperturbed scenes. This module uses existing techniques, is not a core contribution of this work, and can accommodate alternative methods. Thank you for encouraging further evaluation under occlusion, depth noise, and segmentation errors. Following your suggestion, we compared the original method, MegaPose [1], FoundPose [2], and PicoPose [3] on 5 Franka scenes under 52 clean or perturbed conditions, giving 260 scene–condition trials per method.

MethodPose successes / trialsPose success ↑CAD depth MAE (cm) ↓Mean fitting time (s)
Original (Sim(3) ICP)226/26086.92%17.271082.95
MegaPose257/26098.85%10.06250.93
FoundPose183/26070.38%9.3116.36
PicoPose253/26097.31%11.3447.57

For the disturbances you highlighted, the following table reports the highest tested intensity in each category. Random perturbations use 5 scenes × 3 seeds; deterministic erosion uses 5 scenes:

Test conditionOriginalMegaPoseFoundPosePicoPose
Clean5/55/55/55/5
Relative depth noise σ = 0.156/1515/1515/1515/15
Target 3D occlusion fraction 0.407/1512/152/1510/15
Target mask dropout fraction 0.4015/1515/151/1514/15
Mask erosion fraction 0.051/55/55/55/5

Pose success requires rotation error ≤10° and base-origin translation error ≤0.20 m relative to a sensor-depth-constrained pseudo-reference.

MegaPose achieves the highest overall pose success rate in these additional tests (257/260, 98.85%). Based on these results, we will consider replacing the current Sim(3) ICP with MegaPose as the default robot registration method in our system.

References

[1] Yann Labbé et al. MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare. CoRL, 2022.

[2] Evin Pınar Örnek et al. FoundPose: Unseen Object Pose Estimation with Foundation Features. ECCV, 2024.

[3] Lihua Liu, Jiehong Lin, Zhenxin Liu, and Kui Jia. PicoPose: Progressive Pixel-to-Pixel Correspondence Learning for Novel Object Pose Estimation. 2025.

3. Contributions of reconstruction, physical checks, memory, and runtime error detection and recovery.

Thank you for suggesting further analysis of how these stages contribute to hardware execution. They are connected within the same task workflow: reconstruction provides the task scene, physical checks examine the simulated interactability of task objects, policy development establishes action and observation logic in that scene, and experience memory carries development experience into hardware execution. Assessing an individual stage therefore also requires considering how it changes the scene, information, and decisions encountered downstream.

Reconstruction and physical checks. Section 3.1 describes how visual reconstruction is followed by configuration of collision geometry, mobility, or articulation, together with settling, simplified grasp, or articulated-motion probes in MuJoCo. Adjustments to collision geometry, object poses, or physical parameters change the environment used for subsequent policy development; visible geometry or pose changes also trigger regression checks from the reference views. This stage therefore directly affects the input to policy development. We are adapting scene outputs from other Real-to-Sim methods to a common policy pipeline to further compare the downstream effects of different scene sources.

Checkpoints and error detection. Section 3.2 and Appendix D.2 describe optional checkpoints beyond the basic observation and action interfaces. A program can yield control while preserving the current scene, allowing the outer agent to inspect intermediate outcomes and decide whether to continue, retry, or recover. For example, if observations after a grasp do not show the expected object state, or subsequent observations indicate that the target has moved, the agent can re-observe and re-localize before selecting subsequent steps. Error detection is driven by current observations and execution feedback, without a simulator-provided task-success signal.

Why memory and recovery are closely coupled. Section 3.3 transfers both the programmatic policy and experience memory to hardware. The former retains task steps and observation–action logic; the latter records effective behaviors, failure causes, perception difficulties, and actions to try or avoid during simulation development. The agent combines this experience with fresh real observations to decide what to do next. Recovery thus spans yielding control, inspecting intermediate outcomes, selecting the next step using experience, and resuming execution. Removing it requires specifying how execution proceeds after a checkpoint, whether failures terminate execution, and how subsequent observations influence action selection. Removing memory likewise changes the decision-relevant information available under the same observations. Because these mechanisms are closely integrated with task execution, attributing a success-rate contribution by simply disabling an independent module is difficult; an informative control also needs to specify these execution rules.

In our current observations, approximately 20% of final scripts involve resumption after an interruption or recovery-like operations in simulation; after transfer to hardware, approximately 60% of tasks involve similar operations. These observations describe the use of checkpoints, re-observation, and recovery in actual tasks. The CaP comparison in §4.3, Table 3 provides a reference for assessing the overall execution mechanism: our CaP baseline transfers the programmatic policy without adding an outer agent to select subsequent steps at checkpoints or initiate recovery after failure, whereas our method allows the agent to determine the next step from fresh real observations and execution outcomes.

4. Development time and computational cost.

For tasks that complete successfully, simulation policy development takes an average of 9.7 minutes per scene, and hardware execution takes an average of 13.9 minutes, totaling 23.6 minutes for these two stages. Together with registration and scene-reconstruction timings, these measurements allow us to summarize the average time per task by stage.

We initially chose Codex because its computer-use capabilities made it convenient to integrate and operate Blender. As the Blender Python interface and MCP tooling matured, we migrated the entire project to the open-source DeepSeek Harness (DSH), making it easier to integrate and test more models through common tool interfaces and to improve reproducibility of the experimental workflow. Codex and DSH are agent execution frameworks; the underlying model is a separate choice.

In the DSH evaluation on the original 18 scenes, Opus 5.5 and Fable 5.1 both pass 18/18. Below we separately report seven-model quality, time, and cost on five recaptured scenes. This batch uses Odin 1 measured depth, is summarized separately from the 18-scene VGGT-Ω-reference results above, and includes different model versions, such as Fable 5.2.

DSH modelMetric pass / 5 scenesDepth MAE (m) ↓Lab ΔE76 ↓SSIM ↑Active time (min) ↓Mean token cost (USD/scene) ↓
GPT-6 Astra5/50.11049.170.73530.307.3531
GPT-6.1 Sol4/50.116711.750.71326.621.1505
GPT-5.6 Sol5/50.107510.920.72936.965.0323
Claude Opus 5.55/50.08568.040.70251.1610.4395
Claude Fable 5.25/50.077311.200.66164.2327.6492
DeepSeek Flash5/50.096711.400.73130.120.5643
Grok 4.70/50.092629.600.54424.408.5414

The evaluation uses five captured scenes (seq01–05), each with four reference views. Depth MAE is calculated in regions with reference depth between 0 and 1.5 m; token costs are averaged over five scenes.

Using DSH / GPT-6 Astra for reconstruction, the recorded mean times by stage are:

StageMean time (minutes)
Robot registration: original Sim(3) ICP18.05
Scene reconstruction: DSH / GPT-6 Astra30.30
Simulation policy development (successful tasks)9.70
Hardware execution (successful tasks)13.90
Sum of the stage means above71.95

The registration time is converted from the 1082.95 seconds reported in Section 2 of this response, and the scene-reconstruction time comes from the table above. Summing the recorded stage means therefore gives approximately 72.0 minutes per task with the original registration method and DSH / GPT-6 Astra. If the registration module is replaced with MegaPose, using its measured mean fitting time of 250.93 seconds (4.18 minutes), the corresponding sum is approximately 58.1 minutes. These totals are estimates obtained by summing stage means, with the simulation and hardware stages averaged over successful tasks.

For token costs, using GPT-6 Astra and OpenRouter pricing, simulation policy development and hardware execution cost an average of USD 2.4127 per task and USD 3.2685 per task, respectively. Including the scene-reconstruction cost reported above, the mean token costs by stage are:

StageMean token cost (USD/task)
Scene reconstruction7.3531
Simulation policy development2.4127
Hardware execution3.2685
Total13.0343

Summing these stage means gives an average token cost of approximately USD 13.03 per task for scene reconstruction, simulation policy development, and hardware execution using GPT-6 Astra.

5. Reference depth and unseen views.

Thank you for the suggestions on depth reference and evaluation views. The original depth reference in §3.1 is VGGT-Ω-estimated depth after robot-based metric scaling; it is a model-derived reference, not independent sensor-measured ground truth. VGGT-Ω is a visual foundation model with strong multi-view geometry estimation capabilities, and we use it to provide a metric-scale depth reference for reconstruction. To further evaluate both this reference and the reconstruction, we collected measured depth for five scenes using Odin 1, a device combining a LiDAR array and a camera with a stated accuracy of ±3 cm (1σ). We compare the reconstructed scene, VGGT depth estimates, and Odin 1 measurements separately:

SceneReconstruction vs. Odin 1 MAE (m) ↓Reconstruction vs. VGGT-Ω MAE (m) ↓VGGT-Ω vs. Odin 1 MAE (m) ↓
Scene 10.10170.06330.0978
Scene 20.11780.11030.1226
Scene 30.13990.14910.1367
Scene 40.09030.10310.0699
Scene 50.10260.11500.0483
Scene-equal mean0.11040.10820.0951

The reconstruction uses DSH / GPT-6 Astra. On the reference views, mean reconstruction MAEs against VGGT-Ω and measured depth are 0.1082 m and 0.1104 m, respectively, and are numerically close.

We also held out a subset of views from the captured sequences. These views were not provided to the pipeline and did not participate in scene reconstruction or iterative refinement. After reconstruction, we evaluated the final scenes from these held-out views:

SceneDepth MAE (m) ↓SSIM ↑Lab ΔE76 ↓
Scene 10.10240.73211.92
Scene 20.12030.7029.14
Scene 30.14070.7248.53
Scene 40.09000.7517.75
Scene 50.10150.7777.70
Scene-equal mean0.11100.7379.01

These results supplement the reference-view evaluation by directly assessing final reconstruction quality from views excluded from reconstruction and refinement.

Reviewer_dJRYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper suggests agentic real-to-sim-to-real (RSR), showing the full agentic control over the scene reconstruction and policy development, having the RSR into a closed-loop framework. It uses depth and segmentation masking, as well as a point cloud representation for the simulation representation. The conversion rate is 80%, and for individual tasks, the performance outperform a competitive baseline by a good margin.

Critical strengths
  • The time is right: many decision points in the RSR stages may be automated via a coding agent, accelerating robotic research.
  • Presentation is strong: examples and figures help understand the concepts well.
  • Clarity: with codes shared and versions well-defined, it shows a great reproducibility.
Critical weaknesses
  • On robustness over agents: performance would vary largely by coding agents yet there's no particular justification for using CODEX/astra. do performance value significantly by agents used? Have you checked smaller / cheaper models? Is there any particular reason for using CODEX only.

  • On baseline: baselines do not consist of non-agentic, conventional pipelines.

Questions for authors
  • In how many instances did real-world robot performance actually change as a result of runtime recovery or experience memory?

  • What failure modes explain the two tasks that succeeded in simulation but failed on hardware?

  • Can you compare against non-agentic counterparts, such as differentiable-rendering scene fitting and the same final policy run without runtime agent intervention, to isolate the contribution of the agent itself?

2 comments

Thank you for your positive assessment of the framework, presentation, and reproducibility. Below, we address model choice and cost, non-agentic controls, recovery and experience memory, and the causes of hardware transfer failures.

1. Model choice, model dependence, and cost-effectiveness (Weakness 1).

We initially chose Codex because its computer-use capabilities made it convenient to operate Blender and integrate it into our workflow. As the Blender Python interface and MCP tooling matured, we migrated the entire project to the open-source DeepSeek Harness (DSH), making it easier to integrate and test more models through common tool interfaces and to improve reproducibility of the experimental workflow. Codex and DSH are agent execution frameworks; the underlying model is a separate choice.

Section 4.4, Table 4 already reports a reconstruction-stage model comparison. We further evaluated GPT-6 Astra, GPT-5.6 Sol, GPT-6.1 Sol, Claude Fable 5.1, Claude Opus 5.5, and Claude Sonnet 5.5 through DSH on the same 18 scenes. Opus 5.5 and Fable 5.1 both pass the visual check on all 18 scenes. Opus 5.5 achieves mean Depth MAE of 0.08078 m, Lab ΔE76 of 8.0104, and SSIM of 0.73375, further reducing depth and color errors and increasing SSIM relative to the original configuration. These depth metrics retain the robot-scaled VGGT-Ω reference.

To further examine lower-cost models, we separately tested seven models on five recaptured scenes, recording reconstruction quality, active execution time, and mean token cost per scene. This batch uses Odin 1 measured depth (a LiDAR array and camera, with stated accuracy of ±3 cm at 1σ), evaluating depth MAE in regions with reference depth between 0 and 1.5 m. We report it separately from the 18-scene experiment; model versions also differ.

DSH modelScenes passing the visual criteria / 5Depth MAE (m) ↓Lab ΔE76 ↓SSIM ↑Active time (min) ↓Mean token cost (USD/scene) ↓
GPT-6 Astra5/50.11049.170.73530.307.3531
GPT-6.1 Sol4/50.116711.750.71326.621.1505
GPT-5.6 Sol5/50.107510.920.72936.965.0323
Claude Opus 5.55/50.08568.040.70251.1610.4395
Claude Fable 5.25/50.077311.200.66164.2327.6492
DeepSeek Flash5/50.096711.400.73130.120.5643
Grok 4.70/50.092629.600.54424.408.5414

In this batch, DeepSeek Flash passes all five scenes at a mean token cost of USD 0.5643 per scene, providing a lower-cost reconstruction configuration. These results show that the method can be implemented with models beyond Codex/Astra and that model choice affects quality and cost. Therefore, Codex/Astra is not a required configuration for our method.

2. Non-agentic controls (Weakness 2; Question 3).

Thank you for suggesting non-agentic controls to further isolate the agent's contribution. Section 4.2, Table 2 already compares scene quality with Re3Sim, RL-GSBridge, RoboSimGS, and SimFoundry. We are conducting downstream comparisons that connect non-agentic Real-to-Sim methods to a common policy pipeline. The substantially different scene output formats require adaptation first. We will provide results as they become available.

3. Effects of runtime recovery and experience memory (Question 1).

Section 3.2 and Appendix D.2 describe optional checkpoints in addition to the basic observation and action interfaces: a program can yield control while preserving the scene state, allowing the outer agent to inspect an intermediate outcome and decide how to proceed. Section 3.3 describes hardware execution with both policy and experience memory, using fresh real observations to continue, adjust, or stop. Memory records effective behaviors, failure causes, and perception difficulties encountered during simulation development.

In our current observations, approximately 20% of final scripts involve resumption after an interruption or recovery-like operations during simulation execution; after transfer to hardware, approximately 60% of tasks involve similar operations. Yielding control at checkpoints so that the agent can inspect intermediate outcomes and decide whether to continue, retry, or recover is part of our execution mechanism. Recovery is therefore closely integrated with observation, decision-making, and step execution, making it difficult to separate it as an independent module and directly attribute a success-rate improvement to it.

The CaP comparison in §4.3, Table 3 provides a reference for evaluating this overall execution mechanism. Our CaP baseline transfers the programmatic policy without adding an outer agent to select subsequent steps at checkpoints or initiate recovery after failure during hardware execution. Our method instead allows the agent to determine the next step from fresh real observations and execution outcomes.

4. Failures after successful simulation (Question 2).

Failures after successful simulation mainly fall into two categories:

  1. Safety stops: Robot actions exceed safety limits and trigger an automatic stop, or a human observer identifies a hazard and presses the emergency stop. For example, excessive downward force while placing a lid on a cup causes the robot to exceed its limits. Such stops also occur in stacking tasks, including stacking brown blocks and stacking snacks.
  2. Failed recovery and timeout after scene disturbance: The robot inadvertently disrupts the scene, for example by knocking over a transparent cup while placing glue inside it, brushing against other fruit during fruit storage and causing it to roll away, or accidentally tipping over a cup while placing its lid. After the disturbance, the robot cannot restore a state from which the task can continue and loses track of the manipulated objects. The agent then repeatedly attempts to re-localize the objects or recover until the execution time limit is reached.

Thank you for these suggestions. We will provide additional controlled results as they become available.

Reviewer_VtLQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Agentic RSR, an agentic Real-to-Sim-to-Real pipeline that (i) reconstructs a metric, task-relevant MuJoCo scene from a workspace video by anchoring scale to a known robot URDF and refining with reference-view depth/appearance feedback, (ii) develops an executable programmatic policy with a Codex agent through three passes (privileged poses → visual interface → randomized robustness), and (iii) transfers the locked policy plus experience memory to real robots (Franka FR3 and Agilex Piper) via a shared task-level interface with safety gating. On 18 scenes, the authors report mean four-view Depth MAE 0.1057 m, Lab ΔE76 11.04, and SSIM 0.6990. For policies, Agentic RSR reports 55.6% Sim SR and 44.4% Real SR (8 of 10 simulation-successful tasks also succeed on hardware, i.e., 80% paired conversion), outperforming Code-as-Policies and Instruct2Act under the paper’s reporting protocol. Ablations focus on reconstruction (model swap, metric feedback, initial calibration).

Critical strengths
  1. End-to-end task-level Real-to-Sim-to-Real closed loop. Unlike papers that stop at scene assets or at sim-only code policies, the work connects workspace-video scene reconstruction, simulation-based programmatic policy development, and a shared task-level interface for transfer into a single pipeline, and evaluates reconstruction and policies on the same 18-task catalog. Coupling these stages in one loop, rather than evaluating each in isolation, is a useful systems recipe for readers building agentic Real-to-Sim-to-Real stacks.
  2. Real-robot experiments on two embodiments validate sim-to-real transfer. Beyond simulation, the locked policies are deployed on physical Franka FR3 and Agilex Piper robots through the shared task-level interface with safety gating, reaching 44.4% Real SR, with 8 of 10 simulation-successful tasks also succeeding on hardware. This hardware evidence makes the transfer claims more credible than sim-only evaluations of code policies and is directly relevant to readers who want deployable agentic systems.
Critical weaknesses
  1. Positioning and empirical comparison with closely related systems could be strengthened. While the paper cites key neighboring works, it does not compare against them head-to-head. Table 1 positions Agentic RSR as the only framework spanning reconstruction, agentic revision, code policy, runtime recovery, and sim-to-real. However, relying primarily on a capability checklist may overstate the system's uniqueness, given that Agentic Real2Sim, SimFoundry, Re3Sim, and CaP-X already address substantial overlapping subsets of this pipeline. For an ICLR contribution, the paper would benefit from a sharper, falsifiable scientific claim regarding the RSR coupling itself, supported by direct empirical comparisons—at least along the shared axes—against Agentic Real2Sim / SimFoundry (for reconstruction-to-downstream policy) and CaP-X (for execution-grounded code transfer). Without these comparisons, the contribution currently reads more as a well-engineered system integration than a distinct algorithmic or scientific advance.
  2. Several aspects of the policy evaluation and headline transfer metrics make direct interpretation difficult.
  • (a) The hardware scenarios for Code-as-Policies (CaP) reportedly "do not fully match the indexed tasks," meaning the Real SR and conversion comparisons against CaP are descriptive rather than strictly paired—even though their presentation in Table 3 invites direct comparison.
  • (b) Instruct2Act and Agentic RSR appear to use different definitions of sim-to-real conversion (an overall rate ratio vs. paired retention conditioned on simulation success), which complicates cross-method comparison.
  • (c) Success is reported as a task-level binary outcome, but key evaluation protocol details—such as attempt budgets, random seeds, and whether test-time runtime agent recovery is applied uniformly across methods—remain unclear.
  • (d) The Depth MAE results and corresponding claims in the abstract could be misread as measuring absolute metric reconstruction accuracy, whereas they appear to be evaluated relative to the same video-reconstruction depth prior. Taken together, these evaluation ambiguities make it difficult to fully assess the claimed performance gains and the headline "80% retention" rate.
Questions for authors
  1. Regarding Depth MAE: Do you have access to independent metric ground truth (e.g., from a calibrated depth sensor or MoCap object poses) on a subset of scenes? If not, would you consider rephrasing the claims in the abstract and Table 2 to more accurately reflect "consistency with the VGGT metric reference," and potentially reporting sensitivity to the choice of depth estimator?
  2. Regarding real-robot evaluation protocol: At test time on the physical robot, is the Codex agent permitted to revise the program online, or is it restricted to selecting and retrying steps within a locked bundle? Could you please clarify the wall-clock time budget, the maximum number of retries, and whether the baseline methods are granted the same runtime recovery privileges?
  3. Regarding policy-level ablations: The current ablation study focuses primarily on the reconstruction stage. Could you share how Sim SR and Real SR are affected if you ablate key policy and transfer components—specifically, removing (i) the privileged first pass, (ii) experience memory during transfer, or (iii) the MuJoCo interaction probes?
4 comments

Thank you for recognizing our workflow connecting reconstruction, policy development, and real-robot execution, and our experiments on two robot platforms. Below, we clarify the contributions, evaluation protocol, and depth reference, and provide additional results.

1. Contributions and comparisons with related systems (Weakness 1).

Our method makes specific contributions to both Real-to-Sim reconstruction and policy development and Sim-to-Real transfer, connecting them through a shared task-level interface into a complete Real-to-Sim-to-Real pipeline.

Real-to-Sim (§3.1): Agents iteratively reconstruct the scene and configure and check task objects’ collision geometry, mobility or articulation, and task-relevant physical interactions, enabling the reconstructed objects to participate in simulated manipulation. Section 4.2, Table 2 already compares reconstruction quality with Re3Sim, RL-GSBridge, RoboSimGS, and SimFoundry; our method achieves lower reported color error and higher SSIM than these baselines. We have since migrated the agent execution platform to DSH and further evaluated GPT-6 Astra, GPT-5.6 Sol, GPT-6.1 Sol, Claude Fable 5.1, Claude Opus 5.5, and Claude Sonnet 5.5 on the same 18 scenes. DSH / Claude Opus 5.5 achieves mean Depth MAE of 0.08078 m, Lab ΔE76 of 8.0104, and SSIM of 0.73375, further reducing depth and color errors and increasing SSIM relative to the original configuration. Depth evaluation retains the robot-scaled VGGT-Ω reference.

Policy development and Sim-to-Real (§§3.2–3.3): Our staged development progresses from privileged poses to visual observations and then randomized visual observations. This first establishes task decomposition and action logic, then verifies visually grounded execution, and finally tests and refines the policy under perturbations to initial object positions, camera parameters, and action execution, targeting efficient development and robust execution. The shared task-level interface then transfers both the programmatic policy and experience memory to hardware. Memory carries effective behaviors, failure causes, and perception experience accumulated in simulation, while fresh real observations guide continuation, retries, and recovery. Section 4.3, Table 3 already compares simulation and hardware policy outcomes with Code as Policies and Instruct2Act. Under our policy-and-experience transfer workflow, 80% of simulation-successful tasks also succeed on hardware.

Agentic Real2Sim, SimFoundry, Re3Sim, and CaP-X primarily focus on either scene reconstruction or programmatic policy development and transfer, as discussed in §2. Our work introduces specific methodological improvements in both Real-to-Sim and policy development and Sim-to-Real: the former provides interactive, visually faithful task scenes, while the latter develops policies in those scenes and transfers both policy and experience to hardware. Together, these advances enable the complete pipeline from reconstruction of a real workspace through simulation policy development to real-world task execution.

To further evaluate how different reconstruction methods affect downstream policies, we are adapting scene outputs from other Real-to-Sim methods to a common policy pipeline. Their scene output formats differ substantially and require adaptation first; we will provide the results as they become available. The CaP-X comparison you requested is not yet complete; we will provide the results when available.

2. Task metrics and conversion (Weakness 2(a–b)).

Thank you for pointing out the issues in our description of task metrics and conversion. The original submission used different descriptions of conversion across methods, which was insufficiently precise. The common aggregate reporting definition is conversion = 100 × Real SR / Sim SR, computed from unrounded success rates. Section 4.1 specifies how success rates are aggregated, and §4.3, Table 3 reports the results for the three methods.

CaP and Instruct2Act transfer their respective programmatic policies, whereas our method transfers both a programmatic policy and experience memory (§3.3). The original submission describes our conversion as paired retention. Because every hardware-successful task in our evaluation also succeeds in simulation, the aggregate rate ratio and paired retention are numerically identical for this task set. The clarified reporting therefore leaves the existing results unchanged.

For CaP, Real SR is the task-equal mean success rate over recorded hardware attempts, and some hardware scenarios do not fully match the indexed tasks (§§4.1 and 4.3). Its hardware comparison should therefore be interpreted as descriptive.

3. Time budgets and runtime intervention (Weakness 2(c); Question 2).

All methods use the same overall time limits: 30 minutes in simulation and 15 minutes on hardware. There is no fixed limit on the number of retries; retries are constrained only by the overall time budget. The simulation-development rules in Appendix D.2 permit retries and resets as needed within the overall time budget. For tasks that complete successfully, simulation policy development and hardware execution take an average of 9.7 minutes and 13.9 minutes, respectively. These are measured mean durations; the 30- and 15-minute values above are time limits.

Our method permits policy-code revision during development. During final evaluation, the policy bundle remains locked, while the agent uses fresh observations and execution feedback to select existing steps, continue, retry, or recover (§1, lines 87–89; §3.2, lines 240–242). Hardware execution requires re-localizing objects and observing before and after consequential actions to determine how to proceed (§3.3). Thus, the policy version is fixed while observation and action decisions remain responsive to the current state. The CaP comparison transfers the programmatic policy without adding an outer-agent mechanism to initiate retries after failure; the program itself may still contain perception and feedback control.

4. Depth MAE and independent measurements (Weakness 2(d); Question 1).

Section 3.1 already specifies that the original depth evaluation uses robot-scaled VGGT-Ω depth as a common reference, measuring consistency with that metric reference. VGGT-Ω is a foundation model with strong multi-view geometry estimation capabilities, but its estimated depth is not independent sensor-measured ground truth.

To directly evaluate both the reference and the reconstruction, we recaptured five scenes with measured depth using Odin 1, which combines a LiDAR array and a camera with a stated accuracy of ±3 cm at 1σ. We separately evaluate reconstruction versus measured depth, reconstruction versus VGGT-Ω, and VGGT-Ω versus measured depth:

SceneReconstruction vs. Odin 1 MAE (m) ↓Reconstruction vs. VGGT-Ω MAE (m) ↓VGGT-Ω vs. Odin 1 MAE (m) ↓
Scene 10.10170.06330.0978
Scene 20.11780.11030.1226
Scene 30.13990.14910.1367
Scene 40.09030.10310.0699
Scene 50.10260.11500.0483
Scene-equal mean0.11040.10820.0951

The reconstructed scenes were generated using DSH / GPT-6 Astra. We calculate depth MAE in regions where the reference depth is between 0 and 1.5 m.

The mean reconstruction MAEs against VGGT-Ω and measured depth are 0.1082 m and 0.1104 m, respectively, and are numerically close. VGGT-Ω has a mean MAE of 0.0951 m against measured depth. These measured-depth results provide independent evidence supplementing the original model-reference evaluation.

Thank you for the responses and additional experiments. These address several of my questions, although my main concerns about the contribution and its empirical support remain.

The conversion definition is now clearer. Given the stated inclusion of all hardware-successful tasks within the simulation-successful tasks, the reported 8/10 task-level retention is consistent with the aggregate rate ratio on the same, equally weighted task set. I also appreciate the explicit qualification that the unmatched CaP hardware results should be interpreted descriptively. The shared time limits and the distinction between development-time code revision and a frozen evaluation bundle clarify the execution protocol.

The five-scene Odin 1 measurements provide meaningful evidence beyond consistency with VGGT. I therefore no longer consider the absence of any independent depth measurement an outstanding issue. However, the new measurements use DSH/GPT-6 Astra, so their support should be scoped to the evaluated configuration. Calibration and registration details would help interpret the measured errors and their relevance to the manipulation tasks.

My main remaining concern is what the results establish beyond the integration of existing reconstruction and agentic policy-development capabilities. I acknowledge the reconstruction comparisons in Table 2 and the policy comparisons in Table 3. However, the common downstream-pipeline evaluation is still in preparation, and the requested CaP-X comparison is not yet complete. These planned experiments cannot yet resolve the question of which proposed improvements account for the downstream gains. I am not asking for full hardware replication of every related system, but a focused comparison testing the central contribution would be valuable.

The runtime clarification also leaves an attribution issue. Equal time limits are reasonable for comparing complete systems, but the proposed agent can select steps, retry, and recover, while CaP lacks the outer-agent retry mechanism. This does not by itself invalidate the system comparison, but it prevents attributing the gains specifically to reconstruction quality or policy transfer. Attempt counts and repeated-run information would help interpret the results. More importantly, the response does not provide the requested policy/transfer ablations for the privileged first pass, experience memory, or MuJoCo interaction probes. A targeted ablation addressing the central mechanism would be more informative than further description of the pipeline.

Overall, the rebuttal clarifies the reporting and adds useful independent geometric evidence. However, the missing controlled downstream comparison and policy/transfer ablations leave my central concerns unresolved. Based on the evidence currently provided, I am maintaining my rating.

Thank you for reading our rebuttal closely and for explaining which concerns the new evidence addressed. Your questions about the conversion metric and the distinction between development and evaluation helped us clarify both the 8/10 paired retention result and the use of a fixed policy bundle during evaluation. We also appreciate your request for a focused downstream comparison. It helped us design experiments that test the roles of scene reconstruction and policy development separately.

The experiments described as ongoing in our initial reply were completed and reported in our November 17 additional-results comment (OpenReview note MnVnSXMyuc, Table 1). We reproduce the full results below. All five settings were evaluated on the same 18 simulation tasks.

Method / ablationScene sourcePhysics checksPrivileged stageTasks completedSuccess rateAvg. development time (min)
Agentic RSR (full method)Ours✓✓14/1877.8%9.7
Using SimFoundry scenesSimFoundry✓✓7/1838.9%21.3
Without physics checksOurs—✓8/1844.4%13.6
Without the privileged stageOurs✓—10/1855.6%16.5
CaP-XOurs✓—9/1850.0%11.6

The SimFoundry comparison holds the policy pipeline fixed, and the CaP-X comparison holds the reconstructed scenes fixed. The two ablations measure how removing the privileged first pass or the MuJoCo physics checks affects simulation task completion. We would appreciate your assessment of these results in light of your concern about which design choices account for the downstream gains.

Comments on the paper

2 comments

As the rebuttal discussion period draws to a close, we sincerely thank all reviewers and the AC for their time and specific feedback throughout the review and discussion. These suggestions have helped us strengthen the experiments and clarify how the individual components contribute to downstream policy development.

Several reviewers raised related concerns about comparisons with existing systems, the role of reconstructed scenes in policy development, component ablations, and development efficiency. We therefore present the additional comparisons and ablations together here, along with supporting materials, for convenient reference. The correspondence to the reviews is as follows (numbering follows the order of items in each original review):

  • Reviewer VtLQ — Critical Weakness 1; items (i) and (iii) of Question 3: The SimFoundry-scene and CaP-X [1] comparisons address the request for direct comparisons with related systems. The ablations of the Privileged stage and physics checks address the simulation-performance aspects of Question 3 concerning the privileged first pass and MuJoCo interaction probes.
  • Reviewer 9dFx — Critical Weaknesses 1 and 3; Questions 2 and 3: Downstream task results using different scene sources and the physics-check ablation provide evidence for the connection between reconstruction and policy development. Simulated success rates and average development times address the effectiveness of the generated simulation environments for policy development and the average development time per task, respectively.
  • Reviewer i33g — Critical Weaknesses 1, 2, and 6: The additional system comparisons, component ablations, and simulation results on the 18 tasks provide further evidence concerning the role of the system design, direct empirical comparisons, and simulated task performance.
  • Reviewer dJRY — Critical Weakness 1: The accompanying quality vs. price comparison figure provides additional results on the trade-off between model quality and cost.

We reconstructed the experimental setup with DSH and Astra and reran the complete simulation evaluation on the same 18 tasks as in the original experiment. Table 1 summarizes the full method, the SimFoundry-scene comparison, the ablations of physics checks and the Privileged stage, and CaP-X.

Table 1: Ablation and comparison results.

Method / AblationScene sourcePhysics checksPrivileged stageSimulated tasks completed ↑Simulated success rate ↑Avg. development time (min) ↓
Ours (full method)Ours✓✓14/1877.8%9.7
Using SimFoundry scenesSimFoundry✓✓7/1838.9%21.3
Without physics checksOurs✗✓8/1844.4%13.6
Without the Privileged stageOurs✓✗10/1855.6%16.5
CaP-X [1]Ours✓✗9/1850.0%11.6

The full method completed 14 of the 18 tasks, achieving a simulated success rate of 77.8% with an average development time of 9.7 minutes. Using SimFoundry scenes completed 7 tasks, with a simulated success rate of 38.9% and an average development time of 21.3 minutes. In the ablations, removing the physics checks or the Privileged stage resulted in simulated success rates of 44.4% and 55.6%, respectively.

In this evaluation, the full method completed 7 more tasks than the SimFoundry-scene setting, and 6 and 4 more tasks than the settings without physics checks and without the Privileged stage, respectively. CaP-X completed 9 tasks with an average development time of 11.6 minutes. The full method achieved the highest simulated success rate and the shortest average development time among the evaluated settings, providing further evidence for the roles of task-scene construction, physics checks, and staged policy development.

We have packaged the videos of the experimental results, the quality vs. price comparison figure, and all 18 final simulation environments in an anonymous materials repository for the reviewers and AC to inspect.

We look forward to your feedback on these additional experiments and materials, and welcome any further questions or requests for clarification. We will continue to follow the discussion closely and do our best to provide thorough, clear, and satisfactory answers before the discussion period ends. Thank you again to all reviewers and the AC for your time and suggestions.

References

[1] Letian Fu et al. CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation. arXiv:2603.22435, 2026. https://arxiv.org/abs/2603.22435.

Area_Chair_sEVHOpenReview ↗

reviewer discussion

Dear Reviewers,

The authors have submitted their rebuttal. Please read their response carefully and engage in the discussion as soon as possible. Let us know whether the rebuttal addresses your concerns and update your final rating accordingly.

Thank you for your time and service to ICLR.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.