- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This submission introduces Agentic RSR, an agentic framework for real2sim2real transfer of robot manipulation policies. It consists of a real2sim stage, which reconstructs the real-world scene into MuJoCo given a real-world video. Given the reconstructed interactive simulation, a few coding agents devise executable motion trajectories for task completion. Afterwards, such motion trajectories are replayed on real robots to complete the same task in real world. Experiments show that it achieves higher real2sim fidelity, and the resulting policies retain higher success rate.
Critical strengths
- The paper is in generally clear written. The presentation is easy to follow.
Critical weaknesses
- The proposed pipeline is a composition of existing tools. Unclear the key uniqueness of the proposed system.
- Table 1 system-level comparison is superficial. Capabilities are cherry-picked to only differentiate the proposed pipeline.
- The real2sim stage reconstructs exactly the same real-world scenes. However, many studies, such as ACDC (Dai et al., 2024), have found that reconstructing similar but variant scenes help the robustness and generalization.
- The robot registration still requires manual alignment. However, many existing pipelines can already automate this step.
- The sim2real step simply replays the motion trajectory on in the real world, which limits the robotic tasks that this system is able to perform. As as result, the proposed pipeline is only demonstrated on simple table-top pick-and-place tasks, with a simple single-arm manipulator.
- Experiment results are limited and weak. Most efforts are put on evaluating the real2sim fidelity. The robot success rate is very low, only 55% even in sim. Given the simplicity of the tasks, it's not impossible that this system has some major downsides.
- Real-world success is also low, further showing that the proposed system is not able to handle robust real2sim transfer.
Questions for authors
See above.
3 comments
Thanks authors for clarification, which further increases my confidence in my original assessment. My concerns regarding this submission still remain, largely:
- Unclear/limited innovations.
- Lack of clear system-level comparison.
- Demonstrated tasks are trivial (I don't want to argue what are PnP tasks, but some tasks such as drawer closing are even simpler than PnP). Also unclear how the proposed method can work with more complicated robot embodiments.
- Low success rates in both sim and real. It's possible the whole system has certain bugs.
1. Contributions in Real-to-Sim and Sim-to-Real. In Real-to-Sim (§3.1), we reconstruct a task-ready simulation from a walk-around RGB video. The agent builds movable and articulated objects with collision geometry and joints, then uses MuJoCo to test task-relevant interactions and correct configurations that fail those tests. In policy development (§3.2), privileged object poses accelerate the development of task logic. The agent then switches to visual observations and refines the policy under perturbations to object poses, camera calibration, and action execution. These perturbations represent errors the robot can encounter on hardware, such as camera misalignment and deviations from commanded poses. The resulting program and execution experience transfer through a shared interface, and the robot re-localizes objects and acts on fresh observations (§3.3).
Among the systems in Table 1 and the policy baselines Code as Policies, CaP-X, and Instruct2Act, Agentic RSR is the only one to demonstrate this complete loop for the same physical task. It starts with a workspace video, a task description, and a known robot model, develops an executable policy in the reconstructed scene, and carries out the task on hardware using live observations.
2. Direct comparisons. The original submission compares four reconstruction baselines in Table 2 and two programmatic policy baselines in Table 3. The rebuttal adds two controlled comparisons on the same 18 simulation tasks. With the policy pipeline held fixed, our scenes supported 14/18 completions versus 7/18 with SimFoundry scenes. With the scenes held fixed, our policy pipeline completed 14/18 versus 9/18 for CaP-X. These comparisons test the effects of scene construction and policy development separately.
3. Task difficulty and embodiment. Our evaluation covers 18 tasks across two robot platforms, Franka FR3 and Agilex Piper. The suite includes stacking and storage tasks comparable to those evaluated by CaP-X and SimFoundry, as well as articulated manipulation and multistep tasks. For example, placing a cup inside a cardboard box and then closing it (F3) requires the reconstructed scene to support both cup placement and lid motion, and the visual policy to coordinate those interactions in sequence. Bowl stacking and tissue extraction add distinct contact and handling demands. Drawer closing is one task in this broader suite.
4. Success rates and the suggested bug. Our protocol is zero-shot with respect to target-task demonstrations. It uses no teleoperated demonstrations, task-specific training datasets, or model fine-tuning. The agent develops each program through interaction with the reconstructed simulator. Under this protocol, the full method completed 14/18 tasks in the additional simulation evaluation and 8/18 in the original hardware evaluation. The follow-up identifies no failing component, reproducible error, or other evidence of a software defect. The suggestion that a bug explains the reported rates remains speculative.
Thank you for the review. We would like to clarify four factual points that materially affect the assessment of our method. Each clarification below refers to evidence already present in the original submission.
1. Real-robot execution uses fresh observations, rather than replaying simulated trajectories.
The review states that the sim-to-real step “simply replays the motion trajectory” in the real world. However, the original submission already describes repeated observations and actions within a single program execution in §3.2 (p. 5, lines 264–269). Section 3.3 further requires the agent to “re-localize objects and assess task state from current real observations rather than reuse simulated coordinates or success claims” (p. 6, lines 275–281), with observations before and after consequential actions (lines 282–289). The transferred policy therefore determines actions from current real-world observations. A locked policy version fixes the code, not its observation-dependent action coordinates. This directly addresses the concern that hardware execution is restricted to replaying a simulated trajectory.
2. Robot registration is automatic.
The review states that robot registration “still requires manual alignment.” However, the original submission already specifies automatic transform estimation in §3.1: multi-scale Sim(3) registration followed by pose and scale refinement (pp. 3–4, lines 155–182). Appendix B.1 specifies the scoring objective, and Table 6 specifies 128 rotation initializations, refinement of the top three candidates, and selection by the lowest final loss (p. 14). The registration transform is determined by this automatic procedure. The original method does not require an operator to manually choose the alignment.
3. Real-to-Sim reconstruction and randomized policy development are separate stages.
The review contrasts reconstructing the captured workspace with ACDC's use of varied scenes for robustness. However, the original submission already includes randomized policy development and explicitly separates these two stages. The Real-to-Sim stage (§3.1) reconstructs the captured workspace, preserving its metric scale, task-relevant geometry, appearance, and interaction structure (p. 3, lines 147–153). The subsequent policy-development stage (§3.2) introduces randomization: after privileged and visual development in the nominal scene, the third pass varies initial object positions, camera pose and calibration, and action execution (p. 5, lines 233–242; Fig. 4). Appendix D.2 states the same randomization settings (p. 23, lines 1222–1223), and §4.3 reports evaluation under randomized simulation conditions (p. 7, lines 356–357). Reconstruction fidelity is the objective of the first stage; adaptation to perturbations is addressed in the subsequent stage. Reconstructing the captured workspace does not mean that policy development uses only an unvarying scene.
4. The evaluated task set is not limited to pick-and-place.
The review describes the experiments as “only demonstrated on simple table-top pick-and-place tasks.” However, the original submission already lists drawer closing (F1), placing a cup and closing the box (F3), bowl stacking (F5), and tissue extraction (P9), alongside placement tasks in Appendix A, Table 5 (p. 14). The evaluation comprises 18 tasks across Franka FR3 and Agilex Piper, with nine tasks per platform. Describing this task catalog as exclusively pick-and-place omits the articulation, stacking, and sequential-action tasks already listed in the original submission.
We hope these references clarify the method and task setting. Could you please clarify which substantive concerns remain after these corrections?