acceptodds
Withdrawn from ICLR 2027

Real2Gym: Building Gyms from Videos, Bringing Skills to Robots

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Real-world videos provide rich demonstrations of manipulation, but turning them into reusable robot skills requires visually aligned environments, executable physical interactions, and mechanisms for learning from experience. We introduce Real2Gym, an agentic Real2Sim2Real framework that turns human and robot demonstrations into interactive simulation gyms and brings skills acquired in simulation to physical robots. The Real2Sim module reconstructs editable scenes, aligns objects and cameras with the input, validates demonstrated or retargeted actions through native physics execution, and generates task-conditioned variations with action-feasibility checks. Within these environments, the agent generates executable code for manipulation stages, observes their outcomes, and distills successful attempts and failures into reusable task procedures, object-relative motions, and recovery strategies. Through a shared perception-and-control interface, these skills guide subsequent execution in simulation and on real robots, with motions adapted to current observations and no updates to the underlying model weights. Extensive evaluations demonstrate that Real2Gym enables high-fidelity simulation environment reconstruction, outperforming GPT-6 Astra Direct Mode by 16.7% in success rate with approximately 74.9% fewer policy-execution tokens across these environments, while exceeding it by 33.3% in physical robot execution success rate across four tasks on a real Franka robot.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Real2Sim2Real, Robot Self-Improvement, Agentic Real2Sim, Agentic Robotics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SSM3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Real2Gym, an agentic Real2Sim2Real framework that converts human or robot demonstrations into interactive Blender/MuJoCo environments and then uses these environments for iterative robot skill acquisition. The pipeline has two major components: (1) a Real2Sim system that reconstructs task-relevant geometry, aligns viewpoints, refines scenes around interaction events, validates interactions using native physics, and generates task-conditioned variations; and (2) an agent that generates executable code at the manipulation-stage level, observes execution outcomes, and distills successful and failed attempts into an object-relative skill library. The underlying foundation model remains frozen; adaptation occurs through the accumulated skill representation.
The evaluation uses 12 DROID and 12 EgoDex scenes. The authors report substantial improvements over direct GPT-5.6 Sol and GPT-6 Astra reconstruction, particularly for viewpoint alignment and simulation success. For agent execution, the skill-conditioned version reaches 91.67%/83.33% success on DROID/EgoDex while substantially reducing model responses and token usage. The paper further demonstrates four tasks on a Franka robot and shows one failure-driven Real2Sim2Real example in which a previously unsuccessful plate-placement task becomes successful after simulation-based skill evolution.
Overall, I find the paper ambitious and the end-to-end system compelling. However, several central claims—particularly reconstruction quality, self-improvement, and efficiency—are currently supported by evaluations that are somewhat circular or confounded. I would like to see substantially stronger ablations and more objective evaluation before being confident in the conclusions.

Critical strengths

The paper tackles an important bottleneck in embodied learning. Instead of treating Real2Sim reconstruction as an endpoint, it explicitly connects reconstruction, simulator interaction, failure diagnosis, reusable experience, and physical deployment. This closed loop is a meaningful systems contribution and distinguishes the work from pipelines whose primary output is simply a digital twin. The event-centered reconstruction and physics validation are particularly sensible. The method focuses refinement around contact/topology-changing events and checks camera alignment, relative object placement, penetration/contact, and appearance before executing the full trajectory under MuJoCo physics. The resulting native trajectory is then transferred back to Blender for visual comparison. This is much more convincing for manipulation than optimizing photometric similarity alone. I also like the decision to represent acquired knowledge as object-relative motions, applicability conditions, effect checks, and recovery rules rather than simply storing complete action sequences. This potentially makes the memory more robust to changes in object configuration.
The experiments cover both robot demonstrations and human egocentric demonstrations, and the reconstruction gains are large. For example, EgoDex simulation-success score increases from 56.59 for GPT-6 Astra to 85.76, while viewpoint alignment rises from 38.67 to 73.75.
Finally, the physical experiments are valuable. The paper does not stop at simulation and demonstrates that simulation-acquired experience can improve a failed real-world task, which is precisely the behavior a Real2Sim2Real system should ultimately provide.

Critical weaknesses

The Real2Sim evaluation is too dependent on an LLM judge, and specifically on the same model family used by the proposed system. Reconstruction quality is scored by GPT-6 Astra with high reasoning effort, while scene construction, policy execution, and skill extraction all use GPT-6 Astra with medium reasoning effort. This introduces a potentially circular evaluation. More concerningly, the evaluation template explicitly contains a [method] field, so it is unclear whether the judge is blind to method identity.
The paper would be considerably stronger with human evaluation plus objective geometry/camera metrics: reprojection error, object-pose error, scale error, camera extrinsic error, contact-event consistency, penetration depth, or success under an independent physics-based task checker. At minimum, evaluation should use multiple independent judges and anonymized method outputs.

The most important Real2Gym components are not adequately ablated. The paper combines reconstruction, event-driven correction, native-physics verification, scene augmentation, stage-level code generation, SAM3/GraspNet tools, skill extraction, object-relative memory, and recovery rules. Yet the main ablation studies model choice/reasoning effort rather than these algorithmic components. The only clean memory comparison is “Ours” versus “Ours (w/ skills).” A convincing ablation should separately remove event-driven correction, physics validation, action-feasibility filtering, environment augmentation, object-relative skill representation, and failure-derived recovery rules.

The claimed benefit of “building gyms” is surprisingly under-evaluated. Task-conditioned augmentation is a central part of the method: geometry, pose, support height, materials, distractors, background, and lighting are varied, followed by physics validation. However, I could not find a quantitative experiment showing that these augmented environments improve robustness or real-world transfer compared with training/exploration only in the seed reconstruction. This seems essential to the title and central thesis.

The policy comparison is confounded by decision granularity. The paper itself explains that GPT-6 Astra Direct Mode makes individual-action decisions, whereas Real2Gym emits code for an entire operation stage. Therefore, a large reduction in responses and tokens is almost guaranteed by construction. A stronger baseline would use GPT-6 Astra with exactly the same stage-level programming interface but without the proposed memory/skill mechanism. Relatedly, the reported efficiency excludes both scene construction and post-episode skill extraction. This is reasonable for evaluating online execution cost, but the paper should avoid implying that it measures total Real2Gym system efficiency. Reconstruction/refinement/physics retries could be expensive.

Evidence for reusable skill learning and self-improvement is still limited. “Ours (w/ skills)” is evaluated on the second execution of the same task, using skills extracted from the first execution. This demonstrates within-task experience reuse, but it does not establish strong cross-task skill transfer or compositional reuse. In Table 2, skill memory improves the aggregate result primarily on DROID, while EgoDex success remains 83.33%.
I would like to see skills learned in task A improve unseen task B, or skills learned under one object arrangement transferred to multiple unseen object/layout changes.

The real-world evaluation is too small for the strength of the claims. There are only four physical tasks and three zero-shot trials per task, all on a Franka with a parallel-jaw gripper. The most compelling Real2Sim2Real self-evolution result is demonstrated on a single plate-placement task. This is an excellent case study, but not yet sufficient to establish robust autonomous self-improvement.

The “from videos” framing should be clarified. DROID provides synchronized robot trajectories in addition to videos, while EgoDex provides 3D hand/finger tracking. The reconstruction method also uses calibrated camera parameters and metric reference cues to constrain scale. It should be stated very clearly which annotations/calibrations are actually used for each experiment. If additional trajectory, tracking, or calibration data are necessary, the headline claim of converting a “single human video” into a gym is somewhat stronger than the demonstrated setup.

Questions for authors

Q1. Are the outputs provided to the GPT-6 Astra reconstruction evaluator anonymized? Does the judge know whether a reconstruction was produced by Real2Gym or a baseline? Q2. Can the authors report objective reconstruction errors—camera-pose error, object-pose/scale error, reprojection error, penetration/contact violations—in addition to LLM-based scores? Q3. What is the contribution of environment augmentation? What happens if the agent explores only in the single reconstructed seed environment? Q4. Could the authors compare against a GPT-6 Astra baseline using the exact same stage-level code-generation interface and perception tools as Real2Gym, but without skill extraction? This would separate interface benefits from memory benefits. Q5. How much computation, wall-clock time, and model-token usage are required to construct one gym, perform event-driven refinement, generate augmentations, and extract skills? The current efficiency results measure only policy execution. Q6. Can the learned skill library transfer across different tasks rather than merely to a second execution of the same task? Q7. For EgoDex, is the provided 3D hand/finger tracking used during reconstruction or retargeting? For DROID, how strongly does the method rely on the recorded robot trajectories?

Reviewer_CKVXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Real2Gym. It reconstructs the real-world scene from videos and aligns some physical properties such as view points and contact penetration. It then augments the scenes by varying object configurations and environmental conditions. Within these environments, an agent generates stage-level manipulation code and distills execution successes and failures into reusable, object-relative skills in the self-improvement manner, and outperforms many state-of-the-art agents.

Critical strengths
  1. The overall frameworks makes sense. With the development of generative models and agents, the reviewer belives such real2gym manner is a very promising approach for tackling real-world manipulation with robustness and generalization.

  2. The agentic approach for manipulation makes sense and is insteresting. The reviewer believes this is the right track to manipulating objects using agents.

Critical weaknesses
  1. The simulated object categories are limited. The simulated rigid and articulated objects are relatively simple (bowl, microwave, drawer ...). The reviewer is interested in whether (1) more complicated objects, such as a microwave with many handles and buttons, or very complicated rigid objects can be simulated; (2) deformable objects can be simulated; (3) tactile and force information can be introduced in the environment.

  2. The physics is limited. Some physical properties of the target objects, such as weight, friction coefficient and centroid are not included in the pipeline. Such parameters may lead to sim-real gap and influence manipulation success rates.

  3. Experiments seems insufficient. While the proposed framework outperforms GPT-6, agent driven methods are still token-consuming and time-consuming. Therefore, the comparisons with VLAs, wams or other pretrained policies may be important, in both the model success rates and time consumption metrics. Besides, the evaluation scenes and tasks are relatively simple.

Questions for authors

Following the weakness part, the questions are:

  1. Can diversity and complication of simulated objects be improved?
  2. What is the physical correspondence between real-world and simulated scenes?
  3. Can the authors provide more and thorough comparisons with other policies, and the evaluation in more complicated scenes and tasks.
Reviewer_VGU2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Real2Gym, a framework for Real2Sim2Real that utilizes recent LLMs (GPT-6 Astra). For Real2Sim, pretrained geometric reconstruction and segmentation models are used to create a 3D scene. Demonstrations are reconstructed using contact keyframes and physics validation. For the Sim2Real portion, an LLM agent interacts with the environments, generates programs, and extracts skills based on successes and failures. These skills can be transferred to real-world policies.

Critical strengths
  1. Real2Gym leverages existing tools well. This paper is effective at building on other work in 3D reconstruction and LLM development. The LLM pipeline works well to integrate these parts instead of building a method from scratch.
  2. The refinement procedure generates physically executable simulation trajectories. Given only a video as input, many of the physics parameters are underspecified. Iteratively refining the physical parameters until the demonstration succeeds in simulation is a good technique to achieve reasonable trajectories without requiring additional measurements.
Critical weaknesses
  1. The Real2Sim reconstruction pipeline is insufficiently specified. Several important components of the reconstruction pipeline are described based on what they do rather than how they do it. For mesh construction (lines 193-201), it is unclear how a mesh is created from the point cloud, and how the incomplete point cloud is filled in. For event-driven refinement, it is unclear how the keyframes are selected (line 204). Similarly for retargeting human hands (line 206).
  2. The role of the reconstructed demonstrations is unclear. Section 3.1 describes reconstructing demonstrated interactions in MuJoCo and augmenting them. Section 3.2 describes an agent for completing a task in some environment. As written, the agent seems to discover these skills instead of learning them from the demonstrations. If this is the case, it is unclear what is gained by reconstructing the demonstrated actions.
  3. The Real2Sim reconstruction pipeline is not evaluated thoroughly. Table 1 compares against GPT-6 Astra and GPT-5.6 Sol, but not against other reconstruction methods such as Agentic Real2Sim [1]. Evaluation is limited to only 24 scenes. Additionally, metrics use only VLM judgement instead of objective measures of 3D reconstruction quality or image similarity. This may introduce bias and miss sources of error. The cost of the Real2Sim pipeline is also not discussed.
  4. Real world experiments are limited, and the complete Real2Sim2Real pipeline is insufficiently evaluated. Real-world comparisons are made using only 3 trials per task, and the complete pipeline seems to be only evaluated on a single scene. The paper claims 100% success rate without stating the number of trials (line 452-453).
Questions for authors
  1. Could the authors please clarify various components of the reconstruction pipeline? (See weakness 1.)
  2. How are the reconstructed demonstrations used for the Sim2Real portion of the pipeline? What is the purpose of the augmentation pipeline?
  3. How does Real2Gym compare to Agentic Real2Sim [1]? Would a direct comparison between these methods be feasible?
  4. What is the cost of the Real2Sim reconstruction? How many tokens does each method use?

[1] Chen, Guanxiong, et al. "Agentic real2sim: Physics-based world modeling with vision-language agents." arXiv preprint arXiv:2607.19190 (2026).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.