- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents Real2Gym, an agentic Real2Sim2Real framework that converts human or robot demonstrations into interactive Blender/MuJoCo environments and then uses these environments for iterative robot skill acquisition. The pipeline has two major components: (1) a Real2Sim system that reconstructs task-relevant geometry, aligns viewpoints, refines scenes around interaction events, validates interactions using native physics, and generates task-conditioned variations; and (2) an agent that generates executable code at the manipulation-stage level, observes execution outcomes, and distills successful and failed attempts into an object-relative skill library. The underlying foundation model remains frozen; adaptation occurs through the accumulated skill representation.
The evaluation uses 12 DROID and 12 EgoDex scenes. The authors report substantial improvements over direct GPT-5.6 Sol and GPT-6 Astra reconstruction, particularly for viewpoint alignment and simulation success. For agent execution, the skill-conditioned version reaches 91.67%/83.33% success on DROID/EgoDex while substantially reducing model responses and token usage. The paper further demonstrates four tasks on a Franka robot and shows one failure-driven Real2Sim2Real example in which a previously unsuccessful plate-placement task becomes successful after simulation-based skill evolution.
Overall, I find the paper ambitious and the end-to-end system compelling. However, several central claims—particularly reconstruction quality, self-improvement, and efficiency—are currently supported by evaluations that are somewhat circular or confounded. I would like to see substantially stronger ablations and more objective evaluation before being confident in the conclusions.
Critical strengths
The paper tackles an important bottleneck in embodied learning. Instead of treating Real2Sim reconstruction as an endpoint, it explicitly connects reconstruction, simulator interaction, failure diagnosis, reusable experience, and physical deployment. This closed loop is a meaningful systems contribution and distinguishes the work from pipelines whose primary output is simply a digital twin.
The event-centered reconstruction and physics validation are particularly sensible. The method focuses refinement around contact/topology-changing events and checks camera alignment, relative object placement, penetration/contact, and appearance before executing the full trajectory under MuJoCo physics. The resulting native trajectory is then transferred back to Blender for visual comparison. This is much more convincing for manipulation than optimizing photometric similarity alone.
I also like the decision to represent acquired knowledge as object-relative motions, applicability conditions, effect checks, and recovery rules rather than simply storing complete action sequences. This potentially makes the memory more robust to changes in object configuration.
The experiments cover both robot demonstrations and human egocentric demonstrations, and the reconstruction gains are large. For example, EgoDex simulation-success score increases from 56.59 for GPT-6 Astra to 85.76, while viewpoint alignment rises from 38.67 to 73.75.
Finally, the physical experiments are valuable. The paper does not stop at simulation and demonstrates that simulation-acquired experience can improve a failed real-world task, which is precisely the behavior a Real2Sim2Real system should ultimately provide.
Critical weaknesses
The Real2Sim evaluation is too dependent on an LLM judge, and specifically on the same model family used by the proposed system. Reconstruction quality is scored by GPT-6 Astra with high reasoning effort, while scene construction, policy execution, and skill extraction all use GPT-6 Astra with medium reasoning effort. This introduces a potentially circular evaluation. More concerningly, the evaluation template explicitly contains a [method] field, so it is unclear whether the judge is blind to method identity.
The paper would be considerably stronger with human evaluation plus objective geometry/camera metrics: reprojection error, object-pose error, scale error, camera extrinsic error, contact-event consistency, penetration depth, or success under an independent physics-based task checker. At minimum, evaluation should use multiple independent judges and anonymized method outputs.
The most important Real2Gym components are not adequately ablated. The paper combines reconstruction, event-driven correction, native-physics verification, scene augmentation, stage-level code generation, SAM3/GraspNet tools, skill extraction, object-relative memory, and recovery rules. Yet the main ablation studies model choice/reasoning effort rather than these algorithmic components. The only clean memory comparison is “Ours” versus “Ours (w/ skills).” A convincing ablation should separately remove event-driven correction, physics validation, action-feasibility filtering, environment augmentation, object-relative skill representation, and failure-derived recovery rules.
The claimed benefit of “building gyms” is surprisingly under-evaluated. Task-conditioned augmentation is a central part of the method: geometry, pose, support height, materials, distractors, background, and lighting are varied, followed by physics validation. However, I could not find a quantitative experiment showing that these augmented environments improve robustness or real-world transfer compared with training/exploration only in the seed reconstruction. This seems essential to the title and central thesis.
The policy comparison is confounded by decision granularity. The paper itself explains that GPT-6 Astra Direct Mode makes individual-action decisions, whereas Real2Gym emits code for an entire operation stage. Therefore, a large reduction in responses and tokens is almost guaranteed by construction. A stronger baseline would use GPT-6 Astra with exactly the same stage-level programming interface but without the proposed memory/skill mechanism. Relatedly, the reported efficiency excludes both scene construction and post-episode skill extraction. This is reasonable for evaluating online execution cost, but the paper should avoid implying that it measures total Real2Gym system efficiency. Reconstruction/refinement/physics retries could be expensive.
Evidence for reusable skill learning and self-improvement is still limited. “Ours (w/ skills)” is evaluated on the second execution of the same task, using skills extracted from the first execution. This demonstrates within-task experience reuse, but it does not establish strong cross-task skill transfer or compositional reuse. In Table 2, skill memory improves the aggregate result primarily on DROID, while EgoDex success remains 83.33%.
I would like to see skills learned in task A improve unseen task B, or skills learned under one object arrangement transferred to multiple unseen object/layout changes.
The real-world evaluation is too small for the strength of the claims. There are only four physical tasks and three zero-shot trials per task, all on a Franka with a parallel-jaw gripper. The most compelling Real2Sim2Real self-evolution result is demonstrated on a single plate-placement task. This is an excellent case study, but not yet sufficient to establish robust autonomous self-improvement.
The “from videos” framing should be clarified. DROID provides synchronized robot trajectories in addition to videos, while EgoDex provides 3D hand/finger tracking. The reconstruction method also uses calibrated camera parameters and metric reference cues to constrain scale. It should be stated very clearly which annotations/calibrations are actually used for each experiment. If additional trajectory, tracking, or calibration data are necessary, the headline claim of converting a “single human video” into a gym is somewhat stronger than the demonstrated setup.
Questions for authors
Q1. Are the outputs provided to the GPT-6 Astra reconstruction evaluator anonymized? Does the judge know whether a reconstruction was produced by Real2Gym or a baseline? Q2. Can the authors report objective reconstruction errors—camera-pose error, object-pose/scale error, reprojection error, penetration/contact violations—in addition to LLM-based scores? Q3. What is the contribution of environment augmentation? What happens if the agent explores only in the single reconstructed seed environment? Q4. Could the authors compare against a GPT-6 Astra baseline using the exact same stage-level code-generation interface and perception tools as Real2Gym, but without skill extraction? This would separate interface benefits from memory benefits. Q5. How much computation, wall-clock time, and model-token usage are required to construct one gym, perform event-driven refinement, generate augmentations, and extract skills? The current efficiency results measure only policy execution. Q6. Can the learned skill library transfer across different tasks rather than merely to a second execution of the same task? Q7. For EgoDex, is the provided 3D hand/finger tracking used during reconstruction or retargeting? For DROID, how strongly does the method rely on the recorded robot trajectories?