- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Skill2Real, an agentic framework for zero-shot sim-to-real robot manipulation. Instead of updating neural network parameters, Skill2Real acquires reusable executable knowledge through repeated simulation rollouts and persistent skill-memory updates. Its Proposer–Verifier–Governor (PVG) framework uses a VLM to generate and execute robot programs, a Verifier to diagnose outcomes using privileged simulation evidence, and a Governor to decide whether candidate updates should be admitted based on subsequent validation rollouts. The framework learns a two-level hierarchy: the Cerebellum stores local manipulation skills, while the Brain composes these skills into long-horizon task programs. The authors report improvements on LIBERO-Pro Long, seven Robosuite tasks, and four real-world tasks on a UR5e robot, with the full hierarchy achieving 78.75% average task completion compared with 27.50% for the baseline without learned skills and 56.25% for Cerebellum-only. The results suggest that simulation experience can be accumulated as reusable procedural memory without fine-tuning the underlying VLM.
Critical strengths
-
A promising direction for agentic robot learning. Given the emerging capabilities of foundation VLMs in embodied tasks, Skill2Real explores a promising and potentially scalable direction: leveraging automated interaction and privileged ground-truth feedback in simulation to acquire reusable procedural knowledge, rather than updating policy parameters. The separation between local skills and task-level composition is intuitive and provides a useful framework for studying how simulation experience can be transferred to real-world manipulation.
-
Encouraging empirical results. The method achieves substantial gains on unseen long-horizon simulation tasks and real-world manipulation, with average real-world task completion increasing from 27.50% without learned skills to 78.75% with the full hierarchy. These results provide initial evidence that simulation-learned skill memory can improve a frozen VLM-based robot system without real-world task-policy fine-tuning.
Critical weaknesses
- Limited evidence for skill transfer and generalization. Although Skill2Real shows promising results on unseen simulation tasks and four real-world tasks, it remains unclear how broadly the learned skills can be reused across tasks, objects, and scenes, or how well the approach extends to more complex physical interactions. Some memory entries appear task-specific, and the real-world evaluation does not establish generalization to challenging contact-rich or deformable-object manipulation. Moreover, the paper lacks quantitative failure-mode analysis to explain whether the observed gains arise from more reliable local manipulation, better task-level planning, or improved recovery. These limitations make it difficult to assess the generality of the learned knowledge and the extent to which the proposed hierarchy explains the performance improvements.
- Unclear efficiency and scalability advantages. The paper motivates Skill2Real as an alternative to learning task-agnostic low-level representations, but does not establish its efficiency relative to simulation-data-driven policy learning. The growing skill memory may increase prompt length and VLM inference latency, potentially limiting real-world deployment, particularly as task coverage expands. The paper does not report memory growth, VLM call counts, end-to-end execution latency, or total learning cost across training stages, nor does it compare against a parameterized policy trained on simulation rollouts under matched budgets. Consequently, it remains unclear whether the proposed approach offers practical efficiency or scalability advantages beyond avoiding gradient-based policy updates.
Questions for authors
-
Inference efficiency. The project demo accelerates robot execution by tens of times, raising concerns about deployment latency. How much of the latency is attributable to the growing skill memory? Could the authors report memory size, VLM call counts, and end-to-end execution time across training stages?
-
Generalization to complex manipulation. Can Skill2Real generalize to more challenging real-world tasks involving sustained contact and precise physical interactions, such as bottle-cap twisting, or deformable-object manipulation, such as cloth folding? These tasks require more than task sequencing and pick-and-place skills, and may be difficult to reproduce faithfully in simulation. The current real-world evaluation covers only four relatively simple tasks, leaving it unclear how well the approach handles more complex manipulation and what limitations arise from the simulation environment and the underlying robot API.
-
Attribution of performance gains. What accounts for the improvement in real-world task completion from 27.50% to 78.75%? Could the authors provide stage-wise success rates or failure-mode analyses to distinguish gains in grasping, placement, task planning, and recovery?
-
Comparison with policy learning. Although Skill2Real avoids gradient-based policy training, it still incurs costs from VLM calls, simulation rollouts, and candidate validation. Could the authors compare it with a VLA policy trained on successful simulation rollouts under matched simulation interaction budgets, reporting task success, total VLM inference cost, GPU-hours, and the cost required to reach a given success rate?
-
Skill reuse and memory scalability. To what extent can skills learned from one task be reused across new tasks, objects, and scenes? Some memory entries appear task-specific (e.g., “Bind a black bowl and its plate”), raising concerns about skill-library growth, context length, and inference latency as task coverage expands, particularly in more complex real-world scenarios. Could the authors quantify cross-task skill reuse and compare full-memory prompting with retrieving only task-relevant skills in terms of task success, prompt-token count, and end-to-end inference latency?
Thank you for the detailed review. We added inference measurements and report the simulation interaction budget alongside the existing ablations.
Q1. Inference efficiency
The complete seven-iteration LIBERO-90 training run took 12.7 hours on average. The corresponding median model generation latency was about 15 seconds on GPT 5.6-sol.
The project video is sped up for viewing. On the real robot, the backend checks inverse kinematics, trajectory feasibility, and safety constraints before executing a motion.
Q2. Generalization to complex manipulation
Our existing packing case illustrates a more demanding Cerebellum capability: precise placement within a confined receptacle. Success requires identifying the usable opening, selecting an interior placement target that accounts for object geometry and available clearance, and executing release and retreat without disrupting the intended placement. The learned Cerebellum procedures guide these local geometric decisions through the shared API and use fresh observations to verify the resulting containment. This case shows that the learned knowledge extends beyond task sequencing to geometry-sensitive local manipulation. It provides evidence for constrained placement. The simulator and robot API define the interactions and control primitives available to learning; improving contact fidelity and adding force-aware or compliant-control APIs provide a direct path to extending our skill-learning framework to more demanding manipulation.
Q3. Attribution of the real-world gains
Without the Cerebellum, the agent may struggle to identify the right object or choose where to grasp it. An ambiguous SAM3 prompt, for example, could produce a mask on the wrong object or the wrong part of the intended object, directing the grasp to the wrong location. In a separate screening batch, some runs stopped before the robot moved: the target remained ambiguous, or the system could not determine where the container opening, support surface, or placement region was.
In one run without Brain memory, the robot grasped the object successfully but could not complete the transfer. The gripper blocked the camera’s view of the held object, and wrist depth returned no valid points. The system could no longer estimate the object’s geometry and did not recover enough information to continue with transport and release. The failure occurred after grasping, when the system needed to handle a missing observation and decide how to proceed.
Q4. Comparison with policy learning under matched budgets
Skill2Real incurs learning costs from model calls, simulation execution, and candidate validation. With GPT-5.6 Sol, the complete seven-iteration LIBERO-90 run uses 3,150 training rollouts and 15,750 validation rollouts, totaling 18,900 simulation rollouts, excluding evaluation, and takes 12.7 hours on average. We have not conducted the requested comparison with a VLA trained on successful simulation rollouts under a matched interaction budget.
Q5. Skill reuse and memory scalability
The Cerebellum is trained on LIBERO-90 and organized by manipulation operation, including inspection, grasping, and transport. The same local procedure can be reused across tasks that require that operation. The Brain is learned task by task. It records how to bind objects, order operations, and check progress for a particular task. At execution time, the Proposer receives one Brain document for the active task. Adding documents for other tasks grows the catalog without directly increasing the Brain context for that task.
Cross-task transfer. Figure 4(a) evaluates frozen LIBERO-90 checkpoints on unseen LIBERO-Pro Long tasks with GPT-6 Astra. Overall success rises from 2.0% at C0 without learned skills to 35.25% at C3 with the learned Cerebellum alone. It reaches 56.25% with the full hierarchy, C3+B4.
Inference profiling. In a separate LIBERO-Pro Long profiling run, median total token usage was approximately 16.50k at C0, 19.28k at C3, and 21.13k at C3+B4. The corresponding median model generation latencies were 14 s, 14 s, and 15 s. From C0 to the full hierarchy, median token usage increased by about 28%, while median generation latency increased by 1 s.