acceptodds
Under review as a conference paper at ICLR 2027

Skill2Real: Agentic Skill Learning for Zero-Shot Sim-to-Real Robot Manipulation

17 views

Abstract

Sim-to-real learning commonly adapts policies or representations to domain variation, often directing the learning budget toward task-agnostic low-level representations at the expense of task-level knowledge. Skill2Real builds on cross-domain grounding from foundation vision-language models (VLMs) to acquire validated, reusable skills in simulation through a shared code-based policy interface. In its asymmetric Proposer–Verifier–Governor (PVG) loop, the Proposer acts from public observations and API returns, the Verifier diagnoses outcomes with privileged simulation evidence, and the Governor admits updates supported by validation rollouts. A Cerebellum learns reusable local manipulation skills; a Brain then composes the frozen library into long-horizon programs with planning and recovery. At deployment, this hierarchy is grounded in real observations through the same interface. On unseen LIBERO-Pro Long, frozen LIBERO-90 skills raise success from 2.0% to 56.3% with Astra and from 0.5% to 49.0% with Opus 5. Robosuite success reaches 85.1–89.4% across seven tasks. On a real UR5e, the same agent's mean completion rises from 27.50% to 78.75% across four tasks with Skill2Real, a controlled gain that, with cross-model improvements, shows Skill2Real adds a capability orthogonal to the foundation model by learning transferable task knowledge in simulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Manipulation, Zero-Shot Sim-to-Real Transfer, Agentic Learning, Hierarchical Skill Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SKiQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Skill2Real, an agentic framework for zero-shot sim-to-real robot manipulation. Instead of updating neural network parameters, Skill2Real acquires reusable executable knowledge through repeated simulation rollouts and persistent skill-memory updates. Its Proposer–Verifier–Governor (PVG) framework uses a VLM to generate and execute robot programs, a Verifier to diagnose outcomes using privileged simulation evidence, and a Governor to decide whether candidate updates should be admitted based on subsequent validation rollouts. The framework learns a two-level hierarchy: the Cerebellum stores local manipulation skills, while the Brain composes these skills into long-horizon task programs. The authors report improvements on LIBERO-Pro Long, seven Robosuite tasks, and four real-world tasks on a UR5e robot, with the full hierarchy achieving 78.75% average task completion compared with 27.50% for the baseline without learned skills and 56.25% for Cerebellum-only. The results suggest that simulation experience can be accumulated as reusable procedural memory without fine-tuning the underlying VLM.

Critical strengths
  1. A promising direction for agentic robot learning. Given the emerging capabilities of foundation VLMs in embodied tasks, Skill2Real explores a promising and potentially scalable direction: leveraging automated interaction and privileged ground-truth feedback in simulation to acquire reusable procedural knowledge, rather than updating policy parameters. The separation between local skills and task-level composition is intuitive and provides a useful framework for studying how simulation experience can be transferred to real-world manipulation.

  2. Encouraging empirical results. The method achieves substantial gains on unseen long-horizon simulation tasks and real-world manipulation, with average real-world task completion increasing from 27.50% without learned skills to 78.75% with the full hierarchy. These results provide initial evidence that simulation-learned skill memory can improve a frozen VLM-based robot system without real-world task-policy fine-tuning.

Critical weaknesses
  1. Limited evidence for skill transfer and generalization. Although Skill2Real shows promising results on unseen simulation tasks and four real-world tasks, it remains unclear how broadly the learned skills can be reused across tasks, objects, and scenes, or how well the approach extends to more complex physical interactions. Some memory entries appear task-specific, and the real-world evaluation does not establish generalization to challenging contact-rich or deformable-object manipulation. Moreover, the paper lacks quantitative failure-mode analysis to explain whether the observed gains arise from more reliable local manipulation, better task-level planning, or improved recovery. These limitations make it difficult to assess the generality of the learned knowledge and the extent to which the proposed hierarchy explains the performance improvements.
  2. Unclear efficiency and scalability advantages. The paper motivates Skill2Real as an alternative to learning task-agnostic low-level representations, but does not establish its efficiency relative to simulation-data-driven policy learning. The growing skill memory may increase prompt length and VLM inference latency, potentially limiting real-world deployment, particularly as task coverage expands. The paper does not report memory growth, VLM call counts, end-to-end execution latency, or total learning cost across training stages, nor does it compare against a parameterized policy trained on simulation rollouts under matched budgets. Consequently, it remains unclear whether the proposed approach offers practical efficiency or scalability advantages beyond avoiding gradient-based policy updates.
Questions for authors
  1. Inference efficiency. The project demo accelerates robot execution by tens of times, raising concerns about deployment latency. How much of the latency is attributable to the growing skill memory? Could the authors report memory size, VLM call counts, and end-to-end execution time across training stages?

  2. Generalization to complex manipulation. Can Skill2Real generalize to more challenging real-world tasks involving sustained contact and precise physical interactions, such as bottle-cap twisting, or deformable-object manipulation, such as cloth folding? These tasks require more than task sequencing and pick-and-place skills, and may be difficult to reproduce faithfully in simulation. The current real-world evaluation covers only four relatively simple tasks, leaving it unclear how well the approach handles more complex manipulation and what limitations arise from the simulation environment and the underlying robot API.

  3. Attribution of performance gains. What accounts for the improvement in real-world task completion from 27.50% to 78.75%? Could the authors provide stage-wise success rates or failure-mode analyses to distinguish gains in grasping, placement, task planning, and recovery?

  4. Comparison with policy learning. Although Skill2Real avoids gradient-based policy training, it still incurs costs from VLM calls, simulation rollouts, and candidate validation. Could the authors compare it with a VLA policy trained on successful simulation rollouts under matched simulation interaction budgets, reporting task success, total VLM inference cost, GPU-hours, and the cost required to reach a given success rate?

  5. Skill reuse and memory scalability. To what extent can skills learned from one task be reused across new tasks, objects, and scenes? Some memory entries appear task-specific (e.g., “Bind a black bowl and its plate”), raising concerns about skill-library growth, context length, and inference latency as task coverage expands, particularly in more complex real-world scenarios. Could the authors quantify cross-task skill reuse and compare full-memory prompting with retrieving only task-relevant skills in terms of task success, prompt-token count, and end-to-end inference latency?

1 comment

Thank you for the detailed review. We added inference measurements and report the simulation interaction budget alongside the existing ablations.

Q1. Inference efficiency

The complete seven-iteration LIBERO-90 training run took 12.7 hours on average. The corresponding median model generation latency was about 15 seconds on GPT 5.6-sol.

The project video is sped up for viewing. On the real robot, the backend checks inverse kinematics, trajectory feasibility, and safety constraints before executing a motion.

Q2. Generalization to complex manipulation

Our existing packing case illustrates a more demanding Cerebellum capability: precise placement within a confined receptacle. Success requires identifying the usable opening, selecting an interior placement target that accounts for object geometry and available clearance, and executing release and retreat without disrupting the intended placement. The learned Cerebellum procedures guide these local geometric decisions through the shared API and use fresh observations to verify the resulting containment. This case shows that the learned knowledge extends beyond task sequencing to geometry-sensitive local manipulation. It provides evidence for constrained placement. The simulator and robot API define the interactions and control primitives available to learning; improving contact fidelity and adding force-aware or compliant-control APIs provide a direct path to extending our skill-learning framework to more demanding manipulation.

Q3. Attribution of the real-world gains

Without the Cerebellum, the agent may struggle to identify the right object or choose where to grasp it. An ambiguous SAM3 prompt, for example, could produce a mask on the wrong object or the wrong part of the intended object, directing the grasp to the wrong location. In a separate screening batch, some runs stopped before the robot moved: the target remained ambiguous, or the system could not determine where the container opening, support surface, or placement region was.

In one run without Brain memory, the robot grasped the object successfully but could not complete the transfer. The gripper blocked the camera’s view of the held object, and wrist depth returned no valid points. The system could no longer estimate the object’s geometry and did not recover enough information to continue with transport and release. The failure occurred after grasping, when the system needed to handle a missing observation and decide how to proceed.

Q4. Comparison with policy learning under matched budgets

Skill2Real incurs learning costs from model calls, simulation execution, and candidate validation. With GPT-5.6 Sol, the complete seven-iteration LIBERO-90 run uses 3,150 training rollouts and 15,750 validation rollouts, totaling 18,900 simulation rollouts, excluding evaluation, and takes 12.7 hours on average. We have not conducted the requested comparison with a VLA trained on successful simulation rollouts under a matched interaction budget.

Q5. Skill reuse and memory scalability

The Cerebellum is trained on LIBERO-90 and organized by manipulation operation, including inspection, grasping, and transport. The same local procedure can be reused across tasks that require that operation. The Brain is learned task by task. It records how to bind objects, order operations, and check progress for a particular task. At execution time, the Proposer receives one Brain document for the active task. Adding documents for other tasks grows the catalog without directly increasing the Brain context for that task.

Cross-task transfer. Figure 4(a) evaluates frozen LIBERO-90 checkpoints on unseen LIBERO-Pro Long tasks with GPT-6 Astra. Overall success rises from 2.0% at C0 without learned skills to 35.25% at C3 with the learned Cerebellum alone. It reaches 56.25% with the full hierarchy, C3+B4.

Inference profiling. In a separate LIBERO-Pro Long profiling run, median total token usage was approximately 16.50k at C0, 19.28k at C3, and 21.13k at C3+B4. The corresponding median model generation latencies were 14 s, 14 s, and 15 s. From C0 to the full hierarchy, median token usage increased by about 28%, while median generation latency increased by 1 s.

Reviewer_BBGZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a zero-shot sim-to-real framework that transfers simulation-learned behavior through code-based robot skills.

The method learns a two-level skill hierarchy: (i) Cerebellum, which learns reusable local manipulation skills (e.g., grasping and placing), and (ii) Brain, which learns to compose these skills into long-horizon tasks with planning and recovery.

Rather than updating model weights, both levels are learned by updating skill memory through a Proposer–Verifier–Governor loop: (i) the Proposer generates and executes robot code and proposes skill updates; (ii) the Verifier diagnoses execution outcomes using privileged simulation information; and (iii) the Governor validates proposed updates through additional rollouts and decides whether to incorporate them into memory. The resulting skill hierarchy is frozen and transferred to the real robot without additional training.

Critical strengths

Effective agentic skill learning: The proposed approach enables API-based policies to improve through experience without updating model weights, making it particularly useful when the underlying model parameters are inaccessible. The detailed component ablations provide evidence for the effectiveness of the proposed learning framework. However, the motivation and evaluation regarding sim-to-real transfer remain less convincing, as discussed in the weaknesses.

Critical weaknesses

Unclear sim-to-real transfer robustness: Although Table 1 demonstrates real-world performance, it does not quantify the sim-to-real performance gap. Reporting simulation results on the corresponding real-world tasks would help assess how well the learned skills transfer across domains. Furthermore, comparisons with existing sim-to-real methods under matched conditions would clarify the proposed method's relative robustness to domain shifts. Authors could tell the method is actually efficient than methods learning low-level representation with this.

Questions for authors
  • How the method handle domain gap? The learning doesn't necessarily resolve the domain gap between simulation and real-world. How learning more robustly on simulation can lead performance boost on real-world?
  • What are the examples of priviledged information that verifier uses? Having some example along with resulting feedback message can make readers better understand.

Suggestions on organization

  • The LIBERO and Robosuite results demonstrate improvements within simulation but do not establish sim-to-real transfer robustness. So I would not highlight in the abstraction, which can make readers confusing.
  • L62: It would be good to elaborate what low-level representation with examples.
  • L79: Define what is public observation. It was hard to grasp what it means for me, but might due to my lack of knowledge. Ignore if authors think it is general term.
  • L199: The superscripts are explained in random place.
1 comment

Thank you for the constructive feedback. We expanded the explanation of what transfers between domains, how the Verifier uses simulation evidence, and how the notation describes the shared interface.

Q1. How does Skill2Real address the sim-to-real domain gap?

Skill2Real assumes a calibrated real-robot backend that implements the same documented API semantics as simulation, without requiring explicitly matched dynamics or scene configurations across domains. The transferred knowledge resides at a higher level of abstraction than the low-level API: it specifies how to organize operations into manipulation procedures, inspect their effects, and recover or proceed toward the task goal. These procedures are grounded in current real observations, while the backend handles perception, coordinate calibration, and motion execution. This separation supports transferring procedural knowledge without replaying simulator-specific trajectories or coordinates. However, it does not eliminate all deployment discrepancies: errors in API-provided estimates or action execution, as well as imperfect hardware calibration, can still degrade task performance.

Q2. What privileged information does the Verifier use?

The Verifier uses evaluation evidence derived from simulator-ground-truth object and gripper states, contact records, and task outcomes, aligned with the executed API calls. In the bowl-transfer example, the evidence shows finger contact and lateral bowl motion during gripper closure, followed by the gripper rising approximately 17.6 cm while the bowl remains on the table. This localizes the failed transition to grasp retention. The Verifier translates that diagnosis into feedback executable through public observations: after a short lift, request a fresh camera view and check whether the bowl has moved with the gripper before continuing transport. The feedback conveys the failed effect and an observable diagnostic check, while excluding privileged coordinates and hidden simulator predicates.

Q3. Simulation results in the abstract

We moved the emphasis in the abstract to the real-robot comparison from Table 1. The numerical LIBERO and Robosuite results remain in Results. LIBERO-90 and Robosuite assess skill acquisition within simulation, and LIBERO-Pro Long assesses generalization to unseen simulation tasks.

Additional evaluation of the domain gap

Evaluating the same tasks in simulation and on the real robot would let us measure the transfer gap directly. Comparing existing sim-to-real methods under the same task and deployment conditions would help characterize how each method handles domain shifts. We leave these comparisons to future work.

L62. Low-level representations

By low-level representations, we mean the numerical features and action encodings used by a parameterized control policy to turn observations into robot commands. On the observation side, a visual encoder can map a camera image to a latent feature vector. On the action side, a policy can encode a command as a vector containing a relative end-effector translation, a rotation, and a gripper command. We added these examples to Related Work.

L79. Public observations

We define public observations by what the Proposer can receive through the robot API during real-robot execution. They include camera images and the robot’s measured end-effector pose. The API leaves the position of the manipulated object for the Proposer to estimate from observations. The simulator’s ground-truth object position is privileged state available for training-time diagnosis. The simulation backend exposes the same types of public observations as the real backend. This definition now appears at first use in the Introduction.

L199. Superscript notation

In Section 3.1, we define the two symbols at first use. Esim\mathcal{E}^{\mathrm{sim}} denotes the simulated robot environment, and Ereal\mathcal{E}^{\mathrm{real}} denotes the physical robot environment. Both expose the same robot API A\mathcal{A}, so a skill program can call the same documented operations in either environment. Each backend implements those operations for its own sensors and robot. For example, the same camera observation call returns a rendered image in simulation and a live image on the real robot.

Reviewer_1P4HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

They introduce Skill2Real, an agentic policy framework that learns executable robot skills through a shared code-based policy interface,with an asymmetric Proposer–Verifier–Governor (PVG) loop that learns a two-level skill hierarchy. The Proposer executes programs from public observations and proposes reusable updates; the Verifier turns privileged simulation outcomes into publicly grounded diagnoses; and the Governor aggregates evidence across validation rollouts to admit supported updates into memory. Their experiments show better performance on unseen LIBERO-Pro Long and real UR5.

Critical strengths
  1. System & Framework: This paper introduce a new system with Proposer–Verifier–Governor scheme, and a hierarchical policy learning strategy in which the Cerebellum learns reusable local manipulation skills and the Brain learns long-horizon compositions conditioned on the frozen Cerebellum library, enabling task-level planning and recovery. Which seems reasonable and interesting.
  2. Better Performance: Success on unseen LIBERO-Pro Long tasks rise from no memory through the Cerebellum to the full hierarchy on frozen LIBERO-90 libraries and the full Brain–Cerebellum hierarchy reaches 78.75% mean task completion versus 56.25% with local skills on the real UR5e.
Critical weaknesses
  1. Limited novelty compare with the existing skill learning methods. For example, ASPIRE[1] already introduces an agentic framework that learns reusable robot skills through execution, failure diagnosis, program refinement, and validation, with demonstrated generalization to unseen long-horizon tasks and sim-to-real transfer. And RoboOS[2] already introduce 1) the Embodied Brain Model (RoboBrain), a multimodal large language model (MLLM) designed for global perception and high-level decision-making; (2) the Cerebellum Skill Library, a modular, plug-and-play toolkit that facilitates seamless execution of multiple skills; The reported improvementsreflect the generalization capabilities of existing foundation models, combined with prompt engineering and agentic workflow design, rather than new sim-to-real learning methods.
  2. Unclear ablation of the hierarchical design. The Brain stage involves additional training iterations after Cerebellum training. It is unclear whether the performance gains come from Brain–Cerebellum hierarchy or simply from additional training.
  3. Limited evaluation of real-world robustness. Appendix A.1 provides the useful real-world examples under different tabletop appearances and lighting conditions. However, this results are qualitative, without the corresponding success rate;

[1]ASPIRE: Agentic /Skills Discovery for Robotics [2]RoboOS: A Hierarchical Embodied Framework for Cross-Embodiment and Multi-Agent Collaboration

Questions for authors
  1. Novelty: Could the authors clarify the key methodological differences between Skill2Real and existing approaches such as ASPIRE [1] and RoboOS [2]? (most important, agent work is always similar for me~)
  2. More ablations of the hierarchical design.: Could the authors provide a comparison with the skill memory trained for the same number of iterations?
  3. Real-world robustness: Could the authors provide quantitative success rates under the different tabletop appearance and lighting condition.

[1]ASPIRE: Agentic /Skills Discovery for Robotics [2]RoboOS: A Hierarchical Embodied Framework for Cross-Embodiment and Multi-Agent Collaboration

1 comment

Thank you for the thoughtful review and for highlighting the hierarchical learning design. Your questions helped us clarify the contributions and add a direct comparison under matched training iterations.

Q1. Relationship to ASPIRE and RoboOS

ASPIRE. ASPIRE already learns reusable skills through execution, diagnosis, repair, and independent validation, and transfers accumulated knowledge to long-horizon tasks. Skill2Real explicitly separates this knowledge into two persistent skills. Stage I learns local manipulation skills; Stage II holds that memory fixed and learns a separate Brain memory for ordering, object binding, progress tracking, and recovery. Thus, task composition is itself learned and retained. Both stages use the same asymmetric learning procedure: the Verifier diagnoses outcomes using privileged simulation evidence, translates the diagnosis into feedback grounded in public observations and API semantics, and the Governor evaluates candidate updates through subsequent validation rollouts. The distinction comes from the organization and acquisition of skill.

RoboOS. RoboOS provides a precedent for the Brain-Cerebellum architecture, combining RoboBrain, modular execution tools, and shared memory for multi-robot coordination. Skill2Real focuses on acquiring explicit procedural skills at both levels through simulation experience. Its Brain and Cerebellum are memories conditioning the Proposer’s program generation, and foundation-model weights remain fixed throughout skill learning. The methodological contribution therefore lies in their staged acquisition, validation, and zero-shot deployment.

Q2. Hierarchy ablation under matched training iterations

We added a comparison on seven Robosuite tasks with the same total of seven training iterations. The full hierarchy uses three Cerebellum iterations followed by four Brain iterations with the Cerebellum frozen; the single-level variants train their respective memory for all seven iterations. At the final iteration, the full hierarchy, Cerebellum-only, and Brain-only schedules achieve 85.1%, 49.4%, and 25.4% with GPT-5.6 Sol and 89.4%, 53.1%, and 36.6% with Opus 5, respectively. Section 5.4 and Figure 8 describe the schedules and learning curves.

The smaller gains from Brain-only training are consistent with a dependency between task composition and local execution. Brain learning improves task ordering, object binding, progress tracking, and recovery, but these decisions rely on successful physical operations. Without a learned library of reliable grasping and placement procedures, local execution failures can limit the benefit of better task-level decisions. With the Cerebellum learned first, Brain training can build on those local procedures to complete longer sequences and preserve progress during recovery.

Q3. Tabletop appearance and lighting

Table A.2 now reports per-task completion on plain and patterned tablecloths. Across the four real-world tasks, the full hierarchy achieves 85.0% mean completion on the plain tablecloth and 72.5% on the patterned tablecloth.

Appendix A.1 also shows a qualitative demonstration of answer-cube pick-and-place under dim side lighting. Repeated trials under controlled illumination would allow us to estimate full-task success rates across lighting conditions. We leave that evaluation to future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.