acceptodds
Under review as a conference paper at ICLR 2027

Latent Skill Acquisition and Composition for Contact-Rich Manipulation

Abstract

Human-level generalization in manipulation requires composing known behaviors to reach novel goals in scenarios never seen in training. The challenge sharpens in contact-rich settings, where behaviors must be recomposed at the granularity of contact rather than of a semantic subtask. Without relying on per-task demonstrations, a promising approach is to leverage test-time compute to perform physical reasoning that searches for novel solutions and improves them incrementally during deployment. To effectively perform search in the large space of motions, we propose SkillReasoner, which learns temporal abstractions of actions (i.e., skills) from unstructured offline interactions, and then reasons over and recomposes them at test time. Crucial to the formulation is a latent skill space learned with varying temporal horizons, so that composition can happen at varying granularity, as required by different stages of the task, while minimizing the number of decisions. To realize a novel goal at test time, the robot searches over latent skill sequences in its imagination, decodes a chosen skill into low-level actions, and replans in closed loop, up to 30 Hz. We demonstrate on a series of contact-rich real-world and simulated tasks that the robot chains behaviors to achieve novel goals that require far longer behaviors than those seen in training, and adapts zero-shot to unseen perturbations, changing dynamics, and constraints, all trained purely from unstructured play data without task-specific demonstrations.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Manipulation, Robot Learning, Manipulation Planning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eUGEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SkillReasoner learns goal-conditioned contact-rich manipulation from unstructured play without task-specific demonstrations. It combines discrete varying-horizon skills spanning 1 to 20 steps with state encoding, keypoint grounding, duration prediction, skill proposals, latent dynamics, and goal-conditioned value learning. Beam search scores imagined sequences using keypoint-based goal costs, variance-based risk, and cost-to-go, then executes one skill open-loop before replanning. Three Push-T variants and four robot tasks using two hours of play demonstrate gains over shared-data baselines and test horizons, search budgets, adaptation, constraints, self-improvement, and crossing-data removal.

Critical strengths
  1. Varying horizons balance coordinated motion reuse and fine adjustments. At equal search depth, uniform sampling over 1 to 20 steps outperforms fixed horizons of 1, 4, and 16 steps and longer ranges in score and execution steps.

  2. Skill tokenization, duration prediction, grounding, dynamics, value, and beam search use shared play data without task labels, allowing failed or mixed-quality interactions to contribute.

  3. Scores exceed shared-play and expert-demonstration baselines across six simulation settings and π0.5, DP, and ACT across four robot tasks trained on two hours of shared play. The planner recovers from perturbations, adapts to friction changes by fine-tuning only dynamics, and handles new keep-out constraints by rejecting plans.

  4. Training episodes have a median of approximately 100 steps and skills span at most 20, whereas test executions require hundreds. Removing all crossing-spanning windows preserves a 0.915 score, supporting composition beyond demonstrated routes.

Critical weaknesses
  1. Table 1 omits prior latent skill planners such as SkiMo and model predictive methods, despite acknowledged precedents. Fixed-horizon variants are compared only against one another on narrow-passage. Gains remain difficult to attribute specifically to varying horizons rather than keypoint costs, learned value, or search.

  2. In Table 1, all play-trained baselines score below 15, which the authors attribute to short-episode hindsight relabeling rather than insufficient tuning. Hyperparameter ranges, training budgets, goal-pose conditioning for π0.5, ACT, and DP, and simulation dataset size are unspecified. Comparable keypoint supervision for baselines is unclear. Physical evaluation has only ten trials per task and no numerical table.

  3. Table 1 averages two runs without intervals or standard deviations. SkillReasoner shares a frozen skill model and varies only value heads. Half the score rewards route progress, while full-task success is 53.0% for multi-room and 34.0% for winding-maze. Medium exceeds High success on narrow-passage (73.0 versus 66.5) and multi-room (63.0 versus 53.0), contradicting the claimed monotonic improvement with more compute.

  4. Claimed benefits of variance prediction, random versus event-based windows, and frozen versus joint training lack supporting plots or tables. Timing disables early stopping and adds duration cost, unlike Table 1. Single-object Push-T tasks use replayed expert actions, leaving robustness to truly unstructured data untested.

Questions for authors
  1. Why do scores and success diverge? On multi-room (full), High scores higher but succeeds less often and takes more steps. Does deeper search sacrifice alignment or increase 720-step timeouts? Please report main-text success, error-bar definitions, and more than two complete training seeds.

  2. How are goal poses supplied to π0.5, ACT, and DP? Do they receive comparable keypoint supervision? How large is the play dataset relative to 50 expert demonstrations? Would longer hindsight horizons or trajectory stitching help? A fixed-horizon planner with identical keypoint costs and value would be the most informative comparison.

  3. Does uniform sampling over 1 to 20 steps remain advantageous on multi-room and winding-maze? Please ablate variance prediction, random windows versus event boundaries, and frozen versus joint training. How is duration d modeled, and how is plan-length bias avoided across depths?

  4. What is latency under the exact Table 1 settings with early stopping enabled and no duration cost? What replanning rate is used on the real robot, and how much data are needed for friction adaptation? Since the best scores require depth 4 at approximately 1 Hz, should the abstract qualify "up to 30 Hz"?

Reviewer_VYXMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work learns varying-horizon latent skills from offline play through observation-conditioned action reconstruction, then trains a skill proposal, endpoint dynamics, and a goal-conditioned value function on frozen representations. At deployment, beam search combines these components with keypoint-based goal evaluation and a variance-based risk proxy to select skills. Evaluation includes three simulated Push-T layouts, four real-robot tasks, and studies of skill horizons, search guidance, and adaptation.

Critical strengths
  1. This work separates local behavior acquisition from goal-directed skill composition, allowing offline play to support tasks without complete task-specific demonstrations. The varying-horizon skill representation provides a coherent way to expose both extended motions and short adjustments to the same planner, addressing a relevant challenge in contact-rich manipulation.
  2. The evaluation combines longer-horizon goal reaching in three simulated layouts with real-robot alignment, transport, rearrangement, and packing. The crossing-removal experiment further examines whether retained local interactions can support traversal without an intact crossing example, providing a concrete test of the intended composition setting.
  3. The supplementary material specifies the proposed system’s module interfaces, training objectives, planning pseudocode, and timing setup. Together with the selected-duration distributions and execution timelines, these details make the learned skill representation and deployment procedure easier to inspect.
Critical weaknesses
  1. The main comparisons are against model-free offline RL and behavior cloning, while this work combines learned dynamics, geometric goal guidance, and explicit search. Without a comparable skill-based model-based planner under similar supervision and planning budgets, these comparisons do not establish the contribution of the proposed skill design beyond existing model-based skill composition.
  2. The behavior cloning baselines use achieved future poses as hindsight goals, but the goal representation, conditioning interface, and corresponding training modifications are not specified for ACT, Diffusion Policy, or π0.5. Without these details or validation on short-horizon goals, their near-zero scores do not establish that the failure comes from the limitations of hindsight-conditioned behavior cloning.
  3. This work attributes its gains to selecting skill durations appropriate to different task stages. However, changing duration ranges at a fixed skill-search depth also changes physical lookahead and open-loop execution length. The reported comparisons therefore do not isolate adaptive duration selection from these effects.
  4. This work uses a keep-out region to demonstrate test-time constraint handling, but candidate feasibility is checked at predicted skill endpoints. A skill can enter and leave the forbidden region between those endpoints. Without a trajectory-level violation analysis, endpoint validity does not establish that the executed behavior respects the constraint.
  5. The search risk proxy comes from keypoint-prediction variance, whereas planning depends on recursively predicted skill transitions. Accurate or confident keypoint decoding does not imply an accurate transition prediction. Given the reported exploitation of model errors, including imagined wall penetration, the lack of evidence that this proxy detects transition errors weakens the reliability of risk-aware search.
  6. The real-robot evaluation uses only ten reset-goal pairs per task and lacks controls that isolate the contribution of varying-horizon skills. These results demonstrate performance on selected task instances, but provide limited evidence that the central design remains effective across a broader range of physical interaction conditions.
Questions for authors
  1. Can the authors compare against a relevant skill-based model-based planner under comparable geometric supervision and planning budgets, to separate the contribution of the proposed skill design from the benefits of model-based search?
  2. Can the authors specify how hindsight goal poses are represented and incorporated into ACT, Diffusion Policy, and π0.5, including the corresponding training modifications? Can these goal-conditioned baselines also be validated on held-out short-horizon goals drawn from the play-data distribution?
  3. Can adaptive and restricted duration selection be compared using the same learned skill representation, while accounting for physical lookahead, open-loop execution length, and planning compute?
  4. How is constraint compliance measured? Can violations be evaluated along the full executed trajectories, including motion within each skill, rather than only through predicted skill endpoints?
  5. Can the authors measure whether the risk proxy identifies transitions with large actual prediction errors or constraint violations, and compare planning with and without this proxy under matched search budgets?
  6. Can the authors include fixed-horizon controls and expand the real-robot evaluation to more reset-goal pairs and a broader range of task-relevant interaction conditions, to assess whether the benefits of varying-horizon skills persist?
Reviewer_1XJBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

As an emergency reviewer, my assessment is mainly based on main paper. I did not read through the supplement material and other literature in fine details. It is overall a good paper with solid methodology, but I would not recommend accepting the paper unless major concerns are resolved.

SkillReasoner provides a solution to contact-rich problem such as pushing T and pushing toy bricks. It learns dynamics though unlabeled offline plays, and encodes states as well as skills of different horizon. In the planning stage, the learned proposal samples skills of different horizons and plan accordingly.

I appreciate their test-time planning&optimization pipeline. However, the tasks are too easy to support their claimed contribution and the their experiments is insufficient. They claimed to have proposed a formulation "latent skill composition" for contact-rich manipulation, but their task choice is too limited to support it. There are abundant contact rich manipulation tasks. Two simple tasks with similar settings are not enough to show it is a general solution.

Critical strengths

Strengths:

  1. The paper is well-structured and the authors provide a clean interactive demonstration on their website.

  2. SkillReasoner learned from offline interactions solely. It does not rely on complete successful training data sequences. Yet it has shown a remarkable performance on push-T task. It plans efficiently (30HZ on a consumer GPU) and wisely(varying horizon) and generalize to other variants smoothly.

Critical weaknesses

Weakness:

  1. SkillReasoner only conducts experiments on relatively simple tasks. That makes SkillReasoner way less impactful. Neither pushing T in a complex maze nor pushing toy bricks are hard. Both action and movements are in low dimensions. There are harder contact-rich manipulation tasks, like dex hand manipulation and tool-use tasks.

  2. The formulation of SkillReasoner resembles Model Predictive Control(MPC). Yet SkillReasoner has not been compared to MPC. I believe there exist simpler ways to solve these tasks. For sanity check, I built a MPC solver to pushT for all 3 mazes. Both Model Predictive Path Integral(MPPI) and Monte Carlo Tree Search(MCTS) have provided efficient and correct solutions. Plus, it is not hard to train a predictor. I think the author should provides comparison between random action proposal/heuristic-based action proposal, against their learned proposal and varying horizon skills. Without that ablation, the core part of the contribution would become questionable.

Questions for authors

Qestions:

On the same task setting, how is SkillReasoner compared to MPC-based algorithms?

Do you think the current tasks are enough to support your claim? If not, could you add limited experiments to showcase it is a formulation of a variety of contact-rich manipulation tasks, or simply narrow your claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.