- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Co2Skill, a physics-based controller that combines whole-body scene interaction with articulated-hand object manipulation. A transformer processes shared and task-specific observation tokens and predicts a latent residual for a frozen PULSE-style motion prior and decoder. Task- and phase-dependent masks select the scene, object, and grasp conditions relevant to the current objective. A goal-conditioned loco-manipulation curriculum combines reference-guided interaction learning with exploration from varied initial states, and removes full-body tracking during later manipulation phases. A cross-task curriculum jointly trains individual skills and selected manipulation-to-scene sequences, preserving physical state and grasp conditions when lifting transitions into a scene task.
The evaluation covers five individual scene-interaction tasks, synthetic-data augmentation for a separate reference-conditioned manipulation tracker, and goal-conditioned pick-move-place. The shared policy performs strongly on the reported scene tasks; complete pick-move-place success is 64.80% on the reported test split and 25.19% on unseen objects. Manipulation-to-scene transitions and simultaneous scene interaction with grasp retention are illustrated qualitatively. The main contribution is the integrated control formulation and training strategy for combining these capabilities without a reference motion spanning the full composed interaction.
Critical strengths
-
The integrated controller addresses a meaningful difficulty at the boundary between scene interaction and dexterous manipulation. The 153-dimensional joint controller combines articulated-hand interaction with locomotion and scene contact. Object-relative grasp targets, simultaneous scene/object conditioning, and state-preserving handover provide a concrete mechanism for attempting a scene task from the configuration produced by lifting an object. Appendix A.3 explicitly preserves body pose, velocities, and object state, and specifies a joint scene/carry objective and contact-based termination. This is a well-motivated integration: successful execution would allow an existing grasp to persist while the rest of the body performs a new interaction, without requiring a demonstration of the entire sequence.
-
The individual-task results provide credible evidence that a shared policy can support a useful range of behaviors. Table 1 reports 100% success on Climb-up, Stand-up, and Climb-down, 93.22% on Sit, and 69.23% on Stair. The reported Stair result improves on both the authors' specialist, at 54.41%, and the adapted MaskedMimic baseline, at 7.81%. Sharing has a task-dependent trade-off: Sit decreases from the specialist's 99.48% to 93.22%, which the paper acknowledges. Table 2(b) additionally demonstrates complete pick-move-place execution on 64.80% of test episodes, rather than only reference tracking. The reported 2.49 cm test release error is explicitly conditional on successful episodes. These results support the practical value of the integration while leaving its cross-task reliability to be established.
Critical weaknesses
-
The defining cross-task composition capability lacks quantitative validation. Section 4.3 proposes manipulation-to-scene transitions with persistent grasping, and Appendix A.3 devotes 35% of training environments to lifting followed by scene interaction. Nevertheless, Tables 1-2 evaluate standalone scene tasks, reference tracking, and pick-move-place. The specific capability of completing a subsequent scene task while retaining a grasp is supported by qualitative examples in Sec. 5.4 and Fig. 3, as the paper explicitly states. No joint sequence-success rates, object-retention rates, or reliability as a function of sequence length or duration are reported. Thus, the evidence shows that selected compositions can occur, but does not establish how reliably the method handles the state mismatch and simultaneous constraints that motivate its main contribution. Quantifying the already trained transitions, with both scene completion and object retention required, is necessary to assess the claimed advance. This concern does not imply that the demonstrated behaviors are invalid; it concerns their unmeasured reliability.
-
The experiments do not isolate the value of the proposed curricula relative to the inherited control architecture and other training changes. The paper explicitly builds on PULSE and TokenHSI-style task tokenization. Table 1's shared-versus-specialist comparison tests policy sharing on individual scene tasks, but does not isolate either the reference/exploration curriculum or cross-task training. Table 2(a) is a useful synthetic-data ablation for the separate MimicManipulator tracker; it does not test those curricula in Co2Skill. Moreover, there is no goal-conditioned manipulation comparator evaluated on the same pick-move-place protocol. The full MaskedManipulator policy (Tessler et al., 2025), which supports target-object and body-goal conditioning, is a relevant point of comparison distinct from its MimicManipulator tracking teacher. The submission correctly acknowledges that tracker completion and its physical pick metric are not directly comparable, so Table 2(a) does not resolve this gap. A useful integrated system can be a contribution, but the present evidence does not establish which new training decisions are necessary or what advantage they provide over appropriate alternatives.
Questions for authors
-
Quantitative validation of the trained compositions. For the manipulation-to-scene transitions sampled in Appendix A.3, what are the full-sequence success rates when both destination-task completion and continued grasp retention are required? Please report evaluation episode counts, sequence lengths/durations, and uncertainty across runs, and distinguish failed pickup, failed scene execution, and object loss. For any longer sequences illustrated in Fig. 3, which transition types and sequence lengths were included in training? Strong quantitative evidence on these existing compositions would materially improve my assessment.
-
Evidence for the curricula. With the same motion prior, task-policy architecture, training data, and a stated comparable interaction budget, how does performance change when cross-task sequence training is removed? How does this compare with switching separately trained specialists at the same physical handover states? A controlled reference-only or exploration-only variant would also help establish the contribution of the reference/exploration mixture in Sec. 4.2. Which proposed training mechanism produces a measurable improvement in complete task or sequence success?
-
Matched goal-conditioned manipulation comparison. Can the authors provide a comparison to a compatible goal-conditioned controller, such as an explicitly adapted MaskedManipulator, with the same objects, starting-state distribution, available goal/grasp information, and physical success criteria? Please state any adaptation required for placement/release and the training-data and interaction budgets. If a complete protocol match is infeasible, which common manipulation subtask can be compared meaningfully? The existing reference-tracker comparison uses a different control interface and success definition, so it cannot establish relative performance on this question.
-
Clarification of evaluation splits and test-time information. Appendix A.1 specifies six mocap objects, a subject split, and combined reference counts, but what identities and episode counts constitute the “Unseen” row in Table 2(b), and how are synthetic train/test references separated? Which training stages exclude those object identities? How are valid object-relative grasp targets obtained for those evaluation objects, and which phase decisions at evaluation depend on reference timestamps, simulator-state conditions, or externally supplied commands? Please also identify the evaluation denominators and whether both Table 2(a) variants use identical evaluation references. These details determine the interpretation of the 25.19% unseen-object completion rate and the extent of reference information required at deployment.
8 comments
[W2, Q2] Evidence for curricula
We evaluate cross-task training and reference/exploration training separately, reporting their effects on task and full-sequence success.
(a) Cross-task training. We compare the shared policy with and without lift-to-scene training environments, retaining the same frozen motion prior and task-policy architecture. Both settings are trained until each task's performance plateaus. Their environment allocations and cumulative interaction counts are:
| Training variant | Scene environments (%) | Manipulation environments (%) | Lift → scene environments (%) | Training transitions |
|---|---|---|---|---|
| With cross-task training (Ours) | 30 | 25 | 45 | 2.621B |
| No cross-task training | 50 | 50 | 0 | 2.823B |
At evaluation, the policy trained without cross-task training switches from lifting to the destination scene task through its task conditions.
| Variant | Lift → Sit (%) ↑ | Lift → Climb-up (%) ↑ | Lift → Reach (Free) (%) ↑ | Lift → Reach (Climb-down) (%) ↑ | Lift → Reach (Stair) (%) ↑ |
|---|---|---|---|---|---|
| With cross-task training (Ours) | 68.09 ± 2.54 | 34.00 ± 0.74 | 59.71 ± 1.10 | 33.21 ± 1.05 | 37.04 ± 1.19 |
| No cross-task training | 0.20 ± 0.10 | 0.09 ± 0.08 | 8.37 ± 0.84 | 0.36 ± 0.21 | 0.51 ± 0.07 |
Both policies are evaluated across five compositions with a maximum of 1,000 control steps per episode. Without cross-task training, full-sequence success remains at 0.09 to 8.37%, despite a slightly larger interaction budget (2.823B vs. 2.621B). The gap appears across all five compositions, including unobstructed reaching, indicating that the difficulty is not confined to terrain traversal. These results suggest that switching task conditions within a shared controller is insufficient for reliable grasp-preserving composition. Explicit cross-task training exposes the policy to the body, object, and grasp states produced by lifting and allows it to learn subsequent scene interaction under these simultaneous constraints.
Individual-task performance before switching. To assess whether the composition deficit reflects weak individual skills, we compare the policy without cross-task training against separately trained task specialists.
| Task / Metric | Specialist SR (%) ↑ | No cross-task SR (%) ↑ |
|---|---|---|
| Sit | 98.44 ± 0.68 | 99.64 ± 0.20 |
| Climb-up | 99.12 ± 0.20 | 99.12 ± 0.17 |
| Reach (overall) | 99.38 ± 0.06 | 96.37 ± 0.66 |
| Reach — Stand-up | 99.95 ± 0.08 | 99.90 ± 0.08 |
| Reach — Climb-down | 99.90 ± 0.08 | 97.22 ± 0.66 |
| Reach — Stair | 98.31 ± 0.20 | 92.02 ± 1.24 |
| Manipulation — Lift | 97.75 ± 2.51 | 98.12 ± 0.85 |
| Manipulation — Transport | 96.63 ± 2.50 | 96.52 ± 1.74 |
| Manipulation — Release | 61.83 ± 3.02 | 66.97 ± 2.84 |
Scene results use three seeds, with 3,072 episodes per model for each of Sit and Climb-up, and 6,144 for Reach. Manipulation uses far goals: the specialist is evaluated over 535 episodes across five seeds, and the policy without cross-task training over 321 episodes across three seeds. Results are mean ± sample standard deviation.
The policy without cross-task training achieves similar individual-task success to the specialists on Sit, Climb-up, and manipulation, although Reach performance is lower, particularly for Stair (92.02% vs. 98.31%). Its low composition success therefore coexists with strong individual skills and is not readily explained by a general failure to learn those skills. Existing scene specialists do not observe the carried object or grasp state, so directly switching to them would also change the observation interface. Our ablation retains that interface and switches task conditions within the shared policy. This provides complementary evidence for the role of composition training, but is not a direct evaluation of separately trained specialists at identical physical handover states. The cross-task ablation and the individual-task comparison are reported in Table 3(b) and Appendix B.1.
(b) Reference guidance and exploration.
Scene-interaction We compare exploration-only, mixed (60% reference environments), and reference-only (100% reference environments) training on Climb-up, which requires lifting the whole body onto the box. All variants use the same motion prior, task-policy architecture, and reference dataset. All models are trained until their task success rates plateau. Reference environments provide tracking rewards without supplying the reference trajectory as a policy observation. Exploration environments sample starting positions at different distances and directions around the box. We evaluate each variant on both reference (Ref.) and exploration (Explr.) initialization groups. Evaluation episodes are equally allocated to the two groups, yielding 6,144 episodes per group per model.
| Variant | Ref. SR (%) ↑ | Ref. error (cm) ↓ | Explr. SR (%) ↑ | Explr. error (cm) ↓ | Evaluated episodes |
|---|---|---|---|---|---|
| Exploration-only | 0.88 ± 0.18 | 11.67 ± 0.49 | 0.00 ± 0.00 | — | 12,288 |
| Ours | 99.66 ± 0.05 | 5.45 ± 0.01 | 97.75 ± 0.17 | 2.99 ± 0.02 | 12,288 |
| Reference-only | 99.71 ± 0.18 | 8.61 ± 0.02 | 75.33 ± 0.77 | 8.22 ± 0.04 | 12,288 |
Success requires remaining within 15 cm of the target for 60 consecutive control steps. Error is measured at the end of this holding period and averaged over successful episodes only; the dash denotes no successful episodes. Results are mean ± sample standard deviation across three seeds.
Mixed training retains high success on reference initializations (99.66% vs. 99.71% for reference-only training) while improving success on exploration initializations from 75.33% to 97.75%. Exploration-only training rarely succeeds in either group. These results support complementary roles for reference guidance and exploration: learning coordinated climbing behavior and extending it to varied starting states.
Manipulation. We use GRAB motions involving five objects: apple, camera, coffeemug, cup, and waterbottle. We remove pelvis/wrist trajectory guidance during approach and grasp formation and replace it with a distance-based reward for approaching the object from a standing pose, keeping the other model and training settings unchanged. We evaluate far-goal pick–move–place at a comparable interaction count and after continuing the same reference-free run until its evaluation release success rate plateaus.
| Variant | Training transitions (#) | Lift SR (%) ↑ | Transport SR (%) ↑ | Release SR (%) ↑ |
|---|---|---|---|---|
| With reference | 1.117B | 97.75 ± 2.51 | 96.63 ± 2.50 | 61.83 ± 3.02 |
| W/o reference | 1.049B | 8.97 ± 2.86 | 0.84 ± 0.89 | 0.32 ± 0.49 |
| W/o reference (trained to plateau) | 2.207B | 48.70 ± 0.56 | 47.69 ± 1.10 | 1.48 ± 1.05 |
Each variant is evaluated on 535 far-goal episodes across five seeds; results are mean ± sample standard deviation.
Replacing trajectory guidance with a distance-based approach reward, while keeping the other model and training settings unchanged, reduces lift success from 97.75% to 8.97% and full pick–move–place success from 61.83% to 0.32% at comparable interaction counts. Even after 2.207B transitions, the reference-free policy reaches only 48.70% lift, 47.69% transport, and 1.48% release success. This ablation identifies partial reference guidance as a contributor to both grasp acquisition and complete task success, beyond the shared architecture and additional training alone.
Persistent lift failures. We examine 20 motions for which the reference-free policy fails every lift attempt even after training to a plateau. Counts are successful lifts / matched evaluation attempts pooled over close and far goals. The subset comprises 170 trials.
| Variant | Successful lifts / attempts | Lift SR (%) ↑ |
|---|---|---|
| With reference | 167/170 | 98.24 |
| W/o reference | 0/170 | 0.00 |
| W/o reference (trained to plateau) | 0/170 | 0.00 |
Our policy succeeds in 167/170 attempts (98.24%) on this subset, while both reference-free checkpoints fail in all attempts. This indicates persistent failures for particular approach/grasp configurations despite the improvement in overall lift success with longer training. We hypothesize that trajectory guidance helps the policy reach configurations that are difficult to discover through exploration alone. The subset is selected retrospectively to characterize failures, not to estimate overall performance; these diagnostics concern grasp acquisition and do not by themselves explain the remaining release-success gap.
Together, the ablations support distinct roles for the curricula: cross-task training improves grasp-preserving composition, and reference guidance combined with exploration improves generalization in scene interaction and task completion in manipulation. The reference-guidance ablations are reported in Appendix B.2.
We thank the reviewer for recognizing the motivation of grasp-preserving scene interaction and the credibility of the individual-task results. We address composition reliability, curriculum contributions, the goal-conditioned manipulation comparison, and evaluation details below.
[W1, Q1] Reliability of manipulation-to-scene composition
All five evaluated lift-to-scene compositions are part of cross-task training; Lift→Sit is shown in Fig. 4 of the revised version. Each consists of a manipulation stage (grasp and lift) followed by one scene task, with the character and object states preserved at handover. Full-sequence success requires completing both the manipulation stage and the scene task while retaining the grasp. Episodes run for at most 1,000 control steps at 30 Hz (33.3 s), terminating earlier upon success or failure. We evaluate three seeds with 2,013 episodes per composition (2,010 for Reach (Stair) and Reach (Climb-down)), and report mean ± sample standard deviation across seeds.
Lift and full-sequence success rates use all episodes as the denominator. Grasp retention is conditional on entering the carry phase and measures the absence of contact-loss termination before the episode ends. Duration is averaged over successful full sequences only.
Failure breakdown.
The first three failure rates use all episodes and are mutually exclusive: pickup failure occurs before carry entry; object loss occurs after lifting when the contact-loss accumulation reaches 10 steps and takes precedence over a simultaneous fall; remaining failures are post-lift falls or timeouts without object loss. The final column reports the percentage of post-lift failed episodes whose failure begins within 1 s of handover (carry entry). Onset is the start of the contact-loss accumulation leading to termination, or the termination step for falls and timeouts.
The policy completes Lift→Sit and Lift→Reach (Free) with success rates of 68.09% and 59.71%, retaining the grasp in 84.58% and 76.17% of lifted episodes, respectively. Successful sequences finish in 8.45 to 15.77 s on average across tasks, well within the 33.3 s limit. These results demonstrate successful scene interaction from manipulation-generated states while maintaining an object grasp.
Success rates are lower for Climb-up and Reach (Stair), where larger weight shifts and whole-body adjustments may challenge grasp stability. Object loss is the largest failure category for Reach (Stair), whereas object loss and post-lift falls or timeouts occur at similar rates for Climb-up. Pickup failure is the most frequent failure category for Reach (Climb-down). Across all five compositions, most post-lift failures begin more than 1 s after handover. The remaining difficulty therefore extends to subsequent scene execution while carrying, rather than being confined to the immediate task transition.
These results are reported in Table 3(b) and Appendix B.1 of the revised manuscript.