acceptodds
Under review as a conference paper at ICLR 2027

CoSkill: Whole-Body Control via Skill Composition for Long-Horizon Human-Environment Interaction

Abstract

Achieving human-level dexterity in complex, unstructured environments requires the seamless integration of whole-body scene interaction and dexterous object manipulation skills. While existing physics-based controllers generate physically plausible behaviors in each domain, they largely address these two capabilities independently. In this paper, we present CoSkill that integrates scene interaction and dexterous manipulation through a unified policy formulation. Built on a pretrained motion prior, the policy uses task and phase dependent observation masks to select information relevant to the current interaction goals. % while training on individual skills and selected transitions exposes the policy to states encountered across task boundaries. We introduce a goal-conditioned loco-manipulation curriculum that combines partial reference guidance for precision with exploration from varied initial states while allowing goal-directed execution beyond the demonstrated trajectories. We further introduce a cross-task curriculum that jointly trains individual skills and selected task sequences, preserving physical states across task boundaries and maintaining grasps during subsequent scene interactions. Together, these support sequential task execution and simultaneous scene interaction with object manipulation. We evaluate sitting, standing, climbing, stair traversal, and goal-directed manipulation, together with sequential execution and with random different conditions. Additionally, we demonstrate skill compositions in indoor environments, illustrating their integration within the same control formulation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Character Animation, Human-Object Interaction, Human-Scene Interaction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bPdFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Co2Skill, a physics-based controller that combines whole-body scene interaction with articulated-hand object manipulation. A transformer processes shared and task-specific observation tokens and predicts a latent residual for a frozen PULSE-style motion prior and decoder. Task- and phase-dependent masks select the scene, object, and grasp conditions relevant to the current objective. A goal-conditioned loco-manipulation curriculum combines reference-guided interaction learning with exploration from varied initial states, and removes full-body tracking during later manipulation phases. A cross-task curriculum jointly trains individual skills and selected manipulation-to-scene sequences, preserving physical state and grasp conditions when lifting transitions into a scene task.

The evaluation covers five individual scene-interaction tasks, synthetic-data augmentation for a separate reference-conditioned manipulation tracker, and goal-conditioned pick-move-place. The shared policy performs strongly on the reported scene tasks; complete pick-move-place success is 64.80% on the reported test split and 25.19% on unseen objects. Manipulation-to-scene transitions and simultaneous scene interaction with grasp retention are illustrated qualitatively. The main contribution is the integrated control formulation and training strategy for combining these capabilities without a reference motion spanning the full composed interaction.

Critical strengths
  1. The integrated controller addresses a meaningful difficulty at the boundary between scene interaction and dexterous manipulation. The 153-dimensional joint controller combines articulated-hand interaction with locomotion and scene contact. Object-relative grasp targets, simultaneous scene/object conditioning, and state-preserving handover provide a concrete mechanism for attempting a scene task from the configuration produced by lifting an object. Appendix A.3 explicitly preserves body pose, velocities, and object state, and specifies a joint scene/carry objective and contact-based termination. This is a well-motivated integration: successful execution would allow an existing grasp to persist while the rest of the body performs a new interaction, without requiring a demonstration of the entire sequence.

  2. The individual-task results provide credible evidence that a shared policy can support a useful range of behaviors. Table 1 reports 100% success on Climb-up, Stand-up, and Climb-down, 93.22% on Sit, and 69.23% on Stair. The reported Stair result improves on both the authors' specialist, at 54.41%, and the adapted MaskedMimic baseline, at 7.81%. Sharing has a task-dependent trade-off: Sit decreases from the specialist's 99.48% to 93.22%, which the paper acknowledges. Table 2(b) additionally demonstrates complete pick-move-place execution on 64.80% of test episodes, rather than only reference tracking. The reported 2.49 cm test release error is explicitly conditional on successful episodes. These results support the practical value of the integration while leaving its cross-task reliability to be established.

Critical weaknesses
  1. The defining cross-task composition capability lacks quantitative validation. Section 4.3 proposes manipulation-to-scene transitions with persistent grasping, and Appendix A.3 devotes 35% of training environments to lifting followed by scene interaction. Nevertheless, Tables 1-2 evaluate standalone scene tasks, reference tracking, and pick-move-place. The specific capability of completing a subsequent scene task while retaining a grasp is supported by qualitative examples in Sec. 5.4 and Fig. 3, as the paper explicitly states. No joint sequence-success rates, object-retention rates, or reliability as a function of sequence length or duration are reported. Thus, the evidence shows that selected compositions can occur, but does not establish how reliably the method handles the state mismatch and simultaneous constraints that motivate its main contribution. Quantifying the already trained transitions, with both scene completion and object retention required, is necessary to assess the claimed advance. This concern does not imply that the demonstrated behaviors are invalid; it concerns their unmeasured reliability.

  2. The experiments do not isolate the value of the proposed curricula relative to the inherited control architecture and other training changes. The paper explicitly builds on PULSE and TokenHSI-style task tokenization. Table 1's shared-versus-specialist comparison tests policy sharing on individual scene tasks, but does not isolate either the reference/exploration curriculum or cross-task training. Table 2(a) is a useful synthetic-data ablation for the separate MimicManipulator tracker; it does not test those curricula in Co2Skill. Moreover, there is no goal-conditioned manipulation comparator evaluated on the same pick-move-place protocol. The full MaskedManipulator policy (Tessler et al., 2025), which supports target-object and body-goal conditioning, is a relevant point of comparison distinct from its MimicManipulator tracking teacher. The submission correctly acknowledges that tracker completion and its physical pick metric are not directly comparable, so Table 2(a) does not resolve this gap. A useful integrated system can be a contribution, but the present evidence does not establish which new training decisions are necessary or what advantage they provide over appropriate alternatives.

Questions for authors
  1. Quantitative validation of the trained compositions. For the manipulation-to-scene transitions sampled in Appendix A.3, what are the full-sequence success rates when both destination-task completion and continued grasp retention are required? Please report evaluation episode counts, sequence lengths/durations, and uncertainty across runs, and distinguish failed pickup, failed scene execution, and object loss. For any longer sequences illustrated in Fig. 3, which transition types and sequence lengths were included in training? Strong quantitative evidence on these existing compositions would materially improve my assessment.

  2. Evidence for the curricula. With the same motion prior, task-policy architecture, training data, and a stated comparable interaction budget, how does performance change when cross-task sequence training is removed? How does this compare with switching separately trained specialists at the same physical handover states? A controlled reference-only or exploration-only variant would also help establish the contribution of the reference/exploration mixture in Sec. 4.2. Which proposed training mechanism produces a measurable improvement in complete task or sequence success?

  3. Matched goal-conditioned manipulation comparison. Can the authors provide a comparison to a compatible goal-conditioned controller, such as an explicitly adapted MaskedManipulator, with the same objects, starting-state distribution, available goal/grasp information, and physical success criteria? Please state any adaptation required for placement/release and the training-data and interaction budgets. If a complete protocol match is infeasible, which common manipulation subtask can be compared meaningfully? The existing reference-tracker comparison uses a different control interface and success definition, so it cannot establish relative performance on this question.

  4. Clarification of evaluation splits and test-time information. Appendix A.1 specifies six mocap objects, a subject split, and combined reference counts, but what identities and episode counts constitute the “Unseen” row in Table 2(b), and how are synthetic train/test references separated? Which training stages exclude those object identities? How are valid object-relative grasp targets obtained for those evaluation objects, and which phase decisions at evaluation depend on reference timestamps, simulator-state conditions, or externally supplied commands? Please also identify the evaluation denominators and whether both Table 2(a) variants use identical evaluation references. These details determine the interpretation of the 25.19% unseen-object completion rate and the extent of reference information required at deployment.

8 comments

We thank the reviewer for recognizing the motivation of grasp-preserving scene interaction and the credibility of the individual-task results. We address composition reliability, curriculum contributions, the goal-conditioned manipulation comparison, and evaluation details below.

[W1, Q1] Reliability of manipulation-to-scene composition

All five evaluated lift-to-scene compositions are part of cross-task training; Lift→Sit is shown in Fig. 4 of the revised version. Each consists of a manipulation stage (grasp and lift) followed by one scene task, with the character and object states preserved at handover. Full-sequence success requires completing both the manipulation stage and the scene task while retaining the grasp. Episodes run for at most 1,000 control steps at 30 Hz (33.3 s), terminating earlier upon success or failure. We evaluate three seeds with 2,013 episodes per composition (2,010 for Reach (Stair) and Reach (Climb-down)), and report mean ± sample standard deviation across seeds.

CompositionTotal episodesLift SR (%) ↑Full-sequence SR (%) ↑Grasp retention (%) ↑Successful sequence duration (s)
Lift → Sit2,01397.54 ± 0.8468.09 ± 2.5484.58 ± 1.288.50 ± 0.06
Lift → Climb-up2,01398.12 ± 0.3134.00 ± 0.7467.56 ± 2.3213.50 ± 0.14
Lift → Reach (Free)2,01397.61 ± 1.0859.71 ± 1.1076.17 ± 1.4810.24 ± 0.11
Lift → Reach (Stair)2,01095.87 ± 0.4837.04 ± 1.1951.57 ± 0.9415.77 ± 0.54
Lift → Reach (Climb-down)2,01071.51 ± 1.4433.21 ± 1.0562.77 ± 3.068.45 ± 0.39

Lift and full-sequence success rates use all episodes as the denominator. Grasp retention is conditional on entering the carry phase and measures the absence of contact-loss termination before the episode ends. Duration is averaged over successful full sequences only.

Failure breakdown.

CompositionPickup failure (%) ↓Post-lift fall / timeout (%) ↓Object loss (%) ↓Failures within 1 s of handover (%)
Lift → Sit2.46 ± 0.8414.41 ± 1.1915.04 ± 1.233.47 ± 0.89
Lift → Climb-up1.88 ± 0.3132.29 ± 1.6931.83 ± 2.351.98 ± 0.44
Lift → Reach (Free)2.39 ± 1.0814.62 ± 1.1623.27 ± 1.673.10 ± 1.47
Lift → Reach (Stair)4.13 ± 0.4812.40 ± 1.4046.44 ± 0.991.31 ± 0.62
Lift → Reach (Climb-down)28.49 ± 1.4411.67 ± 1.9226.64 ± 2.5216.90 ± 2.50

The first three failure rates use all episodes and are mutually exclusive: pickup failure occurs before carry entry; object loss occurs after lifting when the contact-loss accumulation reaches 10 steps and takes precedence over a simultaneous fall; remaining failures are post-lift falls or timeouts without object loss. The final column reports the percentage of post-lift failed episodes whose failure begins within 1 s of handover (carry entry). Onset is the start of the contact-loss accumulation leading to termination, or the termination step for falls and timeouts.

The policy completes Lift→Sit and Lift→Reach (Free) with success rates of 68.09% and 59.71%, retaining the grasp in 84.58% and 76.17% of lifted episodes, respectively. Successful sequences finish in 8.45 to 15.77 s on average across tasks, well within the 33.3 s limit. These results demonstrate successful scene interaction from manipulation-generated states while maintaining an object grasp.

Success rates are lower for Climb-up and Reach (Stair), where larger weight shifts and whole-body adjustments may challenge grasp stability. Object loss is the largest failure category for Reach (Stair), whereas object loss and post-lift falls or timeouts occur at similar rates for Climb-up. Pickup failure is the most frequent failure category for Reach (Climb-down). Across all five compositions, most post-lift failures begin more than 1 s after handover. The remaining difficulty therefore extends to subsequent scene execution while carrying, rather than being confined to the immediate task transition.

These results are reported in Table 3(b) and Appendix B.1 of the revised manuscript.

[W2, Q2] Evidence for curricula

We evaluate cross-task training and reference/exploration training separately, reporting their effects on task and full-sequence success.

(a) Cross-task training. We compare the shared policy with and without lift-to-scene training environments, retaining the same frozen motion prior and task-policy architecture. Both settings are trained until each task's performance plateaus. Their environment allocations and cumulative interaction counts are:

Training variantScene environments (%)Manipulation environments (%)Lift → scene environments (%)Training transitions
With cross-task training (Ours)3025452.621B
No cross-task training505002.823B

At evaluation, the policy trained without cross-task training switches from lifting to the destination scene task through its task conditions.

VariantLift → Sit (%) ↑Lift → Climb-up (%) ↑Lift → Reach (Free) (%) ↑Lift → Reach (Climb-down) (%) ↑Lift → Reach (Stair) (%) ↑
With cross-task training (Ours)68.09 ± 2.5434.00 ± 0.7459.71 ± 1.1033.21 ± 1.0537.04 ± 1.19
No cross-task training0.20 ± 0.100.09 ± 0.088.37 ± 0.840.36 ± 0.210.51 ± 0.07

Both policies are evaluated across five compositions with a maximum of 1,000 control steps per episode. Without cross-task training, full-sequence success remains at 0.09 to 8.37%, despite a slightly larger interaction budget (2.823B vs. 2.621B). The gap appears across all five compositions, including unobstructed reaching, indicating that the difficulty is not confined to terrain traversal. These results suggest that switching task conditions within a shared controller is insufficient for reliable grasp-preserving composition. Explicit cross-task training exposes the policy to the body, object, and grasp states produced by lifting and allows it to learn subsequent scene interaction under these simultaneous constraints.

Individual-task performance before switching. To assess whether the composition deficit reflects weak individual skills, we compare the policy without cross-task training against separately trained task specialists.

Task / MetricSpecialist SR (%) ↑No cross-task SR (%) ↑
Sit98.44 ± 0.6899.64 ± 0.20
Climb-up99.12 ± 0.2099.12 ± 0.17
Reach (overall)99.38 ± 0.0696.37 ± 0.66
Reach — Stand-up99.95 ± 0.0899.90 ± 0.08
Reach — Climb-down99.90 ± 0.0897.22 ± 0.66
Reach — Stair98.31 ± 0.2092.02 ± 1.24
Manipulation — Lift97.75 ± 2.5198.12 ± 0.85
Manipulation — Transport96.63 ± 2.5096.52 ± 1.74
Manipulation — Release61.83 ± 3.0266.97 ± 2.84

Scene results use three seeds, with 3,072 episodes per model for each of Sit and Climb-up, and 6,144 for Reach. Manipulation uses far goals: the specialist is evaluated over 535 episodes across five seeds, and the policy without cross-task training over 321 episodes across three seeds. Results are mean ± sample standard deviation.

The policy without cross-task training achieves similar individual-task success to the specialists on Sit, Climb-up, and manipulation, although Reach performance is lower, particularly for Stair (92.02% vs. 98.31%). Its low composition success therefore coexists with strong individual skills and is not readily explained by a general failure to learn those skills. Existing scene specialists do not observe the carried object or grasp state, so directly switching to them would also change the observation interface. Our ablation retains that interface and switches task conditions within the shared policy. This provides complementary evidence for the role of composition training, but is not a direct evaluation of separately trained specialists at identical physical handover states. The cross-task ablation and the individual-task comparison are reported in Table 3(b) and Appendix B.1.

(b) Reference guidance and exploration.

Scene-interaction We compare exploration-only, mixed (60% reference environments), and reference-only (100% reference environments) training on Climb-up, which requires lifting the whole body onto the box. All variants use the same motion prior, task-policy architecture, and reference dataset. All models are trained until their task success rates plateau. Reference environments provide tracking rewards without supplying the reference trajectory as a policy observation. Exploration environments sample starting positions at different distances and directions around the box. We evaluate each variant on both reference (Ref.) and exploration (Explr.) initialization groups. Evaluation episodes are equally allocated to the two groups, yielding 6,144 episodes per group per model.

VariantRef. SR (%) ↑Ref. error (cm) ↓Explr. SR (%) ↑Explr. error (cm) ↓Evaluated episodes
Exploration-only0.88 ± 0.1811.67 ± 0.490.00 ± 0.00—12,288
Ours99.66 ± 0.055.45 ± 0.0197.75 ± 0.172.99 ± 0.0212,288
Reference-only99.71 ± 0.188.61 ± 0.0275.33 ± 0.778.22 ± 0.0412,288

Success requires remaining within 15 cm of the target for 60 consecutive control steps. Error is measured at the end of this holding period and averaged over successful episodes only; the dash denotes no successful episodes. Results are mean ± sample standard deviation across three seeds.

Mixed training retains high success on reference initializations (99.66% vs. 99.71% for reference-only training) while improving success on exploration initializations from 75.33% to 97.75%. Exploration-only training rarely succeeds in either group. These results support complementary roles for reference guidance and exploration: learning coordinated climbing behavior and extending it to varied starting states.

Manipulation. We use GRAB motions involving five objects: apple, camera, coffeemug, cup, and waterbottle. We remove pelvis/wrist trajectory guidance during approach and grasp formation and replace it with a distance-based reward for approaching the object from a standing pose, keeping the other model and training settings unchanged. We evaluate far-goal pick–move–place at a comparable interaction count and after continuing the same reference-free run until its evaluation release success rate plateaus.

VariantTraining transitions (#)Lift SR (%) ↑Transport SR (%) ↑Release SR (%) ↑
With reference1.117B97.75 ± 2.5196.63 ± 2.5061.83 ± 3.02
W/o reference1.049B8.97 ± 2.860.84 ± 0.890.32 ± 0.49
W/o reference (trained to plateau)2.207B48.70 ± 0.5647.69 ± 1.101.48 ± 1.05

Each variant is evaluated on 535 far-goal episodes across five seeds; results are mean ± sample standard deviation.

Replacing trajectory guidance with a distance-based approach reward, while keeping the other model and training settings unchanged, reduces lift success from 97.75% to 8.97% and full pick–move–place success from 61.83% to 0.32% at comparable interaction counts. Even after 2.207B transitions, the reference-free policy reaches only 48.70% lift, 47.69% transport, and 1.48% release success. This ablation identifies partial reference guidance as a contributor to both grasp acquisition and complete task success, beyond the shared architecture and additional training alone.

Persistent lift failures. We examine 20 motions for which the reference-free policy fails every lift attempt even after training to a plateau. Counts are successful lifts / matched evaluation attempts pooled over close and far goals. The subset comprises 170 trials.

VariantSuccessful lifts / attemptsLift SR (%) ↑
With reference167/17098.24
W/o reference0/1700.00
W/o reference (trained to plateau)0/1700.00

Our policy succeeds in 167/170 attempts (98.24%) on this subset, while both reference-free checkpoints fail in all attempts. This indicates persistent failures for particular approach/grasp configurations despite the improvement in overall lift success with longer training. We hypothesize that trajectory guidance helps the policy reach configurations that are difficult to discover through exploration alone. The subset is selected retrospectively to characterize failures, not to estimate overall performance; these diagnostics concern grasp acquisition and do not by themselves explain the remaining release-success gap.

Together, the ablations support distinct roles for the curricula: cross-task training improves grasp-preserving composition, and reference guidance combined with exploration improves generalization in scene interaction and task completion in manipulation. The reference-guidance ablations are reported in Appendix B.2.

Reviewer_zVtJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Whole-body-scene interaction and object manipulation are two different tasks in robot control. This work proposes a framework that unifies two tasks within a single policy. The application include behaviors such as picking up an object and sitting down (or move to another position) while holding it. Built on a pretrained motion prior, it uses masked transformer inputs, a goal-conditioned loco-manipulation training curriculum, and finally cross-task training to ensure the policy generalizes between different tasks. It outperforms baselines methods by a large margin.

Critical strengths
  1. I personally haven't seen a work that unified locomotion and manipulation so far, so the application domain is novel. The paper is clearly written and the problem it is trying to solve is a important one in embodied AI. I think the evaluation is thorough and solid.

  2. While the framework has multiple stages and seems tailored for unifying locamotion and manipulation, many ideas such as cross-task training (while not groundbreaking) can serve as practical & reusable components for future explorations.

Critical weaknesses

None.

Questions for authors

None.

1 comment

We thank the reviewer for the positive assessment and for recognizing the clarity of our presentation, the importance of the problem, and the thoroughness of our evaluation. We also appreciate the observation that components such as cross-task training can be practical and reusable for future research.

The revised manuscript adds evidence on this point. Without cross-task training, full-sequence success on manipulation-to-scene compositions stays below 9%, compared with 33.21–68.09% with it, although the ablated policy performs the individual tasks comparably to specialists (Table 3(b), Appendix B.1). We also add a failure analysis of the compositions, reference-guidance ablations (Appendix B.2), and a comparison with the goal-conditioned MaskedManipulator baseline (Appendix B.3). All changes are marked in blue and summarized in our general response.

Reviewer_8QRdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Co²Skill presents a shared physics-based controller for simulated humanoids that combines whole-body scene interaction with dexterous object manipulation. A task-conditioned transformer uses observation masks to predict latent residuals over a frozen PULSE motion prior and decoder. It further presents a goal-conditioned curriculum combining partial reference guidance with exploration, and a cross-task curriculum that preserves physical states and grasp conditions during manipulation-to-scene transition

Critical strengths

The framework addresses a meaningful coordination problem. Scene-interaction skills must accommodate the posture, velocities, object state, and grasp produced by preceding manipulation. Training on these continuous physical states while retaining grasp conditions is a well-motivated approach. Integrating furniture and terrain interaction with grasp formation, transport, and release within a shared controller is a potentially useful systems contribution.

The individual-task results demonstrate meaningful capability. The shared policy outperforms the adapted MaskedMimic baseline across all five scene tasks. Stair success improves from 54.41% for the per-task specialist to 69.23% for the shared policy, although sitting success decreases from 99.48% to 93.22%. The synthetic-data study also improves held-out reference-conditioned lifting success from 56.67% to 90.00%. These results provide useful evidence for policy sharing and data augmentation within the evaluated settings.

Critical weaknesses

The central cross-task composition capability lacks quantitative validation. Table 1 evaluates individual scene tasks, while Table 2 evaluates reference-conditioned lifting and pick–move–place. The selected manipulation-to-scene transitions used in training, such as lifting an object and then sitting or climbing while maintaining the grasp, are supported primarily by qualitative demonstrations. Full-sequence completion rates, grasp-retention statistics, execution durations, and uncertainty are needed to assess their reliability. Without these measurements, the evidence remains incomplete for the sequential and simultaneous interactions emphasized by the title and cross-task curriculum.

The incremental contribution of the representation and curricula is insufficiently isolated and explained. The paper cites TokenHSI and explicitly credits its task-token formulation. However, TokenHSI already investigates simultaneous carrying and scene interaction, as well as training-based adaptation to improve skill transitions; MaskedManipulator already supports sparse-goal whole-body manipulation. Co²Skill’s potential advance therefore depends on its particular integration and training mechanisms. The paper provides no controlled ablations isolating reference/exploration mixing or cross-task training, and no matched sparse-goal manipulation comparison. Table 2(a) evaluates data augmentation in the MimicManipulator tracking teacher, rather than comparing against the MaskedManipulator sparse-goal student.

In addition, the method section relies heavily on text and a single overview figure. Diagrams showing token contents, phase-dependent masks, changes in reference supervision, and state handover would make the proposed mechanisms and their differences from prior work substantially clearer.

Questions for authors

Cross-task reliability: What are the full-sequence completion rates, grasp-retention rates, and execution durations for the trained lift→sit, lift→climb, and lift→stair sequences? What are the main failure modes during handover and subsequent scene interaction?

Cross-task curriculum: How does composition performance change when explicit cross-task training is removed while retaining the shared architecture and a comparable total training budget? Please also report the resulting training exposure for each task.

Reference-guided exploration: What proportions of reference-initialized and exploration environments are used for each applicable task, and do these proportions change during training? Are there ablations against reference-only or exploration-only training that establish the benefit of the mixture?

Grasp-target selection at evaluation: Appendix A.1 describes grasp generation and filtering. How is a particular object-relative grasp target selected during evaluation, especially for unseen objects? Does this require reference motions, per-object preprocessing, or manual selection?

Comparison with closely related methods: Can the authors provide a matched goal-conditioned manipulation comparison with the MaskedManipulator student, controlling for training data and supplied goal information? If direct adaptation is impractical, what specific differences prevent a meaningful comparison?

7 comments

We thank the reviewer for recognizing the meaningful coordination problem and the evidence for policy sharing and data augmentation. We have conducted additional experiments addressing the reviewer’s concerns on cross-task composition, training curricula, and goal-conditioned manipulation, and provide further methodological details below.

[W1, Q1] Quantitative validation of cross-task composition

We evaluate the trained lift-to-scene compositions over three seeds, with 2,013 episodes per composition (2,010 for Reach (Stair) and Reach (Climb-down)). Each episode lasts at most 1,000 control steps at 30 Hz. Full-sequence success requires lifting the object and completing the destination task within the episode limit, without a fall or losing object contact. Results are reported as mean ± sample standard deviation across seeds.

Task success rates and timing measurements.

CompositionTotal episodesLift SR (%) ↑Full-sequence SR (%) ↑Grasp retention (%) ↑Successful sequence duration (s)
Lift → Sit2,01397.54 ± 0.8468.09 ± 2.5484.58 ± 1.288.50 ± 0.06
Lift → Climb-up2,01398.12 ± 0.3134.00 ± 0.7467.56 ± 2.3213.50 ± 0.14
Lift → Reach (Free)2,01397.61 ± 1.0859.71 ± 1.1076.17 ± 1.4810.24 ± 0.11
Lift → Reach (Stair)2,01095.87 ± 0.4837.04 ± 1.1951.57 ± 0.9415.77 ± 0.54
Lift → Reach (Climb-down)2,01071.51 ± 1.4433.21 ± 1.0562.77 ± 3.068.45 ± 0.39

Lift and full-sequence success rates use all episodes as the denominator. Grasp retention is conditional on reaching the carry phase and means that no contact-loss termination occurs before the episode ends, even if the destination task is not completed. Durations are measured on successful episodes only.

Failure modes and timing relative to handover.

CompositionPickup failure (%) ↓Post-lift fall / timeout (%) ↓Object loss (%) ↓Failures within 1 s of handover (%)
Lift → Sit2.46 ± 0.8414.41 ± 1.1915.04 ± 1.233.47 ± 0.89
Lift → Climb-up1.88 ± 0.3132.29 ± 1.6931.83 ± 2.351.98 ± 0.44
Lift → Reach (Free)2.39 ± 1.0814.62 ± 1.1623.27 ± 1.673.10 ± 1.47
Lift → Reach (Stair)4.13 ± 0.4812.40 ± 1.4046.44 ± 0.991.31 ± 0.62
Lift → Reach (Climb-down)28.49 ± 1.4411.67 ± 1.9226.64 ± 2.5216.90 ± 2.50

The first three failure rates are computed over all episodes and grouped into mutually exclusive categories: pickup failure before carry entry, object loss after lifting, and post-lift falls or timeouts without object loss. Object loss is triggered by 10 accumulated contact-loss steps and takes precedence over a simultaneous fall.

Handover marks entry into the carry phase. The final column reports failures beginning within 1 s of handover as a percentage of all post-lift failed episodes. Failure onset is the start of the contact-loss accumulation leading to termination, or the termination step for falls and timeouts.

The policy successfully carries objects into subsequent scene interactions, achieving full-sequence success rates of 68.09% for Sit and 59.71% for Reach (Free), with grasp retention of 84.58% and 76.17%. Performance is lower for Climb-up and Reach (Stair), where larger weight shifts and whole-body adjustments may challenge grasp stability. Reach (Climb-down) presents an additional challenge during grasp acquisition. This may reflect different support-surface conditions and stepping-induced whole-body adjustments that destabilize the contact points.

These results are reported in Table 3(b) and Appendix B.1 of the revised manuscript.

[W3, Q2] Cross-task curriculum ablation

To isolate the effect of explicit cross-task training, we compare the shared policy with and without lift-to-scene training environments. Both variants retain the same motion prior and shared architecture and are trained until their evaluation success rates plateau. The environment allocation ratios below remain fixed throughout training.

Training variantScene environments (%)Manipulation environments (%)Lift → scene environments (%)Training transitions
With cross-task training (Ours)3025452.621B
No cross-task training505002.823B

Composition with and without cross-task environments.

We report full-sequence success rates (%) under a 1,000-step episode limit.

VariantLift → Sit (%) ↑Lift → Climb-up (%) ↑Lift → Reach (Free) (%) ↑Lift → Reach (Climb-down) (%) ↑Lift → Reach (Stair) (%) ↑
With cross-task training (Ours)68.09 ± 2.5434.00 ± 0.7459.71 ± 1.1033.21 ± 1.0537.04 ± 1.19
No cross-task training0.20 ± 0.100.09 ± 0.088.37 ± 0.840.36 ± 0.210.51 ± 0.07

The policy with cross-task training achieves substantially higher full-sequence success across all five compositions: 33.21 to 68.09%, compared with 0.09 to 8.37% without it. This gain persists despite the ablated policy receiving slightly more training interactions (2.823B vs. 2.621B). The results support the benefit of training on the states produced by lifting, beyond sharing a controller across individual tasks. This comparison is reported in Table 3(b), with the training setup in Appendix B.1.

[W3, Q3] Reference/exploration proportions and ablation

For scene tasks using reference guidance, a designated subset of environments receives reference-tracking supervision through the reward. The remaining environments support task-directed exploration, initializing the character in randomly sampled poses at positions around the scene target. The tracking target is not supplied as a policy observation, keeping the policy goal-conditioned in both environment groups. This division combines reference-guided learning of coordinated interactions with exploration from diverse initial states.

Scene interaction task. We compare exploration-only, mixed (Ours), and reference-only training on Climb-up, which requires coordinated whole-body motion to raise the character onto the box. Reference environments make up 0%, 60%, and 100% of the training environments, respectively.

VariantRef. SR (%) ↑Ref. error (cm) ↓Explr. SR (%) ↑Explr. error (cm) ↓Evaluated episodes
Exploration-only0.88 ± 0.1811.67 ± 0.490.00 ± 0.00—12,288
Ours99.66 ± 0.055.45 ± 0.0197.75 ± 0.172.99 ± 0.0212,288
Reference-only99.71 ± 0.188.61 ± 0.0275.33 ± 0.778.22 ± 0.0412,288

Evaluation episodes are equally allocated to the reference (Ref.) and exploration (Explr.) groups, yielding 6,144 episodes per group per model. Success requires remaining within 15 cm of the target for 60 consecutive control steps. Error is the distance to the target at the completion of this holding period, averaged over successful episodes only. Results are reported as mean ± sample standard deviation across three seeds.

The mixed policy achieves similarly high success to reference-only training in the reference group (99.66% vs. 99.71%), while improving exploration-group success from 75.33% to 97.75%. Exploration-only training yields very low success in both groups. These results support combining reference guidance with exploration for reliable climb-up performance across both evaluation groups.

Manipulation. We use GRAB motions involving five objects: apple, camera, coffeemug, cup, and waterbottle. Our reference-guidance reward uses pelvis and wrist trajectories to provide dense supervision for approaching the object and forming a grasp. To assess its contribution, we replace this trajectory reward with a distance-based approach reward as a reference-free variant. All other model and training settings are kept unchanged.

Training transitions and task success.

We evaluate the reference-guidance ablation on far goals, which require transport to a different table. “With reference” denotes our policy. “W/o reference” is evaluated at 1.049B training transitions; “W/o reference (trained to plateau)” is the same reference-free training run continued until its release success rate plateaus, at 2.207B transitions.

VariantTraining transitions (#)Lift SR (%) ↑Transport SR (%) ↑Release SR (%) ↑
With reference1.117B97.75 ± 2.5196.63 ± 2.5061.83 ± 3.02
W/o reference1.049B8.97 ± 2.860.84 ± 0.890.32 ± 0.49
W/o reference (trained to plateau)2.207B48.70 ± 0.5647.69 ± 1.101.48 ± 1.05

Each variant is evaluated on 535 far-goal episodes across five seeds; results are mean ± sample standard deviation.

Reference guidance makes grasp acquisition substantially more sample-efficient: at comparable training budgets, lift success increases from 8.97% without guidance to 97.75% with it. Continuing the reference-free run to approximately twice our training budget improves lift success to 48.70%, but does not close the gap; release success also remains low at 1.48%, compared with 61.83% with guidance. Alongside the Climb-up results, this supports using reference rewards to guide the acquisition of coordinated interactions while task-directed exploration develops goal-conditioned behavior.

Persistent lift failures for specific motions.

We examine 20 reference motions for which the reference-free variant fails to lift the object in every trial even after training to a plateau. Counts are successful lifts over matched evaluation attempts pooled over far and close goals. The subset comprises 170 trials.

VariantSuccessful lifts / attemptsLift SR (%) ↑
With reference167/17098.24
W/o reference0/1700.00
W/o reference (trained to plateau)0/1700.00

Our method succeeds in 167/170 attempts (98.24%) on this subset, whereas both reference-free checkpoints fail in all attempts. We hypothesize that these motions involve approach and grasp configurations that are difficult to discover through exploration alone, and that trajectory guidance helps the policy reach them.

Both ablations are reported in Appendix B.2 of the revised manuscript.

Comments on the paper

1 comment

We sincerely thank Reviewers 8QRd, zVtJ, and bPdF for the careful reading and the constructive feedback on our paper, and the AC and SAC for handling the review process. We are glad that the reviewers found the problem important and well motivated ("addresses a meaningful coordination problem", 8QRd; "a meaningful difficulty at the boundary between scene interaction and dexterous manipulation", bPdF), the integration and its application useful ("a potentially useful systems contribution", 8QRd; "the application domain is novel", zVtJ), and the individual-task results credible ("credible evidence that a shared policy can support a useful range of behaviors", bPdF; "the evaluation is thorough and solid", zVtJ).

We are equally grateful for the concerns and questions the reviewers raised, which helped us improve the paper. The main concerns of 8QRd and bPdF point to the same gap: the manipulation-to-scene compositions were shown only qualitatively, and the contribution of each training curriculum was not isolated. In response, we have run the requested experiments and revised the manuscript as follows:

  • Quantitative evaluation of manipulation-to-scene compositions, with lift success, full-sequence success, grasp retention, and duration (Table 3(b)) and a failure breakdown (Appendix B.1, Table 5) — 8QRd W1/Q1, bPdF W1/Q1
  • Quantitative evaluation of transitions between scene tasks against specialist switching (Table 3(a))
  • Cross-task training ablation (Table 3(b), Appendix B.1), with the individual-task performance of the ablated policy (Table 6) — 8QRd W3/Q2, bPdF W2/Q2
  • Reference-guidance and exploration ablations on Climb-up and on manipulation (Appendix B.2, Table 7) — 8QRd W3/Q3, bPdF W2/Q2
  • Comparison with the goal-conditioned MaskedManipulator baseline, using our manipulation specialist under close and far goals (Appendix B.3, Table 8) — 8QRd W4/Q5, bPdF W2/Q3
  • Figure and paragraph on phase-dependent task conditioning (Fig. 3, Sec. 4.1) — 8QRd W5
  • Success criteria for transitions and compositions (Sec. 5.2)
  • Clarification of the data splits, the meaning of unseen objects, and the grasp-target extraction (Appendix A.1) — 8QRd Q4, bPdF Q4
  • Corrections of the cross-task environment allocation (Appendix A.3) and of the mocap sequence counts (Appendix A.1)
  • Updated statements in the introduction, experiments, and conclusion, which now refer to the quantitative composition evaluation

All added or modified content in the revised manuscript is marked in blue. Point-by-point answers are posted under each review. We hope that these revisions resolve the reviewers' concerns and convey more clearly what our unified controller for scene interaction and dexterous manipulation offers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.