acceptodds
Under review as a conference paper at ICLR 2027

ManiUnit: A Manipulation Skill Dataset and Benchmark for Long-Horizon Tasks

Abstract

Long-horizon mobile manipulation requires a robot to navigate multi-room environments and execute a sequence of manipulation skills under a single natural language instruction. Learning and evaluating these skills present three challenges: similar observations under a fixed task instruction may make skill selection ambiguous; even when a preceding skill succeeds, the robot state inherited by the next skill may deviate from its demonstrated starting states and affect execution; and task-level metrics hinder skill-specific diagnosis, while early failures leave later skills untested. We therefore introduce ManiUnit, a manipulation skill dataset and benchmark built from 50 BEHAVIOR-1K activities. Its dataset contains 137,899 segments across 21 skill types and 417 subtasks, and its benchmark contains 1,260 test instances. Correspondingly, ManiUnit pairs each segment with an explicit subtask instruction; measures sensitivity to perturbations of the robot's starting base position or joint configuration; and restores intermediate simulator states and defines local success conditions so that each skill can be evaluated without executing preceding stages. Evaluations of representative vision-language-action (VLA) policies show that similar aggregate scores can hide substantial per-skill differences. The tested starting-state perturbations also degrade execution: on the full benchmark, joint perturbations reduce success rates by approximately 56% relative to those from demonstrated starting states. On two long-horizon activities, a skill policy trained on ManiUnit segments achieves 78.7% local manipulation success, compared with 49.3% for a task policy trained on complete demonstrations. The trained skills further support complete-task execution on these activities, as coordinating the task and skill policies through a planner raises full-task success from 4.0% to 18.0%.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Skill-Level Dataset, Skill-Level Evaluation, Long-Horizon Mobile Manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qJoTOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

ManiUnit reorganizes demonstrations from 50 BEHAVIOR-1K activities into 137,899 manipulation segments, covering 21 skill types and 417 subtasks. Its benchmark uses restored simulator states, local success conditions, and controlled robot-state variations to evaluate individual skills; the paper reports 78.7% local success for skill-level training versus 49.3% for task-level training.

Critical strengths
  1. Explicit subtask instructions and restored states make it easier to diagnose skill-specific failures.
  2. Controlled base and joint perturbations expose how sensitive policies are to changes in their starting state.
  3. The paper is clearly organized and presented, and the amount of work is commendable.
Critical weaknesses
  1. The benchmark is primarily simulation-based. Maybe need more sim2real discussion.
  2. Policy composition is evaluated on only two complete activities, with overall full-task success remaining at 18%.
  3. I believe the paper lacks discussion of highly relevant work on long-horizon tasks, including RoboCerebra, RoboMemArena, RoboMM, and others.
Questions for authors

I have no extra questions.

7 comments
AuthorsOpenReview ↗

Response to Reviewer qJoT (1/5): Overview

Thank you for your positive assessment and encouraging review. We appreciate your recognition of (1) our explicit subtask instructions and restored states for diagnosing skill-specific failures, (2) our controlled base and joint perturbations for exposing starting-state sensitivity, and (3) the clear organization, presentation, and substantial effort of our work.

Your comments on simulation-to-real transfer, the scope and performance of complete-task policy composition, and related work on long-horizon manipulation have been very helpful in strengthening the paper. In response, we have clarified these points and revised the manuscript (marked in blue) as follows:

  1. Simulation-to-real discussion (W-1): Added a dedicated discussion on the importance of simulation for long-horizon VLA research, simulation-to-real gaps in OmniGibson, and how our data-curation pipeline and evaluation findings can inform real-world data collection and testing (revised in Sec. 5 and Appendix E).
  2. Complete-task scope and 18.0% success rate (W-2): Clarified why complete-task validation focuses on two activities, analyzed the main factors behind the 18.0% full-task success rate, and highlighted how decoupling manipulation skills provides a modular path to improve full-task execution.
  3. Related long-horizon works (W-3): Expanded our related-work discussion to cover RoboCerebra, RoboMemArena, RoboMM (RoboTron-Mani), RoboMME, and MemoryVLA, clarifying how ManiUnit complements them through decoupled skill learning for mobile manipulation and independent skill diagnosis under starting-state perturbations (revised in Appendix A).

We hope that our point-by-point responses below ((2/5)–(4/5)) can resolve your concerns, and we would be glad to clarify any remaining questions.

AuthorsOpenReview ↗

Response to Reviewer qJoT (2/5): W-1

W-1: The benchmark is primarily simulation-based. Maybe need more sim2real discussion.

Thank you for this constructive suggestion. We have noted this limitation in Sec. 5 and added a sim-to-real discussion in Appendix E around three aspects: (1) the importance of simulation for long-horizon VLA research, (2) simulation-to-real gaps in the OmniGibson ecosystem, and (3) how our curation and evaluation pipeline can inform real-world data collection and evaluation.

(1) Importance of simulation for long-horizon VLA research

Simulation is important for long-horizon VLA research for two main reasons:

  • Low-cost, safe exploration for a challenging task. Long-horizon mobile manipulation remains an open challenge: even in simulation, task success rates are still low, indicating that current VLA policies require extensive exploration and iteration. Conducting such large-scale training and evaluation directly on physical hardware entails high manual reset costs as well as safety and hardware risks; simulation therefore provides a safe, low-cost environment for scalable experimentation [1, 2].
  • Realistic, controllable environments for repeatable skill diagnosis. Modern simulators closely model everyday household environments while keeping states controllable across trials. For example, in BEHAVIOR-1K [3], OmniGibson provides multi-room homes with realistic rendering, articulated furniture, fluids and particles, and irreversible state changes, while still allowing exact state resets that are difficult to achieve across repeated physical trials. Building on this controllability, ManiUnit starts directly from restored mid-task states to isolate and test individual skills without executing preceding stages, and applies controlled base-position or joint-configuration perturbations. In addition, because the simulator provides direct access to object states and relations, we can define local BDDL goals to check the completion of each individual skill.

(2) Simulation-to-real gaps

Although OmniGibson provides realistic rendering and physics [3], prior sim-to-real studies [1, 3, 4] highlight three main gaps when transferring policies to physical robots:

  • Visual and sensor differences. Without camera and lighting calibration, rendered RGB streams differ from real camera observations due to lighting changes, sensor noise, and occlusion, which can affect perception on physical hardware [1, 3].
  • Contact and whole-body control. Simulators simplify fine-grained contact and material dynamics, such as assisted grasping and particle approximations in OmniGibson. On physical mobile manipulators, grasping and motion execution are more sensitive to friction changes, slip, and control errors across the base, torso, and arms, which can accumulate across stages [3, 4].
  • Scene resets and goal checking. Unlike simulation, physical evaluation cannot directly restore mid-task states or query object states for local BDDL goals, so testing individual skills requires manual scene resets and separate goal verification.

Recent work in OmniGibson also shows that training across diverse scene variants [5] or combining simulation data with real-world demonstrations can help narrow these gaps.

(3) Implications for real-world data collection and evaluation

While ManiUnit is built in simulation, our data-curation pipeline and evaluation findings can also inform real-world data collection and testing:

  • Skill-level data curation. Real-world household data collection similarly produces long trajectories that mix navigation, pauses, and multiple operations [4], matching the format of our source simulation demonstrations. After collection, these trajectories can therefore be annotated in the same way and processed with ManiUnit to segment individual skills and pair each segment with an explicit subtask instruction. This consistent data construction also makes real demonstrations easier to combine with simulation data for sim-to-real transfer.
  • Starting-state diversity and skill testing. Our benchmark isolates individual skills and shows that starting-state deviations left by preceding stages substantially degrade execution. For real-world data collection, this suggests gathering demonstrations with varied or perturbed starting poses to improve policy robustness. For real-world evaluation, researchers can likewise test subtasks separately and introduce controlled perturbations to assess robustness and locate bottlenecks before full-task execution.
AuthorsOpenReview ↗

Response to Reviewer qJoT (3/5): W-2

W-2: Policy composition is evaluated on only two complete activities, with overall full-task success remaining at 18%.

Thank you for raising this point. Below, we clarify (1) why our complete-task study focuses on two activities and (2) why overall full-task success is 18.0%.

(1) Why complete-task validation focuses on two activities

We focus our complete-task study on two activities for two main reasons:

  • Skill-level focus of ManiUnit. ManiUnit primarily targets skill-level data curation across 50 activities and independent skill diagnosis across 21 skill types (1,260 test instances). The complete-task study serves as a task-level validation to test whether these learned skills can be composed to execute full tasks.
  • Computational cost and task selection. Scaling policy composition to many BEHAVIOR-1K activities is computationally expensive. Within our computational budget, we selected Turning On Radio and Picking Up Trash because they cover representative manipulation skills such as Pick Up and Place (which account for 73.8% of ManiUnit segments) across both short and multi-room task horizons.

(2) Why overall full-task success is 18.0%

We attribute the 18.0% full-task success rate to two main factors:

  • Inherent task difficulty. Multi-room mobile manipulation in BEHAVIOR-1K remains challenging for current VLA policies: even the winning entry of the 2025 BEHAVIOR Challenge achieved only 12.4% average full-task success across the 50 held-out activities.
  • Error and state-drift accumulation across stages. Although skill-level training raises local manipulation success from 49.3% to 78.7%, full-task completion requires every navigation and manipulation stage in the sequence to succeed consecutively [2], while state deviations from preceding steps accumulate across transitions. Accordingly, two-stage Turning On Radio reaches 24.0% full-task success, whereas seven-stage Picking Up Trash reaches 12.0% despite a 26.0% mean Q-score.

To address these challenges, ManiUnit isolates manipulation skills for dedicated training and independent diagnosis. This decouples long-horizon household tasks into navigation, skill-level manipulation, and planning, providing a path to improve each component separately and steadily raise full-task success. Under the same setup on these two activities, composing our skill policy with the task policy already raises overall full-task success from 4.0% to 18.0% and improves mean Q-score from 11.0% to 25.0%, demonstrating the value of skill-level learning.

Reviewer_Y3cMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ManiUnit reorganizes demonstrations from 50 BEHAVIOR-1K activities into 137,899 manipulation segments covering 21 skill types and 417 subtasks. It provides explicit subtask instructions and independent skill evaluation through restored simulator states, local success conditions, and controlled starting-state perturbations. Experiments reveal differences in skill performance and sensitivity to starting states.

Critical strengths
  • Useful independent skill evaluation. Restored states and local BDDL goals allow manipulation stages to be evaluated without executing preceding stages. Careful handling of grasp constraints, object changes, and instance validation strengthens the benchmark’s diagnostic value.

  • Practical value supported by initial experiments. Curated instructions and traceable segment metadata support targeted skill training. Evaluations expose skill differences hidden by aggregate scores.

Critical weaknesses
  • Limited evaluation independence and coverage. Original evaluation starts come from training demonstrations, and perturbed tests reuse these anchors. This supports controlled diagnosis but leaves generalization to unseen episodes or scenes insufficiently established. The benchmark also uses only 20 anchors per skill type, providing limited coverage of diverse categories such as Pick & Place.

  • Incomplete attribution of performance gains. The skill vs. task comparison changes both training segments and instruction granularity, while complete-task evaluation additionally introduces a planner. These experiments demonstrate the benefit of the combined setup but do not isolate the contribution of ManiUnit’s data curation. Complete-task validation is also restricted to two activities.

Questions for authors
  • Are the complete-task test episodes disjoint from policy training, planner training, and the retrieval database? Please also clarify the evaluation anchors’ coverage across episodes and subtasks.

  • Under identical instructions and training budgets, how does ManiUnit compare with unrefined manipulation segments? Does the advantage persist when using the same planner?

  • How were the perturbation magnitudes selected, and how closely do they match the state deviations produced by successfully executed preceding skills?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Y3cM (1/4): Overview

Thank you for your careful review and positive assessment. We appreciate your recognition of (1) a useful independent skill evaluation protocol that evaluates manipulation stages without executing preceding stages, with careful handling of grasp constraints, object changes, and instance validation, and (2) the practical value of our curated instructions and traceable segment metadata in supporting targeted skill training and exposing skill differences hidden by aggregate scores.

Your comments on evaluation independence and coverage, generalization, the attribution of performance gains, and perturbation selection have been very helpful in strengthening the paper. In response, we have added new experiments and revised the manuscript (marked in blue) as follows:

  1. Evaluation independence, coverage, and generalization (W-1, Q-1): Clarified that complete-task test episodes are disjoint from training and retrieval data (Appendix D.3), added the episode, activity, and subtask coverage breakdown of the evaluation anchors (Table R1; revised in Appendix B.1), and detailed our generalization efforts, including a multi-object recombination test and scene-variation pipeline.
  2. Sources of gains (W-2, Q-2): Added controlled ablations isolating segment extraction, boundary refinement, and subtask instructions (Table R2) and validating long-horizon effectiveness under the same planner (Table R3; revised in Sec. 4 and Appendix D.4).
  3. Perturbation selection and realism (Q-3): Clarified how our perturbation ranges were determined and compared the synthetic perturbations with actual states from preceding-skill rollouts (Table R4) and complete long-horizon rollouts (Table R5; revised in Appendix C.3).

We hope that our point-by-point responses below ((2/4)–(4/4)) can resolve your concerns, and we would be glad to clarify any remaining questions.

AuthorsOpenReview ↗

Response to Reviewer Y3cM (2/4): W-1, Q-1

W-1: Original evaluation starts come from training demonstrations, and perturbed tests reuse these anchors. This supports controlled diagnosis but leaves generalization to unseen episodes or scenes insufficiently established. The benchmark also uses only 20 anchors per skill type, providing limited coverage of diverse categories such as Pick & Place.

Q-1: Are the complete-task test episodes disjoint from policy training, planner training, and the retrieval database? Please also clarify the evaluation anchors' coverage across episodes and subtasks.

Thank you for raising this important point. We would like to clarify that the complete-task test episodes in Sec. 4.4 are disjoint from all training and retrieval data: all policy training, planner training, and retrieval databases are built exclusively from the training split, whereas these test episodes come from the official BEHAVIOR-1K test split (which provides initial scene configurations without demonstration trajectories).

Below, we present (1) the actual coverage of the evaluation anchors, (2) why each skill type uses 20 anchors, and (3) our efforts toward evaluating generalization.

(1) Actual coverage of the evaluation suite

Because each skill type has only 20 anchors, we selected them to maximize diversity under three principles:

  • Distinct episodes. Every anchor is drawn from a different source episode, so even skills with a single subtask, such as Ignite or Wipe, are evaluated on 20 different demonstrations.
  • All activities. Every source activity is represented in the evaluation.
  • As many subtasks as possible. Within each skill type, anchors cover as many distinct subtasks as 20 anchors allow.

As summarized in Table R1, the 420 Full anchors therefore come from 420 distinct source episodes and cover all 50 activities and 120 subtasks. The 19 skill types with at most 20 subtasks cover all of their subtasks (e.g., Chop 18, Pour 14, Open Door 9), and Pick Up and Place each span 20 distinct subtasks.

Table R1: Coverage of the Full evaluation suite. Each anchor yields Original, Base, and Joint test instances.

SuiteAnchorsSkill typesSubtasksActivitiesSource episodes
Full4202112050420

(2) Why 20 anchors per skill type, and further expansion

The choice of 20 anchors balances evaluation coverage against cost, and we provide two ways to extend coverage beyond it:

  • Current evaluation cost. Each of the 21 skill types has 20 anchors, and each anchor yields three test instances (Original, Base, and Joint), repeated three times (21 × 20 × 3 × 3 = 3,780 rollouts per policy). A single rollout takes ~23 minutes on two RTX 4090 GPUs, so evaluating one policy on Full costs about 2,900 RTX 4090 GPU-hours.
  • Customizable test sets and expansion. We will open-source our complete test-set construction pipeline so researchers can build custom test sets, and we are expanding subtask-level test sets covering more subtasks, objects, and spatial contexts in our open-source repository.

(3) Generalization efforts

  • Why the current benchmark does not yet include tests with new objects or unseen scenes. Existing policies still struggle in the demonstrated scenes: even under Original starts, StarVLA-PI and StarVLA-GR00T reach only 60.1% and 63.5% on Full, and success drops further under perturbed starts, especially with Joint Perturbation. Because skill execution still leaves substantial room for improvement in familiar scenes, the current benchmark concentrates on starting-state sensitivity.
  • A generalization test that we built and will release. While constructing this benchmark, we also built a harder Pick Up test that recombines objects from 10 categories (e.g., radio, frying pan, and baseball cap) and three supporting surfaces (tables, floors, and countertops) into 15 unseen layouts, each placing four objects of different categories on one surface. For each layout, we keep the scene fixed and change only the instruction and local goal (e.g., "pick up the radio from the table" vs. "pick up the baseball cap from the table") so each object becomes the target once. In our preliminary evaluation, baseline policies trained only on the source demonstrations struggled in these unfamiliar layouts, so we did not include this test in the initial paper. Following your suggestion, we will release this test as a separate high-difficulty challenge for evaluating generalization.
  • Object and scene variations already in the paper. The paper also uses Pick Up to test three scene variations (Table 1), varying target position, target color, and distractor presence one factor at a time while keeping the robot's starting configuration fixed. Researchers can apply the same variations to other skills or build test sets tailored to their own needs.
AuthorsOpenReview ↗

Response to Reviewer Y3cM (3/4): W-2, Q-2

W-2: The skill vs. task comparison changes both training segments and instruction granularity, while complete-task evaluation additionally introduces a planner. These experiments demonstrate the benefit of the combined setup but do not isolate the contribution of ManiUnit's data curation. Complete-task validation is also restricted to two activities.

Q-2: Under identical instructions and training budgets, how does ManiUnit compare with unrefined manipulation segments? Does the advantage persist when using the same planner?

Thank you for this valuable suggestion. We first clarify that ManiUnit's data curation comprises two steps: (1) extracting and merging source annotations into segments via manually defined rules, and (2) refining segment boundaries and writing explicit subtask instructions. To directly examine the effect of each factor, we train two controlled models and evaluate them at both the skill (Sec. 4.3) and complete-task (Sec. 4.4) levels:

  • Policy A: ManiUnit segments with complete-task instructions (isolating subtask instructions).
  • Policy B: Unrefined segments with subtask instructions (isolating boundary refinement).

(1) Skill-level comparison

As shown in Table R2:

  • Segment extraction (Task policy vs. Policy A): With task instructions fixed, extracted segments allow the model to focus learning on manipulation, raising overall success from 49.3% to 68.0%.
  • Subtask instructions (Policy A vs. Skill policy): With segments fixed, subtask instructions allow the model to map each subtask independently, further raising success from 68.0% to 78.7%.
  • Boundary refinement (Policy B vs. Skill policy): With instructions and training epochs fixed, refining boundaries trims surrounding navigation and inactivity, raising success from 73.3% to 78.7%.

Table R2: Local manipulation success rates in Section 4.3 (%). Overall averages Original, Base, and Joint conditions across the 75 test instances. † Reported in the manuscript.

PolicyTraining dataInstructionOriginalBaseJointOverall
Task policy†Complete demonstrationsTask60.056.032.049.3
Policy AManiUnit segmentsTask72.068.064.068.0
Policy BUnrefined segmentsSubtask80.072.068.073.3
Skill policy†ManiUnit segmentsSubtask84.076.076.078.7

(2) Task-level comparison under the same planner

As shown in Table R3, the advantage persists in complete tasks. We only replace the manipulation policy with Policy A or Policy B under the exact same planner. Since Policy A is trained with the task instruction, the planner only switches between the task policy and Policy A while keeping the complete-task prompt throughout execution. The results confirm that trimming data to focus the model on manipulation improves overall full-task success, while explicit subtask instructions enable clearer stage modeling and further boost performance.

Table R3: Long-horizon task execution with policy composition. Full-task success rates (SR, %) and mean Q-scores (Q, %). † Reported in the manuscript.

SystemRadio SRRadio QTrash SRTrash QOverall SROverall Q
Task policy only†4.04.04.018.04.011.0
Task policy + Policy A + Planner10.010.04.020.77.015.4
Task policy + Policy B + Planner20.020.08.024.614.022.3
Task policy + Skill policy + Planner†24.024.012.026.018.025.0

(3) Why complete-task validation focuses on two activities

We focus our complete-task study on two activities for two main reasons:

  • Skill-level focus of ManiUnit. ManiUnit primarily targets skill-level data curation across 50 activities and independent skill diagnosis across 21 skill types (1,260 test instances). The complete-task study serves as a task-level validation to test whether these learned skills can be composed to execute full tasks.
  • Computational cost and task selection. Scaling policy composition to many BEHAVIOR-1K activities is computationally expensive. Within our computational budget, we selected Turning On Radio and Picking Up Trash because they cover representative manipulation skills such as Pick Up and Place (which account for 73.8% of ManiUnit segments) across both short and multi-room task horizons.
Reviewer_3vjGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ManiUnit, a skill-level dataset and benchmark for long-horizon robotic manipulation, built from BEHAVIOR-1K demonstrations. It splits full-task trajectories into manipulation segments with subtask instructions and supports independent skill evaluation through restored simulator states and local success conditions. The benchmark also tests robustness to changes in the robot's starting state. Experiments on several VLA models show differences in skill-level performance, while additional experiments on two activities show improvements from skill-level training and policy composition.

Critical strengths
  1. A useful skill-level dataset. ManiUnit provides 137,899 manipulation segments across 21 skill types and 417 subtasks, each with a subtask instruction. The authors also provide details on how the segments are extracted, merged, and refined. This makes existing BEHAVIOR-1K demonstrations easier to use for skill-level training and analysis.

  2. A practical skill-level evaluation protocol. By restoring intermediate simulator states and defining local BDDL goals, the benchmark can evaluate individual skills without running the preceding steps. The authors also handle issues such as grasp constraints during state restoration. The starting-state perturbations provide a useful way to test VLA robustness and identify failures that may be hidden by overall task success.

Critical weaknesses
  1. Dataset balance and coverage. ManiUnit covers 50 BEHAVIOR-1K activities, but Pick Up and Place account for 73.8% of all segments, while some skills appear in only one activity. The benchmark also uses only 20 starting-state anchors per skill type. More diverse examples would make the evaluation more convincing, especially for less common skills.
  2. What drives the improvements? In Sec. 4.3, the task and skill policies differ in both training data and instructions. It is unclear how much of the gain comes from skill-level training versus explicit subtask instructions. Similarly, Sec. 4.4 introduces a planner alongside the skill policy, but does not test the task policy with the same planner. These controls are needed to explain the reported improvements.
  3. Starting-state perturbations. ManiUnit tests robustness by changing the robot's base position or joint configuration. However, the paper does not show whether these perturbations resemble the states left by preceding skills in actual rollouts. It would be useful to compare the two, since this mismatch is one of the main motivations for the benchmark.
  4. Limited generalization evaluation. Most tests use starting states taken from existing demonstrations. Although the paper includes some perturbation tests, it provides little evidence of how well the learned skills work with new objects or in unseen scenes. Generalization beyond the original demonstrations remains unclear.
Questions for authors
  1. How representative are the 20 evaluation anchors per skill type, especially for skills with few training examples? Can the authors evaluate on more diverse starting states or objects?
  2. Can the authors separate the effects of skill-level training and explicit subtask instructions in Sec. 4.3? Also, how does the task policy perform when using the same planner in Sec. 4.4?
  3. Have the authors compared the synthetic starting-state perturbations with the states produced by preceding skills in actual long-horizon rollouts?
  4. Can the authors report annotation error rates or consistency checks for the segment boundaries, subtask instructions, and local BDDL goals?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 3vjG (1/6): Overview

Thank you for your careful and constructive review. We appreciate your recognition of (1) our curated skill-level dataset, which reorganizes BEHAVIOR-1K demonstrations with explicit subtask instructions for targeted skill training and analysis, and (2) our practical evaluation protocol, which uses restored states, local BDDL goals, and starting-state perturbations to test individual skills independently and expose failures hidden by overall task success.

Your comments on dataset balance and evaluation coverage, the sources of the reported gains, perturbation realism, generalization, and annotation quality have been very helpful in strengthening the paper. In response, we have added new experiments and revised the manuscript (marked in blue) as follows:

  1. Dataset balance and evaluation coverage (W-1, Q-1): Clarified the long-tailed skill distribution, added the episode, activity, and subtask coverage breakdown of the evaluation anchors (Table R1; revised in Appendix B.1), and detailed our open-source pipeline for test-set expansion.
  2. Sources of gains (W-2, Q-2): Added controlled ablations isolating skill-level training data from subtask instructions (Table R2) and validating long-horizon effectiveness under the same planner (Table R3; revised in Sec. 4 and Appendix D.4).
  3. Perturbation realism (W-3, Q-3): Clarified how our perturbation ranges were determined and compared the synthetic perturbations with actual states from preceding-skill rollouts (Table R4) and complete long-horizon rollouts (Table R5; revised in Appendix C.3).
  4. Generalization (W-4): Detailed our efforts toward evaluating generalization, including a multi-object recombination test to be released and our scene-variation pipeline.
  5. Annotation quality (Q-4): Clarified our three-expert review across segmentation rules, subtask instructions, and local goals, together with simulation-based BDDL validation (revised in Appendix B.4).

We hope that our point-by-point responses below ((2/6)–(6/6)) can resolve your concerns, and we would be glad to clarify any remaining questions.

AuthorsOpenReview ↗

Response to Reviewer 3vjG (2/6): W-1, Q-1

W-1: ManiUnit covers 50 BEHAVIOR-1K activities, but Pick Up and Place account for 73.8% of all segments, while some skills appear in only one activity. The benchmark also uses only 20 starting-state anchors per skill type. More diverse examples would make the evaluation more convincing, especially for less common skills.

Q-1: How representative are the 20 evaluation anchors per skill type, especially for skills with few training examples? Can the authors evaluate on more diverse starting states or objects?

Thank you for raising this point. We organize our response around three aspects: (1) why the training data are long-tailed, (2) the actual coverage of the evaluation suite, and (3) why each skill type uses 20 anchors and how we will extend evaluation beyond them.

(1) Why the training data are long-tailed

We explain this distribution from three aspects:

  • It matches the distribution of real manipulation tasks. When everyday manipulation tasks are decomposed into individual operations, most of these operations pick up or place objects. Existing datasets and benchmarks show the same trend: Pick Up and Place appear in 50 and 45 of the 50 BEHAVIOR-1K activities, respectively, and pick-and-place tasks account for 115 of the 130 tasks (88%) in LIBERO [1] and 31 of the 50 tasks (62%) in RoboTwin 2.0 [2].
  • Less common skills still have a reasonable amount of data. At the other end of the distribution, the five least frequent skill types (Attach, Hang, Ignite, Press, and Wipe) each contain about 200 segments.
  • The imbalance can be handled in both training and evaluation. Because every segment is indexed by skill type and subtask, users can rebalance training through sampling weights or select data for a particular skill. In evaluation, every skill type receives equal weight in the aggregate score.

(2) Actual coverage of the evaluation suite

Because each skill type has only 20 anchors, we selected them to maximize diversity under three principles:

  • Distinct episodes. Every anchor is drawn from a different source episode, so even skills with a single subtask, such as Ignite or Wipe, are evaluated on 20 different demonstrations.
  • All activities. Every source activity is represented in the evaluation.
  • As many subtasks as possible. Within each skill type, anchors cover as many distinct subtasks as 20 anchors allow.

As summarized in Table R1, the 420 Full anchors therefore come from 420 distinct source episodes and cover all 50 activities and 120 subtasks. The 19 skill types with at most 20 subtasks cover all of their subtasks (e.g., Chop 18, Pour 14, Open Door 9), and Pick Up and Place each span 20 distinct subtasks.

Table R1: Coverage of the Full evaluation suite. Each anchor yields Original, Base, and Joint test instances.

SuiteAnchorsSkill typesSubtasksActivitiesSource episodes
Full4202112050420

(3) Why 20 anchors per skill type, and further expansion

The choice of 20 anchors balances evaluation coverage against cost, and we provide two ways to extend coverage beyond it:

  • Current evaluation cost. In the current Full suite, each of the 21 skill types has 20 anchors, and each anchor yields three test instances (Original, Base, and Joint). To account for the stochasticity of policy rollouts, each test instance is repeated three times, giving 21 × 20 × 3 × 3 = 3,780 rollouts per policy. A single rollout takes about 23 minutes on two RTX 4090 GPUs, one serving the policy and one running the simulator. Evaluating a single policy on Full therefore costs about 2,900 RTX 4090 GPU-hours.
  • Customizable test sets and expansion. We will open-source our complete test-set construction pipeline so researchers can build custom test sets, and we are expanding subtask-level test sets covering more subtasks, objects, and spatial contexts in our open-source repository.

[1] Bo Liu et al. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. In Advances in Neural Information Processing Systems, volume 36, pp. 44776–44791. Curran Associates, Inc., 2023.

[2] Tianxing Chen et al. RoboTwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. In Forty-third International Conference on Machine Learning, 2026.

AuthorsOpenReview ↗

Response to Reviewer 3vjG (3/6): W-2, Q-2

W-2: In Sec. 4.3, the task and skill policies differ in both training data and instructions. It is unclear how much of the gain comes from skill-level training versus explicit subtask instructions. Similarly, Sec. 4.4 introduces a planner alongside the skill policy, but does not test the task policy with the same planner.

Q-2: Can the authors separate the effects of skill-level training and explicit subtask instructions in Sec. 4.3? Also, how does the task policy perform when using the same planner in Sec. 4.4?

Thank you for this valuable suggestion. Following your suggestion, we first separate skill-level training data from subtask instructions under the Sec. 4.3 protocol, and then evaluate task-level control under the same planner under the Sec. 4.4 protocol.

(1) Skill-level comparison

To measure how much of the gain comes from skill-level training versus explicit subtask instructions, we train Policy A under the same experimental setup as Sec. 4.3, using the ManiUnit skill data but pairing each segment with the complete-task instruction of its source activity. As shown in Table R2, the results reveal two findings:

  • Training data gain: Comparing the task policy and Policy A, with the task instruction fixed, training on ManiUnit segments raises overall success from 49.3% to 68.0%. That is, segmenting the data allows the model to focus more on the manipulation stages and improves performance.
  • Instruction gain: Comparing Policy A and the skill policy, with the training segments fixed, replacing the task instruction with subtask instructions further raises overall success from 68.0% to 78.7%. Using distinct prompts for different subtasks resolves stage ambiguity under similar visual observations and allows the model to learn each operation more clearly.

Table R2: Local manipulation success rates in Section 4.3 (%). Overall averages Original, Base, and Joint conditions across the 75 test instances. † Reported in the manuscript.

PolicyTraining dataInstructionOriginalBaseJointOverall
Task policy†Complete demonstrationsTask60.056.032.049.3
Policy AManiUnit segmentsTask72.068.064.068.0
Skill policy†ManiUnit segmentsSubtask84.076.076.078.7

(2) Task-level comparison under the same planner

As shown in Table R3, long-horizon task evaluation under the Sec. 4.4 setup shows the same trend. In Sec. 4.4, the task policy already serves as the default controller and is trained only with complete-task instructions, so running it alone under the planner with task instructions reduces to Task policy only. We therefore evaluate Task policy + Policy A + Planner, where the planner switches between the task policy and Policy A while keeping the complete-task instruction throughout execution.

Table R3: Long-horizon task execution with policy composition. Full-task success rates (SR, %) and mean Q-scores (Q, %). † Reported in the manuscript.

SystemRadio SRRadio QTrash SRTrash QOverall SROverall Q
Task policy only†4.04.04.018.04.011.0
Task policy + Policy A + Planner10.010.04.020.77.015.4
Task policy + Skill policy + Planner†24.024.012.026.018.025.0

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.