- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ManiUnit reorganizes demonstrations from 50 BEHAVIOR-1K activities into 137,899 manipulation segments, covering 21 skill types and 417 subtasks. Its benchmark uses restored simulator states, local success conditions, and controlled robot-state variations to evaluate individual skills; the paper reports 78.7% local success for skill-level training versus 49.3% for task-level training.
Critical strengths
- Explicit subtask instructions and restored states make it easier to diagnose skill-specific failures.
- Controlled base and joint perturbations expose how sensitive policies are to changes in their starting state.
- The paper is clearly organized and presented, and the amount of work is commendable.
Critical weaknesses
- The benchmark is primarily simulation-based. Maybe need more sim2real discussion.
- Policy composition is evaluated on only two complete activities, with overall full-task success remaining at 18%.
- I believe the paper lacks discussion of highly relevant work on long-horizon tasks, including RoboCerebra, RoboMemArena, RoboMM, and others.
Questions for authors
I have no extra questions.
7 comments
Response to Reviewer qJoT (2/5): W-1
W-1: The benchmark is primarily simulation-based. Maybe need more sim2real discussion.
Thank you for this constructive suggestion. We have noted this limitation in Sec. 5 and added a sim-to-real discussion in Appendix E around three aspects: (1) the importance of simulation for long-horizon VLA research, (2) simulation-to-real gaps in the OmniGibson ecosystem, and (3) how our curation and evaluation pipeline can inform real-world data collection and evaluation.
(1) Importance of simulation for long-horizon VLA research
Simulation is important for long-horizon VLA research for two main reasons:
- Low-cost, safe exploration for a challenging task. Long-horizon mobile manipulation remains an open challenge: even in simulation, task success rates are still low, indicating that current VLA policies require extensive exploration and iteration. Conducting such large-scale training and evaluation directly on physical hardware entails high manual reset costs as well as safety and hardware risks; simulation therefore provides a safe, low-cost environment for scalable experimentation [1, 2].
- Realistic, controllable environments for repeatable skill diagnosis. Modern simulators closely model everyday household environments while keeping states controllable across trials. For example, in BEHAVIOR-1K [3], OmniGibson provides multi-room homes with realistic rendering, articulated furniture, fluids and particles, and irreversible state changes, while still allowing exact state resets that are difficult to achieve across repeated physical trials. Building on this controllability, ManiUnit starts directly from restored mid-task states to isolate and test individual skills without executing preceding stages, and applies controlled base-position or joint-configuration perturbations. In addition, because the simulator provides direct access to object states and relations, we can define local BDDL goals to check the completion of each individual skill.
(2) Simulation-to-real gaps
Although OmniGibson provides realistic rendering and physics [3], prior sim-to-real studies [1, 3, 4] highlight three main gaps when transferring policies to physical robots:
- Visual and sensor differences. Without camera and lighting calibration, rendered RGB streams differ from real camera observations due to lighting changes, sensor noise, and occlusion, which can affect perception on physical hardware [1, 3].
- Contact and whole-body control. Simulators simplify fine-grained contact and material dynamics, such as assisted grasping and particle approximations in OmniGibson. On physical mobile manipulators, grasping and motion execution are more sensitive to friction changes, slip, and control errors across the base, torso, and arms, which can accumulate across stages [3, 4].
- Scene resets and goal checking. Unlike simulation, physical evaluation cannot directly restore mid-task states or query object states for local BDDL goals, so testing individual skills requires manual scene resets and separate goal verification.
Recent work in OmniGibson also shows that training across diverse scene variants [5] or combining simulation data with real-world demonstrations can help narrow these gaps.
(3) Implications for real-world data collection and evaluation
While ManiUnit is built in simulation, our data-curation pipeline and evaluation findings can also inform real-world data collection and testing:
- Skill-level data curation. Real-world household data collection similarly produces long trajectories that mix navigation, pauses, and multiple operations [4], matching the format of our source simulation demonstrations. After collection, these trajectories can therefore be annotated in the same way and processed with ManiUnit to segment individual skills and pair each segment with an explicit subtask instruction. This consistent data construction also makes real demonstrations easier to combine with simulation data for sim-to-real transfer.
- Starting-state diversity and skill testing. Our benchmark isolates individual skills and shows that starting-state deviations left by preceding stages substantially degrade execution. For real-world data collection, this suggests gathering demonstrations with varied or perturbed starting poses to improve policy robustness. For real-world evaluation, researchers can likewise test subtasks separately and introduce controlled perturbations to assess robustness and locate bottlenecks before full-task execution.
Response to Reviewer qJoT (3/5): W-2
W-2: Policy composition is evaluated on only two complete activities, with overall full-task success remaining at 18%.
Thank you for raising this point. Below, we clarify (1) why our complete-task study focuses on two activities and (2) why overall full-task success is 18.0%.
(1) Why complete-task validation focuses on two activities
We focus our complete-task study on two activities for two main reasons:
- Skill-level focus of ManiUnit. ManiUnit primarily targets skill-level data curation across 50 activities and independent skill diagnosis across 21 skill types (1,260 test instances). The complete-task study serves as a task-level validation to test whether these learned skills can be composed to execute full tasks.
- Computational cost and task selection. Scaling policy composition to many BEHAVIOR-1K activities is computationally expensive. Within our computational budget, we selected Turning On Radio and Picking Up Trash because they cover representative manipulation skills such as Pick Up and Place (which account for 73.8% of ManiUnit segments) across both short and multi-room task horizons.
(2) Why overall full-task success is 18.0%
We attribute the 18.0% full-task success rate to two main factors:
- Inherent task difficulty. Multi-room mobile manipulation in BEHAVIOR-1K remains challenging for current VLA policies: even the winning entry of the 2025 BEHAVIOR Challenge achieved only 12.4% average full-task success across the 50 held-out activities.
- Error and state-drift accumulation across stages. Although skill-level training raises local manipulation success from 49.3% to 78.7%, full-task completion requires every navigation and manipulation stage in the sequence to succeed consecutively [2], while state deviations from preceding steps accumulate across transitions. Accordingly, two-stage Turning On Radio reaches 24.0% full-task success, whereas seven-stage Picking Up Trash reaches 12.0% despite a 26.0% mean Q-score.
To address these challenges, ManiUnit isolates manipulation skills for dedicated training and independent diagnosis. This decouples long-horizon household tasks into navigation, skill-level manipulation, and planning, providing a path to improve each component separately and steadily raise full-task success. Under the same setup on these two activities, composing our skill policy with the task policy already raises overall full-task success from 4.0% to 18.0% and improves mean Q-score from 11.0% to 25.0%, demonstrating the value of skill-level learning.
Response to Reviewer qJoT (1/5): Overview
Thank you for your positive assessment and encouraging review. We appreciate your recognition of (1) our explicit subtask instructions and restored states for diagnosing skill-specific failures, (2) our controlled base and joint perturbations for exposing starting-state sensitivity, and (3) the clear organization, presentation, and substantial effort of our work.
Your comments on simulation-to-real transfer, the scope and performance of complete-task policy composition, and related work on long-horizon manipulation have been very helpful in strengthening the paper. In response, we have clarified these points and revised the manuscript (marked in blue) as follows:
We hope that our point-by-point responses below ((2/5)–(4/5)) can resolve your concerns, and we would be glad to clarify any remaining questions.