- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces LADDER, a long-horizon robot manipulation benchmark built on SimplerEnv, RobotArena-∞, and RoboLab. It covers two single-arm robot embodiments with parallel-jaw grippers, eight task categories, and horizons ranging from one to ten pick-and-place subgoals. The authors evaluate frozen low-level policies conditioned on complete task instructions, VLM-based systems that decompose tasks and orchestrate low-level policies, and systems that execute through motion primitives or task-and-motion planning. The reported results show that the evaluated low-level policies degrade rapidly as task horizon increases under zero-shot composition. High-level orchestration mitigates this degradation, although performance remains constrained by execution errors. The strongest tool-based systems achieve higher success rates. Real-world experiments on WidowX and Franka robots support the simulation results.
Critical strengths
- The authors have implemented multiple evaluation systems and conducted evaluations of a reasonable scale. They report their findings in detail. This provides a relatively reliable comparison of similar systems on long-horizon tasks. The results also corroborate some intuitive conclusions.
- Long-horizon tasks are worth investigating, particularly for understanding the compositional generalization capabilities of VLAs. This work contributes to research in this area.
Critical weaknesses
- I believe there is a logical flaw in the experimental design of this paper. The benchmark aims to evaluate zero-shot skill composition, yet the models evaluated on SimplerEnv and RobotArena are trained on BridgeData, which contains very little long-horizon manipulation data. Unsurprisingly, these models perform poorly. In contrast, the π₀.₅ model trained on DROID, which contains some long-horizon demonstrations, shows substantially better relative performance as a standalone policy on RoboLab. How these models would perform after sufficient training on relevant long-horizon, multitask data remains to be seen. Overall, I find the evaluation and comparison of low-level policies on this benchmark to be of limited significance. They merely confirm an unsurprising result and offer little meaningful insight.
- Much of the effort in this work is devoted to evaluating three categories of control systems. However, the benchmark itself offers limited originality and innovation in its tasks and data.
Questions for authors
How would the performance of the low-level policy and its corresponding VoLo system change after retraining the policy with a substantial amount of relevant long-horizon training data?
As the discussion deadline approaches, the authors kindly encourage the reviewer to review the responses. We have carefully considered and addressed the comments and would greatly appreciate any further feedback!