- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces ManipArena, a comprehensive real-robot benchmark for studying manipulation capabilities and generalization. It comprises 15 tabletop tasks, five exploratory mobile-manipulation tasks, and 10,812 demonstration trajectories. The benchmark combines schema-defined task variations, stratified evaluation conditions, subgoal-based scoring, and multiple levels of language annotation. The experiments evaluate different policy configurations and systematically investigate the effects of training-data quantity, language granularity, pretraining, and single-task versus multi-task training. Beyond providing a dataset and evaluation platform, the paper offers empirical insights into how these training choices affect real-world manipulation performance.
Critical strengths
-
Comprehensive and well-developed benchmark. The paper brings together a substantial real-robot dataset, diverse manipulation tasks, structured variation specifications, multi-level language annotations, and a detailed evaluation protocol. These components form a coherent resource rather than a collection of isolated demonstrations. The controlled setting is well suited to examining training choices while retaining the challenges of physical execution.
-
Extensive experiments with insightful findings. The evaluation goes beyond reporting a policy leaderboard and systematically examines data quantity, language granularity, pretraining, and task sharing. Controls such as consistent test-time instructions in the language study and architecture-matched pretraining comparisons strengthen the interpretation of the results. I find these analyses particularly valuable because they provide practical insights into training generalist manipulation policies, independently of whether a particular model achieves the highest aggregate score.
Critical weaknesses
- The distinction from existing benchmarks could be articulated more clearly. RoboChallenge [1] already provides controlled real-robot evaluation with stage-based scoring, RoboArena [2] supports comparative real-world policy evaluation, and ManipulationNet [3] addresses both physical skills and embodied reasoning. ManipArena appears complementary through its combination of structured task variations, language annotations, and systematic training studies, but the current presentation could better explain which evaluation questions this combination uniquely enables or addresses more effectively. A structured comparison covering task coverage, variation control, evaluation granularity, and supported diagnostic analyses would make the contribution easier to assess. This is primarily a concern about positioning and demonstrating the benchmark’s additional value, rather than the breadth or completeness of the experimental effort.
Questions for authors
- Could you provide a more explicit comparison with [1–3], identifying the evaluation capabilities and diagnostic questions that ManipArena adds? A concrete example showing how the proposed variation structure or subgoal-level analysis reveals a finding that an aggregate task score would obscure would be particularly useful. This does not require direct numerical comparisons across incompatible robot platforms.
References
[1] Adina Yakefu et al. RoboChallenge: Large-scale Real-robot Evaluation of Embodied Policies. arXiv:2510.17950, 2025.
[2] Pranav Atreya et al. RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies. arXiv:2506.18123, 2025.
[3] Yiting Chen et al. ManipulationNet: An Infrastructure for Benchmarking Real-World Robot Manipulation with Physical Skill Challenges and Embodied Multimodal Reasoning. arXiv:2603.04363, 2026.
2 comments
Thank you for the detailed comparison and the concrete diagnostic examples. This response substantially addresses my concern about ManipArena’s positioning relative to existing benchmarks. I consider my concern about benchmark differentiation largely addressed and maintain my weak-accept recommendation.
Response to Reviewer 3PT1
Q1
Compared to recent real-world robotic evaluation benchmarks, ManipArena complements them through two core strengths: (1) hierarchical out-of-domain evaluation (separating in-domain, visual/configuration-shift, and held-out OOD trials) to measure genuine generalization capabilities, and (2) a comprehensive training–evaluation–simulation suite that provides a complete infrastructure for controlled policy training and diagnosis.
We firstly provide the detailed comparisons between concurrent benchmarks and our ManipArena as follows.
We then provide two concrete examples of what ManipArena's evaluation reveals beyond aggregate success rates. 1) Subgoal scoring identifies where policies fail. On ordered button pressing, π0.5 (Unified), DreamZero, and Fast-WAM all record 0% success, with partial-credit scores of 31/100, 3/100, and 0/100, respectively. π0.5 reliably presses the first target before losing track of the sequence, while DreamZero and Fast-WAM struggle to locate the first target (Tables 9 and 18). 2) Stratifying by variation type reveals opposite effects of joint training. The π0.5 single-task specialists and the 15-task joint policy (Unified–L1) achieve nearly identical overall success rates (19.3% vs. 20.0%). With joint training, the mean partial-credit score rises from 37.0 to 53.1 on Configuration-OOD tasks and falls from 47.2 to 39.7 on Object-OOD tasks (Tables 4 and 9 and Appendix N, Table 19).