acceptodds
Under review as a conference paper at ICLR 2027

ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation

21 views

Abstract

Vision-Language-Action (VLA) models and world-action models have emerged as central paradigms for general-purpose robotic intelligence, yet their empirical progress remains constrained by the absence of evaluation protocols that are both physically realistic and diagnostically controlled. Simulator-centric benchmarks provide scale and reproducibility, but cannot fully capture the reality gap induced by perception noise, contact dynamics, latency, calibration error, and hardware constraints. Conversely, real-robot evaluations are often fragmented across platforms, scenes, objects, and scoring rules, making fair comparison and failure attribution difficult. We introduce ManipArena, a standardized real-robot evaluation framework for studying manipulation generalization under matched physical conditions. ManipArena comprises 20 tasks, 10,812 expert trajectories, 13.5M frames, and approximately 188 robot hours across tabletop and mobile manipulation. The framework combines schema-defined task variation, stratified in-domain, visualshift, and semantic-OOD trials, subtask-level partial-credit scoring, three-level language annotations, low-level motor signals, and paired real-to-sim environments reconstructed from physical scenes. Using ManipArena, we evaluate seven tabletop configurations spanning VLA and world-action-model policies. The results show that real-robot conclusions depend not only on architecture, but also on model provenance, fine-tuning regime, data sampling, and annotation granularity. ManipArena thus provides a reproducible and interpretable foundation for diagnosing capability boundaries and failure modes in embodied generalization.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3PT1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces ManipArena, a comprehensive real-robot benchmark for studying manipulation capabilities and generalization. It comprises 15 tabletop tasks, five exploratory mobile-manipulation tasks, and 10,812 demonstration trajectories. The benchmark combines schema-defined task variations, stratified evaluation conditions, subgoal-based scoring, and multiple levels of language annotation. The experiments evaluate different policy configurations and systematically investigate the effects of training-data quantity, language granularity, pretraining, and single-task versus multi-task training. Beyond providing a dataset and evaluation platform, the paper offers empirical insights into how these training choices affect real-world manipulation performance.

Critical strengths
  1. Comprehensive and well-developed benchmark. The paper brings together a substantial real-robot dataset, diverse manipulation tasks, structured variation specifications, multi-level language annotations, and a detailed evaluation protocol. These components form a coherent resource rather than a collection of isolated demonstrations. The controlled setting is well suited to examining training choices while retaining the challenges of physical execution.

  2. Extensive experiments with insightful findings. The evaluation goes beyond reporting a policy leaderboard and systematically examines data quantity, language granularity, pretraining, and task sharing. Controls such as consistent test-time instructions in the language study and architecture-matched pretraining comparisons strengthen the interpretation of the results. I find these analyses particularly valuable because they provide practical insights into training generalist manipulation policies, independently of whether a particular model achieves the highest aggregate score.

Critical weaknesses
  1. The distinction from existing benchmarks could be articulated more clearly. RoboChallenge [1] already provides controlled real-robot evaluation with stage-based scoring, RoboArena [2] supports comparative real-world policy evaluation, and ManipulationNet [3] addresses both physical skills and embodied reasoning. ManipArena appears complementary through its combination of structured task variations, language annotations, and systematic training studies, but the current presentation could better explain which evaluation questions this combination uniquely enables or addresses more effectively. A structured comparison covering task coverage, variation control, evaluation granularity, and supported diagnostic analyses would make the contribution easier to assess. This is primarily a concern about positioning and demonstrating the benchmark’s additional value, rather than the breadth or completeness of the experimental effort.
Questions for authors
  1. Could you provide a more explicit comparison with [1–3], identifying the evaluation capabilities and diagnostic questions that ManipArena adds? A concrete example showing how the proposed variation structure or subgoal-level analysis reveals a finding that an aggregate task score would obscure would be particularly useful. This does not require direct numerical comparisons across incompatible robot platforms.

References

[1] Adina Yakefu et al. RoboChallenge: Large-scale Real-robot Evaluation of Embodied Policies. arXiv:2510.17950, 2025.

[2] Pranav Atreya et al. RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies. arXiv:2506.18123, 2025.

[3] Yiting Chen et al. ManipulationNet: An Infrastructure for Benchmarking Real-World Robot Manipulation with Physical Skill Challenges and Embodied Multimodal Reasoning. arXiv:2603.04363, 2026.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3PT1

Q1

Compared to recent real-world robotic evaluation benchmarks, ManipArena complements them through two core strengths: (1) hierarchical out-of-domain evaluation (separating in-domain, visual/configuration-shift, and held-out OOD trials) to measure genuine generalization capabilities, and (2) a comprehensive training–evaluation–simulation suite that provides a complete infrastructure for controlled policy training and diagnosis.

We firstly provide the detailed comparisons between concurrent benchmarks and our ManipArena as follows.

DimensionRoboChallengeRoboArenaManipulationNetManipArena (Ours)
Task Coverage & Data30 static tabletop tasks across 4 platforms; demos lack force/current signals.Open-ended single-arm DROID tasks; no paired training set.5 tabletop tasks across Physical Skills and Embodied Reasoning; no training dataset.15 tabletop tasks (10 execution, 5 semantic) and 5 mobile manipulation tasks; 10,812 trajectories with L1–L3 labels, joint velocities/currents, and paired simulation scenes.
Variation ControlReference frames guide object reset; lighting/background uncontrolled; no explicit OOD split.Initial states aligned within A/B pairs; no cross-session standardization.Standardized physical kits and AprilTag masks; no defined ID/OOD split.Fixed lighting and task schemas; 4 ID / 4 Shift / 2 held-out OOD conditions per task across Object-OOD (7 tasks) and Configuration-OOD (8 tasks).
Evaluation Granularity10 trials × 10-point stage scoring with retry penalties.0–100 progress scores and pairwise preferences.Video-audited completion, declutter rate, and time efficiency.10 trials × 10-point subgoal scoring; success at ≥9/10; results stratified by ID/Shift/OOD.
Supported DiagnosticsVLA ranking and preliminary same-robot multi-task studies.Ranking and episode efficiency of pretrained policies.Task-level benchmarking across systems and control methods.Controlled studies of data scale, language granularity (L1–L3), single- vs. multi-task training, pretraining vs. training from scratch, and Real2Sim co-training.

We then provide two concrete examples of what ManipArena's evaluation reveals beyond aggregate success rates. 1) Subgoal scoring identifies where policies fail. On ordered button pressing, π0.5\pi_{0.5} (Unified), DreamZero, and Fast-WAM all record 0% success, with partial-credit scores of 31/100, 3/100, and 0/100, respectively. π0.5\pi_{0.5} reliably presses the first target before losing track of the sequence, while DreamZero and Fast-WAM struggle to locate the first target (Tables 9 and 18). 2) Stratifying by variation type reveals opposite effects of joint training. The π0.5\pi_{0.5} single-task specialists and the 15-task joint policy (Unified–L1) achieve nearly identical overall success rates (19.3% vs. 20.0%). With joint training, the mean partial-credit score rises from 37.0 to 53.1 on Configuration-OOD tasks and falls from 47.2 to 39.7 on Object-OOD tasks (Tables 4 and 9 and Appendix N, Table 19).

Thank you for the detailed comparison and the concrete diagnostic examples. This response substantially addresses my concern about ManipArena’s positioning relative to existing benchmarks. I consider my concern about benchmark differentiation largely addressed and maintain my weak-accept recommendation.

Reviewer_8x5SOpenReview ↗
Rating
4(+1)
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ManipArena, a real-robot benchmark for evaluating manipulation policies under controlled distribution shifts. It includes 20 tasks, 10,812 demonstrations, and approximately 188 robot hours across tabletop and mobile manipulation. The benchmark provides stratified evaluation, subtask-level scoring, and three levels of language annotations. The authors also compare different VLA and WAM policies and study the effects of data quantity, language granularity, and multi-task training.

Critical strengths
  1. I find the scale and design of the real-robot benchmark particularly valuable. The combination of diverse manipulation tasks, controlled distribution shifts, and partial-credit scoring provides a useful setting for evaluating generalization. The substantial real-robot data collection is also an important contribution.

  2. I appreciate the systematic experimental comparisons across different policy families and training settings. The authors' rebuttal further strengthens the paper by providing additional OOD breakdowns and a clearer explanation of the language-granularity results. These analyses make the benchmark's diagnostic value more convincing.

Critical weaknesses

My original concerns have been sufficiently addressed by the authors' responses. The additional breakdown across Object-OOD and Configuration-OOD tasks helps clarify generalization behavior, while the stage-level analysis explains the different performance trends of L2 and L3 annotations.

One minor limitation is that the real-robot evaluation uses relatively few trials per condition, which may affect the precision of individual comparisons. However, I understand the practical cost of physical experiments and do not consider this a major concern for the paper.

Overall, I appreciate the authors' detailed responses and the additional analyses. Both of my original concerns have been addressed, and I am now more confident in the contribution and experimental findings. I am happy to raise my rating from 3 to 4 (Clear Acceptance). I believe ManipArena is a useful contribution to real-world robot learning, and I look forward to seeing the additional analyses incorporated into the final version.

Questions for authors

None.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8x5S (1/2)

We thank Reviewer 8x5S for the constructive review and insightful questions. Below, we address W1 (Q1) and W2 (Q2) in detail.

W1 & Q1

We clarify the cross-condition performance trends and provide a diagnostic breakdown across generalization types as follows.

  1. Relative trends across evaluation conditions. Across policy families, relative rankings remain consistent from in-domain trials (T1–T4) to visual/configuration shifts (T5–T8) and held-out OOD trials (T9–T10), while absolute scores decrease progressively with shift severity. Because in-domain trials account for at most 40% success (4 of 10 trials) per task, task success rates above 40% directly reflect completions under shift and held-out conditions. Stronger policies such as WALL-OSS-0.5 and π0.5\pi_{0.5} (Unified–L3) achieve ≥50%\ge 50\% success on 6–8 of the 15 tasks (reaching 80%–100% on several tasks, including held-out OOD trials). In contrast, weaker baselines such as Fast-WAM earn partial credit on early subgoals but never reach 50% success on any task, indicating that their full completions remain largely confined to in-domain trials.

  2. Breakdown across generalization categories. Appendix N (Table 19) divides the 15 tabletop tasks into 7 Object-OOD tasks (appearance shifts in T5–T8 and unseen object categories in T9–T10) and 8 Configuration-OOD tasks (unseen spatial layouts and procedural orders). We summarize representative models below (Tables 5, 9, 10, and 20):

ModelOverall (/1500 / SR)Object-OOD (Mean / SR)Configuration-OOD (Mean / SR)
WALL-OSS-0.5 (Spec.)950.8 / 38.0%67.6 / 44.3%59.7 / 32.5%
WALL-WM (Unified)868.0 / 31.3%62.1 / 34.3%54.2 / 28.8%
WALL-OSS-0.5 (Unified)864.8 / 34.0%63.2 / 44.3%52.8 / 25.0%
π0.5\pi_{0.5} (Unified–L2)862.3 / 31.3%55.9 / 35.7%58.9 / 27.5%
π0.5\pi_{0.5} (Unified–L3)839.8 / 37.3%56.0 / 44.3%56.0 / 31.2%
π0.5\pi_{0.5} (Unified–L1)702.5 / 20.0%39.7 / 18.6%53.1 / 21.2%
π0.5\pi_{0.5} (Spec.)626.3 / 19.3%47.2 / 25.7%37.0 / 13.8%
DreamZero (Unified)500.3 / 11.3%37.1 / 11.4%30.0 / 11.2%
Fast-WAM (Unified)464.8 / 10.7%31.1 / 11.4%30.8 / 10.0%

This breakdown reveals two patterns obscured by aggregate totals: (i) For π0.5\pi_{0.5}, joint multi-task training (Unified–L1 vs. Spec.) raises mean partial-credit scores from 37.0 to 53.1 on Configuration-OOD tasks via shared spatial and sequential priors, while lowering scores from 47.2 to 39.7 on Object-OOD tasks. (ii) Under L1 training, WALL-OSS-0.5 (Unified) and π0.5\pi_{0.5} (Unified–L1) perform similarly on Configuration-OOD tasks (52.8 vs. 53.1), showing that their overall gap comes primarily from Object-OOD generalization (63.2 vs. 39.7)—a gap largely narrowed when π0.5\pi_{0.5} is trained with L2/L3 grounded annotations (raising Object-OOD mean scores to 55.9–56.0 and L3 success rate to 44.3%).

AuthorsOpenReview ↗

Response to Reviewer 8x5S (2/2)

W2 & Q2

We explain this difference through two stage-level effects observed across our task rubrics (Sec. 4.2, Table 3; Appendices F and L, Tables 10 and 18).

  1. L3 improves final-stage completion near the success threshold. All three π0.5\pi_{0.5} models in Sec. 4.2 are evaluated with identical concise L1 instructions at test time, differing only in training labels. Under our rubric, a trial counts as a success only at ≥9/10\ge 9/10 points, requiring completion of all core stages prior to arm retraction (Sec. 3.4). On multi-object semantic tasks (shape classification, item pairing) and precision alignment tasks (ring threading, spoon placement), L2 reliably completes early grasping and transport subgoals (scoring 6–7 points per trial, counted as partial credit but 0% success) yet often misses the final classification or insertion step. Training with L3's scene-level descriptions improves relational and spatial grounding at this final stage, pushing borderline trials past 9/10 (e.g., raising success rate from 30% to 80% on shape classification, and from 10% to 50% on ring threading with only an 8-point score gain from 63 to 71).

  2. L2 maintains higher stability on multi-step execution and contact tasks. Because test time provides only short L1 prompts, L3—trained on verbose scene descriptions—faces a larger train–test prompt gap on repetitive multi-step tasks than L2's compact action labels. Consequently, L3 is more likely to stall early or stop mid-sequence on long-horizon execution tasks (scoring 0 vs. 34 on drawer manipulation in Fig. 4, and 55 vs. 90 on block placement), whereas L2 also makes steadier early-stage progress (such as grasping and bimanual cable handoff) on hard contact tasks (water pouring and cable insertion) where both models record 0% success. These intermediate gains give L2 a +60-point lead on Execution tasks (529.75 vs. 469.75) and a higher overall score (862.25 vs. 839.75). Together, L2 favors multi-step execution stability while L3 improves final-stage completion, illustrating why pairing partial credit with success rate is essential for stage-level diagnosis.

Thank you for the detailed responses. I now have a better understanding of the generalization results across different OOD conditions, as well as why L2 achieves higher partial-credit scores while L3 performs better in final-stage completion. These explanations have addressed my original concerns and made the experimental findings clearer to me.

Reviewer_TCytOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ManipArena, an evaluation of a number of generalist policies on table-top and mobile manipulation tasks, with the evaluation comprising of 20 tasks, 10812 trajectories, and 188 robot hours. The authors aim to evaluate how well different policies are as generalists. To do so their evaluation scales a number of different axes that test generalization, like different objects, different tasks, different language instruction granularities, etc. The policies they evaluate include zero shot policies (from VLA/WAM families), as well as policies they fine-tune themselves on data they collected for each of the tasks. The per-task evaluations are designed to test generalization; specifically each task involves 10 evaluations per policy, with evals 1-4 being in-domain, evals 5-8 being controlled visual or configuration shifts, and evals 9-10 using held-out conditions. The authors fine-tune pi05 on their data, and that more data makes the policy worse. The authors in their evaluation then find that different policies perform well on different tasks, and the conclusions about which policy is better is very task dependent. The authors evaluate with three different levels of language instruction specificity (L1, L2, L3) and find that L3, the most specific language instructions, help the most/lead to the best policy performance. Finally the authors explore training policies from scratch (like pi05) to run evals in which different policies share the same pre-training, and they also create simulated versions of scenes (I think for simulated evaluation).

Critical strengths

(1) The paper claims to have run a large number of real world evals (over 10k trajectories) which is a comprehensive eval.

Critical weaknesses

(1) It is unclear what the actual contribution of the paper is. ManipArena, as the authors describe it, seems to be an evaluation of different policies the authors have performed, not a benchmark. If so, this contribution while interesting is limited, as simply running an evaluation would not be a sufficient contribution for ICLR. Is this a benchmark the authors plan to keep active?

(2) The paper is difficult to read, and I strongly suspect it is AI written. AI writing is not bad by itself, but the writing is extremely haphazard, with concepts mentioned before they are properly introduced, with experimental results described without any context or motivation, etc.

(3) The choice of experiments in the paper is highly questionable. The authors mix together several different things, like evaluating zero shot policies, fine-tuning policies with varying number of demos, testing simulation co-training, etc, each of which is poorly described and very haphazard, and together it becomes very unclear what the paper is even trying to do. If the paper was trying to propose a benchmark or at least evaluate various policies, why are the authors training policies at all? Can the authors clearly explain in a few sentences what the overall point of the paper is? Why are there fine-tuning experiments and pre-training from scratch experiments?

(4) Why is there a simulation component? And what is the simulation component exactly? It is not sufficiently explained in the main body of the text. The authors mention in the abstract/introduction that simulation evals poorly capture real-world phenomena like observation noise, delays, etc, which makes sense, but then it is unclear why the authors introduce a simulation component and what the point of this is.

(5) The paper, likely because it was AI written, itself points out a number of flaws in its own experiments, which I will repeat here. “with an initial single-policy evaluation rather than a cross-model mobile comparison” —-> in describing the mobile manipulation experiments, the authors mention that they only evaluate one model and not multiple. Isn’t this a flaw? Why don’t the authors evaluate across multiple models? Another example: after observing that with pi05 fine-tuning more demos seem to hurt performance, the authors mention “neither superiority nor the cause of the non-monotonic curve is established” —-> this is not a good thing. The authors should analyze why they got the results they did.

(6) The final conclusion (figure 5b) seems to be that models that are fine-tuned on the authors’ demos do the best. This is an expected conclusion. If the main result is that the authors have evaluated several models, there is not a clear takeaway that readers can learn about the performance of the various models. From the conclusion: “Initialization and training recipe comparisons expose task-dependent behavior under the evaluated settings, not universal data-scaling or task-sharing effects” —> this again reinforces that there seems to not be any takeaway.

Questions for authors

See weaknesses section for most of my questions. Some other ones below:



(1) The authors set up their evaluation environments with green screens and with pre-defined tasks. If the authors aim for this to be a benchmark that is actively maintained, how will the authors guard against overfitting to these scenes/tasks? Furthermore, how will the authors maintain reproducibility over time? It is in general very difficult to keep the scenes and objects exactly the same over long periods of time.

(2) Why are the trajectory counts in table 2 not round numbers? If a fixed set of policies is evaluated a certain number of times, one would expect it to be.

(3) There are many statements that simply do not make any sense because they are stated with no context. For example, “primary recipes use one training seed; seed 42 is documented for the data sweep. A separate four-task, three-seed check uses 20k updates.” —> could the authors clarify what they are trying to say here? This is just one example out of many.

(4) Table 4 is not sufficiently explained. Is WALL-OSS fine-tuned like pi05 is, or is it tested zero shot?

(5) Table 6 shows results for which there is a control over pre-training. So do the authors start from scratch weights? From paligemma weights? Even then different policy architectures like WALL-OSS and pi05 have different params and are not comparable. Also how is 702.5 possible out of 1500?

(6) “Transfer is task-dependent: simulated-data training improves press_button (31 to 65/100), whereas real-data training is strongest on pick_fruits (78 versus 51/100). Cotraining does not uniformly dominate either source.” —-> it is very confusing what the authors are studying. Are they studying pre-training, evaluation, sim co-training? Could you clarify?

(7) “the two-trial held-out band cannot give precise task-wise OOD estimates” —> what does this mean?

4 comments
AuthorsOpenReview ↗

Response to Reviewer TCyt (1/3)

W1

We clarify our contribution in terms of benchmark positioning and released assets for scene reproduction and extensibility.

  • ManipArena is a fully open-source, reproducible real-robot benchmark, not a one-off model evaluation report (Sec. 1, Sec. 3). While we do not host an online remote-access robot testing service, we open-source and maintain all resources needed to set up and run the benchmark so that other labs can faithfully reproduce the exact same evaluation scenes.

  • Specifically, we release: (1) Dataset and setup specifications: 10,812 teleoperated demonstrations with standardized configurations across 20 tasks in LeRobot v2.1 format, with three-level language annotations, joint velocities, and motor currents (Appendix I–J); (2) Extensible task schemas: formal definitions of OOD for controlled generalization evaluation; and (3) Paired simulation environments: reconstructed from our physical scenes with matching task definitions and data interfaces to support scalable evaluations (Sec. 3.5, Appendix J).

W2

We explain the overall paper structure to address the readability issue:

  • Paper organization: The paper follows a clear progression. Sec. 1–2 discuss the limitations of simulation-only and fragmented real-robot evaluations; Sec. 3 introduces our hardware platforms (3.1), task schemas (3.2), the 4/4/2 stratified test protocol (3.3), 10-point subgoal scoring (3.4), and multi-level language plus paired simulation components (3.5); Sec. 4 keeps the robot setup and scoring rules fixed and runs five controlled studies examining data quantity (4.1), language annotation detail (4.2), single- vs. multi-task training (4.3), policy architectures and pretraining (4.4), and simulated vs. real training data (4.5).

  • Manuscript revision: In the revised manuscript, we have rewritten the opening of Sec. 4 and the transitions of each subsection to clearly introduce the motivation and setup before presenting results.

  • AI Use Statement: As stated on Page 10 (AI Use Statement), AI tools were used only for language editing, LaTeX formatting, checking table arithmetic, and generating L3 scene descriptions (Sec. 3.5).

W3

We clarify our experimental design across the following four aspects:

  • No zero-shot evaluations were run: The mentioned zero-shot policies are not conducted in this paper. All eight VLA/WAM configurations are fine-tuned on the exact same ManipArena training split before evaluation (Sec. 4.4, Appendix B–C).

  • Why fine-tuning is necessary: In robot learning, existing VLA/WAM models are pretrained on different robot arms, camera layouts, and action spaces, and cannot directly adapt to our robot embodiment without fine-tuning (Appendix C). Fine-tuning on a shared downstream dataset is therefore necessary to compare these models fairly on the same hardware.

  • Why we include from-scratch experiments: Our main comparison (Table 5) fine-tunes each model from its official pretrained checkpoint as a system-level comparison. Table 6 adds from-scratch runs (initializing from base VLM weights without upstream robot pretraining) both to measure the gain from robot pretraining and to compare different model architectures on the same downstream data without private robot pretraining datasets.

W4

We explain the necessarity of the simulation component as follows:

  • Cost barrier of real-robot testing: Real-world evaluation is expensive—evaluating a single policy across 15 tabletop tasks (150 trials) takes ~15 robot-hours and 30 evaluator-hours (>2,000 physical trials conducted overall), and many research groups do not have access to identical dual-arm hardware.

  • Why there is no contradiction: Our simulation component consists of 1:1 paired Real2Sim environments reconstructed from our physical tabletop workspace, sharing identical task schemas, camera views, and model training data interfaces with the real robot. This provides scalable Real2Sim evaluation and synthetic data support, i.e., researchers without identical bimanual hardware can train policies on our real-robot demonstrations and run reproducible evaluations in paired simulation, while real-robot testing remains our final validation criterion.

  • Co-training and alignment validation: Sec. 4.5 (Table 14) fine-tunes π0.5\pi_{0.5} using real data, simulated data, and a 50% real+sim co-training mix, and tests all checkpoints on the real robot, confirming strong consistency between our reconstructed simulation and the real tasks (e.g., Sim-only matches Real-only on put_blocks at 85 vs. 83/100, and improves press_button from 31 to 65/100 on the real robot).

AuthorsOpenReview ↗

Response to Reviewer TCyt (2/3)

W5

We address both points by explaining hardware consistency constraints on the mobile track and our budget-control analysis of the non-monotonic π0.5\pi_{0.5} data curve:

  • Why only one model was tested on mobile manipulation: Stating an initial single-policy evaluation is an honest description of our scope rather than an inconsistency. After we collected the 2,872 mobile trajectories and finished testing π0.5\pi_{0.5}, the mobile bimanual robot used for data collection suffered a major hardware failure. Because replacing actuators and base parts alters the robot's physical behavior relative to the data-collection hardware, we stopped testing additional models to maintain strict hardware consistency. Importantly, (1) this does not affect any conclusions of our 15-task tabletop comparisons, and (2) we state this scope clearly while releasing all 2,872 62D mobile trajectories for community research (Sec. 3.1, Appendix E, I–J).

  • Analyzing the non-monotonic π0.5\pi_{0.5} data curve: Stating that the exact cause is not uniquely established is a statistical caveat for single-seed runs, not a lack of analysis. To investigate why Full-60k (277/500) scored lower than 200/task (312/500), we specifically ran the Full-60k vs. Full-120k control in Sec. 4.1 (Appendix G Table 11): doubling training updates lowered offline training loss (0.008163 →\rightarrow 0.006867) but reduced the real-robot score (277 →\rightarrow 248/500). This points to two clear factors: at a fixed 60k budget, using the full dataset reduces how many times each trajectory is sampled during training; when trained longer to 120k steps, the policy overfits offline demonstrations rather than improving real-world execution.

W6

We clarify the shared fine-tuning premise of Figure 5b and summarize four primary empirical takeaways:

  • Clarifying Figure 5b: Every model in Fig. 5b (and Table 5) is fine-tuned on the exact same ManipArena demonstrations—there are no unadapted zero-shot models in the plot (Sec. 4.4, Appendix B–C). Thus, Fig. 5b does not show the expected result that fine-tuning beats zero-shot; instead, it reveals a >2$\times$ real-robot performance gap (442 to 951/1500) across different VLA/WAM architectures and training recipes on identical data, with 36.6% headroom remaining.

  • Four concrete takeaways: (1) Multi-task training affects architectures in opposite ways: 15-task joint training improves π0.5\pi_{0.5} (+76.3) but hurts WALL-OSS-0.5 (-86.0; Table 4); (2) More data is not automatically better under fixed update budgets: 200 demos/task outperforms full-data training on both 5-task (312 vs. 277/500) and 15-task (853 vs. 727.3/1500) tests (Fig. 3, Table 12); (3) Training language detail shapes different skills: when tested with the same abstract L1 prompts, grounded action labels (L2) yield the highest execution score (529.75/1000), while adding scene descriptions (L3) doubles semantic task SR (28% →\rightarrow 56%; Table 3); and (4) Upstream robot pretraining is crucial: pretrained weights bring large within-architecture gains of +357.5 and +541.8 points (Table 6).

Q1

We address the raised concerns as follows:

  • Why green screens? As described in Sec. 3.1 and Appendix M, green screens and fixed studio lights remove uncontrolled background distractions (passersby, background clutter changes, daylight shifts) and allow fast, consistent setup reconstruction across time, relocations, or different labs (Fig. 2).

  • Preventing overfitting via stratified testing: Each task's 10 test trials follow a 4 + 4 + 2 split (Sec. 3.3, Appendix N Table 19). Only T1–T4 match the training distribution, while T5–T8 change object poses, sizes, or materials, and T9–T10 test unseen OOD objects (7 tasks) or held-out spatial/ordering combinations (8 tasks). Together with abstract L1-only test prompts (Sec. 4.2) and evaluating the final EMA checkpoint without selecting checkpoints on test scores (Appendix C), models cannot score well by memorizing training scenes.

  • Empirical consistency checks: Evaluating the same frozen π0.5\pi_{0.5} checkpoint across two separate 15-task (150-trial) batches (Appendix P.3, Table 22) gave 644.5/1500 and 702.5/1500, a difference of only 3.87 points per 100. Training and deploying three independent random seeds on four tasks (Appendix P.1, Table 21) also showed small SDs of 0.58–3.79 / 100.

AuthorsOpenReview ↗

Response to Reviewer TCyt (3/3)

Q2

As noted in Sec. 3 (Line 199: 20 tasks, 10,812 demonstrations, a stratified 10-trial evaluation protocol) and Table 2's caption (ManipArena task families and demonstration counts), the counts 5,157 / 2,783 / 2,872 in Table 2 are human-teleoperated training demonstrations (451–645 valid trajectories per task after removing interrupted takes; Appendix I Table 15), not policy evaluation trials. Policy evaluations strictly run round multiples of 10 trials per task (e.g., 150 trials per 15-task model and 1,200 trials in Table 5; Appendix D).

Q3

We clarify the context regarding physical evaluation cost, fixed-seed main comparisons, and the scope of our 20k-step multi-seed check: (1) evaluating one policy across 15 tasks requires 150 real-robot trials (~15 robot-hours + 30 evaluator-hours; Appendix D), making it infeasible to train and physically test multiple seeds for every 60k-step baseline; (2) our main 60k-step runs therefore fix seed 42 (Appendix C); and (3) to check sensitivity to random seeds, Appendix P.1 (Table 21) trained and evaluated three independent seeds (42, 43, 44) for 20k updates on four tasks, finding small SDs (0.58–3.79 / 100). That sentence in Sec. 4 simply clarifies that we do not use these 20k-step SDs as error bars for the 60k-step main experiments.

Q4

We confirm that all policies in Table 4 are fine-tuned and clarify the Unified vs. Specialist comparison setup as follows. As stated in Sec. 4.3 (Line 380), Table 5's caption (Spec. = task-specific fine-tuning; Unified = unified multi-task), Appendix B (Line 660: baselines are trained on the ManipArena demonstrations), and Appendix D Table 8 (Line 744: WALL-OSS-0.5 fine-tuning: 8$\times$A800, 24h), WALL-OSS-0.5 is fine-tuned on ManipArena just like π0.5\pi_{0.5}. Table 4 reports the score difference between 15-task joint fine-tuning (Unified) and single-task fine-tuning (Specialist) for each model.

Q5

We clarify from-scratch weight initialization and comparability alongside how fractional scores arise from our subgoal rubrics:

  • From-scratch weights and comparison: From-scratch starts from each model's base VLM weights (e.g., PaliGemma) with randomly initialized action modules and no upstream robot pretraining weights, trained directly on ManipArena. Table 6 primarily measures within-architecture gains from robot pretraining (π0.5\pi_{0.5}: +357.5; WALL-OSS-0.5: +541.8), while also providing a reference across architectures when private robot pretraining datasets are removed (Appendix O Table 20, alongside WALL-WM 868.0 and Fast-WAM 464.75).

  • Origin of 702.5 / 1500: Please see the scoring rubrics in Appendix L Table 18 (Page 21). Each trial is worth 10 points, and several multi-step tasks assign 0.5 or 0.25 points to subgoals—such as pick_items (4 cycles of 1 pt pick + 1.25 pts place, giving 67.5/100 for π0.5\pi_{0.5} Unified on Task 9 in Table 9), put_stationery (1.5 pts unzip/zip), and pair_up_items (2.5 pts per placement)—summing across 15 tasks to 702.5/1500.

Q6

Sec. 4.5 (Appendix H Table 14) studies downstream fine-tuning data mixtures: starting from the same pretrained π0.5\pi_{0.5} checkpoint, we fine-tune three models on the three paired tasks using (1) 100% real demos, (2) 100% paired simulated demos, and (3) a 50% real + 50% simulated co-training mix, and evaluate all three models on the real robot (10 trials per task).

Q7

We explain this statement by contrasting per-task SR resolution under two OOD trials with suite-level aggregation across all 15 tabletop tasks. Each task's 10 trials are split into trials 1–4 (in-domain), 5–8 (visual/layout shift), and 9–10 (2 held-out OOD trials) (Sec. 3.3, Eq. 2). For one task alone, 2 OOD trials can only yield a binary success rate of 0%, 50%, or 100%, which is too coarse for precise single-task OOD estimates (Sec. 3.3 Line 266; Appendix P.2 Line 1265). However, pooled across all 15 tabletop tasks, this held-out band contains 15×2=3015 \times 2 = 30 real-robot OOD trials (300 partial-credit points), providing a solid benchmark-level measure of OOD generalization.

I appreciate the authors' rebuttal response.

I read through the authors' statements in the rebuttal, and while one or two comments make sense, most of my originally pointed out weaknesses still hold, and with the major issues of very poor readability, unclear takeaways, and poor experimental design, I don't think this paper meets the bar for ICLR. As such I will keep my score as a 1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.