acceptodds
Under review as a conference paper at ICLR 2027

TOWER: Benchmarking Bimanual Manipulation across Simulation and Reality

Abstract

Complex robotic manipulation requires interpreting visual scenes and anticipating how actions change object contacts and structural support. Evaluating these capabilities calls for diverse, long-horizon tasks in which the physical consequences of earlier actions shape subsequent execution. To this end, we introduce TOWER, a novel Jenga-based benchmark for sustained contact and structural change, comprising six configurable task families involving bimanual coordination and tool use. Jenga provides repeatable structures in which local manipulations alter the support and stability needed for subsequent actions. A Real2Sim pipeline builds corresponding simulated environments for scalable evaluation. We evaluate four policy architectures across twelve configurations using separately trained real and simulated policies. Progress metrics reveal substantial partial completion hidden by binary task success, while taller configurations generally exhibit lower progress. Real–sim evaluation shows strong progress correlation (Pearson r = 0.880) and a normalized MMRV of 0.1217. TOWER provides a unified testbed for studying how policies respond to interaction-induced scene changes and supports systematic policy evaluation across reality and simulation. Code is available at https://anonymous.4open.science/r/artifact-3d0e01332a1edd93.

Primary area:
datasets and benchmarks
Keywords:
robotic manipulation, real-to-sim, contact-rich manipulation, bimanual coordination, long-horizon tasks, robot learning benchmarks, policy evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KbMPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a sim+real benchmark for bimanual manipulation using Jenga blocks. The real setup uses 4 RealSense D435if cameras and 2 NERO robot arms. The authors tested 4 architectures (OpenWAM, StarVLA-alpha, Fast-WAM, pi-0.5) on 6 task families (transfer, extract, stack, insert, etc.). Each task has two variants: 2 layers and 3 layers.

The stated contributions are: benchmark, real2sim pipeline, and metrics. The real and simulated data is highly correlated.

Critical strengths
  • Real2Sim and Sim2Real are important topics, and this paper demonstrates that the former is achievable on a handful of carefully designed tasks.
  • Benchmarks are important in robotics, and this paper proposes a new one.
  • The authors have taken care to ensure that the sim and real are highly correlated.
Critical weaknesses
  • It is not clear how the real data will benefit the community (as a benchmark). Except for its purpose in ensuring the accuracy of the sim data, the real data doesn't seem to be useful for benchmarking. As a result, this appears to be a simulation-only benchmark.
  • Sim and real policies are trained separately, so no sim2real transfer was explored.
  • No new algorithm is proposed, nor any substantive insight as to which methods are better or why.
  • All the videos in the supplemental show success.
  • Some of the metrics and analysis are a bit confusing and hard to follow.

Overall, I'm borderline on this paper. I lean barely toward acceptance, but only if the authors can provide a clear story for how this benchmark will benefit the research community.

Questions for authors
  • Is the simulation being proposed for the community to compare methods in the future? Will the simulation data be free for others to download, or will there be a way to upload results so the evaluation process is hidden from authors?
  • Is the real data being proposed as a benchmark? Or is it solely for the purpose of validating the sim data?
  • What insights can be gained from the benchmark as it exists? Are some methods better than others, and why?
  • What lessons were learned when creating the benchmark that will be useful for both extending the benchmark and for creating new unrelated benchmarks?
  • Please justify the denominator in Eq (8). All things being equal, higher success rate leads to higher ODR, which seems counter-intuitive since lower ODR is desirable.
  • Fig 6. Which model is this? And why are there 2 columns at 100%? What does the vertical axis represent? TSR=0 indicates that this is the mean over all episodes where the model failed to succeed? It's a bit confusing.
  • Fig 7. What are the numbers on the horizontal axis?
  • Fig 1A. Top-row is not ordered, but 2nd row appears to be an ordered sequence. But 2nd row images are too small to really know what is happening, and the text is not clear.
  • Fig 1C. What is matched failure mode?
  • L89 "planned sequence length, interaction dependencies, and the underlying structural demands" . I understand the 1st of these, but not the other 2. How are the structural demands for 3 layers more than those for 2 layers? Please explain.
  • L92 "normalized mean maximum rank violation (MMRV)" - needs citation here (SIMPLER, https://arxiv.org/pdf/2405.05941)
  • L314 use hyphen, not en-dash
  • What are reviewers supposed to get out of the link to the source code? Only one task is covered, no assets are provided, nor checkpoint weights, etc. It would be good to clarify what can be expected from the repo if the paper is accepted.
2 comments
AuthorsOpenReview ↗

Response to Reviewer KbMP

Thank you for the detailed review and for recognizing the importance of Real2Sim/Sim2Real and our care in keeping SIM and REAL correlated. We start with your main question.

How TOWER benefits the community

  1. A benchmark for sustained contact and structural change. Six task families, twelve configurations, 12,000 demonstrations and shared scoring of success, progress and disturbance. The released SIM environment, 3D assets, data and scorer let anyone compare methods.
  2. A visually aligned, physics-grounded Real2Sim pipeline. Reconstruction, camera alignment, robot-response fitting and force-gauge contact measurement, reusable for other contact-rich setups.
  3. A paired real-world and simulation study. SIM tracks hardware (partial r = 0.78 after removing length; MMRV 0.1217).
  4. New: simulator-generated data transfers. Scripted controllers generated all 6,000 SIM demonstrations; OpenWAM trained only on SIM data reaches 92.5% LHPR zero-shot on the real robot (A-2L).

Weaknesses

W1. Value of the real data

Beyond validating SIM, the real data does not seem useful for benchmarking.

REAL is also a hardware benchmark: its protocols and reference scores let labs with a bimanual platform and Jenga blocks compare on hardware. REAL-trained policies can then be run cheaply in SIM: OpenWAM reaches 100% / 88.3% LHPR on A-2L / A-3L, matching SIM-trained checkpoints.

W2. Sim2real transfer

Policies are trained separately, so no sim2real transfer was explored.

We have now tested transfer in both directions (point 4, W1; full metrics in the overall response). Separate training was deliberate: it isolates whether SIM evaluates policies as REAL does.

W3. Contribution and insights

No new algorithm, nor insight into which methods are better or why.

TOWER is a benchmark contribution, and its results answer this (overall response, point 2):

  • World-action models lead on hardware. Both WAMs, which are also trained to predict future video, beat both VLAs in REAL LHPR (85.3% / 77.5% vs. 71.1% / 52.0%).
  • Which pretraining matters. Robot-action pretraining alone is not enough (π0.5 is last in LHPR and ODR in both domains), but OpenWAM's embodied pretraining matters: without it, a single multi-task OpenWAM for all 12 settings, trained with the same recipe, falls from 32.5% to 10.0% TSR in SIM (p ≈ 10⁻⁷).

W4. Success-only videos

All supplementary videos show success.

They are training demonstrations, successful by construction. Our project page will add policy rollouts, including failures; Figure 5 and Appendix H already analyze failures.

W5. Clarity

Some metrics and analysis are hard to follow.

We revised Figures 1, 4, 6 and 7 and the height explanation (Q6–Q10).

Questions

Q1. Community use. Yes. The SIM environment, 3D assets, scorer and SIM data will be freely downloadable, and we will host a hidden test split with an evaluation server.

Q2. REAL as a benchmark. Both: it validates SIM and is a hardware benchmark (W1).

Q3. Methods. Yes; see W3.

Q4. Lessons. Compose tasks from reusable actions; build the simulator from measurements; keep a calibration only if held-out error improves; pair SIM and REAL from the start; score intermediate progress.

Q5. ODR denominator. When success comes from completing a failed operation, C rises by one and (1−s)(1-s) falls by one, so E=C+(1−s)E=C+(1-s) is unchanged. Only a terminal-state failure (all operations done, final check failed) loses its extra counted step on success, which slightly raises ODR. Removing the failure term also leaves model rankings unchanged in both domains (Appendix G.3, Table 16).

Q6. Figure 6. All four models and both domains. TSR = 0/1 marks settings whose ten episodes all failed/succeeded; both 100% columns are TSR = 1, and vertical position only separated overlapping points. It is now a histogram colored by policy.

Q7. Figure 7. Planned operations at 2→3 layers (e.g., 4→6); axis title and legend added.

Q8. Figure 1A. The top row is now labeled an unordered action vocabulary; the middle row shows four enlarged, numbered Task F moments.

Q9. Figure 1C. The same type of structural failure in REAL and SIM, now titled “Similar structural failures”.

Q10. L89. A third layer depends on more earlier actions: it rests on earlier placements or is exposed by earlier pushes and extractions, so early errors propagate further. Length alone does not explain the drop: E grows by 50% yet loses 33.5 points, while C and D double yet lose 3–7.

Q11–Q12. SIMPLER is now cited; hyphens fixed.

Q13. Repository. It shows the simulator interfaces, policy adapters and the scored Task B runner. On acceptance, the release adds SIM environments and 3D assets for all six tasks and both heights, the scorer, SIM demonstrations, SIM-trained OpenWAM checkpoints and evaluation scripts.

Thanks to the authors for satisfactorily answering all my questions, and for revising the manuscript.

Reviewer_CjTROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TOWER, a Jenga-based benchmark for evaluating long-horizon bimanual manipulation involving contact-rich interactions and structural changes. It consists of 6 task families across 12 configurations, covering block transfer, extraction, insertion, pushing, and tool use. The authors construct a corresponding simulation environment through Real2Sim reconstruction and physics calibration, and evaluate four policy architectures trained separately in simulation and reality. Experiments demonstrate that operation-level progress metrics capture partial completion beyond binary success, and report strong sim-real performance correlation.

Critical strengths
  • Valid and meaningful problem formulation. Evaluating manipulation where earlier actions change object contacts, structural stability, and the feasibility of subsequent actions is really important for long-horizon robotic manipulation.

  • Substantial engineering and data collection effort. The benchmark includes 12,000 demonstrations, 960 evaluation episodes, and a physical bimanual platform. The Real2Sim pipeline incorporates careful camera, robot dynamics, and friction calibration.

  • Well-documented evaluation protocol. The authors provide detailed task definitions, operation-level metrics, annotation procedures, and extensive results in the appendix. Measuring intermediate progress and non-target disturbance is useful beyond binary task success.

Critical weaknesses
  • Highly structured and limited task diversity and generalization. All six tasks involve the same rigid wooden blocks in a controlled environment with predefined action sequences. Several tasks are essentially repeated pick-and-place operations. It is unclear whether this benchmark captures the challenges of more complex, unstructured manipulation or offers insights beyond this specific setting.

  • Insufficient evidence for the claimed manipulation capabilities. The paper emphasizes adaptation to structural changes, but does not experimentally isolate this capability. The observation that performance decreases with longer action sequences is unsurprising and does not necessarily demonstrate contact reasoning or adaptation.

  • Limited validity of sim-real correspondence claims. The tasks here only require rigid blocks and very limited generalization to visual disturbances or physics dynamics change, hence despite the reported Pearson correlation is at a high 0.880, it's not validating the proposed benchmark or real2sim pipeline can be used to evaluate more complex tasks. It would be more convincing to show strong results that establish reliable simulation-based prediction of real-world policy performance.

  • Some questionable evaluation choices. Only ten trials are conducted per policy-task configuration. The disturbance metric uses a nonstandard denominator based on successful operations rather than actual attempts, potentially confounding disturbance with task progress. Inter-annotator agreement is evaluated only on the simplest task.

Questions for authors
  1. Does performance on TOWER tasks correlates with performance on more complex, non-block manipulation tasks?

  2. Evaluation metrics: Why is ODR normalized by successful operations rather than attempted operations? How robust are the results to alternative disturbance metrics?

  3. I think the paper title is making an over-claim: given there is only one class of highly structured tasks, the title should not claim it's benchmarking "Bimanual Manipulation" in general?

2 comments
AuthorsOpenReview ↗

Response to Reviewer CjTR

Thank you for the constructive review, and for finding the problem formulation valid and important for long-horizon manipulation and the evaluation protocol well documented. We address each point below.

Weaknesses

W1. Task diversity and controlled structure

Are these mostly repetitive pick-and-place tasks with limited relevance?

TOWER targets the capability you identified as important: manipulation where earlier actions change contacts, support and the feasibility of later actions. Only Tasks A and B are transfer tasks, and in B every placement becomes the support for later blocks. Tasks C–F require extraction, insertion, pushing and tool use. Standardized blocks are deliberate: they make configurations reproducible, enable a measured Real2Sim counterpart and operation-level scoring, and separate structural reasoning from object recognition. The failure modes TOWER exposes (jammed insertion, push-induced instability, extraction disturbing neighbors; Figure 5) are generic contact phenomena.

W2. Evidence for structural adaptation

Longer sequences alone do not demonstrate contact reasoning or adaptation.

Performance in TOWER is not explained by length alone. At equal planned length, structural tasks are much harder. With 4 operations, transfer (A-2L) reaches 90.9% LHPR versus 54.1% for insertion (D-3L); with 6, 92.3% (A-3L) versus 71.5% for pushing (E-2L); with 8, 95.3% (B-2L) versus 76.9% for tool use (F-2L). Height shows the same pattern: Task A becomes longer but not harder (90.9% → 92.3%), while E, F and B, where each action changes the support for later actions, drop 27–34 points (Section 4.4). These matched-length contrasts separate structural demand from sequence length.

W3. Simulation-based prediction of real performance

Does the correspondence support evaluating policies beyond this restricted setting?

We now provide two kinds of direct evidence.

  • Prediction. After removing length, SIM progress still predicts REAL progress (partial r = 0.78, 95% CI [0.53, 0.90]); configuration difficulty (r = 0.89) and within-configuration rankings (MMRV 0.1217) also agree.
  • Transfer. A SIM-trained OpenWAM checkpoint reaches 92.5% / 85.0% LHPR zero-shot on hardware (A-2L / A-3L); a REAL-trained checkpoint evaluated in SIM reaches 100% / 88.3%, matching SIM-trained checkpoints.

Rigid blocks do not make this easy: millimeter-scale contact errors cause collapse or jamming. The pipeline is built from measurements with held-out checks (Appendix E), not task-specific tuning, so the procedure is not specific to Jenga blocks (Q1).

W4. Trial count, ODR denominator and annotation coverage

Are ten trials, the ODR denominator and a Task A-only audit adequate?

Ten trials per setting cover 4 policies × 12 configurations × 2 domains (960 human-scored episodes) despite manual resets; the bootstrap CIs above quantify uncertainty. ODR is addressed in Q2. Agreement now covers all six tasks: task success agrees in 98.75% of episodes in both domains (≥ 97.5% on every task), and disturbance totals agree within two occurrences in 95.00% / 94.58% (REAL / SIM).

Questions

Q1. Correlation with non-block manipulation

Do TOWER scores correlate with more complex, non-block tasks?

TOWER is a controlled diagnostic, not a proxy for every manipulation task. Its value is to isolate structural dependence with reproducible configurations, paired SIM/REAL evaluation and operation-level scores, which unstructured benchmarks cannot provide. Extending the pipeline to other objects is future work.

Q2. ODR normalization and robustness

Why use this denominator instead of attempts, and do alternative metrics change the conclusions?

The denominator is not successes alone: each failed episode adds one counted step, E=C+(1−s)E=C+(1-s). Retries are not counted, because realignment and re-contact cannot be reliably split into attempts from video. Per-step normalization matters: in REAL, π0.5 and StarVLA-α both cause 277 disturbances, which per-episode counts cannot separate; ODR can (0.556 vs. 0.428, over 498 vs. 647 counted steps). Rankings are also robust to removing the failure term: dividing by successful operations only leaves model-level rankings unchanged in both SIM and REAL (Appendix G.3, Table 16).

Q3. Title and scope

Does the title overstate general bimanual-manipulation coverage?

TOWER's scope is the physical consequence of manipulation: each action changes the support and contact that later actions depend on. The abstract and introduction state this scope where TOWER is first introduced (“Jenga-based benchmark for sustained contact and structural change”); the title names the capability evaluated within it. The six task families combine the core atomic actions of bimanual manipulation (grasp, place, push, extract, insert, adjust) with tool use and arm-specific roles; e.g., in Task F one arm uses a tool to expose blocks that the other extracts and places.

Thank you to the authors for the detailed follow-up. I have no further questions an decided to maintain the current recommendation.

Reviewer_jEWQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TOWER provides six Jenga-based task families across twelve two-/three-layer configurations, with matched real and simulated environments. The benchmark includes transfer, extraction, insertion, pushing, and tool use, scored by task success, operation progress, and non-target disturbance. Four policy architectures are trained separately in each domain using 12,000 demonstrations across both domains and evaluated over 960 episodes. A Real2Sim pipeline aligns scene geometry, cameras, friction, and robot response. The reported cross-domain raw progress correlation is 0.880, with normalized MMRV of 0.1217.

Critical strengths
  1. The benchmark targets a useful combination of constrained contact, changing support relationships, and bimanual tool use. Extraction alters the structure available for later actions, while Task F requires tool preparation followed by coordinated extraction and placement. The task suite, paired environments, and balanced evaluation make a substantial experimental resource. Progress scoring also captures differences hidden by binary success: Fast-WAM on REAL D鈥?3L completes 77.5% of planned operations despite no successful episodes (Section 4.3).

  2. Simulator construction is supported by measured friction and held-out checks of camera alignment and robot response. Appendix E reports validation errors and records cases where optimization worsens the nominal calibration. The physical measurements and validation protocol give the simulated counterpart a firmer basis than visual matching alone.

Critical weaknesses
  1. Cross-domain agreement is less consistent than the aggregate correlation alone suggests. Figure 4 pools tasks with two to sixteen planned operations, so shared sequence length can contribute to raw-count correlation. Normalizing the rounded means in Tables 16鈥?17 by planned length gives an approximate Pearson correlation of 0.67; Appendix G.3 defines this statistic but does not report its value. There are also large ranking reversals: on E鈥?3L, OpenWAM goes from 32.2% LHPR in SIM to 92.2% in REAL, while StarVLA goes from 57.8% to 25.6% (Table 15). The reported normalized MMRV partly captures this disagreement, but uncertainty estimates for correspondence and rankings are missing. With ten episodes per setting, separately trained policies, and different demonstration sources in the two domains, the stability of these comparisons remains unclear. The results characterize the evaluated pipelines; they do not isolate how much simulator calibration contributes to their agreement.

  2. The annotation audit covers an easier subset of the scoring problem. All results use human labels, with double annotation and adjudication, but the agreement statistics cover only 80 Task A episodes and omit disturbance counts (Appendix B.2, Table 7). Insertion completion, prescribed push displacement, and graspable protrusion in Tasks D鈥揊 are judged from video without quantitative pose thresholds in Table 5. Separately, disturbance scoring requires detecting non-target block-center movement greater than 10 mm. The Task A audit leaves the consistency of these contact-sensitive judgments and ODR measurements unverified.

Questions for authors
  1. Please report the normalized agreement statistics defined in Appendix G.3 and uncertainty on the cross-domain comparisons. Which training budgets and checkpoint-selection rules were matched? Are repeat-run results available to assess reversals such as E鈥?3L?

  2. Is there pre-adjudication agreement evidence for Tasks C鈥揊 and disturbance counts? In particular, how are the 10 mm non-target displacement threshold and the insertion, push, and tool-operation completion criteria applied consistently from video?

2 comments
AuthorsOpenReview ↗

Response to Reviewer jEWQ

Thank you for the careful review, and for recognizing the task suite and paired environments as a substantial experimental resource and the measured simulator as better grounded than visual matching. We address each point below.

Weaknesses

W1. Correspondence beyond shared length, reversals and separate training

Shared plan length, separate training and ranking reversals may weaken the aggregate agreement.

Correspondence holds after removing length. Controlling for planned length, the partial correlation between SIM and REAL progress is r = 0.78 (95% bootstrap CI [0.53, 0.90]). Length alone predicts REAL progress with r = 0.69, so SIM carries substantial information beyond length. SIM and REAL also agree on which configurations are hard (configuration-level normalized r = 0.89, CI [0.74, 0.96]).

Normalized MMRV (0.1217) compares policies within each configuration, so shared length cannot inflate it. E–3L is the largest single reversal. TOWER reports it at the episode level rather than averaging it away; exposing such cases is the purpose of paired evaluation.

On separate training, we added direct transfer tests. OpenWAM trained on teleoperated REAL demonstrations and evaluated in SIM reaches 100% / 88.3% LHPR on A-2L / A-3L, matching checkpoints trained on scripted SIM demonstrations (100% / 86.7%). A SIM-trained checkpoint reaches 92.5% / 85.0% on hardware. On Task A, policies learned from real demonstrations run in the calibrated simulator at the level of SIM-trained policies, complementing the held-out calibration errors in Appendix E.

W2. Coverage of the annotation audit

Task A alone does not establish reliability for contact-sensitive tasks or disturbances.

We extended the audit to all 960 episodes (80 per task and domain). Task-success agreement is 98.75% in both domains and at least 97.5% on every task, including the contact-sensitive Tasks D–F. Exact LHPR agreement is 90.63% / 91.88% (REAL / SIM), and disturbance totals agree within two occurrences in 95.00% / 94.58% of episodes. Appendix B.2 gives the per-task table.

Questions

Q1. Normalized statistics, training budgets and repeat runs

What are the normalized agreement and uncertainty, matched budgets, selection rules and repeat-run results?

Over the 48 policy–configuration pairs, length-normalized progress gives r = 0.67 (95% CI [0.45, 0.82]) and Lin's concordance 0.63 (defined in Appendix G.4; the values will be added there in the final version). Normalization caps progress at 100% and saturates on the easier configurations, which compresses the scale; we therefore also report the length-controlled partial correlation (0.78).

Each architecture uses one default training recipe, shared by both domains and trained on the same 500 demonstrations per configuration (Appendix G.2, Table 15). Retraining all 96 policy–configuration–domain checkpoints several times is not feasible within the rebuttal period, so we quantify uncertainty by bootstrap (CIs above). The transfer experiments also add independently trained OpenWAM checkpoints: REAL-trained OpenWAM evaluated in SIM reproduces the SIM-trained A-3L result (88.3% vs. 86.7% LHPR).

Q2. Independent ratings and video-based criteria

Is there pre-adjudication evidence for C–F and disturbances, and how are the thresholds applied?

Yes. Agreement is computed on each annotator's original rating, before adjudication; a third evaluator then resolves disagreements against the video (Appendix B.2). The criteria (Appendix B.1) are applied identically in both domains. An insertion counts when the block stays in its prescribed slot for at least 10 s after release. A push counts when the prescribed displacement and end state are reached, and tool operations require a graspable protrusion and stable placement. A disturbance is a non-target block whose center moves more than 10 mm during an operation, counted once per operation, excluding the target and the foundation. Because motion near 10 mm is hard to resolve from video, we report disturbance agreement within two occurrences per episode; ODR uses the adjudicated counts.

Thank you for the detailed response, which addresses most of my previous concerns. I maintain my score.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Overall Response to All Reviewers

We thank all reviewers for their careful and constructive reviews. Reviewers found the problem formulation “valid and meaningful” and important for long-horizon manipulation (CjTR), the task suite and paired environments “a substantial experimental resource” (jEWQ), the measured simulator better grounded “than visual matching alone” (jEWQ), and the sim–real correspondence carefully established (KbMP). The rebuttal adds four results that strengthen these points.

1. Direct sim↔real transfer (KbMP, CjTR, jEWQ)

A single multi-task OpenWAM checkpoint trained only in SIM reaches 92.5% LHPR on the real robot zero-shot (A-2L), above the REAL-trained checkpoint (75.0%). OpenWAM trained on REAL demonstrations and evaluated in SIM matches SIM-trained performance (100% / 88.3% vs. 100% / 86.7% LHPR, 2L / 3L). TOWER therefore supports both training for real deployment and evaluating real-trained policies in SIM. Values are % except ODR; bold marks the better value within each task.

Real → Sim

TaskTrain → EvalTSR ↑PSR ↑GSR ↑LHPR ↑ODR ↓
A-2LSIM → SIM100.0100.0100.0100.00.00
A-2LREAL → SIM100.0100.0100.0100.00.00
A-3LSIM → SIM70.080.093.386.70.13
A-3LREAL → SIM80.086.790.088.30.05

Sim → Real

TaskTrain → EvalTSR ↑PSR ↑GSR ↑LHPR ↑ODR ↓
A-2LREAL → REAL0.050.0100.075.00.00
A-2LSIM → REAL90.090.095.092.50.05
A-3LREAL → REAL90.090.093.391.70.11
A-3LSIM → REAL70.080.090.085.00.41

2. What the results show about methods (KbMP)

  • World-action models lead on hardware. Both world-action models (WAMs), which are also trained to predict future video, rank first and second in REAL TSR and LHPR and beat both VLAs on four of six tasks (B, C, D, F). Paper results, 120 episodes per policy and domain:
PolicyVideo-prediction trainingRobot dataREAL TSR ↑REAL LHPR ↑REAL ODR ↓SIM TSR ↑SIM LHPR ↑SIM ODR ↓
π0.5–✓35.852.00.55626.741.40.851
StarVLA-α––40.871.10.42845.075.20.369
Fast-WAM✓–56.777.50.49323.359.50.502
OpenWAM✓✓53.385.30.29645.078.10.386
  • Which pretraining matters. Robot-action pretraining alone is not enough: π0.5 has it, yet has the lowest LHPR and highest ODR in both domains (table above). OpenWAM's embodied pretraining does matter. A new SIM experiment trains a single multi-task OpenWAM for all 12 settings twice, with the same data, recipe and steps, changing only the initialization:
Multi-task OpenWAM, initialized fromTSR ↑GSR ↑PSR ↑LHPR ↑ODR ↓
OpenWAM-Alpha32.565.345.656.51.002
Wan2.2 only10.031.313.923.61.779

Values are % except ODR; GSR, PSR, LHPR and ODR are ratios of summed counts.

On the same 120 scenes, 28 are solved only with OpenWAM-Alpha and 1 only with Wan2.2 (exact McNemar test, p ≈ 10⁻⁷).

3. Length-controlled correspondence with uncertainty (jEWQ, CjTR)

Controlling for planned length, the SIM–REAL partial correlation is r = 0.78 (95% bootstrap CI [0.53, 0.90]), while length alone predicts REAL progress with r = 0.69. SIM and REAL also agree on which configurations are hard (configuration-level normalized r = 0.89, CI [0.74, 0.96]), and normalized MMRV (0.1217) compares policies within each configuration, where length is shared.

4. Annotation reliability on all tasks (jEWQ, CjTR)

Agreement now covers all 960 episodes across Tasks A–F. Task-success agreement is 98.75% in both domains and at least 97.5% on every task; exact LHPR agreement is 90.63% / 91.88% (REAL / SIM), and disturbance totals agree within two occurrences in 95.00% / 94.58% of episodes (Appendix B.2).

Figures 1, 4, 6 and 7, the introduction, Section 4 and Appendices B.2, G.2–G.5 and I.1 are revised in the PDF (substantive changes in blue). Results 1–3 are new for this rebuttal and will be added to the final version.

AuthorsOpenReview ↗

Change Log of the Revised PDF

We thank all reviewers for their suggestions. We have revised the manuscript accordingly. Here is the change log:

  • As suggested by Reviewers jEWQ and CjTR, we extended the annotation-agreement analysis to all 960 episodes across Tasks A–F, including disturbance counts (Appendix B.2).
  • As suggested by Reviewer jEWQ, we clarified the correlation and MMRV reporting (Section 4.2; Appendix G.4).
  • As suggested by Reviewer jEWQ, we added the default training configuration of every baseline, shared by REAL and SIM (Appendix G.2, Table 15).
  • As suggested by Reviewer KbMP, we redrew Figure 6 and added the axis title, layer legend and caption explanation to Figure 7.
  • As suggested by Reviewer KbMP, we enlarged and numbered the Task F moments in Figure 1A and renamed Figure 1C to “Similar structural failures”.
  • As suggested by Reviewer KbMP, we explained in the introduction why taller towers add dependent operations.
  • As suggested by Reviewer KbMP, we cited SIMPLER for MMRV and corrected hyphenation throughout.
  • As suggested by Reviewers CjTR and KbMP, we added an ODR sensitivity analysis: dividing disturbances by successful operations only, without the failure term, leaves the model-level rankings unchanged in both domains (Appendix G.3, Table 16).
  • As suggested by Reviewer KbMP, we listed the release contents, the hidden test split and the failure rollouts (Appendix I.1).
  • We also split Figure 4 into per-policy panels, added task-wise ODR (Section 4.2, Table 3; later tables are renumbered, e.g. the agreement table is now Table 8 and the complete metrics Tables 17–18), and clarified ODR pooling and G/P notation (Sections 3.3 and 4.2; Appendix G). New and substantively revised text and captions are marked in blue.

If the authors have addressed your concerns or provided clarifications, please join the discussion to indicate whether your concerns have been resolved or if you have any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.