- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a sim+real benchmark for bimanual manipulation using Jenga blocks. The real setup uses 4 RealSense D435if cameras and 2 NERO robot arms. The authors tested 4 architectures (OpenWAM, StarVLA-alpha, Fast-WAM, pi-0.5) on 6 task families (transfer, extract, stack, insert, etc.). Each task has two variants: 2 layers and 3 layers.
The stated contributions are: benchmark, real2sim pipeline, and metrics. The real and simulated data is highly correlated.
Critical strengths
- Real2Sim and Sim2Real are important topics, and this paper demonstrates that the former is achievable on a handful of carefully designed tasks.
- Benchmarks are important in robotics, and this paper proposes a new one.
- The authors have taken care to ensure that the sim and real are highly correlated.
Critical weaknesses
- It is not clear how the real data will benefit the community (as a benchmark). Except for its purpose in ensuring the accuracy of the sim data, the real data doesn't seem to be useful for benchmarking. As a result, this appears to be a simulation-only benchmark.
- Sim and real policies are trained separately, so no sim2real transfer was explored.
- No new algorithm is proposed, nor any substantive insight as to which methods are better or why.
- All the videos in the supplemental show success.
- Some of the metrics and analysis are a bit confusing and hard to follow.
Overall, I'm borderline on this paper. I lean barely toward acceptance, but only if the authors can provide a clear story for how this benchmark will benefit the research community.
Questions for authors
- Is the simulation being proposed for the community to compare methods in the future? Will the simulation data be free for others to download, or will there be a way to upload results so the evaluation process is hidden from authors?
- Is the real data being proposed as a benchmark? Or is it solely for the purpose of validating the sim data?
- What insights can be gained from the benchmark as it exists? Are some methods better than others, and why?
- What lessons were learned when creating the benchmark that will be useful for both extending the benchmark and for creating new unrelated benchmarks?
- Please justify the denominator in Eq (8). All things being equal, higher success rate leads to higher ODR, which seems counter-intuitive since lower ODR is desirable.
- Fig 6. Which model is this? And why are there 2 columns at 100%? What does the vertical axis represent? TSR=0 indicates that this is the mean over all episodes where the model failed to succeed? It's a bit confusing.
- Fig 7. What are the numbers on the horizontal axis?
- Fig 1A. Top-row is not ordered, but 2nd row appears to be an ordered sequence. But 2nd row images are too small to really know what is happening, and the text is not clear.
- Fig 1C. What is matched failure mode?
- L89 "planned sequence length, interaction dependencies, and the underlying structural demands" . I understand the 1st of these, but not the other 2. How are the structural demands for 3 layers more than those for 2 layers? Please explain.
- L92 "normalized mean maximum rank violation (MMRV)" - needs citation here (SIMPLER, https://arxiv.org/pdf/2405.05941)
- L314 use hyphen, not en-dash
- What are reviewers supposed to get out of the link to the source code? Only one task is covered, no assets are provided, nor checkpoint weights, etc. It would be good to clarify what can be expected from the repo if the paper is accepted.
2 comments
Thanks to the authors for satisfactorily answering all my questions, and for revising the manuscript.
Response to Reviewer KbMP
Thank you for the detailed review and for recognizing the importance of Real2Sim/Sim2Real and our care in keeping SIM and REAL correlated. We start with your main question.
How TOWER benefits the community
Weaknesses
W1. Value of the real data
REAL is also a hardware benchmark: its protocols and reference scores let labs with a bimanual platform and Jenga blocks compare on hardware. REAL-trained policies can then be run cheaply in SIM: OpenWAM reaches 100% / 88.3% LHPR on A-2L / A-3L, matching SIM-trained checkpoints.
W2. Sim2real transfer
We have now tested transfer in both directions (point 4, W1; full metrics in the overall response). Separate training was deliberate: it isolates whether SIM evaluates policies as REAL does.
W3. Contribution and insights
TOWER is a benchmark contribution, and its results answer this (overall response, point 2):
W4. Success-only videos
They are training demonstrations, successful by construction. Our project page will add policy rollouts, including failures; Figure 5 and Appendix H already analyze failures.
W5. Clarity
We revised Figures 1, 4, 6 and 7 and the height explanation (Q6–Q10).
Questions
Q1. Community use. Yes. The SIM environment, 3D assets, scorer and SIM data will be freely downloadable, and we will host a hidden test split with an evaluation server.
Q2. REAL as a benchmark. Both: it validates SIM and is a hardware benchmark (W1).
Q3. Methods. Yes; see W3.
Q4. Lessons. Compose tasks from reusable actions; build the simulator from measurements; keep a calibration only if held-out error improves; pair SIM and REAL from the start; score intermediate progress.
Q5. ODR denominator. When success comes from completing a failed operation, C rises by one and (1−s) falls by one, so E=C+(1−s) is unchanged. Only a terminal-state failure (all operations done, final check failed) loses its extra counted step on success, which slightly raises ODR. Removing the failure term also leaves model rankings unchanged in both domains (Appendix G.3, Table 16).
Q6. Figure 6. All four models and both domains. TSR = 0/1 marks settings whose ten episodes all failed/succeeded; both 100% columns are TSR = 1, and vertical position only separated overlapping points. It is now a histogram colored by policy.
Q7. Figure 7. Planned operations at 2→3 layers (e.g., 4→6); axis title and legend added.
Q8. Figure 1A. The top row is now labeled an unordered action vocabulary; the middle row shows four enlarged, numbered Task F moments.
Q9. Figure 1C. The same type of structural failure in REAL and SIM, now titled “Similar structural failures”.
Q10. L89. A third layer depends on more earlier actions: it rests on earlier placements or is exposed by earlier pushes and extractions, so early errors propagate further. Length alone does not explain the drop: E grows by 50% yet loses 33.5 points, while C and D double yet lose 3–7.
Q11–Q12. SIMPLER is now cited; hyphens fixed.
Q13. Repository. It shows the simulator interfaces, policy adapters and the scored Task B runner. On acceptance, the release adds SIM environments and 3D assets for all six tasks and both heights, the scorer, SIM demonstrations, SIM-trained OpenWAM checkpoints and evaluation scripts.