- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
BREE introduces a roughly $624 bimanual platform with standardized hardware, calibration, printable task assets, and eight manipulation tasks. Experiments evaluate five policies and investigate policy transfer and demonstration pooling across two independently assembled instances.
Critical strengths
Accessible and comprehensive infrastructure. BREE standardizes the complete experimental setup, including sensing, lighting, calibration, and task assets. This addresses practical barriers to reproducing real-world manipulation experiments at low cost.
Useful benchmark design. Eight tasks cover different coordination patterns, with explicit evaluation protocols and stage-level scoring. Evaluating five policy families demonstrates the benchmark’s ability to distinguish capabilities and diagnose failures.
Critical weaknesses
Cross-instance reproducibility is only partially demonstrated. Evaluation involves two instances, and substantial performance differences remain: SmolVLA’s Plate Loading success falls from 95% to 65%, while RDT’s Cylinder Insertion success falls from 40% to 20% (Table 8). Preserved intermediate-stage performance does not establish reproducibility of complete-task outcomes. The four inference seeds also do not measure variability across independently trained policies. Evidence for multi-instance scaling is insufficient. Pooling experiments primarily concern one task and do not demonstrate a consistent advantage at matched data budgets. In Table 5, 25 demonstrations from each instance achieve 55% on instance A, versus 65% using 50 demonstrations from A alone. Adding five target-instance demonstrations on another task demonstrates useful adaptation, but does not isolate a general scaling benefit.
Questions for authors
What are the quantitative calibration errors and their variability across repeated assembly/calibration? Were the two instances built by independent operators, and how variable are cross-instance performance differences across independent training seeds?
4 comments
Response to Reviewer FyjT
Q1: Lack of quantitative calibration error and repeatability measurements.
Response: We conducted two additional experiments to quantitatively evaluate BREE's calibration accuracy and repeatability: cross-instance visual geometric consistency across workspace configurations and calibration repeatability under repeated robot-arm reassembly.
Cross-instance visual alignment achieves an overall error of 18.20 mm-equivalent RMS. We evaluated two independently assembled BREE instances, each with two SO-101 arms, at three predefined joint configurations (P1–P3), with three repeated executions per configuration (36 measurements in total). AprilTag image-center displacements were measured relative to Instance A references and converted to millimeter-equivalent distances using the reference AprilTag scale.
Table R3. Visual geometric consistency (mm-equivalent).
| Metric | Left Arm | Right Arm | Overall |
|---|---|---|---|
| BREE: Cross-instance RMS | 14.73 | 21.11 | 18.20 |
| BREE: Maximum cross-instance deviation | 18.16 | 28.05 | — |
| Standard SO-101: Cross-instance RMS | — | — | 65.77 |
BREE calibration achieved an overall cross-instance RMS error of only 18.20 mm-equivalent, compared with 65.77 mm-equivalent under standard SO-101 calibration, representing a 72.3% reduction. Despite combined geometric variations from the robot arms, camera mounts, and aluminum frames, BREE maintains centimeter-level cross-instance visual alignment.
As shown in Table 3 of the paper, BREE calibration achieves 90% success after arm replacement, compared with 0% using standard SO-101 calibration, demonstrating its effectiveness in maintaining policy performance across robot arms.
Repeated reassembly and recalibration produce consistent visual alignment for the same arm. We evaluated three SO-101 arms (Arms 1, 2, and 3), each undergoing three independent cycles of disassembly, reinstallation, and BREE recalibration. After each cycle, AprilTag image-center positions were measured at the same three fixed locations, yielding 27 alignment records.
Table R4. Within-arm visual displacement across repeated reassembly and recalibration cycles (px).
| Metric | Arm 1 | Arm 2 | Arm 3 | Overall |
|---|---|---|---|---|
| Mean within-arm displacement | 5.9 | 5.7 | 7.8 | 6.5 |
| Maximum within-arm displacement | 8.6 | 10.7 | 15.8 | 15.8 |
The mean within-arm displacement across repeated reassembly and recalibration was 6.5 px, with a maximum of 15.8 px. These results demonstrate consistent visual alignment after independent reassembly and recalibration.
Q2: Independent Operators
Response: BREE Instances A and B were independently assembled and calibrated by two different operators, following the same construction specifications and calibration procedures. The cross-instance evaluation therefore includes differences introduced by independent assembly and calibration, although the experiments are limited to two physical instances.
Dear Authors,
Thanks for your comment and the added experiments on the cross-instance validation and supplement evidence. Most of my concerns are resolved, and I've raise my rating to a weak acceptance.
Thank you for your support
Thank you for your positive feedback and for recognizing our work. Your constructive comments have helped us improve our experiments and strengthen the paper. We truly appreciate your time and suggestions!
Response to Reviewer FyjT
We thank the reviewer for the careful assessment and constructive feedback on cross-instance reproducibility and multi-instance scaling. We have conducted additional experiments to examine cross-instance performance under different training seeds, multi-instance data aggregation, and geometric consistency after calibration. We address the concerns below.
W1: Limited evidence for cross-instance reproducibility and differences in full-task success rates.
Response: BREE enables cross-instance policy transfer, with several policies maintaining similar full-task success rates across independently assembled instances. As shown in Table 8, ACT and π0.5 achieve identical success rates on Switch Pushing across both instances, while π0.5 maintains 95% success on Instance B compared with 100% on Instance A for Cylinder Insertion.
Table 8. Benchmark reproduction across BREE instances. The same policy checkpoints are evaluated on Instances A and B, reporting average stage success / full-task success (%).
For relatively long-horizon bimanual manipulation, full-task success alone cannot fully reflect task execution progress. We therefore use stage-wise success as a complementary metric to capture completed operations and identify where failures occur. As discussed in Section 4.4, stage-wise success characterizes the manipulation capabilities retained after cross-instance transfer, while full-task success measures complete-task outcomes. Together, they provide a more comprehensive evaluation of cross-instance reproducibility.
W2: Variability Across Training Seeds inference seeds do not measure variability across independently trained policies
Response: We conducted additional experiments using four different training seeds (1000–1003) for SmolVLA on Plate Loading, with all policies trained on demonstrations collected from Instance A. Each trained policy was evaluated on both BREE instances using the same four inference seeds (0–3) as in the original evaluation, with 20 rollouts per instance. The results show that cross-instance performance remains consistent across different training seeds.
Table R1. Full-task success rates (%) across training seeds (1000-1003) using 50 demonstrations from Instance A.
Although success rates vary across training seeds, the cross-instance gaps remain stable at 25–30 percentage points, with an average gap of 27.5 percentage points. This indicates consistent cross-instance performance across independently trained policies, despite training variability.
W3. Evidence for data scaling is insufficient.
Response: As suggested, we have included more evidence to demonstrate BREE's data scaling (see Table R2). We included ACT on Pick-and-Place, SmolVLA on Drawer Placement, and X-VLA on Trajectory Pushing. Training budgets range from 25 to 100 demonstrations, with all evaluations conducted on Instance A.
The following results have further proved the multi-instance scaling property in our framework, i.e., demonstrations collected independently on different BREE instances can be directly combined for policy training, while maintaining performance close to training with data from a single instance.
As indicated in Table R2, when the mixed-instance dataset increases from 50 to 100 demonstrations, success rates improve from 60% to 80% for ACT, 45% to 75% for SmolVLA, and 45% to 60% for X-VLA.
Table R2. Full-task success rates (%) evaluated on BREE Instance A under different training data configurations. A and B denote two independently assembled BREE instances.