- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents LIBERO-Industry, a benchmark that extends LIBERO toward industrial settings such as high-mix, low-volume manufacturing. It tests generalist robot policies under conditions household benchmarks leave out: parts arriving on moving conveyors, limited time per part, and placement into fixtures of different precision. The tasks form three suites, grouped by whether the source and the destination move. In D2S the robot unloads parts from a conveyor into a static fixture, in S2D it loads parts into an indexing carrier, and in D2D both ends move. Each suite has three difficulty levels, from a tray to an object-specific groove to a form-fitting inlay, with more scene variation at each level. In total there are 216 procedurally generated tasks, 180 for training and 36 held out. Each task's time budget follows from the conveyor speed and the spacing of parts and slots. In simulation, five public policies fine-tuned on the original LIBERO, including and OpenVLA-OFT, are tested on the held-out tasks without further training, and none completes a single task. A step-by-step diagnostic with moves gradually from a LIBERO-Spatial scene to a multi-object conveyor scene. Performance drops at each step and collapses once more than one object must be handled. Finally, the authors build a real conveyor cell with an AUBO-i5H arm. They report that co-trained with LIBERO-Industry simulation data and 95 real demonstrations performs better there than the same model co-trained with original LIBERO data.
Critical strengths
-
The gap the paper targets is real. In most VLA benchmarks the scene waits for the robot, so timing barely matters. But here the belt speed, arrival interval and dwell time are task parameters. The episode horizon is also derived from the transport schedule rather than set as a fixed step limit. I think this is a sensible design that others could reuse.
-
The paper involves a lot of engineering work. For example, the authors built a physical D2S conveyor cell and collected 95 teleoperated demos with it. They built a Panda-to-Aubo conversion and recovery pipeline. They built a scripted demo-collection pipeline with replay-based admission.
Critical weaknesses
- The main benchmark result doesn't tell us whether the benchmark works.
All five policies get 0.0% SR on all 36 held-out tasks. PCR_T is zero everywhere except on S2D, which gets 1.1%. Those checkpoints were fine-tuned on original LIBERO only, "without further fine-tuning on LIBERO-Industry", so the numbers show that LIBERO policies don't transfer to a new set of objects, fixtures and conveyor dynamics. That is expected, and it can't be pinned on any one industrial constraint. PCR_S (11.2–12.5%) doesn't separate the policies either, and because it credits pre-existing destination workpieces that stay valid, "a nonzero score therefore need not indicate a new target transfer". For a benchmark paper, the missing piece is direct evidence that the held-out tasks can be learned from the training split and that the benchmark can rank methods. One or two policies trained on the 180 tasks and evaluated on the 36, broken down by suite, level and speed, would go a long way.
- The training data and the evaluation seem to use different robots.
The paper doesn't say which robot is used in the simulated evaluation. The released demos appear to have been collected on an Aubo:
- In the supplement, each demo group in
simulation/hdf5/industry_d2d_lv3_sample.hdf5(data/demo_1todata/demo_10) has the attributesrobot: AuboI5andgripper: CTAG2F90DGripper. - The supplement's README describes this file as "ten accepted Aubo demonstrations" (README.md, line 9). It says the Industry videos were "rendered at 384 x 384 with AuboI5, CTAG2F90DGripper" (README.md, line 26).
- The videos in
videos/autocollection/show an orange AUBO arm.
The evaluation, on the other hand, appears to use a Panda:
- The five evaluated checkpoints are original-LIBERO fine-tunes (Table 12, p. 25), and LIBERO uses a Franka Panda (Liu et al., 2023).
- The setup section says the evaluation follows "the original LIBERO simulation settings" (L368, p. 7).
- The rollouts shown in Fig. 9 (p. 28) and the task renders in Fig. 5 (p. 17) appear to show a Panda.
- Some Issue in real-robot comparison:
S1 and S2 differ in more than process dynamics. Compared with S1's sim data, the S2 sim data:
- uses the same objects as the real cell (school bus, mini roller, Mario, Thomas in Table 16, L1621–1622, p. 31; all four are listed in Table 5, L977–979, p. 19)
- has about twice as many frames
- was collected on the Aubo directly, while S1 is "Aubo-native original-LIBERO recovered-v2" data produced by the Panda-to-Aubo conversion.
The perturbation results in Fig. 4C are higher than the unperturbed ones in Fig. 4B. For example, S2 gets 58.3% with OOD objects (L468, p. 9), against 34.7% and 23.6% without perturbation (L454–456). Neither the text nor the caption gives the speed or the number of trials for Fig. 4C, so I can't compare the two.
- Some parts in appendix look like unfinished:
- "The exact weight revisions and model-specific inference settings should be recorded in the released evaluation configurations" (App. C.1, L1335–1337, p. 25).
- "the final evaluation must also freeze a precedence rule for episodes that satisfy more than one failure condition" (App. C.3.2, L1391–1392, p. 26).
- "The historical values are not yet tied to frozen denominators, seeds, or confidence intervals" (App. C.3.2, L1433–1434, p. 27). These are the values behind Fig. 3.
- "Results report the complete success–deadline curve for each task" (App. B.4.4, L1154–1155, p. 22). No such curve appears in the paper.
- Throughput is listed as something the paper measures in the introduction (L86 and L99–100, p. 2) and defined in Eq. 17 (L1162–1166, p. 22), but no throughput number is reported.
- Table 7 is described as "the parameters that must be exported with the benchmark release" (L1091–1092, p. 21). This is again phrased as something still to be done.
Questions for authors
- Which robot is used in the simulated evaluation in Table 3, and in which embodiment is the training split released?
- What speed and how many trials were used for Fig. 4C? Why are the perturbed results higher than the nominal ones in Fig. 4B?
- The authors should check the wording in the appendix and report the missing parameters and curves.
7 comments
Response to Reviewer 4Tq5 [2/5] — W1(c–d)
W1(c) — Interpretation of SR, PCR_T and PCR_S
W1(c): “PCR_S (11.2–12.5%) doesn't separate the policies either, and because it credits pre-existing destination workpieces that stay valid, "a nonzero score therefore need not indicate a new target transfer".”
A: We introduced PCR_S alongside PCR_T to separate target-transfer completion from the overall slotting outcome. In an ongoing material-flow process, the robot must place incoming workpieces while preserving the valid placement of workpieces already at the destination. PCR_T measures the fraction of required new transfers completed. PCR_S additionally accounts for pre-existing workpieces by measuring valid destination placement across the full set of goal workpieces. Their inclusion reflects the placement requirements of the operation.
Two Aubo examples in Appendix C.7 illustrate this distinction:
- In the failed S2D evaluation rollout shown in Appendix C.7, panel (a), three workpieces require transfer and two are already validly placed. No new transfer completes, giving PCR_T = 0/3 = 0%, while the two existing placements remain valid, giving PCR_S = 2/5 = 40%.
- In the successful D2S training demonstration shown in Appendix C.7, panel (b), both new transfers complete and the existing placement remains valid, giving PCR_T = 2/2 = 100% and PCR_S = 3/3 = 100%.
For the 36 held-out configurations, the BDDL-declared initial-scene reference is 12.31%, computed by macro-averaging the task-level ratios N_base/(N_target + N_base). This is a reference derived from task composition, not a measured no-action policy result. The reported 11.2–12.5% PCR_S values are close to this reference and should not be interpreted as evidence of substantial new-transfer capability or a reliable policy ranking.
W1(d) — In-domain learnability and policy discrimination
W1(d): “For a benchmark paper, the missing piece is direct evidence that the held-out tasks can be learned from the training split and that the benchmark can rank methods. One or two policies trained on the 180 tasks and evaluated on the 36, broken down by suite, level and speed, would go a long way.”
A: We trained two baselines on the available LIBERO-Industry training demonstrations and evaluated complete-task success and partial completion by suite, difficulty level, and configured conveyor velocity. We fine-tuned π₀.₅ and OpenVLA-OFT for 30,000 updates using the same 1,412 Aubo simulation demonstrations. Our available demonstration set covers 147 of the 180 training configurations. Both policies were evaluated in Aubo simulation on the same 36 held-out configurations, with ten saved initial states per configuration. Each suite–level has 40 episodes per policy. SR denotes complete-task success (successes/episodes in parentheses); PCR values average all episodes, including failures.
| Suite–level | π₀.₅ SR (%) | π₀.₅ PCR_T (%) | π₀.₅ PCR_S (%) | OpenVLA-OFT SR (%) | OpenVLA-OFT PCR_T (%) | OpenVLA-OFT PCR_S (%) |
|---|---|---|---|---|---|---|
| D2D–L1 | 62.50 (25/40) | 85.83 | 85.83 | 42.50 (17/40) | 67.50 | 67.50 |
| D2D–L2 | 5.00 (2/40) | 20.83 | 20.83 | 0.00 (0/40) | 5.00 | 5.00 |
| D2D–L3 | 0.00 (0/40) | 0.00 | 40.25 | 0.00 (0/40) | 0.00 | 40.75 |
| D2S–L1 | 10.00 (4/40) | 46.67 | 46.67 | 0.00 (0/40) | 35.83 | 35.83 |
| D2S–L2 | 0.00 (0/40) | 1.67 | 1.67 | 0.00 (0/40) | 0.00 | 0.00 |
| D2S–L3 | 0.00 (0/40) | 0.00 | 33.33 | 0.00 (0/40) | 0.00 | 32.50 |
| S2D–L1 | 40.00 (16/40) | 66.67 | 66.67 | 7.50 (3/40) | 45.00 | 45.00 |
| S2D–L2 | 0.00 (0/40) | 3.33 | 3.33 | 0.00 (0/40) | 1.67 | 1.67 |
| S2D–L3 | 0.00 (0/40) | 4.17 | 33.00 | 0.00 (0/40) | 0.00 | 31.50 |
| Overall | 13.06 (47/360) | 25.46 | 36.84 | 5.56 (20/360) | 17.22 | 28.86 |
The observed overall SR difference is 7.50 percentage points, with successful completion concentrated at Level 1. π₀.₅ additionally completes two Level-2 episodes; neither policy completes a Level-3 episode. Zero SR can still accompany partial progress: OpenVLA-OFT completes no D2S–L1 episode but achieves 35.83% PCR_T. PCR_S also credits pre-existing valid placements, so higher PCR_S alone does not establish new-transfer completion.
Detailed difficulty-wise and suite-by-level results are reported in Appendix C.5, Tables 21–22. Table 23 in the same subsection gives the complete breakdown for all 36 held-out configurations, including both policies’ SR, PCR_T and PCR_S and the signed source and destination transport velocities.
Velocities in Table 23 are signed transport velocities in m/s; zero denotes a static endpoint. Other task properties also vary across configurations.
Response to Reviewer 4Tq5 [3/5] — W2 / Q1; W3
W2(a,b) / Q1 — Training-data and evaluation embodiments
W2(a,b): “The training data and the evaluation seem to use different robots.”
Q1: “Which robot is used in the simulated evaluation in Table 3, and in which embodiment is the training split released?”
A: Table 3 evaluation. The simulation evaluation in Table 3 uses Panda. These checkpoints were fine-tuned on original LIBERO using Panda and received no additional fine-tuning on LIBERO-Industry demonstrations.
Industry demonstrations and evaluation. The Industry demonstrations used for the new simulation baseline and the S2 physical study use Aubo. The released demonstration sample cited in your review also uses Aubo. The new baseline is trained and evaluated in Aubo simulation. The embodiments used for demonstration collection, training and evaluation are:
| Experiment | Evaluation setting | Where reported | Training embodiment | Evaluation embodiment |
|---|---|---|---|---|
| Original-LIBERO checkpoint transfer | Simulation | Main paper, Table 3 | Panda | Panda |
| Industry-trained π₀.₅ and OpenVLA-OFT simulation baselines | Simulation | Main paper, §4.2, Table 4; Appendix C.5 | Aubo | Aubo |
| S2: Industry simulation + physical data | Real-world | Main paper, Figure 4B–C | Aubo | Aubo |
Embodiment note. Here, Aubo denotes AuboI5 with the CTAG gripper in simulation and AUBO-i5H with the CTAG gripper for physical collection and evaluation.
Note on S1 conversion. For S1, we convert original-LIBERO demonstrations collected with Panda to Aubo demonstrations by adapting the source end-effector trajectories and executing the adapted trajectories in Aubo simulation. The resulting Aubo demonstrations are used for co-training. Further implementation details are provided in Appendix D.1, “Trajectory Conversion and Recovery.”
W3(a) — S1/S2 comparison settings
W3(a): “Some Issue in real-robot comparison: S1 and S2 differ in more than process dynamics. Compared with S1's sim data, the S2 sim data: uses the same objects as the real cell (school bus, mini roller, Mario, Thomas in Table 16, L1621–1622, p. 31; all four are listed in Table 5, L977–979, p. 19) has about twice as many frames was collected on the Aubo directly, while S1 is "Aubo-native original-LIBERO recovered-v2" data produced by the Panda-to-Aubo conversion.”
A: Our physical study compares two simulation-data sources under a shared co-training protocol: the same π₀.₅ base checkpoint, 95 physical demonstrations, 9:1 simulation–real sampling ratio, and 30,000 training updates. Figure 4C additionally evaluates objects excluded from both training datasets. At 0.03 m/s, S2 achieves 58.3% target-transfer completion, compared with 33.3% for S1, over 72 planned transfers per policy.
These results assess the downstream utility of the two co-training conditions in the tested workcell. Separating the contributions of object coverage, training-data volume, and demonstration conversion requires additional controlled physical experiments. These experiments cannot be completed within the rebuttal period; we plan to extend this analysis in a future revision. We also plan to clarify the distinction between stored frames and sampled training inputs in the final protocol description.
W3(b) / Q2 — Figure 4C conditions and perturbation results
W3(b): “The perturbation results in Fig. 4C are higher than the unperturbed ones in Fig. 4B. For example, S2 gets 58.3% with OOD objects (L468, p. 9), against 34.7% and 23.6% without perturbation (L454–456). Neither the text nor the caption gives the speed or the number of trials for Fig. 4C, so I can't compare the two.”
Q2: “What speed and how many trials were used for Fig. 4C? Why are the perturbed results higher than the nominal ones in Fig. 4B?”
A: Figure 4C uses a conveyor speed of 0.03 m/s, with 24 trials per policy per condition and three planned transfers per trial. The reported percentages measure target-transfer completion (PCR_T) over 72 planned transfers. At the same speed, S2 achieves 58.3% with the selected OOD objects, compared with 34.7% under the nominal condition in Figure 4B.
The selected OOD objects may offer more favorable grasping geometry and clearance in this workcell, potentially reducing collision-related interruptions and their effects on subsequent transfers. This is a possible explanation for the observed result; establishing its contribution requires a controlled comparison of object geometry and interruption frequency. We plan to investigate this in future physical experiments. The present result therefore characterizes performance on the selected OOD object set.
Response to Reviewer 4Tq5 [1/5] — W1(a–b)
We are grateful for your time devoted to reviewing this paper and for your comprehensive feedback about our work. Please see our detailed responses to your comments and suggestions below.
W1(a) — Interpretation of the zero-success benchmark results?
W1(a): “The main benchmark result doesn't tell us whether the benchmark works. All five policies get 0.0% SR on all 36 held-out tasks. PCR_T is zero everywhere except π0.5 on S2D, which gets 1.1%.”
A: The zero SR in Table 3 means that none of the five original-LIBERO-adapted checkpoints completed the full required workload in the 360 held-out evaluation episodes per policy. These checkpoints received no additional LIBERO-Industry training. The result therefore characterizes their cross-benchmark transfer under the combined task requirements.
To evaluate held-out performance after in-domain training, we fine-tuned π₀.₅ and OpenVLA-OFT for 30,000 updates using the same 1,412 Aubo simulation demonstrations. Our available demonstration set covers 147 of the 180 training configurations. Both policies were evaluated in Aubo simulation on the same 36 held-out configurations, with ten saved initial states per configuration (360 episodes per policy). For metrics, PCR_T measures target-transfer completion; PCR_S additionally includes pre-existing valid destination workpieces. Both are averaged across evaluation episodes.
After in-domain training, π₀.₅ and OpenVLA-OFT achieve held-out complete-task SRs of 13.06% and 5.56%, respectively. Successful completion and the 7.50-percentage-point separation support the benchmark’s use for assessing learned execution and comparing these two adapted policies under industrial process constraints, complementing the transfer evaluation on Panda.
W1(b) — Cross-benchmark transfer and constraint attribution
W1(b): “Those checkpoints were fine-tuned on original LIBERO only, "without further fine-tuning on LIBERO-Industry", so the numbers show that LIBERO policies don't transfer to a new set of objects, fixtures and conveyor dynamics. That is expected, and it can't be pinned on any one industrial constraint.”
A: We interpret Table 3 as evidence of a transfer gap under combined changes in objects, fixtures and dynamics. To examine sensitivity to specific requirements beyond this aggregate result, we report four targeted comparisons.
All comparisons use the same frozen original-LIBERO-finetuned π₀.₅ policy on Panda at 20 Hz, with four layouts × 20 paired trials per condition and layout family. C1/C2/C4 allow 400 actions; C3 uses schedule-derived limits of 271–811 actions.
C1/C4 triplets follow D2D-, D2S- and S2D-derived layouts. SR measures complete-task success. In C3, PCR_T is the valid source-transfer count divided by three.
These results provide within-setting diagnostic evidence rather than assigning Table 3’s zero SR to one constraint. The stationary D2D floor in C2 and near-floor SR in C3 limit estimation of additional motion and timing penalties. C2 and C3 differ in workload, so their SR gap is not a pure timing effect. Static C4 does not test moving-slot locking, and the comparisons do not establish a global constraint ranking.
Together, these findings support LIBERO-Industry’s focus on complete-task reliability under changing industrial operating conditions and demonstrate its utility for diagnosing the tested transfer failures. The full protocols, condition-wise results and visual examples are provided in Appendix C.6, “Targeted Comparisons of Industrial Task Requirements.”