acceptodds
Under review as a conference paper at ICLR 2027

LIBERO-Industry: Benchmarking Robotic Manipulation under Industrial Process Constraints

Abstract

Industrial manipulation in flexible manufacturing, e.g., high-mix, low-volume (HMLV) electronics assembly, requires robot policies to adapt to changing products and production conditions. Such settings impose industrial process constraints, including product variants, moving material flows, finite execution windows, and tight geometric tolerances. However, existing evaluation frameworks do not systematically capture these constraints when assessing generalist robot policies. To address this gap, we introduce LIBERO-Industry, a benchmark built on the widely used robot learning benchmark LIBERO that translates industrial process constraints into controllable task conditions and measurable success criteria. LIBERO-Industry comprises three industrial workflow scenarios: Static-to-Dynamic (S2D) loading, Dynamic-to-Static (D2S) unloading, and Dynamic-to-Dynamic (D2D) transfer. The benchmark contains 216 tasks that systematically vary object geometry, conveyor speed, interaction-window duration, and geometric acceptance-region size. Experiments across five generalist robot policies, including vision-language-action and world-action models, reveal progressive performance degradation as industrial constraints are introduced and combined; benchmark code and datasets will be made publicly available to support reproducible evaluation. Real-world D2S experiments further show higher physical task success when sim-real co-training uses LIBERO-Industry rather than standard LIBERO simulation data under matched training conditions. Code and data will be made publicly available.

Primary area:
datasets and benchmarks
Keywords:
robot learning, robotic manipulation, industrial manipulation, industrial robotics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4Tq5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LIBERO-Industry, a benchmark that extends LIBERO toward industrial settings such as high-mix, low-volume manufacturing. It tests generalist robot policies under conditions household benchmarks leave out: parts arriving on moving conveyors, limited time per part, and placement into fixtures of different precision. The tasks form three suites, grouped by whether the source and the destination move. In D2S the robot unloads parts from a conveyor into a static fixture, in S2D it loads parts into an indexing carrier, and in D2D both ends move. Each suite has three difficulty levels, from a tray to an object-specific groove to a form-fitting inlay, with more scene variation at each level. In total there are 216 procedurally generated tasks, 180 for training and 36 held out. Each task's time budget follows from the conveyor speed and the spacing of parts and slots. In simulation, five public policies fine-tuned on the original LIBERO, including π0.5\pi_{0.5} and OpenVLA-OFT, are tested on the held-out tasks without further training, and none completes a single task. A step-by-step diagnostic with π0.5\pi_{0.5} moves gradually from a LIBERO-Spatial scene to a multi-object conveyor scene. Performance drops at each step and collapses once more than one object must be handled. Finally, the authors build a real conveyor cell with an AUBO-i5H arm. They report that π0.5\pi_{0.5} co-trained with LIBERO-Industry simulation data and 95 real demonstrations performs better there than the same model co-trained with original LIBERO data.

Critical strengths
  1. The gap the paper targets is real. In most VLA benchmarks the scene waits for the robot, so timing barely matters. But here the belt speed, arrival interval and dwell time are task parameters. The episode horizon is also derived from the transport schedule rather than set as a fixed step limit. I think this is a sensible design that others could reuse.

  2. The paper involves a lot of engineering work. For example, the authors built a physical D2S conveyor cell and collected 95 teleoperated demos with it. They built a Panda-to-Aubo conversion and recovery pipeline. They built a scripted demo-collection pipeline with replay-based admission.

Critical weaknesses
  1. The main benchmark result doesn't tell us whether the benchmark works.

All five policies get 0.0% SR on all 36 held-out tasks. PCR_T is zero everywhere except π0.5\pi_{0.5} on S2D, which gets 1.1%. Those checkpoints were fine-tuned on original LIBERO only, "without further fine-tuning on LIBERO-Industry", so the numbers show that LIBERO policies don't transfer to a new set of objects, fixtures and conveyor dynamics. That is expected, and it can't be pinned on any one industrial constraint. PCR_S (11.2–12.5%) doesn't separate the policies either, and because it credits pre-existing destination workpieces that stay valid, "a nonzero score therefore need not indicate a new target transfer". For a benchmark paper, the missing piece is direct evidence that the held-out tasks can be learned from the training split and that the benchmark can rank methods. One or two policies trained on the 180 tasks and evaluated on the 36, broken down by suite, level and speed, would go a long way.

  1. The training data and the evaluation seem to use different robots.

The paper doesn't say which robot is used in the simulated evaluation. The released demos appear to have been collected on an Aubo:

  • In the supplement, each demo group in simulation/hdf5/industry_d2d_lv3_sample.hdf5 (data/demo_1 to data/demo_10) has the attributes robot: AuboI5 and gripper: CTAG2F90DGripper.
  • The supplement's README describes this file as "ten accepted Aubo demonstrations" (README.md, line 9). It says the Industry videos were "rendered at 384 x 384 with AuboI5, CTAG2F90DGripper" (README.md, line 26).
  • The videos in videos/autocollection/ show an orange AUBO arm.

The evaluation, on the other hand, appears to use a Panda:

  • The five evaluated checkpoints are original-LIBERO fine-tunes (Table 12, p. 25), and LIBERO uses a Franka Panda (Liu et al., 2023).
  • The setup section says the evaluation follows "the original LIBERO simulation settings" (L368, p. 7).
  • The π0.5\pi_{0.5} rollouts shown in Fig. 9 (p. 28) and the task renders in Fig. 5 (p. 17) appear to show a Panda.
  1. Some Issue in real-robot comparison:

S1 and S2 differ in more than process dynamics. Compared with S1's sim data, the S2 sim data:

  • uses the same objects as the real cell (school bus, mini roller, Mario, Thomas in Table 16, L1621–1622, p. 31; all four are listed in Table 5, L977–979, p. 19)
  • has about twice as many frames
  • was collected on the Aubo directly, while S1 is "Aubo-native original-LIBERO recovered-v2" data produced by the Panda-to-Aubo conversion.

The perturbation results in Fig. 4C are higher than the unperturbed ones in Fig. 4B. For example, S2 gets 58.3% with OOD objects (L468, p. 9), against 34.7% and 23.6% without perturbation (L454–456). Neither the text nor the caption gives the speed or the number of trials for Fig. 4C, so I can't compare the two.

  1. Some parts in appendix look like unfinished:
  • "The exact weight revisions and model-specific inference settings should be recorded in the released evaluation configurations" (App. C.1, L1335–1337, p. 25).
  • "the final evaluation must also freeze a precedence rule for episodes that satisfy more than one failure condition" (App. C.3.2, L1391–1392, p. 26).
  • "The historical values are not yet tied to frozen denominators, seeds, or confidence intervals" (App. C.3.2, L1433–1434, p. 27). These are the values behind Fig. 3.
  • "Results report the complete success–deadline curve for each task" (App. B.4.4, L1154–1155, p. 22). No such curve appears in the paper.
  • Throughput is listed as something the paper measures in the introduction (L86 and L99–100, p. 2) and defined in Eq. 17 (L1162–1166, p. 22), but no throughput number is reported.
  • Table 7 is described as "the parameters that must be exported with the benchmark release" (L1091–1092, p. 21). This is again phrased as something still to be done.
Questions for authors
  1. Which robot is used in the simulated evaluation in Table 3, and in which embodiment is the training split released?
  2. What speed and how many trials were used for Fig. 4C? Why are the perturbed results higher than the nominal ones in Fig. 4B?
  3. The authors should check the wording in the appendix and report the missing parameters and curves.
7 comments
AuthorsOpenReview ↗

Response to Reviewer 4Tq5 [1/5] — W1(a–b)

We are grateful for your time devoted to reviewing this paper and for your comprehensive feedback about our work. Please see our detailed responses to your comments and suggestions below.

W1(a) — Interpretation of the zero-success benchmark results?

W1(a): “The main benchmark result doesn't tell us whether the benchmark works. All five policies get 0.0% SR on all 36 held-out tasks. PCR_T is zero everywhere except π0.5 on S2D, which gets 1.1%.”

A: The zero SR in Table 3 means that none of the five original-LIBERO-adapted checkpoints completed the full required workload in the 360 held-out evaluation episodes per policy. These checkpoints received no additional LIBERO-Industry training. The result therefore characterizes their cross-benchmark transfer under the combined task requirements.

To evaluate held-out performance after in-domain training, we fine-tuned π₀.₅ and OpenVLA-OFT for 30,000 updates using the same 1,412 Aubo simulation demonstrations. Our available demonstration set covers 147 of the 180 training configurations. Both policies were evaluated in Aubo simulation on the same 36 held-out configurations, with ten saved initial states per configuration (360 episodes per policy). For metrics, PCR_T measures target-transfer completion; PCR_S additionally includes pre-existing valid destination workpieces. Both are averaged across evaluation episodes.

PolicyComplete-task SR % (successes/episodes)PCR_T (%)PCR_S (%)
π₀.₅13.06 (47/360)25.4636.84
OpenVLA-OFT5.56 (20/360)17.2228.86

After in-domain training, π₀.₅ and OpenVLA-OFT achieve held-out complete-task SRs of 13.06% and 5.56%, respectively. Successful completion and the 7.50-percentage-point separation support the benchmark’s use for assessing learned execution and comparing these two adapted policies under industrial process constraints, complementing the transfer evaluation on Panda.

W1(b) — Cross-benchmark transfer and constraint attribution

W1(b): “Those checkpoints were fine-tuned on original LIBERO only, "without further fine-tuning on LIBERO-Industry", so the numbers show that LIBERO policies don't transfer to a new set of objects, fixtures and conveyor dynamics. That is expected, and it can't be pinned on any one industrial constraint.”

A: We interpret Table 3 as evidence of a transfer gap under combined changes in objects, fixtures and dynamics. To examine sensitivity to specific requirements beyond this aggregate result, we report four targeted comparisons.

All comparisons use the same frozen original-LIBERO-finetuned π₀.₅ policy on Panda at 20 Hz, with four layouts × 20 paired trials per condition and layout family. C1/C2/C4 allow 400 actions; C3 uses schedule-derived limits of 271–811 actions.

ComparisonResultsInterpretation
C1: Workpiece substitution in static layoutsAkita controls: 78/80, 70/80, 80/80; substituted workpieces: 0–19/80, 1–25/80, 15–48/80, respectively.Workpiece substitution contributes to the transfer gap even without motion.
C2: Source/destination motionSource motion changes D2S success from 51/80 to 28/80; destination motion changes S2D from 76/80 to 69/80. D2D remains 0/80, including the stationary control.D2S and S2D show sensitivity to the tested motion changes.
C3: Three-transfer execution under different schedulesD2S/S2D/D2D complete-task SR: 2/720, 0/960, 0/960; final PCR_T: 9.40%, 32.92%, 1.08%.Partial progress persists while completing all three transfers remains difficult.
C4: Static fixture opening narrowed from 50 to 2 mmSuccess changes from 67/80 to 11/80, 47/80 to 1/80, and 77/80 to 24/80, respectively.Tighter openings increase completion difficulty for the same workpiece in matched layouts.

C1/C4 triplets follow D2D-, D2S- and S2D-derived layouts. SR measures complete-task success. In C3, PCR_T is the valid source-transfer count divided by three.

These results provide within-setting diagnostic evidence rather than assigning Table 3’s zero SR to one constraint. The stationary D2D floor in C2 and near-floor SR in C3 limit estimation of additional motion and timing penalties. C2 and C3 differ in workload, so their SR gap is not a pure timing effect. Static C4 does not test moving-slot locking, and the comparisons do not establish a global constraint ranking.

Together, these findings support LIBERO-Industry’s focus on complete-task reliability under changing industrial operating conditions and demonstrate its utility for diagnosing the tested transfer failures. The full protocols, condition-wise results and visual examples are provided in Appendix C.6, “Targeted Comparisons of Industrial Task Requirements.”

AuthorsOpenReview ↗

Response to Reviewer 4Tq5 [2/5] — W1(c–d)

W1(c) — Interpretation of SR, PCR_T and PCR_S

W1(c): “PCR_S (11.2–12.5%) doesn't separate the policies either, and because it credits pre-existing destination workpieces that stay valid, "a nonzero score therefore need not indicate a new target transfer".”

A: We introduced PCR_S alongside PCR_T to separate target-transfer completion from the overall slotting outcome. In an ongoing material-flow process, the robot must place incoming workpieces while preserving the valid placement of workpieces already at the destination. PCR_T measures the fraction of required new transfers completed. PCR_S additionally accounts for pre-existing workpieces by measuring valid destination placement across the full set of goal workpieces. Their inclusion reflects the placement requirements of the operation.

Two Aubo examples in Appendix C.7 illustrate this distinction:

  • In the failed S2D evaluation rollout shown in Appendix C.7, panel (a), three workpieces require transfer and two are already validly placed. No new transfer completes, giving PCR_T = 0/3 = 0%, while the two existing placements remain valid, giving PCR_S = 2/5 = 40%.
  • In the successful D2S training demonstration shown in Appendix C.7, panel (b), both new transfers complete and the existing placement remains valid, giving PCR_T = 2/2 = 100% and PCR_S = 3/3 = 100%.

For the 36 held-out configurations, the BDDL-declared initial-scene reference is 12.31%, computed by macro-averaging the task-level ratios N_base/(N_target + N_base). This is a reference derived from task composition, not a measured no-action policy result. The reported 11.2–12.5% PCR_S values are close to this reference and should not be interpreted as evidence of substantial new-transfer capability or a reliable policy ranking.

W1(d) — In-domain learnability and policy discrimination

W1(d): “For a benchmark paper, the missing piece is direct evidence that the held-out tasks can be learned from the training split and that the benchmark can rank methods. One or two policies trained on the 180 tasks and evaluated on the 36, broken down by suite, level and speed, would go a long way.”

A: We trained two baselines on the available LIBERO-Industry training demonstrations and evaluated complete-task success and partial completion by suite, difficulty level, and configured conveyor velocity. We fine-tuned π₀.₅ and OpenVLA-OFT for 30,000 updates using the same 1,412 Aubo simulation demonstrations. Our available demonstration set covers 147 of the 180 training configurations. Both policies were evaluated in Aubo simulation on the same 36 held-out configurations, with ten saved initial states per configuration. Each suite–level has 40 episodes per policy. SR denotes complete-task success (successes/episodes in parentheses); PCR values average all episodes, including failures.

Suite–levelπ₀.₅ SR (%)π₀.₅ PCR_T (%)π₀.₅ PCR_S (%)OpenVLA-OFT SR (%)OpenVLA-OFT PCR_T (%)OpenVLA-OFT PCR_S (%)
D2D–L162.50 (25/40)85.8385.8342.50 (17/40)67.5067.50
D2D–L25.00 (2/40)20.8320.830.00 (0/40)5.005.00
D2D–L30.00 (0/40)0.0040.250.00 (0/40)0.0040.75
D2S–L110.00 (4/40)46.6746.670.00 (0/40)35.8335.83
D2S–L20.00 (0/40)1.671.670.00 (0/40)0.000.00
D2S–L30.00 (0/40)0.0033.330.00 (0/40)0.0032.50
S2D–L140.00 (16/40)66.6766.677.50 (3/40)45.0045.00
S2D–L20.00 (0/40)3.333.330.00 (0/40)1.671.67
S2D–L30.00 (0/40)4.1733.000.00 (0/40)0.0031.50
Overall13.06 (47/360)25.4636.845.56 (20/360)17.2228.86

The observed overall SR difference is 7.50 percentage points, with successful completion concentrated at Level 1. π₀.₅ additionally completes two Level-2 episodes; neither policy completes a Level-3 episode. Zero SR can still accompany partial progress: OpenVLA-OFT completes no D2S–L1 episode but achieves 35.83% PCR_T. PCR_S also credits pre-existing valid placements, so higher PCR_S alone does not establish new-transfer completion.

Detailed difficulty-wise and suite-by-level results are reported in Appendix C.5, Tables 21–22. Table 23 in the same subsection gives the complete breakdown for all 36 held-out configurations, including both policies’ SR, PCR_T and PCR_S and the signed source and destination transport velocities.

Velocities in Table 23 are signed transport velocities in m/s; zero denotes a static endpoint. Other task properties also vary across configurations.

AuthorsOpenReview ↗

Response to Reviewer 4Tq5 [3/5] — W2 / Q1; W3

W2(a,b) / Q1 — Training-data and evaluation embodiments

W2(a,b): “The training data and the evaluation seem to use different robots.”

Q1: “Which robot is used in the simulated evaluation in Table 3, and in which embodiment is the training split released?”

A: Table 3 evaluation. The simulation evaluation in Table 3 uses Panda. These checkpoints were fine-tuned on original LIBERO using Panda and received no additional fine-tuning on LIBERO-Industry demonstrations.

Industry demonstrations and evaluation. The Industry demonstrations used for the new simulation baseline and the S2 physical study use Aubo. The released demonstration sample cited in your review also uses Aubo. The new baseline is trained and evaluated in Aubo simulation. The embodiments used for demonstration collection, training and evaluation are:

ExperimentEvaluation settingWhere reportedTraining embodimentEvaluation embodiment
Original-LIBERO checkpoint transferSimulationMain paper, Table 3PandaPanda
Industry-trained π₀.₅ and OpenVLA-OFT simulation baselinesSimulationMain paper, §4.2, Table 4; Appendix C.5AuboAubo
S2: Industry simulation + physical dataReal-worldMain paper, Figure 4B–CAuboAubo

Embodiment note. Here, Aubo denotes AuboI5 with the CTAG gripper in simulation and AUBO-i5H with the CTAG gripper for physical collection and evaluation.

Note on S1 conversion. For S1, we convert original-LIBERO demonstrations collected with Panda to Aubo demonstrations by adapting the source end-effector trajectories and executing the adapted trajectories in Aubo simulation. The resulting Aubo demonstrations are used for co-training. Further implementation details are provided in Appendix D.1, “Trajectory Conversion and Recovery.”

W3(a) — S1/S2 comparison settings

W3(a): “Some Issue in real-robot comparison: S1 and S2 differ in more than process dynamics. Compared with S1's sim data, the S2 sim data: uses the same objects as the real cell (school bus, mini roller, Mario, Thomas in Table 16, L1621–1622, p. 31; all four are listed in Table 5, L977–979, p. 19) has about twice as many frames was collected on the Aubo directly, while S1 is "Aubo-native original-LIBERO recovered-v2" data produced by the Panda-to-Aubo conversion.”

A: Our physical study compares two simulation-data sources under a shared co-training protocol: the same π₀.₅ base checkpoint, 95 physical demonstrations, 9:1 simulation–real sampling ratio, and 30,000 training updates. Figure 4C additionally evaluates objects excluded from both training datasets. At 0.03 m/s, S2 achieves 58.3% target-transfer completion, compared with 33.3% for S1, over 72 planned transfers per policy.

These results assess the downstream utility of the two co-training conditions in the tested workcell. Separating the contributions of object coverage, training-data volume, and demonstration conversion requires additional controlled physical experiments. These experiments cannot be completed within the rebuttal period; we plan to extend this analysis in a future revision. We also plan to clarify the distinction between stored frames and sampled training inputs in the final protocol description.

W3(b) / Q2 — Figure 4C conditions and perturbation results

W3(b): “The perturbation results in Fig. 4C are higher than the unperturbed ones in Fig. 4B. For example, S2 gets 58.3% with OOD objects (L468, p. 9), against 34.7% and 23.6% without perturbation (L454–456). Neither the text nor the caption gives the speed or the number of trials for Fig. 4C, so I can't compare the two.”

Q2: “What speed and how many trials were used for Fig. 4C? Why are the perturbed results higher than the nominal ones in Fig. 4B?”

A: Figure 4C uses a conveyor speed of 0.03 m/s, with 24 trials per policy per condition and three planned transfers per trial. The reported percentages measure target-transfer completion (PCR_T) over 72 planned transfers. At the same speed, S2 achieves 58.3% with the selected OOD objects, compared with 34.7% under the nominal condition in Figure 4B.

The selected OOD objects may offer more favorable grasping geometry and clearance in this workcell, potentially reducing collision-related interruptions and their effects on subsequent transfers. This is a possible explanation for the observed result; establishing its contribution requires a controlled comparison of object geometry and interruption frequency. We plan to investigate this in future physical experiments. The present result therefore characterizes performance on the selected OOD object set.

Reviewer_GCNXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on benchmarking robotic manipulation under industrial process constraints. The authors argue for the necessity of standardized evaluation that systematically captures product variants, moving material flows, finite execution windows, and tight geometric tolerances when assessing generalist robot policies. To this end, they introduce LIBERO-INDUSTRY, a novel benchmark built upon LIBERO, which defines three task scenarios: Static-to-Dynamic (S2D) loading, Dynamic-to-Static (D2S) unloading, and Dynamic-to-Dynamic (D2D) transfer, with a total of 216 controllable tasks. Evaluated in simulation and via real-world experiments on an AUBO-i5H arm, policies trained with LIBERO-INDUSTRY data achieve substantially improved physical task success compared to policies trained on standard LIBERO under matched training conditions.

Critical strengths
  1. The proposed LIBERO-INDUSTRY benchmark systematically translates four industrial process constraints into controllable task conditions and measurable success criteria, with three task suites and 216 total tasks for evaluating generalist robot policies.
  2. The paper conducts experiments across five generalist robot policies in simulation and presents real-world experiments on an AUBO-i5H arm to demonstrate improved physical task success under matched training conditions.
Critical weaknesses
  1. No quantitative baseline results from approaches other than VLA/WAM policies (traditional industrial controllers, reinforcement-learning-based policies, modular perception-planning-action pipelines, etc.) are reported on LIBERO-INDUSTRY. Without these baselines, it cannot be determined whether low policy performance originates from intrinsic task difficulty or inherent limitations of the evaluated VLA/WAM generalist robot policies.
  2. The benchmark is restricted exclusively to conveyor-based pick-and-place transfer tasks. It does not incorporate other practically meaningful industrial manipulation tasks. As a result, the scope for evaluating generalist robot policies toward diverse factory-relevant operations remains limited.
Questions for authors
  1. All 216 benchmark tasks are limited to conveyor-based pick-and-place transfer. Are there inherent technical or design constraints preventing other categories of industrial manipulation tasks from being integrated into this benchmark?
  2. All five policies achieve 0% full-task success on the held-out simulation set. Is the held-out test split intentionally extremely hard, or does it have limited discriminative power for policy comparison?
  3. Real-world validation is only provided for D2S. Are S2D and D2D physically realizable on real robot hardware under current simulation definitions?
5 comments
AuthorsOpenReview ↗

Response to Reviewer GCNX [1/4] — W1

W1 — Baselines beyond VLA/WAM policies

W1: "No quantitative baseline results from approaches other than VLA/WAM policies (traditional industrial controllers, reinforcement-learning-based policies, modular perception-planning-action pipelines, etc.) are reported on LIBERO-INDUSTRY. Without these baselines, it cannot be determined whether low policy performance originates from intrinsic task difficulty or inherent limitations of the evaluated VLA/WAM generalist robot policies."

A: We evaluated a configured classical non-VLA reference on nine selected configurations spanning D2S, S2D and D2D at Levels 1–3. It combines scripted sequencing, geometric grasp/place computation and Intrinsic motion planning on Panda at 20 Hz, using privileged simulator poses, CAD geometry and contact/goal feedback. Implementation reference: libero-classical-manipulation. An AI agent used development rollouts on initial state 0 to iteratively adjust controller settings, including grasp orientation, release clearance, settling time and carrier selection. Each final pipeline was frozen and evaluated once on a reserved initial state under the original task deadline and complete-task success criteria. Deadlines measure simulated execution time, with simulation paused during planning. The full protocol and all nine results are reported in Appendix C.8.

Configuration typeTask variant (protocol ID)Complete successNative PCRT\mathrm{PCR}_TNew transfers at endpointAgent configuration minDevelopment trialsMain adjustment
D2S L1006_V5 (ID 12)1 of 1100.00%3 of 310.335Higher release clearance; shorter settling and vertical travel
D2S L2003 · Thomas/Mini Roller/School Bus · V3 (ID 16)0 of 166.70%2 of 33.212Nearest equivalent groove yaw
D2S L3004 · Thomas/Porter/Mini Roller · V3 (ID 20)0 of 150.00%1 of 28.793Thomas grasp yaw fixed at 67.5 degrees
S2D L1002 · pump bottle · V5 (ID 24)0 of 166.70%2 of 3reused; no new tuning0Reused existing controller; no additional changes
S2D L2001 · flat pump bottle · V3 (ID 28)0 of 133.30%1 of 310.075Transport settling; release height/XY; post-pick carrier selection
S2D L3001 · upright pump bottle · V3 (ID 32)0 of 10.00%0 of 310.043Side grasps; bilateral-contact approach stop
D2D L1008_V5 (ID 0)0 of 133.30%1 of 310.023Post-pick carrier selection; higher release
D2D L2001 · Black Taipan · V3/V5 (ID 4)0 of 10.00%0 of 24.582Post-pick compatible-carrier selection
D2D L3002 · Black Taipan · V5/M8 (ID 8)0 of 10.00%0 of 39.152Nearest-source ordering

SR requires all native task goals. PCRT\mathrm{PCR}_T measures new target-transfer completion and excludes pre-existing destination workpieces. Configuration time reports recorded AI-agent elapsed time; human active time was not measured.

The added results show that LIBERO-Industry is challenging but achievable. Its task conditions model industrial process requirements, including changing workpieces, material flow, finite execution windows and fixture-specific placement conditions. Case-specific configuration enables the classical reference to complete all three D2S L1 transfers in 13.95 seconds, within the 14-second deadline (Appendix C.8). Partial transfers across the other suites demonstrate further execution capability, while the remaining failures show that satisfying the combined requirements also challenges the tested classical approach.

The original-LIBERO-adapted VLA/WAM results reveal a cross-benchmark transfer gap, and the added in-domain VLA evaluation demonstrates both learnability and remaining policy limitations. After fine-tuning, π₀.₅ and OpenVLA-OFT achieve overall complete-task SRs of 13.06% and 5.56%, respectively. Level-1 SR reaches 37.50% and 16.67%, but Level-2 SR remains 1.67% and 0%, with neither policy completing a Level-3 episode (Appendix C.5). Thus, achievable execution coexists with limited reliability under the broader generalization conditions at higher levels. LIBERO-Industry makes this gap measurable and supports evaluating further training and adaptation toward reliable industrial execution.

AuthorsOpenReview ↗

Response to Reviewer GCNX [2/4] — W2 / Q1

W2 / Q1 — Task scope and extensibility

W2: "The benchmark is restricted exclusively to conveyor-based pick-and-place transfer tasks. It does not incorporate other practically meaningful industrial manipulation tasks. As a result, the scope for evaluating generalist robot policies toward diverse factory-relevant operations remains limited."

Q1: "All 216 benchmark tasks are limited to conveyor-based pick-and-place transfer. Are there inherent technical or design constraints preventing other categories of industrial manipulation tasks from being integrated into this benchmark?"

A: LIBERO-Industry focuses on workstation-level material transfer to evaluate whether generalist policies can maintain reliable completion under changing products and operating conditions. Workpiece transfer belongs to the broader handling application area, for which IFR reports approximately 280,000 new industrial robot installations in 2023 [2, slide 13]. In high-mix, low-volume production, flexibility and workcell reconfigurability are particularly important because products and production setups change [3, §4.1.2]. This motivates studying transfer reliability across industrial process conditions within a shared task family.

Our coverage of selected IFR application categories is:

IFR application categoryScope of LIBERO-Industry
Handling operations/machine tending (110)Workstation-level workpiece transfer through D2S, S2D, and D2D
Welding and soldering (160)Outside the current release
Dispensing (170)Outside the current release
Processing (190)Outside the current release
Assembling and disassembling (200)Outside the current release; fixture-constrained placement is evaluated as a transfer task

Application categories and codes follow the IFR classification [4, Table 1.2].

Regarding W2, the 216 configurations cover D2S unloading, S2D loading, and D2D transfer. D2S requires acquisition from a moving source; S2D requires placement during a destination service period and retention when transport resumes; D2D combines source-acquisition and destination-service opportunities. Together, these configurations evaluate transfer reliability under workpiece variation, material flow, execution windows, and geometric tolerance.

Regarding Q1, the conveyor focus is a deliberate scope choice for the current release, reflecting the task families we have implemented and validated. We extend beyond LIBERO framework which specifies task instructions, initial states, and goal predicates separately, providing a basis for defining additional task families [5, §4.1]. Extending the benchmark to other industrial operations would require task-specific assets and interaction models, success criteria, demonstrations, and feasibility validation, together with additional sensing or control interfaces where needed. Broader industrial manipulation tasks remain future work.

AuthorsOpenReview ↗

Response to Reviewer GCNX [3/4] — Q2

Q2 — Held-out difficulty and policy discrimination

"All five policies achieve 0% full-task success on the held-out simulation set. Is the held-out test split intentionally extremely hard, or does it have limited discriminative power for policy comparison?"

A: In-domain training produces nonzero held-out complete-task success and distinguishes the two tested policies. The five zero-SR results concern original-LIBERO-adapted checkpoints evaluated on Panda without LIBERO-Industry training. We separately fine-tuned π0.5\pi_{0.5} and OpenVLA-OFT for 30,000 updates using the same 1,412 Aubo simulation demonstrations covering 147/180 training configurations.

Both adapted policies were evaluated in Aubo simulation on all 36 held-out configurations, with ten saved initial states each and the prescribed task-specific deadlines: 360 episodes per policy. Level and suite–level summaries describe the same episodes.

SR measures complete-task success, with successes/episodes in parentheses. PCRT\mathrm{PCR}_T measures target-transfer completion; PCRS\mathrm{PCR}_S measures overall slotting, including pre-existing valid destination workpieces. Both PCR metrics average all episodes, including failures. OFT denotes OpenVLA-OFT; all scores are percentages.

Evaluation groupπ0.5\pi_{0.5} SRπ0.5\pi_{0.5} PCR$_T$π0.5\pi_{0.5} PCR$_S$OFT SROFT PCR$_T$OFT PCR$_S$
Level 1 (all suites)37.50 (45/120)66.3966.3916.67 (20/120)49.4449.44
Level 2 (all suites)1.67 (2/120)8.618.610.00 (0/120)2.222.22
Level 3 (all suites)0.00 (0/120)1.3935.530.00 (0/120)0.0034.92
D2D–L162.50 (25/40)85.8385.8342.50 (17/40)67.5067.50
D2D–L25.00 (2/40)20.8320.830.00 (0/40)5.005.00
D2D–L30.00 (0/40)0.0040.250.00 (0/40)0.0040.75
D2S–L110.00 (4/40)46.6746.670.00 (0/40)35.8335.83
D2S–L20.00 (0/40)1.671.670.00 (0/40)0.000.00
D2S–L30.00 (0/40)0.0033.330.00 (0/40)0.0032.50
S2D–L140.00 (16/40)66.6766.677.50 (3/40)45.0045.00
S2D–L20.00 (0/40)3.333.330.00 (0/40)1.671.67
S2D–L30.00 (0/40)4.1733.000.00 (0/40)0.0031.50
Overall13.06 (47/360)25.4636.845.56 (20/360)17.2228.86

The overall SR difference is 7.50 percentage points, with the largest separation at Level 1 (37.50% versus 16.67%). Partial completion provides additional resolution: OpenVLA-OFT achieves 35.83% target-transfer completion on D2S–L1 despite zero complete-task SR. Together, these results demonstrate achievable held-out execution and support comparison of the two adapted policies across graded task requirements, with substantial room for improvement at the higher levels.

Reviewer_XpcEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LIBERO-INDUSTRY, an extension of LIBERO for material transfer under conveyor motion, finite interaction windows, and placement constraints. It contains 216 task configurations across static-to-dynamic, dynamic-to-static, and dynamic-to-dynamic suites. Experiments evaluate five LIBERO-adapted policies, analyze a cumulative stress progression, and compare two simulation-real co-training settings on a physical D2S workcell.

Critical strengths
  1. The benchmark explicitly specifies transport schedules, completion predicates, and episode horizons. These provide a systematic setting for studying manipulation under motion and timing constraints.

  2. The physical experiments provide encouraging evidence for the proposed simulation data. At 0.05 m/s, target-transfer completion improves from 4.2% to 23.6%, using the same physical demonstrations and optimization budget (Section 4.4).

Critical weaknesses
  1. The task contribution is incremental. The 216 tasks combine object, speed, direction, and difficulty variations within three source-destination motion settings (Table 2). All retain a repeated pick-and-place objective. These variations increase execution difficulty, but behavioral coverage remains narrow. Relative to DOM/DynamicVLA and DOMINO, the distinct evaluation capability is not sufficiently demonstrated. The industrial-precision motivation also needs stronger support: fixtures follow a 1 cm clearance rule, and moving-slot transport uses kinematic locking (Appendices B.1 and B.5.2). The effect of these choices on physical difficulty is not evaluated.

  2. The main evaluation provides little separation between policies. All five checkpoints are adapted to original LIBERO, without further training on LIBERO-INDUSTRY (Appendix C.1). Every method obtains 0/360 complete-task successes, with almost no target transfers (Table 3). This demonstrates a substantial transfer gap, but leaves the benchmark's value for comparing learned capabilities unclear. No policy trained on the proposed simulation training split is evaluated on its held-out tasks. The cumulative progression in Figure 3 changes several factors together, so it also cannot isolate failures caused by the proposed process constraints.

Questions for authors

The following additional evidence could materially change my assessment:

  1. Can the authors report held-out simulation performance after training a strong baseline on the 180-task training split, including results by difficulty level?

  2. What evidence links the simulated precision settings to physical placement difficulty, given the 1 cm fixture-clearance rule and kinematic slot locking? A sensitivity analysis or physical results broken down by difficulty level would help address this concern.

6 comments
AuthorsOpenReview ↗

Response to Reviewer XpcE [1/5] — W1(a): Task scope

We sincerely thank you for your time devoted to reviewing this paper and your constructive comments. Please see our detailed responses to your comments and suggestions below.

W1(a) — Task scope and contribution

W1(a): “The task contribution is incremental. The 216 tasks combine object, speed, direction, and difficulty variations within three source-destination motion settings (Table 2). All retain a repeated pick-and-place objective. These variations increase execution difficulty, but behavioral coverage remains narrow.”

A: Our work focuses on evaluating generalist policies for industrial workpiece transfer, using repeated pick-and-place tasks that model unloading, loading, and conveyor-to-conveyor transfer. These operations fall within the broader Handling category, which accounted for approximately 280,000 new industrial robot installations in 2023, about 52% of the global total [A, slide 13]. These transfer operations are considered in the context of high-mix, low-volume production, the manufacturing setting motivating our benchmark, where changing products and setups create a need for flexibility and workcell reconfigurability [B, §4.1.2]. We translate these industrial requirements into the following benchmark conditions and evaluation questions:

Industrial requirement and evidenceBenchmark conditionsEvaluation question
Workpiece variation. Flexible manufacturing involves changes in products, feeding locations, and pick/place parameters [C, §1].Different workpieces and corresponding receiving fixturesCan the policy complete the required transfers when workpieces change?
Material flow. Material transport and production-line organization provide the engineering basis for workstation operation [D, Ch. 10 and §16.1].Static, continuous, and intermittent source/destination motionCan the policy coordinate acquisition and delivery under these motion conditions?
Execution windows. Moving conveyor items provide limited pickup time [E].Acquisition/delivery opportunities, execution budgets, and target-transfer countsCan the policy complete all target transfers within the specified opportunities and budget?
Geometric tolerance. Handling new parts requires placing them correctly into processing equipment [B, §3.2.2].Fixture-specific position, orientation, seating, and retention conditionsDoes placement satisfy the fixture-specific acceptance conditions?

The 216 entries are task configurations, rather than distinct manipulation behaviors (§3.4 and Table 2). D2S evaluates timely acquisition from a moving source; S2D evaluates delivery during a destination service period and valid placement after transport resumes; D2D combines acquisition and delivery opportunities. Workpiece and fixture variations further change the requirements for acceptable placement. Complete-task SR requires completion of the entire specified workload. These configurations therefore evaluate transfer reliability across industrial operating conditions, while broader industrial skills remain outside the present scope.

References [A]–[M] are listed in Part 5.

AuthorsOpenReview ↗

Response to Reviewer XpcE [2/5] — W1(b): Benchmark comparison

W1(b) — Comparison with dynamic benchmarks

W1(b): “Relative to DOM/DynamicVLA and DOMINO, the distinct evaluation capability is not sufficiently demonstrated.”

A: LIBERO-Industry distinctly evaluates the coordination of source-acquisition opportunities, destination-service opportunities, and fixture-specific placement conditions during repeated workpiece transfer. The differences concern the following task requirements.

First, DOM evaluates general dynamic manipulation, including responses to changing object motion and repeated collection, while DOMINO evaluates dynamic interception and tracking (DOM [F, §4.2 and Appendix D]; DOMINO [G, §2.3]). In contrast, LIBERO-Industry evaluates coordinated acquisition and delivery under separately specified source-arrival and destination-service schedules, an important capability in industrial manipulation. Conveyor speed and workpiece pitch define source-arrival intervals, while alternating transport and stationary dwell phases define destination-service opportunities. D2D requires coordinating both opportunities for each required transfer. Missing either opportunity can leave a transfer incomplete as workpieces leave the acquisition region or receiving fixtures resume transport (§§3.2–3.4; Tables 9–10).

Second, DOMINO includes general grasping and placement tasks, such as grasping a moving mug and hanging it on a rack [H]. In contrast, LIBERO-Industry additionally evaluates fine-grained placement requirements under a shared transfer objective. Trays, object-matched grooves, and inlays define explicit levels of receiving conditions with corresponding placement predicates. These requirements are relevant to machine tending, where workpieces must be placed correctly into processing equipment [B, §3.2.2].

Evaluation requirementDynamicVLADOMINOPhysManiLIBERO-Industry
Dynamic manipulation within a task time budget✓✓✓ (a)✓
Placement onto a moving receiving target×✓✓✓
Source-arrival intervals defined by conveyor speed and workpiece pitch×××✓
Intermittent destination service with explicit transport and dwell phases×××✓
Transfer jointly constrained by source-acquisition and destination-service opportunities×××✓
Graded receiving fixtures with corresponding acceptance conditions×××✓
Completion requirements covering multiple target objects✓✓ (b)×✓

✓ denotes explicit coverage in reported tasks. × denotes that the specific setting was not identified in the examined task definitions and release materials. Coverage includes reported simulation and real-robot tasks.

(a) PhysMani’s explicitly reported time budget is the 60-second limit used in its real-robot evaluation [I].

(b) DOMINO includes bulk emptying of five objects, with success checked across the object set. LIBERO-Industry uses repeated individual transfers [H].

Additional comparison with PhysMani. PhysMani covers general dynamic manipulation tasks, such as moving receptacles, dynamic insertion, and conveyor tasks [I, §4 and Appendix A.6]. By comparison, LIBERO-Industry evaluates repeated transfers jointly constrained by source-arrival schedules, intermittent destination service, and graded fixture acceptance conditions. This joint process setting is not reported in PhysMani. Successful execution requires coordinating acquisition and delivery opportunities while satisfying placement conditions for all target transfers. The detailed comparison of reported task definitions is now included in Appendix F.3.

AuthorsOpenReview ↗

Response to Reviewer XpcE [3/5] — W1(c)/Q2: Precision and locking

W1(c) / Q2 — Placement precision and physical difficulty

W1(c): “The industrial-precision motivation also needs stronger support: fixtures follow a 1 cm clearance rule, and moving-slot transport uses kinematic locking (Appendices B.1 and B.5.2). The effect of these choices on physical difficulty is not evaluated.”

Q2: “What evidence links the simulated precision settings to physical placement difficulty, given the 1 cm fixture-clearance rule and kinematic slot locking? A sensitivity analysis or physical results broken down by difficulty level would help address this concern.”

A: Precision setting. The 1 cm rule specifies a geometric opening allowance, distinct from demonstrated robot positioning accuracy or repeatability [J]. We added a clearance sensitivity analysis to quantify its effect on simulated placement difficulty, with results summarized in the accompanying table and full details in Appendix C.4 (“Fixture-Clearance Sensitivity”). Industrial allowances are task-specific; one brake-pad placement system uses slots 2 mm wider than its plates [K, §6].

We use the Akita black bowl, an original-LIBERO object, to reduce confounding from unfamiliar workpieces for the LIBERO-fine-tuned policy. We test grooves and inlays at 2, 5, 10, 20 and 50 mm total extra width, holding bowl geometry, wall height (25.27 mm) and contact parameters fixed within each fixture family.

  • Geometry/passive capture, without a policy: The same 20 prescribed poses per width comprise centred yaw controls and ±2/4/8/16 mm offsets along X or Y. Fixed-X/Y collision queries hold horizontal position and orientation constant during descent. Passive capture restores the same start, releases the bowl 30 mm above the inlet and simulates 2 s, requiring valid seating in the final five samples.
  • Policy evaluation: Frozen original-LIBERO-fine-tuned π₀.₅/Panda uses four static layouts × 20 matched trials per width and fixture, with 400 actions at 20 Hz. Complete-task success requires five consecutive valid released, floor-supported, contained, upright (≤10°) and stable seating samples; yaw is unrestricted.
Extra widthFixed X/Y (both)Groove captureInlay captureGroove SR (%)Inlay SR (%)
2 mm4/2012/2014/2013.7517.50
5 mm8/2014/2014/2015.0020.00
10 mm12/2016/2016/2026.2525.00
20 mm16/2017/2019/2041.2553.75
50 mm20/2020/2020/2083.7577.50

Fixed-X/Y and capture counts describe 20 prescribed cases, not random-trial probabilities. SR is complete-task success over 80 policy episodes per width and fixture.

The sweeps show that smaller clearances admit fewer tested approaches and reduce complete-task policy success within both fixture families. Passive recovery explains why an initially blocked fixed-X/Y approach can still seat successfully; initial offsets therefore should not be interpreted as final placement accuracy. The two fixture families use different reference collision profiles, so this comparison does not isolate fixture shape. These results quantify static simulation sensitivity, without validating hardware accuracy or establishing a uniquely optimal 1 cm setting. The effect of kinematic locking during moving-slot transport remains unquantified by these tests. Appendix C.4 provides the construction definitions, protocols, scoring thresholds and example schematics/simulation frames.

Locking: motivation and approximation. Contact-model assumptions and numerical solvers can affect simulated frictional behaviour [M], and MuJoCo documents gradual slip associated with its contact model [L]. We use kinematic locking as an idealized model of retained fixture transport. It maintains the workpiece’s pose relative to the moving carrier, suppressing subsequent sliding and rotation.

Locking example and scope. Appendix B.5.4 (“Transport-Assistance Illustration and Limitations”) presents the mechanism using a preloaded Industry workpiece without a learned policy. Native assistance maintained the three prescribed preloaded object–carrier configurations. However, disabling assistance also removes other transport corrections, and all three controls lost seating before carrier motion. Two elevated groove/inlay controls additionally triggered locking despite floor gaps of 19.5/23.7 mm, exposing incomplete seating-depth checks. This diagnostic therefore illustrates implementation behaviour. The isolated effect of locking on transport difficulty and its physical validity require further validation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for the valuable time devoted to reviewing this paper and the helpful comments. The paper and the appendix are revised following the suggestions from all the reviewers. The revisions are in blue color.

On the one hand, we have revised the paper to improve its clarity and readability:

  • In Section 2, we clarify the comparison with DOM/DynamicVLA and DOMINO, with a detailed task-definition comparison in Appendix F.3 (XpcE).
  • In Section 4.2, we clarify PCR_S by reporting the 12.31% composition-derived initial-scene reference and distinguishing it from new target transfers (4Tq5).
  • In Sections 4.1–4.2, we add in-domain adaptation settings and held-out results for π₀.₅ and OpenVLA-OFT, with extended results in Appendix C.5 (XpcE, GCNX, 4Tq5).
  • In Section 4.3, we clarify that D0–D6 is a cumulative diagnostic progression and remove causal interpretations of individual constraints (XpcE, 4Tq5).
  • In Section 4.4, we clarify the physical evaluation protocol, training conditions, and trial and placement denominators (4Tq5).

On the other hand, we have revised the appendix to provide additional evaluation details:

  • In Appendix B.4, we specify suite-specific episode horizons at 20 Hz, distinguish nominal task requirements from achieved policy throughput, and replace unfinished release wording and benchmark-takt labels with evaluated timing parameters and source-arrival intervals (4Tq5).
  • In Appendix B.5.4, we add a transport-assistance illustration and explain why it does not isolate the effect of locking; we also report incomplete seating-depth checks and the limits of physical validation (XpcE).
  • In Appendix C.1, we report the released checkpoint identifiers and the model-specific inference and action-execution settings (4Tq5).
  • In Appendix C.2, we add the task-composition table and explain how the 12.31% initial-scene reference is calculated across the 36 held-out configurations; this reference is not a measured no-action result (4Tq5).
  • In Appendix C.3, we specify failure-classification precedence and report D0–D6 success counts, evaluation denominators, and post-hoc 95% Wilson intervals, together with representative failure trajectories (4Tq5).
  • In Appendix C.4, we add geometric-feasibility, passive-capture, and frozen-policy clearance results, and distinguish fixture clearance from robot positioning accuracy (XpcE).
  • In Appendix C.5, we report the in-domain training data and coverage for π₀.₅ and OpenVLA-OFT, held-out results by suite, difficulty level, and configured conveyor velocity, and achieved simulated target-transfer throughput (XpcE, GCNX, 4Tq5).
  • In Appendix C.6, we add the C1–C4 targeted comparisons of workpiece variation, endpoint motion, execution windows, and fixture clearance, together with their controls, action budgets, and scoring definitions (XpcE, 4Tq5).
  • In Appendix C.7, we add two scoring examples showing how PCR_T counts new target transfers and PCR_S additionally includes pre-existing valid destination workpieces; the training demonstration is identified separately from held-out evaluation evidence (4Tq5).
  • In Appendix C.8, we add the configured classical non-VLA reference on nine selected configurations, reporting complete-task success and target-transfer completion together with its privileged-input, one-state, and paused-planning limitations (GCNX).
  • In Appendix D, we clarify the Panda and Aubo embodiments, the policy interface, and the trajectory-conversion and replay-validated recovery procedure (4Tq5).
  • In Appendices E.2–E.4, we detail physical-data collection and splits and the sim–real co-training settings, including the shared physical demonstrations and unequal simulation-data volumes in S1 and S2 (4Tq5).
  • In Appendices F.1–F.3, we clarify the scope of the 216 configurations, extension requirements, and limits of D2S-only physical validation, and add a detailed comparison with DOM/DynamicVLA, DOMINO, and PhysMani (XpcE, GCNX).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.