- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces LIBERO-MAX, a benchmark for evaluating robot policy robustness under environmental changes. The benchmark constructs 8,000 paired Base/Dynamic cases across eight types of changes while keeping the pre-event execution history fixed, and evaluates 14 robot policies including VLA, VLA+WAM, and WAM models. The author show that current policies suffer substantial performance degradation after online changes and provide further analyses of policy families, intervention timing, and query cadence, along with a simple recovery study using image restoration.
Critical strengths
-
The paper studies an interesting problem that is less explored in existing robot benchmarks: whether robot policies can adapt when the environment changes during execution. This provides a useful perspective beyond traditional evaluations that mainly consider changes before execution.
-
The paper provides a large-scale experimental evaluation across 14 robot policies and multiple types of environmental changes. The extensive experiments and additional analyses help reveal different failure patterns of current robot policies and provide a useful understanding of their limitations under dynamic conditions.
Critical weaknesses
-
The first concern is that each Base case is paired with only one Dynamic event. Although the benchmark covers eight types of changes, it is unclear why the event is selected for each case in the current version of the paper. This construction may introduce bias, because different tasks may have different sensitivities to different changes. Therefore, the evaluation may depend on the specific task-event combination and not fully reflect the policy’s adaptation ability. If this is the case, I am concerned that the benchmark comparison may be affected by the specific task-event assignment.
-
Another concern is whether the designed interventions and their magnitudes realistically reflect changes encountered in real robotic scenarios. Although these events may exist in real environments and the authors check the feasibility of these interventions in simulation, I am concerned that some simulated changes may be too abrupt or have unrealistic magnitudes, making them closer to artificial perturbations.
Questions for authors
I appreciate the effort and contribution of this work. My main concerns are the above mentioned weaknesses, and clarification on these points would help me better understand the benchmark.
-
For the Base/Dynamic case construction, each Base case is paired with only one Dynamic event. Could the authors clarify how the event type is selected for each case and whether this may introduce bias due to different task-event sensitivities?
-
For the simulated interventions, could the authors further explain whether the type, magnitude, and implementation of these changes are representative of real-world robotic scenarios? In particular, how do the authors ensure that the simulated changes are not overly abrupt or artificial?
If the authors can provide explanations for these concerns, it would help address my concerns and I will increase my rating.
8 comments
Response to Reviewer o9tz [2/3]
W2 and Q2: Realism of the Type, Magnitude, and Onset of the Changes
Each change corresponds to a common real situation, and its magnitude is moderate in physical units. Our new experiments show that the findings also hold for milder and gradual versions of the same changes.
Type and magnitude. The table gives the two frozen settings of each event in physical units, together with a real world counterpart.
| Event | Two frozen settings, one per case | Real world counterpart |
|---|---|---|
| Target relocation | Horizontal displacement of 6 or 12 cm in one of 8 directions, on the same support | An object nudged or repositioned by a person during a reach |
| Receptacle relocation | The same displacement, applied to the destination | A plate or basket moved while the robot approaches |
| Camera shift | External camera translated by (+6, 0, +2) cm with yaw +18° and vertical FOV +12°, or by (−6, +3, 0) cm with yaw −22° and FOV −12° | A bumped or remounted camera, or a loose mount |
| Illumination switch | Scene lights scaled by 0.2 or 1.8, with camera exposure scaled by 0.75 or 1.25 | Lights switched off or on, a window blind, or a change in sunlight |
| Sensor noise onset | Gaussian noise with SD 24 or 36 on 8 bit images, plus a moving black occluder covering 8% or 16% of the image | Sensor degradation, or a smudged or partially occluded lens |
| Visual theme switch | RGB channel permutation with channel gains, for example BRG with gains of 0.70, 1.15, and 1.30 | A change in white balance or color temperature |
| Distractor burst | Five task scene objects placed 9 to 17 cm from the target | A person setting items down near the workspace |
| Obstacle insertion | One object placed 14 cm from the target on the approach side, for example a soup can of 6.2 × 7.6 × 6.2 cm or a mug of 12.5 × 9.6 × 10.6 cm | An item placed in the path of the robot |
These are reproducible stress levels chosen to be clearly visible while physically valid. We do not claim that they match the frequency of real deployment events.
Milder changes on the same cases. To test whether the findings depend on the chosen magnitude, we scaled each event to 25% and 50% of its magnitude, holding the case, initial state, executed prefix, and direction fixed. We use the 800 LIBERO-MAX Lite cases with π0.5, X-VLA, and HiMem-WAM, the policies of the camera controls in the paper. The table reports the change in success rate relative to Base, in percentage points, with 99 or 100 pairs per cell.
| Event | Strength | π0.5 | X-VLA | HiMem-WAM |
|---|---|---|---|---|
| Target relocation | 25%, 1.5 to 3 cm | −1.0 | −2.0 | −1.0 |
| Target relocation | 50%, 3 to 6 cm | −13.0 | −18.0 | −15.2 |
| Target relocation | 100%, 6 to 12 cm | −28.0 | −45.0 | −33.0 |
| Receptacle relocation | 25%, 1.5 to 3 cm | −1.0 | −3.0 | +2.0 |
| Receptacle relocation | 50%, 3 to 6 cm | −7.0 | −19.0 | −10.0 |
| Receptacle relocation | 100%, 6 to 12 cm | −15.0 | −45.0 | −30.0 |
| Sensor noise onset | 25%, SD 6 to 9, 2% to 4% occluded | −3.0 | −8.0 | −5.1 |
| Sensor noise onset | 50%, SD 12 to 18, 4% to 8% occluded | −6.0 | −12.0 | −10.1 |
| Sensor noise onset | 100%, SD 24 to 36, 8% to 16% occluded | −28.0 | −34.0 | −27.0 |
Losses decrease smoothly with the magnitude but remain clear at half strength. A target relocation of only 3 to 6 cm still reduces success by 13.0, 18.0, and 15.2 points for π0.5, X-VLA, and HiMem-WAM. Quarter strength relocations of 1.5 to 3 cm, which are within typical grasp tolerance, have little effect. We also varied the illumination strength and the number of distractors. Illumination stays benign at every strength, and distractor losses stay small at every count, as in the main results. The findings therefore do not rely on extreme magnitudes.
Response to Reviewer o9tz [3/3]
Gradual onset. The main track applies each change in one simulator update. This is realistic for observation changes, since a light switch or a camera bump is effectively instantaneous at the 20 Hz control rate, but abrupt for objects that are moved or placed. We therefore added gradual versions of these physical events, where the same final change unfolds over 1.5 s from the same trigger, as one point on the continuum between an instantaneous change and no change. The target or destination slides at 4 to 8 cm/s, as when a person pushes it. The obstacle is lowered into the path from 10 cm above, as when a person sets it down. The five distractors are set down one after another. The table reports the change in success rate relative to Base, in percentage points, with 99 or 100 pairs per cell.
| Event | Onset | π0.5 | X-VLA | HiMem-WAM |
|---|---|---|---|---|
| Target relocation | Abrupt | −28.0 | −45.0 | −33.0 |
| Target relocation | Gradual | −27.0 | −40.0 | −26.3 |
| Receptacle relocation | Abrupt | −15.0 | −45.0 | −30.0 |
| Receptacle relocation | Gradual | −11.0 | −47.0 | −28.0 |
| Obstacle insertion | Abrupt | −7.0 | −5.0 | +1.0 |
| Obstacle insertion | Gradual | −10.0 | −4.0 | −5.0 |
| Distractor burst | Abrupt | −7.0 | −4.0 | −14.0 |
| Distractor burst | Gradual | −7.0 | +1.0 | −10.0 |
Gradual onset produces losses of similar size, and none of the 12 differences from abrupt onset is significant under McNemar's test. The findings are therefore not an artifact of abrupt changes. A slower movement acts like a smaller displacement when the robot arrives, and the milder changes above show that the loss shrinks continuously with the displacement.
Implementation validity. Every case passes automated simulator checks for a stable state, valid placement on the support and within the workspace, visibility, a nonzero change in the observation, and no new unrelated contact. Across the original runs of four policies, all 11,051 logged geometric events matched their registered displacements or placements.
We will add the physical parameter table to Appendix B and the severity and gradual onset results to new appendices. We will include all these results and revisions in the revised PDF, which we will upload soon during the discussion period. We hope that these results address the concerns of the reviewer.
Response to Reviewer o9tz [1/3]
We thank the reviewer for recognizing that whether robot policies adapt to changes during execution is an interesting and underexplored problem, which complements evaluations that focus on changes before execution, and for appreciating the large scale evaluation across 14 policies and multiple types of changes. We address the two concerns below.
W1 and Q1: Event Assignment and Task and Event Interactions
The event of each case is assigned by a fixed and balanced design before any policy is run. A new fully crossed experiment shows that this assignment does not bias the reported losses or the policy ranking.
How events are assigned. The assignment never uses outcomes. Each selection cell combines a source category, an event type, and one of two parameter settings. For LIBERO-Plus, each of the 7 × 8 × 2 cells receives 50 tasks, which gives 5,600 cases. For LIBERO-PRO, each of the 10 × 8 × 2 cells receives 15 task and initial state configurations, which gives 2,400 cases. A task is eligible for an event only if the event can be applied validly. For example, relocation requires a movable target on a planar support, and obstacle insertion requires an unused object on the same support. Every policy is evaluated on the same 8,000 cases, so all policies face exactly the same pairs of task and event. We agree that this balancing alone does not rule out interactions between tasks and events, so we tested them directly.
A fully crossed experiment. We drew 60 LIBERO-Plus tasks, stratified by suite and without looking at outcomes, from the 2,039 tasks that are eligible for all eight events. We applied every event to every task with the official case generator, which gives 60 × 8 = 480 crossed cases. Each pair of task and event receives one case with one of the two parameter settings of the benchmark, and the two settings are split about evenly for each event. For the originally assigned event of each task, the generator reproduces the released benchmark case exactly. We evaluated seven policies covering the VLA, VLA+WAM, and WAM families, each with matched Base rollouts.
The table reports success rates and the gap in percentage points. The event profile correlation is the Spearman correlation between the losses of the eight events on the crossed cases and in the main benchmark, where 1 means the same ordering. The last column compares the loss on the originally assigned event of each task with the loss of the same event applied to the other tasks, where a positive value means that the assigned event is less damaging.
The results answer the question at three levels.
The benchmark comparison therefore does not depend on the specific pairing of tasks and events. We will move the assignment rules from Appendix B.2 into Section 3.2 and add the crossed experiment and the coverage matrix of tasks and events to a new appendix.