acceptodds
Under review as a conference paper at ICLR 2027

LIBERO-MAX: Do Robot Policies Adapt When the World Changes?

12 views

Abstract

Robots must often continue a task after a target moves, the viewpoint shifts, or an obstacle appears, even though their earlier observations and committed actions reflect the previous scene. Many simulation robustness benchmarks fix external conditions at reset, leaving this temporal challenge underexamined. We introduce LIBERO-MAX, a benchmark of 8,000 paired cases spanning eight types of changes to geometry, observations, appearance, clutter, and paths. Each pair compares task execution with and without a mid-task event, holding the task, initial state, policy seed, and pre-event action sequence fixed. This controlled comparison distinguishes event-associated regressions from failures already present without the change. Across fourteen current VLA, hybrid, and world-action policies, events reduce success by 11.0–25.7 percentage points. Event profiles reveal shared vulnerabilities to geometry and observation changes, while policy-family rankings interleave. Camera controls show that robustness reflects both competence under the changed conditions and the trajectory from which they are encountered; varying query cadence does not eliminate the gap. Together, the paired protocol and temporal diagnostics establish LIBERO-MAX as a reproducible testbed for diagnosing failures under mid-execution changes and measuring progress toward robot policies that remain effective as the world changes.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Models, VLA

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_o9tzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LIBERO-MAX, a benchmark for evaluating robot policy robustness under environmental changes. The benchmark constructs 8,000 paired Base/Dynamic cases across eight types of changes while keeping the pre-event execution history fixed, and evaluates 14 robot policies including VLA, VLA+WAM, and WAM models. The author show that current policies suffer substantial performance degradation after online changes and provide further analyses of policy families, intervention timing, and query cadence, along with a simple recovery study using image restoration.

Critical strengths
  1. The paper studies an interesting problem that is less explored in existing robot benchmarks: whether robot policies can adapt when the environment changes during execution. This provides a useful perspective beyond traditional evaluations that mainly consider changes before execution.

  2. The paper provides a large-scale experimental evaluation across 14 robot policies and multiple types of environmental changes. The extensive experiments and additional analyses help reveal different failure patterns of current robot policies and provide a useful understanding of their limitations under dynamic conditions.

Critical weaknesses
  1. The first concern is that each Base case is paired with only one Dynamic event. Although the benchmark covers eight types of changes, it is unclear why the event is selected for each case in the current version of the paper. This construction may introduce bias, because different tasks may have different sensitivities to different changes. Therefore, the evaluation may depend on the specific task-event combination and not fully reflect the policy’s adaptation ability. If this is the case, I am concerned that the benchmark comparison may be affected by the specific task-event assignment.

  2. Another concern is whether the designed interventions and their magnitudes realistically reflect changes encountered in real robotic scenarios. Although these events may exist in real environments and the authors check the feasibility of these interventions in simulation, I am concerned that some simulated changes may be too abrupt or have unrealistic magnitudes, making them closer to artificial perturbations.

Questions for authors

I appreciate the effort and contribution of this work. My main concerns are the above mentioned weaknesses, and clarification on these points would help me better understand the benchmark.

  1. For the Base/Dynamic case construction, each Base case is paired with only one Dynamic event. Could the authors clarify how the event type is selected for each case and whether this may introduce bias due to different task-event sensitivities?

  2. For the simulated interventions, could the authors further explain whether the type, magnitude, and implementation of these changes are representative of real-world robotic scenarios? In particular, how do the authors ensure that the simulated changes are not overly abrupt or artificial?

If the authors can provide explanations for these concerns, it would help address my concerns and I will increase my rating.

8 comments
AuthorsOpenReview ↗

Response to Reviewer o9tz [1/3]

We thank the reviewer for recognizing that whether robot policies adapt to changes during execution is an interesting and underexplored problem, which complements evaluations that focus on changes before execution, and for appreciating the large scale evaluation across 14 policies and multiple types of changes. We address the two concerns below.

W1 and Q1: Event Assignment and Task and Event Interactions

The event of each case is assigned by a fixed and balanced design before any policy is run. A new fully crossed experiment shows that this assignment does not bias the reported losses or the policy ranking.

How events are assigned. The assignment never uses outcomes. Each selection cell combines a source category, an event type, and one of two parameter settings. For LIBERO-Plus, each of the 7 × 8 × 2 cells receives 50 tasks, which gives 5,600 cases. For LIBERO-PRO, each of the 10 × 8 × 2 cells receives 15 task and initial state configurations, which gives 2,400 cases. A task is eligible for an event only if the event can be applied validly. For example, relocation requires a movable target on a planar support, and obstacle insertion requires an unused object on the same support. Every policy is evaluated on the same 8,000 cases, so all policies face exactly the same pairs of task and event. We agree that this balancing alone does not rule out interactions between tasks and events, so we tested them directly.

A fully crossed experiment. We drew 60 LIBERO-Plus tasks, stratified by suite and without looking at outcomes, from the 2,039 tasks that are eligible for all eight events. We applied every event to every task with the official case generator, which gives 60 × 8 = 480 crossed cases. Each pair of task and event receives one case with one of the two parameter settings of the benchmark, and the two settings are split about evenly for each event. For the originally assigned event of each task, the generator reproduces the released benchmark case exactly. We evaluated seven policies covering the VLA, VLA+WAM, and WAM families, each with matched Base rollouts.

The table reports success rates and the gap in percentage points. The event profile correlation is the Spearman correlation between the losses of the eight events on the crossed cases and in the main benchmark, where 1 means the same ordering. The last column compares the loss on the originally assigned event of each task with the loss of the same event applied to the other tasks, where a positive value means that the assigned event is less damaging.

PolicyBase success rateDynamic success rateGapEvent profile correlation with the main benchmarkAssigned event minus the same event on other tasks
MolmoAct290.475.2−15.20.85+5.0
GR00T N1.784.257.1−27.10.87+1.7
OpenVLA-OFT79.456.7−22.70.79−6.5
VLA-JEPA89.667.9−21.70.77+2.8
π0.595.079.8−15.20.81+7.2
X-VLA70.242.1−28.10.90+6.1
HiMem-WAM89.273.5−15.60.90+4.3

The results answer the question at three levels.

  1. The ordering of events is preserved. For every policy, the losses of the eight events on the crossed cases correlate with those in the main benchmark, with a correlation between 0.77 and 0.90.
  2. The assignment does not inflate the losses. Each value in the last column is based on only 60 assigned pairs, and none is significantly different from zero under a bootstrap over tasks. Pooled over the seven policies, the difference is 3.0 points and also not significant. If anything, the assigned events are slightly less damaging, so the frozen assignment does not inflate the reported losses.
  3. The policy ranking is preserved. Ranking the seven policies by Dynamic success on the crossed cases reproduces the order of the main benchmark with a Kendall rank correlation of 0.90. The only swap is between π0.5 and MolmoAct2, which differ by 1.2 points in the main benchmark.

The benchmark comparison therefore does not depend on the specific pairing of tasks and events. We will move the assignment rules from Appendix B.2 into Section 3.2 and add the crossed experiment and the coverage matrix of tasks and events to a new appendix.

AuthorsOpenReview ↗

Response to Reviewer o9tz [2/3]

W2 and Q2: Realism of the Type, Magnitude, and Onset of the Changes

Each change corresponds to a common real situation, and its magnitude is moderate in physical units. Our new experiments show that the findings also hold for milder and gradual versions of the same changes.

Type and magnitude. The table gives the two frozen settings of each event in physical units, together with a real world counterpart.

EventTwo frozen settings, one per caseReal world counterpart
Target relocationHorizontal displacement of 6 or 12 cm in one of 8 directions, on the same supportAn object nudged or repositioned by a person during a reach
Receptacle relocationThe same displacement, applied to the destinationA plate or basket moved while the robot approaches
Camera shiftExternal camera translated by (+6, 0, +2) cm with yaw +18° and vertical FOV +12°, or by (−6, +3, 0) cm with yaw −22° and FOV −12°A bumped or remounted camera, or a loose mount
Illumination switchScene lights scaled by 0.2 or 1.8, with camera exposure scaled by 0.75 or 1.25Lights switched off or on, a window blind, or a change in sunlight
Sensor noise onsetGaussian noise with SD 24 or 36 on 8 bit images, plus a moving black occluder covering 8% or 16% of the imageSensor degradation, or a smudged or partially occluded lens
Visual theme switchRGB channel permutation with channel gains, for example BRG with gains of 0.70, 1.15, and 1.30A change in white balance or color temperature
Distractor burstFive task scene objects placed 9 to 17 cm from the targetA person setting items down near the workspace
Obstacle insertionOne object placed 14 cm from the target on the approach side, for example a soup can of 6.2 × 7.6 × 6.2 cm or a mug of 12.5 × 9.6 × 10.6 cmAn item placed in the path of the robot

These are reproducible stress levels chosen to be clearly visible while physically valid. We do not claim that they match the frequency of real deployment events.

Milder changes on the same cases. To test whether the findings depend on the chosen magnitude, we scaled each event to 25% and 50% of its magnitude, holding the case, initial state, executed prefix, and direction fixed. We use the 800 LIBERO-MAX Lite cases with π0.5, X-VLA, and HiMem-WAM, the policies of the camera controls in the paper. The table reports the change in success rate relative to Base, in percentage points, with 99 or 100 pairs per cell.

EventStrengthπ0.5X-VLAHiMem-WAM
Target relocation25%, 1.5 to 3 cm−1.0−2.0−1.0
Target relocation50%, 3 to 6 cm−13.0−18.0−15.2
Target relocation100%, 6 to 12 cm−28.0−45.0−33.0
Receptacle relocation25%, 1.5 to 3 cm−1.0−3.0+2.0
Receptacle relocation50%, 3 to 6 cm−7.0−19.0−10.0
Receptacle relocation100%, 6 to 12 cm−15.0−45.0−30.0
Sensor noise onset25%, SD 6 to 9, 2% to 4% occluded−3.0−8.0−5.1
Sensor noise onset50%, SD 12 to 18, 4% to 8% occluded−6.0−12.0−10.1
Sensor noise onset100%, SD 24 to 36, 8% to 16% occluded−28.0−34.0−27.0

Losses decrease smoothly with the magnitude but remain clear at half strength. A target relocation of only 3 to 6 cm still reduces success by 13.0, 18.0, and 15.2 points for π0.5, X-VLA, and HiMem-WAM. Quarter strength relocations of 1.5 to 3 cm, which are within typical grasp tolerance, have little effect. We also varied the illumination strength and the number of distractors. Illumination stays benign at every strength, and distractor losses stay small at every count, as in the main results. The findings therefore do not rely on extreme magnitudes.

AuthorsOpenReview ↗

Response to Reviewer o9tz [3/3]

Gradual onset. The main track applies each change in one simulator update. This is realistic for observation changes, since a light switch or a camera bump is effectively instantaneous at the 20 Hz control rate, but abrupt for objects that are moved or placed. We therefore added gradual versions of these physical events, where the same final change unfolds over 1.5 s from the same trigger, as one point on the continuum between an instantaneous change and no change. The target or destination slides at 4 to 8 cm/s, as when a person pushes it. The obstacle is lowered into the path from 10 cm above, as when a person sets it down. The five distractors are set down one after another. The table reports the change in success rate relative to Base, in percentage points, with 99 or 100 pairs per cell.

EventOnsetπ0.5X-VLAHiMem-WAM
Target relocationAbrupt−28.0−45.0−33.0
Target relocationGradual−27.0−40.0−26.3
Receptacle relocationAbrupt−15.0−45.0−30.0
Receptacle relocationGradual−11.0−47.0−28.0
Obstacle insertionAbrupt−7.0−5.0+1.0
Obstacle insertionGradual−10.0−4.0−5.0
Distractor burstAbrupt−7.0−4.0−14.0
Distractor burstGradual−7.0+1.0−10.0

Gradual onset produces losses of similar size, and none of the 12 differences from abrupt onset is significant under McNemar's test. The findings are therefore not an artifact of abrupt changes. A slower movement acts like a smaller displacement when the robot arrives, and the milder changes above show that the loss shrinks continuously with the displacement.

Implementation validity. Every case passes automated simulator checks for a stable state, valid placement on the support and within the workspace, visibility, a nonzero change in the observation, and no new unrelated contact. Across the original runs of four policies, all 11,051 logged geometric events matched their registered displacements or placements.

We will add the physical parameter table to Appendix B and the severity and gradual onset results to new appendices. We will include all these results and revisions in the revised PDF, which we will upload soon during the discussion period. We hope that these results address the concerns of the reviewer.

Reviewer_xqwbOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

LIBERO-MAX evaluates robot policies under environmental changes introduced during execution. Each Dynamic rollout shares an identical executed prefix with a no-event Base rollout, allowing paired analysis of the outcomes. The benchmark covers eight event types and fourteen policies, all of which show performance losses. Camera-timing controls, query-cadence sweeps, and an observation-restoration intervention examine several sources of these failures.

Critical strengths
  1. Replaying the same executed prefix in Base and Dynamic controls the history before an event. This gives the paired outcome analysis a stronger basis than comparing independently sampled rollouts.
  2. The study covers fourteen policies and goes beyond aggregate success rates. Paired outcomes, camera controls, cadence sweeps, and the restoration experiment provide useful evidence about where performance is degraded.
Critical weaknesses
  1. Table 6 defines the events, Appendix F varies camera strength on matched cases, and Appendix H specifies sensor-noise and occlusion levels. The physical scale of several interventions remains hard to judge: relocation distances and directions, absolute camera position/yaw/FOV changes, illumination scales, obstacle dimensions and placements, and onset profiles. From my perspetive, what deployment events motivate these magnitudes? Beyond the camera study, varying severity on the same source cases would separate event strength from differences in instance difficulty.

  2. The fourteen-policy evaluation leaves two practical questions open. Do the measured sensitivities carry over to physical robots, and can an agent that explicitly reasons about recovery handle these events? A small real-robot study of target displacement, camera movement, or obstacle introduction would address the first. A GPT-6 Astra Direct/Hybrid recovery baseline, following the recent embodied-policy study [1], would address the second. Its observations, action/tool adapter, execution budget, treatment of queued actions, and whether physics pauses during reasoning must be specified for comparison with the native policies.

[1] Systematically Exploring the Capabilities of GPT-6 Astra as Embodied Policies. arXiv:2609.38537. https://arxiv.org/abs/2609.38537

  1. Appendix G states that pi0.5 uses H=10 instead of the primary H=50, while X-VLA and GR00T decode H=Q. The authors correctly describe the latter as serving-setting comparisons rather than an isolated Q sweep. Why are these horizon choices necessary, and how do they affect comparison with the primary protocol and camera controls? Where supported, a fixed-H sweep or queue-cancellation control would separate feedback cadence from prediction horizon and stale-action exposure.
Questions for authors

Please see the weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer xqwb [1/3]

We thank the reviewer for recognizing the value of replaying the executed prefix and of the evidence beyond aggregate success rates, and for the precise suggestions on intervention scale and serving protocol. We address the concerns below.

W1: Physical Scale of the Interventions and Severity on the Same Cases

Physical scale. We will add the following table to Appendix B, together with distributions over cases and before and after images with scale references.

EventTwo frozen settings, one per caseReal world counterpart
Target relocationHorizontal displacement of 6 or 12 cm in one of 8 directions, on the same supportAn object nudged or repositioned by a person during a reach
Receptacle relocationThe same displacement, applied to the destinationA plate or basket moved while the robot approaches
Camera shiftExternal camera translated by (+6, 0, +2) cm with yaw +18° and vertical FOV +12°, or by (−6, +3, 0) cm with yaw −22° and FOV −12°A bumped or remounted camera, or a loose mount
Illumination switchScene lights scaled by 0.2 or 1.8, with camera exposure scaled by 0.75 or 1.25Lights switched off or on, a window blind, or a change in sunlight
Sensor noise onsetGaussian noise with SD 24 or 36 on 8 bit images, plus a moving black occluder covering 8% or 16% of the imageSensor degradation, or a smudged or partially occluded lens
Visual theme switchRGB channel permutation with channel gains, for example BRG with gains of 0.70, 1.15, and 1.30A change in white balance or color temperature
Distractor burstFive task scene objects placed 9 to 17 cm from the targetA person setting items down near the workspace
Obstacle insertionOne object placed 14 cm from the target on the approach side, for example a soup can of 6.2 × 7.6 × 6.2 cm or a mug of 12.5 × 9.6 × 10.6 cmAn item placed in the path of the robot

These are reproducible stress levels chosen to be clearly visible while physically valid. The object stays on its support, within the workspace, and visible, and no new unrelated contact occurs. We do not claim that they match the frequency of real deployment events.

Severity on the same cases. Holding the case, initial state, executed prefix, and direction fixed, we scaled each event to 25%, 50%, and 100% of its magnitude. We use the 800 LIBERO-MAX Lite cases with π0.5, X-VLA, and HiMem-WAM, the policies of the camera controls in the paper. The table reports the change in success rate relative to Base, in percentage points, with 99 or 100 pairs per cell.

EventStrengthπ0.5X-VLAHiMem-WAM
Target relocation25%, 1.5 to 3 cm−1.0−2.0−1.0
Target relocation50%, 3 to 6 cm−13.0−18.0−15.2
Target relocation100%, 6 to 12 cm−28.0−45.0−33.0
Receptacle relocation25%, 1.5 to 3 cm−1.0−3.0+2.0
Receptacle relocation50%, 3 to 6 cm−7.0−19.0−10.0
Receptacle relocation100%, 6 to 12 cm−15.0−45.0−30.0
Sensor noise onset25%, SD 6 to 9, 2% to 4% occluded−3.0−8.0−5.1
Sensor noise onset50%, SD 12 to 18, 4% to 8% occluded−6.0−12.0−10.1
Sensor noise onset100%, SD 24 to 36, 8% to 16% occluded−28.0−34.0−27.0

On identical cases, the loss grows with the magnitude of the change. Quarter strength relocations of 1.5 to 3 cm, which are within typical grasp tolerance, have little effect. Half strength already reduces success significantly under McNemar's test for all three policies on target relocation and sensor noise, and for X-VLA and HiMem-WAM on receptacle relocation. Full strength reduces success by 15.0 to 45.0 points. We also varied the illumination strength and the number of distractors. Illumination stays benign at every strength, and distractor losses stay small at every count, at most 14 points, as in the main results. Together with the camera strength sweep in Appendix F.4, this gives a magnitude resolved view for six of the eight events. The losses therefore scale with the physical size of the change and do not reflect differences in instance difficulty.

AuthorsOpenReview ↗

Response to Reviewer xqwb [2/3]

Onset profile. In the main track each change is applied in one simulator update and then persists. We added gradual versions of the physical events, where the same final change unfolds over 1.5 s from the same trigger, as one point on the continuum between an instantaneous change and no change. The target or destination slides at 4 to 8 cm/s, the obstacle is lowered into the path from 10 cm above, and the five distractors are set down one after another. The table reports the change in success rate relative to Base, in percentage points, with 99 or 100 pairs per cell.

EventOnsetπ0.5X-VLAHiMem-WAM
Target relocationAbrupt−28.0−45.0−33.0
Target relocationGradual−27.0−40.0−26.3
Receptacle relocationAbrupt−15.0−45.0−30.0
Receptacle relocationGradual−11.0−47.0−28.0
Obstacle insertionAbrupt−7.0−5.0+1.0
Obstacle insertionGradual−10.0−4.0−5.0
Distractor burstAbrupt−7.0−4.0−14.0
Distractor burstGradual−7.0+1.0−10.0

Gradual onset produces losses of similar size, and none of the 12 differences from abrupt onset is significant under McNemar's test. A slower movement acts like a smaller displacement when the robot arrives, and the severity results above show that the loss shrinks continuously with the displacement. We will add the severity sweep and the gradual variants to new appendices.

W2: GPT-6 Astra Recovery Baseline

Thank you for the pointer. We implemented both modes of [1] on LIBERO-MAX with the specifications the reviewer lists.

  1. Observations: the same agent view and wrist images, proprioception, and instruction as the native policies. No event flag is given, so the agent must notice the change from its observations.
  2. Direct mode: Astra outputs end effector delta actions for the next k ≤ 20 control steps, calibrated at about 7 mm per step for a unit action. The actions are executed, and Astra is called again.
  3. Hybrid mode, following [1]: Astra supervises π0.5. Every 10 control steps it may let π0.5 continue, cancel the queued actions of π0.5 and requery it, reinstruct π0.5 with a corrected target or subgoal, or issue a short corrective end effector move.
  4. Execution budget and queued actions: the same step limit as all policies. In Hybrid, the queue of π0.5 is kept unless Astra cancels it. In Direct, the k actions of Astra are executed before the next call.
  5. Physics: paused during reasoning, as in [1].
  6. Pairing and cases: the Dynamic arm replays the complete decision log of the Base arm up to the trigger, so the executed prefix is identical. We use LIBERO-Plus cases from LIBERO-MAX Lite, chosen in a fixed order before any run, with 80 cases for π0.5 and Hybrid and the first 32 of them for Direct. Astra uses medium reasoning effort.
AgentCasesBase success rateDynamic success rateChange from Base to DynamicAstra calls per episodeTokens per episodeReasoning time per episode
π0.58081.365.0−16.3N/AN/AN/A
GPT-6 Astra Hybrid8080.062.5−17.519.415.2k57 s
GPT-6 Astra Direct320.00.00.029.324.0k182 s

Hybrid notices the events but does not recover more. Across the 80 Dynamic episodes, Astra cancelled the queue of π0.5 146 times after the event, reinstructed π0.5 88 times, and took direct corrective moves 129 times. A typical reasoning step reads: "A second, plain white mug has appeared in front of the yellow-handled target mug. Refresh the plan". Yet Hybrid reaches 62.5% Dynamic success compared with 65.0% for π0.5 alone, and the difference is not significant under McNemar's test. Noticing a change and replanning at the level of language and subgoals is therefore not sufficient. The bottleneck is executing a corrected low level approach from the intermediate state.

Direct does not complete any task, even without events. The action adapter itself works. Without events, Direct brings the gripper within 10 cm of the target in 11 of 32 episodes and lifts the target in 6, compared with 29 and 26 for π0.5, but it never achieves the precise grasp and placement needed to finish. This agrees with the finding of [1] that direct low level control is weak.

We will add this study as a new appendix and cite [1] in the related work.

AuthorsOpenReview ↗

Response to Reviewer xqwb [3/3]

W2: Transfer to Physical Robots

We agree that this is an important question. A physical study that runs the paired protocol at a meaningful scale is not feasible within the discussion period, and we leave it as future work. LIBERO-MAX follows the simulation only evaluation of LIBERO, LIBERO-Plus, and LIBERO-PRO, and prior work has found that policy rankings in carefully constructed simulation correlate with real robot rankings [2]. The events are physically realizable with simple hardware, such as moving an object by hand, remounting a camera, switching a light, or placing an object in the path. The paired protocol also transfers directly: record a Base run, replay its prefix, and introduce the change at the same trigger. We will state this limitation and the transfer protocol in Appendix I.2.

W3: Prediction Horizon, Query Interval, and Stale Actions

Thank you for catching this. It is a documentation error in Table 9, not a protocol difference. The primary π0.5 evaluation used the released pi05_libero configuration, which returns H = 10 actions and executes Q = 5. We verified this on the original traces of the full 8,000 pair run. All 604,956 model queries returned 10 actions, all 679,716 intervals between consecutive queries were 5 steps, and all 16,000 Base and Dynamic outcomes match the submitted scores. The cadence sweep and the camera controls therefore use the same protocol as the primary evaluation.

For X-VLA and GR00T N1.7, the network horizon is also fixed throughout the sweep. X-VLA always generates 30 actions, and GR00T generates 40 network timesteps decoded into 16 commands. Only the number Q executed before the next observation changes, and "H = Q" referred to the length of the action list returned to the runner. The sweep is therefore already a cadence comparison at a fixed horizon.

We also ran the queue cancellation control on the 800 Lite cases. At the event, all queued actions are discarded and the policy is queried at once on the changed scene, with the same prefix, event state, and step limit. A paired control forces the same requery without an event. The table reports the change in success rate relative to Base, in percentage points.

PolicyActions executed per queryQueued actions discarded at the eventDynamicDynamic with cancel and requeryForced requery without an event
π0.552.0−14.1−12.6+1.1
X-VLA3011.3−23.0−25.2+0.4
HiMem-WAM104.4−13.2−14.1+1.9

Cancelling the queue recovers only 1.5 points for π0.5 and does not help X-VLA or HiMem-WAM, and the forced requery alone has no effect. The gap is therefore not an artifact of the horizon or queue length. We will correct Table 9 and Appendix G.1 and relabel Table 16 with the network horizon, returned actions, and executed actions.

We will include all these results and revisions in the revised PDF, which we will upload soon during the discussion period.

[1] Systematically Exploring the Capabilities of GPT-6 Astra as Embodied Policies.

[2] Evaluating Real-World Robot Manipulation Policies in Simulation.

Reviewer_3CvdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LIBERO-MAX, a benchmark for evaluating whether robot policies can complete tasks when the environment changes during execution. It contains 8,000 paired cases derived from LIBERO-Plus and LIBERO-PRO, covering eight event types. Each pair shares the task, initial state, and executed action prefix before a deterministic trigger; the Dynamic condition then introduces a persistent environmental change. Fourteen released policies are evaluated under their native serving protocols, with success-rate decreases of 11.0–25.7 percentage points. Additional experiments examine camera-change timing, policy query cadence, and image restoration under sensor noise. An 800-pair Lite subset supports more economical evaluation.

Critical strengths

The paired evaluation protocol provides a useful and carefully controlled measurement of performance under execution-time changes. Matching the executed action prefix prevents pre-event trajectory divergence from confounding the Base–Dynamic comparison. The four-way outcome decomposition and conditional regression analysis also help distinguish event-associated failures from low baseline competence. These choices strengthen the benchmark’s diagnostic value beyond reporting aggregate success rates alone. The benchmark is a substantial empirical resource with practical evaluation coverage. Evaluating fourteen released policies on a common set of 8,000 cases provides a useful picture of current capabilities across diverse environmental changes. The camera-timing controls and query-cadence experiments provide additional insight into execution conditions, while the Lite subset makes the benchmark more accessible for iteration. The paper appropriately treats cross-family comparisons as descriptive rather than attributing differences solely to model architecture.

Critical weaknesses

Post-event solvability within the remaining execution budget is insufficiently validated. The automated checks establish physical validity, including support, workspace, visibility, and collision constraints, but do not establish that the task remains completable after the intervention within the remaining steps. The acknowledged incomplete human feasibility audit leaves this question open, particularly for relocation and obstacle events. Consequently, some failures could reflect insufficient remaining time or problematic intervention configurations rather than limitations in policy adaptation. A stratified completion audit or results from a privileged-state reference controller, together with timeout statistics, would strengthen the interpretation of the reported performance gaps. Such a controller would provide evidence of achievable performance, rather than a strict solvability ceiling. The experiments only partially separate the distinctive challenges of execution-time change from static robustness and action-execution effects. The paired protocol measures the total effect of introducing an event, but does not fully distinguish difficulty under the changed environment, dependence on execution history, and continued execution of previously predicted actions. The camera Reset–Mid-task comparison is informative, but covers only one event type and three policies. Notably, Mid-task success exceeds Reset success for all three tested policies (+11.1, +4.0, and +3.8 percentage points), highlighting the potential benefit of progress made before the change. However, the comparison also changes visited states, exposure duration, and remaining budget, leaving the separate contributions of changed-view difficulty and execution history unresolved. For X-VLA and GR00T N1.7, jointly varying prediction horizon and query interval further limits causal interpretation of the cadence sweep. Broader event-at-start comparisons and a matched diagnostic that clears pending actions and immediately re-queries at event onset would help establish which limitations are specific to the dynamic setting.

Questions for authors

Post-event feasibility and time budget: What evidence is currently available that intervened tasks remain completable within the remaining step budget? Is the budget adjusted following geometry changes, and what fraction of Dynamic failures are timeouts? Please distinguish automated validity checks from demonstrated task completion. Replay fidelity: How are policy memory, observation history, pending action queues, and random-number-generator states synchronized between the paired conditions? Has a no-event replay control been run, and, if so, what trajectory and outcome agreement does it achieve? Static versus execution-time changes: Beyond the camera study, is there existing evidence comparing the same intervention applied at task start and during execution? Even a representative additional event would help clarify how broadly the temporal findings extend. Pending-action effects: Are results available for a matched condition that uses an oracle event signal to discard pending actions and immediately request a fresh prediction, while retaining the same pre-event prefix and task budget? This would help distinguish limitations of native serving protocols from limitations in post-event control. Primary evaluation metric: Which metric should users prioritize when comparing policies on LIBERO-MAX, and how should Dynamic success rate, the paired success-rate gap, and conditional regression be interpreted together when Base competence differs substantially?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3Cvd [1/3]

We thank the reviewer for the constructive and detailed review, and for recognizing the paired protocol as a carefully controlled measurement, the outcome decomposition and conditional regression, and the benchmark as a substantial empirical resource. We refer to the two weaknesses as W1 and W2 and to the five questions as Q1 to Q5, in the order of the review. We address them below.

W1 and Q1: Completion After the Event and Step Budget

We agree that physical validity is not the same as demonstrated completion. We will rename the requirement "continued feasibility" in Appendix B.1 to "physical validity", so that it describes only what the automated checks establish, and we now report completion evidence separately.

Completion in the original runs. On 7,039 of the 8,000 cases, or 88.0%, at least one of the 13 policies received a fresh query after the event and then completed the Dynamic task within the original step limit.

Completion from the exact state after the event. Before any run and without looking at outcomes, we drew a stratified sample of 160 LIBERO-MAX Lite cases with 20 per event, and the event fired in 155 of them. For each, we restored the exact simulator state after the event, cancelled the pending actions, and let π0.5 continue with up to 10 sampling seeds within the remaining native budget. A success shows that this specific state is completable, while a failure does not prove infeasibility. As weaker evidence, we also count cases that another of the 13 policies completed after the same event, and cases whose changed layout was completed by π0.5, X-VLA, or HiMem-WAM when present from the start.

The table reports the number of sampled states with completion evidence. Each row includes the evidence of the rows above it.

Completion evidenceTargetReceptacleCameraSensorIlluminationThemeDistractorObstacleAll
Sampled states1920201919182020155
Completed from the exact state by π0.51512151418181718127
Or completed by another policy on the same case1614191619182019141
Or the same changed layout completed from the start1617191619182019144

π0.5 completes 82% of the sampled states from the exact state, close to its own Base success of 86% on the same cases, so coverage is limited mainly by the competence of the witness policy. Including the weaker evidence, 93% of the sampled cases are demonstrably completable. Most importantly for the paired loss, π0.5 loses 28 sampled pairs because of the event, and only 3 of them, or 1.9% of the sample, have no completion evidence.

Budget and timeouts. The step budget is the same for all events and is not adjusted after geometry changes. The trigger fires early, with 96.7% of events within the first third of the limit. Episodes end only on success or at the step limit, so every failure reaches the limit, and a timeout label does not by itself mean that more time would help. We therefore tested the budget directly. On the 800 Lite cases with π0.5, X-VLA, and HiMem-WAM, the policies of the camera controls in the paper, we reran all arms so that every arm receives the full step limit after the event.

The table reports the gap between Dynamic and Base in percentage points, and the share of Dynamic failures that succeed with the extra time.

PolicyGap at the native limitGap with the full limit after the eventDynamic failures rescued by extra time
π0.5−14.1−13.83.2%
X-VLA−23.0−23.01.2%
HiMem-WAM−13.2−13.21.8%

The gap is essentially unchanged, and Base gains a similar number of successes from the extra time. The losses are therefore not caused by insufficient remaining time. We will add this completion evidence and the budget control to a new appendix.

AuthorsOpenReview ↗

Response to Reviewer 3Cvd [2/3]

W2 and Q4: Pending Actions and Oracle Requery

We implemented the matched condition the reviewer describes on the same 800 Lite cases and three policies. It keeps the same checkpoint, prefix, event state, and step limit. At the event, only the pending actions are discarded, and the policy is queried at once on the first observation after the event. Because this uses an oracle event signal, any recovery measures what is available under that privilege. A paired control forces the same requery without an event, which separates the effect of requerying in general from responding to the event.

The table reports the change in success rate relative to Base, in percentage points.

PolicyActions executed per queryQueued actions discarded at the eventDynamicDynamic with cancel and requeryForced requery without an event
π0.552.0−14.1−12.6+1.1
X-VLA3011.3−23.0−25.2+0.4
HiMem-WAM104.4−13.2−14.1+1.9

Discarding the stale queue and requerying at once recovers only 1.5 points for π0.5 and does not help X-VLA or HiMem-WAM, whose fresh predictions after the event are themselves poor. The forced requery alone has no effect. The original runs contain a natural version of the same condition, namely pairs whose event coincided with a scheduled query so that no stale action was executed. On these pairs, success still fell by 24.9 points for OpenVLA-OFT, 20.5 points for VLA-JEPA, and 27.3 points for X-VLA. Native serving with stale actions is therefore not the main source of the loss, and the limitation lies in control from the changed state. We will add this control to Section 4.5.

W2 and Q3: The Same Change at Task Start and During Execution

We extended the Reset comparison from the camera event to all eight events, on the same cases, policies, seeds, and success criteria. Together with the cancel condition above, this separates the three components the reviewer lists. Dynamic gives the total effect of the event. Dynamic with cancel removes the previously predicted actions. Reset applies the same change before execution and removes the intermediate state and the progress made before the change.

The table reports the change in success rate relative to Base, in percentage points, averaged over the three policies with 100 pairs per event and policy.

Event familyReset (at task start)Dynamic (during execution)Dynamic with cancel (during execution)
Geometry: target and receptacle relocation−26.5−32.7−31.7
Observation: camera shift and sensor noise−35.8−28.8−28.8
Appearance and clutter: illumination, theme, and distractors−7.7−2.6−4.7
Path: obstacle insertion−5.3−3.7−3.7

For observation and appearance changes, Reset is as damaging or more damaging than Dynamic, by 7.0 and 5.1 points. These losses mainly reflect the difficulty of the changed condition, and progress made before the change helps, which generalizes the camera result the reviewer highlights. For geometry changes, Dynamic is 6.2 points worse than Reset and still 5.2 points worse after cancelling the previously predicted actions, with the same direction for each policy. Both differences are significant under a bootstrap over cases. This part of the loss depends on the execution history, that is, on meeting the change from an intermediate state.

The reviewer rightly notes that Reset and Dynamic also differ in remaining budget, exposure duration, and visited states. For the budget, when every arm receives the full step limit after the event, Dynamic stays 6.8 points below Reset for geometry events and 7.7 points above Reset for observation events. Exposure over the whole episode is the static difficulty that Reset is meant to capture. For visited states, Dynamic with cancel removes the committed actions, so its remaining difference from Reset reflects the state from which the policy continues, which is the execution history effect itself. We will add this decomposition to Section 4.5 and the results for each event to a new appendix.

AuthorsOpenReview ↗

Response to Reviewer 3Cvd [3/3]

W2: Prediction Horizon and Query Interval for X-VLA and GR00T

The statement in Section 4.6 and Appendix G.1 that X-VLA and GR00T N1.7 use H = Q is inaccurate. On reexamining the serving code and traces, we found that the network horizon is fixed throughout the cadence sweep. X-VLA always generates a 30 step chunk, and the GR00T network produces 40 timesteps decoded into 16 robot commands. Only the number Q of actions executed before the next query changes, and "H = Q" described the length of the list returned to the runner. The sweep therefore already varies the query interval at a fixed network horizon, which is the comparison the reviewer asks for. We rederived all 18 cadence settings from the original runs, and all success counts and paired outcomes reproduce exactly. We will correct Section 4.6 and Appendix G.1 and relabel Table 16 with the network horizon, returned actions, and executed actions.

Q2: Replay Synchronization and Replay Without an Event

  1. Prefix: the Dynamic arm executes the recorded Base action chunks verbatim until the trigger. The policy is not queried during the prefix, so no randomness on the policy side is consumed. All 8,243,686 executed prefix actions in the original runs matched the logged chunks.
  2. Memory and observation history: π0.5, X-VLA, and HiMem-WAM condition each query on the current observation and proprioception. X-VLA resets its internal queue at every query. For HiMem-WAM, the released LIBERO inference code creates a fresh memory state at every call, so its memory gating acts within a call and no state is carried across queries. Replaying actions without inference therefore leaves no policy state to synchronize for these policies. For the other policies, the control across 13 policies below tests for residual effects.
  3. Pending actions: the queue at the event is exactly the remainder of the recorded Base chunk.
  4. Random number generators: π0.5 and X-VLA draw their sampling noise from per query seeds derived from the policy seed and the query position, so the first query after the event uses the same noise as the corresponding Base query would.

To measure replay fidelity, we ran the full Dynamic pipeline with an identity event on all 800 Lite cases. The table reports the change in success rate relative to Base, in percentage points.

PolicyReplay without an eventReplay with the real eventOutcome agreement of replay without an event with Base
π0.5+0.1−14.194.1%
X-VLA+0.4−23.095.9%
HiMem-WAM+1.1−13.293.6%

Outcome agreement. Replay without an event changes success by at most 1.1 points, and none of these changes is significant under McNemar's test. Across 13 policies with 16 cases each, replay without an event matched Base on 194 of 208 outcomes, the same level as two ordinary Base runs at 193 of 208, while the real event on the same cases reduced success by 21.2 points.

Trajectory agreement. The remaining disagreements come from GPU and rendering nondeterminism, which affects rerunning Base itself in the same way. For π0.5 and X-VLA, replay without an event is bitwise identical to Base in all 17 cases where two ordinary Base runs are bitwise identical. Aggregate success rates are reproducible, as our Lite reruns give 79.4, 62.0, and 72.0 for the three policies compared with 79.3, 62.0, and 72.5 in the paper. We will add these details to Appendix B.3 and the replay control to a new appendix.

Q5: Primary Evaluation Metric

We recommend Dynamic success rate over all assigned cases as the primary ranking metric, since it measures task completion under the event and does not reward low Base competence. It should be read together with Base success rate and two diagnostics. The paired gap gives the net effect of the event for each policy. Conditional regression, the share of the policy's own Base successes lost after the event, exposes fragility hidden by a low Base rate. For example, SmolVLA has the smallest gap at 11.0 points but loses 56.1% of its Base successes. Because conditional regression uses a policy specific denominator, it should not be used alone to rank policies. We will state this recommendation in Section 4.1 and Appendix I.1.

We will include all these results and revisions in the revised PDF, which we will upload soon during the discussion period.

Reviewer_eXYAOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces a Libero-based benchmark for evaluating policy robustness to changes applied during policy execution. The changes are applied as events conditioned on the state of the task. An event can relocate objects of interest, introduce new objects, change camera parameters, and change scene appearance. Different VLA, hybrid, and WAM policies are evaluated on the benchmark, all showing a decrease in their performance.

Critical strengths
  • Detailed experiments on camera parameter perturbations.
  • The paper is easy to follow.
Critical weaknesses
  • Only one type of event trigger is evaluated, which is pre-grasp proximity. Since most of the policies evaluated are memoryless, using pre-grasp changes mostly amounts to applying changes pre-execution with a different initial robot state.
  • The evaluated policies are not controlled for their parameter count, architecture, training objective, or dataset. Therefore, even if there were a significant difference between the VLA and WAM policies, the reason could not be disentangled.
  • The experiments other than camera parameter changes do not evaluate the policy performance when the same changes are applied pre-execution. Furthermore, the experiments with camera parameter changes show that policies obtain worse performance when changes are applied before execution than during execution. Due to the absence of pre-execution change evaluation for other event types, it is hard to isolate the true effect of applying changes mid-execution from the effect of having changes from the beginning.
  • As far as I understood from the writing and the shared figures in the appendix, the changes are not dynamic. They are instantaneous changes applied mid-execution, which makes the Dynamic naming misleading. Aside from some of the evaluated observation and appearance changes that can be instantaneous, it is hard to assess policies' real-world robustness to dynamic changes due to the evaluated instantaneous changes.
  • Success rate comparisons do not show the true effect of the changes on the policies. Adding sub-task (moving to the correct object, grasping, carrying the object to the target, correctly placing) success comparisons would strengthen the paper.
  • Geometry and Observation are broad categories for event types. Appearance events could be under Observation, and Path events could be under Geometry.
Questions for authors
  • Do Base and Dynamic evaluations share the same number of policy execution steps? If so, have you done any experiments where policies in Dynamic evaluation run for the same number of steps after the event happens?
  • Did you run the policy during replay and discard the actions? Was fixed environment and policy seeding not enough to obtain the same prefix action prediction?
7 comments
AuthorsOpenReview ↗

Response to Reviewer eXYA [1/3]

We thank the reviewer for the careful reading and for recognizing the detailed camera parameter experiments and the clarity of the paper. We address the concerns below.

W1: Generalization Beyond the Pre-Grasp Trigger

To test whether the effect of a change extends beyond the approach phase, we added a trigger that fires when the gripper first grasps the target. The change then arrives while the object is already in the gripper and persists through transport. A change applied before execution cannot create this situation, since every episode starts with the object at rest in the scene. In 96% of the triggered π0.5 episodes, the object is lifted by at least 3 cm within one second. We applied the five events that remain defined once the target is held. Each is paired with a control that uses the same trigger and replay without an event.

The table reports the change in success rate relative to this control, in percentage points, on the 64 to 83 episodes per cell where the trigger fired.

Event after the graspπ0.5X-VLAHiMem-WAM
Receptacle relocation−8.5−63.9−33.8
Camera shift−16.9−60.6−1.2
Sensor noise onset−22.2−28.8−28.2
Illumination switch0.00.0−1.4
Visual theme switch−5.10.0+6.5

Changes after the grasp remain damaging, so LIBERO-MAX also measures robustness at a stage of the task that no change before execution can reach. Receptacle relocation reduces success by 8.5, 63.9, and 33.8 points for π0.5, X-VLA, and HiMem-WAM. Camera shift and sensor noise onset also remain damaging, except camera shift for HiMem-WAM, which is robust to this event in the main benchmark as well. Illumination and visual theme switches have small effects, as before the grasp.

We address the comparison with the same change applied before execution, which the reviewer also raises in W1, in W3 below.

W2: Architecture Effects

We agree, and the paper draws no causal conclusions about architecture. Table 2 states that it "compares policies rather than isolating an architecture-family effect," and Takeaway 2 states that "the comparison cannot isolate the effect of architecture." As a benchmark, LIBERO-MAX evaluates released policies as they are deployed, like LIBERO, LIBERO-Plus, and LIBERO-PRO, and all its causal statements are made within a policy with a fixed checkpoint, serving protocol, and prefix. We will state this scope explicitly in Section 4.3.

AuthorsOpenReview ↗

Response to Reviewer eXYA [2/3]

W3: The Same Change Before and During Execution

To test whether a change during execution is equivalent to the same change applied before execution, we compare three conditions on the same cases.

  1. Reset (pre-execution): the change is applied before execution starts. This measures the difficulty of the changed scene itself.
  2. Dynamic (mid-execution): the change is applied at the trigger during execution, as in the benchmark.
  3. Dynamic with cancel (mid-execution): the same as Dynamic, but at the event the actions queued from the old scene are discarded and the policy is queried at once on the changed scene.

The third condition is needed because Dynamic differs from Reset in two ways. The policy may still execute actions planned for the old scene, and it continues from an intermediate robot state. Dynamic with cancel removes the first difference and corresponds exactly to a memoryless policy restarting in the changed scene from a different robot state. Its gap to Reset therefore measures whether the robot state at which the change arrives matters.

We run all eight events on the 800 LIBERO-MAX Lite cases with π0.5, X-VLA, and HiMem-WAM, the policies of the camera study in the paper, keeping the seeds and success criteria of Base. The table reports the change in success rate relative to Base, in percentage points, averaged over the three policies with 100 pairs per event and policy.

Event familyReset (pre-execution)Dynamic (mid-execution)Dynamic with cancel (mid-execution)
Geometry: target and receptacle relocation−26.5−32.7−31.7
Observation: camera shift and sensor noise−35.8−28.8−28.8
Appearance and clutter: illumination, theme, and distractors−7.7−2.6−4.7
Path: obstacle insertion−5.3−3.7−3.7

Our results show that mid-execution changes are not generally equivalent to pre-execution changes, and that their relative difficulty depends on the event type.

  1. Geometry changes are harder mid-execution. The change mid-execution costs 6.2 points more than the same change pre-execution, with the same direction for each policy at 8.0, 4.0, and 6.5 points. With the queued actions cancelled, it still costs 5.2 points more. Both differences are significant under a bootstrap over cases, and the gap remains 6.8 points when every arm receives the full step limit after the event. Meeting the change with the arm already approaching the old location is therefore harder than starting in the changed scene, and this is not explained by stale actions or remaining time.
  2. Observation and appearance changes are as hard or harder pre-execution. Reset costs 7.0 and 5.1 points more than Dynamic, consistent with the camera study in the paper. For these events the changed condition itself is the main difficulty, and progress made before the change helps.

LIBERO-MAX measures the total effect of a change on a task already under way, and Reset shows which part of this effect is specific to the timing.

W4: Gradual Onset of the Changes

We clarify that Dynamic refers to when a change occurs, not to continuous motion. In LIBERO-MAX, the environment changes during execution, after the policy has started acting, and the change then persists. We will state this definition explicitly in Section 3.1.

To examine whether our findings extend beyond instantaneous changes, we added gradual versions of the physical events. The same final change unfolds over 1.5 s from the same trigger, as one point on the continuum between an instantaneous change and no change. The target or destination slides at 4 to 8 cm/s, the obstacle is lowered into the path from 10 cm above, and the five distractors are set down one after another.

The table reports the change in success rate relative to Base, in percentage points, with 99 or 100 pairs per cell.

EventOnsetπ0.5X-VLAHiMem-WAM
Target relocationAbrupt−28.0−45.0−33.0
Target relocationGradual−27.0−40.0−26.3
Receptacle relocationAbrupt−15.0−45.0−30.0
Receptacle relocationGradual−11.0−47.0−28.0
Obstacle insertionAbrupt−7.0−5.0+1.0
Obstacle insertionGradual−10.0−4.0−5.0
Distractor burstAbrupt−7.0−4.0−14.0
Distractor burstGradual−7.0+1.0−10.0

Gradual onset produces losses of similar size, and none of the 12 differences from abrupt onset is significant under McNemar's test. Our findings therefore extend beyond instantaneous changes. A slower movement acts like a smaller displacement when the robot arrives, and on the same cases halving the target relocation more than halves the loss, so performance degrades continuously along this continuum.

AuthorsOpenReview ↗

Response to Reviewer eXYA [3/3]

W5: Subtask Analysis

We computed the four subtasks from the logged end effector and object poses, on the 576 Lite cases whose destination is a movable object. Reach means the end effector comes within 10 cm of the target. Pick means the target is lifted by at least 3 cm. Carry means the held target comes within 10 cm of the destination. Place means task success.

The table reports the percentage of episodes that complete each stage.

PolicyConditionReachPickCarryPlace
π0.5Base92.489.986.880.0
π0.5Dynamic90.684.478.668.4
X-VLABase77.368.463.760.2
X-VLADynamic70.854.045.836.1
HiMem-WAMBase87.885.883.374.8
HiMem-WAMDynamic85.178.872.659.5

Reaching is barely affected, with drops of only 1.8 to 6.5 points, and the loss accumulates over picking, carrying, and placing. Under geometry events, the policies usually still pick and carry the object but fail to place it at the moved destination. On the 164 cases with geometry events, success among episodes that complete carry drops from 91.9%, 98.0%, and 87.8% in Base to 79.0%, 47.5%, and 59.3% in Dynamic for π0.5, X-VLA, and HiMem-WAM.

W6: Event Grouping

We agree that the families are a presentation choice, and the eight events remain the units of analysis. Under the grouping into geometry and observation that the reviewer suggests, geometry losses exceed observation losses for 13 of the 14 policies in Table 12, so the conclusions are unchanged. We will add this grouping to Appendix E.2.

Q1: Step Budget After the Event

Base and Dynamic share the same step limit, which is not adjusted after events, and episodes end early only on success. The trigger fires early, with 96.7% of events within the first third of the limit. To test the budget directly, we reran all arms so that every arm receives the full step limit after the event.

The table reports the gap between Dynamic and Base in percentage points, and the share of Dynamic failures that succeed with the extra time.

PolicyGap at the native limitGap with the full limit after the eventDynamic failures rescued by extra time
π0.5−14.1−13.83.2%
X-VLA−23.0−23.01.2%
HiMem-WAM−13.2−13.21.8%

The gap is essentially unchanged, so the failures are not caused by a shortage of time.

Q2: Prefix Replay and Seeding

We do not run the policy during the prefix. Base records every action chunk, and Dynamic executes the recorded chunks verbatim until the trigger, after which the policy is queried normally. All 8,243,686 executed prefix actions in the original runs matched the logged chunks.

Seeding is not enough because GPU inference and rendering are not bitwise deterministic across processes. In a rerun with identical seeds, the first camera frame differed in two pixels by one intensity level, which already changed the first action of π0.5. Aggregate success rates are reproducible, as our Lite reruns give 79.4, 62.0, and 72.0 for the three policies compared with 79.3, 62.0, and 72.5 in the paper, but individual trajectories can diverge. Replay guarantees an identical prefix by construction.

To check that replay itself does not change outcomes, we ran the full Dynamic pipeline with an identity event on all 800 Lite cases. The table reports the change in success rate relative to Base, in percentage points.

PolicyReplay without an eventReplay with the real eventOutcome agreement of replay without an event with Base
π0.5+0.1−14.194.1%
X-VLA+0.4−23.095.9%
HiMem-WAM+1.1−13.293.6%

Replay without an event changes success by at most 1.1 points, and none of these changes is significant under McNemar's test.

We will include all these results and revisions in the revised PDF, which we will upload soon during the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Authors’ Summary of the Discussion Period

Dear Area Chair,

We thank the reviewers for their engagement during the discussion period. Three of the four reviewers now rate the paper at weak acceptance, and two reviewers raised their ratings. Reviewer o9tz raised the rating from weak rejection to weak acceptance, noting that "the new fully crossed experiment is helpful" and that the new magnitude and gradual-change experiments "address most of my second concern." Reviewer eXYA raised the rating from clear rejection to weak rejection and wrote that the new triggers and gradual changes improve "the benchmark's usefulness for assessing the policies' real-world robustness." Reviewer xqwb maintained weak acceptance and is "very satisfied with the added GPT-6 experiment." Reviewer 3Cvd rates the paper at weak acceptance but unfortunately did not respond during the discussion period. We have nevertheless addressed each of their questions in our response and in the revised PDF.

The main concerns and our resolutions, all now in the revised PDF and marked in blue:

  • Changes during versus before execution (eXYA, 3Cvd). We compared Reset, Dynamic, and Dynamic with cancel for all eight events. Geometry changes are harder during execution, and restarting from the event state without the change does not reduce success. (Sec 4.5, Table 4, Appx J)
  • Protocol validity (3Cvd, eXYA). Replay without an event leaves success unchanged, the full step limit after the event leaves the gap unchanged, and the tasks remain completable after the event. (Appx K, Tables 23 to 25)
  • Task and event assignment (o9tz). A fully crossed cohort of 60 tasks, 8 events, and 7 policies shows that the assignment does not bias the losses or the rankings. (Appx M)
  • Magnitude and onset of the changes (xqwb, o9tz, eXYA). We report the physical scale of every event, smaller magnitudes on the same cases, gradual onset, and a second trigger after the grasp. The losses persist in each setting. (Appx B.3, L.1 to L.4)
  • Recovery with explicit reasoning (xqwb). GPT-6 Astra supervising π0.5 notices the events but does not recover the loss. (Sec 4.6, Appx N)

In the final round, eXYA raised four remaining points. Each is answered by results already in the revised PDF, as we detailed in our reply. In short, the extra loss under a change during execution is not explained by the robot state at the event, the time budget, or run-to-run variation.

Together, these results support the central finding of the paper. All fourteen released policies lose 11.0 to 25.7 points when the world changes during execution, and the loss comes from the change itself rather than from replay, time, or stale actions. LIBERO-MAX makes this ability measurable with paired cases and provides a controlled testbed for robot policies that must adapt as the world changes.

We sincerely thank you and the reviewers for the time and care devoted to our submission. The reviewers' questions led to substantial new analyses and controls, and we believe the revised paper is considerably stronger as a result. Thank you for your consideration.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.