acceptodds
Under review as a conference paper at ICLR 2027

LIBERO-Dynamic: Benchmarking Robotic Manipulation in Dynamic Environments

Abstract

Most simulation benchmarks for robotic manipulation keep target objects static and treat policy inference as instantaneous. These assumptions simplify repeatable evaluation, but they hide how a moving target evolves during computation and how stale an observation may be when its predicted action is executed. We introduce LIBERO-Dynamic, a controlled dynamic extension of all 40 LIBERO tasks. It drives objects through contact friction along three trajectory families and three speed tiers, while continuously advancing the physical world according to calibrated, model-specific inference latency. The same motion mechanism also supports training: we replay verified static demonstrations and compensate robot actions online using the target’s displacement relative to the reference, producing 9,000 quality-filtered dynamic trajectories. We evaluate six pretrained policies on a shared set of 1,800 dynamic conditions per configuration, compare execution strategies and action horizons with the checkpoint fixed, and fine-tune three architectures using dynamic and mixed data. The experiments show that faster motion degrades every evaluated policy, but not uniformly; execution timing and horizon can change the behavior of the same policy; and, in the measured sweep, asynchronous replanning benefits from frequent feedback only after the action queue can be sustained without underflow. Dynamic demonstrations improve each evaluated architecture under at least one training recipe, while the magnitude of improvement varies with temporal conditioning, data composition, and training budget. In a small-scale physical study using policies trained on real-robot dynamic demonstrations, we observe execution-strategy differences consistent with those found in simulation, with task-dependent effects, providing complementary evidence for the corresponding benchmark findings. By making object motion, inference latency, and action execution explicit and controllable within a broad task suite, LIBERO-Dynamic provides a reproducible platform for evaluating and improving robot policies for moving-target manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotic manipulation, dynamic environments, benchmarking, vision-language-action models, inference latency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bWrWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

As I understand it, the paper makes target objects move in LIBERO. It generates enough dynamic demonstrations. The work compares models, execution strategies, and fine-tuning recipes. Then design a small real-robot study. To me, the main contribution is a unified evaluation of dynamic tasks.

Critical strengths

1.The problem is practically relevant and important today. Targets keep moving while the model is computing (time mismatch), which matters in real deployment.

2.The execution comparisons are useful. It shows how much deployment settings affect the same model.

3.The action-queue analysis is informative. It explains why shorter action horizons can also hurt performance.

Critical weaknesses
  1. The added value over existing work is unclear(this is important, it'll be better if you clear it). DOMINO already studies dynamic manipulation, and ReflexBench accounts for inference latency. I am not convinced what important insights this benchmark adds beyond them(Simply on Libero?).

2.The timing model may not reflect real deployment well, there's a huge huge gap between sim and real, especially LATENCY; real exp. is not enough, I am not sure whether the reported efficiency and strategy rankings would hold on, whether the benefit will cross the huge gap.maybe a much more clear report helps, to prove its value;

3.The generalization value of dynamic training is not established. Training and testing seem that share tasks, paths, and speeds, please prove the true generalization over 'dynamic' not just starting phase and soon;

Questions for authors

Please see Critical Weaknesses

6 comments
AuthorsOpenReview ↗

Response to W1: Added Value Beyond Existing Dynamic Benchmarks

Thank you for asking us to clarify the contribution beyond applying dynamic motion to LIBERO.

LIBERO-Dynamic provides a common setting to study how manipulation capability, training recipes and execution timing interact. Its added value is not latency modeling or asynchronous execution alone: the experiments show why static performance and low latency are insufficient predictors of dynamic performance, and why frequent feedback must be balanced against continuous action supply.

Relationship to prior work. We recognize DOMINO and ReflexBench as important foundations.

DOMINO covers 35 dynamic tasks. Its paper describes kinematic target motion and stopping autonomous motion upon gripper contact; the public code defaults to this behavior but allows task-level overrides. Our objects instead move through frictional contact with a moving platform. First gripper contact does not disable that external drive: subsequent motion is determined by the physics solver, and lifting can separate an object from the platform. Thus, first contact is not equivalent to stable grasping; inadequate tracking can still lead to tipping or slipping. This comparison concerns DOMINO's published/default configuration, not an absence of rigid-body contact physics.

In the inspected DOMINO pi0.5 deployment wrapper, policy inference returns before actions execute; we did not find explicit scene advancement by measured inference duration in that path. LIBERO-Dynamic advances physical time while waiting for actions. We limit this observation to the inspected path, not every DOMINO implementation.

ReflexBench explicitly supports latency modeling and synchronous/asynchronous evaluation across six reaction-sensitive tasks. The ReflexVLA paper also reports static LIBERO results. Our complementary scope retains 40 task objectives from LIBERO's four suites under common dynamic paths, speeds and execution protocols, spanning objects, spatial relations, goals and longer manipulation sequences. Neither asynchronous execution nor history input itself is claimed as new.

What the common evaluation reveals.

  1. Static scores and latency alone are insufficient. The six evaluated models have high reported static LIBERO scores, yet diverge under common dynamic conditions and differ in sensitivity to nonzero target speeds. Models with similar inference latency also differ in dynamic performance. The static scores are official reports, not protocol-matched reevaluations, so static-to-dynamic gaps cannot be attributed solely to motion.

  2. Execution settings matter even at fixed weights. The same pi0.5 checkpoint yields different success, completion-time and smoothness trade-offs across execution configurations. In the corrected common-30-second comparison, Async/Sync/RTC achieve 48.39%/40.00%/37.83% success. These are deployment operating points with different K values (6/10/16) and calibrated model latencies (about 147/147/170 ms), not an isolated strategy-only causal comparison.

  3. Shorter action horizons are not always better. Fixed-weight asynchronous sweeps expose two regimes: very short execution lengths can exhaust the queue before new actions arrive, accompanying lower success; within the tested underflow-free regime, shorter open-loop execution is associated with higher success. Horizon selection must balance feedback freshness, latency and queue continuity rather than simply minimize K.

  4. Dynamic-data benefits depend on the model and recipe. In the training-step-matched Joint comparison, the history-frame variant benefits more from dynamic-only fine-tuning, and its best tested execution length changes. Dynamic-only versus mixed training also behaves differently across models. This motivates studying temporal inputs, recipes and execution length together, not claiming a universal recipe or a proven architectural cause.

Together, these results provide a controlled basis for studying model-training-execution interactions. We appreciate the opportunity to make these contributions clearer.

Sources: DOMINO Sec. 2.2 and App. A.2; versioned evaluation configuration; inspected pi0.5 wrapper; Reflex Sec. III and Table II.

AuthorsOpenReview ↗

Response to W2: Timing Calibration and Matched Real-Robot Comparisons

Thank you for requesting a clearer real-deployment assessment. We agree that simulation alone cannot establish transfer of efficiency or strategy rankings.

The added real-robot comparison supports the efficiency trend: both Async and RTC complete more operations than Sync in the continuous task for both tested models, while conveyor success rates are close. The specific rankings remain task- and model-dependent.

Why calibrate a fixed model latency? Parallel simulation may run on A100/H100/H20 hardware, where inference time depends on hardware and competing CPU/GPU load rather than the target robot deployment. We independently calibrate each relevant configuration on a single RTX 5090 without other GPU computation and use that model-call latency in simulation. The observed steady-state fluctuations of roughly 2-3 ms support a fixed-value approximation for these model calls under this deployment condition, not a complete end-to-end latency model. Real-robot runs incur actual elapsed time, with no added artificial inference wait. Camera, communication and control-chain delays are not measured by the model-latency column.

Matched real-robot settings. All six conditions use 15 Hz control, 10 denoising steps and K=8, on a single RTX 5090 without competing GPU computation. Within each model, Sync/Async/RTC share weights, normalization and input processing. Async/RTC have zero observed queue underflow; Sync's intentional inference waits are not asynchronous underflow.

Conveyor handover: 50 attempts per condition, each limited to 30 s. Placing the apple in the bowl is success; timeout, dropping or knocking it off is failure. All six conditions share conveyor settings, objects and the initial-condition arrangement.

Whack-a-mole: one continuous one-hour run per condition. A valid press extinguishes the currently lit target; only then is the next target selected randomly. All six conditions use the same target-selection and triggering rules, not necessarily identical realized target sequences.

ModelExecutionModel latency (ms)Conveyor successes / 50Success rateValid presses in 1 h
pi0.5 (JAX)Sync61.49 +/- 2.172448%546
pi0.5 (JAX)Async61.49 +/- 2.642550%655
pi0.5 (JAX)RTC78.32 +/- 2.392346%685
History-frame JointSync161.43 +/- 2.811734%428
History-frame JointAsync161.43 +/- 2.261836%630
History-frame JointRTC178.91 +/- 2.571632%617

Here "+/-" denotes the measured steady-state fluctuation range, NOT a standard deviation or confidence interval. These are model-call latencies only. The press counts are also hourly throughputs because each run lasted one hour. There is only one continuous run per condition: its hundreds of presses are not independent experimental replicates, and we do not claim statistical significance.

Comparison with simulation and limits. In the corrected, wait-inclusive 30 s simulation comparison, mean capped completion times over all episodes are 20.51 s (Async), 23.13 s (Sync) and 22.58 s (RTC); unsuccessful episodes contribute 30 s, not an observed completion time. These use the same pi0.5 PyTorch checkpoint at 20 Hz but K=6/10/16 and calibrated model latencies of about 147/147/170 ms, respectively. They compare deployment operating points, not only the strategy label.

In the real continuous task, relative to Sync, Async throughput increases by 20.0% for pi0.5 and 47.2% for history-frame Joint; RTC increases it by 25.5% and 44.2%. Conveyor outcomes differ by only 1-2 successes out of 50. RTC leads pi0.5 throughput, whereas Async leads Joint throughput, so there is no universal ranking.

The tasks, metrics and configurations differ across simulation and hardware: real pi0.5 uses JAX rather than PyTorch, control runs at 15 rather than 20 Hz, and Joint uses a different real-image input resolution. Each uses its configuration-specific latency. The results support the practical value of overlapping computation and execution, rather than an exact numerical or ranking reproduction across simulation and hardware; the gains cannot all be attributed to model latency alone. Thank you for motivating this more explicit real-deployment comparison.

AuthorsOpenReview ↗

Response to W3: Held-Out Geometry and Speed Generalization

Thank you for asking us to test dynamic generalization beyond changes in starting phase.

The new geometry/speed holdout evaluation shows aggregate improvements over the corresponding static baseline in 11 of 12 fine-tuning comparisons, with one decrease. This supports dynamic generalization beyond starting-phase changes, specifically on held-out paths and interpolated speeds within known tasks and motion families, with model- and recipe-dependent benefits.

Holdout construction. Dynamic fine-tuning uses 960 demonstrations at only 0.03 and 0.09 m/s. We construct path holdouts from the actual demonstration geometries, excluding overlap with those dynamic-training paths, rather than merely changing starting phase. The unseen speed is 0.06 m/s, between the two training speeds.

Strict holdout categoryConditionsEvaluation episodes
Unseen path / seen speed228684
Seen path / unseen speed98294
Unseen path / unseen speed130390
Total4561,368

An additional 24 repeated-geometry controls are excluded from this strict holdout set. "Unseen" here refers to the dynamic-training conditions; tasks and motion families remain known.

Evaluation and training controls. Each model/recipe/K combination is evaluated on all 456 conditions with evaluation seeds 1001, 1002 and 1003, totaling 1,368 episodes. Success rate is pooled successes divided by pooled episodes, not a claim of three independently trained models. These are evaluation seeds, not independent training-seed replications.

The three models share holdout conditions, evaluation seeds and time budget. We report both fixed K=4 and K=8, using asynchronous execution at 20 Hz with a 30 s simulation-time limit including waits. Inputs and native action-chunk lengths retain each model's settings, so this is not a fully architecture-matched comparison.

All six fine-tuning branches start independently from their respective static baselines, run for 3,000 updates with effective batch size 128, and use the predetermined final checkpoint. Dynamic-only updates contain 128 dynamic examples; mixed updates contain 64 dynamic plus 64 static examples. Total updates and sample budgets are matched, but mixed training has half as much dynamic exposure; recipe differences therefore cannot be attributed solely to adding static data.

Overall strict-holdout results. Entries are success rates (%); parentheses show changes in percentage points from the same-model, same-K static baseline.

ModelKStatic baselineDynamic-onlyStatic + dynamic
pi0.5454.9765.35 (+10.38)65.13 (+10.16)
pi0.5853.0059.87 (+6.87)60.60 (+7.60)
Single-frame Joint433.9237.65 (+3.73)36.62 (+2.70)
Single-frame Joint835.0935.89 (+0.80)34.87 (-0.22)
History-frame Joint440.2047.37 (+7.16)47.95 (+7.75)
History-frame Joint838.7448.03 (+9.28)48.76 (+10.01)

The pi0.5 gains range from 6.87 to 10.38 points and history-frame Joint gains from 7.16 to 10.01 points. Single-frame Joint improves less and decreases by 0.22 points under mixed training at K=8. Mixed training is not universally better than dynamic-only: it is slightly higher for history-frame Joint at both K values, lower for single-frame Joint, and varies with K for pi0.5.

Thus, the test goes beyond starting-phase variation through geometric and speed holdouts, while the supported result remains aggregate held-out improvement with model/recipe dependence. We do not infer per-subset gains from pooled scores, statistical significance, robustness across independent training seeds, or generalization to unseen tasks or extrapolated speeds. Thank you for suggesting this more stringent test.

Reviewer_3XFSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LIBERO-Dynamic extends 40 LIBERO tasks with contact-driven object motion and calibrated inference delays, and provides 9,000 dynamic demonstrations converted from static references. The paper evaluates six policies, studies execution strategies and action horizons, and examines dynamic-data fine-tuning with complementary real-robot experiments.

Critical strengths

The benchmark combines controlled object motion, inference timing, and execution protocols within an established task suite. The same-checkpoint sweeps and explicit queue-underflow measurements provide useful diagnostics beyond aggregate success rates.

The training improvements persist in the fixed-horizon comparison in Table 8, supporting benefits beyond horizon retuning. The paper also clearly distinguishes condition-matched adaptation, oracle operating points, and comparisons involving different checkpoints or training budgets.

Critical weaknesses

The physical-time budget needs clarification. Success is defined within 600 control steps, but synchronous inference waits are excluded from the reported step counts (Sections 3.5 and A.10); if the termination counter uses the same convention, strategies receive different physical-time budgets. Moreover, the reported time-efficiency metric does not measure elapsed completion time, which is particularly relevant to this benchmark.

Deployment configurations need clearer documentation. Appendix A.11 identifies 60K-step Joint checkpoints without clearly mapping them to the released variants, while Table 10 uses one denoising step for Joint Async and ten for RTC without reporting real-deployment latency or underflow. These details matter for reproducibility and for interpreting the physical execution-strategy results.

Questions for authors

Does the 600-step limit count every physics advance, including synchronous waits and asynchronous underflow? Please report success under a common simulation-time cutoff and completion time including all waits.

Please identify the exact FastWAM/Joint checkpoints and sampling configurations, and clarify whether Table 1’s speed-zero baselines were reevaluated under the same deployment protocol as the dynamic results. What measured latency and underflow accompany the real-robot Async and RTC configurations?

Table 7 reports (E_I=0.451) for mixed-data FastWAM-Joint at 43.8% success, implying approximately 971 inference requests per episode under the stated formula. Is this a table error or a different query-counting convention?

7 comments
AuthorsOpenReview ↗

Response to W1/Q1: Common Physical-Time Budget and Wait-Inclusive Completion Times

Response to W1/Q1: You are correct that the original Sync limit excluded inference waits. Under a common, wait-inclusive 30-second simulation budget, the compared Async, Sync, and RTC operating points achieve 48.39%, 40.00%, and 37.83% success, respectively. Thank you for identifying this issue; we apologize for the ambiguity in the original termination rule and efficiency metric.

The original 600-control-step counter omitted synchronous inference waits; asynchronous queue-underflow cycles were already counted. The new Sync evaluation enforces a hard cutoff using the MuJoCo simulation clock, including object motion during startup and subsequent inference waits. Timing starts at the first policy observation, and episodes end early on success. Success is detected at control endpoints; the new Sync runs also check the exact cutoff state, not every physics substep during a wait.

All rows below use the same pi0.5 PyTorch weights and 20 Hz control. Each setting covers 40 tasks x 3 motion modes x 5 trajectories x 3 speeds = 1,800 conditions. The first three rows use the common 30-second budget. Async K=6 and RTC K=16 use existing, checked records for these conditions; the Sync results are new runs.

StrategyKSuccesses / 1,800Success rateSuccessful-episode mean (s)Capped all-episode mean (s)
Async687148.39%10.4020.51
Sync, new 30 s cutoff1072040.00%12.8223.13
RTC, queue threshold 101668137.83%10.3922.58
Sync, original 600-step limit*1077543.06%*14.12*23.04*

Both time metrics include startup and subsequent waits; overlapping inference and action execution are not counted twice. For the common-cutoff rows, the capped mean assigns each successful episode its completion time and each unsuccessful episode 30 s. This is not a claim that failures completed at 30 s. Successful-episode means concern different subsets of episodes across strategies and should not be interpreted as a matched-task speed comparison.

*The historical Sync row is NOT SR@30s. Its successful-episode mean includes successes after 30 s; 52 of its successful episodes ended after that cutoff. Its 23.04 s mean was computed post hoc by clipping successful termination times to 30 s and assigning failures 30 s. This does not replace strict-cutoff evaluation. The old and new Sync experiments are independent runs, so the 3.06-percentage-point difference cannot be attributed entirely to the cutoff correction.

For completeness, the new Sync sweep covers nine K values, each with 1,800 episodes, totaling 16,200 episodes:

KSuccesses / 1,800SR@30sSuccessful-episode mean (s)Capped all-episode mean (s)
1472.61%25.3429.88
221411.89%19.8428.79
447026.11%15.5826.23
661133.94%14.3524.69
869938.83%13.5123.60
1072040.00%12.8223.13
1671639.78%11.8922.80
3253129.50%10.6224.28
5037520.83%10.5625.95

Async exceeds the new Sync operating point by 8.39 percentage points. However, K differs across strategies, and calibrated model latency is approximately 147.20 ms for Sync/Async versus 169.80 ms for RTC. This is an operating-point comparison, not an isolation of execution strategy at identical K and latency. The success-rate ordering is descriptive, not a significance or universal-ranking claim. Sync K=10 exceeds K=16 by only four successes; we do not claim a uniquely optimal horizon.

We agree that the original E_T, a dimensionless score combining success rate and relative control-step usage, is not elapsed completion time. We will clarify this distinction and correct the termination-rule description in the revision, reporting wait-inclusive simulation-time metrics separately.

AuthorsOpenReview ↗

Response to Q3: Correction of the Inference-Request Efficiency Entry

Response to Q3: This is a table-value error, not a different request-counting convention. For mixed-data FastWAM-Joint, E_I should be corrected from 0.451 to 3.27; the success rate remains 43.8%. Thank you for checking the arithmetic, and we apologize for the incorrect entry.

The metric counts successful episodes per 1,000 inference requests:

EI=1000Nsuccessful episodesNinference requests.E_I = 1000\frac{N_{\mathrm{successful\ episodes}}}{N_{\mathrm{inference\ requests}}}.

The denominator is the total number of inference requests over the evaluated episodes, not just the successful episodes. Neither the definition nor the counting convention changes.

Table 7 rowSuccess rateE_I
Mixed-data FastWAM-Joint, original entry43.8%0.451 (incorrect)
Mixed-data FastWAM-Joint, corrected entry43.8%3.27

Using the reported rounded values, the corrected entry implies an average of

=\frac{1000\times0.438}{3.27}\approx133.94$$ requests per episode. Your calculation for the original entry is correct: 1000 x 0.438 / 0.451 is approximately 971.18. These are implied averages from the displayed values, not newly measured request totals. We will correct this cell and make the formula and denominator explicit in the revised manuscript.
AuthorsOpenReview ↗

Response to W2/Q2: Checkpoint Mapping, Sampling Configuration, and Real-Deployment Measurements

Thank you for carefully checking the checkpoint provenance and deployment settings.

The 60K checkpoints are our reproduced static bases for fine-tuning, not alternative names for the official releases used in Table 1. The one-step entry for Joint Async in Table 10 is a labeling error and should read 10 steps. We apologize for this error. Below we clarify the checkpoint mapping, the source of the speed-zero baselines, and the measured real-robot latency and underflow.

1. Checkpoint mapping and evaluation configurations.

Model / purposeCheckpoint and sourceDynamic evaluationSuccesses / 1,800
FastWAM / Table 1Official native release: libero_uncond_2cam224.ptAsync, current frame, K=8577 (32.06%)
Single-frame FastWAM Joint / Table 1Official release: step_034720.ptAsync, current frame, K=8731 (40.61%)
Single-frame Joint / fine-tuning baseOur reproduced 60K static base: joint_static60k_step_060000.ptAsync, current frame, K=8749 (41.61%)
History-frame Joint / fine-tuning baseOur reproduced 60K static base: step_060000.pt, five-history-frame / 16-step action-prefix variantAsync, five history frames, K=4817 (45.39%)

All four dynamic evaluations use 20 Hz control. The two single-frame Joint checkpoints use K=8 and a calibrated latency of approximately 64.39 ms. Their static-base performance is close (41.61% versus 40.61%). In Appendix A.11, dynamic-only and mixed-data fine-tuning start from the same reproduced static base within each Joint variant, and improvements are calculated relative to that base. We do not count the difference between bases as a fine-tuning gain. The history-frame row uses K=4, so the rows above are not a fixed-K architectural comparison.

2. Correction of the real-robot denoising-step entry.

The Joint Async entry in the original Table 10 was mislabeled as one denoising step; the correct value is 10, as for RTC. We will correct this entry in the revised manuscript. For the real-robot configurations reported below, Sync, Async, and RTC use 10 denoising steps, 15 Hz control, and K=8. Within each model, the three strategies share the same weights, normalization, and input processing.

3. Were the speed-zero baselines reevaluated?

No. Table 1's speed-zero success rates are quoted from the original official LIBERO reports, rather than rerun under our dynamic deployment protocol. They serve as references for the original static performance. Therefore, the entire zero-to-nonzero gap should not be attributed solely to object motion. Our speed-sensitivity analysis uses the nonzero-speed sweep under a common protocol. We will state this distinction explicitly in the revised table caption.

4. Measured real-robot latency and queue underflow.

The following three-model measurements document the deployment configurations corresponding to the original manuscript's main real-robot comparison. Each model was run on a single RTX 5090 without other GPU computation.

ModelSync latency (ms)Async latency (ms)RTC latency (ms)Async / RTC underflow
pi0.5 (JAX)61.49 +/- 2.1761.49 +/- 2.6478.32 +/- 2.390 / 0
Single-frame Joint131.64 +/- 3.01131.64 +/- 3.01145.79 +/- 2.170 / 0
History-frame Joint161.43 +/- 2.81161.43 +/- 2.26178.91 +/- 2.570 / 0

These are model-call latencies, not end-to-end system latencies. The +/- values denote measured steady-state fluctuation ranges, not standard deviations or confidence intervals. Sync's intentional inference waits are distinct from asynchronous queue underflow. History-frame Joint uses its corresponding real-robot input resolution, which differs from simulation. Simulation uses separately calibrated deployment delays; the real robot incurs actual runtime without additional artificial waits.

The additional task experiments in Supplementary Table S2 evaluate only pi0.5 (JAX) and history-frame Joint. Their corresponding latency values are the same as those above; S2 does not contain new single-frame Joint task results. We distinguish these additional two-model task experiments from the three-model deployment information requested here.

In summary, we will correct the Table 10 label and explicitly document the checkpoint mapping, baseline sources, and measured deployment settings in the revised manuscript.

Reviewer_gpBdOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces LIBERO-Dynamic, an extension of the LIBERO simulation benchmark where objects are subject to a variety of motion patterns and object speeds and perception-action delays are modeled. Using contact-driven motion, the authors construct a dataset of 9000 dynamic trajectories. Comprehensive evaluations show improvements across several policies when dynamic data is introduced and additional physical demonstrations on a real robot highlight the impact of execution timing on dynamic behavior.

Critical strengths

The paper very clearly answers the question of why LIBERO-Dynamic is useful to the community. The evaluations are comprehensive and each one serves a different purpose that substantiates the claims, for instance:

  • We can see that the dynamic success rate decreases as the speed of the object increases, with the first drop being the most substantial. Additionally, the result that some models are more sensitive to this trend than others is a meaningful contribution.
  • The analysis of execution timing's impact is demonstrated on both simulated and real-world tasks and across multiple execution strategies.
  • There is a clear takeaway that dynamic data improves the models, though this is validated in simulation only which is acknowledged in the limitations.

Overall, I think the paper makes clear contributions and is valuable to the community. My initial recommendation is a clear accept, though that might be subject to change depending on the outcome of the discussions.

Critical weaknesses

I have not seen any critical weaknesses. The only counterintuitive result I saw was in Section 4.4. I was surprised that some models like pi_0.5 have significant improvements by combining static and dynamic data, whereas both single-frame Joint and history-frame Joint have a worse success rate with static + dynamic data compared to only dynamic data. From the paper, this trend does not seem clear to me. An analysis of the conditions under which the task fails might clarify this result.

Questions for authors
  1. Based on the trajectory overlay videos and appendix A.1, I am curious about the 25 tasks that contain multiple independently moving objects. Do all objects in those tasks move according to the same motion pattern and speed? In the videos, it appeared to be the case. If so, would introducing a mix of motion patterns and speeds have any impact on the success rate?
  2. Why do some examples use PyTorch for pi_0.5 and pi_0 and others use JAX? While this is a minor implementation detail, I was wondering if it was due to some practical limitation for strategy sweeps.
  3. What are the seven tasks used for training the real robot policies? That is not specified in A.14 and I could not find the details elsewhere.
14 comments
AuthorsOpenReview ↗

Response to W1: Why Mixed Static and Dynamic Training Affects Models Differently

Thank you for highlighting this counterintuitive result and suggesting a failure-condition analysis. For pi0.5, mixed training recovers part of the drawer-opening performance lost after dynamic-only fine-tuning while retaining most of the improvement on the moka-pot task. For history-frame Joint, additional failures in sequential two-object manipulation offset improvements on other tasks. An additional StarVLA experiment also favors mixed over dynamic-only training. Together, these observations suggest model- and task-dependent responses to the training recipe, rather than a universally beneficial mixture or an established architectural cause.

1. Task-level analysis of pi0.5

The following pi0.5 and history-frame Joint analyses use per-episode records at K=8, with 1,800 dynamic evaluation conditions per configuration and 45 conditions per task.

pi0.5 taskStatic baseDynamic-only fine-tuningStatic + dynamic fine-tuning
Turn on the stove and place the moka pot9/4534/4530/45
Open the middle drawer45/4523/4538/45

Overall success rates are 55.94%, 54.78%, and 63.78%, respectively. Mixed training recovers some drawer-opening performance while retaining most of the moka-pot improvement. This is consistent with retaining previously acquired manipulation capabilities while adapting to dynamic conditions, but these cases alone do not establish a forgetting mechanism.

2. Where history-frame Joint loses performance

Task groupDynamic-only fine-tuningStatic + dynamic fine-tuning
Sequential two-object pick-and-place180/270146/270

The mixed model has 34 fewer successes in this group but 29 more on the remaining tasks, for a net decrease of only five successes. In one matched test condition, it lifts the cheese box but never contacts the second target, the butter, whereas the dynamic-only model completes the task. Thus, completing the first grasp does not guarantee completion of the subsequent manipulation. These records locate the failure stage; they do not determine why the training recipe changes that behavior.

For single-frame Joint, we retain the manuscript's mixed-training success rate of 43.8%. The task-level trajectory analysis above concerns pi0.5 and history-frame Joint, rather than establishing the mechanism for every model.

3. An additional model-level check with StarVLA

We additionally compare the official StarVLA base, dynamic-only fine-tuning, and static + dynamic fine-tuning:

Training recipeSuccesses / episodesSuccess rateChange from base
Official base, no additional fine-tuning577/180032.06%N/A
Dynamic-only fine-tuning642/180035.67%+3.61 pp
Static + dynamic fine-tuning769/180042.72%+10.67 pp

All three use 20 Hz asynchronous execution, K=4, the same 1,800 dynamic conditions, and a 30-second time limit including inference waits. Mixed training exceeds dynamic-only training by 7.06 percentage points, agreeing with the pi0.5 trend rather than the decrease observed for the Joint variants. This checks the recipe response in another model; it does not isolate architecture as its cause. Architecture, pretraining, and training configuration may all contribute.

In summary, the task records explain which operations contribute to the aggregate changes, and StarVLA provides an additional example of a different model's recipe response. We interpret these as evidence of model- and task-dependent trade-offs, not proof that one recipe or architectural explanation applies universally. Thank you for motivating this analysis.

AuthorsOpenReview ↗

Response to Q1 (Part 1): Heterogeneous Object Motion and Mixed Speeds

Thank you for examining the videos and asking about heterogeneous motion. Your observation is correct: in the original evaluation, objects within a scene use the same speed tier and motion family. We additionally test mixed speeds, mixed motion families, and their combination. Mixed-condition success rates usually lie between the corresponding homogeneous endpoints, rather than consistently below both. This part gives the speed-mixing setup and complete results; the motion-family and joint-mixing results will follow separately.

1. Scope and comparison protocol

For each of pi0.5, single-frame Joint, and history-frame Joint, we evaluate the existing static-base, dynamic-only, and static + dynamic checkpoints. K is fixed as listed in each row. Every mixed condition is compared with homogeneous conditions using the same checkpoint and K; K is not necessarily identical across rows.

The speed-mixing comparison uses the common subset of 20 two-object tasks, not all 25 original multi-object tasks. We use 20 Hz asynchronous execution, a 30-second physical-time limit, early termination on success, and holding when the action queue is empty.

We pair two distinct speeds from 0.03, 0.06, and 0.09 m/s and include both assignments of the speeds to the objects. Each homogeneous-speed endpoint has 300 episodes. The mixed-speed pairs (0.03, 0.06), (0.06, 0.09), and (0.03, 0.09) have 226, 225, and 231 episodes, respectively. The two assignments are pooled using total successes divided by total episodes.

2. Complete speed-mixing results

Each cell reports homogeneous A / mixed A+B / homogeneous B, in percent; the mixed value is bold. Here, "mixed training" means static + dynamic fine-tuning, whereas "mixed speeds" refers to evaluation conditions.

Model / training recipe (K)0.03 + 0.06 m/s0.06 + 0.09 m/s0.03 + 0.09 m/s
pi0.5, static base (4)66.67 / 61.95 / 55.0055.00 / 54.67 / 48.6766.67 / 58.87 / 48.67
pi0.5, dynamic-only (4)71.67 / 70.35 / 64.3364.33 / 54.67 / 48.3371.67 / 56.28 / 48.33
pi0.5, mixed training (4)79.67 / 76.55 / 72.0072.00 / 63.11 / 56.3379.67 / 65.80 / 56.33
Single-frame Joint, static base (8)51.67 / 46.46 / 40.6740.67 / 39.56 / 34.3351.67 / 45.89 / 34.33
Single-frame Joint, dynamic-only (4)76.00 / 61.06 / 57.6757.67 / 49.78 / 39.6776.00 / 56.28 / 39.67
Single-frame Joint, mixed training (8)58.00 / 46.90 / 45.3345.33 / 37.33 / 30.3358.00 / 37.66 / 30.33
History-frame Joint, static base (4)57.33 / 45.58 / 42.6742.67 / 37.78 / 33.6757.33 / 45.45 / 33.67
History-frame Joint, dynamic-only (8)83.67 / 78.76 / 79.3379.33 / 67.11 / 57.3383.67 / 74.46 / 57.33
History-frame Joint, mixed training (8)84.33 / 80.97 / 75.3375.33 / 67.56 / 55.3384.33 / 69.26 / 55.33

In 26 of the 27 pooled comparisons, the mixed result lies between the two homogeneous endpoints. The exception is dynamic-only history-frame Joint at K=8 for 0.03 + 0.06 m/s: 78.76%, approximately 0.57 percentage points below the lower endpoint of 79.33%. Before pooling the two speed assignments, 44 of the 54 comparisons lie between their endpoints, indicating that object assignment matters.

In summary, heterogeneous speeds affect performance but do not universally make the evaluated conditions harder than both homogeneous endpoints. These are descriptive comparisons on the common task subset, not strictly episode-paired tests or evidence of statistical significance; they should not be generalized to every original multi-object task. The next parts address motion-family mixing and jointly mixed speeds and motion families.

AuthorsOpenReview ↗

Response to Q1 (Part 2): Mixed Motion Families at a Fixed Speed

Thank you for suggesting that objects should also have different motion patterns.

At a fixed speed of 0.06 m/s, 23 of the 27 mixed-motion comparisons lie between their corresponding homogeneous endpoints. The four exceptions deviate by at most 0.36 percentage points. Thus, heterogeneous motion patterns affect performance but are not uniformly harder than both homogeneous conditions in these comparisons.

We combine two of the straight, curved, and zigzag motion families. Each comparison uses the same checkpoint and K as its homogeneous endpoints. The homogeneous-mode endpoints use a common subset of 22 two-object tasks, with 110 episodes per homogeneous condition. We retain 20 Hz asynchronous execution, a 30-second physical-time limit including inference waits, early termination on success, and holding when the action queue is empty. K is fixed within each row, not necessarily across rows.

Each cell below is homogeneous A / mixed A+B / homogeneous B, in percent. The middle value is bold; above/below marks a result outside the endpoint interval. "Mixed training" means static + dynamic training, not mixed motion at evaluation.

Model / training recipe (K)Straight + curvedCurved + zigzagStraight + zigzag
pi0.5 static base (4)61.82 / 51.97 / 46.3646.36 / 53.88 / 62.7361.82 / 62.88 / 62.73 (above)
pi0.5 dynamic-only (4)61.82 / 62.57 / 63.6463.64 / 63.51 / 64.55 (below)61.82 / 62.86 / 64.55
pi0.5 mixed training (4)76.36 / 71.42 / 68.1868.18 / 69.47 / 70.9176.36 / 72.88 / 70.91
Single-frame Joint static base (8)42.73 / 43.49 / 44.5544.55 / 40.06 / 37.2742.73 / 39.71 / 37.27
Single-frame Joint dynamic-only (4)55.45 / 53.01 / 51.8251.82 / 55.74 / 60.9155.45 / 57.43 / 60.91
Single-frame Joint mixed training (8)44.55 / 43.48 / 42.7342.73 / 43.36 / 44.5544.55 / 44.19 / 44.55 (below)
History-frame Joint static base (4)46.36 / 43.47 / 40.9140.91 / 41.89 / 43.6446.36 / 44.68 / 43.64
History-frame Joint dynamic-only (8)82.73 / 78.24 / 75.4575.45 / 75.76 / 75.45 (above)82.73 / 78.61 / 75.45
History-frame Joint mixed training (8)79.09 / 79.83 / 80.9180.91 / 71.38 / 65.4579.09 / 71.56 / 65.45

The exceptions are pi0.5 static base for straight + zigzag (62.88%, 0.15 pp above), pi0.5 dynamic-only for curved + zigzag (63.51%, 0.13 pp below), single-frame Joint mixed training for straight + zigzag (44.19%, 0.36 pp below), and history-frame Joint dynamic-only for curved + zigzag (75.76%, 0.31 pp above).

These are descriptive comparisons on the corresponding common task subsets, not strictly episode-paired tests or evidence of statistical significance or absent interactions. They do not establish the same behavior for every original multi-object task. The following parts provide the jointly mixed speed-and-motion results, separated by model to keep each table readable.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Supplementary Material Update: Additional Real-Robot Demonstrations

Thank you for your constructive feedback. We have updated the supplementary material to make the additional real-robot demonstrations available for review.

After extracting the ZIP, please open Double_Click_to_Open.html and select New demos (the Real-robot measurements and demonstrations section). It contains 12 illustrative recordings covering two models, two tasks, and three execution modes, with four selectable camera views per recording. These clips illustrate execution behavior; the quantitative results come from the full experiments described in our responses, not from this small gallery alone.

This is primarily a demo-focused update. Some original-study figures and tables are still from the earlier version and have not yet been fully synchronized with the rebuttal updates. We will incorporate the corresponding updates into the revised manuscript and supplementary material together. In the meantime, please refer to our official responses for the updated quantitative results. The historical evaluation code is also explicitly labelled as an unchanged reference snapshot, rather than the implementation of the new wait-inclusive 30-second comparison.

We welcome any further questions or suggestions and would be happy to clarify any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.