- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
As I understand it, the paper makes target objects move in LIBERO. It generates enough dynamic demonstrations. The work compares models, execution strategies, and fine-tuning recipes. Then design a small real-robot study. To me, the main contribution is a unified evaluation of dynamic tasks.
Critical strengths
1.The problem is practically relevant and important today. Targets keep moving while the model is computing (time mismatch), which matters in real deployment.
2.The execution comparisons are useful. It shows how much deployment settings affect the same model.
3.The action-queue analysis is informative. It explains why shorter action horizons can also hurt performance.
Critical weaknesses
- The added value over existing work is unclear(this is important, it'll be better if you clear it). DOMINO already studies dynamic manipulation, and ReflexBench accounts for inference latency. I am not convinced what important insights this benchmark adds beyond them(Simply on Libero?).
2.The timing model may not reflect real deployment well, there's a huge huge gap between sim and real, especially LATENCY; real exp. is not enough, I am not sure whether the reported efficiency and strategy rankings would hold on, whether the benefit will cross the huge gap.maybe a much more clear report helps, to prove its value;
3.The generalization value of dynamic training is not established. Training and testing seem that share tasks, paths, and speeds, please prove the true generalization over 'dynamic' not just starting phase and soon;
Questions for authors
Please see Critical Weaknesses
6 comments
Response to W2: Timing Calibration and Matched Real-Robot Comparisons
Thank you for requesting a clearer real-deployment assessment. We agree that simulation alone cannot establish transfer of efficiency or strategy rankings.
The added real-robot comparison supports the efficiency trend: both Async and RTC complete more operations than Sync in the continuous task for both tested models, while conveyor success rates are close. The specific rankings remain task- and model-dependent.
Why calibrate a fixed model latency? Parallel simulation may run on A100/H100/H20 hardware, where inference time depends on hardware and competing CPU/GPU load rather than the target robot deployment. We independently calibrate each relevant configuration on a single RTX 5090 without other GPU computation and use that model-call latency in simulation. The observed steady-state fluctuations of roughly 2-3 ms support a fixed-value approximation for these model calls under this deployment condition, not a complete end-to-end latency model. Real-robot runs incur actual elapsed time, with no added artificial inference wait. Camera, communication and control-chain delays are not measured by the model-latency column.
Matched real-robot settings. All six conditions use 15 Hz control, 10 denoising steps and K=8, on a single RTX 5090 without competing GPU computation. Within each model, Sync/Async/RTC share weights, normalization and input processing. Async/RTC have zero observed queue underflow; Sync's intentional inference waits are not asynchronous underflow.
Conveyor handover: 50 attempts per condition, each limited to 30 s. Placing the apple in the bowl is success; timeout, dropping or knocking it off is failure. All six conditions share conveyor settings, objects and the initial-condition arrangement.
Whack-a-mole: one continuous one-hour run per condition. A valid press extinguishes the currently lit target; only then is the next target selected randomly. All six conditions use the same target-selection and triggering rules, not necessarily identical realized target sequences.
| Model | Execution | Model latency (ms) | Conveyor successes / 50 | Success rate | Valid presses in 1 h |
|---|---|---|---|---|---|
| pi0.5 (JAX) | Sync | 61.49 +/- 2.17 | 24 | 48% | 546 |
| pi0.5 (JAX) | Async | 61.49 +/- 2.64 | 25 | 50% | 655 |
| pi0.5 (JAX) | RTC | 78.32 +/- 2.39 | 23 | 46% | 685 |
| History-frame Joint | Sync | 161.43 +/- 2.81 | 17 | 34% | 428 |
| History-frame Joint | Async | 161.43 +/- 2.26 | 18 | 36% | 630 |
| History-frame Joint | RTC | 178.91 +/- 2.57 | 16 | 32% | 617 |
Here "+/-" denotes the measured steady-state fluctuation range, NOT a standard deviation or confidence interval. These are model-call latencies only. The press counts are also hourly throughputs because each run lasted one hour. There is only one continuous run per condition: its hundreds of presses are not independent experimental replicates, and we do not claim statistical significance.
Comparison with simulation and limits. In the corrected, wait-inclusive 30 s simulation comparison, mean capped completion times over all episodes are 20.51 s (Async), 23.13 s (Sync) and 22.58 s (RTC); unsuccessful episodes contribute 30 s, not an observed completion time. These use the same pi0.5 PyTorch checkpoint at 20 Hz but K=6/10/16 and calibrated model latencies of about 147/147/170 ms, respectively. They compare deployment operating points, not only the strategy label.
In the real continuous task, relative to Sync, Async throughput increases by 20.0% for pi0.5 and 47.2% for history-frame Joint; RTC increases it by 25.5% and 44.2%. Conveyor outcomes differ by only 1-2 successes out of 50. RTC leads pi0.5 throughput, whereas Async leads Joint throughput, so there is no universal ranking.
The tasks, metrics and configurations differ across simulation and hardware: real pi0.5 uses JAX rather than PyTorch, control runs at 15 rather than 20 Hz, and Joint uses a different real-image input resolution. Each uses its configuration-specific latency. The results support the practical value of overlapping computation and execution, rather than an exact numerical or ranking reproduction across simulation and hardware; the gains cannot all be attributed to model latency alone. Thank you for motivating this more explicit real-deployment comparison.
Response to W3: Held-Out Geometry and Speed Generalization
Thank you for asking us to test dynamic generalization beyond changes in starting phase.
The new geometry/speed holdout evaluation shows aggregate improvements over the corresponding static baseline in 11 of 12 fine-tuning comparisons, with one decrease. This supports dynamic generalization beyond starting-phase changes, specifically on held-out paths and interpolated speeds within known tasks and motion families, with model- and recipe-dependent benefits.
Holdout construction. Dynamic fine-tuning uses 960 demonstrations at only 0.03 and 0.09 m/s. We construct path holdouts from the actual demonstration geometries, excluding overlap with those dynamic-training paths, rather than merely changing starting phase. The unseen speed is 0.06 m/s, between the two training speeds.
| Strict holdout category | Conditions | Evaluation episodes |
|---|---|---|
| Unseen path / seen speed | 228 | 684 |
| Seen path / unseen speed | 98 | 294 |
| Unseen path / unseen speed | 130 | 390 |
| Total | 456 | 1,368 |
An additional 24 repeated-geometry controls are excluded from this strict holdout set. "Unseen" here refers to the dynamic-training conditions; tasks and motion families remain known.
Evaluation and training controls. Each model/recipe/K combination is evaluated on all 456 conditions with evaluation seeds 1001, 1002 and 1003, totaling 1,368 episodes. Success rate is pooled successes divided by pooled episodes, not a claim of three independently trained models. These are evaluation seeds, not independent training-seed replications.
The three models share holdout conditions, evaluation seeds and time budget. We report both fixed K=4 and K=8, using asynchronous execution at 20 Hz with a 30 s simulation-time limit including waits. Inputs and native action-chunk lengths retain each model's settings, so this is not a fully architecture-matched comparison.
All six fine-tuning branches start independently from their respective static baselines, run for 3,000 updates with effective batch size 128, and use the predetermined final checkpoint. Dynamic-only updates contain 128 dynamic examples; mixed updates contain 64 dynamic plus 64 static examples. Total updates and sample budgets are matched, but mixed training has half as much dynamic exposure; recipe differences therefore cannot be attributed solely to adding static data.
Overall strict-holdout results. Entries are success rates (%); parentheses show changes in percentage points from the same-model, same-K static baseline.
| Model | K | Static baseline | Dynamic-only | Static + dynamic |
|---|---|---|---|---|
| pi0.5 | 4 | 54.97 | 65.35 (+10.38) | 65.13 (+10.16) |
| pi0.5 | 8 | 53.00 | 59.87 (+6.87) | 60.60 (+7.60) |
| Single-frame Joint | 4 | 33.92 | 37.65 (+3.73) | 36.62 (+2.70) |
| Single-frame Joint | 8 | 35.09 | 35.89 (+0.80) | 34.87 (-0.22) |
| History-frame Joint | 4 | 40.20 | 47.37 (+7.16) | 47.95 (+7.75) |
| History-frame Joint | 8 | 38.74 | 48.03 (+9.28) | 48.76 (+10.01) |
The pi0.5 gains range from 6.87 to 10.38 points and history-frame Joint gains from 7.16 to 10.01 points. Single-frame Joint improves less and decreases by 0.22 points under mixed training at K=8. Mixed training is not universally better than dynamic-only: it is slightly higher for history-frame Joint at both K values, lower for single-frame Joint, and varies with K for pi0.5.
Thus, the test goes beyond starting-phase variation through geometric and speed holdouts, while the supported result remains aggregate held-out improvement with model/recipe dependence. We do not infer per-subset gains from pooled scores, statistical significance, robustness across independent training seeds, or generalization to unseen tasks or extrapolated speeds. Thank you for suggesting this more stringent test.
Response to W1: Added Value Beyond Existing Dynamic Benchmarks
Thank you for asking us to clarify the contribution beyond applying dynamic motion to LIBERO.
LIBERO-Dynamic provides a common setting to study how manipulation capability, training recipes and execution timing interact. Its added value is not latency modeling or asynchronous execution alone: the experiments show why static performance and low latency are insufficient predictors of dynamic performance, and why frequent feedback must be balanced against continuous action supply.
Relationship to prior work. We recognize DOMINO and ReflexBench as important foundations.
DOMINO covers 35 dynamic tasks. Its paper describes kinematic target motion and stopping autonomous motion upon gripper contact; the public code defaults to this behavior but allows task-level overrides. Our objects instead move through frictional contact with a moving platform. First gripper contact does not disable that external drive: subsequent motion is determined by the physics solver, and lifting can separate an object from the platform. Thus, first contact is not equivalent to stable grasping; inadequate tracking can still lead to tipping or slipping. This comparison concerns DOMINO's published/default configuration, not an absence of rigid-body contact physics.
In the inspected DOMINO pi0.5 deployment wrapper, policy inference returns before actions execute; we did not find explicit scene advancement by measured inference duration in that path. LIBERO-Dynamic advances physical time while waiting for actions. We limit this observation to the inspected path, not every DOMINO implementation.
ReflexBench explicitly supports latency modeling and synchronous/asynchronous evaluation across six reaction-sensitive tasks. The ReflexVLA paper also reports static LIBERO results. Our complementary scope retains 40 task objectives from LIBERO's four suites under common dynamic paths, speeds and execution protocols, spanning objects, spatial relations, goals and longer manipulation sequences. Neither asynchronous execution nor history input itself is claimed as new.
What the common evaluation reveals.
Static scores and latency alone are insufficient. The six evaluated models have high reported static LIBERO scores, yet diverge under common dynamic conditions and differ in sensitivity to nonzero target speeds. Models with similar inference latency also differ in dynamic performance. The static scores are official reports, not protocol-matched reevaluations, so static-to-dynamic gaps cannot be attributed solely to motion.
Execution settings matter even at fixed weights. The same pi0.5 checkpoint yields different success, completion-time and smoothness trade-offs across execution configurations. In the corrected common-30-second comparison, Async/Sync/RTC achieve 48.39%/40.00%/37.83% success. These are deployment operating points with different K values (6/10/16) and calibrated model latencies (about 147/147/170 ms), not an isolated strategy-only causal comparison.
Shorter action horizons are not always better. Fixed-weight asynchronous sweeps expose two regimes: very short execution lengths can exhaust the queue before new actions arrive, accompanying lower success; within the tested underflow-free regime, shorter open-loop execution is associated with higher success. Horizon selection must balance feedback freshness, latency and queue continuity rather than simply minimize K.
Dynamic-data benefits depend on the model and recipe. In the training-step-matched Joint comparison, the history-frame variant benefits more from dynamic-only fine-tuning, and its best tested execution length changes. Dynamic-only versus mixed training also behaves differently across models. This motivates studying temporal inputs, recipes and execution length together, not claiming a universal recipe or a proven architectural cause.
Together, these results provide a controlled basis for studying model-training-execution interactions. We appreciate the opportunity to make these contributions clearer.
Sources: DOMINO Sec. 2.2 and App. A.2; versioned evaluation configuration; inspected pi0.5 wrapper; Reflex Sec. III and Table II.