acceptodds
Under review as a conference paper at ICLR 2027

The World Does Not Pause: Real-Time Benchmarks and Training-Free Acceleration for Dynamic Manipulation

Abstract

Dynamic manipulation requires timely actions as the world continues to evolve during policy inference. In vision-language-action (VLA) and world action models (WAM), inference latency can leave actions conditioned on outdated observations. We introduce Dynamic-LIBERO and Dynamic-RoboTwin, which combine controlled target motion with inference-time world evolution while largely preserving established task goals for frozen-policy evaluation. We propose PACE (Policy Acceleration through Cached Execution), a training-free runtime that reuses observation representations and action computation across replanning calls and integration steps, and compiles the remaining computation without policy fine-tuning. Across two simulated benchmarks and two policies, PACE accelerates inference by up to 3.8× and improves dynamic task success by up to 42.8 percentage points over standard deployments, achieving the highest success in all eight dynamic settings. On a physical robot, PACE reduces per-chunk inference stalls from 159 to 69 ms, maintains or improves static-task success, shortens completion times, and increases dynamic success counts. Together, these results highlight that reducing inference latency is critical for successful dynamic manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Inference Acceleration, Dynamic Manipulation, World Action Model, Vision-Language-Action Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2xtiOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

While policy is computing actions, scene and robot’s state continue to change, and inference latency can cause the action to correspond to outdated observations.

This work establishes Dynamic-LIBERO and Dynamic-RoboTwin benchmarks and proposes PACE, a caching acceleration method, to investigate how faster inference can improve dynamic manipulation.

Critical strengths

The motivation is sound, and dynamic manipulation is a valuable research topic; the benchmark is designed as a simple modification of commonly used benchmarks, making it easy to understand and use; PACE improving dynamic manipulation through accelerated reasoning is intuitive.

The benchmark is expected to facilitate research in this area within the robotics community;

Critical weaknesses

Some of figures and discussion are rather rough, making it difficult to read for readers unfamiliar with the background knowledge relevant to this work; for example, Figure 3 and Section 6.2. I have no objection to using LLMs to generate figures or text, but they should be reviewed and polished by humans to ensure that it is clearly expressed and easy to understand.

Experiments in simulation are based on a limited number of policies, real-world experiments are also conducted with few repetitions; since the method does not involve large scale training, consideration should be given to supplementing the work with additional experiments to support its main idea.

PACE’s acceleration stems from a combination of various approaches, such as reusing different types of computations; however, the specific contributions of each of these approaches to the overall performance require further discussion.

Questions for authors

It is reasonable to implement a new benchmark by adding a layer of lightweight modifications to an existing benchmark; however, considering portability to other benchmarks, such as RoboDojo, it would be helpful to discuss what interface support this benchmark requires for underlying benchmark.

See weakness; consider providing some discussion or additional details for each point to support the main idea of the work.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 2xti

We sincerely thank you for your careful reading of our paper and your constructive comments.

W1: Clarity of Figure 3 and Section 6.2

We redrew Figure 3 with larger labels and separate input, encoding, and action-generation panels. Sections 5.1-5.3 and Appendix E.1, Table 11 explain the order: velocity reuse first, then residual reuse or full evaluation on a miss, with every native integration update retained. Section 6.2 distinguishes paused-world Original/Modified-only from measured-delay Static. FastWAM's Q=32, r=10 queue leaves 22 old actions that can move the robot before installing a chunk conditioned on an earlier observation. Appendix B.2 specifies replay and fallback. Zero-tail Q=r=10 yields 90.0%/92.0% Baseline/PACE Static success and PACE moving-target gains of 14.0/18.0 percentage points at v1/v2 (Appendix E.2, Table 12).

W2: Policy coverage and experimental repetitions

Simulation covers FastWAM and π0.5 across 50 Dynamic-LIBERO and 45 Dynamic-RoboTwin variants. We added queue, fewer-step, periodic-reuse, ungated-reuse, and component comparisons (Appendices E.2-E.4, Tables 12-14), plus direct BAC/VLA-Cache evaluations on Dynamic-LIBERO (Appendix E.8, Table 18; v1=1.2 cm/s):

PolicyMethodStatic SR (%)Static latency (ms)v1 SR (%)v1 latency (ms)
FastWAMBAC18.6334.57.8336.1
FastWAMVLA-Cache32.2284.019.3264.6
π0.5BAC94.8185.876.6183.4
π0.5VLA-Cache77.689.460.688.7

PACE reaches 43.0%/81.2% v1 success on FastWAM/π0.5 (Table 2), exceeding BAC by 35.2/4.6 and VLA-Cache by 23.7/20.6 percentage points. This supports PACE's dynamic-performance advantage over existing caching methods.

Physical evaluation now covers five tasks with 25 trials per method and condition. Conveyor-belt grasping and rotating cup-rack placement test moving-object acquisition and moving-target alignment (Appendix E.5, Figure 6, Table 15):

TaskConditionBaseline SRPACE SRBaseline latency (ms)PACE latency (ms)
Conveyor-belt object graspingStatic25/2525/25157.968.4
Conveyor-belt object graspingDynamic9/2512/25159.669.0
Rotating cup-rack placementStatic11/2512/25156.868.9
Rotating cup-rack placementDynamic3/255/25159.269.6

Dynamic success gains are 12/8 percentage points with 56.1%-56.8% lower latency, approximately 2.3x acceleration. Together with 2.86-3.76x simulation speedups (Section 6.3, Table 2), these results show substantial acceleration and higher observed dynamic success in both simulated and physical environments. On T1-T3, Static success is maintained or improved; at 3.5 cm/s, each task improves from 0/25 to 3/25 (Section 6.5, Table 4; Appendix D.1). Appendix E.6, Table 16 reports two-sided 95% Wilson intervals.

We have also uploaded videos of the newly added real-world experiments as supplementary material.

W3: Contributions of individual reuse mechanisms

Observation reuse saves encoding; residual reuse skips hidden transformations while retaining the current embedding and output head; velocity reuse bypasses both but retains every integration update (Appendix E.1, Table 11). Appendix E.4, Table 14 reports latency increases when each component is removed:

Removed componentFastWAM / LIBERO (ms)π0.5 / LIBERO (ms)FastWAM / RoboTwin (ms)π0.5 / RoboTwin (ms)
Visual reuse+48.6+13.9+17.2+3.0
Residual reuse+85.8+12.2+29.5+7.8
Velocity reuse+223.6+131.8+118.9+167.5

Every removal increases latency and lowers both backbones' Dynamic-LIBERO dynamic success (Section 6.4, Table 3). Each module contributes to acceleration and closed-loop performance, supporting PACE's ordered runtime architecture across three complementary boundaries: observation encoding, cross-call hidden transformations, and within-chunk velocities.

Q1: Portability to benchmarks such as RoboDojo

The RoboCasa examples show that localized simulator adaptation can add controlled target motion to existing scenes (Appendix E.7, Figure 7, Table 17). Adapters connect observation and inference timing, world stepping, action execution, and motion hooks under a shared world clock while preserving task semantics and success criteria (Appendices A.2, B.1-B.2). We will continue expanding the dynamic datasets and task suites, adding Dynamic-RoboCasa and Dynamic-RoboDojo in subsequent releases and publishing usable benchmark-modification tools.

Q2: Additional discussion addressing the weaknesses

The revised presentation, expanded comparisons, physical trials, and component analysis address these points (Sections 5.1-5.3, 6.2-6.5; Appendices E.1-E.8).

Thank you to the author for the detailed response; the explanation has addressed my main concerns. I will maintain my current positive rating and increase the confidence level for that rating.

Thank you for your thoughtful review and constructive feedback. We are glad that our response has addressed your main concerns, and we sincerely appreciate your time and careful consideration of our work.

Reviewer_SpseOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study the impact of inference latency on dynamic robotic manipulation, where the environment continues to change while a VLA policy is generating actions. They introduce Dynamic-LIBERO and Dynamic-RoboTwin, two benchmarks that extend existing manipulation tasks with moving targets and continuous world evolution during inference. The authors also propose PACE, a training free acceleration framework that combines visual feature caching, cross-chunk residual reuse, and velocity reuse during action generation. Experiments on FastWAM and π0.5 show up to 3.8× inference speedup and improvements in dynamic task success. The authors further validate the approach on a real robot, showing reduced inference stalls and some improvements in manipulation performance.

Critical strengths

The paper studies an important practical problem that is often overlooked in existing VLA evaluation. I appreciate that the authors connect inference efficiency with actual task success rather than reporting latency reduction alone. The two dynamic benchmarks provide a useful evaluation setting, and the comparison between paused and continuously evolving environments in Table 1 is particularly interesting. The experiments also cover two different policy architectures, several acceleration baselines, and real robot experiments. The ablation results suggest that the proposed caching components contribute to the overall acceleration.

Critical weaknesses

My main concern is the limited methodological novelty. The authors propose PACE, but most of its components are closely related to existing caching and reuse techniques. Visual caching, residual reuse, and velocity reuse are individually quite straightforward, and the paper does not clearly explain what new technical insight is gained from combining them. Although the reported improvements are impressive, I would like to see stronger comparisons with existing methods such as VLA-Cache and BAC, which are discussed in the related work but not evaluated. The comparison with AdaVLA also uses only its sampler without channel pruning, which makes it difficult to judge the advantage over the complete method. Another concern is the evaluation protocol. In Table 1, FastWAM success on Dynamic-LIBERO drops from 91.2% to 0.4% simply by allowing the world to evolve during inference, even with stationary targets. This is a surprisingly large difference. Appendix B.2 explains that inference delays are simulated through replay, and the robot holds its pose when the action queue is exhausted. I am not fully convinced that the resulting performance drop reflects typical asynchronous execution rather than specific executor settings. The real robot experiments are useful, but they use synchronous execution and only ten trials per condition. At higher target speeds, the improvement is often just one additional successful trial, making the practical benefit less conclusive.

Questions for authors

Could the authors explain why FastWAM experiences such a dramatic performance drop in the static setting of Table 1, and whether the same behavior occurs under a different asynchronous execution strategy? It would also be helpful to report how frequently the action queue becomes empty during inference. Regarding the method itself, can the authors provide comparisons with VLA-Cache, BAC, or a simpler combination of existing acceleration techniques under matched latency budgets? I would also like to understand how sensitive PACE is to its caching thresholds, particularly when objects move rapidly or become partially occluded. Finally, could the authors provide uncertainty estimates for the real robot success rates to better support the reported improvements?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Spse (pt1)

We sincerely thank you for your careful reading of our paper and your constructive comments.

W1: Methodological novelty and runtime composition

PACE organizes established caches into an ordered runtime: observation reuse reduces encoding; residual reuse skips hidden transformations while retaining the current embedding and output head; velocity reuse bypasses both. The velocity gate is checked first, so residual or full-network evaluation runs only when a new velocity is needed. Every path retains native integration updates, weights, and the action interface (Sections 5.1-5.3, Figure 3; Appendix E.1, Table 11). Appendix E.4, Table 14 reports component latency effects:

Removed componentFastWAM / LIBERO (ms)π0.5 / LIBERO (ms)FastWAM / RoboTwin (ms)π0.5 / RoboTwin (ms)
Visual reuse+48.6+13.9+17.2+3.0
Residual reuse+85.8+12.2+29.5+7.8
Velocity reuse+223.6+131.8+118.9+167.5

Removing any module increases latency, demonstrating a measurable contribution at each reuse boundary; velocity reuse has the largest effect. In Section 6.4, Table 3, removing any component lowers dynamic success for both backbones on Dynamic-LIBERO. Together, these results support PACE's runtime architecture: ordered reuse across the complementary boundaries of observation encoding, cross-call hidden transformations, and within-chunk velocity evaluation.

W2: Comparisons with VLA-Cache and BAC

We now directly evaluate both methods on Dynamic-LIBERO (Appendix E.8, Table 18; pt2, Q3). At v1, PACE reaches 43.0% on FastWAM versus 7.8% for BAC and 19.3% for VLA-Cache; on π0.5, it reaches 81.2% versus 76.6% and 60.6%, respectively. PACE therefore achieves higher dynamic success than both existing caching methods. Appendix E.3, Table 13 also compares fewer-step Euler, periodic within-chunk reuse, and ungated multi-level reuse.

W3: AdaVLA configuration

Section 6.1 and Appendix C.5 specify that the evaluated AdaVLA configuration uses its curvature-based adaptive sampler on π0.5 with channel pruning disabled. Against this configuration, PACE reduces Dynamic-LIBERO latency from 153.8 to 97.3 ms and increases v1/v2 success from 75.8%/58.6% to 81.2%/65.9%, gains of 5.4/7.3 percentage points (Table 2). On Dynamic-RoboTwin, latency falls from 124.4 to 82.3 ms and v1 success rises from 17.8% to 44.4%, combining faster inference with higher task success.

W4: Dependence on executor settings

Section 6.2 and Appendix B.2 clarify delay replay, queue replacement, and empty-queue fallback. For FastWAM on Dynamic-LIBERO, zero-tail Q=r=10 admission retains the native prediction length H=32; Baseline/PACE Static success reaches 90.0%/92.0%; moving-target success is 32.0%/12.0% versus 46.0%/30.0% at v1/v2, gains of 14.0/18.0 percentage points. PACE takes 155.9 ms versus 628.5 ms (Appendix E.2, Table 12). Queue handling strongly affects Static performance, while PACE's moving-target advantage persists under both tested settings, which share the same replay semantics.

W5: Real-robot repetitions, dynamics, and practical benefit

The physical evaluation now covers five tasks, with 25 trials per method and condition. Added conveyor-belt grasping and rotating cup-rack placement introduce motion during acquisition and placement alignment (Appendix E.5, Figure 6, Table 15):

TaskConditionBaseline SRPACE SRBaseline latency (ms)PACE latency (ms)
Conveyor-belt object graspingStatic25/2525/25157.968.4
Conveyor-belt object graspingDynamic9/2512/25159.669.0
Rotating cup-rack placementStatic11/2512/25156.868.9
Rotating cup-rack placementDynamic3/255/25159.269.6

Dynamic success increases on both tasks while inference latency falls by 56.1%-56.8%, approximately 2.3x acceleration. Together with the 2.86-3.76x simulation speedups (Section 6.3, Table 2), these results show substantial acceleration and improved observed dynamic task success in both simulation and physical deployment. The original T1-T3 trials also use the expanded count: at 3.5 cm/s, PACE succeeds in 3/25 trials on each task versus 0/25 for Baseline. Static success is maintained or improved, with successful Static-trial completion time reduced by 4.3%-26.5% (Section 6.5, Table 4). Physical execution remains synchronous with identical fine-tuned weights (Appendix D.1); Q5 reports uncertainty.

We have also uploaded videos of the newly added real-world experiments as supplementary material.

AuthorsOpenReview ↗

Response to Reviewer Spse (pt2)

Q1: Static degradation and alternative queue strategies

Section 6.2 distinguishes paused-world Modified-only (91.2% FastWAM success) from measured-delay Static (0.4%). With Q=32, r=10, 22 old actions remain at replanning. Their relative end-effector increments change robot state during delay replay, leaving the new chunk conditioned on an outdated observation. Appendix E.2, Table 12 compares queue admission at the native prediction length:

Queue settingMethodLatency (ms)Static (%)v1 (%)v2 (%)
Q=32, r=10Baseline598.60.40.20.8
Q=32, r=10PACE159.182.443.027.8
Q=10, r=10Baseline628.590.032.012.0
Q=10, r=10PACE155.992.046.030.0

Zero-tail admission restores Baseline Static success to 90.0%, while PACE retains higher dynamic success under both settings. Appendix B.2 specifies sequential replay: queued actions advance dynamics; empty-queue fallback holds the controlled pose while advancing target motion and refreshing kinematics.

Q2: Queue exhaustion during inference

Appendix E.2 reports means of 222.2/59.0 old-action ticks and 12.2/2.9 empty-queue hold ticks per episode for Baseline/PACE, quantifying cumulative old-action execution and waiting.

Q3: Existing caching methods and matched-latency comparisons

Appendix E.3, Table 13 compares frozen policies on Dynamic-LIBERO at v1=1.2 cm/s and v2=2.0 cm/s:

PolicyConfigurationLatency (ms)Static (%)v1 (%)v2 (%)
FastWAMFewer-step Euler, K=1164.374.034.020.0
FastWAMPeriodic within-chunk reuse, N=2161.479.639.225.0
FastWAMUngated multi-level reuse110.50.00.02.0
FastWAMPACE, K=10159.182.443.027.8
π0.5Fewer-step Euler, K=2127.0100.074.062.0
π0.5Ungated multi-level reuse70.884.056.054.0
π0.5PACE, K=1097.394.281.265.9

PACE outperforms fewer-step Euler and periodic reuse at closely matched FastWAM latency; on π0.5, it is faster and more successful than two-step Euler. Faster ungated reuse has lower success on both policies.

We added direct BAC/VLA-Cache comparisons (Appendix E.8, Table 18), with separately measured latencies for Static and v1:

PolicyMethodStatic SR (%)Static latency (ms)v1 SR (%)v1 latency (ms)
FastWAMBAC18.6334.57.8336.1
FastWAMVLA-Cache32.2284.019.3264.6
π0.5BAC94.8185.876.6183.4
π0.5VLA-Cache77.689.460.688.7

PACE's v1 success of 43.0%/81.2% (FastWAM/π0.5; Table 2) exceeds BAC by 35.2/4.6 and VLA-Cache by 23.7/20.6 percentage points. π0.5 VLA-Cache takes 88.7 ms at v1 but reaches 60.6% success; PACE reaches 81.2% with the 97.3 ms latency reported in Table 2. These results support ordered, selective reuse for dynamic closed-loop performance.

Q4: Threshold sensitivity and partial occlusion

Appendices C.2-C.3 and E.4 specify change thresholds and bounded reuse. Motion or partial occlusion that pushes image change above the visual threshold triggers fresh encoding. Section 6.3, Figure 4 and Appendix B.3 sweep 0-3.0 cm/s at fixed thresholds; Appendix E.3 compares gated and ungated reuse, assessing motion performance and change-based selection.

Q5: Uncertainty in real-robot success

Appendix E.6, Table 16 reports two-sided 95% Wilson intervals across all five tasks, with 25 trials per method and condition. Added Dynamic results are:

TaskBaseline successesBaseline SR [95% CI] (%)PACE successesPACE SR [95% CI] (%)
Conveyor-belt object grasping9/2536.0 [20.2, 55.5]12/2548.0 [30.0, 66.5]
Rotating cup-rack placement3/2512.0 [4.2, 30.0]5/2520.0 [8.9, 39.1]

Both tasks show higher observed Dynamic success with 56.1%-56.8% lower inference latency, approximately 2.3x acceleration (Appendix E.5, Table 15). Together with 2.86-3.76x simulation speedups and higher dynamic success (Section 6.3, Table 2), these results show substantial acceleration and improved observed dynamic task performance in simulated and physical environments.

Thank you again for your feedback. We would appreciate it if you could let us know whether our response has addressed your concerns. If there are any remaining questions or concerns, we would be happy to discuss them further and provide additional clarification.

Reviewer_dCZYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how inference latency affects dynamic robot manipulation. It introduces Dynamic-LIBERO and Dynamic-RoboTwin, where targets move and the environment advances during policy inference. It also proposes PACE, a training-free runtime method.

Critical strengths
  1. The paper addresses a practical problem. In real deployment, a good action may become unsuitable if it arrives too late. Evaluating latency together with closed-loop task success is more useful than reporting inference speed alone.

  2. The benchmark includes useful controls. The Original, Modified-only, and Static settings help separate the effects of scene changes from those of inference-time world evolution. Using matched initial states and motion seeds across methods is also a good choice.

Critical weaknesses

The real-robot evidence is still limited and designed tasks are not-that complex and dynamic. Each real-robot condition has only ten trials. So the results are not convincing.

Questions for authors
  1. How does PACE compare with fewer-step sampling and fixed-interval reuse at similar inference latency?
2 comments
AuthorsOpenReview ↗

Response to Reviewer dCZY

We sincerely thank you for your careful reading of our paper and your constructive comments.

W1: Real-robot task complexity, dynamics, and number of trials

We expanded the physical evaluation to five tasks, with 25 trials per method and condition. The added conveyor-belt grasping and rotating cup-rack placement tasks require acquiring a moving object and aligning a cup handle with a peg whose position and orientation change during placement. Their Static and Dynamic results are reported in Appendix E.5, Figure 6 and Table 15:

TaskConditionBaseline SRPACE SRBaseline latency (ms)PACE latency (ms)
Conveyor-belt object graspingStatic25/2525/25157.968.4
Conveyor-belt object graspingDynamic9/2512/25159.669.0
Rotating cup-rack placementStatic11/2512/25156.868.9
Rotating cup-rack placementDynamic3/255/25159.269.6

The Dynamic success gains are 12 and 8 percentage points, respectively, with 56.1%-56.8% lower inference latency across the displayed conditions (2.28-2.31x speedup). The original T1-T3 tasks also use the expanded trial protocol (Section 6.5, Table 4; Appendix D.1): PACE maintains or improves Static success counts and shortens successful Static-trial completion times; at 3.5 cm/s, each task improves from 0/25 to 3/25 successes. Appendix E.6, Table 16 reports two-sided 95% Wilson intervals for each method-condition setting. Together with the simulation results (2.86-3.76x speedup; Section 6.3, Table 2), these experiments demonstrate substantial inference acceleration and higher observed dynamic-task success in both simulation and physical deployment.

We have also uploaded videos of the newly added real-world experiments as supplementary material.

Q1: Fewer-step sampling and fixed-interval reuse at similar inference latency

Appendix E.3, Table 13 compares fewer-step sampling, fixed-interval within-chunk reuse, and ungated multi-level reuse on Dynamic-LIBERO with frozen weights and measured-delay replay, at v1=1.2 cm/s and v2=2.0 cm/s:

PolicyConfigurationLatency (ms)Static (%)v1 (%)v2 (%)
FastWAMFewer-step Euler, K=1164.374.034.020.0
FastWAMPeriodic within-chunk reuse, N=2161.479.639.225.0
FastWAMUngated multi-level reuse110.50.00.02.0
FastWAMPACE, K=10159.182.443.027.8
π0.5Fewer-step Euler, K=2127.0100.074.062.0
π0.5Ungated multi-level reuse70.884.056.054.0
π0.5PACE, K=1097.394.281.265.9

At closely matched FastWAM latency, PACE's v1/v2 success exceeds one-step Euler by 9.0/7.8 percentage points and periodic reuse by 3.8/2.8 points. On π0.5, PACE has both lower latency and higher moving-target success than two-step Euler. PACE also achieves higher moving-target success than the faster ungated reuse on both policies, while retaining the native integration grid and using change gates to select reuse.

We also added direct VLA-Cache and BAC evaluations (Appendix E.8, Table 18). The table reports separately measured Static and v1 latencies:

PolicyMethodStatic SR (%)Static latency (ms)v1 SR (%)v1 latency (ms)
FastWAMBAC18.6334.57.8336.1
FastWAMVLA-Cache32.2284.019.3264.6
π0.5BAC94.8185.876.6183.4
π0.5VLA-Cache77.689.460.688.7

PACE's v1 success is 43.0% on FastWAM and 81.2% on π0.5 (Table 2), exceeding BAC/VLA-Cache by 35.2/23.7 and 4.6/20.6 percentage points, respectively. These results reinforce the benefit of ordered, change-gated reuse for dynamic task success.

Thank you again for your feedback. We would appreciate it if you could let us know whether our response has addressed your concerns. If there are any remaining questions or concerns, we would be happy to discuss them further and provide additional clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.