acceptodds
Under review as a conference paper at ICLR 2027

RealtimeWAM: How Fast Can I Run My World Action Model?

25 views

Abstract

World Action Models (WAMs) combine visual dynamics modeling with action generation, but their high inference latency limits responsive robot control. Recent efforts accelerate inference by removing explicit future video generation at test time, as in FastWAM, an approach that requires a specially tailored architectural design. More general caching strategies exploit feature redundancy, but redundancy alone does not capture the changing computational demands of closed-loop control. To address these challenges, we present RealtimeWAM, a general, training-free framework that coordinates parallel execution with adaptive computation for low latency inference across diverse WAM architectures. We exploit layerwise dependencies to overlap observation processing with prediction. However, concurrent branches still compete for GPU resources, limiting the benefit of parallel execution. We therefore adapt computation throughout the pipeline through selective reuse, caching observation features in visually stable regions and reusing Transformer residuals while reserving additional refinement for small predicted adjustments. We evaluate RealtimeWAM on FastWAM and OpenWAM across RoboTwin, LIBERO, and LIBERO-Plus. On an RTX 4090, measured mean inference latencies are 24.09 and 63.09 ms, corresponding to average speedups of 8.90× and 10.67×. Average success rates are 82.75% and 87.41%, respectively, within 0.02 and 0.53 percentage points of native inference. Across five real-world tasks, RealtimeWAM improves average success rates over native inference by 17.2 and 37.2 percentage points on FastWAM and OpenWAM, respectively. Project page: https://anonymous.4open.science/w/realtimewam/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CPm8OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper focuses on improving the inference latency of World Action Models (WAMs). Unlike previous works that focus on architectural changes, this paper proposes coordinating parallel execution with adaptive computation to achieve low-latency inference across diverse WAM architectures, resulting in a training-free framework called RealtimeWAM.| On an RTX 4090, RealtimeWAM leads to approximate speedups of 8.90× and 10.67× in measured mean inference latencies for FastWAM and OpenWAM. The average success rates are comparable to those of native inference, respectively, within 0.02 and 0.53 percentage points.

Critical strengths
  1. This work proposes to improve the inference latency of WAMs at the execution level, which can speed up diverse WAMs without the need for of changing architectures. This infra level improvement can (nearly) universally benefit many existing and future WAM designs.

  2. The significant speedups with negligible performance drops, evidenced by the experiments, are pronounced.

Critical weaknesses
  1. There are three major proposed techniques: 1). Dependency-aware parallel execution; 2). Hardware-aware token reuse; 3). Motion-adaptive refinement. Table 2 provides the only latency in a bottom-up manner from the native inference. The ablation studies provide both latency and success rate in a top-down manner from the full proposed method. A bottom-up ablation study providing both latency and success rate can largely strengthen the paper, which can provide a better understanding of the success-latency trade-off for each component. From my point of view, the first component seems lossless, while the other two rely on some hypotheses. The extra ablation can help me better understand the proposed techniques.
Questions for authors
  1. Please read the weaknesses

  2. I will also be interested in the success-latency performance under the distribution shift. For example, training on LIBERO and evaluating on LIBERO-Plus will be a good benchmark. Can you also add the extra evaluation on LIBERO-Plus with the existing checkpoints on LIBERO? (at least for RealtimeWAM)

1 comment

We thank the reviewer for recognizing the value of execution-level acceleration. The requested bottom-up ablation helps clarify each component's contribution; we report it below together with the LIBERO-Plus protocol. Below, F denotes the number of full Transformer evaluations within the ten denoising steps.

W1. Bottom-up ablation with both success rate and latency.

Starting from native OpenWAM inference, we add compilation and parallel execution at a fixed 10F budget, and then add observation-token FFN reuse and adaptive computation separately and in combination:

ConfigurationRoboTwin success rateMean request latency
Native inference, 10F93.85%672.83 ms
+ Compilation and CUDA Graph, 10F93.85%450.90 ms
+ Parallel execution and associated kernels, 10F93.85%223.27 ms
+ Observation-token FFN reuse, 10F93.46%217.24 ms
+ Adaptive 2F/4F93.78%71.34 ms
+ Observation-token FFN reuse + adaptive 2F/4F93.40%63.09 ms

The last three rows build on parallel execution. Latencies average the four evaluation settings; on RoboTwin alone, the last two rows take 72.10 / 63.76 ms (paper Table 3). Compilation and parallel execution keep the computation unchanged and match native outputs to numerical precision (paper Table 8), so they share the native success rate.

Parallel execution gives a 2.02× speedup. FFN reuse cuts first-forward latency from 37.12 to 31.62 ms at about a 0.4-point drop; adaptive computation lowers the average number of evaluations to about 2.8 with almost no change in success. For reference, fixed 2F and fixed 4F give 89.62% / 54.70 ms and 93.68% / 95.26 ms.

Q2. Evaluating the LIBERO checkpoints on LIBERO-Plus.

The paper's LIBERO-Plus results already follow this protocol: we directly evaluate the LIBERO-finetuned checkpoints with all weights frozen under distribution shift.

BackboneNative success rateRealtimeWAM success rate
FastWAM49.80%51.20%
OpenWAM64.75%63.85%

RealtimeWAM changes success by +1.40 and −0.90 points relative to native inference.

Reviewer_2UxfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to reduce the cost of both action-only and joint video–action on existing World Action Models without retraining them. The approach combines a layerwise schedule that overlaps observation processing and prediction with selective reuse of observation-token FFN outputs. The reported results on FastWAM and OpenWAM suggest that the integrated runtime reduces latency while largely preserving task success rates in simulation and improving performance on dynamic real-world robot tasks.

Critical strengths
  1. Well-supported efficiency gains: The reported acceleration is substantial across both WAM backbones, including in comparisons with compiled baselines. The fixed-budget ablations (Tables 2 and 7) distinguish the contribution of the execution design from gains due to fewer Transformer evaluations.
  2. Clever design: Using the model’s coarse action to select the refinement budget avoids training a separate selector and ties computation to the current control request. The RoboTwin comparison against random allocation is a particularly useful control: matching budget usage tests whether motion-guided decisions improve task completion. This supports the allocation rule itself, beyond establishing that the model can tolerate fewer evaluations.
Critical weaknesses

Missing native caching baseline: OpenWAM already provides a similarity-based cache that skips model evaluations. The main comparison uses dense Native, and the compilation control does not evaluate this built-in adaptive option. Its omission leaves the proposed refinement policy without a direct comparison against the backbone's existing mechanism for deciding when to skip computation.

Questions for authors

Other than the mentioned weakness, I have these additional questions:

  1. About the real-world protocol: How many actions from each 32-action prediction are executed before replanning, when is the next observation captured, and what happens during inference? What measurements establish the approximately 90 ms conveyor threshold?
  2. How were the caching thresholds and computation budgets selected for RealtimeWAM and the competing methods? Were these settings calibrated on separate validation data and fixed before evaluation?
1 comment

We thank the reviewer for recognizing the execution design and the matched-budget comparison, and for suggesting the native caching baseline. We report this comparison below and clarify parameter selection and the real-world protocol. Below, F denotes the number of full Transformer evaluations within the ten denoising steps.

W1. Comparison with OpenWAM's native similarity-based cache.

We include OpenWAM's official native cache with its compiled execution path, a cosine-similarity threshold of 0.99, and at most three consecutive skipped evaluations, following the same evaluation protocol as the main experiments. All methods perform ten Euler updates.

MethodRoboTwin success rateMean request latency
OpenWAM native cache93.22%194.20 ms
RealtimeWAM, fixed 4F93.68%95.26 ms
RealtimeWAM, fixed 2F89.62%54.70 ms
RealtimeWAM, adaptive 2F/4F93.40%63.76 ms

The native cache averages 4.00 evaluations. At the same count, fixed 4F is 2.04× faster; the adaptive configuration is 3.05× faster with 0.18 points higher success.

Q1. Real-world protocol and the ~90 ms reference.

All methods follow the same synchronous protocol: each prediction contains 32 actions, all 32 actions are executed at 20 Hz, and the next inference cycle follows. A new observation is captured after the action chunk finishes, and the controller holds the final target pose during inference. Each task, backbone, and inference configuration is evaluated for 50 trials.

In the three-shuttlecock stacking task, OpenWAM's inference latency decreases from 640.28 ms to 68.61 ms and success increases from 4% to 68%; at the 1 m/min belt speed, the object moves about 10.7 mm and 1.1 mm during these two inference times. The ~90 ms value comes from our earlier conveyor runs, where success visibly dropped once latency exceeded roughly this level; these runs were not a systematic measurement. To measure it directly, we add a latency sweep on the same task: we keep OpenWAM with RealtimeWAM and the control protocol unchanged, inject additional delay before each action chunk is executed, and evaluate 50 trials per setting:

Total inference latency69 ms (no injection)90 ms120 ms160 ms210 ms
Success rate68%62%34%22%14%

Q2. How were thresholds and computation budgets selected?

We adapt each baseline from its published caching mechanism and reference implementation, and keep the following settings fixed during evaluation:

MethodFixed setting
VLA-CacheVisual similarity threshold 0.996; layerwise reuse growth 0.55
TeaCacheAccumulated relative-change threshold 0.2
ToCaToken refresh ratio 0.3; full refresh every 3 steps; cache-age weight 0.25
BACACS refresh budget 5; BU selection budget 3 sublayers
OpenWAM native cacheCosine-similarity threshold 0.99; at most 3 consecutive skips

Each method keeps its own refresh criterion within the shared ten-step denoising schedule.

RealtimeWAM sets the action-magnitude threshold per benchmark: 8 cm on RoboTwin and 33 cm on LIBERO, held fixed across all tasks within each benchmark, with 2F/4F budgets on both. The thresholds differ because the two simulation environments differ in spatial scale, robot motion range, and action representation. Observation-token FFN refresh capacity is determined by offline latency profiling on the target GPU.

Reviewer_7CHxOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RealtimeWAM, a training-free framework for accelerating world action model inference. The method combines parallel execution, selective reuse of observation-token features, and motion-adaptive reuse of Transformer residuals. The authors evaluate it on FastWAM and OpenWAM across simulation benchmarks and five real-world manipulation tasks, reporting substantial latency reductions while preserving simulated task success and improving real-world success under their evaluation protocol.

Critical strengths

Substantial inference speedup. The proposed method achieves a substantial reduction in WAM inference latency while preserving the model's performance.

Thorough experimental evaluation. The authors evaluate the method across multiple settings and provide detailed ablations of its main components. These experiments help assess both the overall effectiveness of the approach and the contribution of its individual design choices.

Critical weaknesses

Limited evaluation across WAM paradigms. The experiments cover FastWAM and OpenWAM, but do not evaluate a video-then-action WAM based on an inverse dynamics model (IDM). As shown in Table 2, the parallel execution stage is a major contributor to the reported speedup. In an IDM-based model, action generation depends on the generated future video, which may limit opportunities for parallel execution. It is therefore unclear how much of the reported speedup would carry over to this paradigm.

Unclear generality of the motion-adaptive criterion. The method uses predicted action magnitude to decide how many full Transformer evaluations to perform. It is not yet clear whether action magnitude alone is a reliable indicator of the computation needed across different tasks. And, for an IDM-based model, it is unclear whether applying this criterion would cause a substantial degradation in task performance.

Questions for authors
  1. How would the proposed parallel execution strategy apply to an IDM-based WAM, where the model generates future video before predicting action? Would the sequential dependency limit the achievable speedup?

  2. How sensitive is motion-adaptive computation to the choice of action-magnitude threshold across tasks? When applied to an IDM-based model, could this criterion substantially degrade task performance?

3 comments

We thank the reviewer for recognizing the speedup and the experimental evaluation, and for the suggestions on IDM-based WAMs and the motion-adaptive criterion. Below, F denotes the number of full Transformer evaluations within the ten denoising steps.

Q1. How does parallel execution apply to an IDM-based WAM, and does the sequential dependency limit the speedup?

IDM preserves the sequential dependency from video generation to action prediction, while computation within each stage can still overlap. In OpenWAM IDM, we overlap observation-frame and future-frame computation within each video layer, and overlap video-condition prefill with the first action forward pass in the action stage. With Video fixed at 10F:

IDM configurationAction budgetMean request latency (RTX 4090)RoboTwin success rate
Native execution10F511.14 ms87.62%
Parallel execution and associated kernels10F413.87 ms87.62%
+ computation reuse2F375.74 ms84.06%
+ adaptive computation2F/4F379 ms87.24%

At 10F/10F, execution optimizations give a 1.24× speedup without changing success. Reducing Action to 2F gives 1.36× at a 3.56-point drop, and adaptive 2F/4F recovers success to 87.24%. All configurations share the same tasks, scenes, and seeds. Because the Video stage must finish first, the IDM speedup comes mainly from overlap within each stage.

Q2. How sensitive is motion-adaptive computation to the threshold across tasks, and could it substantially degrade an IDM-based model?

Small motions mostly occur in contact phases such as grasping, placing, and alignment, which demand higher position accuracy. We therefore assign 4F to requests whose coarse predicted magnitude is at most 8 cm and 2F to the rest. We first examine how approximation error depends on action magnitude in a paired study, and then evaluate the allocation in closed loop.

The paired study covers all 50 RoboTwin tasks: Clean and Randomized settings with 2 seeds each, giving 200 reference episodes, from which observations are sampled uniformly in time to obtain 1,430 fixed observations. For each observation, we run 2F, 4F, and 10F with the same images, state, instruction, and initial noise, measure the bimanual end-effector position deviation of 2F and 4F from 10F, and group observations by the coarse magnitude available at the routing decision. Joint denotes the original OpenWAM and IDM the variant in Q1.

ModelAction magnitudeObservationsAbsolute deviation, 2F / 4FRelative deviation, 2F / 4F
Joint≤ 8 cm5264.41 / 1.11 mm16.63% / 3.99%
Joint> 8 cm9044.69 / 1.38 mm3.79% / 1.03%
IDM≤ 8 cm6684.12 / 0.96 mm15.59% / 3.66%
IDM> 8 cm7624.43 / 0.96 mm4.05% / 0.89%

Deviation covers the first 24 executed actions; relative deviation divides it by the 10F displacement scale, with a 2 cm floor. 4F reduces the deviation on all 1,430 observations. The absolute 2F deviation stays at about 4–5 mm across magnitudes, so small motions have about four times the relative deviation of large ones, and 4F brings it to the level of 2F on large motions.

Small-motion requests also concentrate in contact phases. We count whether the first 24 executed actions of each observation include a gripper open or close event:

Coarse action magnitudeObservationsChunks with a gripper event
0–4 cm29275.3%
4–8 cm23469.2%
8–12 cm24234.7%
12–20 cm2673.0%
> 20 cm3951.8%

The shares are 72.6% at or below 8 cm and 11.0% above, and the trend holds in 48 of the 50 tasks.

To evaluate the allocation, we compare motion-magnitude gating at different thresholds with random allocation that uses the same 2F/4F proportions as the 8 cm gate:

ConfigurationRoboTwin success rateMean request latency
Motion-magnitude gate, 4 cm91.62%60.11 ms
Motion-magnitude gate, 8 cm (default)93.40%63.76 ms
Motion-magnitude gate, 12 cm93.48%67.09 ms
Random allocation91.08%64.77 ms

At the same budget, the 8 cm gate exceeds random allocation by 2.32 points; 8 cm and 12 cm are nearly equal, and 4 cm is lower. The threshold comparison uses the main-experiment protocol. On IDM, the gate reaches 87.24%, 0.38 points below native execution and 3.18 points above fixed 2F (Q1).

Thank you for addressing my questions and providing additional results. The new results further support the effectiveness of the proposed method. Overall, I consider this a strong paper. After considering your response, I will keep my rating.

Dear Reviewer,

Thank you very much for your positive feedback and for confirming that our responses have addressed your questions. We sincerely appreciate your constructive comments and the time you spent on the review and discussion.

Thank you again, and we wish you a pleasant day!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.