acceptodds
Under review as a conference paper at ICLR 2027

STAIRCASE POLICY: Streaming Inference for World-Action Models with Large Action Chunks

Abstract

World-Action Models (WAMs) improve robotic manipulation by conditioning action generation on predicted future observations, but future prediction adds further inference overhead to already expensive iterative action generation. Action chunking can amortize this cost over multiple actions, yet performance degrades over long execution horizons because later actions remain conditioned on stale observations. We introduce STAIRCASE POLICY, a streaming inference and training framework that turns a flow-matching VLA into a JEPA-style WAM and partitions a large action chunk into sub-chunks at staggered denoising stages. Near-term actions are executed as soon as they become available, while later actions continue to be refined. At each sub-chunk boundary, the future latent is re-predicted from the latest observation and used to update all unexecuted actions, enabling long-horizon execution without repeated full policy inference. The resulting future-prediction error can further serve as a signal for adaptive chunking. S-WAM achieves 97.7% on LIBERO and 87.9% on LIBERO-Plus, and improves performance across multiple policy backbones and real-robot tasks. It reaches 292.7 executed actions per second, 3.62× the throughput of conventional execution at comparable accuracy, while reducing time-to-first-action from 123.6 to 73.3 ms. With additional inference optimizations, throughput further increases to 642.9 actions per second.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Models, Model Efficiency, Action Chunk

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CufpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes STAIRCASE POLICY, a streaming inference framework for World-Action Models that aims to improve the efficiency of long action-chunk execution. It divides each chunk into sub-chunks that are progressively denoised and executed. During execution, new observations are used to refresh future latent predictions and refine the remaining actions, without repeatedly invoking the expensive VLA backbone. The authors also introduce a training procedure aligned with this inference schedule and explore prediction error for adaptive replanning. Experiments on LIBERO, LIBERO-Plus, DOMINO, and nine real-robot tasks show improved success rates and inference efficiency. The method is also evaluated across three VLA backbones.

Critical strengths
  1. The paper addresses a practical issue in VLA deployment. Longer action chunks reduce inference overhead but suffer from stale observations. The proposed streaming schedule provides a reasonable way to incorporate new observations without repeatedly running the full backbone, offering a useful trade-off between efficiency and closed-loop responsiveness.

  2. The experiments are fairly extensive, covering multiple benchmarks, three backbones, and real-robot evaluations. The improvements on dynamic real-robot tasks are particularly encouraging, with success increasing from 25.8% to 44.2% over the long-chunk baseline. The reported latency and training-cost analyses are also useful.

Critical weaknesses
  1. The ablations do not sufficiently separate the contributions of staircase denoising, observation refresh, and future latent prediction. Since the action expert already receives updated visual features, it is unclear how much additional benefit comes from predicting future latents. The streaming-aligned training procedure also introduces several changes simultaneously. Without isolating these components, the necessity of the proposed world-model mechanism remains unclear.

  2. The novelty over existing streaming and reactive VLA methods is not fully established. Methods such as TIDAL, πR2, and prior action-chunk correction approaches already explore updating actions with fresh observations while avoiding repeated expensive backbone inference. Although the proposed denoising schedule differs, its benefits over these alternatives are not directly demonstrated.

  3. The efficiency claims need a more careful interpretation. Table 2 shows that S-WAM has lower throughput than vanilla execution with the same 50-action horizon (292.7 vs. 402.5 actions/s). The reported 3.62× speedup is relative to the short-horizon baseline, rather than a reduction in inference cost at the same execution horizon. End-to-end control latency and actual responsiveness on the robot are also not sufficiently evaluated.

Questions for authors
  1. Can the authors provide ablations separating staircase denoising, observation refresh, and future latent prediction? In particular, how does the method perform when only the current visual features are refreshed, without future prediction?

  2. How much of the improvement comes from streaming-aligned training rather than the inference schedule itself?

  3. Can the authors provide a more direct comparison with TIDAL, πR2, or lightweight online action correction methods under matched backbone and inference settings?

  4. Could the authors clarify the throughput comparison and report end-to-end control latency or reaction time on the real robot?

  5. Why is the improvement on DOMINO relatively modest (19.3% vs. 16.2%), with some individual tasks showing performance degradation?

6 comments
AuthorsOpenReview ↗

Response - 1

W1 & Q1: Contributions of staircase denoising, observation refresh, and future-latent prediction

We thank the reviewer for the suggestion. S-WAM integrates staircase denoising, observation refresh, and streaming-aligned training: staircase denoising enables progressive sub-chunk execution, observation refresh updates unexecuted actions using new observations, and streaming-aligned training supervises intermediate readouts. We ablate future-latent prediction, observation refresh, and the training procedure while retaining staircase inference. Each is important for S-WAM's performance; removing future-latent prediction alone reduces success from 97.7 to 89.2.

1. Component ablations

Each variant modifies only the specified setting, with all other settings unchanged.

Modification (LaWAM)SpatialObjectGoalLongAvg
Full S-WAM98.6100.097.095.097.7
Remove future-latent prediction and its loss (current visual features still refreshed every 10 actions)89.497.291.678.489.2
Remove observation refresh (current and future features fixed at chunk start)77.683.094.072.681.8
Standard training (retain staircase inference)40.812.450.225.832.3

Future-latent prediction and its training provide an additional 8.5-point gain beyond refreshing current visual features alone, with the largest improvement on LIBERO-Long (95.0 vs. 78.4). Without observation refresh, success drops to 81.8, comparable to full-chunk open-loop execution (82.4). Using standard training with staircase inference further reduces success to 32.3, highlighting the need for aligned training.

Replacing only the predicted future latent at inference with one from another episode reduces success from 97.7 to 4.9, further indicating that the trained policy relies on future-latent conditioning.

2. Ablations of streaming-aligned training objectives

We further ablate individual training designs. The table reports success-rate changes (percentage points) relative to full S-WAM; training–inference alignment is examined in Q2.

Removed or modified componentSpatialObjectGoalLongAvg
Loss on non-executed positions (weight 0.25 → 0)0.00.0−1.0−1.6−0.7
Future-prediction horizon (next sub-chunk → full-chunk end)+0.4−3.0−6.2−9.4−4.6
Block action-loss gradients through the future latent−11.4−8.0−7.6−18.8−11.5

Blocking action-loss gradients to the future predictor causes the largest degradation (−11.5 points), showing the importance of learning future representations useful for action generation, beyond merely predicting future features. Changing the prediction horizon also hurts performance (−4.6), while the auxiliary loss on non-executed positions has a smaller effect (−0.7).


Q2: How much improvement comes from streaming-aligned training versus staircase inference?

We thank the reviewer for the question. We conduct a training–inference cross-comparison. Neither change alone achieves the full performance: streaming-aligned training with standard inference reaches 25.6, standard training with staircase inference reaches 32.3, and combining both reaches 97.7.

The first four rows use the same S-WAM architecture and future predictor, varying only training and inference. The last two use the vanilla model as references. Standard inference replans every 10 actions (Hexec=10H_{\mathrm{exec}}=10); staircase inference invokes the VLM once per 50 actions and refreshes conditioning and emits a sub-chunk every 10 actions.

TrainingInferenceSpatialObjectGoalLongAvg
Standard (S-WAM architecture)Standard87.497.890.081.889.3
Standard (S-WAM architecture)Staircase40.812.450.225.832.3
Streaming-aligned (S-WAM)Standard45.42.248.85.825.6
Streaming-aligned (S-WAM)Staircase98.6100.097.095.097.7
Reference: VanillaStandard96.298.495.690.695.2
Reference: VanillaStaircase71.251.077.258.464.5

Under staircase inference, streaming-aligned training improves success from 32.3 to 97.7. Conversely, switching the streaming-trained model to standard inference reduces success to 25.6 (or 7.1 with Hexec=50H_{\mathrm{exec}}=50). Applying staircase inference directly to vanilla also reduces success from 95.2 to 64.5.

These results demonstrate that streaming-aligned training and staircase inference are strongly interdependent, rather than providing independently additive gains. Streaming-aligned training enables intermediate action readouts and refreshed conditioning during staircase inference, rather than independently improving standard inference. We will include these ablations in the revision.

AuthorsOpenReview ↗

Response - 2

W2 & Q3: Comparison with TIDAL, πR², and Existing Action Chunk Correction Methods

Thank you for the suggestion. We adapted the core mechanisms of TIDAL, πR², and RNR-DP to π0.5 and evaluated them under the same single-GPU synchronous setting. S-WAM achieves the highest success rate (96.5%) and 346.8 actions/s, comparable to TIDAL (335.3 actions/s) and 2.16× and 2.29× faster than RNR-DP and πR², respectively. The three baselines process 24× more training samples than S-WAM.

1. Comparison with TIDAL, πR², and RNR-DP

All methods use the same π0.5 backbone, training data, and initialization, with an action chunk length of 50. We adapted πR² and RNR-DP from their official code and implemented TIDAL based on its paper:

  • πR²: Performs one denoising step and executes 10 actions per call, requiring 5 VLM calls per 50 actions. The original method uses asynchronous VLM inference on a separate GPU; we use synchronous execution on a single GPU for a consistent comparison.
  • RNR-DP: Preserves its noise relay mechanism but changes the relay interval from one action to 10 actions, with 5 noise levels and 5 VLM calls per 50 actions.
  • TIDAL: Calls the VLM once per 50 actions and uses a CNN to extract motion features from pixel differences every 10 actions, followed by one-step action generation. We adapt its original setting from (H, N, K) = (16, 4, 4) to (50, 10, 5).

S-WAM uses batch 8 × 10K training steps, while vanilla uses batch 8 × 30K steps. All three adapted baselines use batch 64 × 30K steps. We measure all methods on the same L40S GPU (batch 1, bf16, eager), including remeasuring vanilla and S-WAM. Thus, the absolute speeds differ from the H100 results in the paper.

Method (π0.5)SpatialObjectGoalLongAvgActions/sTTFA (ms)
Vanilla, H_exec = 1094.699.091.687.493.2106.793.7
Vanilla, H_exec = 5066.467.672.858.666.4533.793.7
πR²93.291.889.486.090.1151.462.6
RNR-DP93.487.688.073.285.6160.662.3
TIDAL95.093.686.490.291.3335.359.2
S-WAM98.098.695.294.296.5346.861.7

Compared with vanilla H_exec = 10, S-WAM achieves 3.25× higher throughput and 34% lower TTFA. Compared with the faster H_exec = 50 baseline, it improves success by approximately 30 percentage points.

πR² and RNR-DP use position-dependent noise schedules, which differ from the shared flow time used during π0.5 pretraining and may require more training to adapt. S-WAM keeps a shared flow time and reads out actions at different denoising stages. TIDAL also caches the VLM context, but refreshes motion features and regenerates actions in one step. In contrast, S-WAM continuously refines unexecuted actions using updated visual and predicted future features.

S-WAM and TIDAL have similar throughput because both use one VLM call per 50 actions and lightweight updates every 10 actions. πR² and RNR-DP require 5 VLM calls under our single-GPU setting, resulting in lower throughput. These results compare the adapted mechanisms under a unified π0.5 setting, rather than the full performance of the original systems.

2. Relation to Action Chunk Correction Methods

Existing methods either terminate an action chunk early and replan based on uncertainty or prediction errors, or correct generated actions using new observations. S-WAM instead refreshes the predicted future representation during generation to refine unexecuted actions. These approaches are complementary: prediction errors can trigger early replanning (Sec. 5.4), while lightweight action correction can also be applied to S-WAM outputs.

AuthorsOpenReview ↗

Response - 3

W3 & Q4: Throughput Comparison and Real-Robot End-to-End Latency

Thank you for the question. The reported 3.62× speedup is relative to H_exec = 10, which achieves comparable success. Compared with H_exec = 50, S-WAM requires more computation but achieves substantially higher success. We will clarify this trade-off and report additional real-robot latency measurements.

1. Throughput Comparison

MethodLIBEROLIBERO-PlusActions/sSpeedupTTFA (ms)
Vanilla, H_exec = 1095.287.280.91.00×123.6
Vanilla, H_exec = 5082.470.6402.54.98×124.2
S-WAM97.787.9292.73.62×73.3

Compared with H_exec = 50, S-WAM uses more computation (1550 vs. 1010 GFLOPs per 50 actions) but improves success by 15.3 points on LIBERO and 17.3 points on LIBERO-Plus.

2. Real-Robot Deployment (xArm 850 + Jetson Thor)

We measure real-robot performance using an Intel RealSense D435 camera and onboard inference on Jetson Thor, with the same CUDA Graph and prompt-cache optimizations. We set the target command rate to 200 Hz and measure the achieved rate for each method; while the next sub-chunk is being computed, the robot does not execute further actions. S-WAM achieves 140.5 Hz, 1.85× that of H_exec = 10 (76.1 Hz). H_exec = 50 reaches a higher rate (157.4 Hz) because it updates only once per 50 actions, but its success rate is much lower (82.4 vs. 97.7 on LIBERO).

MethodCommand Rate (Hz) ↑SpeedupTTFA (ms) ↓
Vanilla, H_exec = 1076.11.00×89.7
Vanilla, H_exec = 20110.31.45×93.1
Vanilla, H_exec = 50157.42.07×92.7
S-WAM140.51.85×75.8

S-WAM requires only 17.4 ms for a sub-chunk update, compared with about 90 ms for a full vanilla policy inference. Its first sub-chunk requires only one denoising step, reducing TTFA to 75.8 ms.

The observation-to-command latency is 104.3 ms at chunk start and 46.1 ms at sub-chunk boundaries for S-WAM, compared with 117.5–120.8 ms for vanilla. This measures the time from image capture to command dispatch, including image acquisition, preprocessing, and inference.

The onboard inference breakdown is shown below (separate profiling with synchronization after each module, so the totals are higher than in the end-to-end runs; ms per 50 actions):

ComponentS-WAMH_exec = 10H_exec = 20H_exec = 50
VLM49.6262.6130.951.7
Visual encoding + future prediction56.056.227.510.8
DiT denoising24.9124.760.624.2
CPU / Others32.0102.247.918.4
Total162.5545.6266.9105.1

S-WAM requires 1 VLM call and 5 DiT forward passes per 50 actions, compared with 5 VLM calls and 25 DiT passes for H_exec = 10. Their visual encoding and future prediction costs are nearly identical (56.0 vs. 56.2 ms). Including these additional costs, S-WAM requires only 0.30× the total inference time of H_exec = 10 (162.5 vs. 545.6 ms).

Overall, S-WAM reduces TTFA through staircase readout and avoids repeated full VLM inference through lightweight sub-chunk updates, achieving a higher measured command rate than H_exec = 10 while refreshing visual feedback just as often.


Q5: Limited Improvements and Task-Level Drops on DOMINO

Thank you for the question. DOMINO is challenging, with low overall success rates. We train only on clean demonstrations and jointly train all tasks, which may contribute to performance variation across tasks. S-WAM improves some tasks but performs worse on others, such as move-can.

Unlike static settings, execution throughput is particularly important in dynamic scenes. In our experiments, frequent full replanning performs poorly: vanilla H_exec = 10 achieves only 2.2%, compared with 16.2% for full-chunk execution (H_exec = 75). S-WAM further achieves 19.3%, combining the efficiency of long action chunks with observation-based refinement of unexecuted actions. This may help maintain action quality, consistent with our findings on static tasks.

Reviewer_NwHVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A new method, called STAIRCASE POLICY, is presented in this paper. The main focus is to improve action throughput when generating large action chunks and executing in a long-horizon setting. The core methodology is to convert a flow-matching VLA into a latent World-Action Model by adding a lightweight future-latent prediction. Large action chunks are divided into sub-chunks. Then, Near-term actions will be executed before the full chunk is generated. When new observations arrive, the proposed method refreshes the predicted future latent and refines the unexecuted actions without repeatedly invoking the full policy.

Extensive experimental results, including benchmarking and real-robot settings, were presented to demonstrate the effectiveness of the proposed method.

Critical strengths
  • The problem and research gap are well defined. Generating long action chunks with large VLM/WAM policies can improve inference costs, but it is challenging to execute them in long-horizon settings.

  • The staircase idea, ie progressively refining the unexecuted chunks, is a simple yet sensible approach.

  • The experimental results are strong, outperformed the baselines. There are also results on using S-WAM with different backbones to show the generality.

Critical weaknesses
  • It is not clear how much the individual proposed components are contributing to the performance gain. For example, how much has the future-latent prediction improved the results? More detailed ablation should have been presented.

  • In the main results table (Table 1), S-WAN achieved Avg. 87.9% and Vanilla, H_exec=10 achieved Avg. 87.2%. The gap is not large and is it statistically significant?

  • More importantly, the high-level concept of generating a large number of action chunks and progressively refining the unexecuted actions is very similar to MGP-Long [Zhuang et al. ICLR 2026], which predicts all action tokens at once and progressively refine the low-confidence unexecuted action tokens. While the underlying technical directions are different, the high-level ideas are very similar. More discussion is needed to clarify the novelty in this work.

Reference Zhuang et al., "Masked Generative Policy for Robotic Control", ICLR 2026

Questions for authors

Please address the points I raised in the Critical Weaknesses section above.

5 comments
AuthorsOpenReview ↗

Response - 1

W1: How much does each component contribute, particularly future-latent prediction?

We thank the reviewer for the suggestion. We conducted a complete component ablation study. All variants use the same training configuration as S-WAM, with only one component removed at a time.

Variant (LaWAM)ModificationSpatialObjectGoalLongAvg
S-WAM—98.6100.097.095.097.7
w/o future predictionRefresh current visual features at each boundary, but remove future prediction and its corresponding world-model loss89.497.291.678.489.2
w/o observation refreshKeep both current and future features fixed at the beginning of the chunk77.683.094.072.681.8
Standard trainingUse standard training with all other settings unchanged; retain staircase readout at inference40.812.450.225.832.3
Reference: Vanilla, H_exec = 10—96.298.495.690.695.2
Reference: Vanilla, H_exec = 50—87.280.688.073.882.4

Removing any component substantially reduces success rates:

  • Without future prediction, LIBERO success drops from 97.7 to 89.2, with the largest degradation on LIBERO-Long (95.0 → 78.4).
  • Without observation refresh, performance drops to a level comparable to full-chunk open-loop execution (81.8 vs. 82.4).
  • Replacing streaming-aligned training with standard training, while keeping all other settings unchanged, reduces the success rate under staircase readout to 32.3.

Furthermore, replacing only the predicted future latent at inference time with one from another episode reduces success from 97.7 to 4.9, confirming that the policy uses the predicted future when generating actions. We will include these ablations in the revised manuscript.


W2: Is the 87.9 vs. 87.2 difference in Table 1 statistically significant?

We thank the reviewer for the question. This difference is not statistically significant, and we do not claim that S-WAM achieves a higher success rate than H_exec = 10.

S-WAM aims to balance success rate and speed: maintaining success rates comparable to short execution horizons (H_exec = 10) while approaching the throughput of long execution horizons (H_exec = 50). Because refreshing at each sub-chunk boundary introduces additional overhead, we do not claim to outperform H_exec = 50 in throughput. Instead, our goal is to accelerate execution while maintaining or slightly improving success rates.

MethodLIBERO-PlusLIBEROActions/sTime-to-first-action (ms)
Vanilla, H_exec = 1087.295.280.9123.6
Vanilla, H_exec = 5070.682.4402.5124.2
S-WAM87.997.7292.773.3

To characterize evaluation variability, we additionally trained and evaluated all three settings on LIBERO with two additional random seeds. Each run follows the full evaluation procedure described in the paper (four suites, 50 trials per task). Run1 corresponds to the results reported in the paper.

Method (LIBERO)Run1Run2Run3Mean ± Std
Vanilla, H_exec = 1095.295.096.395.5 ± 0.7
Vanilla, H_exec = 5082.480.183.782.1 ± 1.8
S-WAM97.796.897.997.5 ± 0.6

S-WAM achieves a mean success rate of 97.5% on LIBERO, 2.0 percentage points higher than H_exec = 10, while delivering 3.62× higher throughput. Compared with H_exec = 50, which uses the same number of model calls, S-WAM achieves a 15.4-point higher success rate at slightly lower throughput. This advantage also holds in more challenging settings: S-WAM improves success by 17.3 points on LIBERO-Plus and achieves 44.2% vs. 25.8% on dynamic real-robot tasks.

AuthorsOpenReview ↗

Response - 2

W3: Relationship to MGP-Long

We thank the reviewer for pointing out this relevant work. We agree that both methods share the same high-level idea: using new observations during execution to refine unexecuted actions. We do not claim to be the first to introduce this idea. We will cite and discuss MGP in the related work. However, the two methods differ substantially in their settings, making a direct comparison inappropriate:

S-WAMMGP-Long
Action representationContinuous actions (flow matching)Discrete VQ tokens
Model2.3B VLA (also evaluated on π0.5 and FLOWER)~7M-parameter masked generative transformer, excluding the VLM
GenerationDivide an action chunk into 5 sub-chunks, sharing a single flow time with readouts at different denoising depthsPredict the entire episode's token sequence at once
Updating unexecuted actions with new observationsRe-predict the future latent to recondition all unexecuted actionsRe-estimate token confidence, re-mask approximately 70% of the lowest-confidence tokens, and regenerate them

MGP's core operation is confidence-based re-masking of discrete tokens, which has no direct counterpart in a continuous flow-matching action head. A comparison would require replacing the action head and tokenizer, making it no longer a comparison under the same backbone.

Within this shared high-level idea, our contribution is a concrete implementation for large flow-matching VLAs, consisting of three components: (1) shared-flow-time staircase readout, which progressively releases continuous action sub-chunks while later sub-chunks continue denoising during execution; (2) future-latent refresh, which re-predicts the future latent from new observations at each sub-chunk boundary, using the cached VLM context to recondition all unexecuted actions; and (3) streaming-aligned training that explicitly supervises these intermediate readouts.

Our ablations show that observation-driven future refresh is critical for maintaining execution quality with long action chunks: retaining staircase denoising without observation refresh achieves 81.8, comparable to open-loop execution (82.4), whereas the full S-WAM with future refresh achieves 97.7. Additionally, staircase readout allows the earliest actions to be executed after only one denoising step, reducing time-to-first-action from 123.6 ms to 73.3 ms.

Dear Reviewer NwHV,

Thank you again for your thoughtful feedback and constructive suggestions.

As the discussion period approaches its conclusion, we wanted to kindly follow up and ask whether our additional experiments and clarifications have helped address your main concerns. We would greatly appreciate any further feedback you may have. If there are any remaining questions or points that would benefit from additional clarification, we would be more than happy to discuss them before the discussion closes.

Thank you very much for your time and consideration.

Sincerely,

The Authors of Submission 22489

Reviewer_GxUsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces STAIRCASE POLICY, a streaming inference framework for world-action models that aims to combine the computational efficiency of long action chunks with feedback during execution. The proposed S-WAM caches the expensive VLM context for an entire chunk, while refreshing current visual features and predicted future features between action subchunks. A shared flow-time schedule enables successive subchunks to be emitted through endpoint predictions at different denoising stages. The training procedure follows this staged schedule and directly supervises the action predictions used for execution. Evaluations on LIBERO, LIBERO-Plus, and DOMINO, across three policy backbones, demonstrate favorable success–inference-efficiency trade-offs against conventional chunk execution. Experiments on an xArm 850, including dynamic turntable tasks, provide additional evidence of the approach’s practical value.

Critical strengths

The paper addresses a practically important tension between the efficiency of long action chunks and the need for timely feedback in robot control. I find the proposed combination of staged action generation and lightweight visual refresh interesting, and the dynamic real-world evaluation provides meaningful support for its practical relevance. The following strengths are particularly consequential to my assessment:

  1. A concrete mechanism for incorporating feedback within long action chunks. S-WAM separates expensive semantic processing from more frequent updates of current and predicted future visual features. This allows new observations to influence actions that have not yet been executed while retaining one VLM evaluation per chunk. The design directly addresses the stale-conditioning problem associated with long action chunks and offers a useful efficiency–responsiveness trade-off.
  2. A coherent combination of staged action readout and corresponding training. The method combines a shared flow-time schedule with endpoint predictions that release different subchunks at successive denoising stages. Its training procedure explicitly supervises these stage-specific predictions under refreshed conditions. This is a thoughtful aspect of the design, because the training objective accounts for how intermediate predictions are actually used during deployment.
  3. Evaluation across multiple backbones, with relevant dynamic real-world validation. Results across LaWAM, π0.5, and FLOWER support the applicability of the overall approach beyond a single host model. The real-world evaluation is particularly valuable: across three dynamic turntable settings, S-WAM achieves an average success rate of 44.2%, compared with 25.8% for the full-length chunk baseline and 10.8% for the shorter-execution baseline. These tasks directly test the paper’s motivation that feedback during chunk execution matters when the environment changes.
Critical weaknesses

While the proposed framework is interesting and the dynamic real-world results are encouraging, several important gaps remain in the comparative evaluation, the attribution of gains to individual components, and the characterization of actual deployment efficiency. These limitations make it difficult to fully assess the method’s distinct contribution and the strength of its efficiency–responsiveness claims.

  1. Insufficient comparison with the most relevant streaming and reactive policies. The experiments primarily compare S-WAM with conventional short- and full-chunk execution, leaving its advantage over closely related streaming or reactive methods insufficiently established. Moreover, the related-work statement that these methods condition an entire chunk on a single observation is too broad: RNR-DP [1] updates pending actions using new observations, while πR² [2] combines cached semantic features with fresh proprioceptive feedback. The paper should more precisely distinguish its shared flow-time readout and future-feature refresh from these existing mechanisms. A comparison with at least one closely related method under comparable backbone, training, and hardware settings would materially strengthen the contribution.
  2. The contribution of the core components, particularly future prediction, is not sufficiently isolated. S-WAM jointly introduces refreshed visual observations, predicted future features, staged action readout, and a corresponding training objective. The current ablations mainly examine scheduling and configuration choices, rather than separating these contributions. In particular, it remains unclear whether future prediction provides benefits beyond simply conditioning on fresh visual features. Comparisons between current-feature-only and current-plus-future-feature conditioning, together with controlled evaluations of staged readout and its training objective, are needed to establish which mechanisms drive the gains. This is particularly consequential to the paper’s emphasis on world-action modeling.
  3. The timing evaluation does not fully characterize end-to-end closed-loop execution. The reported actions-per-second metric measures amortized inference throughput and excludes physical execution or simulator stepping; it therefore does not directly establish the robot’s actual control frequency or observation-to-response latency. Since each refresh depends on observations obtained after executing a subchunk, the handling of computation and possible waiting at subchunk boundaries requires clarification. Furthermore, the “compute-matched” setting matches backbone and denoiser call counts rather than total computation, because S-WAM additionally refreshes visual encodings and future predictions. Onboard timing traces covering sensor acquisition, inference, command delivery, and boundary waiting would provide a more complete assessment of the claimed efficiency–responsiveness trade-off.

[1] Zhuoqun Chen, Xiu Yuan, Tongzhou Mu, and Hao Su. Responsive Noise-Relaying Diffusion Policy: Responsive and Efficient Visuomotor Control. Transactions on Machine Learning Research, 2025.

[2] Sungjae Park and Shubham Tulsiani. πR²: Reactive Real-time Flow Policies. arXiv preprint arXiv:2607.26055, 2026.

Questions for authors
  1. How does S-WAM differ from closely related methods such as RNR-DP [1] and πR² [2] in its use of fresh observations, denoising schedule, and computational cost? Please clarify these distinctions and reconsider the statement that prior methods condition an entire chunk on a single observation. Could you provide a matched comparison with at least one closely related streaming or reactive policy, reporting both task success and inference or response latency?

  2. Does predicted future conditioning provide a measurable benefit beyond refreshing current visual features? Could you compare current-feature-only conditioning with current-plus-future-feature conditioning while keeping the observation schedule and training budget comparable? Additionally, could you separate the benefits of staged action readout from those of its corresponding training objective? These comparisons would help determine whether the improvements depend on future modeling and the proposed generation mechanism, rather than primarily on more frequent visual feedback.

  3. On the xArm 850 and Jetson Thor deployment, what are the actual control frequency and end-to-end observation-to-command latency, including sensor acquisition and transmission? How are control commands handled while a new subchunk is being computed, and are there waiting periods at subchunk boundaries? Could you provide an onboard timing breakdown for S-WAM and the short- and full-chunk baselines, using equivalent optimization settings and including the additional visual encoding and future-prediction costs? Please also specify the hardware used for the main throughput measurements and its relationship to the onboard results.

[1] Zhuoqun Chen, Xiu Yuan, Tongzhou Mu, and Hao Su. Responsive Noise-Relaying Diffusion Policy: Responsive and Efficient Visuomotor Control. Transactions on Machine Learning Research, 2025.

[2] Sungjae Park and Shubham Tulsiani. πR²: Reactive Real-time Flow Policies. arXiv preprint arXiv:2607.26055, 2026.

7 comments
AuthorsOpenReview ↗

Response - 1

W1 & Q1: Differences from RNR-DP, πR², and Other Streaming Methods

Thank you for pointing this out. Our statement that these methods use only one observation for an entire action chunk was too broad. RNR-DP updates unexecuted actions using new observations, while πR² combines cached semantic features with updated proprioceptive feedback. We will revise the related work to clarify their differences in observation updates, denoising schedules, and computational costs. We also compare RNR-DP, πR², and TIDAL under the same backbone, training data, and hardware. S-WAM achieves the highest success rate (96.5%) and throughput comparable to TIDAL (346.8 vs. 335.3 actions/s), while being 2.16× and 2.29× faster than RNR-DP and πR². All four methods have similar TTFA, and the three adapted baselines process 24× as many training samples as S-WAM.

1. Differences in Mechanisms

S-WAMRNR-DPπR²TIDAL
Use of new observationsRefreshes visual features and predicts future latents every 10 actions to condition unexecuted actionsEncodes the latest observation at each control stepUses updated proprioceptive states, with asynchronous visual-language feature updatesRegenerates actions using updated proprioceptive states and motion features every 4 steps
DenoisingShared flow time across the chunk; sub-chunks read out at different stagesPosition-dependent noise levels, with one denoising step per updatePosition-dependent noise levels, with one denoising step per callOne-step generation from fresh noise
Original inference design1 VLM call, 5 lightweight updates, and 5 DiT passes per 50 actions1 network pass per action (no VLM in the original method)1 DiT pass per call; asynchronous VLM on another GPU1 VLM call per 16 steps and 1 action-head pass per 4 steps
Future predictionYesNoNoNo

S-WAM keeps a shared flow time and uses new observations to predict future latents and refine unexecuted actions without rerunning the VLM. The table summarizes the original designs; the experiments below use a common update interval of 10 actions.

2. Comparison Under the Same Backbone, Training Data, and Hardware

We conduct this comparison on π0.5, while W2 and W3 use the main LaWAM backbone, explaining the different S-WAM success rates (96.5% vs. 97.7%). All adapted methods use the same backbone, training data, initialization, and chunk length of 50. We adapt RNR-DP and πR² from their official code and implement TIDAL based on its paper:

  • RNR-DP: We preserve its position-dependent noise relay. Since its original per-action updates would require a VLM call per action on π0.5, we relay every 10 actions with 5 noise levels, requiring 5 VLM calls per 50 actions.
  • πR²: We preserve its position-dependent noise schedule and one-step denoising, executing 10 actions per call with 5 VLM calls per 50 actions. We replace its original asynchronous, two-GPU VLM updates with synchronous single-GPU updates.
  • TIDAL: We cache VLM intent features once per 50 actions and use a CNN to extract pixel-difference motion features every 10 actions for one-step action generation. We adapt (H, N, K) from (16, 4, 4) to (50, 10, 5).

The three adapted baselines use batch 64 × 30K training steps, processing 24× as many samples as S-WAM (batch 8 × 10K steps); vanilla uses batch 8 × 30K steps. We measure all methods on one L40S GPU (batch 1, bf16, eager, synchronous inference), including remeasuring vanilla and S-WAM. Thus, absolute speeds differ from the H100 results in the paper.

Method (π0.5)SpatialObjectGoalLongAvgActions/sTTFA (ms)
Vanilla, H_exec = 1094.699.091.687.493.2106.793.7
Vanilla, H_exec = 5066.467.672.858.666.4533.793.7
πR²93.291.889.486.090.1151.462.6
RNR-DP93.487.688.073.285.6160.662.3
TIDAL95.093.686.490.291.3335.359.2
S-WAM98.098.695.294.296.5346.861.7

Compared with vanilla H_exec = 10, S-WAM achieves 3.25× higher throughput and 34% lower TTFA; compared with H_exec = 50, it improves success by about 30 points. S-WAM and TIDAL have similar throughput because both use one VLM call per 50 actions. RNR-DP and πR² require 5 VLM calls under our single-GPU setting, resulting in lower throughput. Their position-dependent noise schedules also differ from π0.5's pretrained shared flow time, which may make adaptation harder. These results compare the adapted mechanisms under a unified π0.5 setting, not the full performance of the original systems.

AuthorsOpenReview ↗

Response - 2

W2 & Q2: Benefits of Future Prediction and Training–Inference Alignment

Thank you for raising these questions. With the same observation refresh interval and training steps, future prediction and its training terms improve success by 8.5 points (97.7 vs. 89.2), showing that the gain is not simply from more frequent visual feedback. Staircase readout without aligned training achieves only 32.3, while aligned training without staircase readout achieves 25.6. Combining both reaches 97.7.

1. Current Visual Features vs. Current and Predicted Future Features

Both models refresh visual features every 10 actions using the same data, initialization, optimization settings, and training steps. They differ only in whether predicted future features and their training terms are included. We also include a no-refresh variant and vanilla references.

Variant (LaWAM)SpatialObjectGoalLongAvg
S-WAM (current + predicted future)98.6100.097.095.097.7
Current features only (remove future prediction and its training terms; refresh every 10 actions)89.497.291.678.489.2
No observation refresh (fix current and future features at chunk start)77.683.094.072.681.8
Reference: Vanilla, H_exec = 1096.298.495.690.695.2
Reference: Vanilla, H_exec = 5087.280.688.073.882.4

The largest improvement is on LIBERO-Long (95.0 vs. 78.4). Without observation refresh, success drops to 81.8, close to full-chunk open-loop execution (82.4). Replacing the predicted future latent with one from another episode at inference time, while keeping other inputs unchanged, reduces success from 97.7 to 4.9, showing that the policy strongly depends on this input.

2. Separating Staircase Readout from Its Training Objective

We use a 2 × 2 comparison of standard vs. streaming-aligned training and standard vs. staircase inference. The first four variants share the same architecture, future predictor, and conditioning inputs. Standard inference replans every 10 actions; staircase inference calls the VLM once per 50 actions and refreshes conditions and reads out a sub-chunk every 10 actions. The last two rows are vanilla references.

TrainingInferenceSpatialObjectGoalLongAvg
Standard training (S-WAM architecture)Standard inference87.497.890.081.889.3
Standard training (S-WAM architecture)Staircase inference40.812.450.225.832.3
Streaming-aligned training (S-WAM)Standard inference45.42.248.85.825.6
Streaming-aligned training (S-WAM)Staircase inference98.6100.097.095.097.7
Reference: vanillaStandard inference96.298.495.690.695.2
Reference: vanillaStaircase inference71.251.077.258.464.5

The two mismatched settings achieve only 32.3 and 25.6, while the full S-WAM reaches 97.7. Streaming-aligned training supervises intermediate readouts and prepares the model for updated conditions during execution.

We further ablate individual training components (success changes relative to full S-WAM, in percentage points):

Removed or Modified ComponentSpatialObjectGoalLongAvg
Loss on non-executed positions (weight 0.25 → 0)0.00.0−1.0−1.6−0.7
Future-prediction horizon (next sub-chunk → full-chunk end)+0.4−3.0−6.2−9.4−4.6
Block action-loss gradients through the future latent−11.4−8.0−7.6−18.8−11.5

Blocking action-loss gradients to the future predictor causes the largest drop (−11.5 points), showing the importance of optimizing future prediction together with action generation. Changing the prediction horizon reduces success by 4.6 points, while removing the loss on non-executed positions has a smaller effect (−0.7 points).

We will include these ablations in the revised paper.

AuthorsOpenReview ↗

Response - 3

W3 & Q3: End-to-End Latency, Command Rate, Sub-Chunk Updates, and Compute Matching

Thank you for highlighting the need for real-robot measurements. Since inference throughput alone does not fully reflect deployment performance, we measure action command rates, observation-to-command latency, and onboard inference costs on an xArm 850 with Jetson Thor. S-WAM receives new visual feedback every 10 actions, like H_exec = 10, but achieves a 1.85× higher command rate (140.5 vs. 76.1 Hz). Its observation-to-command latency at sub-chunk boundaries is 46.1 ms, less than half of vanilla (117.5–120.8 ms).

1. Action Command Rate and Observation-to-Command Latency

We use an Intel RealSense D435 and the same CUDA Graph and prompt-cache optimizations for all methods. The target command rate is 200 Hz; we report the achieved rates below. The achieved rates include any time the robot spends waiting for inference.

MethodCommand Rate (Hz) ↑SpeedupTTFA (ms) ↓Observation-to-Command Latency (ms) ↓
Vanilla, H_exec = 1076.11.00×89.7117.5
Vanilla, H_exec = 20110.31.45×93.1120.8
Vanilla, H_exec = 50157.42.07×92.7119.0
S-WAM140.51.85×75.8104.3 (chunk start) / 46.1 (sub-chunk boundary)

H_exec = 50 reaches the highest command rate because it updates only once per 50 actions. S-WAM refreshes visual feedback every 10 actions, as H_exec = 10 does, while achieving 1.85× its command rate.

Observation-to-command latency is measured from the camera timestamp to SDK command dispatch. It includes image acquisition and transfer (15.8 ms), waiting for the policy to read the image (~12 ms), preprocessing (2.8–6.3 ms), inference (70.5/14.6 ms for S-WAM at chunk start/sub-chunk boundaries; 84–87 ms for vanilla), and command dispatch (~0.3 ms). The policy and robot client run in the same process. S-WAM also reduces TTFA because its first sub-chunk requires only one denoising step.

2. Robot Control and Waiting at Sub-Chunk Boundaries

During the computation of a new sub-chunk, the robot does not continue executing actions from the previous sub-chunk. It waits for the next sub-chunk before continuing execution. S-WAM takes only 17.4 ms for this update, including preprocessing, visual encoding, future prediction, and one DiT forward pass, compared with about 90 ms for a full vanilla policy inference. This waiting is included in the command rates above.

3. Onboard Inference Breakdown

We profile each component separately under the same optimizations, synchronizing after every module, so the totals are higher than in the end-to-end runs above (ms per 50 actions):

ComponentS-WAMH_exec = 10H_exec = 20H_exec = 50
VLM49.6262.6130.951.7
Visual encoding + future prediction56.056.227.510.8
DiT denoising24.9124.760.624.2
CPU / Others32.0102.247.918.4
Total162.5545.6266.9105.1

Including visual encoding and future prediction, S-WAM takes only 0.30× the inference time of H_exec = 10 (162.5 vs. 545.6 ms). It is slower than H_exec = 50 (105.1 ms), mainly due to additional visual encoding and future prediction (56.0 vs. 10.8 ms). S-WAM requires 1 VLM call and 5 DiT passes per 50 actions, compared with 5 VLM calls and 25 DiT passes for H_exec = 10.

The paper's Table 2 uses a single H100 GPU (batch 1, eager), while onboard measurements use Jetson Thor with CUDA Graph and prompt cache. Absolute speeds are therefore not directly comparable, but continuous end-to-end inference speedups over H_exec = 10 are similar (3.20× on Thor vs. 3.62× on H100). We will clarify the H100 setting in Appendix B.

4. Clarifying "Compute-Matched"

Our use of "compute-matched" referred to matched backbone and denoiser call counts, not total computation. We will use "matched number of model calls" and report inference FLOPs:

Per 50 ActionsS-WAMVanilla, H_exec = 10Vanilla, H_exec = 50
GFLOPs155050601010

Compared with H_exec = 50, S-WAM uses more computation (1550 vs. 1010 GFLOPs) for visual and future-feature updates, but improves success from 82.4 to 97.7. Compared with H_exec = 10, S-WAM uses only 0.31× the FLOPs (1550 vs. 5060 GFLOPs) with comparable success.

We will add the real-robot results, inference breakdown, FLOPs, and measurement details to the revised paper.

Comments on the paper

1 comment

Dear Reviewer,

Thank you again for your thoughtful feedback.

With less than 20 hours remaining in the discussion period, we wanted to gently ask whether our responses and revised manuscript have addressed your main concerns.

We would greatly appreciate any remaining feedback and look forward to hearing from you.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.