acceptodds
Withdrawn from ICLR 2027

Stream-in-Stream: Teaching Vision-Language-Action Models when Their Actions Expire

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action (VLA) models are mostly trained and evaluated in scenes that hold still while they compute. We study the stream-in-stream setting, in which the world keeps moving during inference, and make action timing measurable and learnable. PhysClock adds granular, settling, rolling, and air-coupled task fami- lies to simulated manipulation and advances the world while a policy computes; PhysClock-Syn labels scripted demonstrations with replay-measured timing: how much delay each decisive action tolerates, and how early and how late each action chunk may start. Fine-tuned on these demonstrations and run synchronously, π0.5 still gains 14 percentage points when the world pauses during its inference. We train a small head on the timing labels to predict the earliest and latest start of each chunk, and a scheduler waits for, keeps, or discards chunks accordingly. In a confirmatory evaluation with contrasts fixed in advance, the head raises success on the three families other than a granular pour by 3.6 and 5.5 points on two poli- cies, mostly in episodes with little timing tolerance, also when attached to frozen policies; a head that predicts success probabilities from the same labels is equiv- alent to it within a pre-specified margin. On a real arm, heads trained on event times read from demonstration videos raise grasps of a falling sheet from 58 to 83 of 200 paired trials in a pre-registered batch, after a smaller one was inconclusive, and catches of a rolling ball from 41 to 60 of 120. A head trained in simulation does not transfer to the real arm, and on a task family held out from training the head brings no measurable gain.

Primary area:
applications to robotics, autonomy, planning
Keywords:
multimodal learning, audio-visual perception, contact-rich manipulation, vision-language-action models, robot learning, dataset

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vZ2kOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work addresses the challenge that scenes that continue to change during VLA inference.

It introduces PhysClock, a dynamic-task benchmark with temporal annotated data, and trains a compact prediction head to estimate the valid activation window of each chunk.

Critical strengths

The problem is highly valuable, and the method is intuitive and effective. The addition of an effective window for predicting action blocks represents only a modest modification to existing common VLA architectures , yet it is effective.

The dynamic evaluation benchmark, expert demonstrations and annotations, and annotations form a comprehensive resource that could advance research on this problem within the community.

The setup for real-world evaluation is presented in considerable detail.

Critical weaknesses

The success-probability head and the validity-window head achieve similar results, but the implications of this finding seems somewhat vague. I believe this is an interesting and valuable finding, but it requires a clearer discussion.

Questions for authors

The labeling approach is reasonable, but obtaining the labels requires multiple simulation replays, which is quite difficult. Could lighter-weight labeling method or alternative supervision signals provide similar benefits?

Data acquisition costs are important for assessing scalability of this method. I think it’s important to discuss roughly how much data is required for each new task, and what are the associated collection and annotation costs.

Reviewer_97CsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies “stream-in-stream” VLA execution, where the physical world continues evolving while the policy is computing. It introduces PhysClock, a simulation benchmark with granular pouring, settling stacks, rolling balls, and falling sheets, together with quasi-static and paused-world counterparts. PhysClock-Syn generates timing labels by replaying demonstrations with shifted decisive actions and delayed chunk activations, defining each chunk’s earliest and latest valid start time. A small window head predicts these validity windows, and a scheduler waits for, keeps, replaces, or discards action chunks according to their predicted timing. The paper evaluates latency effects, window-based scheduling, value-based alternatives, out-of-distribution conditions, multiple VLA backbones, and real-robot tasks. The method improves simulation success by 3.6–5.5 points on the primary dynamic task families and improves falling-sheet and rolling-ball performance on hardware, but simulation-trained heads do not transfer directly to the real arm and show no gain on a held-out task family.

Critical strengths
  • The paper identifies and formalizes an important but under-measured failure mode of VLA policies: actions can become invalid while inference is still running. PhysClock and PhysClock-Syn provide matched dynamic, paused, and quasi-static conditions plus explicit timing-tolerance labels, offering a useful benchmark and diagnostic perspective for real-time robotic control.

  • The evaluation is unusually careful for a robotics paper. The confirmatory protocol uses fresh episodes, fixed contrasts, multiple training runs, confidence intervals, multiple-comparison correction, frozen-policy plug-in heads, value-head comparisons, and paired real-robot trials. This substantially strengthens the credibility of the reported timing benefits.

Critical weaknesses
  • The learned timing labels do not exactly match deployment semantics. Replay labels assume that the robot holds its pose before activation and then follows a scripted expert, whereas deployment uses a previously active learned chunk and a VLA policy. The method also assumes one decisive action and a known feasible interval per episode. This mismatch may limit how well the predicted windows represent the actual value of switching among learned policy chunks.

  • The demonstrated generality is limited. The primary endpoint excludes the pour family, where the head provides no measurable gain; the simulation-trained head does not transfer to the real arm; and the held-out task family shows no improvement. These results suggest that the method currently requires task- and platform-specific timing labels rather than providing a generally transferable timing module.

Questions for authors
  1. Clarification: How are PhysClock-Syn replay labels generated relative to the deployment behavior? In particular, how sensitive are the learned windows to replacing the scripted-expert continuation and pose-holding behavior with the previously active learned chunk and the learned VLA policy?

  2. Material validation: How much task-specific annotation and replay engineering are required to define decisive events, feasible intervals, and timing labels? Can the current labeling and scheduling procedure handle episodes with multiple decisive actions or long-horizon subtasks?

  3. Material validation: What explains the failure of simulation-trained heads to transfer to the real arm and the null result on the held-out family? How many real event-time proxy labels are needed for the reported hardware gains, and do the gains persist under a fixed compute/control-rate budget against the strongest streaming baselines?

Reviewer_jRJcOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper deals with the problem of executing VLA-generated actions in dynamic environments, where the actions might become outdated because of inference latency. The authors introduce PhysClock, which is a benchmark for time-critical robotic manipulation, and they propose a validity-window prediction head that estimates the earliest and latest times when an action chunk can be successfully executed.

Critical strengths
  1. The paper deals with an important and practical problem: The paper addresses inference latency in VLA models, which is a particularly important challenge for robotic manipulation tasks in dynamic environments. Learning when an action remains valid is an interesting perspective besides simply accelerating inference.

  2. A benchmark and timing supervision: PhysClock introduces diverse dynamic manipulation tasks and measures action timing tolerance, which provides a useful benchmark for studying latency-aware robotic control.

  3. Simple and flexible method: The proposed validity-window prediction head is lightweight and can be integrated into different VLA architectures, including frozen pretrained policies.

Critical weaknesses
  1. The methodological novelty is limited: The proposed approach mainly combines a lightweight timing prediction head with a heuristic scheduler. Similar concepts have been explored in adaptive action chunking, asynchronous control, and value-based action selection.

  2. The paper may be written with AI without much polishing. It is difficult to follow. The paper would benefit if it would be rewritten more clearly.

Questions for authors
  1. What camera and setting is used in the real robot experiment? Can you provide some illustrative figures that would show how the robot performs on the dynamic tasks?

  2. If the model itself still has a lot of latency, is the predicted validity window useful in practice?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.