acceptodds
Under review as a conference paper at ICLR 2027

Premover: Fast Vision-Language-Action Control via Early Execution During Instruction Delivery

Abstract

Vision-Language-Action (VLA) policies are typically evaluated under the assumption that the robot starts acting only after the user has finished typing or speaking. In real interactions, however, entering an instruction can take several seconds, leaving the policy idle for a substantial fraction of the interaction. Partial instructions may already contain sufficient information to begin acting before the full instruction arrives. We introduce Premover, a parameter-lightweight module that reduces interaction latency by overlapping instruction delivery with robot execution while keeping the VLA backbone frozen. Premover uses a learned vision-language focus map to identify where the current partial instruction refers in the visual scene, and an action readiness gate to determine when execution can begin. On the SO-101 robot with speech and online transcription, Premover reduces mean wall-clock time by 17.9%, from 60.5 s to 49.7s, while maintaining task success rates. In simulation on LIBERO with at the speaking rate, Premover reduces mean wall-clock time by 8.3% while maintaining comparable success to waiting for the complete instruction before execution.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, robotic manipulation, embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_N8VhOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles the challenge of reducing end-to-end interaction time between a user and a VLA policy based on typing or speaking the language command. The authors propose Premover, a lightweight module on top of a frozen VLA, based on a contrastive focus map and action-readiness gate, that begins execution from a partial streaming instruction, once its informative enough for the policy to begin executing action chunks; it continues updating as more instruction tokens stream in. They validate in LIBERO, VLA-Arena, and the SO-101 arm and modestly decrease interaction time (6-18%).

Critical strengths

Action-readiness gate: the manuscript's most convincing result is the ablation done in Table 4 that shows the action-readiness gate carries most of the benefit, raising success rate from 58.4 (acting immediately with naive streaming) to 92.2 (with very minor improvement in clock-time). It suggests that we can start acting earlier for simple pick and place tasks.

Critical weaknesses

Missing baselines: there are a few baselines that are not present which I think are necessary to assess the benefits conferred via Premover. Listed is one below and one in the questions.

(i) VLM as Gate: firstly, I think using an LLM directly as the readiness gate should be included. While readiness labels are annotated offline with Qwen judging when a prefix is sufficient, the obvious baseline is to use the same VLM online as judge to gate when an instruction contains enough information to begin execution. If this works, the action-readiness gate adds little, beside being lightweight

Minor Improvements: the gains reported (6-8% in simulation and 18% on real-world) are minor. This would be fine for complex tasks, but in pick and place tasks where the task instructions clearly delineate progression throughout a task, e.g., I can perform 'pick up' before determining where to 'place', the improvements are weak.

Questions for authors

Can you include the VLM as Gate baseline? that I mention above. Also, can you consider this ablation:

(Ablation): Full Prompt + Focus-Map: on the real-robot, Premover is more successful than full-prompt (86% vs. 81%), which doesn't make sense because all we are doing is starting earlier. My hunch is that this is due to visual reweighting with the contrastive alignment objective. Without this ablation, I can't tell how much gain comes from early execution.

5 comments
AuthorsOpenReview ↗

Author Response [1/2]

Dear Reviewer N8Vh,

Thank you for your constructive review and suggestions. We have evaluated an online VLM-as-Gate baseline and a Full-prompt + Focus-map control, and address the magnitude of the latency gains below. We have revised the manuscript accordingly and highlighted the changes in blue for ease of reference. For clarity, we address the individual points within each comment separately below.


[Weakness 1]

VLM as Gate: firstly, I think using an LLM directly as the readiness gate should be included. While readiness labels are annotated offline with Qwen judging when a prefix is sufficient, the obvious baseline is to use the same VLM online as judge to gate when an instruction contains enough information to begin execution. If this works, the action-readiness gate adds little, beside being lightweight

[Response 1]

Thank you for suggesting this baseline. We have evaluated Qwen3.5-2B, the same model used for offline readiness annotation, as an online readiness gate. Unlike offline annotation, the online VLM judges whether the current instruction prefix contains enough information to begin execution using only the words received so far and the current camera observation. Premover achieved higher success and lower Wall than VLM-as-Gate, with substantially less GPU memory. All methods are evaluated with π0.5\pi_{0.5} on LIBERO at the speaking rate, following the setting of Table 2.

MethodSR (%) ↑Wall (s) ↓ΔSR (pp)ΔWall (s)Readiness release (word index)Additional GPU memory
Full-prompt97.908.17––11.6–
VLM-as-Gate89.298.10−8.62−0.076.55.0 GiB
Premover97.577.49−0.33−0.688.213.6 MB (heads)

Although Qwen released execution earlier than Premover (6.5 vs. 8.2 words), its success rate was 8.28 pp lower (89.29% vs. 97.57%). Including Qwen inference overhead, its Wall was also higher (8.10 vs. 7.49 s), nearly the same as Full-prompt (8.17 s). Qwen additionally required 5.0 GiB of GPU memory, compared with 13.6 MB for Premover's heads.

These results show that using Qwen directly as an online readiness gate does not match Premover's success–latency trade-off in this setting. The learned readiness gate therefore offers benefits beyond its smaller memory footprint.

We have added this baseline and its failure analysis to Section 4.5.4.


[Weakness 2]

Minor Improvements: the gains reported (6-8% in simulation and 18% on real-world) are minor. This would be fine for complex tasks, but in pick and place tasks where the task instructions clearly delineate progression throughout a task, e.g., I can perform 'pick up' before determining where to 'place', the improvements are weak.

[Response 2]

We agree that the average latency improvements are modest in some settings. However, the gains vary considerably across tasks and instruction-delivery rates. In simulation, Premover reduced mean Wall by 5.9-8.3% at the speaking rate, with larger reductions of 11.0-23.5% at the typing rate. On the real robot, the average reduction was 17.9%, with larger improvements on individual tasks, including 38.5% on Cup→Cup (53.5 to 32.9 s) and 26.4% on Block→Cup (66.6 to 49.0 s).

Our primary goal is to demonstrate that the instruction-delivery interval, which conventional VLAs spend waiting, can be used for robot execution without substantially compromising task success. The achievable gain naturally depends on the task and how much execution can overlap with the remaining instruction. Our results demonstrate that this previously unused interval offers a practical opportunity to reduce end-to-end interaction time, even for relatively simple manipulation tasks.

We have discussed the factors affecting the available overlap in Section 5.

(Our response continues in the next comment.)

AuthorsOpenReview ↗

Author Response [2/2]

[Question 1]

Can you include the VLM as Gate baseline?

[Response 1]

Thank you for raising this question. We have addressed it in our response to Weakness 1 above.


[Question 2]

Full Prompt + Focus-Map: on the real-robot, Premover is more successful than full-prompt (86% vs. 81%), which doesn't make sense because all we are doing is starting earlier. My hunch is that this is due to visual reweighting with the contrastive alignment objective. Without this ablation, I can't tell how much gain comes from early execution.

[Response 2]

We appreciate the suggestion to separate the effects of visual reweighting and early execution. To address this concern, we have evaluated a Full-prompt + Focus-map control that uses the complete instruction with the same trained focus-map reweighting, including the contrastive alignment objective.

TaskFull-promptFull-prompt + Focus-mapPremover
Block→Cup657075
Cat→Cup959590
Cup→Cup909595
Push Cup859090
Drawer708080
Mean81.0 (81/100)86.0 (86/100)86.0 (86/100)

Full-prompt + Focus-map improved mean success from 81.0% to 86.0% (+5.0 pp), matching Premover. This shows that focus-map reweighting improves task success even when execution starts after the full instruction, rather than the gain coming from early execution alone. Premover also achieved lower mean Wall than Full-prompt + Focus-map (49.7 vs. 56.9 s, a 12.7% reduction), while maintaining the same mean success. Thus, Premover retains a timing advantage beyond the success improvement provided by focus-map reweighting.

We have added these control results to Section 4.5.2 and Appendix F.1, with successful-episode Wall also reported in Appendix F.2.

Thanks for the additional experiments. The LLM/VLM (alternative streaming) baselines in Table 7 address my first concern. The Full-prompt + Focus-map control answers my question, but it shows the real-robot success gain comes from the focus map, not early execution (86.0% for both). So early execution saves only 12.7% rather than the reported 17.9%, and the new pre-motion analysis suggests the focus map accounts for at least part of this. It would be good to report this in the abstract and introduction, since they currently overstate the effect of early execution. My original concerns about the size of the gains and the reliance on pick-and-place instruction structure remain, so I am keeping my score.

Reviewer_a623OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies execution before a user has finished delivering an instruction. Premover adds a supervised vision-language focus map and an action-readiness gate to a frozen VLA policy, and uses agreement between region-conditioned action chunks to limit execution when later parts of the instruction remain ambiguous. It evaluates two backbones on LIBERO and VLA-Arena, and deploys π0.5 on an SO-101 robot with TTS and online transcription. Across the four backbone/benchmark combinations at the speaking rate, mean host wall-clock time decreases by approximately 5.9–8.3%, with overall success within 0.6 percentage points of Full-prompt. The real-robot single-object results show 60.5s versus 49.7s mean time and 81% versus 86% success. The setting assumes a fixed intended task rather than corrections after execution begins.

Critical strengths

Evidence across models and a physical robot. The speaking-rate results are consistent at the aggregate level across two backbones and two benchmarks. The robot experiments use online transcription, and the joint-displacement plot supports the claim that actual physical motion can begin before instruction delivery ends. This provides more persuasive feasibility evidence than simulation alone.

Critical weaknesses
  1. The source of the timing gain is unclear. Premover changes both execution timing and visual conditioning, while success-rate differences also affect average runtime. A control using the same modules but waiting for the full instruction would help isolate the benefit of early execution.

  2. Simulation speedups depend on the timing setup. Words arrive in wall-clock time, but active actions run without pacing to a physical control rate. The reported waiting fractions and speedups therefore depend on simulator throughput. A fixed-control-rate evaluation would clarify their practical relevance.

  3. Readiness generalization remains unclear. The evaluation holds out demonstrations or initial states, not instruction wording, and Fig.5 mainly shows agreement with annotated cutoffs. Tests on unseen paraphrases or shifted disambiguation points would strengthen the claim. The roughly five-percentage-point success drops on VLA-Arena Extrapolation also deserve discussion.

  4. Real-robot validation is limited in scope. Experiments use a single robot platform and TTS-generated instructions on a small set of tasks. Tests with spontaneous human speech would provide stronger evidence that the benefits extend to natural interaction.

Questions for authors
  1. What evidence shows that Premover improves on an LLM that incrementally outputs executable sub-instructions to the same VLA, using only the input received so far?
  2. How much speedup remains against a full-instruction baseline with identical focus/consensus modules? Could differences in success rate explain part of the reported 17.9% gain?
  3. Why are active simulation steps unpaced? Is there evidence that the speedup persists when actions follow a fixed physical control rate?
  4. Does evaluation include instruction phrasings unseen during readiness training? Separately, how do you explain the Extrapolation success drops in Tables 6 and 8?
6 comments
AuthorsOpenReview ↗

Author Response [1/5]

Dear Reviewer a623,

Thank you for your careful review and constructive suggestions. We have conducted additional timing analyses, evaluated a Full-prompt + Focus-map/consensus module control, tested incremental LLM baselines, and repeated the simulation evaluation at a fixed 20 Hz control rate. We have revised the manuscript accordingly and highlighted the changes in blue for ease of reference. For clarity, we address the individual points within each comment separately below.


[Weakness 1]

The source of the timing gain is unclear. Premover changes both execution timing and visual conditioning, while success-rate differences also affect average runtime. A control using the same modules but waiting for the full instruction would help isolate the benefit of early execution.

[Response 1-1]

Success-rate differences also affect average runtime

We apologize that our original presentation did not clearly explain how success-rate differences could affect Wall averaged over all episodes. We have therefore reanalyzed Wall using successful episodes in both simulation and real-robot experiments. The results show that Premover retained a lower mean Wall than Full-prompt across all four simulation configurations and both real-robot task groups. We report the detailed results below.

Simulation. Premover achieved a lower mean Wall than Full-prompt on successful episodes across all four backbone–benchmark combinations, with reductions of 7.1–16.1%. The table below compares all-episode and successful-episode-only Wall (s).

Backbone × BenchmarkAll: FullAll: PremoverSuccess-only: FullSuccess-only: Premover
π0.5\pi_{0.5} × LIBERO8.17 ± 0.027.49 ± 0.028.042 ± 0.0027.330 ± 0.024
GR00T-N1.7 × LIBERO8.60 ± 0.038.07 ± 0.058.235 ± 0.0247.647 ± 0.030
π0.5\pi_{0.5} × VLA-Arena10.37 ± 0.059.63 ± 0.088.380 ± 0.0517.030 ± 0.012
GR00T-N1.7 × VLA-Arena8.12 ± 0.057.64 ± 0.036.751 ± 0.0495.779 ± 0.059

Real-robot experiments. Premover also retained a lower mean Wall on successful episodes in both task groups: 41.9 vs. 49.4 s on single-object tasks (a 15.2% reduction) and 50.2 vs. 63.3 s on disambiguation tasks (a 20.7% reduction). The table below compares all-episode and successful-episode-only Wall, with all times reported in seconds.

TaskAll: Full-prompt (s)All: Premover (s)Success-only: Full-prompt (s)Success-only: Premover (s)
Block→Cup66.649.047.731.0
Cat→Cup48.438.845.432.7
Cup→Cup53.532.945.729.7
Push Cup54.954.747.048.5
Drawer79.173.261.467.5
Single-object Mean60.549.749.441.9
On Cup83.677.849.642.3
Gray Cat79.371.055.054.1
Orange Cat101.881.285.254.2
Disambiguation Mean88.276.763.350.2

Across all four simulation configurations and both real-robot task groups, Premover retained a lower mean Wall when unsuccessful episodes were excluded. These results show that the timing advantage persists under successful-episode-only evaluation. We have summarized these findings in Sections 4.2 and 4.3. The all-episode and successful-episode averages are reported together in Appendix F.2 (Table 10) and Appendix G.1 (Table 14).


[Response 1-2]

A control using the same modules but waiting for the full instruction would help isolate the benefit of early execution.

Following your suggestion, we have evaluated a Full-prompt + Focus-map control on the five single-object real-robot tasks. Premover retained a lower mean Wall than this control, which uses the same trained focus-map while waiting for the complete instruction. Under our execution rule, consensus-guided execution is inactive once the complete instruction is available.

MethodSR (%) ↑Wall (s) ↓Success-only Wall (s) ↓
Full-prompt81.060.549.4
Full-prompt + Focus-map86.056.950.3
Premover86.049.741.9

Premover achieved a 12.7% lower mean Wall than Full-prompt + Focus-map (49.7 vs. 56.9 s), with both methods achieving an observed mean success rate of 86.0%. Mean Wall also remained lower among successful episodes (41.9 vs. 50.3 s). These results show that Premover’s timing advantage persists against a full-instruction control using the same trained visual reweighting. We have included these control results in Section 4.5.2 and Appendix F.1. The successful-episode Wall comparison is also reported in Appendix F.2.

(Our response continues in the next comment.)

AuthorsOpenReview ↗

Author Response [2/5]

[Weakness 2]

Simulation speedups depend on the timing setup. Words arrive in wall-clock time, but active actions run without pacing to a physical control rate. The reported waiting fractions and speedups therefore depend on simulator throughput. A fixed-control-rate evaluation would clarify their practical relevance.

[Response 2]

We appreciate the concern about how simulator timing affects the reported speedups. Our original evaluation measured host wall-clock runtime using the same calibration and timing rules for all methods. Wall included policy inference and all method-specific overheads, allowing a direct comparison under that setup.

To test whether the gains also hold with control-rate pacing, we have repeated the LIBERO evaluation with π0.5\pi_{0.5} at its default 20 Hz control frequency. Every simulator step, including active action steps, was paced to take at least 50 ms.

MethodSR (%) ↑Wall (s) ↓
Full-prompt97.8813.33
Premover97.6411.85

Averaged over all LIBERO tasks and three evaluation seeds (0, 1, and 2), Premover remained faster than Full-prompt under the default 20 Hz control frequency, reducing mean Wall time from 13.33 to 11.85 s (11.1%) while maintaining comparable success rates (97.64% vs. 97.88%). This confirms that Premover's timing advantage persists even when active simulation steps are paced to a physical control rate. We have added these results to Appendix E (Table 8), with a reference in Section 4.3.


[Weakness 3]

Readiness generalization remains unclear. The evaluation holds out demonstrations or initial states, not instruction wording, and Fig.5 mainly shows agreement with annotated cutoffs. Tests on unseen paraphrases or shifted disambiguation points would strengthen the claim. The roughly five-percentage-point success drops on VLA-Arena Extrapolation also deserve discussion.

[Response 3-1]

Readiness generalization remains unclear.

Our current evaluation focuses on readiness after target-task adaptation, rather than zero-shot generalization to unseen instructions. As with task-specific VLA adaptation, training the readiness gate and focus-map on target task demonstrations is an intended part of deploying Premover, while the VLA backbone remains frozen. Accordingly, we evaluated on held-out initial states, but did not separately test unseen instruction phrasings or shifted disambiguation points. Figure 5 shows that readiness scores align with annotated cutoffs within this setting, but does not rule out instruction-specific cutoff learning. The demonstration and initial-state splits are described in Appendix A.1. We have qualified the interpretation of Figure 5 in Section 4.4 and added readiness generalization to unseen instruction phrasings and changes in when the target is identified as future work in Section 5.


[Response 3-2]

The roughly five-percentage-point success drops on VLA-Arena Extrapolation also deserve discussion.

We acknowledge the success drops on VLA-Arena Extrapolation. At the speaking rate, Premover's success was roughly five percentage points lower than Full-prompt on π0.5\pi_{0.5} and GR00T-N1.7. ( Tables 11 and 13 in the revised manuscript; Tables 6 and 8 in the original submission)

A closer examination suggests that the results vary with instruction structure, while performance on unseen-object cases remains close to Full-prompt. For instructions combining spatial relations, one possible explanation is that the information needed to begin an initial action is available before all spatial constraints governing subsequent actions are resolved. The larger drops at the typing rate are consistent with longer execution under incomplete information, although the typing and speaking protocols also differ.

At the same time, Premover also improved success on other VLA-Arena tasks. With π0.5\pi_{0.5} at 150 WPM, success increased from 69.52% to 81.90% (+12.38 pp) on the kiwi-to-white-bowl task with the stove on, and from 67.62% to 69.52% (+1.90 pp) on the apple-to-bowl task with a moving obstacle. These gains may partly come from focus-map reweighting, which emphasizes the relevant object and destination regions when predicting actions and may help the frozen policy execute more reliably in these scenes. We have added robustness to instructions that combine multiple spatial relations as a direction for future work in Section 5.

(Our response continues in the next comment.)

AuthorsOpenReview ↗

Author Response [3/5]

[Weakness 4]

Real-robot validation is limited in scope. Experiments use a single robot platform and TTS-generated instructions on a small set of tasks. Tests with spontaneous human speech would provide stronger evidence that the benefits extend to natural interaction.

[Response 4]

We acknowledge that our real-robot evaluation is limited to a single robot platform and a controlled speech setting. We evaluated Premover on the SO-101 across eight tasks, with 20 trials per task and method (160 trials per method). We used TTS-generated audio to provide consistent speech input across methods, while processing the instructions through online ASR. Under this setting, Premover reduced mean Wall by 17.9% (60.5 to 49.7 s) on the five single-object tasks.

We believe that these experiments demonstrate the feasibility and latency benefits of streaming execution on a physical robot. We have clarified the use of TTS-generated speech and online transcription in Section 4.2 and identified unscripted human speech and other robot platforms as future evaluation directions in Section 5.


[Question 1]

What evidence shows that Premover improves on an LLM that incrementally outputs executable sub-instructions to the same VLA, using only the input received so far?

[Response 1]

Thank you for your suggestion. We have evaluated Qwen3.5-2B and Qwen3.5-9B as text-only incremental sub-instruction generators, along with a vision-enabled Qwen3.5-2B baseline that additionally receives camera observations. Each model receives only the streaming prefix, the input received so far, and outputs either WAIT or a single executable sub-instruction. The models are instructed to propose actions based on the available information, such as picking up an identified object, without assuming missing objects, attributes, or destinations. Generated sub-instructions are passed to the same frozen VLA used by Premover to predict robot actions, while WAIT responses defer execution. All methods are evaluated with π0.5\pi_{0.5} on LIBERO at the speaking rate, following the setting of Table 2.

MethodSR (%) ↑Wall (s) ↓ΔSR (pp)ΔWall (s)Additional GPU memory
Full-prompt97.908.17–––
Qwen3.5-2B (text)76.748.02−21.17−0.155.0 GiB
Qwen3.5-9B (text)92.008.53−5.90+0.3618.5 GiB
Qwen3.5-2B (text + camera)69.888.75−28.02+0.585.1 GiB
Premover97.577.49−0.33−0.6813.6 MB

Among the incremental LLM baselines, Qwen3.5-2B achieved 76.74% success with 8.02 s, while Qwen3.5-9B reached 92.00% success with 8.53 s. Adding camera observations to the 2B model did not improve performance (69.88% success, 8.75 s Wall). In comparison, Premover achieved 97.57% success with 7.49 s Wall, close to Full-prompt's 97.90% success but with lower latency. Most failures of the LLM baselines occur when execution starts before the annotated readiness cutoff (Figure 5). For example, Qwen3.5-9B starts before the cutoff in 60% of episodes, where its success drops to 87.9%. With camera input, the 2B model also often named visible objects that had not yet been mentioned. The LLM/VLM baselines also required 5.0-18.5 GiB of additional GPU memory, compared with 13.6 MB for Premover's heads.

These results show that Premover achieves a better success–latency trade-off than the evaluated incremental LLM baselines, without requiring a separate LLM at inference time. We have added the baseline setup, results, and failure analysis to Section 4.5.4.

(Our response continues in the next comment.)

Reviewer_tZ54OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Premover, a parameter-lightweight module that reduces interaction latency by overlapping instruction delivery with robot execution while keeping the VLA backbone frozen. Premover uses a learned vision-language focus map to identify where the current partial instruction refers in the visual scene, and an action readiness gate to determine when execution can begin. Both components are trained jointly on features from the frozen VLA backbone. The focus map is supervised with segmentation masks and a region-contrastive loss, and the readiness gate is supervised with prefix-level labels derived from a readiness cutoff annotated by a language model. They evaluate Premover with two frozen VLA backbones in simulation.

Critical strengths
  • The problem framing is useful and new for VLAs. Treating instruction delivery time as part of end-to-end latency is a fair point that standard VLA evaluation ignores, and it is orthogonal to work on faster inference.

  • The method is cheap and backbone-agnostic. It keeps the backbone frozen, adds small heads, and is shown on two different backbones and two benchmarks with three seeds and matched episodes. The timing protocol in the Appendix is careful, with real host-time arrival, replayed ASR timestamps, and all method overheads counted in wall time. This makes the simulation comparisons credible.

Critical weaknesses
  • The real-robot speedup may not be mainly from overlap. Premover can save at most the instruction window, which Figure 5c puts at about 6.1 s, yet the mean single-object saving is about 10.8 s, and Cup→Cup drops from 53.5 to 32.9 s. Wall averages over failed episodes that can run to the 900-step limit, so the gap is largely driven by Premover having higher success than Full-prompt. It is also unclear why acting on less language should beat acting on the full instruction.

  • The gain over simple baselines is small and sometimes absent. In simulation, the speaking-rate saving is only about 6 to 8 percent. On π0.5 with VLA-Arena, Noun-triggered reaches the same success, 63.5 versus 63.7, at a lower Wall, 8.97 versus 9.63 s, so the claim that simple triggers "often incur substantial drops in success" does not hold there. In Table 5, Fixed 75% gets 93.6 versus 93.8 success with only 0.18 s more Wall, and the ablations in Tables 4 and 5 use a single seed while seed-level std is around 0.5 to 1.3 points. Adding the focus map raises success by 1.6 points but also raises Wall. It is hard to see that the learned gate and focus map clearly beat a well-tuned heuristic.

  • Generalization of the readiness gate is not tested. The heads are trained per backbone-benchmark pair on demos of the same tasks used for evaluation, and LIBERO and the robot tasks use a small, fixed set of instruction strings. The gate could simply memorize the cutoff word index per instruction. There is no test on held-out tasks, paraphrased instructions, or instructions where key words come late.

Questions for authors
  • Report real-robot Wall on successful episodes only, or report time-to-first-motion and time-to-completion separately, so the overlap effect is separated from the success effect?

  • Add a Full-prompt baseline that uses the trained focus-map reweighting? This tells whether the success gains on the robot and on disambiguation come from streaming or from the reweighting.

  • How is the target noun found for Noun-triggered at test time? Is it from an oracle annotation or detected online?

  • How does the gate behave on paraphrased or held-out instructions, and on instructions where the object is named last?

4 comments
AuthorsOpenReview ↗

Author Response [1/4]

Dear Reviewer tZ54,

Thank you for the constructive review and for recognizing the importance of accounting for instruction-delivery time. Following your suggestions, we have conducted additional real-robot timing analyses, evaluated a Full-prompt + Focus-map control, and repeated the ablations with three evaluation seeds. We have revised the manuscript accordingly and highlighted the changes in blue for ease of reference. For clarity, we address the individual points within each comment separately below.


[Weakness 1]

The real-robot speedup may not be mainly from overlap. Premover can save at most the instruction window, which Figure 5c puts at about 6.1 s, yet the mean single-object saving is about 10.8 s, and Cup→Cup drops from 53.5 to 32.9 s. Wall averages over failed episodes that can run to the 900-step limit, so the gap is largely driven by Premover having higher success than Full-prompt. It is also unclear why acting on less language should beat acting on the full instruction.

[Response 1-1]

Wall averages over failed episodes that can run to the 900-step limit.

We agree that including failed episodes can affect mean Wall. However, our additional analysis shows that Premover retained a lower mean Wall even when we excluded unsuccessful episodes.

Successful-episode-only Wall. To assess the effect of failed episodes on mean Wall, we have separately analyzed Wall using only successful episodes. As shown in the table below, Premover averaged 41.9 s compared with 49.4 s for Full-prompt on single-object tasks, and 50.2 s compared with 63.3 s on disambiguation tasks.

TaskAll: FullAll: PremoverSuccess-only: FullSuccess-only: Premover
Block→Cup66.649.047.731.0
Cat→Cup48.438.845.432.7
Cup→Cup53.532.945.729.7
Push Cup54.954.747.048.5
Drawer79.173.261.467.5
Single-object Mean60.549.749.441.9
On Cup83.677.849.642.3
Gray Cat79.371.055.054.1
Orange Cat101.881.285.254.2
Disambiguation Mean88.276.763.350.2

[Response 1-2]

Premover can save at most the instruction window.

We acknowledge that instruction–execution overlap alone cannot explain a time saving larger than the instruction-delivery window. To better understand the observed Wall reduction, we have separately analyzed when the robot begins physical motion and when it completes the task.

Time-to-first-motion & Time-to-completion. The instruction-delivery window limits the time saved through overlap alone, but not the overall Wall reduction. Pre-motion delay, the time between action generation and actual physical motion, can also contribute to the timing difference.

To examine this, we have analyzed time-to-first-motion and time-to-completion separately. Premover began moving earlier than Full-prompt on single-object tasks (6.4 vs. 15.7 s) and disambiguation tasks (8.8 vs. 13.3 s). It also achieved lower mean time-to-completion on successful episodes in both task groups. All times were measured from the start of instruction delivery.

Task groupMethodTime-to-first-motion (s)Time-to-completion (s)Pre-motion delay (s)
Single-objectFull-prompt15.749.49.7
Single-objectPremover6.441.93.2 (from release)
DisambiguationFull-prompt13.363.35.4
DisambiguationPremover8.850.22.6 (from release)

For Full-prompt, pre-motion delay is measured from the first generated action to the first detectable joint motion. For Premover, “from release” means that this delay is measured from when the readiness gate enables execution to the first detectable joint motion. As illustrated in Figure 5(c), physical motion can begin substantially later than instruction delivery. Our additional analysis shows that Full-prompt took 9.7 s from its first generated action to start motion, compared with 2.8 s for Full-prompt + Focus-map.

One possible explanation for this delay is that the policy may output small joint movements, making the robot appear to pause. Both methods use the same pretrained VLA policy and standard LeRobot asynchronous execution framework. Focus-map reweighting changes the visual features used for action prediction, which may help the robot start moving earlier.

These results suggest that reduced pre-motion delay, together with instruction–execution overlap, may help explain why the Wall reduction exceeds the instruction-delivery window.

(Our response continues in the next comment.)

AuthorsOpenReview ↗

Author Response [2/4]

[Response 1-3]

It is also unclear why acting on less language should beat acting on the full instruction.

To address this concern, we have evaluated a Full-prompt + Focus-map control that uses the complete instruction with the same Focus-map, under the same five-task real-robot setting.

TaskFull-promptFull-prompt + Focus-mapPremover
Block→Cup657075
Cat→Cup959590
Cup→Cup909595
Push Cup859090
Drawer708080
Mean81.0 (81/100)86.0 (86/100)86.0 (86/100)

Full-prompt + Focus-map improved mean success from 81.0% to 86.0% (+5.0 pp), matching Premover. This shows that focus-map reweighting improves task success even when the complete instruction is available, rather than acting on less language. Premover also achieved lower mean Wall than Full-prompt + Focus-map (49.7 s vs. 56.9 s), despite their identical mean success rates. These results indicate that focus-map reweighting contributes to higher success, while Premover enables earlier execution without sacrificing the observed mean success rate.

We have added these results to Section 4.5.2 and Appendix F.1. Section 4.2 also discusses how focus-map reweighting may contribute to the higher real-robot success rates.


[Weakness 2]

The gain over simple baselines is small and sometimes absent. In simulation, the speaking-rate saving is only about 6 to 8 percent. On π0.5 with VLA-Arena, Noun-triggered reaches the same success, 63.5 versus 63.7, at a lower Wall, 8.97 versus 9.63 s, so the claim that simple triggers "often incur substantial drops in success" does not hold there. In Table 5, Fixed 75% gets 93.6 versus 93.8 success with only 0.18 s more Wall, and the ablations in Tables 4 and 5 use a single seed while seed-level std is around 0.5 to 1.3 points. Adding the focus map raises success by 1.6 points but also raises Wall. It is hard to see that the learned gate and focus map clearly beat a well-tuned heuristic.

[Response 2-1]

The gain over simple baselines is small and sometimes absent.

We agree that Premover's gains over simple baselines vary across benchmarks and tasks. However, the gains are substantially larger in other settings. On GR00T-N1.7/LIBERO, Premover achieves 93.8% success compared with 77.2% for Noun-triggered (+16.6 pp). On real-robot tasks, the gains are even larger: +31.0 pp in success rate for single-object tasks (86.0% vs. 55.0%) and +37.5 pp in partial success score for disambiguation tasks (71.7% vs. 34.2%). These results show that Premover can substantially improve success over simple baselines, even though the gains vary across settings.

Furthermore, simple baselines rely on information not available from the current instruction prefix alone. Fixed 75% requires the final instruction length, while Noun-triggered uses an oracle target-noun annotation. Premover instead decides when to start using only the current streaming prefix and observation. Across two backbones and two benchmarks, it reduces mean Wall by 5.9–8.3% relative to Full-prompt at the speaking rate, with success-rate differences within 0.6 percentage points.

The revised discussion in Section 4.3 no longer includes the original broad claim about success drops. The oracle target-noun annotation is specified in Section 4.1, and the need to know the complete instruction length for fixed-fraction baselines is explained in Section 4.5.3.

(Our response continues in the next comment.)

AuthorsOpenReview ↗

Author Response [3/4]

[Response 2-2]

The ablations in Tables 4 and 5 use a single seed

To address concerns about single-seed ablations, we repeated the component and fixed-fraction evaluations, now reported in Tables 5 and 6 of the revised manuscript, with two additional seeds (1 and 2), keeping all other experimental conditions unchanged.

Table 5. Component ablation. The original trends remain in the three-seed averages. The readiness gate increased mean success by 34.81 percentage points while reducing Wall by 0.59 s. Adding the focus map further improved success by 1.26 percentage points, with a 0.08 s increase in Wall.

StreamingReadyFocusSR (%) ↑Wall (s) ↓
✓57.69 ± 0.868.58 ± 0.05
✓✓92.50 ± 0.257.99 ± 0.02
✓✓✓93.76 ± 0.468.07 ± 0.05

Table 6. Fixed-fraction execution baselines. Among the settings with Full-prompt-level success, Premover has the lowest mean Wall (8.07 s vs. 8.26 s for Fixed 75% and 8.60 s for Full-prompt). The three-seed averages preserve the original success–latency trend.

SettingSR ↑ (%)Wall ↓ (s)
Fixed 0% (Right-away)57.69 ± 0.868.58 ± 0.05
Fixed 25%68.86 ± 0.938.04 ± 0.06
Fixed 50%85.71 ± 0.267.80 ± 0.01
Fixed 75%93.83 ± 0.398.26 ± 0.04
Fixed 100% (Full-prompt)93.95 ± 0.258.60 ± 0.03
Premover93.76 ± 0.468.07 ± 0.05

We have updated Tables 5 and 6 and the corresponding discussions in Sections 4.5.2 and 4.5.3 with these three-seed results.


[Weakness 3]

Generalization of the readiness gate is not tested. The heads are trained per backbone-benchmark pair on demos of the same tasks used for evaluation, and LIBERO and the robot tasks use a small, fixed set of instruction strings. The gate could simply memorize the cutoff word index per instruction. There is no test on held-out tasks, paraphrased instructions, or instructions where key words come late.

[Response 3]

We agree that our evaluation does not establish generalization to held-out tasks, paraphrased instructions, or instructions where key words arrive late. The heads were trained on demonstrations of the evaluated tasks, with only initial states held out. We therefore cannot rule out that the gate learns instruction-specific cutoff positions.

However, environment-specific fine-tuning is common in VLA deployment, particularly when transferring to a different environment, such as from LIBERO to LIBERO-Plus. Premover follows a similar adaptation setting: the readiness and focus heads can be trained using demonstrations collected for the target environment, while keeping the VLA backbone frozen during head training. Our goal is therefore to enable early execution after adaptation, rather than to claim zero-shot generalization to unseen instructions.

The demonstration and initial-state splits are described in Appendix A.1. We have added generalization to unseen instruction phrasings and changes in when the target is identified as future work in Section 5.


[Question 1]

Report real-robot Wall on successful episodes only, or report time-to-first-motion and time-to-completion separately, so the overlap effect is separated from the success effect?

[Response 1]

Thank you for raising this point. We have addressed it in our response to Weakness 1 above.

(Our response continues in the next comment.)

AuthorsOpenReview ↗

Author Response [4/4]

[Question 2]

Add a Full-prompt baseline that uses the trained focus-map reweighting? This tells whether the success gains on the robot and on disambiguation come from streaming or from the reweighting.

[Response 2]

To separate the effects of focus-map reweighting and streaming execution, we evaluated a Full-prompt + Focus-map control under the same five-task real-robot setting. Please see the table in [Response 1-3] for the task-level results.

  • Success rate. Full-prompt + Focus-map improved mean success from 81.0% to 86.0% (+5.0 pp), matching Premover's 86.0%. This suggests that visual reweighting contributes to the success improvement even without streaming execution.

  • Wall time. Premover achieved a mean Wall of 49.7 s, compared with 56.9 s for Full-prompt + Focus-map (12.7% reduction), while both achieved 86.0% mean success. Premover retains a timing advantage over the focus-map control at the same observed mean success.

We have added these control results to Section 4.5.2 and Appendix F.1.


[Question 3]

How is the target noun found for Noun-triggered at test time? Is it from an oracle annotation or detected online?

[Response 3]

We apologize that our original explanation was not sufficiently clear. We specified the target noun using oracle information. At test time, Noun-triggered began execution when that noun appeared in the streaming prefix; it did not infer the target noun online. Premover does not require oracle target-noun annotations at test time and determines when to start from the current streaming prefix and observation. We have clarified the oracle target-noun annotation and the trigger rule for Noun-triggered in Section 4.1.


[Question 4]

How does the gate behave on paraphrased or held-out instructions, and on instructions where the object is named last?

[Response 4]

Thank you for raising this question. Premover is designed for a target-task adaptation setting, where the readiness and focus heads are trained on demonstrations from the target tasks while keeping the VLA backbone frozen. Our evaluation follows this setting, using held-out initial states rather than unseen tasks or instruction phrasings.

While our readiness supervision is designed to delay execution until the target-identifying information arrives, we have not evaluated how the learned gate behaves on unseen paraphrases or instructions where the object is named last. We recognize this and have revised Section 5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.