acceptodds
Under review as a conference paper at ICLR 2027

Fast-VLA: You Only Think Once for Robot Manipulation

Abstract

Vision–language–action (VLA) policies commonly place a multi-billion-parameter model inside the feedback loop, repeatedly recomputing task context whenever they predict a new action chunk. Yet task understanding changes far more slowly than the robot state. We introduce Fast-VLA, which invokes a frozen 4B VLM once per episode to contextualize the instruction and initial views as fixed, multi-layer key–value task memory. During execution, a lightweight online actor fuses learned time- and arm-indexed queries with current visual and proprioceptive feedback to form action queries. These queries retrieve task context from the fixed memory and predict action chunks without re-running the foundation model. Fast-VLA achieves 92.46% and 92.40% success on RoboTwin 2.0 Clean and Randomized, and 95.95% and 79.11% on LIBERO and LIBERO-Plus evaluation. Against three continually conditioned baselines on the same A100, it reduces RoboTwin amortized policy inference time by 10.9–17.1× and requires 1.22 GiB of post-prefill online resident memory. Task-memory swaps further show that the actor relies on episode-matched task memory. These results show that task context can be established once while closed-loop control remains responsive throughout execution.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Efficient Robot Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2i2JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents Fast-VLA, an efficient vision-language-action framework that decouples episode-level task understanding from continuous robot control. The proposed method invokes a frozen 4B vision-language model (VLM) once at the beginning of each episode to construct a fixed, multi-layer key-value task memory from the language instruction and initial visual observations.

During execution, a lightweight online actor processes current visual observations, short-term visual history when available, and proprioceptive feedback. Through learned action queries and cross-attention mechanisms, the actor retrieves task-relevant information from the cached memory and predicts action chunks in a single forward pass without repeatedly invoking the large VLM.

The authors evaluate Fast-VLA on RoboTwin 2.0, LIBERO, and LIBERO-Plus. The proposed framework achieves competitive manipulation success rates while substantially reducing amortized inference latency and online GPU memory consumption. Additional ablations investigate the importance of task memory, online observations, memory refresh frequency, and different memory encoders.

Critical strengths
  1. The paper addresses an important practical limitation of current VLA architectures: the substantial computational overhead associated with repeatedly executing large vision-language backbones during robot control. The proposed separation between episode-level task understanding and lightweight online control is intuitive and well motivated. By reusing task representations across action predictions while maintaining access to current visual and proprioceptive feedback, Fast-VLA offers a promising direction for reducing inference costs without completely sacrificing closed-loop control. The overall framework is also relatively straightforward, making the proposed approach potentially attractive for practical robotic applications.

  2. The reported experimental results are impressive, particularly on RoboTwin 2.0, where Fast-VLA achieves 92.46% and 92.40% success in the Clean and Randomized settings, respectively. Despite maintaining competitive task success rates, the method achieves a substantial reduction in inference latency compared with several strong VLA and WAM baselines. The reported 10.9–17.1× reduction in amortized inference time, together with the reduced online GPU memory footprint, demonstrates a compelling performance-efficiency trade-off. Importantly, the authors include the one-time VLM prefill cost in their amortized latency measurements, making the reported efficiency gains more informative than comparisons based solely on the online actor's forward-pass latency.

  3. The paper provides evaluations across three established manipulation benchmarks, including robustness assessments on LIBERO-Plus. The ablation studies are particularly useful for understanding the proposed architecture. Memory swapping experiments demonstrate that the online actor relies on episode-matched task representations, while comparisons involving different memory constructions and encoders help establish the contribution of the multi-layer KV memory. Furthermore, the memory refresh experiments provide evidence that repeatedly invoking the VLM is not necessary to achieve competitive performance in the evaluated settings. Overall, these experiments provide meaningful empirical support for the proposed efficiency-oriented architecture.

Critical weaknesses
  1. My primary concern is the absence of real-world robotic experiments. Although the simulation results are promising, the paper does not provide experimental validation on physical robotic systems. This limitation is particularly important given the paper's central motivation of enabling efficient and responsive robot manipulation. While simulation benchmarks provide useful measurements of policy performance, they cannot fully capture the practical challenges encountered during real-world deployment, including perception uncertainty, control inaccuracies, hardware communication overhead, and unexpected environmental disturbances. Moreover, the reported inference efficiency is measured through offline profiling on an A100 GPU rather than through end-to-end execution on a physical robotic platform. Consequently, it remains unclear how the proposed computational improvements translate into practical control performance and responsiveness in real-world settings. I believe that real-world validation is particularly important for an efficiency-oriented VLA framework, as computational speedups alone do not necessarily guarantee improved robotic execution. Even a relatively small-scale evaluation on physical manipulation tasks, accompanied by measurements of actual policy latency and execution success, would substantially strengthen the practical significance of the paper.

  2. The proposed framework constructs task memory once at the beginning of an episode and keeps it fixed throughout execution. Although the online actor continuously incorporates current observations, the paper provides limited analysis of scenarios where the environment changes substantially after the initial memory construction. For example, unexpected object movements, execution failures, or interruptions may introduce discrepancies between the initial task context and the current environment. The existing robustness evaluations are encouraging, but they primarily consider perturbations present when the episode begins, rather than controlled changes introduced during execution. Additional experiments involving mid-episode disturbances would help clarify the limitations of maintaining fixed task memory.

  3. The reported RoboTwin configuration predicts and executes 50 actions before replanning. While this execution schedule contributes to the low amortized inference cost, it also limits the frequency at which the policy can respond to new observations. This raises questions about how well the proposed approach would perform in manipulation tasks requiring more frequent feedback, particularly those involving precise contact, unexpected execution errors, or dynamic object interactions. Although the paper provides detailed inference latency measurements, it does not sufficiently investigate how the performance-efficiency trade-off changes with shorter action execution horizons. Such an analysis would provide a more complete understanding of the practical benefits of Fast-VLA for responsive robotic control.

Questions for authors
  1. Could the authors provide real-world robotic experiments to validate the proposed framework? Given that the primary motivation is improving inference efficiency for robot manipulation, it would be particularly valuable to evaluate Fast-VLA on a physical robotic platform and report both task success rates and actual inference latency. Ideally, such experiments would include comparisons with representative VLA baselines under the same hardware and control conditions. This would help establish whether the substantial computational advantages demonstrated in simulation translate into meaningful benefits during real-world execution.

  2. How does Fast-VLA handle significant environmental changes after the initial task memory has been constructed? For example, if a target object is unexpectedly moved during execution, can the online actor adapt reliably using only current visual feedback and the fixed task memory? Could the authors provide controlled experiments involving mid-episode perturbations, execution interruptions, or unexpected object movements to investigate this behavior?

  3. How sensitive is the proposed method to the number of actions executed between replanning steps? In particular, could the authors evaluate shorter execution horizons, such as 10 or 20 steps on RoboTwin, and report the corresponding task success rates and inference latency? This would help clarify whether the proposed efficiency advantages can be maintained in settings requiring more frequent feedback and responsive closed-loop control.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 2i2J

We thank the reviewer for the concrete experimental suggestions, which we followed. The General Response summarizes the new real-robot and dynamic-object results; here we report the key numbers and address the execution-horizon question.

Q1 / W1. "Could the authors provide real-world robotic experiments ... report both task success rates and actual inference latency ... comparisons with representative VLA baselines under the same hardware and control conditions."

Yes (General Response 1; §4.5, Table R3, Figures R1–R3, App. F.6). On an SO-101 arm with one RTX 3090, all methods are trained on the same 100 demonstrations, execute 50 actions per call, and are evaluated from the same 20 initial scenes, each reproduced with IoU > 0.8 for every object before the trial starts:

MethodLarge-model callsSuccessLatency per call (median)
π0.5every chunk15/20219 ms
DM0.5 (same VLM as Fast-VLA)every chunk13/20382 ms
Fast-VLAonce per episode15/2022 ms (first call 173 ms, incl. VLM prefill)

Fast-VLA matches π0.5 and exceeds DM0.5 by 10 points, with 10× and 17× lower per-call latency.

Q2 / W2. "If a target object is unexpectedly moved during execution, can the online actor adapt reliably using only current visual feedback and the fixed task memory?"

On the real robot we observed both the ball and the target box being moved during execution (App. F.6, Figure R3). When a failed grasp pushed the ball aside, Fast-VLA, with its memory still built from the first frame, re-grasped the ball from its current observations and completed the task; when the held ball pushed the target box, it released the ball into the box at the box's new position. Our 100 training demonstrations contain no such disturbances. In comparable trials, π0.5 did not re-grasp a pushed-away ball, and DM0.5 pushed the box away with an empty gripper; neither completed the task (Figure R3, rows C–D). In simulation (General Response 2), on DOMINO (35 RoboTwin tasks in which objects keep moving), Fast-VLA with memory fixed at the first frame reaches 34.2%, second only to the dynamics-specific DynamicWAM (38.2%) and well above PUMA (17.2%), π0.5 (9.6%) and OpenVLA-OFT (9.1%) (§4.4, Table R1). With another episode's memory in 10% of training samples, success with mismatched memory rises from 6.26%/8.36% to 82.90%/81.70% (§4.4, Table 4).

Q3 / W3. "How sensitive is the proposed method to the number of actions executed between replanning steps? ... evaluate shorter execution horizons, such as 10 or 20 steps on RoboTwin."

We re-evaluated the paper checkpoint on all 50 RoboTwin tasks (Clean and Randomized, 50 episodes per task and setting), predicting 50 actions and executing the first k before the next actor call, with memory still built once per episode (§4.4, Table R2, App. F.3):

kCleanRand.MeanActor calls / episodeAmortized ms/step (A100)
1093.8092.5693.18~255.01
2094.4093.7294.06~122.80
50 (paper setting)93.2892.3692.82~51.46

Mean success stays within 92.82–94.06% across horizons, so Fast-VLA can re-observe the scene as often as every 10 steps without refreshing its memory or losing success. Because the VLM is not re-run, shorter horizons only add actor calls: at k = 10 the cost is 5.01 ms/step, still 3.2× lower than DM0.5 at its 50-step schedule (16.069 ms/step, Table 2).

The revised PDF marks all changes in blue. We hope these results address the reviewer's main concerns and would be grateful if the reviewer would reconsider the score.

Reviewer_8nDQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Fast-VLA is a two-stage VLA architecture that decouples task understanding from closed-loop control. A frozen 4B VLM (DM0.5) encodes the instruction and initial multi-view frames into fixed, layer-wise KV task memory once per episode, then exits the control loop entirely. During execution, a lightweight (~272M) online actor fuses learned time/arm-indexed queries with current visual and proprioceptive feedback, retrieves task context from the fixed memory, and predicts 50-step action chunks in a single forward pass. On RoboTwin 2.0 it achieves 92.46%/92.40% success (Clean/Randomized) with 1.437 ms/step amortized inference—10.9–17.1× faster than comparable WAM/VLA baselines at 1.22 GiB online resident memory—and 95.95%/79.11% on LIBERO/LIBERO-Plus. Memory-swap interventions confirm the actor genuinely depends on episode-matched task memory.

Critical strengths
  1. Clear and well-motivated thesis. The paper identifies a genuine inefficiency—re-running a multi-billion-parameter VLM for every action chunk when task context barely changes—and frames it around two natural timescales (stable task understanding vs. fast control), which is intuitive and persuasive.
  2. Significant efficiency gains with minimal performance cost. 10.9–17.1× amortized speedup and ~10× memory reduction over strong baselines, while staying within ~1 point of DM0.5 on RoboTwin and exceeding Faster-WAM variants on LIBERO-Plus—an excellent speed/accuracy trade-off.
  3. Thorough empirical validation of the core claim. The memory-swap experiments (same-task different-episode memory collapses to ~6–8% success) are a clean, convincing test that the actor actually uses the episode-matched memory rather than ignoring it.
Critical weaknesses
  1. Baseline comparison is not apples-to-apples, weakening the efficiency claim. The comparison mixes two method families with fundamentally different inference profiles: CoT-style VLAs (π₀, π₀.5, DM0.5) that spend compute on per-step deliberation, and WAM-style models that pay heavy costs on test-time future imagination. Fast-VLA, however, is architecturally a π-type VLA (frozen VLM + lightweight action head, one-shot chunk prediction) and is naturally fast—so much of the reported 10.9–17.1× speedup reflects this architectural asymmetry rather than the "think once" idea itself. More importantly, against the most directly comparable π-type baselines, the advantage largely evaporates: Fast-VLA amortizes to 1.437 ms/step while π₀ achieves 1.580 and π₀.5 1.758 ms/step—only a 1.10–1.22× speedup that is arguably within profiling noise, at substantially lower success rates (62.16%/79.75% vs. 92.43%). A more convincing evaluation would compare against π-type baselines under identical action-chunking and replanning schedules.
  2. Cost comparison rests on a single benchmark and unaligned training. Same-hardware latency is only measured on RoboTwin 2.0; LIBERO cost numbers come from heterogeneous sources. Meanwhile, training configurations are not aligned with baselines—Fast-VLA trains for 200k steps with a 256 batch on RoboTwin (~150 GPU-hours) and 30k steps on LIBERO, while baseline compute budgets are uncontrolled. Without matched training-compute or training-step fair comparisons, it is difficult to verify whether the gains come from the method or from different optimization investments, and the experimental comprehensiveness suffers as a result.
  3. The think-once assumption limits generalization to dynamic, long-horizon settings. Fixing task memory at episode start assumes the task goal and the relevant scene context remain stationary. The paper validates robustness only against static perturbations of the initial scene (lighting, texture, layout, viewpoint), where nothing substantively changes after memory construction. Scenarios involving object displacement mid-episode, multi-stage goals that require re-planning, or newly introduced instructions are explicitly out of scope—and the refresh-frequency ablation (Table 8), while supporting the efficiency claim, also shows performance degrades when memory is refreshed, suggesting the actor cannot easily recover from stale context. This raises a fair concern about generalization on long-horizon, dynamically changing tasks.
  4. No real-robot validation. All results are in simulation (RoboTwin 2.0, LIBERO, LIBERO-Plus). For a paper whose central contribution is inference efficiency, the absence of real-robot deployment leaves open whether the amortized-latency advantages survive real-system bottlenecks (sensor frequency, communication overhead, actuation limits), and whether fixed-memory control remains safe when physical contact dynamics deviate from simulation.
Questions for authors

See Weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 8nDQ

We thank the reviewer for the detailed feedback. The General Response summarizes the shared results; we address each concern below.

W1. "Much of the reported speedup reflects this architectural asymmetry rather than the 'think once' idea ... compare under identical action-chunking and replanning schedules."

The key difference is how often the VLM runs. π0 and π0.5 re-run their 3B VLM at every action call; Fast-VLA runs its 4B VLM once per episode, and every later call uses only the 272M actor. The π0/π0.5 rows in Table 2 already use the same 50/50 schedule as Fast-VLA, which is why their amortized gap looks small: with ~237-step episodes, each method is called only about five times per episode. At this schedule, Fast-VLA reaches 92.43% versus 62.16% and 79.75%, with 1.6–1.8× lower per-call time (44.87 vs. 72.38/80.51 ms) and 5.1–5.3× lower online memory (1.22 vs. 6.26/6.47 GiB). On the real robot, with the same 50-action schedule on one RTX 3090, Fast-VLA matches π0.5 in success (15/20 each) at 10× lower per-call latency (22 vs. 219 ms; General Response 1, Table R3).

The advantage grows with more frequent replanning. On LIBERO, where π0 and π0.5 use their official 50/5 and 10/5 schedules, amortized time is 16.577 and 17.466 ms/step versus 3.975 for Fast-VLA (4.2–4.4×, Table 3). On RoboTwin, executing only 10 of the 50 predicted actions keeps 93.18% at 5.01 ms/step, still 3.2× faster than DM0.5 at 50/50 (General Response 3; Table R2).

Comparisons that isolate "think once".

  • Same VLM, same 50/50 schedule: DM0.5 re-runs its 4B model every chunk; Fast-VLA calls the same VLM once. Fast-VLA is 11.2× faster (1.437 vs. 16.069 ms/step) at 1.02 points lower mean success (Table 2).
  • Same actor and memory, matched training, only the refresh schedule changes (Table 8): refreshing every chunk costs 3.12× the amortized time for +1.80/−1.23 points on LIBERO/LIBERO-Plus.
  • Same VLM on a real robot: Fast-VLA 15/20 at 22 ms per call versus DM0.5 13/20 at 382 ms (General Response 1; Table R3).

W2a. "Same-hardware latency is only measured on RoboTwin 2.0; LIBERO cost numbers come from heterogeneous sources."

All cost columns in Table 3 (ms/chunk, amortized ms/step, GiB) were measured by us on the same A100 under the same protocol as Table 2 (Section 4 "Cost measurement", Appendix B, Table 11); only success rates marked ♭ come from compilations. The real-robot study adds on-robot latency for all methods on one RTX 3090.

W2b. "Training configurations are not aligned with baselines." Published downstream settings on RoboTwin (App. F.5, Table R4):

MethodAction pretrainingFine-tuning (batch × steps)
Fast-VLAnone256 × 200k
DM0.5yes (not disclosed)1024 × 100k
LingBot-VLA 2.0~60k h robot + human data1024 × 50k
Faster-WAM (Noah)none (video-model init)512 × 58.7k
X-VLA290k episodes128 × 60k

Our downstream budget (51.2M samples) is half of DM0.5's official recipe and equal to LingBot-VLA 2.0's, without action pretraining. Most baselines do not report downstream GPU-hours, so we also ran same-data comparisons: DOMINO (10,500 demonstrations) and the real robot (100 demonstrations).

W3. "The think-once assumption limits generalization to dynamic, long-horizon settings ... Table 8 ... shows performance degrades when memory is refreshed."

Moving objects. On the real robot, Fast-VLA still completed trials in which the ball or the target box was pushed, while π0.5 and DM0.5 did not in comparable trials (Figure R3). On DOMINO, Fast-VLA with memory fixed at the first frame and its actor trained from scratch reaches 34.2%, second only to the dynamics-specific DynamicWAM (38.2%) and well above PUMA (17.2%) and π0.5 (9.6%) (General Response 2; Table R1).

Mismatched memory. The Table 4 collapse reflects the training distribution: with another episode's memory in 10% of training samples, other-episode success rises from 6.26%/8.36% to 82.90%/81.70% (§4.4, Table 4).

Reading Table 8. Table 8 compares separately trained models, each given fresh memory at every refresh, so it does not measure how the actor handles scene changes. Relative to building memory once per episode, refreshing every chunk gives +1.80 points on LIBERO, while every refresh schedule lowers LIBERO-Plus by 1.23–5.46 points and increases inference cost.

Long horizon. Fast-VLA keeps 90.25% on the long-horizon group (901–1700 steps), the highest in Table 7; many RoboTwin tasks are multi-stage and bimanual (Figure 4 shows attention moving between sub-goals).

W4. "No real-robot validation." See General Response 1 (§4.5, Table R3, Figures R1–R3, App. F.6).

The revised PDF marks all changes in blue. We hope this clarifies that the efficiency comes from the once-per-episode VLM call and that fixed memory remains competitive when objects move.

Reviewer_5TaUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Fast-VLA, an efficient vision-language-action architecture that separates task understanding from closed-loop action generation. A frozen 4B VLM processes the language instruction and initial observations once per episode, storing multi-layer KV features as task memory. A lightweight online actor then uses current visual observations and proprioceptive states to retrieve relevant information from this memory and predict action chunks without repeatedly executing the VLM.

The method achieves 92.43% average success on RoboTwin 2.0, 95.95% on LIBERO, and 79.11% on LIBERO-Plus. It substantially reduces inference latency compared with several existing VLA and WAM baselines. The paper also provides ablations on task memory, visual inputs, and memory refresh frequency to validate the proposed architecture.

Critical strengths
  1. An effective architecture for efficient VLA inference. Separating high-level task conditioning from frequent low-level control is a reasonable and practically important direction. The proposed memory-based architecture achieves a substantial reduction in inference cost while maintaining competitive manipulation performance. This is particularly valuable for deploying generalist robot policies on compute-constrained platforms.

  2. Strong empirical results with useful architectural ablations. The experiments cover RoboTwin 2.0, LIBERO, and LIBERO-Plus, with same-hardware inference measurements. The reported 10.9–17.1× speedup on RoboTwin is impressive. The memory-swap and refresh-frequency experiments also provide useful evidence about the contribution of task memory and the feasibility of reusing it throughout an episode.

Critical weaknesses
  1. The generality of fixed episode-level memory remains unclear. The proposed method assumes that task context can be established from the initial observations and remain unchanged throughout execution. However, practical manipulation frequently involves changes in object configurations, unexpected interactions, and new subgoals. Interestingly, Table 4 shows that replacing the memory with one from another episode of the same task drastically reduces success. This suggests that the memory encodes substantial episode-specific scene information rather than only high-level task semantics. It is unclear how well this design generalizes to more dynamic environments or longer-horizon tasks requiring replanning.

  2. The practical efficiency claims need more complete deployment validation. Although the amortized inference latency and online memory footprint are impressive, the reported low resident memory excludes the VLM initialization peak, which is approximately 12 GiB. More importantly, the experiments are entirely simulation-based. It remains unclear whether the reported advantages translate to real robots operating with dynamic observations, tighter feedback requirements, and practical memory-management overhead. The absence of a simpler initial-scene memory baseline also makes it difficult to isolate how much of the benefit comes specifically from the pretrained VLM representations.

Questions for authors
  1. Can the authors evaluate the method under unexpected scene changes, object disturbances, or changes in task instructions after the initial VLM call? How would the performance compare with periodically refreshing task memory?

  2. How much does the pretrained VLM contribute beyond providing initial-scene features? For example, could a lightweight visual encoder combined with language embeddings achieve similar performance using the same online actor?

  3. Can the authors clarify the end-to-end deployment cost, including VLM initialization, memory offloading, and peak GPU memory? It would also be valuable to understand the achievable closed-loop control frequency on practical robot hardware.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 5TaU

We thank the reviewer for the positive assessment and the questions. The General Response summarizes the shared results; below we address the remaining points.

W1 / Q1. Behavior under scene changes, and comparison with periodic refresh.

(a) Objects moving after the memory is built. On the real robot (Figure R3), Fast-VLA completed the task both in a trial where a failed grasp pushed the ball aside and in one where the held ball pushed the target box, whereas π0.5 and DM0.5 did not complete comparable trials (General Response 2). On DOMINO, where objects keep moving, Fast-VLA with memory fixed at the first frame reaches 34.2%, second only to the dynamics-specific DynamicWAM (38.2%) (Table R1).

(b) The Table 4 collapse. The memory is built from the instruction and the initial views, so it contains episode-specific grounding by design. The collapse reflects the training distribution, in which the memory always matches the episode: with another episode's memory in 10% of training samples, other-episode success rises from 6.26%/8.36% to 82.90%/81.70% (Clean/Rand.), with own-memory success at 92.32%/90.79% (General Response 2; §4.4, Table 4).

(c) Periodic refresh. Table 8 compares refresh schedules, each with matched training. Refreshing every chunk changes success by +1.80 (LIBERO) and −1.23 (LIBERO-Plus) points while increasing amortized time 3.12×; refreshing every 2 or 4 chunks lowers LIBERO-Plus by 3.07 and 5.46 points. Re-reading the scene more often needs only the actor: executing 10 of the 50 predicted actions per call keeps 93.18% on RoboTwin at 5.01 ms/step (General Response 3; Table R2).

(d) Instruction changes. A new instruction defines a new task context. A natural event-triggered extension is to re-run the prefill once on the new instruction and current views (108.87 ms on A100) and continue with the same actor, so the cost scales with the number of instruction changes rather than the number of chunks. We keep this in the limitations (Appendix E).

W1. Longer-horizon tasks. Table 7 groups all 50 RoboTwin tasks by step budget: Fast-VLA keeps 90.25% on the long group (901–1700 steps), the highest among the listed methods, and 87% on put_bottles_dustbin, which uses up to 34 actor calls on one memory.

W2 / Q2. "How much does the pretrained VLM contribute beyond providing initial-scene features?"

We replaced the 4B VLM with a frozen SigLIP2-so400m encoder: its image features for the two initial views and its text features for the instruction are mapped by a learned per-layer linear projection to the same KV format; the actor and recipe are unchanged (Table 9, App. F.4).

Task memory (same actor)LIBEROLIBERO-Plus
None (constant zero KV, Table 5)69.9053.69
SigLIP2 image + text features (new)91.9571.53
PaliGemma v1 (Table 9)95.3573.15
DM0.5 (ours)95.9579.11

The lightweight memory recovers most of the standard-LIBERO success but is 4.0 points below DM0.5 there and 7.6 points below on LIBERO-Plus. PaliGemma, whose memory has the same 9.8-MB size, reaches 95.35/73.15, so the gap is not explained by memory size: the pretrained VLM memory helps most under perturbations.

W2 / Q3. End-to-end deployment cost.

  • RoboTwin (A100, Table 2 note and Appendix C): the first chunk takes 153.74 ms (108.87 ms VLM prefill + 44.87 ms actor) and each later chunk 44.87 ms; peak memory during the prefill is 12.11 GiB.
  • After the prefill, the VLM leaves the control loop. If it is offloaded, online resident memory is 1.22 GiB, of which the reusable task memory is 174.6 MiB; if the GPU has room for the 12.11-GiB peak, the VLM can stay resident with no transfer cost. Loading the weights from disk is a one-time start-up cost.
  • Control frequency: the actor alone supports about 22 calls per second on A100.
  • Real robot (RTX 3090): each call after the first takes 22 ms (173 ms including the prefill), versus 219 ms for π0.5 and 382 ms for DM0.5 (General Response 1; §4.5, Table R3).

W2. Real-robot validation. See General Response 1: on an SO-101 arm, with all methods trained on the same 100 demonstrations and evaluated from the same 20 IoU-checked initial scenes, Fast-VLA reaches 15/20, matching π0.5 (15/20) and above DM0.5 (13/20), which uses the same VLM (§4.5, Table R3, Figures R1–R3, App. F.6).

The revised PDF marks all changes in blue. We hope these results address the reviewer's concerns.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response and Summary of Revisions (1/2)

We thank all reviewers for their careful reading and constructive feedback. We appreciate the recognition that separating episode-level task understanding from fast closed-loop control is well motivated [5TaU, 8nDQ, 2i2J], that the efficiency gains come at little cost in success [8nDQ, 2i2J], and that the memory-swap experiments are a clean test of how the actor uses task memory [8nDQ, 2i2J].

The main shared questions concerned real-robot deployment, scene changes after the memory is built, and how often the policy re-reads the scene. We summarize the new results below; details are in the individual replies. The revised PDF marks all changes in blue; new tables and figures are numbered R1, R2, …, and all original numbers are unchanged.

1. Real robot [5TaU, 8nDQ, 2i2J] (§4.5, Table R3, Figures R1–R3, App. F.6). On an SO-101 arm with one RTX 3090, all methods are fine-tuned on the same 100 demonstrations and evaluated from the same 20 initial scenes, each reproduced with IoU > 0.8 for every object before the trial starts. Fast-VLA reaches 15/20, matching π0.5 (15/20) and above DM0.5 (13/20), which uses the same VLM, while each call after the first takes 22 ms versus 219 ms and 382 ms.

Table R3: Real robot (SO-101, RTX 3090), 20 trials per method; latency is the median wall time per policy call.

MethodLarge-model callsSuccessLatency per call
π0.5every chunk15/20219 ms
DM0.5 (same VLM)every chunk13/20382 ms
Fast-VLAonce per episode15/2022 ms (first call 173 ms)

2. Scene changes after memory construction [5TaU, 8nDQ, 2i2J].

  • Real robot (App. F.6, Figure R3). Contact during grasping sometimes moves an object after the memory is built, and Figure R3 shows Fast-VLA handling both cases: when a failed grasp pushed the ball aside, it re-grasped the ball and completed the task; when the held ball pushed the target box, it released the ball into the box at the box's new position. Our 100 demonstrations contain no such disturbances. In comparable trials, π0.5 did not re-grasp a pushed-away ball, and DM0.5 pushed the box away with an empty gripper and did not complete the task.
  • Moving objects in simulation (§4.4, Table R1, App. F.1). On DOMINO (35 RoboTwin tasks with continuously moving objects), Fast-VLA with memory fixed at the first frame and its actor trained from scratch reaches 34.2%, second only to the dynamics-specific DynamicWAM (38.2%) and above PUMA (17.2%), π0.5 (9.6%) and OpenVLA-OFT (9.1%).

Table R1: DOMINO level 1 success (%), 35 tasks, same training data.

MethodSuccess
OpenVLA-OFT9.1
π0.59.6
PUMA17.2
DynamicWAM38.2
Fast-VLA (from scratch)34.2
  • Mismatched memory (§4.4, Table 4, App. F.2). The collapse in Table 4 raised the question of whether fixed memory is an architectural limitation. We hypothesized that it instead stems from the training distribution, in which the memory always matches the episode. To test this, we retrained the actor with another episode's memory in 10% of training samples: success with such memory rises from 6.26%/8.36% to 82.90%/81.70% (Clean/Rand.), with own-memory success at 92.32%/90.79%. These results indicate that sensitivity to episode-mismatched memory depends strongly on the training distribution.

Table 4: RoboTwin 2.0 task-memory swaps (%).

Memory conditionCleanRand.
Swapped language segment35.1829.66
Same task, different episode6.268.36
Swapped initial-view segment2.642.62
Different task0.620.52
Original memory92.4692.40
Swap-aware training (p = 0.1): same task, different episode82.9081.70
Swap-aware training (p = 0.1): original memory92.3290.79

3. Re-reading the scene more often [2i2J, 8nDQ] (§4.4, Table R2, App. F.3). Executing 10 or 20 of the 50 predicted actions per call gives 93.18% and 94.06% on RoboTwin (92.82% at 50). Because the VLM is not re-run, k = 10 costs 5.01 ms/step, still 3.2× lower than DM0.5 at its 50-step schedule.

Table R2: RoboTwin 2.0 success (%) when executing the first k of 50 predicted actions per call (default k = 50); memory built once.

kCleanRand.MeanCalls/ep.ms/step
1093.8092.5693.18~255.01
2094.4093.7294.06~122.80
5093.2892.3692.82~51.46

4. Value of VLM memory [5TaU] (Table 9, App. F.4). A lightweight SigLIP2 memory reaches 91.95%/71.53% on LIBERO/LIBERO-Plus, below every VLM memory including PaliGemma at the same 9.8-MB size, with the largest gap to DM0.5 on LIBERO-Plus.

Task memory (same actor)SizeLIBEROLIBERO-Plus
None–69.9053.69
SigLIP2 + linear KV9.8 MB91.9571.53
PaliGemma v19.8 MB95.3573.15
DM0.5 (ours)79.7 MB95.9579.11
AuthorsOpenReview ↗

General Response and Summary of Revisions (2/2)

5. Training budget [8nDQ] (App. F.5, Table R4). Our downstream budget is half of DM0.5's official recipe and equal to LingBot-VLA 2.0's, without action pretraining.

MethodAction pretrainingBatch × steps
Fast-VLAnone256 × 200k
DM0.5yes (not disclosed)1024 × 100k
LingBot-VLA 2.0~60k h1024 × 50k

Summary of revisions in the PDF (changes in blue; original numbering unchanged):

  • §4.4: new rows in Table 4 (swap-aware training) and Table 9 (SigLIP2 memory); new Table R1 (DOMINO) and Table R2 (execution horizon), with the corresponding text.
  • §4.5 (new): real-robot evaluation, Table R3 and Figure R1.
  • Appendix F (new): protocols for DOMINO (F.1), swap-aware training (F.2), execution horizon (F.3) and SigLIP2 memory (F.4); training budgets (F.5, Table R4); real-robot setup, scene-reset check, and examples of objects moved during execution for Fast-VLA, π0.5 and DM0.5 (F.6, Figures R2–R3).
  • Conclusion: one sentence on swap-aware training and the real-robot result.

We thank the reviewers again and welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.