acceptodds
Under review as a conference paper at ICLR 2027

LA4VLA: Learning to Act without Seeing via Language-Action Pretraining

14 views

Abstract

Vision-Language-Action (VLA) models are commonly pretrained on robot demonstrations by jointly mapping visual observations and language instructions to actions. However, dense visual-action supervision can dominate the comparatively sparse language-action signal. As a result, policies may rely on visual shortcuts rather than learn how language conditions action execution, making them sensitive to visual variations. To address this limitation, we propose LA4VLA, a language-action pretraining framework that enables policies to acquire language-conditioned action priors without visual observations. These priors capture reusable manipulation skills shared across tasks and scenes, reducing reliance on scene-specific visual cues. Specifically, LA4VLA decomposes expert demonstration trajectories into atomic action segments and pairs each segment with a corresponding low-level action description. This yields LA-33K, a dataset of 33K Language-Action (LA) episodes derived entirely from existing demonstrations without additional robot data collection. We further develop LA4VLA-1B, a lightweight 1B-parameter VLA model, and investigate three paradigms for incorporating language-action supervision into VLA learning: LA-only pretraining, sequential LA-to-VLA pretraining, and mixed LA-VLA pretraining. Across simulation and real-world tasks, LA-pretrained policies consistently outperform matched VLA-pretrained counterparts, while combining LA and VLA supervision leads to further gains. In particular, mixed LA-VLA pretraining improves the average success rate of LA4VLA-1B over the no-pretraining baseline by up to 17.8 and 45.0 percentage points in simulation and real-world tasks, respectively. These results establish LA4VLA as an effective and complementary pretraining strategy for building stronger and more robust VLA policies. We release our code, data, and model weights to facilitate future research on language-action pretraining for VLA models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLA, Embodied AI

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_nG3UOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether language-action pretraining without visual observations can improve downstream VLA policies. The authors segment DROID demonstrations into atomic action episodes and combine VLM-generated annotations with human verification to construct LA-33K. They compare LA-only, VLA-only, sequential LA-to-VLA, and mixed pretraining across simulation benchmarks, visual perturbations, a second VLA architecture, and real-world robot tasks. The main value of the work lies in its construction of atomic language-action data and its systematic empirical comparison of pretraining strategies using paired data and matched training budgets.

Critical strengths
  1. LA and VLA pretraining use the same atomic action segments and the same total number of optimization steps, helping isolate the effect of visual conditioning during pretraining. The appendix also includes a comparison against pretraining on the original, unsegmented DROID demonstrations.
  2. Relative to matched VLA pretraining, LA improves success rates by 3.22, 0.90, and 10.01 percentage points on MetaWorld, LIBERO, and the evaluated LIBERO-Plus conditions, respectively. Average real-world task success increases from 48.3% to 81.7%. These results motivate further investigation, particularly for real-world manipulation and performance under visual perturbations.
  3. The paper describes the action taxonomy, keyframe detection, automatic segmentation, and human filtering procedures, and reports inter-annotator agreement. The real-world tasks also require language use by assigning different goals to the same initial scene.
Critical weaknesses
  1. Section 3 and Appendix F provide evidence that the LA-pretrained model can follow directional instructions without visual input. However, could masking visual input introduce a distribution shift for the VLA-trained model, whereas this condition is already encountered during LA pretraining? If so, to what extent might the diagnostic differences reflect this mismatch rather than differences in language-action grounding?
  2. The conflicting-state experiment provides useful supporting evidence. Would a more informative additional experiment be to evaluate the policies after downstream finetuning, holding visual observations and robot states fixed while varying the language instructions, and comparing whether the different pretraining schemes reliably produce the corresponding actions?
Questions for authors
  1. Could you correct the description of the relationship to Qwen-VLA and clarify the additional empirical insights established by the atomic dataset and paired comparisons?
  2. After completing downstream visual fine-tuning, when the language instruction is varied while keeping the image and robot state constant, does a model initialized with LA perform the corresponding action more reliably than one initialized with VLA?
  3. Since VLA models are trained using images but have those images abruptly removed during testing, does this create a discrepancy between training and testing conditions? If so, could the observed performance decline be partly due to an inability to adapt to the missing modality, rather than serving as definitive proof that the model failed to learn the language-action associations?
5 comments
AuthorsOpenReview ↗

Response to the Reviewer

We thank the reviewer for highlighting our controlled pretraining comparisons and the design of our real-world tasks, which use language to specify different goals in the same scene. Below, we address each concern in turn and clarify our experimental design and findings.

R1.Clarifying the visual-intervention diagnostic (W1).

Same-scene controls for distribution shift. We test two conditions that retain the image-input configuration used during VLA training. Visual-unaligned uses an image from the same scene that is unaligned with the current instruction, and Visual-conflict uses an image from a trajectory in a similar scene associated with the opposite motion direction. The language instruction and robot state remain fixed (Section 3.1). Using images from the same or similar scenes limits distribution shift due to unrelated scene differences, so the comparison focuses on how changing the visual–instruction correspondence affects the predicted action. With the original paired observation, the policy achieves DAR 0.98 and DCS 0.95. Replacing it with an unaligned same-scene image reduces these values to 0.66 and 0.37; under conflicting visual input, they further decrease to 0.35 and 0.03 (Table 1). These results show that even under comparable visual input distributions, with the instruction and robot state held fixed, mismatched visual cues can pull the VLA policy’s predictions away from the instructed direction.

Behavioral and representational evidence of language–action grounding. In our real-world tasks, we vary the language-specified target while holding the initial scene and robot state fixed. After the same visual finetuning protocol, the LA-pretrained policy achieves 81.7% average success compared with 48.3% for the VLA-pretrained policy (Table 3). This shows that the final visual policy follows language-specified goals more reliably after LA pretraining. The t-SNE plots in Figure 9 (right panels, Appendix F) show that the inputs to the action head cluster more clearly by instructed motion direction after LA pretraining, providing qualitative evidence that the policy learns to associate language instructions with the actions they specify. Together, the real-world results and t-SNE visualizations provide behavioral and representational evidence that LA pretraining strengthens language–action grounding.

R2.Language-dependent behavior after downstream visual finetuning (W2 and Q2).

As the reviewer noted among the strengths, our real-world tasks assign different language-specified goals to a shared initial scene. This design closely matches the central requirement of the suggested evaluation: after downstream visual finetuning, different instructions share the same initial visual scene and robot states, while requiring different target actions.

The policy must select among four buttons, three shelf positions, or four evaluated grid cells according to language. The initial image and state alone cannot identify the intended target, and all policies receive full visual input during execution. LA improves success over matched VLA pretraining from 50.0% to 85.0% on Press Button, from 40.0% to 65.0% on Place Book, and from 55.0% to 95.0% on Place Drink, increasing the average from 48.3% to 81.7% (Section 5.5, Table 3; setup in Appendix D).

These results show that, after visual finetuning, LA-pretrained policies perform more reliably on tasks where language specifies different goals under shared initial visual scenes and robot states.

R3. Relationship to Qwen-VLA (Q1).

We will correct and expand the comparison in Section 2 to clarify the shared use of vision-free action pretraining and the differences in dataset construction and empirical focus. Qwen-VLA pretrains its action decoder without images, using task instructions and robot-specific control prompts. Our work focuses on constructing LA-33K, a dataset of vision-agnostic atomic instruction–action pairs derived from existing robot demonstrations, and systematically evaluating this supervision through paired experiments with matched training budgets. On the same atomic episodes, LA-only pretraining improves downstream performance over VLA-only pretraining, and combining LA and VLA pretraining yields further gains.

AuthorsOpenReview ↗

Response to the Reviewer

R4.Visual inputs in the diagnostic and downstream evaluation (Q3).

In both simulation and real-world experiments, all downstream policies receive visual observations during finetuning and evaluation. Black-image substitution is used only in the Visual-removed condition of the diagnostic, where it changes the visual content at test time while preserving the image-input structure. As shown in R1, directional alignment also drops in Visual-unaligned and Visual-conflict, where the policy receives images from the same or similar scenes and the instruction and robot state remain fixed (Table 1). As shown in R2, with the initial scene and robot state held fixed, the LA-pretrained policy more reliably completes real-world tasks in which language specifies the target (81.7% vs. 48.3% for matched VLA pretraining, Table 3). Together, R1 and R2 show that limitations in the VLA policy’s language–action grounding persist even when visual input is available in the same format during training and testing.

AuthorsOpenReview ↗

We look forward to hearing from you

As the discussion period is drawing to a close, we would greatly appreciate any further feedback you may have on our responses. We would be happy to provide additional clarification and address any remaining questions or concerns. If our responses have adequately addressed your concerns, we would be grateful if you could consider raising your score. Thank you for your time and consideration.

Reviewer_oSVgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem that conventional vision-language-action (VLA) models may rely excessively on visual inputs during joint training and fail to adequately learn the correspondence between language and actions. It proposes LA4VLA, a VLA framework based on vision-free language-action pretraining.

Experiments are conducted on MetaWorld, LIBERO, LIBERO-Plus, and real-world robot manipulation tasks. The results show that, under the same data and training conditions, LA pretraining without visual input achieves higher average success rates on all three simulation benchmarks than VLA pretraining with visual input. Further combining LA and VLA data through sequential or mixed pretraining can lead to higher downstream task success rates.

Critical strengths

The construction of atomic-level language-action data and the pretraining strategy have practical value. Based on existing DROID robot demonstrations, the authors construct a dataset of 33,116 atomic action segments using robot-state keyframe detection, VLM-based temporal segmentation, and human-quality screening. On this basis, the authors systematically compare vision-free LA pretraining, VLA pretraining, and strategies that combine the two, providing an experimental basis for investigating the role of language-action supervision in VLA pretraining.

The experimental comparisons are relatively comprehensive and include cross-task and real-world robot validation. The authors construct LA and VLA training samples from the same atomic-action data and control the model architecture, training budget, and downstream finetuning protocol, making the comparison between training with and without visual input reasonably fair. The experiments cover MetaWorld, LIBERO, LIBERO-Plus, and real-world robot manipulation tasks, with additional validation on the StarVLA architecture. LA pretraining achieves consistent improvements in average performance across multiple benchmarks, particularly on LIBERO-Plus and real-world robot tasks, supporting the effectiveness and transferability of the method.

Critical weaknesses
  1. It remains unclear how well atomic-action language constrains action scale and how well it generalizes. The atomic language descriptions in LA-33K mainly specify action categories, motion directions, and object-holding states, while deliberately downplaying object identity and scene information. This design helps the model learn motion patterns that can be reused across tasks, but it may also prevent language conditions from sufficiently distinguishing different action execution requirements.

  2. The atomic-action descriptions downplay high-level task goals and object information. Although the existing downstream experiments demonstrate skill transfer, they do not sufficiently establish whether this pretraining generalizes to complex language instructions and novel combinations of actions. Therefore, further validation is needed to determine the extent to which atomic-level language supervision can learn action control capabilities and which information must be supplemented through subsequent visual finetuning.

Questions for authors
  1. How do atomic language instructions distinguish different action scales?

  2. For training samples with identical atomic language instructions and similar initial robot states but different actual action magnitudes, how does the model determine which action to generate?

  3. Does MixPT differ in effectiveness from random visual masking during training?

  4. How much do keyframe cues actually contribute to atomic-action dataset construction?

4 comments
AuthorsOpenReview ↗

Response to the Reviewer

We thank the reviewer for recognizing the value of LA-33K and the controlled pretraining comparisons. Below, we address each concern in turn and provide additional experimental results and clarifications.

R1.How atomic instructions constrain action scale (W1 and Q1).

Atomic instructions primarily constrain action type, motion direction, and interaction state. During LA pretraining, complete continuous action sequences provide supervision for the demonstrated magnitudes. During downstream visual finetuning, object and target locations provide additional context for adapting execution scales to the scene.

In the Objects Layout evaluation on LIBERO-Plus, changes in object positions require the policy to adjust its movement distances. Under the same visual finetuning protocol, LA pretraining improves success from 59.94% without pretraining to 81.41% (Table 2), providing task-level evidence that the policy can adapt to changes in object layout. We use directional commands in the diagnostic study because their explicit motion directions make instruction following easy to quantify and visualize.

R2.Transfer to complex tasks and the role of downstream visual finetuning (W2).

The high-level goals and object information omitted from atomic LA descriptions are introduced during downstream visual finetuning. LA pretraining learns how language conditions local action execution, using complete continuous action trajectories as supervision. Downstream finetuning then uses complete task demonstrations: task-level instructions specify the overall goal, while visual observations identify the relevant objects, target locations, and current scene configuration. Together with robot states, these inputs provide the context needed to adapt local action priors to specific execution requirements and coordinate actions across task stages. This training process allows atomic language-action supervision to support complex task learning, with task-specific grounding and multi-stage behavior learned during downstream finetuning.

The long-horizon results support this transfer. On LIBERO-Long, LA improves success from 88.2% with matched VLA pretraining to 92.0%, while LA-VLA and MixPT both reach 93.4% (Table 2). Our evaluated LIBERO-Plus conditions are also based on LIBERO-10, the long-horizon suite, under layout, background, sensor-noise, and camera-viewpoint changes. LA improves average success from 59.01% to 69.02%, and MixPT further reaches 70.40%.

These results demonstrate that the benefits of atomic supervision extend to multi-stage long-horizon manipulation and persist under visual and scene variations. 

R3.Action generation with varying demonstrated magnitudes (Q2).

During LA pretraining, the action head learns a distribution over complete action sequences conditioned on the instruction and robot state, which accommodates variation in demonstrated magnitudes. When the instruction and robot state leave the action magnitude ambiguous, the model generates a plausible trajectory from this learned conditional distribution. During downstream visual finetuning, camera observations provide information about object positions and obstacles. The policy learns to use this spatial information to locate the instructed target and determine how far to move in the current scene.

AuthorsOpenReview ↗

Response to the Reviewer

R4.Relationship between MixPT and random visual masking (Q3).

On our paired atomic data, random visual masking can produce the same types of training examples used in MixPT. However, whether those examples retain explicit action supervision depends on the instructions. LA-33K’s atomic instructions specify the demonstrated local actions independently of the images, preserving explicit language–action supervision when visual input is masked. In the original VLA demonstrations, task-level prompts describe the overall goal and provide limited supervision for individual demonstrated actions on their own. Masking images during pretraining therefore leaves those actions underspecified by language, which can weaken the training signal and hurt pretraining performance.

To compare their effectiveness, we added a DROID-VLA + random visual masking experiment. We applied masking during pretraining on the original DROID demonstrations using the same budget as DROID-VLA. We set the per-example visual masking probability to 0.5 to match the proportion of examples without images in MixPT’s 1:1 mixture of LA and VLA examples. We then finetuned the model on MetaWorld with full visual input under the same downstream protocol as the DROID-VLA baseline (Appendix C.3).

PretrainingEasyMediumHardVery HardAverage
DROID-VLA (Table 14)88.274.571.786.080.11
DROID-VLA + visual-input dropout (new)90.772.768.372.075.9
LA-only (Table 14)93.281.865.092.083.00
MixPT (Table 2)88.994.566.7100.087.53

Random visual masking on the original DROID data lowered average success by 4.2 percentage points. MixPT achieved 87.53% with paired atomic data. These results support the role of language–action (LA) pretraining in MixPT. LA-33K pairs vision-agnostic atomic instructions with demonstrated actions, giving the policy explicit supervision to learn language–action associations when images are withheld.

R5.Role of keyframe cues in atomic segmentation (Q4).

We use keyframe cues as practical aids to help the VLM locate the start and end of short atomic actions. These cues help the VLM narrow down candidate segment boundaries, especially around grasp and release events and transitions between motion phases. We derive these cues from robot-state trajectories using rules (Appendix A.1). Gripper transitions identify candidate grasp or release phases, while low-motion intervals provide cues around alignment, placing, or phase transitions. The VLM uses these timestamps as soft hints alongside video observations, task context, and the action vocabulary to determine semantically appropriate segments. Human verification subsequently checks the consistency of temporal boundaries, action labels, and language descriptions and filters low-quality proposals.

AuthorsOpenReview ↗

We look forward to hearing from you

As the discussion period is drawing to a close, we would greatly appreciate any further feedback you may have on our responses. We would be happy to provide additional clarification and address any remaining questions or concerns. If our responses have adequately addressed your concerns, we would be grateful if you could consider raising your score. Thank you for your time and consideration.

Based on the authors’ response and additional experimental results provided during the rebuttal, the reviewer will maintain overall positive assessment and the original score

Reviewer_ukRgOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LA4VLA, a pretraining framework that learns from language instructions and proprioceptive states without visual observations before downstream vision-language-action finetuning. The authors construct dataset by segmenting existing DROID demonstrations into approximately 33,000 atomic action episodes with low-level language descriptions, using VLM-generated proposals and human verification. They compare LA-only, VLA-only, sequential LA-to-VLA, and mixed LA/VLA pretraining. Evaluation on MetaWorld, LIBERO, selected LIBERO-Plus perturbations, and three real-world manipulation task families shows that LA pretraining improves downstream success rates over matched VLA pretraining.

Critical strengths
  1. The motivation of the paper is clear. VLA models are pretrained on visual observations and language instructions and actions. The author claim that dense visual-action supervision can dominate the comparatively sparse language-action signal, and then propose a method to reduce reliance on scene-specific visual cues. Serveral simulation and real-world manipulation experiments are conducted to verify the performance of its LA-VLA model.
  2. The paper is well written and easy to understand.
Critical weaknesses
  1. My main concern is the experiments do not directly test the main hypothesis. The paper aims to reduce reliance on misleading visual cues by train without vsion. But removing vision also removes useful information, such as object positions and obstacles. Language and robot states alone cannot provide this information. The downstream gains show that LA pretraining might helps, but they do not show that the final policy learns to use vision more appropriately. The gains could instead come from learning general motion patterns and the precision of behavior. Stronger evidence connecting the existing results to the main hypothesis helps score.

  2. Furthermore I think the key comparisons are missing. Other approaches, such as stronger task-preserving image augmentation from previous work, could reduce visual shortcuts while keeping useful visual information, or more detailed label instruction annotations. It remains unclear whether removing vision is better than other ways of improving multimodal training under the same budget. These comparisons are important to justify the proposed approach and make people clear that why removing vision is the right solution.

Questions for authors
  1. Removing vision during pretraining prevents the use of both useful and not useful information, but does not by itself explain why the downstream policy would selectively become less dependent on spurious cues after visual finetuning?
  2. Whether MixPT is equivalent to visual-input dropout?
3 comments
AuthorsOpenReview ↗

Response to the Reviewer

R1.Experimental evidence for our main hypothesis (W1).

Visual finetuning introduces task-relevant scene information.  LA pretraining focuses on learning reusable action priors from atomic instruction–action pairs. During downstream visual finetuning, camera observations provide information about object positions and obstacles, allowing the policy to locate the instructed target and adapt its movements to the scene. The final policy can therefore use this information to locate the instructed target and adapt its movements to the scene.

Evidence for reduced reliance on misleading visual cues.  We add Gaussian noise to camera observations during real-world evaluation after visual finetuning, introducing task-irrelevant interference that makes visual cues less reliable (Section 5.5, Table 4). Under the same perturbation, the LA-pretrained policy achieves higher success than the VLA-pretrained policy on both Press Button (55.0% vs. 20.0%) and Place Drink (80.0% vs. 65.0%), with a 25-percentage-point advantage on average (67.5% vs. 42.5%). Together with the fixed-scene results below, this finding supports our interpretation that LA pretraining helps the final policy rely less on misleading visual cues when following language instructions.

The final policy follows language-specified targets more reliably.  Within each real-world task, we hold the initial scene and robot state fixed and vary only the instruction specifying the target. Under the same visual finetuning protocol, the LA-pretrained policy achieves 81.7% average success, compared with 48.3% for the VLA-pretrained policy (Section 5.5, Table 3). The higher success rate supports stronger language-guided target selection and execution in the final visual policy.

t-SNE visualizations show clearer alignment between action representations and instructed directions.  In Figure 9 (right panels, Appendix F), we compare the inputs to the action head using the same robot states and directional instructions for both policies, with visual input masked. The representations cluster more clearly by instructed direction after LA pretraining, providing qualitative evidence that the policy learns to associate language instructions with the actions they specify.

R2.Alternative visual augmentation approaches (W2).

We agree that stronger task-preserving visual augmentation can also improve multimodal learning. Our research question is whether explicitly constructed LA supervision provides an effective, transferable, and complementary pretraining signal. The matched comparisons are designed to answer this question; they do not establish that removing vision is universally preferable to every augmentation strategy. The additional gains from LA-VLA and MixPT further demonstrate the value of combining LA supervision with visual grounding. Stronger augmentation may be complementary to this approach, and we do not claim superiority over augmentation methods that were not directly evaluated.

However, to directly address the reviewer’s question about task-preserving visual augmentation, we ran an additional DROID-VLA + Random Overlay experiment. We used Random Overlay, first introduced in SODA [1] and subsequently used to train robotic manipulation policies [2, 3, 4]. During pretraining, we blended the original DROID camera observations with randomly sampled images and kept the task instructions, robot states, and action targets unchanged. We used the same pretraining budget and visual finetuning protocol as the DROID-VLA baseline. Using the same MetaWorld evaluation harness, we report success rates (%) for the four difficulty groups and their mean.

PretrainingEasyMediumHardVery HardAverage
DROID-VLA (Table 14)88.274.571.786.080.11
DROID-VLA + Random Overlay (new)86.871.871.784.078.6
LA-only (Table 14)93.281.865.092.083.00

Random Overlay scored 1.5 percentage points below DROID-VLA and 4.4 points below LA-only. Under the same training budget, LA pretraining was more effective than conventional VLA pretraining with Random Overlay, a task-preserving image augmentation method that retains visual input. This supports our choice to learn language-conditioned action priors from vision-agnostic atomic instructions during pretraining and use visual observations during downstream finetuning.

References

[1] Hansen and Wang. Generalization in Reinforcement Learning by Soft Data Augmentation. ICRA 2021. 

[2] Zhuang et al. Enhancing Visual Domain Robustness in Behaviour Cloning via Saliency-Guided Augmentation. CoRL 2024.

[3] Zhuang et al. MirrorDuo: Reflection-Consistent Visuomotor Learning from Mirrored Demonstration Pairs. CoRL 2025.

[4] R. Wang et al. PALM: Enhanced Generalizability for Local Visuomotor Policies via Perception Alignment. IEEE Robotics and Automation Letters, 2026.

AuthorsOpenReview ↗

Response to the Reviewer

R3.Fine-grained annotation control (W2).

We report a paired experiment that tests the effect of LA pretraining while holding annotation granularity fixed (Sections 5.1 and 5.2). In this comparison, the Atomic VLA baseline receives exactly the same fine-grained atomic instructions as LA. LA-33K-V pairs the atomic segments in LA-33K with their corresponding images and retains the same instructions, robot states, and action targets. Both settings use the same architecture, pretraining budget, and downstream protocol.

Results from Table 2.

BenchmarkAtomic VLA (%)LA (%)
MetaWorld79.7883.00
LIBERO94.4095.30
LIBERO-Plus59.0169.02

With annotation granularity held fixed, LA achieves higher success rates across the three benchmarks. Because both settings use the same detailed instruction annotations, the comparison suggests that the LA pretraining setup itself contributes to the gains.

R4.How LA pretraining helps reduce reliance on spurious cues after visual finetuning (Q1).

LA pretraining gives the policy a language-conditioned basis for selecting actions. During visual finetuning, the policy learns to use scene observations to locate the instructed target and adapt its movements. This gives the final policy a way to rely less on spurious visual cues while still using task-relevant visual information to execute the instructed action.

The fixed-scene evaluation supports this explanation. When the initial scene and robot state are held fixed and only the instruction changes, the LA-pretrained policy more reliably acts on the target specified by language (Table 3). We also add Gaussian noise to the camera observations to introduce task-irrelevant visual interference while keeping the instruction and target fixed. Under this interference, the LA-pretrained policy still completes the instructed tasks more reliably, achieving 67.5% average success compared with 42.5% for the VLA-pretrained policy (Table 4).

R5.MixPT and visual-input dropout (Q2).

On our paired atomic data, visual-input dropout can produce the same types of training examples used in MixPT. However, whether those examples retain explicit action supervision depends on the instructions. LA-33K’s atomic instructions specify the demonstrated local actions independently of the images, preserving explicit language–action supervision when visual input is disabled. In the original VLA demonstrations, task-level prompts describe the overall goal and provide limited supervision for individual demonstrated actions on their own. Dropping images during pretraining therefore leaves those actions underspecified by language, which can weaken the training signal and hurt pretraining performance.

To test whether dropout on conventional task-level data can reproduce the gains of LA pretraining, we added a DROID-VLA + visual-input dropout experiment. We first pretrained the model on the original DROID demonstrations with visual-input dropout, using the same pretraining budget as DROID-VLA. We set the per-example dropout probability to 0.5 to match the proportion of examples without images in MixPT’s 1:1 mixture of LA and VLA examples. We then finetuned the model on MetaWorld with full visual input, following the same downstream protocol as the DROID-VLA baseline (Appendix C.3).

PretrainingEasyMediumHardVery HardAverage
DROID-VLA (Table 14)88.274.571.786.080.11
DROID-VLA + visual-input dropout (new)90.772.768.372.075.9
LA-only (Table 14)93.281.865.092.083.00
MixPT (Table 2)88.994.566.7100.087.53

Adding dropout to DROID-VLA reduced average success by 4.2 percentage points. With the same target proportion of examples without images, MixPT achieved 87.53% using paired atomic data, compared with 75.9% for visual-input dropout on the original DROID demonstrations. This comparison highlights the role of LA-33K’s atomic instructions in providing explicit action supervision when images are withheld.

AuthorsOpenReview ↗

We look forward to hearing from you

As the discussion period is drawing to a close, we would greatly appreciate any further feedback you may have on our responses. We would be happy to provide additional clarification and address any remaining questions or concerns. If our responses have adequately addressed your concerns, we would be grateful if you could consider raising your score. Thank you for your time and consideration.

Reviewer_39G9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper starts with a simple concern: robot models might look like they’re following instructions, but actually rely heavily on visual cues to decide what to do. To address this, the authors break existing robot demonstrations into short action segments and give each segment a specific instruction. They then pretrain the model using only language, robot state, and actions, without images, so it has to learn a clearer connection between instructions and movements.

This approach improves performance in both simulation and real-world tasks, especially when visual inputs change or become noisy. Combining it with normal vision-language-action training brings further gains. The main takeaway is that learning how instructions map to actions separately can make robot policies stronger and more robust, though the paper doesn’t establish improvements in broader language reasoning or debate abilities.

Critical strengths
  • A simple, well-motivated idea, backed by a controlled comparison using the same action data with and without images.
  • Strong results across simulation, real robots, and another model architecture, showing the approach is useful beyond one specific setup.
Critical weaknesses
  • Removing images prevents the model from taking visual shortcuts, but that alone doesn’t prove it learns a stronger connection between language and actions. It might simply learn to rely less on vision, much like randomly dropping images during training. A comparison with that simpler approach would help clarify whether the proposed method specifically improves instruction following or mainly makes the model less sensitive to visual changes.

  • Even without images, the model still receives the robot’s current state, such as its position, pose, and gripper status. Those signals may already hint at the next action. For example, if the gripper has just closed around an object, lifting it might be a common next step in the training data. Since the demonstrations are split into short action segments, some actions could be predicted from the starting state without carefully reading the instruction. The direction-following tests provide useful evidence, but removing or shuffling instructions would help show how much the model actually depends on language.

  • The direct evidence for better instruction following mostly comes from simple commands like “move left” or “move upward.” That shows the model can connect basic instructions to movements, but it doesn’t tell much about more complex requests, such as “move the cup first, then place the book, without touching the bottle.” Those require understanding order, constraints, and how actions fit together. The downstream results are encouraging, but more targeted tests would be needed to show whether the method improves this broader ability, rather than mainly strengthening familiar basic movements.

Questions for authors

seed weakness

3 comments
AuthorsOpenReview ↗

Response to the Reviewer

We thank the reviewer for highlighting our controlled comparisons and the consistent gains across simulation, real-world tasks, and model architectures. Below, we address each concern in turn and provide additional experimental results and clarifications.

R1. Conventional visual-input dropout versus explicit LA supervision (W1).

Following the reviewer’s suggestion, we added a DROID-VLA + visual-input dropout experiment. We applied dropout during pretraining on the original DROID demonstrations with their task-level instructions, using the same pretraining budget as DROID-VLA. We set the per-example visual-input dropout probability to 0.5 to match MixPT’s 1:1 mixture of LA and VLA examples. We then finetuned the model on MetaWorld with full visual input using the same downstream protocol as the baseline (Appendix C.3).

PretrainingEasyMediumHardVery HardAverage
DROID-VLA (Table 14)88.274.571.786.080.11
DROID-VLA + visual-input dropout (new)90.772.768.372.075.9
LA-only (Table 14)93.281.865.092.083.00
MixPT (Table 2)88.994.566.7100.087.53

Adding visual-input dropout to DROID-VLA lowered average success by 4.2 percentage points. With the same target proportion of training examples without images, MixPT achieved 87.53% on MetaWorld using paired atomic data, 11.6 percentage points above the dropout baseline. This comparison supports the value of LA-33K’s atomic instructions, which provide explicit supervision for the demonstrated actions even when images are withheld.

R2.Potential shortcuts through proprioceptive state (W2).

We examine this concern in downstream real-world tasks and in a diagnostic of the pretrained policy. Our real-world tasks test language-dependent target selection after visual finetuning. Different target instructions share the same initial scene and robot configuration, while requiring selection among four buttons, three shelf positions, or four evaluated grid cells. Thus, the initial image and state cannot uniquely determine the intended target.

Real-world success rates (Section 5.5, Table 3).

TaskVLA pretrainingLA pretraining
Press Button50.0%85.0%
Place Book40.0%65.0%
Place Drink55.0%95.0%
Average48.3%81.7%

The LA-pretrained policy’s higher success rate across different targets with identical initial images and robot states provides evidence that it uses the instruction to select the intended target.

The conflicting-state evaluation in Appendix F provides complementary evidence at the pretraining stage. We keep the evaluation instruction fixed and replace its state with one from an instruction–action pair associated with the opposite motion direction.

Direction-following results (Appendix F, Table 15).

Evaluation conditionDAR ↑DCS ↑
Original state0.990.85
Conflicting state0.950.80

The predicted direction therefore remains aligned with language despite the substituted state, providing evidence against an explanation based solely on state-associated motion patterns. Together, language-dependent goal selection under shared initial conditions and instruction following under conflicting states support the use of language in downstream execution and action prediction.

AuthorsOpenReview ↗

Response to the Reviewer

R3.Purpose of the atomic diagnostic and transfer to long-horizon manipulation (W3).

LA pretraining learns language-conditioned priors for local actions from atomic instruction–action pairs. Downstream visual finetuning adapts these priors to full tasks using task-level instructions and scene observations. We use directional commands in the diagnostic because their expected motions are easy to quantify and visualize. Our downstream evaluation also includes more complex instructions that specify multiple actions. LA pretraining improves success on these long-horizon tasks.

On LIBERO-Long, success rises from 88.2% with VLA-only to 92.0% with LA-only and reaches 93.4% with either LA-VLA or MixPT (Table 2). In the microwave task, for example, the robot approaches, grasps, and lifts the yellow-and-white mug, aligns it with the microwave opening, and inserts and releases it inside. It then withdraws the gripper, moves it to the door, and closes the microwave. The LIBERO-Long gains show that, after visual finetuning, the LA-pretrained policy completes tasks that require this kind of ordered, multi-step coordination more reliably. 

Furthermore, the gains extend to long-horizon tasks under perturbations. On the LIBERO-10 tasks in LIBERO-Plus, LA-only improves success under all four perturbations (object layout, background, image noise, and camera viewpoint), raising the average from 59.01% to 69.02%. MixPT reaches 70.40%. These results show that the local action priors learned during LA pretraining help the finetuned policy coordinate multiple actions to complete long-horizon tasks even when the scene or visual observations change.

AuthorsOpenReview ↗

We look forward to hearing from you

As the discussion period is drawing to a close, we would greatly appreciate any further feedback you may have on our responses. We would be happy to provide additional clarification and address any remaining questions or concerns. If our responses have adequately addressed your concerns, we would be grateful if you could consider raising your score. Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank the Area Chair and all four reviewers for their time and thoughtful feedback. The initial reviews included three Weak Accept ratings (3) and one Weak Reject rating (2). During the rebuttal, we responded to all four reviews and added two experimental baselines suggested by the reviewers. Two reviewers followed up during the discussion. After considering our response and the additional experimental results, Reviewer oSVg stated that they would maintain their overall positive assessment and their original score of 3 (Weak Accept). Reviewer nG3U noted that the results in Table 3 addressed the real-world aspect of their question. At the time of writing, we have not received follow-up comments from the other two reviewers and would welcome any further questions they may have.

Across the reviews, the reviewers highlighted several strengths of LA4VLA.

Clear motivation and presentation. Reviewers 39G9 and ukRg appreciated the clear motivation for studying language–action supervision, and ukRg highlighted the paper’s readability.

Practical value of LA-33K. Reviewer oSVg highlighted the value of deriving atomic instruction–action pairs from existing robot demonstrations, and nG3U appreciated the transparent description of segmentation, human verification, and annotator agreement.

Controlled experimental design. Reviewers 39G9, oSVg, and nG3U recognized that using the same atomic demonstrations and matched training budgets makes it possible to study the effect of visual input during pretraining.

Breadth and consistency of the results. Reviewers highlighted the gains across simulation and real-world manipulation, including under visual perturbations, as well as evidence of transfer to a second VLA architecture.

Language-dependent real-world evaluation. Reviewer nG3U highlighted the use of different language-specified goals within a shared initial scene and confirmed during the discussion that these experiments addressed their question about instruction following after visual fine-tuning.

In response to the reviewers’ questions, we added experiments and clarified the following points.

Alternative training strategies. We added matched-budget visual-input dropout and Random Overlay baselines and clarified how the paired atomic-data comparison controls for annotation granularity.

Language–action grounding. We clarified what the visual-conflict diagnostic supports and how the conflicting-state test and shared-scene robot tasks provide complementary evidence of instruction following.

Action scale and long-horizon transfer. We clarified the respective roles of atomic LA supervision and downstream visual fine-tuning in learning local motions, adapting movement distances, and completing multi-stage tasks.

Dataset construction. We clarified how robot-state keyframes guide candidate segmentation and how human reviewers verify the resulting instruction–action pairs.

Relationship to Qwen-VLA. We will explicitly credit Qwen-VLA’s visual-input ablation and clarify how LA-33K extends that work through matched comparisons of LA-only, VLA-only, sequential, and mixed pretraining.

Finally, we highlight the main contributions of LA4VLA.

Language–action pretraining. We introduce a framework that separates language-conditioned action learning from visually grounded VLA training, allowing policies to learn action priors that transfer across scenes and tasks.

The LA-33K dataset. We construct vision-agnostic instruction–action pairs by segmenting existing expert demonstrations into atomic actions and pairing each segment with a low-level instruction, without collecting additional robot data.

Transfer and robustness. We show that LA pretraining improves manipulation across simulation benchmarks, model architectures, and real-world tasks. It also improves robustness to visual perturbations and yields further gains when combined with standard VLA pretraining.

We appreciate the reviewers’ suggestions and will incorporate the additional experiments and clarifications into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.