- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates whether language-action pretraining without visual observations can improve downstream VLA policies. The authors segment DROID demonstrations into atomic action episodes and combine VLM-generated annotations with human verification to construct LA-33K. They compare LA-only, VLA-only, sequential LA-to-VLA, and mixed pretraining across simulation benchmarks, visual perturbations, a second VLA architecture, and real-world robot tasks. The main value of the work lies in its construction of atomic language-action data and its systematic empirical comparison of pretraining strategies using paired data and matched training budgets.
Critical strengths
- LA and VLA pretraining use the same atomic action segments and the same total number of optimization steps, helping isolate the effect of visual conditioning during pretraining. The appendix also includes a comparison against pretraining on the original, unsegmented DROID demonstrations.
- Relative to matched VLA pretraining, LA improves success rates by 3.22, 0.90, and 10.01 percentage points on MetaWorld, LIBERO, and the evaluated LIBERO-Plus conditions, respectively. Average real-world task success increases from 48.3% to 81.7%. These results motivate further investigation, particularly for real-world manipulation and performance under visual perturbations.
- The paper describes the action taxonomy, keyframe detection, automatic segmentation, and human filtering procedures, and reports inter-annotator agreement. The real-world tasks also require language use by assigning different goals to the same initial scene.
Critical weaknesses
- Section 3 and Appendix F provide evidence that the LA-pretrained model can follow directional instructions without visual input. However, could masking visual input introduce a distribution shift for the VLA-trained model, whereas this condition is already encountered during LA pretraining? If so, to what extent might the diagnostic differences reflect this mismatch rather than differences in language-action grounding?
- The conflicting-state experiment provides useful supporting evidence. Would a more informative additional experiment be to evaluate the policies after downstream finetuning, holding visual observations and robot states fixed while varying the language instructions, and comparing whether the different pretraining schemes reliably produce the corresponding actions?
Questions for authors
- Could you correct the description of the relationship to Qwen-VLA and clarify the additional empirical insights established by the atomic dataset and paired comparisons?
- After completing downstream visual fine-tuning, when the language instruction is varied while keeping the image and robot state constant, does a model initialized with LA perform the corresponding action more reliably than one initialized with VLA?
- Since VLA models are trained using images but have those images abruptly removed during testing, does this create a discrepancy between training and testing conditions? If so, could the observed performance decline be partly due to an inability to adapt to the missing modality, rather than serving as definitive proof that the model failed to learn the language-action associations?
5 comments
Response to the Reviewer
R4.Visual inputs in the diagnostic and downstream evaluation (Q3).
In both simulation and real-world experiments, all downstream policies receive visual observations during finetuning and evaluation. Black-image substitution is used only in the Visual-removed condition of the diagnostic, where it changes the visual content at test time while preserving the image-input structure. As shown in R1, directional alignment also drops in Visual-unaligned and Visual-conflict, where the policy receives images from the same or similar scenes and the instruction and robot state remain fixed (Table 1). As shown in R2, with the initial scene and robot state held fixed, the LA-pretrained policy more reliably completes real-world tasks in which language specifies the target (81.7% vs. 48.3% for matched VLA pretraining, Table 3). Together, R1 and R2 show that limitations in the VLA policy’s language–action grounding persist even when visual input is available in the same format during training and testing.
We look forward to hearing from you
As the discussion period is drawing to a close, we would greatly appreciate any further feedback you may have on our responses. We would be happy to provide additional clarification and address any remaining questions or concerns. If our responses have adequately addressed your concerns, we would be grateful if you could consider raising your score. Thank you for your time and consideration.
Response to the Reviewer
We thank the reviewer for highlighting our controlled pretraining comparisons and the design of our real-world tasks, which use language to specify different goals in the same scene. Below, we address each concern in turn and clarify our experimental design and findings.
R1.Clarifying the visual-intervention diagnostic (W1).
Same-scene controls for distribution shift. We test two conditions that retain the image-input configuration used during VLA training. Visual-unaligned uses an image from the same scene that is unaligned with the current instruction, and Visual-conflict uses an image from a trajectory in a similar scene associated with the opposite motion direction. The language instruction and robot state remain fixed (Section 3.1). Using images from the same or similar scenes limits distribution shift due to unrelated scene differences, so the comparison focuses on how changing the visual–instruction correspondence affects the predicted action. With the original paired observation, the policy achieves DAR 0.98 and DCS 0.95. Replacing it with an unaligned same-scene image reduces these values to 0.66 and 0.37; under conflicting visual input, they further decrease to 0.35 and 0.03 (Table 1). These results show that even under comparable visual input distributions, with the instruction and robot state held fixed, mismatched visual cues can pull the VLA policy’s predictions away from the instructed direction.
Behavioral and representational evidence of language–action grounding. In our real-world tasks, we vary the language-specified target while holding the initial scene and robot state fixed. After the same visual finetuning protocol, the LA-pretrained policy achieves 81.7% average success compared with 48.3% for the VLA-pretrained policy (Table 3). This shows that the final visual policy follows language-specified goals more reliably after LA pretraining. The t-SNE plots in Figure 9 (right panels, Appendix F) show that the inputs to the action head cluster more clearly by instructed motion direction after LA pretraining, providing qualitative evidence that the policy learns to associate language instructions with the actions they specify. Together, the real-world results and t-SNE visualizations provide behavioral and representational evidence that LA pretraining strengthens language–action grounding.
R2.Language-dependent behavior after downstream visual finetuning (W2 and Q2).
As the reviewer noted among the strengths, our real-world tasks assign different language-specified goals to a shared initial scene. This design closely matches the central requirement of the suggested evaluation: after downstream visual finetuning, different instructions share the same initial visual scene and robot states, while requiring different target actions.
The policy must select among four buttons, three shelf positions, or four evaluated grid cells according to language. The initial image and state alone cannot identify the intended target, and all policies receive full visual input during execution. LA improves success over matched VLA pretraining from 50.0% to 85.0% on Press Button, from 40.0% to 65.0% on Place Book, and from 55.0% to 95.0% on Place Drink, increasing the average from 48.3% to 81.7% (Section 5.5, Table 3; setup in Appendix D).
These results show that, after visual finetuning, LA-pretrained policies perform more reliably on tasks where language specifies different goals under shared initial visual scenes and robot states.
R3. Relationship to Qwen-VLA (Q1).
We will correct and expand the comparison in Section 2 to clarify the shared use of vision-free action pretraining and the differences in dataset construction and empirical focus. Qwen-VLA pretrains its action decoder without images, using task instructions and robot-specific control prompts. Our work focuses on constructing LA-33K, a dataset of vision-agnostic atomic instruction–action pairs derived from existing robot demonstrations, and systematically evaluating this supervision through paired experiments with matched training budgets. On the same atomic episodes, LA-only pretraining improves downstream performance over VLA-only pretraining, and combining LA and VLA pretraining yields further gains.