This paper asks whether a VLA policy that has learned a task can still carry it out when the instruction is phrased in richer, more conversational language that still describes the same task. The authors first audit the native instructions in LIBERO and SimplerEnv by using IDS. They find that all instructions are short, direct commands (IDS 1–2).
Using OpenVLA as a case study, they then present three diagnostics. First, replacing the native command with a denser equivalent (IDS 3) cuts LIBERO-Spatial success from 83.2% to 17.6%, with the scene and rollout held fixed. Second, action fine-tuning substantially degrades the VQA ability inherited from the underlying Prismatic VLM. Third, simply training on Direct Mix narrows the gap only modestly and slightly hurts performance on the original commands.
Based on these findings, the authors propose STAR, a fine-tuning objective with two frozen teachers. Representation alignment keeps the student’s instruction representations close to those of the pre-action VLM. Action distillation pulls the student’s action distribution, under both instruction forms, toward that of the source VLA conditioned on the sparse command.
To evaluate the approach, the authors build a three-level benchmark of dense instructions. The instructions are generated by DeepSeek-V4-Flash and screened by GPT-5.6-Sol as an LLM judge. On LIBERO, STAR raises average dense-instruction success from 7.2% to 24.3%, while success on the original commands falls modestly, from 73.4% to 69.7%. It also improves the balance between the two instruction types on SimplerEnv, although a substantial gap remains.
I might just leave all my concerns and weaknesses here.
-
A natural alternative is to have an LLM rewrite the dense instruction into a concise command before passing it to the unchanged policy. This needs one call per episode, not per action step, so the latency cost seems acceptable. This simplest baseline is missing.
-
The empirical results are quite confusing. After STAR, OpenVLA reaches 42.4% / 21.5%/9.2% success on LIBERO. Without any adaptation, π0.5 already achieves 95.0% / 86.8% / 59.0% across the three levels, and InstructVLA reaches 67.7% on Simple. These results are superior to STAR-finetuned OpenVLA.
-
Following up on the previous point, the diagnoses in Sec.2 also only rely on OpenVLA results. It is unclear whether STAR transfers to flow-based or diffusion-based policies.
-
There are some existing works that study complex natural language instruction for VLA that are not discussed. Understanding that there are some 2026 conference papers / arxiv papers, [1-2] I would not undermine the novelty of this work based on that. Just for awareness. But InstructVLA (cited) requires substaintial discussion, as it also studies keeping reasoning along with the action capacity.
[1] LangGap: Diagnosing and Closing the Language Gap in Vision-Language-Action Models
[2] When Does Language Matter? Multilingual Instructions Reveal Step-wise Language Sensitivity in Vision-Language-Action Models
-
There is no human annotator to validate the quality of deepseek generated content. Does it reflect human language habits? Or is it just injecting noise to make the problem more complex?
-
Experiments - There is no hyper-parameter sensetivity analysis for \lambda_rep and \lambda_KD. 2. All experiments are conducted in simulation (LIBERO and SimplerEnv), without any physical robot evaluation. This setting is addressing a realistic problem of complex natural language instruction, I would be surprised not to see the real demo of human robot interaction.
Thank you for the response and for running the additional experiments! Overall, my concerns have been partially addressed, and will raise the score.