acceptodds
Under review as a conference paper at ICLR 2027

From Sparse to Dense: Diagnosing Task-Equivalent Language Robustness in Vision-Language-Action Models

Abstract

Vision-Language-Action (VLA) models are commonly trained and evaluated with concise task commands, leaving open whether learned control remains accessible when the same executable intent is expressed through richer, task-equivalent language. An audit of LIBERO and SimplerEnv finds that their native instructions concentrate on direct, low-context expressions. To study this uncovered regime, we construct a three-level task-equivalent benchmark with controlled dense language and evaluate models under matched rollouts that vary only the language realization. These evaluations reveal a substantial Sparse–Dense Gap whose magnitude depends strongly on the model and task. Further diagnostics show that paired sparse–dense action training yields limited dense gains and that action fine-tuning reduces inherited visual-language capability. Guided by these findings, STAR, a Sparse-to-dense Task Alignment Regularization framework, uses a lineage-matched pre-action VLM and the source VLA as complementary representation and action anchors. Across LIBERO, STAR raises dense macro success from 7.2% to 24.3% while retaining 69.7% sparse success, and it improves the corresponding sparse–dense balance in SimplerEnv, demonstrating that task-equivalent dense-language access is a distinct robustness problem that can be systematically measured and improved.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language-Action Models, Language Generalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XficOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary
  • This paper studies how VLA models perform when the same task is expressed through more complex language. The authors introduce the Sparse–Dense Gap and define three instruction-density levels (Simple, Middle, and Hard) to evaluate this issue.
  • The authors also propose STAR, which combines sparse–dense action supervision, pretrained VLM representation alignment, and action distillation from VLAs.
  • On LIBERO, STAR improves average dense-instruction success, while preserving sparse success. The paper also includes evaluations on SimplerEnv and several ablation studies.
Critical strengths
  • I really like the definition of the three instruction-density levels (Simple, Middle, and Hard). It provides a clear and systematic way to study language robustness in VLA models, while keeping the underlying task unchanged.
  • Nice empirical evaluation across different VLA models and benchmarks. The results show that language robustness varies quite a bit across models, and the ablations help explain the contributions of the different STAR components.
Critical weaknesses
  • Overlap with existing work: The weakness identified in this paper seems closely related to the From Intention to Execution paper. It would be great if the authors could more clearly differentiate the Sparse–Dense Gap from the findings in that paper. Also, the observation that action fine-tuning degrades pretrained VLM capabilities has already been explored in works such as LAP. I think the paper could do a better job positioning these findings relative to prior work.
  • Missing stronger baselines: There are many existing approaches that could potentially address this problem through instruction rewriting, reasoning, or verification, such as Hi Robot, ECoT, and CoVer-VLA. A simple baseline that uses an LLM to rewrite dense instructions into sparse instructions before feeding them into the VLA would also be interesting. Without these comparisons, it's a bit difficult to assess how much STAR improves over existing approaches.
Questions for authors
  • How is the Sparse–Dense Gap different from the intention–execution gap identified in From Intention to Execution? It would be helpful to clarify the key differences.
  • Could the authors consider adding stronger baselines, such as instruction rewriting with an LLM or reasoning-based approaches like Hi Robot and ECoT? In particular, I'm curious how well a simple dense-to-sparse rewriting pipeline would perform compared to STAR.
  • How does the VQA capability degradation and recovery analysis differ from prior work such as LAP? It would be great to clarify what new insights STAR provides beyond existing approaches for preserving pretrained VLM capabilities.
1 comment

Thank you for the response and for running the additional experiments! Overall, my concerns have been partially addressed, and will raise the score.

Reviewer_MQMdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether VLA policies can execute familiar tasks when the instruction is rephrased in richer but task-equivalent language. It constructs dense instruction variants from LIBERO/SimplerEnv instructions and evaluates language changes under matched rollout conditions. STAR combines paired sparse and dense action supervision, representation alignment to the preaction VLM, and action distribution distillation from the source VLA. On LIBERO, STAR increases OpenVLA's dense macro success from 7.2% to 24.3% (Direct Mix: 11.2%), while sparse success decreases from 73.4% to 69.7%. Additional experiments examine other policies and VQA retention.

Critical strengths
  • With task, scene and initial state held fixed and only the language changed, policies with high sparse success fall largely on dense forms (e.g., OpenVLA-OFT, CogVLA), while π0.5\pi_{0.5} retains strong performance on most dense conditions. This implies that the three graded levels benchmark success are more informative than benchmark success.

  • The dense mean of STAR exceeds Direct Mix on every LIBERO suite, and the spatial ablation supports benefits from representation and action alignment. Fractal variant aggregation improves greatly on dense success relative to Sparse-FT while maintaining sparse performance.

Critical weaknesses
  • The rewriter never sees the scene, but the middle and hard prompts ask for scene descriptions with competing details, so accepted examples in Table S2 conflict with the scene or with this work's own preservation rules. The Bridge Middle example identifies an eggplant on a cutting board, whereas the corresponding task visualization in Figure S5 shows the eggplant in a sink with none of those objects, and the simple drawer instruction adds a precondition "once you're done with those dishes".

  • Direct Mix is the only matched adaptation baseline, and there is no instruction-canonicalization baseline (since an LLM judge can verify that each dense instruction maps to the sparse command so it's a natural baseline) to assess whether dense interpretation can be handled before the existing controller when simple retains the source command. The dense performance of STAR stays low, 24.3% macro, 9.2% on Hard, ~10% on Goal and Long, which is far below unadapted π0.5 and is demonstrated only on OpenVLA. All results are single runs without confidence intervals or different seeds and the evidence beyond LIBERO is weak.

Questions for authors
  • Why is Fractal-VM simple 0.0 for all methods while VA simple reaches 49.5% and VM hard is nonzero?

  • What fraction of middle and hard instructions do humans map to the correct task on a sample given the first frame?

  • How large is the STAR training pool, I mean rewrites per task or per demonstration? How is VQA evaluated on an action tokenized policy, and why can STAR recover it without any text supervision?

  • Could you report STAR evaluated on human written or other llm rewrites and on held-out tasks?

Reviewer_uV5rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether a VLA policy that has learned a task can still carry it out when the instruction is phrased in richer, more conversational language that still describes the same task. The authors first audit the native instructions in LIBERO and SimplerEnv by using IDS. They find that all instructions are short, direct commands (IDS 1–2).

Using OpenVLA as a case study, they then present three diagnostics. First, replacing the native command with a denser equivalent (IDS 3) cuts LIBERO-Spatial success from 83.2% to 17.6%, with the scene and rollout held fixed. Second, action fine-tuning substantially degrades the VQA ability inherited from the underlying Prismatic VLM. Third, simply training on Direct Mix narrows the gap only modestly and slightly hurts performance on the original commands.

Based on these findings, the authors propose STAR, a fine-tuning objective with two frozen teachers. Representation alignment keeps the student’s instruction representations close to those of the pre-action VLM. Action distillation pulls the student’s action distribution, under both instruction forms, toward that of the source VLA conditioned on the sparse command.

To evaluate the approach, the authors build a three-level benchmark of dense instructions. The instructions are generated by DeepSeek-V4-Flash and screened by GPT-5.6-Sol as an LLM judge. On LIBERO, STAR raises average dense-instruction success from 7.2% to 24.3%, while success on the original commands falls modestly, from 73.4% to 69.7%. It also improves the balance between the two instruction types on SimplerEnv, although a substantial gap remains.

I might just leave all my concerns and weaknesses here.

  1. A natural alternative is to have an LLM rewrite the dense instruction into a concise command before passing it to the unchanged policy. This needs one call per episode, not per action step, so the latency cost seems acceptable. This simplest baseline is missing.

  2. The empirical results are quite confusing. After STAR, OpenVLA reaches 42.4% / 21.5%/9.2% success on LIBERO. Without any adaptation, π0.5 already achieves 95.0% / 86.8% / 59.0% across the three levels, and InstructVLA reaches 67.7% on Simple. These results are superior to STAR-finetuned OpenVLA.

  3. Following up on the previous point, the diagnoses in Sec.2 also only rely on OpenVLA results. It is unclear whether STAR transfers to flow-based or diffusion-based policies.

  4. There are some existing works that study complex natural language instruction for VLA that are not discussed. Understanding that there are some 2026 conference papers / arxiv papers, [1-2] I would not undermine the novelty of this work based on that. Just for awareness. But InstructVLA (cited) requires substaintial discussion, as it also studies keeping reasoning along with the action capacity. [1] LangGap: Diagnosing and Closing the Language Gap in Vision-Language-Action Models [2] When Does Language Matter? Multilingual Instructions Reveal Step-wise Language Sensitivity in Vision-Language-Action Models

  5. There is no human annotator to validate the quality of deepseek generated content. Does it reflect human language habits? Or is it just injecting noise to make the problem more complex?

  6. Experiments - There is no hyper-parameter sensetivity analysis for \lambda_rep and \lambda_KD. 2. All experiments are conducted in simulation (LIBERO and SimplerEnv), without any physical robot evaluation. This setting is addressing a realistic problem of complex natural language instruction, I would be surprised not to see the real demo of human robot interaction.

Critical strengths

A relevant, clearly scoped question.

Critical weaknesses
  • Positioning and missing baselines.

The paper doesn’t adequately position STAR against existing work on complex or indirect instructions for VLAs.

The paper cites InstructVLA [1] only as an evaluated model, yet it addresses closely related questions. It keeps multimodal reasoning alongside action learning and introduces SimplerEnv-Instruct. That benchmark seems to conflict with the claim that current work covers only direct, low-context commands.

The most natural alternative is missing: use an LLM to rewrite the dense instruction into a concise command for the unchanged policy. This needs one call per episode, not per action step, and is plausibly easy here, since the dense instructions were themselves generated from the sparse ones by an LLM. InstructVLA’s GPT-4o-assisted baseline and VLA Grounder (cited) are close to this, but neither is compared. The only adaptation baseline is Direct Mix, and its training budget is not specified.

  • Experimental setup and results

This work only diagnoses with OpenVLA and validates STAR with OpenVLA. STAR remains well below existing models evaluated without adaptation.

Questions for authors
  1. LLM rewriting baseline. Have the authors considered using an LLM to convert the dense instruction into a concise command before passing it to the original, unadapted policy?

  2. Comparison/Diagnose with stronger existing models. How would STAR apply to continuous-action, flow-based or diffusion-based models?

  3. The dense instructions are generated by DeepSeek-V4-Flash and validated only by another LLM. Was any human evaluation done to check that they preserve the task and resemble how people actually instruct robots?

  4. Hyperparameter sensitivity and real-world evaluation. How sensitive is STAR to λ_rep and λ_KD, and to the distillation temperature and alignment layer?

1 comment

Thank you for the detailed rebuttal. The authors have partially addressed my concerns. After carefully reviewing the rebuttal and the comments from other reviewers, I have decided to maintain my initial rating.

I would encourage the authors to provide a more in-depth analysis of why the simple baselines struggle, particularly on the "hard" tasks. Such an analysis would help better motivate the proposed method and provide stronger insights into its advantages over existing approaches.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.