acceptodds
Withdrawn from ICLR 2027

In-Context Structured Demonstration enables Generalizable Vision-Language-Action Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action (VLA) models often degrade sharply in out-of-distribution (OOD) scenarios. In-context imitation learning (ICIL) enables test-time adaptation by retrieving expert demonstrations and appending them as contextual prompts. Existing frameworks, however, underutilize these trajectories: by conditioning on raw observations and continuous actions alone, the policy observes what the expert did without the task structure that explains why. We propose StellaVLA, a framework that advances ICIL by distilling in-context structured rationales into VLA models. First, an offline pipeline extracts a hierarchical rationale from raw expert demonstrations without manual annotation, articulating both high-level semantic sub-goals and fine-grained kinematic movements through a language-as-action paradigm. The same rendering applies to real-robot, human-hand and XR-retargeted demos. Second, a parallel dual-training paradigm uses these rationale-augmented trajectories as supervision, training the VLA to predict continuous actions and to articulate the underlying rationale from one shared representation. During deployment, an asymmetric inference strategy strips away the language head, enabling high-frequency control guided by KV-cached expert rationale. Extensive experiments across simulation benchmarks (e.g., LIBERO and VLA-Arena) and real-world manipulation show that StellaVLA effectively exploits heterogeneous demonstrations as structured context, consistently improving both in-distribution performance and OOD robustness, achieving state-of-the-art performance without any robotics pretraining.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Model, In-Context Learning, Generaliztion

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_odJtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a new method for ICIL for VLA models as they suffer from OOD called StellaVLA. StellaVLA has two expert heads during training, one which predicts actions and one which predicts supervised language output which is task related (subtask description, 2D trace, 3D motion). This language expert is dropped during inference and the supervision data for it is produced using an LLM on the given training data. StellaVLA achieves high results on LIBERO, LIBERO-Plus, VLA-Arena and real robot tasks. Ablations show that the model needs either no or correct in-context conditioning or the performance is dropping heavily.

Critical strengths
  • Experiments include a wrong-task intervention test which shows that the method is actually relying on the given demonstrations
  • The overall structure of the paper is easy to follow
  • Very good results on the trained models on different benchmarks and real robot tasks
Critical weaknesses

Weaknesses

  • Experiments are not part of the contribution as they are necessary to show that the proposed method is working.
  • The claim that OOD robustness comes from the dual training setup is not given, as shown in Table 8 in the appendix and should be reframed in the paper.
  • The paper says that ICIL methods usually only look at what an expert did, but now why. It is not clear how StellaVLA does this differently and if it is evaluated.
  • The StarVLA-OFT baseline removes the retrieved demonstration and the spatial-language supervision. This makes it unclear which of these two actually contributes to the proposed method, as neither baseline is evaluated.
  • It is not clear how retrieval works for LIBERO-Plus and the VLA-Arena L1/L2. Especially in regards to the language descriptions. Which demonstrations were used and how do the instructions differ from the demos used for training?
  • It is not clear how StellaVLA differs from the LAP (Zha et al.) model. Differentiating it would strengthen the contribution and novelty of the paper. Also including a related work section about embodied reasoning as auxiliary supervision without decoding it at inference would be helpful to put this work into perspective.

Minor Weaknesses

  • Line 81 the word Context is written with a capital letter in the beginning but nothing else of “in-context structured distillation”
  • It is not clear if the evaluated models are only one seed or multiple ones
  • It is not clear how well the Qwen model can divide the tasks into subtasks and it is not evaluated against a hand-labeled control sub-dataset
  • There is also more related work which should be cited and included in the paper to compare to the proposed method, like Knowledge Insulating VLAs (Driess et al.) and ECoT-Lite (Chen et al.) and other work in this direction.
Questions for authors
  • Is the semantic segmentation evaluated somehow using a hand annotated validation set?
  • Why is the success for some VLA-Arena L0 tasks lower compared to the L1 and L2 variants?
  • How do the demos differ from VLA-Arena L0 and L1/L2?
  • What is the difference from StellaVLA to LAP?

Please also consider all weaknesses also as a question that should be answered.

Reviewer_QignOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StellaVLA, an in-context imitation learning(ICIL) framework that leverage structured context demonstration rather than raw visual observation and continuous low-level action which were used in prior works of ICIL. An offline pipeline converts raw expert demonstration into structured context demonstration compromising semantic subgoals, visual keyframes, and verbalized kinematic information. During training, an action expert and a spatial-language expert operate in parallel on representations from a shared vision-language backbone and are jointly trained to predict continuous action chunks and structured rationales, respectively. At inference, the spatial-language expert is not executed, and the model reuses the key–value cache of the fixed demonstration prefix to reduce latency.

Critical strengths
  1. Clear Problem Formulation and Motivation

The paper identifies a concrete limitation of existing ICIL method which is using raw expert demonstration comprising raw visual observation and continuous low-level action. Due to this fact, existing ICIL methods may underuse the reference demonstration and may fall back on their prior pretrained knowledge. They may also imitate the expert’s trajectory without capturing why each action was taken, which is a well-posed and important gap for generalization under distribution shift.

  1. Well-designed interface for using cross- and off- embodiment demonstration.

The structured representation lets demonstrations from different embodiments and off-embodiments share a single context format without requiring a shared action space.

Critical weaknesses
  1. The claim that StellaVLA learns “why” rather than only “what” remains insufficiently substantiated.

The 3D motion and 2D trace are deterministic spatial-language descriptions of the ground-truth action. Predicting these alternative representations may provide useful auxiliary supervision but does not itself show an understanding of why an action is appropriate or why behind the expert demonstration. Semantic subtask labels could provide procedural guidance, but their contribution is not separately ablated. Moreover, under object-layout changes, StellaVLA shows virtually no improvement over the match control. The authors attribute this to demonstration kinematics no longer matching the scene and the policy falling back on live observations. This explanation does not demonstrate that an abstract procedure transfers when the original motion descriptions become inapplicable. Although the result does not prove trajectory copying, it leaves the claimed benefit of reusable rationale-based guidance insufficiently supported.

  1. The OOD comparisons and ablations do not sufficiently isolate the method’s benefit.

While the reported OOD gains support the effectiveness of StellaVLA’s overall system, neither LIBERO-Plus nor VLA-Arena compares against prior ICIL methods. And comparison with Retrieval-VLA is restricted to LIBERO which is ID scenario. Although the matched control StarVLA-OFT is compared in Libero-Plus, this control removes retrieved demonstration. To show the benefit and effectiveness of structured demonstration, the experiment should compare with not only StarVLA-OFT without retrieved demonstration but also StarVLA-OFT with retrieved raw demonstration. Moreover, this matched control is absent from VLA-Arena where leaderboard baselines also lack demonstration context. Lastly, Table 8 shows that StellaVLA achieves its highest LIBERO-Plus score without the auxiliary language loss (86.9% at λ=0\lambda = 0 vs. 85.1% at the default). Since the structured demonstration is still provided at λ=0\lambda = 0, this suggests that the OOD gains stem from demonstration conditioning rather than dual training, while whether the structure itself matters remains untested without the raw-demonstration control above.

Questions for authors
  1. What is retrieval accuracy on each LIBERO-Plus axis, especially language perturbation and on VLA-Arena L1/L2? What is the procedure when no exact task match exists?

  2. In Table 3, what does each modality condition contain? Does the image-only demonstration consist of keyframes alone, or keyframes with raw actions? How do the evaluation-time and training-time settings differ? Also can author reports not only averaged result of LIBERO-Plus but only axis-wise result of LIBERO-Plus in table 3?

  3. The semantic sub-task label appears central to the claimed procedural guidance, but it is not separately ablated. Could the author ablate it and report the experiment?

Reviewer_StrUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

StellaVLA conditions a Qwen3-VL policy on an expert demonstration rendered as subgoal images, task descriptions, robot states, and verbalized motion. An auxiliary language objective supervises subtask and motion predictions alongside continuous action regression; inference uses only the action output and caches the demonstration prefix. The paper evaluates this combination on LIBERO, LIBERO-Plus, VLA-Arena, and a Piper robot, including human-hand and retargeted demonstrations. The most convincing evidence concerns the usefulness of structured textual context in simulation. The attribution of generalization to rationale supervision and cross-embodiment transfer is substantially less secure.

Critical strengths
  1. The modality interventions provide useful evidence about what the policy consumes. In Table 3, removing demonstration images at evaluation preserves the 98.8% LIBERO average, whereas retaining images without the structured guidance reduces it to 92.9%. Training with text-only demonstrations also retains nearly the full reported LIBERO-Plus performance, 85.0% versus 85.1%. These observations make a substantive case for investigating compact textual representations of demonstrations, particularly when visual correspondence is unreliable. The paper also acknowledges that the no-demonstration intervention is outside the LIBERO checkpoint's training distribution.

  2. The architecture gives a concrete way to retain contextual conditioning at practical action latency. Action placeholders precede the auxiliary language targets, so action prediction need not consume generated rationales. Caching the fixed demonstration prefix reduces the reported forward latency from 183 to 91 ms (Table 9), and Appendix C describes a working hardware control configuration. This is a useful integration of structured context and continuous action prediction, even though learning from reasoning without generating it at deployment is not itself new.

Critical weaknesses
  1. The supplied evaluation code conflicts with the stated LIBERO-Plus protocol on two consequential points. Section 4.2, p. 8, states that retrieval uses the perturbed instruction and that no privileged task identity enters the language comparison. However, examples/LIBERO/eval_files/serve_stellavla.py, _resolve_demo_task_idx (lines 172–181), recovers the original task from task_classification metadata indexed by the evaluation task ID when instruction matching fails. The supplied LIBERO-Plus launcher enables this option (lines 45–49). This can supply the clean task identity precisely when language retrieval fails, so the 95.3% language result cannot be interpreted as robustness to the perturbed instruction without resolving this discrepancy. Furthermore, run_lp_full.sh lines 68–73 exclude 600 tasks by default and state that the reported score used this setting, whereas the manuscript claims the official protocol without disclosing this exclusion. The code establishes what the supplied configuration does; it does not establish which configuration produced every table entry. Exact run provenance and comparisons on the same task set are necessary before the headline robustness improvement is reliable.

  2. The evidence supports a narrower contribution than the rationale-imitation and cross-embodiment claims. Table 8's best robustness result uses no auxiliary language loss: 86.9% versus 85.1% for the proposed default. Its 1.2-point in-distribution gain is also selected on benchmark success, as Appendix D.2 explicitly acknowledges. Thus the dual objective has not demonstrated the claimed OOD benefit. On hardware, the source-invariance test changes predicted actions by only 0.0014–0.0016 standard deviations between sources, but withholding the demonstration also changes them very little, 0.0041–0.0045 (Table 5). As Appendix C.1 recognizes, these measurements cannot distinguish useful cross-embodiment transfer from largely ignoring all demonstration sources. This distinction matters because reasoning supervision without test-time rationale generation already appears in Chen et al., Training Strategies for Efficient Embodied Reasoning (2025), which is not discussed. The novel case therefore needs to rest on what the structured retrieved demonstration adds, rather than on eliminating rationale decoding. The existing results suggest that contribution but do not establish its advertised breadth.

Questions for authors
  1. Which exact retrieval configuration and task list produced Table 1 and the LIBERO-Plus ablations? Did the language evaluation ever use the metadata fallback? Please provide the language-only retrieval result, including retrieval failures, and establish whether the matched control and published baselines were evaluated on the same retained tasks. These are protocol clarification requests, not suggestions to expand the benchmark.

  2. Does a fixed hardware checkpoint improve in closed-loop success or progress with a human/XR demonstration versus no demonstration, holding initial conditions fixed? Appendix C.1 identifies this missing comparison; it would directly determine whether the claimed cross-embodiment benefit is present.

  3. Beyond the benchmark-selected LIBERO improvement, is there evidence that the auxiliary rationale objective improves generalization over the same structured-context policy with λ = 0? If not, the contribution should be framed around structured context and the observed accuracy–robustness trade-off.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.