acceptodds
Withdrawn from ICLR 2027

What Makes Good In-Context Examples for Vision-Language-Action Models?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Language-Action models (VLAs) have demonstrated strong capabilities across a wide range of physical manipulation tasks. To enable more versatile generalization, recent studies have sought to improve their In-Context Learning (ICL) ability, i.e., to perform new physical tasks by conditioning on a few related demonstrations at inference time. However, what information in these demonstrations actually drives ICL performance remains poorly understood. Prior work largely focuses on demonstrating ICL gains, leaving the roles of different context components underexplored. In this paper, we systematically investigate what makes in-context demonstrations effective for VLAs. We find that the action tokens in the demonstrations play the most important role, whereas removing images and language results in only a marginal decrease in performance. Our analysis also reveals a surprising fact: the pairing between observations and actions is largely unnecessary. Given a selected context, the correct mapping between images and action tokens is actually not required to perform ICL. Shuffling the ground-truth action tokens among demonstrations in the context window has little effect on the performance. We further find that ICL becomes more beneficial as tasks require greater generalization, while increasing the number of demonstrations can eventually hurt performance. Together, these findings provide practical insights into effective context construction and raise fundamental questions about how current VLAs use demonstrations for in-context learning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, In-context Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_246yOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates what makes effective in-context demonstrations for Vision-Language-Action (VLA) models. Through extensive experiments on pi0.5 and InternVLA-A1.5, the authors analyze the effects of different modalities, observation-action correspondence, and task difficulty. The experimental results suggest that action information plays a dominant role, accurate observation-action correspondence is less critical than expected, and more demonstrations do not necessarily improve performance.

Critical strengths
  1. The paper investigates an important and underexplored question about how VLAs utilize in-context demonstrations.
  2. The paper presents several interesting findings, particularly the dominance of action information and the limited importance of observation-action correspondence, although some experimental designs could be more rigorous.
  3. The paper examines multiple aspects of in-context demonstrations across different robotic manipulation benchmarks.
Critical weaknesses

My major concerns are listed as follows:

  1. The pi0.5-RICL model employs nearest-neighbor action interpolation during inference, which directly incorporates retrieved actions into the final predictions. This mechanism may influence the findings regarding action dominance and observation-action correspondence. Please provide more ablations.
  2. The shuffled actions are drawn from visually similar demonstrations of the same task, meaning that these actions may already be highly similar. Therefore, the small performance drop after shuffling may not necessarily indicate that observation-action correspondence is not important. Please analyze the differences between shuffled actions and consider experiments involving diverse actions.
  3. Although the paper considers both pi0.5 and InternVLA-A1.5, most key analysis are conducted only on pi0.5-RICL. Please conduct similar experiments on more VLA models to verify whether these findings generalize across different architectures. I think the current experiments are not sufficient.
  4. While the paper demonstrates that action information is the most important component of in-context demonstrations, it provides limited analysis of why this is the case. For example, does the model primarily benefit from task-specific action priors, action distributions, or other information?
  5. The paper has several writing and formatting problems. For example, an unresolved LaTeX reference ("Table ??") appears in line 300, and several figures have overly small text and crowded layouts, making them difficult to read. The overall presentation should be improved.

If the authors can address my concerns, I am willing to increase my rating.

Questions for authors
  1. The current experiments only compare full context with individual modalities (action-only, vision-only, and language-only). However, these results do not directly establish the contribution of each modality to the full context. Please include leave-one-modality-out experiments to better support the conclusions.
Reviewer_K5rgOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the mechanisms underlying RICL by examining demonstration modalities, observation–action pairing, task difficulty, and the number of demonstrations. The results suggest that action information is most important for RICL, while observation–action pairing has a relatively small impact. In contrast, using context from an incorrect task substantially degrades performance. ICL gains increase with task difficulty, whereas the effect of demonstration count is inconsistent.

Critical strengths
  1. The paper investigates RICL’s underlying mechanisms from multiple perspectives.
  2. The simulation experiments cover multiple LIBERO evaluation settings.
Critical weaknesses
  1. The analysis is restricted to RICL. For a paper investigating the mechanisms of in-context learning in vision-language-action models, examining a single paradigm is insufficient to support broader conclusions. The study should include additional approaches, such as In-Context Imitation Learning via Next-Token Prediction and In-Context Imitation Learning with Visual Reasoning.
  2. The presentation is poor, and many figures and tables have a pronounced AI-generated aesthetic. Substantial editing and refinement are needed during the rebuttal period. In its current form, the paper does not meet the standard for acceptance at ICLR 2027.
  3. Several experiments remain incomplete. For example, line 446 states that it is unclear whether the identified mechanisms apply to InternVLA, while line 368 labels a result as provisional because the corresponding rollout records are unavailable. These unresolved issues suggest that the paper is not yet ready for publication.

Minor issue: The table reference in line 300 is incorrectly formatted.

Questions for authors

See the weaknesses.

Reviewer_kJXqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates what makes effective in-context demonstrations for VLA models. The results suggest that demonstration actions carry most of the useful contextual information, while shuffling observation–action pairings has little impact on aggregate performance. The paper also finds that context is more beneficial under challenging distribution shifts and that increasing the number of demonstrations does not always improve performance.

Critical strengths

Investigating which components of demonstrations actually contribute to VLA in-context learning is an interesting and important question.

Critical weaknesses
  1. Why do the authors use action-only, vision-only, and language-only experiments instead of leave-one-modality-out ablations to directly quantify the contribution of each modality?

  2. How much does the nearest-neighbor action interpolation mechanism in π0.5-RICL contribute to the overall performance? Do the main findings regarding action-only context still hold when this mechanism is disabled?

  3. In Section 5.2, the authors shuffle action sequences only among retrieved demonstrations from the same task, thereby preserving the task-specific action distribution. The small performance degradation may simply indicate that the model exploits task-level action statistics, rather than demonstrating that the correct observation–action correspondence is unimportant.

Questions for authors

Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.