- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates what makes effective in-context demonstrations for Vision-Language-Action (VLA) models. Through extensive experiments on pi0.5 and InternVLA-A1.5, the authors analyze the effects of different modalities, observation-action correspondence, and task difficulty. The experimental results suggest that action information plays a dominant role, accurate observation-action correspondence is less critical than expected, and more demonstrations do not necessarily improve performance.
Critical strengths
- The paper investigates an important and underexplored question about how VLAs utilize in-context demonstrations.
- The paper presents several interesting findings, particularly the dominance of action information and the limited importance of observation-action correspondence, although some experimental designs could be more rigorous.
- The paper examines multiple aspects of in-context demonstrations across different robotic manipulation benchmarks.
Critical weaknesses
My major concerns are listed as follows:
- The pi0.5-RICL model employs nearest-neighbor action interpolation during inference, which directly incorporates retrieved actions into the final predictions. This mechanism may influence the findings regarding action dominance and observation-action correspondence. Please provide more ablations.
- The shuffled actions are drawn from visually similar demonstrations of the same task, meaning that these actions may already be highly similar. Therefore, the small performance drop after shuffling may not necessarily indicate that observation-action correspondence is not important. Please analyze the differences between shuffled actions and consider experiments involving diverse actions.
- Although the paper considers both pi0.5 and InternVLA-A1.5, most key analysis are conducted only on pi0.5-RICL. Please conduct similar experiments on more VLA models to verify whether these findings generalize across different architectures. I think the current experiments are not sufficient.
- While the paper demonstrates that action information is the most important component of in-context demonstrations, it provides limited analysis of why this is the case. For example, does the model primarily benefit from task-specific action priors, action distributions, or other information?
- The paper has several writing and formatting problems. For example, an unresolved LaTeX reference ("Table ??") appears in line 300, and several figures have overly small text and crowded layouts, making them difficult to read. The overall presentation should be improved.
If the authors can address my concerns, I am willing to increase my rating.
Questions for authors
- The current experiments only compare full context with individual modalities (action-only, vision-only, and language-only). However, these results do not directly establish the contribution of each modality to the full context. Please include leave-one-modality-out experiments to better support the conclusions.