- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies neural geometry in vision-language-action models. The authors explore whether a concept can be retrieved from activations in the model, and the actual geometry of the concept in the activation space. To be specific, the authors study variables including end-effector position, target position, distance, contact, task phase, wrist orientation, and gripper, showing the interpretability and geometry of all these factors in vision-language-action models. The authors also study some of these observations on real-world data.
Critical strengths
- The setting of the paper is meaningful. Studying the geometry and interpretability of vision-language-action models can facilitate further understanding of their internal representations for environment understanding and action generation.
- The experiment is overall comprehensive, studying most of the factors related to the environment and manipulation phases. A lot of analysis methods are leveraged to verify different representations.
- Some of the observations are insightful. Also, the authors extend some of the observations to real-world settings and other models.
Critical weaknesses
- The real-world and cross-model validation is still somewhat limited. Although the authors explore this part in the paper in Sec. 4, such as π0.5 and MolmoAct2 on real-world DROID data, some of the stronger causal analyses in Sec. 3 cannot be reproduced in real-world settings or on other models.
- The paper contains a lot of interesting figures, covering a variety of analyses, but some of them are not clearly explained or self-contained in the main text. Some figures are related to important observations in the paper, but the explanations of the basic settings, how such experiments are conducted, and how the results support the claims are too brief in the main text. A reader needs to frequently check the appendix for more information. For example, Figs. 2 and 8. I think it would be better if more important information could be included in the main text, while moving some less important text to the appendix. The authors can better balance the content between the two parts.
Questions for authors
Could the authors clarify how much of the similarity across models may come from using the same labels, rather than from the models actually learning similar representations?
General response
We sincerely thank all three reviewers for their careful reading and constructive feedback. Below, we summarize the new experiments and the main points of discussion.
New Experiments
1. Shared labels
2. Effects of the wrist edit
3. Gripper edits run in a closed loop.
Clarifications of wording/scope
Other comments, mainly from reviewers cszC and cSiP, helped us state our results more precisely. We will make these clarifications in our camera-ready version:
Main findings the reviewers agree with
We thank the reviewers for their time. Their feedback has helped us make our claims more precise and bolster the experimental evidence.