- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper proposes a VLA conditioned on key poses. It extracts key poses from expert demonstrations. Then, it conditions a VLA on these keypose sequences as a form of task description. This allows for one-shot transfer, since a main source of information in the expert trajectory is the macro sequence of actions (key poses) rather than low level controls.They run experiments both in simulation and on hardware showing the effectiveness of their approach.
Critical strengths
SE3 equivariance is a nice property to exploit for robotic systems. Most VLAs just use brute force learning from data, whereas taking advantage of equivariances could improve data efficiency. While SE3 equivariance has been studied before in general, applying it to VLA is still useful.
I like the idea of object-centric conditioning and key poses sequences, which is basically a goal-conditioned method. This is a nice alternative to just using text and images.
Experiments seem thorough.
Critical weaknesses
This paper is somehow very hard for me to follow. I think part of the problem is using EquiKey and EquiKey-VLA as names of the two different components, as these are very similar sounding. The writing also seems to lack an overall structure that motivates and combines the different ideas, it reads like a bunch of disparate ideas connected for this one paper, even though I dont think that is accurate.
I think the equivariance is oversold. Even if Equikey preserves it, the overall framework does not due to learned nature of perception and the VLA. Please correct me if I am missing something.
Questions for authors
- Even if Equikey preserves SE3 equivariance, the VLA does not, correct? If true, this somewhat breaks the logic of the paper because SE3 equivariance in terms of actions would not be preserved. Of course you can still use data augmentation, but this does not guarantee equivariance either.
- Even if some of the internal processing is equivariant, is the perception as well? Maybe run some ablations on the same scene with SE3 transformations, and see what the error of the corresponding poses are. Also check the action outputs to see if the actions have corresponding changes or not.
- How well does this handle clutter? Object-centric methods may be especially susceptible to clutter because distinguishing between objects and their boundaries becomes difficult
- Why do you have to retrain the key pose predictor on the expert trajectory? This would be much stronger if key pose estimator was transferable to new scenes.
- If I understand correctly, you finetuned the pi0.5 baseline on the same 40 tasks as used to train Equikey-vla. Correct?
Rebuttal Discussion
Dear Reviewers,
The authors have now submitted their rebuttal and responded to the concerns raised in the reviews. We would greatly appreciate it if you could take some time to review their responses and participate in the discussion, particularly regarding whether your concerns have been adequately addressed.
Thank you very much for your time and valuable contributions!